Bug in parsing numbers in PPSmalltalkGrammar?

> I'm extending Coral to support Gnu Smalltalk syntax, and ran into
> something strange.
>
> I asked Lukas and he said that since others would be interested, I
> should repost the question here. So:
>
> Gnu Smalltalk's class variable declarations take the form of
> [<identifier> := <expression>.]* at the end of an object definition.
> To that end I wrote a #classVars production like so:
>
>   classVars
>       ^ (assignment , expression , $. asParser) star
>       "Possibly I need to support the lack of a dot on the last
> expression. Not relevant at the moment"
>
> This didn't work as expected. Looking further, I found this oddity:
>
>   p := PPSmalltalkParser new.
>   p number end parse: '1.' "=> #(#(nil #($1) nil) 1.0)"
>
> I would expect this to fail, because "1." isn't a number (but see below).
>
> Indeed, ripping out the parser of #number:
>
>   n := ($- asParser optional , #digit asParser plus , ($. asParser ,
> #digit asParser plus) optional) end.
>   n parse: '1.' "=> end of input expected at 1"
>
> So I experimented a bit, and it seems like#number is being overly
> eager in consuming that $. character, and Number >> #readFrom:'s
> reading '1.':
>
>   Number readFrom: '1.' readStream "=> 1.0"
>
>   p := PPSmalltalkParser new.
>   p number end parse: '1' "=> #(#(nil #($1) nil) 1)"
>
>   p := PPSmalltalkParser new.
>   p number end parse: '1.0' "=> #(#(nil #($1) #($. #($0))) 1.0)"
>
>   p := PPSmalltalkParser new.
>   p number end parse: '1. ' "=> end of input expected at 2"
>
> Or am I being silly?
>
> frank
>

> frank
>        just to be clear, we do not like the GNU syntax for Coral, else we would have done it before.
>        See my reply to your question in the coral mailing-lits
>
>        I think that
>                Eval []
>                Class []
>                Import []
>
>
> but I do not want nested definition nor class extension or class definition as in GNU. So if you want a GNU syntax
> then make sure that it is optional and that we can get coral without it.
>
> Stef
>
> On Aug 11, 2011, at 10:48 PM, Frank Shearar wrote:
>
>> I'm extending Coral to support Gnu Smalltalk syntax, and ran into
>> something strange.
>>
>> I asked Lukas and he said that since others would be interested, I
>> should repost the question here. So:
>>
>> Gnu Smalltalk's class variable declarations take the form of
>> [<identifier> := <expression>.]* at the end of an object definition.
>> To that end I wrote a #classVars production like so:
>>
>>   classVars
>>       ^ (assignment , expression , $. asParser) star
>>       "Possibly I need to support the lack of a dot on the last
>> expression. Not relevant at the moment"
>>
>> This didn't work as expected. Looking further, I found this oddity:
>>
>>   p := PPSmalltalkParser new.
>>   p number end parse: '1.' "=> #(#(nil #($1) nil) 1.0)"
>>
>> I would expect this to fail, because "1." isn't a number (but see below).
>>
>> Indeed, ripping out the parser of #number:
>>
>>   n := ($- asParser optional , #digit asParser plus , ($. asParser ,
>> #digit asParser plus) optional) end.
>>   n parse: '1.' "=> end of input expected at 1"
>>
>> So I experimented a bit, and it seems like#number is being overly
>> eager in consuming that $. character, and Number >> #readFrom:'s
>> reading '1.':
>>
>>   Number readFrom: '1.' readStream "=> 1.0"
>>
>>   p := PPSmalltalkParser new.
>>   p number end parse: '1' "=> #(#(nil #($1) nil) 1)"
>>
>>   p := PPSmalltalkParser new.
>>   p number end parse: '1.0' "=> #(#(nil #($1) #($. #($0))) 1.0)"
>>
>>   p := PPSmalltalkParser new.
>>   p number end parse: '1. ' "=> end of input expected at 2"
>>
>> Or am I being silly?
>>
>> frank
>>
>
>

> Hi Frank,
>
>> I asked Lukas and he said that since others would be interested, I
>> should repost the question here. So:
>
> Thanks!
>
>> Gnu Smalltalk's class variable declarations take the form of
>> [<identifier> := <expression>.]* at the end of an object definition.
>> To that end I wrote a #classVars production like so:
>>
>>   classVars
>>       ^ (assignment , expression , $. asParser) star
>>       "Possibly I need to support the lack of a dot on the last
>> expression. Not relevant at the moment"
>
> This seems to be indeed the same problem as the simpler case below ...
>
>> This didn't work as expected. Looking further, I found this oddity:
>>
>>   p := PPSmalltalkParser new.
>>   p number end parse: '1.' "=> #(#(nil #($1) nil) 1.0)"
>>
>> I would expect this to fail, because "1." isn't a number (but see below).
>
> This fails in Pharo 1.3. In fact, in Pharo the following statements
> return 'false':
>
>  | s |
>  s := '1.' readStream.
>  Number readFrom: s.
>  s atEnd
>
> That is, #readFrom: really only consumes what it can. I don't know
> what Smalltalk you use, but I guess your image consumes the $. and
> returns 'true'. Not sure if this can be called a bug, but it is
> definitely a bit strange. The definition of a valid number literal is
> quite different among different Smalltalk dialects, thus I decided to
> rely on the built-in implementation of Number class>>#readFrom:.

>
>
> On Friday, 12 August 2011, Frank Shearar <[hidden email]> wrote:
>> On 11 August 2011 22:05, Lukas Renggli <[hidden email]> wrote:
>>> Hi Frank,
>>>
>>>> I asked Lukas and he said that since others would be interested, I
>>>> should repost the question here. So:
>>>
>>> Thanks!
>>>
>>>> Gnu Smalltalk's class variable declarations take the form of
>>>> [<identifier> := <expression>.]* at the end of an object definition.
>>>> To that end I wrote a #classVars production like so:
>>>>
>>>>   classVars
>>>>       ^ (assignment , expression , $. asParser) star
>>>>       "Possibly I need to support the lack of a dot on the last
>>>> expression. Not relevant at the moment"
>>>
>>> This seems to be indeed the same problem as the simpler case below ...
>>>
>>>> This didn't work as expected. Looking further, I found this oddity:
>>>>
>>>>   p := PPSmalltalkParser new.
>>>>   p number end parse: '1.' "=> #(#(nil #($1) nil) 1.0)"
>>>>
>>>> I would expect this to fail, because "1." isn't a number (but see
>>>> below).
>>>
>>> This fails in Pharo 1.3. In fact, in Pharo the following statements
>>> return 'false':
>>>
>>>  | s |
>>>  s := '1.' readStream.
>>>  Number readFrom: s.
>>>  s atEnd
>>>
>>> That is, #readFrom: really only consumes what it can. I don't know
>>> what Smalltalk you use, but I guess your image consumes the $. and
>>> returns 'true'. Not sure if this can be called a bug, but it is
>>> definitely a bit strange. The definition of a valid number literal is
>>> quite different among different Smalltalk dialects, thus I decided to
>>> rely on the built-in implementation of Number class>>#readFrom:.
>>
>> I'm using Squeak (trunk), and there we have
>>
>>    Number readFrom: '1.' readStream "=> 1.0"
>>    Number readFrom: '1' readStream "=> 1"
>>
>> So what might be happening is that the stream contains '1.' but
>> Pharo's version doesn't care, and returns an Integer?
>>
>> Ah. Ahem. Indeed, that's what's happening. Number gets '1.', and in
>> Pharo that means 1 while in Squeak it means 1.0.
>>
>> I find it a bit strange that PPSmalltalkGrammar sends off '1.', but I
>> also think that Squeak's doing the wrong thing.
>
> PPSmalltalkGrammar doesn't send off anything, it just delegates to the
> number reader by passing over its stream. In return PPSmalltalkGrammar
> expects a number and that the number reader leaves the stream at the end of
> the number. This is the same than the refactoring browser parser and the
> standard compiler do in Pharo (I believe).
>
> Lukas
>
> --
> Lukas Renggli
> www.lukas-renggli.ch
>

> Again, while we're at it…
>
> The definition of PPSmalltalkGrammar>>#binary looks wrong. It would
> not parse -- (double dash) as a binary selector. Camillo and I added a
> few tests for this, and a quick fix, but the way that rule was written
> suggests it was voluntary to forbid dashes after the first character.
>
> I guess it should be scrutinized a bit more, especially since I think
> various dialects accept different lengths and characters for binary
> selectors…

> On 12 August 2011 01:52, Damien Pollet <[hidden email]> wrote:
>> Again, while we're at it…
>>
>> The definition of PPSmalltalkGrammar>>#binary looks wrong. It would
>> not parse -- (double dash) as a binary selector. Camillo and I added a
>> few tests for this, and a quick fix, but the way that rule was written
>> suggests it was voluntary to forbid dashes after the first character.
>>
>> I guess it should be scrutinized a bit more, especially since I think
>> various dialects accept different lengths and characters for binary
>> selectors…
>
> I saw your change and committed some more fixes.
>
> Up until a few years ago no Smalltalk allowed a dash in the last place
> of a binary selector. I fixed this a while ago in the refactoring
> browser, but didn't adapt the PPSmalltalkGrammar.
>
> Lukas
>
> --
> Lukas Renggli
> www.lukas-renggli.ch
>
>

> Hi Stef,
>
> I'm reporting a bug in PetitSmalltalk, so why don't we concentrate on that?
>
> frank
>
> On 11 August 2011 21:56, Stéphane Ducasse <[hidden email]> wrote:
>> frank
>>        just to be clear, we do not like the GNU syntax for Coral, else we would have done it before.
>>        See my reply to your question in the coral mailing-lits
>>
>>        I think that
>>                Eval []
>>                Class []
>>                Import []
>>
>>
>> but I do not want nested definition nor class extension or class definition as in GNU. So if you want a GNU syntax
>> then make sure that it is optional and that we can get coral without it.
>>
>> Stef
>>
>> On Aug 11, 2011, at 10:48 PM, Frank Shearar wrote:
>>
>>> I'm extending Coral to support Gnu Smalltalk syntax, and ran into
>>> something strange.
>>>
>>> I asked Lukas and he said that since others would be interested, I
>>> should repost the question here. So:
>>>
>>> Gnu Smalltalk's class variable declarations take the form of
>>> [<identifier> := <expression>.]* at the end of an object definition.
>>> To that end I wrote a #classVars production like so:
>>>
>>>   classVars
>>>       ^ (assignment , expression , $. asParser) star
>>>       "Possibly I need to support the lack of a dot on the last
>>> expression. Not relevant at the moment"
>>>
>>> This didn't work as expected. Looking further, I found this oddity:
>>>
>>>   p := PPSmalltalkParser new.
>>>   p number end parse: '1.' "=> #(#(nil #($1) nil) 1.0)"
>>>
>>> I would expect this to fail, because "1." isn't a number (but see below).
>>>
>>> Indeed, ripping out the parser of #number:
>>>
>>>   n := ($- asParser optional , #digit asParser plus , ($. asParser ,
>>> #digit asParser plus) optional) end.
>>>   n parse: '1.' "=> end of input expected at 1"
>>>
>>> So I experimented a bit, and it seems like#number is being overly
>>> eager in consuming that $. character, and Number >> #readFrom:'s
>>> reading '1.':
>>>
>>>   Number readFrom: '1.' readStream "=> 1.0"
>>>
>>>   p := PPSmalltalkParser new.
>>>   p number end parse: '1' "=> #(#(nil #($1) nil) 1)"
>>>
>>>   p := PPSmalltalkParser new.
>>>   p number end parse: '1.0' "=> #(#(nil #($1) #($. #($0))) 1.0)"
>>>
>>>   p := PPSmalltalkParser new.
>>>   p number end parse: '1. ' "=> end of input expected at 2"
>>>
>>> Or am I being silly?
>>>
>>> frank
>>>
>>
>>
>

>
>
> On Friday, 12 August 2011, Frank Shearar <[hidden email]> wrote:
>> On 11 August 2011 22:05, Lukas Renggli <[hidden email]> wrote:
>>> Hi Frank,
>>>
>>>> I asked Lukas and he said that since others would be interested, I
>>>> should repost the question here. So:
>>>
>>> Thanks!
>>>
>>>> Gnu Smalltalk's class variable declarations take the form of
>>>> [<identifier> := <expression>.]* at the end of an object definition.
>>>> To that end I wrote a #classVars production like so:
>>>>
>>>>   classVars
>>>>       ^ (assignment , expression , $. asParser) star
>>>>       "Possibly I need to support the lack of a dot on the last
>>>> expression. Not relevant at the moment"
>>>
>>> This seems to be indeed the same problem as the simpler case below ...
>>>
>>>> This didn't work as expected. Looking further, I found this oddity:
>>>>
>>>>   p := PPSmalltalkParser new.
>>>>   p number end parse: '1.' "=> #(#(nil #($1) nil) 1.0)"
>>>>
>>>> I would expect this to fail, because "1." isn't a number (but see
>>>> below).
>>>
>>> This fails in Pharo 1.3. In fact, in Pharo the following statements
>>> return 'false':
>>>
>>>  | s |
>>>  s := '1.' readStream.
>>>  Number readFrom: s.
>>>  s atEnd
>>>
>>> That is, #readFrom: really only consumes what it can. I don't know
>>> what Smalltalk you use, but I guess your image consumes the $. and
>>> returns 'true'. Not sure if this can be called a bug, but it is
>>> definitely a bit strange. The definition of a valid number literal is
>>> quite different among different Smalltalk dialects, thus I decided to
>>> rely on the built-in implementation of Number class>>#readFrom:.
>>
>> I'm using Squeak (trunk), and there we have
>>
>>    Number readFrom: '1.' readStream "=> 1.0"
>>    Number readFrom: '1' readStream "=> 1"
>>
>> So what might be happening is that the stream contains '1.' but
>> Pharo's version doesn't care, and returns an Integer?
>>
>> Ah. Ahem. Indeed, that's what's happening. Number gets '1.', and in
>> Pharo that means 1 while in Squeak it means 1.0.
>>
>> I find it a bit strange that PPSmalltalkGrammar sends off '1.', but I
>> also think that Squeak's doing the wrong thing.
>
> PPSmalltalkGrammar doesn't send off anything, it just delegates to the
> number reader by passing over its stream. In return PPSmalltalkGrammar
> expects a number and that the number reader leaves the stream at the end of
> the number. This is the same than the refactoring browser parser and the
> standard compiler do in Pharo (I believe).

> On 11 August 2011 23:30, Lukas Renggli <[hidden email]> wrote:
>>
>>
>> On Friday, 12 August 2011, Frank Shearar <[hidden email]> wrote:
>>> On 11 August 2011 22:05, Lukas Renggli <[hidden email]> wrote:
>>>> Hi Frank,
>>>>
>>>>> I asked Lukas and he said that since others would be interested, I
>>>>> should repost the question here. So:
>>>>
>>>> Thanks!
>>>>
>>>>> Gnu Smalltalk's class variable declarations take the form of
>>>>> [<identifier> := <expression>.]* at the end of an object definition.
>>>>> To that end I wrote a #classVars production like so:
>>>>>
>>>>>   classVars
>>>>>       ^ (assignment , expression , $. asParser) star
>>>>>       "Possibly I need to support the lack of a dot on the last
>>>>> expression. Not relevant at the moment"
>>>>
>>>> This seems to be indeed the same problem as the simpler case below ...
>>>>
>>>>> This didn't work as expected. Looking further, I found this oddity:
>>>>>
>>>>>   p := PPSmalltalkParser new.
>>>>>   p number end parse: '1.' "=> #(#(nil #($1) nil) 1.0)"
>>>>>
>>>>> I would expect this to fail, because "1." isn't a number (but see
>>>>> below).
>>>>
>>>> This fails in Pharo 1.3. In fact, in Pharo the following statements
>>>> return 'false':
>>>>
>>>>  | s |
>>>>  s := '1.' readStream.
>>>>  Number readFrom: s.
>>>>  s atEnd
>>>>
>>>> That is, #readFrom: really only consumes what it can. I don't know
>>>> what Smalltalk you use, but I guess your image consumes the $. and
>>>> returns 'true'. Not sure if this can be called a bug, but it is
>>>> definitely a bit strange. The definition of a valid number literal is
>>>> quite different among different Smalltalk dialects, thus I decided to
>>>> rely on the built-in implementation of Number class>>#readFrom:.
>>>
>>> I'm using Squeak (trunk), and there we have
>>>
>>>    Number readFrom: '1.' readStream "=> 1.0"
>>>    Number readFrom: '1' readStream "=> 1"
>>>
>>> So what might be happening is that the stream contains '1.' but
>>> Pharo's version doesn't care, and returns an Integer?
>>>
>>> Ah. Ahem. Indeed, that's what's happening. Number gets '1.', and in
>>> Pharo that means 1 while in Squeak it means 1.0.
>>>
>>> I find it a bit strange that PPSmalltalkGrammar sends off '1.', but I
>>> also think that Squeak's doing the wrong thing.
>>
>> PPSmalltalkGrammar doesn't send off anything, it just delegates to the
>> number reader by passing over its stream. In return PPSmalltalkGrammar
>> expects a number and that the number reader leaves the stream at the end of
>> the number. This is the same than the refactoring browser parser and the
>> standard compiler do in Pharo (I believe).
>
> OK. So when parsing '1.' readStream. PPSmalltalkGrammar will send off
> (a stream containing) '1.' to something (in this case Number class >>
> #readFrom:), which will eat the Number. Then PPSmalltalkGrammar picks
> up the stream with whatever's still left unparsed.
>
> In Pharo's case that means that 1 is returned, leaving '.' on the
> stream; in Squeak's case Number eats the '1.', returns 1.0 and then
> leaves the stream empty.
>
> Is that an accurate restatement of the issue?
>
> frank
>
>> Lukas
>>
>> --
>> Lukas Renggli
>> www.lukas-renggli.ch
>>
>
>

> Lukas
>
> On 12 August 2011 12:50, Frank Shearar <[hidden email]> wrote:
>> On 11 August 2011 23:30, Lukas Renggli <[hidden email]> wrote:
>>>
>>>
>>> On Friday, 12 August 2011, Frank Shearar <[hidden email]> wrote:
>>>> On 11 August 2011 22:05, Lukas Renggli <[hidden email]> wrote:
>>>>> Hi Frank,
>>>>>
>>>>>> I asked Lukas and he said that since others would be interested, I
>>>>>> should repost the question here. So:
>>>>>
>>>>> Thanks!
>>>>>
>>>>>> Gnu Smalltalk's class variable declarations take the form of
>>>>>> [<identifier> := <expression>.]* at the end of an object definition.
>>>>>> To that end I wrote a #classVars production like so:
>>>>>>
>>>>>>   classVars
>>>>>>       ^ (assignment , expression , $. asParser) star
>>>>>>       "Possibly I need to support the lack of a dot on the last
>>>>>> expression. Not relevant at the moment"
>>>>>
>>>>> This seems to be indeed the same problem as the simpler case below ...
>>>>>
>>>>>> This didn't work as expected. Looking further, I found this oddity:
>>>>>>
>>>>>>   p := PPSmalltalkParser new.
>>>>>>   p number end parse: '1.' "=> #(#(nil #($1) nil) 1.0)"
>>>>>>
>>>>>> I would expect this to fail, because "1." isn't a number (but see
>>>>>> below).
>>>>>
>>>>> This fails in Pharo 1.3. In fact, in Pharo the following statements
>>>>> return 'false':
>>>>>
>>>>>  | s |
>>>>>  s := '1.' readStream.
>>>>>  Number readFrom: s.
>>>>>  s atEnd
>>>>>
>>>>> That is, #readFrom: really only consumes what it can. I don't know
>>>>> what Smalltalk you use, but I guess your image consumes the $. and
>>>>> returns 'true'. Not sure if this can be called a bug, but it is
>>>>> definitely a bit strange. The definition of a valid number literal is
>>>>> quite different among different Smalltalk dialects, thus I decided to
>>>>> rely on the built-in implementation of Number class>>#readFrom:.
>>>>
>>>> I'm using Squeak (trunk), and there we have
>>>>
>>>>    Number readFrom: '1.' readStream "=> 1.0"
>>>>    Number readFrom: '1' readStream "=> 1"
>>>>
>>>> So what might be happening is that the stream contains '1.' but
>>>> Pharo's version doesn't care, and returns an Integer?
>>>>
>>>> Ah. Ahem. Indeed, that's what's happening. Number gets '1.', and in
>>>> Pharo that means 1 while in Squeak it means 1.0.
>>>>
>>>> I find it a bit strange that PPSmalltalkGrammar sends off '1.', but I
>>>> also think that Squeak's doing the wrong thing.
>>>
>>> PPSmalltalkGrammar doesn't send off anything, it just delegates to the
>>> number reader by passing over its stream. In return PPSmalltalkGrammar
>>> expects a number and that the number reader leaves the stream at the end of
>>> the number. This is the same than the refactoring browser parser and the
>>> standard compiler do in Pharo (I believe).
>>
>> OK. So when parsing '1.' readStream. PPSmalltalkGrammar will send off
>> (a stream containing) '1.' to something (in this case Number class >>
>> #readFrom:), which will eat the Number. Then PPSmalltalkGrammar picks
>> up the stream with whatever's still left unparsed.
>>
>> In Pharo's case that means that 1 is returned, leaving '.' on the
>> stream; in Squeak's case Number eats the '1.', returns 1.0 and then
>> leaves the stream empty.
>>
>> Is that an accurate restatement of the issue?
>>
>> frank
>>
>>> Lukas
>>>
>>> --
>>> Lukas Renggli
>>> www.lukas-renggli.ch
>>>
>>
>>
>
>
>
> --
> Lukas Renggli
> www.lukas-renggli.ch
>
>