Fraction and ScaledDecimal should be not be serialized as Float in STON format

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Fraction and ScaledDecimal should be not be serialized as Float in STON format

Julien
Hello,

I realised that in the current implementation, when a number is serialised by STON it is either as an integer literal or as a float literal.

There is a risk to loose precision, especially if what you serialise is a ScaledDecimal or a fraction.

I propose a simple change to fix this:

1. Add STONWriter>>#isInJsonMode method which returns true if the STONWriter wants to write JSON

2. Add:

Fraction>>#stonOn: stonWriter
    stonWriter isInJsonMode
        ifTrue: [ ^ super stonOn: stonWriter ].
    
    stonWriter writeObject: self streamMap: [ :dictionary |
        dictionary at: #numerator put: numerator.
        dictionary at: #denominator put: denominator ]

3. Add:

ScaledDecimal>>#stonOn: stonWriter
    stonWriter isInJsonMode
        ifTrue: [ ^ super stonOn: stonWriter ].
    
    stonWriter writeObject: self streamMap: [ :dictionary |
        dictionary at: #numerator put: numerator.
        dictionary at: #denominator put: denominator.
        dictionary at: #scale put: scale ]


This change has a main drawback, storing fractions and scaled decimal will result in a huge overhead.

Maybe for ScaledDecimal we can use Pharo’s literal (e.g. 1.12s2).

For fractions I don’t know what literal could be used?

Or maybe the overhead is fine?

Cheers,

Julien

---
Julien Delplanque
Doctorant à l’Université de Lille
Bâtiment B 40, Avenue Halley 59650 Villeneuve d'Ascq
Numéro de téléphone: +333 59 35 86 40

Reply | Threaded
Open this post in threaded view
|

Re: Fraction and ScaledDecimal should be not be serialized as Float in STON format

Sven Van Caekenberghe-2
Hi Julien,

Good and interesting point.

Your summary is correct: STON, inheriting from JSON so to speak, only knowns about integer and float numbers. All other Smalltalk numbers get converted, which results in a loss of type and precision. That might not be a perfect situation, but nobody complained so far.

If we would change it, efficiency has to be taken into account: speed/memory efficiency as well as human typing efficiency/complexity. Compatibility, especially between different Smalltalk and other object languages, is also an aspect.

Apart from the current approach, the full STON approach could be, for example for 1/3 and 1.5s2

 Fraction { #numerator:1, #denominator:3 }
 ScaledDecimal { #numerator:3, #denominator:2, #scale:2}

For some types, STON uses shorter notations, either using positional arguments

 Point [ 1, 2 ]
 Fraction [ 1, 3 ]
 ScaledFraction [ 3, 2, 2 ]

Or a string argument, when there is a clear external representation

 DateAndTime [ '2018-09-18T16:39:10.325129+02:00' ]
 Fraction [ '1/3' ]
 ScaledFraction [ '3/2s2' ]

Of course, we could also extend the basic number parser and allow the native (stored) notation

 1/3
 3/2s2

Note how that last one should use the more exact #storeOn: notation, not the #printOn:

I'll have to think about this a bit more, as I can really not choose right now.

Sven

> On 18 Sep 2018, at 10:34, Julien <[hidden email]> wrote:
>
> Hello,
>
> I realised that in the current implementation, when a number is serialised by STON it is either as an integer literal or as a float literal.
>
> There is a risk to loose precision, especially if what you serialise is a ScaledDecimal or a fraction.
>
> I propose a simple change to fix this:
>
> 1. Add STONWriter>>#isInJsonMode method which returns true if the STONWriter wants to write JSON
>
> 2. Add:
>
> Fraction>>#stonOn: stonWriter
>     stonWriter isInJsonMode
>         ifTrue: [ ^ super stonOn: stonWriter ].
>    
>     stonWriter writeObject: self streamMap: [ :dictionary |
>         dictionary at: #numerator put: numerator.
>         dictionary at: #denominator put: denominator ]
>
> 3. Add:
>
> ScaledDecimal>>#stonOn: stonWriter
>     stonWriter isInJsonMode
>         ifTrue: [ ^ super stonOn: stonWriter ].
>    
>     stonWriter writeObject: self streamMap: [ :dictionary |
>         dictionary at: #numerator put: numerator.
>         dictionary at: #denominator put: denominator.
>         dictionary at: #scale put: scale ]
>
>
> This change has a main drawback, storing fractions and scaled decimal will result in a huge overhead.
>
> Maybe for ScaledDecimal we can use Pharo’s literal (e.g. 1.12s2).
>
> For fractions I don’t know what literal could be used?
>
> Or maybe the overhead is fine?
>
> Cheers,
>
> Julien
>
> ---
> Julien Delplanque
> Doctorant à l’Université de Lille
> http://juliendelplanque.be/phd.html
> Equipe Rmod, Inria
> Bâtiment B 40, Avenue Halley 59650 Villeneuve d'Ascq
> Numéro de téléphone: +333 59 35 86 40
>


Reply | Threaded
Open this post in threaded view
|

Re: Fraction and ScaledDecimal should be not be serialized as Float in STON format

Julien
Hello,

Le 18 sept. 2018 à 16:45, Sven Van Caekenberghe <[hidden email]> a écrit :

Hi Julien,

Good and interesting point.

Your summary is correct: STON, inheriting from JSON so to speak, only knowns about integer and float numbers. All other Smalltalk numbers get converted, which results in a loss of type and precision. That might not be a perfect situation, but nobody complained so far.

That’s what I guessed to be the reason of this feature. :-)


If we would change it, efficiency has to be taken into account: speed/memory efficiency as well as human typing efficiency/complexity. Compatibility, especially between different Smalltalk and other object languages, is also an aspect.

Apart from the current approach, the full STON approach could be, for example for 1/3 and 1.5s2

Fraction { #numerator:1, #denominator:3 }
ScaledDecimal { #numerator:3, #denominator:2, #scale:2}

For some types, STON uses shorter notations, either using positional arguments 

Point [ 1, 2 ]
Fraction [ 1, 3 ]
ScaledFraction [ 3, 2, 2 ]

Or a string argument, when there is a clear external representation

DateAndTime [ '2018-09-18T16:39:10.325129+02:00' ]
Fraction [ '1/3' ]
ScaledFraction [ '3/2s2' ]

Of course, we could also extend the basic number parser and allow the native (stored) notation

1/3
3/2s2

Note how that last one should use the more exact #storeOn: notation, not the #printOn: 

I'll have to think about this a bit more, as I can really not choose right now.

Sven

Tell me if you need help to implement this feature, I’ll be happy to contribute! :-)

Cheers,

Julien

---
Julien Delplanque
Doctorant à l’Université de Lille
Bâtiment B 40, Avenue Halley 59650 Villeneuve d'Ascq
Numéro de téléphone: +333 59 35 86 40
Reply | Threaded
Open this post in threaded view
|

Re: Fraction and ScaledDecimal should be not be serialized as Float in STON format

Sven Van Caekenberghe-2
In reply to this post by Sven Van Caekenberghe-2


> On 18 Sep 2018, at 16:45, Sven Van Caekenberghe <[hidden email]> wrote:
>
> Of course, we could also extend the basic number parser and allow the native (stored) notation
>
> 1/3
> 3/2s2
>
> Note how that last one should use the more exact #storeOn: notation, not the #printOn:
>
> I'll have to think about this a bit more, as I can really not choose right now.

I am still not sure if we should do this or not, but here is a POC implementation of the above.

It implements the following syntax extension (see the class comment of STON for the rest).

number
  int
  int denominator
  int denominator scale
  int frac
  int exp
  int frac exp

denominator
  / digits

scale
  s digits


STONWriter>>#writeFraction: fraction
        jsonMode
                ifTrue: [ self writeFloat: fraction asFloat ]
                ifFalse: [
                        writeStream
                                print: fraction numerator;
                                nextPut: $/;
                                print: fraction denominator ]

STONWriter>>#writeScaledDecimal: scaledDecimal
        jsonMode
                ifTrue: [ self writeFloat: scaledDecimal asFloat ]
                ifFalse: [
                        writeStream
                                print: scaledDecimal numerator;
                                nextPut: $/;
                                print: scaledDecimal denominator;
                                nextPut: $s;
                                print: scaledDecimal scale ]

Fraction>>#stonOn: stonWriter
        stonWriter writeFraction: self

ScaledDecimal>>#stonOn: stonWriter
        stonWriter writeScaledDecimal: self

STONReader>>parseNumber
        | negated number |
        negated := readStream peekFor: $-.
        number := self parseNumberInteger.
        (readStream peekFor: $/)
                ifTrue: [
                        number := Fraction numerator: number denominator: self parseNumberInteger.
                        (readStream peekFor: $s)
                                ifTrue: [ number := ScaledDecimal newFromNumber: number scale: self parseNumberInteger ] ]
                ifFalse: [
                        (readStream peekFor: $.)
                                ifTrue: [ number := number + self parseNumberFraction ].
                        ((readStream peekFor: $e) or: [ readStream peekFor: $E ])
                                ifTrue: [ number := number * self parseNumberExponent ] ].
        negated
                ifTrue: [ number := number negated ].
        self consumeWhitespace.
        ^ number

Sven


Reply | Threaded
Open this post in threaded view
|

Re: Fraction and ScaledDecimal should be not be serialized as Float in STON format

Sven Van Caekenberghe-2
I decided to add it for real:

https://github.com/svenvc/ston/commit/9c83e3cc2f00cab83e57f2e10a139d6ecef3cb30

add support for Fraction and ScaledDecimal as direct numeric literals (in STON mode, not in JSON mode) with units tests

> On 25 Sep 2018, at 14:13, Sven Van Caekenberghe <[hidden email]> wrote:
>
>
>
>> On 18 Sep 2018, at 16:45, Sven Van Caekenberghe <[hidden email]> wrote:
>>
>> Of course, we could also extend the basic number parser and allow the native (stored) notation
>>
>> 1/3
>> 3/2s2
>>
>> Note how that last one should use the more exact #storeOn: notation, not the #printOn:
>>
>> I'll have to think about this a bit more, as I can really not choose right now.
>
> I am still not sure if we should do this or not, but here is a POC implementation of the above.
>
> It implements the following syntax extension (see the class comment of STON for the rest).
>
> number
>  int
>  int denominator
>  int denominator scale
>  int frac
>  int exp
>  int frac exp
>
> denominator
>  / digits
>
> scale
>  s digits
>
>
> STONWriter>>#writeFraction: fraction
> jsonMode
> ifTrue: [ self writeFloat: fraction asFloat ]
> ifFalse: [
> writeStream
> print: fraction numerator;
> nextPut: $/;
> print: fraction denominator ]
>
> STONWriter>>#writeScaledDecimal: scaledDecimal
> jsonMode
> ifTrue: [ self writeFloat: scaledDecimal asFloat ]
> ifFalse: [
> writeStream
> print: scaledDecimal numerator;
> nextPut: $/;
> print: scaledDecimal denominator;
> nextPut: $s;
> print: scaledDecimal scale ]
>
> Fraction>>#stonOn: stonWriter
> stonWriter writeFraction: self
>
> ScaledDecimal>>#stonOn: stonWriter
> stonWriter writeScaledDecimal: self
>
> STONReader>>parseNumber
> | negated number |
> negated := readStream peekFor: $-.
> number := self parseNumberInteger.
> (readStream peekFor: $/)
> ifTrue: [
> number := Fraction numerator: number denominator: self parseNumberInteger.
> (readStream peekFor: $s)
> ifTrue: [ number := ScaledDecimal newFromNumber: number scale: self parseNumberInteger ] ]
> ifFalse: [
> (readStream peekFor: $.)
> ifTrue: [ number := number + self parseNumberFraction ].
> ((readStream peekFor: $e) or: [ readStream peekFor: $E ])
> ifTrue: [ number := number * self parseNumberExponent ] ].
> negated
> ifTrue: [ number := number negated ].
> self consumeWhitespace.
> ^ number
>
> Sven
>


Reply | Threaded
Open this post in threaded view
|

Re: Fraction and ScaledDecimal should be not be serialized as Float in STON format

Stephane Ducasse-3
This is cool because indeed scaledDecimal are nice objects :)

On Tue, Oct 9, 2018 at 9:23 PM Sven Van Caekenberghe <[hidden email]> wrote:

>
> I decided to add it for real:
>
> https://github.com/svenvc/ston/commit/9c83e3cc2f00cab83e57f2e10a139d6ecef3cb30
>
> add support for Fraction and ScaledDecimal as direct numeric literals (in STON mode, not in JSON mode) with units tests
>
> > On 25 Sep 2018, at 14:13, Sven Van Caekenberghe <[hidden email]> wrote:
> >
> >
> >
> >> On 18 Sep 2018, at 16:45, Sven Van Caekenberghe <[hidden email]> wrote:
> >>
> >> Of course, we could also extend the basic number parser and allow the native (stored) notation
> >>
> >> 1/3
> >> 3/2s2
> >>
> >> Note how that last one should use the more exact #storeOn: notation, not the #printOn:
> >>
> >> I'll have to think about this a bit more, as I can really not choose right now.
> >
> > I am still not sure if we should do this or not, but here is a POC implementation of the above.
> >
> > It implements the following syntax extension (see the class comment of STON for the rest).
> >
> > number
> >  int
> >  int denominator
> >  int denominator scale
> >  int frac
> >  int exp
> >  int frac exp
> >
> > denominator
> >  / digits
> >
> > scale
> >  s digits
> >
> >
> > STONWriter>>#writeFraction: fraction
> >       jsonMode
> >               ifTrue: [ self writeFloat: fraction asFloat ]
> >               ifFalse: [
> >                       writeStream
> >                               print: fraction numerator;
> >                               nextPut: $/;
> >                               print: fraction denominator ]
> >
> > STONWriter>>#writeScaledDecimal: scaledDecimal
> >       jsonMode
> >               ifTrue: [ self writeFloat: scaledDecimal asFloat ]
> >               ifFalse: [
> >                       writeStream
> >                               print: scaledDecimal numerator;
> >                               nextPut: $/;
> >                               print: scaledDecimal denominator;
> >                               nextPut: $s;
> >                               print: scaledDecimal scale ]
> >
> > Fraction>>#stonOn: stonWriter
> >       stonWriter writeFraction: self
> >
> > ScaledDecimal>>#stonOn: stonWriter
> >       stonWriter writeScaledDecimal: self
> >
> > STONReader>>parseNumber
> >       | negated number |
> >       negated := readStream peekFor: $-.
> >       number := self parseNumberInteger.
> >       (readStream peekFor: $/)
> >               ifTrue: [
> >                       number := Fraction numerator: number denominator: self parseNumberInteger.
> >                       (readStream peekFor: $s)
> >                               ifTrue: [ number := ScaledDecimal newFromNumber: number scale: self parseNumberInteger ] ]
> >               ifFalse: [
> >                       (readStream peekFor: $.)
> >                               ifTrue: [ number := number + self parseNumberFraction ].
> >                       ((readStream peekFor: $e) or: [ readStream peekFor: $E ])
> >                               ifTrue: [ number := number * self parseNumberExponent ] ].
> >       negated
> >               ifTrue: [ number := number negated ].
> >       self consumeWhitespace.
> >       ^ number
> >
> > Sven
> >
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Fraction and ScaledDecimal should be not be serialized as Float in STON format

Julien
In reply to this post by Sven Van Caekenberghe-2
Sorry, I forgot to answer. This change is indeed really nice.

Thank you for it!

Cheers,

Julien

---
Julien Delplanque
Doctorant à l’Université de Lille
Bâtiment B 40, Avenue Halley 59650 Villeneuve d'Ascq
Numéro de téléphone: +333 59 35 86 40

Le 9 oct. 2018 à 21:22, Sven Van Caekenberghe <[hidden email]> a écrit :

I decided to add it for real:

https://github.com/svenvc/ston/commit/9c83e3cc2f00cab83e57f2e10a139d6ecef3cb30

add support for Fraction and ScaledDecimal as direct numeric literals (in STON mode, not in JSON mode) with units tests

On 25 Sep 2018, at 14:13, Sven Van Caekenberghe <[hidden email]> wrote:



On 18 Sep 2018, at 16:45, Sven Van Caekenberghe <[hidden email]> wrote:

Of course, we could also extend the basic number parser and allow the native (stored) notation

1/3
3/2s2

Note how that last one should use the more exact #storeOn: notation, not the #printOn:

I'll have to think about this a bit more, as I can really not choose right now.

I am still not sure if we should do this or not, but here is a POC implementation of the above.

It implements the following syntax extension (see the class comment of STON for the rest).

number
int
int denominator
int denominator scale
int frac
int exp
int frac exp

denominator
/ digits

scale
s digits


STONWriter>>#writeFraction: fraction
jsonMode
ifTrue: [ self writeFloat: fraction asFloat ]
ifFalse: [
writeStream
print: fraction numerator;
nextPut: $/;
print: fraction denominator ]

STONWriter>>#writeScaledDecimal: scaledDecimal
jsonMode
ifTrue: [ self writeFloat: scaledDecimal asFloat ]
ifFalse: [
writeStream
print: scaledDecimal numerator;
nextPut: $/;
print: scaledDecimal denominator;
nextPut: $s;
print: scaledDecimal scale ]

Fraction>>#stonOn: stonWriter
stonWriter writeFraction: self

ScaledDecimal>>#stonOn: stonWriter
stonWriter writeScaledDecimal: self

STONReader>>parseNumber
| negated number |
negated := readStream peekFor: $-.
number := self parseNumberInteger.
(readStream peekFor: $/)
ifTrue: [
number := Fraction numerator: number denominator: self parseNumberInteger.
(readStream peekFor: $s)
ifTrue: [ number := ScaledDecimal newFromNumber: number scale: self parseNumberInteger ] ]
ifFalse: [
(readStream peekFor: $.)
ifTrue: [ number := number + self parseNumberFraction ].
((readStream peekFor: $e) or: [ readStream peekFor: $E ])
ifTrue: [ number := number * self parseNumberExponent ] ].
negated
ifTrue: [ number := number negated ].
self consumeWhitespace.
^ number

Sven