Inconsistencies in NumberPrintPolicy

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Inconsistencies in NumberPrintPolicy

Karyn Hurley
I am seeing inconsistencies in the NumberPrintPolicy and would like some feedback.    

If I use this:
l := 10.0.
s := (NumberPrintPolicy print: l using: '#0.0##') asString.  

the value in s, when inspected is '10.0'.   Perfect.

But if I use this number:
l := 100000.0.
s := (NumberPrintPolicy print: l using: '#0.0##') asString.

the value is s is now '99999.992'.  

I've tried various changes to the format I pass in with no success.   I can use s:= l printString but we're trying to maintain some consistency in the string we output and simply using printString doesn't help us with that.

Thanks,

Karyn Hurley
IBM
Karyn Hurley
Reply | Threaded
Open this post in threaded view
|

Re: Inconsistencies in NumberPrintPolicy

Boris Popov, DeepCove Labs (SNN)
Karyn,

Are you specifically trying to format a float this way or would the below better serve your needs?

l := 100000.0s1.
s := (NumberPrintPolicy print: l using: '#0.0##') asString.

'100000.0'

-Boris


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Karyn Hurley
Sent: Thursday, October 25, 2012 10:50 AM
To: [hidden email]
Subject: [vwnc] Inconsistencies in NumberPrintPolicy

I am seeing inconsistencies in the NumberPrintPolicy and would like some
feedback.    

If I use this:
l := 10.0.
s := (NumberPrintPolicy print: l using: '#0.0##') asString.  

the value in s, when inspected is '10.0'.   Perfect.

But if I use this number:
l := 100000.0.
s := (NumberPrintPolicy print: l using: '#0.0##') asString.

the value is s is now '99999.992'.  

I've tried various changes to the format I pass in with no success.   I can
use s:= l printString but we're trying to maintain some consistency in the string we output and simply using printString doesn't help us with that.

Thanks,

Karyn Hurley
IBM



--
View this message in context: http://forum.world.st/Inconsistencies-in-NumberPrintPolicy-tp4652934.html
Sent from the VisualWorks mailing list archive at Nabble.com.
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Reply | Threaded
Open this post in threaded view
|

Re: Inconsistencies in NumberPrintPolicy

Georg Heeg
In reply to this post by Karyn Hurley
The issue has only to do with floating point numbers have a small precision.

Just try
l := 100000.0s.
s := (NumberPrintPolicy print: l using: '#0.0##') asString.


Georg

Georg Heeg eK, Dortmund und Köthen, HR Dortmund A 12812
Wallstraße 22, 06366 Köthen
Tel. +49-3496-214328, Fax +49-3496-214712


-----Ursprüngliche Nachricht-----
Von: [hidden email] [mailto:[hidden email]] Im Auftrag
von Karyn Hurley
Gesendet: Donnerstag, 25. Oktober 2012 16:50
An: [hidden email]
Betreff: [vwnc] Inconsistencies in NumberPrintPolicy

I am seeing inconsistencies in the NumberPrintPolicy and would like some
feedback.    

If I use this:
l := 10.0.
s := (NumberPrintPolicy print: l using: '#0.0##') asString.  

the value in s, when inspected is '10.0'.   Perfect.

But if I use this number:
l := 100000.0.
s := (NumberPrintPolicy print: l using: '#0.0##') asString.

the value is s is now '99999.992'.  

I've tried various changes to the format I pass in with no success.   I can
use s:= l printString but we're trying to maintain some consistency in the
string we output and simply using printString doesn't help us with that.

Thanks,

Karyn Hurley
IBM



--
View this message in context:
http://forum.world.st/Inconsistencies-in-NumberPrintPolicy-tp4652934.html
Sent from the VisualWorks mailing list archive at Nabble.com.
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc



_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Reply | Threaded
Open this post in threaded view
|

Re: Inconsistencies in NumberPrintPolicy

Karyn Hurley
In reply to this post by Boris Popov, DeepCove Labs (SNN)
That would work - I was also looking at 100000.0 asFixedPoint: 1.   That does basically the same thing, right?

Thanks!  
Karyn Hurley
Reply | Threaded
Open this post in threaded view
|

Re: Inconsistencies in NumberPrintPolicy

Boris Popov, DeepCove Labs (SNN)
In reply to this post by Boris Popov, DeepCove Labs (SNN)
Yup, all the same,

100000.0s
denominator:  1
numerator:  100000
scale:  1

100000.0s1
denominator:  1
numerator:  100000
scale:  1

100000.0 asFixedPoint: 1
denominator:  1
numerator:  100000
scale:  1

-Boris


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Karyn Hurley
Sent: Thursday, October 25, 2012 11:06 AM
To: [hidden email]
Subject: Re: [vwnc] Inconsistencies in NumberPrintPolicy

That would work - I was also looking at 100000.0 asFixedPoint: 1.   That does
basically the same thing, right?

Thanks!  



--
View this message in context: http://forum.world.st/Inconsistencies-in-NumberPrintPolicy-tp4652934p4652938.html
Sent from the VisualWorks mailing list archive at Nabble.com.
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Reply | Threaded
Open this post in threaded view
|

Re: Inconsistencies in NumberPrintPolicy

Karyn Hurley
I just stumbled on the real pinch point I think.   I'm starting with a string.   I can use the various suggestions on a number and retain my precision but how do I handle the case where I pass in a string '123456.789' and change it to a number 123456.789?  

I appreciate all the feedback, thank you!

Karyn Hurley
Karyn Hurley
Reply | Threaded
Open this post in threaded view
|

Re: Inconsistencies in NumberPrintPolicy

Boris Popov, DeepCove Labs (SNN)
In reply to this post by Boris Popov, DeepCove Labs (SNN)
Karyn,

You'll need to decide on what the appropriate action is yourself depending on the source and formatting of your data, but this may help:

Locale current readNumberFrom: '123456.789' readStream type: FixedPoint

self:  123456.789s
denominator:  1000
numerator:  123456789
scale:  3

-Boris

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Karyn Hurley
Sent: Thursday, October 25, 2012 12:39 PM
To: [hidden email]
Subject: Re: [vwnc] Inconsistencies in NumberPrintPolicy

I just stumbled on the real pinch point I think.   I'm starting with a
string.   I can use the various suggestions on a number and retain my
precision but how do I handle the case where I pass in a string '123456.789'
and change it to a number 123456.789?  

I appreciate all the feedback, thank you!

Karyn Hurley



--
View this message in context: http://forum.world.st/Inconsistencies-in-NumberPrintPolicy-tp4652934p4652949.html
Sent from the VisualWorks mailing list archive at Nabble.com.
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Reply | Threaded
Open this post in threaded view
|

Re: Inconsistencies in NumberPrintPolicy

John Brant-2
In reply to this post by Georg Heeg
On 10/25/2012 10:01 AM, Georg Heeg wrote:
> The issue has only to do with floating point numbers have a small precision.
>
> Just try
> l := 100000.0s.
> s := (NumberPrintPolicy print: l using: '#0.0##') asString.

Yes, but 100000.0 is exactly representable in floating point. The
problem is that the number format has more digits precision than the
float. If you use "NumberPrintPolicy print: 100000.0 using: '#0.0'", you
get 100000.0.

The NumberPrintPolicy>>print:on:using: performs its calculations
directly on the number passed in so it can introduce some errors when
you use floats. I think the "number" in this method should be converted
to a fraction when it is used:
        shiftedNumber := number asFraction / commaScalingFactor.
If you change this, then "NumberPrintPolicy print: 100000.0 using:
'0.0##'" returns 100000.0 as expected. To do this though, you'll need to
add a Number>>asFraction method:
        asFraction
                ^self asRational
Since the asFraction method is part of the ANSI standard, I think it
should be there anyway. Without the asFraction change, the following
causes a divide by zero error:

NumberPrintPolicy print: 1.0 using:
'#0.0###############################################'

With the change, you get the expected 1.0.

Finally, if the asFraction send is added, the exponentLength code should
also be changed not to use #log and #asDouble. Currently, this throws an
error:
   NumberPrintPolicy print: (1/3 * (10 ** -10000)) using: '0.0#e######0'
Here's a simple change that allows the above code to work (and return
3.33e-10001):

        (exponentLength ~~ nil and: [shiftedNumber isZero not]) ifTrue:
                [[shiftedNumber < 1 and: [shiftedNumber > -1]] whileTrue:
[shiftedNumber := shiftedNumber * 10. exponentShift := exponentShift + 1].
                [shiftedNumber >= 10 or: [shiftedNumber <= -10]] whileTrue:
[shiftedNumber := shiftedNumber / 10. exponentShift := exponentShift - 1]].

Someone might want to change this code to calculate the exponentShift in
log time instead of the linear time it is currently using though...


John Brant
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Reply | Threaded
Open this post in threaded view
|

Re: Inconsistencies in NumberPrintPolicy

Paul Baumann
In reply to this post by Karyn Hurley
A Double will give more range than Float while being similar to the floats you've started with. Floats have about six decimals of precision. You can find rounding and truncating problems with either, but that is just the nature of floating point.

| l s |
l := 100000.0d.
l := (Double readFrom: '100000.0' readStream).
s := (NumberPrintPolicy print: l using: '#0.0##') asString.

 '100000.0'



-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Karyn Hurley
Sent: Thursday, October 25, 2012 10:50
To: [hidden email]
Subject: [vwnc] Inconsistencies in NumberPrintPolicy

I am seeing inconsistencies in the NumberPrintPolicy and would like some
feedback.

If I use this:
l := 10.0.
s := (NumberPrintPolicy print: l using: '#0.0##') asString.

the value in s, when inspected is '10.0'.   Perfect.

But if I use this number:
l := 100000.0.
s := (NumberPrintPolicy print: l using: '#0.0##') asString.

the value is s is now '99999.992'.

I've tried various changes to the format I pass in with no success.   I can
use s:= l printString but we're trying to maintain some consistency in the
string we output and simply using printString doesn't help us with that.

Thanks,

Karyn Hurley
IBM



--
View this message in context: http://forum.world.st/Inconsistencies-in-NumberPrintPolicy-tp4652934.html
Sent from the VisualWorks mailing list archive at Nabble.com.
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange, Inc. (ICE), its subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.


_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Reply | Threaded
Open this post in threaded view
|

Re: Inconsistencies in NumberPrintPolicy

Karyn Hurley
In reply to this post by Karyn Hurley
Thanks to all who responded.   It's great to get such fast feedback from so many users!

Karyn Hurley
IBM
Karyn Hurley
Reply | Threaded
Open this post in threaded view
|

Re: Inconsistencies in NumberPrintPolicy

Holger Guhl
If you are a Support customer you can benefit from Res98614. It provides a refined algorithm for
printing in order to suppress bad effects from the inherent inexactness of floating point
representations. With this Resolution you'll have more flexibility in your coding. The proposed
solution to convert to a FixedPoint is good, but it may be a burden to maintain all possible code
for this issue.

Holger Guhl
--
Senior Consultant * Certified Scrum Master * [hidden email]
Tel: +49 231 9 75 99 21 * Fax: +49 231 9 75 99 20
Georg Heeg eK Dortmund
Handelsregister: Amtsgericht Dortmund  A 12812



Am 26.10.2012 13:43, schrieb Karyn Hurley:

> Thanks to all who responded.   It's great to get such fast feedback from so
> many users!
>
> Karyn Hurley
> IBM
>
>
>
> --
> View this message in context: http://forum.world.st/Inconsistencies-in-NumberPrintPolicy-tp4652934p4653016.html
> Sent from the VisualWorks mailing list archive at Nabble.com.
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Reply | Threaded
Open this post in threaded view
|

Re: Inconsistencies in NumberPrintPolicy

Andres Valloud-6
In reply to this post by John Brant-2
FYI all this stuff is in the queue to be fixed.  Please let me know if
you find other types of brokenness.

On 10/25/2012 10:33 AM, John Brant wrote:

> On 10/25/2012 10:01 AM, Georg Heeg wrote:
>> The issue has only to do with floating point numbers have a small
>> precision.
>>
>> Just try
>> l := 100000.0s.
>> s := (NumberPrintPolicy print: l using: '#0.0##') asString.
>
> Yes, but 100000.0 is exactly representable in floating point. The
> problem is that the number format has more digits precision than the
> float. If you use "NumberPrintPolicy print: 100000.0 using: '#0.0'", you
> get 100000.0.
>
> The NumberPrintPolicy>>print:on:using: performs its calculations
> directly on the number passed in so it can introduce some errors when
> you use floats. I think the "number" in this method should be converted
> to a fraction when it is used:
>      shiftedNumber := number asFraction / commaScalingFactor.
> If you change this, then "NumberPrintPolicy print: 100000.0 using:
> '0.0##'" returns 100000.0 as expected. To do this though, you'll need to
> add a Number>>asFraction method:
>      asFraction
>          ^self asRational
> Since the asFraction method is part of the ANSI standard, I think it
> should be there anyway. Without the asFraction change, the following
> causes a divide by zero error:
>
> NumberPrintPolicy print: 1.0 using:
> '#0.0###############################################'
>
> With the change, you get the expected 1.0.
>
> Finally, if the asFraction send is added, the exponentLength code should
> also be changed not to use #log and #asDouble. Currently, this throws an
> error:
>    NumberPrintPolicy print: (1/3 * (10 ** -10000)) using: '0.0#e######0'
> Here's a simple change that allows the above code to work (and return
> 3.33e-10001):
>
>      (exponentLength ~~ nil and: [shiftedNumber isZero not]) ifTrue:
>          [[shiftedNumber < 1 and: [shiftedNumber > -1]] whileTrue:
> [shiftedNumber := shiftedNumber * 10. exponentShift := exponentShift + 1].
>          [shiftedNumber >= 10 or: [shiftedNumber <= -10]] whileTrue:
> [shiftedNumber := shiftedNumber / 10. exponentShift := exponentShift - 1]].
>
> Someone might want to change this code to calculate the exponentShift in
> log time instead of the linear time it is currently using though...
>
>
> John Brant
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc