Inconsistencies in NumberPrintPolicy
Locked
Inconsistencies in NumberPrintPolicy
 
|
I am seeing inconsistencies in the NumberPrintPolicy and would like some feedback.
If I use this: l := 10.0. s := (NumberPrintPolicy print: l using: '#0.0##') asString. the value in s, when inspected is '10.0'. Perfect. But if I use this number: l := 100000.0. s := (NumberPrintPolicy print: l using: '#0.0##') asString. the value is s is now '99999.992'. I've tried various changes to the format I pass in with no success. I can use s:= l printString but we're trying to maintain some consistency in the string we output and simply using printString doesn't help us with that. Thanks, Karyn Hurley IBM
Karyn Hurley
|
Re: Inconsistencies in NumberPrintPolicy
|
|
Karyn,
Are you specifically trying to format a float this way or would the below better serve your needs? l := 100000.0s1. s := (NumberPrintPolicy print: l using: '#0.0##') asString. '100000.0' -Boris -----Original Message----- From: [hidden email] [mailto:[hidden email]] On Behalf Of Karyn Hurley Sent: Thursday, October 25, 2012 10:50 AM To: [hidden email] Subject: [vwnc] Inconsistencies in NumberPrintPolicy I am seeing inconsistencies in the NumberPrintPolicy and would like some feedback. If I use this: l := 10.0. s := (NumberPrintPolicy print: l using: '#0.0##') asString. the value in s, when inspected is '10.0'. Perfect. But if I use this number: l := 100000.0. s := (NumberPrintPolicy print: l using: '#0.0##') asString. the value is s is now '99999.992'. I've tried various changes to the format I pass in with no success. I can use s:= l printString but we're trying to maintain some consistency in the string we output and simply using printString doesn't help us with that. Thanks, Karyn Hurley IBM -- View this message in context: http://forum.world.st/Inconsistencies-in-NumberPrintPolicy-tp4652934.html Sent from the VisualWorks mailing list archive at Nabble.com. _______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc _______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc |
Re: Inconsistencies in NumberPrintPolicy
|
|
In reply to this post by Karyn Hurley
The issue has only to do with floating point numbers have a small precision.
Just try l := 100000.0s. s := (NumberPrintPolicy print: l using: '#0.0##') asString. Georg Georg Heeg eK, Dortmund und Köthen, HR Dortmund A 12812 Wallstraße 22, 06366 Köthen Tel. +49-3496-214328, Fax +49-3496-214712 -----Ursprüngliche Nachricht----- Von: [hidden email] [mailto:[hidden email]] Im Auftrag von Karyn Hurley Gesendet: Donnerstag, 25. Oktober 2012 16:50 An: [hidden email] Betreff: [vwnc] Inconsistencies in NumberPrintPolicy I am seeing inconsistencies in the NumberPrintPolicy and would like some feedback. If I use this: l := 10.0. s := (NumberPrintPolicy print: l using: '#0.0##') asString. the value in s, when inspected is '10.0'. Perfect. But if I use this number: l := 100000.0. s := (NumberPrintPolicy print: l using: '#0.0##') asString. the value is s is now '99999.992'. I've tried various changes to the format I pass in with no success. I can use s:= l printString but we're trying to maintain some consistency in the string we output and simply using printString doesn't help us with that. Thanks, Karyn Hurley IBM -- View this message in context: http://forum.world.st/Inconsistencies-in-NumberPrintPolicy-tp4652934.html Sent from the VisualWorks mailing list archive at Nabble.com. _______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc _______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc |
Re: Inconsistencies in NumberPrintPolicy
|
|
In reply to this post by Boris Popov, DeepCove Labs (SNN)
That would work - I was also looking at 100000.0 asFixedPoint: 1. That does basically the same thing, right?
Thanks!
Karyn Hurley
|
Re: Inconsistencies in NumberPrintPolicy
|
|
In reply to this post by Boris Popov, DeepCove Labs (SNN)
Yup, all the same,
100000.0s denominator: 1 numerator: 100000 scale: 1 100000.0s1 denominator: 1 numerator: 100000 scale: 1 100000.0 asFixedPoint: 1 denominator: 1 numerator: 100000 scale: 1 -Boris -----Original Message----- From: [hidden email] [mailto:[hidden email]] On Behalf Of Karyn Hurley Sent: Thursday, October 25, 2012 11:06 AM To: [hidden email] Subject: Re: [vwnc] Inconsistencies in NumberPrintPolicy That would work - I was also looking at 100000.0 asFixedPoint: 1. That does basically the same thing, right? Thanks! -- View this message in context: http://forum.world.st/Inconsistencies-in-NumberPrintPolicy-tp4652934p4652938.html Sent from the VisualWorks mailing list archive at Nabble.com. _______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc _______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc |
Re: Inconsistencies in NumberPrintPolicy
|
|
I just stumbled on the real pinch point I think. I'm starting with a string. I can use the various suggestions on a number and retain my precision but how do I handle the case where I pass in a string '123456.789' and change it to a number 123456.789?
I appreciate all the feedback, thank you! Karyn Hurley
Karyn Hurley
|
Re: Inconsistencies in NumberPrintPolicy
|
|
In reply to this post by Boris Popov, DeepCove Labs (SNN)
Karyn,
You'll need to decide on what the appropriate action is yourself depending on the source and formatting of your data, but this may help: Locale current readNumberFrom: '123456.789' readStream type: FixedPoint self: 123456.789s denominator: 1000 numerator: 123456789 scale: 3 -Boris -----Original Message----- From: [hidden email] [mailto:[hidden email]] On Behalf Of Karyn Hurley Sent: Thursday, October 25, 2012 12:39 PM To: [hidden email] Subject: Re: [vwnc] Inconsistencies in NumberPrintPolicy I just stumbled on the real pinch point I think. I'm starting with a string. I can use the various suggestions on a number and retain my precision but how do I handle the case where I pass in a string '123456.789' and change it to a number 123456.789? I appreciate all the feedback, thank you! Karyn Hurley -- View this message in context: http://forum.world.st/Inconsistencies-in-NumberPrintPolicy-tp4652934p4652949.html Sent from the VisualWorks mailing list archive at Nabble.com. _______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc _______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc |
Re: Inconsistencies in NumberPrintPolicy
|
|
In reply to this post by Georg Heeg
On 10/25/2012 10:01 AM, Georg Heeg wrote:
> The issue has only to do with floating point numbers have a small precision. > > Just try > l := 100000.0s. > s := (NumberPrintPolicy print: l using: '#0.0##') asString. Yes, but 100000.0 is exactly representable in floating point. The problem is that the number format has more digits precision than the float. If you use "NumberPrintPolicy print: 100000.0 using: '#0.0'", you get 100000.0. The NumberPrintPolicy>>print:on:using: performs its calculations directly on the number passed in so it can introduce some errors when you use floats. I think the "number" in this method should be converted to a fraction when it is used: shiftedNumber := number asFraction / commaScalingFactor. If you change this, then "NumberPrintPolicy print: 100000.0 using: '0.0##'" returns 100000.0 as expected. To do this though, you'll need to add a Number>>asFraction method: asFraction ^self asRational Since the asFraction method is part of the ANSI standard, I think it should be there anyway. Without the asFraction change, the following causes a divide by zero error: NumberPrintPolicy print: 1.0 using: '#0.0###############################################' With the change, you get the expected 1.0. Finally, if the asFraction send is added, the exponentLength code should also be changed not to use #log and #asDouble. Currently, this throws an error: NumberPrintPolicy print: (1/3 * (10 ** -10000)) using: '0.0#e######0' Here's a simple change that allows the above code to work (and return 3.33e-10001): (exponentLength ~~ nil and: [shiftedNumber isZero not]) ifTrue: [[shiftedNumber < 1 and: [shiftedNumber > -1]] whileTrue: [shiftedNumber := shiftedNumber * 10. exponentShift := exponentShift + 1]. [shiftedNumber >= 10 or: [shiftedNumber <= -10]] whileTrue: [shiftedNumber := shiftedNumber / 10. exponentShift := exponentShift - 1]]. Someone might want to change this code to calculate the exponentShift in log time instead of the linear time it is currently using though... John Brant _______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc |
Re: Inconsistencies in NumberPrintPolicy
|
|
In reply to this post by Karyn Hurley
A Double will give more range than Float while being similar to the floats you've started with. Floats have about six decimals of precision. You can find rounding and truncating problems with either, but that is just the nature of floating point.
| l s | l := 100000.0d. l := (Double readFrom: '100000.0' readStream). s := (NumberPrintPolicy print: l using: '#0.0##') asString. '100000.0' -----Original Message----- From: [hidden email] [mailto:[hidden email]] On Behalf Of Karyn Hurley Sent: Thursday, October 25, 2012 10:50 To: [hidden email] Subject: [vwnc] Inconsistencies in NumberPrintPolicy I am seeing inconsistencies in the NumberPrintPolicy and would like some feedback. If I use this: l := 10.0. s := (NumberPrintPolicy print: l using: '#0.0##') asString. the value in s, when inspected is '10.0'. Perfect. But if I use this number: l := 100000.0. s := (NumberPrintPolicy print: l using: '#0.0##') asString. the value is s is now '99999.992'. I've tried various changes to the format I pass in with no success. I can use s:= l printString but we're trying to maintain some consistency in the string we output and simply using printString doesn't help us with that. Thanks, Karyn Hurley IBM -- View this message in context: http://forum.world.st/Inconsistencies-in-NumberPrintPolicy-tp4652934.html Sent from the VisualWorks mailing list archive at Nabble.com. _______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange, Inc. (ICE), its subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired. _______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc |
Re: Inconsistencies in NumberPrintPolicy
|
|
In reply to this post by Karyn Hurley
Thanks to all who responded. It's great to get such fast feedback from so many users!
Karyn Hurley IBM
Karyn Hurley
|
Re: Inconsistencies in NumberPrintPolicy
|
|
If you are a Support customer you can benefit from Res98614. It provides a refined algorithm for
printing in order to suppress bad effects from the inherent inexactness of floating point representations. With this Resolution you'll have more flexibility in your coding. The proposed solution to convert to a FixedPoint is good, but it may be a burden to maintain all possible code for this issue. Holger Guhl -- Senior Consultant * Certified Scrum Master * [hidden email] Tel: +49 231 9 75 99 21 * Fax: +49 231 9 75 99 20 Georg Heeg eK Dortmund Handelsregister: Amtsgericht Dortmund A 12812 Am 26.10.2012 13:43, schrieb Karyn Hurley: > Thanks to all who responded. It's great to get such fast feedback from so > many users! > > Karyn Hurley > IBM > > > > -- > View this message in context: http://forum.world.st/Inconsistencies-in-NumberPrintPolicy-tp4652934p4653016.html > Sent from the VisualWorks mailing list archive at Nabble.com. > _______________________________________________ > vwnc mailing list > [hidden email] > http://lists.cs.uiuc.edu/mailman/listinfo/vwnc _______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc |
Re: Inconsistencies in NumberPrintPolicy
|
|
In reply to this post by John Brant-2
FYI all this stuff is in the queue to be fixed. Please let me know if
you find other types of brokenness. On 10/25/2012 10:33 AM, John Brant wrote: > On 10/25/2012 10:01 AM, Georg Heeg wrote: >> The issue has only to do with floating point numbers have a small >> precision. >> >> Just try >> l := 100000.0s. >> s := (NumberPrintPolicy print: l using: '#0.0##') asString. > > Yes, but 100000.0 is exactly representable in floating point. The > problem is that the number format has more digits precision than the > float. If you use "NumberPrintPolicy print: 100000.0 using: '#0.0'", you > get 100000.0. > > The NumberPrintPolicy>>print:on:using: performs its calculations > directly on the number passed in so it can introduce some errors when > you use floats. I think the "number" in this method should be converted > to a fraction when it is used: > shiftedNumber := number asFraction / commaScalingFactor. > If you change this, then "NumberPrintPolicy print: 100000.0 using: > '0.0##'" returns 100000.0 as expected. To do this though, you'll need to > add a Number>>asFraction method: > asFraction > ^self asRational > Since the asFraction method is part of the ANSI standard, I think it > should be there anyway. Without the asFraction change, the following > causes a divide by zero error: > > NumberPrintPolicy print: 1.0 using: > '#0.0###############################################' > > With the change, you get the expected 1.0. > > Finally, if the asFraction send is added, the exponentLength code should > also be changed not to use #log and #asDouble. Currently, this throws an > error: > NumberPrintPolicy print: (1/3 * (10 ** -10000)) using: '0.0#e######0' > Here's a simple change that allows the above code to work (and return > 3.33e-10001): > > (exponentLength ~~ nil and: [shiftedNumber isZero not]) ifTrue: > [[shiftedNumber < 1 and: [shiftedNumber > -1]] whileTrue: > [shiftedNumber := shiftedNumber * 10. exponentShift := exponentShift + 1]. > [shiftedNumber >= 10 or: [shiftedNumber <= -10]] whileTrue: > [shiftedNumber := shiftedNumber / 10. exponentShift := exponentShift - 1]]. > > Someone might want to change this code to calculate the exponentShift in > log time instead of the linear time it is currently using though... > > > John Brant > _______________________________________________ > vwnc mailing list > [hidden email] > http://lists.cs.uiuc.edu/mailman/listinfo/vwnc > vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc |
«
Return to VisualWorks
|
1 view|%1 views
| Free forum by Nabble | Edit this page |