On Wed, Feb 18, 2009 at 12:21 PM, Michael van der Gulik
<[hidden email]> wrote:
Could you give me an example? It isn't immediately intuitive to me what incorrect results might occur. Rounding errors maybe?
Well, I had to go back and figure out what they were ( I don't have the exact examples that caused this issue in the first place). However, if you have an intermediate calculation that results in a fraction more precise than what you need, and you make a scaledDecimal out of it, you can get weird results. The example:
33.33 + 33.33 + 33.33 + 33.33 = 133.32 "in real math"
x := 33.333s2. "This prints as 33.33s2, giving the illusion that you have it exactly as this amount"
x + x + x + x = 133.33s2 "Not accurate, depending on exactly what you are trying to measure"
y := 33.333 asFixedDecimal: 2. "This prints as 33.33"
y + y + y + y = 133.32 "that is, the fixed decimal of scale 2 truely truncated the value, and the result is correct, depending on what you are measuring"
So, if you need to keep the original precision and you are just want to present it at a lower precision, ScaledDecimal is exactly what you want. If, on the other hand, you want the number to be at exactly the precision that you tell it it should be, then FixedDecimal is what you want.
-Chris
_______________________________________________
Beginners mailing list
[hidden email]
http://lists.squeakfoundation.org/mailman/listinfo/beginners
Locked
