4.85 asStringUsingFormat: #(1 1 false) = 4.8

2 messages
 Yes, this kind of thing is inherent in using binary floating points. Few decimal fractions can be exactly represented by a binary float. Ones ending in ".85" can never be exactly represented. Strings such as "4.85" and "20.85" are rounded to the nearest representable binary float. Depending on the integer part, the closest float can be either greater than or less than the ideal value. So when further decimal rounding is done, sometimes down is the correct direction, sometimes up is. Converting the float to a fraction can show you what the exact value of a float is, and how it compares to the decimal string. 4.85 asStringUsingFormat: #(1 1 false) = 4.8 4.85 asFraction < (485/100) --> true 20.85 asStringUsingFormat: #(1 1 false) = 20.9 20.85 asFraction > (2085 / 100) --> true 1.85 => 1.9 1.85 asFraction > (185 / 100) --> true 5.85 => 5.8 5.85 asFraction < (585 / 100) --> true Regards, -Martin On 3/4/21 12:32 PM, Marten Feldtmann via Glass wrote: 4.85 asStringUsingFormat: #(1 1 false) = 4.8 20.85 asStringUsingFormat: #(1 1 false) = 20.9 1.85 => 1.9 5.85 => 5.8 So, actually this is the typical floating point problem ... :-( ```_______________________________________________ Glass mailing list [hidden email] https://lists.gemtalksystems.com/mailman/listinfo/glass ``` _______________________________________________ Glass mailing list [hidden email] https://lists.gemtalksystems.com/mailman/listinfo/glass