Smalltalk › Instantiations

Weird Float behaviour

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

5 messages Options

stepha...@etas.com

Weird Float behaviour

Hello Instantiations team,

I noticed a strange behaviour while working with floats. The usecase I am working on is rounding 169.38125 to four digits after the comma. So I would expect 169.3813 as result. But I get 169.3812.

The actual rounding is done by a dll of our own but the problem appears to be about the float that is passed to the dll as a parameter.

Then I discovered that this is because 169.38125 does not have an exact float-64 representation but is casted to the nearest one (1.69381249999999994315658113919e2) which happens to be a lower one and that value is in principle then rounded down correctly.

But if I inspect 1.6938125 I see in the inspector the hex representation 40652C3333333333 which is to be inspected, but for 1.69381249999999994315658113919e2 I see 40652C3333333334, which I do not expect. and if I compare both numbers with < then

1.6938125 < 1.69381249999999994315658113919e2 evaluates to true which it should not do. I would expect here that either both numbers would be equal or that the former would be greater. I checked the method < and there a virtual machine primitive is called. Is this a bug?

We currently use VA 8.0.2 but will soon migrate to 9.2 64-bit. But I was able to reproduce the issue on both versions.

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/2f9e935d-f6ef-4af5-9758-72cd74a76725%40googlegroups.com.

Mariano Martinez Peck-2

Re: Weird Float behaviour

Hi Stephan,

Maybe someone else from this forum may be able to help you. Off the top of my head I don't see anything obviously wrong. Maybe you prefer to use ScaledDecimals for what you are trying to do?

In any case, if you are a supported customer, feel free to contact customer support at [hidden email] and we will do our best to help you.

Best regards,

On Mon, Nov 4, 2019 at 8:15 AM 'Stephan Henke' via VA Smalltalk <[hidden email]> wrote:

Hello Instantiations team,

I noticed a strange behaviour while working with floats. The usecase I am working on is rounding 169.38125 to four digits after the comma. So I would expect 169.3813 as result. But I get 169.3812.
The actual rounding is done by a dll of our own but the problem appears to be about the float that is passed to the dll as a parameter.
Then I discovered that this is because 169.38125 does not have an exact float-64 representation but is casted to the nearest one (1.69381249999999994315658113919e2) which happens to be a lower one and that value is in principle then rounded down correctly.
But if I inspect 1.6938125 I see in the inspector the hex representation 40652C3333333333 which is to be inspected, but for 1.69381249999999994315658113919e2 I see 40652C3333333334, which I do not expect. and if I compare both numbers with < then

1.6938125 < 1.69381249999999994315658113919e2 evaluates to true which it should not do. I would expect here that either both numbers would be equal or that the former would be greater. I checked the method < and there a virtual machine primitive is called. Is this a bug?
We currently use VA 8.0.2 but will soon migrate to 9.2 64-bit. But I was able to reproduce the issue on both versions.

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/2f9e935d-f6ef-4af5-9758-72cd74a76725%40googlegroups.com.

Mariano Martinez Peck
Software Engineer, Instantiations Inc.

Email: [hidden email]

Twitter: https://twitter.com/MartinezPeck

LinkedIn: www.linkedin.com/in/mariano-martinez-peck

Blog: https://marianopeck.wordpress.com/

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/CAOUkibF0KSN%2Bt5mAdf572%3DwsgUVbyLY3SdSoRM_GLJSqyW%2BsGA%40mail.gmail.com.

Seth Berman

Re: Weird Float behaviour

Greetings Stephan,

Thanks for your question. For a timely response from Instantiations, I would definitely propose [hidden email].

This is a community forum, and while some of us from the company monitor it closely, our active support queue (with tracking system) is where all of us at the company are highly engaged and you will receive a quick response and dedication of resource.

I have created a case on your behalf and you will get a response from us shortly.

- Seth

On Tuesday, November 5, 2019 at 8:56:26 AM UTC-5, Mariano Martinez Peck wrote:

Hi Stephan,

Maybe someone else from this forum may be able to help you. Off the top of my head I don't see anything obviously wrong. Maybe you prefer to use ScaledDecimals for what you are trying to do?

In any case, if you are a supported customer, feel free to contact customer support at [hidden email] and we will do our best to help you.

Best regards,

On Mon, Nov 4, 2019 at 8:15 AM 'Stephan Henke' via VA Smalltalk <[hidden email]> wrote:
Hello Instantiations team,

I noticed a strange behaviour while working with floats. The usecase I am working on is rounding 169.38125 to four digits after the comma. So I would expect 169.3813 as result. But I get 169.3812.
The actual rounding is done by a dll of our own but the problem appears to be about the float that is passed to the dll as a parameter.
Then I discovered that this is because 169.38125 does not have an exact float-64 representation but is casted to the nearest one (1.69381249999999994315658113919e2) which happens to be a lower one and that value is in principle then rounded down correctly.
But if I inspect 1.6938125 I see in the inspector the hex representation 40652C3333333333 which is to be inspected, but for 1.69381249999999994315658113919e2 I see 40652C3333333334, which I do not expect. and if I compare both numbers with < then

1.6938125 < 1.69381249999999994315658113919e2 evaluates to true which it should not do. I would expect here that either both numbers would be equal or that the former would be greater. I checked the method < and there a virtual machine primitive is called. Is this a bug?
We currently use VA 8.0.2 but will soon migrate to 9.2 64-bit. But I was able to reproduce the issue on both versions.

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/va-smalltalk/2f9e935d-f6ef-4af5-9758-72cd74a76725%40googlegroups.com?utm_medium=email&utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/msgid/va-smalltalk/2f9e935d-f6ef-4af5-9758-72cd74a76725%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter';return true;" onclick="this.href='https://groups.google.com/d/msgid/va-smalltalk/2f9e935d-f6ef-4af5-9758-72cd74a76725%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter';return true;">https://groups.google.com/d/msgid/va-smalltalk/2f9e935d-f6ef-4af5-9758-72cd74a76725%40googlegroups.com.

--
Mariano Martinez Peck
Software Engineer, Instantiations Inc.
Email: [hidden email]
Twitter: <a href="https://twitter.com/MartinezPeck" target="_blank" rel="nofollow" onmousedown="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Ftwitter.com%2FMartinezPeck\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFPV-7Bnc-U6phGEh-VZU0iUtY7vw';return true;" onclick="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Ftwitter.com%2FMartinezPeck\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFPV-7Bnc-U6phGEh-VZU0iUtY7vw';return true;">https://twitter.com/MartinezPeck
LinkedIn: <a href="https://www.linkedin.com/in/mariano-mart%C3%ADnez-peck/" target="_blank" rel="nofollow" onmousedown="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.linkedin.com%2Fin%2Fmariano-mart%25C3%25ADnez-peck%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGyJTUAWXPstaw4J3OpFUYRyFAqmw';return true;" onclick="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.linkedin.com%2Fin%2Fmariano-mart%25C3%25ADnez-peck%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGyJTUAWXPstaw4J3OpFUYRyFAqmw';return true;">www.linkedin.com/in/mariano-martinez-peck
Blog: <a href="https://marianopeck.wordpress.com/" target="_blank" rel="nofollow" onmousedown="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fmarianopeck.wordpress.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHAOaIsyMIYgmQWdQZRuKRdD6gBfw';return true;" onclick="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fmarianopeck.wordpress.com%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHAOaIsyMIYgmQWdQZRuKRdD6gBfw';return true;">https://marianopeck.wordpress.com/

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/05cc83c3-47a7-453c-8488-b2afcdd47cd2%40googlegroups.com.

Hans-Martin Mosner-3

Re: Weird Float behaviour

In reply to this post by stepha...@etas.com

Please note that this is not a bug with VA Smalltalk but with your understanding of how floating point numbers work. All binary floating point numbers will work the same, independently of the programming language in which they are used.

Float numbers have binary fractional parts, not decimal fractional parts, so as you already found out they cannot represent decimal fractions exactly (in general; all decimal fractions that are also finite binary fractions such as 0.5, 0.25, 0.125 and their integer multiples such as 0.75, 0.625 can be represented exactly). The 4/5 decimal rounding rule will therefore fail for numbers whose binary fractional representation is infinite.

If you want to round fractional numbers according to the standard decimal rounding rules, do not use Floating Point!

Use ScaledDecimal numbers or work with Fractions (which are always exact) and convert to a decimal representation when the numbers need to be displayed.

Cheers,

Hans-Martin

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/6552a945-8de0-4258-891c-e8044722b0bb%40googlegroups.com.

stepha...@etas.com

Re: Weird Float behaviour

Hallo Hans-Martin,

thank you for your comment. I already was on that point. The behaviour that I describe being weird ist that:

1.6938125 < 1.69381249999999994315658113919e2

That should not be true because the former value is larger.

Am Dienstag, 19. November 2019 14:27:10 UTC+1 schrieb Hans-Martin Mosner:

Please note that this is not a bug with VA Smalltalk but with your understanding of how floating point numbers work. All binary floating point numbers will work the same, independently of the programming language in which they are used.
Float numbers have binary fractional parts, not decimal fractional parts, so as you already found out they cannot represent decimal fractions exactly (in general; all decimal fractions that are also finite binary fractions such as 0.5, 0.25, 0.125 and their integer multiples such as 0.75, 0.625 can be represented exactly). The 4/5 decimal rounding rule will therefore fail for numbers whose binary fractional representation is infinite.

If you want to round fractional numbers according to the standard decimal rounding rules, do not use Floating Point!

Use ScaledDecimal numbers or work with Fractions (which are always exact) and convert to a decimal representation when the numbers need to be displayed.

Cheers,
Hans-Martin

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/21de8b9d-f87e-4b91-ba1c-1417078c199d%40googlegroups.com.