Float precision

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Float precision

Markus Lampert
Evaluating the following:

0.0 to: 10 by: 0.1 do: [ :v |
    Transcript show: v printString; crlf]

gives some interesting numbers for 3.6, 3.7, 3.8, 3.9 and everything above 5.5.
Is this expected given the rounding effects of Float?

Thanks,
Markus




Reply | Threaded
Open this post in threaded view
|

Re: Float precision

Michael Haupt-3
Hi,

Am 22.12.2010 um 06:19 schrieb Markus Lampert <[hidden email]>:
> Evaluating the following:
>
> 0.0 to: 10 by: 0.1 do: [ :v |
>    Transcript show: v printString; crlf]
>
> gives some interesting numbers for 3.6, 3.7, 3.8, 3.9 and everything above 5.5.
> Is this expected given the rounding effects of Float?

inaccuracies are always expected with floating-point numbers. If you want accurate results, you should use BigDecimal (IIRC) or SmallInteger and manual decimal point management.

Best,

Michael
Reply | Threaded
Open this post in threaded view
|

Re: Float precision

David T. Lewis
On Wed, Dec 22, 2010 at 08:04:11AM +0100, Michael Haupt wrote:

> Hi,
>
> Am 22.12.2010 um 06:19 schrieb Markus Lampert <[hidden email]>:
> > Evaluating the following:
> >
> > 0.0 to: 10 by: 0.1 do: [ :v |
> >    Transcript show: v printString; crlf]
> >
> > gives some interesting numbers for 3.6, 3.7, 3.8, 3.9 and everything above 5.5.
> > Is this expected given the rounding effects of Float?
>
> inaccuracies are always expected with floating-point numbers. If you want accurate results, you should use BigDecimal (IIRC) or SmallInteger and manual decimal point management.
>
> Best,
>
> Michael

Yes. And that would be ScaledDecimal (not BigDecimal).

This is a common source of confusion. Wikipedia gives a good overview
of the topic:

http://en.wikipedia.org/wiki/Floating_point

Dave

Reply | Threaded
Open this post in threaded view
|

Re: Float precision

Michael Haupt-3
Hi David,

Am 22.12.2010 um 14:38 schrieb "David T. Lewis" <[hidden email]>:
> Yes. And that would be ScaledDecimal (not BigDecimal).

thanks for the correction, I've been occupied with Java code lately. :-)

Best,

Michael

Reply | Threaded
Open this post in threaded view
|

Re: Float precision

Markus Lampert
In reply to this post by David T. Lewis
Thanks, I was hoping I did something wrong ... ;)

Markus



----- Original Message ----

> From: David T. Lewis <[hidden email]>
> To: The general-purpose Squeak developers list
><[hidden email]>
> Sent: Wed, December 22, 2010 5:38:01 AM
> Subject: Re: [squeak-dev] Float precision
>
> On Wed, Dec 22, 2010 at 08:04:11AM +0100, Michael Haupt wrote:
> >  Hi,
> >
> > Am 22.12.2010 um 06:19 schrieb Markus Lampert <[hidden email]>:
> >  > Evaluating the following:
> > >
> > > 0.0 to: 10 by: 0.1 do:  [ :v |
> > >    Transcript show: v printString; crlf]
> >  >
> > > gives some interesting numbers for 3.6, 3.7, 3.8, 3.9 and  everything above
>5.5.
> > > Is this expected given the rounding effects  of Float?
> >
> > inaccuracies are always expected with floating-point  numbers. If you want
>accurate results, you should use BigDecimal (IIRC) or  SmallInteger and manual
>decimal point management.
>
> >
> >  Best,
> >
> > Michael
>
> Yes. And that would be ScaledDecimal (not  BigDecimal).
>
> This is a common source of confusion. Wikipedia gives a good  overview
> of the topic:
>
> http://en.wikipedia.org/wiki/Floating_point
>
> Dave
>
>