ScaledDecimal conversion and equality question

I had a surprise today when I summed some money amounts basically too
different ways and then compared them.  I expected them to be equal, and
they looked equal with printString ('1818.80s'), but they did not test
equal.  Upon inspecting them, I saw that the fraction in one case had a very
large numerator and denominator while the other had the expected small
numerator and denominator.  (The gcd of the numerator and denominator of
each pair was 1, so the fractions were in their simplest forms.)

Of course, the representation of Float 1818.8 is not quite exact and I am
sure this leads to the difference.  I have solved it for my application by
changing my Number>>asMoney method as follows:

   asMoney

       ^ScaledDecimal newFromNumber: (self * 1000) rounded / 1000 scale: 2



but my question is whether I should have needed to do this.  That is, if you
evaluate the following in a workspace, should j and k be equal or not?

| j k |

j := ScaledDecimal newFromNumber: 1818.8 scale: 2.

k := ScaledDecimal newFromNumber: (1818.8 * 100) rounded / 100 scale: 2.

j = k  "==> false"



-- Frank

[hidden email]

On Thu, 1 Mar 2001 15:13:17 -0600, "Frank Sergeant"
<[hidden email]> wrote:

I ran across the same problem working on ANSI Tests for
<scaledDecimal>.  It also seemed counter-intuitive to me.

Both Dolphin 3.0 and VWNC 3.0 say they are not equivalent.  It
actually makes sense when you find out both store a fraction
internally, and both say the difference is 0.00s (well one is + the
other -) but internally a small fraction.

I always heard one had to use Binary Coded Decimal (BCD) for money, or
you open yourself up to all manner of precision evils.

Dolphin 3.0
| j k |
j := ScaledDecimal newFromNumber: 1818.8 scale: 2.
k := ScaledDecimal newFromNumber: (1818.8 * 100) rounded / 100 scale:
2.
j - k.  "==> -0.00s ==> -1/21990232555540"
j = k  "==> false, j hash ==> 39427 & k==> 9091""

VWNC 3.0
| j k |
j := 1818.8 asFixedPoint: 2. " 1818.80s"
k := ((1818.8 * 100) rounded / 100) asFixedPoint: 2. " 1818.80s"
j - k.  "==> 0.00s ==> 1 / 20477"
j = k  "==> false"


--
Richard A. Harmon          "The only good zombie is a dead zombie"
[hidden email]           E. G. McCarthy

"Richard A. Harmon" <[hidden email]> wrote in message
news:[hidden email]...
> On Thu, 1 Mar 2001 15:13:17 -0600, "Frank Sergeant"
> <[hidden email]> wrote:
>
> >I had a surprise today when I summed some money amounts basically too
> >different ways and then compared them.  I expected them to be equal, and
> >they looked equal with printString ('1818.80s'), but they did not test
> >equal.  Upon inspecting them, I saw that the fraction in one case had a
very
> >large numerator and denominator while the other had the expected small
> >numerator and denominator.  (The gcd of the numerator and denominator of
> >each pair was 1, so the fractions were in their simplest forms.)
> >
> >Of course, the representation of Float 1818.8 is not quite exact and I am
> >sure this leads to the difference.  I have solved it for my application
by
> >changing my Number>>asMoney method as follows:
> >
> >   asMoney
> >
> >       ^ScaledDecimal newFromNumber: (self * 1000) rounded / 1000 scale:
2
> >
> >
> >
> >but my question is whether I should have needed to do this.  That is, if
you
Hmm...
QKS Smalltalk and SmallScript return "true".

SmallScript/QKS-Smalltalk
| j k |
j := 1818.8 asScaledDecimal: 100. " 1818.80s"
k := ((1818.8 * 100) rounded / 100) asScaledDecimal: 100. " 1818.80s"
j - k.  "==> 0.00s ==> 1 / 20477"
j = k  "==> false"

Noting that "asFixedPoint: precision" --> "asScaledDecimal: 10 ^^
precision".

When I put the ScaledDecimal class in sometime in 1994-1995, I implemented
it using <Integer> arithmetic and a decimal place scale factor to avoid the
precise set of issues your encountering. To ensure BCD style rounding
requires using integer arithmetic.

I.e., the whole point of implementing ScaledDecimal (a BCD equivalent) is to
avoid the kind of rounding issues one gets from <Float> and <Fraction>
conversions. I raised this very set of issues during the ANSI standards
discussion, but clearly I failed to convince anyone if it is still in VWNC.

It is trivial to implement such a class (~ 20 methods). There is no reason
not to make an open source version of this basic class (CampSmalltalk). It
should not require core system support except for the compiler numeric
literals form (N.DsP).  Since that is presumably just calling the
ScaledDecimal class, one should be able to hi-jack its call pretty easily.

class name      =ScaledDecimal
  extends       =Real
  inst-vars     ='unscaledValue,
                  scale'.

Where the unscaled value in our example was:
    181880
and the scale was:
    100

-- Dave Simmons [www.qks.com / www.smallscript.com]
  "Effectively solving a problem begins with how you express it."

>
> --
> Richard A. Harmon          "The only good zombie is a dead zombie"
> [hidden email]           E. G. McCarthy

On Sat, 03 Mar 2001 19:28:54 GMT, "David Simmons" <[hidden email]>
wrote:

I agree David.  I did a freely available implementation for Squeak 2.7
almost a year ago.  It also includes the <Duration> protocol, and all
but a couple of every other missing ANSI message, and also changes to
existing messages that differed from ANSI behavior.  The
<scaledDecimal> and <Duration> implementations should be pretty
portable (I even have a SIF version of the files).

The bad news is I looked at the VWNC 3.0 implementation figuring that
was a pretty safe bet on how one goes about a ScaledDecimal.

The really bad news is I didn't know what I was doing (don't quibble
over the use of past tense, folks), and I think my code is butt ugly.
If it were a dog folks would ask me to shave its butt and make it walk
backwards.  

The good news is the Squeak version of my stuff passed all the then
available Camp Smalltalk ANSI Tests.  The bad news is the ANSI Tests
weren't complete.  The good news is they are now complete for Dolphin.
The really bad news is I completed them for Dolphin.

I don't think it would take much to change my implementation to use
integers internally instead of fractions.  It might be a good idea if
somebody who actually knew what they were doing did it.  Maybe I'll
try it in Dolphin which is now my dialect of choice.

The Squeak ANSI Compatibility stuff is still available at my web page:
 
         http://homepage2.rconnect.com/raharmon/

Just click on "Previous home page" to the right under the title at the
top of the page.


--
Richard A. Harmon          "The only good zombie is a dead zombie"
[hidden email]           E. G. McCarthy

"Richard A. Harmon" <[hidden email]> wrote in message
news:[hidden email]...
> On Sat, 03 Mar 2001 19:28:54 GMT, "David Simmons" <[hidden email]>
> wrote:

Thansk to David and Richard for the feedback on this.

> >> >evaluate the following in a workspace, should j and k be equal or not?
> >> >
> >> >| j k |
> >> >j := ScaledDecimal newFromNumber: 1818.8 scale: 2.
> >> >k := ScaledDecimal newFromNumber: (1818.8 * 100) rounded / 100 scale:
2.
> >> >j = k  "==> false"
 ...
> >> Both Dolphin 3.0 and VWNC 3.0 say they are not equivalent.  It
 ...
> >Hmm...
> >QKS Smalltalk and SmallScript return "true".
 ...
> >I.e., the whole point of implementing ScaledDecimal (a BCD equivalent) is
to
> >avoid the kind of rounding issues one gets from <Float> and <Fraction>
> >conversions.

I think the above paragraph is the key.  Does Object Arts agree?

> I don't think it would take much to change my implementation to use
> integers internally instead of fractions.

I don't think this makes any difference.  That is, the problem doesn't seem
to have anything to do with storing the number as a fraction, but has
everything to do with how the number is converted.  In other words, what do
we really mean when we say the scale is 2?  Should the number be coerced to
the integer "(number * 10^scale) rounded" or not?

I've implemented something (close) to the above coercion for my application,
so I am happy, but I wanted to call it to the Dolphin community's attention
in case it was a bug.


-- Frank
[hidden email]

Richard,

> > >I had a surprise today when I summed some money amounts basically too
> > >different ways and then compared them.  I expected them to be equal,
and
> > >they looked equal with printString ('1818.80s'), but they did not test
> > >equal.
[snip]
> > >but my question is whether I should have needed to do this.  That is,
if
> > >you evaluate the following in a workspace, should j and k be equal or
not?
> > >
> > >| j k |
> > >j := ScaledDecimal newFromNumber: 1818.8 scale: 2.
> > >k := ScaledDecimal newFromNumber: (1818.8 * 100) rounded / 100 scale:
2.
> > >j = k  "==> false"

You are right, the problem is that the floating point precision is causing
the values to be different -- even though they look the same to the naked
eye.
When I ran your example (using VW5i.3 Beta) I wound up with the same results
you did.  Note that on my machine the precision is:

Floating-point machine parameters
===========================
Radix                                   : 2
Machine Precision                : 5.96046e-8
Negative Machine Precision   : 2.98023e-8
Smallest Number                  : 1.4013e-45
Largest Number                    : 3.40282e38

The following code worked fine:

| j k |
j := 1818.8 asFixedPoint: 2.
k := ((1818.8 * 100) rounded / 100) asFixedPoint: 2.
j equalTo: k

or

| j k |
j := 1818.8 asFixedPoint: 2.
k := ((1818.8 * 100) rounded / 100) asFixedPoint: 2.
j relativelyEqualTo: k upTo: 0.01.

Where the method "equalTo:" and "relativelyEqualTo:upTo:" are patterned
after the ideas in the first chapter of "Object-Oriented Implementation Of
Numerical Methods", by Didier Besset.  The concept is that if the numbers
being compared are equal up to the machine precision, then they can not be
distinguished and therefore can be considered equal.  Since the machine
precision is much much smaller than monetary denominations this approach
works fine (at least for your example).  From some testing I did, it appears
that the discrepancy in the numbers occurs between 1.0e-7 and 1.0e-8 which
is consistent with my machine precision.

Well, hope that helps.

Regards,

Randy

"Randy A. Ynchausti" <[hidden email]> wrote in message
news:vmko6.6865$[hidden email]...
> When I ran your example (using VW5i.3 Beta) I wound up with the same
results
> you did.  Note that on my machine the precision is:
 ...
> The following code worked fine:
> | j k |
> j := 1818.8 asFixedPoint: 2.
> k := ((1818.8 * 100) rounded / 100) asFixedPoint: 2.
> j equalTo: k

Thank you.


-- Frank
[hidden email]

"Frank Sergeant" <[hidden email]> wrote in message news:sgko6.3661

[...snip...]

>  ...
> > >I.e., the whole point of implementing ScaledDecimal (a BCD equivalent)
is seem
> to have anything to do with storing the number as a fraction, but has
> everything to do with how the number is converted.  In other words, what
do
> we really mean when we say the scale is 2?  Should the number be coerced
to
> the integer "(number * 10^scale) rounded" or not?

The main reason to use integers is to avoid having to truncate to an integer
every time you perform a numeric operation involving other scaled decimals
or integer values (which is the common case).

It is most efficient to work with integer values since, for typical work
with things like money, the integer values (incl scale) will all be
<SmallIntegers>. Hence no intermediate float/fraction objects are getting
generated during operations. The only generated objects will be the new
instances of <ScaledDecimal> that form the results.

I.e., all such operations can be performed by selecting the appropriate
scale factor and applying it to one of the two operands. Then performing the
operation directly with the unscaledValue. Finally creating a new instance
of scaled decimal with that "appropriate" scale factor and the unscaled
value result (multiplication and division require appropriate adjustment one
more time for the scale factor). Given those conversions, working with
Integer values is the most efficient in terms of execution time and in terms
of least number of intermediate objects instantiated.

-------------
NOTE: An additional (pattern) technique for improving numeric efficiency is
to define subclasses of the numeric types such as <Float> and
<ScaledDecimal> called <FloatRegister> and <ScaledDecimalRegister>. These
classes are then implemented so that when they are the receiver of a
message, they modify themselves "in-place" and return <self> rather than
generating a new result instance.

** to implement <FloatRegister> efficiently
   probably requires VM access [which you have in Squeak]
   it almost certainly requires the ability to write primitives
   or some equivalent **

E.g.,
    |fr1|
    fr1 := FloatRegister new. "0.0"
    fr1 + 10.
    Transcript << fr1.        "Outputs 10.0"

Clever use of this kind of an object can significantly improve performance
by avoiding the need for new instances to be created since the <self> is
just over-written. You can do even better if you cache the instances in some
pool variables or other useful place. With this technique one can begin to
write code that achieves pretty good floating point or other performance.

You also want to provide a #value: method to set their value directly from
some other object.
-------------


There is a "potentially" open question as to whether <Float> should have a
higher or lower promotion precedence than a <ScaledDecimal>.

I.e.,

    result := scaled_decimal op float

The promotion rule will determine the class of the <result>.

My presumption has been that scaled decimal is higher precedence than a
<Float> (to preserve the scaled decimal precision). So, in that case, when
an op is performed using a <Float> it is "logically" converted to a
<ScaledDecimal>. In reality, this may be optimized within the method for
float_op_scaledDecimal.

-- Dave Simmons [www.qks.com / www.smallscript.com]
  "Effectively solving a problem begins with how you express it."

>
> I've implemented something (close) to the above coercion for my
application,
> so I am happy, but I wanted to call it to the Dolphin community's
attention
> in case it was a bug.
>
>
> -- Frank
> [hidden email]

On Sun, 04 Mar 2001 00:20:15 GMT, [hidden email] (Richard A.
Harmon) wrote:
[snip]
>I did a freely available implementation for Squeak 2.7
>almost a year ago.  It also includes the <Duration> protocol, and all
>but a couple of every other missing ANSI message, and also changes to
>existing messages that differed from ANSI behavior.  The
><scaledDecimal> and <Duration> implementations should be pretty
>portable (I even have a SIF version of the files).
>The <scaledDecimal> and <Duration> implementations should be pretty
>portable (I even have a SIF version of the files).
[snip]

That should read:

=====
I did a freely available implementation for Squeak 2.7 almost a year
ago.  It also includes the <scaledDecimal>, <DateAndTime>, and
<Duration> protocols, and all but a couple of every other missing ANSI
message, and also changes to existing messages that differed from ANSI
behavior.

The  <scaledDecimal>, <DateAndTime>, and <Duration> implementations
should be pretty portable (I even have a SIF version of the files).
=====


--
Richard A. Harmon          "The only good zombie is a dead zombie"
[hidden email]           E. G. McCarthy

On Sat, 3 Mar 2001 23:18:45 -0600, "Frank Sergeant"
<[hidden email]> wrote:

I think your right, and I was a little (a lot?) off.

Ed Shirk passed on some information, and I think I was stumbling over
a mistaken understanding and a Dolphin 3.0 and VWNC 3.0 anomaly.

It seems counter-intuitive that two ScaledDecimals that say they are
818.80s (same float to the same scale) are not equivalent.  From the
Draft ANSI Standard:

        The meaning of "equivalent" cannot be precisely defined but the
        intent is that two objects are considered equivalent if they can
        be used interchangeably. Conforming protocols may choose to more
        precisely define the meaning of "equivalent".  The value of

                receiver = comparand

        is true if and only if the value of

                comparand = receiver

        would also be true. If the value of

                receiver = comparand

        is true then the receiver and comparand must have equivalent hash
        values.  Or more formally:

                receiver = comparand =>
                receiver hash = comparand hash

I think this is more than just an ANSI incompatibility in VWNC 3.0
(and Dolphin 3.0).  Given an ANSI Test like:

        | j x k |
        j := 1818.80s.
        x := ((1818.8 * 100) rounded / 100 asFixedPoint: 2). " 1818.80s"
        k := ((1818.80 asFixedPoint: 2) * 100) / 100. " 1818.80s"
        should: [
                j hash = x hash  "==>  true"
                & j = x  "==>   true"].
        should: [
                j hash = k hash  "==>  true"
                & j = k  "==>  false"].


This shows an anomaly in VW regardless of ANSI.  I think VW expects of
numbers that the hashes differ if they are not #=.


I still can't see what I'm missing if they are both 1818.8 to two
decimal places regardless of how they were created.  I "think" two
BCDs would be equal regardless of how they were created.



--
Richard A. Harmon          "The only good zombie is a dead zombie"
[hidden email]           E. G. McCarthy

On Sat, 3 Mar 2001 22:32:15 -0700, "Randy A. Ynchausti"
<[hidden email]> wrote:
[snip]

Thanks Randy.  That helps.

In the ANSI Tests I finally switched to using #closeTo: on anything
vaguely having to do with floats.  Very clumsy.  I think this might be
wrong as there may be a genuine anomaly in VW (NC 3.0 tested).  Sample
ANSI Test.

        | j x k |
        j := 1818.80s.
        x := ((1818.8 * 100) rounded / 100 asFixedPoint: 2). "
1818.80s"
        k := ((1818.80 asFixedPoint: 2) * 100) / 100. " 1818.80s"
        should: [
           j hash = x hash  "==>  true"
           & j = x  "==>   true"].
        should: [
           j hash = k hash  "==>  true"
           & j = k  "==>  false"].
>You are right, the problem is that the floating point precision is causing
>the values to be different -- even though they look the same to the naked
>eye.

More than "look the same to the naked eye".  All three "say" they are
the same FixedPoint number.  I'm not sure one should poke into the
implementation or precision to find out they are not.  They are
supposed to represent exact amounts which can be created any number of
ways but print as 1818.80s.

I think there can only be one exact amount.  Or what am I missing?


--
Richard A. Harmon          "The only good zombie is a dead zombie"
[hidden email]           E. G. McCarthy

[hidden email] (Richard A. Harmon) wrote (abridged):
> should: [
> j hash = k hash  "==>  true"
> & j = k  "==>  false"].

This uses AND instead of IMPLIES. It is permitted for the hash values to
be equal even if the objects themselves are not equal. It is called a
"hash collision". It's inevitable, eg if hash values are 32-bit and the
objects are longish strings with more than 32 bits of information in
them.

The test should be something like:
    should: [(j hash = k hash) or: [j ~= k]].

  Dave Harris, Nottingham, UK | "Weave a circle round him thrice,
      [hidden email]      |   And close your eyes with holy dread,
                              |  For he on honey dew hath fed
 http://www.bhresearch.co.uk/ |   And drunk the milk of Paradise."

On Sun, 4 Mar 2001 18:07 +0000 (GMT Standard Time), [hidden email]
(Dave Harris) wrote:

Thanks Dave. I just looked right past that it was implies.  I know I
made the same mistake elsewhere in tests.



--
Richard A. Harmon          "The only good zombie is a dead zombie"
[hidden email]           E. G. McCarthy

Richard A. Harmon wrote:

> I still can't see what I'm missing if they are both 1818.8 to two
> decimal places regardless of how they were created.  I "think" two
> BCDs would be equal regardless of how they were created.

I think the deep issue here is whether it is permissible and appropriate for
a <scaledDecimal> to hold *more* digits-worth of precision than is stated by
its #scale.

My personal opinion is that it is inappropriate.  However, I can't find
anything in the ANSI stuff which states this, or even definitely implies it
clearly.  For instance <scaledDecimal>#+ is required only to have *at least*
as many digits as the receiver.  Also the spec of <float> does not even have
a specialisation of #asScaledDecimal:

Try this in Dolphin:

| sd |
sd := (202/100) asScaledDecimal: 1.
sd.             "correctly reports --> 2.0s"
sd scale.    "correctly reports --> 1"
sd = 2.0s.   "--> false"
sd * 5.        "--> 10.1s"

You see that "sd" is calculating to higher precision than it is willing to
admit to holding.

    -- chris

On Mon, 5 Mar 2001 07:35:48 -0000, "Chris Uppal"
<[hidden email]> wrote:

Part of my confusion was that I was mixing myself up with questions
about how ANSI <scaledDecimal> is intended to work, and questions
about the VW FixedPoint implementation.  I haven't spent the time to
figure out VW FixedPoint so I'll leave that to another time.

Chris, I agree with you.  I looked again at the draft of the ANSI
standard, and noted that it say <integer>, <Fraction>, and
<scaledDecimal> values are represented exactly, and <Float> values are
approximations.

Secondly, it seems to me that how a dialect represents a
<scaledDecimal> value shouldn't matter.  So in your example:

        | sd |
        sd := (202/100) asScaledDecimal: 1.
        sd.          "correctly reports --> 2.0s"
        sd scale.    "correctly reports --> 1"

But

        sd = 2.0s.   "--> false"

should answer true, and

        sd * 5.      "--> 10.1s"

should answer 10.0s, and because

        asFraction Definition: <number>
            Answer a fraction that reasonably approximates the
receiver. If
            the receiver is an integral value the result may be
<integer>.


        sd asFraction.      "--> 2 or 2/1"

as should

        ((202/100) asScaledDecimal: 1) asFraction.

It seems to me that any implementation should act as if it were
implemented as a Binary Coded Decimal (BCD).

>You see that "sd" is calculating to higher precision than it is willing to
>admit to holding.

I think this was what was confusing me.  I now think of a
<scaledDecimal> as an exact value representation that should act as a
BCD.  There is only one representation of any specific value.  Any
specific value should be equivalent, print the same, and have the same
hash value.

Thinking about it this way makes this an irrelevant issue.  It is like
just like the <integer> value 3.  The actual number of bytes the
representation takes to hold 3 doesn't matter.  It may be a
LargeInteger or SmallInteger.  I don't need to rely on the
representation unless I need to do something like pass it outside of
Smalltalk.

I think this works.  So given:

        #* Definition: <number>
            Answer a number whose value is the result of multiplying
the receiver
            and operand, . . .
            If the return value conforms to <scaledDecimal> then the
scale of the
            result is at least the scale of the receiver . . . .

        2.1s * 2.10s "==> either 4.4s or 4.41s"
        2.10s * 2.1s "==> 4.41s"


--
Richard A. Harmon          "The only good zombie is a dead zombie"
[hidden email]           E. G. McCarthy

Richard A. Harmon wrote:

> It seems to me that any implementation should act as if it were
> implemented as a Binary Coded Decimal (BCD).

Not much to say, except that I agree with you.  I can't see the point of
<scaledDecimal> unless it *does* round to the given scale.  (Why not just
use <fraction> otherwise?)  But I don't think that it's required by ANSI.

OTOH (there's always another hand in my world), I believe that in many
contexts the rounding algorithm is specified by law/contract/custom, rather
than by the whim of the class's implementor, so I'd guess that
<scaledDecimal> isn't as useful as it might be, even then.

Hmm, how about a plugin RoundingPolicy?

> Richard A. Harmon          "The only good zombie is a dead zombie"

    -- chris