How to? DoubleArray

Hello all,
Anyone implemented a DoubleArray ?
(Array with IEEE 754 double precision 64 bits floats)

If not, how  to declare such a class ? byteArray, wordArray ?
I did not see a doubleWordArray...
How to convert one element to a Float and vice versa ?

I did not see anything about endianness conversion. where is it ?
Or Is squeak only for little endian machines ?

  Nicolas,

> Anyone implemented a DoubleArray ?
> (Array with IEEE 754 double precision 64 bits floats)

  As far as I know, the answer is "not yet".  (I wouldn't be surprized
that somebody have done it, though.)

> If not, how  to declare such a class ? byteArray, wordArray ?
> I did not see a doubleWordArray...
> How to convert one element to a Float and vice versa ?

  I would go with a subclass of ArrayedCollection.  Look at
ShortIntegerArray, for example.  It is "opposite" of what you would
like to do (two 16-bit quantities in one 32-bit word), but together
with FloatArray implementation, it should give you a sense of what is
needed.

> I did not see anything about endianness conversion. where is it ?
> Or Is squeak only for little endian machines ?

  Of course, it runs on both endianness.  Look at the
ShortIntegerArray, and other classes that implement
#restoreEndianness.

-- Yoshiki

  I happened to have implemented a whole set of vector operations
primitives, and a syntax tree transformer whose output utilizes such
primitives, even if there is a ifTrue:ifFalse: messages.

nicolas cellier wrote:
> Anyone implemented a DoubleArray ?

I thought about it but decided against it. Mainly because implementing
runtime efficient operations (contrary to "just" space efficient
storage) requires that a) the doubles are stored in host order  which is
not supported by the VM (Squeak's Floats are always in MSB order which
will soon prove a hilarious artifact of a long distance past when men
were real men, women were real women, and big endian Macs were real big
endian Macs) and b) various platforms require doubles to be 8-byte
aligned (and Squeak only guarantees 4-byte alignment). Because of that,
having double-arrays would only be useful for efficient storage but if
you need efficient storage you might as well use FloatArrays which cut
your storage requirements further in half.

> If not, how  to declare such a class ? byteArray, wordArray ?
> I did not see a doubleWordArray...

Because there is no such thing.

> How to convert one element to a Float and vice versa ?

You'll either have to write primitives to do the conversion or you'd
have to construct the Float from two words.

> I did not see anything about endianness conversion. where is it ?
> Or Is squeak only for little endian machines ?

It's in the VM and deals exclusively with 32bit quantities.

Cheers,
   - Andreas

Yoshiki Ohshima wrote:
>> I did not see anything about endianness conversion. where is it ?
>> Or Is squeak only for little endian machines ?
>
>   Of course, it runs on both endianness.  Look at the
> ShortIntegerArray, and other classes that implement
> #restoreEndianness.

Actually, it's better *not* to look at #restoreEndianness - it's a red
herring (and the name is wrong, too). The method doesn't really
"restore" anything; all it does is swapping (byte, short, word)
quantities if called on a little endian platform (in other words, it's
always a no-op on big-enders and will be broken if called "the wrong
number of times" on little-enders[*]). A better place to look at is
ShortIntegerArray and friends (see the class-side methods, in particular
#startUp) since they have to implement endianness conversion manually
for lack of VM support.

[*] Even the definition of "wrong number of times" depends on whether
the array in question was loaded in a big- or little-endian format, or
if it was internal to begin with. And of course, if you load a
little-endian array on a big-endian platform and expect
#restoreEndianness to fix that, it won't work either. Confused? Well, I
told you not to look there ;-)


Cheers,
   - Andreas

  Andreas,

  Good explanation.

  The "," after the word "ShortIntegerArray" meant that "look at
ShortIntegerArray", and then look at other classes with some endian
related issues^^;

> [*] Even the definition of "wrong number of times" depends on whether
> the array in question was loaded in a big- or little-endian format, or
> if it was internal to begin with. And of course, if you load a
> little-endian array on a big-endian platform and expect
> #restoreEndianness to fix that, it won't work either. Confused? Well, I
> told you not to look there ;-)

  Yup.

  Still, if you really like to have DoubleArray, perhaps for the
reason to communicate with external libraries or for some other
reasons such as writing your own performance primitives on numbers
with better precision, you could do it, I think.

-- Yoshiki

Yoshiki Ohshima wrote:
I know. It just read a lot like "look at these methods for how to do it"
and it's almost exactly how not to do it ;-)

>   Still, if you really like to have DoubleArray, perhaps for the
> reason to communicate with external libraries or for some other
> reasons such as writing your own performance primitives on numbers
> with better precision, you could do it, I think.

Oh, sure, you absolutely could. My main point was that the naive
intuition is wrong for both of the reasons cited (problems with host
order and problems with alignment) - if that is not an issue (either
because space is the primary concern or because cross-platform behavior
isn't terribly relevant) it can make a ton of sense to use it.

Cheers,
   - Andreas

Thanks Yoshiki and Andreas.

WHY I WANT DoubleArray :
--------------------------------

My aim is to interface with external library (LAPACK), this is what i call
Smallapack.

I want efficiency both in space and operation: i would not allocate these ten
thousand Float needed at each 100 by 100 matrix operation, and wouldn't
execute these many Float * or +, but LAPACK/BLAS will take major part of work
concerning operations.

FloatArray class is just fine, but some processing require double precision
arithmetic, and sorry Andreas, you cannot decide that in my place. So this is
why DoubleArray...

HOW I WOULD DO:
------------------------

I was thinking of using external heap for storing matrices and pass direct
pointers through FFI external calls. This is how i implemented in VW. No
primitive to write, no marshalling or copying, all Smalltalk solution...

For sure, direct pointer are dangerous, but operations on pointers are
protected in some low level methods, that should not be used directly (no way
to prevent someone to bypass warnings in comments and private category, so
not fully safe...).

I need at least some Smalltalk storage for surviving to snapshots. So a first
implementation of DoubleArray would just need methods for swapping bytes
(conform to current machine endianness) at startup, exchanging to/from
external heap and accessing.

Have you any objection or better ideas to suggest ?
**********************************************

Same optimized methods as FloatArray would be appreciated, but i do not need
them for now. Is it a lot more than copy/pasting FloatArray implementation ?

ABOUT BYTE SWAPPING :
------------------------------

Concerning your clue about #restoreEndianness, thank you again, it helped me
as a good starting point.

But i feel Squeak implementation is a little bit more tricky and less uniform
than VW.
In VW byte swapping service is centralized in a separate class
(UninterpretedBytes)  at which do register classes interested in endianness:
this is easier than more general startup service because you do not have to
bother with class initialization order, UninterpretedBytes do that for its
clients...
And secondly, previous endianness is stored in a single place (class var
WasBigEndian in UninterpretedBytes) instead of the LastSaveOrder trick in
ShortIntegerArray and ShortRunArray.
Will i have to repeat such a trick at each new endian-dependent class ? (i
would also have FloatComplex and DoubleComplex for instance).
By the way, i saw no startup message in MatrixTransform2x3, where is this
stuff ?
(i cannot test if swapped, only have one little endian machine).

Beside, tracking #endianness #isLittleEndian and #restoreEndianness, I did
 not manage to understand when those bytes where swapped when writing to a
 file, though i see very well how they are when reading, but i do not mind.
 However i crashed a 3.8  image trying to test some assertions. Unfortunately
 non reproducible...

out of memory

1105559676 ArrayedCollection class>newFromStream:
1105559268 [] in UndefinedObject>DoIt
1105559176 BlockContext>ensure:
1105558928 UndefinedObject>DoIt
1105552676 Compiler>evaluate:in:to:notifying:ifFail:logged:
1105552768 [] in ParagraphEditor>evaluateSelection
1105552584 BlockContext>on:do:
1105552448 ParagraphEditor>evaluateSelection
1105552356 ParagraphEditor>inspectIt
1105552264 TextMorphEditor>inspectIt:
1105552016 ParagraphEditor>dispatchOnCharacter:with:
1105551924 TextMorphEditor>dispatchOnCharacter:with:
1105551832 ParagraphEditor>readKeyboard
1105551740 TextMorphEditor>readKeyboard
1105550876 [] in TextMorph>keyStroke:
1105550784 TextMorph>handleInteraction:fromEvent:
1105550692 TextMorphForEditView>handleInteraction:fromEvent:
1105550548 TextMorph>keyStroke:
1105550456 TextMorphForEditView>keyStroke:
1105550364 TextMorph>handleKeystroke:
1105549996 KeyboardEvent>sentTo:
1105549904 Morph>handleEvent:
1105549812 Morph>handleFocusEvent:
1105550088 [] in HandMorph>sendFocusEvent:to:clear:
1105550180 [] in PasteUpMorph>becomeActiveDuring:
1105549720 BlockContext>on:do:
1105549628 PasteUpMorph>becomeActiveDuring:
1105549404 HandMorph>sendFocusEvent:to:clear:
1105549312 HandMorph>sendEvent:focus:clear:
1105549220 HandMorph>sendKeyboardEvent:
1105549128 HandMorph>handleEvent:
1105548908 HandMorph>processEvents
1105549000 [] in WorldState>doOneCycleNowFor:
1105548816 SequenceableCollection>do:
1105548724 WorldState>handsDo:
1105548632 WorldState>doOneCycleNowFor:
1105548172 WorldState>doOneCycleFor:
1105548080 PasteUpMorph>doOneCycle
1105304784 [] in Project class>spawnNewProcess
1105304968 [] in BlockContext>newProcess

Smalltalk doIt was maybe

| tmp tmp2 file |

tmp := MatrixTransform2x3 identity.
[file := (FileStream fileNamed: 'foo') binary.
tmp writeOn: file]
 ensure: [file close].

[file := (FileStream oldFileNamed: 'foo') binary.
tmp2 := MatrixTransform2x3 newFromStream: file]
 ensure: [file close].
^tmp2

or maybe the same code with

tmp :=MatrixTransform2x3 newFrom: #(1 0 3 0 0 512).

(a copy paste error form ShortRunArray test).

I do not know if it would help much., if any has an idea...

-------------------------------------------------------

nicolas cellier wrote:
> Have you any objection or better ideas to suggest ?

This is a good solution if your main goal is to interface LAPACK or BLAS.

> Same optimized methods as FloatArray would be appreciated, but i do not need
> them for now. Is it a lot more than copy/pasting FloatArray implementation ?

Mostly the primitives would look different. I'd probably recommend you
write them in a DLL to begin with - the overhead of unpacking stuff
that's been packaged for FFI in a "regular" plugin is probably more work
than just writing C functions that take "double*" args.

> But i feel Squeak implementation is a little bit more tricky and less uniform
> than VW.

That is quite possible. Thus far there simply wasn't a need for this
facility outside of those few places that you have observed.

> Will i have to repeat such a trick at each new endian-dependent class ? (i
> would also have FloatComplex and DoubleComplex for instance).
> By the way, i saw no startup message in MatrixTransform2x3, where is this
> stuff ?
> (i cannot test if swapped, only have one little endian machine).

MatrixTransform2x3 (and all other single precision float arrays) don't
need the explicit byte swapping since the VM does endian reversal for
32bit quantities internally.

> Beside, tracking #endianness #isLittleEndian and #restoreEndianness, I did
>  not manage to understand when those bytes where swapped when writing to a
>  file, though i see very well how they are when reading, but i do not mind.

It's tricky business. Stay away from it for the time being; just getting
all the other things to work will be plenty of stuff to do for now.


Cheers,
   - Andreas

Andreas, many thanks again for your answers