FFI problem: too many arguments

I am trying to interface LAPACK with squeak (Smallapack).
Unfortunately, some methods will have too many arguments...

I do not feel like hand writing wrapper functions to group several arguments
in an array, it's boring...

Moreover, i would have to write this in FORTRAN, compile and link: that's
against my starting principle that a user come along with its pre-compiled
LAPACK library, and then all Smallapack code is Smalltalk.

Number of arguments is probably far enough for true Smalltalk code.
But very short for the external world.
I do not experience the same limit in VW (it must be 255 i think).
Could the limit be augmented ? Anyone has a better idea ?

--------------------------------
Here is an example (a usefull one) that fails:

xgeevWithjobvl: jobvl jobvr: jobvr n: n a: a lda: lda wr: wr wi: wi vl: vl
ldvl: ldvl vr: vr ldvr: ldvr work: work lwork: lwork info: info length:
lengthOfjobvl length: lengthOfjobvr
 "
*  Purpose
*  =======
*  SGEEV computes for an N-by-N real nonsymmetric matrix A, the
*  eigenvalues and, optionally, the left and/or right eigenvectors.
*  The right eigenvector v(j) of A satisfies
*                   A * v(j) = lambda(j) * v(j)
*  where lambda(j) is its eigenvalue.
*  The left eigenvector u(j) of A satisfies
*                u(j)**H * A = lambda(j) * u(j)**H
*  where u(j)**H denotes the conjugate transpose of u(j).
*  The computed eigenvectors are normalized to have Euclidean norm
*  equal to 1 and largest component real.
"

 <cdecl: long 'sgeev_'( char *   char *   long * float *  * float *  * float *  
* float *  * float *  *   long *   long   long )>
 ^self externalCallFailed

There is no fix for this problem. In Squeak, the number of parameters
for a method invocation is limited to 15.

Cheers,
   - Andreas

nicolas cellier wrote:

Andreas Raabwrote on Wed, 15 Feb 2006 22:13:26 -0500:
> There is no fix for this problem. In Squeak, the number of parameters
> for a method invocation is limited to 15.

Not quite true (in theory) - the VI4 version of Squeak could have up to
256 parameters (if my quick impression of the code is correct).

http://minnow.cc.gatech.edu/squeak/2119
 
In practice Nicolas will almost certainly not want to go that route
given that nobody else will, so the answer is that indeed there is no
fix for this problem.

-- Jecel

On 16.02.2006, at 15:22, Jecel Assumpcao Jr wrote:

There are a lot of upper bounds in the squeak bytecode, e.g. jump
offset, number of temps (256), number of instVars, number of Literals...

That's quite bad as soon as you try anything with code generation that
has different properties than Smalltalk code. Or when you try to  
annotate
code (e.g., for tracing or debugging or meta programming).

I'd love to see a vm and bytecode design with all these artificial  
limitations fixed...
in SmalltalkX Claus extended the bytecode a long time ago to have no  
limits at all
(offsets can grow indefinitly).

      Marcus

Marcus Denker wrote on Thu, 16 Feb 2006 14:51:38 +0100:
> There are a lot of upper bounds in the squeak bytecode, e.g. jump
> offset, number of temps (256), number of instVars, number of Literals...

I don't like such limits myself, but can certainly understand the hacker
urge to optimize the lower level as far as possible with the idea of
hiding the resulting limitations in higher layers. Chuck Moore (the
Forth one, not the PowerPC one) is a great example of this, as were the
Apple II and the original Mac.
 
The original Self VM didn't have this kind of limit and the change to a
more Squeak-like style in the 4.1 VM didn't add any limits that I
remember. You might find the one I am building right now (in VHDL, but
there is a simulation in Squeak) interesting:

http://www.merlintec.com:8080/Hardware/Oliver

The only limit is that indirection vectors must be at depth 16 or less
in the stack, but that is so trivial to compensate in software that I
didn't think it was worth eliminating.

Though that page describes the 16 bit version, a 32, 36, 48 or 64 bit
"VM" would share the same bytecodes and even the same image format. In
the image, numbers are always stored as arbitrary precision integers and
when loaded they become either a SmallInteger with immediate encoding or
a boxed LargeInteger depending on their value. So a 24 bit number, for
example, would be read as a LargeInteger in the 16 bit machine but as a
SmallInteger on a 32 bit one. The actual bits for a bitmap are also
saved in the image as an arbitrary precision integer. The only
restriction in swapping images among processors with different sizes is
that the 16 bit one can't handle more than 32K objects.

The Transputer had an additional trick to implement width independent
machine code: not only was there a "prefix" instruction to extend the
following instruction but there was also a "negative prefix" one so you
didn't have to worry about 16rFFFF being -1 on a 16 bit T222 but 65535
on a 32 bit T800. I originally had this too, but by replacing the
"PC:=PC+OP" style of jumps with a "PC substituteLowBitsWith: OP"
operation (the first is easier to do in software but the latter is
easier in hardware) and adding a "push small constant" instruction that
I could get by without negative numbers.

-- Jecel

Le Jeudi 16 Février 2006 14:51, Marcus Denker a écrit :
As long as you have a Compiler with BlockClosure, you do not need a jump
instruction in the byte code: jump is just an optimisation.

Temporary variables and literals can each be gathered in a single array, all
you need is #at: #at:put: #perform:withArguments, and when accessing the
arrays, do arithmetics with small numbers so that they are not coded as
literals themselves.

This his how I did overpass byte code limits for arbitrary complex symbolic
expression in VW, transparently to the user (the byte code limit did raise
exception that were handled at compiler level).
Generated code were not optimal, but it was still more efficient than
sub-expression substitution.

Maybe I should publish it someday (some clean up necessary).
It must be transposable to Squeak I think (BlockClosure needed).

Unfortunately, this does not solve my problem of number of arguments...

> I'd love to see a vm and bytecode design with all these artificial
> limitations fixed...
> in SmalltalkX Claus extended the bytecode a long time ago to have no
> limits at all
> (offsets can grow indefinitly).
>
>       Marcus

That would be great, but i presume futur is far away...

Maybe a trick is possible at FFI level ? (pass an array of arguments)

You should have a look at ByteSurgeon in Squeak :)

Stef

On 16 févr. 06, at 19:59, nicolas cellier wrote:

Le Jeudi 16 Février 2006 20:28, vous avez écrit :
> You should have a look at ByteSurgeon in Squeak :)
>
> Stef

Thanks, I'll have a look at it. Seems good to try extensions of Smalltalk.

Now, it seems that the too-many-arguments-problem has one solution:

IDENTIFICATION OF EXISTING PRIMITIVE:
-------------------------------------------------

It appear that function FFIPlugin>>#primitiveCalloutWithArgs do exactly what
 i require:
it will invoke the external method with an arbitrary number of arguments
packed into an Array (in order to have a single method argument).

Just have to discover the hack how to invoke it, without creating the
ExternalMethod by hand... (I do not want to paraphrase Parser <cdecl:...>
interpretation).

PROPOSED SOLUTION (modification of compiler)
----------------------------

I noticed that #primitiveCallout is already a hack in the compiler (primitive
number 120 is hard coded in the Parser).

It is simple to add another hack that would still generate the
ExternalFunction as first literal but send #invokeWithArguments: instead of
primitive 120 in case the arguments size do not match... (in
MethodNode>>#generate: ).

By now, when arguments size do not match the ExternalMethod argTypes size +
 1, the primitive simply fail (cf. FFIPlugin>>#primitiveCallout).
We do not get a warning at compile time... That's bad.

Better behaviour would be to signal size mismatch at parser level, then if
number of method arguments is 1, signal the user that this may work if the
arguments is an array, propose to proceed or to abort.

The hack is not beautiful, but neither is actual code.
This would solve my problems and possible future problems of guys wishing to
use FFI.

REQUEST TO PROCEED WITH IMPLEMENTATION
------------------------------------------------------------

I did not check the 3.9 image. Is there work in progress in such area ?
If not, I can proceed unless anyone has objections ?

nicolas cellier wrote:
> Maybe a trick is possible at FFI level ? (pass an array of arguments)

That is indeed possible. For example:

MyClass>>initialize
   "Set up an ffi call"
   MyFFICall := ExternalLibraryFunction
     name: 'myfunction'
     module: 'mymodule'
     callType: FFICallTypeCDecl
     returnType: ExternalType char
     argumentTypes: {
       ExternalType ulong.
       ExternalType ulong.
       ExternalType ulong
       "..."
     }

This is exactly equivalent to, e.g.,

myFunction: arg1 with: arg2 with: arg3
   <cdecl: char 'myfunction' (ulong ulong ulong) module: 'mymodule'>

and can later be invoked via:

   MyFFICall invokeWithArguments: #(1 2 3).


Cheers,
   - Andreas

I propose a patch in the Parser to check externalFunction number of arguments
against method selector number of arguments and notify the user in case of
mismatch.

Current Parser implementation does silently accept the method. If there is a
mismatch, the primitive 120 (external call) will fail at execution time...
very lately in my opinion, and hard to debug...

This correction is for 3.8.
Some of the changed methods have been redefined in 3.9, so this patch does not
apply to 3.9, but has to be rewritten (easy).

>There is no fix for this problem. In Squeak, the number of parameters for a method invocation is limited to 15.

I think there may be a relatively simple solution for extending the number of arguments to 31.  I realize that it is only a factor of two, and folks may consider it not worth worrying about this unless all limits are to be relieved but, if I'm correct, I'd be remiss if I didn't point this out.

 It looks to me like the method headers can take another bit of argCount as is, and the double-extended-do-anything bytecode can specify up to 31 arguments on a send.  I added the DEDA bytecode for other reasons, and may just never have bothered extending argcount at the time, figuring such code is, um, not generally found in Smalltalk programs.

So  if I'm not mistaken about the argCount bits in the method header and the DEDA bytecode, all that remains is a bit of compiler work and a few methods in CompiledMethod which shouldn't add up to more than an hour or two for someone who knows the territory.

FWIW

        - Dan

Le Dimanche 19 Février 2006 01:42, Dan Ingalls a écrit :
Very true, lot of arguments surely indicate lack of an abstraction level
(think of creating a new class).
This was just in case of external world linking. We cannot change external
world, we must adapt.

> So  if I'm not mistaken about the argCount bits in the method header and
> the DEDA bytecode, all that remains is a bit of compiler work and a few
> methods in CompiledMethod which shouldn't add up to more than an hour or
> two for someone who knows the territory.
>
> FWIW
>
>  - Dan

I have another idea (workaround): fake this limit passing a single array
argument:
 - transform method calls (recvr key1: arg1: ... keyn: argn) into (recvr
key1:key2:keyn: {arg1. arg2. ... argn})
 - transform method implementation: argj would be replaced by (arg1 at: j)
This can be done transparently at code generation from ParseNode.
Some class like MessageAsTempNode can easily do the work.

So i started inquiring bytecode generation and found this: the DEDA you are
speaking of also have a limitation to 256 variables. And i did not see these
limitations enforced anywhere in the code.
Only the single byte code limit is tested for in LeafNode>>#code:type:, and
the code for DEDA is temporarily encoded (type*256+index). Should index
overpass the 256 limit, there might be strange results, changing the type of
operation and probably crashing the VM...

Protecting this code would be easy
(index > 255) ifTrue: [self error: 'bycode limit exceeded'].

Maybe i missed a protection elsewhere ? I will try to exhibit a test case
latter.

On 19.02.2006, at 01:42, Dan Ingalls wrote:

>> There is no fix for this problem. In Squeak, the number of  
>> parameters for a method invocation is limited to 15.
>
> I think there may be a relatively simple solution for extending the  
> number of arguments to 31.  I realize that it is only a factor of  
> two, and folks may consider it not worth worrying about this unless  
> all limits are to be relieved

Everytime someone proposes a small improvement, people are very fast  
to argue that "there will be a much more general solution to this  
soon". Or "those will be replaced
anyway" or the common "genius trap": Not doing the small increment  
because there are much better, much more general solution thinkable  
(and when those are done,
even better, even more general one are thinkable, which in turn means  
to postpone again. And again, to infinity...)

If we look into the history of Squeak, we see that those grand  
schemes seldom where realized. Or if they are realized, it takes a  
long time, with both
old an new used in parallel.

So I have come to the conclusion that the "not perfect unimportant  
small improvement" is indeed very important.

- Even if you plan to throw out a subsystem next week, do fix the  
trivial bug in it today.
- Even if something will be obsolete soon, include the improvements  
that have already been done.

That does not mean to stop improving on the grand scale: Improvement  
needs to be continuous, on all levels, all the time. Beeing able to  
do change in the
small is the pre-requesite of beeing able to do change in the large.

> but, if I'm correct, I'd be remiss if I didn't point this out.
>

Yes, indeed. thanks a lot! I will put it on mantis so it's not lost.  
Maybe someone find the time (or necesity) to implement it.

        Marcus

Le Dimanche 19 Février 2006 11:45, nicolas cellier a écrit :
I answer to myself: it seems that the case cannot happen:

number of inst var is protected at class creation
at ClassBuilder>>#computeFormat:instSize:forSuper:ccIndex:

number of literals is protected at
MethodNode>>#generate:

number of temps (and/or args) is protected at:
CompiledMethod>>#newBytes:trailerBytes:nArgs:nTemps:nStack:nLits:primitive:

However tricky as i can be, i saw that automatic temporaries allocated for
optimizing to:do: loops are not counted in this last protection.
So i gathered 64 declared temps and a to:do: loop, and i reached code
  self halt.  "Shouldn't get here"
in TempVariableNode>>#sizeForStorePop:

I am close to the bug, but cannot reveal it because the limit of temporaries
is not 256 as i stated incorrectly but 64. That's what i call luck.

So no need to protect again in LeafNode>>#code:type:

However, i would really prefer a notification of exceeded number of temps
rather than this delicious halt with shouldn't get here comment...

By the way, if i proceed the halt, i am saved from crash by a MNU
#adaptToInteger:andSend:.

I did post on mantis (http://bugs.impara.de/view.php?id=2920) for that.
I did not include a patch: better think of rewriting some part of code
generation rather than patching.

On 19.02.2006, at 15:44, nicolas cellier wrote:
>
>
> I did post on mantis (http://bugs.impara.de/view.php?id=2920) for  
> that.
> I did not include a patch: better think of rewriting some part of code
> generation rather than patching.


I actually quite like the code-generator that Anthony Hannan did for
the ClosureCompiler... (IRBuilder). Simple to re-use for other  
compilers,
overall quite fun to hack on.

       Marcus

Le Dimanche 19 Février 2006 15:57, vous avez écrit :
Then it seems that i would better rely on this one. I will upload.
But i saw that some work is on progress on 3.9 image, and that seems to be on
the "old" compiler...

What are the Squeak plans ?
Include anthony's in the future base image ?
Does it need more testing first ?
Are there any pros and cons i should be aware of ?

Nicolas

On 19.02.2006, at 16:11, nicolas cellier wrote:


>
> Then it seems that i would better rely on this one. I will upload.
> But i saw that some work is on progress on 3.9 image, and that  
> seems to be on
> the "old" compiler...
>

We should not stop fixing the old Compiler just because there might  
be something
else in the future.

The new Compiler's last version requires 3.9a (not the newest, I need  
to fix a bunch
of things for that). I should make it simpler to obtain an image.

I'l make a page describing the process and make a ready to use image.  
The best
thing right now for playing is to dowload the ByteSurgeon image:

   http://www.iam.unibe.ch/~scg/Research/ByteSurgeon/

But this will change: The new Compiler will be a simple package to  
load in 3.9a
very soon.

> What are the Squeak plans ?

Good Question.... I'm slowly working on getting the compiler into  
shape for
beeing in theory a replacement. But I have to admit that efforts  
stalled a
bit after I got it far enough for the experiments I wanted to do...

(Which included to make the compiler not closure-only, now it's  
configurable
by a preference to either generate old-style blocks or BlockClosures).

> Include anthony's in the future base image ?

I would like to see it replacing the old one eventually, but for that  
some
work needs to be done that I didn't yet do... e.g. we need
better error-reporting for SmaCC based parsers, and boostrapping  
(recreating
parser with SmaCC) need old compiler right now.
And of course bugs need to be weeded out.

> Does it need more testing first ?

Definitly.

> Are there any pros and cons i should be aware of ?

Pro:
  - much easier to understand and change.
  - provides a lot of very interesting infrastructure
    vital for what Squeak is all about (inventing the future):
     -- better AST Design (parent pointer, visitor)
        (perfect for eToy like stuff, syntax experiments, pretty  
printers, analyzers)
     -- Backend (IRBuilder) can be used independendly (kind of a  
"Squeak Macro Assember")
         (it's even retargable towards the actual bytecode set used,  
cool for experimenting
          with hacked VMs, makes VM evolution easier).
     -- SmaCC based Parser. Easy to adapt and change.
     -- easy manipulation of bytecode with ByteSurgeon
- optional Block Closure support.

Con:
  - Compiling is slower (factor 3)
  - Consists of much more Classes ;-)
  - error reporting worse then the old one.
  - Bugs (e.g. decompiling not yet 100% working)


    Marcus