help needed with the new FFI plugin...

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

help needed with the new FFI plugin...

Eliot Miranda-2
 
Hi All,

    today I committed the threaded FFI plugin along with the threaded VM and hope to be able to support people in doing threaded FFI work on the x86 platforms really soon now (Esteban, amongst others, this means you ;) ).  But to put the image-level code that uses the new FFI plugin into production the standard VM needs to be upgraded to not crash with the ExternalFunction image changes the new threaded FFI plugin wants.  The main change is that ExternalFunction changes size, gaining a stackSize inst var which is used to tell the FFI plugin how much stack spacer to reserve for marshalling (its computed on first use).  The best way for this to happen is to start using the ThreadedFFIPlugin as the standard plugin in the Interpreter VM. The ThreadedFFIPlugin overall is simpler than the old FFI plugin (no assembler support code) and better (fully reentrant).  But for this to happen we may need to implement the platform=specific parts of the ThreadedFFIPlugin (still entirely Smalltalk code) for PowerPC and ARM (and perhaps others, SPARC??).  I understand what has to be done and I can explain it to anyone interested in working on the problem, but I don't have any hardware with which I can test the results so its pretty pointless my doing the work and producing something inevitably broken.  So are there people out there who would be interested in doing the port to PPC and ARM?  It means fleshing out the classes 
    ThreadedARMFFIPlugin ThreadedFFICalloutStateForARM
    ThreadedPPCBEFFIPlugin ThreadedFFICalloutStateForPPC
and having a solid understanding of how alloca works to do stack allocation and how the ABIs on ARM and PPC pass registers.

here's hoping there are a few good low-level people who would be interested.
Let me know.
--
best,
Eliot

Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] help needed with the new FFI plugin...

Igor Stasenko

On 22 July 2011 02:34, Eliot Miranda <[hidden email]> wrote:

> Hi All,
>     today I committed the threaded FFI plugin along with the threaded VM and
> hope to be able to support people in doing threaded FFI work on the x86
> platforms really soon now (Esteban, amongst others, this means you ;) ).
>  But to put the image-level code that uses the new FFI plugin into
> production the standard VM needs to be upgraded to not crash with the
> ExternalFunction image changes the new threaded FFI plugin wants.  The main
> change is that ExternalFunction changes size, gaining a stackSize inst var
> which is used to tell the FFI plugin how much stack spacer to reserve for
> marshalling (its computed on first use).  The best way for this to happen is
> to start using the ThreadedFFIPlugin as the standard plugin in the
> Interpreter VM. The ThreadedFFIPlugin overall is simpler than the old FFI
> plugin (no assembler support code) and better (fully reentrant).  But for
> this to happen we may need to implement the platform=specific parts of
> the ThreadedFFIPlugin (still entirely Smalltalk code) for PowerPC and ARM
> (and perhaps others, SPARC??).  I understand what has to be done and I can
> explain it to anyone interested in working on the problem, but I don't have
> any hardware with which I can test the results so its pretty pointless my
> doing the work and producing something inevitably broken.  So are there
> people out there who would be interested in doing the port to PPC and ARM?
>  It means fleshing out the classes
>     ThreadedARMFFIPlugin ThreadedFFICalloutStateForARM
>     ThreadedPPCBEFFIPlugin ThreadedFFICalloutStateForPPC
> and having a solid understanding of how alloca works to do stack allocation
> and how the ABIs on ARM and PPC pass registers.
> here's hoping there are a few good low-level people who would be interested.
> Let me know.

I wonder if you find anyone today using these archs, not saying using
squeak/pharo on them + knowing low-level details + wanting to help :)

> --
> best,
> Eliot
>

--
Best regards,
Igor Stasenko AKA sig.
Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: [Pharo-project] help needed with the new FFI plugin...

stephane ducasse-2
In reply to this post by Eliot Miranda-2

+1

On Jul 22, 2011, at 10:07 AM, Janko Mivšek wrote:

> Wow, Eliot, thanks for that so long waited function a lot!
>
> Threaded FFI is namelly a must for Squeak/Pharo to be used seriously
> everywhere where a connection to the other worlds is needed. Which is a
> common situation when working on "enterprise" level information systems.
>
> Thanks again! Thanks also to all investors/donators who invested to this
> so important work!
>
> Janko
>
> S, Eliot Miranda piše:
>> Hi All,
>>
>>    today I committed the threaded FFI plugin along with the threaded VM
>> and hope to be able to support people in doing threaded FFI work on the
>> x86 platforms really soon now (Esteban, amongst others, this means you
>> ;) ).  But to put the image-level code that uses the new FFI plugin into
>> production the standard VM needs to be upgraded to not crash with the
>> ExternalFunction image changes the new threaded FFI plugin wants.  The
>> main change is that ExternalFunction changes size, gaining a stackSize
>> inst var which is used to tell the FFI plugin how much stack spacer to
>> reserve for marshalling (its computed on first use).  The best way for
>> this to happen is to start using the ThreadedFFIPlugin as the standard
>> plugin in the Interpreter VM. The ThreadedFFIPlugin overall is simpler
>> than the old FFI plugin (no assembler support code) and better (fully
>> reentrant).  But for this to happen we may need to implement the
>> platform=specific parts of the ThreadedFFIPlugin (still entirely
>> Smalltalk code) for PowerPC and ARM (and perhaps others, SPARC??).  I
>> understand what has to be done and I can explain it to anyone interested
>> in working on the problem, but I don't have any hardware with which I
>> can test the results so its pretty pointless my doing the work and
>> producing something inevitably broken.  So are there people out there
>> who would be interested in doing the port to PPC and ARM?  It means
>> fleshing out the classes
>>    ThreadedARMFFIPlugin ThreadedFFICalloutStateForARM
>>    ThreadedPPCBEFFIPlugin ThreadedFFICalloutStateForPPC
>> and having a solid understanding of how alloca works to do stack
>> allocation and how the ABIs on ARM and PPC pass registers.
>>
>> here's hoping there are a few good low-level people who would be interested.
>> Let me know.
>> --
>> best,
>> Eliot
>>
>
> --
> Janko Mivšek
> Aida/Web
> Smalltalk Web Application Server
> http://www.aidaweb.si
>

Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] help needed with the new FFI plugin...

stephane ducasse-2
In reply to this post by Eliot Miranda-2

In september Guido and Javier will be visiting us for 3 months and one of them will be payed to work on the project
we exchanged about some months ago (header C parser and ....)
So at least we will get that.

Stef

On Jul 22, 2011, at 1:34 AM, Eliot Miranda wrote:

> Hi All,
>
>     today I committed the threaded FFI plugin along with the threaded VM and hope to be able to support people in doing threaded FFI work on the x86 platforms really soon now (Esteban, amongst others, this means you ;) ).  But to put the image-level code that uses the new FFI plugin into production the standard VM needs to be upgraded to not crash with the ExternalFunction image changes the new threaded FFI plugin wants.  The main change is that ExternalFunction changes size, gaining a stackSize inst var which is used to tell the FFI plugin how much stack spacer to reserve for marshalling (its computed on first use).  The best way for this to happen is to start using the ThreadedFFIPlugin as the standard plugin in the Interpreter VM. The ThreadedFFIPlugin overall is simpler than the old FFI plugin (no assembler support code) and better (fully reentrant).  But for this to happen we may need to implement the platform=specific parts of the ThreadedFFIPlugin (still entirely Smalltalk code) for PowerPC and ARM (and perhaps others, SPARC??).  I understand what has to be done and I can explain it to anyone interested in working on the problem, but I don't have any hardware with which I can test the results so its pretty pointless my doing the work and producing something inevitably broken.  So are there people out there who would be interested in doing the port to PPC and ARM?  It means fleshing out the classes
>     ThreadedARMFFIPlugin ThreadedFFICalloutStateForARM
>     ThreadedPPCBEFFIPlugin ThreadedFFICalloutStateForPPC
> and having a solid understanding of how alloca works to do stack allocation and how the ABIs on ARM and PPC pass registers.
>
> here's hoping there are a few good low-level people who would be interested.
> Let me know.
> --
> best,
> Eliot
>
>

Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] help needed with the new FFI plugin...

Eliot Miranda-2
In reply to this post by Igor Stasenko
 


On Fri, Jul 22, 2011 at 9:08 AM, Ricardo Moran <[hidden email]> wrote:
Yes, indeed. Right now, for most use cases we use the SerialPort class, but we still have some FFI code for other stuff hanging around. I plan to write it as a plugin as soon as I have the time.

But getting the FFI working on ARM and/or PPC is more effective than writing yet another plugin, no?  We can always add security to the FFI (essentially some way of disabling internal calls within the FFI plugin, e.g. based on cryptographic keys to enable it, or whatever).  So I really do appeal to people to help me put the effort in here.  I can't justify working on the FFI on ARM or PPC in my Cadence Newspeak gig; we're entirely on x86.  I /can/ work on the threaded FFI as we need it at Cadence, just as many other people need it.  But for me to move the threaded FFI ahead it needs to be on all platforms and that means I need help on the platforms I can't justify working on.

So folks, please give serious consideration to putting effort into something that may be a little harder at first but will yield major benefits slightly later.  Let's defer gratification :)

 
Cheers,
RIcho


On Fri, Jul 22, 2011 at 10:37 AM, Bert Freudenberg <[hidden email]> wrote:
It's not part of Etoys yet. If its low-level interface indeed requires FFI, then that interface would have to be rewritten as plugin for Etoys to include it.

- Bert -

On 22.07.2011, at 14:55, karl ramberg wrote:

What about the Physical Etoys project ?
 
Karl

On Fri, Jul 22, 2011 at 2:20 PM, Bert Freudenberg <[hidden email]> wrote:

On 22.07.2011, at 02:10, Eliot Miranda wrote:

>
>
> On Thu, Jul 21, 2011 at 4:57 PM, Igor Stasenko <[hidden email]> wrote:
> On 22 July 2011 02:34, Eliot Miranda <[hidden email]> wrote:
> > Hi All,
> >     today I committed the threaded FFI plugin along with the threaded VM and
> > hope to be able to support people in doing threaded FFI work on the x86
> > platforms really soon now (Esteban, amongst others, this means you ;) ).
> >  But to put the image-level code that uses the new FFI plugin into
> > production the standard VM needs to be upgraded to not crash with the
> > ExternalFunction image changes the new threaded FFI plugin wants.  The main
> > change is that ExternalFunction changes size, gaining a stackSize inst var
> > which is used to tell the FFI plugin how much stack spacer to reserve for
> > marshalling (its computed on first use).  The best way for this to happen is
> > to start using the ThreadedFFIPlugin as the standard plugin in the
> > Interpreter VM. The ThreadedFFIPlugin overall is simpler than the old FFI
> > plugin (no assembler support code) and better (fully reentrant).  But for
> > this to happen we may need to implement the platform=specific parts of
> > the ThreadedFFIPlugin (still entirely Smalltalk code) for PowerPC and ARM
> > (and perhaps others, SPARC??).  I understand what has to be done and I can
> > explain it to anyone interested in working on the problem, but I don't have
> > any hardware with which I can test the results so its pretty pointless my
> > doing the work and producing something inevitably broken.  So are there
> > people out there who would be interested in doing the port to PPC and ARM?
> >  It means fleshing out the classes
> >     ThreadedARMFFIPlugin ThreadedFFICalloutStateForARM
> >     ThreadedPPCBEFFIPlugin ThreadedFFICalloutStateForPPC
> > and having a solid understanding of how alloca works to do stack allocation
> > and how the ABIs on ARM and PPC pass registers.
> > here's hoping there are a few good low-level people who would be interested.
> > Let me know.
>
> I wonder if you find anyone today using these archs, not saying using
> squeak/pharo on them + knowing low-level details + wanting to help :)
>
> Right, chances are thin.  But then if no one is interested then the work doesn't need to be done.  Bert, you mentioned the other day that PPC was still relevant for etoys.  But am I right in thinking etoys never uses the FFI?

Right. Etoys should run on both ARM and PPC, but it does not use FFI.

- Bert -

















--
best,
Eliot

Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] help needed with the new FFI plugin...

stephane ducasse-2


On Jul 22, 2011, at 6:25 PM, Eliot Miranda wrote:

>
>
> On Fri, Jul 22, 2011 at 9:08 AM, Ricardo Moran <[hidden email]> wrote:
> Yes, indeed. Right now, for most use cases we use the SerialPort class, but we still have some FFI code for other stuff hanging around. I plan to write it as a plugin as soon as I have the time.
>
> But getting the FFI working on ARM and/or PPC is more effective than writing yet another plugin, no?  We can always add security to the FFI (essentially some way of disabling internal calls within the FFI plugin, e.g. based on cryptographic keys to enable it, or whatever).  So I really do appeal to people to help me put the effort in here.  I can't justify working on the FFI on ARM or PPC in my Cadence Newspeak gig; we're entirely on x86.  I /can/ work on the threaded FFI as we need it at Cadence, just as many other people need it.  But for me to move the threaded FFI ahead it needs to be on all platforms and that means I need help on the platforms I can't justify working on.
>
> So folks, please give serious consideration to putting effort into something that may be a little harder at first but will yield major benefits slightly later.  Let's defer gratification :)

Personally we would love but to do that you need to know.
This is the key question behind increasing the mass of people knowing. This is why we organize the School and why the blog of mariano is good or the effort of igor to
make everybody been able to compile the vm. It is a slow and long process.
There is a difference between will and can. At least to me there are plenty of things I would like to do but I'm quite sure I will not be able to do them in this life.

Stef
Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] help needed with the new FFI plugin...

Ricardo Moran
In reply to this post by Eliot Miranda-2
 


On Fri, Jul 22, 2011 at 1:25 PM, Eliot Miranda <[hidden email]> wrote:
 


On Fri, Jul 22, 2011 at 9:08 AM, Ricardo Moran <[hidden email]> wrote:
Yes, indeed. Right now, for most use cases we use the SerialPort class, but we still have some FFI code for other stuff hanging around. I plan to write it as a plugin as soon as I have the time.

But getting the FFI working on ARM and/or PPC is more effective than writing yet another plugin, no?  We can always add security to the FFI (essentially some way of disabling internal calls within the FFI plugin, e.g. based on cryptographic keys to enable it, or whatever).  So I really do appeal to people to help me put the effort in here.  I can't justify working on the FFI on ARM or PPC in my Cadence Newspeak gig; we're entirely on x86.  I /can/ work on the threaded FFI as we need it at Cadence, just as many other people need it.  But for me to move the threaded FFI ahead it needs to be on all platforms and that means I need help on the platforms I can't justify working on.

So folks, please give serious consideration to putting effort into something that may be a little harder at first but will yield major benefits slightly later.  Let's defer gratification :)


Hi Eliot,

I would love to help but I don't have the hardware nor the knowledge to do it. Sorry...

Cheers,
Richo
 
 
Cheers,
RIcho


On Fri, Jul 22, 2011 at 10:37 AM, Bert Freudenberg <[hidden email]> wrote:
It's not part of Etoys yet. If its low-level interface indeed requires FFI, then that interface would have to be rewritten as plugin for Etoys to include it.

- Bert -

On 22.07.2011, at 14:55, karl ramberg wrote:

What about the Physical Etoys project ?
 
Karl

On Fri, Jul 22, 2011 at 2:20 PM, Bert Freudenberg <[hidden email]> wrote:

On 22.07.2011, at 02:10, Eliot Miranda wrote:

>
>
> On Thu, Jul 21, 2011 at 4:57 PM, Igor Stasenko <[hidden email]> wrote:
> On 22 July 2011 02:34, Eliot Miranda <[hidden email]> wrote:
> > Hi All,
> >     today I committed the threaded FFI plugin along with the threaded VM and
> > hope to be able to support people in doing threaded FFI work on the x86
> > platforms really soon now (Esteban, amongst others, this means you ;) ).
> >  But to put the image-level code that uses the new FFI plugin into
> > production the standard VM needs to be upgraded to not crash with the
> > ExternalFunction image changes the new threaded FFI plugin wants.  The main
> > change is that ExternalFunction changes size, gaining a stackSize inst var
> > which is used to tell the FFI plugin how much stack spacer to reserve for
> > marshalling (its computed on first use).  The best way for this to happen is
> > to start using the ThreadedFFIPlugin as the standard plugin in the
> > Interpreter VM. The ThreadedFFIPlugin overall is simpler than the old FFI
> > plugin (no assembler support code) and better (fully reentrant).  But for
> > this to happen we may need to implement the platform=specific parts of
> > the ThreadedFFIPlugin (still entirely Smalltalk code) for PowerPC and ARM
> > (and perhaps others, SPARC??).  I understand what has to be done and I can
> > explain it to anyone interested in working on the problem, but I don't have
> > any hardware with which I can test the results so its pretty pointless my
> > doing the work and producing something inevitably broken.  So are there
> > people out there who would be interested in doing the port to PPC and ARM?
> >  It means fleshing out the classes
> >     ThreadedARMFFIPlugin ThreadedFFICalloutStateForARM
> >     ThreadedPPCBEFFIPlugin ThreadedFFICalloutStateForPPC
> > and having a solid understanding of how alloca works to do stack allocation
> > and how the ABIs on ARM and PPC pass registers.
> > here's hoping there are a few good low-level people who would be interested.
> > Let me know.
>
> I wonder if you find anyone today using these archs, not saying using
> squeak/pharo on them + knowing low-level details + wanting to help :)
>
> Right, chances are thin.  But then if no one is interested then the work doesn't need to be done.  Bert, you mentioned the other day that PPC was still relevant for etoys.  But am I right in thinking etoys never uses the FFI?

Right. Etoys should run on both ARM and PPC, but it does not use FFI.

- Bert -

















--
best,
Eliot



Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] help needed with the new FFI plugin...

Levente Uzonyi-2
In reply to this post by Eliot Miranda-2
 
On Fri, 22 Jul 2011, Eliot Miranda wrote:

> But getting the FFI working on ARM and/or PPC is more effective than writing yet another plugin, no?  We can always add security to the FFI (essentially some
> way of disabling internal calls within the FFI plugin, e.g. based on cryptographic keys to enable it, or whatever).  So I really do appeal to people to help me
> put the effort in here.  I can't justify working on the FFI on ARM or PPC in my Cadence Newspeak gig; we're entirely on x86.  I /can/ work on the threaded FFI
> as we need it at Cadence, just as many other people need it.  But for me to move the threaded FFI ahead it needs to be on all platforms and that means I need
> help on the platforms I can't justify working on.
>
> So folks, please give serious consideration to putting effort into something that may be a little harder at first but will yield major benefits slightly later.
> Let's defer gratification :)

QEMU can be used for emulating various CPUs, so if you don't have an ARM,
PPC or SPARC hardware, you can still develop/test FFI with it.

Threaded FFI is great, it opens new possibilities, but nowadays the
feature I miss the most is the ability to call functions in 64-bit
(x86_64) libraries (from a 32-bit VM/image).


Levente
Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] help needed with the new FFI plugin...

David T. Lewis
 
On Sat, Jul 23, 2011 at 01:50:49PM +0200, Levente Uzonyi wrote:
>
> Threaded FFI is great, it opens new possibilities, but nowadays the
> feature I miss the most is the ability to call functions in 64-bit
> (x86_64) libraries (from a 32-bit VM/image).

As far as I know, mixing 32-bit and 64-bit libraries in a single
executable (VM) is not feasible, at least not on Linux. Maybe it
would work in the case of statically linked libraries, I'm not
certain.

The traditional FFI does not work for 64-bit VMs, and I expect
that this needs to be resolved for any new FFI implementation as
well, because some of the issues were in the interpreter itself,
not just the FFI plugin. The issue also may affect other things,
e.g. Igor's proposed #primLoadModule primitive crashes the VM
if compiled for 64 bit.

I had intended to push the fixes for 64-bit FFI as a VM team
initiative, but it's a complex set of changes that require
coordination of VMMaker (oscog plus traditional), as well as
of all the platforms (oscog, trunk, all platforms) so IMO this
would be a lot to take on right now.

But I guess that I should ask - is there interest in implementing
fixes for 64-bit FFI?

Some background:

Mantis report and patches to implement the fixes, circa 2008. I
assume some bit rot will have set in by this time:
 <http://bugs.squeak.org/view.php?id=7237>

Status of the fixes, as tested on various platforms:
 <http://lists.squeakfoundation.org/pipermail/vm-dev/2008-May/001945.html>

The last discussion relative to VM team priorities:
 <http://lists.squeakfoundation.org/pipermail/vm-dev/2009-July/002853.html>

Dave