Re: [Pharo-project] Can OSProcess functionality be implemented using FFI instead of plugin?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
42 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] Can OSProcess functionality be implemented using FFI instead of plugin?

Eliot Miranda-2
 


On Sun, May 6, 2012 at 9:14 AM, Nicolas Cellier <[hidden email]> wrote:
2012/5/6 Igor Stasenko <[hidden email]>:
> On 6 May 2012 17:08, Nicolas Cellier <[hidden email]> wrote:
>> 2012/5/6 Guillermo Polito <[hidden email]>:
>>>
>>>
>>> On Sun, May 6, 2012 at 2:16 PM, Esteban Lorenzano <[hidden email]>
>>> wrote:
>>>>
>>>> <snip>
>>>> I'm more worried about having all-platforms-specific-stuff inside image...
>>>> but we can mitigate that with fuel, and making loadable packages when
>>>> running the image... I don't know, I'm just thinking while writing, so, this
>>>> is probably stupid :)
>>>>
>>>
>>> Just think how many times you took a development image and used it in
>>> several platforms.  At least I don't.  Same happened when I used Eclipse.  I
>>> didn't share my eclipses between systems.  Even, I had several eclipse
>>> installations with their own plugins (just like images, hehe).
>>>
>>> Probably with jenkins, metacello, and kernel/bootstrap we can generate
>>> distributions per platform (With the possibility of an all-in-one
>>> distribution for the ones who like that).
>>>
>>> Guille
>>
>> Yes, I understand that we can live without this feature...
>> - If we can reconstruct images easily (one of the goal of Pharo) - I
>> mean not only code, but any object (eventually with Fuel)
>> - If we solve the bootstrap problem (or if we can still prepare an
>> image for cross platform startup)
>> - If we don't forget to always talk (send messages) thru an abstract
>> layer, and never directly name the target library,
>>
>> Since I didn't have all these tools in the past, I were forced to use
>> development images across different platform a lot, and yes, it was
>> not following the mainstream rules (a la "we can reconstruct all from
>> scratch") but damn powerful.
>> For deploying applications, it also is very powerful and cheap.
>> Personally, I would feel sore to lose it.
>>
>
> But look at the root of what we are talking about: N bytes in VM
> versus M bytes in image to support certain functionality.
> I think if you need it, you will make sure that those bytes is there
> and properly packaged with you application.
>

Unfortunately, it's more than moving code...
What I mean is that when I need to pass an O_NONBLOCK flag to a FFI
call, it's going to be a problem because I have to know how this
information is encoded on each and every platform I want to support.

But there are solutions to this which mean you *don't* have to know.  I wrote a prototype for VisualWorks that maps a SharedPool to these externally defined variables.  Here's how it works.

For each group of C constants, e.g. i/o constants, one populates a subclasss of SharedPoolForC with the variables one wants to define, and in a class-side method one defines the set of include files per platform that one should pull in to evaluate the constants.  SharedPoolForC has code in it to automatically generate a C program, and compile it, e.g. to provide a shared library/dll for the current platform.  The C program is essentially a name-value dictionary that maps from the abstract name #O_NONBLOCK to the size and value for a particular platform.  SharedPoolForC also contains code to load the shared library/dll, extract the values and update the pool variables automatically.

The deployment scheme is as follows, at start-up the system asks each SharedPoolForC subclass to check the platform and see if the platform has changed.  If it hasn't changed, nothing needs to happen.  If it has changed the system attempts to locate the shared library/dll for the current platform (the platform name is embedded in the dll's name), and update the pool variables from that dll, raising an exception if unavailable (and the exception could be mapped into a warning or an error to suit).  So to deploy e.g. a one-click one needs to generate the set of dlls fort the platforms one wants to deploy on.

The development scheme is simply to run a method on the SharedPoolForC when one adds some class variables and/or changes the set of include files which turns the crank, generating, compiling and loading the C file to get the value(s) for the new variable(s).

An alternative scheme would generate a program that would print e.g. STON, which could be parsed or evaluated to compute the values.  This would have the advantage that the definitions of the values are readable and editable by mere humans.  So I think I'd discard the shared library/dll approach and keep it simple.


At least, in C code, I just care about the symbolic information and
have a relatively portable sentence.
To me that's one of the highest hurdle with FFI, because this is the
kind of complexity I wish I never cared of.
That's just a flavour of #define/#ifdef hell.
It can be worse if you want to interface with IPC (which has lot's of
different flavours).
Same for functions defined by macros that just use machine specific
structure layout... We can no longer use these structures as opaque
handles.

> You can ship your product and use it on multiple platforms with ease,
> granted that appropriate platform-specific code is loaded into your
> image.
> With distribution using VMs it's a bit different story: it is a
> barrier with high entry cost. Especially if you think about all those
> RPMs, which is controlled by third-party maintainers. It is not that
> easy to directly control it, and much much slower in case if you need
> to deal with some problems.
>

I agree on the principle, I always prefer to develop in FFI than hack
the VM, I'm far far more efficient on the former.
Nonetheless, I don't think FFI can magically solve all our problems.
In certain ways it can make them worse.

Nicolas

>> Nicolas
>>
>
>
>
> --
> Best regards,
> Igor Stasenko.
>




--
best,
Eliot

Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] Can OSProcess functionality be implemented using FFI instead of plugin?

Igor Stasenko

On 7 May 2012 20:15, Eliot Miranda <[hidden email]> wrote:

>
>
>
> On Sun, May 6, 2012 at 9:14 AM, Nicolas Cellier <[hidden email]> wrote:
>>
>> 2012/5/6 Igor Stasenko <[hidden email]>:
>> > On 6 May 2012 17:08, Nicolas Cellier <[hidden email]> wrote:
>> >> 2012/5/6 Guillermo Polito <[hidden email]>:
>> >>>
>> >>>
>> >>> On Sun, May 6, 2012 at 2:16 PM, Esteban Lorenzano <[hidden email]>
>> >>> wrote:
>> >>>>
>> >>>> <snip>
>> >>>> I'm more worried about having all-platforms-specific-stuff inside image...
>> >>>> but we can mitigate that with fuel, and making loadable packages when
>> >>>> running the image... I don't know, I'm just thinking while writing, so, this
>> >>>> is probably stupid :)
>> >>>>
>> >>>
>> >>> Just think how many times you took a development image and used it in
>> >>> several platforms.  At least I don't.  Same happened when I used Eclipse.  I
>> >>> didn't share my eclipses between systems.  Even, I had several eclipse
>> >>> installations with their own plugins (just like images, hehe).
>> >>>
>> >>> Probably with jenkins, metacello, and kernel/bootstrap we can generate
>> >>> distributions per platform (With the possibility of an all-in-one
>> >>> distribution for the ones who like that).
>> >>>
>> >>> Guille
>> >>
>> >> Yes, I understand that we can live without this feature...
>> >> - If we can reconstruct images easily (one of the goal of Pharo) - I
>> >> mean not only code, but any object (eventually with Fuel)
>> >> - If we solve the bootstrap problem (or if we can still prepare an
>> >> image for cross platform startup)
>> >> - If we don't forget to always talk (send messages) thru an abstract
>> >> layer, and never directly name the target library,
>> >>
>> >> Since I didn't have all these tools in the past, I were forced to use
>> >> development images across different platform a lot, and yes, it was
>> >> not following the mainstream rules (a la "we can reconstruct all from
>> >> scratch") but damn powerful.
>> >> For deploying applications, it also is very powerful and cheap.
>> >> Personally, I would feel sore to lose it.
>> >>
>> >
>> > But look at the root of what we are talking about: N bytes in VM
>> > versus M bytes in image to support certain functionality.
>> > I think if you need it, you will make sure that those bytes is there
>> > and properly packaged with you application.
>> >
>>
>> Unfortunately, it's more than moving code...
>> What I mean is that when I need to pass an O_NONBLOCK flag to a FFI
>> call, it's going to be a problem because I have to know how this
>> information is encoded on each and every platform I want to support.
>
>
> But there are solutions to this which mean you *don't* have to know.  I wrote a prototype for VisualWorks that maps a SharedPool to these externally defined variables.  Here's how it works.
>
> For each group of C constants, e.g. i/o constants, one populates a subclasss of SharedPoolForC with the variables one wants to define, and in a class-side method one defines the set of include files per platform that one should pull in to evaluate the constants.  SharedPoolForC has code in it to automatically generate a C program, and compile it, e.g. to provide a shared library/dll for the current platform.  The C program is essentially a name-value dictionary that maps from the abstract name #O_NONBLOCK to the size and value for a particular platform.  SharedPoolForC also contains code to load the shared library/dll, extract the values and update the pool variables automatically.
>
> The deployment scheme is as follows, at start-up the system asks each SharedPoolForC subclass to check the platform and see if the platform has changed.  If it hasn't changed, nothing needs to happen.  If it has changed the system attempts to locate the shared library/dll for the current platform (the platform name is embedded in the dll's name), and update the pool variables from that dll, raising an exception if unavailable (and the exception could be mapped into a warning or an error to suit).  So to deploy e.g. a one-click one needs to generate the set of dlls fort the platforms one wants to deploy on.
>
> The development scheme is simply to run a method on the SharedPoolForC when one adds some class variables and/or changes the set of include files which turns the crank, generating, compiling and loading the C file to get the value(s) for the new variable(s).
>

Yes, i remember you told about it some years ago ;)

But this requires a platform with compiler installed. otherwise your
system won't be able to adopt to new environment. But of course there
is a solution to that as well: keep a database for every platform in
image,
and detect platform on startup, hoping that everything is in sync :)
And if not, then it usually means that guy using either non-standard
distribution, or custom built kernel etc.. At this point i would stop
worrying about it, since if he is good enough to be able to customize
own system, he can deal with FFI troubles in no time..
and usually those guys having compiler installed :)

Also,  if system supports binary distributions, then we're in same boat
as any other binary compiled and distributed for that platform.. (need
to elaborate that?)

> An alternative scheme would generate a program that would print e.g. STON, which could be parsed or evaluated to compute the values.  This would have the advantage that the definitions of the values are readable and editable by mere humans.  So I think I'd discard the shared library/dll approach and keep it simple.
>
>
Indeed. Again, generate C code.. a bit more code doesn't hurts.


--
Best regards,
Igor Stasenko.
Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] [Vm-dev] Re: Can OSProcess functionality be implemented using FFI instead of plugin?

Mariano Martinez Peck
 
Hi all,

Sorry for reviving an old thread but I thought it was better to continue the discussion here because of the context. 
As you may have read, the other day I released a first approeach to a subset of OSProcess based on FFI (posix_spwan() family of functions):


 And with that in mind, I wanted to share a few things with you. The main 2 problems I found with implementing this with FFI was:

1) We have all already agree and discussed that fork+exec cannot be done in separate FFI calls. So at the very min you need either a plugin method that does the fork()+exec() OR wrapping a lib like posix_spwan()

2) The other main problem, is, as you all said (and mostly  Nicolas), is the problems with the preprocessor (constants, macros, etc).

With all that said, I was able to get my stuff working. However, I am still using some primitives of OSProcess plugin because of 2). 

I read Eliot idea and what I don't like is the need of a C compiler in the user machine. I think that's a high constrain. Then Igor suggested that WE (developers and maintainers of a certain tool) are the ones that compiles the little C program to extract constant values etc and then WE provide as part of our source code, some packages with some SharedPool depending on the platform/OS. And Igor approach looked a bit better to me.

Then Nicolas made a point that if we plan to manage all that complexity at the image level it may become a hell too. 

So.... what if we take a simpler (probably not better) approach and we consider the "c program that exports constants and sizes" a VM Plugin? Let's say we have a UnixPreprocessorPlugin (that would work for OSX, Linux and other's Unix I imagine for the time being) which provides a function (that is exported) which answers an array of arrays. For each constant, we include the name of the constant, the value, and the sizeof().  Then from image side, we simply do one FFI call, we get the large array and we adapt it to a SharedPool or whatever kind of object representing that info. 

I know that different users will need different constants. But let's say the infrastructure (plugin etc) is already done. And let's say I am a user that I want to build something with FFI and I need some constants that I see are not defined. Then I can simply add the ones I need in the plugin, and next VM release will have those. If Cog gets moved to Github, then this is even easier. Everybody can do a PR with the constants he needs. And in fact, if we have the infrastructure in place, I think that we each of us spend half an hour, we may have almost everything we need. 

For example, I can add myself all those for signals (to use kill() from FFI), all those from fcntl (to make none blocking pipes), all those from wait()/waitpid() family (so that I can do a waitpid() with WNOHANG), etc etc etc.

I know it's not the best approach but it's something that could be done very easily and would allow A LOT of stuff to be moved to FFI just because we have no access to preprocess constants or sizeof()  (to know how to allocate). I also know this won't cover macros and other stuff. But still.

If you think this is a good idea, I can spend the time to do it. 

Cheers, 









On Thu, May 10, 2012 at 10:09 AM, Nick Ager <[hidden email]> wrote:
<snip>
Well, like opendbx, maybe because opengl has quite standard interface...
</snip>

and

<snip>
It's not that it's not doable, it's that we gonna reinvent gaz plant
and it gonna be so boring...
I'd like to see a proof of concept, even if we restrict to libc, libm,
kernel.dll, msvcrt.dll ...
</snip>

<snip>
Is the unix style select()
ubiquitous or should I use WaitForMultipleObject() on Windows? Are
specification of read/write streams implementation machine independant
(bsd/sysv/others...)
</snip>

Perhaps *a* way forward is to try to find existing projects which have already created cross-platform abstractions for platform specific functionality. Then we can use FFI to access that interface in a similar way to OpenGL and OpenDBX. For example NodeJs works across unixes - perhaps they have a useful cross-platform abstraction, boost  has abstractions of IPC etc 

Nick



--
Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] [Vm-dev] Re: Can OSProcess functionality be implemented using FFI instead of plugin?

Levente Uzonyi
 
On Sat, 16 Jan 2016, Mariano Martinez Peck wrote:

(No quote, thanks google.)

Forking a process is easy compared to communicating with it.
How would you do the latter with FFI?

Levente
Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] [Vm-dev] Re: Can OSProcess functionality be implemented using FFI instead of plugin?

Mariano Martinez Peck
 


On Sat, Jan 16, 2016 at 11:33 AM, Levente Uzonyi <[hidden email]> wrote:

On Sat, 16 Jan 2016, Mariano Martinez Peck wrote:

(No quote, thanks google.)

Forking a process is easy compared to communicating with it. How would you do the latter with FFI?

Hi Levente,

I am not sure if I understand the question. For OSSubprocess I use 2 ways of communicating the parent and the child, either with pipes or with regular files, both cases doing a dup2() of those to the standard streams. Same way as OSProcess does as far as I am concern.

Cheers,
 


--
Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] [Vm-dev] Re: Can OSProcess functionality be implemented using FFI instead of plugin?

David T. Lewis
In reply to this post by Mariano Martinez Peck
 
On Sat, Jan 16, 2016 at 11:00:39AM -0300, Mariano Martinez Peck wrote:
>
> So.... what if we take a simpler (probably not better) approach and we
> consider the "c program that exports constants and sizes" a VM Plugin?

I think that is a good approach. It gives you what you need, and you can
probably do it in slang without the need for any external C code.

Dave
Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] [Vm-dev] Re: Can OSProcess functionality be implemented using FFI instead of plugin?

kilon.alios
 
I think the most important thing is to create something that is easy to maintain. Also as much I appreciate doing the most in the image I also don't underestimate the elegance of C for low level stuff and the fact that it comes with great deal of documentation on the subject. Doesn't C offer already a solution to this problem ?

On Sat, 16 Jan 2016 at 23:28, David T. Lewis <[hidden email]> wrote:

On Sat, Jan 16, 2016 at 11:00:39AM -0300, Mariano Martinez Peck wrote:
>
> So.... what if we take a simpler (probably not better) approach and we
> consider the "c program that exports constants and sizes" a VM Plugin?

I think that is a good approach. It gives you what you need, and you can
probably do it in slang without the need for any external C code.

Dave
Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] [Vm-dev] Re: Can OSProcess functionality be implemented using FFI instead of plugin?

Levente Uzonyi
In reply to this post by Mariano Martinez Peck
 
On Sat, 16 Jan 2016, Mariano Martinez Peck wrote:

(Still no quote.)

How will you read the output of the process without having your image's
process blocked in the FFI callout?
How will you make sure that writes to input of the process won't block the
FFI callout?

Levente
Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] [Vm-dev] Re: Can OSProcess functionality be implemented using FFI instead of plugin?

Mariano Martinez Peck
 


On Sat, Jan 16, 2016 at 9:37 PM, Levente Uzonyi <[hidden email]> wrote:

On Sat, 16 Jan 2016, Mariano Martinez Peck wrote:

(Still no quote.)

How will you read the output of the process without having your image's process blocked in the FFI callout?
How will you make sure that writes to input of the process won't block the FFI callout


I don't see how your question are related to what I proposed of creating a plugin who exports constants. And I also don't understand what would change in that regard wether it is via FFI or plugin. Could you explain please?
 
In any case, for the reading, I simply use none blocking pipes. Something like this:

fcntl(descriptor, F_SETFL, flags | O_NONBLOCK)

That way the read operations won't block and simply answer what is available on the pipe. Then, from image side, is up to the user to decide how to get it. Could be a polling loop or whatever. 

Something I wanted to give it a try one day is to use blocking pipes but with threaded FFI callouts. But until that's in place, using non blocking pipes plus a image side polling is that I am using. 

The writing to the stdin I think it's blocking, although I think there is non blocking writing possibility too, but I never tried.

Does this answer your question?



--
Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] [Vm-dev] Re: Can OSProcess functionality be implemented using FFI instead of plugin?

Mariano Martinez Peck
In reply to this post by David T. Lewis
 


On Sat, Jan 16, 2016 at 6:28 PM, David T. Lewis <[hidden email]> wrote:

On Sat, Jan 16, 2016 at 11:00:39AM -0300, Mariano Martinez Peck wrote:
>
> So.... what if we take a simpler (probably not better) approach and we
> consider the "c program that exports constants and sizes" a VM Plugin?

I think that is a good approach. It gives you what you need, and you can
probably do it in slang without the need for any external C code.


Hi Dave,

Thanks! I guess I will only be able to give it a try if the rest also agrees that would be worth and to have some green light
for integration. 

BTW, I was investigating a bit over internet and it seems that it may be easier with X Macros?? See this for example:


Cheers! 



--
Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] [Vm-dev] Re: Can OSProcess functionality be implemented using FFI instead of plugin?

Eliot Miranda-2
In reply to this post by Mariano Martinez Peck
 
Hi Mariano,

On Sat, Jan 16, 2016 at 6:00 AM, Mariano Martinez Peck <[hidden email]> wrote:
 
Hi all,

Sorry for reviving an old thread but I thought it was better to continue the discussion here because of the context. 
As you may have read, the other day I released a first approeach to a subset of OSProcess based on FFI (posix_spwan() family of functions):


 And with that in mind, I wanted to share a few things with you. The main 2 problems I found with implementing this with FFI was:

1) We have all already agree and discussed that fork+exec cannot be done in separate FFI calls. So at the very min you need either a plugin method that does the fork()+exec() OR wrapping a lib like posix_spwan()

2) The other main problem, is, as you all said (and mostly  Nicolas), is the problems with the preprocessor (constants, macros, etc).

With all that said, I was able to get my stuff working. However, I am still using some primitives of OSProcess plugin because of 2). 

I read Eliot idea and what I don't like is the need of a C compiler in the user machine. I think that's a high constrain. Then Igor suggested that WE (developers and maintainers of a certain tool) are the ones that compiles the little C program to extract constant values etc and then WE provide as part of our source code, some packages with some SharedPool depending on the platform/OS. And Igor approach looked a bit better to me.

You misunderstand the proposal.  The C compiler is needed /only when changing the set of constants/, i.e. when /developing/ the interface.  The C compiler is /not/ needed when deploying.

The idea is to
a) at development time, e.g. when a new variable is added to a SharedPool containing platform constants, a C program is autogenerated that outputs in some format a description of the names and values of all the constants defined in the pool.  One convenient notation is e.g. STON.  For the purposes of this discussion let's assume we're using ston, but any format the image an parse (or indeed a shared object the image can load on teh current pkatform) will do.  The output of the autogenerated C program would be called something like <SharedPoolName>.<PlatformName>.ston, e.g. UnixConstants.MacOSX64.ston or UnixConstants.Linux32.ston.  The ston files can easily be parsed by facilities in the Smalltalk image.

b) when deploying the system to a set of platforms one includes all the relevant platform-specific ston files.

c) at startup the image checks its current platform.  If the platform is the same that it was saved on, no action is taken.  But if the platform as changed then the relevant ston file is selected, parsed, and the values for the variables in the shared pool updated to reflect the values of the current platform.

So the C compiler is only needed when developing the interface, not when deploying it.

Then Nicolas made a point that if we plan to manage all that complexity at the image level it may become a hell too. 

So.... what if we take a simpler (probably not better) approach and we consider the "c program that exports constants and sizes" a VM Plugin? Let's say we have a UnixPreprocessorPlugin (that would work for OSX, Linux and other's Unix I imagine for the time being) which provides a function (that is exported) which answers an array of arrays. For each constant, we include the name of the constant, the value, and the sizeof().  Then from image side, we simply do one FFI call, we get the large array and we adapt it to a SharedPool or whatever kind of object representing that info. 

This is what I suggestred in teh first place.  That what is autogenerated is a shared object (be it a plgin or a dll doesn't matter, it is machine code generated by a C compiler form an autogenerated C program compiled with the platform's C compiler) that can be loaded at run-time and interrogated to fetch the values of a set of variables.  ut I think that the textual notation suggested above is simpler.  The test files are easier to distribute and change.


I know that different users will need different constants. But let's say the infrastructure (plugin etc) is already done. And let's say I am a user that I want to build something with FFI and I need some constants that I see are not defined. Then I can simply add the ones I need in the plugin, and next VM release will have those. If Cog gets moved to Github, then this is even easier. Everybody can do a PR with the constants he needs. And in fact, if we have the infrastructure in place, I think that we each of us spend half an hour, we may have almost everything we need. 

For example, I can add myself all those for signals (to use kill() from FFI), all those from fcntl (to make none blocking pipes), all those from wait()/waitpid() family (so that I can do a waitpid() with WNOHANG), etc etc etc.

I know it's not the best approach but it's something that could be done very easily and would allow A LOT of stuff to be moved to FFI just because we have no access to preprocess constants or sizeof()  (to know how to allocate). I also know this won't cover macros and other stuff. But still.

If you think this is a good idea, I can spend the time to do it. 

Cheers, 









On Thu, May 10, 2012 at 10:09 AM, Nick Ager <[hidden email]> wrote:
<snip>
Well, like opendbx, maybe because opengl has quite standard interface...
</snip>

and

<snip>
It's not that it's not doable, it's that we gonna reinvent gaz plant
and it gonna be so boring...
I'd like to see a proof of concept, even if we restrict to libc, libm,
kernel.dll, msvcrt.dll ...
</snip>

<snip>
Is the unix style select()
ubiquitous or should I use WaitForMultipleObject() on Windows? Are
specification of read/write streams implementation machine independant
(bsd/sysv/others...)
</snip>

Perhaps *a* way forward is to try to find existing projects which have already created cross-platform abstractions for platform specific functionality. Then we can use FFI to access that interface in a similar way to OpenGL and OpenDBX. For example NodeJs works across unixes - perhaps they have a useful cross-platform abstraction, boost  has abstractions of IPC etc 

Nick



--




--
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] [Vm-dev] Re: Can OSProcess functionality be implemented using FFI instead of plugin?

Eliot Miranda-2
In reply to this post by Mariano Martinez Peck
 


On Sat, Jan 16, 2016 at 6:00 AM, Mariano Martinez Peck <[hidden email]> wrote:
 
Hi all,

Sorry for reviving an old thread but I thought it was better to continue the discussion here because of the context. 
As you may have read, the other day I released a first approeach to a subset of OSProcess based on FFI (posix_spwan() family of functions):


 And with that in mind, I wanted to share a few things with you. The main 2 problems I found with implementing this with FFI was:

1) We have all already agree and discussed that fork+exec cannot be done in separate FFI calls. So at the very min you need either a plugin method that does the fork()+exec() OR wrapping a lib like posix_spwan()

2) The other main problem, is, as you all said (and mostly  Nicolas), is the problems with the preprocessor (constants, macros, etc).

With all that said, I was able to get my stuff working. However, I am still using some primitives of OSProcess plugin because of 2). 

I read Eliot idea and what I don't like is the need of a C compiler in the user machine. I think that's a high constrain. Then Igor suggested that WE (developers and maintainers of a certain tool) are the ones that compiles the little C program to extract constant values etc and then WE provide as part of our source code, some packages with some SharedPool depending on the platform/OS. And Igor approach looked a bit better to me.



You misunderstand the proposal.  The C compiler is needed /only when changing the set of constants/, i.e. when /developing/ the interface.  The C compiler is /not/ needed when deploying.

The idea is to
a) at development time, e.g. when a new variable is added to a SharedPool containing platform constants, a C program is autogenerated that outputs in some format a description of the names and values of all the constants defined in the pool.  One convenient notation is e.g. STON.  For the purposes of this discussion let's assume we're using ston, but any format the image an parse (or indeed a shared object the image can load on teh current pkatform) will do.  The output of the autogenerated C program would be called something like <SharedPoolName>.<PlatformName>.ston, e.g. UnixConstants.MacOSX64.ston or UnixConstants.Linux32.ston.  The ston files can easily be parsed by facilities in the Smalltalk image.

b) when deploying the system to a set of platforms one includes all the relevant platform-specific ston files.

c) at startup the image checks its current platform.  If the platform is the same that it was saved on, no action is taken.  But if the platform as changed then the relevant ston file is selected, parsed, and the values for the variables in the shared pool updated to reflect the values of the current platform.

So the C compiler is only needed when developing the interface, not when deploying it.


Then Nicolas made a point that if we plan to manage all that complexity at the image level it may become a hell too. 

So.... what if we take a simpler (probably not better) approach and we consider the "c program that exports constants and sizes" a VM Plugin? Let's say we have a UnixPreprocessorPlugin (that would work for OSX, Linux and other's Unix I imagine for the time being) which provides a function (that is exported) which answers an array of arrays. For each constant, we include the name of the constant, the value, and the sizeof().  Then from image side, we simply do one FFI call, we get the large array and we adapt it to a SharedPool or whatever kind of object representing that info. 



This is what I suggestred in teh first place.  That what is autogenerated is a shared object (be it a plgin or a dll doesn't matter, it is machine code generated by a C compiler form an autogenerated C program compiled with the platform's C compiler) that can be loaded at run-time and interrogated to fetch the values of a set of variables.  But I think that the textual notation suggested above is simpler.  The test files are easier to distribute and change.  Shared objects and plugins have a habit of going stale, and there needs to be metadata in there to describe the set of constants etc, which is tricky to generate and parse because it is binary (pointer sizes, etc, etc).  Instead a simple textual format should be much more robust.  One could even edit by hand to add new constants.  It would be easy to make the textual file a versioned file.  Etc, etc.
 

I know that different users will need different constants. But let's say the infrastructure (plugin etc) is already done. And let's say I am a user that I want to build something with FFI and I need some constants that I see are not defined. Then I can simply add the ones I need in the plugin, and next VM release will have those. If Cog gets moved to Github, then this is even easier. Everybody can do a PR with the constants he needs. And in fact, if we have the infrastructure in place, I think that we each of us spend half an hour, we may have almost everything we need. 

For example, I can add myself all those for signals (to use kill() from FFI), all those from fcntl (to make none blocking pipes), all those from wait()/waitpid() family (so that I can do a waitpid() with WNOHANG), etc etc etc.

I know it's not the best approach but it's something that could be done very easily and would allow A LOT of stuff to be moved to FFI just because we have no access to preprocess constants or sizeof()  (to know how to allocate). I also know this won't cover macros and other stuff. But still.

If you think this is a good idea, I can spend the time to do it. 

Cheers, 









On Thu, May 10, 2012 at 10:09 AM, Nick Ager <[hidden email]> wrote:
<snip>
Well, like opendbx, maybe because opengl has quite standard interface...
</snip>

and

<snip>
It's not that it's not doable, it's that we gonna reinvent gaz plant
and it gonna be so boring...
I'd like to see a proof of concept, even if we restrict to libc, libm,
kernel.dll, msvcrt.dll ...
</snip>

<snip>
Is the unix style select()
ubiquitous or should I use WaitForMultipleObject() on Windows? Are
specification of read/write streams implementation machine independant
(bsd/sysv/others...)
</snip>

Perhaps *a* way forward is to try to find existing projects which have already created cross-platform abstractions for platform specific functionality. Then we can use FFI to access that interface in a similar way to OpenGL and OpenDBX. For example NodeJs works across unixes - perhaps they have a useful cross-platform abstraction, boost  has abstractions of IPC etc 

Nick



--




--
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] [Vm-dev] Re: Can OSProcess functionality be implemented using FFI instead of plugin?

Eliot Miranda-2
In reply to this post by Levente Uzonyi
 


On Sat, Jan 16, 2016 at 4:37 PM, Levente Uzonyi <[hidden email]> wrote:

On Sat, 16 Jan 2016, Mariano Martinez Peck wrote:

(Still no quote.)

How will you read the output of the process without having your image's process blocked in the FFI callout?
How will you make sure that writes to input of the process won't block the FFI callout?

This presupposes the threaded FFI.  The threaded FFI allows the VM to make any number of blocking calls, adding a new thread to run the VM whenever the VM is stalled when the heartbeat beats.  hence one can freely read and write to/from i/o blocking i/o streams (including pipes and sockets) or blocking database connexions, etc, all without stating that the FFI call must be done in a special way, since all calls through the FFI can block without blocking the VM.

Note that the scheme is also amenable to plugins, but the plugins must be rewritten to include the release vm/acquire vm calls around a blocking call.  With the threaded VM the FFI includes these calls around every FFI call.

HTH
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] [Vm-dev] Re: Can OSProcess functionality be implemented using FFI instead of plugin?

Mariano Martinez Peck
In reply to this post by Eliot Miranda-2
 


On Sat, Jan 16, 2016 at 11:02 PM, Eliot Miranda <[hidden email]> wrote:


On Sat, Jan 16, 2016 at 6:00 AM, Mariano Martinez Peck <[hidden email]> wrote:
 
Hi all,

Sorry for reviving an old thread but I thought it was better to continue the discussion here because of the context. 
As you may have read, the other day I released a first approeach to a subset of OSProcess based on FFI (posix_spwan() family of functions):


 And with that in mind, I wanted to share a few things with you. The main 2 problems I found with implementing this with FFI was:

1) We have all already agree and discussed that fork+exec cannot be done in separate FFI calls. So at the very min you need either a plugin method that does the fork()+exec() OR wrapping a lib like posix_spwan()

2) The other main problem, is, as you all said (and mostly  Nicolas), is the problems with the preprocessor (constants, macros, etc).

With all that said, I was able to get my stuff working. However, I am still using some primitives of OSProcess plugin because of 2). 

I read Eliot idea and what I don't like is the need of a C compiler in the user machine. I think that's a high constrain. Then Igor suggested that WE (developers and maintainers of a certain tool) are the ones that compiles the little C program to extract constant values etc and then WE provide as part of our source code, some packages with some SharedPool depending on the platform/OS. And Igor approach looked a bit better to me.



You misunderstand the proposal. 

I think I did. But let me confirm that below ;)
 
The C compiler is needed /only when changing the set of constants/, i.e. when /developing/ the interface.  The C compiler is /not/ needed when deploying.

The idea is to
a) at development time, e.g. when a new variable is added to a SharedPool containing platform constants, a C program is autogenerated that outputs in some format a description of the names and values of all the constants defined in the pool.  One convenient notation is e.g. STON.  For the purposes of this discussion let's assume we're using ston, but any format the image an parse (or indeed a shared object the image can load on teh current pkatform) will do.  The output of the autogenerated C program would be called something like <SharedPoolName>.<PlatformName>.ston, e.g. UnixConstants.MacOSX64.ston or UnixConstants.Linux32.ston.  The ston files can easily be parsed by facilities in the Smalltalk image.

b) when deploying the system to a set of platforms one includes all the relevant platform-specific ston files.


OK. But let me ask something. Below you said "be it a plugin or a dll doesn't matter". To autogenerate the C program, I must know which header files to include for each platform and probably a few others things. For example, besides exporting the value,  I would also like to export the sizeof(). At that depends how was the VM compiled, right?   So...my question is...if such a autogenerated C code could be part of the VM building (considering all the settings being assume when building), cannot I reuse the knowledge the VM already has? Like which header files to include, if it was compiled 32 bits or 64 bits, which C compiler to use, etc..

What I mean is if it would be easier if I take the SharedPool at VM building time, and from there I autogenerate (and run) the C code that would generate the output. Then, when we "deploy" the VM, we can deploy it with relevant platform specific ston files as you said. 

 
c) at startup the image checks its current platform.  If the platform is the same that it was saved on, no action is taken.  But if the platform as changed then the relevant ston file is selected, parsed, and the values for the variables in the shared pool updated to reflect the values of the current platform.

So the C compiler is only needed when developing the interface, not when deploying it.

 
OK 
 

Then Nicolas made a point that if we plan to manage all that complexity at the image level it may become a hell too. 

So.... what if we take a simpler (probably not better) approach and we consider the "c program that exports constants and sizes" a VM Plugin? Let's say we have a UnixPreprocessorPlugin (that would work for OSX, Linux and other's Unix I imagine for the time being) which provides a function (that is exported) which answers an array of arrays. For each constant, we include the name of the constant, the value, and the sizeof().  Then from image side, we simply do one FFI call, we get the large array and we adapt it to a SharedPool or whatever kind of object representing that info. 



This is what I suggestred in teh first place.  That what is autogenerated is a shared object (be it a plgin or a dll doesn't matter, it is machine code generated by a C compiler form an autogenerated C program compiled with the platform's C compiler) that can be loaded at run-time and interrogated to fetch the values of a set of variables

OK, got it. But still, it would be easier if the "platform" in this case is the "machine where we build the VM we will then distribute" right? i mean, I would like to put this in the CI jobs that automatically builds the VM, and not myself building for each platform. 

I mean, my main doubt is if this job of autogenerating C code, compile it, run it, export text file, and distribute text file with the VM, could be done as part of the VM building. 

 
.  But I think that the textual notation suggested above is simpler.  The test files are easier to distribute and change.  Shared objects and plugins have a habit of going stale, and there needs to be metadata in there to describe the set of constants etc, which is tricky to generate and parse because it is binary (pointer sizes, etc, etc).  Instead a simple textual format should be much more robust.  One could even edit by hand to add new constants.  It would be easy to make the textual file a versioned file.  Etc, etc.
 

OK. Got it. And do you think using X Macros for the autogenerated C (from the SharedPool) is a good idea?
And then I simply write a text file out of it. 

 

I know that different users will need different constants. But let's say the infrastructure (plugin etc) is already done. And let's say I am a user that I want to build something with FFI and I need some constants that I see are not defined. Then I can simply add the ones I need in the plugin, and next VM release will have those. If Cog gets moved to Github, then this is even easier. Everybody can do a PR with the constants he needs. And in fact, if we have the infrastructure in place, I think that we each of us spend half an hour, we may have almost everything we need. 

For example, I can add myself all those for signals (to use kill() from FFI), all those from fcntl (to make none blocking pipes), all those from wait()/waitpid() family (so that I can do a waitpid() with WNOHANG), etc etc etc.

I know it's not the best approach but it's something that could be done very easily and would allow A LOT of stuff to be moved to FFI just because we have no access to preprocess constants or sizeof()  (to know how to allocate). I also know this won't cover macros and other stuff. But still.

If you think this is a good idea, I can spend the time to do it. 

Cheers, 









On Thu, May 10, 2012 at 10:09 AM, Nick Ager <[hidden email]> wrote:
<snip>
Well, like opendbx, maybe because opengl has quite standard interface...
</snip>

and

<snip>
It's not that it's not doable, it's that we gonna reinvent gaz plant
and it gonna be so boring...
I'd like to see a proof of concept, even if we restrict to libc, libm,
kernel.dll, msvcrt.dll ...
</snip>

<snip>
Is the unix style select()
ubiquitous or should I use WaitForMultipleObject() on Windows? Are
specification of read/write streams implementation machine independant
(bsd/sysv/others...)
</snip>

Perhaps *a* way forward is to try to find existing projects which have already created cross-platform abstractions for platform specific functionality. Then we can use FFI to access that interface in a similar way to OpenGL and OpenDBX. For example NodeJs works across unixes - perhaps they have a useful cross-platform abstraction, boost  has abstractions of IPC etc 

Nick



--




--
_,,,^..^,,,_
best, Eliot



--
Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] [Vm-dev] Re: Can OSProcess functionality be implemented using FFI instead of plugin?

Eliot Miranda-2
 
Hi Mariano,

On Sat, Jan 16, 2016 at 6:25 PM, Mariano Martinez Peck <[hidden email]> wrote:


On Sat, Jan 16, 2016 at 11:02 PM, Eliot Miranda <[hidden email]> wrote:


On Sat, Jan 16, 2016 at 6:00 AM, Mariano Martinez Peck <[hidden email]> wrote:
 
Hi all,

Sorry for reviving an old thread but I thought it was better to continue the discussion here because of the context. 
As you may have read, the other day I released a first approeach to a subset of OSProcess based on FFI (posix_spwan() family of functions):


 And with that in mind, I wanted to share a few things with you. The main 2 problems I found with implementing this with FFI was:

1) We have all already agree and discussed that fork+exec cannot be done in separate FFI calls. So at the very min you need either a plugin method that does the fork()+exec() OR wrapping a lib like posix_spwan()

2) The other main problem, is, as you all said (and mostly  Nicolas), is the problems with the preprocessor (constants, macros, etc).

With all that said, I was able to get my stuff working. However, I am still using some primitives of OSProcess plugin because of 2). 

I read Eliot idea and what I don't like is the need of a C compiler in the user machine. I think that's a high constrain. Then Igor suggested that WE (developers and maintainers of a certain tool) are the ones that compiles the little C program to extract constant values etc and then WE provide as part of our source code, some packages with some SharedPool depending on the platform/OS. And Igor approach looked a bit better to me.



You misunderstand the proposal. 

I think I did. But let me confirm that below ;)
 
The C compiler is needed /only when changing the set of constants/, i.e. when /developing/ the interface.  The C compiler is /not/ needed when deploying.

The idea is to
a) at development time, e.g. when a new variable is added to a SharedPool containing platform constants, a C program is autogenerated that outputs in some format a description of the names and values of all the constants defined in the pool.  One convenient notation is e.g. STON.  For the purposes of this discussion let's assume we're using ston, but any format the image an parse (or indeed a shared object the image can load on teh current pkatform) will do.  The output of the autogenerated C program would be called something like <SharedPoolName>.<PlatformName>.ston, e.g. UnixConstants.MacOSX64.ston or UnixConstants.Linux32.ston.  The ston files can easily be parsed by facilities in the Smalltalk image.

b) when deploying the system to a set of platforms one includes all the relevant platform-specific ston files.


OK. But let me ask something. Below you said "be it a plugin or a dll doesn't matter". To autogenerate the C program, I must know which header files to include for each platform and probably a few others things. For example, besides exporting the value,  I would also like to export the sizeof(). At that depends how was the VM compiled, right?   So...my question is...if such a autogenerated C code could be part of the VM building (considering all the settings being assume when building), cannot I reuse the knowledge the VM already has? Like which header files to include, if it was compiled 32 bits or 64 bits, which C compiler to use, etc..

I actually said that using text is easier than a dll.  So I'm saying  autogenerate a C program that outputs name-value pairs in some convenient textual representation, e.g. ston.  But answering your question...

The knowledge in the VM as to what header files are included *applies only to the include files the VM uses*.  The VM uses a subset of the platform.  It doesn't for example include any headers that define a database interface.  It doesn't include header files that define the interface to a UI tooklit such at GTK.  Etc, etc.  So in fact the VM *doesn't* include the knowledge one needs to determine the set of include files for an arbitrary FFI interface.  And even so, the include files that it does use are in the VM's platform source files, and that information is not readily accessible.

Let me summarise.  No, the VM cannot be used to determine the set of include files needed to generate constants used in an arbitrary FFI interface.

What I mean is if it would be easier if I take the SharedPool at VM building time, and from there I autogenerate (and run) the C code that would generate the output. Then, when we "deploy" the VM, we can deploy it with relevant platform specific ston files as you said. 

No.  The VM is something that provides an FFI.  It doesn't *define* an FFI.   One must be able to develop an FFI interface without needing to rebuild the VM.  So computing the values of constants should be *separate* from building a VM.  Now let me give you more of an example.

Let's say we define a subclass of SharedPool called FFISharedPool.   FFISharedPool 's job is to manage autogenerating a C file, compiling it for the platform, and organizing parsing the relevant output.  Let's say we use a convention like class-side pragmas to define include files, and compiler flags.  The VM provides two crucial pieces of information:

1. the platform name
2. the word size

One can't run a Mac OS VM on Linux, and one can't run a 64-bit VM on a 32-bit operating system.  So taking this information from the VM accurately tells the current system what ABI (application binary interface) to use, and that's what's important in generating the right constants. 

So we use these two pieces of information to index the method pragmas that tell us what specific files to include.

Let's imagine we subclass FFISharedPool to add a shared pool for constants for an SQL database.  We might have a class declaration like

FFISharedPool subclass: #MYSQLInterface
instanceVariableNames: ''
classVariableNames: 'MYSQL_DEFAULT_AUTH MYSQL_ENABLE_CLEARTEXT_PLUGIN MYSQL_INIT_COMMAND MYSQL_OPT_BIND MYSQL_OPT_CAN_HANDLE_EXPIRED_PASSWORDS MYSQL_OPT_COMPRESS MYSQL_OPT_CONNECT_ATTR_DELETE MYSQL_OPT_CONNECT_ATTR_RESET'
poolDictionaries: ''
category: 'MYSQLInterface-Pools'

The job of FFISharedPool is to compute the right values for the class variables on every platform we want to deploy the MYSQL interface on.

So we need to know the relevant include files and C flags for each platform/word-size combination.  A few of them might look like


MYSQLInterface class methods for platform information
mac32
    "I describe the include files and C flags to use when developing a 32-bit MYSQL FFI interface on Mac OS X"
    <platformName: 'Mac OS' wordSize: 4>
    <cFlags: #('-m32') includeFiles: #('/opt/mysql/include32')>
    ^self "all the info is in the pragmas"

mac64
    "I describe the include files and C flags to use when developing a 64-bit MYSQL FFI interface on Mac OS X"
    <platformName: 'Mac OS' wordSize: 8>
    <cFlags: #('-m64') includeFiles: #('/opt/mysql/include64')>

The above might cause FFISharedPool to autogenerate files called MYSQLInterface.mac32.c & MYSQLInterface.mac64.c.  And these, when run, might output ston notation to MYSQLInterface.mac32.ston & MYSQLInterface.mac64.ston (or maybe to stdout which has to be redirected to MYSQLInterface.mac32.ston; whatever).

Now, you might use pragmas, or you might answer a Dictionary instance.  What ever style pleases you and seems convenient and readable.  But these methods define the necessary metadata (C flags, include paths, and ...?) for FFISharedPool to autogenerate the C program that, when compiled with the supplied C flags and run on the current platform, outputs the values for the constants the shared pool wants to define.


You can get fancy and have FFISharedPool autogenerate the C programs whenever one adds or removes a constant name.  Or you can require the programmer run something, e.g. MYSQLInterface generateInterfaces.  It's really nice if FFISharedPool submits the file to the C compiler automatically, but this can only work for e.g. 32 & 64 bit versions on a single platform.  You have to compile the autogenerated program on the relevant platform, with the necessary libraries and include files installed.

You could imagine a set of servers for different platforms so one could submit the autogenerated program for compilation and execution on each platform.  That's a facility I'd make it easy to implement.  I could imagine that a programmer whose company develops an FFI interface and deploys it on a number of platforms would love to be able to automate compiling and running the relevant autogenerated code on a set of servers.  I could imagine the Pharo community providing a set of servers upon which lots of software is installed for precisely this purpose. That means that people could develop FFI interfaces without even having to have the C compiler installed on their platform.

You could also add a C parser to FFISharedPool  that parses the post-preprocessed code and extracts function declarations.  But the important thing is autogenerating the C program so that it generates easily parsable output containing the values for the constants.  You can extend the system in interesting ways once you ave this core functionality implemented.

So once the program is autogenerated and compiled for the current platform, it is run and its output collected in a file whose name can be recognised by FFISharedPool.


Now the class side of FFISharedPool might be declared as

FFIShardPool class
instanceVariableNames: 'platformName wordSize'

and on start-up FFIShardPool could examine its subclasses, and for each whose platformName & wordSize do not match the current platform, search for all the matching FOOInterface.plat.ston files, parse them and update the subclasses' variables, and update that pool's platformName & wordSize.  It could emit a warning on the Transcript or stdout (headful vs headless) indicating which subclasses it couldn't find the relevant FOOInterface.plat.ston files for.

But the end result is that

a) providing the system is deployed with FOOInterface.plat.ston files for each interface and platform used, a cross-platform application can be deployed *that does not require a C compiler*.
b) providing that a system's FOOInterface files have been initialized on the intended platform, a platform-specific application can be deployed for a single platform *without needing the ston files*.

Does this make more sense now?

c) at startup the image checks its current platform.  If the platform is the same that it was saved on, no action is taken.  But if the platform as changed then the relevant ston file is selected, parsed, and the values for the variables in the shared pool updated to reflect the values of the current platform.

So the C compiler is only needed when developing the interface, not when deploying it.

 
OK 
 

Then Nicolas made a point that if we plan to manage all that complexity at the image level it may become a hell too. 

So.... what if we take a simpler (probably not better) approach and we consider the "c program that exports constants and sizes" a VM Plugin? Let's say we have a UnixPreprocessorPlugin (that would work for OSX, Linux and other's Unix I imagine for the time being) which provides a function (that is exported) which answers an array of arrays. For each constant, we include the name of the constant, the value, and the sizeof().  Then from image side, we simply do one FFI call, we get the large array and we adapt it to a SharedPool or whatever kind of object representing that info. 



This is what I suggestred in teh first place.  That what is autogenerated is a shared object (be it a plgin or a dll doesn't matter, it is machine code generated by a C compiler form an autogenerated C program compiled with the platform's C compiler) that can be loaded at run-time and interrogated to fetch the values of a set of variables

OK, got it. But still, it would be easier if the "platform" in this case is the "machine where we build the VM we will then distribute" right? i mean, I would like to put this in the CI jobs that automatically builds the VM, and not myself building for each platform. 

NO!  For example, why would a company that has some proprietary arithmetic package implemented in its secret labs in C or C++ and accessed through the FFI want to have that code on the Pharo community's build servers?
 

I mean, my main doubt is if this job of autogenerating C code, compile it, run it, export text file, and distribute text file with the VM, could be done as part of the VM building. 

For fuck's sake.  Developing an FFI is not something one does when building a VM.  It is something one does wen using the system.  f you want to do this you *use a plugin*.  The FFI is a different beast.  It is to allow programers to interface to external librarys that are *independent from teh VM*.

I'm not going to answer this one again.  OK?

 

 
.  But I think that the textual notation suggested above is simpler.  The test files are easier to distribute and change.  Shared objects and plugins have a habit of going stale, and there needs to be metadata in there to describe the set of constants etc, which is tricky to generate and parse because it is binary (pointer sizes, etc, etc).  Instead a simple textual format should be much more robust.  One could even edit by hand to add new constants.  It would be easy to make the textual file a versioned file.  Etc, etc.
 

OK. Got it. And do you think using X Macros for the autogenerated C (from the SharedPool) is a good idea?
And then I simply write a text file out of it. 

 

I know that different users will need different constants. But let's say the infrastructure (plugin etc) is already done. And let's say I am a user that I want to build something with FFI and I need some constants that I see are not defined. Then I can simply add the ones I need in the plugin, and next VM release will have those. If Cog gets moved to Github, then this is even easier. Everybody can do a PR with the constants he needs. And in fact, if we have the infrastructure in place, I think that we each of us spend half an hour, we may have almost everything we need. 

For example, I can add myself all those for signals (to use kill() from FFI), all those from fcntl (to make none blocking pipes), all those from wait()/waitpid() family (so that I can do a waitpid() with WNOHANG), etc etc etc.

I know it's not the best approach but it's something that could be done very easily and would allow A LOT of stuff to be moved to FFI just because we have no access to preprocess constants or sizeof()  (to know how to allocate). I also know this won't cover macros and other stuff. But still.

If you think this is a good idea, I can spend the time to do it. 

Cheers, 









On Thu, May 10, 2012 at 10:09 AM, Nick Ager <[hidden email]> wrote:
<snip>
Well, like opendbx, maybe because opengl has quite standard interface...
</snip>

and

<snip>
It's not that it's not doable, it's that we gonna reinvent gaz plant
and it gonna be so boring...
I'd like to see a proof of concept, even if we restrict to libc, libm,
kernel.dll, msvcrt.dll ...
</snip>

<snip>
Is the unix style select()
ubiquitous or should I use WaitForMultipleObject() on Windows? Are
specification of read/write streams implementation machine independant
(bsd/sysv/others...)
</snip>

Perhaps *a* way forward is to try to find existing projects which have already created cross-platform abstractions for platform specific functionality. Then we can use FFI to access that interface in a similar way to OpenGL and OpenDBX. For example NodeJs works across unixes - perhaps they have a useful cross-platform abstraction, boost  has abstractions of IPC etc 

Nick



--




--
_,,,^..^,,,_
best, Eliot



--



--
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] [Vm-dev] Re: Can OSProcess functionality be implemented using FFI instead of plugin?

Mariano Martinez Peck
 
Hi Eliot,

Thanks, much clearer now. Sometimes I am slow :)    I was confused because I was only thinking in libc kind of lib (very kernel and very likely used by the VM). But when you gave the SQL example, then I did get the general nature you were trying to explain. So it's clear now. 

I would like to add 2 more comments:

1) Do you agree that besides the name / value it would also help having the result of  sizeof ? Otherwise, I may still find problems when I need to allocate from FFI and it's not clear size of a struct (as it was my case same days ago).  So in this case, it would be kind of an array rather than a key / value pairs.


Thanks,


On Sun, Jan 17, 2016 at 12:40 AM, Eliot Miranda <[hidden email]> wrote:
Hi Mariano,

On Sat, Jan 16, 2016 at 6:25 PM, Mariano Martinez Peck <[hidden email]> wrote:


On Sat, Jan 16, 2016 at 11:02 PM, Eliot Miranda <[hidden email]> wrote:


On Sat, Jan 16, 2016 at 6:00 AM, Mariano Martinez Peck <[hidden email]> wrote:
 
Hi all,

Sorry for reviving an old thread but I thought it was better to continue the discussion here because of the context. 
As you may have read, the other day I released a first approeach to a subset of OSProcess based on FFI (posix_spwan() family of functions):


 And with that in mind, I wanted to share a few things with you. The main 2 problems I found with implementing this with FFI was:

1) We have all already agree and discussed that fork+exec cannot be done in separate FFI calls. So at the very min you need either a plugin method that does the fork()+exec() OR wrapping a lib like posix_spwan()

2) The other main problem, is, as you all said (and mostly  Nicolas), is the problems with the preprocessor (constants, macros, etc).

With all that said, I was able to get my stuff working. However, I am still using some primitives of OSProcess plugin because of 2). 

I read Eliot idea and what I don't like is the need of a C compiler in the user machine. I think that's a high constrain. Then Igor suggested that WE (developers and maintainers of a certain tool) are the ones that compiles the little C program to extract constant values etc and then WE provide as part of our source code, some packages with some SharedPool depending on the platform/OS. And Igor approach looked a bit better to me.



You misunderstand the proposal. 

I think I did. But let me confirm that below ;)
 
The C compiler is needed /only when changing the set of constants/, i.e. when /developing/ the interface.  The C compiler is /not/ needed when deploying.

The idea is to
a) at development time, e.g. when a new variable is added to a SharedPool containing platform constants, a C program is autogenerated that outputs in some format a description of the names and values of all the constants defined in the pool.  One convenient notation is e.g. STON.  For the purposes of this discussion let's assume we're using ston, but any format the image an parse (or indeed a shared object the image can load on teh current pkatform) will do.  The output of the autogenerated C program would be called something like <SharedPoolName>.<PlatformName>.ston, e.g. UnixConstants.MacOSX64.ston or UnixConstants.Linux32.ston.  The ston files can easily be parsed by facilities in the Smalltalk image.

b) when deploying the system to a set of platforms one includes all the relevant platform-specific ston files.


OK. But let me ask something. Below you said "be it a plugin or a dll doesn't matter". To autogenerate the C program, I must know which header files to include for each platform and probably a few others things. For example, besides exporting the value,  I would also like to export the sizeof(). At that depends how was the VM compiled, right?   So...my question is...if such a autogenerated C code could be part of the VM building (considering all the settings being assume when building), cannot I reuse the knowledge the VM already has? Like which header files to include, if it was compiled 32 bits or 64 bits, which C compiler to use, etc..

I actually said that using text is easier than a dll.  So I'm saying  autogenerate a C program that outputs name-value pairs in some convenient textual representation, e.g. ston.  But answering your question...

The knowledge in the VM as to what header files are included *applies only to the include files the VM uses*.  The VM uses a subset of the platform.  It doesn't for example include any headers that define a database interface.  It doesn't include header files that define the interface to a UI tooklit such at GTK.  Etc, etc.  So in fact the VM *doesn't* include the knowledge one needs to determine the set of include files for an arbitrary FFI interface.  And even so, the include files that it does use are in the VM's platform source files, and that information is not readily accessible.

Let me summarise.  No, the VM cannot be used to determine the set of include files needed to generate constants used in an arbitrary FFI interface.

What I mean is if it would be easier if I take the SharedPool at VM building time, and from there I autogenerate (and run) the C code that would generate the output. Then, when we "deploy" the VM, we can deploy it with relevant platform specific ston files as you said. 

No.  The VM is something that provides an FFI.  It doesn't *define* an FFI.   One must be able to develop an FFI interface without needing to rebuild the VM.  So computing the values of constants should be *separate* from building a VM.  Now let me give you more of an example.

Let's say we define a subclass of SharedPool called FFISharedPool.   FFISharedPool 's job is to manage autogenerating a C file, compiling it for the platform, and organizing parsing the relevant output.  Let's say we use a convention like class-side pragmas to define include files, and compiler flags.  The VM provides two crucial pieces of information:

1. the platform name
2. the word size

One can't run a Mac OS VM on Linux, and one can't run a 64-bit VM on a 32-bit operating system.  So taking this information from the VM accurately tells the current system what ABI (application binary interface) to use, and that's what's important in generating the right constants. 

So we use these two pieces of information to index the method pragmas that tell us what specific files to include.

Let's imagine we subclass FFISharedPool to add a shared pool for constants for an SQL database.  We might have a class declaration like

FFISharedPool subclass: #MYSQLInterface
instanceVariableNames: ''
classVariableNames: 'MYSQL_DEFAULT_AUTH MYSQL_ENABLE_CLEARTEXT_PLUGIN MYSQL_INIT_COMMAND MYSQL_OPT_BIND MYSQL_OPT_CAN_HANDLE_EXPIRED_PASSWORDS MYSQL_OPT_COMPRESS MYSQL_OPT_CONNECT_ATTR_DELETE MYSQL_OPT_CONNECT_ATTR_RESET'
poolDictionaries: ''
category: 'MYSQLInterface-Pools'

The job of FFISharedPool is to compute the right values for the class variables on every platform we want to deploy the MYSQL interface on.

So we need to know the relevant include files and C flags for each platform/word-size combination.  A few of them might look like


MYSQLInterface class methods for platform information
mac32
    "I describe the include files and C flags to use when developing a 32-bit MYSQL FFI interface on Mac OS X"
    <platformName: 'Mac OS' wordSize: 4>
    <cFlags: #('-m32') includeFiles: #('/opt/mysql/include32')>
    ^self "all the info is in the pragmas"

mac64
    "I describe the include files and C flags to use when developing a 64-bit MYSQL FFI interface on Mac OS X"
    <platformName: 'Mac OS' wordSize: 8>
    <cFlags: #('-m64') includeFiles: #('/opt/mysql/include64')>

The above might cause FFISharedPool to autogenerate files called MYSQLInterface.mac32.c & MYSQLInterface.mac64.c.  And these, when run, might output ston notation to MYSQLInterface.mac32.ston & MYSQLInterface.mac64.ston (or maybe to stdout which has to be redirected to MYSQLInterface.mac32.ston; whatever).

Now, you might use pragmas, or you might answer a Dictionary instance.  What ever style pleases you and seems convenient and readable.  But these methods define the necessary metadata (C flags, include paths, and ...?) for FFISharedPool to autogenerate the C program that, when compiled with the supplied C flags and run on the current platform, outputs the values for the constants the shared pool wants to define.


You can get fancy and have FFISharedPool autogenerate the C programs whenever one adds or removes a constant name.  Or you can require the programmer run something, e.g. MYSQLInterface generateInterfaces.  It's really nice if FFISharedPool submits the file to the C compiler automatically, but this can only work for e.g. 32 & 64 bit versions on a single platform.  You have to compile the autogenerated program on the relevant platform, with the necessary libraries and include files installed.

You could imagine a set of servers for different platforms so one could submit the autogenerated program for compilation and execution on each platform.  That's a facility I'd make it easy to implement.  I could imagine that a programmer whose company develops an FFI interface and deploys it on a number of platforms would love to be able to automate compiling and running the relevant autogenerated code on a set of servers.  I could imagine the Pharo community providing a set of servers upon which lots of software is installed for precisely this purpose. That means that people could develop FFI interfaces without even having to have the C compiler installed on their platform.

You could also add a C parser to FFISharedPool  that parses the post-preprocessed code and extracts function declarations.  But the important thing is autogenerating the C program so that it generates easily parsable output containing the values for the constants.  You can extend the system in interesting ways once you ave this core functionality implemented.

So once the program is autogenerated and compiled for the current platform, it is run and its output collected in a file whose name can be recognised by FFISharedPool.


Now the class side of FFISharedPool might be declared as

FFIShardPool class
instanceVariableNames: 'platformName wordSize'

and on start-up FFIShardPool could examine its subclasses, and for each whose platformName & wordSize do not match the current platform, search for all the matching FOOInterface.plat.ston files, parse them and update the subclasses' variables, and update that pool's platformName & wordSize.  It could emit a warning on the Transcript or stdout (headful vs headless) indicating which subclasses it couldn't find the relevant FOOInterface.plat.ston files for.

But the end result is that

a) providing the system is deployed with FOOInterface.plat.ston files for each interface and platform used, a cross-platform application can be deployed *that does not require a C compiler*.
b) providing that a system's FOOInterface files have been initialized on the intended platform, a platform-specific application can be deployed for a single platform *without needing the ston files*.

Does this make more sense now?

c) at startup the image checks its current platform.  If the platform is the same that it was saved on, no action is taken.  But if the platform as changed then the relevant ston file is selected, parsed, and the values for the variables in the shared pool updated to reflect the values of the current platform.

So the C compiler is only needed when developing the interface, not when deploying it.

 
OK 
 

Then Nicolas made a point that if we plan to manage all that complexity at the image level it may become a hell too. 

So.... what if we take a simpler (probably not better) approach and we consider the "c program that exports constants and sizes" a VM Plugin? Let's say we have a UnixPreprocessorPlugin (that would work for OSX, Linux and other's Unix I imagine for the time being) which provides a function (that is exported) which answers an array of arrays. For each constant, we include the name of the constant, the value, and the sizeof().  Then from image side, we simply do one FFI call, we get the large array and we adapt it to a SharedPool or whatever kind of object representing that info. 



This is what I suggestred in teh first place.  That what is autogenerated is a shared object (be it a plgin or a dll doesn't matter, it is machine code generated by a C compiler form an autogenerated C program compiled with the platform's C compiler) that can be loaded at run-time and interrogated to fetch the values of a set of variables

OK, got it. But still, it would be easier if the "platform" in this case is the "machine where we build the VM we will then distribute" right? i mean, I would like to put this in the CI jobs that automatically builds the VM, and not myself building for each platform. 

NO!  For example, why would a company that has some proprietary arithmetic package implemented in its secret labs in C or C++ and accessed through the FFI want to have that code on the Pharo community's build servers?
 

I mean, my main doubt is if this job of autogenerating C code, compile it, run it, export text file, and distribute text file with the VM, could be done as part of the VM building. 

For fuck's sake.  Developing an FFI is not something one does when building a VM.  It is something one does wen using the system.  f you want to do this you *use a plugin*.  The FFI is a different beast.  It is to allow programers to interface to external librarys that are *independent from teh VM*.

I'm not going to answer this one again.  OK?

 

 
.  But I think that the textual notation suggested above is simpler.  The test files are easier to distribute and change.  Shared objects and plugins have a habit of going stale, and there needs to be metadata in there to describe the set of constants etc, which is tricky to generate and parse because it is binary (pointer sizes, etc, etc).  Instead a simple textual format should be much more robust.  One could even edit by hand to add new constants.  It would be easy to make the textual file a versioned file.  Etc, etc.
 

OK. Got it. And do you think using X Macros for the autogenerated C (from the SharedPool) is a good idea?
And then I simply write a text file out of it. 

 

I know that different users will need different constants. But let's say the infrastructure (plugin etc) is already done. And let's say I am a user that I want to build something with FFI and I need some constants that I see are not defined. Then I can simply add the ones I need in the plugin, and next VM release will have those. If Cog gets moved to Github, then this is even easier. Everybody can do a PR with the constants he needs. And in fact, if we have the infrastructure in place, I think that we each of us spend half an hour, we may have almost everything we need. 

For example, I can add myself all those for signals (to use kill() from FFI), all those from fcntl (to make none blocking pipes), all those from wait()/waitpid() family (so that I can do a waitpid() with WNOHANG), etc etc etc.

I know it's not the best approach but it's something that could be done very easily and would allow A LOT of stuff to be moved to FFI just because we have no access to preprocess constants or sizeof()  (to know how to allocate). I also know this won't cover macros and other stuff. But still.

If you think this is a good idea, I can spend the time to do it. 

Cheers, 









On Thu, May 10, 2012 at 10:09 AM, Nick Ager <[hidden email]> wrote:
<snip>
Well, like opendbx, maybe because opengl has quite standard interface...
</snip>

and

<snip>
It's not that it's not doable, it's that we gonna reinvent gaz plant
and it gonna be so boring...
I'd like to see a proof of concept, even if we restrict to libc, libm,
kernel.dll, msvcrt.dll ...
</snip>

<snip>
Is the unix style select()
ubiquitous or should I use WaitForMultipleObject() on Windows? Are
specification of read/write streams implementation machine independant
(bsd/sysv/others...)
</snip>

Perhaps *a* way forward is to try to find existing projects which have already created cross-platform abstractions for platform specific functionality. Then we can use FFI to access that interface in a similar way to OpenGL and OpenDBX. For example NodeJs works across unixes - perhaps they have a useful cross-platform abstraction, boost  has abstractions of IPC etc 

Nick



--




--
_,,,^..^,,,_
best, Eliot



--



--
_,,,^..^,,,_
best, Eliot



--
Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] [Vm-dev] Re: Can OSProcess functionality be implemented using FFI instead of plugin?

Levente Uzonyi
In reply to this post by Mariano Martinez Peck
 
On Sat, 16 Jan 2016, Mariano Martinez Peck wrote:

(Again no quote.)

My question was obviously unrelated to the plugin with the constants.
I was just wondering if there's a reason to use FFI instead of OSProcess.
Based on your answers, I assume that
- Your solution is Unix/Mac only, since dup and dup2 don't work on
Windows. At least they didn't work when I wrote the ProcessWrapper plugin.
- You have the drawbacks of non-blocking streams: polling, repeated write
attempts.
- You have to solve the problem of different platforms (constant values,
library names and paths, etc), something that OSProcess handles by
default.

All this makes me wonder:
- Why do you want this FFI-based solution at all?
- How will it be better than OSProcess?
- If you decide to make a plugin as well to tackle some of the problems,
why make it a mixed solution instead of plugin-only?

Levente
Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: [Pharo-project] [Vm-dev] Re: Can OSProcess functionality be implemented using FFI instead of plugin?

Levente Uzonyi
In reply to this post by Eliot Miranda-2
 
On Sat, 16 Jan 2016, Eliot Miranda wrote:

>
>
> On Sat, Jan 16, 2016 at 4:37 PM, Levente Uzonyi <[hidden email]> wrote:
>
>       On Sat, 16 Jan 2016, Mariano Martinez Peck wrote:
>
>       (Still no quote.)
>
>       How will you read the output of the process without having your image's process blocked in the FFI callout?
>       How will you make sure that writes to input of the process won't block the FFI callout?
>
>
> This presupposes the threaded FFI.  The threaded FFI allows the VM to make any number of blocking calls, adding a new thread to run the VM whenever the VM is stalled when the heartbeat beats.  hence one can freely read and write to/from i/o blocking i/o streams
> (including pipes and sockets) or blocking database connexions, etc, all without stating that the FFI call must be done in a special way, since all calls through the FFI can block without blocking the VM.
I think it was you who said (in a discussion with Craig) that the threaded
FFI was not production ready. Is it ready for produciton now?

Levente

>
> Note that the scheme is also amenable to plugins, but the plugins must be rewritten to include the release vm/acquire vm calls around a blocking call.  With the threaded VM the FFI includes these calls around every FFI call.
>
> HTH
> _,,,^..^,,,_
> best, Eliot
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: [Pharo-project] [Vm-dev] Re: Can OSProcess functionality be implemented using FFI instead of plugin?

Eliot Miranda-2

Hi Levente,

> On Jan 17, 2016, at 7:30 AM, Levente Uzonyi <[hidden email]> wrote:
>
>> On Sat, 16 Jan 2016, Eliot Miranda wrote:
>>
>> On Sat, Jan 16, 2016 at 4:37 PM, Levente Uzonyi <[hidden email]> wrote:
>>
>>      On Sat, 16 Jan 2016, Mariano Martinez Peck wrote:
>>
>>      (Still no quote.)
>>
>>      How will you read the output of the process without having your image's process blocked in the FFI callout?
>>      How will you make sure that writes to input of the process won't block the FFI callout?
>> This presupposes the threaded FFI.  The threaded FFI allows the VM to make any number of blocking calls, adding a new thread to run the VM whenever the VM is stalled when the heartbeat beats.  hence one can freely read and write to/from i/o blocking i/o streams
>> (including pipes and sockets) or blocking database connexions, etc, all without stating that the FFI call must be done in a special way, since all calls through the FFI can block without blocking the VM.
>
> I think it was you who said (in a discussion with Craig) that the threaded FFI was not production ready. Is it ready for produciton now?

No, but I expect this is the year it will be.  Spur provides pinning, so the VM infrastructure is there.  The Pharo community plus some commercial relationships that have developed are providing funding.  Esteban Lorenzano and I want to collaborate on this and I hope to get help from some other people, such as Ronie Salgado.  And Mariano is working on an important part of the problem.  So I feel there's sufficient momentum for us to realize the threaded FFI this year.

> Levente


_,,,^..^,,,_ (phone)

>> Note that the scheme is also amenable to plugins, but the plugins must be rewritten to include the release vm/acquire vm calls around a blocking call.  With the threaded VM the FFI includes these calls around every FFI call.
>> HTH
>> _,,,^..^,,,_
>> best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: [Pharo-project] [Vm-dev] Re: Can OSProcess functionality be implemented using FFI instead of plugin?

kilon.alios
 
wow impressive work guys , thats awesome news.

On Sun, Jan 17, 2016 at 6:53 PM Eliot Miranda <[hidden email]> wrote:

Hi Levente,

> On Jan 17, 2016, at 7:30 AM, Levente Uzonyi <[hidden email]> wrote:
>
>> On Sat, 16 Jan 2016, Eliot Miranda wrote:
>>
>> On Sat, Jan 16, 2016 at 4:37 PM, Levente Uzonyi <[hidden email]> wrote:
>>
>>      On Sat, 16 Jan 2016, Mariano Martinez Peck wrote:
>>
>>      (Still no quote.)
>>
>>      How will you read the output of the process without having your image's process blocked in the FFI callout?
>>      How will you make sure that writes to input of the process won't block the FFI callout?
>> This presupposes the threaded FFI.  The threaded FFI allows the VM to make any number of blocking calls, adding a new thread to run the VM whenever the VM is stalled when the heartbeat beats.  hence one can freely read and write to/from i/o blocking i/o streams
>> (including pipes and sockets) or blocking database connexions, etc, all without stating that the FFI call must be done in a special way, since all calls through the FFI can block without blocking the VM.
>
> I think it was you who said (in a discussion with Craig) that the threaded FFI was not production ready. Is it ready for produciton now?

No, but I expect this is the year it will be.  Spur provides pinning, so the VM infrastructure is there.  The Pharo community plus some commercial relationships that have developed are providing funding.  Esteban Lorenzano and I want to collaborate on this and I hope to get help from some other people, such as Ronie Salgado.  And Mariano is working on an important part of the problem.  So I feel there's sufficient momentum for us to realize the threaded FFI this year.

> Levente


_,,,^..^,,,_ (phone)

>> Note that the scheme is also amenable to plugins, but the plugins must be rewritten to include the release vm/acquire vm calls around a blocking call.  With the threaded VM the FFI includes these calls around every FFI call.
>> HTH
>> _,,,^..^,,,_
>> best, Eliot
123