Smalltalk › Pharo › Pharo Smalltalk Users

[Ann] OSSubprocess 1.0.0

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

12 messages Options

Guillermo Polito

[Ann] OSSubprocess 1.0.0

Hi all,

News from the OSSubprocess side. For those who do not know it, OSSubprocess is the library to call external processes from Pharo.

These are the main points of this release.

- 64bits support

- Issues Working on both Pharo 6 and 7

- Issue #34: Adding tests to validate that the return code is correctly handled

- Issue #27: Cleanup: remove OSSubprocess-Tests package

- PR #26: Add stress tests

This version works for Pharo >=6. Users of Pharo5 will need to stick with v0.2.5, because of non compatible changes in uFFI (required mostly for 64 bits).

Additionally:

- We moved the repository to pharo-contribution to remove pressure from Mariano for releases and integrations

- We updated the configuration in the catalog

- We decided to move to a 1.0.0 versioning schema because 0.* does look like the library is not stable and it actually is (We have been using it in pillar and iceberg for quite some time).

You can load this version from the catalog or using with:

Metacello new

baseline: 'OSSubprocess';

repository: 'github://pharo-contributions/OSSubprocess:v1.0.0/repository';

load.

Detailed changes log in

- https://github.com/pharo-contributions/OSSubprocess/releases/tag/v1.0.0

- https://github.com/pharo-contributions/OSSubprocess/compare/v0.2.5...v1.0.0

If you have any issues (or PR!), we wait for your issue entry in https://github.com/pharo-contributions/OSSubprocess/issues

Have a nice weekend,

Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - http://www.cnrs.fr

Web: http://guillep.github.io

Phone: +33 06 52 70 66 13

Sean P. DeNigris

Re: [Ann] OSSubprocess 1.0.0

Administrator

Guillermo Polito wrote
> If you have any issues…

IMHO the biggest issue is incompatibility with OSProcess. This seems not to
matter when one is loading Project A depending on OSSP in a clean image, but
the second one tries to load Project A into an image where Project B already
depends on OSP - KABOOM! Game over.

On a more philosophical note, I never understood why OSSP was created in the
first place since we have OSP; and especially why the API is different,
making switching more difficult. I'm sure there was a good reason to do
OSSP, but it might be helpful to clarify in the docs (so tradeoffs between
OSP and OSSP can be understood by prospective users).

-----
Cheers,
Sean
--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Cheers,
Sean

Guillermo Polito

Re: [Ann] OSSubprocess 1.0.0

Hi Sean

On Fri, May 18, 2018 at 9:49 PM, Sean P. DeNigris <[hidden email]> wrote:

Guillermo Polito wrote
> If you have any issues…

IMHO the biggest issue is incompatibility with OSProcess. This seems not to
matter when one is loading Project A depending on OSSP in a clean image, but
the second one tries to load Project A into an image where Project B already
depends on OSP - KABOOM! Game over.

Well, I cannot say much about why this happens. I've seen it happening, but I try not to depend on projects using OSProcess in that case.

Can we also say the same the other way around?

If we have OSSubprocess loaded and I download a project using OSProcess, does it break the same?

I understand that the incompatibilities bother, but putting all the fault in OSSubprocess and not in OSProcess seems not fair :).

Maybe both should be modified to be compatible? Dunno...

On a more philosophical note, I never understood why OSSP was created in the
first place since we have OSP;

I cannot answer this, I'll let Mariano or somebody from the Pharo board on this one.

and especially why the API is different,
making switching more difficult.

I found OSProcess API particularly bad. It's never clear to me whether to use OSProcess, PipeableSomething or CommandWhatever (I don't even remember the names of the things that **do work** and I have to look them every time).

To be fair, to me even sometimes OSProcess is too verbose. Most people only want to do

output := OSProcess executeCommand: 'ls'.

Blocking. Recovering stdout. And that's all.

So in that sense, I think that letting us move on and rethink the API is not bad at all...

Then, we can think on providing some compatibility layer to help people in migrating.

So, if anyone want to take action on this point...

I'm sure there was a good reason to do
OSSP, but it might be helpful to clarify in the docs (so tradeoffs between
OSP and OSSP can be understood by prospective users).

Well from a technical point of view, if you want to choose from one or another today I'd say:

- OSSP is written mostly in FFI, what means that it is easier to modify and ship.

- I'm sure OSSP works: it has way more tests and we are using it for several projects ourselves. (Please notice I do not say that OSP does not work, I say I have no way to be sure that it works because tests do not run). So we are eating our own food:

- we use them to run iceberg tests

- pillar is using it to compile pdfs

I've put myself quite some effort to have a working CI, with stress tests, 64bits. I've just tried to run OSP tests and they do not even run in Pharo because they use FileDirectory (and even latest version is updated to work on Cuis)... Having a compatibility layer to make such a project run on squeak, cuis and Pharo looks like a gigantic task for a project that seems so core to me.

-----
Cheers,
Sean
--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - http://www.cnrs.fr

Web: http://guillep.github.io

Phone: +33 06 52 70 66 13

Sean P. DeNigris

Re: [Ann] OSSubprocess 1.0.0

Administrator

Guillermo Polito wrote
> I try not to depend on projects using OSProcess in that case.

That is not exactly a robust solution! What is the only project providing
the functionality you need depends on OSP?!

Guillermo Polito wrote
> I understand that the incompatibilities bother, but putting all the fault
> in OSSubprocess and not in OSProcess seems not fair :)

Ha ha, well OSProcess was born in 2005, and then OSSP appeared reusing some
of its functionality just recently, so I'd say it's fair indeed!

Guillermo Polito wrote
> Maybe both should be modified to be compatible?

Maybe. I don't really understand the source of the incompatibility well
enough.

Guillermo Polito wrote
> I found OSProcess API particularly bad. It's never clear to me whether to
> use OSProcess, PipeableSomething or CommandWhatever (I don't even remember
> the names of the things that **do work** and I have to look them every
> time).

The things you're describing are not accidental complexity, but the fact
that OSP (and its companion CommandShell) actually have a far wider feature
set than just simple commands. CommandShell makes many shell-like things
possible in-image, but one doesn't have to use or even know anything about
all that to do simple things. That said, every time I did into CS and see
all the things that are possible I am amazed.

Guillermo Polito wrote
> Most people only want to do
> output := OSProcess executeCommand: 'ls'.
> Blocking. Recovering stdout. And that's all.

(PipeableOSProcess command: 'ls') output

That doesn't seem horribly complex to do in OSP ;)

Guillermo Polito wrote
> Well from a technical point of view, if you want to choose from one or
> another today I'd say…

Ah, interesting and informative. Thank you.

-----
Cheers,
Sean
--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Cheers,
Sean

David T. Lewis

Re: [Ann] OSSubprocess 1.0.0

On Wed, May 23, 2018 at 07:11:29AM -0700, Sean P. DeNigris wrote:

> Guillermo Polito wrote
> > I try not to depend on projects using OSProcess in that case.
>
> That is not exactly a robust solution! What is the only project providing
> the functionality you need depends on OSP?!
>
>
> Guillermo Polito wrote
> > I understand that the incompatibilities bother, but putting all the fault
> > in OSSubprocess and not in OSProcess seems not fair :)
>
> Ha ha, well OSProcess was born in 2005, and then OSSP appeared reusing some
> of its functionality just recently, so I'd say it's fair indeed!
>

Actually, OSProcess was born in 1999 :-)

First announcement:

http://lists.squeakfoundation.org/pipermail/squeak-dev/1999-June/016553.html

And this appears to be where I first starting thinking about the problem:

http://lists.squeakfoundation.org/pipermail/squeak-dev/1999-January/025236.html

http://lists.squeakfoundation.org/pipermail/squeak-dev/1999-May/023712.html

Kudos to Mariano for working closely with me as he developed the FFI based
functionality in OSSP, he did a really good job of working out issues wherever
possible. FFI based solutions work at a different level of abstraction than
VM plugins, and there is a role for both.

Dave

Mariano Martinez Peck

Re: [Ann] OSSubprocess 1.0.0

In reply to this post by Sean P. DeNigris

Hi guys,

First, I would like to say that David was always very helpful, positive and we have discussed and worked together a lot for this.

When I was sponsored to build OSSubprocess I already got the "we would like this have this", so maybe some additional opinion from the ESUG board may help clarify. From what I understand/remember/believe, the main reasons for building OSSubprocess were:

1) FFI-based tool. Low the barrier of having to modify the VM to make a fix. Despite all our efforts, we still have way more people able to deal with FFI than with the VM. Even more if the VM is now shared under a common project, in which is not always easy to plug Pharo-specific stuff.

2) Cleaner/nicer API. You may or may not agree with this and that's perfect (we all have different opinions). When I was using OSProcess I always thought the API was not as I would like to, so my goal was to provide a API closer to my wishes. Again, not necessary better.

3) Smaller functionality than OSProcess: OSProcess is huge...it allows you to fork squeak images, fork() + exec*() whatever you want, runs on Windows, etc. We wanted something smaller...just enough to be able to fork processes. That is... to capture the most common/easy process forking.. But if you really need something very advance, you may still go to bare metals and use OSProcess approach (fork + exec).

4) Added infrastructure. We wanted the project to be on Git, with a very detailed documentation, very high test coverage, and CI integration.

-----

So...that is about the reasons... as to why it breaks when OSProcess is loaded to, it's because for OSSubprocess I took the wonderful idea from OSProcess of the child reaper. This is a process that runs on the Smalltalk side and handles exception (via semaphore) when the child has died. This is (currently) only possible thanks to OSProcess primitive. That means OSSubprocess still needs (one or very little) OSProcess primitives down in the VM. When you load either of the projects, they launch the child reaper process, register the semaphore etc etc. So the problem is when you loaded one and you then try to load the second one, the initialization of the second one you plug its own semaphore etc etc and so the previous gets broken as it doesn't have a working child reaper process.

If someone has a better idea on how to solve this, then please speak up.

Cheers,

On Wed, May 23, 2018 at 11:12 AM Sean P. DeNigris <[hidden email]> wrote:

Guillermo Polito wrote
> I try not to depend on projects using OSProcess in that case.

That is not exactly a robust solution! What is the only project providing
the functionality you need depends on OSP?!

Guillermo Polito wrote
> I understand that the incompatibilities bother, but putting all the fault
> in OSSubprocess and not in OSProcess seems not fair :)

Ha ha, well OSProcess was born in 2005, and then OSSP appeared reusing some
of its functionality just recently, so I'd say it's fair indeed!

Guillermo Polito wrote
> Maybe both should be modified to be compatible?

Maybe. I don't really understand the source of the incompatibility well
enough.

Guillermo Polito wrote
> I found OSProcess API particularly bad. It's never clear to me whether to
> use OSProcess, PipeableSomething or CommandWhatever (I don't even remember
> the names of the things that **do work** and I have to look them every
> time).

The things you're describing are not accidental complexity, but the fact
that OSP (and its companion CommandShell) actually have a far wider feature
set than just simple commands. CommandShell makes many shell-like things
possible in-image, but one doesn't have to use or even know anything about
all that to do simple things. That said, every time I did into CS and see
all the things that are possible I am amazed.

Guillermo Polito wrote
> Most people only want to do
> output := OSProcess executeCommand: 'ls'.
> Blocking. Recovering stdout. And that's all.

(PipeableOSProcess command: 'ls') output

That doesn't seem horribly complex to do in OSP ;)

Guillermo Polito wrote
> Well from a technical point of view, if you want to choose from one or
> another today I'd say…

Ah, interesting and informative. Thank you.

-----
Cheers,
Sean
--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Mariano
http://marianopeck.wordpress.com

Sean P. DeNigris

Re: [Ann] OSSubprocess 1.0.0

Administrator

Mariano Martinez Peck wrote
> From what I understand/remember/believe, the main reasons for
> building OSSubprocess were:

Thanks, Mariano!! Very informative :)

-----
Cheers,
Sean
--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Cheers,
Sean

Sean P. DeNigris

Re: [Ann] OSSubprocess 1.0.0

Administrator

In reply to this post by David T. Lewis

David T. Lewis wrote
> FFI based solutions work at a different level of abstraction than
> VM plugins, and there is a role for both.

Thanks for the context, David. Now that we understand the different niches,
the main problem is that they can not be loaded in the same image without
breaking :/

-----
Cheers,
Sean
--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Cheers,
Sean

Mariano Martinez Peck

Re: [Ann] OSSubprocess 1.0.0

On Wed, May 23, 2018 at 2:46 PM Sean P. DeNigris <[hidden email]> wrote:

David T. Lewis wrote
> FFI based solutions work at a different level of abstraction than
> VM plugins, and there is a role for both.

Thanks for the context, David. Now that we understand the different niches,
the main problem is that they can not be loaded in the same image without
breaking :/

The problem is to find a solution that works for both, Squeak and Pharo as well as for OSProcess and OSSubprocess.

I guess one possibility is to modify both, OSProcess and OSSubprocess initialization of the child reaper so that they install the same "generic" child reapear. Aside from initializating the child repear, they should be added into an "observer list". When, when it comes the second project to get loaded it which check that a child reaper is already registered..in which case he just register itself as "observer".

This child reaper should be generic enough (cannot be coupled to WHAT to do when the semaphore is signaled). Using Announcements (or other mechanisim that would work for Pharo and Squeak) we could simply "notify" the registered observers and each observer would do whatever is needed (what is actually now hardcoded in each child reaper)

Maybe that works...just an idea.

-----
Cheers,
Sean
--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Mariano
http://marianopeck.wordpress.com

Thierry Goubier

Re: [Ann] OSSubprocess 1.0.0

Hi Mariano,

2018-05-23 19:57 GMT+02:00 Mariano Martinez Peck <[hidden email]>:

>
>
> On Wed, May 23, 2018 at 2:46 PM Sean P. DeNigris <[hidden email]>
> wrote:
>>
>> David T. Lewis wrote
>> > FFI based solutions work at a different level of abstraction than
>> > VM plugins, and there is a role for both.
>>
>> Thanks for the context, David. Now that we understand the different
>> niches,
>> the main problem is that they can not be loaded in the same image without
>> breaking :/
>>
>
> The problem is to find a solution that works for both, Squeak and Pharo as
> well as for OSProcess and OSSubprocess.
>
> I guess one possibility is to modify both, OSProcess and OSSubprocess
> initialization of the child reaper so that they install the same "generic"
> child reapear. Aside from initializating the child repear, they should be
> added into an "observer list". When, when it comes the second project to get
> loaded it which check that a child reaper is already registered..in which
> case he just register itself as "observer".
>
> This child reaper should be generic enough (cannot be coupled to WHAT to do
> when the semaphore is signaled). Using Announcements (or other mechanisim
> that would work for Pharo and Squeak) we could simply "notify" the
> registered observers and each observer would do whatever is needed (what is
> actually now hardcoded in each child reaper)
>
> Maybe that works...just an idea.

I think that looks like a nice design. Just one question: do we need
to track down whether the child reaper event should be directed to
OSProcess or to OSSubprocess? Wondering if there is an issue with one
of them trying to close the other one's resource (but I haven't
checked, so I may be wrong).

Thierry

>
>>
>>
>>
>> -----
>> Cheers,
>> Sean
>> --
>> Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
>>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com

Mariano Martinez Peck

Re: [Ann] OSSubprocess 1.0.0

Hi Thierry. Excellent question. For OSSubprocess this is not an issue because I look up by pid in my list of forked children... So if not found I do nothing... At least that's what I remember (away from my machine now).

So at worst it would be a performance problem when both loaded.

As for osprocess I don't recall if this would be a problem but I imagine it won't. Maybe David can tell.

On Wed, May 23, 2018, 5:31 PM Thierry Goubier <[hidden email]> wrote:

Hi Mariano,

2018-05-23 19:57 GMT+02:00 Mariano Martinez Peck <[hidden email]>:
>
>
> On Wed, May 23, 2018 at 2:46 PM Sean P. DeNigris <[hidden email]>
> wrote:
>>
>> David T. Lewis wrote
>> > FFI based solutions work at a different level of abstraction than
>> > VM plugins, and there is a role for both.
>>
>> Thanks for the context, David. Now that we understand the different
>> niches,
>> the main problem is that they can not be loaded in the same image without
>> breaking :/
>>
>
> The problem is to find a solution that works for both, Squeak and Pharo as
> well as for OSProcess and OSSubprocess.
>
> I guess one possibility is to modify both, OSProcess and OSSubprocess
> initialization of the child reaper so that they install the same "generic"
> child reapear. Aside from initializating the child repear, they should be
> added into an "observer list". When, when it comes the second project to get
> loaded it which check that a child reaper is already registered..in which
> case he just register itself as "observer".
>
> This child reaper should be generic enough (cannot be coupled to WHAT to do
> when the semaphore is signaled). Using Announcements (or other mechanisim
> that would work for Pharo and Squeak) we could simply "notify" the
> registered observers and each observer would do whatever is needed (what is
> actually now hardcoded in each child reaper)
>
> Maybe that works...just an idea.

I think that looks like a nice design. Just one question: do we need
to track down whether the child reaper event should be directed to
OSProcess or to OSSubprocess? Wondering if there is an issue with one
of them trying to close the other one's resource (but I haven't
checked, so I may be wrong).

Thierry

>
>>
>>
>>
>> -----
>> Cheers,
>> Sean
>> --
>> Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
>>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com

David T. Lewis

Re: [Ann] OSSubprocess 1.0.0

Hi Mariano, hi Thierry,

I cannot follow up for a few days but I like Mariano's proposal and I
think that it will work. Maybe move the child reaper process to a small
OSChildWatcher class and have interested clients (OSP, OSSP, others)
subscribe to update notifications.

Dave

> Hi Thierry. Excellent question. For OSSubprocess this is not an issue
> because I look up by pid in my list of forked children... So if not found
> I
> do nothing... At least that's what I remember (away from my machine now).
> So at worst it would be a performance problem when both loaded.
>
> As for osprocess I don't recall if this would be a problem but I imagine
> it
> won't. Maybe David can tell.
>
>
> On Wed, May 23, 2018, 5:31 PM Thierry Goubier <[hidden email]>
> wrote:
>
>> Hi Mariano,
>>
>> 2018-05-23 19:57 GMT+02:00 Mariano Martinez Peck
>> <[hidden email]>:
>> >
>> >
>> > On Wed, May 23, 2018 at 2:46 PM Sean P. DeNigris
>> <[hidden email]>
>> > wrote:
>> >>
>> >> David T. Lewis wrote
>> >> > FFI based solutions work at a different level of abstraction than
>> >> > VM plugins, and there is a role for both.
>> >>
>> >> Thanks for the context, David. Now that we understand the different
>> >> niches,
>> >> the main problem is that they can not be loaded in the same image
>> without
>> >> breaking :/
>> >>
>> >
>> > The problem is to find a solution that works for both, Squeak and
>> Pharo
>> as
>> > well as for OSProcess and OSSubprocess.
>> >
>> > I guess one possibility is to modify both, OSProcess and OSSubprocess
>> > initialization of the child reaper so that they install the same
>> "generic"
>> > child reapear. Aside from initializating the child repear, they should
>> be
>> > added into an "observer list". When, when it comes the second project
>> to
>> get
>> > loaded it which check that a child reaper is already registered..in
>> which
>> > case he just register itself as "observer".
>> >
>> > This child reaper should be generic enough (cannot be coupled to WHAT
>> to do
>> > when the semaphore is signaled). Using Announcements (or other
>> mechanisim
>> > that would work for Pharo and Squeak) we could simply "notify" the
>> > registered observers and each observer would do whatever is needed
>> (what
>> is
>> > actually now hardcoded in each child reaper)
>> >
>> > Maybe that works...just an idea.
>>
>> I think that looks like a nice design. Just one question: do we need
>> to track down whether the child reaper event should be directed to
>> OSProcess or to OSSubprocess? Wondering if there is an issue with one
>> of them trying to close the other one's resource (but I haven't
>> checked, so I may be wrong).
>>
>>
>>
>> Thierry
>>
>> >
>> >>
>> >>
>> >>
>> >> -----
>> >> Cheers,
>> >> Sean
>> >> --
>> >> Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html
>> >>
>> >
>> >
>> > --
>> > Mariano
>> > http://marianopeck.wordpress.com
>>
>>
>