needed one OSProcessPlugin for darwin/osx

in my spare time today I was going to run the Sunits for OSProcess  
under mac carbon 3.8.10b15.
However I'm unable to find a version of the plugin anywhere and I'm a  
bit lazy and don't want to
face the hassle of trying to build a plugin from scratch. Therefore  
if someone has a pre-build plugin for os-x
somewhere please email a link or how to obtain it.

--
========================================================================
===
John M. McIntosh <[hidden email]> 1-800-477-2659
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

On Mon, Feb 27, 2006 at 10:57:45AM -0800, John M McIntosh wrote:
> in my spare time today I was going to run the Sunits for OSProcess  
> under mac carbon 3.8.10b15.
> However I'm unable to find a version of the plugin anywhere and I'm a  
> bit lazy and don't want to
> face the hassle of trying to build a plugin from scratch. Therefore  
> if someone has a pre-build plugin for os-x
> somewhere please email a link or how to obtain it.

Hi John,

In case you don't get someone to send a pre-built OSPP:

If you are building a VM with VMMaker, I would expect that adding
OSPP to the build should be pretty easy. I'm not familiar with the
Mac, but here is more or less what I think you'll need to know:

- There are no external support files or additional headers required.

- Get OSProcessPlugin from SqueakMap. If the carbon VM supports
Ian's asynchronous IO routines, then also grab AioPlugin. If X window
is of any interest, then also grab XDisplayControlPlugin.

- You will need to put FilePlugin.h and SocketPlugin.h in the build
path for OSPP (and for AioPlugin if you build that).

- You will need to include the sqaio.h file from the unix platform
sources for both OSPP and AioPlugin, and also define SQAIO_H to point
to this file (#define SQAIO_H "sqaio.h"). This is in OSPP only for
backward compatibility, but you still need to have it for the build.
The #define is also for backward compatibility, as the file used
to be named "aio.h" rather than "sqaio.h".

- If you build XDisplayControlPlugin, you'll need to get the X headers
and libraries into the build and link paths. I don't know how this
would be done on the Mac, but the acinclude.m4  Makefile.inc files
in platforms/unix/plugins/XDisplayControlPlugin might provide a hint.

As far as I know, that should be all you need to build the plugins.

When you run the tests, I'd suggest that you load both OSProcess
and CommandShell, because CommandShell will add a lot of additional
unit tests when you run "OSProcess allTestResults". Get the latest
version of both packages, as I recently added a few tweaks to make
the tests fail more gracefully on Mac.

The unit tests will write a results file when they are completed, but
there is a pretty good chance that they will not get that far if one
of the tests hangs up due to pthread signal handling problems (and I
think this is quite likely). However, you will also get useful output
written to stdout as the tests run, so if you can run the VM from a
console window on the Mac, you should be able to see the output there
(and capture it to a file). Or course you can also run the unit tests
in the usual manner with the test runner.

I do not expect that you will find endianness problems. I do expect
that there will be problems related to signal handling and errno
handling with pthreads, as I have not done anything in OSPP to
accomodate this.  Most notably, the Squeak process that handles
child process exit status relies on catching the SIGCHLD signal
and signaling a Squeak semaphore.  This may not work properly in
a pthread environment, as the SIGCHLD may not be delivered to the
right pthread.

Dave

David thank you the instructions, I was hoping to avoid having to  
follow all of them.
I'll do other things this week, and perhaps revisit the issue in a  
couple of weeks..

On 28-Feb-06, at 4:01 AM, David T. Lewis wrote:

--
========================================================================
===
John M. McIntosh <[hidden email]> 1-800-477-2659
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

OK, let me know if I can be of help.

Dave

On Wed, Mar 01, 2006 at 02:38:55AM -0800, John M McIntosh wrote:

ok, after some clues from David I built a mostly broken unix vm,  
however it was enough to build a OSProcessPlugin as a side effect.

In testing I see that calling primitiveForkSqueakWithoutSigHandler
crashes things, I'll suspect forking the carbon vm is a no no.

clues are...

[

squeak: could not load plugin `XDisplayControlPlugin'
ioFindExternalFunctionIn(primitiveForkSqueakWithoutSigHandler, 4251344)
Switching to process 4134 thread 0x6977]
Program received signal:  "EXC_BAD_ACCESS".

(gdb) call (int) printAllStacks()
SmallInteger
149288092 ProcessorScheduler class>idleProcess
149211076 [] in ProcessorScheduler class>startUp
149211168 [] in BlockContext>newProcess
$1 = 259985308
(gdb) bt
#0  0x0002bd24 in resume (aProcess=131462172) at /Users/johnmci/
Documents/Squeak3.8.0/src/vm/interp.c:21212
#1  0x0002eb98 in synchronousSignal (aSemaphore=131461696) at /Users/
johnmci/Documents/Squeak3.8.0/src/vm/interp.c:22390
#2  0x00005b38 in checkForInterrupts () at /Users/johnmci/Documents/
Squeak3.8.0/src/vm/interp.c:3134
#3  0x000103fc in interpret () at /Users/johnmci/Documents/
Squeak3.8.0/src/vm/interp.c:8247
#4  0x9002b1e0 in _pthread_body ()

Anyway other things work, geting file permissions, working directory  
etc etc, and I guess it's enough for folks who want limited support  
under os-x using the carbon vm at 3.8.10 level.  IF anyone wants to  
fix or alter the sunits so that they won't fork the vm if platform is  
Mac OS & not unix then feel free...


On 1-Mar-06, at 3:57 AM, David T. Lewis wrote:

--
========================================================================
===
John M. McIntosh <[hidden email]> 1-800-477-2659
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

On Thu, Mar 02, 2006 at 12:33:02AM -0800, John M McIntosh wrote:
> Anyway other things work, geting file permissions, working directory  
> etc etc, and I guess it's enough for folks who want limited support  
> under os-x using the carbon vm at 3.8.10 level.  IF anyone wants to  
> fix or alter the sunits so that they won't fork the vm if platform is  
> Mac OS & not unix then feel free...

Probably safer would be to just force a #primitiveFail in
UnixOSProcessPlugin>>primitiveForkSqueakWithoutSigHandler, and
let the tests fail.

Does an ordinary fork-and-exec (not forkSqueak) work? For example
does UnixProcessTestCase>>testRunCommand pass?

Dave

On Thu, Mar 02, 2006 at 12:33:02AM -0800, John M McIntosh wrote:
> ok, after some clues from David I built a mostly broken unix vm,  
> however it was enough to build a OSProcessPlugin as a side effect.
>
> In testing I see that calling primitiveForkSqueakWithoutSigHandler
> crashes things, I'll suspect forking the carbon vm is a no no.

Hi John,

If you have a chance, could you please try something for me?
Change the method UnixOSProcessAccessor>>canForwardExternalSignals
to answer false, then save your image and restart.

This will replace the event-driven SIGCHLD handler with a polling
loop, and it should work around the problem of SIGCHLD being handled
in some pthread other than the interpreter thread.

I think this should prevent the VM crashes you were getting, and
it may allow you to run external programs with OSProcess. (But I'll
be quite surprised if #forkSqueak works, because I have no clue how
or if all those pthreads would be restarted in a child Squeak.)

If the #canForwardExternalSignals workaround helps, we could change
it in OSProcess to be:

canForwardExternalSignals
        "Answer true if it is possible to forward OS signals to a
         Smalltalk Semaphore."
        ^ OSProcess isUnixMac not

Dave

p.s. I started reading up on pthreads, and I'm recoiling in horror
at the thought. There appears to be no way for the interpreter
thread to find the identity of other running threads, hence no
straightforward way to set the signal masks such that unix signals
can be reliably delivered to the interpreter thread (which needs
to do a #signalSemaphoreWithIndex:). I can imagine a workaround,
but it in't gonna be pretty.

Ok, I'm running a few tests this am, mind perhaps there are no OS-X  
OSProcessPlugin users out there?
however thank you for taking the time to look into this.

First I went in and mangled MacOSProcessAccessor and MacProcess to  
remove the overrides for access to unix superclass logic
then I prim failed the forksqueak prim call, and made the suggested  
change to UnixOSProcessAccessor>>canForwardExternalSignals

Testing

UnixProcessAccessorTestCase>testIsExecutableForUserInGroup
accessor isExecutable: '/tmp/delete.me'
                                        forUser: 99999
                                        inGroup: accessor primGetGid
fails, but I'll note:
-r-xr-x---   1 johnmci  wheel  0 Mar  4 13:47 /tmp/delete.me

testIsReadableForUserInGroup
didn't look likely related to above
testIsWriteableForUserInGroup
didn't look likely related to above

testChDir
for this accessor primGetCurrentWorkingDirectory
returns /private/tmp on the mac

UnixProcessFileLockingtestCase
wants to fork a headless squeak....
They all fail then...

UnixProcessFileLockTestcase
They all RUN (GREEN)

UnixProcessTestCase
5 run, 13 error because they all attempt to fork a squeak vm.

Likely I should look at the ancient  code for headless squeak, if  
there is any visible demand?

On 4-Mar-06, at 7:28 AM, David T. Lewis wrote:

--
========================================================================
===
John M. McIntosh <[hidden email]> 1-800-477-2659
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

Thanks John,

A key question is whether forking a simple process works:

    "(UnixProcessTestCase selector: #testRunCommand) run"

Forking a running Squeak image is a rather tricky business. By
contrast, a fork-and-exec to run an external program is simple,
and has a good chance of working as long as we steer clear of
the pthreads signal handling for the time being.

Let me know if #testRunCommand passes.

Dave


On Sat, Mar 04, 2006 at 02:22:40PM -0800, John M McIntosh wrote:

Hi John,

I managed to overcome my squeamishness about pthreads long enough
to write pthread-aware signal forwarding for OSPP. This is at:

  <http://www.squeaksource.com/OSProcessPlugin/VMConstruction-Plugins-OSProcessPlugin-dtl.3.mcz>

The approach I took is:
- OSPP plugin initialization remembers pthread ID of the interpreter.
- After calling signal forwarding primitive, OSPP sets signal handler
  to forward SIGFOO to Squeak semaphore.
- On receipt of signal, the handler in OSPP checks to see if it is
  executing in the context of the interpreter pthread.
    if in interpreter thread:
        signal the Squeak semaphore.
    otherwise:
        set sigmask to prevent future delivery of SIGFOO to this pthread.
        resend the signal to the interpreter pthread.

This seems to all work fine on my Linux system (meaning that it
does not break anything), and I did some tests to make sure the
signal re-sending mechanism works. So the idea would be that on
OS X, Squeak would be able to handle signals such as SIGCHLD by
having any non-interpreter threads that receive the signal just
resend it to the correct interpreter pthread, and also mask the
signal so it does not end up going to the wrong pthread next time.

If you try this, you would want to change #canForwardExternalSignals
back to true.

I would still like to know if #testRunCommand passes on your current
configuration, that's an important data point.

Thanks,

Dave


On Sat, Mar 04, 2006 at 06:18:10PM -0500, David T. Lewis wrote:

Ah, so back to checking this, low interesting (non-existent as far as  
I can tell) means slow turn around, Mind getting all the plugins to  
work with MacIntel has
been a chore.

So
(UnixProcessTestCase selector: #testRunCommand) run
does run and does not throw any errors...

However lets see, build debug version of VM with GCC 4.0  oops...

dlopen(/Users/johnmci/Documents/Squeak3.8.0/build/Development/Plugins/
UnixOSProcessPlugin, 10):
Symbol not found: _argCnt.  Referenced from: /Users/johnmci/Documents/
Squeak3.8.0/build/Development/Plugins/UnixOSProcessPlugin.
Expected in: flat namespace

(gdb) call (int) printAllStacks()
Process
129010824 UnixOSProcessAccessor>forwardSigChld
129010732 UnixOSProcessAccessor>sigChldSemaphore
129010640 UnixOSProcessAccessor>finalize
129007008 OSProcessAccessor class>startUp:
129006916 ThisOSProcess class>startUp:
126364916 [] in SystemDictionary>send:toClassesNamedIn:with:
126364692 OrderedCollection>do:
126364600 SystemDictionary>send:toClassesNamedIn:with:
126364508 SystemDictionary>processStartUpList:
126227260 SmalltalkImage>snapshot:andQuit:embedded:
126227168 SmalltalkImage>snapshot:andQuit:

Mmm, much reading, turns out in gcc 4.0 xcode decides that -
fvisibility=hidden  a *new* feature to hide or not export symbols in  
the linker. Nice.
Of course this worked before because I was compiling with GCC3.3 for  
powerpc during development and GCC 3.3 for Deployment powerpc. However
macIntel uses GCC 4.0 so loading the UnixOSProcessPlugin would work  
on ppc, but not on macintel. Guess it's good I discovered this  
problem before
someone else did.


On 4-Mar-06, at 3:18 PM, David T. Lewis wrote:

> (UnixProcessTestCase selector: #testRunCommand) run

--
========================================================================
===
John M. McIntosh <[hidden email]> 1-800-477-2659
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

On Sun, Mar 12, 2006 at 04:15:06PM -0800, John M McIntosh wrote:
> Ah, so back to checking this, low interesting (non-existent as far as  
> I can tell) means slow turn around, Mind getting all the plugins to  
> work with MacIntel has
> been a chore.
>
> So
> (UnixProcessTestCase selector: #testRunCommand) run
> does run and does not throw any errors...

Excellent, that mean that running an external program and getting
its exit status works with the basic mechanism of polling for exit
status. The event-driven signal forwarding probably is not working
yet, but that is the subject of the pthread-aware issue, which may
or may not be fixed by the update to OSPP for pthreads.

> However lets see, build debug version of VM with GCC 4.0  oops...
>
> dlopen(/Users/johnmci/Documents/Squeak3.8.0/build/Development/Plugins/
> UnixOSProcessPlugin, 10):
> Symbol not found: _argCnt.  Referenced from: /Users/johnmci/Documents/
> Squeak3.8.0/build/Development/Plugins/UnixOSProcessPlugin.
> Expected in: flat namespace

This would be from platforms/unix/sqUnixMain.c, which make these
available to plugins as follows:

int main(int argc, char **argv, char **envp)
{
  /* Make parameters global for access from plugins */
  argCnt= argc;
  argVec= argv;
  envVec= envp;


These would not be in the Mac OS sources, hence symbol not found.

Dave

Actually I did add these to the mac carbon source so I can run the  
unix command line processing, mind I actually added them years ago  
for OSProcess plugin btw. The fact I got a symbol not found is due to  
choices made by Apple in the gcc 3.3 to 4.0.1 migration. Usage of a  
compiler flag solved the problem by un-hiding
global symbols.

On 12-Mar-06, at 6:57 PM, David T. Lewis wrote:

--
========================================================================
===
John M. McIntosh <[hidden email]> 1-800-477-2659
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

On Sun, Mar 12, 2006 at 11:25:52PM -0800, John M McIntosh wrote:
> Actually I did add these to the mac carbon source so I can run the  
> unix command line processing, mind I actually added them years ago  
> for OSProcess plugin btw.

Ah, right you are! Thanks for that.

Dave