Is this a legitimate use of ExternalAddress?

I'm trying to do something that will make my life much easier, but I'm
not sure if it is valid or not. Probably 20% of the time I run my app,
it works flawlessly. But, the other 80% causes GP faults, trashes the
image, etc. Bad times. I am interfacing to DLLs, and I can assure that
the calling convention is correct, parameters, etc.

In the DLL I'm interfacing with, I pass a state structure around to
every external function. Likewise, there are callbacks into Dolphin
where the first parameter is a pointer to that state structure. So,
originally I put void* for every parameter where the structure would
get passed and then used an ExternalAddress object to hold the
structure pointer (note: I don't need to access the data in the state
structure, just pass it around).

Well, to make things easier on myself, I wanted to accomplish two
goals: type safety and ease of use. The idea was to subclass
ExternalAddress (call it Foo), and let that object duplicate a lot of
the functionality of the library functions. For example:

FooLibrary>>initFoo: pFoo
    <cdecl: void initFoo void*>

Would become:

FooLibrary>>initFoo: pFoo
    <cdecl: void initFoo Foo>

And I could also have:

Foo>>initFoo
    FooLibrary default initFoo: self

At least, that's the end goal. This actually appears to be working just
fine, I only mention it as background, and just in case my later
problems are caused by doing something wrong above.

Now, the next part of the equation is the callbacks. The DLL will pass
the pFoo structure to each callback (some with extra parameters). I
wanted to be able and make the callbacks easily, so I wrapped up the
actual callback register function like so:

FooLibrary>>RegisterCB: pFoo externalCallback: cb
    "Private - ..."
    <cdecl: void registerCB Foo void*>

FooLibrary>>registerCB: pFoo block: aBlock
    self RegisterCB: pFoo externalCallback:
        (ExternalCallback
            block: aBlock
            descriptor: (ExternalDescriptor fromString: 'cdecl: void
Foo')) asParameter

Now when I open up a worksheet and try it out like so:

foo := Foo new.
foo initFoo.
FooLibrary default registerCB: foo block: [ :aFoo | "do nothing" ].
FooLibrary default run. "this will run some tests"

My results are random. Sometimes it's perfect. Othertimes it's a GP
fault that either trashes the image or locks up Dolphin until I kill
it. I'm still learning Dolphin (and ST for that matter), so chances are
this is entirely my fault. But here are some questions:

1. Can I safely subclass ExternalAddress like I want?
2. Am I setting up the external calls correctly (the Foo parameter) if
(1) is true?
3. Do my external callbacks/descriptors look right?
4. In the docs for ExternalDescriptor, the docs use a syntax I'm not
aware of.. ##(ExternalDescriptor ...). What does this do? Do I need it?

Lastly, as a plea to Andy and Blair: can Dolphin handle GP faults a
little more gracefully? I understand that many are unrecoverable, but
right now, when I get one, I pretty much assume that I need to quit and
restart the image. :-(

Thanks for any help everyone!

Jeff M.

Jeff,

> FooLibrary>>registerCB: pFoo block: aBlock
>     self RegisterCB: pFoo externalCallback:
>         (ExternalCallback
>             block: aBlock
>             descriptor: (ExternalDescriptor fromString: 'cdecl: void
> Foo')) asParameter

The ExternalCallback which you create here is transitory, and will be reaped by
the GC very quickly -- possibly even before the external call is entered.  That
will free and perhaps overwrite its memory.  If you use an ExternalCallback
then you /must/ ensure that you keep a Dolphin-level reference to that object
"live" for as least as long as the external library is using the pointer to it.

The same applies to all other objects, of course, but it seems (I speak from
sad experience) to be easier to forget about ExternalCallbacks than most
objects -- maybe because I don't think of them as "objects" so much as function
pointers (which is wrong but tempting).


> 1. Can I safely subclass ExternalAddress like I want?

As far as I know, yes.  At least /I/ do it, and it seems to work well...


> 2. Am I setting up the external calls correctly (the Foo parameter) if
> (1) is true?

No, as above.


> 3. Do my external callbacks/descriptors look right?

They look /plausible/, including the use of Foo as an opaque pointer --
difficult to tell whether they are /right/ without knowing the external API
you're wrapping.


> 4. In the docs for ExternalDescriptor, the docs use a syntax I'm not
> aware of.. ##(ExternalDescriptor ...). What does this do? Do I need it?

The ##( ... ) syntax is nothing to do with external interfacing and you can
probably ignore it.  I don't think the example in the documentation is
particularly well-chosen.

The syntax tells the compiler to evaluate the expression in the brackets, and
embed a reference to the resulting value directly into the compiled method.
It's a generalisation of literals in that you can have a literal of any class,
with any kind of initialisation, not just the usual Integer, Float, String,
etc.

A very useful feature, but not one to over-use.  In the example it is just
being used as a simple optimisation -- instead of creating a new
ExternalDescriptor each time, it just re-uses the same one over-and-over.
Arguably unnecessary for the application, but it's only three extra characters,
and doesn't change the meaning of the method (in this case) in any way, so...


> Lastly, as a plea to Andy and Blair: can Dolphin handle GP faults a
> little more gracefully? I understand that many are unrecoverable, but
> right now, when I get one, I pretty much assume that I need to quit and
> restart the image. :-(

I find Dolphin is pretty robust (at least D5, I'm less convinced about D6 --
which seems to be a little unstable compared with earlier versions).  An
isolated GP Fault /reading/ a location is (in my experience to date) usually
benign, and I ignore it unless I have reason to suspect some deeper problem.  A
GPF /writing/ a location should be taken as a fatal warning (although writing
to the first 1000 bytes or so of address-space is often OK[*]) -- save your
work and abandon that image.  An absolute death blow is if you start getting
GPFs while you are not doing anything -- that's the garbage collector
attempting to follow invalid links within the image, and implies that the image
is totally irrecoverable.  Save (if you can) your work /do not overwrite the
existing files/, and abandon that image pronto.

([*] The question is: what had the errant external DLL written to /before/ it
tried to write to the illegal address ?  The GPF itself is, as far as I know,
completely benign -- the access to that memory failed, or it wouldn't have been
detected as a GPF.)

Unfortunately, one of the best ways to create unrecoverable situations is to
hand out pointers to Dolphin objects which get GCed before the external library
has finished using them -- which seems to be what you've been doing ;-)

The general rule when writing external interfacing code is to save the image
/before/ you try anything new, and give the new feature a good testing before
you trust it not to have corrupted that image.

I also find it very helpful to work with VC debugging the DLL if at all
possible -- that way you can step right into the external call and verify that
pointers and other parameters that C sees are what you expect.

BTW, you can instruct Dolphin to disconnect completely from a DLL that it has
loaded (so you can modify and recompile it in VC, say), by evaluating
    MyExternalLibrary closeDefault; clear.
without having to close and restart Dolphin.

Modifying system classes in Smalltalk has been compared with performing brain
surgery on yourself.  External interfacing is perhaps more like writing down
the instructions for that brain surgery, to be followed by autistic 12-year old
once you are safely anesthetised -- it calls for tact, delicacy, and care.
That said, I do a fair bit of external interfacing, and I don't find that
aspect of it especially difficult; the bulk of the problems come from trying to
work with the weird APIs that people dream up....

    -- chris

Chris Uppal wrote:
>
> The ExternalCallback which you create here is transitory, and will be reaped by
> the GC very quickly -- possibly even before the external call is entered.  That
> will free and perhaps overwrite its memory.  If you use an ExternalCallback
> then you /must/ ensure that you keep a Dolphin-level reference to that object
> "live" for as least as long as the external library is using the pointer to it.

Ah. This makes a lot of sense now. If the GC didn't run for a while,
then odds are it would work as expected (until it did run).

So, I've now got a new object that wraps up the ExternalCallback and
stores it in an instance variable. Simple enough. Now, the first time
the program runs (test from a worksheet), everything is perfect for the
full duration of the test. The second run (ie, hilighting the same test
code and evaluating) results in a GPF, though. Same code, same callback
block.

I believe I'm going to attribute this to memory not getting cleaned up
properly, but if you can recall any particular instance where you had a
similar problem (and what the resolution was), I'd be very happy to
hear about it. :-)

> The ##( ... ) syntax is nothing to do with external interfacing and you can
> probably ignore it.  I don't think the example in the documentation is
> particularly well-chosen.
>
> The syntax tells the compiler to evaluate the expression in the brackets, and
> embed a reference to the resulting value directly into the compiled method.
> It's a generalisation of literals in that you can have a literal of any class,
> with any kind of initialisation, not just the usual Integer, Float, String,
> etc.

Simple enough. Thanks. I'd always tried it in a worksheet to test the
syntax, and it always gave me grief. Now I know why. :-)

> I also find it very helpful to work with VC debugging the DLL if at all
> possible -- that way you can step right into the external call and verify that
> pointers and other parameters that C sees are what you expect.

I was hoping to avoid doing this (at least for now). I can see the
light at the end of the tunnel (where using Smalltalk could provide me
with great efficiency gains), but the setup getting there (at least for
external interfacing) is causing that light to dim daily. :-(

> Modifying system classes in Smalltalk has been compared with performing brain
> surgery on yourself.  External interfacing is perhaps more like writing down
> the instructions for that brain surgery, to be followed by autistic 12-year old
> once you are safely anesthetised -- it calls for tact, delicacy, and care.

Haha. Adding that to my list of running quotes. Fantastic. :-)

Jeff M.

I've also found ExternalCallback to be quite a challenge. I try to stay very
close to the other uses of ExternalCallback. Take a look at
Locale class>>systemLacales:
Locale>>dateFormats:
Locale>>timeFormats
Canvas>>fonts:do:

As to the callback working the first time but not the second time, it may be
related to the DLL you are calling. Is there any chance that it has stored
and reused the old pointer? You could test this by duplicating the workspace
code, so that instead of executing the identical code in two steps, you
execute the equivalent code in one step. If you use the same variable name
for the second callback, it might be GC the first reference. If you use a
separate name for the second callback, the first one might be the one being
executed.

As to the (in)ability to continue from a GPF, this is arguably an
unfortunate byproduct of the otherwise wonderful fact that your development
image is in the same address space as the application being developed. If
you were in C, the development environment would launch your application in
a separate OS process, and when you pass a bad pointer to a DLL, it only
kills your debugged application. In Smalltalk, the bad pointer will have the
chance to damage the development environment. I don't think there is much
that can be done to prevent this. As long as you are in Smalltalk, you can
trust the environment to protect you. Once you call a DLL, that DLL can
write into any of your process space--you don't even have to pass it a bad
pointer.

James Foster

"Jeff M." <[hidden email]> wrote in message
news:[hidden email]...

Jeff,

> So, I've now got a new object that wraps up the ExternalCallback and
> stores it in an instance variable. Simple enough. Now, the first time
> the program runs (test from a worksheet), everything is perfect for the
> full duration of the test. The second run (ie, hilighting the same test
> code and evaluating) results in a GPF, though. Same code, same callback
> block.

That definitely shouldn't happen.

On the assumption that you haven't missed anything which needs to be protected
(easy to do until you get into the habit of thinking in C while programming in
Smalltalk), it sounds like a bug in the external library which is holding onto
and using a pointer after it has been told not to (you have told it that the
pointer is now invalid ?).

One quick check would be to execute the:
    MyExternalLibrary closeDefault; clear.
incantation -- which should completely unload the external library.  If doing
that between two runs of your workspace test makes the problem go away, then it
is almost certainly a bug (or design misfeature) in the external library, or an
omission in the way you are driving it.


> > I also find it very helpful to work with VC debugging the DLL if at all
> > possible -- that way you can step right into the external call and
> > verify that pointers and other parameters that C sees are what you
> > expect.
>
> I was hoping to avoid doing this (at least for now). I can see the
> light at the end of the tunnel (where using Smalltalk could provide me
> with great efficiency gains), but the setup getting there (at least for
> external interfacing) is causing that light to dim daily. :-(

I think it depends on what you are doing.  If you are attempting to connect
directly to some third party's DLL, then running that under VC could be a real
chore (unless they supply it with a working VC debug build project, etc --
which is not the typical case).  OTOH, if you are connecting to your own code,
or (as I have done several times recently) to your own wrapper DLL for some
third party's non-DLL library, then it's extremely simple -- just don't exit VC
before you test your code from Dolphin ;-)

Well, OK, it a /little/ more complicated than that -- you have to tell Dolphin
to load the DLL from VC's debug folder, and tell VC to "attach" the Dolphin
process -- but I promise you that it is still pretty painless.  In fact it is
rather nice -- it feels as if you are extending at least some of Smalltalk's
dynamic-ness into the static C/C++ world.  I have even (just once) got
modify-and-continue to work from C++ while debugging a DLL called from
Dolphin -- the /only/ time in about 10 years that that feature has ever worked
for me...

    -- chris