Interesting stripping problem

Blair, Ian,

I just spent a few hours tracing down a problem with deploying some of my
executables.  It would have been quicker had I actually _read_ the crash
dumps  ~:0

The problem turned out to be an "unused" method that referenced Ian's
profiler.  Lagoon was getting into trouble tracing references in
ProfilerSetupDialog.

Out of the box, Lagoon's error message was "no such resource".  A little
hacking added the resource identifier, which had a nil class, so it was
reported as nil.Default view, which was not much more help than the original
message.  Some debugging strings later (this is where reading the crash
dumps would have made a huge difference<g>), it was fairly clear that
ProfilerSetupDialog was involved.  I finally took the hint when I tried to
uninstall the profiler (just to see what would happen) and was informed that
about 1/3 of my image was going to go with it.  Deleting the unused method
appears to have fixed the problem.

Clearly a deployed executable has no use for a source level profiler, but
I'm wondering whether there is something we could do to make it easier to
find a problem like this.

Have a good one,

Bill

--
Wilhelm K. Schwab, Ph.D.
[hidden email]

Bill,

> I just spent a few hours tracing down a problem with deploying some
> of my executables.  It would have been quicker had I actually _read_
> the crash dumps  ~:0

Strange.  I can't really see why a reference in a method that is going
to be removed during deployment (I assume?) would cause problems like
that.  Could there have been a zombie ProfilerSetupDialog hanging
around - that might have done something nasty that was solved by
removing the only remaining reference to its class.

I can't say that I've ever seen problems like this though.  I'm sure one
reason is that your applications are a lot more complex that mine but
the main reason is probably that <smug mode> I always deploy using a
clean image which only contains the packages I want in the deployment
</smug> :-)  It's probably not an option for you?

--
Ian

Use the Reply-To address to contact me.
Mail sent to the From address is ignored.

Bill,

> The problem turned out to be an "unused" method that referenced Ian's
> profiler.

I find it worthwhile encoding such development-time-only references using the
    Smalltalk at: #ClassName
idiom.

> I finally took the hint
> when I tried to uninstall the profiler (just to see what would happen)
> and was informed that about 1/3 of my image was going to go with it.
> Deleting the unused method appears to have fixed the problem.
>
> Clearly a deployed executable has no use for a source level profiler, but
> I'm wondering whether there is something we could do to make it easier to
> find a problem like this.

One thing would be a more "upfront" way of reviewing all the dependents of a
package.  This has been discussed many times, and it is easy enough to add for
ourselves (I'm sure I'm not the only Dolphin user to have done so), but it
still seems as if it'd be a worthwhile addition to the base system (or maybe
only the 'Pro' edition ;-)

Another feature that would have saved me some time in the past is some sort of
"assert not stripped" facility, which would cause a deployment time error (with
walkback if possible, or at least a stack dump) if the stripper attempted to
remove methods or classes marked in that way.

I recently had some trouble tracking down why some of my methods were being
stripped when "clearly" they should not be.  It turned out to be because the
stripper removes methods in the 'subclass responsibility' virtual category
(which I think should be at optional behaviour rather than hard-coded) and I
use that category to mark methods that have simple default implementation that
will sometimes suffice (e.g. 'do nothing'), but which should always be reviewed
as a candidate for overriding.  I knew fairly early on *that* the methods were
being stripped, but it took some hours to figure out *why*, so an
assert-not-stripped feature would have helped, I think.

    -- chris

"Chris Uppal" <[hidden email]> wrote in message
news:3ffffdb7$0$61063$[hidden email]...
> Bill,
>
> > The problem turned out to be an "unused" method that referenced Ian's
> > profiler.
>
> I find it worthwhile encoding such development-time-only references using
the
>     Smalltalk at: #ClassName
> idiom.
>
> > I finally took the hint
> > when I tried to uninstall the profiler (just to see what would happen)
> > and was informed that about 1/3 of my image was going to go with it.
> > Deleting the unused method appears to have fixed the problem.
> >
> > Clearly a deployed executable has no use for a source level profiler,
but
> > I'm wondering whether there is something we could do to make it easier
to
> > find a problem like this.
>
> One thing would be a more "upfront" way of reviewing all the dependents of
a
> package.  This has been discussed many times, and it is easy enough to add
for
> ourselves (I'm sure I'm not the only Dolphin user to have done so), but it
> still seems as if it'd be a worthwhile addition to the base system (or
maybe
> only the 'Pro' edition ;-)

The D6 package browser has a Dependents tree tab similar to the
Prerequisites tree tab.

>
> Another feature that would have saved me some time in the past is some
sort of
> "assert not stripped" facility, which would cause a deployment time error
(with
> walkback if possible, or at least a stack dump) if the stripper attempted
to
> remove methods or classes marked in that way.
>
> I recently had some trouble tracking down why some of my methods were
being
> stripped when "clearly" they should not be.  It turned out to be because
the
> stripper removes methods in the 'subclass responsibility' virtual category
> (which I think should be at optional behaviour rather than hard-coded) and
I
> use that category to mark methods that have simple default implementation
that
> will sometimes suffice (e.g. 'do nothing'), but which should always be
reviewed
> as a candidate for overriding.  I knew fairly early on *that* the methods
were
> being stripped, but it took some hours to figure out *why*, so an
> assert-not-stripped feature would have helped, I think.

I think you're right that stripping subclass responsibility methods should
be optional, in fact I thought it was. It is treated as a development method
category, but would be better handled in the same way as the 'should not
implement' category (#1471).

The D6 image stripper emits a log that makes it much easier to track down
problems such as this. Some useful features of this are:
1) Its in XML, so you can load it up into a DOM and query it, make
assertions, etc.

2) A complete application manifest that lists all classes, methods and
resources that remain in the deployed application. The manifest can be used
to define the source environment of a browser, effectively allowing one to
open a class browser in a development system that is constrained to see only
the classes and methods in the application. Definitions/References searches
from the browser will be similarly constrained. You may be quite surprised
at what ends up in your application, and browsing it in this way makes it
easy to determine the original source of an unwanted reference.

3) A list of sent but not implemented messages. This does not conclusively
prove that you will not get a DNU in your application, since the
messages may not be implemented in all the required classes and it only
includes directly sent messages in methods and resources, but it is
extremely useful in diagnosing stripping issues.

Related to (3), the D6 package browser has a 'Browse/Unimplemented Messages'
command that allows one to see a list of messages sent from a package and
its prerequisites that are not implemented in any of those packages. This
makes it easier to identify cases where a manual prerequisite must be added,
and is also a useful "lint" check that may identify errors in the
application. This list should be empty before you deploy your application.
Then if there are any entries in the same list in the deployment log you
will know that they have been introduced by the stripping process. Sometimes
there may be false positives in this list, but all should be investigated.

Regards

Blair

Ian,

> Strange.  I can't really see why a reference in a method that is going
> to be removed during deployment (I assume?) would cause problems like
> that.

Actually, I forgot about it and Lagoon found it :)


> Could there have been a zombie ProfilerSetupDialog hanging
> around - that might have done something nasty that was solved by
> removing the only remaining reference to its class.

There are no zombies now, and I doubt there were before, because I use the
Profiler class methods rather than the dialog.


> I can't say that I've ever seen problems like this though.  I'm sure one
> reason is that your applications are a lot more complex that mine but
> the main reason is probably that <smug mode> I always deploy using a
> clean image which only contains the packages I want in the deployment
> </smug> :-)  It's probably not an option for you?

It's an option, just not my first choice.  There is a fair amount of work to
do any time I "move in" to a new image, and some of that will be pretty
unpleasant to automate.  Unless I am certain it gets done the same way every
time, I would be concerned about deploying from an image that is very
different from the one I use to develop and run tests.  However, I will
probably get there eventually.

Have a good one,

Bill

--
Wilhelm K. Schwab, Ph.D.
[hidden email]

Blair,

> > One thing would be a more "upfront" way of reviewing all the dependents
> > of a package.  This has been discussed many times, and it is easy
> > enough to add for ourselves (I'm sure I'm not the only Dolphin user to
> > have done so), but it still seems as if it'd be a worthwhile addition
> > to the base system (or maybe only the 'Pro' edition ;-)
>
> The D6 package browser has a Dependents tree tab similar to the
> Prerequisites tree tab.

Good!


> I think you're right that stripping subclass responsibility methods should
> be optional, in fact I thought it was. It is treated as a development
> method category, but would be better handled in the same way as the
> 'should not implement' category (#1471).

Even better ;-)


> 2) A complete application manifest that lists all classes, methods and
> resources that remain in the deployed application. The manifest can be
> used to define the source environment of a browser, effectively allowing
> one to open a class browser in a development system that is constrained
> to see only the classes and methods in the application.
> Definitions/References searches from the browser will be similarly
> constrained.

Better still.  In fact it sounds really cool -- so much so that I'm tempted to
hack together something similar for use now...


> 3) A list of sent but not implemented messages. This does not conclusively
> prove that you will not get a DNU in your application, since the
> messages may not be implemented in all the required classes and it only
> includes directly sent messages in methods and resources, but it is
> extremely useful in diagnosing stripping issues.

Better and better :-)


> Related to (3), the D6 package browser has a 'Browse/Unimplemented
> Messages' command that allows one to see a list of messages sent from a
> package and its prerequisites that are not implemented in any of those
> packages.

I've had something similar for a while now, and it's been useful.  Two
enhancements that I keep thinking I should add (but have never got around to)
are:
    - some way of marking a package with a list of "false positives" that can
safely be ignored, to save me the effort of checking the same selectors over
and over.
    - some way of marking a "manual prerequisite" of a package with *why* it
was added in the first place.  I've occasionally resorted to removing the
manual prerequisite, rechecking for undefined selectors, and then restoring it,
just to see if it was still needed.

(OK, yes, I could just use the package comment, but -- for some reason -- that
had never occurred to me before now.)

BTW, I still think that the PB would benefit from foregrounding the version
string and the manual prerequisites.  Perhaps by adding the version string to
the 'Comment' tab, and the manual prerequisites to the 'Prerequisites' tab.  Or
perhaps a tab dedicated to the "properties" would be an easier way of rescuing
those very important (IMO) attributes from undeserved obscurity.  I'd also be
tempted to move the package comment tab to the front (I've recently started
doing that in my image anyway, but I think it's worth _considering_ for the
base image).

Be that as it may, the new stuff sounds excellent -- I'm looking forward to it.

    -- chris