Runtime Image Stripping Woes

At 04:07 PM 2/14/2008, Andre Schnoor wrote:

Alan Knight wrote:

The most obvious technique to diagnose this if it worked the day before is to run it on the version from the day before, dump out the list of stuff it things it's saving/deleting, do it with today's version, and then diff them.

This is what I wanted to do, but Store doesn't support product configurations (with versioned prerequisites) beyond the bundle scope. I will need to load all the bundles and packages manually, picking the right version for each and see. Whether this will lead to the same results as running the automated script, dunno. It's worth a try.

Yes, that would be good, and has been on the agenda for some time, but not had the resources to get done.

However, you can fake that up in a number of ways. What we do internally is version the build scripts as files associated with a known bundle.

Yep. My plan is to move away from using Store in the build script. Step 1 would be to load from Store and create and save the parcel files. I've got a ParcelBuilder that can save the whole product as a single parcel. Step 2 would load this parcel in the way you describe. Step 3 would then seal the image and strip a few critical things (command line, compiler, etc).

Yes, I definitely wouldn't want to be using Store load into a runtime image, and then have to get rid of Store and associated code.

In 7.6, some of that functionality, notably the exception handling and logging, has been removed from runtime packager and put into the base image.

That's a very good move.

Thanks.

Yes, I definitely wouldn't want to be using Store load into a runtime image, and then have to get rid of Store and associated code. 

Here's an update on what I found out so far (incl. a bugfix for the
message tracer):

I compared my application versions (pre and post crash) manually and
package-wise, but could not find any suspicious changes. I guessed some
butterfly effect must have happened, i.e. a random change in the order
of methods being processed by the tracer may have caused some of them to
slip through. The tracer works in an iterative, multi-stage fashion
("repeat until no more new classes or selectors are found"), which makes
it likely sensitive to sequence issues.

My observation was that all accidentally stripped methods (6 in my case)
were

1) only referenced once in the image
2) referenced by the #initialize method of some other class
3) absolutely vital for running /any/ runtime image

Point 3 made me believe there must be a bug in the tracer and/or the
assumption of initially referenced start items was wrong. Further
experimentation showed that almost every stripping problem was related
somehow to #initialize. What is so special about #initialize? This
method is

1) implemented in almost every class
2) probably the one most frequently sent to super

I suspected super implementations to generally slip the tracer under
certain circumstances. The tracer uses a set of symbols to keep track of
methods already visited. This had that special smell :-) My vague guess
was: Even if I don't yet fully understand how the tracer works, this may
cause superclass or subclass implementations of the same selector to be
not considered for #traceMethod: where they should. I tried a patch that
keeps track of visited methods directly. All the stripping issues were
instantly gone   8-)

Here's my fix:

1) Add an instvar 'tracedMethods' to RuntimeBuilderMessageTracer
2) Initialize it with an empty IdentitySet.
3) Edit the methods #traceSelector: and #traceMethod: as stated below.

In addition to the fix, one must ensure Subsystem withAllSubclasses is
kept by default (except DevelopmentSystem), otherwise vital code for
runtime startup will be crippled. I just added them to my kept classes,
but there should be a more appropriate place to code this permanently
for all projects.

I would love to see everyone who experienced stripping issues (James?)
try out the fix and see if it works for them. Actually the tracer is a
well crafted piece of code after all and it shouldn't be abandoned just
because it had issues.

Enjoy!

Andre

BTW: I have no idea how the tracer could have ever worked before and
what particular change in my code made it fail. This is probably one of
the software engineering misteries this Murphy guy talked about.


--
Here are the patched methods:

#initialize
    ...
    "Add this initializer:"
    tracedMethods := IdentitySet new.


#traceMethod: aMethod
    "Add this guard clause:"
    (tracedMethods includes: aMethod)
        ifTrue:[ ^self ].
    tracedMethods add: aMethod.
    ...
   (keep everything else)
    ...


#traceSelector:
    | methods |
    sentSelectors add: aSelector.
    methods := invertedSelectors at: aSelector ifAbsent: [ ^self "not a
selector" ].
    methods do: [ :mth |
        " do not trace methods twice"
        (tracedMethods includes: mth) ifFalse:[
            " see if the class is being kept so far "
            ( referencedClasses includes: (
                mth mclass isMeta
                    ifTrue: [ mth mclass soleInstance ]
                    ifFalse: [ mth mclass ] ) ) ifTrue: [
                " make sure this method is not deleted "
                ( ( items deletedMethods includesKey: mth mclass ) and: [
                ( items deletedMethods at: mth mclass ) includes:
aSelector ] ) ifFalse: [
                        " ok - trace the individual method "
                        self traceMethod: mth ] ] ] ].



_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Thanks. I've created AR 53941. However, I'm curious what version you're using. Subsystem and its subclasses have been automatically kept by default since version 7.4.1, so I wonder why they would be missing for you.

At 12:47 PM 2/22/2008, Andre Schnoor wrote:

Here's an update on what I found out so far (incl. a bugfix for the message tracer):

I compared my application versions (pre and post crash) manually and package-wise, but could not find any suspicious changes. I guessed some butterfly effect must have happened, i.e. a random change in the order of methods being processed by the tracer may have caused some of them to slip through. The tracer works in an iterative, multi-stage fashion ("repeat until no more new classes or selectors are found"), which makes it likely sensitive to sequence issues.

My observation was that all accidentally stripped methods (6 in my case) were

1) only referenced once in the image
2) referenced by the #initialize method of some other class
3) absolutely vital for running /any/ runtime image

Point 3 made me believe there must be a bug in the tracer and/or the assumption of initially referenced start items was wrong. Further experimentation showed that almost every stripping problem was related somehow to #initialize. What is so special about #initialize? This method is

1) implemented in almost every class
2) probably the one most frequently sent to super

I suspected super implementations to generally slip the tracer under certain circumstances. The tracer uses a set of symbols to keep track of methods already visited. This had that special smell :-) My vague guess was: Even if I don't yet fully understand how the tracer works, this may cause superclass or subclass implementations of the same selector to be not considered for #traceMethod: where they should. I tried a patch that keeps track of visited methods directly. All the stripping issues were instantly gone   8-)

Here's my fix:

1) Add an instvar 'tracedMethods' to RuntimeBuilderMessageTracer
2) Initialize it with an empty IdentitySet.
3) Edit the methods #traceSelector: and #traceMethod: as stated below.

In addition to the fix, one must ensure Subsystem withAllSubclasses is kept by default (except DevelopmentSystem), otherwise vital code for runtime startup will be crippled. I just added them to my kept classes, but there should be a more appropriate place to code this permanently for all projects.

I would love to see everyone who experienced stripping issues (James?) try out the fix and see if it works for them. Actually the tracer is a well crafted piece of code after all and it shouldn't be abandoned just because it had issues.

Enjoy!

Andre

BTW: I have no idea how the tracer could have ever worked before and what particular change in my code made it fail. This is probably one of the software engineering misteries this Murphy guy talked about.


--
Here are the patched methods:

#initialize
   ...
   "Add this initializer:"
   tracedMethods := IdentitySet new.


#traceMethod: aMethod
   "Add this guard clause:"
   (tracedMethods includes: aMethod)
       ifTrue:[ ^self ].
   tracedMethods add: aMethod.
   ...
  (keep everything else)
   ...


#traceSelector:
   | methods |
   sentSelectors add: aSelector.
   methods := invertedSelectors at: aSelector ifAbsent: [ ^self "not a selector" ].
   methods do: [ :mth |
       " do not trace methods twice"
       (tracedMethods includes: mth) ifFalse:[
           " see if the class is being kept so far "
           ( referencedClasses includes: (
               mth mclass isMeta
                   ifTrue: [ mth mclass soleInstance ]
                   ifFalse: [ mth mclass ] ) ) ifTrue: [
               " make sure this method is not deleted "
               ( ( items deletedMethods includesKey: mth mclass ) and: [
               ( items deletedMethods at: mth mclass ) includes: aSelector ] ) ifFalse: [
                       " ok - trace the individual method "
                       self traceMethod: mth ] ] ] ].

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Alan Knight wrote:

Thanks. I've created AR 53941. However, I'm curious what version you're using. Subsystem and its subclasses have been automatically kept by default since version 7.4.1, so I wonder why they would be missing for you.

Including these classes explicitly was my first attempt to solve the issue and it led to some progress (albeit not a solution). After generally fixing the tracer, this might no longer be necessary.

I must note that, by "including" a class in my custom build system, also all of its methods are kept. This is probably why I saw some progress in the beginning.

Andre

 

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Alan Knight wrote:

Thanks. I've created AR 53941. However, I'm curious what version you're using. Subsystem and its subclasses have been automatically kept by default since version 7.4.1, so I wonder why they would be missing for you.

After testing this, I can confirm you are right: These classes are included anyway.  Before applying the patch, some methods slipped the tracer, because of its bug. They were reached exclusively from subsystems (e.g. DelaySystem), which led me to the false assumtpion not all of them were included correctly. After the fix, this no longer happens.

BTW: It might make sense to nil the instvar tracedMethods after the scan completed. They likely introduce cylic links that prevent the huge sets from being GCed. I haven't tested this, though, but it shouldn't harm.

Andre


_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Dear Steve,
    my package StoreUnloading in the OR may interest you.  If your Store-less image has no Undeclareds, it should still have none when Store has been loaded and unloaded again.  You will want to tweak exemptions / additions in the #unloadStore method to suit your app.

(The package also contains a few unloading-related fixes to Store methods and locations which I recommended become part of the base in 7.6 but which had not been added to it when last I looked in early Jan.  See email quoted below.)

HTH
          Yours faithfully
             Niall Ross

From: Alan Knight [[hidden email]]:

Yes, I definitely wouldn't want to be using Store load into a runtime image, and then have to get rid of Store and associated code. 

Just to be contrarian: we've been doing that for several years with automated scripts, using RTP to strip. It works well for us, and over the years unloading Store and PDP has become easier.

 

On a historical note, we never could get that to work well with Envy, even though it had a single "unload Envy" command. There we had to automate saving all applications to .st files, then automate filing them all in to a standard VisualWorks image.

 

Steve

-------- Original Message --------

Subject:	Re: [FIX][STORE] StoreUnloading and some fixes
Date:	Tue, 09 Oct 2007 14:49:23 -0400
From:	Alan Knight [hidden email]
To:	Niall Ross [hidden email], [hidden email], [hidden email]
References:	[hidden email]

Thanks Niall. This sounds very nice.

At 02:26 PM 10/9/2007, Niall Ross wrote:

Dear Sames et al,
   I've replicated a package StoreUnloading to the OR.  This package does two things:

   - It provides a menupick for clean unloading of Store from the image.

   - It fixes two Store bugs in VW7.5 and earlier:
       * bug in updating bundle information when switching Store logins
       * mispackaged #dbTrace method causes unloading problems

I think these two bugfixes should become part of the base image.  The rest of this email describes them (some stuff about the package's nominal purpose is in a P.S. just FYI).

Updating bundle information bugfix
=======================
Logically, a store bundle contains a single set of subpundles in a given image at a given time, regardless of which database(s) that bundle has been reconciled with, or which database the image is connected to;  that is certainly how the average user will think of it.  However in practice, this data is stored in the database informations of _each_ database connection, and it is updated in the database information for the current database _only_ when the user edits the bundle;  The base release of Store does _not_ upgrade the list for one database when that for another is the most recent load or reconcile.

The result is that an image with a bundle for which it had multiple database informations could have different contents lists and orders in each, causing eg. newly-added packages to jump from the bundle to the Registry's top-level list in the following scenario
   a) Bundle loaded from one database, or created with Store unloaded, with certain pundles.
   b) Bundle reconciled to other database and version with additional pundles loaded and/or changed order
   c) Reconnect to first database or unload store (i.e. 'reconnect' to 'no database')
The fixes in this package ensure that the additional pundles. and their order, are applied to all database informations.

#dbTrace mispackaged
================
The DatabaseConnectionInformation>>dbTrace method should be in PackageCategories as it can be required even when Store is unloaded, since Bundles insist on having a nonce DatabaseConectionBundleInformation to manage their contents and file descriptors even when Store is absent.  This has nil dbTrace of course but, in its present location, the dbTrace accessor used to verify that fact will have been unloaded with Store!  (The distribution base image gets away with it by leaving its database links in, but that means that when you reattach, you see pundles marked as unchanged which have in fact been changed while you were unattached; such changes are not known to Store until you choose to rereconcile.  This is not an ideal UI strategy and users who unload and reload Store, e.g. when checking packaging, dependencies, etc., may want to discard all links, or specific database links, so that they are not mislead into wrongly thinking that a pundle is unchanged.)

All the code in StoreUnloading relates to the first bugfix above except for the #dbTrace override (which is the second bugfix:  it should be moved to PackageCategories in the base and is only overridden here FYI) and the VisualLauncher>>unloadStore method.

HTH.
         Yours faithfully
            Niall Ross

P.S. Store unloading behaviour:  VisualLauncher>>unloadStore provides a template for unloading Store, adaptable to an applications needs (and the package comment includes a VisualLauncher>>reloadStore method for inversion as needed).  Itadds an 'Unload Store' menupick to the Launcher's Store menu, active only when not logged-in to Store. 
Invoking it will
   - ask user to confirm, warning that store-using packages must be unloaded
       * store-using packages e.g. RBStoreExtensions, must be removed beforehand or added to #unloadStore's list
   - prompt user to save repository settings
   - prompt user to discard database links
   - remove all StoreFor* parcels and all their prereqs recursively
       * unloadStore method has exemptedNames temp user can populate; 
confirmer lists these
   - cleanse and unload StoreBase bundle (StoreBase parcel has bundle structure, so unloading parcel does not unload all)
   - condense changes (to remove store referencess from changes, thence remove all Store Undeclareds)
       * (if image has other Undeclareds matching Store's then Undeclareds may remain)

The assumption is that users will edit #unloadStore and #reloadStore to suit their application and Store usage.  These methods could of course be developed to be dynamic inverses (unload saving the roots it found in the image to a shared var for reload's use), to find store-dependent pundles recursively and unload them as well, and so on;  that can wait till anyone needs it.

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc