code missing from loaded packages

Hello,

I am experiencing strange problems with code (entire classes -- part of a package) not being loaded into the image when doing an 'upgrade' using metacello.

There is a certain version of all packages already loaded into extent and I am doing an upgrade. For that I use our metacello config and launch it to load the next version.
This seems to be going all well: it mentions all packages being fetched and loaded, etc...

However, after the load, I can find class definitions that are missing from certain packages, although the monticello tools indicate that the package should be loaded completely without changes in the image. But when I launch the 'changes' button, they do pop up.

Any ideas where I should be looking at to find out the reason of such a problem?

Johan

Johan,

Are the classes present in the image or are they missing as well?

If the classes are present then we are just dealing with a Monticello issue ...

Dale

----- Original Message -----
| From: "Johan Brichau" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Tuesday, May 22, 2012 5:06:25 AM
| Subject: [GS/SS Beta] code missing from loaded packages
|
| Hello,
|
| I am experiencing strange problems with code (entire classes -- part
| of a package) not being loaded into the image when doing an
| 'upgrade' using metacello.
|
| There is a certain version of all packages already loaded into extent
| and I am doing an upgrade. For that I use our metacello config and
| launch it to load the next version.
| This seems to be going all well: it mentions all packages being
| fetched and loaded, etc...
|
| However, after the load, I can find class definitions that are
| missing from certain packages, although the monticello tools
| indicate that the package should be loaded completely without
| changes in the image. But when I launch the 'changes' button, they
| do pop up.
|
| Any ideas where I should be looking at to find out the reason of such
| a problem?
|
| Johan

Hi Dale,

A previous version of one of those classes is present, yes. Another one is a new class.
Mind that I'm launching the update from a topaz with manual migration and manual commit.

I might be hitting 2 different issues (I think). And I think the only thing I need is a way to be notified of incomplete package loads during a metacello/gofer load process in topaz? It seems that monticello in gemtools tells me when I'm trying to load a class which does not have the superclass present, but that this package loads silently in metacello?

- The first issue happens because I needed to load a particular package manually (using Gofer) before launching the metacello load itself. This is necessary because the metacello load of the entire project has a load order that conflicts with an internal class refactoring. More specifically: we moved up an instance variable to a superclass but because of the load order, the Gemstone compiler complains that the same instance variable is being defined in a subclass (because that subclass is loaded later on -- in a different package). It now seems that this package did not load a class because the (new) superclass was not present. I need to verify a bit more, but I think that is the case for at least one of the missing classes. If that is the case, I understand what is happening but I am missing an error notification ;-)

- The other issue seems different because the class is already present and a new version is being loaded. This new version is a (simple) class redefinition where additional instance variables are being declared. Although I have a suspicion that there might be similar problems as the one above, but then with temporary variables in subclasses. Could that be? Can a class definition be refused if there are still methods with the same temporary variable names present?

scouting on... ;-)

Johan

On 22 May 2012, at 17:16, Dale Henrichs wrote:

Dale,

I misunderstood your question:

No, the classes are missing from the image. That is the problem, actually ;-)
I need to reload the package manually to load those classes. But the metacello load's trace told me that it successfully loaded that package already... silently leaving out a couple of classes?

On 22 May 2012, at 19:11, Johan Brichau wrote:

>> Are the classes present in the image or are they missing as well?

Johan,

These two issues do ring a bell when it comes to problems with Monticello ... I don't think Metacello is implicated in this ... it uses Monticello to do the actual loads and it shouldn't be handling errors during the load. It does handle fetch errors (for retrying) but it shouldn't be catching Errors during the load.

Off hand, I think the big difference between Topaz and GemTools  for doing Monticello loads is that Warnings may be swallowed (are you using GsDeployer?) and logged in Topaz while GemTools will bring up a notifier (no UI in Topaz). With that said, I would think that handled warnings should be logged.

The second thing that I'm suspicious of is that when an error occurs during a definition load, the Error is caught and the definition is scheduled to be tried again later ... in GemStone this can wreak a bit of havoc for class definitions since it can cause the class definition creation order to be changed ... with that said, I would expect the second load to fail and that should result in an Error ...

In both of your issues, I would expect errors to be thrown and logged ... I am surprised that they are sinking without a trace ...

Perhaps you should instrument up the MCPackageLoader>>basicLoad and MCPakcageLoader>>tryToLoad: ... there you can see that I've got quite a bit of monkey business going on with handling and retrying various "known issues". Perhaps your scenario is wandering between the cracks ...

MCPackageLoader>>basicLoad is doing the retry of #errorDefintions, so this is where I expect things to be going wrong ... the first load Error during MCPakcageLoader>>tryToLoad: is "expected" and the definition should be added to the #errorDefinitions and retried later ...

Dale

----- Original Message -----
| From: "Johan Brichau" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Tuesday, May 22, 2012 10:11:44 AM
| Subject: Re: [GS/SS Beta] code missing from loaded packages
|
| Hi Dale,
|
| A previous version of one of those classes is present, yes. Another
| one is a new class.
| Mind that I'm launching the update from a topaz with manual migration
| and manual commit.
|
| I might be hitting 2 different issues (I think). And I think the only
| thing I need is a way to be notified of incomplete package loads
| during a metacello/gofer load process in topaz? It seems that
| monticello in gemtools tells me when I'm trying to load a class
| which does not have the superclass present, but that this package
| loads silently in metacello?
|
| - The first issue happens because I needed to load a particular
| package manually (using Gofer) before launching the metacello load
| itself. This is necessary because the metacello load of the entire
| project has a load order that conflicts with an internal class
| refactoring. More specifically: we moved up an instance variable to
| a superclass but because of the load order, the Gemstone compiler
| complains that the same instance variable is being defined in a
| subclass (because that subclass is loaded later on -- in a different
| package). It now seems that this package did not load a class
| because the (new) superclass was not present. I need to verify a bit
| more, but I think that is the case for at least one of the missing
| classes. If that is the case, I understand what is happening but I
| am missing an error notification ;-)
|
| - The other issue seems different because the class is already
| present and a new version is being loaded. This new version is a
| (simple) class redefinition where additional instance variables are
| being declared. Although I have a suspicion that there might be
| similar problems as the one above, but then with temporary variables
| in subclasses. Could that be? Can a class definition be refused if
| there are still methods with the same temporary variable names
| present?
|
| scouting on... ;-)
|
| Johan
|
| On 22 May 2012, at 17:16, Dale Henrichs wrote:
|
| > Johan,
| >
| > Are the classes present in the image or are they missing as well?
| >
| > If the classes are present then we are just dealing with a
| > Monticello issue ...
| >
| > Dale
| >
| > ----- Original Message -----
| > | From: "Johan Brichau" <[hidden email]>
| > | To: "GemStone Seaside beta discussion"
| > | <[hidden email]>
| > | Sent: Tuesday, May 22, 2012 5:06:25 AM
| > | Subject: [GS/SS Beta] code missing from loaded packages
| > |
| > | Hello,
| > |
| > | I am experiencing strange problems with code (entire classes --
| > | part
| > | of a package) not being loaded into the image when doing an
| > | 'upgrade' using metacello.
| > |
| > | There is a certain version of all packages already loaded into
| > | extent
| > | and I am doing an upgrade. For that I use our metacello config
| > | and
| > | launch it to load the next version.
| > | This seems to be going all well: it mentions all packages being
| > | fetched and loaded, etc...
| > |
| > | However, after the load, I can find class definitions that are
| > | missing from certain packages, although the monticello tools
| > | indicate that the package should be loaded completely without
| > | changes in the image. But when I launch the 'changes' button,
| > | they
| > | do pop up.
| > |
| > | Any ideas where I should be looking at to find out the reason of
| > | such
| > | a problem?
| > |
| > | Johan
|
|

Johan,

The other thing that is different when using topaz for upgrades is that autoMigrate is false by default ... there might be a few more cracks in Monticello when it comes to autoMigrate being turned off...

Dale

----- Original Message -----
| From: "Johan Brichau" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Tuesday, May 22, 2012 5:06:25 AM
| Subject: [GS/SS Beta] code missing from loaded packages
|
| Hello,
|
| I am experiencing strange problems with code (entire classes -- part
| of a package) not being loaded into the image when doing an
| 'upgrade' using metacello.
|
| There is a certain version of all packages already loaded into extent
| and I am doing an upgrade. For that I use our metacello config and
| launch it to load the next version.
| This seems to be going all well: it mentions all packages being
| fetched and loaded, etc...
|
| However, after the load, I can find class definitions that are
| missing from certain packages, although the monticello tools
| indicate that the package should be loaded completely without
| changes in the image. But when I launch the 'changes' button, they
| do pop up.
|
| Any ideas where I should be looking at to find out the reason of such
| a problem?
|
| Johan

Hi Dale,

In my attempts to get more information on erroneous class load, I am now getting an even more strange result after loading and migrating.

First, it's correct that the initial load of the package issues a warning that several classes have compile errors but that a second attempt should work fine. The (automatic) retry seems to go fine for all those classes except one. I missed that warning earlier because my "commitOnAlmostOutOfMemoryDuring:" code around the metacello load was swallowing the warnings:

MCPlatformSupport commitOnAlmostOutOfMemoryDuring: [
  [
        (Smalltalk at: #ConfigurationOfNextPlan) loadLatestProductionVersionInGemStone.
  ]
                on: Warning
                do: [:ex | ex resume ]].

But the class definition of that one class is thus not loaded correctly (i.e. the new instance variables are missing from the class definition). However, all instance migrations and all methods referencing these variables are operating without errors. The instvars "seem to exist" but when testing, alle instances are sharing the same value for those instvars. When I was first reporting this issue, I got errors when doing the instance migration... which now seems to pass silently as well.

Right now, my only solution is to load the package twice before starting the instance migrations.
I am operating in topaz with a metacello load (see code above -- now with warning logger ;-) followed by a "GsDeployer new doBulkmigrate".

That's it so far... I'm still trying to trace what's going wrong with the retried compilation...

Johan

On 23 May 2012, at 18:52, Dale Henrichs wrote:

Johan.

Sorry I didn't get back to you earlier....I was under the weather over the weekend ...

Answers embedded below...

----- Original Message -----
| From: "Johan Brichau" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Monday, June 4, 2012 1:09:53 AM
| Subject: Re: [GS/SS Beta] code missing from loaded packages
|
| Hi Dale,
|
| In my attempts to get more information on erroneous class load, I am
| now getting an even more strange result after loading and migrating.
|
| First, it's correct that the initial load of the package issues a
| warning that several classes have compile errors but that a second
| attempt should work fine. The (automatic) retry seems to go fine for
| all those classes except one.

I seem to recall that there are some manipulations of instance variables that can be problematic depending upon the order of the operations: moving IVs up or down the hierarchy without an intermediate remove are suspect ...

Is is possible that you're doing one of these types of IV moves?


| I missed that warning earlier because
| my "commitOnAlmostOutOfMemoryDuring:" code around the metacello load
| was swallowing the warnings:
|
| MCPlatformSupport commitOnAlmostOutOfMemoryDuring: [
|   [
| (Smalltalk at: #ConfigurationOfNextPlan)
| loadLatestProductionVersionInGemStone.
|   ]
|                 on: Warning
|                 do: [:ex | ex resume ]].

Yeah, in GsDeploy, I've got a `Transcript show` in my Warning handlers to avoid having important Warnings sink without a trace:)

|
| But the class definition of that one class is thus not loaded
| correctly (i.e. the new instance variables are missing from the
| class definition). However, all instance migrations and all methods
| referencing these variables are operating without errors. The
| instvars "seem to exist" but when testing, alle instances are
| sharing the same value for those instvars. When I was first
| reporting this issue, I got errors when doing the instance
| migration... which now seems to pass silently as well.

You will probably find that you "functional" IVs are sitting in the UndefinedSymbols global ... You can inspect UndefinedSymbols from the Tools menu (a cleaning of the UndefinedSymbols is an important step: `UndefinedSymbolsTest new cleanUndefinedSymbols`...

The UndefinedSymbols mechanism is supposed to be triggered when a class is changed and the methods referencing the undefined symbols are then recompiled ... at least that's how it's supposed to work ...

|
| Right now, my only solution is to load the package twice before
| starting the instance migrations.
| I am operating in topaz with a metacello load (see code above -- now
| with warning logger ;-) followed by a "GsDeployer new
| doBulkmigrate".

I'd be interested to understand the "geometry" of the IV changes, i.e., order and classes involved (before and after) .... There area likely to be  a couple of holes in the algortithms...

|
| That's it so far... I'm still trying to trace what's going wrong with
| the retried compilation...
|
| Johan
|
| On 23 May 2012, at 18:52, Dale Henrichs wrote:
|
| > Johan,
| >
| > The other thing that is different when using topaz for upgrades is
| > that autoMigrate is false by default ... there might be a few more
| > cracks in Monticello when it comes to autoMigrate being turned
| > off...
| >
| > Dale
| >
| > ----- Original Message -----
| > | From: "Johan Brichau" <[hidden email]>
| > | To: "GemStone Seaside beta discussion"
| > | <[hidden email]>
| > | Sent: Tuesday, May 22, 2012 5:06:25 AM
| > | Subject: [GS/SS Beta] code missing from loaded packages
| > |
| > | Hello,
| > |
| > | I am experiencing strange problems with code (entire classes --
| > | part
| > | of a package) not being loaded into the image when doing an
| > | 'upgrade' using metacello.
| > |
| > | There is a certain version of all packages already loaded into
| > | extent
| > | and I am doing an upgrade. For that I use our metacello config
| > | and
| > | launch it to load the next version.
| > | This seems to be going all well: it mentions all packages being
| > | fetched and loaded, etc...
| > |
| > | However, after the load, I can find class definitions that are
| > | missing from certain packages, although the monticello tools
| > | indicate that the package should be loaded completely without
| > | changes in the image. But when I launch the 'changes' button,
| > | they
| > | do pop up.
| > |
| > | Any ideas where I should be looking at to find out the reason of
| > | such
| > | a problem?
| > |
| > | Johan
|
|

Dale,

> Sorry I didn't get back to you earlier....I was under the weather over the weekend ...

No problem at all. Hope you're feeling better!
I have to context-switch between this and many other tasks, so I'm not making progress fast on this one.

> I seem to recall that there are some manipulations of instance variables that can be problematic depending upon the order of the operations: moving IVs up or down the hierarchy without an intermediate remove are suspect ...

Yes, it's such a case.
The class that fails to load correctly already defined an IV that was now pulled up the hierarchy. The entire hierarchy of classes gets mentioned in the failed compilation warning.

I'm trying to trace exactly why this does not get compiled correctly and silently fails. At least it fails consistently for our current version update, so I can reproduce it easily.

Hope to get back asap with more...

Johan