Unloading packages with Environments (was Re: [squeak-dev] Re: Towards SqueakCore)

>
>
> On Sat, Mar 30, 2013 at 4:52 AM, Frank Shearar <[hidden email]>
> wrote:
>>
>> On 25 February 2013 18:51, David T. Lewis <[hidden email]> wrote:
>> > On Mon, Feb 25, 2013 at 04:57:54PM +0000, Frank Shearar wrote:
>> >> On 25 February 2013 16:37, Bert Freudenberg <[hidden email]>
>> >> wrote:
>> >> > On 2013-02-25, at 17:07, Frank Shearar <[hidden email]>
>> >> > wrote:
>> >> >
>> >> >> On 25 February 2013 15:35, David T. Lewis <[hidden email]>
>> >> >> wrote:
>> >> >>> On Mon, Feb 25, 2013 at 03:01:18PM +0000, Frank Shearar wrote:
>> >> >>>> On 25 February 2013 14:49, Bert Freudenberg <[hidden email]>
>> >> >>>> wrote:
>> >> >>>>>
>> >> >>>>> On 2013-02-25, at 15:33, Frank Shearar <[hidden email]>
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>>> On 14 February 2013 16:50, Frank Shearar
>> >> >>>>>> <[hidden email]> wrote:
>> >> >>>>>>> No, this takes the Trunk image as is and runs the job. No
>> >> >>>>>>> stripping
>> >> >>>>>>> has occurred. Noone's actually said "oh, that's a sane idea
>> >> >>>>>>> someone
>> >> >>>>>>> should do that" yet (except possibly for you, by virtue of not
>> >> >>>>>>> saying
>> >> >>>>>>> "are you crazy?" :) ), so I haven't done it yet.
>> >> >>>>>>>
>> >> >>>>>>> So the installation is essentially a no-op.
>> >> >>>>>>>
>> >> >>>>>>> With green lights on these builds, we can now verify that
>> >> >>>>>>> removing
>> >> >>>>>>> these packages won't break anything. At least, to the limit of
>> >> >>>>>>> the
>> >> >>>>>>> tests, which isn't much at all. So if we _do_ actually strip
>> >> >>>>>>> these
>> >> >>>>>>> guys out of Trunk, the builds should stay green. If they don't,
>> >> >>>>>>> we'll
>> >> >>>>>>> need to roll back the change and investigate.
>> >> >>>>>>>
>> >> >>>>>>> But I don't actually know how to strip them out of Trunk. I
>> >> >>>>>>> _suspect_
>> >> >>>>>>> it'll be via a Monticello config map of some description.
>> >> >>>>>>
>> >> >>>>>> Anyone? Bert? How do we strip packages out of Trunk?
>> >> >>>>>
>> >> >>>>> Manually once, when you prepare the Core image. Otherwise, if we
>> >> >>>>> put it in the trunk update stream, the packages would be unloaded from
>> >> >>>>> everyone's working images when updating.
>> >> >>>>
>> >> >>>> Given that we're early in 4.5's release cycle, if we do this kind
>> >> >>>> of
>> >> >>>> thing - stripping unnecessary things out of trunk - we ought to do
>> >> >>>> it
>> >> >>>> now rather than later. And announce loudly what we're doing, why,
>> >> >>>> and
>> >> >>>> how people can load in their favourite not-Core packages.
>> >> >>>
>> >> >>> NO.
>> >> >>>
>> >> >>> If Squeak developers work with images that contain just "their
>> >> >>> favourite
>> >> >>> non-Core packages" then none of the non-Core packages get
>> >> >>> maintained.
>> >> >>
>> >> >> Clearly we disagree quite a bit on this, but I don't think anyone
>> >> >> should actually _work_ on a Core image. _Noone_ should be able to
>> >> >> work
>> >> >> effectively in a minimal image because such a thing wouldn't even
>> >> >> have
>> >> >> a Browser!
>> >> >
>> >> > We don't disagree on this at all. People should be working in a full
>> >> > image.
>> >>
>> >> OK, cool. Finding out points of agreement helps in finding out where
>> >> we disagree :)
>> >>
>> >> >>> It is important to work on making packages loadable and unloadable,
>> >> >>> but
>> >> >>> you do not need to break the trunk image in order to accomplish
>> >> >>> this.
>> >> >>
>> >> >> Who's breaking trunk? I want to remove packages that are already
>> >> >> unloadable,
>> >> >
>> >> > Yes, but not from *user's* images, from the *core* image that is only
>> >> > used by the CI to build a full image.
>> >> >
>> >> >> and have CI jobs that run every time Trunk updates that
>> >> >> verifies that the packages can (a) reload and (b) run as well as we
>> >> >> already know now (which is not very). To be precise: Universes (for
>> >> >> example) might be smashed beyond recognition, and there is no way to
>> >> >> know, because it has something like 23% method coverage.
>> >> >
>> >> > Right.
>> >> >
>> >> >> Removing these packages from Trunk _will_ affect people updating
>> >> >> from
>> >> >> Trunk, I agree. How else do we make Trunk simpler, if not by
>> >> >> removing
>> >> >> stuff?
>> >> >
>> >> > I think we may be talking past each other. There are two types of
>> >> > packages we want to "remove from trunk":
>> >> >
>> >> > full = core packages + unloadable packages + deprecated packages
>> >> >
>> >> > So there are unloadable packages which should still exist in a full
>> >> > image that everyone is using. And there are deprecated packages which should
>> >> > not exist in a full image.
>> >> >
>> >> > Both core packages and unloadable packages are in Trunk. The only
>> >> > packages that we actually want to remove for good are the deprecated ones.
>> >> > The mechanics of unloading a deprecated package is by committing an empty
>> >> > version (which will remove all the classes and methods from user's images
>> >> > when updating), and eventually removing the now empty package from the
>> >> > update map (and bumping the squeak-version package to keep update numbers
>> >> > continuous).
>> >> >
>> >> > Maybe that's what you were asking about?
>> >> >
>> >> >> (It's also simple enough to reinstall your favourite packages:
>> >> >> Installer squeakmap update; install: 'WhateverItIs (someVersion)'.)
>> >> >
>> >> > Sure, that's how you would re-install a deprecated package.
>> >> >
>> >> >>> Also, please recognize that not everybody wants to do all their
>> >> >>> work in
>> >> >>> a clean image built by a robot. I for example do most of my work in
>> >> >>> a
>> >> >>> Squeak3.11alpha image that has been continuously updated from the
>> >> >>> update
>> >> >>> stream for quite a while. I also periodically check things against
>> >> >>> a
>> >> >>> fresh 4.4 image, but my usual pattern is to use one image over a
>> >> >>> long
>> >> >>> period of time with the trunk update stream, and most of my updates
>> >> >>> to
>> >> >>> trunk come from that old image.
>> >> >>
>> >> >> OK, but then we can't have a minimal core without an entire reboot.
>> >> >> I
>> >> >> mean, seriously, making a minimal core by unloading packages is an
>> >> >> order of magnitude harder when you cannot ever actually unload the
>> >> >> packages!
>> >> >>
>> >> >> So, given the resistance to my slow, gradual unwinding of the
>> >> >> tangle,
>> >> >> what's your counterproposal?
>> >> >>
>> >> >> frank
>> >> >
>> >> >
>> >> > I still fail to see the problem. For purposes of the build server,
>> >> > you once make a Core image by unloading all possible packages. This becomes
>> >> > the new "master image" from which all subsequent releases are built. Then
>> >> > you use that as the base to run tests, build full images, etc.
>> >> >
>> >> > Does that sound reasonable?
>> >>
>> >> Ah! OK, right. So, to see if I understand correctly:
>> >> * We _don't_ issue an empty version for the packages we're interested
>> >> in unloading. (Just for argument's sake, I'll talk about Universes.)
>> >> * If at some point in the future we decide to deprecate Universes,
>> >> _then_ we commit the empty version, and update the map. That decision
>> >> may be deferred for an arbitrary period after unloading it from Core.
>> >> * We take a clean Trunk image from CI, unload all unloadable packages,
>> >> and store that as our new master 4.5 image. (This will likely be a
>> >> manual step, and the clean image committed to the squeak-ci script
>> >> repository.)
>> >> * The stripped master remains stripped, and the existing SqueakTrunk
>> >> job publishes updated copies of the core image as per normal.
>> >> * We adjust the ReleaseSqueakTrunk CI job to, via
>> >> ReleaseBuilderForSqueak4dot5, loads the unloadable packages from
>> >> SqueakTrunk's output to create a "full fat" version.
>> >>
>> >> Dave, sound good?
>> >>
>> >
>> > Yes :)
>>
>> OK, so I thought I'd start the ball rolling with inspiration from
>> SmalltalkImage >> #unloadAllKnownPackages -
>>
>> #('Nebraska' 'Universes' 'XML-Parser') do: [:pkgName|
>>     (MCPackage named: pkgName) unload.
>>     MCMcmUpdater disableUpdatesOfPackage: pkgName].
>>
>> I've stored this as Squeak4.5.image in the squeak-ci repository. This
>> is the new base that we update from trunk. ReleaseSqueakTrunk then
>> loads these packages back into the image.
>>
>> That's the theory. In practice, the above is insufficient because it
>> leaves obsolete classes lying around. So I added
>>
>> MCWorkingCopy flushObsoletePackageInfos.
>> MCFileBasedRepository flushAllCaches.
>> MCDefinition clearInstances.
>> Behavior flushObsoleteSubclasses.
>> ChangeSet current clear.
>> ChangeSet current name: 'Unnamed1'.
>> Smalltalk flushClassNameCache.
>>
>> This too is insufficient: The SqueakCI-Benchmarking package happens to
>> depend on XML-Parser, so I reload the latest version of that
>> package... and things break because there are AnObsoleteXMLWriter
>> instances.
>>
>> What have I missed?
>
>
> This is when the tedium starts.  Use PointerFinder to track down the
> reference path to the obsolete XMLWriter, then determine (e.g. find or
> invent) the initialization(s) that eliminate that path (either by deleting
> the thing because it shouldn't be referred to, or evaluating something that
> replaces it).

>> frank
>>
>> > Dave
>> >
>> >> frank
>> >>
>> >> > - Bert -
>> >> >
>> >> >
>> >
>>
>
>
>
> --
> best,
> Eliot
>
>
>

> On 01/04/13 5:07 PM, Frank Shearar wrote:
>>
>>
>> Right. That tells me there's an Alias still pointing to the XMLWriter
>> class.
>
>
> It could be as simple as (made up code below):
>
> PerfMeasure>>xmlWriterClass
>     ^ XMLWriter
>
> Just re-compiling would fix the obsolete reference, or code like:
>     ^ Smalltalk at: #XMLWriter
> would fix it.
>
>> So my half-baked theory is that something's missing in the
>>
>> Environment work and how it integrates with Monticello?
>
>
> I doubt it's the direct responsibility of either (Environment or
> Monticello), but maybe a nice solution could be facilitated by Environment
> and used by Monticello. It could be a lot of work since Monticello is
> cross-platform (however only the format has to be compatible, while platform
> implementations could differ).

>
>
>
> On Mon, Apr 1, 2013 at 2:07 PM, Frank Shearar <[hidden email]>
> wrote:
>
>> Right. That tells me there's an Alias still pointing to the XMLWriter
>> class. So my half-baked theory is that something's missing in the
>> Environment work and how it integrates with Monticello?
>
>
> I don't think that's it. I mean sure, Environments is incomplete and buggy,
> but the behaviour you describe can happen in Squeak 4.4 too. If there are
> any instances of XMLWriter when you unload the class, they get migrated to
> obsolete instances. The thing to do is, as Eliot suggested, figure out
> what's pointing to them and how to sever those references before you unload
> the class.

> On 1 April 2013 23:06, Colin Putney <[hidden email]> wrote:
>>
>>
>>
>> On Mon, Apr 1, 2013 at 2:07 PM, Frank Shearar <[hidden email]>
>> wrote:
>>
>>> Right. That tells me there's an Alias still pointing to the XMLWriter
>>> class. So my half-baked theory is that something's missing in the
>>> Environment work and how it integrates with Monticello?
>>
>>
>> I don't think that's it. I mean sure, Environments is incomplete and buggy,
>> but the behaviour you describe can happen in Squeak 4.4 too. If there are
>> any instances of XMLWriter when you unload the class, they get migrated to
>> obsolete instances. The thing to do is, as Eliot suggested, figure out
>> what's pointing to them and how to sever those references before you unload
>> the class.
>
> And what's pointing to the class is an Alias. In a normally updated
> image (one in which XML-Parser has not been removed, PointerFinder
> shows a single path to XMLWriter:
>
> CLASS: SmalltalkImage class
> superclass: Object class
> subclasses: Array
> 383: XMLWriter class
>
> If I hack Environment >> #forgetClass:logged: and Environment >>
> #removeClass: to dump their arguments to the Transcript I can see that
> they're called. This invalidates my Environments-Monticello
> integration hypothesis: the bug, whatever it is, lies in Environment.
> After an unload (Monticello browser, right click, unload),
> PointerFinder on: XMLWriter says:
>
> globals: Environment
> bindings: IdentityDictionary
> array: Array
> 1301: Alias
> source: Association
> value: AnObsoleteXMLWriter class
>
> Of course, you should not even be able to evaluate "PointerFinder on:
> XMLWriter" - the class is supposed to be gone, and you should be told
> "there's no such class".
>
> So my next hypothesis is that unloading the package doesn't remove the
> association from the top level Environment's bindings instvar.

>
>
>
> On Tue, Apr 2, 2013 at 6:45 AM, Frank Shearar <[hidden email]>
> wrote:
>
>>
>>
>> uSo it _looks_ like Environments has a slight case of split brain: the
>> contents and bindings instvars seem to do the same job (inspecting
>> "Environment default" and looking at the contents of contents and
>> bindings), and it looks like we're moving things over to bindings
>> (#assocationAt: pulls stuff out of contents, while
>> #associationOrUndeclaredAt: is more recent, and pulls stuff out of
>> bindings. #migrate copies contents' contents into bindings).
>>
>> Does that sound like a fair assessment? If so, I suspect that we
>> simply haven't pushed the migration far enough yet.
>
>
> Sort of.
>
> The 'contents' dictionary is for classes and globals that are declared
> within the environment. The 'bindings' dictionary is for bindings that are
> referenced from methods compiled within the environment. So when you create
> a class, it goes into 'contents', and when you refer to the class in a
> method, that binding gets copied (or aliased) into 'bindings', possibly of
> another environment.