About unloading of packages in the most recent Squeak 4.1 trunk

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
51 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: About unloading of packages in the most recent Squeak 4.1 trunk

Hannes Hirzel
The comment of class MCMUpdater is currently

<citation>
MCMcmUpdater provides utility methods for updating Monticello packages
from Monticello configurations.

When Monticello configurations are stored in a repository,
MCMcmUpdater acts as an update stream. It first ensures that each
configuration map has been loaded in sequence, then updates the last
configuration map to the most recent version for each specified
package, and finally loads these versions to produce a fully updated
configuration.
</citation>


This class went through a small though important change (see
discussion in this thread). It  should inform people who have not
followed it.

In addition I'd like to have the class comment to give more details so
that a non-specialist in version control / update mechanism issues can
understand it.

My two questions:

1) Is it correct to say that MCMUpdater  handles each Monticello
package so-to-say as having its own update stream? I mean the
succession of Monticello mcz files?

2) Which object is meant here by 'configuration map'?


I'd like to add to the comment.

<proposalForAddition>
MCMUpdater maintains in the variable 'SkipPackages' list of packages
it no longer updates because they have been unloaded for example by
Smalltalk unloadAllKnownPackages.
</proposalForAddition>


And we need a description of the class variables

DefaultUpdateURL
LastUpdateMap
SkipPackages
UpdateMissingPackages

And I'd like to have more.

Could somebody help please?

--Hannes



On 8/25/10, Chris Cunningham <[hidden email]> wrote:
> Oh, and for anyone trying this, you will need to load the
> MonticelloConfigurations before Monticello changes.
>
>

Reply | Threaded
Open this post in threaded view
|

Re: About unloading of packages in the most recent Squeak 4.1 trunk

Bert Freudenberg

On 26.08.2010, at 08:13, Hannes Hirzel wrote:

> The comment of class MCMUpdater is currently
>
> <citation>
> MCMcmUpdater provides utility methods for updating Monticello packages
> from Monticello configurations.
>
> When Monticello configurations are stored in a repository,
> MCMcmUpdater acts as an update stream. It first ensures that each
> configuration map has been loaded in sequence, then updates the last
> configuration map to the most recent version for each specified
> package, and finally loads these versions to produce a fully updated
> configuration.
> </citation>
>
>
> This class went through a small though important change (see
> discussion in this thread). It  should inform people who have not
> followed it.
>
> In addition I'd like to have the class comment to give more details so
> that a non-specialist in version control / update mechanism issues can
> understand it.
>
> My two questions:
>
> 1) Is it correct to say that MCMUpdater  handles each Monticello
> package so-to-say as having its own update stream? I mean the
> succession of Monticello mcz files?

Stating it that way would not be helpful IMHO.

> 2) Which object is meant here by 'configuration map'?

The update configuration loaded from the repository (e.g. "update-xy.42.mcm")

> I'd like to add to the comment.
>
> <proposalForAddition>
> MCMUpdater maintains in the variable 'SkipPackages' list of packages
> it no longer updates because they have been unloaded for example by
> Smalltalk unloadAllKnownPackages.
> </proposalForAddition>

... or because the user does not want them to be updated for some other reason. Being unloaded is just one. I'd rather say that you can prevent packages from being loaded or updated by adding their name to the SkipPackages list.

> And we need a description of the class variables
>
> DefaultUpdateURL

String: the URL of the default repository to update from (MCMUpdater can be used with multiple repos, this one is used when you click the "Update Squeak" menu entry)

> LastUpdateMap

Dictionary of Integer: version number of the latest loaded update map per repository. An update will start from the first config map newer than that.

> SkipPackages

Set of Strings: names of packages to not update (empty by default)

> UpdateMissingPackages

Boolean: if true (default), new packages in the update config map will be loaded unless they are in SkipPackages. If false, packages not currently loaded are always ignored (which may lead to inconsistencies - use at your own risk).

- Bert -



Reply | Threaded
Open this post in threaded view
|

Re: About unloading of packages in the most recent Squeak 4.1 trunk

Andreas.Raab
In reply to this post by Bert Freudenberg
On 8/24/2010 1:16 AM, Bert Freudenberg wrote:
> Manually re-ordering the update map is just a work-around for not being able to update multiple packages together. If we could, the order would not matter for moving classes/methods around.

You know there's a funny thing there: I actually think it may be
possible to load multiple packages together these days as long as we're
incrementally updating. We've fixed so many things in MC loading that I
believe it *may* just be safe to do that.

Except that ... if you would then use that to load all the unloaded
packages back in you'd may try to load the traits package together with
a package using traits or the FFI package together with a package using
the FFI etc. and that can't possibly work because in order to load the
second package the first must be *installed*. Which requires loading
them sequentially :-)

So there you have it. It's kind of a tricky situation.

Cheers,
   - Andreas

Reply | Threaded
Open this post in threaded view
|

Re: About unloading of packages in the most recent Squeak 4.1 trunk

Bert Freudenberg

On 27.08.2010, at 05:18, Andreas Raab wrote:

> On 8/24/2010 1:16 AM, Bert Freudenberg wrote:
>> Manually re-ordering the update map is just a work-around for not being able to update multiple packages together. If we could, the order would not matter for moving classes/methods around.
>
> You know there's a funny thing there: I actually think it may be possible to load multiple packages together these days as long as we're incrementally updating. We've fixed so many things in MC loading that I believe it *may* just be safe to do that.
>
> Except that ... if you would then use that to load all the unloaded packages back in you'd may try to load the traits package together with a package using traits or the FFI package together with a package using the FFI etc. and that can't possibly work because in order to load the second package the first must be *installed*. Which requires loading them sequentially :-)
>
> So there you have it. It's kind of a tricky situation.
>
> Cheers,
>  - Andreas

Well, my point was we only need to disturb the "good" loading order to be able to move classes or methods from one package to the next. If we didn't need to do that, the order could remain strictly by package dependency.

I guess for installing new packages we would not want the simultaneous load, for the reasons you describe. So first update the loaded packages together, then install the missing ones one-by-one?

This still leaves the questions of the order of initializers and pre/post install scripts. Not quite sure about that.

- Bert -


Reply | Threaded
Open this post in threaded view
|

Re: About unloading of packages in the most recent Squeak 4.1 trunk

Hannes Hirzel
In reply to this post by Bert Freudenberg
Bert,
Somebody (you, Bert?) updated the class comment of MCMcmUpdater  one
or two day ago. It is now more comprehensive and answers many of the
question. However some new come up. See below.

--Hannes

On 8/26/10, Bert Freudenberg <[hidden email]> wrote:
>
> On 26.08.2010, at 08:13, Hannes Hirzel wrote:
>
.....
>> 2) Which object is meant here by 'configuration map'?
>
> The update configuration loaded from the repository (e.g.
> "update-xy.42.mcm")

Could you give an actual example of a configuration map with packages
in the unloaded packages list? (I.e name config map xy uses packages
p1, p2, p3)
I assume  it is a file in the repository?
And loaded into the image is it an instance of MCConfiguration?

Is the Monticello Configuration browser used to create them? Why do I
not see any of the configuration maps there?

==================================
The current class comment of MCMcmUpdater
==================================


MCMcmUpdater provides utility methods for updating Monticello packages
from Monticello configurations.

When Monticello configurations are stored in a repository (or
repositories), MCMcmUpdater acts as an update stream. It first ensures
that each configuration map has been loaded in sequence, then updates
the last configuration map to the most recent version for each
specified package, and finally loads these versions to produce a fully
updated configuration.

Currently if a set of packages are unloaded from the image, using this
class to reload them may cause problems, depending on what
dependencies those classes have.  Success is not assured.  Removing
packages via SmalltalkImage>>unloadAllKnownPackages will be
successful, it flags the packages removed so that they are not loaded
by this utility.

If you wish to not have MCMcmUpdater update packages, there are two
ways to handle this:

1) To have MCMcmUpdater not update any packages not currently in the
image set the UpdateMissingPackages preference to false:
                MCMcmUpdater updateMissingPackages: false
        Note that any new packages added to the repositories will not be
picked up when this is turned off.
2) To have MCMcmUpdater not update a specific package, evaluate
                MCMcmUpdater disableUpdatesOfPackage: <packageName>

Class Variables definitions:

DefaultUpdateURL - String: the URL that will be checked by default for
updates.  This would be set for a common standard location to check.

LastUpdateMap - Dictionary of Integer: version number of the last
loaded update map per repository.  Keeps track of the last
configuration map, so that the utility will not have to run through
the full history in the repositories each time you ask to update.

SkipPackages - Set of Strings: names of packages to not update in
MCMcmUpdater (empty by default).

UpdateMissingPackages - Boolean: if true (default), new packages in
the update config map will be loaded unless they are in SkipPackages.
If false, packages not currently loaded in the image will not be
loaded by MCMcmUpdater.  (This can be dangerous if packages are split
- use at your own risk).

Reply | Threaded
Open this post in threaded view
|

Re: About unloading of packages in the most recent Squeak 4.1 trunk

Bert Freudenberg

On 27.08.2010, at 19:26, Hannes Hirzel wrote:

> Bert,
> Somebody (you, Bert?) updated the class comment of MCMcmUpdater  one
> or two day ago.

It was Chris, I just committed his package to Trunk.

> It is now more comprehensive and answers many of the
> question. However some new come up. See below.
>
> --Hannes
>
> On 8/26/10, Bert Freudenberg <[hidden email]> wrote:
>>
>> On 26.08.2010, at 08:13, Hannes Hirzel wrote:
>>
> .....
>>> 2) Which object is meant here by 'configuration map'?
>>
>> The update configuration loaded from the repository (e.g.
>> "update-xy.42.mcm")
>
> Could you give an actual example of a configuration map with packages
> in the unloaded packages list? (I.e name config map xy uses packages
> p1, p2, p3)

Just open the trunk repo, scroll to the bottom in the left list, and click the "update" entry. There you see all the configs. Click "Browse" top open an mcconfig browser on any of them.

> I assume  it is a file in the repository?

Yes.

> And loaded into the image is it an instance of MCConfiguration?

Yes.

> Is the Monticello Configuration browser used to create them?

Yes.

> Why do I not see any of the configuration maps there?

Because they are in the repository. There is no config map in the image. It is only loaded from the repository when updating.

- Bert -

> ==================================
> The current class comment of MCMcmUpdater
> ==================================
>
>
> MCMcmUpdater provides utility methods for updating Monticello packages
> from Monticello configurations.
>
> When Monticello configurations are stored in a repository (or
> repositories), MCMcmUpdater acts as an update stream. It first ensures
> that each configuration map has been loaded in sequence, then updates
> the last configuration map to the most recent version for each
> specified package, and finally loads these versions to produce a fully
> updated configuration.
>
> Currently if a set of packages are unloaded from the image, using this
> class to reload them may cause problems, depending on what
> dependencies those classes have.  Success is not assured.  Removing
> packages via SmalltalkImage>>unloadAllKnownPackages will be
> successful, it flags the packages removed so that they are not loaded
> by this utility.
>
> If you wish to not have MCMcmUpdater update packages, there are two
> ways to handle this:
>
> 1) To have MCMcmUpdater not update any packages not currently in the
> image set the UpdateMissingPackages preference to false:
> MCMcmUpdater updateMissingPackages: false
> Note that any new packages added to the repositories will not be
> picked up when this is turned off.
> 2) To have MCMcmUpdater not update a specific package, evaluate
> MCMcmUpdater disableUpdatesOfPackage: <packageName>
>
> Class Variables definitions:
>
> DefaultUpdateURL - String: the URL that will be checked by default for
> updates.  This would be set for a common standard location to check.
>
> LastUpdateMap - Dictionary of Integer: version number of the last
> loaded update map per repository.  Keeps track of the last
> configuration map, so that the utility will not have to run through
> the full history in the repositories each time you ask to update.
>
> SkipPackages - Set of Strings: names of packages to not update in
> MCMcmUpdater (empty by default).
>
> UpdateMissingPackages - Boolean: if true (default), new packages in
> the update config map will be loaded unless they are in SkipPackages.
> If false, packages not currently loaded in the image will not be
> loaded by MCMcmUpdater.  (This can be dangerous if packages are split
> - use at your own risk).
>


Reply | Threaded
Open this post in threaded view
|

Re: About unloading of packages in the most recent Squeak 4.1 trunk

Hannes Hirzel
Thank you for your explanations, Bert. I now have a more complete view
how the update process works. In particular noteworthy is the fact
that we could have more than one in the future.

I was able to spot the configuration map in the trunk. The map is
called  'update-...mcm" and contains the load order of the packages
(see below).

I found a method #unloadablePackages in  the ScriptLoader class.

^ #('OmniBrowser' 'PlusTools' 'Flash' 'FFI' 'StarSqueak' 'Speech'
'Movie' 'FlexibleVocabularies' '39Deprecated' '39Deprecated'
'PreferenceBrowser' 'ReleaseBuilder'
        'SUnitUI' 'Protocols' 'Sounds')


Within the code of #unloadAllKnownPackages  (class SmalltalkImage)
we have
#( 'ReleaseBuilder' 'ScriptLoader'
                '311Deprecated' '39Deprecated'
                'Universes' 'SMLoader' 'SMBase' 'Installer-Core'
                'VersionNumberTests' 'VersionNumber'
                'Services-Base' 'PreferenceBrowser' 'Nebraska'
                'ToolBuilder-MVC' 'ST80'
                'CollectionsTests' 'GraphicsTests' 'KernelTests'  'MorphicTests'
                'MultilingualTests' 'NetworkTests' 'ToolsTests' 'TraitsTests'
                'SystemChangeNotification-Tests' 'FlexibleVocabularies'
                'EToys' 'Protocols' 'XML-Parser' 'Tests' 'SUnitGUI'

There is some overlap here. Is the class ScriptLoader still used?
Otherwise I would say it should be deleted to avoid confusion. It is
just one class in a package of it's own and the comment says that
ScriptLoader  is a dummy class.

--Hannes


--------------------------------------------------------------------------------------------------------------------------------

update-nice.150.mcm  (the current version of the configuration map)

Squeak-Version-ar.4662
311Deprecated-nice.2
39Deprecated-ar.19
Balloon-ar.16
Collections-nice.370
CollectionsTests-nice.170
Compression-ar.20
Monticello-ar.397
Kernel-nice.482
Compiler-nice.170
EToys-nice.73
Exceptions-sn.29
Files-ar.83
FlexibleVocabularies-ar.12
Graphics-nice.147
GraphicsTests-nice.26
Installer-Core-nice.339
KernelTests-nice.157
MonticelloConfigurations-dtl.77
System-nice.359
Morphic-eem.460
Multilingual-nice.125
MorphicExtras-nice.91
MorphicTests-ar.17
MultilingualTests-HenrikSperreJohansen.10
Nebraska-nice.31
Network-ar.78
NetworkTests-ar.17
PackageInfo-Base-ar.43
PreferenceBrowser-ar.46
Protocols-nice.31
SMBase-nice.112
SMLoader-ar.59
ST80-eem.117
SUnit-ar.80
SUnitGUI-ar.49
ScriptLoader-nice.332
Services-Base-ar.45
Sound-nice.21
SystemChangeNotification-Tests-bp.11
Tests-eem.90
ToolBuilder-Kernel-eem.36
ToolBuilder-MVC-dtl.20
ToolBuilder-Morphic-ar.66
ToolBuilder-SUnit-nice.14
Tools-nice.261
ToolsTests-tbn.5
Traits-nice.285
TraitsTests-ar.9
TrueType-nice.15
Universes-nice.45
VersionNumber-ar.2
XML-Parser-ar.32
ReleaseBuilder-nice.50
ShoutCore-laza.19
VersionNumberTests-nice.3
HelpSystem-Core-tbn.49
HelpSystem-Tests-tbn.12
Help-Squeak-Project-it.5


The Utilities class does the actual update

Utilities updateFromServer.

This is called by the menu entry.


On 8/27/10, Bert Freudenberg <[hidden email]> wrote:

>
> On 27.08.2010, at 19:26, Hannes Hirzel wrote:
>
>> Bert,
>> Somebody (you, Bert?) updated the class comment of MCMcmUpdater  one
>> or two day ago.
>
> It was Chris, I just committed his package to Trunk.
>
>> It is now more comprehensive and answers many of the
>> question. However some new come up. See below.
>>
>> --Hannes
>>
>> On 8/26/10, Bert Freudenberg <[hidden email]> wrote:
>>>
>>> On 26.08.2010, at 08:13, Hannes Hirzel wrote:
>>>
>> .....
>>>> 2) Which object is meant here by 'configuration map'?
>>>
>>> The update configuration loaded from the repository (e.g.
>>> "update-xy.42.mcm")
>>
>> Could you give an actual example of a configuration map with packages
>> in the unloaded packages list? (I.e name config map xy uses packages
>> p1, p2, p3)
>
> Just open the trunk repo, scroll to the bottom in the left list, and click
> the "update" entry. There you see all the configs. Click "Browse" top open
> an mcconfig browser on any of them.
>
>> I assume  it is a file in the repository?
>
> Yes.
>
>> And loaded into the image is it an instance of MCConfiguration?
>
> Yes.
>
>> Is the Monticello Configuration browser used to create them?
>
> Yes.
>
>> Why do I not see any of the configuration maps there?
>
> Because they are in the repository. There is no config map in the image. It
> is only loaded from the repository when updating.
>
> - Bert -
>
>> ==================================
>> The current class comment of MCMcmUpdater
>> ==================================
>>
>>
>> MCMcmUpdater provides utility methods for updating Monticello packages
>> from Monticello configurations.
>>
>> When Monticello configurations are stored in a repository (or
>> repositories), MCMcmUpdater acts as an update stream. It first ensures
>> that each configuration map has been loaded in sequence, then updates
>> the last configuration map to the most recent version for each
>> specified package, and finally loads these versions to produce a fully
>> updated configuration.
>>
>> Currently if a set of packages are unloaded from the image, using this
>> class to reload them may cause problems, depending on what
>> dependencies those classes have.  Success is not assured.  Removing
>> packages via SmalltalkImage>>unloadAllKnownPackages will be
>> successful, it flags the packages removed so that they are not loaded
>> by this utility.
>>
>> If you wish to not have MCMcmUpdater update packages, there are two
>> ways to handle this:
>>
>> 1) To have MCMcmUpdater not update any packages not currently in the
>> image set the UpdateMissingPackages preference to false:
>> MCMcmUpdater updateMissingPackages: false
>> Note that any new packages added to the repositories will not be
>> picked up when this is turned off.
>> 2) To have MCMcmUpdater not update a specific package, evaluate
>> MCMcmUpdater disableUpdatesOfPackage: <packageName>
>>
>> Class Variables definitions:
>>
>> DefaultUpdateURL - String: the URL that will be checked by default for
>> updates.  This would be set for a common standard location to check.
>>
>> LastUpdateMap - Dictionary of Integer: version number of the last
>> loaded update map per repository.  Keeps track of the last
>> configuration map, so that the utility will not have to run through
>> the full history in the repositories each time you ask to update.
>>
>> SkipPackages - Set of Strings: names of packages to not update in
>> MCMcmUpdater (empty by default).
>>
>> UpdateMissingPackages - Boolean: if true (default), new packages in
>> the update config map will be loaded unless they are in SkipPackages.
>> If false, packages not currently loaded in the image will not be
>> loaded by MCMcmUpdater.  (This can be dangerous if packages are split
>> - use at your own risk).
>>
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: About unloading of packages in the most recent Squeak 4.1 trunk

Hannes Hirzel
Hello


I did a retest with the most recent trunk image

The image
Squeak 4.2alpha #10437  - 17.5MB


SmalltalkImage current unloadAllKnownPackages


The image got down to 10.3MB



Further up in this thread Levente writes:
     Hm, I think Casey forgot to do [Smalltalk cleanUp: false] before
saving the image. Doing that saves 2.8MB.

I did
   Smalltalk cleanUp: false
but I only got 10.1MB

Smalltalk cleanUp: true
brought it down to 10.0MB

It opens fine with a Windows Cog VM from
http://www.mirandabanda.org/files/Cog/VM/
(I have not done any further testing yet).

As a whole I am pleased with the result.

So to summarize - a reduced image like the one I got with this process
should take trunk updates properly in most cases?


-- Hannes

Reply | Threaded
Open this post in threaded view
|

Re: About unloading of packages in the most recent Squeak 4.1 trunk

Bert Freudenberg
On 31.08.2010, at 14:19, Hannes Hirzel wrote:

> SmalltalkImage current unloadAllKnownPackages
>
> a reduced image like the one I got with this process should take trunk updates properly in most cases?

Well, hopefully. Try and let us know ;)

- Bert -


Reply | Threaded
Open this post in threaded view
|

Re: About unloading of packages in the most recent Squeak 4.1 trunk

Hannes Hirzel
On 8/31/10, Bert Freudenberg <[hidden email]> wrote:

> On 31.08.2010, at 14:19, Hannes Hirzel wrote:
>
>> SmalltalkImage current unloadAllKnownPackages
>>
>> a reduced image like the one I got with this process should take trunk
>> updates properly in most cases?
>
> Well, hopefully. Try and let us know ;)
>
> - Bert -
>

I updated image which had been through
   SmalltalkImage current unloadAllKnownPackages
and was in the version 10437.

The update process went through. No walkbacks.

Problems:

1) The feedback given is
        Update completed. Current update number: 8546.
     This number appears now in the 'About Squeak' menu.


2) If I right-click on the background I get
    PasteUpMorph doesNotunderstand:  #isStackBackgroundMorph


Conclusion: It does not work proplery yet. However as Andreas, Eliot
and Nicolas currently are doing a lot of refactoring I do not mind. I
will conduct this test later taking an updated regular trunk image, do
the unloading, wait two weeks and then the update.

--Hannes

Reply | Threaded
Open this post in threaded view
|

Re: About unloading of packages in the most recent Squeak 4.1 trunk

Hannes Hirzel
I did another test

I took an image with 10490 and the packages unloaded and upgraded it
to 10502. The same result as with the last test. The version number is
not correct anymore and Morphic is actually broken. If I create a
Morphic project and then want to delete it it gives a walkback.

I am looking forward to a fixed MVC to continue with unloading.
A candiate would then be to unload Morphic and start trying to load
Juan's version of Morphic or Tweak or something else.

Thread starts with
http://lists.squeakfoundation.org/pipermail/squeak-dev/2010-September/153360.html

currently at
http://lists.squeakfoundation.org/pipermail/squeak-dev/2010-September/153489.html
where David T. Lewis managed to get the debugger window in MVC as well
(like Andreas and Florin)

--Hannes


On 9/6/10, Hannes Hirzel <[hidden email]> wrote:

> On 8/31/10, Bert Freudenberg <[hidden email]> wrote:
>> On 31.08.2010, at 14:19, Hannes Hirzel wrote:
>>
>>> SmalltalkImage current unloadAllKnownPackages
>>>
>>> a reduced image like the one I got with this process should take trunk
>>> updates properly in most cases?
>>
>> Well, hopefully. Try and let us know ;)
>>
>> - Bert -
>>
>
> I updated image which had been through
>    SmalltalkImage current unloadAllKnownPackages
> and was in the version 10437.
>
> The update process went through. No walkbacks.
>
> Problems:
>
> 1) The feedback given is
>         Update completed. Current update number: 8546.
>      This number appears now in the 'About Squeak' menu.
>
>
> 2) If I right-click on the background I get
>     PasteUpMorph doesNotunderstand:  #isStackBackgroundMorph
>
>
> Conclusion: It does not work proplery yet. However as Andreas, Eliot
> and Nicolas currently are doing a lot of refactoring I do not mind. I
> will conduct this test later taking an updated regular trunk image, do
> the unloading, wait two weeks and then the update.
>
> --Hannes
>

123