Obsoletes...

Hi all,

I just took the todays Trunk image (Squeak4.5-13148#712) (NameVersion-Update#CIJob)
and ran
        SystemNavigation default obsoleteBehaviors
and got
        {AnObsoleteBlueSmallLandColorTheme . AnObsoleteSmallLandColorTheme . AnObsoleteSmallLandColorTheme class . AnObsoleteBlueSmallLandColorTheme class . AnObsoleteToolBuilderTests class . AnObsoleteToolBuilderTests . AnObsoleteBindingTest class . AnObsoleteBindingTest}

We should fix that.
It breaks at least
        Smalltalk unloadReloadablePackages

What is the best way?
A postscript in a System Package change?

Best
        -Tobias

On 29 December 2013 12:12, Tobias Pape <[hidden email]> wrote:
> Hi all,
>
> I just took the todays Trunk image (Squeak4.5-13148#712) (NameVersion-Update#CIJob)

Ah, that's nice & specific! Thanks! (Nit: we might as well not bother
with update numbers, because our versioning is broken (*). The
SqueakTrunk builds' TrunkImage artifacts will have an "update number"
hundreds lower than a "full fat" image.

(*) It's broken because an update number doesn't map in any meaningful
way to what's in the image. "Update number" just means "the sum of all
MC numbers of all packages in the latest update map that this image
currently contains." If your Collections package is one ahead of trunk
(because you have a local hack) while your Kernel package is one
behind, you have _the same update number_.

> and ran
>         SystemNavigation default obsoleteBehaviors
> and got
>         {AnObsoleteBlueSmallLandColorTheme . AnObsoleteSmallLandColorTheme . AnObsoleteSmallLandColorTheme class . AnObsoleteBlueSmallLandColorTheme class . AnObsoleteToolBuilderTests class . AnObsoleteToolBuilderTests . AnObsoleteBindingTest class . AnObsoleteBindingTest}
>
> We should fix that.
> It breaks at least
>         Smalltalk unloadReloadablePackages

It also manifests as failing tests:
* http://build.squeak.org/job/SqueakTrunk/712/testReport/junit/Tests.Release/ReleaseTest/testNoObsoleteClasses/
* http://build.squeak.org/job/SqueakTrunk/712/testReport/junit/Tests.Release/ReleaseTest/testUndeclared/

> What is the best way?
> A postscript in a System Package change?

I've been hoping someone would tell me! But Chris told us that the
SqueakTrunk base image's state is a little broken, and that he doesn't
see these tests failing. That would be well worth verifying. If so,
the solution is to hand-roll another base image for SqueakTrunk. I
have a candidate image that I've not quite finished preparing.

As an aside, I think the reason the SqueakTrunk base image broke, at
least in part, was caused by Nicolas and I adding new _necessary_
packages to trunk. For instance, when I factored out
ToolBuilder-Tests, the base image was told not to update packages it
didn't have, so as not to reload unloaded packages (ST80, Universes,
...)... but that also meant it didn't get UpdateStream and
ToolBuilder-Tests.

frank

Hi

[Below, a "fix" to remove those obsoletes…]

On 29.12.2013, at 14:15, Frank Shearar <[hidden email]> wrote:

> On 29 December 2013 12:12, Tobias Pape <[hidden email]> wrote:
>> Hi all,
>>
>> I just took the todays Trunk image (Squeak4.5-13148#712) (NameVersion-Update#CIJob)
>
> Ah, that's nice & specific! Thanks!
(I wrote this mail prior to the other and hoped the order
would stick)
Which is true.
However, for the typically downloaded half-fat TrunkImage from FTP or jenkins,
it is meaningful :)
I think it is comparable to the local revision number in hg: Not meaningful
around the globe but one can work with it in several scenarios.

Well..

Ok. Sounds reasonable.
Apart from that, I try to get the image in a state I can do
>> unloadReloadablePackages
[1]

In that process I crashed the current cog vm several times[2]
as well as other vms with out-of-memory issues.
(You can try it:
        SystemNavigation default obsoleteBehaviors chasePointers
)
Sadly,
        SystemNavigation default obsoleteBehaviors explorePointers
yields no meaningful output :(


Ok. Different approach. The color theme stuff.
By clicking around I found, ColorTheme's CurrentTheme held unto
AnObsoleteBlueSmallLandColorTheme. Hence:

        SystemNavigation default obsoleteBehaviors size. "8"
        ColorTheme current: nil.
        ColorTheme current.
        Behavior flushObsoleteSubclasses.
        Smalltalk garbageCollect; garbageCollect.
        SystemNavigation default obsoleteBehaviors size. " 4"

So, 4 less, but the Menus look different now.

Here, my approach on finding the remaining culprits.
        [| haystack |
        "my pointer finder"
        haystack :=  [:needle | SystemNavigation default allObjectsSelect: [:obj | obj pointsTo: needle]].

        "First, onto the binding test."
        (haystack value: SystemNavigation default obsoleteBehaviors last " AnObsoleteBindingTest"
        ) in: [:stuff |
          "only 2nd and 3d seem interesting"
          (((stuff first: 3) last: 2) collect: [:needle2 | haystack value: needle2]) in: [:stuff2 |
            "first of 1st is interesting"
            (haystack value: stuff2 first first) in: [:stuff3 |
              "first is interesting"
              (haystack value: stuff3 first) "there's an export..." in: [:stuff4 |
                (haystack value: stuff4 first)
                "Smalltalk Enviornment, ie Environment default !!"
        ]]]].

        "Ok, see if the culprit is there"
        [ |assocs |
        assocs := (Environment default exports instVarNamed: 'namespace') associationsSelect: [:assoc |
          assoc value isBehavior
            ifTrue: [assoc value isObsolete]
            ifFalse: [false]]. "an IdentityDictionary(#BindingTest->AnObsoleteBindingTest )"

        assocs do: [:a | Environment default exports forgetName: a key]
        ] value.

        Behavior flushObsoleteSubclasses.
        Smalltalk garbageCollect; garbageCollect.
        SystemNavigation default obsoleteBehaviors size. " 2"

        "Ok, 2 to go"
        "Onto the toolbuildertest"
        (haystack value: SystemNavigation default obsoleteBehaviors first "AnObsoleteToolBuilderTests")
        in: [:stuff5 |
                stuff5 second " SUnitToolBuilderTests class" in: [:suspicious |
                        suspicious superclass " AnObsoleteToolBuilderTests class"
                        "Gotcha!"] ].
        "I don't know... redo?"

        "Redo.. we add artificial instvar here to assure new class."
        ToolBuilderTests subclass: #SUnitToolBuilderTests
                instanceVariableNames: 'foo'
                classVariableNames: ''
                poolDictionaries: ''
                category: 'SUnitGUI-ToolBuilder'.
        ToolBuilderTests subclass: #SUnitToolBuilderTests
                instanceVariableNames: ''
                classVariableNames: ''
                poolDictionaries: ''
                category: 'SUnitGUI-ToolBuilder'.

        Behavior flushObsoleteSubclasses.
        Smalltalk garbageCollect; garbageCollect.
        SystemNavigation default obsoleteBehaviors size. " 0"
Yay!

I attached a better Script to handle this and on the way created the
name uncle class :P

Shall I make a System revision with post script?

Best
        -Tobias


       
[1] Nit: unloadAllKnownPackages is gone forever, right?
[2] See vm-dev

On Sun, 29 Dec 2013, Frank Shearar wrote:

The number only applies to Trunk images. There's intended to be only one
of those (not two or three as it is now).

One word: Environments.

>
>> What is the best way?
>> A postscript in a System Package change?

A postscript won't help, because new obsolete classes will be introduced
because of Environments.

Using a smaller image to build back the full Trunk image doesn't fit the
Trunk process.


Levente

>
> frank
>
>

On 29 December 2013 16:37, Levente Uzonyi <[hidden email]> wrote:
No. The obsolete ColorThemes come from unloading the SmallLand themes,
and have nothing to do with Environments. And please take a moment to
review Tests-fbs.280 in the Inbox, which uses Environments for all the
Monticello tests. You'll probably need Monticello-fbs.581, also in the
Inbox.

The two changes need review.

>>> What is the best way?
>>> A postscript in a System Package change?
>
>
> A postscript won't help, because new obsolete classes will be introduced
> because of Environments.

Explain?

It will.

frank

> Levente
>
>>
>> frank
>>
>>
>

On Sun, 29 Dec 2013, Frank Shearar wrote:

If you run the tests using the Test Runner, then close it, there will be
new obsolete classes. If you explore the pointers to them, you'll find
that some of them are referenced from the global environment's exports.

This might be because the exports of the global environment are broken.
Instead of the public IdentityDictionary, another one is held by the only
export of the global environment. It has all kind of weird stuff, like
#'_windows_which_were_not_collapsed'.

The import chain is also "broken", there's an empty Import at the end. It
doesn't cause any harm, but it's unnecessary.

I mean, it was designed to be done the other way: The Trunk image always
contains all packages managed by the Trunk process, and some packages can
be unloaded to create a smaller image, but that's not a Trunk image
anymore.
This is why the version numbers are "broken".


Levente

On 29 December 2013 18:18, Levente Uzonyi <[hidden email]> wrote:
Ah, OK. I'd actually thought that Smalltalk globals' exports was
empty: my recent hacking on Environments lead me to believe this
because a new Environment couldn't resolve Object, even after
importing Smalltalk globals, unless I previously executed "Smalltalk
globals exportSelf".

So I'm a bit surprised to see exports involved at all.

They're still broken. The only real "version number" we have is a
vector mapping packages to MC version numbers. The update number is
the sum of these version numbers, so it's kind've like the magnitude
of the vector.

But like Tobias says, _assuming_ a full-fat Trunk, and _assuming_ no
fancy trickery like having locally loaded versions higher or lower
than those in trunk, the update number in your image is comparable to
mine.

It occurs to me that we do have a sorta-kinda fuzzy version number in
the update map number.

frank

> Levente
>
>>
>> frank
>>
>>> Levente
>>>
>>>>
>>>> frank

On Sun, Dec 29, 2013 at 5:28 PM, Tobias Pape <[hidden email]> wrote:

Hi

[Below, a "fix" to remove those obsoletes…]

On 29.12.2013, at 14:15, Frank Shearar <[hidden email]> wrote:

> On 29 December 2013 12:12, Tobias Pape <[hidden email]> wrote:
>> Hi all,
>>
>> I just took the todays Trunk image (Squeak4.5-13148#712) (NameVersion-Update#CIJob)
>
> Ah, that's nice & specific! Thanks!

(I wrote this mail prior to the other and hoped the order
would stick)

> (Nit: we might as well not bother
> with update numbers, because our versioning is broken (*). The
> SqueakTrunk builds' TrunkImage artifacts will have an "update number"
> hundreds lower than a "full fat" image.
>
> (*) It's broken because an update number doesn't map in any meaningful
> way to what's in the image. "Update number" just means "the sum of all
> MC numbers of all packages in the latest update map that this image
> currently contains." If your Collections package is one ahead of trunk
> (because you have a local hack) while your Kernel package is one
> behind, you have _the same update number_.

Which is true.
However, for the typically downloaded half-fat TrunkImage from FTP or jenkins,
it is meaningful :)
I think it is comparable to the local revision number in hg: Not meaningful
around the globe but one can work with it in several scenarios.

>
>> and ran
>>        SystemNavigation default obsoleteBehaviors
>> and got
>>        {AnObsoleteBlueSmallLandColorTheme . AnObsoleteSmallLandColorTheme . AnObsoleteSmallLandColorTheme class . AnObsoleteBlueSmallLandColorTheme class . AnObsoleteToolBuilderTests class . AnObsoleteToolBuilderTests . AnObsoleteBindingTest class . AnObsoleteBindingTest}
>>
>> We should fix that.
>> It breaks at least
>>        Smalltalk unloadReloadablePackages
>
> It also manifests as failing tests:
> * http://build.squeak.org/job/SqueakTrunk/712/testReport/junit/Tests.Release/ReleaseTest/testNoObsoleteClasses/
> * http://build.squeak.org/job/SqueakTrunk/712/testReport/junit/Tests.Release/ReleaseTest/testUndeclared/

Well..

>
>> What is the best way?
>> A postscript in a System Package change?
>
> I've been hoping someone would tell me! But Chris told us that the
> SqueakTrunk base image's state is a little broken, and that he doesn't
> see these tests failing. That would be well worth verifying. If so,
> the solution is to hand-roll another base image for SqueakTrunk. I
> have a candidate image that I've not quite finished preparing.
>
> As an aside, I think the reason the SqueakTrunk base image broke, at
> least in part, was caused by Nicolas and I adding new _necessary_
> packages to trunk. For instance, when I factored out
> ToolBuilder-Tests, the base image was told not to update packages it
> didn't have, so as not to reload unloaded packages (ST80, Universes,
> ...)... but that also meant it didn't get UpdateStream and
> ToolBuilder-Tests.

Ok. Sounds reasonable.
Apart from that, I try to get the image in a state I can do
>> unloadReloadablePackages
[1]

In that process I crashed the current cog vm several times[2]
as well as other vms with out-of-memory issues.
(You can try it:
        SystemNavigation default obsoleteBehaviors chasePointers
)
Sadly,
        SystemNavigation default obsoleteBehaviors explorePointers
yields no meaningful output :(

It seems PointersExplorer works on single objects, not a array of objects. So if you look at each item in the list of obsoletes you get valid results:

SystemNavigation default obsoleteBehaviors first explorePointers

Cheers,

Karl

Ok. Different approach. The color theme stuff.
By clicking around I found, ColorTheme's CurrentTheme held unto
AnObsoleteBlueSmallLandColorTheme. Hence:

        SystemNavigation default obsoleteBehaviors size. "8"
        ColorTheme current: nil.
        ColorTheme current.
        Behavior flushObsoleteSubclasses.
        Smalltalk garbageCollect; garbageCollect.
        SystemNavigation default obsoleteBehaviors size. " 4"

So, 4 less, but the Menus look different now.

Here, my approach on finding the remaining culprits.
        [| haystack |
        "my pointer finder"
        haystack :=  [:needle | SystemNavigation default allObjectsSelect: [:obj | obj pointsTo: needle]].

        "First, onto the binding test."
        (haystack value: SystemNavigation default obsoleteBehaviors last " AnObsoleteBindingTest"
        ) in: [:stuff |
          "only 2nd and 3d seem interesting"
          (((stuff first: 3) last: 2) collect: [:needle2 | haystack value: needle2]) in: [:stuff2 |
            "first of 1st is interesting"
            (haystack value: stuff2 first first) in: [:stuff3 |
              "first is interesting"
              (haystack value: stuff3 first) "there's an export..." in: [:stuff4 |
                (haystack value: stuff4 first)
                "Smalltalk Enviornment, ie Environment default !!"
        ]]]].

        "Ok, see if the culprit is there"
        [ |assocs |
        assocs := (Environment default exports instVarNamed: 'namespace') associationsSelect: [:assoc |
          assoc value isBehavior
            ifTrue: [assoc value isObsolete]
            ifFalse: [false]]. "an IdentityDictionary(#BindingTest->AnObsoleteBindingTest )"

        assocs do: [:a | Environment default exports forgetName: a key]
        ] value.

        Behavior flushObsoleteSubclasses.
        Smalltalk garbageCollect; garbageCollect.
        SystemNavigation default obsoleteBehaviors size. " 2"

        "Ok, 2 to go"
        "Onto the toolbuildertest"
        (haystack value: SystemNavigation default obsoleteBehaviors first "AnObsoleteToolBuilderTests")
        in: [:stuff5 |
                stuff5 second " SUnitToolBuilderTests class" in: [:suspicious |
                        suspicious superclass " AnObsoleteToolBuilderTests class"
                        "Gotcha!"] ].
        "I don't know... redo?"

        "Redo.. we add artificial instvar here to assure new class."
        ToolBuilderTests subclass: #SUnitToolBuilderTests
                instanceVariableNames: 'foo'
                classVariableNames: ''
                poolDictionaries: ''
                category: 'SUnitGUI-ToolBuilder'.
        ToolBuilderTests subclass: #SUnitToolBuilderTests
                instanceVariableNames: ''
                classVariableNames: ''
                poolDictionaries: ''
                category: 'SUnitGUI-ToolBuilder'.

        Behavior flushObsoleteSubclasses.
        Smalltalk garbageCollect; garbageCollect.
        SystemNavigation default obsoleteBehaviors size. " 0"
Yay!

I attached a better Script to handle this and on the way created the
name uncle class :P

Shall I make a System revision with post script?

Best
        -Tobias



[1] Nit: unloadAllKnownPackages is gone forever, right?
[2] See vm-dev