[MC] Overrides and condenseChanges?

Hi Guys -

I'm planning to do a condenseChanges/condenseSources for the Croquet
release and was wondering what this will do to MC overrides (IIRC, the
previous versions are used from the changes file). Can somebody give me
a primer on what I need to keep in mind when I do this?

Thanks,
   - Andreas

Hi,
> Hi Guys -
>
> I'm planning to do a condenseChanges/condenseSources for the Croquet
> release and was wondering what this will do to MC overrides (IIRC, the
> previous versions are used from the changes file). Can somebody give me
> a primer on what I need to keep in mind when I do this?
>
> Thanks,
>    - Andreas

I have noticed condenceSources no longer works in 3.8

Hi Brent -

Brent Pinkney wrote:
> I have noticed condenceSources no longer works in 3.8

Which means what exactly? Since I'll have to fix it any insights, clues,
leads are both relevant and welcome.

Cheers,
   - Andreas

Am 05.04.2006 um 10:45 schrieb Andreas Raab:

> Hi Guys -
>
> I'm planning to do a condenseChanges/condenseSources for the  
> Croquet release and was wondering what this will do to MC overrides  
> (IIRC, the previous versions are used from the changes file).

It will break in rather subtle ways that you'll discover only later -  
we screwed up a few images before the fix.

> Can somebody give me a primer on what I need to keep in mind when I  
> do this?

To make overrides survive the condensing, install this fix first:

        http://bugs.impara.de/view.php?id=2514

It will keep all versions of overridden methods. This is not optimal,  
it would suffice to keep the most recent version for each overriding  
package back to the latest non-overridden version. But there are not  
too many overrides in the system so it doesn't matter much.

Very useful to see if the overrides are correct is this:

        http://bugs.impara.de/view.php?id=2727

It shows a method's category in its version history - without this  
it's almost impossible to fiddle with overrides, like removing an  
override without removing the original method.

To be sure everything went right, go over each and every MC package  
and make sure it does not have unsuspected changes (select the  
repository to compare to, press the "changes" button). If all  
packages were clean before, this should be easy to automate (yet  
another reason for having a clean base).

- Bert -

Hi Adreas,

> Brent Pinkney wrote:
> > I have noticed condenceSources no longer works in 3.8
>
> Which means what exactly? Since I'll have to fix it any insights, clues,
> leads are both relevant and welcome.

I apologise for leaving you with just the bad news: I was dragged away at just
that time for other work stuff.

When I evaluate 'Smalltalk condenseSources' I get a DNU which can be traced to
the following line in #condenseSources:

My version of the method is 'yo 2/24/2005 18:00'. I have formatted the
following for clarity:

        newVersionString := FillInTheBlank
                request: 'Please designate the version for the new source code file...'
                initialAnswer: SourceFileVersionString.

SourceFileVersionString is a class variable which is nil in my image. If I
inspect SystemDictionary and set it to '3', then the method continues and
blows up at the lines:

        "Write all sources with fileIndex 1"
        f := FileStream newFileNamed: SmalltalkImage current sourcesName.
        f header; timeStamp.

The DNU occurs baucase f is nil. f is nil bacause

        SmalltalkImage current sourcesName

returns '/usr/lib/squeak/3.7-7/SqueakV3.sources' which does not exist.

I stopped at this point.

I am willing to assist in correcting this.

Brent

Hi -

Brent Pinkney wrote:
> When I evaluate 'Smalltalk condenseSources' I get a DNU which can be traced to
> the following line in #condenseSources:

Yeah, I see. This one is fairly easy to fix (and I'll bite my tongue
about SystemDictionary vs. SmalltalkImage) but there seems to be another
problem which bothers me a lot more.

Running condenseSources, I end up with a warning that the limit of the
sources file is being reached (during the process) and indeed the
sources file says it's >32MB. However, if I simply file out all classes
I end up with an 18MB file overall. Not sure what this is but it's very,
very troubling.

Cheers,
   - Andreas

On 5-Apr-06, at 12:39 PM, Andreas Raab wrote:
I don't suppose that the method sources are being (re)written to the  
*old* source files, possibly as well as to the new one? Just a  
possibility raised by the surprising size of the file....

Alternatively, it isn't some artifact of multibytefilethingummy  
writing the source in UTF-16 or whatever?

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: CCC: Crash if Carry Clear

tim Rowledge wrote:
> I don't suppose that the method sources are being (re)written to the
> *old* source files, possibly as well as to the new one? Just a
> possibility raised by the surprising size of the file....

Interesting idea. BTW, the deeper I dig into this stuff the more broken
things get. Like this:

f1 := FileStream newFileNamed: 'f1.st'.
f2 := FileStream newFileNamed: 'f2.st'.
ExternalSettings fileOutOn: f1 moveSource: false toFile: 0.
ExternalSettings fileOutOn: f2 moveSource: true toFile: 1.
f1 close.
f2 close.

The result are two files of different sizes with the latter one
completely broken.

Cheers,
   - Andreas

tim Rowledge wrote:
> Alternatively, it isn't some artifact of multibytefilethingummy writing
> the source in UTF-16 or whatever?

Close. I found it. And I don't know how to say it gently but this
qualifies for the worst code I have *EVER* seen in Squeak (I was so
pissed when I found it, I was seriously looking for a sand bag to
deliver a few punches). Have a look at what lurks in
MultiByteFileStream>>open:forWrite:

self localName = (FileDirectory localNameFor: SmalltalkImage current
sourcesName)
        ifTrue: [converter := MacRomanTextConverter new]
        ifFalse: [converter := UTF8TextConverter new]

E.g., when a file has the same LOCAL name as the current image's source
name the converter is set to MacRoman? WTF??? And not in openSourceFiles
(where we actually know that we're dealing with the sources file), no
*right there* in #open:forWrite:. That's freaking unbelievable.

Which, of course, means that when you create *any* file that happens to
have the same name as your sources file (which naturally happens when
you do a condenseSources) it'll end up with a @#^! up text converter.
And the fact that MacRomanTextConverter doesn't raise an error when
invoked with wide characters is just another major bug here since it
would have pointed out immediately that something is wrong and prevented
me from wasting a whole day hunting down that bug in the midst of trying
to get a release out of the door.

So what happens is this: #condenseSources opens the file, the converter
is set to MacRoman. You start filing out and everything works just fine,
until you run into a wide character (like in JapaneseEnvironment).
MacRomanTextConverter (instead of raising an error) lets this slip
through, the "buffer1" iVar in FileStream (instead of raising an error)
lets this slip through, too, and FileStream from here on writes every
character as four bytes.

By my counting, there are at least four major bugs in the above:
1) The code in MultiByteStream>>open:forWrite:
2) The fact that MacRomanTextConverter doesn't raise an error for wide
characters
3) The fact that the sources file is in a non-default (and undetectable)
encoding which means that just fixing #1 will break with the current
source file unless extra care is taken
4) The fact that ByteString automatically mutates into WideString via
#become: (which I always considered dubious; by now I'm convinced it's a
bug)
And of course, when you try to figure out what's going on, you're not
going to use the current sources file name so the series of bugs just
disappear.

I am so mad right now, I'll stop right here.
  - A.

On 5-Apr-06, at 2:09 PM, Andreas Raab wrote:

> tim Rowledge wrote:
>> Alternatively, it isn't some artifact of multibytefilethingummy  
>> writing the source in UTF-16 or whatever?
>
> Close. I found it. And I don't know how to say it gently but this  
> qualifies for the worst code I have *EVER* seen in Squeak

Well that seems a very good candidate for the original problem in  
condenseSources and certainly has a good chance of wining the  
'dreadfullest dreadful code' competition.

It doesn't however explain why your example snippet -
f1 := FileStream newFileNamed: 'f1.st'.
f2 := FileStream newFileNamed: 'f2.st'.
ExternalSettings fileOutOn: f1 moveSource: false toFile: 0.
ExternalSettings fileOutOn: f2 moveSource: true toFile: 1.
f1 close.
f2 close.
- was causing a problem. I found the answer to that though :-)

The clue was that the fileout code - untidy, ugly, poorly formatted  
though it be - is reasonably functional but somehow a completely  
strange set of source strings were getting written out in the f2  
case. What happens is that down a few levels from  
fileOutOn:moveSource:toFile: the list of categories is enumerated and  
each method in the cat is written out. When you reset the method  
source pointer we then set things up for the next problem.

Some classes have somehow ended up with a set of categories like
'--all--'
'private'
'accessing'
'--all--'
and that where the composting excrement impacts the rotating  
impeller. The first '--all--' that we see in a browser is an artifact  
and not really in the category list. The latter one is really there  
due to some ancient futzup that I have vague memories about from  
maybe 3.5ish time. So, we write out the methods in 'private' and  
update them. Then in 'accessing' and finally '--all--' which  
carefully includes all the selectors.... you can see where this is  
going, right? Now we have some methods with source pointers fairly  
randomly set into the file we're writing to but with incorrect  
fileIndex values since after all you were just doing an example. We  
write out each method with crapulated source and hey presto! A  
totally screwed file. Yippee.

Solution? Somebody can write a script to scan all classes and their  
metaclasses for categories explicitly called '--all--' and
a) move any methods actually, really, in the category (which you  
can't find by asking for the methods in '--all--' of course) to some  
where safe
b) deleting the stupid explicit '--all--' category.

Sigh.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Loyalty oaths.Secret searches.No-fly lists.Detention without legal  
recourse.Remind me - who won the cold war?

On 5-Apr-06, at 3:07 PM, tim Rowledge wrote:
>
> Solution? Somebody can write a script to scan all classes and their  
> metaclasses for categories explicitly called '--all--' and
> a) move any methods actually, really, in the category (which you  
> can't find by asking for the methods in '--all--' of course) to  
> some where safe
> b) deleting the stupid explicit '--all--' category.
Supposedly Utilities fixUpProblemsWithAllCategory will do this.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: SDD: Scratch Disk and Die

On 5-Apr-06, at 3:20 PM, tim Rowledge wrote:
Hah! Such irony; it claims that *only* ExternalSettings still has the  
problem. Not your lucky day I guess.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: NBRM: Unconditional No BRanch Multiple

Great catch! Thanks for finding this (it was the next thing on my TODO
list to check).

Cheers,
   - Andreas

tim Rowledge wrote:

Mantis 3435 has a solution for this and is ready to go.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Useful Latin Phrases:- Ventis secundis, tene cursum. = Go with the flow.

On 5 avr. 06, at 21:39, Andreas Raab wrote:

> Hi -
>
> Brent Pinkney wrote:
>> When I evaluate 'Smalltalk condenseSources' I get a DNU which can  
>> be traced to the following line in #condenseSources:
>
> Yeah, I see. This one is fairly easy to fix (and I'll bite my  
> tongue about SystemDictionary vs. SmalltalkImage) but there seems  
> to be another problem which bothers me a lot more.

I know I would have love to find the time and energy to continue to  
implement the solution I proposed, but
life decided otherwise. I hope that I will sometime to continue.