Version Histories (was Whats Happening with 3.10. etc.)

Version Histories
was
Whats Happening with 3.10. And how is it going to
proceed?

Hi tim,

>tim Rowledge tim at rowledge.org
>Tue Jan 9 18:21:10 UTC 2007 wrote:
>
>
>On 9-Jan-07, at 10:12 AM, Ralph Johnson wrote:
>
>>
>>> -My final wish is to, as soon as possible, get
back to
>>> a state where version histories are present enough
so
>>> those of us who bug track have a way of finding
out
>>> when and how things changed. Who changed them. And
a
>>> good chance of guessing why and with what intent.
I am
>>> indeed having difficulty with that with 3.9-7067.
>>
>> I'm not sure what you mean by this.  We want to
have a complete
>> history of what we did so it is possible to go back
and time to find
>> when things stopped working.  Why can't you do this
for 3.9?
>
>Ralph, the problem is that during the 3.9 dev cycle
the changes file  
>ran out space and was compressed. So all those
version histories went  
>away :-(
>
>Now , I *do* have a partly complete rework of the
source accessing  
>stuff (and bugger all spare time to do anything with
it) that could  
>avoid that problem.

My suggestion is still to share the code in its
current state by uploading it to the mantis issue.
Something good may come of that.

Your still in the drivers seat, it's your issue.

A stable repair of the source/changes system needs to
be in place before version histories can be gotten at
easily.

Version histories have to be gotten at easily before
broken code can be analysed easily.
And that has to happen before good tests and/or fixes
can be proposed with confidence.

Yours in service, --Jerome Peace


It would, technically, be plausible to add that  
>to a 3.8.1 image and then load all the packages
Marcus built during  
>the 3.9 work (at huge cost to his sanity, sleep and
social life) and  
>*not* do the compression. That should get us a 3.9
with all version  
>info intact. A lot of work - and I'm not volunteering
to do it! - but  
>you might consider it as an option. The extra good
news is that with  
>a more flexible source code access system in place
one could consider  
>ways of storing version history off to the side in
some manner so  
>that you can have it or not at will. Just one of the
benefits of  
>getting away from a string assumption of integer
indices into text  
>files as your only source access.
>
+1
>
>tim
>


 
____________________________________________________________________________________
Need a quick answer? Get one in minutes from people who know.
Ask your question on www.Answers.yahoo.com

Here is my schema, such as it is thus far, for improving the source  
referencing. It's not complete. I need some suggestions for ways to  
tackle a few items.

The problem we face is that a lot needs to be changed in order to use  
anything other than indices into file; so many facilities rely upon  
it. Some serious refactoring would be needed in assorted changelist,  
version listing etc methods. There are complications in ImageSegment  
code too. RemoteString is pretty yucky. Class comments are also mixed  
up in file/index  encoding assumptions. All the source compression,  
tempname-in-method etc code will need altering.

During the writing (and indeed installing later) of the code we need  
all our normal tools to keep working so we can stay sane while doing  
the writing.

The basic idea is to add a proper oop for a source reference object  
so that we can later implement classes that use a database, a web  
search, access multiple files, decompile, guess or whatever.  
Initially I propose simply using an integer with a simple encoding  
scheme (not the rather covoluted one currently in use - take a look  
at StandardSourceFileArray>fileIndexFromSourcePointer: and friends)  
and just reusing the files.

First possibly contentious idea
========================
Do this work in a 3.8.1 image to avoid the change in sources/changes  
files done during the 3.9 cycle and some issues with the introduction  
of Traits. I am *not* suggesting abandoning all the hard work in 3.9  
and anyone implying that I did so will get a late night visitation.  
You Don't Want That.
*After* incorporating the improved source referencing, add the 3.9  
packages but leave out the source condensing step(s). This would  
leave the SqueakV3.sources file untouched and could either leave us  
with a changes file that is simply appended to (and obviously quite  
big) or slightly reformatted (and still pretty big).

Step 1
Add a new source reference ivar to the method properties object that  
all methods now have. Well, except that method properties were added  
somewhere during 3.9 - so they'd have to be added early. Drat. Ideas  
on the minimum disruptive way to add this?

Step 2
Add new method creation methods that do not use the trailerBytes  
stuff. We are not using them yet...

Step 3
Change source access method(s) to check the value of the source  
reference in the methodProperties and use the 'old' access if it is  
nil - which of course it is for now. Also change endPC similarly.

Step 4
change #generate methods to refer to a global flag to say whether  
they use trailerbytes or not. Set the flag *before* that... DAMHIKT!

Step 5
A big do-it to flip the global flag, recompile all methods and  
thereby set the new shiny source pointers.  This is where things can  
get very complicated. We have to decide what to do about the changes  
file and all those historical versions.
a) we could effectively condenseSources so that there is no need to  
worry about handling encrypted pointers to older versions. This would  
of course mean a new .sources file.
b) we could condenseChanges so that any methods in the changelog that  
are also in the old sources file have a correct back pointer, thus  
losing most but not all history.
c) we could try to be very clever and copy the entire history chain  
for each method across to a new changes file, keeping history.
d) we could be insane and try to make code that works out when a  
history pointer is 'old format' and still handle it and then just  
append new format sources to the current changes. I'm not going to  
write that one myself...
Ideas?

Step 6
change the #generate methods to ignore trailerbytes completely, drop  
the global flag and remove it
remove all the redundant method creation methods  etc

Step 7
rework ImageSegment, source code compression, abandonSources, etc  
etc. Ideas?

There is a *lot* needing changing to do this well. I'm horrified how  
poorly factored and written some of this core code is and how it has  
just accreted more and more crap over the last few years. Yuck. Yuck.  
Bad taste in mouth.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: EFBI: Emulate Five-volt Battery Intermittently

On 1/11/07, tim Rowledge <[hidden email]> wrote:
> Here is my schema, such as it is thus far, for improving the source
> referencing. It's not complete. I need some suggestions for ways to
> tackle a few items.

The history of a software project is valuable, and should be
preserved.  But the changes file does not seem to me to be the right
way to do it.

First, most programmers doesn't really care about history back in the
90s.  The people who do care are very important people, so this is not
a reason for ignoring history, but it does indicate that perhaps it is
not necessary to duplicate history with every image.

Second, we are going to split the image into pieces.  In particular,
we plan to have a KernelImage that has no GUI to speak of; Morphic
will be loaded from Monticello.  Keeping the history of Morphic in the
changes file will be pretty wierd at this point.  And it will seem
wierd to have a 2 meg image with a 15 meg changes file.

So, I have an alternative idea.  I haven't looked into this much,
perhaps you can tell me why it won't work.  The idea is to change ONLY
the code to look at past versions of a method, nothing else.  I
imagine that this code looks back through the changes file and, at
some point, says "nothing there".  At that point, we'd change it to
look in the "history database".  I think that the only place that past
versions is used is in a few "version browsers", such as the
VersionsBrowser and the ClassCommentVersionsBrowser.  It seems like it
would be easier to fix them than to fix CompiledMethods.

Anyway, once these version browsers know how to look in a database for
history instead of in the changes file, we can compress the entire
changes file into the sources file, and our final release of 3.10
would have an empty changes file.  When people look at version
history, the version history tools would still look at the changes
file for the changes that they made, but would look in the history
database for older changes.  The current tools would all work the same
way that they do now, but the changes file would be a lot smaller.

What is wrong with this?

-Ralph

 > What is wrong with this?

Mostly it doesn't address the real issue and that's the limitation of
the sources and changes file to 32MB each. That we can preserve the
history by having a larger changes file is nice but not the primary goal
as far as I am concerned.

Cheers,
   - Andreas

Ralph Johnson wrote:

On 11-Jan-07, at 12:36 PM, Andreas Raab wrote:

> > What is wrong with this?
>
> Mostly it doesn't address the real issue and that's the limitation  
> of the sources and changes file to 32MB each. That we can preserve  
> the history by having a larger changes file is nice but not the  
> primary goal as far as I am concerned.

I'd go a bit further by complaining that the issue is really the deep  
assumption of sources being in files and having the files referred to  
by index and having the index and file position encrypted into rather  
less than a SmallInteger's worth of bits.

I agree with Ralph about the desirability of splitting things out,  
keeping histories in some outside dataset etc but I'm fairly sure we  
need to clean out the basic mechanism in order to do that. If we want  
to be able to keep the histories at all I think it makes sense to  
clean things up in a 3.8.1 image before adding the 3.9 work rather  
than trying to extract and synthesize it later.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
The best packed information most resembles random noise.