[squeak-dev] SkipLists removal

SkipLists contains code under SqueakL as well as under MIT.  Two
contributors have not signed the agreement and appear to be
uncontactable.  SkipLists are not used in the image and are easily
extractable.  To that end I have done so and published a new version of
Collections without them and an external package.

References:

http://bugs.squeak.org/view.php?id=7359

http://source.squeak.org/trunk/Collections-klc.93.mcz

http://www.squeaksource.com/SkipLists.html

Ken

On Sat, Jul 04, 2009 at 01:08:14PM -0500, Ken Causey wrote:
> SkipLists contains code under SqueakL as well as under MIT.  Two
> contributors have not signed the agreement and appear to be
> uncontactable.  SkipLists are not used in the image and are easily
> extractable.  To that end I have done so and published a new version of
> Collections without them and an external package.

I just did a "help... -> update code from server" and my local
image is now up to date with your change. Nice!

Dave

On Sat, 2009-07-04 at 14:44 -0400, David T. Lewis wrote: Oh good, I have to admit I wasn't absolutely sure there wasn't something
more I needed to do.

Ken

Ken Causey wrote:
> Oh good, I have to admit I wasn't absolutely sure there wasn't something
> more I needed to do.

If the change is local to a single package there is not much you need to
do. Monticello is reasonable in this case and the updater will just load
the version of the package. Where it gets nasty is:

* When you rename classes (this may have been addressed in one of the
later MC versions?) as Monticello will delete the old class mercilessly,
leaving all the instances in the dust

* When you move classes between packages where you need to issue an
update.mcm so that you have the destination package before the source
package so that the class doesn't get deleted in the middle (of course,
some of the time the update.mcm will be in the right order and you get
lucky ;-)

* When you have system methods that are modified during load and rely on
new methods in the package. This is by far the biggest issue (and a
problem even with Change Sets). For example,
http://bugs.squeak.org/view.php?id=6535 contains a change to
OrderedCollection>>copyFrom:to: which utilizes postCopyFrom:to: that is
defined in the same place. In change sets you can at least manually
reorder the methods. Monticello does it unpredictably so it may work for
one person and not for another (I wish MC would load methods in a
well-defined order).

To deal with this, you need to make a separate MC package version with
*just* the prerequisite method (in this case postCopyFrom:to:) and issue
an update.mcm for that precise version of the package; this will ensure
the updater loads that precise version of the package before going on to
load the version of the method that utilizes it. When there's a lot of
dependencies this is obviously not a very rewarding process...

Cheers,
   - Andreas

On Sat, 2009-07-04 at 12:18 -0700, Andreas Raab wrote: This change is in fact all about packages, moving code from one package
to another.  But the move in this case happens to be to a package now
outside of trunk so only the packages from which code disappeared remain
and need to be loaded.  Should be OK I think.

Ken Causey wrote:
Yes. This is only ever an issue if you have instances. As long as you
are just shuffling unused code around, Monticello can take care of it
perfectly. So SkipList should be fine; but it would be tricky to move
Array from Collections to Kernel without great caution.

Cheers,
   - Andreas

On Sat, Jul 04, 2009 at 12:18:11PM -0700, Andreas Raab wrote:
No Monticello version deals correctly with this case.
DeltaStreams does, however. SystemEditor can handle this case,
but the Monticello snapshot mechanism simply looks at the state
of the system and stores no metadata about what changed between
versions

> * When you move classes between packages where you need to issue an
> update.mcm so that you have the destination package before the source
> package so that the class doesn't get deleted in the middle (of course,
> some of the time the update.mcm will be in the right order and you get
> lucky ;-)

With MC1.6, if the packages are loaded together, it should
always work, but I don't think I ever tested this. MCM's
probably load packages together; I've never looked at them.

> * When you have system methods that are modified during load and rely on
> new methods in the package. This is by far the biggest issue (and a
> problem even with Change Sets). For example,
> http://bugs.squeak.org/view.php?id=6535 contains a change to
> OrderedCollection>>copyFrom:to: which utilizes postCopyFrom:to: that is
> defined in the same place. In change sets you can at least manually
> reorder the methods. Monticello does it unpredictably so it may work for
> one person and not for another (I wish MC would load methods in a
> well-defined order).

This is never a problem in MC1.6, period. In MC1.6, the new code is
never run until the entire package is loaded; during the load,
the system is static; only after the load is the new code. MC1.6
is able to move instance variables and rename core Monticello
methods during the load, with never a hiccup.

MC1.5 is an improved version of Monticello with a number of bug
fixes and improvements:
- compatibility with both the 3.8 and the 3.9 format of .mcz
  files.
- Better handling of out-of-order loading: loading extensions
  and subclasses of not-yet-loaded classes is delayed until the
  missing class is loaded.
- More versatile package naming; branching and real-world version
  numbers are supported: Monticello-mtf.experimental.1.5.147
- A new dual changes tool, that can quickly compare arbitrary
  versions without loading
- A recent history of commit messages, so you can reuse them
  easily (merged from Pharo)
- Completely rewritten image scanning code, resulting in about
  10x faster snapshotting.
- And some others.

MC1.6 is identical to MC1.5, except that it has a vastly smarter
loader, which first compiles all classes and methods outside the
live class hierarchy, then migrates all methods, classes, and
instances during a single mass-become primitive, making it truly
atomic, and up to 6x faster at loading.

There are currently 2 outstanding bugs in MC1.6:
- No traits support in SystemEditor
  - SystemEditor is the engine that manages the compilation and
    migration of the temporary classes and methods outside the
    class hierarchy. It needs to know a lot about the internal
    structure of Classes to do this, but it was never taught
    about Traits, and I have not yet finished this coding.
- Obsolete classes are sometimes left behind

And one bug in both MC1.5 and 1.6:
- Class initialization methods are run in a different order than
  previous Monticellos.
  - This causes problems when loading some packages from
    Universes, although I forget which packages.

--
Matthew Fulmer -- http://mtfulmer.wordpress.com/

On Sat, Jul 04, 2009 at 04:17:39PM -0400, Matthew Fulmer wrote:
And one bug in MC1.5 but not 1.6:
- MC1.5's loader is marginally more sensitive to method load
  order than MC1.0, because it removes and re-adds methods,
  rather than simply replace them. I'm not sure why it does
  this.

--
Matthew Fulmer -- http://mtfulmer.wordpress.com/

Hi Ken.

I was the main and first contributor of SkipLists (LC). It is OK with me if you decide to remove them (I did them for fun and somehow they got included long ago). I just wanted to say that I've signed the agreement and would sign it again as many times as necessary.


/Leandro Caniglia (lcaniglia caesarsystems com)



Ken Causey wrote:

SkipLists contains code under SqueakL as well as under MIT.  Two
contributors have not signed the agreement and appear to be
uncontactable.  SkipLists are not used in the image and are easily
extractable.  To that end I have done so and published a new version of
Collections without them and an external package.

References:

http://bugs.squeak.org/view.php?id=7359

http://source.squeak.org/trunk/Collections-klc.93.mcz

http://www.squeaksource.com/SkipLists.html

Ken
  

Thanks Leandro.  The primary issue is that Scott Crosby who updated
SkipList a fair amount has not signed to my knowledge.  In fact, for
Etoys 4.0 it appears Yoshiki basically reverted to your earlier version
to address the issue.

However I think it is preferable to have the more up to date version
(including by the way a fix from Nicolas Cellier for which I published a
new version today) under SqueakL outside of the image.  I'm a proponent
of a smaller and simpler core.

The SkipLists squeaksource project

(bad URL below by the way: http://www.squeaksource.com/skiplists.html )

is writable by all, but if you wish to be listed as a developer or admin
just let me know.

Ken

On Sat, 2009-07-04 at 18:00 -0300, Leandro wrote:

Matthew Fulmer wrote:
Wow. This sounds like an excellent reason to switch to MC 1.6 since this
has been a constant nightmare in the back of my for dealing with
incremental updates. Is 1.6 ready for wide-spread use?

Cheers,
   - Andreas

On Sat, 2009-07-04 at 14:56 -0700, Andreas Raab wrote: Keep in mind that 1.6 does not support Traits.  Frankly, I'm a nut about
minimalism and would prefer Traits as an external package if that is at
all possible.  But then that would potentially offload support for
loading packages with Traits to whoever takes on it's maintenance.

Ken

Matthew Fulmer wrote: I am not exactly sure but I think it does this because it is trying to
keep the changes file correct for extensions and overrides to work.

Keith

Andreas Raab wrote: Yes and no,

 I have been banging on about this for 6 months. It used to be ok, but
last time I tried it it left a trail of obsolete classes.

Andreas, have you had a look at my new sceencasts there is a demo of
MC1.6 loading Seaside.

http://www.vimeo.com/groups/squeak/

Keith

On Sat, Jul 04, 2009 at 02:56:04PM -0700, Andreas Raab wrote:
Nearly.

The atomic loader cannot handle Traits, and this is non-trivial
to fix. However, Monticello could fall back to the sequential
loader when it detects a package with traits, as it is able to
handle them. As there are currently very few packages using
traits, this would make the reliability of mc1.6 available to
nearly all developers. Currently, the choice between the atomic
loader and the sequential loader is a user-settable preference
(Preferences > monticello > useAtomicLoader), set false in 1.5
and true in 1.6. This is the only difference between the two
versions, the setting of this switch.

One problem that users will nearly immediately run into,
however, is a long-standing bug regarding the order in which
class initialization methods are run. This causes some packages
(I forget a good example) to fail to load, as the multiple
initialization methods are run in a different order than what
MC1.0 would have run them. Arguably it's bad practice to rely on
the order in which class initialization methods are run, but I
see no reason to give MC1.6 the right to break packages that
used to work.

Those are the only two bugs I know of that actually cause MC1.6
to load a package improperly. I used MC1.6 myself for nearly a
year, and fixed any bugs that caused improper loads. The above
two are the only ones I know of that remain.

It has another bug in which obsolete classes are sometimes left
around, but that is less visible than the above two.

--
Matthew Fulmer -- http://mtfulmer.wordpress.com/

On Sat, Jul 04, 2009 at 12:18:11PM -0700, Andreas Raab wrote:
Speaking of deleting things mercilessly, I'm doing some housecleaning to
move a few methods from the base image into the VMMaker package (see
Mantis 7221). These are just some C code generator methods that apparently
were not recatogorized back when the VMMaker package was split out of the
base image. I have added them to the VMMaker package (VMMaker-dtl.125 on
SqueakSource), and have intentially left them in the image (Kernel and
Collections packages on source.squeak.org/trunk) on the theory that other
VMMaker projects will expect to find these methods in the image for some
time to come.

Some time in the future (maybe in 6 months or so) the recategorized methods
will be removed from the trunk project. In the mean time anyone who loads the
latest VMMaker, Kernel and Collections will have an annoying conflict to deal
with.  As far as I know this is an inherently manual process that just requires
some follow up actions, but if there are any Monticello tricks I should be
aware of, please let me know.

Dave