Some notes about SqueakMap dead in trunk images...

...I know :)

But so far I am unclear as to why this is so. I have now tested LOTS of
trunk images backwards and it seems to be broken all the way down to
7179! Funny thing is - my 3.10.1-7175 image works fine.

And there may be a hint here: The 7175 image can be opened by an old
3-9-8 VM. But the later 7179 images can NOT be opened due to some kind
of image format change.

Can that be related to this issue?

Note that you can easily try out an image/VM combo using:

        "SMSqueakMap default reload"

This will try to load the map.xxx.sgz file with the highest number in
the "sm" directory. The failure is the primitive loading the
ImageSegment - so it is a bit hard to figure out what is going on here.

We have had issues in this area before on Linux and FreeBSD, later
kernels. This turned out to be related to buglets in the VM that was
only discovered when the kernel loaded the Squeak process higher up in
the virtual OS memory.

Now... I have no idea if this is the same problem popping up again, in
some shape or form.

regards, Göran

2010/3/15 Göran Krampe <[hidden email]>:
Yep, primitive 99...

On 3/15/2010 1:26 PM, Göran Krampe wrote:
> But so far I am unclear as to why this is so. I have now tested LOTS of
> trunk images backwards and it seems to be broken all the way down to
> 7179! Funny thing is - my 3.10.1-7175 image works fine.
>
> And there may be a hint here: The 7175 image can be opened by an old
> 3-9-8 VM. But the later 7179 images can NOT be opened due to some kind
> of image format change.

Bingo. The issue is that the image format changed for closures and
there's been a bug in the image segment loading code that makes the
primitive fail in cases where the image format is older, but compatible.
I've posted a fix for that in VMMaker but I really think SM should stop
using image segments for data exchange. Image segments are not a
reliable long-term interchange format in an evolving system.

Obviously, this fix requires new VMs.

Cheers,
   - Andreas

On Mon, 15 Mar 2010, Nicolas Cellier wrote:
This won't work if there's no checkpoint available. SMSqueakMap default
loadFull should do it.

>>
>> This will try to load the map.xxx.sgz file with the highest number in the
>> "sm" directory. The failure is the primitive loading the ImageSegment - so
>> it is a bit hard to figure out what is going on here.
>>
>
> Yep, primitive 99...

It's probably not a vm issue, because it works in 3.10.2-7179 (if it
doesn't work for you, just delete the sm folder and try again) with the
latest vm.


Levente

On Mon, 15 Mar 2010, Andreas Raab wrote:
Did you check if this fix solves the SqueakMap issue?


Levente

>
> Cheers,
>  - Andreas
>
>

On 3/15/2010 4:07 PM, Levente Uzonyi wrote:
> It's probably not a vm issue, because it works in 3.10.2-7179 (if it
> doesn't work for you, just delete the sm folder and try again) with the
> latest vm.

It's definitely a VM bug. The VM knows if it's running a pre-closure or
closure image. Due to a mistaken consistency check, image segment
loading will fail if you try to load an image segment from a pre-closure
image into a closure image. That's why it works with old images but not
with closure-converted ones.

Cheers,
   - Andreas

On Mon, 15 Mar 2010, Andreas Raab wrote:

Hah, nice find.


Levente

>
> Cheers,
>  - Andreas
>
>

Hi!

Andreas Raab wrote:
Ah, good that we know what it is then.

> I've posted a fix for that in VMMaker but I really think SM should stop
> using image segments for data exchange. Image segments are not a
> reliable long-term interchange format in an evolving system.

Sure, I am all in agreement. I am slightly amazed we got away with it
this long :)

When it comes to persistence I am pretty sold on CouchDB these days. If
you just have the ability to make trivial HTTP GETs and parse JSON -
then you can load stuff from a CouchDB instance.

Anyway, I really need to sleep but we should take SM by the balls and
make a plan for how to evolve it into the next 10 years :)

SqueakMap was born somewhere in 2002 IIRC.

> Obviously, this fix requires new VMs.
>
> Cheers,
>   - Andreas

Mmmm.

regards, Göran

Hi Göran, I'm glad you're on for getting SqueakMap going, thank you.
CouchDB sounds like it might be a great solution for SqueakMap.  If,
however, you do decide to consider a near drop-in replacement for
ImageSegments, I have been using
MaObjectSerializer>>#fileOut:toFileNamed:in: to save/load my objects
into files intead of ImageSegments for the very reason Andreas said.
With MaObjectSerialization, it is a logical serialization, not
physical, which is of course image independent but also safer because
the Buffer domain maintains full control.  MaObjectSerializer
class>>#objectFromStream loads.  I would be happy help with this if
you want.

I am obviously a big fan of SqueakMap.  I think a bump in
usability/accessibility could reignite and elevate SqueakMap
significantly upward for another 10 years..

Regards,
  Chris

2010/3/15 Göran Krampe <[hidden email]>:

On 3/15/10 9:32 PM, "Göran Krampe" <[hidden email]> wrote:
I think the data could be an .obj as I was using all this years.
I wish get rid of SM in the Core , but sure is good to have it in Basic.
I put my hands on it if all agree.

Edgar

2010/3/15 Göran Krampe <[hidden email]>: Fixing the SqueakSource publish bug would also make SqueakMap a more
obvious choise for distributing stuff.

Karl

Chris Muller wrote:
How much code would that "add" to the image?

The short term fix is of course to use a different kind of
serialization, and hey, perhaps a SmartRefStream works? I haven't had
the time to try.

Andreas did a small XML hack a while back, and any kind of serialization
works as a short term fix. The "only" problem is how to make sure
other/older images works too - at least back to 3.8.

> I am obviously a big fan of SqueakMap.  I think a bump in
> usability/accessibility could reignite and elevate SqueakMap
> significantly upward for another 10 years..

Yeah, and I really would like to move to a different model altogether. A
model which is more distributed. But that is a different story than
fixing SqueakMap for 4.1.

> Regards,
>   Chris

regards, Göran

Chris Muller wrote:
How much code would that "add" to the image?

The short term fix is of course to use a different kind of
serialization, and hey, perhaps a SmartRefStream works? I haven't had
the time to try.

Andreas did a small XML hack a while back, and any kind of serialization
works as a short term fix. The "only" problem is how to make sure
other/older images works too - at least back to 3.8.

> I am obviously a big fan of SqueakMap.  I think a bump in
> usability/accessibility could reignite and elevate SqueakMap
> significantly upward for another 10 years..

Yeah, and I really would like to move to a different model altogether. A
model which is more distributed. But that is a different story than
fixing SqueakMap for 4.1.

> Regards,
>   Chris

regards, Göran

Hi Göran,

> How much code would that "add" to the image?

Umm, it looks like Ma object serialization itself is about 600
methods, and the other pre-req packages would add another 700 or so
methods.  No changes are made to the base system, of course, and you
only have to deal with about 2 or 3 methods of API.  So I think this
has little bearing on anything real for humans (unlike a
non-functional SM, which does); not space, not performance, not
maintainability.  So I do have trouble to understand people's
obsession with this particular metric, but that's another story.

> The short term fix is of course to use a different kind of serialization,
> and hey, perhaps a SmartRefStream works? I haven't had the time to try.

It probably would, and that would be the first thing to try.  Like Ma
object serialization, you would have just 2 or 3 methods max to deal
with, API-wise.  But it might suffer the same risk of
compatibility-issues across all current + future Squeak versions
because it is a physical, not logical, serialization.

> Andreas did a small XML hack a while back, and any kind of serialization
> works as a short term fix. The "only" problem is how to make sure
> other/older images works too - at least back to 3.8.

I agree.

> Yeah, and I really would like to move to a different model altogether. A
> model which is more distributed. But that is a different story than fixing
> SqueakMap for 4.1.

I really like off-line cache it has already.  I just hope we can get
SM working for the trunk image.