Smalltalk › Pharo › Pharo Smalltalk Developers

Moving data into new package/image version

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

11 messages Options

Sean P. DeNigris

Moving data into new package/image version

Administrator

I'm porting some apps that were using simple image-based persistence (ReferenceStream) from 1.3 to 1.4.

The first thing that I ran into was FS class name changes (e.g. FSReference -> FileReference). Since the class wasn't present, I got an errorKeyNotFound [1]. Naively tried:

Smalltalk globals
at: #FSReference put: FileReference;
at: #FSFilesystem put: FileSystem;
at: #FSAbsolutePath put: AbsolutePath;
at: #FSUnixStore put: UnixStore.

But got "primitive #primGetPosition: in MultiByteFileStream failed" from ReferenceStream>>next

I have a few ideas, but has anyone gone successfully through this type of process before? What's my best bet? I've been following Fuel and STON (which is quite nice for parsing json from REST APIs b.t.w.), but I'm not clear if either would be helpful here.

What do you suggest?

Thanks,
Sean

[1] Stack trace (heavily snipped)
SystemDictionary(Dictionary)>>errorKeyNotFound:
Receiver: a SystemDictionary(lots of globals)
Arguments and temporary variables:
aKey: #FSReference
Receiver's instance variables:
tally: 2920
array: an Array(#MemoryHandle->MemoryHandle nil nil #BlockClosuresTestCase->Blo...etc...
cachedClassNames: nil
cachedNonClassNames: nil

...

ReferenceStream(DataStream)>>readInstance
Receiver: a ReferenceStream
Arguments and temporary variables:
instSize: 2
aSymbol: #FSReference
refPosn: 50
anObject: nil
newClass: nil
Receiver's instance variables:
byteStream: MultiByteFileStream: '/Volumes/Sean''s Universe/sean/.jenkins/jobs/...etc...
topCall: nil
basePos: 0
references: an IdentityDictionary()
objects: an IdentityDictionary(0->#(nil) 5->a Resources 10->#Resources 21->an O...etc...
currentReference: 55
fwdRefEnds: an IdentityDictionary()
blockers: an IdentityDictionary()
skipping: an IdentitySet()

Cheers,
Sean

Mariano Martinez Peck

Re: Moving data into new package/image version

On Sat, May 12, 2012 at 3:03 PM, Sean P. DeNigris <[hidden email]> wrote:

I'm porting some apps that were using simple image-based persistence
(ReferenceStream) from 1.3 to 1.4.

The first thing that I ran into was FS class name changes (e.g. FSReference
-> FileReference). Since the class wasn't present, I got an errorKeyNotFound
[1]. Naively tried:

Smalltalk globals
at: #FSReference put: FileReference;
at: #FSFilesystem put: FileSystem;
at: #FSAbsolutePath put: AbsolutePath;
at: #FSUnixStore put: UnixStore.

But got "primitive #primGetPosition: in MultiByteFileStream failed" from
ReferenceStream>>next

I have a few ideas, but has anyone gone successfully through this type of
process before? What's my best bet? I've been following Fuel and STON (which
is quite nice for parsing json from REST APIs b.t.w.), but I'm not clear if
either would be helpful here.

Sorry, I don't have ideas of how to solve that. However, I can only say that Fuel *does* suppor that type of problem (if I understood the problem correctly).
The following test demonstrates an example:

FLMigrationTest >> testClassRename

    | pointClass aPoint resultPoint pointClassName |
    pointClass := self newClassWithInstanceVariableNames: 'x y'.
    pointClassName := pointClass name.
    aPoint := pointClass new.
    aPoint instVarNamed: 'x' put: 7.
    aPoint instVarNamed: 'y' put: 11.

    self serialize: aPoint.
    pointClass renameSilently: (pointClassName, 'Renamed') asSymbol.

    self materializer
        migrateClassNamed: pointClassName
        toClass: pointClass.

    resultPoint := self materialized.

    self assert: (resultPoint instVarNamed: 'x') = 7.
    self assert: (resultPoint instVarNamed: 'y') = 11.

What do you suggest?

Thanks,
Sean

[1] Stack trace (heavily snipped)
SystemDictionary(Dictionary)>>errorKeyNotFound:
Receiver: a SystemDictionary(lots of globals)
Arguments and temporary variables:
aKey: #FSReference
Receiver's instance variables:
tally: 2920
array: an Array(#MemoryHandle->MemoryHandle nil nil
#BlockClosuresTestCase->Blo...etc...
cachedClassNames: nil
cachedNonClassNames: nil

...

ReferenceStream(DataStream)>>readInstance
Receiver: a ReferenceStream
Arguments and temporary variables:
instSize: 2
aSymbol: #FSReference
refPosn: 50
anObject: nil
newClass: nil
Receiver's instance variables:
byteStream: MultiByteFileStream: '/Volumes/Sean''s
Universe/sean/.jenkins/jobs/...etc...
topCall: nil
basePos: 0
references: an IdentityDictionary()
objects: an IdentityDictionary(0->#(nil) 5->a Resources 10->#Resources
21->an O...etc...
currentReference: 55
fwdRefEnds: an IdentityDictionary()
blockers: an IdentityDictionary()
skipping: an IdentitySet()

--
View this message in context: http://forum.world.st/Moving-data-into-new-package-image-version-tp4628820.html
Sent from the Pharo Smalltalk mailing list archive at Nabble.com.

--
Mariano
http://marianopeck.wordpress.com

Sean P. DeNigris

Re: Moving data into new package/image version

Administrator

Mariano Martinez Peck wrote

Sorry, I don't have ideas of how to solve that.

I ended up using OB to rename the classes in the old image to match the new class names in the new image, then serialized with ReferenceStream, then deserialized in the new image.

Mariano Martinez Peck wrote

I can only say
that Fuel *does* suppor that type of problem

Would Fuel be better than ReferenceStream for persistence? I thought the use case was saving rarely, but loading often; where serialization is balanced between the two.

Cheers,
Sean

Mariano Martinez Peck

Re: Moving data into new package/image version

On Wed, May 16, 2012 at 10:37 PM, Sean P. DeNigris <[hidden email]> wrote:

Mariano Martinez Peck wrote

>
> Sorry, I don't have ideas of how to solve that.

I ended up using OB to rename the classes in the old image to match the new
class names in the new image, then serialized with ReferenceStream, then
deserialized in the new image.

Good idea!

Mariano Martinez Peck wrote

> I can only say
> that Fuel *does* suppor that type of problem

Would Fuel be better than ReferenceStream for persistence?

Yes, much better.

I thought the use
case was saving rarely, but loading often; where serialization is balanced
between the two.

Well, even if that's true, that doesn't mean that Fuel serialization is slow. In fact, it is between 5x and 10x faster than ReferenceStream ;) and like 30x when loading. (I should check the paper for the numbers..i don't remember them).
Fuel is *much* faster than ReferenceStream while also providing more features (as the one you have just experimented).

--
View this message in context: http://forum.world.st/Moving-data-into-new-package-image-version-tp4628820p4630561.html

Sent from the Pharo Smalltalk mailing list archive at Nabble.com.

--
Mariano
http://marianopeck.wordpress.com

Igor Stasenko

Re: Moving data into new package/image version

A bit tangent to your problem, Sean.
Just want to tell you trick which i use:

Rule #1. never make more than 1 reference to any class external to
your package in your code. an exception is kernel classes.

which means, that you are allowed to do 'Set new' or 'Array new', in
any place of your code
but never 'SomeFooClass new' more than once.

If your code having such dependency, localize it in single method:

fooClass
^ FooClass

and then use 'self fooClass' everywhere..
Like that, if you will figure out one day, that sand moves under your
feets, you'll have to change only single method. :)

--
Best regards,
Igor Stasenko.

Sean P. DeNigris

Re: Moving data into new package/image version

Administrator

Igor Stasenko wrote

Rule #1. never make more than 1 reference to any class external to
your package in your code. an exception is kernel classes.

Thanks, Igor. Of course I'm lazy and don't always do this, and in this case, I was treating FS as part of the kernel, I guess a little too prematurely. I sometimes extend this policy even to some kernel classes both because Pharo is progressing so quickly and it can help with testing...

Sean

Cheers,
Sean

Yanni Chiu

Re: Moving data into new package/image version

In reply to this post by Sean P. DeNigris

On 16/05/12 4:37 PM, Sean P. DeNigris wrote:
>
> Mariano Martinez Peck wrote
>>
>> Sorry, I don't have ideas of how to solve that.
> I ended up using OB to rename the classes in the old image to match
> the new class names in the new image, then serialized with
> ReferenceStream, then deserialized in the new image.

When I don't have the old image anymore, I create a subclass in the new
image (e.g. FileReference subclass: FSReference), do the import, then
fix up the class references.

Mariano - I think the class rename feature is not available in Fuel-1.7
which is the current stable release. Is there a target date for 1.8 to
become stable?

Mariano Martinez Peck

Re: Moving data into new package/image version

On Thu, May 17, 2012 at 9:37 PM, Yanni Chiu <[hidden email]> wrote:

On 16/05/12 4:37 PM, Sean P. DeNigris wrote:

Mariano Martinez Peck wrote

Sorry, I don't have ideas of how to solve that.

I ended up using OB to rename the classes in the old image to match
the new class names in the new image, then serialized with
ReferenceStream, then deserialized in the new image.

When I don't have the old image anymore, I create a subclass in the new
image (e.g. FileReference subclass: FSReference), do the import, then
fix up the class references.

wow..how many good ideas :)

Mariano - I think the class rename feature is not available in Fuel-1.7
which is the current stable release.

That's correct.

Is there a target date for 1.8 to
become stable?

this week :) And we will include it in Pharo 2.0 also.

--
Mariano
http://marianopeck.wordpress.com

Sean P. DeNigris

Re: Moving data into new package/image version

Administrator

In reply to this post by Yanni Chiu

Yanni Chiu wrote

I create a subclass in the new
image (e.g. FileReference subclass: FSReference), do the import, then
fix up the class references.

That could be very useful. How do you "fix up the class references"? What methods should I look at?

Cheers,
Sean

fstephany

Re: Moving data into new package/image version

In reply to this post by Yanni Chiu

> When I don't have the old image anymore, I create a subclass in the new
> image (e.g. FileReference subclass: FSReference), do the import, then
> fix up the class references.

That's what I also did when Migrating from 1.3 to 1.4. It worked fine.

Yanni Chiu

Re: Moving data into new package/image version

In reply to this post by Sean P. DeNigris

On 17/05/12 3:51 PM, Sean P. DeNigris wrote:
>
> That could be very useful. How do you "fix up the class references"? What
> methods should I look at?

Brute force: find all references to the old class (select class name,
cmd-shift-N). Go through the list of methods, paste the new class name,
accept the change.

This works for code. If you have instances of the class, then I guess
even more brute force is needed. Or, you could go back to your original
method, using class re-naming. (Remember that this technique started
with not having an old image, but only having the exported objects).