Moving data into new package/image version

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Moving data into new package/image version

Sean P. DeNigris
Administrator
I'm porting some apps that were using simple image-based persistence (ReferenceStream) from 1.3 to 1.4.

The first thing that I ran into was FS class name changes (e.g. FSReference -> FileReference). Since the class wasn't present, I got an errorKeyNotFound [1]. Naively tried:

    Smalltalk globals
        at: #FSReference put: FileReference;
        at: #FSFilesystem put: FileSystem;
        at: #FSAbsolutePath put: AbsolutePath;
        at: #FSUnixStore put: UnixStore.

But got "primitive #primGetPosition: in MultiByteFileStream failed" from ReferenceStream>>next

I have a few ideas, but has anyone gone successfully through this type of process before? What's my best bet? I've been following Fuel and STON (which is quite nice for parsing json from REST APIs b.t.w.), but I'm not clear if either would be helpful here.

What do you suggest?

Thanks,
Sean

[1] Stack trace (heavily snipped)
SystemDictionary(Dictionary)>>errorKeyNotFound:
        Receiver: a SystemDictionary(lots of globals)
        Arguments and temporary variables:
                aKey: #FSReference
        Receiver's instance variables:
                tally: 2920
                array: an Array(#MemoryHandle->MemoryHandle nil nil #BlockClosuresTestCase->Blo...etc...
                cachedClassNames: nil
                cachedNonClassNames: nil

...

ReferenceStream(DataStream)>>readInstance
        Receiver: a ReferenceStream
        Arguments and temporary variables:
                instSize: 2
                aSymbol: #FSReference
                refPosn: 50
                anObject: nil
                newClass: nil
        Receiver's instance variables:
                byteStream: MultiByteFileStream: '/Volumes/Sean''s Universe/sean/.jenkins/jobs/...etc...
                topCall: nil
                basePos: 0
                references: an IdentityDictionary()
                objects: an IdentityDictionary(0->#(nil) 5->a Resources 10->#Resources 21->an O...etc...
                currentReference: 55
                fwdRefEnds: an IdentityDictionary()
                blockers: an IdentityDictionary()
                skipping: an IdentitySet()
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: Moving data into new package/image version

Mariano Martinez Peck


On Sat, May 12, 2012 at 3:03 PM, Sean P. DeNigris <[hidden email]> wrote:
I'm porting some apps that were using simple image-based persistence
(ReferenceStream) from 1.3 to 1.4.

The first thing that I ran into was FS class name changes (e.g. FSReference
-> FileReference). Since the class wasn't present, I got an errorKeyNotFound
[1]. Naively tried:

   Smalltalk globals
       at: #FSReference put: FileReference;
       at: #FSFilesystem put: FileSystem;
       at: #FSAbsolutePath put: AbsolutePath;
       at: #FSUnixStore put: UnixStore.

But got "primitive #primGetPosition: in MultiByteFileStream failed" from
ReferenceStream>>next

I have a few ideas, but has anyone gone successfully through this type of
process before? What's my best bet? I've been following Fuel and STON (which
is quite nice for parsing json from REST APIs b.t.w.), but I'm not clear if
either would be helpful here.


Sorry, I don't have ideas of how to solve that. However, I can only say that Fuel *does* suppor that type of problem (if I understood the problem correctly).
The following test demonstrates an example:

FLMigrationTest >> testClassRename
   
    | pointClass aPoint resultPoint pointClassName |
    pointClass := self newClassWithInstanceVariableNames: 'x y'.
    pointClassName := pointClass name.
    aPoint := pointClass new.
    aPoint instVarNamed: 'x' put: 7.   
    aPoint instVarNamed: 'y' put: 11.
   
    self serialize: aPoint.
    pointClass renameSilently: (pointClassName, 'Renamed') asSymbol.

    self materializer
        migrateClassNamed: pointClassName
        toClass: pointClass.
       
    resultPoint := self materialized.

    self assert: (resultPoint instVarNamed: 'x') = 7.
    self assert: (resultPoint instVarNamed: 'y') = 11.




 
What do you suggest?

Thanks,
Sean

[1] Stack trace (heavily snipped)
SystemDictionary(Dictionary)>>errorKeyNotFound:
       Receiver: a SystemDictionary(lots of globals)
       Arguments and temporary variables:
               aKey:   #FSReference
       Receiver's instance variables:
               tally:  2920
               array:  an Array(#MemoryHandle->MemoryHandle nil nil
#BlockClosuresTestCase->Blo...etc...
               cachedClassNames:       nil
               cachedNonClassNames:    nil

...

ReferenceStream(DataStream)>>readInstance
       Receiver: a ReferenceStream
       Arguments and temporary variables:
               instSize:       2
               aSymbol:        #FSReference
               refPosn:        50
               anObject:       nil
               newClass:       nil
       Receiver's instance variables:
               byteStream:     MultiByteFileStream: '/Volumes/Sean''s
Universe/sean/.jenkins/jobs/...etc...
               topCall:        nil
               basePos:        0
               references:     an IdentityDictionary()
               objects:        an IdentityDictionary(0->#(nil) 5->a Resources 10->#Resources
21->an O...etc...
               currentReference:       55
               fwdRefEnds:     an IdentityDictionary()
               blockers:       an IdentityDictionary()
               skipping:       an IdentitySet()

--
View this message in context: http://forum.world.st/Moving-data-into-new-package-image-version-tp4628820.html
Sent from the Pharo Smalltalk mailing list archive at Nabble.com.




--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: Moving data into new package/image version

Sean P. DeNigris
Administrator
Mariano Martinez Peck wrote
Sorry, I don't have ideas of how to solve that.
I ended up using OB to rename the classes in the old image to match the new class names in the new image, then serialized with ReferenceStream, then deserialized in the new image.

Mariano Martinez Peck wrote
I can only say
that Fuel *does* suppor that type of problem
Would Fuel be better than ReferenceStream for persistence? I thought the use case was saving rarely, but loading often; where serialization is balanced between the two.
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: Moving data into new package/image version

Mariano Martinez Peck


On Wed, May 16, 2012 at 10:37 PM, Sean P. DeNigris <[hidden email]> wrote:

Mariano Martinez Peck wrote
>
> Sorry, I don't have ideas of how to solve that.
I ended up using OB to rename the classes in the old image to match the new
class names in the new image, then serialized with ReferenceStream, then
deserialized in the new image.


Good idea!
 

Mariano Martinez Peck wrote
> I can only say
> that Fuel *does* suppor that type of problem
Would Fuel be better than ReferenceStream for persistence?

Yes, much better.
 
I thought the use
case was saving rarely, but loading often; where serialization is balanced
between the two.

Well, even if that's true, that doesn't mean that Fuel serialization is slow. In fact, it is between 5x and 10x faster than ReferenceStream ;) and like 30x when loading. (I should check the paper for the numbers..i don't remember them).
Fuel is *much* faster than ReferenceStream while also providing more features (as the one you have just experimented).


 

--
View this message in context: http://forum.world.st/Moving-data-into-new-package-image-version-tp4628820p4630561.html
Sent from the Pharo Smalltalk mailing list archive at Nabble.com.




--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: Moving data into new package/image version

Igor Stasenko
A bit tangent to your problem, Sean.
Just want to tell you trick which i use:

Rule #1. never make more than 1 reference to any class external to
your package in your code. an exception is kernel classes.

which means, that you are allowed to do 'Set new' or 'Array new', in
any place of your code
but never 'SomeFooClass new' more than once.

If your code having such dependency, localize it in single method:

fooClass
  ^ FooClass

and then use 'self fooClass' everywhere..
Like that, if you will figure out one day, that sand moves under your
feets, you'll have to change only single method. :)

--
Best regards,
Igor Stasenko.

Reply | Threaded
Open this post in threaded view
|

Re: Moving data into new package/image version

Sean P. DeNigris
Administrator
Igor Stasenko wrote
Rule #1. never make more than 1 reference to any class external to
your package in your code. an exception is kernel classes.
Thanks, Igor. Of course I'm lazy and don't always do this, and in this case, I was treating FS as part of the kernel, I guess a little too prematurely. I sometimes extend this policy even to some kernel classes both because Pharo is progressing so quickly and it can help with testing...

Sean
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: Moving data into new package/image version

Yanni Chiu
In reply to this post by Sean P. DeNigris
On 16/05/12 4:37 PM, Sean P. DeNigris wrote:
>
> Mariano Martinez Peck wrote
>>
>> Sorry, I don't have ideas of how to solve that.
> I ended up using OB to rename the classes in the old image to match
> the new class names in the new image, then serialized with
> ReferenceStream, then deserialized in the new image.

When I don't have the old image anymore, I create a subclass in the new
image (e.g. FileReference subclass: FSReference), do the import, then
fix up the class references.

Mariano - I think the class rename feature is not available in Fuel-1.7
which is the current stable release. Is there a target date for 1.8 to
become stable?


Reply | Threaded
Open this post in threaded view
|

Re: Moving data into new package/image version

Mariano Martinez Peck


On Thu, May 17, 2012 at 9:37 PM, Yanni Chiu <[hidden email]> wrote:
On 16/05/12 4:37 PM, Sean P. DeNigris wrote:

Mariano Martinez Peck wrote

Sorry, I don't have ideas of how to solve that.
I ended up using OB to rename the classes in the old image to match
the new class names in the new image, then serialized with
ReferenceStream, then deserialized in the new image.

When I don't have the old image anymore, I create a subclass in the new
image (e.g. FileReference subclass: FSReference), do the import, then
fix up the class references.


wow..how many good ideas :)
 
Mariano - I think the class rename feature is not available in Fuel-1.7
which is the current stable release.

That's correct.
 
Is there a target date for 1.8 to
become stable?


this week :)  And we will include it in Pharo 2.0 also. 



--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: Moving data into new package/image version

Sean P. DeNigris
Administrator
In reply to this post by Yanni Chiu
Yanni Chiu wrote
I create a subclass in the new
image (e.g. FileReference subclass: FSReference), do the import, then
fix up the class references.
That could be very useful. How do you "fix up the class references"? What methods should I look at?
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: Moving data into new package/image version

fstephany
In reply to this post by Yanni Chiu

> When I don't have the old image anymore, I create a subclass in the new
> image (e.g. FileReference subclass: FSReference), do the import, then
> fix up the class references.

That's what I also did when Migrating from 1.3 to 1.4. It worked fine.

Reply | Threaded
Open this post in threaded view
|

Re: Moving data into new package/image version

Yanni Chiu
In reply to this post by Sean P. DeNigris
On 17/05/12 3:51 PM, Sean P. DeNigris wrote:
>
> That could be very useful. How do you "fix up the class references"? What
> methods should I look at?

Brute force: find all references to the old class (select class name,
cmd-shift-N). Go through the list of methods, paste the new class name,
accept the change.

This works for code. If you have instances of the class, then I guess
even more brute force is needed. Or, you could go back to your original
method, using class re-naming. (Remember that this technique started
with not having an old image, but only having the exported objects).