Object serialization in Pharo

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Object serialization in Pharo

Simon Denier-3
Hi folks

I'm looking at the code and comments of ReferenceStream for object  
serialization and I wonder what is the best strategy to serialize a  
partial model. I mean, I want to store all elements found while  
traversing the graph of the model, but I dont want to store some  
attributes of those elements (i.e., transient attributes).

Any sample?

--
  Simon




_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Reply | Threaded
Open this post in threaded view
|

Re: Object serialization in Pharo

Chris Muller-3
"Ma object serialization" was designed for this purpose.  You simply
override #maTransientVariables to answer an array of the variable
names that are transient.  That part of the graph will not be
traversed during serialization and nil will be stored in its place.



On Tue, Sep 15, 2009 at 8:31 AM, Simon Denier <[hidden email]> wrote:

> Hi folks
>
> I'm looking at the code and comments of ReferenceStream for object
> serialization and I wonder what is the best strategy to serialize a
> partial model. I mean, I want to store all elements found while
> traversing the graph of the model, but I dont want to store some
> attributes of those elements (i.e., transient attributes).
>
> Any sample?
>
> --
>  Simon
>
>
>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Reply | Threaded
Open this post in threaded view
|

Re: Object serialization in Pharo

Schwab,Wilhelm K
In reply to this post by Simon Denier-3
Simon,

I wish you well and will be eager to hear progress reports.  Long ago, I looked at ReferenceStream (or maybe SmartReferenceStream), and did not like what I found.  IIRC, the output for simple examples was much larger (of course, it might not continue to scale so unfavorably) compared that that produced by Dolphin's binary filer.  Another concern was schema changes.

Dolphin has an orderly conversion process that will raise an error if it is not prepared for what it finds; Squeak seemed to want to fix it itself with the aid of the user (not gonna fly in an end-user app or a server process, which is pretty much everything).

Dolphin locates conversion methods in the class of the object being converted or in the class of a proxy for it (that sounded scary to me when I read about it, but it is exactly what one would want in practice).  In contrast, Squeak puts the conversion methods in ReferenceStream.  Yuk!!!  Sorry, don't know what just came over me :)  Of course one can always modify ReferenceStream[*], but imagine the chaos if ReferenceStream were to become widely used.

I have used serializers to good effect and would like to continue doing so.  At the moment, I would look again, but would lean away from ReferenceStream.  SIXX (an XML serializer) is a possibility.  As I recall, its output was not terse (but might compress nicely??).  There originally was minimal to no support for shape changes, but that has been addressed; I cannot speak for how well it has been done because I have not had time to look.  It is on my todo list and might float to the surface soon, though initially I would not need to care about shape changes.

HTH

Bill


[*] I think we recently discussed a Squeak fork that would not allow touching base classes due to packaging limitations.  Not good.  I am not defending the ReferenceStream example, but it would be a big problem on that system, if I understood that thread.


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Simon Denier
Sent: Tuesday, September 15, 2009 8:32 AM
To: Pharo Development
Subject: [Pharo-project] Object serialization in Pharo

Hi folks

I'm looking at the code and comments of ReferenceStream for object serialization and I wonder what is the best strategy to serialize a partial model. I mean, I want to store all elements found while traversing the graph of the model, but I dont want to store some attributes of those elements (i.e., transient attributes).

Any sample?

--
  Simon




_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Reply | Threaded
Open this post in threaded view
|

Re: Object serialization in Pharo

Henrik Sperre Johansen

On Sep 15, 2009, at 4:16 46PM, Schwab,Wilhelm K wrote:

> Simon,
>
> I wish you well and will be eager to hear progress reports.  Long  
> ago, I looked at ReferenceStream (or maybe SmartReferenceStream),  
> and did not like what I found.  IIRC, the output for simple examples  
> was much larger (of course, it might not continue to scale so  
> unfavorably) compared that that produced by Dolphin's binary filer.  
> Another concern was schema changes.
>
> Dolphin has an orderly conversion process that will raise an error  
> if it is not prepared for what it finds; Squeak seemed to want to  
> fix it itself with the aid of the user (not gonna fly in an end-user  
> app or a server process, which is pretty much everything).
>
> Dolphin locates conversion methods in the class of the object being  
> converted or in the class of a proxy for it (that sounded scary to  
> me when I read about it, but it is exactly what one would want in  
> practice).
Sounds alot like BOSS in VisualWorks :)
The only annoying thing I've found using that, is the fact you never  
get prompted to update the conversion method when you modify a class  
which already contains a binaryReadBlockFrom: method. (Kind of hard to  
know if you store objects of the kind to disk if it doesn't have one  
already)
Believe it or not, but it's quite easy to forget when doing some  
change, and you have no tests importing old variations of all the  
objects you file to disk in your project.

> In contrast, Squeak puts the conversion methods in ReferenceStream.  
> Yuk!!!  Sorry, don't know what just came over me :)  Of course one  
> can always modify ReferenceStream[*], but imagine the chaos if  
> ReferenceStream were to become widely used.
IIRC from looking briefly at it when reviewing some removal patch  
which touched it, it checks first if class has conversion method, THEN  
it tries to autoconvert (silently if it considers itself  
"successful", yay).
Also, if I read the code correctly, it only checks for class structure  
differences, so if you've, say, switched the order of two instvars  
since you stored an old object to disk, it'll file in just fine, but  
with the data in incorrect places...
In short, I imagine using ReferenceStream could lead to some _very_  
frustrating debugging sessions later on in the project, consider  
yourself warned :)

Cheers,
Henry

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Reply | Threaded
Open this post in threaded view
|

Re: Object serialization in Pharo

Simon Denier-3
In reply to this post by Chris Muller-3

On 15 sept. 09, at 16:08, Chris Muller wrote:

> "Ma object serialization" was designed for this purpose.  You simply
> override #maTransientVariables to answer an array of the variable
> names that are transient.  That part of the graph will not be
> traversed during serialization and nil will be stored in its place.

OK thanks, the wiki offering the doc is down for the moment, but the  
seaside book offers a few hints about the subject.

>
>
>
> On Tue, Sep 15, 2009 at 8:31 AM, Simon Denier  
> <[hidden email]> wrote:
>> Hi folks
>>
>> I'm looking at the code and comments of ReferenceStream for object
>> serialization and I wonder what is the best strategy to serialize a
>> partial model. I mean, I want to store all elements found while
>> traversing the graph of the model, but I dont want to store some
>> attributes of those elements (i.e., transient attributes).
>>
>> Any sample?
>>
>> --
>>  Simon
>>
>>
>>
>>
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

--
  Simon




_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Reply | Threaded
Open this post in threaded view
|

Re: Object serialization in Pharo

Stéphane Ducasse
In reply to this post by Schwab,Wilhelm K

On Sep 15, 2009, at 4:16 PM, Schwab,Wilhelm K wrote:

> Simon,
>
> I wish you well and will be eager to hear progress reports.  Long  
> ago, I looked at ReferenceStream (or maybe SmartReferenceStream),  
> and did not like what I found.  IIRC, the output for simple examples  
> was much larger (of course, it might not continue to scale so  
> unfavorably) compared that that produced by Dolphin's binary filer.  
> Another concern was schema changes.
>
> Dolphin has an orderly conversion process that will raise an error  
> if it is not prepared for what it finds; Squeak seemed to want to  
> fix it itself with the aid of the user (not gonna fly in an end-user  
> app or a server process, which is pretty much everything).
>
> Dolphin locates conversion methods in the class of the object being  
> converted or in the class of a proxy for it (that sounded scary to  
> me when I read about it, but it is exactly what one would want in  
> practice).  In contrast, Squeak puts the conversion methods in  
> ReferenceStream.  Yuk!!!  Sorry, don't know what just came over  
> me :)  Of course one can always modify ReferenceStream[*], but  
> imagine the chaos if ReferenceStream were to become widely used.
>
> I have used serializers to good effect and would like to continue  
> doing so.  At the moment, I would look again, but would lean away  
> from ReferenceStream.

I started to write some tests in the PharoTaskForces.
If people want to join effort because before changing I would like to  
know what is there.

> SIXX (an XML serializer) is a possibility.  As I recall, its output  
> was not terse (but might compress nicely??).  There originally was  
> minimal to no support for shape changes, but that has been  
> addressed; I cannot speak for how well it has been done because I  
> have not had time to look.  It is on my todo list and might float to  
> the surface soon, though initially I would not need to care about  
> shape changes.
>
> HTH
>
> Bill
>
>
> [*] I think we recently discussed a Squeak fork that would not allow  
> touching base classes due to packaging limitations.

really?

> Not good.  I am not defending the ReferenceStream example, but it  
> would be a big problem on that system, if I understood that thread.

I would like to fix that
We need a good object serializer.


>
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]
> ] On Behalf Of Simon Denier
> Sent: Tuesday, September 15, 2009 8:32 AM
> To: Pharo Development
> Subject: [Pharo-project] Object serialization in Pharo
>
> Hi folks
>
> I'm looking at the code and comments of ReferenceStream for object  
> serialization and I wonder what is the best strategy to serialize a  
> partial model. I mean, I want to store all elements found while  
> traversing the graph of the model, but I dont want to store some  
> attributes of those elements (i.e., transient attributes).
>
> Any sample?
>
> --
>  Simon
>
>
>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Reply | Threaded
Open this post in threaded view
|

Re: Object serialization in Pharo

Stéphane Ducasse
In reply to this post by Henrik Sperre Johansen
Ok good to know

Stef
On Sep 15, 2009, at 4:48 PM, Henrik Johansen wrote:

>
> On Sep 15, 2009, at 4:16 46PM, Schwab,Wilhelm K wrote:
>
>> Simon,
>>
>> I wish you well and will be eager to hear progress reports.  Long
>> ago, I looked at ReferenceStream (or maybe SmartReferenceStream),
>> and did not like what I found.  IIRC, the output for simple examples
>> was much larger (of course, it might not continue to scale so
>> unfavorably) compared that that produced by Dolphin's binary filer.
>> Another concern was schema changes.
>>
>> Dolphin has an orderly conversion process that will raise an error
>> if it is not prepared for what it finds; Squeak seemed to want to
>> fix it itself with the aid of the user (not gonna fly in an end-user
>> app or a server process, which is pretty much everything).
>>
>> Dolphin locates conversion methods in the class of the object being
>> converted or in the class of a proxy for it (that sounded scary to
>> me when I read about it, but it is exactly what one would want in
>> practice).
> Sounds alot like BOSS in VisualWorks :)
> The only annoying thing I've found using that, is the fact you never
> get prompted to update the conversion method when you modify a class
> which already contains a binaryReadBlockFrom: method. (Kind of hard to
> know if you store objects of the kind to disk if it doesn't have one
> already)
> Believe it or not, but it's quite easy to forget when doing some
> change, and you have no tests importing old variations of all the
> objects you file to disk in your project.
>
>> In contrast, Squeak puts the conversion methods in ReferenceStream.
>> Yuk!!!  Sorry, don't know what just came over me :)  Of course one
>> can always modify ReferenceStream[*], but imagine the chaos if
>> ReferenceStream were to become widely used.
> IIRC from looking briefly at it when reviewing some removal patch
> which touched it, it checks first if class has conversion method, THEN
> it tries to autoconvert (silently if it considers itself
> "successful", yay).
> Also, if I read the code correctly, it only checks for class structure
> differences, so if you've, say, switched the order of two instvars
> since you stored an old object to disk, it'll file in just fine, but
> with the data in incorrect places...
> In short, I imagine using ReferenceStream could lead to some _very_
> frustrating debugging sessions later on in the project, consider
> yourself warned :)
>
> Cheers,
> Henry
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Reply | Threaded
Open this post in threaded view
|

Re: Object serialization in Pharo

Miguel Cobá
In reply to this post by Simon Denier-3
El mar, 15-09-2009 a las 15:31 +0200, Simon Denier escribió:
> Hi folks
>
> I'm looking at the code and comments of ReferenceStream for object  
> serialization and I wonder what is the best strategy to serialize a  
> partial model. I mean, I want to store all elements found while  
> traversing the graph of the model, but I dont want to store some  
> attributes of those elements (i.e., transient attributes).
>
> Any sample?

SIXX or JSON are very good options (JSON doens't serialize dates so..)

>
> --
>   Simon
>
>
>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
--
Miguel Cobá
http://miguel.leugim.com.mx


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Reply | Threaded
Open this post in threaded view
|

Re: Object serialization in Pharo

Schwab,Wilhelm K
In reply to this post by Stéphane Ducasse
Stef,

The packaging constraint had nothing to do with Pharo.  See the [ANN] thread on #readStream replacing ReadStream on: (around 4 Sept).  SecureSqueak is the system with the apparent limitation; it really does have that constraint, it will not be of much use to me.

Unit tests for a serializer would be well-taken, but I will largely have to start over.  Choosing a filer seems to be the first step.  Squeak's ReferenceStream did not strike me as a good option; if you wish to argue in its favor, please do so.  Otherwise, I would like to see SIXX get a good evaluation, as well as any other options that should be considered.

Bill


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Stéphane Ducasse
Sent: Tuesday, September 15, 2009 10:06 AM
To: [hidden email]
Subject: Re: [Pharo-project] Object serialization in Pharo


On Sep 15, 2009, at 4:16 PM, Schwab,Wilhelm K wrote:

> Simon,
>
> I wish you well and will be eager to hear progress reports.  Long ago,
> I looked at ReferenceStream (or maybe SmartReferenceStream), and did
> not like what I found.  IIRC, the output for simple examples was much
> larger (of course, it might not continue to scale so
> unfavorably) compared that that produced by Dolphin's binary filer.  
> Another concern was schema changes.
>
> Dolphin has an orderly conversion process that will raise an error if
> it is not prepared for what it finds; Squeak seemed to want to fix it
> itself with the aid of the user (not gonna fly in an end-user app or a
> server process, which is pretty much everything).
>
> Dolphin locates conversion methods in the class of the object being
> converted or in the class of a proxy for it (that sounded scary to me
> when I read about it, but it is exactly what one would want in
> practice).  In contrast, Squeak puts the conversion methods in
> ReferenceStream.  Yuk!!!  Sorry, don't know what just came over me :)  
> Of course one can always modify ReferenceStream[*], but imagine the
> chaos if ReferenceStream were to become widely used.
>
> I have used serializers to good effect and would like to continue
> doing so.  At the moment, I would look again, but would lean away from
> ReferenceStream.

I started to write some tests in the PharoTaskForces.
If people want to join effort because before changing I would like to know what is there.

> SIXX (an XML serializer) is a possibility.  As I recall, its output
> was not terse (but might compress nicely??).  There originally was
> minimal to no support for shape changes, but that has been addressed;
> I cannot speak for how well it has been done because I have not had
> time to look.  It is on my todo list and might float to the surface
> soon, though initially I would not need to care about shape changes.
>
> HTH
>
> Bill
>
>
> [*] I think we recently discussed a Squeak fork that would not allow
> touching base classes due to packaging limitations.

really?

> Not good.  I am not defending the ReferenceStream example, but it
> would be a big problem on that system, if I understood that thread.

I would like to fix that
We need a good object serializer.


>
>
> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]
> ] On Behalf Of Simon Denier
> Sent: Tuesday, September 15, 2009 8:32 AM
> To: Pharo Development
> Subject: [Pharo-project] Object serialization in Pharo
>
> Hi folks
>
> I'm looking at the code and comments of ReferenceStream for object
> serialization and I wonder what is the best strategy to serialize a
> partial model. I mean, I want to store all elements found while
> traversing the graph of the model, but I dont want to store some
> attributes of those elements (i.e., transient attributes).
>
> Any sample?
>
> --
>  Simon
>
>
>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Reply | Threaded
Open this post in threaded view
|

Re: Object serialization in Pharo

Stéphane Ducasse
> Stef,
>
> The packaging constraint had nothing to do with Pharo.  See the  
> [ANN] thread on #readStream replacing ReadStream on: (around 4  
> Sept).  SecureSqueak is the system with the apparent limitation; it  
> really does have that constraint, it will not be of much use to me.
>
> Unit tests for a serializer would be well-taken, but I will largely  
> have to start over.  Choosing a filer seems to be the first step.  
> Squeak's ReferenceStream did not strike me as a good option; if you  
> wish to argue in its favor, please do so.  Otherwise, I would like  
> to see SIXX get a good evaluation, as well as any other options that  
> should be considered.

I'm sure that I want an XML parser in the way but why not.
I started to write some tests just to learn and document. I would even  
have no problem to throw away everything
it there is a better solution. Now what tests will let enable us to  
change and nothing what we change.

Stef

>
> Bill
>
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]
> ] On Behalf Of Stéphane Ducasse
> Sent: Tuesday, September 15, 2009 10:06 AM
> To: [hidden email]
> Subject: Re: [Pharo-project] Object serialization in Pharo
>
>
> On Sep 15, 2009, at 4:16 PM, Schwab,Wilhelm K wrote:
>
>> Simon,
>>
>> I wish you well and will be eager to hear progress reports.  Long  
>> ago,
>> I looked at ReferenceStream (or maybe SmartReferenceStream), and did
>> not like what I found.  IIRC, the output for simple examples was much
>> larger (of course, it might not continue to scale so
>> unfavorably) compared that that produced by Dolphin's binary filer.
>> Another concern was schema changes.
>>
>> Dolphin has an orderly conversion process that will raise an error if
>> it is not prepared for what it finds; Squeak seemed to want to fix it
>> itself with the aid of the user (not gonna fly in an end-user app  
>> or a
>> server process, which is pretty much everything).
>>
>> Dolphin locates conversion methods in the class of the object being
>> converted or in the class of a proxy for it (that sounded scary to me
>> when I read about it, but it is exactly what one would want in
>> practice).  In contrast, Squeak puts the conversion methods in
>> ReferenceStream.  Yuk!!!  Sorry, don't know what just came over me :)
>> Of course one can always modify ReferenceStream[*], but imagine the
>> chaos if ReferenceStream were to become widely used.
>>
>> I have used serializers to good effect and would like to continue
>> doing so.  At the moment, I would look again, but would lean away  
>> from
>> ReferenceStream.
>
> I started to write some tests in the PharoTaskForces.
> If people want to join effort because before changing I would like  
> to know what is there.
>
>> SIXX (an XML serializer) is a possibility.  As I recall, its output
>> was not terse (but might compress nicely??).  There originally was
>> minimal to no support for shape changes, but that has been addressed;
>> I cannot speak for how well it has been done because I have not had
>> time to look.  It is on my todo list and might float to the surface
>> soon, though initially I would not need to care about shape changes.
>>
>> HTH
>>
>> Bill
>>
>>
>> [*] I think we recently discussed a Squeak fork that would not allow
>> touching base classes due to packaging limitations.
>
> really?
>
>> Not good.  I am not defending the ReferenceStream example, but it
>> would be a big problem on that system, if I understood that thread.
>
> I would like to fix that
> We need a good object serializer.
>
>
>>
>>
>> -----Original Message-----
>> From: [hidden email]
>> [mailto:[hidden email]
>> ] On Behalf Of Simon Denier
>> Sent: Tuesday, September 15, 2009 8:32 AM
>> To: Pharo Development
>> Subject: [Pharo-project] Object serialization in Pharo
>>
>> Hi folks
>>
>> I'm looking at the code and comments of ReferenceStream for object
>> serialization and I wonder what is the best strategy to serialize a
>> partial model. I mean, I want to store all elements found while
>> traversing the graph of the model, but I dont want to store some
>> attributes of those elements (i.e., transient attributes).
>>
>> Any sample?
>>
>> --
>> Simon
>>
>>
>>
>>
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Reply | Threaded
Open this post in threaded view
|

Re: Object serialization in Pharo

Schwab,Wilhelm K
Stef,

Henry's warning tells me I'm not alone in being afraid of Squeak's serializer.  A while ago, I had settled on SIXX as the likely best option for me, but will certainly consider other options.  For the Dolphin filer, my tests mostly center on loading objects with old schemas.  It was fairly easy to do because I had old data in abundance, and simply loaded some of it into new images.  For Pharo, we might write tests that compile a class, serialize, make some changes to the class and reload the "old" data. The details will probably depend on the serializer(s) we select.

My plan has been to start out using SIXX for network transport of data between like images (where schema changes are not a concern) and then begin to look at archival uses if that goes well.

Does that answer your question?

Bill


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Stéphane Ducasse
Sent: Tuesday, September 15, 2009 1:19 PM
To: [hidden email]
Subject: Re: [Pharo-project] Object serialization in Pharo

> Stef,
>
> The packaging constraint had nothing to do with Pharo.  See the [ANN]
> thread on #readStream replacing ReadStream on: (around 4 Sept).  
> SecureSqueak is the system with the apparent limitation; it really
> does have that constraint, it will not be of much use to me.
>
> Unit tests for a serializer would be well-taken, but I will largely  
> have to start over.  Choosing a filer seems to be the first step.  
> Squeak's ReferenceStream did not strike me as a good option; if you
> wish to argue in its favor, please do so.  Otherwise, I would like to
> see SIXX get a good evaluation, as well as any other options that
> should be considered.

I'm sure that I want an XML parser in the way but why not.
I started to write some tests just to learn and document. I would even have no problem to throw away everything it there is a better solution. Now what tests will let enable us to change and nothing what we change.

Stef

>
> Bill
>
>
> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]
> ] On Behalf Of Stéphane Ducasse
> Sent: Tuesday, September 15, 2009 10:06 AM
> To: [hidden email]
> Subject: Re: [Pharo-project] Object serialization in Pharo
>
>
> On Sep 15, 2009, at 4:16 PM, Schwab,Wilhelm K wrote:
>
>> Simon,
>>
>> I wish you well and will be eager to hear progress reports.  Long
>> ago, I looked at ReferenceStream (or maybe SmartReferenceStream), and
>> did not like what I found.  IIRC, the output for simple examples was
>> much larger (of course, it might not continue to scale so
>> unfavorably) compared that that produced by Dolphin's binary filer.
>> Another concern was schema changes.
>>
>> Dolphin has an orderly conversion process that will raise an error if
>> it is not prepared for what it finds; Squeak seemed to want to fix it
>> itself with the aid of the user (not gonna fly in an end-user app or
>> a server process, which is pretty much everything).
>>
>> Dolphin locates conversion methods in the class of the object being
>> converted or in the class of a proxy for it (that sounded scary to me
>> when I read about it, but it is exactly what one would want in
>> practice).  In contrast, Squeak puts the conversion methods in
>> ReferenceStream.  Yuk!!!  Sorry, don't know what just came over me :)
>> Of course one can always modify ReferenceStream[*], but imagine the
>> chaos if ReferenceStream were to become widely used.
>>
>> I have used serializers to good effect and would like to continue
>> doing so.  At the moment, I would look again, but would lean away
>> from ReferenceStream.
>
> I started to write some tests in the PharoTaskForces.
> If people want to join effort because before changing I would like to
> know what is there.
>
>> SIXX (an XML serializer) is a possibility.  As I recall, its output
>> was not terse (but might compress nicely??).  There originally was
>> minimal to no support for shape changes, but that has been addressed;
>> I cannot speak for how well it has been done because I have not had
>> time to look.  It is on my todo list and might float to the surface
>> soon, though initially I would not need to care about shape changes.
>>
>> HTH
>>
>> Bill
>>
>>
>> [*] I think we recently discussed a Squeak fork that would not allow
>> touching base classes due to packaging limitations.
>
> really?
>
>> Not good.  I am not defending the ReferenceStream example, but it
>> would be a big problem on that system, if I understood that thread.
>
> I would like to fix that
> We need a good object serializer.
>
>
>>
>>
>> -----Original Message-----
>> From: [hidden email]
>> [mailto:[hidden email]
>> ] On Behalf Of Simon Denier
>> Sent: Tuesday, September 15, 2009 8:32 AM
>> To: Pharo Development
>> Subject: [Pharo-project] Object serialization in Pharo
>>
>> Hi folks
>>
>> I'm looking at the code and comments of ReferenceStream for object
>> serialization and I wonder what is the best strategy to serialize a
>> partial model. I mean, I want to store all elements found while
>> traversing the graph of the model, but I dont want to store some
>> attributes of those elements (i.e., transient attributes).
>>
>> Any sample?
>>
>> --
>> Simon
>>
>>
>>
>>
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Reply | Threaded
Open this post in threaded view
|

Re: Object serialization in Pharo

Stéphane Ducasse
> Stef,
>
> Henry's warning tells me I'm not alone in being afraid of Squeak's  
> serializer.  A while ago, I had settled on SIXX as the likely best  
> option for me, but will certainly consider other options.  For the  
> Dolphin filer, my tests mostly center on loading objects with old  
> schemas.  It was fairly easy to do because I had old data in  
> abundance, and simply loaded some of it into new images.  For Pharo,  
> we might write tests that compile a class, serialize, make some  
> changes to the class and reload the "old" data. The details will  
> probably depend on the serializer(s) we select.
>
> My plan has been to start out using SIXX for network transport of  
> data between like images (where schema changes are not a concern)  
> and then begin to look at archival uses if that goes well.
>
> Does that answer your question?

ok we will evaluate SIXX.
Now my point is that we should have good solutions: so if  
referenceStream can be fix we fix it else we trash it.
Half-baked solutions just hinder progress.

My goal is this community is to set the cap (fun for a lighthouse) and  
the cap for 1.2 and other releases.
I want a neat nice robust working documented system and each step  
going in that direction is an important one.
Stef


>
> Bill
>
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]
> ] On Behalf Of Stéphane Ducasse
> Sent: Tuesday, September 15, 2009 1:19 PM
> To: [hidden email]
> Subject: Re: [Pharo-project] Object serialization in Pharo
>
>> Stef,
>>
>> The packaging constraint had nothing to do with Pharo.  See the [ANN]
>> thread on #readStream replacing ReadStream on: (around 4 Sept).
>> SecureSqueak is the system with the apparent limitation; it really
>> does have that constraint, it will not be of much use to me.
>>
>> Unit tests for a serializer would be well-taken, but I will largely
>> have to start over.  Choosing a filer seems to be the first step.
>> Squeak's ReferenceStream did not strike me as a good option; if you
>> wish to argue in its favor, please do so.  Otherwise, I would like to
>> see SIXX get a good evaluation, as well as any other options that
>> should be considered.
>
> I'm sure that I want an XML parser in the way but why not.
> I started to write some tests just to learn and document. I would  
> even have no problem to throw away everything it there is a better  
> solution. Now what tests will let enable us to change and nothing  
> what we change.
>
> Stef
>>
>> Bill
>>
>>
>> -----Original Message-----
>> From: [hidden email]
>> [mailto:[hidden email]
>> ] On Behalf Of Stéphane Ducasse
>> Sent: Tuesday, September 15, 2009 10:06 AM
>> To: [hidden email]
>> Subject: Re: [Pharo-project] Object serialization in Pharo
>>
>>
>> On Sep 15, 2009, at 4:16 PM, Schwab,Wilhelm K wrote:
>>
>>> Simon,
>>>
>>> I wish you well and will be eager to hear progress reports.  Long
>>> ago, I looked at ReferenceStream (or maybe SmartReferenceStream),  
>>> and
>>> did not like what I found.  IIRC, the output for simple examples was
>>> much larger (of course, it might not continue to scale so
>>> unfavorably) compared that that produced by Dolphin's binary filer.
>>> Another concern was schema changes.
>>>
>>> Dolphin has an orderly conversion process that will raise an error  
>>> if
>>> it is not prepared for what it finds; Squeak seemed to want to fix  
>>> it
>>> itself with the aid of the user (not gonna fly in an end-user app or
>>> a server process, which is pretty much everything).
>>>
>>> Dolphin locates conversion methods in the class of the object being
>>> converted or in the class of a proxy for it (that sounded scary to  
>>> me
>>> when I read about it, but it is exactly what one would want in
>>> practice).  In contrast, Squeak puts the conversion methods in
>>> ReferenceStream.  Yuk!!!  Sorry, don't know what just came over  
>>> me :)
>>> Of course one can always modify ReferenceStream[*], but imagine the
>>> chaos if ReferenceStream were to become widely used.
>>>
>>> I have used serializers to good effect and would like to continue
>>> doing so.  At the moment, I would look again, but would lean away
>>> from ReferenceStream.
>>
>> I started to write some tests in the PharoTaskForces.
>> If people want to join effort because before changing I would like to
>> know what is there.
>>
>>> SIXX (an XML serializer) is a possibility.  As I recall, its output
>>> was not terse (but might compress nicely??).  There originally was
>>> minimal to no support for shape changes, but that has been  
>>> addressed;
>>> I cannot speak for how well it has been done because I have not had
>>> time to look.  It is on my todo list and might float to the surface
>>> soon, though initially I would not need to care about shape changes.
>>>
>>> HTH
>>>
>>> Bill
>>>
>>>
>>> [*] I think we recently discussed a Squeak fork that would not allow
>>> touching base classes due to packaging limitations.
>>
>> really?
>>
>>> Not good.  I am not defending the ReferenceStream example, but it
>>> would be a big problem on that system, if I understood that thread.
>>
>> I would like to fix that
>> We need a good object serializer.
>>
>>
>>>
>>>
>>> -----Original Message-----
>>> From: [hidden email]
>>> [mailto:[hidden email]
>>> ] On Behalf Of Simon Denier
>>> Sent: Tuesday, September 15, 2009 8:32 AM
>>> To: Pharo Development
>>> Subject: [Pharo-project] Object serialization in Pharo
>>>
>>> Hi folks
>>>
>>> I'm looking at the code and comments of ReferenceStream for object
>>> serialization and I wonder what is the best strategy to serialize a
>>> partial model. I mean, I want to store all elements found while
>>> traversing the graph of the model, but I dont want to store some
>>> attributes of those elements (i.e., transient attributes).
>>>
>>> Any sample?
>>>
>>> --
>>> Simon
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Pharo-project mailing list
>>> [hidden email]
>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>>
>>> _______________________________________________
>>> Pharo-project mailing list
>>> [hidden email]
>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>
>>
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Reply | Threaded
Open this post in threaded view
|

Re: Object serialization in Pharo

Schwab,Wilhelm K
Stef,

Sounds great.  ReferenceStream has some real problems, and it might be very difficult to change in ways that would not orphan existing data, if in fact there is any - projects perhaps??  It might be best to adopt an existing serializer or build a clean one of our own.  With some care and a few compatibility methods, we can probably write tests that can work with multiple serializers, and perhaps even profile them at the same time.

Bill
 

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Stéphane Ducasse
Sent: Tuesday, September 15, 2009 3:18 PM
To: [hidden email]
Subject: Re: [Pharo-project] Object serialization in Pharo

> Stef,
>
> Henry's warning tells me I'm not alone in being afraid of Squeak's
> serializer.  A while ago, I had settled on SIXX as the likely best
> option for me, but will certainly consider other options.  For the
> Dolphin filer, my tests mostly center on loading objects with old
> schemas.  It was fairly easy to do because I had old data in
> abundance, and simply loaded some of it into new images.  For Pharo,
> we might write tests that compile a class, serialize, make some
> changes to the class and reload the "old" data. The details will
> probably depend on the serializer(s) we select.
>
> My plan has been to start out using SIXX for network transport of data
> between like images (where schema changes are not a concern) and then
> begin to look at archival uses if that goes well.
>
> Does that answer your question?

ok we will evaluate SIXX.
Now my point is that we should have good solutions: so if referenceStream can be fix we fix it else we trash it.
Half-baked solutions just hinder progress.

My goal is this community is to set the cap (fun for a lighthouse) and the cap for 1.2 and other releases.
I want a neat nice robust working documented system and each step going in that direction is an important one.
Stef


>
> Bill
>
>
> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]
> ] On Behalf Of Stéphane Ducasse
> Sent: Tuesday, September 15, 2009 1:19 PM
> To: [hidden email]
> Subject: Re: [Pharo-project] Object serialization in Pharo
>
>> Stef,
>>
>> The packaging constraint had nothing to do with Pharo.  See the [ANN]
>> thread on #readStream replacing ReadStream on: (around 4 Sept).
>> SecureSqueak is the system with the apparent limitation; it really
>> does have that constraint, it will not be of much use to me.
>>
>> Unit tests for a serializer would be well-taken, but I will largely
>> have to start over.  Choosing a filer seems to be the first step.
>> Squeak's ReferenceStream did not strike me as a good option; if you
>> wish to argue in its favor, please do so.  Otherwise, I would like to
>> see SIXX get a good evaluation, as well as any other options that
>> should be considered.
>
> I'm sure that I want an XML parser in the way but why not.
> I started to write some tests just to learn and document. I would even
> have no problem to throw away everything it there is a better
> solution. Now what tests will let enable us to change and nothing what
> we change.
>
> Stef
>>
>> Bill
>>
>>
>> -----Original Message-----
>> From: [hidden email]
>> [mailto:[hidden email]
>> ] On Behalf Of Stéphane Ducasse
>> Sent: Tuesday, September 15, 2009 10:06 AM
>> To: [hidden email]
>> Subject: Re: [Pharo-project] Object serialization in Pharo
>>
>>
>> On Sep 15, 2009, at 4:16 PM, Schwab,Wilhelm K wrote:
>>
>>> Simon,
>>>
>>> I wish you well and will be eager to hear progress reports.  Long
>>> ago, I looked at ReferenceStream (or maybe SmartReferenceStream),
>>> and did not like what I found.  IIRC, the output for simple examples
>>> was much larger (of course, it might not continue to scale so
>>> unfavorably) compared that that produced by Dolphin's binary filer.
>>> Another concern was schema changes.
>>>
>>> Dolphin has an orderly conversion process that will raise an error
>>> if it is not prepared for what it finds; Squeak seemed to want to
>>> fix it itself with the aid of the user (not gonna fly in an end-user
>>> app or a server process, which is pretty much everything).
>>>
>>> Dolphin locates conversion methods in the class of the object being
>>> converted or in the class of a proxy for it (that sounded scary to
>>> me when I read about it, but it is exactly what one would want in
>>> practice).  In contrast, Squeak puts the conversion methods in
>>> ReferenceStream.  Yuk!!!  Sorry, don't know what just came over me
>>> :) Of course one can always modify ReferenceStream[*], but imagine
>>> the chaos if ReferenceStream were to become widely used.
>>>
>>> I have used serializers to good effect and would like to continue
>>> doing so.  At the moment, I would look again, but would lean away
>>> from ReferenceStream.
>>
>> I started to write some tests in the PharoTaskForces.
>> If people want to join effort because before changing I would like to
>> know what is there.
>>
>>> SIXX (an XML serializer) is a possibility.  As I recall, its output
>>> was not terse (but might compress nicely??).  There originally was
>>> minimal to no support for shape changes, but that has been
>>> addressed; I cannot speak for how well it has been done because I
>>> have not had time to look.  It is on my todo list and might float to
>>> the surface soon, though initially I would not need to care about
>>> shape changes.
>>>
>>> HTH
>>>
>>> Bill
>>>
>>>
>>> [*] I think we recently discussed a Squeak fork that would not allow
>>> touching base classes due to packaging limitations.
>>
>> really?
>>
>>> Not good.  I am not defending the ReferenceStream example, but it
>>> would be a big problem on that system, if I understood that thread.
>>
>> I would like to fix that
>> We need a good object serializer.
>>
>>
>>>
>>>
>>> -----Original Message-----
>>> From: [hidden email]
>>> [mailto:[hidden email]
>>> ] On Behalf Of Simon Denier
>>> Sent: Tuesday, September 15, 2009 8:32 AM
>>> To: Pharo Development
>>> Subject: [Pharo-project] Object serialization in Pharo
>>>
>>> Hi folks
>>>
>>> I'm looking at the code and comments of ReferenceStream for object
>>> serialization and I wonder what is the best strategy to serialize a
>>> partial model. I mean, I want to store all elements found while
>>> traversing the graph of the model, but I dont want to store some
>>> attributes of those elements (i.e., transient attributes).
>>>
>>> Any sample?
>>>
>>> --
>>> Simon
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Pharo-project mailing list
>>> [hidden email]
>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>>
>>> _______________________________________________
>>> Pharo-project mailing list
>>> [hidden email]
>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>
>>
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project