Any serializer can customize serialization strategy?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Any serializer can customize serialization strategy?

hernanmd
Is there any serializer (Fuel/StOMP/Ma object
serialization/BOSS/SRP/SIXX??) which let me attach an external
compressor/decompressor for specific Strings to the
serialization/deserialization process? With external I mean it could
be in a separate binary (I could arrange the binding though).
Thanks

Hernán

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Reply | Threaded
Open this post in threaded view
|

Re: Any serializer can customize serialization strategy?

Paul Baumann
Hernán,

There are several ways to do it with SRP.

The encoding supports embedded data (variable and fixed width), so you could do inline serialization. It is clever enough that it is even possible to do this with positional access even to base64 encoded data. Tricks like that are used for example to update individual values in a persistent index. There are simpler techniques that should work for any marshaler...

All marshalers should have some form of pre-save and post-load exchange. SRP has many ways to pre-save and post-load while maintaining relative references. SRP can even do post-load exchanges after the rest of the object graph is loaded and still keep relative references (without using a #become:).

Save a proxy in place of the object. The proxy would be a class that you create to do what you want as a post-load actions. You can create the proxy as part of a pre-save action. The proxy can have the encoded binary ByteArray you want embedded in the data. The post-load of the proxy would typically unmarshal the embedded data. Here is an example of a class-based exchange:

SpecialSaveObject>>srpSavePreMap: aContext
        ^MyOtherMarshalProxy newFor: self

MyOtherMarshalerProxy.binData

MyOtherMarshalerProxy class>>newFor: anObject
        ^self new save: anObject

MyOtherMarshalProxy>>save: anObject
        binData := self marshaler saveAsBinary: anObject.

MyOtherMarshalerProxy>>srpLoadPostMap: aContext
        ^OtherMarshaler loadFromBinary: binData

Depending on your needs, you might want your object to remember the embedded data. In that case, you could store the encoded data in an inst var that is lazy-initialized as a pre-save action and save only that inst var. A post-load action could perhaps read the other attributes from the encoded data that you saved.

SRP also allows any object to answer back "extended attributes" that are saved with an object (#srpExtendedAttributes) and can be restored after loading (#srpExtendedAttributes:). Extended attributes are typically data you want associated with an object that is not directly referenced by the object. You could save/load encoded data along with the SRP saved object.

Paul Baumann



-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Hernán Morales Durand
Sent: Monday, October 10, 2011 15:36
To: Pharo Development; The general-purpose Squeak developers list; vwnc mailing list
Subject: [vwnc] Any serializer can customize serialization strategy?

Is there any serializer (Fuel/StOMP/Ma object
serialization/BOSS/SRP/SIXX??) which let me attach an external
compressor/decompressor for specific Strings to the
serialization/deserialization process? With external I mean it could
be in a separate binary (I could arrange the binding though).
Thanks

Hernán

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange, Inc. (ICE), its subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Reply | Threaded
Open this post in threaded view
|

Re: Any serializer can customize serialization strategy?

hernanmd
Hi Paul,

Thanks for your detailed reply. What you describe seems suitable for
my needs (I'm interested in simplest techniques, my problem is complex
enough to try premature improvements at the tool level :) however I
cannot comment more until I do some real experiments with SRP.
Thanks again

Hernán

2011/10/12 Paul Baumann <[hidden email]>:

> Hernán,
>
> There are several ways to do it with SRP.
>
> The encoding supports embedded data (variable and fixed width), so you could do inline serialization. It is clever enough that it is even possible to do this with positional access even to base64 encoded data. Tricks like that are used for example to update individual values in a persistent index. There are simpler techniques that should work for any marshaler...
>
> All marshalers should have some form of pre-save and post-load exchange. SRP has many ways to pre-save and post-load while maintaining relative references. SRP can even do post-load exchanges after the rest of the object graph is loaded and still keep relative references (without using a #become:).
>
> Save a proxy in place of the object. The proxy would be a class that you create to do what you want as a post-load actions. You can create the proxy as part of a pre-save action. The proxy can have the encoded binary ByteArray you want embedded in the data. The post-load of the proxy would typically unmarshal the embedded data. Here is an example of a class-based exchange:
>
> SpecialSaveObject>>srpSavePreMap: aContext
>        ^MyOtherMarshalProxy newFor: self
>
> MyOtherMarshalerProxy.binData
>
> MyOtherMarshalerProxy class>>newFor: anObject
>        ^self new save: anObject
>
> MyOtherMarshalProxy>>save: anObject
>        binData := self marshaler saveAsBinary: anObject.
>
> MyOtherMarshalerProxy>>srpLoadPostMap: aContext
>        ^OtherMarshaler loadFromBinary: binData
>
> Depending on your needs, you might want your object to remember the embedded data. In that case, you could store the encoded data in an inst var that is lazy-initialized as a pre-save action and save only that inst var. A post-load action could perhaps read the other attributes from the encoded data that you saved.
>
> SRP also allows any object to answer back "extended attributes" that are saved with an object (#srpExtendedAttributes) and can be restored after loading (#srpExtendedAttributes:). Extended attributes are typically data you want associated with an object that is not directly referenced by the object. You could save/load encoded data along with the SRP saved object.
>
> Paul Baumann
>
>
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Hernán Morales Durand
> Sent: Monday, October 10, 2011 15:36
> To: Pharo Development; The general-purpose Squeak developers list; vwnc mailing list
> Subject: [vwnc] Any serializer can customize serialization strategy?
>
> Is there any serializer (Fuel/StOMP/Ma object
> serialization/BOSS/SRP/SIXX??) which let me attach an external
> compressor/decompressor for specific Strings to the
> serialization/deserialization process? With external I mean it could
> be in a separate binary (I could arrange the binding though).
> Thanks
>
> Hernán
>
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>
> This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange, Inc. (ICE), its subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.
>

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Reply | Threaded
Open this post in threaded view
|

Re: Any serializer can customize serialization strategy?

Paul Baumann
Hernán,

Here are a few more SRP specific tips that may be helpful...

A relatively new addition to SRP is the ability to marshal "distinct" object graphs embedded within a larger marshaling context. Look at the #Distinct related code in #storeTypeArrays. Class and inst var store types can be declared through #srpDataTypeMap declarations. Your SrpConfiguration can declare the proxy to save and marshaler to save the distinct object with (SRP is the default). SRP sends #srpSavedAs: with the object (a proxy in this case) that SRP saved as. It is possible to note the stream position that your distinct object is saved at.

SRP also has a marshal directive for saving Proxy type objects. Proxies can be declared by class or inst var (by way of #srpDataTypeMap declarations) or by one of several other pre-save mapping options. You'd customize your SrpConfiguration to answer the proxy (#proxyFor:context:). You could for example save an object as a proxy that would do an SQL query to get the data from an RDB.

Either SrpContext.userData or SrpConfiguration.userData can be used to manage your proxies.

Either saving a #Proxy or #Distinct may serve your needs, or at least show a pattern that you can customize. The SRP encoding is designed to be infinitely extendable and portable. When using SRP, you should always start with a SrpConfiguration that you'd customize for your needs or for different marshaling purposes.

The default SRP configurations are intended for greatest portability. SRP encodes metastate and state in as many bytes as other typical marshaling frameworks encode just state. If you need even faster and tighter encoding then use SrpMetastateTables to externalize your most commonly used metastates.

Paul Baumann


-----Original Message-----
From: Hernán Morales Durand [mailto:[hidden email]]
Sent: Friday, October 14, 2011 01:26
To: Paul Baumann
Cc: Pharo Development; The general-purpose Squeak developers list; vwnc mailing list
Subject: Re: [vwnc] Any serializer can customize serialization strategy?

Hi Paul,

Thanks for your detailed reply. What you describe seems suitable for
my needs (I'm interested in simplest techniques, my problem is complex
enough to try premature improvements at the tool level :) however I
cannot comment more until I do some real experiments with SRP.
Thanks again

Hernán

2011/10/12 Paul Baumann <[hidden email]>:

> Hernán,
>
> There are several ways to do it with SRP.
>
> The encoding supports embedded data (variable and fixed width), so you could do inline serialization. It is clever enough that it is even possible to do this with positional access even to base64 encoded data. Tricks like that are used for example to update individual values in a persistent index. There are simpler techniques that should work for any marshaler...
>
> All marshalers should have some form of pre-save and post-load exchange. SRP has many ways to pre-save and post-load while maintaining relative references. SRP can even do post-load exchanges after the rest of the object graph is loaded and still keep relative references (without using a #become:).
>
> Save a proxy in place of the object. The proxy would be a class that you create to do what you want as a post-load actions. You can create the proxy as part of a pre-save action. The proxy can have the encoded binary ByteArray you want embedded in the data. The post-load of the proxy would typically unmarshal the embedded data. Here is an example of a class-based exchange:
>
> SpecialSaveObject>>srpSavePreMap: aContext
>        ^MyOtherMarshalProxy newFor: self
>
> MyOtherMarshalerProxy.binData
>
> MyOtherMarshalerProxy class>>newFor: anObject
>        ^self new save: anObject
>
> MyOtherMarshalProxy>>save: anObject
>        binData := self marshaler saveAsBinary: anObject.
>
> MyOtherMarshalerProxy>>srpLoadPostMap: aContext
>        ^OtherMarshaler loadFromBinary: binData
>
> Depending on your needs, you might want your object to remember the embedded data. In that case, you could store the encoded data in an inst var that is lazy-initialized as a pre-save action and save only that inst var. A post-load action could perhaps read the other attributes from the encoded data that you saved.
>
> SRP also allows any object to answer back "extended attributes" that are saved with an object (#srpExtendedAttributes) and can be restored after loading (#srpExtendedAttributes:). Extended attributes are typically data you want associated with an object that is not directly referenced by the object. You could save/load encoded data along with the SRP saved object.
>
> Paul Baumann
>
>
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Hernán Morales Durand
> Sent: Monday, October 10, 2011 15:36
> To: Pharo Development; The general-purpose Squeak developers list; vwnc mailing list
> Subject: [vwnc] Any serializer can customize serialization strategy?
>
> Is there any serializer (Fuel/StOMP/Ma object
> serialization/BOSS/SRP/SIXX??) which let me attach an external
> compressor/decompressor for specific Strings to the
> serialization/deserialization process? With external I mean it could
> be in a separate binary (I could arrange the binding though).
> Thanks
>
> Hernán
>
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>
> This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange, Inc. (ICE), its subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.
>


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange, Inc. (ICE), its subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Reply | Threaded
Open this post in threaded view
|

Re: [Pharo-project] [squeak-dev] Any serializer can customize serialization strategy?

Masashi UMEZAWA-2
In reply to this post by hernanmd
Hi,

Hernán Morales Durand <[hidden email]> wrote:
>Is there any serializer (Fuel/StOMP/Ma object
>serialization/BOSS/SRP/SIXX??) which let me attach an external
>compressor/decompressor for specific Strings to the
>serialization/deserialization process?

Mariano Martinez Peck <[hidden email]> wrote:
> I think StOMP should support this as well:
> http://stomp.smalltalk-users.jp/home/how-to-use-stomp/hook-methods

Yes. StOMP does support such kind of custom serialization.

Pattern 1: Using Memento.

StOMP has two basic hook methods (stompWriteValue/stompReadValue).
With a memento pattern, you can customize
serialization/deserialization behavior easily.

SpecificString>> stompWriteValue
 ^ StringCompressedMemento on: self

StringCompressedMemento>> stompReadValue
 ^ SpecificString on: self uncompressToString

SpecificString is a mere wrapper class (holding original string). And
StringCompressedMemento should implement compress/uncompress logic.

Pattern 2: Subclassing Writer/Reader

Pattern 1 is usable, but it is basically suitable for pointer objects.
For String-like objects, defining a new String wrapper and Memento is
overwhelming. You can also subclass StompWriter/Reader for adding
custom serialization/deserialization behaviors.

MyStompWriter>>writeString: aString
 (self isSpecificString: aString) ifTrue: [^super writeString: (self
compressString: aString)].
 ^ super writeString: string

MyStompReader>>readByteString
 | rawString |
 rawString := self basicReadObject asString.
 ^ (self isCompressedString: rawString)
        ifTrue: [self uncompressString: rawString]
        ifFalse: [rawString]

You have to add some marking headers to the original string for
determining it should be compressed/uncompressed. (Yes. this is a
little awkward, but works well).

I've attached working sample codes. So please see for details.

Best regards,
--
[:masashi | ^umezawa]

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Stomp-Exmaple1.st (4K) Download Attachment
Stomp-Example2.st (2K) Download Attachment