Any serializer can customize serialization strategy?
Locked
 
|
Is there any serializer (Fuel/StOMP/Ma object
serialization/BOSS/SRP/SIXX??) which let me attach an external compressor/decompressor for specific Strings to the serialization/deserialization process? With external I mean it could be in a separate binary (I could arrange the binding though). Thanks Hernán |
Re: [squeak-dev] Any serializer can customize serialization strategy?
|
|
On Mon, Oct 10, 2011 at 9:35 PM, Hernán Morales Durand <[hidden email]> wrote: Is there any serializer (Fuel/StOMP/Ma object In Fuel you can do: String >> fuelSubstitution ^ ExternalCompressor compress: self String >> fuelAccept: aVisitorBut the problem is that if #compress: answers an instance of String, you should be careful that if #shouldCompressString: asnwers true to the compressed string, then you will end up in an infinitive loop. See: http://rmod.lille.inria.fr/web/pier/software/Fuel/Hooks and FLHookedSubstitutionTest I think StOMP should support this as well: http://stomp.smalltalk-users.jp/home/how-to-use-stomp/hook-methods The previous hook you let you replace the original String with its compression during serialization. Which make sense because if you are going to compress it is better to do it during serialization. With external I mean it could I am not sure if I understood. Thanks -- Mariano http://marianopeck.wordpress.com |
Re: [squeak-dev] Any serializer can customize serialization strategy?
|
|
I forgot to say something..... notice that using this hook is slower than a plain serialization. So what I mean is that if the strings are not that big, then it may happens that the usage of the hook + the compression is slower than a plain serialization + the regular string. I let you know because you may not expect that. I should benchmark the overhead of the hook....
Cheers On Mon, Oct 10, 2011 at 9:54 PM, Mariano Martinez Peck <[hidden email]> wrote:
-- Mariano http://marianopeck.wordpress.com |
|
|
In reply to this post by Mariano Martinez Peck
Thanks Mariano,
That's pretty much what I want, but... why to expose the visitor interface? ;) Just to give a little feedback of what I have in my mind: mySerializer setEngine: ExternalCompressor for: ( MyClass -> #myStringIVar ) or mySerializer setEngine: ExternalCompressor for: incomingStream Now with machines generating daily 6 TBytes of raw data, one of the keys will be data compression. 2011/10/10 Mariano Martinez Peck <[hidden email]>: > > > On Mon, Oct 10, 2011 at 9:35 PM, Hernán Morales Durand > <[hidden email]> wrote: >> >> Is there any serializer (Fuel/StOMP/Ma object >> serialization/BOSS/SRP/SIXX??) which let me attach an external >> compressor/decompressor for specific Strings to the >> serialization/deserialization process? > > In Fuel you can do: > > String >> fuelSubstitution > ^ ExternalCompressor compress: self > > String >> fuelAccept: aVisitor > ^ (ExternalCompressor shouldCompressString: self) > ifTrue: [aVisitor visitSubstitution: self] > ifFalse: [super fuelAccept: aVisitor] > > But the problem is that if #compress: answers an instance of String, you > should be careful that if #shouldCompressString: asnwers true to the > compressed string, then you will end up in an infinitive loop. > > See: http://rmod.lille.inria.fr/web/pier/software/Fuel/Hooks and > FLHookedSubstitutionTest > > I think StOMP should support this as well: > http://stomp.smalltalk-users.jp/home/how-to-use-stomp/hook-methods > > The previous hook you let you replace the original String with its > compression during serialization. Which make sense because if you are going > to compress it is better to do it during serialization. > >> With external I mean it could >> be in a separate binary (I could arrange the binding though). > > I am not sure if I understood. > > > >> >> Thanks >> >> Hernán >> > > > > -- > Mariano > http://marianopeck.wordpress.com > > |
Re: [squeak-dev] Any serializer can customize serialization strategy?
|
|
On Tue, Oct 11, 2011 at 12:56 AM, Hernán Morales Durand <[hidden email]> wrote: Thanks Mariano, Good question. Honesty, it was the only way we have found to do it with Fuel. I think (but I am not sure) that one of the drawbacks of the pickle format is that it analysis phase (as we call it in Fuel) make some hooks more complicated to implement than a classic serializer. Just to give a little feedback of what I have in my mind: Well, you can also create an specific Fuel cluster for MyClass that does exactly that. Basically you have to create FLMyClassCluster, implement methods like #serializeWith: and #materializeFrom: and there you serialize/materialize the specific #myStringIVar with a special way and the rest as normal. Then you subclass the FLDefaultMapper, say YourMapper and you implement #visitXXX Then you have to implement MyClass >> visitXXX. Of course, you should be able to generalize such cluster and be able to reuse it for different classes or instVars. I am not saying this is the best way. I am just telling you one possible solution you can do by extending Fuel for your needs. Of course, you may need to understand how Fuel works first. But of course, we can help if you want. or it looks like ;) 2011/10/10 Mariano Martinez Peck <[hidden email]>: -- Mariano http://marianopeck.wordpress.com |
Re: [squeak-dev] Any serializer can customize serialization strategy?
|
|
Hi Hernan, Could your "specific Strings" be instances of a variant of RemoteString that compresses and decompresses? Then they will occupy less memory in your image and it would be easier to serializer and deserialize the proxy object.
Thanks for the feedback, I also don't like the visitor interface we have for customizations. In fact, I think it is not a visitor but a double-dispatch. And I would prefer to merge the two hooks Mariano proposed to something like:
String >> fuelMapOn: aDefaultMapper
(ExternalCompressor shouldCompressString: self)
ifTrue: [aDefaultMapper mapSubstitution: self by: (ExternalCompressor compress: self)]
ifFalse: [super fuelMapOn: aDefaultMapper]
Do you like more this way? On Tue, Oct 11, 2011 at 4:10 PM, Mariano Martinez Peck <[hidden email]> wrote:
|
|
|
In reply to this post by hernanmd
Hi Paul,
Thanks for your detailed reply. What you describe seems suitable for my needs (I'm interested in simplest techniques, my problem is complex enough to try premature improvements at the tool level :) however I cannot comment more until I do some real experiments with SRP. Thanks again Hernán 2011/10/12 Paul Baumann <[hidden email]>: > Hernán, > > There are several ways to do it with SRP. > > The encoding supports embedded data (variable and fixed width), so you could do inline serialization. It is clever enough that it is even possible to do this with positional access even to base64 encoded data. Tricks like that are used for example to update individual values in a persistent index. There are simpler techniques that should work for any marshaler... > > All marshalers should have some form of pre-save and post-load exchange. SRP has many ways to pre-save and post-load while maintaining relative references. SRP can even do post-load exchanges after the rest of the object graph is loaded and still keep relative references (without using a #become:). > > Save a proxy in place of the object. The proxy would be a class that you create to do what you want as a post-load actions. You can create the proxy as part of a pre-save action. The proxy can have the encoded binary ByteArray you want embedded in the data. The post-load of the proxy would typically unmarshal the embedded data. Here is an example of a class-based exchange: > > SpecialSaveObject>>srpSavePreMap: aContext > ^MyOtherMarshalProxy newFor: self > > MyOtherMarshalerProxy.binData > > MyOtherMarshalerProxy class>>newFor: anObject > ^self new save: anObject > > MyOtherMarshalProxy>>save: anObject > binData := self marshaler saveAsBinary: anObject. > > MyOtherMarshalerProxy>>srpLoadPostMap: aContext > ^OtherMarshaler loadFromBinary: binData > > Depending on your needs, you might want your object to remember the embedded data. In that case, you could store the encoded data in an inst var that is lazy-initialized as a pre-save action and save only that inst var. A post-load action could perhaps read the other attributes from the encoded data that you saved. > > SRP also allows any object to answer back "extended attributes" that are saved with an object (#srpExtendedAttributes) and can be restored after loading (#srpExtendedAttributes:). Extended attributes are typically data you want associated with an object that is not directly referenced by the object. You could save/load encoded data along with the SRP saved object. > > Paul Baumann > > > > -----Original Message----- > From: [hidden email] [mailto:[hidden email]] On Behalf Of Hernán Morales Durand > Sent: Monday, October 10, 2011 15:36 > To: Pharo Development; The general-purpose Squeak developers list; vwnc mailing list > Subject: [vwnc] Any serializer can customize serialization strategy? > > Is there any serializer (Fuel/StOMP/Ma object > serialization/BOSS/SRP/SIXX??) which let me attach an external > compressor/decompressor for specific Strings to the > serialization/deserialization process? With external I mean it could > be in a separate binary (I could arrange the binding though). > Thanks > > Hernán > > _______________________________________________ > vwnc mailing list > [hidden email] > http://lists.cs.uiuc.edu/mailman/listinfo/vwnc > > This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange, Inc. (ICE), its subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired. > |
|
|
In reply to this post by tinchodias
Hi Martin
2011/10/13 Martin Dias <[hidden email]>: > Hi Hernan, > Could your "specific Strings" be instances of a variant of RemoteString that > compresses and decompresses? Then they will occupy less memory in your image > and it would be easier to serializer and deserialize the proxy object. I'd like to keep my objects cross-dialect, so if that's not an issue it would be ok to use a RemoteString. I don't see why it would be easier to serialize/deserialize? > Thanks for the feedback, I also don't like the visitor interface we have for > customizations. In fact, I think it is not a visitor but a double-dispatch. > And I would prefer to merge the two hooks Mariano proposed to something > like: > String >> > fuelMapOn: aDefaultMapper (ExternalCompressor shouldCompressString: self) > ifTrue: [aDefaultMapper mapSubstitution: self by: (ExternalCompressor > compress: self)] ifFalse: [super fuelMapOn: aDefaultMapper] > > Do you like more this way? Absolutely :) > Martin > > On Tue, Oct 11, 2011 at 4:10 PM, Mariano Martinez Peck > <[hidden email]> wrote: >> >> >> On Tue, Oct 11, 2011 at 12:56 AM, Hernán Morales Durand >> <[hidden email]> wrote: >>> >>> Thanks Mariano, >>> >>> That's pretty much what I want, but... why to expose the visitor >>> interface? ;) >> >> Good question. Honesty, it was the only way we have found to do it with >> Fuel. I think (but I am not sure) that one of the drawbacks of the pickle >> format is that it analysis phase (as we call it in Fuel) make some hooks >> more complicated to implement than a classic serializer. >> >>> >>> Just to give a little feedback of what I have in my mind: >>> >>> mySerializer setEngine: ExternalCompressor for: ( MyClass -> >>> #myStringIVar ) >>> >> >> Well, you can also create an specific Fuel cluster for MyClass that does >> exactly that. Basically you have to create FLMyClassCluster, implement >> methods like #serializeWith: and #materializeFrom: and there you >> serialize/materialize the specific #myStringIVar with a special way and the >> rest as normal. Then you subclass the FLDefaultMapper, say YourMapper and >> you implement #visitXXX >> Then you have to implement MyClass >> visitXXX. >> Of course, you should be able to generalize such cluster and be able to >> reuse it for different classes or instVars. >> >> I am not saying this is the best way. I am just telling you one possible >> solution you can do by extending Fuel for your needs. Of course, you may >> need to understand how Fuel works first. But of course, we can help if you >> want. >> >>> >>> or >>> >>> mySerializer setEngine: ExternalCompressor for: incomingStream >>> >>> Now with machines generating daily 6 TBytes of raw data, one of the >>> keys will be data compression. >>> >> >> it looks like ;) >> >> >>> >>> 2011/10/10 Mariano Martinez Peck <[hidden email]>: >>> > >>> > >>> > On Mon, Oct 10, 2011 at 9:35 PM, Hernán Morales Durand >>> > <[hidden email]> wrote: >>> >> >>> >> Is there any serializer (Fuel/StOMP/Ma object >>> >> serialization/BOSS/SRP/SIXX??) which let me attach an external >>> >> compressor/decompressor for specific Strings to the >>> >> serialization/deserialization process? >>> > >>> > In Fuel you can do: >>> > >>> > String >> fuelSubstitution >>> > ^ ExternalCompressor compress: self >>> > >>> > String >> fuelAccept: aVisitor >>> > ^ (ExternalCompressor shouldCompressString: self) >>> > ifTrue: [aVisitor visitSubstitution: self] >>> > ifFalse: [super fuelAccept: aVisitor] >>> > >>> > But the problem is that if #compress: answers an instance of String, >>> > you >>> > should be careful that if #shouldCompressString: asnwers true to the >>> > compressed string, then you will end up in an infinitive loop. >>> > >>> > See: http://rmod.lille.inria.fr/web/pier/software/Fuel/Hooks and >>> > FLHookedSubstitutionTest >>> > >>> > I think StOMP should support this as well: >>> > http://stomp.smalltalk-users.jp/home/how-to-use-stomp/hook-methods >>> > >>> > The previous hook you let you replace the original String with its >>> > compression during serialization. Which make sense because if you are >>> > going >>> > to compress it is better to do it during serialization. >>> > >>> >> With external I mean it could >>> >> be in a separate binary (I could arrange the binding though). >>> > >>> > I am not sure if I understood. >>> > >>> > >>> > >>> >> >>> >> Thanks >>> >> >>> >> Hernán >>> >> >>> > >>> > >>> > >>> > -- >>> > Mariano >>> > http://marianopeck.wordpress.com >>> > >>> > >>> >> >> >> >> -- >> Mariano >> http://marianopeck.wordpress.com >> > > |
Re: [squeak-dev] Any serializer can customize serialization strategy?
|
|
On Fri, Oct 14, 2011 at 2:40 AM, Hernán Morales Durand <[hidden email]> wrote: Hi Martin I don't know it has sense, also it depends on your case... but if MyRemoteString just has a instVar 'filename' or 'fileIndex' to fetch the real string from disk when necessary... it should be easy to serialize instances of such MyRemoteString.
thanks... probably for Fuel 1.8!
|
Re: [squeak-dev] Any serializer can customize serialization strategy?
|
|
In reply to this post by Mariano Martinez Peck
Hi,
Hernán Morales Durand <[hidden email]> wrote: >Is there any serializer (Fuel/StOMP/Ma object >serialization/BOSS/SRP/SIXX??) which let me attach an external >compressor/decompressor for specific Strings to the >serialization/deserialization process? Mariano Martinez Peck <[hidden email]> wrote: > I think StOMP should support this as well: > http://stomp.smalltalk-users.jp/home/how-to-use-stomp/hook-methods Yes. StOMP does support such kind of custom serialization. Pattern 1: Using Memento. StOMP has two basic hook methods (stompWriteValue/stompReadValue). With a memento pattern, you can customize serialization/deserialization behavior easily. SpecificString>> stompWriteValue ^ StringCompressedMemento on: self StringCompressedMemento>> stompReadValue ^ SpecificString on: self uncompressToString SpecificString is a mere wrapper class (holding original string). And StringCompressedMemento should implement compress/uncompress logic. Pattern 2: Subclassing Writer/Reader Pattern 1 is usable, but it is basically suitable for pointer objects. For String-like objects, defining a new String wrapper and Memento is overwhelming. You can also subclass StompWriter/Reader for adding custom serialization/deserialization behaviors. MyStompWriter>>writeString: aString (self isSpecificString: aString) ifTrue: [^super writeString: (self compressString: aString)]. ^ super writeString: string MyStompReader>>readByteString | rawString | rawString := self basicReadObject asString. ^ (self isCompressedString: rawString) ifTrue: [self uncompressString: rawString] ifFalse: [rawString] You have to add some marking headers to the original string for determining it should be compressed/uncompressed. (Yes. this is a little awkward, but works well). I've attached working sample codes. So please see for details. Best regards, -- [:masashi | ^umezawa] |
Stomp-Exmaple1.st (4K)
«
Return to Pharo Smalltalk Developers
|
1 view|%1 views
| Free forum by Nabble | Edit this page |