ESUG SummerTalk - Fuel, binary object serializer

Interesting project. There are so many uses for object serializers.

Personally I am looking at storing objects in NoSQL databases, and
serializing them is an option that will work in a lot of cases.

Using
a serializer for copying objects is also useful.

Does Fuel support “cutting off” instance variables, like SIXX’s
Object>>sixxIgnorableInstVarNames ?

I always thought SIXX
(http://www.mars.dti.ne.jp/~umejava/smalltalk/sixx/index.html ) was
state-of-the-art in this field.

It seems to have many features, and it
is supported on many dialects.  Are your ideas so great that it really
makes sense to create yet another serializer?

Wouldn’t improving SIXX
be better?

I see three main uses for serializers:

-       Storing objects in databases and files.
-       Deep copy without manually having to implement copy methods.
-       Comparing objects.

The last use is complex: Given two (possibly complex) object graphs,
you could serialize both to a (separate) byte array, and then compare
the two byte arrays to implement #=. In theory this could be a lot
easier than manually implementing the comparison. However, depending
on the serializer and the contents being serialized, the byte arrays
could represent two “equal” objects, but still have different byte
contents:

-       “Equal” objects, represented by unequal objects: One graph has
Integer 0, the other one has Float 0.0
-       Objects stored in different order: A Smalltalk Set being equal in
the two graphs can have the element stored using different ordering.
-       Object IDs which should not be part of the comparison, would make
the two byte arrays different.

I am sure there are cases where a serializer could be used to
implement #=, but I have so far not seen it used.

I am looking at using Riak ( http://wiki.basho.com/ ) with Pharo. When
storing a “business object” you  have two choices:

-       Store a binary BLOB representing your object graph.
-       Store the object graph as JSON data.

The last option essentially means you must do something similar to OR
mapping, so I would prefer the first option. With Riak, you will soon
be able to store additional indexed properties, which you later will
be able to query. So you store your whole business object as a BLOB,
together with those properties you need to query.

I never tested SIXX with anything large. Saving space and time is of
course a valid reason to make a new serializer.

If you use SIXX you
might need to compress the XML before storing it, adding even more
overhead.

I really look forward to improvements in Fuel. Good luck!

Runar
_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Sounds awesome!

The streaming feature is appealing.

I might take a look at it. Won't hurt as a redundant repo backup tactic

sebastian 

o/ 

On May 24, 2011, at 5:39 PM, Martin Dias wrote:

Hi folks. I am really happy to announce that ESUG is sponsoring me for Fuel development through the ESUG SummerTalk. I am Martin Dias, a student at Buenos Aires, Argentina. The idea behind this SummerTalk is to implement Fuel, a binary, fast and general-purpose object graph serializer in Pharo. It is based on VisualWorks' Parcels ideas.

Actually, the project has already started since several months. Tristan Bourgois and I started with the project while doing an internship with RMoD, INRIA. Since a couple of months, Mariano Martinez Peck joined the team, and now he is the official mentor in the SummerTalk.

ESUG website for SummertTalk: http://www.esug.org/wiki/pier/Promotion/SummerTalk/SummerTalk2011

The website with all the necessary information is here: http://rmod.lille.inria.fr/web/pier/software/Fuel
It even includes slides explaining the algorithm. In addition, a paper is in progress.

For the moment, Fuel already provides the following features:

- Fast pickle format. It is much faster to materialize than to serialize.
- Correctly support class reshape (when the class of serialized objects has changed).
- Serialize ANY kind of object. For the moment there is no object to our knowledge that we cannot serialize and materialize.
- Be able to completely serialize classes and traits (not just a global name).
- Support cycles and avoid duplicates in the graph.
- Integration to Moose with an extension to export and import their models.
- Detection of globals: for example if you serialize Transcript, it is not duplicated and instead managed as a global reference.
- Solve common problems like Set rehash.
- Buffered writing: we use a buffered write stream for the serialization part (thanks Sven!).
- No need of special support from the VM.
- Try to have a good object oriented design.
- Well tested (about 120 tests, for the moment).
- Large set of benchmarks (even benchmarks for Moose extension).

And of course, there are a lot features for the future. You can see some of them in the website and some in the issue tracker: http://code.google.com/p/fuel/issues/list

We really appreciate all kind of feedback and comments. If you want to try it, check in the website how to do it. It is extremely easy.

Once again, I want to thank a lot to ESUG for sponsoring the project. I plan to create a "news" section in the website with some RSS. I will keep you informed.

Best regards,
Martin

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

On 24/05/11 4:39 PM, Martin Dias wrote:

We really appreciate all kind of feedback and comments. If you want to
try it, check in the website how to do it. It is extremely easy.

I had a brief look and will look some more. I may try to use it to serialize a Pier kernel.

In another use case, I'd like to serialize from one image, and deserialize in another image - under end user control. The issue here is that "nasty" code could be introduced: e.g. capture the Fuel output, deserialize, add nasty code, re-serialize, then send onward for import to image. Would it be possible to have some sort of "virus" filter? Maybe something like the Star Trek transporter that can filter out nasty stuff before re-materializing. :) For a start, maybe an inclusion list and/or an exclusion list of classes and globals would be useful.

On 25/05/11 11:22 AM, Mariano Martinez Peck wrote:

- globals behaviors (classes and traits) are managed (by default) in
kind of "light" serialization. Where we only serialize the global name
which means that the class has to be present in Smalltalk globals in the
image that you want to materialize.

You can change the default behavior and be able to completely serialize
a class/trait. But this is much more complicated and it is still work on
process (ClassBuilder is not your best friend).

Okay. I think all that is needed is a "safe mode" option when *de-serializing*, which would discard any non-light classes or global objects that appear in the stream.

On 25/05/11 11:53 AM, Mariano Martinez Peck wrote:

Sorry Yanni, I didn't follow. Could you please explain a bit more? what
do you want to serialize? do you want to be able to choose some classes
as light and some as non-light? where do you want to materialize ? in
the same image or in another one ?   When you said discard....what would
you do with the instances of those non-light classes for example? you
don't materialize them? and what happens to the objects that were
pointing to them ?  why would be the scenario useful for ? security ?

====
Yes, security. Here's my first post again, with different formatting:

In another use case, I'd like to serialize from one image, and deserialize in another image - *under end user control*. [e.g. web app]

The issue here is that "nasty" code could be introduced:

- capture the Fuel output
- deserialize, add nasty code, re-serialize
- then send onward for import to image.

Would it be possible to have some sort of "virus" filter?

====

So a simple "safe-mode" option on de-serialization would probably be sufficient.

_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

 

I am sure there are cases where a serializer could be used to
implement #=, but I have so far not seen it used.

I am looking at using Riak ( http://wiki.basho.com/ ) with Pharo. When
storing a “business object” you  have two choices:

-       Store a binary BLOB representing your object graph.
-       Store the object graph as JSON data.

The last option essentially means you must do something similar to OR
mapping, so I would prefer the first option. With Riak, you will soon
be able to store additional indexed properties, which you later will
be able to query. So you store your whole business object as a BLOB,
together with those properties you need to query.

Ok, I understand. So, we can provide something like this API:

byteArray := FLSerializer serializeInMemory: myObject

And

myRecoveredObject := FLMaterializer materializeFromByteArray: aByteArray

would something like that help ?
 

hehehehe I culdn't resist it. I was reading:
http://www.parashift.com/c++-faq-lite/serialization.html
and I read:*
"Like the Transporter on Star Trek, it's all about taking something
complicated and turning it into a flat sequence of 1s and 0s, then taking
that sequence of 1s and 0s (possibly at another place, possibly at another
time) and reconstructing the original complicated "something." "*




--
View this message in context: http://forum.world.st/ESUG-SummerTalk-Fuel-binary-object-serializer-tp3548159p4429702.html
Sent from the Seaside General mailing list archive at Nabble.com.
_______________________________________________
seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside