serializing object graphs / ordering graph anchors

Before I start a utility myself I want to ask if there is such a tool already.

I'm using sixx for object graph serialization. It works quite good but it has some severe limits. Well, the limits are not really sixx dependent but a problem of usual serializers. My problem is the size of the stack such a tool generates. Especially in gemstone this appears to be a problem because the stack is limited per gem.

If I look at the serializers I know the mechanics are simple:

1. take an object and look it up in a table.
2. if the object is in the table it has already been serialized. We take a reference from the table and write it down
3. if it isn't in the table it means the object has not been serialized and the contents need to be written. After that the object with a reference representation is written to the table
4. while writing the content of an object (inst vars) I encounter simple objects (direct string representation) or complex objects (composite). In the latter case we start over from 1.

If the object graph is well interconnected than the odds are high that a stack will grow high until the first reference can be written. With a small pier kernel I already matched a stack size of 1000.

One solution I can think of is to collect and order objects in the graph that cover smaller segments from the overall graph. This way I would construct a sorted collection that is sorted by stack depth (lowest first). The collection will then be serialized leading to a much smaller possible stack sizes. It might be there are cases it can't work but I think for most cases it will do its job. And it will be slow of course.

Does anybody know if such a tool exists?

Norbert

Am 02.06.2011 um 20:10 schrieb Jon Paynter:

Can you raise the stack limit on your gem processes?
here we have a "standard" smalltalk stack depth of 10k, and it works well for deeply recursive processes.

It works with 10k as stack size. I just don't know for how long it will be ok. In the worst case in a highly interconnected graph the stack size can be as deep as the number of objects you have. And then 10k objects is not that much. 

Do you have the setting of 10k stack size all the time or just for doing stuff that needs it (it is a setting of the stone not the client gem). I mean the stack size can be cumbersome to adjust but on the other hand it can save you at the same time. For normal operation someone can assume something has gone wrong if the stack size is exceeded. 

So I'm trying to get a gut feeling how to tackle this sorts of problems.

thank you,

Norbert

On Thu, Jun 2, 2011 at 3:38 AM, Norbert Hartl <[hidden email]> wrote:

Before I start a utility myself I want to ask if there is such a tool already.

I'm using sixx for object graph serialization. It works quite good but it has some severe limits. Well, the limits are not really sixx dependent but a problem of usual serializers. My problem is the size of the stack such a tool generates. Especially in gemstone this appears to be a problem because the stack is limited per gem.

If I look at the serializers I know the mechanics are simple:

1. take an object and look it up in a table.
2. if the object is in the table it has already been serialized. We take a reference from the table and write it down
3. if it isn't in the table it means the object has not been serialized and the contents need to be written. After that the object with a reference representation is written to the table
4. while writing the content of an object (inst vars) I encounter simple objects (direct string representation) or complex objects (composite). In the latter case we start over from 1.

If the object graph is well interconnected than the odds are high that a stack will grow high until the first reference can be written. With a small pier kernel I already matched a stack size of 1000.

One solution I can think of is to collect and order objects in the graph that cover smaller segments from the overall graph. This way I would construct a sorted collection that is sorted by stack depth (lowest first). The collection will then be serialized leading to a much smaller possible stack sizes. It might be there are cases it can't work but I think for most cases it will do its job. And it will be slow of course.

Does anybody know if such a tool exists?

Norbert

Am 03.06.2011 um 19:06 schrieb Jon Paynter:

> Lastly, since 10k is the max allowed there isnt any room to increase it further.

Looking at the 2.4 SysAdminGuide on p. 382 I read a maximum of 1000000.