Smalltalk › Cincom › VisualWorks

[vwnc] How do you serialize your data?

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

16 messages Options

Stefan Schmiedl

[vwnc] How do you serialize your data?

If you do so at all. I'm currently working without database backend, so
I have to.

I want to access the application data of a remotely running (headless)
application for local testing and debugging. I tried it via BOSS, but
(just as is stated in the docs) it does not work with cyclic references.

For small data sets SIXX works just like I want it to, but the (XML)
file size and time for writing it soon gets quite out of hand.

Is there something out there "dumping anything to a file" like SIXX
does, but faster and more compact like BOSS?

Thanks,
s.
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Stew MacLean

Re: [vwnc] How do you serialize your data?

Hi Stefan,

I use BOSS with good results, an have built a simple persistence
framework around it.

You may want to experiment a bit further, as I believe it does handle
cyclic references.

You can run into problems if you don't break links to graphic objects
(eg Dependents). You either need to break the references, save then
restore them, or do a deep copy of your structure, that breaks the
references during the process of copying.

I have no experience of them, but you may want to check out VOSS and
Gemstone as well.

HTH,

Stewart

>-----Original Message-----
>From: [hidden email] [mailto:[hidden email]] On
Behalf

>Of Stefan Schmiedl
>Sent: 22 May 2008 5:02 a.m.
>To: [hidden email]
>Subject: [vwnc] How do you serialize your data?
>
>If you do so at all. I'm currently working without database backend, so
>I have to.
>
>I want to access the application data of a remotely running (headless)
>application for local testing and debugging. I tried it via BOSS, but
>(just as is stated in the docs) it does not work with cyclic

references.

>
>For small data sets SIXX works just like I want it to, but the (XML)
>file size and time for writing it soon gets quite out of hand.
>
>Is there something out there "dumping anything to a file" like SIXX
>does, but faster and more compact like BOSS?
>
>Thanks,
>s.
>_______________________________________________
>vwnc mailing list
>[hidden email]
>http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Rob Vens-2

Re: [vwnc] How do you serialize your data?

In reply to this post by Stefan Schmiedl

BOSS indeed should be able to handle cyclic references, so I would be curious to what problems you encountered. another strategy (in fact I offer my users both options) is to use parcels, which are able to store objects very well.

2008/5/21 Stefan Schmiedl <[hidden email]>:

If you do so at all. I'm currently working without database backend, so
I have to.

I want to access the application data of a remotely running (headless)
application for local testing and debugging. I tried it via BOSS, but
(just as is stated in the docs) it does not work with cyclic references.

For small data sets SIXX works just like I want it to, but the (XML)
file size and time for writing it soon gets quite out of hand.

Is there something out there "dumping anything to a file" like SIXX
does, but faster and more compact like BOSS?

Thanks,
s.
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Andre Schnoor

Re: [vwnc] How do you serialize your data?

Am 22.05.2008 um 09:31 schrieb Rob Vens:

> BOSS indeed should be able to handle cyclic references ...

Nope. Not in all cases. There is a bug I posted eight years ago. IIRC,
BOSS can't handle cyclic structures that contain SortedCollections. I
needed to change my fileIn/Out code such that cycles are temporarily
removed and restored later.

Andre

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Rob Vens-2

Re: [vwnc] How do you serialize your data?

I assume you are referring to representBinaryOn: to file out your objects?
Is this bug also present in parcelling out your objects?

2008/5/22 Andre Schnoor <[hidden email]>:

Am 22.05.2008 um 09:31 schrieb Rob Vens:

BOSS indeed should be able to handle cyclic references ...

Nope. Not in all cases. There is a bug I posted eight years ago. IIRC, BOSS can't handle cyclic structures that contain SortedCollections. I needed to change my fileIn/Out code such that cycles are temporarily removed and restored later.

Andre

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Andre Schnoor

Re: [vwnc] How do you serialize your data?

Am 22.05.2008 um 12:40 schrieb Rob Vens:

> I assume you are referring to representBinaryOn: to file out your
> objects?
> Is this bug also present in parcelling out your objects?

I never parcelled out objects, just using BOSS. Yes, I am using
representBinaryOn: and a more generic preFileOut/postFileOut protocol
on the document level (top-level object).

Andre
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Runar Jordahl

Re: [vwnc] How do you serialize your data?

In reply to this post by Stefan Schmiedl

I had a post about this a while ago:
http://www.cincomsmalltalk.com/userblogs/runarj/blogView?showComments=true&printTitle=Smalltalk_Serializers&entry=3355044917

I think State Replication Protocol (SRP) looks like a good alternative
to BOSS and SIXX. Sadly, no one is maintaining it now.

Runar
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Stefan Schmiedl

Re: [vwnc] How do you serialize your data?

In reply to this post by Rob Vens-2

On Thu, 22 May 2008 09:31:29 +0200
"Rob Vens" <[hidden email]> wrote:

> BOSS indeed should be able to handle cyclic references, so I would be
> curious to what problems you encountered. another strategy (in fact I offer
> my users both options) is to use parcels, which are able to store objects
> very well.

I actually managed to reproduce the problem, yay!

Take a virgin image, load the BOSS parcel, file in the attached
changeset, execute 'TestData new beforeAndAfter' and you should
see something similar to the attached screenshot.

There are three members of 2 groups 'before' and only 1 member of 2
groups 'after'.

The code defines two classes, 'Group' and 'Member', where aGroup has a
Set of its Members, and a Member has a Set of its Groups. I'm creating
two groups, add three members to both, six members to only one. Then
the data is BOSSed to a ByteArray, read back in and only one Member is
in two groups.

At that point I started looking for alternatives :-)

s.

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

bossproblem.st (7K) Download Attachment

Screenshot.png (25K) Download Attachment

Steven Kelly

Re: [vwnc] How do you serialize your data?

In reply to this post by Stefan Schmiedl

I haven't tested this, but I think you should try sending "bos
expectCycles: true".
Steve

> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On
> Behalf Of Stefan Schmiedl
> Sent: 22 May 2008 22:01
> To: Rob Vens
> Cc: [hidden email]
> Subject: Re: [vwnc] How do you serialize your data?
>
> On Thu, 22 May 2008 09:31:29 +0200
> "Rob Vens" <[hidden email]> wrote:
>
> > BOSS indeed should be able to handle cyclic references, so I would

be
> > curious to what problems you encountered. another strategy (in fact
I

> offer
> > my users both options) is to use parcels, which are able to store
> objects
> > very well.
>
> I actually managed to reproduce the problem, yay!
>
> Take a virgin image, load the BOSS parcel, file in the attached
> changeset, execute 'TestData new beforeAndAfter' and you should
> see something similar to the attached screenshot.
>
> There are three members of 2 groups 'before' and only 1 member of 2
> groups 'after'.
>
> The code defines two classes, 'Group' and 'Member', where aGroup has a
> Set of its Members, and a Member has a Set of its Groups. I'm creating
> two groups, add three members to both, six members to only one. Then
> the data is BOSSed to a ByteArray, read back in and only one Member is
> in two groups.
>
> At that point I started looking for alternatives :-)
>
> s.

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Stefan Schmiedl

Re: [vwnc] How do you serialize your data?

In reply to this post by Stew MacLean

Hi Stewart,

thanks for your reply.

On Thu, 22 May 2008 18:38:18 +1200
"Stewart MacLean" <[hidden email]> wrote:

> You may want to experiment a bit further, as I believe it does handle
> cyclic references.

Are you doing something different to what I posted in reply to Rob's
post above?

> You can run into problems if you don't break links to graphic objects
> (eg Dependents). You either need to break the references, save then
> restore them, or do a deep copy of your structure, that breaks the
> references during the process of copying.

No, nothing fancy, just "plain" objects with "primitive" associated
data. That's why I was quite astonished at the non-working
dump-and-restore, but shrugged it off as I found the remark "no cycles"
in the docs.

s.

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Stefan Schmiedl

Re: [vwnc] How do you serialize your data?

In reply to this post by Steven Kelly

Hi Steve,

your reply helps a lot here.

On Thu, 22 May 2008 22:39:20 +0300
"Steven Kelly" <[hidden email]> wrote:

> I haven't tested this, but I think you should try sending "bos
> expectCycles: true".

WOW! What a difference :-)

Now I only wonder, why the Application Developer's Guide
says on page 14-1

Also, avoid circular references, such as an application
model that holds onto a window that holds onto the application model,
and so on.

and does not mention this nice feature anywhere. The method comment
of expectCycles: shows what a lucky hand I have:

"Set whether the receiver should, on writing, expect to see cycles,
especially cycles where a Set or Dictionary refers back to itself..."

I mean, I *could* have gone with an OrderedCollection, but no, I
have to use a Set to avoid duplicate entries :-)

Thanks,
s.
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Stefan Schmiedl

[vwnc] State Replication Protocol, was Re: How do you serialize your data?

In reply to this post by Runar Jordahl

Hi Runar,

SRP looks very interesting. Thank you for pointing this out!

On Thu, 22 May 2008 16:17:45 +0200
"Runar Jordahl" <[hidden email]> wrote:

> I had a post about this a while ago:
> http://www.cincomsmalltalk.com/userblogs/runarj/blogView?showComments=true&printTitle=Smalltalk_Serializers&entry=3355044917
>
> I think State Replication Protocol (SRP) looks like a good alternative
> to BOSS and SIXX. Sadly, no one is maintaining it now.
>

So I've downloaded the zip and installed the parcels, switched the
tests to use SUnitToo and all 61 ran fine. Using SRP looks simple
enough for me to manage:

SrpConfiguration default saveObjectAsBinary: anObject.
SrpConfiguration default loadObjectFrom: aByteArray.

are what I find from a quick glance through the most easily accessible
documentation.

So I'll compare my BOSS code to my SRP code:

saveToBytesBos
| dataCollection bos stream |
stream := (ByteArray new: 1200) writeStream.
dataCollection := Array with: groups with: members.
bos := (BinaryObjectStorage onNew: stream)
expectCycles: true;
yourself.
[bos nextPutAll: dataCollection] ensure: [bos close].
^stream contents

loadFromBytesBos: aByteArray
| bos array |
bos := BinaryObjectStorage onOldNoScan: aByteArray readStream.
array := bos contents.
^(self class new)
groups: array first;
members: array last;
yourself

More or less a copy straight from the docs. Now SRP:

saveToBytesSrp
| dataCollection |
dataCollection := Array with: groups with: members.
^SrpConfiguration default saveObjectAsBinary: dataCollection

loadFromBytesSrp: aByteArray
^SRP.SrpConfiguration default loadObjectFrom: aByteArray

Looks nicer, doesn't it? Next I compared the representations of my
posted test data:

BOSS: 1040 bytes
SRP: 207 bytes

Another plus. What about runtime? I'm probably comparing apples to
oranges here, due to my lack of knowledge about how things ought to be
done, but, well, that's the way I currently would have done them. So
here's the copy of my Totally Unscientific Benchmark Workspace:

|td|
ObjectMemory garbageCollect.
td := TestData new.
Time millisecondsToRun: [10000 timesRepeat: [ td saveToBytesBos ]]
11806 11897 11634 11988

|td|
ObjectMemory garbageCollect.
td := TestData new.
Time millisecondsToRun: [10000 timesRepeat: [ td saveToBytesSrp ]]
1925 1928 1942 1945

|td dump|
ObjectMemory garbageCollect.
td := TestData new.
dump := TestData new saveToBytesBos.
Time millisecondsToRun: [10000 timesRepeat: [ td loadFromBytesBos: dump ]]
3727 3691 3625 3717

|td dump|
ObjectMemory garbageCollect.
td := TestData new.
dump := TestData new saveToBytesSrp.
Time millisecondsToRun: [10000 timesRepeat: [ td loadFromBytesSrp: dump ]]
2217 2244 2217 2312

The four numbers below each segment are the results of selecting and
printing the segment repeatedly. Totalling the dump/restore gives
BOSS : SRP :: 16 : 4

So either I'm doing something horribly wrong, or SRP is seriously cool.

Let's see how things work out with larger datasets next week.

s.
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Philipp Bunge

Re: [vwnc] How do you serialize your data?

In reply to this post by Stefan Schmiedl

Hi,

I am currently working on a flex-messaging server which mimics Adobe
BlazeDS' [1] API and uses AMF [2] to serialize data over the network.
AMF is a binary format, designed to be very compact, deals with
cyclic references and allows objects to define their own
serialization (see also [3]).

Is there an interest in serializing data for storage using AMF? Is
there an interest in creating services for Flex/AIR using VisualWorks
or another Smalltalk? My code is not yet ready for release, but I
might make an additional effort if people are interested.

Kind regards,
Philipp

[1] http://opensource.adobe.com/wiki/display/blazeds/
[2] http://en.wikipedia.org/wiki/Action_Message_Format
[3] http://livedocs.adobe.com/flex/201/langref/flash/utils/
IExternalizable.html

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Rob Vens-2

Re: [vwnc] State Replication Protocol, was Re: How do you serialize your data?

In reply to this post by Stefan Schmiedl

Sounds cool indeed. I will also run tests with my big datasets, which serialise fine with either BOSS or Parcels (btw they are complex object graphs with cycles).
Could be this is one of those Smalltalk gems gathering dust on obscure places...

2008/5/23 Stefan Schmiedl <[hidden email]>:

Hi Runar,

SRP looks very interesting. Thank you for pointing this out!

On Thu, 22 May 2008 16:17:45 +0200
"Runar Jordahl" <[hidden email]> wrote:

> I had a post about this a while ago:
> http://www.cincomsmalltalk.com/userblogs/runarj/blogView?showComments=true&printTitle=Smalltalk_Serializers&entry=3355044917
>
> I think State Replication Protocol (SRP) looks like a good alternative
> to BOSS and SIXX. Sadly, no one is maintaining it now.
>

So I've downloaded the zip and installed the parcels, switched the
tests to use SUnitToo and all 61 ran fine. Using SRP looks simple
enough for me to manage:

SrpConfiguration default saveObjectAsBinary: anObject.
SrpConfiguration default loadObjectFrom: aByteArray.

are what I find from a quick glance through the most easily accessible
documentation.

So I'll compare my BOSS code to my SRP code:

saveToBytesBos
| dataCollection bos stream |
stream := (ByteArray new: 1200) writeStream.
dataCollection := Array with: groups with: members.
bos := (BinaryObjectStorage onNew: stream)
expectCycles: true;
yourself.
[bos nextPutAll: dataCollection] ensure: [bos close].
^stream contents

loadFromBytesBos: aByteArray
| bos array |
bos := BinaryObjectStorage onOldNoScan: aByteArray readStream.
array := bos contents.
^(self class new)
groups: array first;
members: array last;
yourself

More or less a copy straight from the docs. Now SRP:

saveToBytesSrp
| dataCollection |
dataCollection := Array with: groups with: members.
^SrpConfiguration default saveObjectAsBinary: dataCollection

loadFromBytesSrp: aByteArray
^SRP.SrpConfiguration default loadObjectFrom: aByteArray

Looks nicer, doesn't it? Next I compared the representations of my
posted test data:

BOSS: 1040 bytes
SRP: 207 bytes

Another plus. What about runtime? I'm probably comparing apples to
oranges here, due to my lack of knowledge about how things ought to be
done, but, well, that's the way I currently would have done them. So
here's the copy of my Totally Unscientific Benchmark Workspace:

|td|
ObjectMemory garbageCollect.
td := TestData new.
Time millisecondsToRun: [10000 timesRepeat: [ td saveToBytesBos ]]
11806 11897 11634 11988

|td|
ObjectMemory garbageCollect.
td := TestData new.
Time millisecondsToRun: [10000 timesRepeat: [ td saveToBytesSrp ]]
1925 1928 1942 1945

|td dump|
ObjectMemory garbageCollect.
td := TestData new.
dump := TestData new saveToBytesBos.
Time millisecondsToRun: [10000 timesRepeat: [ td loadFromBytesBos: dump ]]
3727 3691 3625 3717

|td dump|
ObjectMemory garbageCollect.
td := TestData new.
dump := TestData new saveToBytesSrp.
Time millisecondsToRun: [10000 timesRepeat: [ td loadFromBytesSrp: dump ]]
2217 2244 2217 2312

The four numbers below each segment are the results of selecting and
printing the segment repeatedly. Totalling the dump/restore gives
BOSS : SRP :: 16 : 4

So either I'm doing something horribly wrong, or SRP is seriously cool.

Let's see how things work out with larger datasets next week.

s.
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Nowak, Helge

Re: [vwnc] How do you serialize your data?

In reply to this post by Philipp Bunge

it would be interested to see how VisualWorks could be serving a Flex/AIR frontend, same for MS Silverlight

Cheers
Helge

-----Ursprüngliche Nachricht-----
Von: [hidden email] [mailto:[hidden email]] Im Auftrag von Philipp Bunge
Gesendet: Freitag, 23. Mai 2008 02:53
An: [hidden email]
Betreff: Re: [vwnc] How do you serialize your data?

Hi,

I am currently working on a flex-messaging server which mimics Adobe BlazeDS' [1] API and uses AMF [2] to serialize data over the network.
AMF is a binary format, designed to be very compact, deals with cyclic references and allows objects to define their own serialization (see also [3]).

Is there an interest in serializing data for storage using AMF? Is there an interest in creating services for Flex/AIR using VisualWorks or another Smalltalk? My code is not yet ready for release, but I might make an additional effort if people are interested.

Kind regards,
Philipp

[1] http://opensource.adobe.com/wiki/display/blazeds/
[2] http://en.wikipedia.org/wiki/Action_Message_Format
[3] http://livedocs.adobe.com/flex/201/langref/flash/utils/
IExternalizable.html

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Maximiliano Taborda

Re: [vwnc] How do you serialize your data?

In reply to this post by Philipp Bunge

Hi.

I'm currently working on a project using Visual Works and Flex for the UI.
I would like to know about your code because I thinking to replace the
current webservices calls to AMF.
Best Regards,

Maxi

2008/5/22 Philipp Bunge <[hidden email]>:

> Hi,
>
> I am currently working on a flex-messaging server which mimics Adobe
> BlazeDS' [1] API and uses AMF [2] to serialize data over the network.
> AMF is a binary format, designed to be very compact, deals with
> cyclic references and allows objects to define their own
> serialization (see also [3]).
>
> Is there an interest in serializing data for storage using AMF? Is
> there an interest in creating services for Flex/AIR using VisualWorks
> or another Smalltalk? My code is not yet ready for release, but I
> might make an additional effort if people are interested.
>
> Kind regards,
> Philipp
>
> [1] http://opensource.adobe.com/wiki/display/blazeds/
> [2] http://en.wikipedia.org/wiki/Action_Message_Format
> [3] http://livedocs.adobe.com/flex/201/langref/flash/utils/
> IExternalizable.html
>
>
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc