[ANN] Utility for Sixx handling
Locked
 
|
I created a little package for reading and writing object graphs with Sixx. The latest version of Sixx has some issues fixed regarding stream handling and encoding.
Based on that the utility tries to solve memory issues with bigger graphs. I incorporated a code snippet I got from Dale some time ago that solves read issues. It installs memory threshold handling so that if temporary object memory gets low a commit is done to subsequent get a file of any size into the image. The same happens for the conversion from string to object via Sixx. Writing of a graph is done using the streaming API of Sixx. The problem here is that nextPut: and nextPutAll: in gemstone are pretty slow operations. I added a buffering stream that lowers the amount of needed operations on the underlying stream. The speedup is immense. If you want to serialize a graph you can do it this way GUGemStoneSixxUtil new writeObject: Object new toFile: 'sixx.xml' inFolder: '/tmp'. Reading it back you do with GUGemStoneSixxUtil new readObjectFromFileNamed: '/tmp/sixx.xml' The buffer can be adjusted with GUGemStoneSixxUtil new streamBufferSize: aNumber ... The default is 100000 which seems to be ok. The package you can get from MCHttpRepository location: 'http://source.selfish.org/mc/misc' user: 'small' password: 'talk' If there are any issues just drop me a note. If you have the username and password set you can write to the repository. Hope you like it, Norbert |
Re: [ANN] Utility for Sixx handling
|
|
Hi Norbert,
Excellent. I had also grabbed Dale's script for importing our sixx file. Much cooler to have it in a utility for reuse. thanks! There's just a problem I am always having with reading sixx files that are exported from Pharo's sixx implementation. There seems to be a difference w.r.t. the root of the tree. When importing a file from Pharo, I have to change the following method to loop over the 'topElem nodes' of the xml tree instead of calling the 'readSixxFromSixxElement' method directly on the rootObj. (see my code of Behavior>>readSixxFrom:context:persistentRoot: below. readSixxFrom: xmlStringOrStream context: aSixxContext persistentRoot: persistentArrayOrNil | topElem rootObj | topElem := SixxXmlUtil parseXml: xmlStringOrStream persistentRoot: persistentArrayOrNil. (persistentArrayOrNil ~~ nil ) ifTrue: [ persistentArrayOrNil add: aSixxContext ]. rootObj := topElem nodes collect: [:n | self readSixxFromSixxElement: n context: aSixxContext]. persistentArrayOrNil ~~ nil ifTrue: [ persistentArrayOrNil size: 0 ]. ^rootObj However, after changing this method, a lot of unit tests are failing. So I am hesitating to submit that as a fix, although it's the only way I can read sixx files exported from Pharo. On top of that, the method is actually _only_ used in Dale's script and the unit tests... On 18 Feb 2011, at 20:21, Norbert Hartl wrote: > I created a little package for reading and writing object graphs with Sixx. The latest version of Sixx has some issues fixed regarding stream handling and encoding. > > Based on that the utility tries to solve memory issues with bigger graphs. I incorporated a code snippet I got from Dale some time ago that solves read issues. It installs memory threshold handling so that if temporary object memory gets low a commit is done to subsequent get a file of any size into the image. The same happens for the conversion from string to object via Sixx. > > Writing of a graph is done using the streaming API of Sixx. The problem here is that nextPut: and nextPutAll: in gemstone are pretty slow operations. I added a buffering stream that lowers the amount of needed operations on the underlying stream. The speedup is immense. > > If you want to serialize a graph you can do it this way > > GUGemStoneSixxUtil new > writeObject: Object new > toFile: 'sixx.xml' > inFolder: '/tmp'. > > Reading it back you do with > > GUGemStoneSixxUtil new > readObjectFromFileNamed: '/tmp/sixx.xml' > > The buffer can be adjusted with > > GUGemStoneSixxUtil new > streamBufferSize: aNumber > ... > > The default is 100000 which seems to be ok. The package you can get from > > MCHttpRepository > location: 'http://source.selfish.org/mc/misc' > user: 'small' > password: 'talk' > > If there are any issues just drop me a note. If you have the username and password set you can write to the repository. > > Hope you like it, > > Norbert > |
Re: [ANN] Utility for Sixx handling
|
|
Johan,
it is still on my list to synchronize the gemstone and the pharo port of Sixx but don't find time right now. It could be that there several differences. For gemstone I changed quite a bit of the implementation. I changed all uses of Stream>>cr, Stream>>tab to use a nextPut: variant in combination with grease to be cross-platform. Furthermore I changed the handling of multi-byte strings. Sixx used to just export the class directlywith the string. This leads to xml elements containing QuadByteString as class and platform specific byte layout as content. The gemstone port exports class "String" for all types of strings (and Symbol for all types of symbol) and the content utf-8 encoded. If the String is read from utf-8 then corresponding multi-byte string is created automatically. This way it is cross-platform. These are the changes I know but I'm pretty sure that those two ports are a bit more out of sync. So there might be subtle differences causing this problems. If I find some time today I can have a look. Now that the xml parser in gemstone caught up with the pharo one it is a good time to sync the sixx ports as well. Which version of the xml parser do you use? And from where did you take the Sixx for pharo? Norbert On 24.02.2011, at 08:28, Johan Brichau wrote: > Hi Norbert, > > Excellent. I had also grabbed Dale's script for importing our sixx file. > Much cooler to have it in a utility for reuse. thanks! > > There's just a problem I am always having with reading sixx files that are exported from Pharo's sixx implementation. There seems to be a difference w.r.t. the root of the tree. When importing a file from Pharo, I have to change the following method to loop over the 'topElem nodes' of the xml tree instead of calling the 'readSixxFromSixxElement' method directly on the rootObj. (see my code of Behavior>>readSixxFrom:context:persistentRoot: below. > > readSixxFrom: xmlStringOrStream context: aSixxContext persistentRoot: persistentArrayOrNil > | topElem rootObj | > topElem := SixxXmlUtil parseXml: xmlStringOrStream persistentRoot: persistentArrayOrNil. > (persistentArrayOrNil ~~ nil ) > ifTrue: [ persistentArrayOrNil add: aSixxContext ]. > rootObj := topElem nodes collect: [:n | self readSixxFromSixxElement: n context: aSixxContext]. > persistentArrayOrNil ~~ nil ifTrue: [ persistentArrayOrNil size: 0 ]. > ^rootObj > > However, after changing this method, a lot of unit tests are failing. So I am hesitating to submit that as a fix, although it's the only way I can read sixx files exported from Pharo. On top of that, the method is actually _only_ used in Dale's script and the unit tests... > > > On 18 Feb 2011, at 20:21, Norbert Hartl wrote: > >> I created a little package for reading and writing object graphs with Sixx. The latest version of Sixx has some issues fixed regarding stream handling and encoding. >> >> Based on that the utility tries to solve memory issues with bigger graphs. I incorporated a code snippet I got from Dale some time ago that solves read issues. It installs memory threshold handling so that if temporary object memory gets low a commit is done to subsequent get a file of any size into the image. The same happens for the conversion from string to object via Sixx. >> >> Writing of a graph is done using the streaming API of Sixx. The problem here is that nextPut: and nextPutAll: in gemstone are pretty slow operations. I added a buffering stream that lowers the amount of needed operations on the underlying stream. The speedup is immense. >> >> If you want to serialize a graph you can do it this way >> >> GUGemStoneSixxUtil new >> writeObject: Object new >> toFile: 'sixx.xml' >> inFolder: '/tmp'. >> >> Reading it back you do with >> >> GUGemStoneSixxUtil new >> readObjectFromFileNamed: '/tmp/sixx.xml' >> >> The buffer can be adjusted with >> >> GUGemStoneSixxUtil new >> streamBufferSize: aNumber >> ... >> >> The default is 100000 which seems to be ok. The package you can get from >> >> MCHttpRepository >> location: 'http://source.selfish.org/mc/misc' >> user: 'small' >> password: 'talk' >> >> If there are any issues just drop me a note. If you have the username and password set you can write to the repository. >> >> Hope you like it, >> >> Norbert >> > |
Re: [ANN] Utility for Sixx handling
|
|
On 24 Feb 2011, at 09:52, Norbert Hartl wrote: > Which version of the xml parser do you use? And from where did you take the Sixx for pharo? Sixx for Pharo is the one in squeaksource and I was using the XMLParser of GS (latest one until before this week) I'm not sure if the written-out xml tree is different, but at least the xml tree that is expected by the method invoked from Dale's script expects a different root node. Johan |
|
|
In reply to this post by NorbertHartl
Thanks. Grabbed this when you first posted it, and finally tried it the other day. Made life in a few places easier, so much appreciated.
|
|
|
Hi Norbert
Just wanted to say thank you too. Your extension has made my life so much easier! Cheers, Max On 25.02.2011, at 21:11, YossiDM wrote: > > Thanks. Grabbed this when you first posted it, and finally tried it the other > day. Made life in a few places easier, so much appreciated. > -- > View this message in context: http://forum.world.st/ANN-Utility-for-Sixx-handling-tp3313388p3325107.html > Sent from the GLASS mailing list archive at Nabble.com. |
|
|
In reply to this post by YossiDM
Norbert
I forgot to mention yesterday, that I had the same problem as Yossi. The sixx xml I was trying to import was generated in a Squeak 3.9 image with 0.3c loaded. Thanks Yossi for the code to iterate over the nodes. Max On 25.02.2011, at 21:11, YossiDM wrote: > > Thanks. Grabbed this when you first posted it, and finally tried it the other > day. Made life in a few places easier, so much appreciated. > -- > View this message in context: http://forum.world.st/ANN-Utility-for-Sixx-handling-tp3313388p3325107.html > Sent from the GLASS mailing list archive at Nabble.com. |
«
Return to GLASS
|
1 view|%1 views
| Free forum by Nabble | Edit this page |