Norbert,
That's what I was guessing:) .... Well, my current project is directed at improving the GemTools experience .... basically it's tODE using Pharo as the client rather than a web-browser ... frankly I am quite excited by the system that is emerging... So in the end, you might not need 64 bit pharo:) Dale ----- Original Message ----- | From: "Norbert Hartl" <[hidden email]> | To: "GemStone Seaside beta discussion" <[hidden email]> | Sent: Monday, October 29, 2012 10:15:31 AM | Subject: Re: [GS/SS Beta] STON and transactions | | | Am 29.10.2012 um 17:46 schrieb Dale Henrichs <[hidden email]>: | | > Norbert ... | > | > | > Hmmm, you are exporting data to Pharo to do analysis ... so why | > don't you use GemStone for your analysis? | > | The term "analyzing" might be misleading. I have a lot of data in | gemstone I like to play with and to develop. My primary development | is in pharo because it is more comfortable than e.g. gemtools. My | workflow is that I develop in pharo and then the code flows towards | to live system in gemstone. While code flows from my pharo image to | the development server and finally to the live system for data it | should be the reverse. Bringing the data from the live to the test | system is easy. I do a gemstone full backup every night. I just | replay the backup on the test system so I have all of the newest | data there. I like to close the gap with bringing the data from the | test system to my development image. That's my motivation. | Especially with my current duty to analyze/play with the data to | model proper code around it. | | > For the topez project I am exporting limited (by depth) object | > graphs from GemStone to Pharo using STON ... you could do | > something similar with this particular project ... if you need to | > run the analysis in Pharo ... | > | > If extracting data from the system that you are analyzing is | > important, then you could extract the data using STON (for now) | > ... import into a separate GemStone db, then analyze on Pharo... | > | > Just thinking out loud:) | > | Agreed. Today I managed to import data into pharo. I wrote a utility | for my domain model that produces a shrinked copy of the data I can | work with. I can export the data via STON. And since today I can | import data into pharo. The STONReader needed to be changed. I was | able to import over a million objects in pharo using approx. 200MB | heap while exporting and now I have a image with 160MB after | startup. Quite ok for me at the moment. The new approach will last | a little while. And then I'm not sure which is needed more urgent: | fuel in gemstone or 64 bit pharo. I think it is the latter. | | thanks, | | Norbert | | > ----- Original Message ----- | > | From: "Norbert Hartl" <[hidden email]> | > | To: "GemStone Seaside beta discussion" | > | <[hidden email]> | > | Sent: Sunday, October 28, 2012 3:55:32 AM | > | Subject: Re: [GS/SS Beta] STON and transactions | > | | > | | > | Am 26.10.2012 um 19:31 schrieb Dale Henrichs | > | <[hidden email]>: | > | | > | > Mariano, | > | > | > | > Agree with everything you say ... my ignorance of Fuel 'fueled' | > | > my | > | > wacky idea that if you could handle class shape changes you | > | > could | > | > handle dialect to dialect changes too:) | > | > | > | That is what I was thinking, too. | > | | > | > As for the port, it' son my list of things to do, but if | > | > Norbert or | > | > anyone finds that they would use Fuel if it were ported, then | > | > it's | > | > worth trying to make time to get the port done … | > | > | > | I think it is a good idea to have fuel ported to gemstone. I | > | don't | > | know if it is urgent enough to distract you from your current | > | work. | > | I'm still trying to make the STON part work. For gemstone I just | > | raised the SPC size and the gem temp object memory and I can save | > | it. Now I'm trying to import it again in pharo. That might work | > | but | > | a new problem is close. The data I'm speaking of is getting | > | bigger | > | and bigger and so it will blow up a pharo image anyway. | > | So I have two choice: | > | | > | - not exporting everything. If I can put a scope on the graph for | > | exportation that could be a way to avoid exporting all of it. If | > | this would work neither the gemstone export nor the pharo import | > | will be a big problem. | > | - store data external. I could store files or something similar. | > | This | > | way it will be slow but I could have at least access to the data | > | from pharo in order to do analyzes. | > | | > | So you see my actual problem shifted slightly away from the | > | serialization/materialization. | > | | > | Norbert | > | | > | > Dale | > | > | > | > ----- Original Message ----- | > | > | From: "Mariano Martinez Peck" <[hidden email]> | > | > | To: "GemStone Seaside beta discussion" | > | > | <[hidden email]> | > | > | Sent: Friday, October 26, 2012 1:18:40 AM | > | > | Subject: Re: [GS/SS Beta] STON and transactions | > | > | | > | > | | > | > | | > | > | | > | > | On Fri, Oct 26, 2012 at 12:52 AM, Dale Henrichs < | > | > | [hidden email] | > | > | > wrote: | > | > | | > | > | | > | > | Norbert, | > | > | | > | > | The fact that Fuel is not recursive makes it an attractive | > | > | option | > | > | for | > | > | GemStone to GemStone transfer if passivate/activate (which is | > | > | recursive) is not viable ... I'd be willing to chip in on a | > | > | port | > | > | of | > | > | Fuel if you think you would use it... | > | > | | > | > | | > | > | | > | > | I think that porting Fuel to GemStone and do GemStone to | > | > | GemStone | > | > | transfer, should work without much effort. I cannot do it | > | > | myself | > | > | right now, but I am willing to help. | > | > | | > | > | | > | > | | > | > | Given the ability to map classes of different shapes, | > | > | | > | > | | > | > | Not sure what you mean. Is what I said about class reshape? | > | > | if | > | > | true, | > | > | I think there was a misunderstood. What I mean is that Fuel | > | > | supports | > | > | the case where when you materialize something, the classes in | > | > | the | > | > | system have changed since the time you serialized. Typical | > | > | cases | > | > | are: add/remove/rename/reorder instance variables, rename | > | > | classes, | > | > | etc. | > | > | | > | > | | > | > | it sounds like there might be a chance of using Fuel for | > | > | Gemstone | > | > | to | > | > | Pharo transfers. The OrderedCollection implementations are | > | > | very | > | > | different on the two platforms and would make a good first | > | > | test | > | > | case. | > | > | | > | > | | > | > | | > | > | Supporting GemStone-Pharo transfer looks more complex. At | > | > | least, | > | > | Fuel | > | > | was never built with this in mind, I mean, we never tried to | > | > | find | > | > | portable representation of the objects. Say Float, we just | > | > | encode | > | > | Floats as they are in Pharo. We do not delegate to a | > | > | compatibility | > | > | layer (as other serializers like SRP do) nor we do any type | > | > | of | > | > | conversion. I don't know the concrete differences with the | > | > | "primitive classes" (Floats, Integers, ByteArray, | > | > | ByteStrings, | > | > | etc) | > | > | of GemStone and Pharo. Maybe there are not many and maybe a | > | > | little | > | > | layer would help. | > | > | Anyway, what I want to say is that GemStone-Pharo transfer | > | > | will | > | > | be | > | > | more work than GemStone to GemStone. And of course, the | > | > | latter is | > | > | needed for the first :) | > | > | | > | > | | > | > | | > | > | | > | > | | > | > | | > | > | Dale | > | > | | > | > | ----- Original Message ----- | > | > | | From: "Norbert Hartl" < [hidden email] > | > | > | | To: "GemStone Seaside beta discussion" < | > | > | | [hidden email] | > | > | | > | > | > | | > | > | | > | > | | Sent: Thursday, October 25, 2012 10:40:32 AM | > | > | | Subject: Re: [GS/SS Beta] STON and transactions | > | > | | | > | > | | | > | > | | Am 25.10.2012 um 17:46 schrieb Dale Henrichs < | > | > | | [hidden email] | > | > | | >: | > | > | | | > | > | | > I haven't even looked at Fuel, so it will be a wild guess | > | > | | > indeed. | > | > | | > | > | > | | > Are you planning on using Fuel to do GemStone to GemStone | > | > | | > transfers | > | > | | > or GemStone to Pharo transfers? GemStone 2.x to GemStone | > | > | | > 3.x | > | > | | > transfers? or GemStone to GemStone of the same major | > | > | | > version? | > | > | | > | > | > | | I was just thinking about transfer from gemstone to pharo. | > | > | | But | > | > | | gemstone to gemstone on same version is also interesting. | > | > | | | > | > | | > If you are talking GemStone to GemStone of same major | > | > | | > version. | > | > | | > You | > | > | | > might want to look at Object>>passivate (and look at | > | > | | > manual). | > | > | | > Object passivation suffers from the same problem (running | > | > | | > out | > | > | | > of | > | > | | > stack space) that SIXX had (before I changed some of the | > | > | | > algorithms to not use recursion) and come to think of it | > | > | | > ... | > | > | | > If | > | > | | > Fuel uses recursion while walking the object graph, it | > | > | | > will | > | > | | > have | > | > | | > stack problems as well … | > | > | | > | > | > | | agreed | > | > | | | > | > | | > Cross dialect or even cross major version transfers | > | > | | > probably | > | > | | > can't | > | > | | > be done well with Fuel (unless it is implemented | > | > | | > differently | > | > | | > than | > | > | | > I imagine) so STON or SIXX are your best bets. BTW, | > | > | | > what's | > | > | | > wrong | > | > | | > with SIXX? | > | > | | > | > | > | | The biggest problem is that there is no pull parser | > | > | | available | > | > | | (I | > | > | | know | > | > | | Ken Treis did one but the xml parser changed quite a lot | > | > | | since | > | > | | then). Writing out a graph depends mainly on the depth | > | > | | traversal | > | > | | management and that there is enough space for a object to | > | > | | reference | > | > | | table. Reading in could be done similar if a pure streaming | > | > | | solution | > | > | | would be available. In Sixx the XML is very verbose | > | > | | producing | > | > | | huge | > | > | | files. Without a pull parser you read in the whole file as | > | > | | string, | > | > | | then convert it into a DOM which is even more bloated. Just | > | > | | to | > | > | | finally walk the DOM and create objects. That makes to | > | > | | problem | > | > | | worse | > | > | | at some magnitude. So STON is much more slim in string size | > | > | | and | > | > | | it | > | > | | is also smaller in code size. Making it more easy to tweak | > | > | | it | > | > | | where | > | > | | needed. | > | > | | | > | > | | Or to say it in numbers. Taking a fresh extent and reading | > | > | | in a | > | > | | couple of 10000s of objects with SIXX took me the last time | > | > | | 40 | > | > | | minutes and left back an extent with a size of 25GB. Now I | > | > | | have | > | > | | something between 2 and 5 million objects. I didn't even | > | > | | think | > | > | | about | > | > | | to try. | > | > | | | > | > | | > With all of that said I can't see it taking more than a | > | > | | > week | > | > | | > or | > | > | | > two | > | > | | > (63.458 hours). Classes are shaped differently in | > | > | | > GemStone, | > | > | | > but | > | > | | > you can look at the passivate/active code for the special | > | > | | > cases | > | > | | > ... it takes a bit of thought but the recursive | > | > | | > algorithms | > | > | | > can be | > | > | | > unwound into loops (makes for horrid debugging) without | > | > | | > too | > | > | | > much | > | > | | > trouble. | > | > | | > | > | > | | That is a good tip. I'll look into the passivate stuff. You | > | > | | are | > | > | | right | > | > | | indeed. The different shapes of objects might be a big | > | > | | problem. | > | > | | I | > | > | | think Fuel has some shape changing support (as SIXX has). | > | > | | But | > | > | | if it | > | > | | comes to the internals of objects that might be a really | > | > | | annoying | > | > | | task to do. | > | > | | | > | > | | thanks for your detailed estimation, | > | > | | | > | > | | Norbert | > | > | | | > | > | | > Dale | > | > | | > | > | > | | > ----- Original Message ----- | > | > | | > | From: "Norbert Hartl" < [hidden email] > | > | > | | > | To: "GemStone Seaside beta discussion" | > | > | | > | < [hidden email] > | > | > | | > | Sent: Thursday, October 25, 2012 8:21:24 AM | > | > | | > | Subject: Re: [GS/SS Beta] STON and transactions | > | > | | > | | > | > | | > | Dale, | > | > | | > | | > | > | | > | I don't know how good you are in "wild guessing" but I | > | > | | > | think | > | > | | > | you | > | > | | > | are | > | > | | > | an expert :). To place an estimation from the gut what | > | > | | > | would | > | > | | > | you | > | > | | > | think how much work it will be to port Fuel to | > | > | | > | GemStone. | > | > | | > | You | > | > | | > | can | > | > | | > | leave Blocks aside. | > | > | | > | | > | > | | > | Norbert | > | > | | > | | > | > | | > | Am 25.10.2012 um 16:52 schrieb Dale Henrichs | > | > | | > | < [hidden email] >: | > | > | | > | | > | > | | > | > Regarding the memory issues ... I assume that the | > | > | | > | > issue | > | > | | > | > is | > | > | | > | > similar | > | > | | > | > to the problem with SIXX in that you're building some | > | > | | > | > very | > | > | | > | > large | > | > | | > | > objects (Strings in Streams?) in temporary memory. | > | > | | > | > For | > | > | | > | > SIXX | > | > | | > | > the | > | > | | > | > trick was to link the root of the large temp object | > | > | | > | > graph | > | > | | > | > in | > | > | | > | > UserGlobals so that the temp object could be | > | > | | > | > committed, | > | > | | > | > thus | > | > | | > | > reducing the pressure on temp obj space ... at the | > | > | | > | > end, | > | > | | > | > the | > | > | | > | > temp | > | > | | > | > object was removed from UserGlobals so that it could | > | > | | > | > be | > | > | | > | > garbage | > | > | | > | > collected ... | > | > | | > | > | > | > | | > | > Regarding the test failures in GemStone ... I will | > | > | | > | > have | > | > | | > | > to | > | > | | > | > take | > | > | | > | > a | > | > | | > | > look. At the time of the port, the tests were | > | > | | > | > passing, | > | > | | > | > but at | > | > | | > | > the | > | > | | > | > moment I don't recall which versions of GemStone I | > | > | | > | > tested | > | > | | > | > STON | > | > | | > | > against ... | > | > | | > | > | > | > | | > | > I have been using STON in 3.1.0.1 , but I have been | > | > | | > | > using | > | > | | > | > it | > | > | | > | > to | > | > | | > | > serialize limited object graphs (I am stopping after | > | > | | > | > n-levels), | > | > | | > | > so | > | > | | > | > I haven't tried to serialize a big object graph from | > | > | | > | > GemStone | > | > | | > | > and | > | > | | > | > haven't run into any errors using STON. | > | > | | > | > | > | > | | > | > Regarding the error related to ScaledDecimal ... I'd | > | > | | > | > like | > | > | | > | > to | > | > | | > | > know | > | > | | > | > what the error is. I can't imagine what kind of error | > | > | | > | > that | > | > | | > | > ScaledDecimal would be throwing from the following | > | > | | > | > code: | > | > | | > | > | > | > | | > | > STON fromStream: (String streamContents: [:stream| | > | > | | > | > (Dictionary new | > | > | | > | > at: Object new put: Object new; | > | > | | > | > yourself) stonOn: (STONWriter on: stream)]) | > | > | | > | > readStream | > | > | | > | > | > | > | | > | > In a somewhat related are, I have found that there | > | > | | > | > are | > | > | | > | > bugs | > | > | | > | > related | > | > | | > | > to using #basicSize, #basicAt: and #basicAtPut: for a | > | > | | > | > handful | > | > | | > | > of | > | > | | > | > classes. I've identified a problems in GsNMethod and | > | > | | > | > ClassSet | > | > | | > | > and | > | > | | > | > this particular problem could affect the operation of | > | > | | > | > STON. | > | > | | > | > | > | > | | > | > Still not sure about the ScaledDecimal problem... | > | > | | > | > | > | > | | > | > Dale | > | > | | > | > ----- Original Message ----- | > | > | | > | > | From: "Norbert Hartl" < [hidden email] > | > | > | | > | > | To: "GemStone Seaside beta discussion" | > | > | | > | > | < [hidden email] > | > | > | | > | > | Sent: Thursday, October 25, 2012 4:53:45 AM | > | > | | > | > | Subject: Re: [GS/SS Beta] STON and transactions | > | > | | > | > | | > | > | | > | > | And finally the problem I've found seems to be a | > | > | | > | > | problem in | > | > | | > | > | the | > | > | | > | > | design of STON (or just me misunderstanding its | > | > | | > | > | purpose). | > | > | | > | > | Anyway | > | > | | > | > | the | > | > | | > | > | following code snippet raises the error | > | > | | > | > | | > | > | | > | > | STON fromStream: (String streamContents: [:stream| | > | > | | > | > | (Dictionary new | > | > | | > | > | at: Object new put: Object new; | > | > | | > | > | yourself) stonOn: (STONWriter on: stream)]) | > | > | | > | > | readStream | > | > | | > | > | | > | > | | > | > | Norbert | > | > | | > | > | | > | > | | > | > | Am 25.10.2012 um 13:37 schrieb Norbert Hartl | > | > | | > | > | < [hidden email] >: | > | > | | > | > | | > | > | | > | > | > This is some sort of monolog :) I just forgot to | > | > | | > | > | > add | > | > | | > | > | > to | > | > | | > | > | > the | > | > | | > | > | > last | > | > | | > | > | > mail that reading back the file in GemStone fails | > | > | | > | > | > with | > | > | | > | > | > the | > | > | | > | > | > same | > | > | | > | > | > error as in pharo. | > | > | | > | > | > | > | > | | > | > | > Norbert | > | > | | > | > | > | > | > | | > | > | > Am 25.10.2012 um 13:35 schrieb Norbert Hartl | > | > | | > | > | > < [hidden email] >: | > | > | | > | > | > | > | > | | > | > | >> Unfortunately only the export worked "somehow". | > | > | | > | > | >> In | > | > | | > | > | >> pharo | > | > | | > | > | >> I | > | > | | > | > | >> cannot | > | > | | > | > | >> read in the STON file produced by gemstone. | > | > | | > | > | >> Tests | > | > | | > | > | >> are | > | > | | > | > | >> green on | > | > | | > | > | >> pharo. On gemstone there are two failing tests | > | > | | > | > | >> that | > | > | | > | > | >> have | > | > | | > | > | >> to do | > | > | | > | > | >> with ScaledDecimal. You said you are working | > | > | | > | > | >> with it | > | > | | > | > | >> and | > | > | | > | > | >> you | > | > | | > | > | >> did | > | > | | > | > | >> not encounter problems? | > | > | | > | > | >> | > | > | | > | > | >> Norbert | > | > | | > | > | >> | > | > | | > | > | >> Am 25.10.2012 um 11:11 schrieb Norbert Hartl | > | > | | > | > | >> < [hidden email] >: | > | > | | > | > | >> | > | > | | > | > | >>> Haha, | > | > | | > | > | >>> | > | > | | > | > | >>> often after I write a mail the situation | > | > | | > | > | >>> changes | > | > | | > | > | >>> somehow | > | > | | > | > | >>> :) | > | > | | > | > | >>> While | > | > | | > | > | >>> fighting in the mist I tried doing a | > | > | | > | > | >>> commitTransaction | > | > | | > | > | >>> instead | > | > | | > | > | >>> of abortTransaction and this works. At least I | > | > | | > | > | >>> can | > | > | | > | > | >>> export | > | > | | > | > | >>> my | > | > | | > | > | >>> object graph now. If there is a good | > | > | | > | > | >>> explanation | > | > | | > | > | >>> I'm | > | > | | > | > | >>> glad | > | > | | > | > | >>> to | > | > | | > | > | >>> hear it :) | > | > | | > | > | >>> | > | > | | > | > | >>> Norbert | > | > | | > | > | >>> | > | > | | > | > | >>> Am 25.10.2012 um 11:03 schrieb Norbert Hartl | > | > | | > | > | >>> < [hidden email] >: | > | > | | > | > | >>> | > | > | | > | > | >>>> Dale, | > | > | | > | > | >>>> | > | > | | > | > | >>>> I loaded the STON code with your snippet and | > | > | | > | > | >>>> this | > | > | | > | > | >>>> worked. | > | > | | > | > | >>>> But it | > | > | | > | > | >>>> didn't solve my problem. I cannot export my | > | > | | > | > | >>>> object | > | > | | > | > | >>>> graph | > | > | | > | > | >>>> with | > | > | | > | > | >>>> STON. I always get temporary object memory | > | > | | > | > | >>>> full. I | > | > | | > | > | >>>> gave | > | > | | > | > | >>>> the | > | > | | > | > | >>>> gem | > | > | | > | > | >>>> 200MB already and still no success. I can give | > | > | | > | > | >>>> it | > | > | | > | > | >>>> even | > | > | | > | > | >>>> more | > | > | | > | > | >>>> but | > | > | | > | > | >>>> that would postpone my problem only a couple | > | > | | > | > | >>>> of | > | > | | > | > | >>>> weeks. | > | > | | > | > | >>>> I | > | > | | > | > | >>>> like | > | > | | > | > | >>>> to find the reason why it doesn't work. | > | > | | > | > | >>>> | > | > | | > | > | >>>> Reading the code I only suspected the | > | > | | > | > | >>>> STONWriterXXX | > | > | | > | > | >>>> classes | > | > | | > | > | >>>> to | > | > | | > | > | >>>> be the culprit because they are created all | > | > | | > | > | >>>> the | > | > | | > | > | >>>> time. | > | > | | > | > | >>>> So | > | > | | > | > | >>>> I | > | > | | > | > | >>>> rewrote the STON code to use an object pool | > | > | | > | > | >>>> for | > | > | | > | > | >>>> the | > | > | | > | > | >>>> writer | > | > | | > | > | >>>> classes but still no success. Having the pool | > | > | | > | > | >>>> I | > | > | | > | > | >>>> had | > | > | | > | > | >>>> one | > | > | | > | > | >>>> central | > | > | | > | > | >>>> instance that gets accessed all the time. So I | > | > | | > | > | >>>> tried | > | > | | > | > | >>>> to | > | > | | > | > | >>>> do | > | > | | > | > | >>>> on | > | > | | > | > | >>>> every 100000 instance request a 'System | > | > | | > | > | >>>> abortTransaction' | > | > | | > | > | >>>> hoping it would solve something. | > | > | | > | > | >>>> That falsified my theory disconnected objects | > | > | | > | > | >>>> can | > | > | | > | > | >>>> fill | > | > | | > | > | >>>> up | > | > | | > | > | >>>> the | > | > | | > | > | >>>> space. For the gem it is like a normal vm, | > | > | | > | > | >>>> right? | > | > | | > | > | >>>> When | > | > | | > | > | >>>> space | > | > | | > | > | >>>> gets low gc chimes in. So my abortTransaction | > | > | | > | > | >>>> doesn't | > | > | | > | > | >>>> do | > | > | | > | > | >>>> anything the gc wouldn't do, right? | > | > | | > | > | >>>> That means the objects that fill up the space | > | > | | > | > | >>>> are | > | > | | > | > | >>>> connected | > | > | | > | > | >>>> somehow. Now I'm asking myself what is the | > | > | | > | > | >>>> best | > | > | | > | > | >>>> way to | > | > | | > | > | >>>> get | > | > | | > | > | >>>> an | > | > | | > | > | >>>> overview over the instances that are in my | > | > | | > | > | >>>> temporary | > | > | | > | > | >>>> object | > | > | | > | > | >>>> memory. Would that be a duty for statmon or is | > | > | | > | > | >>>> there | > | > | | > | > | >>>> another | > | > | | > | > | >>>> way? | > | > | | > | > | >>>> | > | > | | > | > | >>>> thanks, | > | > | | > | > | >>>> | > | > | | > | > | >>>> Norbert | > | > | | > | > | >>>> | > | > | | > | > | >>> | > | > | | > | > | >> | > | > | | > | > | > | > | > | | > | > | | > | > | | > | > | | > | > | | > | | > | > | | > | | > | > | | | > | > | | | > | > | | > | > | | > | > | | > | > | | > | > | -- | > | > | Mariano | > | > | http://marianopeck.wordpress.com | > | > | | > | > | | > | | > | | | |
Free forum by Nabble | Edit this page |