STON and transactions

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

STON and transactions

NorbertHartl
Dale,

I loaded the STON code with your snippet and this worked. But it didn't solve my problem. I cannot export my object graph with STON. I always get temporary object memory full. I gave the gem 200MB already and still no success. I can give it even more but that would postpone my problem only a couple of weeks. I like to find the reason why it doesn't work.

 Reading the code I only suspected the STONWriterXXX classes to be the culprit because they are created all the time. So I rewrote the STON code to use an object pool for the writer classes but still no success. Having the pool I had one central instance that gets accessed all the time. So I tried to do on every 100000 instance request a 'System abortTransaction' hoping it would solve something.
That falsified my theory disconnected objects can fill up the space. For the gem it is like a normal vm, right? When space gets low gc chimes in. So my abortTransaction doesn't do anything the gc wouldn't do, right?
That means the objects that fill up the space are connected somehow. Now I'm asking myself what is the best way to get an overview over the instances that are in my temporary object memory. Would that be a duty for statmon or is there another way?

thanks,

Norbert

Reply | Threaded
Open this post in threaded view
|

Re: STON and transactions

NorbertHartl
Haha,

often after I write a mail the situation changes somehow :) While fighting in the mist I tried doing a commitTransaction instead of abortTransaction and this works. At least I can export my object graph now. If there is a good explanation I'm glad to hear it :)

Norbert

Am 25.10.2012 um 11:03 schrieb Norbert Hartl <[hidden email]>:

> Dale,
>
> I loaded the STON code with your snippet and this worked. But it didn't solve my problem. I cannot export my object graph with STON. I always get temporary object memory full. I gave the gem 200MB already and still no success. I can give it even more but that would postpone my problem only a couple of weeks. I like to find the reason why it doesn't work.
>
> Reading the code I only suspected the STONWriterXXX classes to be the culprit because they are created all the time. So I rewrote the STON code to use an object pool for the writer classes but still no success. Having the pool I had one central instance that gets accessed all the time. So I tried to do on every 100000 instance request a 'System abortTransaction' hoping it would solve something.
> That falsified my theory disconnected objects can fill up the space. For the gem it is like a normal vm, right? When space gets low gc chimes in. So my abortTransaction doesn't do anything the gc wouldn't do, right?
> That means the objects that fill up the space are connected somehow. Now I'm asking myself what is the best way to get an overview over the instances that are in my temporary object memory. Would that be a duty for statmon or is there another way?
>
> thanks,
>
> Norbert
>

Reply | Threaded
Open this post in threaded view
|

Re: STON and transactions

NorbertHartl
Unfortunately only the export worked "somehow". In pharo I cannot read in the STON file produced by gemstone. Tests are green on pharo. On gemstone there are two failing tests that have to do with ScaledDecimal.  You said you are working with it and you did not encounter problems?

Norbert

Am 25.10.2012 um 11:11 schrieb Norbert Hartl <[hidden email]>:

> Haha,
>
> often after I write a mail the situation changes somehow :) While fighting in the mist I tried doing a commitTransaction instead of abortTransaction and this works. At least I can export my object graph now. If there is a good explanation I'm glad to hear it :)
>
> Norbert
>
> Am 25.10.2012 um 11:03 schrieb Norbert Hartl <[hidden email]>:
>
>> Dale,
>>
>> I loaded the STON code with your snippet and this worked. But it didn't solve my problem. I cannot export my object graph with STON. I always get temporary object memory full. I gave the gem 200MB already and still no success. I can give it even more but that would postpone my problem only a couple of weeks. I like to find the reason why it doesn't work.
>>
>> Reading the code I only suspected the STONWriterXXX classes to be the culprit because they are created all the time. So I rewrote the STON code to use an object pool for the writer classes but still no success. Having the pool I had one central instance that gets accessed all the time. So I tried to do on every 100000 instance request a 'System abortTransaction' hoping it would solve something.
>> That falsified my theory disconnected objects can fill up the space. For the gem it is like a normal vm, right? When space gets low gc chimes in. So my abortTransaction doesn't do anything the gc wouldn't do, right?
>> That means the objects that fill up the space are connected somehow. Now I'm asking myself what is the best way to get an overview over the instances that are in my temporary object memory. Would that be a duty for statmon or is there another way?
>>
>> thanks,
>>
>> Norbert
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: STON and transactions

NorbertHartl
This is some sort of monolog :) I just forgot to add to the last mail that reading back the file in GemStone fails with the same error as in pharo.

Norbert

Am 25.10.2012 um 13:35 schrieb Norbert Hartl <[hidden email]>:

> Unfortunately only the export worked "somehow". In pharo I cannot read in the STON file produced by gemstone. Tests are green on pharo. On gemstone there are two failing tests that have to do with ScaledDecimal.  You said you are working with it and you did not encounter problems?
>
> Norbert
>
> Am 25.10.2012 um 11:11 schrieb Norbert Hartl <[hidden email]>:
>
>> Haha,
>>
>> often after I write a mail the situation changes somehow :) While fighting in the mist I tried doing a commitTransaction instead of abortTransaction and this works. At least I can export my object graph now. If there is a good explanation I'm glad to hear it :)
>>
>> Norbert
>>
>> Am 25.10.2012 um 11:03 schrieb Norbert Hartl <[hidden email]>:
>>
>>> Dale,
>>>
>>> I loaded the STON code with your snippet and this worked. But it didn't solve my problem. I cannot export my object graph with STON. I always get temporary object memory full. I gave the gem 200MB already and still no success. I can give it even more but that would postpone my problem only a couple of weeks. I like to find the reason why it doesn't work.
>>>
>>> Reading the code I only suspected the STONWriterXXX classes to be the culprit because they are created all the time. So I rewrote the STON code to use an object pool for the writer classes but still no success. Having the pool I had one central instance that gets accessed all the time. So I tried to do on every 100000 instance request a 'System abortTransaction' hoping it would solve something.
>>> That falsified my theory disconnected objects can fill up the space. For the gem it is like a normal vm, right? When space gets low gc chimes in. So my abortTransaction doesn't do anything the gc wouldn't do, right?
>>> That means the objects that fill up the space are connected somehow. Now I'm asking myself what is the best way to get an overview over the instances that are in my temporary object memory. Would that be a duty for statmon or is there another way?
>>>
>>> thanks,
>>>
>>> Norbert
>>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: STON and transactions

NorbertHartl
And finally the problem I've found seems to be a problem in the design of STON (or just me misunderstanding its purpose). Anyway the following code snippet raises the error

STON fromStream: (String streamContents: [:stream|
        (Dictionary new
                at: Object new put: Object new;
                yourself) stonOn: (STONWriter on: stream)]) readStream

Norbert

Am 25.10.2012 um 13:37 schrieb Norbert Hartl <[hidden email]>:

> This is some sort of monolog :) I just forgot to add to the last mail that reading back the file in GemStone fails with the same error as in pharo.
>
> Norbert
>
> Am 25.10.2012 um 13:35 schrieb Norbert Hartl <[hidden email]>:
>
>> Unfortunately only the export worked "somehow". In pharo I cannot read in the STON file produced by gemstone. Tests are green on pharo. On gemstone there are two failing tests that have to do with ScaledDecimal.  You said you are working with it and you did not encounter problems?
>>
>> Norbert
>>
>> Am 25.10.2012 um 11:11 schrieb Norbert Hartl <[hidden email]>:
>>
>>> Haha,
>>>
>>> often after I write a mail the situation changes somehow :) While fighting in the mist I tried doing a commitTransaction instead of abortTransaction and this works. At least I can export my object graph now. If there is a good explanation I'm glad to hear it :)
>>>
>>> Norbert
>>>
>>> Am 25.10.2012 um 11:03 schrieb Norbert Hartl <[hidden email]>:
>>>
>>>> Dale,
>>>>
>>>> I loaded the STON code with your snippet and this worked. But it didn't solve my problem. I cannot export my object graph with STON. I always get temporary object memory full. I gave the gem 200MB already and still no success. I can give it even more but that would postpone my problem only a couple of weeks. I like to find the reason why it doesn't work.
>>>>
>>>> Reading the code I only suspected the STONWriterXXX classes to be the culprit because they are created all the time. So I rewrote the STON code to use an object pool for the writer classes but still no success. Having the pool I had one central instance that gets accessed all the time. So I tried to do on every 100000 instance request a 'System abortTransaction' hoping it would solve something.
>>>> That falsified my theory disconnected objects can fill up the space. For the gem it is like a normal vm, right? When space gets low gc chimes in. So my abortTransaction doesn't do anything the gc wouldn't do, right?
>>>> That means the objects that fill up the space are connected somehow. Now I'm asking myself what is the best way to get an overview over the instances that are in my temporary object memory. Would that be a duty for statmon or is there another way?
>>>>
>>>> thanks,
>>>>
>>>> Norbert
>>>>
>>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: STON and transactions

Dale Henrichs
Regarding the memory issues ... I assume that the issue is similar to the problem with SIXX in that you're building some very large objects (Strings in Streams?) in temporary memory. For SIXX the trick was to link the root of the large temp object graph in UserGlobals so that the temp object could be committed, thus reducing the pressure on temp obj space ... at the end, the temp object was removed from UserGlobals so that it could be garbage collected ...

Regarding the test failures in GemStone ... I will have to take a look. At the time of the port, the tests were passing, but at the moment I don't recall which versions of GemStone I tested STON against ...

I have been using STON in 3.1.0.1, but I have been using it to serialize limited object graphs (I am stopping after n-levels), so I haven't tried to serialize a big object graph from GemStone and haven't run into any errors using STON.

Regarding the error related to ScaledDecimal ... I'd like to know what the error is. I can't imagine what kind of error that ScaledDecimal would be throwing from the following code:

  STON fromStream: (String streamContents: [:stream|
        (Dictionary new
                at: Object new put: Object new;
                yourself) stonOn: (STONWriter on: stream)]) readStream

In a somewhat related are, I have found that there are bugs related to using #basicSize, #basicAt: and #basicAtPut: for a handful of classes. I've identified a problems in GsNMethod and ClassSet and this particular problem could affect the operation of STON.

Still not sure about the ScaledDecimal problem...

Dale
----- Original Message -----
| From: "Norbert Hartl" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Thursday, October 25, 2012 4:53:45 AM
| Subject: Re: [GS/SS Beta] STON and transactions
|
| And finally the problem I've found seems to be a problem in the
| design of STON (or just me misunderstanding its purpose). Anyway the
| following code snippet raises the error
|
| STON fromStream: (String streamContents: [:stream|
| (Dictionary new
| at: Object new put: Object new;
| yourself) stonOn: (STONWriter on: stream)]) readStream
|
| Norbert
|
| Am 25.10.2012 um 13:37 schrieb Norbert Hartl <[hidden email]>:
|
| > This is some sort of monolog :) I just forgot to add to the last
| > mail that reading back the file in GemStone fails with the same
| > error as in pharo.
| >
| > Norbert
| >
| > Am 25.10.2012 um 13:35 schrieb Norbert Hartl <[hidden email]>:
| >
| >> Unfortunately only the export worked "somehow". In pharo I cannot
| >> read in the STON file produced by gemstone. Tests are green on
| >> pharo. On gemstone there are two failing tests that have to do
| >> with ScaledDecimal.  You said you are working with it and you did
| >> not encounter problems?
| >>
| >> Norbert
| >>
| >> Am 25.10.2012 um 11:11 schrieb Norbert Hartl <[hidden email]>:
| >>
| >>> Haha,
| >>>
| >>> often after I write a mail the situation changes somehow :) While
| >>> fighting in the mist I tried doing a commitTransaction instead
| >>> of abortTransaction and this works. At least I can export my
| >>> object graph now. If there is a good explanation I'm glad to
| >>> hear it :)
| >>>
| >>> Norbert
| >>>
| >>> Am 25.10.2012 um 11:03 schrieb Norbert Hartl
| >>> <[hidden email]>:
| >>>
| >>>> Dale,
| >>>>
| >>>> I loaded the STON code with your snippet and this worked. But it
| >>>> didn't solve my problem. I cannot export my object graph with
| >>>> STON. I always get temporary object memory full. I gave the gem
| >>>> 200MB already and still no success. I can give it even more but
| >>>> that would postpone my problem only a couple of weeks. I like
| >>>> to find the reason why it doesn't work.
| >>>>
| >>>> Reading the code I only suspected the STONWriterXXX classes to
| >>>> be the culprit because they are created all the time. So I
| >>>> rewrote the STON code to use an object pool for the writer
| >>>> classes but still no success. Having the pool I had one central
| >>>> instance that gets accessed all the time. So I tried to do on
| >>>> every 100000 instance request a 'System abortTransaction'
| >>>> hoping it would solve something.
| >>>> That falsified my theory disconnected objects can fill up the
| >>>> space. For the gem it is like a normal vm, right? When space
| >>>> gets low gc chimes in. So my abortTransaction doesn't do
| >>>> anything the gc wouldn't do, right?
| >>>> That means the objects that fill up the space are connected
| >>>> somehow. Now I'm asking myself what is the best way to get an
| >>>> overview over the instances that are in my temporary object
| >>>> memory. Would that be a duty for statmon or is there another
| >>>> way?
| >>>>
| >>>> thanks,
| >>>>
| >>>> Norbert
| >>>>
| >>>
| >>
| >
|
|
Reply | Threaded
Open this post in threaded view
|

Re: STON and transactions

NorbertHartl

Am 25.10.2012 um 16:52 schrieb Dale Henrichs <[hidden email]>:

> Regarding the memory issues ... I assume that the issue is similar to the problem with SIXX in that you're building some very large objects (Strings in Streams?) in temporary memory. For SIXX the trick was to link the root of the large temp object graph in UserGlobals so that the temp object could be committed, thus reducing the pressure on temp obj space ... at the end, the temp object was removed from UserGlobals so that it could be garbage collected …
>
The out of memory appears when serializing. Here there is just a file stream where content is written to. That is what makes me wonder.

> Regarding the test failures in GemStone ... I will have to take a look. At the time of the port, the tests were passing, but at the moment I don't recall which versions of GemStone I tested STON against ...
>
> I have been using STON in 3.1.0.1, but I have been using it to serialize limited object graphs (I am stopping after n-levels), so I haven't tried to serialize a big object graph from GemStone and haven't run into any errors using STON.
>
> Regarding the error related to ScaledDecimal ... I'd like to know what the error is. I can't imagine what kind of error that ScaledDecimal would be throwing from the following code:
>
>  STON fromStream: (String streamContents: [:stream|
>        (Dictionary new
>                at: Object new put: Object new;
>                yourself) stonOn: (STONWriter on: stream)]) readStream
>
> In a somewhat related are, I have found that there are bugs related to using #basicSize, #basicAt: and #basicAtPut: for a handful of classes. I've identified a problems in GsNMethod and ClassSet and this particular problem could affect the operation of STON.
>
> Still not sure about the ScaledDecimal problem…
>
The ScaledDecimal problem appears when running the tests. The code above just demonstrates another problem. Talking to sven he told me objects as map keys are not supported by design. Maybe there will be an option to enable using them but that was the error I was experiencing.

Norbert


> Dale
> ----- Original Message -----
> | From: "Norbert Hartl" <[hidden email]>
> | To: "GemStone Seaside beta discussion" <[hidden email]>
> | Sent: Thursday, October 25, 2012 4:53:45 AM
> | Subject: Re: [GS/SS Beta] STON and transactions
> |
> | And finally the problem I've found seems to be a problem in the
> | design of STON (or just me misunderstanding its purpose). Anyway the
> | following code snippet raises the error
> |
> | STON fromStream: (String streamContents: [:stream|
> | (Dictionary new
> | at: Object new put: Object new;
> | yourself) stonOn: (STONWriter on: stream)]) readStream
> |
> | Norbert
> |
> | Am 25.10.2012 um 13:37 schrieb Norbert Hartl <[hidden email]>:
> |
> | > This is some sort of monolog :) I just forgot to add to the last
> | > mail that reading back the file in GemStone fails with the same
> | > error as in pharo.
> | >
> | > Norbert
> | >
> | > Am 25.10.2012 um 13:35 schrieb Norbert Hartl <[hidden email]>:
> | >
> | >> Unfortunately only the export worked "somehow". In pharo I cannot
> | >> read in the STON file produced by gemstone. Tests are green on
> | >> pharo. On gemstone there are two failing tests that have to do
> | >> with ScaledDecimal.  You said you are working with it and you did
> | >> not encounter problems?
> | >>
> | >> Norbert
> | >>
> | >> Am 25.10.2012 um 11:11 schrieb Norbert Hartl <[hidden email]>:
> | >>
> | >>> Haha,
> | >>>
> | >>> often after I write a mail the situation changes somehow :) While
> | >>> fighting in the mist I tried doing a commitTransaction instead
> | >>> of abortTransaction and this works. At least I can export my
> | >>> object graph now. If there is a good explanation I'm glad to
> | >>> hear it :)
> | >>>
> | >>> Norbert
> | >>>
> | >>> Am 25.10.2012 um 11:03 schrieb Norbert Hartl
> | >>> <[hidden email]>:
> | >>>
> | >>>> Dale,
> | >>>>
> | >>>> I loaded the STON code with your snippet and this worked. But it
> | >>>> didn't solve my problem. I cannot export my object graph with
> | >>>> STON. I always get temporary object memory full. I gave the gem
> | >>>> 200MB already and still no success. I can give it even more but
> | >>>> that would postpone my problem only a couple of weeks. I like
> | >>>> to find the reason why it doesn't work.
> | >>>>
> | >>>> Reading the code I only suspected the STONWriterXXX classes to
> | >>>> be the culprit because they are created all the time. So I
> | >>>> rewrote the STON code to use an object pool for the writer
> | >>>> classes but still no success. Having the pool I had one central
> | >>>> instance that gets accessed all the time. So I tried to do on
> | >>>> every 100000 instance request a 'System abortTransaction'
> | >>>> hoping it would solve something.
> | >>>> That falsified my theory disconnected objects can fill up the
> | >>>> space. For the gem it is like a normal vm, right? When space
> | >>>> gets low gc chimes in. So my abortTransaction doesn't do
> | >>>> anything the gc wouldn't do, right?
> | >>>> That means the objects that fill up the space are connected
> | >>>> somehow. Now I'm asking myself what is the best way to get an
> | >>>> overview over the instances that are in my temporary object
> | >>>> memory. Would that be a duty for statmon or is there another
> | >>>> way?
> | >>>>
> | >>>> thanks,
> | >>>>
> | >>>> Norbert
> | >>>>
> | >>>
> | >>
> | >
> |
> |

Reply | Threaded
Open this post in threaded view
|

Re: STON and transactions

NorbertHartl
In reply to this post by Dale Henrichs
Dale,

I don't know how good you are in "wild guessing" but I think you are an expert :). To place an estimation from the gut what would you think how much work it will be to port Fuel to GemStone. You can leave Blocks aside.

Norbert

Am 25.10.2012 um 16:52 schrieb Dale Henrichs <[hidden email]>:

> Regarding the memory issues ... I assume that the issue is similar to the problem with SIXX in that you're building some very large objects (Strings in Streams?) in temporary memory. For SIXX the trick was to link the root of the large temp object graph in UserGlobals so that the temp object could be committed, thus reducing the pressure on temp obj space ... at the end, the temp object was removed from UserGlobals so that it could be garbage collected ...
>
> Regarding the test failures in GemStone ... I will have to take a look. At the time of the port, the tests were passing, but at the moment I don't recall which versions of GemStone I tested STON against ...
>
> I have been using STON in 3.1.0.1, but I have been using it to serialize limited object graphs (I am stopping after n-levels), so I haven't tried to serialize a big object graph from GemStone and haven't run into any errors using STON.
>
> Regarding the error related to ScaledDecimal ... I'd like to know what the error is. I can't imagine what kind of error that ScaledDecimal would be throwing from the following code:
>
>  STON fromStream: (String streamContents: [:stream|
>        (Dictionary new
>                at: Object new put: Object new;
>                yourself) stonOn: (STONWriter on: stream)]) readStream
>
> In a somewhat related are, I have found that there are bugs related to using #basicSize, #basicAt: and #basicAtPut: for a handful of classes. I've identified a problems in GsNMethod and ClassSet and this particular problem could affect the operation of STON.
>
> Still not sure about the ScaledDecimal problem...
>
> Dale
> ----- Original Message -----
> | From: "Norbert Hartl" <[hidden email]>
> | To: "GemStone Seaside beta discussion" <[hidden email]>
> | Sent: Thursday, October 25, 2012 4:53:45 AM
> | Subject: Re: [GS/SS Beta] STON and transactions
> |
> | And finally the problem I've found seems to be a problem in the
> | design of STON (or just me misunderstanding its purpose). Anyway the
> | following code snippet raises the error
> |
> | STON fromStream: (String streamContents: [:stream|
> | (Dictionary new
> | at: Object new put: Object new;
> | yourself) stonOn: (STONWriter on: stream)]) readStream
> |
> | Norbert
> |
> | Am 25.10.2012 um 13:37 schrieb Norbert Hartl <[hidden email]>:
> |
> | > This is some sort of monolog :) I just forgot to add to the last
> | > mail that reading back the file in GemStone fails with the same
> | > error as in pharo.
> | >
> | > Norbert
> | >
> | > Am 25.10.2012 um 13:35 schrieb Norbert Hartl <[hidden email]>:
> | >
> | >> Unfortunately only the export worked "somehow". In pharo I cannot
> | >> read in the STON file produced by gemstone. Tests are green on
> | >> pharo. On gemstone there are two failing tests that have to do
> | >> with ScaledDecimal.  You said you are working with it and you did
> | >> not encounter problems?
> | >>
> | >> Norbert
> | >>
> | >> Am 25.10.2012 um 11:11 schrieb Norbert Hartl <[hidden email]>:
> | >>
> | >>> Haha,
> | >>>
> | >>> often after I write a mail the situation changes somehow :) While
> | >>> fighting in the mist I tried doing a commitTransaction instead
> | >>> of abortTransaction and this works. At least I can export my
> | >>> object graph now. If there is a good explanation I'm glad to
> | >>> hear it :)
> | >>>
> | >>> Norbert
> | >>>
> | >>> Am 25.10.2012 um 11:03 schrieb Norbert Hartl
> | >>> <[hidden email]>:
> | >>>
> | >>>> Dale,
> | >>>>
> | >>>> I loaded the STON code with your snippet and this worked. But it
> | >>>> didn't solve my problem. I cannot export my object graph with
> | >>>> STON. I always get temporary object memory full. I gave the gem
> | >>>> 200MB already and still no success. I can give it even more but
> | >>>> that would postpone my problem only a couple of weeks. I like
> | >>>> to find the reason why it doesn't work.
> | >>>>
> | >>>> Reading the code I only suspected the STONWriterXXX classes to
> | >>>> be the culprit because they are created all the time. So I
> | >>>> rewrote the STON code to use an object pool for the writer
> | >>>> classes but still no success. Having the pool I had one central
> | >>>> instance that gets accessed all the time. So I tried to do on
> | >>>> every 100000 instance request a 'System abortTransaction'
> | >>>> hoping it would solve something.
> | >>>> That falsified my theory disconnected objects can fill up the
> | >>>> space. For the gem it is like a normal vm, right? When space
> | >>>> gets low gc chimes in. So my abortTransaction doesn't do
> | >>>> anything the gc wouldn't do, right?
> | >>>> That means the objects that fill up the space are connected
> | >>>> somehow. Now I'm asking myself what is the best way to get an
> | >>>> overview over the instances that are in my temporary object
> | >>>> memory. Would that be a duty for statmon or is there another
> | >>>> way?
> | >>>>
> | >>>> thanks,
> | >>>>
> | >>>> Norbert
> | >>>>
> | >>>
> | >>
> | >
> |
> |

Reply | Threaded
Open this post in threaded view
|

Re: STON and transactions

Dale Henrichs
In reply to this post by NorbertHartl


----- Original Message -----
| From: "Norbert Hartl" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Thursday, October 25, 2012 8:06:35 AM
| Subject: Re: [GS/SS Beta] STON and transactions
|
|
| Am 25.10.2012 um 16:52 schrieb Dale Henrichs <[hidden email]>:
|
| > Regarding the memory issues ... I assume that the issue is similar
| > to the problem with SIXX in that you're building some very large
| > objects (Strings in Streams?) in temporary memory. For SIXX the
| > trick was to link the root of the large temp object graph in
| > UserGlobals so that the temp object could be committed, thus
| > reducing the pressure on temp obj space ... at the end, the temp
| > object was removed from UserGlobals so that it could be garbage
| > collected …
| >
| The out of memory appears when serializing. Here there is just a file
| stream where content is written to. That is what makes me wonder.

You mentioned that a commit solved the out of memory problem?

To get to the bottom of this, you can wrap the code with an AlmostOutOfMemory handler:

    AlmostOutOfMemory enable.
    [...] onException: AlmostOutOfMemory do:[ :ex | ex halt ].

and then use:

  System _vmInstanceCounts: 0

which will return an array of associations whose key is a class and whose value is an array with the first slot the instance count and the second slot the total size of the instances in bytes.

You can sort the array to find the classes that are contributing the most to the bloat and then go from there...

other values for _vmInstanceCounts: arg (unless it's changed recently):
  0 = all generations
  1 = new gen only
  2 = pom gen only
  3 = old only
  4 = perm only
  5 = code only

|
| > Regarding the test failures in GemStone ... I will have to take a
| > look. At the time of the port, the tests were passing, but at the
| > moment I don't recall which versions of GemStone I tested STON
| > against ...
| >
| > I have been using STON in 3.1.0.1, but I have been using it to
| > serialize limited object graphs (I am stopping after n-levels), so
| > I haven't tried to serialize a big object graph from GemStone and
| > haven't run into any errors using STON.
| >
| > Regarding the error related to ScaledDecimal ... I'd like to know
| > what the error is. I can't imagine what kind of error that
| > ScaledDecimal would be throwing from the following code:
| >
| >  STON fromStream: (String streamContents: [:stream|
| >        (Dictionary new
| >                at: Object new put: Object new;
| >                yourself) stonOn: (STONWriter on: stream)])
| >                readStream
| >
| > In a somewhat related are, I have found that there are bugs related
| > to using #basicSize, #basicAt: and #basicAtPut: for a handful of
| > classes. I've identified a problems in GsNMethod and ClassSet and
| > this particular problem could affect the operation of STON.
| >
| > Still not sure about the ScaledDecimal problem…
| >
| The ScaledDecimal problem appears when running the tests. The code
| above just demonstrates another problem. Talking to sven he told me
| objects as map keys are not supported by design. Maybe there will be
| an option to enable using them but that was the error I was
| experiencing.

I will be able to see the ScaledDecimal problem for myself then.

I didn't realize that STON had the limitation of not serializing Dictionaries with Object keys ...

Dale
Reply | Threaded
Open this post in threaded view
|

Re: STON and transactions

NorbertHartl

Am 25.10.2012 um 17:31 schrieb Dale Henrichs <[hidden email]>:

>
>
> ----- Original Message -----
> | From: "Norbert Hartl" <[hidden email]>
> | To: "GemStone Seaside beta discussion" <[hidden email]>
> | Sent: Thursday, October 25, 2012 8:06:35 AM
> | Subject: Re: [GS/SS Beta] STON and transactions
> |
> |
> | Am 25.10.2012 um 16:52 schrieb Dale Henrichs <[hidden email]>:
> |
> | > Regarding the memory issues ... I assume that the issue is similar
> | > to the problem with SIXX in that you're building some very large
> | > objects (Strings in Streams?) in temporary memory. For SIXX the
> | > trick was to link the root of the large temp object graph in
> | > UserGlobals so that the temp object could be committed, thus
> | > reducing the pressure on temp obj space ... at the end, the temp
> | > object was removed from UserGlobals so that it could be garbage
> | > collected …
> | >
> | The out of memory appears when serializing. Here there is just a file
> | stream where content is written to. That is what makes me wonder.
>
> You mentioned that a commit solved the out of memory problem?
>
> To get to the bottom of this, you can wrap the code with an AlmostOutOfMemory handler:
>
>    AlmostOutOfMemory enable.
>    [...] onException: AlmostOutOfMemory do:[ :ex | ex halt ].
>
> and then use:
>
>  System _vmInstanceCounts: 0
>
> which will return an array of associations whose key is a class and whose value is an array with the first slot the instance count and the second slot the total size of the instances in bytes.
>
> You can sort the array to find the classes that are contributing the most to the bloat and then go from there...
>
> other values for _vmInstanceCounts: arg (unless it's changed recently):
>  0 = all generations
>  1 = new gen only
>  2 = pom gen only
>  3 = old only
>  4 = perm only
>  5 = code only
>
I think that is exactly what I was asking for. Cool!

> |
> | > Regarding the test failures in GemStone ... I will have to take a
> | > look. At the time of the port, the tests were passing, but at the
> | > moment I don't recall which versions of GemStone I tested STON
> | > against ...
> | >
> | > I have been using STON in 3.1.0.1, but I have been using it to
> | > serialize limited object graphs (I am stopping after n-levels), so
> | > I haven't tried to serialize a big object graph from GemStone and
> | > haven't run into any errors using STON.
> | >
> | > Regarding the error related to ScaledDecimal ... I'd like to know
> | > what the error is. I can't imagine what kind of error that
> | > ScaledDecimal would be throwing from the following code:
> | >
> | >  STON fromStream: (String streamContents: [:stream|
> | >        (Dictionary new
> | >                at: Object new put: Object new;
> | >                yourself) stonOn: (STONWriter on: stream)])
> | >                readStream
> | >
> | > In a somewhat related are, I have found that there are bugs related
> | > to using #basicSize, #basicAt: and #basicAtPut: for a handful of
> | > classes. I've identified a problems in GsNMethod and ClassSet and
> | > this particular problem could affect the operation of STON.
> | >
> | > Still not sure about the ScaledDecimal problem…
> | >
> | The ScaledDecimal problem appears when running the tests. The code
> | above just demonstrates another problem. Talking to sven he told me
> | objects as map keys are not supported by design. Maybe there will be
> | an option to enable using them but that was the error I was
> | experiencing.
>
> I will be able to see the ScaledDecimal problem for myself then.
>
> I didn't realize that STON had the limitation of not serializing Dictionaries with Object keys ...
>
As far as I understand it was an deliberate act to be close to JSON. Technically (code wise) there is no limitation in this regard. Sven sent me a piece of code to enable object key for review. So I think the limitation can be circumvented in a short time frame

Norbert

Reply | Threaded
Open this post in threaded view
|

Re: STON and transactions

Dale Henrichs
In reply to this post by NorbertHartl
I haven't even looked at Fuel, so it will be a wild guess indeed.

Are you planning on using Fuel to do GemStone to GemStone transfers or GemStone to Pharo transfers? GemStone 2.x to GemStone 3.x transfers? or GemStone to GemStone of the same major version?

If you are talking GemStone to GemStone of same major version. You might want to look at Object>>passivate (and look at manual). Object passivation suffers from the same problem (running out of stack space) that SIXX had (before I changed some of the algorithms to not use recursion) and come to think of it ... If Fuel uses recursion while walking the object graph, it will have stack problems as well ...

Cross dialect or even cross major version transfers probably can't be done well with Fuel (unless it is implemented differently than I imagine) so STON or SIXX are your best bets. BTW, what's wrong with SIXX?

With all of that said I can't see it taking more than a week or two (63.458 hours). Classes are shaped differently in GemStone, but you can look at the passivate/active code for the special cases ... it takes a bit of thought but the recursive algorithms can be unwound into loops (makes for horrid debugging) without too much trouble.

Dale

----- Original Message -----
| From: "Norbert Hartl" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Thursday, October 25, 2012 8:21:24 AM
| Subject: Re: [GS/SS Beta] STON and transactions
|
| Dale,
|
| I don't know how good you are in "wild guessing" but I think you are
| an expert :). To place an estimation from the gut what would you
| think how much work it will be to port Fuel to GemStone. You can
| leave Blocks aside.
|
| Norbert
|
| Am 25.10.2012 um 16:52 schrieb Dale Henrichs <[hidden email]>:
|
| > Regarding the memory issues ... I assume that the issue is similar
| > to the problem with SIXX in that you're building some very large
| > objects (Strings in Streams?) in temporary memory. For SIXX the
| > trick was to link the root of the large temp object graph in
| > UserGlobals so that the temp object could be committed, thus
| > reducing the pressure on temp obj space ... at the end, the temp
| > object was removed from UserGlobals so that it could be garbage
| > collected ...
| >
| > Regarding the test failures in GemStone ... I will have to take a
| > look. At the time of the port, the tests were passing, but at the
| > moment I don't recall which versions of GemStone I tested STON
| > against ...
| >
| > I have been using STON in 3.1.0.1, but I have been using it to
| > serialize limited object graphs (I am stopping after n-levels), so
| > I haven't tried to serialize a big object graph from GemStone and
| > haven't run into any errors using STON.
| >
| > Regarding the error related to ScaledDecimal ... I'd like to know
| > what the error is. I can't imagine what kind of error that
| > ScaledDecimal would be throwing from the following code:
| >
| >  STON fromStream: (String streamContents: [:stream|
| >        (Dictionary new
| >                at: Object new put: Object new;
| >                yourself) stonOn: (STONWriter on: stream)])
| >                readStream
| >
| > In a somewhat related are, I have found that there are bugs related
| > to using #basicSize, #basicAt: and #basicAtPut: for a handful of
| > classes. I've identified a problems in GsNMethod and ClassSet and
| > this particular problem could affect the operation of STON.
| >
| > Still not sure about the ScaledDecimal problem...
| >
| > Dale
| > ----- Original Message -----
| > | From: "Norbert Hartl" <[hidden email]>
| > | To: "GemStone Seaside beta discussion"
| > | <[hidden email]>
| > | Sent: Thursday, October 25, 2012 4:53:45 AM
| > | Subject: Re: [GS/SS Beta] STON and transactions
| > |
| > | And finally the problem I've found seems to be a problem in the
| > | design of STON (or just me misunderstanding its purpose). Anyway
| > | the
| > | following code snippet raises the error
| > |
| > | STON fromStream: (String streamContents: [:stream|
| > | (Dictionary new
| > | at: Object new put: Object new;
| > | yourself) stonOn: (STONWriter on: stream)]) readStream
| > |
| > | Norbert
| > |
| > | Am 25.10.2012 um 13:37 schrieb Norbert Hartl
| > | <[hidden email]>:
| > |
| > | > This is some sort of monolog :) I just forgot to add to the
| > | > last
| > | > mail that reading back the file in GemStone fails with the same
| > | > error as in pharo.
| > | >
| > | > Norbert
| > | >
| > | > Am 25.10.2012 um 13:35 schrieb Norbert Hartl
| > | > <[hidden email]>:
| > | >
| > | >> Unfortunately only the export worked "somehow". In pharo I
| > | >> cannot
| > | >> read in the STON file produced by gemstone. Tests are green on
| > | >> pharo. On gemstone there are two failing tests that have to do
| > | >> with ScaledDecimal.  You said you are working with it and you
| > | >> did
| > | >> not encounter problems?
| > | >>
| > | >> Norbert
| > | >>
| > | >> Am 25.10.2012 um 11:11 schrieb Norbert Hartl
| > | >> <[hidden email]>:
| > | >>
| > | >>> Haha,
| > | >>>
| > | >>> often after I write a mail the situation changes somehow :)
| > | >>> While
| > | >>> fighting in the mist I tried doing a commitTransaction
| > | >>> instead
| > | >>> of abortTransaction and this works. At least I can export my
| > | >>> object graph now. If there is a good explanation I'm glad to
| > | >>> hear it :)
| > | >>>
| > | >>> Norbert
| > | >>>
| > | >>> Am 25.10.2012 um 11:03 schrieb Norbert Hartl
| > | >>> <[hidden email]>:
| > | >>>
| > | >>>> Dale,
| > | >>>>
| > | >>>> I loaded the STON code with your snippet and this worked.
| > | >>>> But it
| > | >>>> didn't solve my problem. I cannot export my object graph
| > | >>>> with
| > | >>>> STON. I always get temporary object memory full. I gave the
| > | >>>> gem
| > | >>>> 200MB already and still no success. I can give it even more
| > | >>>> but
| > | >>>> that would postpone my problem only a couple of weeks. I
| > | >>>> like
| > | >>>> to find the reason why it doesn't work.
| > | >>>>
| > | >>>> Reading the code I only suspected the STONWriterXXX classes
| > | >>>> to
| > | >>>> be the culprit because they are created all the time. So I
| > | >>>> rewrote the STON code to use an object pool for the writer
| > | >>>> classes but still no success. Having the pool I had one
| > | >>>> central
| > | >>>> instance that gets accessed all the time. So I tried to do
| > | >>>> on
| > | >>>> every 100000 instance request a 'System abortTransaction'
| > | >>>> hoping it would solve something.
| > | >>>> That falsified my theory disconnected objects can fill up
| > | >>>> the
| > | >>>> space. For the gem it is like a normal vm, right? When space
| > | >>>> gets low gc chimes in. So my abortTransaction doesn't do
| > | >>>> anything the gc wouldn't do, right?
| > | >>>> That means the objects that fill up the space are connected
| > | >>>> somehow. Now I'm asking myself what is the best way to get
| > | >>>> an
| > | >>>> overview over the instances that are in my temporary object
| > | >>>> memory. Would that be a duty for statmon or is there another
| > | >>>> way?
| > | >>>>
| > | >>>> thanks,
| > | >>>>
| > | >>>> Norbert
| > | >>>>
| > | >>>
| > | >>
| > | >
| > |
| > |
|
|
Reply | Threaded
Open this post in threaded view
|

Re: STON and transactions

Mariano Martinez Peck


On Thu, Oct 25, 2012 at 5:46 PM, Dale Henrichs <[hidden email]> wrote:
I haven't even looked at Fuel, so it will be a wild guess indeed.

Are you planning on using Fuel to do GemStone to GemStone transfers or GemStone to Pharo transfers? GemStone 2.x to GemStone 3.x transfers? or GemStone to GemStone of the same major version?

If you are talking GemStone to GemStone of same major version. You might want to look at Object>>passivate (and look at manual). Object passivation suffers from the same problem (running out of stack space) that SIXX had (before I changed some of the algorithms to not use recursion) and come to think of it ... If Fuel uses recursion while walking the object graph, it will have stack problems as well ...

Hi dale. Fuel does not use recursion. Instead, we use a stack where we pop/push objects and the key thing is that we first write objects (without references) and then the references (these are done in different steps). So you won't have the problem of the stack (if I understand it correctly)
 

Cross dialect or even cross major version transfers probably can't be done well with Fuel (unless it is implemented differently than I imagine) so STON or SIXX are your best bets. BTW, what's wrong with SIXX?


Indeed, cross dialect is more difficult with Fuel than with STON/SIXX, but Fuel was designed with other goals in mind. 
 
With all of that said I can't see it taking more than a week or two (63.458 hours). Classes are shaped differently in GemStone, but you can look at the passivate/active code for the special cases ... it takes a bit of thought but the recursive algorithms can be unwound into loops (makes for horrid debugging) without too much trouble.


I would love if someone makes Fuel work on Gemstone :)
 
Dale

----- Original Message -----
| From: "Norbert Hartl" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Thursday, October 25, 2012 8:21:24 AM
| Subject: Re: [GS/SS Beta] STON and transactions
|
| Dale,
|
| I don't know how good you are in "wild guessing" but I think you are
| an expert :). To place an estimation from the gut what would you
| think how much work it will be to port Fuel to GemStone. You can
| leave Blocks aside.
|
| Norbert
|
| Am 25.10.2012 um 16:52 schrieb Dale Henrichs <[hidden email]>:
|
| > Regarding the memory issues ... I assume that the issue is similar
| > to the problem with SIXX in that you're building some very large
| > objects (Strings in Streams?) in temporary memory. For SIXX the
| > trick was to link the root of the large temp object graph in
| > UserGlobals so that the temp object could be committed, thus
| > reducing the pressure on temp obj space ... at the end, the temp
| > object was removed from UserGlobals so that it could be garbage
| > collected ...
| >
| > Regarding the test failures in GemStone ... I will have to take a
| > look. At the time of the port, the tests were passing, but at the
| > moment I don't recall which versions of GemStone I tested STON
| > against ...
| >
| > I have been using STON in <a href="tel:3.1.0.1" value="+333101">3.1.0.1, but I have been using it to
| > serialize limited object graphs (I am stopping after n-levels), so
| > I haven't tried to serialize a big object graph from GemStone and
| > haven't run into any errors using STON.
| >
| > Regarding the error related to ScaledDecimal ... I'd like to know
| > what the error is. I can't imagine what kind of error that
| > ScaledDecimal would be throwing from the following code:
| >
| >  STON fromStream: (String streamContents: [:stream|
| >        (Dictionary new
| >                at: Object new put: Object new;
| >                yourself) stonOn: (STONWriter on: stream)])
| >                readStream
| >
| > In a somewhat related are, I have found that there are bugs related
| > to using #basicSize, #basicAt: and #basicAtPut: for a handful of
| > classes. I've identified a problems in GsNMethod and ClassSet and
| > this particular problem could affect the operation of STON.
| >
| > Still not sure about the ScaledDecimal problem...
| >
| > Dale
| > ----- Original Message -----
| > | From: "Norbert Hartl" <[hidden email]>
| > | To: "GemStone Seaside beta discussion"
| > | <[hidden email]>
| > | Sent: Thursday, October 25, 2012 4:53:45 AM
| > | Subject: Re: [GS/SS Beta] STON and transactions
| > |
| > | And finally the problem I've found seems to be a problem in the
| > | design of STON (or just me misunderstanding its purpose). Anyway
| > | the
| > | following code snippet raises the error
| > |
| > | STON fromStream: (String streamContents: [:stream|
| > |   (Dictionary new
| > |           at: Object new put: Object new;
| > |           yourself) stonOn: (STONWriter on: stream)]) readStream
| > |
| > | Norbert
| > |
| > | Am 25.10.2012 um 13:37 schrieb Norbert Hartl
| > | <[hidden email]>:
| > |
| > | > This is some sort of monolog :) I just forgot to add to the
| > | > last
| > | > mail that reading back the file in GemStone fails with the same
| > | > error as in pharo.
| > | >
| > | > Norbert
| > | >
| > | > Am 25.10.2012 um 13:35 schrieb Norbert Hartl
| > | > <[hidden email]>:
| > | >
| > | >> Unfortunately only the export worked "somehow". In pharo I
| > | >> cannot
| > | >> read in the STON file produced by gemstone. Tests are green on
| > | >> pharo. On gemstone there are two failing tests that have to do
| > | >> with ScaledDecimal.  You said you are working with it and you
| > | >> did
| > | >> not encounter problems?
| > | >>
| > | >> Norbert
| > | >>
| > | >> Am 25.10.2012 um 11:11 schrieb Norbert Hartl
| > | >> <[hidden email]>:
| > | >>
| > | >>> Haha,
| > | >>>
| > | >>> often after I write a mail the situation changes somehow :)
| > | >>> While
| > | >>> fighting in the mist I tried doing a commitTransaction
| > | >>> instead
| > | >>> of abortTransaction and this works. At least I can export my
| > | >>> object graph now. If there is a good explanation I'm glad to
| > | >>> hear it :)
| > | >>>
| > | >>> Norbert
| > | >>>
| > | >>> Am 25.10.2012 um 11:03 schrieb Norbert Hartl
| > | >>> <[hidden email]>:
| > | >>>
| > | >>>> Dale,
| > | >>>>
| > | >>>> I loaded the STON code with your snippet and this worked.
| > | >>>> But it
| > | >>>> didn't solve my problem. I cannot export my object graph
| > | >>>> with
| > | >>>> STON. I always get temporary object memory full. I gave the
| > | >>>> gem
| > | >>>> 200MB already and still no success. I can give it even more
| > | >>>> but
| > | >>>> that would postpone my problem only a couple of weeks. I
| > | >>>> like
| > | >>>> to find the reason why it doesn't work.
| > | >>>>
| > | >>>> Reading the code I only suspected the STONWriterXXX classes
| > | >>>> to
| > | >>>> be the culprit because they are created all the time. So I
| > | >>>> rewrote the STON code to use an object pool for the writer
| > | >>>> classes but still no success. Having the pool I had one
| > | >>>> central
| > | >>>> instance that gets accessed all the time. So I tried to do
| > | >>>> on
| > | >>>> every 100000 instance request a 'System abortTransaction'
| > | >>>> hoping it would solve something.
| > | >>>> That falsified my theory disconnected objects can fill up
| > | >>>> the
| > | >>>> space. For the gem it is like a normal vm, right? When space
| > | >>>> gets low gc chimes in. So my abortTransaction doesn't do
| > | >>>> anything the gc wouldn't do, right?
| > | >>>> That means the objects that fill up the space are connected
| > | >>>> somehow. Now I'm asking myself what is the best way to get
| > | >>>> an
| > | >>>> overview over the instances that are in my temporary object
| > | >>>> memory. Would that be a duty for statmon or is there another
| > | >>>> way?
| > | >>>>
| > | >>>> thanks,
| > | >>>>
| > | >>>> Norbert
| > | >>>>
| > | >>>
| > | >>
| > | >
| > |
| > |
|
|



--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: STON and transactions

NorbertHartl
In reply to this post by Dale Henrichs

Am 25.10.2012 um 17:46 schrieb Dale Henrichs <[hidden email]>:

> I haven't even looked at Fuel, so it will be a wild guess indeed.
>
> Are you planning on using Fuel to do GemStone to GemStone transfers or GemStone to Pharo transfers? GemStone 2.x to GemStone 3.x transfers? or GemStone to GemStone of the same major version?
>
I was just thinking about transfer from gemstone to pharo. But gemstone to gemstone on same version is also interesting.

> If you are talking GemStone to GemStone of same major version. You might want to look at Object>>passivate (and look at manual). Object passivation suffers from the same problem (running out of stack space) that SIXX had (before I changed some of the algorithms to not use recursion) and come to think of it ... If Fuel uses recursion while walking the object graph, it will have stack problems as well …
>
agreed

> Cross dialect or even cross major version transfers probably can't be done well with Fuel (unless it is implemented differently than I imagine) so STON or SIXX are your best bets. BTW, what's wrong with SIXX?
>
The biggest problem is that there is no pull parser available (I know Ken Treis did one but the xml parser changed quite a lot since then). Writing out a graph depends mainly on the depth traversal management and that there is enough space for a object to reference table. Reading in could be done similar if a pure streaming solution would be available. In Sixx the XML is very verbose producing huge files. Without a pull parser you read in the whole file as string, then convert it into a DOM which is even more bloated. Just to finally walk the DOM and create objects. That makes to problem worse at some magnitude. So STON is much more slim in string size and it is also smaller in code size. Making it more easy to tweak it where needed.

Or to say it in numbers. Taking a fresh extent and reading in a couple of 10000s of objects with SIXX took me the last time 40 minutes and left back an extent with a size of 25GB. Now I have something between 2 and 5 million objects. I didn't even think about to try.

> With all of that said I can't see it taking more than a week or two (63.458 hours). Classes are shaped differently in GemStone, but you can look at the passivate/active code for the special cases ... it takes a bit of thought but the recursive algorithms can be unwound into loops (makes for horrid debugging) without too much trouble.
>
That is a good tip. I'll look into the passivate stuff. You are right indeed. The different shapes of objects might be a big problem. I think Fuel has some shape changing support (as SIXX has). But if it comes to the internals of objects that might be a really annoying task to do.

thanks for your detailed estimation,

Norbert

> Dale
>
> ----- Original Message -----
> | From: "Norbert Hartl" <[hidden email]>
> | To: "GemStone Seaside beta discussion" <[hidden email]>
> | Sent: Thursday, October 25, 2012 8:21:24 AM
> | Subject: Re: [GS/SS Beta] STON and transactions
> |
> | Dale,
> |
> | I don't know how good you are in "wild guessing" but I think you are
> | an expert :). To place an estimation from the gut what would you
> | think how much work it will be to port Fuel to GemStone. You can
> | leave Blocks aside.
> |
> | Norbert
> |
> | Am 25.10.2012 um 16:52 schrieb Dale Henrichs <[hidden email]>:
> |
> | > Regarding the memory issues ... I assume that the issue is similar
> | > to the problem with SIXX in that you're building some very large
> | > objects (Strings in Streams?) in temporary memory. For SIXX the
> | > trick was to link the root of the large temp object graph in
> | > UserGlobals so that the temp object could be committed, thus
> | > reducing the pressure on temp obj space ... at the end, the temp
> | > object was removed from UserGlobals so that it could be garbage
> | > collected ...
> | >
> | > Regarding the test failures in GemStone ... I will have to take a
> | > look. At the time of the port, the tests were passing, but at the
> | > moment I don't recall which versions of GemStone I tested STON
> | > against ...
> | >
> | > I have been using STON in 3.1.0.1, but I have been using it to
> | > serialize limited object graphs (I am stopping after n-levels), so
> | > I haven't tried to serialize a big object graph from GemStone and
> | > haven't run into any errors using STON.
> | >
> | > Regarding the error related to ScaledDecimal ... I'd like to know
> | > what the error is. I can't imagine what kind of error that
> | > ScaledDecimal would be throwing from the following code:
> | >
> | >  STON fromStream: (String streamContents: [:stream|
> | >        (Dictionary new
> | >                at: Object new put: Object new;
> | >                yourself) stonOn: (STONWriter on: stream)])
> | >                readStream
> | >
> | > In a somewhat related are, I have found that there are bugs related
> | > to using #basicSize, #basicAt: and #basicAtPut: for a handful of
> | > classes. I've identified a problems in GsNMethod and ClassSet and
> | > this particular problem could affect the operation of STON.
> | >
> | > Still not sure about the ScaledDecimal problem...
> | >
> | > Dale
> | > ----- Original Message -----
> | > | From: "Norbert Hartl" <[hidden email]>
> | > | To: "GemStone Seaside beta discussion"
> | > | <[hidden email]>
> | > | Sent: Thursday, October 25, 2012 4:53:45 AM
> | > | Subject: Re: [GS/SS Beta] STON and transactions
> | > |
> | > | And finally the problem I've found seems to be a problem in the
> | > | design of STON (or just me misunderstanding its purpose). Anyway
> | > | the
> | > | following code snippet raises the error
> | > |
> | > | STON fromStream: (String streamContents: [:stream|
> | > | (Dictionary new
> | > | at: Object new put: Object new;
> | > | yourself) stonOn: (STONWriter on: stream)]) readStream
> | > |
> | > | Norbert
> | > |
> | > | Am 25.10.2012 um 13:37 schrieb Norbert Hartl
> | > | <[hidden email]>:
> | > |
> | > | > This is some sort of monolog :) I just forgot to add to the
> | > | > last
> | > | > mail that reading back the file in GemStone fails with the same
> | > | > error as in pharo.
> | > | >
> | > | > Norbert
> | > | >
> | > | > Am 25.10.2012 um 13:35 schrieb Norbert Hartl
> | > | > <[hidden email]>:
> | > | >
> | > | >> Unfortunately only the export worked "somehow". In pharo I
> | > | >> cannot
> | > | >> read in the STON file produced by gemstone. Tests are green on
> | > | >> pharo. On gemstone there are two failing tests that have to do
> | > | >> with ScaledDecimal.  You said you are working with it and you
> | > | >> did
> | > | >> not encounter problems?
> | > | >>
> | > | >> Norbert
> | > | >>
> | > | >> Am 25.10.2012 um 11:11 schrieb Norbert Hartl
> | > | >> <[hidden email]>:
> | > | >>
> | > | >>> Haha,
> | > | >>>
> | > | >>> often after I write a mail the situation changes somehow :)
> | > | >>> While
> | > | >>> fighting in the mist I tried doing a commitTransaction
> | > | >>> instead
> | > | >>> of abortTransaction and this works. At least I can export my
> | > | >>> object graph now. If there is a good explanation I'm glad to
> | > | >>> hear it :)
> | > | >>>
> | > | >>> Norbert
> | > | >>>
> | > | >>> Am 25.10.2012 um 11:03 schrieb Norbert Hartl
> | > | >>> <[hidden email]>:
> | > | >>>
> | > | >>>> Dale,
> | > | >>>>
> | > | >>>> I loaded the STON code with your snippet and this worked.
> | > | >>>> But it
> | > | >>>> didn't solve my problem. I cannot export my object graph
> | > | >>>> with
> | > | >>>> STON. I always get temporary object memory full. I gave the
> | > | >>>> gem
> | > | >>>> 200MB already and still no success. I can give it even more
> | > | >>>> but
> | > | >>>> that would postpone my problem only a couple of weeks. I
> | > | >>>> like
> | > | >>>> to find the reason why it doesn't work.
> | > | >>>>
> | > | >>>> Reading the code I only suspected the STONWriterXXX classes
> | > | >>>> to
> | > | >>>> be the culprit because they are created all the time. So I
> | > | >>>> rewrote the STON code to use an object pool for the writer
> | > | >>>> classes but still no success. Having the pool I had one
> | > | >>>> central
> | > | >>>> instance that gets accessed all the time. So I tried to do
> | > | >>>> on
> | > | >>>> every 100000 instance request a 'System abortTransaction'
> | > | >>>> hoping it would solve something.
> | > | >>>> That falsified my theory disconnected objects can fill up
> | > | >>>> the
> | > | >>>> space. For the gem it is like a normal vm, right? When space
> | > | >>>> gets low gc chimes in. So my abortTransaction doesn't do
> | > | >>>> anything the gc wouldn't do, right?
> | > | >>>> That means the objects that fill up the space are connected
> | > | >>>> somehow. Now I'm asking myself what is the best way to get
> | > | >>>> an
> | > | >>>> overview over the instances that are in my temporary object
> | > | >>>> memory. Would that be a duty for statmon or is there another
> | > | >>>> way?
> | > | >>>>
> | > | >>>> thanks,
> | > | >>>>
> | > | >>>> Norbert
> | > | >>>>
> | > | >>>
> | > | >>
> | > | >
> | > |
> | > |
> |
> |

Reply | Threaded
Open this post in threaded view
|

Re: STON and transactions

Mariano Martinez Peck


> With all of that said I can't see it taking more than a week or two (63.458 hours). Classes are shaped differently in GemStone, but you can look at the passivate/active code for the special cases ... it takes a bit of thought but the recursive algorithms can be unwound into loops (makes for horrid debugging) without too much trouble.
>
That is a good tip. I'll look into the passivate stuff. You are right indeed. The different shapes of objects might be a big problem. I think Fuel has some shape changing support (as SIXX has).

Yes, Fuel supports the reshape: add/remove/renamed an instVar, class rename, etc. 
 
But if it comes to the internals of objects that might be a really annoying task to do.

thanks for your detailed estimation,

Norbert

> Dale
>
> ----- Original Message -----
> | From: "Norbert Hartl" <[hidden email]>
> | To: "GemStone Seaside beta discussion" <[hidden email]>
> | Sent: Thursday, October 25, 2012 8:21:24 AM
> | Subject: Re: [GS/SS Beta] STON and transactions
> |
> | Dale,
> |
> | I don't know how good you are in "wild guessing" but I think you are
> | an expert :). To place an estimation from the gut what would you
> | think how much work it will be to port Fuel to GemStone. You can
> | leave Blocks aside.
> |
> | Norbert
> |
> | Am 25.10.2012 um 16:52 schrieb Dale Henrichs <[hidden email]>:
> |
> | > Regarding the memory issues ... I assume that the issue is similar
> | > to the problem with SIXX in that you're building some very large
> | > objects (Strings in Streams?) in temporary memory. For SIXX the
> | > trick was to link the root of the large temp object graph in
> | > UserGlobals so that the temp object could be committed, thus
> | > reducing the pressure on temp obj space ... at the end, the temp
> | > object was removed from UserGlobals so that it could be garbage
> | > collected ...
> | >
> | > Regarding the test failures in GemStone ... I will have to take a
> | > look. At the time of the port, the tests were passing, but at the
> | > moment I don't recall which versions of GemStone I tested STON
> | > against ...
> | >
> | > I have been using STON in <a href="tel:3.1.0.1" value="+333101">3.1.0.1, but I have been using it to
> | > serialize limited object graphs (I am stopping after n-levels), so
> | > I haven't tried to serialize a big object graph from GemStone and
> | > haven't run into any errors using STON.
> | >
> | > Regarding the error related to ScaledDecimal ... I'd like to know
> | > what the error is. I can't imagine what kind of error that
> | > ScaledDecimal would be throwing from the following code:
> | >
> | >  STON fromStream: (String streamContents: [:stream|
> | >        (Dictionary new
> | >                at: Object new put: Object new;
> | >                yourself) stonOn: (STONWriter on: stream)])
> | >                readStream
> | >
> | > In a somewhat related are, I have found that there are bugs related
> | > to using #basicSize, #basicAt: and #basicAtPut: for a handful of
> | > classes. I've identified a problems in GsNMethod and ClassSet and
> | > this particular problem could affect the operation of STON.
> | >
> | > Still not sure about the ScaledDecimal problem...
> | >
> | > Dale
> | > ----- Original Message -----
> | > | From: "Norbert Hartl" <[hidden email]>
> | > | To: "GemStone Seaside beta discussion"
> | > | <[hidden email]>
> | > | Sent: Thursday, October 25, 2012 4:53:45 AM
> | > | Subject: Re: [GS/SS Beta] STON and transactions
> | > |
> | > | And finally the problem I've found seems to be a problem in the
> | > | design of STON (or just me misunderstanding its purpose). Anyway
> | > | the
> | > | following code snippet raises the error
> | > |
> | > | STON fromStream: (String streamContents: [:stream|
> | > |         (Dictionary new
> | > |                 at: Object new put: Object new;
> | > |                 yourself) stonOn: (STONWriter on: stream)]) readStream
> | > |
> | > | Norbert
> | > |
> | > | Am 25.10.2012 um 13:37 schrieb Norbert Hartl
> | > | <[hidden email]>:
> | > |
> | > | > This is some sort of monolog :) I just forgot to add to the
> | > | > last
> | > | > mail that reading back the file in GemStone fails with the same
> | > | > error as in pharo.
> | > | >
> | > | > Norbert
> | > | >
> | > | > Am 25.10.2012 um 13:35 schrieb Norbert Hartl
> | > | > <[hidden email]>:
> | > | >
> | > | >> Unfortunately only the export worked "somehow". In pharo I
> | > | >> cannot
> | > | >> read in the STON file produced by gemstone. Tests are green on
> | > | >> pharo. On gemstone there are two failing tests that have to do
> | > | >> with ScaledDecimal.  You said you are working with it and you
> | > | >> did
> | > | >> not encounter problems?
> | > | >>
> | > | >> Norbert
> | > | >>
> | > | >> Am 25.10.2012 um 11:11 schrieb Norbert Hartl
> | > | >> <[hidden email]>:
> | > | >>
> | > | >>> Haha,
> | > | >>>
> | > | >>> often after I write a mail the situation changes somehow :)
> | > | >>> While
> | > | >>> fighting in the mist I tried doing a commitTransaction
> | > | >>> instead
> | > | >>> of abortTransaction and this works. At least I can export my
> | > | >>> object graph now. If there is a good explanation I'm glad to
> | > | >>> hear it :)
> | > | >>>
> | > | >>> Norbert
> | > | >>>
> | > | >>> Am 25.10.2012 um 11:03 schrieb Norbert Hartl
> | > | >>> <[hidden email]>:
> | > | >>>
> | > | >>>> Dale,
> | > | >>>>
> | > | >>>> I loaded the STON code with your snippet and this worked.
> | > | >>>> But it
> | > | >>>> didn't solve my problem. I cannot export my object graph
> | > | >>>> with
> | > | >>>> STON. I always get temporary object memory full. I gave the
> | > | >>>> gem
> | > | >>>> 200MB already and still no success. I can give it even more
> | > | >>>> but
> | > | >>>> that would postpone my problem only a couple of weeks. I
> | > | >>>> like
> | > | >>>> to find the reason why it doesn't work.
> | > | >>>>
> | > | >>>> Reading the code I only suspected the STONWriterXXX classes
> | > | >>>> to
> | > | >>>> be the culprit because they are created all the time. So I
> | > | >>>> rewrote the STON code to use an object pool for the writer
> | > | >>>> classes but still no success. Having the pool I had one
> | > | >>>> central
> | > | >>>> instance that gets accessed all the time. So I tried to do
> | > | >>>> on
> | > | >>>> every 100000 instance request a 'System abortTransaction'
> | > | >>>> hoping it would solve something.
> | > | >>>> That falsified my theory disconnected objects can fill up
> | > | >>>> the
> | > | >>>> space. For the gem it is like a normal vm, right? When space
> | > | >>>> gets low gc chimes in. So my abortTransaction doesn't do
> | > | >>>> anything the gc wouldn't do, right?
> | > | >>>> That means the objects that fill up the space are connected
> | > | >>>> somehow. Now I'm asking myself what is the best way to get
> | > | >>>> an
> | > | >>>> overview over the instances that are in my temporary object
> | > | >>>> memory. Would that be a duty for statmon or is there another
> | > | >>>> way?
> | > | >>>>
> | > | >>>> thanks,
> | > | >>>>
> | > | >>>> Norbert
> | > | >>>>
> | > | >>>
> | > | >>
> | > | >
> | > |
> | > |
> |
> |




--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: STON and transactions

Dale Henrichs
In reply to this post by NorbertHartl
Norbert,

The fact that Fuel is not recursive makes it an attractive option for GemStone to GemStone transfer if passivate/activate (which is recursive) is not viable ... I'd be willing to chip in on a port of Fuel if you think you would use it...

Given the ability to map classes of different shapes, it sounds like there might be a chance of using Fuel for Gemstone to Pharo transfers. The OrderedCollection implementations are very different on the two platforms and would make a good first test case.

Dale

----- Original Message -----
| From: "Norbert Hartl" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Thursday, October 25, 2012 10:40:32 AM
| Subject: Re: [GS/SS Beta] STON and transactions
|
|
| Am 25.10.2012 um 17:46 schrieb Dale Henrichs <[hidden email]>:
|
| > I haven't even looked at Fuel, so it will be a wild guess indeed.
| >
| > Are you planning on using Fuel to do GemStone to GemStone transfers
| > or GemStone to Pharo transfers? GemStone 2.x to GemStone 3.x
| > transfers? or GemStone to GemStone of the same major version?
| >
| I was just thinking about transfer from gemstone to pharo. But
| gemstone to gemstone on same version is also interesting.
|
| > If you are talking GemStone to GemStone of same major version. You
| > might want to look at Object>>passivate (and look at manual).
| > Object passivation suffers from the same problem (running out of
| > stack space) that SIXX had (before I changed some of the
| > algorithms to not use recursion) and come to think of it ... If
| > Fuel uses recursion while walking the object graph, it will have
| > stack problems as well …
| >
| agreed
|
| > Cross dialect or even cross major version transfers probably can't
| > be done well with Fuel (unless it is implemented differently than
| > I imagine) so STON or SIXX are your best bets. BTW, what's wrong
| > with SIXX?
| >
| The biggest problem is that there is no pull parser available (I know
| Ken Treis did one but the xml parser changed quite a lot since
| then). Writing out a graph depends mainly on the depth traversal
| management and that there is enough space for a object to reference
| table. Reading in could be done similar if a pure streaming solution
| would be available. In Sixx the XML is very verbose producing huge
| files. Without a pull parser you read in the whole file as string,
| then convert it into a DOM which is even more bloated. Just to
| finally walk the DOM and create objects. That makes to problem worse
| at some magnitude. So STON is much more slim in string size and it
| is also smaller in code size. Making it more easy to tweak it where
| needed.
|
| Or to say it in numbers. Taking a fresh extent and reading in a
| couple of 10000s of objects with SIXX took me the last time 40
| minutes and left back an extent with a size of 25GB. Now I have
| something between 2 and 5 million objects. I didn't even think about
| to try.
|
| > With all of that said I can't see it taking more than a week or two
| > (63.458 hours). Classes are shaped differently in GemStone, but
| > you can look at the passivate/active code for the special cases
| > ... it takes a bit of thought but the recursive algorithms can be
| > unwound into loops (makes for horrid debugging) without too much
| > trouble.
| >
| That is a good tip. I'll look into the passivate stuff. You are right
| indeed. The different shapes of objects might be a big problem. I
| think Fuel has some shape changing support (as SIXX has). But if it
| comes to the internals of objects that might be a really annoying
| task to do.
|
| thanks for your detailed estimation,
|
| Norbert
|
| > Dale
| >
| > ----- Original Message -----
| > | From: "Norbert Hartl" <[hidden email]>
| > | To: "GemStone Seaside beta discussion"
| > | <[hidden email]>
| > | Sent: Thursday, October 25, 2012 8:21:24 AM
| > | Subject: Re: [GS/SS Beta] STON and transactions
| > |
| > | Dale,
| > |
| > | I don't know how good you are in "wild guessing" but I think you
| > | are
| > | an expert :). To place an estimation from the gut what would you
| > | think how much work it will be to port Fuel to GemStone. You can
| > | leave Blocks aside.
| > |
| > | Norbert
| > |
| > | Am 25.10.2012 um 16:52 schrieb Dale Henrichs
| > | <[hidden email]>:
| > |
| > | > Regarding the memory issues ... I assume that the issue is
| > | > similar
| > | > to the problem with SIXX in that you're building some very
| > | > large
| > | > objects (Strings in Streams?) in temporary memory. For SIXX the
| > | > trick was to link the root of the large temp object graph in
| > | > UserGlobals so that the temp object could be committed, thus
| > | > reducing the pressure on temp obj space ... at the end, the
| > | > temp
| > | > object was removed from UserGlobals so that it could be garbage
| > | > collected ...
| > | >
| > | > Regarding the test failures in GemStone ... I will have to take
| > | > a
| > | > look. At the time of the port, the tests were passing, but at
| > | > the
| > | > moment I don't recall which versions of GemStone I tested STON
| > | > against ...
| > | >
| > | > I have been using STON in 3.1.0.1, but I have been using it to
| > | > serialize limited object graphs (I am stopping after n-levels),
| > | > so
| > | > I haven't tried to serialize a big object graph from GemStone
| > | > and
| > | > haven't run into any errors using STON.
| > | >
| > | > Regarding the error related to ScaledDecimal ... I'd like to
| > | > know
| > | > what the error is. I can't imagine what kind of error that
| > | > ScaledDecimal would be throwing from the following code:
| > | >
| > | >  STON fromStream: (String streamContents: [:stream|
| > | >        (Dictionary new
| > | >                at: Object new put: Object new;
| > | >                yourself) stonOn: (STONWriter on: stream)])
| > | >                readStream
| > | >
| > | > In a somewhat related are, I have found that there are bugs
| > | > related
| > | > to using #basicSize, #basicAt: and #basicAtPut: for a handful
| > | > of
| > | > classes. I've identified a problems in GsNMethod and ClassSet
| > | > and
| > | > this particular problem could affect the operation of STON.
| > | >
| > | > Still not sure about the ScaledDecimal problem...
| > | >
| > | > Dale
| > | > ----- Original Message -----
| > | > | From: "Norbert Hartl" <[hidden email]>
| > | > | To: "GemStone Seaside beta discussion"
| > | > | <[hidden email]>
| > | > | Sent: Thursday, October 25, 2012 4:53:45 AM
| > | > | Subject: Re: [GS/SS Beta] STON and transactions
| > | > |
| > | > | And finally the problem I've found seems to be a problem in
| > | > | the
| > | > | design of STON (or just me misunderstanding its purpose).
| > | > | Anyway
| > | > | the
| > | > | following code snippet raises the error
| > | > |
| > | > | STON fromStream: (String streamContents: [:stream|
| > | > | (Dictionary new
| > | > | at: Object new put: Object new;
| > | > | yourself) stonOn: (STONWriter on: stream)]) readStream
| > | > |
| > | > | Norbert
| > | > |
| > | > | Am 25.10.2012 um 13:37 schrieb Norbert Hartl
| > | > | <[hidden email]>:
| > | > |
| > | > | > This is some sort of monolog :) I just forgot to add to the
| > | > | > last
| > | > | > mail that reading back the file in GemStone fails with the
| > | > | > same
| > | > | > error as in pharo.
| > | > | >
| > | > | > Norbert
| > | > | >
| > | > | > Am 25.10.2012 um 13:35 schrieb Norbert Hartl
| > | > | > <[hidden email]>:
| > | > | >
| > | > | >> Unfortunately only the export worked "somehow". In pharo I
| > | > | >> cannot
| > | > | >> read in the STON file produced by gemstone. Tests are
| > | > | >> green on
| > | > | >> pharo. On gemstone there are two failing tests that have
| > | > | >> to do
| > | > | >> with ScaledDecimal.  You said you are working with it and
| > | > | >> you
| > | > | >> did
| > | > | >> not encounter problems?
| > | > | >>
| > | > | >> Norbert
| > | > | >>
| > | > | >> Am 25.10.2012 um 11:11 schrieb Norbert Hartl
| > | > | >> <[hidden email]>:
| > | > | >>
| > | > | >>> Haha,
| > | > | >>>
| > | > | >>> often after I write a mail the situation changes somehow
| > | > | >>> :)
| > | > | >>> While
| > | > | >>> fighting in the mist I tried doing a commitTransaction
| > | > | >>> instead
| > | > | >>> of abortTransaction and this works. At least I can export
| > | > | >>> my
| > | > | >>> object graph now. If there is a good explanation I'm glad
| > | > | >>> to
| > | > | >>> hear it :)
| > | > | >>>
| > | > | >>> Norbert
| > | > | >>>
| > | > | >>> Am 25.10.2012 um 11:03 schrieb Norbert Hartl
| > | > | >>> <[hidden email]>:
| > | > | >>>
| > | > | >>>> Dale,
| > | > | >>>>
| > | > | >>>> I loaded the STON code with your snippet and this
| > | > | >>>> worked.
| > | > | >>>> But it
| > | > | >>>> didn't solve my problem. I cannot export my object graph
| > | > | >>>> with
| > | > | >>>> STON. I always get temporary object memory full. I gave
| > | > | >>>> the
| > | > | >>>> gem
| > | > | >>>> 200MB already and still no success. I can give it even
| > | > | >>>> more
| > | > | >>>> but
| > | > | >>>> that would postpone my problem only a couple of weeks. I
| > | > | >>>> like
| > | > | >>>> to find the reason why it doesn't work.
| > | > | >>>>
| > | > | >>>> Reading the code I only suspected the STONWriterXXX
| > | > | >>>> classes
| > | > | >>>> to
| > | > | >>>> be the culprit because they are created all the time. So
| > | > | >>>> I
| > | > | >>>> rewrote the STON code to use an object pool for the
| > | > | >>>> writer
| > | > | >>>> classes but still no success. Having the pool I had one
| > | > | >>>> central
| > | > | >>>> instance that gets accessed all the time. So I tried to
| > | > | >>>> do
| > | > | >>>> on
| > | > | >>>> every 100000 instance request a 'System
| > | > | >>>> abortTransaction'
| > | > | >>>> hoping it would solve something.
| > | > | >>>> That falsified my theory disconnected objects can fill
| > | > | >>>> up
| > | > | >>>> the
| > | > | >>>> space. For the gem it is like a normal vm, right? When
| > | > | >>>> space
| > | > | >>>> gets low gc chimes in. So my abortTransaction doesn't do
| > | > | >>>> anything the gc wouldn't do, right?
| > | > | >>>> That means the objects that fill up the space are
| > | > | >>>> connected
| > | > | >>>> somehow. Now I'm asking myself what is the best way to
| > | > | >>>> get
| > | > | >>>> an
| > | > | >>>> overview over the instances that are in my temporary
| > | > | >>>> object
| > | > | >>>> memory. Would that be a duty for statmon or is there
| > | > | >>>> another
| > | > | >>>> way?
| > | > | >>>>
| > | > | >>>> thanks,
| > | > | >>>>
| > | > | >>>> Norbert
| > | > | >>>>
| > | > | >>>
| > | > | >>
| > | > | >
| > | > |
| > | > |
| > |
| > |
|
|
Reply | Threaded
Open this post in threaded view
|

Re: STON and transactions

Mariano Martinez Peck


On Fri, Oct 26, 2012 at 12:52 AM, Dale Henrichs <[hidden email]> wrote:
Norbert,

The fact that Fuel is not recursive makes it an attractive option for GemStone to GemStone transfer if passivate/activate (which is recursive) is not viable ... I'd be willing to chip in on a port of Fuel if you think you would use it...

I think that porting Fuel to GemStone and do GemStone to GemStone transfer, should work without much effort. I cannot do it myself right now, but I am willing to help. 
 

Given the ability to map classes of different shapes,

Not sure what you mean. Is what I said about class reshape? if true, I think there was a misunderstood. What I mean is that Fuel supports the case where when you materialize something, the classes in the system have changed since the time you serialized. Typical cases are: add/remove/rename/reorder instance variables, rename classes, etc. 
 
it sounds like there might be a chance of using Fuel for Gemstone to Pharo transfers. The OrderedCollection implementations are very different on the two platforms and would make a good first test case.

Supporting GemStone-Pharo transfer looks more complex. At least, Fuel was never built with this in mind, I mean, we never tried to find portable representation of the objects. Say Float, we just encode Floats as they are in Pharo. We do not delegate to a compatibility layer (as other serializers like SRP do) nor we do any type of conversion. I don't know the concrete differences with the "primitive classes" (Floats, Integers, ByteArray, ByteStrings, etc) of GemStone and Pharo. Maybe there are not many and maybe a little layer would help. 
Anyway, what I want to say is that GemStone-Pharo transfer will be more work than GemStone to GemStone. And of course, the latter is needed for the first :)

 

Dale

----- Original Message -----
| From: "Norbert Hartl" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Thursday, October 25, 2012 10:40:32 AM
| Subject: Re: [GS/SS Beta] STON and transactions
|
|
| Am 25.10.2012 um 17:46 schrieb Dale Henrichs <[hidden email]>:
|
| > I haven't even looked at Fuel, so it will be a wild guess indeed.
| >
| > Are you planning on using Fuel to do GemStone to GemStone transfers
| > or GemStone to Pharo transfers? GemStone 2.x to GemStone 3.x
| > transfers? or GemStone to GemStone of the same major version?
| >
| I was just thinking about transfer from gemstone to pharo. But
| gemstone to gemstone on same version is also interesting.
|
| > If you are talking GemStone to GemStone of same major version. You
| > might want to look at Object>>passivate (and look at manual).
| > Object passivation suffers from the same problem (running out of
| > stack space) that SIXX had (before I changed some of the
| > algorithms to not use recursion) and come to think of it ... If
| > Fuel uses recursion while walking the object graph, it will have
| > stack problems as well …
| >
| agreed
|
| > Cross dialect or even cross major version transfers probably can't
| > be done well with Fuel (unless it is implemented differently than
| > I imagine) so STON or SIXX are your best bets. BTW, what's wrong
| > with SIXX?
| >
| The biggest problem is that there is no pull parser available (I know
| Ken Treis did one but the xml parser changed quite a lot since
| then). Writing out a graph depends mainly on the depth traversal
| management and that there is enough space for a object to reference
| table. Reading in could be done similar if a pure streaming solution
| would be available. In Sixx the XML is very verbose producing huge
| files. Without a pull parser you read in the whole file as string,
| then convert it into a DOM which is even more bloated. Just to
| finally walk the DOM and create objects. That makes to problem worse
| at some magnitude. So STON is much more slim in string size and it
| is also smaller in code size. Making it more easy to tweak it where
| needed.
|
| Or to say it in numbers. Taking a fresh extent and reading in a
| couple of 10000s of objects with SIXX took me the last time 40
| minutes and left back an extent with a size of 25GB. Now I have
| something between 2 and 5 million objects. I didn't even think about
| to try.
|
| > With all of that said I can't see it taking more than a week or two
| > (63.458 hours). Classes are shaped differently in GemStone, but
| > you can look at the passivate/active code for the special cases
| > ... it takes a bit of thought but the recursive algorithms can be
| > unwound into loops (makes for horrid debugging) without too much
| > trouble.
| >
| That is a good tip. I'll look into the passivate stuff. You are right
| indeed. The different shapes of objects might be a big problem. I
| think Fuel has some shape changing support (as SIXX has). But if it
| comes to the internals of objects that might be a really annoying
| task to do.
|
| thanks for your detailed estimation,
|
| Norbert
|
| > Dale
| >
| > ----- Original Message -----
| > | From: "Norbert Hartl" <[hidden email]>
| > | To: "GemStone Seaside beta discussion"
| > | <[hidden email]>
| > | Sent: Thursday, October 25, 2012 8:21:24 AM
| > | Subject: Re: [GS/SS Beta] STON and transactions
| > |
| > | Dale,
| > |
| > | I don't know how good you are in "wild guessing" but I think you
| > | are
| > | an expert :). To place an estimation from the gut what would you
| > | think how much work it will be to port Fuel to GemStone. You can
| > | leave Blocks aside.
| > |
| > | Norbert
| > |
| > | Am 25.10.2012 um 16:52 schrieb Dale Henrichs
| > | <[hidden email]>:
| > |
| > | > Regarding the memory issues ... I assume that the issue is
| > | > similar
| > | > to the problem with SIXX in that you're building some very
| > | > large
| > | > objects (Strings in Streams?) in temporary memory. For SIXX the
| > | > trick was to link the root of the large temp object graph in
| > | > UserGlobals so that the temp object could be committed, thus
| > | > reducing the pressure on temp obj space ... at the end, the
| > | > temp
| > | > object was removed from UserGlobals so that it could be garbage
| > | > collected ...
| > | >
| > | > Regarding the test failures in GemStone ... I will have to take
| > | > a
| > | > look. At the time of the port, the tests were passing, but at
| > | > the
| > | > moment I don't recall which versions of GemStone I tested STON
| > | > against ...
| > | >
| > | > I have been using STON in <a href="tel:3.1.0.1" value="+333101">3.1.0.1, but I have been using it to
| > | > serialize limited object graphs (I am stopping after n-levels),
| > | > so
| > | > I haven't tried to serialize a big object graph from GemStone
| > | > and
| > | > haven't run into any errors using STON.
| > | >
| > | > Regarding the error related to ScaledDecimal ... I'd like to
| > | > know
| > | > what the error is. I can't imagine what kind of error that
| > | > ScaledDecimal would be throwing from the following code:
| > | >
| > | >  STON fromStream: (String streamContents: [:stream|
| > | >        (Dictionary new
| > | >                at: Object new put: Object new;
| > | >                yourself) stonOn: (STONWriter on: stream)])
| > | >                readStream
| > | >
| > | > In a somewhat related are, I have found that there are bugs
| > | > related
| > | > to using #basicSize, #basicAt: and #basicAtPut: for a handful
| > | > of
| > | > classes. I've identified a problems in GsNMethod and ClassSet
| > | > and
| > | > this particular problem could affect the operation of STON.
| > | >
| > | > Still not sure about the ScaledDecimal problem...
| > | >
| > | > Dale
| > | > ----- Original Message -----
| > | > | From: "Norbert Hartl" <[hidden email]>
| > | > | To: "GemStone Seaside beta discussion"
| > | > | <[hidden email]>
| > | > | Sent: Thursday, October 25, 2012 4:53:45 AM
| > | > | Subject: Re: [GS/SS Beta] STON and transactions
| > | > |
| > | > | And finally the problem I've found seems to be a problem in
| > | > | the
| > | > | design of STON (or just me misunderstanding its purpose).
| > | > | Anyway
| > | > | the
| > | > | following code snippet raises the error
| > | > |
| > | > | STON fromStream: (String streamContents: [:stream|
| > | > |       (Dictionary new
| > | > |               at: Object new put: Object new;
| > | > |               yourself) stonOn: (STONWriter on: stream)]) readStream
| > | > |
| > | > | Norbert
| > | > |
| > | > | Am 25.10.2012 um 13:37 schrieb Norbert Hartl
| > | > | <[hidden email]>:
| > | > |
| > | > | > This is some sort of monolog :) I just forgot to add to the
| > | > | > last
| > | > | > mail that reading back the file in GemStone fails with the
| > | > | > same
| > | > | > error as in pharo.
| > | > | >
| > | > | > Norbert
| > | > | >
| > | > | > Am 25.10.2012 um 13:35 schrieb Norbert Hartl
| > | > | > <[hidden email]>:
| > | > | >
| > | > | >> Unfortunately only the export worked "somehow". In pharo I
| > | > | >> cannot
| > | > | >> read in the STON file produced by gemstone. Tests are
| > | > | >> green on
| > | > | >> pharo. On gemstone there are two failing tests that have
| > | > | >> to do
| > | > | >> with ScaledDecimal.  You said you are working with it and
| > | > | >> you
| > | > | >> did
| > | > | >> not encounter problems?
| > | > | >>
| > | > | >> Norbert
| > | > | >>
| > | > | >> Am 25.10.2012 um 11:11 schrieb Norbert Hartl
| > | > | >> <[hidden email]>:
| > | > | >>
| > | > | >>> Haha,
| > | > | >>>
| > | > | >>> often after I write a mail the situation changes somehow
| > | > | >>> :)
| > | > | >>> While
| > | > | >>> fighting in the mist I tried doing a commitTransaction
| > | > | >>> instead
| > | > | >>> of abortTransaction and this works. At least I can export
| > | > | >>> my
| > | > | >>> object graph now. If there is a good explanation I'm glad
| > | > | >>> to
| > | > | >>> hear it :)
| > | > | >>>
| > | > | >>> Norbert
| > | > | >>>
| > | > | >>> Am 25.10.2012 um 11:03 schrieb Norbert Hartl
| > | > | >>> <[hidden email]>:
| > | > | >>>
| > | > | >>>> Dale,
| > | > | >>>>
| > | > | >>>> I loaded the STON code with your snippet and this
| > | > | >>>> worked.
| > | > | >>>> But it
| > | > | >>>> didn't solve my problem. I cannot export my object graph
| > | > | >>>> with
| > | > | >>>> STON. I always get temporary object memory full. I gave
| > | > | >>>> the
| > | > | >>>> gem
| > | > | >>>> 200MB already and still no success. I can give it even
| > | > | >>>> more
| > | > | >>>> but
| > | > | >>>> that would postpone my problem only a couple of weeks. I
| > | > | >>>> like
| > | > | >>>> to find the reason why it doesn't work.
| > | > | >>>>
| > | > | >>>> Reading the code I only suspected the STONWriterXXX
| > | > | >>>> classes
| > | > | >>>> to
| > | > | >>>> be the culprit because they are created all the time. So
| > | > | >>>> I
| > | > | >>>> rewrote the STON code to use an object pool for the
| > | > | >>>> writer
| > | > | >>>> classes but still no success. Having the pool I had one
| > | > | >>>> central
| > | > | >>>> instance that gets accessed all the time. So I tried to
| > | > | >>>> do
| > | > | >>>> on
| > | > | >>>> every 100000 instance request a 'System
| > | > | >>>> abortTransaction'
| > | > | >>>> hoping it would solve something.
| > | > | >>>> That falsified my theory disconnected objects can fill
| > | > | >>>> up
| > | > | >>>> the
| > | > | >>>> space. For the gem it is like a normal vm, right? When
| > | > | >>>> space
| > | > | >>>> gets low gc chimes in. So my abortTransaction doesn't do
| > | > | >>>> anything the gc wouldn't do, right?
| > | > | >>>> That means the objects that fill up the space are
| > | > | >>>> connected
| > | > | >>>> somehow. Now I'm asking myself what is the best way to
| > | > | >>>> get
| > | > | >>>> an
| > | > | >>>> overview over the instances that are in my temporary
| > | > | >>>> object
| > | > | >>>> memory. Would that be a duty for statmon or is there
| > | > | >>>> another
| > | > | >>>> way?
| > | > | >>>>
| > | > | >>>> thanks,
| > | > | >>>>
| > | > | >>>> Norbert
| > | > | >>>>
| > | > | >>>
| > | > | >>
| > | > | >
| > | > |
| > | > |
| > |
| > |
|
|



--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: STON and transactions

Dale Henrichs
Mariano,

Agree with everything you say ... my ignorance of Fuel 'fueled' my wacky idea that if you could handle class shape changes you could handle dialect to dialect changes too:)

As for the port, it' son my list of things to do, but if Norbert or anyone finds that they would use Fuel if it were ported, then it's worth trying to make time to get the port done ...

Dale

----- Original Message -----
| From: "Mariano Martinez Peck" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Friday, October 26, 2012 1:18:40 AM
| Subject: Re: [GS/SS Beta] STON and transactions
|
|
|
|
| On Fri, Oct 26, 2012 at 12:52 AM, Dale Henrichs < [hidden email]
| > wrote:
|
|
| Norbert,
|
| The fact that Fuel is not recursive makes it an attractive option for
| GemStone to GemStone transfer if passivate/activate (which is
| recursive) is not viable ... I'd be willing to chip in on a port of
| Fuel if you think you would use it...
|
|
|
| I think that porting Fuel to GemStone and do GemStone to GemStone
| transfer, should work without much effort. I cannot do it myself
| right now, but I am willing to help.
|
|
|
| Given the ability to map classes of different shapes,
|
|
| Not sure what you mean. Is what I said about class reshape? if true,
| I think there was a misunderstood. What I mean is that Fuel supports
| the case where when you materialize something, the classes in the
| system have changed since the time you serialized. Typical cases
| are: add/remove/rename/reorder instance variables, rename classes,
| etc.
|
|
| it sounds like there might be a chance of using Fuel for Gemstone to
| Pharo transfers. The OrderedCollection implementations are very
| different on the two platforms and would make a good first test
| case.
|
|
|
| Supporting GemStone-Pharo transfer looks more complex. At least, Fuel
| was never built with this in mind, I mean, we never tried to find
| portable representation of the objects. Say Float, we just encode
| Floats as they are in Pharo. We do not delegate to a compatibility
| layer (as other serializers like SRP do) nor we do any type of
| conversion. I don't know the concrete differences with the
| "primitive classes" (Floats, Integers, ByteArray, ByteStrings, etc)
| of GemStone and Pharo. Maybe there are not many and maybe a little
| layer would help.
| Anyway, what I want to say is that GemStone-Pharo transfer will be
| more work than GemStone to GemStone. And of course, the latter is
| needed for the first :)
|
|
|
|
|
|
| Dale
|
| ----- Original Message -----
| | From: "Norbert Hartl" < [hidden email] >
| | To: "GemStone Seaside beta discussion" < [hidden email]
| | >
|
|
| | Sent: Thursday, October 25, 2012 10:40:32 AM
| | Subject: Re: [GS/SS Beta] STON and transactions
| |
| |
| | Am 25.10.2012 um 17:46 schrieb Dale Henrichs < [hidden email]
| | >:
| |
| | > I haven't even looked at Fuel, so it will be a wild guess indeed.
| | >
| | > Are you planning on using Fuel to do GemStone to GemStone
| | > transfers
| | > or GemStone to Pharo transfers? GemStone 2.x to GemStone 3.x
| | > transfers? or GemStone to GemStone of the same major version?
| | >
| | I was just thinking about transfer from gemstone to pharo. But
| | gemstone to gemstone on same version is also interesting.
| |
| | > If you are talking GemStone to GemStone of same major version.
| | > You
| | > might want to look at Object>>passivate (and look at manual).
| | > Object passivation suffers from the same problem (running out of
| | > stack space) that SIXX had (before I changed some of the
| | > algorithms to not use recursion) and come to think of it ... If
| | > Fuel uses recursion while walking the object graph, it will have
| | > stack problems as well …
| | >
| | agreed
| |
| | > Cross dialect or even cross major version transfers probably
| | > can't
| | > be done well with Fuel (unless it is implemented differently than
| | > I imagine) so STON or SIXX are your best bets. BTW, what's wrong
| | > with SIXX?
| | >
| | The biggest problem is that there is no pull parser available (I
| | know
| | Ken Treis did one but the xml parser changed quite a lot since
| | then). Writing out a graph depends mainly on the depth traversal
| | management and that there is enough space for a object to reference
| | table. Reading in could be done similar if a pure streaming
| | solution
| | would be available. In Sixx the XML is very verbose producing huge
| | files. Without a pull parser you read in the whole file as string,
| | then convert it into a DOM which is even more bloated. Just to
| | finally walk the DOM and create objects. That makes to problem
| | worse
| | at some magnitude. So STON is much more slim in string size and it
| | is also smaller in code size. Making it more easy to tweak it where
| | needed.
| |
| | Or to say it in numbers. Taking a fresh extent and reading in a
| | couple of 10000s of objects with SIXX took me the last time 40
| | minutes and left back an extent with a size of 25GB. Now I have
| | something between 2 and 5 million objects. I didn't even think
| | about
| | to try.
| |
| | > With all of that said I can't see it taking more than a week or
| | > two
| | > (63.458 hours). Classes are shaped differently in GemStone, but
| | > you can look at the passivate/active code for the special cases
| | > ... it takes a bit of thought but the recursive algorithms can be
| | > unwound into loops (makes for horrid debugging) without too much
| | > trouble.
| | >
| | That is a good tip. I'll look into the passivate stuff. You are
| | right
| | indeed. The different shapes of objects might be a big problem. I
| | think Fuel has some shape changing support (as SIXX has). But if it
| | comes to the internals of objects that might be a really annoying
| | task to do.
| |
| | thanks for your detailed estimation,
| |
| | Norbert
| |
| | > Dale
| | >
| | > ----- Original Message -----
| | > | From: "Norbert Hartl" < [hidden email] >
| | > | To: "GemStone Seaside beta discussion"
| | > | < [hidden email] >
| | > | Sent: Thursday, October 25, 2012 8:21:24 AM
| | > | Subject: Re: [GS/SS Beta] STON and transactions
| | > |
| | > | Dale,
| | > |
| | > | I don't know how good you are in "wild guessing" but I think
| | > | you
| | > | are
| | > | an expert :). To place an estimation from the gut what would
| | > | you
| | > | think how much work it will be to port Fuel to GemStone. You
| | > | can
| | > | leave Blocks aside.
| | > |
| | > | Norbert
| | > |
| | > | Am 25.10.2012 um 16:52 schrieb Dale Henrichs
| | > | < [hidden email] >:
| | > |
| | > | > Regarding the memory issues ... I assume that the issue is
| | > | > similar
| | > | > to the problem with SIXX in that you're building some very
| | > | > large
| | > | > objects (Strings in Streams?) in temporary memory. For SIXX
| | > | > the
| | > | > trick was to link the root of the large temp object graph in
| | > | > UserGlobals so that the temp object could be committed, thus
| | > | > reducing the pressure on temp obj space ... at the end, the
| | > | > temp
| | > | > object was removed from UserGlobals so that it could be
| | > | > garbage
| | > | > collected ...
| | > | >
| | > | > Regarding the test failures in GemStone ... I will have to
| | > | > take
| | > | > a
| | > | > look. At the time of the port, the tests were passing, but at
| | > | > the
| | > | > moment I don't recall which versions of GemStone I tested
| | > | > STON
| | > | > against ...
| | > | >
| | > | > I have been using STON in 3.1.0.1 , but I have been using it
| | > | > to
| | > | > serialize limited object graphs (I am stopping after
| | > | > n-levels),
| | > | > so
| | > | > I haven't tried to serialize a big object graph from GemStone
| | > | > and
| | > | > haven't run into any errors using STON.
| | > | >
| | > | > Regarding the error related to ScaledDecimal ... I'd like to
| | > | > know
| | > | > what the error is. I can't imagine what kind of error that
| | > | > ScaledDecimal would be throwing from the following code:
| | > | >
| | > | > STON fromStream: (String streamContents: [:stream|
| | > | > (Dictionary new
| | > | > at: Object new put: Object new;
| | > | > yourself) stonOn: (STONWriter on: stream)])
| | > | > readStream
| | > | >
| | > | > In a somewhat related are, I have found that there are bugs
| | > | > related
| | > | > to using #basicSize, #basicAt: and #basicAtPut: for a handful
| | > | > of
| | > | > classes. I've identified a problems in GsNMethod and ClassSet
| | > | > and
| | > | > this particular problem could affect the operation of STON.
| | > | >
| | > | > Still not sure about the ScaledDecimal problem...
| | > | >
| | > | > Dale
| | > | > ----- Original Message -----
| | > | > | From: "Norbert Hartl" < [hidden email] >
| | > | > | To: "GemStone Seaside beta discussion"
| | > | > | < [hidden email] >
| | > | > | Sent: Thursday, October 25, 2012 4:53:45 AM
| | > | > | Subject: Re: [GS/SS Beta] STON and transactions
| | > | > |
| | > | > | And finally the problem I've found seems to be a problem in
| | > | > | the
| | > | > | design of STON (or just me misunderstanding its purpose).
| | > | > | Anyway
| | > | > | the
| | > | > | following code snippet raises the error
| | > | > |
| | > | > | STON fromStream: (String streamContents: [:stream|
| | > | > | (Dictionary new
| | > | > | at: Object new put: Object new;
| | > | > | yourself) stonOn: (STONWriter on: stream)]) readStream
| | > | > |
| | > | > | Norbert
| | > | > |
| | > | > | Am 25.10.2012 um 13:37 schrieb Norbert Hartl
| | > | > | < [hidden email] >:
| | > | > |
| | > | > | > This is some sort of monolog :) I just forgot to add to
| | > | > | > the
| | > | > | > last
| | > | > | > mail that reading back the file in GemStone fails with
| | > | > | > the
| | > | > | > same
| | > | > | > error as in pharo.
| | > | > | >
| | > | > | > Norbert
| | > | > | >
| | > | > | > Am 25.10.2012 um 13:35 schrieb Norbert Hartl
| | > | > | > < [hidden email] >:
| | > | > | >
| | > | > | >> Unfortunately only the export worked "somehow". In pharo
| | > | > | >> I
| | > | > | >> cannot
| | > | > | >> read in the STON file produced by gemstone. Tests are
| | > | > | >> green on
| | > | > | >> pharo. On gemstone there are two failing tests that have
| | > | > | >> to do
| | > | > | >> with ScaledDecimal. You said you are working with it and
| | > | > | >> you
| | > | > | >> did
| | > | > | >> not encounter problems?
| | > | > | >>
| | > | > | >> Norbert
| | > | > | >>
| | > | > | >> Am 25.10.2012 um 11:11 schrieb Norbert Hartl
| | > | > | >> < [hidden email] >:
| | > | > | >>
| | > | > | >>> Haha,
| | > | > | >>>
| | > | > | >>> often after I write a mail the situation changes
| | > | > | >>> somehow
| | > | > | >>> :)
| | > | > | >>> While
| | > | > | >>> fighting in the mist I tried doing a commitTransaction
| | > | > | >>> instead
| | > | > | >>> of abortTransaction and this works. At least I can
| | > | > | >>> export
| | > | > | >>> my
| | > | > | >>> object graph now. If there is a good explanation I'm
| | > | > | >>> glad
| | > | > | >>> to
| | > | > | >>> hear it :)
| | > | > | >>>
| | > | > | >>> Norbert
| | > | > | >>>
| | > | > | >>> Am 25.10.2012 um 11:03 schrieb Norbert Hartl
| | > | > | >>> < [hidden email] >:
| | > | > | >>>
| | > | > | >>>> Dale,
| | > | > | >>>>
| | > | > | >>>> I loaded the STON code with your snippet and this
| | > | > | >>>> worked.
| | > | > | >>>> But it
| | > | > | >>>> didn't solve my problem. I cannot export my object
| | > | > | >>>> graph
| | > | > | >>>> with
| | > | > | >>>> STON. I always get temporary object memory full. I
| | > | > | >>>> gave
| | > | > | >>>> the
| | > | > | >>>> gem
| | > | > | >>>> 200MB already and still no success. I can give it even
| | > | > | >>>> more
| | > | > | >>>> but
| | > | > | >>>> that would postpone my problem only a couple of weeks.
| | > | > | >>>> I
| | > | > | >>>> like
| | > | > | >>>> to find the reason why it doesn't work.
| | > | > | >>>>
| | > | > | >>>> Reading the code I only suspected the STONWriterXXX
| | > | > | >>>> classes
| | > | > | >>>> to
| | > | > | >>>> be the culprit because they are created all the time.
| | > | > | >>>> So
| | > | > | >>>> I
| | > | > | >>>> rewrote the STON code to use an object pool for the
| | > | > | >>>> writer
| | > | > | >>>> classes but still no success. Having the pool I had
| | > | > | >>>> one
| | > | > | >>>> central
| | > | > | >>>> instance that gets accessed all the time. So I tried
| | > | > | >>>> to
| | > | > | >>>> do
| | > | > | >>>> on
| | > | > | >>>> every 100000 instance request a 'System
| | > | > | >>>> abortTransaction'
| | > | > | >>>> hoping it would solve something.
| | > | > | >>>> That falsified my theory disconnected objects can fill
| | > | > | >>>> up
| | > | > | >>>> the
| | > | > | >>>> space. For the gem it is like a normal vm, right? When
| | > | > | >>>> space
| | > | > | >>>> gets low gc chimes in. So my abortTransaction doesn't
| | > | > | >>>> do
| | > | > | >>>> anything the gc wouldn't do, right?
| | > | > | >>>> That means the objects that fill up the space are
| | > | > | >>>> connected
| | > | > | >>>> somehow. Now I'm asking myself what is the best way to
| | > | > | >>>> get
| | > | > | >>>> an
| | > | > | >>>> overview over the instances that are in my temporary
| | > | > | >>>> object
| | > | > | >>>> memory. Would that be a duty for statmon or is there
| | > | > | >>>> another
| | > | > | >>>> way?
| | > | > | >>>>
| | > | > | >>>> thanks,
| | > | > | >>>>
| | > | > | >>>> Norbert
| | > | > | >>>>
| | > | > | >>>
| | > | > | >>
| | > | > | >
| | > | > |
| | > | > |
| | > |
| | > |
| |
| |
|
|
|
|
| --
| Mariano
| http://marianopeck.wordpress.com
|
|
Reply | Threaded
Open this post in threaded view
|

Re: STON and transactions

NorbertHartl

Am 26.10.2012 um 19:31 schrieb Dale Henrichs <[hidden email]>:

> Mariano,
>
> Agree with everything you say ... my ignorance of Fuel 'fueled' my wacky idea that if you could handle class shape changes you could handle dialect to dialect changes too:)
>
That is what I was thinking, too.

> As for the port, it' son my list of things to do, but if Norbert or anyone finds that they would use Fuel if it were ported, then it's worth trying to make time to get the port done …
>
I think it is a good idea to have fuel ported to gemstone. I don't know if it is urgent enough to distract you from your current work. I'm still trying to make the STON part work. For gemstone I just raised the SPC size and the gem temp object memory and I can save it. Now I'm trying to import it again in pharo. That might work but a new problem is close. The data I'm speaking of is getting bigger and bigger and so it will blow up a pharo image anyway.
So I have two choice:

- not exporting everything. If I can put a scope on the graph for exportation that could be a way to avoid exporting all of it. If this would work neither the gemstone export nor the pharo import will be a big problem.
- store data external. I could store files or something similar. This way it will be slow but I could have at least access to the data from pharo in order to do analyzes.

So you see my actual problem shifted slightly away from the serialization/materialization.

Norbert

> Dale
>
> ----- Original Message -----
> | From: "Mariano Martinez Peck" <[hidden email]>
> | To: "GemStone Seaside beta discussion" <[hidden email]>
> | Sent: Friday, October 26, 2012 1:18:40 AM
> | Subject: Re: [GS/SS Beta] STON and transactions
> |
> |
> |
> |
> | On Fri, Oct 26, 2012 at 12:52 AM, Dale Henrichs < [hidden email]
> | > wrote:
> |
> |
> | Norbert,
> |
> | The fact that Fuel is not recursive makes it an attractive option for
> | GemStone to GemStone transfer if passivate/activate (which is
> | recursive) is not viable ... I'd be willing to chip in on a port of
> | Fuel if you think you would use it...
> |
> |
> |
> | I think that porting Fuel to GemStone and do GemStone to GemStone
> | transfer, should work without much effort. I cannot do it myself
> | right now, but I am willing to help.
> |
> |
> |
> | Given the ability to map classes of different shapes,
> |
> |
> | Not sure what you mean. Is what I said about class reshape? if true,
> | I think there was a misunderstood. What I mean is that Fuel supports
> | the case where when you materialize something, the classes in the
> | system have changed since the time you serialized. Typical cases
> | are: add/remove/rename/reorder instance variables, rename classes,
> | etc.
> |
> |
> | it sounds like there might be a chance of using Fuel for Gemstone to
> | Pharo transfers. The OrderedCollection implementations are very
> | different on the two platforms and would make a good first test
> | case.
> |
> |
> |
> | Supporting GemStone-Pharo transfer looks more complex. At least, Fuel
> | was never built with this in mind, I mean, we never tried to find
> | portable representation of the objects. Say Float, we just encode
> | Floats as they are in Pharo. We do not delegate to a compatibility
> | layer (as other serializers like SRP do) nor we do any type of
> | conversion. I don't know the concrete differences with the
> | "primitive classes" (Floats, Integers, ByteArray, ByteStrings, etc)
> | of GemStone and Pharo. Maybe there are not many and maybe a little
> | layer would help.
> | Anyway, what I want to say is that GemStone-Pharo transfer will be
> | more work than GemStone to GemStone. And of course, the latter is
> | needed for the first :)
> |
> |
> |
> |
> |
> |
> | Dale
> |
> | ----- Original Message -----
> | | From: "Norbert Hartl" < [hidden email] >
> | | To: "GemStone Seaside beta discussion" < [hidden email]
> | | >
> |
> |
> | | Sent: Thursday, October 25, 2012 10:40:32 AM
> | | Subject: Re: [GS/SS Beta] STON and transactions
> | |
> | |
> | | Am 25.10.2012 um 17:46 schrieb Dale Henrichs < [hidden email]
> | | >:
> | |
> | | > I haven't even looked at Fuel, so it will be a wild guess indeed.
> | | >
> | | > Are you planning on using Fuel to do GemStone to GemStone
> | | > transfers
> | | > or GemStone to Pharo transfers? GemStone 2.x to GemStone 3.x
> | | > transfers? or GemStone to GemStone of the same major version?
> | | >
> | | I was just thinking about transfer from gemstone to pharo. But
> | | gemstone to gemstone on same version is also interesting.
> | |
> | | > If you are talking GemStone to GemStone of same major version.
> | | > You
> | | > might want to look at Object>>passivate (and look at manual).
> | | > Object passivation suffers from the same problem (running out of
> | | > stack space) that SIXX had (before I changed some of the
> | | > algorithms to not use recursion) and come to think of it ... If
> | | > Fuel uses recursion while walking the object graph, it will have
> | | > stack problems as well …
> | | >
> | | agreed
> | |
> | | > Cross dialect or even cross major version transfers probably
> | | > can't
> | | > be done well with Fuel (unless it is implemented differently than
> | | > I imagine) so STON or SIXX are your best bets. BTW, what's wrong
> | | > with SIXX?
> | | >
> | | The biggest problem is that there is no pull parser available (I
> | | know
> | | Ken Treis did one but the xml parser changed quite a lot since
> | | then). Writing out a graph depends mainly on the depth traversal
> | | management and that there is enough space for a object to reference
> | | table. Reading in could be done similar if a pure streaming
> | | solution
> | | would be available. In Sixx the XML is very verbose producing huge
> | | files. Without a pull parser you read in the whole file as string,
> | | then convert it into a DOM which is even more bloated. Just to
> | | finally walk the DOM and create objects. That makes to problem
> | | worse
> | | at some magnitude. So STON is much more slim in string size and it
> | | is also smaller in code size. Making it more easy to tweak it where
> | | needed.
> | |
> | | Or to say it in numbers. Taking a fresh extent and reading in a
> | | couple of 10000s of objects with SIXX took me the last time 40
> | | minutes and left back an extent with a size of 25GB. Now I have
> | | something between 2 and 5 million objects. I didn't even think
> | | about
> | | to try.
> | |
> | | > With all of that said I can't see it taking more than a week or
> | | > two
> | | > (63.458 hours). Classes are shaped differently in GemStone, but
> | | > you can look at the passivate/active code for the special cases
> | | > ... it takes a bit of thought but the recursive algorithms can be
> | | > unwound into loops (makes for horrid debugging) without too much
> | | > trouble.
> | | >
> | | That is a good tip. I'll look into the passivate stuff. You are
> | | right
> | | indeed. The different shapes of objects might be a big problem. I
> | | think Fuel has some shape changing support (as SIXX has). But if it
> | | comes to the internals of objects that might be a really annoying
> | | task to do.
> | |
> | | thanks for your detailed estimation,
> | |
> | | Norbert
> | |
> | | > Dale
> | | >
> | | > ----- Original Message -----
> | | > | From: "Norbert Hartl" < [hidden email] >
> | | > | To: "GemStone Seaside beta discussion"
> | | > | < [hidden email] >
> | | > | Sent: Thursday, October 25, 2012 8:21:24 AM
> | | > | Subject: Re: [GS/SS Beta] STON and transactions
> | | > |
> | | > | Dale,
> | | > |
> | | > | I don't know how good you are in "wild guessing" but I think
> | | > | you
> | | > | are
> | | > | an expert :). To place an estimation from the gut what would
> | | > | you
> | | > | think how much work it will be to port Fuel to GemStone. You
> | | > | can
> | | > | leave Blocks aside.
> | | > |
> | | > | Norbert
> | | > |
> | | > | Am 25.10.2012 um 16:52 schrieb Dale Henrichs
> | | > | < [hidden email] >:
> | | > |
> | | > | > Regarding the memory issues ... I assume that the issue is
> | | > | > similar
> | | > | > to the problem with SIXX in that you're building some very
> | | > | > large
> | | > | > objects (Strings in Streams?) in temporary memory. For SIXX
> | | > | > the
> | | > | > trick was to link the root of the large temp object graph in
> | | > | > UserGlobals so that the temp object could be committed, thus
> | | > | > reducing the pressure on temp obj space ... at the end, the
> | | > | > temp
> | | > | > object was removed from UserGlobals so that it could be
> | | > | > garbage
> | | > | > collected ...
> | | > | >
> | | > | > Regarding the test failures in GemStone ... I will have to
> | | > | > take
> | | > | > a
> | | > | > look. At the time of the port, the tests were passing, but at
> | | > | > the
> | | > | > moment I don't recall which versions of GemStone I tested
> | | > | > STON
> | | > | > against ...
> | | > | >
> | | > | > I have been using STON in 3.1.0.1 , but I have been using it
> | | > | > to
> | | > | > serialize limited object graphs (I am stopping after
> | | > | > n-levels),
> | | > | > so
> | | > | > I haven't tried to serialize a big object graph from GemStone
> | | > | > and
> | | > | > haven't run into any errors using STON.
> | | > | >
> | | > | > Regarding the error related to ScaledDecimal ... I'd like to
> | | > | > know
> | | > | > what the error is. I can't imagine what kind of error that
> | | > | > ScaledDecimal would be throwing from the following code:
> | | > | >
> | | > | > STON fromStream: (String streamContents: [:stream|
> | | > | > (Dictionary new
> | | > | > at: Object new put: Object new;
> | | > | > yourself) stonOn: (STONWriter on: stream)])
> | | > | > readStream
> | | > | >
> | | > | > In a somewhat related are, I have found that there are bugs
> | | > | > related
> | | > | > to using #basicSize, #basicAt: and #basicAtPut: for a handful
> | | > | > of
> | | > | > classes. I've identified a problems in GsNMethod and ClassSet
> | | > | > and
> | | > | > this particular problem could affect the operation of STON.
> | | > | >
> | | > | > Still not sure about the ScaledDecimal problem...
> | | > | >
> | | > | > Dale
> | | > | > ----- Original Message -----
> | | > | > | From: "Norbert Hartl" < [hidden email] >
> | | > | > | To: "GemStone Seaside beta discussion"
> | | > | > | < [hidden email] >
> | | > | > | Sent: Thursday, October 25, 2012 4:53:45 AM
> | | > | > | Subject: Re: [GS/SS Beta] STON and transactions
> | | > | > |
> | | > | > | And finally the problem I've found seems to be a problem in
> | | > | > | the
> | | > | > | design of STON (or just me misunderstanding its purpose).
> | | > | > | Anyway
> | | > | > | the
> | | > | > | following code snippet raises the error
> | | > | > |
> | | > | > | STON fromStream: (String streamContents: [:stream|
> | | > | > | (Dictionary new
> | | > | > | at: Object new put: Object new;
> | | > | > | yourself) stonOn: (STONWriter on: stream)]) readStream
> | | > | > |
> | | > | > | Norbert
> | | > | > |
> | | > | > | Am 25.10.2012 um 13:37 schrieb Norbert Hartl
> | | > | > | < [hidden email] >:
> | | > | > |
> | | > | > | > This is some sort of monolog :) I just forgot to add to
> | | > | > | > the
> | | > | > | > last
> | | > | > | > mail that reading back the file in GemStone fails with
> | | > | > | > the
> | | > | > | > same
> | | > | > | > error as in pharo.
> | | > | > | >
> | | > | > | > Norbert
> | | > | > | >
> | | > | > | > Am 25.10.2012 um 13:35 schrieb Norbert Hartl
> | | > | > | > < [hidden email] >:
> | | > | > | >
> | | > | > | >> Unfortunately only the export worked "somehow". In pharo
> | | > | > | >> I
> | | > | > | >> cannot
> | | > | > | >> read in the STON file produced by gemstone. Tests are
> | | > | > | >> green on
> | | > | > | >> pharo. On gemstone there are two failing tests that have
> | | > | > | >> to do
> | | > | > | >> with ScaledDecimal. You said you are working with it and
> | | > | > | >> you
> | | > | > | >> did
> | | > | > | >> not encounter problems?
> | | > | > | >>
> | | > | > | >> Norbert
> | | > | > | >>
> | | > | > | >> Am 25.10.2012 um 11:11 schrieb Norbert Hartl
> | | > | > | >> < [hidden email] >:
> | | > | > | >>
> | | > | > | >>> Haha,
> | | > | > | >>>
> | | > | > | >>> often after I write a mail the situation changes
> | | > | > | >>> somehow
> | | > | > | >>> :)
> | | > | > | >>> While
> | | > | > | >>> fighting in the mist I tried doing a commitTransaction
> | | > | > | >>> instead
> | | > | > | >>> of abortTransaction and this works. At least I can
> | | > | > | >>> export
> | | > | > | >>> my
> | | > | > | >>> object graph now. If there is a good explanation I'm
> | | > | > | >>> glad
> | | > | > | >>> to
> | | > | > | >>> hear it :)
> | | > | > | >>>
> | | > | > | >>> Norbert
> | | > | > | >>>
> | | > | > | >>> Am 25.10.2012 um 11:03 schrieb Norbert Hartl
> | | > | > | >>> < [hidden email] >:
> | | > | > | >>>
> | | > | > | >>>> Dale,
> | | > | > | >>>>
> | | > | > | >>>> I loaded the STON code with your snippet and this
> | | > | > | >>>> worked.
> | | > | > | >>>> But it
> | | > | > | >>>> didn't solve my problem. I cannot export my object
> | | > | > | >>>> graph
> | | > | > | >>>> with
> | | > | > | >>>> STON. I always get temporary object memory full. I
> | | > | > | >>>> gave
> | | > | > | >>>> the
> | | > | > | >>>> gem
> | | > | > | >>>> 200MB already and still no success. I can give it even
> | | > | > | >>>> more
> | | > | > | >>>> but
> | | > | > | >>>> that would postpone my problem only a couple of weeks.
> | | > | > | >>>> I
> | | > | > | >>>> like
> | | > | > | >>>> to find the reason why it doesn't work.
> | | > | > | >>>>
> | | > | > | >>>> Reading the code I only suspected the STONWriterXXX
> | | > | > | >>>> classes
> | | > | > | >>>> to
> | | > | > | >>>> be the culprit because they are created all the time.
> | | > | > | >>>> So
> | | > | > | >>>> I
> | | > | > | >>>> rewrote the STON code to use an object pool for the
> | | > | > | >>>> writer
> | | > | > | >>>> classes but still no success. Having the pool I had
> | | > | > | >>>> one
> | | > | > | >>>> central
> | | > | > | >>>> instance that gets accessed all the time. So I tried
> | | > | > | >>>> to
> | | > | > | >>>> do
> | | > | > | >>>> on
> | | > | > | >>>> every 100000 instance request a 'System
> | | > | > | >>>> abortTransaction'
> | | > | > | >>>> hoping it would solve something.
> | | > | > | >>>> That falsified my theory disconnected objects can fill
> | | > | > | >>>> up
> | | > | > | >>>> the
> | | > | > | >>>> space. For the gem it is like a normal vm, right? When
> | | > | > | >>>> space
> | | > | > | >>>> gets low gc chimes in. So my abortTransaction doesn't
> | | > | > | >>>> do
> | | > | > | >>>> anything the gc wouldn't do, right?
> | | > | > | >>>> That means the objects that fill up the space are
> | | > | > | >>>> connected
> | | > | > | >>>> somehow. Now I'm asking myself what is the best way to
> | | > | > | >>>> get
> | | > | > | >>>> an
> | | > | > | >>>> overview over the instances that are in my temporary
> | | > | > | >>>> object
> | | > | > | >>>> memory. Would that be a duty for statmon or is there
> | | > | > | >>>> another
> | | > | > | >>>> way?
> | | > | > | >>>>
> | | > | > | >>>> thanks,
> | | > | > | >>>>
> | | > | > | >>>> Norbert
> | | > | > | >>>>
> | | > | > | >>>
> | | > | > | >>
> | | > | > | >
> | | > | > |
> | | > | > |
> | | > |
> | | > |
> | |
> | |
> |
> |
> |
> |
> | --
> | Mariano
> | http://marianopeck.wordpress.com
> |
> |

Reply | Threaded
Open this post in threaded view
|

Re: STON and transactions

Dale Henrichs
Norbert ...


Hmmm, you are exporting data to Pharo to do analysis ... so why don't you use GemStone for your analysis?

For the topez project I am exporting limited (by depth) object graphs from GemStone to Pharo using STON ... you could do something similar with this particular project ... if you need to run the analysis in Pharo ...

If extracting data from the system that you are analyzing is important, then you could extract the data using STON (for now) ... import into a separate GemStone db, then analyze on Pharo...

Just thinking out loud:)

Dale

----- Original Message -----
| From: "Norbert Hartl" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Sunday, October 28, 2012 3:55:32 AM
| Subject: Re: [GS/SS Beta] STON and transactions
|
|
| Am 26.10.2012 um 19:31 schrieb Dale Henrichs <[hidden email]>:
|
| > Mariano,
| >
| > Agree with everything you say ... my ignorance of Fuel 'fueled' my
| > wacky idea that if you could handle class shape changes you could
| > handle dialect to dialect changes too:)
| >
| That is what I was thinking, too.
|
| > As for the port, it' son my list of things to do, but if Norbert or
| > anyone finds that they would use Fuel if it were ported, then it's
| > worth trying to make time to get the port done …
| >
| I think it is a good idea to have fuel ported to gemstone. I don't
| know if it is urgent enough to distract you from your current work.
| I'm still trying to make the STON part work. For gemstone I just
| raised the SPC size and the gem temp object memory and I can save
| it. Now I'm trying to import it again in pharo. That might work but
| a new problem is close. The data I'm speaking of is getting bigger
| and bigger and so it will blow up a pharo image anyway.
| So I have two choice:
|
| - not exporting everything. If I can put a scope on the graph for
| exportation that could be a way to avoid exporting all of it. If
| this would work neither the gemstone export nor the pharo import
| will be a big problem.
| - store data external. I could store files or something similar. This
| way it will be slow but I could have at least access to the data
| from pharo in order to do analyzes.
|
| So you see my actual problem shifted slightly away from the
| serialization/materialization.
|
| Norbert
|
| > Dale
| >
| > ----- Original Message -----
| > | From: "Mariano Martinez Peck" <[hidden email]>
| > | To: "GemStone Seaside beta discussion"
| > | <[hidden email]>
| > | Sent: Friday, October 26, 2012 1:18:40 AM
| > | Subject: Re: [GS/SS Beta] STON and transactions
| > |
| > |
| > |
| > |
| > | On Fri, Oct 26, 2012 at 12:52 AM, Dale Henrichs <
| > | [hidden email]
| > | > wrote:
| > |
| > |
| > | Norbert,
| > |
| > | The fact that Fuel is not recursive makes it an attractive option
| > | for
| > | GemStone to GemStone transfer if passivate/activate (which is
| > | recursive) is not viable ... I'd be willing to chip in on a port
| > | of
| > | Fuel if you think you would use it...
| > |
| > |
| > |
| > | I think that porting Fuel to GemStone and do GemStone to GemStone
| > | transfer, should work without much effort. I cannot do it myself
| > | right now, but I am willing to help.
| > |
| > |
| > |
| > | Given the ability to map classes of different shapes,
| > |
| > |
| > | Not sure what you mean. Is what I said about class reshape? if
| > | true,
| > | I think there was a misunderstood. What I mean is that Fuel
| > | supports
| > | the case where when you materialize something, the classes in the
| > | system have changed since the time you serialized. Typical cases
| > | are: add/remove/rename/reorder instance variables, rename
| > | classes,
| > | etc.
| > |
| > |
| > | it sounds like there might be a chance of using Fuel for Gemstone
| > | to
| > | Pharo transfers. The OrderedCollection implementations are very
| > | different on the two platforms and would make a good first test
| > | case.
| > |
| > |
| > |
| > | Supporting GemStone-Pharo transfer looks more complex. At least,
| > | Fuel
| > | was never built with this in mind, I mean, we never tried to find
| > | portable representation of the objects. Say Float, we just encode
| > | Floats as they are in Pharo. We do not delegate to a
| > | compatibility
| > | layer (as other serializers like SRP do) nor we do any type of
| > | conversion. I don't know the concrete differences with the
| > | "primitive classes" (Floats, Integers, ByteArray, ByteStrings,
| > | etc)
| > | of GemStone and Pharo. Maybe there are not many and maybe a
| > | little
| > | layer would help.
| > | Anyway, what I want to say is that GemStone-Pharo transfer will
| > | be
| > | more work than GemStone to GemStone. And of course, the latter is
| > | needed for the first :)
| > |
| > |
| > |
| > |
| > |
| > |
| > | Dale
| > |
| > | ----- Original Message -----
| > | | From: "Norbert Hartl" < [hidden email] >
| > | | To: "GemStone Seaside beta discussion" <
| > | | [hidden email]
| > | | >
| > |
| > |
| > | | Sent: Thursday, October 25, 2012 10:40:32 AM
| > | | Subject: Re: [GS/SS Beta] STON and transactions
| > | |
| > | |
| > | | Am 25.10.2012 um 17:46 schrieb Dale Henrichs <
| > | | [hidden email]
| > | | >:
| > | |
| > | | > I haven't even looked at Fuel, so it will be a wild guess
| > | | > indeed.
| > | | >
| > | | > Are you planning on using Fuel to do GemStone to GemStone
| > | | > transfers
| > | | > or GemStone to Pharo transfers? GemStone 2.x to GemStone 3.x
| > | | > transfers? or GemStone to GemStone of the same major version?
| > | | >
| > | | I was just thinking about transfer from gemstone to pharo. But
| > | | gemstone to gemstone on same version is also interesting.
| > | |
| > | | > If you are talking GemStone to GemStone of same major
| > | | > version.
| > | | > You
| > | | > might want to look at Object>>passivate (and look at manual).
| > | | > Object passivation suffers from the same problem (running out
| > | | > of
| > | | > stack space) that SIXX had (before I changed some of the
| > | | > algorithms to not use recursion) and come to think of it ...
| > | | > If
| > | | > Fuel uses recursion while walking the object graph, it will
| > | | > have
| > | | > stack problems as well …
| > | | >
| > | | agreed
| > | |
| > | | > Cross dialect or even cross major version transfers probably
| > | | > can't
| > | | > be done well with Fuel (unless it is implemented differently
| > | | > than
| > | | > I imagine) so STON or SIXX are your best bets. BTW, what's
| > | | > wrong
| > | | > with SIXX?
| > | | >
| > | | The biggest problem is that there is no pull parser available
| > | | (I
| > | | know
| > | | Ken Treis did one but the xml parser changed quite a lot since
| > | | then). Writing out a graph depends mainly on the depth
| > | | traversal
| > | | management and that there is enough space for a object to
| > | | reference
| > | | table. Reading in could be done similar if a pure streaming
| > | | solution
| > | | would be available. In Sixx the XML is very verbose producing
| > | | huge
| > | | files. Without a pull parser you read in the whole file as
| > | | string,
| > | | then convert it into a DOM which is even more bloated. Just to
| > | | finally walk the DOM and create objects. That makes to problem
| > | | worse
| > | | at some magnitude. So STON is much more slim in string size and
| > | | it
| > | | is also smaller in code size. Making it more easy to tweak it
| > | | where
| > | | needed.
| > | |
| > | | Or to say it in numbers. Taking a fresh extent and reading in a
| > | | couple of 10000s of objects with SIXX took me the last time 40
| > | | minutes and left back an extent with a size of 25GB. Now I have
| > | | something between 2 and 5 million objects. I didn't even think
| > | | about
| > | | to try.
| > | |
| > | | > With all of that said I can't see it taking more than a week
| > | | > or
| > | | > two
| > | | > (63.458 hours). Classes are shaped differently in GemStone,
| > | | > but
| > | | > you can look at the passivate/active code for the special
| > | | > cases
| > | | > ... it takes a bit of thought but the recursive algorithms
| > | | > can be
| > | | > unwound into loops (makes for horrid debugging) without too
| > | | > much
| > | | > trouble.
| > | | >
| > | | That is a good tip. I'll look into the passivate stuff. You are
| > | | right
| > | | indeed. The different shapes of objects might be a big problem.
| > | | I
| > | | think Fuel has some shape changing support (as SIXX has). But
| > | | if it
| > | | comes to the internals of objects that might be a really
| > | | annoying
| > | | task to do.
| > | |
| > | | thanks for your detailed estimation,
| > | |
| > | | Norbert
| > | |
| > | | > Dale
| > | | >
| > | | > ----- Original Message -----
| > | | > | From: "Norbert Hartl" < [hidden email] >
| > | | > | To: "GemStone Seaside beta discussion"
| > | | > | < [hidden email] >
| > | | > | Sent: Thursday, October 25, 2012 8:21:24 AM
| > | | > | Subject: Re: [GS/SS Beta] STON and transactions
| > | | > |
| > | | > | Dale,
| > | | > |
| > | | > | I don't know how good you are in "wild guessing" but I
| > | | > | think
| > | | > | you
| > | | > | are
| > | | > | an expert :). To place an estimation from the gut what
| > | | > | would
| > | | > | you
| > | | > | think how much work it will be to port Fuel to GemStone.
| > | | > | You
| > | | > | can
| > | | > | leave Blocks aside.
| > | | > |
| > | | > | Norbert
| > | | > |
| > | | > | Am 25.10.2012 um 16:52 schrieb Dale Henrichs
| > | | > | < [hidden email] >:
| > | | > |
| > | | > | > Regarding the memory issues ... I assume that the issue
| > | | > | > is
| > | | > | > similar
| > | | > | > to the problem with SIXX in that you're building some
| > | | > | > very
| > | | > | > large
| > | | > | > objects (Strings in Streams?) in temporary memory. For
| > | | > | > SIXX
| > | | > | > the
| > | | > | > trick was to link the root of the large temp object graph
| > | | > | > in
| > | | > | > UserGlobals so that the temp object could be committed,
| > | | > | > thus
| > | | > | > reducing the pressure on temp obj space ... at the end,
| > | | > | > the
| > | | > | > temp
| > | | > | > object was removed from UserGlobals so that it could be
| > | | > | > garbage
| > | | > | > collected ...
| > | | > | >
| > | | > | > Regarding the test failures in GemStone ... I will have
| > | | > | > to
| > | | > | > take
| > | | > | > a
| > | | > | > look. At the time of the port, the tests were passing,
| > | | > | > but at
| > | | > | > the
| > | | > | > moment I don't recall which versions of GemStone I tested
| > | | > | > STON
| > | | > | > against ...
| > | | > | >
| > | | > | > I have been using STON in 3.1.0.1 , but I have been using
| > | | > | > it
| > | | > | > to
| > | | > | > serialize limited object graphs (I am stopping after
| > | | > | > n-levels),
| > | | > | > so
| > | | > | > I haven't tried to serialize a big object graph from
| > | | > | > GemStone
| > | | > | > and
| > | | > | > haven't run into any errors using STON.
| > | | > | >
| > | | > | > Regarding the error related to ScaledDecimal ... I'd like
| > | | > | > to
| > | | > | > know
| > | | > | > what the error is. I can't imagine what kind of error
| > | | > | > that
| > | | > | > ScaledDecimal would be throwing from the following code:
| > | | > | >
| > | | > | > STON fromStream: (String streamContents: [:stream|
| > | | > | > (Dictionary new
| > | | > | > at: Object new put: Object new;
| > | | > | > yourself) stonOn: (STONWriter on: stream)])
| > | | > | > readStream
| > | | > | >
| > | | > | > In a somewhat related are, I have found that there are
| > | | > | > bugs
| > | | > | > related
| > | | > | > to using #basicSize, #basicAt: and #basicAtPut: for a
| > | | > | > handful
| > | | > | > of
| > | | > | > classes. I've identified a problems in GsNMethod and
| > | | > | > ClassSet
| > | | > | > and
| > | | > | > this particular problem could affect the operation of
| > | | > | > STON.
| > | | > | >
| > | | > | > Still not sure about the ScaledDecimal problem...
| > | | > | >
| > | | > | > Dale
| > | | > | > ----- Original Message -----
| > | | > | > | From: "Norbert Hartl" < [hidden email] >
| > | | > | > | To: "GemStone Seaside beta discussion"
| > | | > | > | < [hidden email] >
| > | | > | > | Sent: Thursday, October 25, 2012 4:53:45 AM
| > | | > | > | Subject: Re: [GS/SS Beta] STON and transactions
| > | | > | > |
| > | | > | > | And finally the problem I've found seems to be a
| > | | > | > | problem in
| > | | > | > | the
| > | | > | > | design of STON (or just me misunderstanding its
| > | | > | > | purpose).
| > | | > | > | Anyway
| > | | > | > | the
| > | | > | > | following code snippet raises the error
| > | | > | > |
| > | | > | > | STON fromStream: (String streamContents: [:stream|
| > | | > | > | (Dictionary new
| > | | > | > | at: Object new put: Object new;
| > | | > | > | yourself) stonOn: (STONWriter on: stream)]) readStream
| > | | > | > |
| > | | > | > | Norbert
| > | | > | > |
| > | | > | > | Am 25.10.2012 um 13:37 schrieb Norbert Hartl
| > | | > | > | < [hidden email] >:
| > | | > | > |
| > | | > | > | > This is some sort of monolog :) I just forgot to add
| > | | > | > | > to
| > | | > | > | > the
| > | | > | > | > last
| > | | > | > | > mail that reading back the file in GemStone fails
| > | | > | > | > with
| > | | > | > | > the
| > | | > | > | > same
| > | | > | > | > error as in pharo.
| > | | > | > | >
| > | | > | > | > Norbert
| > | | > | > | >
| > | | > | > | > Am 25.10.2012 um 13:35 schrieb Norbert Hartl
| > | | > | > | > < [hidden email] >:
| > | | > | > | >
| > | | > | > | >> Unfortunately only the export worked "somehow". In
| > | | > | > | >> pharo
| > | | > | > | >> I
| > | | > | > | >> cannot
| > | | > | > | >> read in the STON file produced by gemstone. Tests
| > | | > | > | >> are
| > | | > | > | >> green on
| > | | > | > | >> pharo. On gemstone there are two failing tests that
| > | | > | > | >> have
| > | | > | > | >> to do
| > | | > | > | >> with ScaledDecimal. You said you are working with it
| > | | > | > | >> and
| > | | > | > | >> you
| > | | > | > | >> did
| > | | > | > | >> not encounter problems?
| > | | > | > | >>
| > | | > | > | >> Norbert
| > | | > | > | >>
| > | | > | > | >> Am 25.10.2012 um 11:11 schrieb Norbert Hartl
| > | | > | > | >> < [hidden email] >:
| > | | > | > | >>
| > | | > | > | >>> Haha,
| > | | > | > | >>>
| > | | > | > | >>> often after I write a mail the situation changes
| > | | > | > | >>> somehow
| > | | > | > | >>> :)
| > | | > | > | >>> While
| > | | > | > | >>> fighting in the mist I tried doing a
| > | | > | > | >>> commitTransaction
| > | | > | > | >>> instead
| > | | > | > | >>> of abortTransaction and this works. At least I can
| > | | > | > | >>> export
| > | | > | > | >>> my
| > | | > | > | >>> object graph now. If there is a good explanation
| > | | > | > | >>> I'm
| > | | > | > | >>> glad
| > | | > | > | >>> to
| > | | > | > | >>> hear it :)
| > | | > | > | >>>
| > | | > | > | >>> Norbert
| > | | > | > | >>>
| > | | > | > | >>> Am 25.10.2012 um 11:03 schrieb Norbert Hartl
| > | | > | > | >>> < [hidden email] >:
| > | | > | > | >>>
| > | | > | > | >>>> Dale,
| > | | > | > | >>>>
| > | | > | > | >>>> I loaded the STON code with your snippet and this
| > | | > | > | >>>> worked.
| > | | > | > | >>>> But it
| > | | > | > | >>>> didn't solve my problem. I cannot export my object
| > | | > | > | >>>> graph
| > | | > | > | >>>> with
| > | | > | > | >>>> STON. I always get temporary object memory full. I
| > | | > | > | >>>> gave
| > | | > | > | >>>> the
| > | | > | > | >>>> gem
| > | | > | > | >>>> 200MB already and still no success. I can give it
| > | | > | > | >>>> even
| > | | > | > | >>>> more
| > | | > | > | >>>> but
| > | | > | > | >>>> that would postpone my problem only a couple of
| > | | > | > | >>>> weeks.
| > | | > | > | >>>> I
| > | | > | > | >>>> like
| > | | > | > | >>>> to find the reason why it doesn't work.
| > | | > | > | >>>>
| > | | > | > | >>>> Reading the code I only suspected the
| > | | > | > | >>>> STONWriterXXX
| > | | > | > | >>>> classes
| > | | > | > | >>>> to
| > | | > | > | >>>> be the culprit because they are created all the
| > | | > | > | >>>> time.
| > | | > | > | >>>> So
| > | | > | > | >>>> I
| > | | > | > | >>>> rewrote the STON code to use an object pool for
| > | | > | > | >>>> the
| > | | > | > | >>>> writer
| > | | > | > | >>>> classes but still no success. Having the pool I
| > | | > | > | >>>> had
| > | | > | > | >>>> one
| > | | > | > | >>>> central
| > | | > | > | >>>> instance that gets accessed all the time. So I
| > | | > | > | >>>> tried
| > | | > | > | >>>> to
| > | | > | > | >>>> do
| > | | > | > | >>>> on
| > | | > | > | >>>> every 100000 instance request a 'System
| > | | > | > | >>>> abortTransaction'
| > | | > | > | >>>> hoping it would solve something.
| > | | > | > | >>>> That falsified my theory disconnected objects can
| > | | > | > | >>>> fill
| > | | > | > | >>>> up
| > | | > | > | >>>> the
| > | | > | > | >>>> space. For the gem it is like a normal vm, right?
| > | | > | > | >>>> When
| > | | > | > | >>>> space
| > | | > | > | >>>> gets low gc chimes in. So my abortTransaction
| > | | > | > | >>>> doesn't
| > | | > | > | >>>> do
| > | | > | > | >>>> anything the gc wouldn't do, right?
| > | | > | > | >>>> That means the objects that fill up the space are
| > | | > | > | >>>> connected
| > | | > | > | >>>> somehow. Now I'm asking myself what is the best
| > | | > | > | >>>> way to
| > | | > | > | >>>> get
| > | | > | > | >>>> an
| > | | > | > | >>>> overview over the instances that are in my
| > | | > | > | >>>> temporary
| > | | > | > | >>>> object
| > | | > | > | >>>> memory. Would that be a duty for statmon or is
| > | | > | > | >>>> there
| > | | > | > | >>>> another
| > | | > | > | >>>> way?
| > | | > | > | >>>>
| > | | > | > | >>>> thanks,
| > | | > | > | >>>>
| > | | > | > | >>>> Norbert
| > | | > | > | >>>>
| > | | > | > | >>>
| > | | > | > | >>
| > | | > | > | >
| > | | > | > |
| > | | > | > |
| > | | > |
| > | | > |
| > | |
| > | |
| > |
| > |
| > |
| > |
| > | --
| > | Mariano
| > | http://marianopeck.wordpress.com
| > |
| > |
|
|
Reply | Threaded
Open this post in threaded view
|

Re: STON and transactions

NorbertHartl

Am 29.10.2012 um 17:46 schrieb Dale Henrichs <[hidden email]>:

> Norbert ...
>
>
> Hmmm, you are exporting data to Pharo to do analysis ... so why don't you use GemStone for your analysis?
>
The term "analyzing" might be misleading. I have a lot of data in gemstone I like to play with and to develop. My primary development is in pharo because it is more comfortable than e.g. gemtools. My workflow is that I develop in pharo and then the code flows towards to live system in gemstone. While code flows from my pharo image to the development server and finally to the live system for data it should be the reverse. Bringing the data from the live to the test system is easy. I do a gemstone full backup every night. I just replay the backup on the test system so I have all of the newest data there. I like to close the gap with bringing the data from the test system to my development image. That's my motivation. Especially with my current duty to analyze/play with the data to model proper code around it.

> For the topez project I am exporting limited (by depth) object graphs from GemStone to Pharo using STON ... you could do something similar with this particular project ... if you need to run the analysis in Pharo ...
>
> If extracting data from the system that you are analyzing is important, then you could extract the data using STON (for now) ... import into a separate GemStone db, then analyze on Pharo...
>
> Just thinking out loud:)
>
Agreed. Today I managed to import data into pharo. I wrote a utility for my domain model that produces a shrinked copy of the data I can work with. I can export the data via STON. And since today I can import data into pharo. The STONReader needed to be changed. I was able to import over a million objects in pharo using approx. 200MB heap while exporting and now I have a image with 160MB after startup. Quite ok for me at the moment.  The new approach will last a little while. And then I'm not sure which is needed more urgent: fuel in gemstone or 64 bit pharo. I think it is the latter.

thanks,

Norbert

> ----- Original Message -----
> | From: "Norbert Hartl" <[hidden email]>
> | To: "GemStone Seaside beta discussion" <[hidden email]>
> | Sent: Sunday, October 28, 2012 3:55:32 AM
> | Subject: Re: [GS/SS Beta] STON and transactions
> |
> |
> | Am 26.10.2012 um 19:31 schrieb Dale Henrichs <[hidden email]>:
> |
> | > Mariano,
> | >
> | > Agree with everything you say ... my ignorance of Fuel 'fueled' my
> | > wacky idea that if you could handle class shape changes you could
> | > handle dialect to dialect changes too:)
> | >
> | That is what I was thinking, too.
> |
> | > As for the port, it' son my list of things to do, but if Norbert or
> | > anyone finds that they would use Fuel if it were ported, then it's
> | > worth trying to make time to get the port done …
> | >
> | I think it is a good idea to have fuel ported to gemstone. I don't
> | know if it is urgent enough to distract you from your current work.
> | I'm still trying to make the STON part work. For gemstone I just
> | raised the SPC size and the gem temp object memory and I can save
> | it. Now I'm trying to import it again in pharo. That might work but
> | a new problem is close. The data I'm speaking of is getting bigger
> | and bigger and so it will blow up a pharo image anyway.
> | So I have two choice:
> |
> | - not exporting everything. If I can put a scope on the graph for
> | exportation that could be a way to avoid exporting all of it. If
> | this would work neither the gemstone export nor the pharo import
> | will be a big problem.
> | - store data external. I could store files or something similar. This
> | way it will be slow but I could have at least access to the data
> | from pharo in order to do analyzes.
> |
> | So you see my actual problem shifted slightly away from the
> | serialization/materialization.
> |
> | Norbert
> |
> | > Dale
> | >
> | > ----- Original Message -----
> | > | From: "Mariano Martinez Peck" <[hidden email]>
> | > | To: "GemStone Seaside beta discussion"
> | > | <[hidden email]>
> | > | Sent: Friday, October 26, 2012 1:18:40 AM
> | > | Subject: Re: [GS/SS Beta] STON and transactions
> | > |
> | > |
> | > |
> | > |
> | > | On Fri, Oct 26, 2012 at 12:52 AM, Dale Henrichs <
> | > | [hidden email]
> | > | > wrote:
> | > |
> | > |
> | > | Norbert,
> | > |
> | > | The fact that Fuel is not recursive makes it an attractive option
> | > | for
> | > | GemStone to GemStone transfer if passivate/activate (which is
> | > | recursive) is not viable ... I'd be willing to chip in on a port
> | > | of
> | > | Fuel if you think you would use it...
> | > |
> | > |
> | > |
> | > | I think that porting Fuel to GemStone and do GemStone to GemStone
> | > | transfer, should work without much effort. I cannot do it myself
> | > | right now, but I am willing to help.
> | > |
> | > |
> | > |
> | > | Given the ability to map classes of different shapes,
> | > |
> | > |
> | > | Not sure what you mean. Is what I said about class reshape? if
> | > | true,
> | > | I think there was a misunderstood. What I mean is that Fuel
> | > | supports
> | > | the case where when you materialize something, the classes in the
> | > | system have changed since the time you serialized. Typical cases
> | > | are: add/remove/rename/reorder instance variables, rename
> | > | classes,
> | > | etc.
> | > |
> | > |
> | > | it sounds like there might be a chance of using Fuel for Gemstone
> | > | to
> | > | Pharo transfers. The OrderedCollection implementations are very
> | > | different on the two platforms and would make a good first test
> | > | case.
> | > |
> | > |
> | > |
> | > | Supporting GemStone-Pharo transfer looks more complex. At least,
> | > | Fuel
> | > | was never built with this in mind, I mean, we never tried to find
> | > | portable representation of the objects. Say Float, we just encode
> | > | Floats as they are in Pharo. We do not delegate to a
> | > | compatibility
> | > | layer (as other serializers like SRP do) nor we do any type of
> | > | conversion. I don't know the concrete differences with the
> | > | "primitive classes" (Floats, Integers, ByteArray, ByteStrings,
> | > | etc)
> | > | of GemStone and Pharo. Maybe there are not many and maybe a
> | > | little
> | > | layer would help.
> | > | Anyway, what I want to say is that GemStone-Pharo transfer will
> | > | be
> | > | more work than GemStone to GemStone. And of course, the latter is
> | > | needed for the first :)
> | > |
> | > |
> | > |
> | > |
> | > |
> | > |
> | > | Dale
> | > |
> | > | ----- Original Message -----
> | > | | From: "Norbert Hartl" < [hidden email] >
> | > | | To: "GemStone Seaside beta discussion" <
> | > | | [hidden email]
> | > | | >
> | > |
> | > |
> | > | | Sent: Thursday, October 25, 2012 10:40:32 AM
> | > | | Subject: Re: [GS/SS Beta] STON and transactions
> | > | |
> | > | |
> | > | | Am 25.10.2012 um 17:46 schrieb Dale Henrichs <
> | > | | [hidden email]
> | > | | >:
> | > | |
> | > | | > I haven't even looked at Fuel, so it will be a wild guess
> | > | | > indeed.
> | > | | >
> | > | | > Are you planning on using Fuel to do GemStone to GemStone
> | > | | > transfers
> | > | | > or GemStone to Pharo transfers? GemStone 2.x to GemStone 3.x
> | > | | > transfers? or GemStone to GemStone of the same major version?
> | > | | >
> | > | | I was just thinking about transfer from gemstone to pharo. But
> | > | | gemstone to gemstone on same version is also interesting.
> | > | |
> | > | | > If you are talking GemStone to GemStone of same major
> | > | | > version.
> | > | | > You
> | > | | > might want to look at Object>>passivate (and look at manual).
> | > | | > Object passivation suffers from the same problem (running out
> | > | | > of
> | > | | > stack space) that SIXX had (before I changed some of the
> | > | | > algorithms to not use recursion) and come to think of it ...
> | > | | > If
> | > | | > Fuel uses recursion while walking the object graph, it will
> | > | | > have
> | > | | > stack problems as well …
> | > | | >
> | > | | agreed
> | > | |
> | > | | > Cross dialect or even cross major version transfers probably
> | > | | > can't
> | > | | > be done well with Fuel (unless it is implemented differently
> | > | | > than
> | > | | > I imagine) so STON or SIXX are your best bets. BTW, what's
> | > | | > wrong
> | > | | > with SIXX?
> | > | | >
> | > | | The biggest problem is that there is no pull parser available
> | > | | (I
> | > | | know
> | > | | Ken Treis did one but the xml parser changed quite a lot since
> | > | | then). Writing out a graph depends mainly on the depth
> | > | | traversal
> | > | | management and that there is enough space for a object to
> | > | | reference
> | > | | table. Reading in could be done similar if a pure streaming
> | > | | solution
> | > | | would be available. In Sixx the XML is very verbose producing
> | > | | huge
> | > | | files. Without a pull parser you read in the whole file as
> | > | | string,
> | > | | then convert it into a DOM which is even more bloated. Just to
> | > | | finally walk the DOM and create objects. That makes to problem
> | > | | worse
> | > | | at some magnitude. So STON is much more slim in string size and
> | > | | it
> | > | | is also smaller in code size. Making it more easy to tweak it
> | > | | where
> | > | | needed.
> | > | |
> | > | | Or to say it in numbers. Taking a fresh extent and reading in a
> | > | | couple of 10000s of objects with SIXX took me the last time 40
> | > | | minutes and left back an extent with a size of 25GB. Now I have
> | > | | something between 2 and 5 million objects. I didn't even think
> | > | | about
> | > | | to try.
> | > | |
> | > | | > With all of that said I can't see it taking more than a week
> | > | | > or
> | > | | > two
> | > | | > (63.458 hours). Classes are shaped differently in GemStone,
> | > | | > but
> | > | | > you can look at the passivate/active code for the special
> | > | | > cases
> | > | | > ... it takes a bit of thought but the recursive algorithms
> | > | | > can be
> | > | | > unwound into loops (makes for horrid debugging) without too
> | > | | > much
> | > | | > trouble.
> | > | | >
> | > | | That is a good tip. I'll look into the passivate stuff. You are
> | > | | right
> | > | | indeed. The different shapes of objects might be a big problem.
> | > | | I
> | > | | think Fuel has some shape changing support (as SIXX has). But
> | > | | if it
> | > | | comes to the internals of objects that might be a really
> | > | | annoying
> | > | | task to do.
> | > | |
> | > | | thanks for your detailed estimation,
> | > | |
> | > | | Norbert
> | > | |
> | > | | > Dale
> | > | | >
> | > | | > ----- Original Message -----
> | > | | > | From: "Norbert Hartl" < [hidden email] >
> | > | | > | To: "GemStone Seaside beta discussion"
> | > | | > | < [hidden email] >
> | > | | > | Sent: Thursday, October 25, 2012 8:21:24 AM
> | > | | > | Subject: Re: [GS/SS Beta] STON and transactions
> | > | | > |
> | > | | > | Dale,
> | > | | > |
> | > | | > | I don't know how good you are in "wild guessing" but I
> | > | | > | think
> | > | | > | you
> | > | | > | are
> | > | | > | an expert :). To place an estimation from the gut what
> | > | | > | would
> | > | | > | you
> | > | | > | think how much work it will be to port Fuel to GemStone.
> | > | | > | You
> | > | | > | can
> | > | | > | leave Blocks aside.
> | > | | > |
> | > | | > | Norbert
> | > | | > |
> | > | | > | Am 25.10.2012 um 16:52 schrieb Dale Henrichs
> | > | | > | < [hidden email] >:
> | > | | > |
> | > | | > | > Regarding the memory issues ... I assume that the issue
> | > | | > | > is
> | > | | > | > similar
> | > | | > | > to the problem with SIXX in that you're building some
> | > | | > | > very
> | > | | > | > large
> | > | | > | > objects (Strings in Streams?) in temporary memory. For
> | > | | > | > SIXX
> | > | | > | > the
> | > | | > | > trick was to link the root of the large temp object graph
> | > | | > | > in
> | > | | > | > UserGlobals so that the temp object could be committed,
> | > | | > | > thus
> | > | | > | > reducing the pressure on temp obj space ... at the end,
> | > | | > | > the
> | > | | > | > temp
> | > | | > | > object was removed from UserGlobals so that it could be
> | > | | > | > garbage
> | > | | > | > collected ...
> | > | | > | >
> | > | | > | > Regarding the test failures in GemStone ... I will have
> | > | | > | > to
> | > | | > | > take
> | > | | > | > a
> | > | | > | > look. At the time of the port, the tests were passing,
> | > | | > | > but at
> | > | | > | > the
> | > | | > | > moment I don't recall which versions of GemStone I tested
> | > | | > | > STON
> | > | | > | > against ...
> | > | | > | >
> | > | | > | > I have been using STON in 3.1.0.1 , but I have been using
> | > | | > | > it
> | > | | > | > to
> | > | | > | > serialize limited object graphs (I am stopping after
> | > | | > | > n-levels),
> | > | | > | > so
> | > | | > | > I haven't tried to serialize a big object graph from
> | > | | > | > GemStone
> | > | | > | > and
> | > | | > | > haven't run into any errors using STON.
> | > | | > | >
> | > | | > | > Regarding the error related to ScaledDecimal ... I'd like
> | > | | > | > to
> | > | | > | > know
> | > | | > | > what the error is. I can't imagine what kind of error
> | > | | > | > that
> | > | | > | > ScaledDecimal would be throwing from the following code:
> | > | | > | >
> | > | | > | > STON fromStream: (String streamContents: [:stream|
> | > | | > | > (Dictionary new
> | > | | > | > at: Object new put: Object new;
> | > | | > | > yourself) stonOn: (STONWriter on: stream)])
> | > | | > | > readStream
> | > | | > | >
> | > | | > | > In a somewhat related are, I have found that there are
> | > | | > | > bugs
> | > | | > | > related
> | > | | > | > to using #basicSize, #basicAt: and #basicAtPut: for a
> | > | | > | > handful
> | > | | > | > of
> | > | | > | > classes. I've identified a problems in GsNMethod and
> | > | | > | > ClassSet
> | > | | > | > and
> | > | | > | > this particular problem could affect the operation of
> | > | | > | > STON.
> | > | | > | >
> | > | | > | > Still not sure about the ScaledDecimal problem...
> | > | | > | >
> | > | | > | > Dale
> | > | | > | > ----- Original Message -----
> | > | | > | > | From: "Norbert Hartl" < [hidden email] >
> | > | | > | > | To: "GemStone Seaside beta discussion"
> | > | | > | > | < [hidden email] >
> | > | | > | > | Sent: Thursday, October 25, 2012 4:53:45 AM
> | > | | > | > | Subject: Re: [GS/SS Beta] STON and transactions
> | > | | > | > |
> | > | | > | > | And finally the problem I've found seems to be a
> | > | | > | > | problem in
> | > | | > | > | the
> | > | | > | > | design of STON (or just me misunderstanding its
> | > | | > | > | purpose).
> | > | | > | > | Anyway
> | > | | > | > | the
> | > | | > | > | following code snippet raises the error
> | > | | > | > |
> | > | | > | > | STON fromStream: (String streamContents: [:stream|
> | > | | > | > | (Dictionary new
> | > | | > | > | at: Object new put: Object new;
> | > | | > | > | yourself) stonOn: (STONWriter on: stream)]) readStream
> | > | | > | > |
> | > | | > | > | Norbert
> | > | | > | > |
> | > | | > | > | Am 25.10.2012 um 13:37 schrieb Norbert Hartl
> | > | | > | > | < [hidden email] >:
> | > | | > | > |
> | > | | > | > | > This is some sort of monolog :) I just forgot to add
> | > | | > | > | > to
> | > | | > | > | > the
> | > | | > | > | > last
> | > | | > | > | > mail that reading back the file in GemStone fails
> | > | | > | > | > with
> | > | | > | > | > the
> | > | | > | > | > same
> | > | | > | > | > error as in pharo.
> | > | | > | > | >
> | > | | > | > | > Norbert
> | > | | > | > | >
> | > | | > | > | > Am 25.10.2012 um 13:35 schrieb Norbert Hartl
> | > | | > | > | > < [hidden email] >:
> | > | | > | > | >
> | > | | > | > | >> Unfortunately only the export worked "somehow". In
> | > | | > | > | >> pharo
> | > | | > | > | >> I
> | > | | > | > | >> cannot
> | > | | > | > | >> read in the STON file produced by gemstone. Tests
> | > | | > | > | >> are
> | > | | > | > | >> green on
> | > | | > | > | >> pharo. On gemstone there are two failing tests that
> | > | | > | > | >> have
> | > | | > | > | >> to do
> | > | | > | > | >> with ScaledDecimal. You said you are working with it
> | > | | > | > | >> and
> | > | | > | > | >> you
> | > | | > | > | >> did
> | > | | > | > | >> not encounter problems?
> | > | | > | > | >>
> | > | | > | > | >> Norbert
> | > | | > | > | >>
> | > | | > | > | >> Am 25.10.2012 um 11:11 schrieb Norbert Hartl
> | > | | > | > | >> < [hidden email] >:
> | > | | > | > | >>
> | > | | > | > | >>> Haha,
> | > | | > | > | >>>
> | > | | > | > | >>> often after I write a mail the situation changes
> | > | | > | > | >>> somehow
> | > | | > | > | >>> :)
> | > | | > | > | >>> While
> | > | | > | > | >>> fighting in the mist I tried doing a
> | > | | > | > | >>> commitTransaction
> | > | | > | > | >>> instead
> | > | | > | > | >>> of abortTransaction and this works. At least I can
> | > | | > | > | >>> export
> | > | | > | > | >>> my
> | > | | > | > | >>> object graph now. If there is a good explanation
> | > | | > | > | >>> I'm
> | > | | > | > | >>> glad
> | > | | > | > | >>> to
> | > | | > | > | >>> hear it :)
> | > | | > | > | >>>
> | > | | > | > | >>> Norbert
> | > | | > | > | >>>
> | > | | > | > | >>> Am 25.10.2012 um 11:03 schrieb Norbert Hartl
> | > | | > | > | >>> < [hidden email] >:
> | > | | > | > | >>>
> | > | | > | > | >>>> Dale,
> | > | | > | > | >>>>
> | > | | > | > | >>>> I loaded the STON code with your snippet and this
> | > | | > | > | >>>> worked.
> | > | | > | > | >>>> But it
> | > | | > | > | >>>> didn't solve my problem. I cannot export my object
> | > | | > | > | >>>> graph
> | > | | > | > | >>>> with
> | > | | > | > | >>>> STON. I always get temporary object memory full. I
> | > | | > | > | >>>> gave
> | > | | > | > | >>>> the
> | > | | > | > | >>>> gem
> | > | | > | > | >>>> 200MB already and still no success. I can give it
> | > | | > | > | >>>> even
> | > | | > | > | >>>> more
> | > | | > | > | >>>> but
> | > | | > | > | >>>> that would postpone my problem only a couple of
> | > | | > | > | >>>> weeks.
> | > | | > | > | >>>> I
> | > | | > | > | >>>> like
> | > | | > | > | >>>> to find the reason why it doesn't work.
> | > | | > | > | >>>>
> | > | | > | > | >>>> Reading the code I only suspected the
> | > | | > | > | >>>> STONWriterXXX
> | > | | > | > | >>>> classes
> | > | | > | > | >>>> to
> | > | | > | > | >>>> be the culprit because they are created all the
> | > | | > | > | >>>> time.
> | > | | > | > | >>>> So
> | > | | > | > | >>>> I
> | > | | > | > | >>>> rewrote the STON code to use an object pool for
> | > | | > | > | >>>> the
> | > | | > | > | >>>> writer
> | > | | > | > | >>>> classes but still no success. Having the pool I
> | > | | > | > | >>>> had
> | > | | > | > | >>>> one
> | > | | > | > | >>>> central
> | > | | > | > | >>>> instance that gets accessed all the time. So I
> | > | | > | > | >>>> tried
> | > | | > | > | >>>> to
> | > | | > | > | >>>> do
> | > | | > | > | >>>> on
> | > | | > | > | >>>> every 100000 instance request a 'System
> | > | | > | > | >>>> abortTransaction'
> | > | | > | > | >>>> hoping it would solve something.
> | > | | > | > | >>>> That falsified my theory disconnected objects can
> | > | | > | > | >>>> fill
> | > | | > | > | >>>> up
> | > | | > | > | >>>> the
> | > | | > | > | >>>> space. For the gem it is like a normal vm, right?
> | > | | > | > | >>>> When
> | > | | > | > | >>>> space
> | > | | > | > | >>>> gets low gc chimes in. So my abortTransaction
> | > | | > | > | >>>> doesn't
> | > | | > | > | >>>> do
> | > | | > | > | >>>> anything the gc wouldn't do, right?
> | > | | > | > | >>>> That means the objects that fill up the space are
> | > | | > | > | >>>> connected
> | > | | > | > | >>>> somehow. Now I'm asking myself what is the best
> | > | | > | > | >>>> way to
> | > | | > | > | >>>> get
> | > | | > | > | >>>> an
> | > | | > | > | >>>> overview over the instances that are in my
> | > | | > | > | >>>> temporary
> | > | | > | > | >>>> object
> | > | | > | > | >>>> memory. Would that be a duty for statmon or is
> | > | | > | > | >>>> there
> | > | | > | > | >>>> another
> | > | | > | > | >>>> way?
> | > | | > | > | >>>>
> | > | | > | > | >>>> thanks,
> | > | | > | > | >>>>
> | > | | > | > | >>>> Norbert
> | > | | > | > | >>>>
> | > | | > | > | >>>
> | > | | > | > | >>
> | > | | > | > | >
> | > | | > | > |
> | > | | > | > |
> | > | | > |
> | > | | > |
> | > | |
> | > | |
> | > |
> | > |
> | > |
> | > |
> | > | --
> | > | Mariano
> | > | http://marianopeck.wordpress.com
> | > |
> | > |
> |
> |

12