[Glass] out-of-mem removing elements from indexed RcIdentityBags

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[Glass] out-of-mem removing elements from indexed RcIdentityBags

GLASS mailing list
Hi there,

I am debugging an out-of-temporary-objectspace issue we ran into because we are removing objects from many different RcIdentityBags that each have 6 equality indexes defined on them in a single transaction. The amount of temp object space required for that is 100 MB. I am sure the issue is because of the indexes because when I execute the same operation after removing all the indexes, it requires only 5 MB of temp object space. When I check which instances are occupying most of the space, I get the *BtreeNode* family of classes rising to the top in bytesize.

Though the manual mentions some things about performance when modifying indexed collections, I am surprised about the size required to execute this operation. So, is there a way to investigate why so many large nodes need to be loaded when removing? How do I interpret the results of _indexStatistics? Is there anything I can try to investigate?

Thanks for any considerations
Johan
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] out-of-mem removing elements from indexed RcIdentityBags

GLASS mailing list
Johan,

I don't think that there's much that you can do to reduce the space
consumption.

The discrepancy in space consumption that you're seeing is due to the
fact that in a BtreeNode, we use 4 slots per object and RcIdentityBags
only takes 1 slot per object.

Right out of the gate, it takes 4x as many page-sized objects to store
the same number of objects.

Since you are looking at each object participating in 6 indexes, the
back of the envelope calculation says that it will take about 24x as
many page-sized objects to store the same set of objects in BtreeNodes ...

So 5MB to 100MB TOC size is in the ballpark for "expected memory
consumption".

If TOC is an issue, I can imagine that there might be ways to handle the
deletion of objcts from collections where you defer the removal to a
batch process that has plenty of TOC available ...

Dale

On 02/05/2015 11:06 PM, Johan Brichau via Glass wrote:

> Hi there,
>
> I am debugging an out-of-temporary-objectspace issue we ran into because we are removing objects from many different RcIdentityBags that each have 6 equality indexes defined on them in a single transaction. The amount of temp object space required for that is 100 MB. I am sure the issue is because of the indexes because when I execute the same operation after removing all the indexes, it requires only 5 MB of temp object space. When I check which instances are occupying most of the space, I get the *BtreeNode* family of classes rising to the top in bytesize.
>
> Though the manual mentions some things about performance when modifying indexed collections, I am surprised about the size required to execute this operation. So, is there a way to investigate why so many large nodes need to be loaded when removing? How do I interpret the results of _indexStatistics? Is there anything I can try to investigate?
>
> Thanks for any considerations
> Johan
> _______________________________________________
> Glass mailing list
> [hidden email]
> http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] out-of-mem removing elements from indexed RcIdentityBags

GLASS mailing list
Thanks for the feedback Dale.

Yes, we will probably work around this by postponing the actual removal from the collection to a service task. I had been experimenting a little with intermediate commits but then we break the assumption that a request can be fully aborted in the Seaside code.

With respect to the temp object space size: I’m setting them at 100MB but would it be safe to allocate more, even when the addition of all gems temp object space largely exceeds the machine’s memory? I’m asking in the sense: does the gem release memory back to the OS ? If the gem only uses the amount of memory it requires and releases it back, we could easily handle peak situations too.

cheers
Johan

> On 06 Feb 2015, at 19:10, Dale Henrichs <[hidden email]> wrote:
>
> Johan,
>
> I don't think that there's much that you can do to reduce the space consumption.
>
> The discrepancy in space consumption that you're seeing is due to the fact that in a BtreeNode, we use 4 slots per object and RcIdentityBags only takes 1 slot per object.
>
> Right out of the gate, it takes 4x as many page-sized objects to store the same number of objects.
>
> Since you are looking at each object participating in 6 indexes, the back of the envelope calculation says that it will take about 24x as many page-sized objects to store the same set of objects in BtreeNodes ...
>
> So 5MB to 100MB TOC size is in the ballpark for "expected memory consumption".
>
> If TOC is an issue, I can imagine that there might be ways to handle the deletion of objcts from collections where you defer the removal to a batch process that has plenty of TOC available ...
>
> Dale
>
> On 02/05/2015 11:06 PM, Johan Brichau via Glass wrote:
>> Hi there,
>>
>> I am debugging an out-of-temporary-objectspace issue we ran into because we are removing objects from many different RcIdentityBags that each have 6 equality indexes defined on them in a single transaction. The amount of temp object space required for that is 100 MB. I am sure the issue is because of the indexes because when I execute the same operation after removing all the indexes, it requires only 5 MB of temp object space. When I check which instances are occupying most of the space, I get the *BtreeNode* family of classes rising to the top in bytesize.
>>
>> Though the manual mentions some things about performance when modifying indexed collections, I am surprised about the size required to execute this operation. So, is there a way to investigate why so many large nodes need to be loaded when removing? How do I interpret the results of _indexStatistics? Is there anything I can try to investigate?
>>
>> Thanks for any considerations
>> Johan
>> _______________________________________________
>> Glass mailing list
>> [hidden email]
>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] out-of-mem removing elements from indexed RcIdentityBags

GLASS mailing list

On 2/6/15 11:12 AM, Johan Brichau wrote:
> Thanks for the feedback Dale.
>
> Yes, we will probably work around this by postponing the actual removal from the collection to a service task. I had been experimenting a little with intermediate commits but then we break the assumption that a request can be fully aborted in the Seaside code.
>
> With respect to the temp object space size: I’m setting them at 100MB but would it be safe to allocate more, even when the addition of all gems temp object space largely exceeds the machine’s memory? I’m asking in the sense: does the gem release memory back to the OS ? If the gem only uses the amount of memory it requires and releases it back, we could easily handle peak situations too.
Johan, this is a good point. Gems dynamically allocate and deallocate
real memory based on demand, so you can get away with running gems whose
total TOC would exceed memory. If you do this, it's probably a good idea
to make sure that you've allocated swap space to cover for memory
overruns. The system will run slower if you overflow to swap space, but
when you run out of swap space Linux starts killing processes ... which
is annoying:)

Dale
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass