[Glass] out-of-mem removing elements from indexed RcIdentityBags
Locked
[Glass] out-of-mem removing elements from indexed RcIdentityBags
 
|
Hi there,
I am debugging an out-of-temporary-objectspace issue we ran into because we are removing objects from many different RcIdentityBags that each have 6 equality indexes defined on them in a single transaction. The amount of temp object space required for that is 100 MB. I am sure the issue is because of the indexes because when I execute the same operation after removing all the indexes, it requires only 5 MB of temp object space. When I check which instances are occupying most of the space, I get the *BtreeNode* family of classes rising to the top in bytesize. Though the manual mentions some things about performance when modifying indexed collections, I am surprised about the size required to execute this operation. So, is there a way to investigate why so many large nodes need to be loaded when removing? How do I interpret the results of _indexStatistics? Is there anything I can try to investigate? Thanks for any considerations Johan _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: [Glass] out-of-mem removing elements from indexed RcIdentityBags
|
|
Johan,
I don't think that there's much that you can do to reduce the space consumption. The discrepancy in space consumption that you're seeing is due to the fact that in a BtreeNode, we use 4 slots per object and RcIdentityBags only takes 1 slot per object. Right out of the gate, it takes 4x as many page-sized objects to store the same number of objects. Since you are looking at each object participating in 6 indexes, the back of the envelope calculation says that it will take about 24x as many page-sized objects to store the same set of objects in BtreeNodes ... So 5MB to 100MB TOC size is in the ballpark for "expected memory consumption". If TOC is an issue, I can imagine that there might be ways to handle the deletion of objcts from collections where you defer the removal to a batch process that has plenty of TOC available ... Dale On 02/05/2015 11:06 PM, Johan Brichau via Glass wrote: > Hi there, > > I am debugging an out-of-temporary-objectspace issue we ran into because we are removing objects from many different RcIdentityBags that each have 6 equality indexes defined on them in a single transaction. The amount of temp object space required for that is 100 MB. I am sure the issue is because of the indexes because when I execute the same operation after removing all the indexes, it requires only 5 MB of temp object space. When I check which instances are occupying most of the space, I get the *BtreeNode* family of classes rising to the top in bytesize. > > Though the manual mentions some things about performance when modifying indexed collections, I am surprised about the size required to execute this operation. So, is there a way to investigate why so many large nodes need to be loaded when removing? How do I interpret the results of _indexStatistics? Is there anything I can try to investigate? > > Thanks for any considerations > Johan > _______________________________________________ > Glass mailing list > [hidden email] > http://lists.gemtalksystems.com/mailman/listinfo/glass _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: [Glass] out-of-mem removing elements from indexed RcIdentityBags
|
|
Thanks for the feedback Dale.
Yes, we will probably work around this by postponing the actual removal from the collection to a service task. I had been experimenting a little with intermediate commits but then we break the assumption that a request can be fully aborted in the Seaside code. With respect to the temp object space size: I’m setting them at 100MB but would it be safe to allocate more, even when the addition of all gems temp object space largely exceeds the machine’s memory? I’m asking in the sense: does the gem release memory back to the OS ? If the gem only uses the amount of memory it requires and releases it back, we could easily handle peak situations too. cheers Johan > On 06 Feb 2015, at 19:10, Dale Henrichs <[hidden email]> wrote: > > Johan, > > I don't think that there's much that you can do to reduce the space consumption. > > The discrepancy in space consumption that you're seeing is due to the fact that in a BtreeNode, we use 4 slots per object and RcIdentityBags only takes 1 slot per object. > > Right out of the gate, it takes 4x as many page-sized objects to store the same number of objects. > > Since you are looking at each object participating in 6 indexes, the back of the envelope calculation says that it will take about 24x as many page-sized objects to store the same set of objects in BtreeNodes ... > > So 5MB to 100MB TOC size is in the ballpark for "expected memory consumption". > > If TOC is an issue, I can imagine that there might be ways to handle the deletion of objcts from collections where you defer the removal to a batch process that has plenty of TOC available ... > > Dale > > On 02/05/2015 11:06 PM, Johan Brichau via Glass wrote: >> Hi there, >> >> I am debugging an out-of-temporary-objectspace issue we ran into because we are removing objects from many different RcIdentityBags that each have 6 equality indexes defined on them in a single transaction. The amount of temp object space required for that is 100 MB. I am sure the issue is because of the indexes because when I execute the same operation after removing all the indexes, it requires only 5 MB of temp object space. When I check which instances are occupying most of the space, I get the *BtreeNode* family of classes rising to the top in bytesize. >> >> Though the manual mentions some things about performance when modifying indexed collections, I am surprised about the size required to execute this operation. So, is there a way to investigate why so many large nodes need to be loaded when removing? How do I interpret the results of _indexStatistics? Is there anything I can try to investigate? >> >> Thanks for any considerations >> Johan >> _______________________________________________ >> Glass mailing list >> [hidden email] >> http://lists.gemtalksystems.com/mailman/listinfo/glass > _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: [Glass] out-of-mem removing elements from indexed RcIdentityBags
|
|
On 2/6/15 11:12 AM, Johan Brichau wrote: > Thanks for the feedback Dale. > > Yes, we will probably work around this by postponing the actual removal from the collection to a service task. I had been experimenting a little with intermediate commits but then we break the assumption that a request can be fully aborted in the Seaside code. > > With respect to the temp object space size: I’m setting them at 100MB but would it be safe to allocate more, even when the addition of all gems temp object space largely exceeds the machine’s memory? I’m asking in the sense: does the gem release memory back to the OS ? If the gem only uses the amount of memory it requires and releases it back, we could easily handle peak situations too. Johan, this is a good point. Gems dynamically allocate and deallocate real memory based on demand, so you can get away with running gems whose total TOC would exceed memory. If you do this, it's probably a good idea to make sure that you've allocated swap space to cover for memory overruns. The system will run slower if you overflow to swap space, but when you run out of swap space Linux starts killing processes ... which is annoying:) Dale _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
«
Return to GLASS
|
1 view|%1 views
| Free forum by Nabble | Edit this page |