[Glass] instances not being garbage collected?
Locked
[Glass] instances not being garbage collected?
 
|
Hi,
My repository has a (large) number of objects that are no longer referenced from the persistent root but that are never garbage collected. When I do: ClassXX allInstances I get a number of instances but when I then do for an instance i: SystemRepository findReferencePathForObject: i The answer is that there is no path (aka 'false' as the second element in the array that is returned). Any ideas how I can investigate this? thx Johan _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: [Glass] instances not being garbage collected?
|
|
Johan, Two thing come to mind right off the bat: 1. did these objects participate in an index? 2. have you cycled all of your gems recently If you are using indexes, strong references to objects will be made from the DepMap (an ObjectTable-like entity that maps objects to their dependents for modification tracking). Objects are managed correctly with respect to the DepMap, however, if you drop a reference to a collection that has an index without first removing the index, the DepMap will keep objects alive. If you are not using indexes the DepMap should be empty.
Long running gems may end up keeping references to otherwise dead objects [1], especially if you haven't haven't set GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE to something like 90:
GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE=90;
in your system.conf. Even with the above, you are not guaranteed to prune all of the persistent references on the vote. You should be suspicious that you are seeing this particular problem if you consistently see a high number of possible dead in your mfc report. Also look at vsd and after an mfc you should see the stone stat PossibleDeadObjs jump up to the value that was reported when your mfc ran ... Then a short while later (after voting is complete) you should see the stone stat DeadNotReclaimedObjs jump ... if DeadNotReclaimedObjs is significantly less than PossibleDeadObjs, then those objects were "voted down" because one or more live gems had a reference to the object in its head. If you've already got GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE set to 90 you might try cycling all of your gems and run another mfc ....
If your PossibleDeadObjs from an mfc is not as high as you think it should be, then I will have to do some more research on finding out if there are any other "hidden" references to objects ...
Dale
On Mon, Jan 20, 2014 at 11:40 AM, Johan Brichau <[hidden email]> wrote: Hi, _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: [Glass] instances not being garbage collected?
|
|
Hi Dale,
The objects that stick around are not part of any indexes. But I would interested to know what to do with such references too at some point. However, right now, I have the issue with a set of objects of different classes, which are referencing each other (which is logical) but their transitive closure is large. I am playing around in a copy of the extent, so it went through a mfc/backup/restore and several mfc already afterwards with no seaside gems running. I fiddled around some more, nilling some instance variables part of those objects to try to 'cut some references' to prune them. Some of them are now gone but I still have a long way to go to get the real stuck object(s). If you think of any places I should search, let me know, in the meantime I hope that I will be able to prune the objects enough to get a better idea. Johan On 20 Jan 2014, at 22:47, Dale Henrichs <[hidden email]> wrote: > Johan, > > Two thing come to mind right off the bat: > 1. did these objects participate in an index? > 2. have you cycled all of your gems recently > > If you are using indexes, strong references to objects will be made from the DepMap (an ObjectTable-like entity that maps objects to their dependents for modification tracking). Objects are managed correctly with respect to the DepMap, however, if you drop a reference to a collection that has an index without first removing the index, the DepMap will keep objects alive. If you are not using indexes the DepMap should be empty. > > Long running gems may end up keeping references to otherwise dead objects [1], especially if you haven't haven't set GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE to something like 90: > > GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE=90; > > in your system.conf. Even with the above, you are not guaranteed to prune all of the persistent references on the vote. You should be suspicious that you are seeing this particular problem if you consistently see a high number of possible dead in your mfc report. Also look at vsd and after an mfc you should see the stone stat PossibleDeadObjs jump up to the value that was reported when your mfc ran ... Then a short while later (after voting is complete) you should see the stone stat DeadNotReclaimedObjs jump ... if DeadNotReclaimedObjs is significantly less than PossibleDeadObjs, then those objects were "voted down" because one or more live gems had a reference to the object in its head. If you've already got GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE set to 90 you might try cycling all of your gems and run another mfc .... > > If your PossibleDeadObjs from an mfc is not as high as you think it should be, then I will have to do some more research on finding out if there are any other "hidden" references to objects ... > > Dale > > > [1] https://code.google.com/p/glassdb/issues/detail?id=136 > > > On Mon, Jan 20, 2014 at 11:40 AM, Johan Brichau <[hidden email]> wrote: > Hi, > > My repository has a (large) number of objects that are no longer referenced from the persistent root but that are never garbage collected. > > When I do: > > ClassXX allInstances > > I get a number of instances but when I then do for an instance i: > > SystemRepository findReferencePathForObject: i > > The answer is that there is no path (aka 'false' as the second element in the array that is returned). > > Any ideas how I can investigate this? > > thx > Johan > _______________________________________________ > Glass mailing list > [hidden email] > http://lists.gemtalksystems.com/mailman/listinfo/glass > _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: [Glass] instances not being garbage collected?
|
|
Dale,
I think I found the blocking reference but it does not make sense to me. In the path that is returned for this object (let's call it A), I notice an RcIdentityBag that is empty. The next object on the reference path is an Array of IdentityBags (thus referenced by the RcIdentityBag) and that one holds on to a WAGemStoneServiceTask, where a VariableContext is holding on to the object A. The object A is part of our DALi framework and because it is not garbage collected, it is building up something very similar to a CR backlog (hence a lot of objects). The step of the one RcIdentityBag to the Array of IdentityBags is the one I do not understand. The RcIdentityBag is the #inProcess bag of WAGemStoneServiceTask but this collection is empty. Nevertheless, one of the IdentityBags (referenced by the RcIdentityBag) is holding on to the WAGemStoneServiceTask instance. I'm now checking to see if this is the main culprit but it definitely seems like it. Johan On 20 Jan 2014, at 23:01, Johan Brichau <[hidden email]> wrote: > Hi Dale, > > The objects that stick around are not part of any indexes. But I would interested to know what to do with such references too at some point. > However, right now, I have the issue with a set of objects of different classes, which are referencing each other (which is logical) but their transitive closure is large. > > I am playing around in a copy of the extent, so it went through a mfc/backup/restore and several mfc already afterwards with no seaside gems running. > > I fiddled around some more, nilling some instance variables part of those objects to try to 'cut some references' to prune them. > Some of them are now gone but I still have a long way to go to get the real stuck object(s). > > If you think of any places I should search, let me know, in the meantime I hope that I will be able to prune the objects enough to get a better idea. > > Johan > > On 20 Jan 2014, at 22:47, Dale Henrichs <[hidden email]> wrote: > >> Johan, >> >> Two thing come to mind right off the bat: >> 1. did these objects participate in an index? >> 2. have you cycled all of your gems recently >> >> If you are using indexes, strong references to objects will be made from the DepMap (an ObjectTable-like entity that maps objects to their dependents for modification tracking). Objects are managed correctly with respect to the DepMap, however, if you drop a reference to a collection that has an index without first removing the index, the DepMap will keep objects alive. If you are not using indexes the DepMap should be empty. >> >> Long running gems may end up keeping references to otherwise dead objects [1], especially if you haven't haven't set GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE to something like 90: >> >> GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE=90; >> >> in your system.conf. Even with the above, you are not guaranteed to prune all of the persistent references on the vote. You should be suspicious that you are seeing this particular problem if you consistently see a high number of possible dead in your mfc report. Also look at vsd and after an mfc you should see the stone stat PossibleDeadObjs jump up to the value that was reported when your mfc ran ... Then a short while later (after voting is complete) you should see the stone stat DeadNotReclaimedObjs jump ... if DeadNotReclaimedObjs is significantly less than PossibleDeadObjs, then those objects were "voted down" because one or more live gems had a reference to the object in its head. If you've already got GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE set to 90 you might try cycling all of your gems and run another mfc .... >> >> If your PossibleDeadObjs from an mfc is not as high as you think it should be, then I will have to do some more research on finding out if there are any other "hidden" references to objects ... >> >> Dale >> >> >> [1] https://code.google.com/p/glassdb/issues/detail?id=136 >> >> >> On Mon, Jan 20, 2014 at 11:40 AM, Johan Brichau <[hidden email]> wrote: >> Hi, >> >> My repository has a (large) number of objects that are no longer referenced from the persistent root but that are never garbage collected. >> >> When I do: >> >> ClassXX allInstances >> >> I get a number of instances but when I then do for an instance i: >> >> SystemRepository findReferencePathForObject: i >> >> The answer is that there is no path (aka 'false' as the second element in the array that is returned). >> >> Any ideas how I can investigate this? >> >> thx >> Johan >> _______________________________________________ >> Glass mailing list >> [hidden email] >> http://lists.gemtalksystems.com/mailman/listinfo/glass >> > _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: [Glass] instances not being garbage collected?
|
|
Dale,
Things get even weirder... First off, the good news is that this instance of WAGemStoneServiceTask is the culprit. After nilling the variable holding on to our DALi object (which is the start of a CR backlog), all those objects got removed from the database in an MFC. However, the things I saw happening with the RcIdentityBag are very strange: Remember that the instance of WAGemStoneServiceTask was not found in the RcIdentityBag but in an IdentityBag that is referenced from this RcIdentityBag. I removed the instance of WAGemStoneServiceTask from the IdentityBag and ran an mfc. After that, the objects were still there and something very weird happened: when I asked for the reference path to the instance of WAGemStoneServiceTask, the RcIdentityBag now contained the object that I removed. So, now, the reference path looked ok. However, after removing it again and running an MFC again, I again get a similar situation as it was before: the instance of WAGemStoneServiceTask could be found in two instances of the IdentityBag but not in the RcIdentityBag. I took a backup at each step. For now I will just nil the same variable in production so I get rid of the rapid object build up. There is a chain of 3 million objects (and their various contents) that gets garbage collected at once which fixes my rapid repository growth since Dec 30th (which is when the cuplrit object got created). Johan On 21 Jan 2014, at 07:55, Johan Brichau <[hidden email]> wrote: > Dale, > > I think I found the blocking reference but it does not make sense to me. > > In the path that is returned for this object (let's call it A), I notice an RcIdentityBag that is empty. The next object on the reference path is an Array of IdentityBags (thus referenced by the RcIdentityBag) and that one holds on to a WAGemStoneServiceTask, where a VariableContext is holding on to the object A. The object A is part of our DALi framework and because it is not garbage collected, it is building up something very similar to a CR backlog (hence a lot of objects). > > The step of the one RcIdentityBag to the Array of IdentityBags is the one I do not understand. The RcIdentityBag is the #inProcess bag of WAGemStoneServiceTask but this collection is empty. Nevertheless, one of the IdentityBags (referenced by the RcIdentityBag) is holding on to the WAGemStoneServiceTask instance. > > I'm now checking to see if this is the main culprit but it definitely seems like it. > > Johan > > On 20 Jan 2014, at 23:01, Johan Brichau <[hidden email]> wrote: > >> Hi Dale, >> >> The objects that stick around are not part of any indexes. But I would interested to know what to do with such references too at some point. >> However, right now, I have the issue with a set of objects of different classes, which are referencing each other (which is logical) but their transitive closure is large. >> >> I am playing around in a copy of the extent, so it went through a mfc/backup/restore and several mfc already afterwards with no seaside gems running. >> >> I fiddled around some more, nilling some instance variables part of those objects to try to 'cut some references' to prune them. >> Some of them are now gone but I still have a long way to go to get the real stuck object(s). >> >> If you think of any places I should search, let me know, in the meantime I hope that I will be able to prune the objects enough to get a better idea. >> >> Johan >> >> On 20 Jan 2014, at 22:47, Dale Henrichs <[hidden email]> wrote: >> >>> Johan, >>> >>> Two thing come to mind right off the bat: >>> 1. did these objects participate in an index? >>> 2. have you cycled all of your gems recently >>> >>> If you are using indexes, strong references to objects will be made from the DepMap (an ObjectTable-like entity that maps objects to their dependents for modification tracking). Objects are managed correctly with respect to the DepMap, however, if you drop a reference to a collection that has an index without first removing the index, the DepMap will keep objects alive. If you are not using indexes the DepMap should be empty. >>> >>> Long running gems may end up keeping references to otherwise dead objects [1], especially if you haven't haven't set GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE to something like 90: >>> >>> GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE=90; >>> >>> in your system.conf. Even with the above, you are not guaranteed to prune all of the persistent references on the vote. You should be suspicious that you are seeing this particular problem if you consistently see a high number of possible dead in your mfc report. Also look at vsd and after an mfc you should see the stone stat PossibleDeadObjs jump up to the value that was reported when your mfc ran ... Then a short while later (after voting is complete) you should see the stone stat DeadNotReclaimedObjs jump ... if DeadNotReclaimedObjs is significantly less than PossibleDeadObjs, then those objects were "voted down" because one or more live gems had a reference to the object in its head. If you've already got GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE set to 90 you might try cycling all of your gems and run another mfc .... >>> >>> If your PossibleDeadObjs from an mfc is not as high as you think it should be, then I will have to do some more research on finding out if there are any other "hidden" references to objects ... >>> >>> Dale >>> >>> >>> [1] https://code.google.com/p/glassdb/issues/detail?id=136 >>> >>> >>> On Mon, Jan 20, 2014 at 11:40 AM, Johan Brichau <[hidden email]> wrote: >>> Hi, >>> >>> My repository has a (large) number of objects that are no longer referenced from the persistent root but that are never garbage collected. >>> >>> When I do: >>> >>> ClassXX allInstances >>> >>> I get a number of instances but when I then do for an instance i: >>> >>> SystemRepository findReferencePathForObject: i >>> >>> The answer is that there is no path (aka 'false' as the second element in the array that is returned). >>> >>> Any ideas how I can investigate this? >>> >>> thx >>> Johan >>> _______________________________________________ >>> Glass mailing list >>> [hidden email] >>> http://lists.gemtalksystems.com/mailman/listinfo/glass >>> >> > _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: [Glass] instances not being garbage collected?
|
|
Johan,
While this seems weird, it can actually make sense. If you watch http://www.youtube.com/watch?v=1rb_A8hwKmc you will see part of a lecture I gave about RC classes. Consider the following code (run in 3.2 to take advantage of the new GsExternalSession). We create an RcIdentityBag, look at its contents and the internal components (all empty), add something to it in one session, look at the contents and internal components (it appears in the collection and in one of the components), remove the contents in another session, look at the contents and internal components (it is not in the collection but is in two of the components), cleanup, and finally look at the contents and components (all empty). The take-away is that the components are used to avoid conflict and the object can be "not in the collection" but still referenced (at least) twice from the components. This is because it is contained in the “additions" for one session and the “removals" for another session. RcIdentityBag>>#’cleanupBag’ is the way to clean up the components. If you remove the object from the removals then it magically reappears in the main collection (since it is still in the additions). | rcBag session1 session2 stream block | rcBag := RcIdentityBag new: 10. UserGlobals at: #'James' put: rcBag. stream := WriteStream on: String new. block := [ | components | components := (rcBag instVarAt: 5) reject: [:each | each isEmpty]. rcBag asArray printOn: stream. stream nextPutAll: ' - '. components printOn: stream. stream lf. ]. System commit. stream lf. block value. session1 := GsExternalSession gemNRS: '!tcp@vienna#netldi:netldi3#task!gemnetobject' stoneNRS: '!tcp@localhost#server!gs64stone3' username: 'DataCurator' password: 'swordfish'. session1 login; executeString: 'James add: 42. System commit.'. System commit. block value. session2 := GsExternalSession gemNRS: '!tcp@vienna#netldi:netldi3#task!gemnetobject' stoneNRS: '!tcp@localhost#server!gs64stone3' username: 'DataCurator' password: 'swordfish'. session2 login; executeString: 'James remove: 42. System commit.'. System commit. block value. session1 logout. session2 logout. rcBag cleanupBag. block value. stream contents. " anArray( ) - anArray( ) anArray( 42) - anArray( anIdentityBag( 42)) anArray( ) - anArray( anIdentityBag( 42), anIdentityBag( 42)) anArray( ) - anArray( ) " James On Jan 21, 2014, at 8:29 AM, Johan Brichau <[hidden email]> wrote: > Dale, > > Things get even weirder... > > First off, the good news is that this instance of WAGemStoneServiceTask is the culprit. After nilling the variable holding on to our DALi object (which is the start of a CR backlog), all those objects got removed from the database in an MFC. > > However, the things I saw happening with the RcIdentityBag are very strange: > > Remember that the instance of WAGemStoneServiceTask was not found in the RcIdentityBag but in an IdentityBag that is referenced from this RcIdentityBag. I removed the instance of WAGemStoneServiceTask from the IdentityBag and ran an mfc. After that, the objects were still there and something very weird happened: when I asked for the reference path to the instance of WAGemStoneServiceTask, the RcIdentityBag now contained the object that I removed. So, now, the reference path looked ok. However, after removing it again and running an MFC again, I again get a similar situation as it was before: the instance of WAGemStoneServiceTask could be found in two instances of the IdentityBag but not in the RcIdentityBag. > > I took a backup at each step. For now I will just nil the same variable in production so I get rid of the rapid object build up. There is a chain of 3 million objects (and their various contents) that gets garbage collected at once which fixes my rapid repository growth since Dec 30th (which is when the cuplrit object got created). > > Johan > > On 21 Jan 2014, at 07:55, Johan Brichau <[hidden email]> wrote: > >> Dale, >> >> I think I found the blocking reference but it does not make sense to me. >> >> In the path that is returned for this object (let's call it A), I notice an RcIdentityBag that is empty. The next object on the reference path is an Array of IdentityBags (thus referenced by the RcIdentityBag) and that one holds on to a WAGemStoneServiceTask, where a VariableContext is holding on to the object A. The object A is part of our DALi framework and because it is not garbage collected, it is building up something very similar to a CR backlog (hence a lot of objects). >> >> The step of the one RcIdentityBag to the Array of IdentityBags is the one I do not understand. The RcIdentityBag is the #inProcess bag of WAGemStoneServiceTask but this collection is empty. Nevertheless, one of the IdentityBags (referenced by the RcIdentityBag) is holding on to the WAGemStoneServiceTask instance. >> >> I'm now checking to see if this is the main culprit but it definitely seems like it. >> >> Johan >> >> On 20 Jan 2014, at 23:01, Johan Brichau <[hidden email]> wrote: >> >>> Hi Dale, >>> >>> The objects that stick around are not part of any indexes. But I would interested to know what to do with such references too at some point. >>> However, right now, I have the issue with a set of objects of different classes, which are referencing each other (which is logical) but their transitive closure is large. >>> >>> I am playing around in a copy of the extent, so it went through a mfc/backup/restore and several mfc already afterwards with no seaside gems running. >>> >>> I fiddled around some more, nilling some instance variables part of those objects to try to 'cut some references' to prune them. >>> Some of them are now gone but I still have a long way to go to get the real stuck object(s). >>> >>> If you think of any places I should search, let me know, in the meantime I hope that I will be able to prune the objects enough to get a better idea. >>> >>> Johan >>> >>> On 20 Jan 2014, at 22:47, Dale Henrichs <[hidden email]> wrote: >>> >>>> Johan, >>>> >>>> Two thing come to mind right off the bat: >>>> 1. did these objects participate in an index? >>>> 2. have you cycled all of your gems recently >>>> >>>> If you are using indexes, strong references to objects will be made from the DepMap (an ObjectTable-like entity that maps objects to their dependents for modification tracking). Objects are managed correctly with respect to the DepMap, however, if you drop a reference to a collection that has an index without first removing the index, the DepMap will keep objects alive. If you are not using indexes the DepMap should be empty. >>>> >>>> Long running gems may end up keeping references to otherwise dead objects [1], especially if you haven't haven't set GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE to something like 90: >>>> >>>> GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE=90; >>>> >>>> in your system.conf. Even with the above, you are not guaranteed to prune all of the persistent references on the vote. You should be suspicious that you are seeing this particular problem if you consistently see a high number of possible dead in your mfc report. Also look at vsd and after an mfc you should see the stone stat PossibleDeadObjs jump up to the value that was reported when your mfc ran ... Then a short while later (after voting is complete) you should see the stone stat DeadNotReclaimedObjs jump ... if DeadNotReclaimedObjs is significantly less than PossibleDeadObjs, then those objects were "voted down" because one or more live gems had a reference to the object in its head. If you've already got GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE set to 90 you might try cycling all of your gems and run another mfc .... >>>> >>>> If your PossibleDeadObjs from an mfc is not as high as you think it should be, then I will have to do some more research on finding out if there are any other "hidden" references to objects ... >>>> >>>> Dale >>>> >>>> >>>> [1] https://code.google.com/p/glassdb/issues/detail?id=136 >>>> >>>> >>>> On Mon, Jan 20, 2014 at 11:40 AM, Johan Brichau <[hidden email]> wrote: >>>> Hi, >>>> >>>> My repository has a (large) number of objects that are no longer referenced from the persistent root but that are never garbage collected. >>>> >>>> When I do: >>>> >>>> ClassXX allInstances >>>> >>>> I get a number of instances but when I then do for an instance i: >>>> >>>> SystemRepository findReferencePathForObject: i >>>> >>>> The answer is that there is no path (aka 'false' as the second element in the array that is returned). >>>> >>>> Any ideas how I can investigate this? >>>> >>>> thx >>>> Johan >>>> _______________________________________________ >>>> Glass mailing list >>>> [hidden email] >>>> http://lists.gemtalksystems.com/mailman/listinfo/glass >>>> >>> >> > > _______________________________________________ > Glass mailing list > [hidden email] > http://lists.gemtalksystems.com/mailman/listinfo/glass _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: [Glass] instances not being garbage collected?
|
|
Hi James,
Are there particular ways to detect when I would need to send #cleanupBag? In this case, I noticed the problem because the 'stuck object' was keeping a lot of other objects from being garbage collected. But if there are just objects that are stuck in these collections, they might suddenly pop up again? Johan On 22 Jan 2014, at 06:29, James Foster <[hidden email]> wrote: > Johan, > > While this seems weird, it can actually make sense. If you watch http://www.youtube.com/watch?v=1rb_A8hwKmc you will see part of a lecture I gave about RC classes. Consider the following code (run in 3.2 to take advantage of the new GsExternalSession). We create an RcIdentityBag, look at its contents and the internal components (all empty), add something to it in one session, look at the contents and internal components (it appears in the collection and in one of the components), remove the contents in another session, look at the contents and internal components (it is not in the collection but is in two of the components), cleanup, and finally look at the contents and components (all empty). The take-away is that the components are used to avoid conflict and the object can be "not in the collection" but still referenced (at least) twice from the components. This is because it is contained in the “additions" for one session and the “removals" for another session. RcIdentityBag>>#’cleanupBag’ is the way to clean up the components. If you remove the object from the removals then it magically reappears in the main collection (since it is still in the additions). > > > | rcBag session1 session2 stream block | > rcBag := RcIdentityBag new: 10. > UserGlobals at: #'James' put: rcBag. > stream := WriteStream on: String new. > block := [ > | components | > components := (rcBag instVarAt: 5) reject: [:each | each isEmpty]. > rcBag asArray printOn: stream. > stream nextPutAll: ' - '. > components printOn: stream. > stream lf. > ]. > System commit. > stream lf. > block value. > session1 := GsExternalSession > gemNRS: '!tcp@vienna#netldi:netldi3#task!gemnetobject' > stoneNRS: '!tcp@localhost#server!gs64stone3' > username: 'DataCurator' > password: 'swordfish'. > session1 login; executeString: 'James add: 42. System commit.'. > System commit. > block value. > session2 := GsExternalSession > gemNRS: '!tcp@vienna#netldi:netldi3#task!gemnetobject' > stoneNRS: '!tcp@localhost#server!gs64stone3' > username: 'DataCurator' > password: 'swordfish'. > session2 login; executeString: 'James remove: 42. System commit.'. > System commit. > block value. > session1 logout. > session2 logout. > rcBag cleanupBag. > block value. > stream contents. > " > anArray( ) - anArray( ) > anArray( 42) - anArray( anIdentityBag( 42)) > anArray( ) - anArray( anIdentityBag( 42), anIdentityBag( 42)) > anArray( ) - anArray( ) > " > > James > > On Jan 21, 2014, at 8:29 AM, Johan Brichau <[hidden email]> wrote: > >> Dale, >> >> Things get even weirder... >> >> First off, the good news is that this instance of WAGemStoneServiceTask is the culprit. After nilling the variable holding on to our DALi object (which is the start of a CR backlog), all those objects got removed from the database in an MFC. >> >> However, the things I saw happening with the RcIdentityBag are very strange: >> >> Remember that the instance of WAGemStoneServiceTask was not found in the RcIdentityBag but in an IdentityBag that is referenced from this RcIdentityBag. I removed the instance of WAGemStoneServiceTask from the IdentityBag and ran an mfc. After that, the objects were still there and something very weird happened: when I asked for the reference path to the instance of WAGemStoneServiceTask, the RcIdentityBag now contained the object that I removed. So, now, the reference path looked ok. However, after removing it again and running an MFC again, I again get a similar situation as it was before: the instance of WAGemStoneServiceTask could be found in two instances of the IdentityBag but not in the RcIdentityBag. >> >> I took a backup at each step. For now I will just nil the same variable in production so I get rid of the rapid object build up. There is a chain of 3 million objects (and their various contents) that gets garbage collected at once which fixes my rapid repository growth since Dec 30th (which is when the cuplrit object got created). >> >> Johan >> >> On 21 Jan 2014, at 07:55, Johan Brichau <[hidden email]> wrote: >> >>> Dale, >>> >>> I think I found the blocking reference but it does not make sense to me. >>> >>> In the path that is returned for this object (let's call it A), I notice an RcIdentityBag that is empty. The next object on the reference path is an Array of IdentityBags (thus referenced by the RcIdentityBag) and that one holds on to a WAGemStoneServiceTask, where a VariableContext is holding on to the object A. The object A is part of our DALi framework and because it is not garbage collected, it is building up something very similar to a CR backlog (hence a lot of objects). >>> >>> The step of the one RcIdentityBag to the Array of IdentityBags is the one I do not understand. The RcIdentityBag is the #inProcess bag of WAGemStoneServiceTask but this collection is empty. Nevertheless, one of the IdentityBags (referenced by the RcIdentityBag) is holding on to the WAGemStoneServiceTask instance. >>> >>> I'm now checking to see if this is the main culprit but it definitely seems like it. >>> >>> Johan >>> >>> On 20 Jan 2014, at 23:01, Johan Brichau <[hidden email]> wrote: >>> >>>> Hi Dale, >>>> >>>> The objects that stick around are not part of any indexes. But I would interested to know what to do with such references too at some point. >>>> However, right now, I have the issue with a set of objects of different classes, which are referencing each other (which is logical) but their transitive closure is large. >>>> >>>> I am playing around in a copy of the extent, so it went through a mfc/backup/restore and several mfc already afterwards with no seaside gems running. >>>> >>>> I fiddled around some more, nilling some instance variables part of those objects to try to 'cut some references' to prune them. >>>> Some of them are now gone but I still have a long way to go to get the real stuck object(s). >>>> >>>> If you think of any places I should search, let me know, in the meantime I hope that I will be able to prune the objects enough to get a better idea. >>>> >>>> Johan >>>> >>>> On 20 Jan 2014, at 22:47, Dale Henrichs <[hidden email]> wrote: >>>> >>>>> Johan, >>>>> >>>>> Two thing come to mind right off the bat: >>>>> 1. did these objects participate in an index? >>>>> 2. have you cycled all of your gems recently >>>>> >>>>> If you are using indexes, strong references to objects will be made from the DepMap (an ObjectTable-like entity that maps objects to their dependents for modification tracking). Objects are managed correctly with respect to the DepMap, however, if you drop a reference to a collection that has an index without first removing the index, the DepMap will keep objects alive. If you are not using indexes the DepMap should be empty. >>>>> >>>>> Long running gems may end up keeping references to otherwise dead objects [1], especially if you haven't haven't set GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE to something like 90: >>>>> >>>>> GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE=90; >>>>> >>>>> in your system.conf. Even with the above, you are not guaranteed to prune all of the persistent references on the vote. You should be suspicious that you are seeing this particular problem if you consistently see a high number of possible dead in your mfc report. Also look at vsd and after an mfc you should see the stone stat PossibleDeadObjs jump up to the value that was reported when your mfc ran ... Then a short while later (after voting is complete) you should see the stone stat DeadNotReclaimedObjs jump ... if DeadNotReclaimedObjs is significantly less than PossibleDeadObjs, then those objects were "voted down" because one or more live gems had a reference to the object in its head. If you've already got GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE set to 90 you might try cycling all of your gems and run another mfc .... >>>>> >>>>> If your PossibleDeadObjs from an mfc is not as high as you think it should be, then I will have to do some more research on finding out if there are any other "hidden" references to objects ... >>>>> >>>>> Dale >>>>> >>>>> >>>>> [1] https://code.google.com/p/glassdb/issues/detail?id=136 >>>>> >>>>> >>>>> On Mon, Jan 20, 2014 at 11:40 AM, Johan Brichau <[hidden email]> wrote: >>>>> Hi, >>>>> >>>>> My repository has a (large) number of objects that are no longer referenced from the persistent root but that are never garbage collected. >>>>> >>>>> When I do: >>>>> >>>>> ClassXX allInstances >>>>> >>>>> I get a number of instances but when I then do for an instance i: >>>>> >>>>> SystemRepository findReferencePathForObject: i >>>>> >>>>> The answer is that there is no path (aka 'false' as the second element in the array that is returned). >>>>> >>>>> Any ideas how I can investigate this? >>>>> >>>>> thx >>>>> Johan >>>>> _______________________________________________ >>>>> Glass mailing list >>>>> [hidden email] >>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass >>>>> >>>> >>> >> >> _______________________________________________ >> Glass mailing list >> [hidden email] >> http://lists.gemtalksystems.com/mailman/listinfo/glass > _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
[Glass] ZnManagingMultiThreadedServer error after backup recovery
|
|
Hi all!
I just tried to move my dev gemstone db to my prototype platform and encouter errors. My dev environment is a virtual image and the prototype is running on an intel nuc system. I took a backup on the virtual system and then I loaded it into my my prototype system. Now, I do get the following error while sending a REST request to the ZincServer "a InternalError occurred (error 2083), Primitive number -1 does not exist in the virtual machine." it is a result of the an Error in the critical Block in "ZnManagingMultiThreadedServer socketStreamOn: socket" that I can't debug due to the typical Gemstone stack errors in the debugger. Did anybody else encouter such issues and is it possible to move/synchronize Gmestone installations this way anyways? Connection the database with Jade or Gemtools is no problem at all. Somehow I have the strange feeling that I might have an issue in my Ubuntu installation, too. Thanks Sebastian _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: [Glass] ZnManagingMultiThreadedServer error after backup recovery
|
|
On Wed, Jan 22, 2014 at 11:06 AM, Sebastian Heidbrink <[hidden email]> wrote: Hi all! not sure I know what an 'intel nuc system' is.... what os/os version are you running on this nuc thing:)
The system thinks you are trying to execute primitive number -1 ... this is very bizarre.... Since you are restoring from a backup, I am curious if there might be subtle GemStone version difference between the system where you produced the backup and the system where you are restoring the backup ... if the GemStone versions are not exactly the same, then that could be the source of your problem ...
If the GemStone versions are exactly the same, then this is a very good way to synchronize ...
When we get this one resolved, I am interested in getting to the bottom of the "typical Gemstone stack errors" issue ...
Dale _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: [Glass] instances not being garbage collected?
|
|
In reply to this post by Johan Brichau-3
Johan,
The conditions for RC cleanup are a bit vague and not well documented. It looks to me like prior to an MFC you should stop all user sessions and run the cleanup but that doesn’t seem completely reasonable. In any case, it explains the earlier mystery! James On Jan 22, 2014, at 10:11 AM, Johan Brichau <[hidden email]> wrote: > Hi James, > > Are there particular ways to detect when I would need to send #cleanupBag? > In this case, I noticed the problem because the 'stuck object' was keeping a lot of other objects from being garbage collected. But if there are just objects that are stuck in these collections, they might suddenly pop up again? > > Johan > > On 22 Jan 2014, at 06:29, James Foster <[hidden email]> wrote: > >> Johan, >> >> While this seems weird, it can actually make sense. If you watch http://www.youtube.com/watch?v=1rb_A8hwKmc you will see part of a lecture I gave about RC classes. Consider the following code (run in 3.2 to take advantage of the new GsExternalSession). We create an RcIdentityBag, look at its contents and the internal components (all empty), add something to it in one session, look at the contents and internal components (it appears in the collection and in one of the components), remove the contents in another session, look at the contents and internal components (it is not in the collection but is in two of the components), cleanup, and finally look at the contents and components (all empty). The take-away is that the components are used to avoid conflict and the object can be "not in the collection" but still referenced (at least) twice from the components. This is because it is contained in the “additions" for one session and the “removals" for another session. RcIdentityBag>>#’cleanupBag’ is the way to clean up the components. If you remove the object from the removals then it magically reappears in the main collection (since it is still in the additions). >> >> >> | rcBag session1 session2 stream block | >> rcBag := RcIdentityBag new: 10. >> UserGlobals at: #'James' put: rcBag. >> stream := WriteStream on: String new. >> block := [ >> | components | >> components := (rcBag instVarAt: 5) reject: [:each | each isEmpty]. >> rcBag asArray printOn: stream. >> stream nextPutAll: ' - '. >> components printOn: stream. >> stream lf. >> ]. >> System commit. >> stream lf. >> block value. >> session1 := GsExternalSession >> gemNRS: '!tcp@vienna#netldi:netldi3#task!gemnetobject' >> stoneNRS: '!tcp@localhost#server!gs64stone3' >> username: 'DataCurator' >> password: 'swordfish'. >> session1 login; executeString: 'James add: 42. System commit.'. >> System commit. >> block value. >> session2 := GsExternalSession >> gemNRS: '!tcp@vienna#netldi:netldi3#task!gemnetobject' >> stoneNRS: '!tcp@localhost#server!gs64stone3' >> username: 'DataCurator' >> password: 'swordfish'. >> session2 login; executeString: 'James remove: 42. System commit.'. >> System commit. >> block value. >> session1 logout. >> session2 logout. >> rcBag cleanupBag. >> block value. >> stream contents. >> " >> anArray( ) - anArray( ) >> anArray( 42) - anArray( anIdentityBag( 42)) >> anArray( ) - anArray( anIdentityBag( 42), anIdentityBag( 42)) >> anArray( ) - anArray( ) >> " >> >> James >> >> On Jan 21, 2014, at 8:29 AM, Johan Brichau <[hidden email]> wrote: >> >>> Dale, >>> >>> Things get even weirder... >>> >>> First off, the good news is that this instance of WAGemStoneServiceTask is the culprit. After nilling the variable holding on to our DALi object (which is the start of a CR backlog), all those objects got removed from the database in an MFC. >>> >>> However, the things I saw happening with the RcIdentityBag are very strange: >>> >>> Remember that the instance of WAGemStoneServiceTask was not found in the RcIdentityBag but in an IdentityBag that is referenced from this RcIdentityBag. I removed the instance of WAGemStoneServiceTask from the IdentityBag and ran an mfc. After that, the objects were still there and something very weird happened: when I asked for the reference path to the instance of WAGemStoneServiceTask, the RcIdentityBag now contained the object that I removed. So, now, the reference path looked ok. However, after removing it again and running an MFC again, I again get a similar situation as it was before: the instance of WAGemStoneServiceTask could be found in two instances of the IdentityBag but not in the RcIdentityBag. >>> >>> I took a backup at each step. For now I will just nil the same variable in production so I get rid of the rapid object build up. There is a chain of 3 million objects (and their various contents) that gets garbage collected at once which fixes my rapid repository growth since Dec 30th (which is when the cuplrit object got created). >>> >>> Johan >>> >>> On 21 Jan 2014, at 07:55, Johan Brichau <[hidden email]> wrote: >>> >>>> Dale, >>>> >>>> I think I found the blocking reference but it does not make sense to me. >>>> >>>> In the path that is returned for this object (let's call it A), I notice an RcIdentityBag that is empty. The next object on the reference path is an Array of IdentityBags (thus referenced by the RcIdentityBag) and that one holds on to a WAGemStoneServiceTask, where a VariableContext is holding on to the object A. The object A is part of our DALi framework and because it is not garbage collected, it is building up something very similar to a CR backlog (hence a lot of objects). >>>> >>>> The step of the one RcIdentityBag to the Array of IdentityBags is the one I do not understand. The RcIdentityBag is the #inProcess bag of WAGemStoneServiceTask but this collection is empty. Nevertheless, one of the IdentityBags (referenced by the RcIdentityBag) is holding on to the WAGemStoneServiceTask instance. >>>> >>>> I'm now checking to see if this is the main culprit but it definitely seems like it. >>>> >>>> Johan >>>> >>>> On 20 Jan 2014, at 23:01, Johan Brichau <[hidden email]> wrote: >>>> >>>>> Hi Dale, >>>>> >>>>> The objects that stick around are not part of any indexes. But I would interested to know what to do with such references too at some point. >>>>> However, right now, I have the issue with a set of objects of different classes, which are referencing each other (which is logical) but their transitive closure is large. >>>>> >>>>> I am playing around in a copy of the extent, so it went through a mfc/backup/restore and several mfc already afterwards with no seaside gems running. >>>>> >>>>> I fiddled around some more, nilling some instance variables part of those objects to try to 'cut some references' to prune them. >>>>> Some of them are now gone but I still have a long way to go to get the real stuck object(s). >>>>> >>>>> If you think of any places I should search, let me know, in the meantime I hope that I will be able to prune the objects enough to get a better idea. >>>>> >>>>> Johan >>>>> >>>>> On 20 Jan 2014, at 22:47, Dale Henrichs <[hidden email]> wrote: >>>>> >>>>>> Johan, >>>>>> >>>>>> Two thing come to mind right off the bat: >>>>>> 1. did these objects participate in an index? >>>>>> 2. have you cycled all of your gems recently >>>>>> >>>>>> If you are using indexes, strong references to objects will be made from the DepMap (an ObjectTable-like entity that maps objects to their dependents for modification tracking). Objects are managed correctly with respect to the DepMap, however, if you drop a reference to a collection that has an index without first removing the index, the DepMap will keep objects alive. If you are not using indexes the DepMap should be empty. >>>>>> >>>>>> Long running gems may end up keeping references to otherwise dead objects [1], especially if you haven't haven't set GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE to something like 90: >>>>>> >>>>>> GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE=90; >>>>>> >>>>>> in your system.conf. Even with the above, you are not guaranteed to prune all of the persistent references on the vote. You should be suspicious that you are seeing this particular problem if you consistently see a high number of possible dead in your mfc report. Also look at vsd and after an mfc you should see the stone stat PossibleDeadObjs jump up to the value that was reported when your mfc ran ... Then a short while later (after voting is complete) you should see the stone stat DeadNotReclaimedObjs jump ... if DeadNotReclaimedObjs is significantly less than PossibleDeadObjs, then those objects were "voted down" because one or more live gems had a reference to the object in its head. If you've already got GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE set to 90 you might try cycling all of your gems and run another mfc .... >>>>>> >>>>>> If your PossibleDeadObjs from an mfc is not as high as you think it should be, then I will have to do some more research on finding out if there are any other "hidden" references to objects ... >>>>>> >>>>>> Dale >>>>>> >>>>>> >>>>>> [1] https://code.google.com/p/glassdb/issues/detail?id=136 >>>>>> >>>>>> >>>>>> On Mon, Jan 20, 2014 at 11:40 AM, Johan Brichau <[hidden email]> wrote: >>>>>> Hi, >>>>>> >>>>>> My repository has a (large) number of objects that are no longer referenced from the persistent root but that are never garbage collected. >>>>>> >>>>>> When I do: >>>>>> >>>>>> ClassXX allInstances >>>>>> >>>>>> I get a number of instances but when I then do for an instance i: >>>>>> >>>>>> SystemRepository findReferencePathForObject: i >>>>>> >>>>>> The answer is that there is no path (aka 'false' as the second element in the array that is returned). >>>>>> >>>>>> Any ideas how I can investigate this? >>>>>> >>>>>> thx >>>>>> Johan >>>>>> _______________________________________________ >>>>>> Glass mailing list >>>>>> [hidden email] >>>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass >>>>>> >>>>> >>>> >>> >>> _______________________________________________ >>> Glass mailing list >>> [hidden email] >>> http://lists.gemtalksystems.com/mailman/listinfo/glass >> > _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: [Glass] ZnManagingMultiThreadedServer error after backup recovery
|
|
In reply to this post by Dale Henrichs-3
Hi Dale,
both systems are based on Ubuntu 12.04.4 LTS One is virtual and the other one is a completely different hardware. The virtual image is absed on Gemstone version 3.1.0.4 the hardware, faulty, db is Gemstone version 3.1.0.1 !!!! I installed the script with the wrong version numer :-/ That was amazing help! Thank you! Sebastian Am 22.01.2014 11:17, schrieb Dale Henrichs:
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: [Glass] instances not being garbage collected?
|
|
In reply to this post by James Foster-9
Johan, I'll check around today and see what I can find out ... Dale On Wed, Jan 22, 2014 at 11:19 AM, James Foster <[hidden email]> wrote: Johan, _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: [Glass] instances not being garbage collected?
|
|
Johan, After reading the comment in #cleanupBag it seems to me that the cleanupBag is something that should be done on a regular basis as part of regular db maintenance ... the sticky wicket is that the cleanup activity could result in a commit conflict if the bag is operated on by another session ...so technically it isn't necessary to shut down the active sessions, but you do want to ensure that the other sessions remain idle while doing the cleanup (at least not accessing the bag(s) in question so it isn't a simple matter of scheduling regular clean up...
In your specific case if you have some off hours production time where at a minimum you've got the web server inactive, you can schedule the cleanup during that period .. Otherwise we need to dig deeper for a solution. I've create a bug[1] for tracking this issue ... I hate to resort to a lock for this purpose, but a generic bag maintenance lock may be the ticket ...
Dale On Thu, Jan 23, 2014 at 7:11 AM, Dale Henrichs <[hidden email]> wrote:
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
In reply to this post by Dale Henrichs-3
Dale,
If I removed a reference to a collection containing an index without first removing the index, can I still clear the contained objects? How can one clean the DepMap?
Thanks,
Thierry On Mon, Jan 20, 2014 at 2:47 PM, Dale Henrichs <[hidden email]> wrote:
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: [Glass] instances not being garbage collected?
|
|
Thierry, IndexManger current getAllNscRoots will give you an IdentitySet of all nscs with indexes ... once you identify the disconnected nsc, you can user #removeAllIndexes to clean up the DepMap correctly...
Dale On Sat, Mar 1, 2014 at 8:45 PM, Thierry Thelliez <[hidden email]> wrote:
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
«
Return to GLASS
|
1 view|%1 views
| Free forum by Nabble | Edit this page |