[tode_st] Could it be that tODE is holding onto objects?

Here could be an interesting part Dale:

  Search oop #  1 (  20658433):   ***SCAN COMPLETED***

  Complete reference path for oop   20658433

    1  oop=27279023617  aMetaTDTopezServer

    2  oop=28621538049  aTDDirectoryNode

    3  oop=27534577921  aDictionary

    4  oop=28683437569  aAssociation

    5  oop=28843284737  aTDDirectoryNode

    6  oop=14222087169  aTDTopezServer

    7  oop=14222075137  aIdentitySet

    8  oop=14222042625  aTodeClientListElement

    9  oop=14222043137  aTDObjectLogEntryBrowser

   10  oop=15775559937  aOrderedCollection

   11  oop=13241066753  aWAObjectLogEntry

   12  oop=13241067777  aGsProcess

   13  oop=13241069569  aGsStackBuffer

   14  oop=  20658433  aDpWebSession

On Wed, Feb 17, 2016 at 2:26 PM, Mariano Martinez Peck <[hidden email]> wrote:

Dale,

Remember that every in a while I face a problem in which I believe I have some WASessions around (and that means callbacks, and that means...a lot of stuff). I am again in that situation. 

I did a 

System abort. 

SystemRepository findAllReferencePathsToObject: ((SystemRepository listInstances: { DpWebSession }) at: 1) first 

And I got the stuff written into the gem log. I am trying to analyze that, and I am seeing a couple of places where tODE is related. Note that I use "ol view -r --continuation" a lot. And debug and inspect from there.

Please see attached file (private mail).  

I am not sure if tODE is the reason (maybe you cache things weakly and since I have many reference paths, then it could be another). But worth checking.

Does this sound any bell? Can you imagine scenarios where you could hold into the elements of the object log?

I will continue checking. Thanks in advance. 

--

Mariano
http://marianopeck.wordpress.com

--

Mariano
http://marianopeck.wordpress.com

On Wed, Feb 17, 2016 at 3:33 PM, Mariano Martinez Peck <[hidden email]> wrote:

Here could be an interesting part Dale:

  Search oop #  1 (  20658433):   ***SCAN COMPLETED***

  Complete reference path for oop   20658433

    1  oop=27279023617  aMetaTDTopezServer

    2  oop=28621538049  aTDDirectoryNode

    3  oop=27534577921  aDictionary

    4  oop=28683437569  aAssociation

    5  oop=28843284737  aTDDirectoryNode

    6  oop=14222087169  aTDTopezServer

    7  oop=14222075137  aIdentitySet

    8  oop=14222042625  aTodeClientListElement

    9  oop=14222043137  aTDObjectLogEntryBrowser

   10  oop=15775559937  aOrderedCollection

   11  oop=13241066753  aWAObjectLogEntry

   12  oop=13241067777  aGsProcess

   13  oop=13241069569  aGsStackBuffer

   14  oop=  20658433  aDpWebSession

Dale.... I have some more info... if I follow all that reference path ... I arrive to this conclusion:

the node in question is:

Inspect /mc//

--------------------

.              -> /mc/

(class)@       -> TDDirectoryNode

(oop)@         -> 28843284737

(committed)@   -> true

builderAspect@ -> nil

contents@      -> aDictionary( 'repos'->/mc/repos/, 'pkgs'->/mc/pkgs/)

name@          -> 'mc'

parent@        -> /

readMe@        -> nil

topez@         -> aTDTopezServer

Then, topez@ aTDTopezServer  -> persistentElementCache -> aTodeClientListElement -> theObject.

And this guy is a TDObjectLogEntryBrowser with:

windowLabel Object Log view -r --continuation

and it indeed has the some entries in  'objectLogEntries'.

How can I get rid of these?  I cannot confirm these are the guilty guys, but I would like to at least give it a try.

Thanks

  

On Wed, Feb 17, 2016 at 2:26 PM, Mariano Martinez Peck <[hidden email]> wrote:

Dale,

Remember that every in a while I face a problem in which I believe I have some WASessions around (and that means callbacks, and that means...a lot of stuff). I am again in that situation. 

I did a 

System abort. 

SystemRepository findAllReferencePathsToObject: ((SystemRepository listInstances: { DpWebSession }) at: 1) first 

And I got the stuff written into the gem log. I am trying to analyze that, and I am seeing a couple of places where tODE is related. Note that I use "ol view -r --continuation" a lot. And debug and inspect from there.

Please see attached file (private mail).  

I am not sure if tODE is the reason (maybe you cache things weakly and since I have many reference paths, then it could be another). But worth checking.

Does this sound any bell? Can you imagine scenarios where you could hold into the elements of the object log?

I will continue checking. Thanks in advance. 

--

Mariano
http://marianopeck.wordpress.com

--

Mariano
http://marianopeck.wordpress.com

--

Mariano
http://marianopeck.wordpress.com

Mariano,

take a look at the tODE command: `gs flushCaches` ... not only does it flush the "standard caches" (see the man page for the list), but it also does a `TDTopezServer clearPersistentElementCacheStack` which wipes out the cache stack in tODE ...

I think you've seen that this might be one of the culprits in your later analysis ... I create this command recently[2] while working on the Object Explorer[1] ... Soon you should be able to use the object explorer for tracking down reference path issues ...

Dale

[1] https://github.com/dalehenrich/obex
[2] https://github.com/dalehenrich/tode/pull/241

On 02/17/2016 09:26 AM, Mariano Martinez Peck wrote:

Dale,

Remember that every in a while I face a problem in which I believe I have some WASessions around (and that means callbacks, and that means...a lot of stuff). I am again in that situation. 

I did a 

System abort. 

SystemRepository findAllReferencePathsToObject: ((SystemRepository listInstances: { DpWebSession }) at: 1) first 

And I got the stuff written into the gem log. I am trying to analyze that, and I am seeing a couple of places where tODE is related. Note that I use "ol view -r --continuation" a lot. And debug and inspect from there.

Please see attached file (private mail).  

I am not sure if tODE is the reason (maybe you cache things weakly and since I have many reference paths, then it could be another). But worth checking.

Does this sound any bell? Can you imagine scenarios where you could hold into the elements of the object log?

I will continue checking. Thanks in advance. 

--

Mariano
http://marianopeck.wordpress.com

--
You received this message because you are subscribed to the Google Groups "tODE" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "tODE" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

On 2/18/16 11:15 AM, Mariano Martinez Peck wrote:

On Thu, Feb 18, 2016 at 2:45 PM, Dale Henrichs <[hidden email]> wrote:

Mariano,

take a look at the tODE command: `gs flushCaches` ... not only does it flush the "standard caches" (see the man page for the list), but it also does a `TDTopezServer clearPersistentElementCacheStack` which wipes out the cache stack in tODE ...

Dale, that did not do it. While the class var 'PersistentElementCacheStack' is now nil, the instances of TDTopezServer still have the instVar 'persistentElementCache' with an IdentitySet that points to lots of TodeClientListElement, which, one of those, the 'theObject' is the TDObjectLogEntryBrowser, whose 'objectLogEntries' are full of stacks (guilty guys).

 

Should I also nil 'persistentElementCache' instVar from each TDTopezServer instance? (I have 7 TDTopezServer instances).

Hmmm that is interesting ... I will have to look into that one a bit more ... I wouldn't think you'd have 7 instances running around without object log references

I think you've seen that this might be one of the culprits in your later analysis ... I create this command recently[2] while working on the Object Explorer[1] ... Soon you should be able to use the object explorer for tracking down reference path issues ...

This is a great initiative (I remember proposing this long ago). However... findReferencePathsTo:  takes a lifetime in this extent I am debugging :(

Time and technology are the only things keeping us from implementing every good idea:)

Yes the repo scans can be very costly ... for 3.4 we are looking into some promising techniques to speed the analysis up a bit.

Take a look at the Class Instance Count Detective[1[ (while you are waiting for a scan to complete)... I can explain a bit more if you are interested. but the scans are done against a backup of the db and if you know what you are looking for you might be able to accomplish a lot in a single pass through the backup - I extended the ScanBackup package[2] to allow one to execute arbitrary blocks during the scan...

Dale

[1] https://github.com/dalehenrich/obex#class-instance-counts-based-on-selected-set-of-instances
[2] https://github.com/dalehenrich/obex/tree/master/repository/Obex-Server-ScanBackup-Model.package/OXScanBackup64.class

 

Dale

[1] https://github.com/dalehenrich/obex
[2] https://github.com/dalehenrich/tode/pull/241

On 02/17/2016 09:26 AM, Mariano Martinez Peck wrote:

Dale,

Remember that every in a while I face a problem in which I believe I have some WASessions around (and that means callbacks, and that means...a lot of stuff). I am again in that situation. 

I did a 

System abort. 

SystemRepository findAllReferencePathsToObject: ((SystemRepository listInstances: { DpWebSession }) at: 1) first 

And I got the stuff written into the gem log. I am trying to analyze that, and I am seeing a couple of places where tODE is related. Note that I use "ol view -r --continuation" a lot. And debug and inspect from there.

Please see attached file (private mail).  

I am not sure if tODE is the reason (maybe you cache things weakly and since I have many reference paths, then it could be another). But worth checking.

Does this sound any bell? Can you imagine scenarios where you could hold into the elements of the object log?

I will continue checking. Thanks in advance. 

--

Mariano
http://marianopeck.wordpress.com

--
You received this message because you are subscribed to the Google Groups "tODE" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email][hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "tODE" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email][hidden email].
For more options, visit https://groups.google.com/d/optout.

--

Mariano
http://marianopeck.wordpress.com

--
You received this message because you are subscribed to the Google Groups "tODE" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "tODE" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

Mariano,

I think I've found the mechanism for keeping old instances of TDTopezServer alive. Take a look at Issue #249[1] ... the commands and rootNode slots in the TDTopezServer class are the root references ...

I've been working with obex and added a `--hard` option the `gs flushCaches` command for another reason and after further work, I've tracked down the TDTopezServer leak ...

I'll add a method that will clean up the references (and share the formula with you ... in the bug), since I know you cannot run the latest tODE .., yet .

Dale

[1] https://github.com/dalehenrich/tode/issues/249
[2] https://github.com/dalehenrich/tode/commit/8dc5ade46abb102d8a9ce2ed7e39f629d06747d0

On 02/19/2016 07:52 AM, Mariano Martinez Peck wrote:

On Fri, Feb 19, 2016 at 12:39 PM, Dale Henrichs <[hidden email]> wrote:

On 2/18/16 11:15 AM, Mariano Martinez Peck wrote:

On Thu, Feb 18, 2016 at 2:45 PM, Dale Henrichs <[hidden email][hidden email]> wrote:

Mariano,

take a look at the tODE command: `gs flushCaches` ... not only does it flush the "standard caches" (see the man page for the list), but it also does a `TDTopezServer clearPersistentElementCacheStack` which wipes out the cache stack in tODE ...

Dale, that did not do it. While the class var 'PersistentElementCacheStack' is now nil, the instances of TDTopezServer still have the instVar 'persistentElementCache' with an IdentitySet that points to lots of TodeClientListElement, which, one of those, the 'theObject' is the TDObjectLogEntryBrowser, whose 'objectLogEntries' are full of stacks (guilty guys).

 

Should I also nil 'persistentElementCache' instVar from each TDTopezServer instance? (I have 7 TDTopezServer instances).

Hmmm that is interesting ... I will have to look into that one a bit more ... I wouldn't think you'd have 7 instances running around without object log references

OK, thanks Dale. I will eagerly wait for you. Note that I would do it myself and directly nil all that as I really want to decrease the extent size. However... I am not doing anything and I am keeping the extent "as is" just to help you found the real issue (if there is none). I know I could do a backup and then restore etc..but that is a bit complicated right now, the extent is quite big, etc etc. 

 

The backup of the extent (compressed) is 5GB. The server has quite good upload speed, so if you want to give it a hand yourself let me know.

If you take the reference path I pasted above, and navigate the graph is quite easy to understand the reference path. 

I think you've seen that this might be one of the culprits in your later analysis ... I create this command recently[2] while working on the Object Explorer[1] ... Soon you should be able to use the object explorer for tracking down reference path issues ...

This is a great initiative (I remember proposing this long ago). However... findReferencePathsTo:  takes a lifetime in this extent I am debugging :(

Time and technology are the only things keeping us from implementing every good idea:)

:)

 

Yes the repo scans can be very costly ... for 3.4 we are looking into some promising techniques to speed the analysis up a bit.

Good to know. 

 

Take a look at the Class Instance Count Detective[1[ (while you are waiting for a scan to complete)... I can explain a bit more if you are interested. but the scans are done against a backup of the db and if you know what you are looking for you might be able to accomplish a lot in a single pass through the backup - I extended the ScanBackup package[2] to allow one to execute arbitrary blocks during the scan...

I do use the ScanBackup tool from James in order to get a list of instances since a long time. Problem with obex is that it needs a fairly new version of tODE...and I cannot still use latet tODE for my stones until I migrate to gsDevKit_home....

 

Dale

[1] https://github.com/dalehenrich/obex#class-instance-counts-based-on-selected-set-of-instances
[2] https://github.com/dalehenrich/obex/tree/master/repository/Obex-Server-ScanBackup-Model.package/OXScanBackup64.class

 

Dale

[1] https://github.com/dalehenrich/obex
[2] https://github.com/dalehenrich/tode/pull/241

On 02/17/2016 09:26 AM, Mariano Martinez Peck wrote:

Dale,

Remember that every in a while I face a problem in which I believe I have some WASessions around (and that means callbacks, and that means...a lot of stuff). I am again in that situation. 

I did a 

System abort. 

SystemRepository findAllReferencePathsToObject: ((SystemRepository listInstances: { DpWebSession }) at: 1) first 

And I got the stuff written into the gem log. I am trying to analyze that, and I am seeing a couple of places where tODE is related. Note that I use "ol view -r --continuation" a lot. And debug and inspect from there.

Please see attached file (private mail).  

I am not sure if tODE is the reason (maybe you cache things weakly and since I have many reference paths, then it could be another). But worth checking.

Does this sound any bell? Can you imagine scenarios where you could hold into the elements of the object log?

I will continue checking. Thanks in advance. 

--

Mariano
http://marianopeck.wordpress.com

--
You received this message because you are subscribed to the Google Groups "tODE" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email][hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "tODE" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--

Mariano
http://marianopeck.wordpress.com

--
You received this message because you are subscribed to the Google Groups "tODE" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email][hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "tODE" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email][hidden email].
For more options, visit https://groups.google.com/d/optout.

--

Mariano
http://marianopeck.wordpress.com

--
You received this message because you are subscribed to the Google Groups "tODE" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "tODE" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.