[Glass] GemTools-1.0-beta.8.7.1-32x can't log in to GS 3.2.4

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
57 messages Options
123
Reply | Threaded
Open this post in threaded view
|

[Glass] GemTools-1.0-beta.8.7.1-32x can't log in to GS 3.2.4

GLASS mailing list
I'm trying to log in to GS 3.2.4 using GemTools-1.0-beta.8.7.1-32x, but it
fails because the netldi reports that "Requested version level 3020, this
server is at level 3022".

Is there an easy way around this?


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] GemTools-1.0-beta.8.7.1-32x can't log in to GS 3.2.4

GLASS mailing list
I seem to have solved it myself by copying libgcirpc-3.2.4-32.so from the
GemStone installation over gciForLinux.so in GemTools.

> I'm trying to log in to GS 3.2.4 using GemTools-1.0-beta.8.7.1-32x, but it
> fails because the netldi reports that "Requested version level 3020, this
> server is at level 3022".
>
> Is there an easy way around this?
>
>
> _______________________________________________
> Glass mailing list
> [hidden email]
> http://lists.gemtalksystems.com/mailman/listinfo/glass
>


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] GemTools-1.0-beta.8.7.1-32x can't log in to GS 3.2.4

GLASS mailing list
Yes that's the ticket ...

On 3/25/15 5:32 AM, Pieter Nagel via Glass wrote:

> I seem to have solved it myself by copying libgcirpc-3.2.4-32.so from the
> GemStone installation over gciForLinux.so in GemTools.
>
>> I'm trying to log in to GS 3.2.4 using GemTools-1.0-beta.8.7.1-32x, but it
>> fails because the netldi reports that "Requested version level 3020, this
>> server is at level 3022".
>>
>> Is there an easy way around this?
>>
>>
>> _______________________________________________
>> Glass mailing list
>> [hidden email]
>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>
>
> _______________________________________________
> Glass mailing list
> [hidden email]
> http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

[Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

       has grown to 22528 megabytes.

    Repository has grown to 22528 megabytes.


    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

       has grown to 22544 megabytes.

    Repository has grown to 22544 megabytes.


--- 03/08/2015 03:31:45 PM UTC ---

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

       has grown to 22560 megabytes.

    Repository has grown to 22560 megabytes.


    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

       has grown to 22576 megabytes.

    Repository has grown to 22576 megabytes.


--- 03/10/2015 03:19:34 AM UTC ---

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

       has grown to 22592 megabytes.

    Repository has grown to 22592 megabytes.


--- 03/10/2015 03:46:39 PM UTC ---

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

       has grown to 22608 megabytes.

    Repository has grown to 22608 megabytes.


    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

       has grown to 22624 megabytes.

    Repository has grown to 22624 megabytes.



My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry



_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list
Lawrence,

Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?

Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...

Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...

The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...

Dale

On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

       has grown to 22528 megabytes.

    Repository has grown to 22528 megabytes.


    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

       has grown to 22544 megabytes.

    Repository has grown to 22544 megabytes.


--- 03/08/2015 03:31:45 PM UTC ---

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

       has grown to 22560 megabytes.

    Repository has grown to 22560 megabytes.


    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

       has grown to 22576 megabytes.

    Repository has grown to 22576 megabytes.


--- 03/10/2015 03:19:34 AM UTC ---

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

       has grown to 22592 megabytes.

    Repository has grown to 22592 megabytes.


--- 03/10/2015 03:46:39 PM UTC ---

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

       has grown to 22608 megabytes.

    Repository has grown to 22608 megabytes.


    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

       has grown to 22624 megabytes.

    Repository has grown to 22624 megabytes.



My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list
Hello Dale, 

  Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL. 

  I’m not running the maintenance VM and I haven’t been doing regular mark for collects. 

  I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup? 

  I’ll also try the ObjectLog trick. 

  I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual. 

  
Best, 

Larry


  
On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Lawrence,

Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?

Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...

Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...

The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...

Dale

On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22528 megabytes.
    Repository has grown to 22528 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22544 megabytes.
    Repository has grown to 22544 megabytes.

--- 03/08/2015 03:31:45 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22560 megabytes.
    Repository has grown to 22560 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22576 megabytes.
    Repository has grown to 22576 megabytes.

--- 03/10/2015 03:19:34 AM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22592 megabytes.
    Repository has grown to 22592 megabytes.

--- 03/10/2015 03:46:39 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22608 megabytes.
    Repository has grown to 22608 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22624 megabytes.
    Repository has grown to 22624 megabytes.


My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list
No problem, the GLASS documentation hasn't been that stellar either:)

Okay with a large db and a relatively small SPC these operations can be very slow... the order of things to do is important ... Hold off on running things until I can give you a rough sequence of things to do ... in a little bit ...

Dale

On 3/25/15 10:18 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL. 

  I’m not running the maintenance VM and I haven’t been doing regular mark for collects. 

  I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup? 

  I’ll also try the ObjectLog trick. 

  I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual. 

  
Best, 

Larry


  
On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Lawrence,

Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?

Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...

Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...

The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...

Dale

On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22528 megabytes.
    Repository has grown to 22528 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22544 megabytes.
    Repository has grown to 22544 megabytes.

--- 03/08/2015 03:31:45 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22560 megabytes.
    Repository has grown to 22560 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22576 megabytes.
    Repository has grown to 22576 megabytes.

--- 03/10/2015 03:19:34 AM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22592 megabytes.
    Repository has grown to 22592 megabytes.

--- 03/10/2015 03:46:39 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22608 megabytes.
    Repository has grown to 22608 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22624 megabytes.
    Repository has grown to 22624 megabytes.


My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass



_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list
In reply to this post by GLASS mailing list
Okay here's the sequence of steps that I think you should take:

  1. expire all of your sessions:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransactions

  2. initalize your object log

  3. run MFC

  [
  System abortTransaction.
  SystemRepository markForCollection ]
    on: Warning
    do: [ :ex |
      Transcript
        cr;
        show: ex description.
      ex resume ]

  4. Then do a backup and restore ... you can use GemTools to do the restore,
      but then you should read the SysAdmin docs[1] for instructions to do the restore
      (I've enclosed link to 3.2 docs, but the procedure and commands should pretty
      much be the same, but it's best to look up the docs for your GemStone version[2]
      and follow those instructions)

As I mentioned earlier, it will probably take a while for each of these operations to complete (object log will be fast and the backup will be fast, if the mfc tosses out the majority of your data) and it is likely that the repository will grow some more during the process (hard to predict this one, tho).

Step 1 will touch every session and every continuation so it is hard to say what percent of the objects are going to be touched (the expensive part), still there are likely to be a lot of those puppies and they will have to be read from disk into the SPC ...

Step 3. is going scan all of the live objects and again it hard to predict exactly how expensive it will be ...

Dale

[1] http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm
[2] http://gemtalksystems.com/techsupport/resources/

On 3/25/15 10:18 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL. 

  I’m not running the maintenance VM and I haven’t been doing regular mark for collects. 

  I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup? 

  I’ll also try the ObjectLog trick. 

  I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual. 

  
Best, 

Larry


  
On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Lawrence,

Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?

Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...

Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...

The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...

Dale

On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22528 megabytes.
    Repository has grown to 22528 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22544 megabytes.
    Repository has grown to 22544 megabytes.

--- 03/08/2015 03:31:45 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22560 megabytes.
    Repository has grown to 22560 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22576 megabytes.
    Repository has grown to 22576 megabytes.

--- 03/10/2015 03:19:34 AM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22592 megabytes.
    Repository has grown to 22592 megabytes.

--- 03/10/2015 03:46:39 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22608 megabytes.
    Repository has grown to 22608 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22624 megabytes.
    Repository has grown to 22624 megabytes.


My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass



_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list
Hello Dale, 

  Well, Step 1 ran for hours and halted in the debugger with this error: 

Error: VM temporary object memory is full 
, almost out of memory, too many markSweeps since last successful scavenge


What do I do now? 

Best, 

  Larry 



On Mar 25, 2015, at 2:15 PM, Dale Henrichs <[hidden email]> wrote:

Okay here's the sequence of steps that I think you should take:

  1. expire all of your sessions:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransactions

  2. initalize your object log

  3. run MFC

  [
  System abortTransaction.
  SystemRepository markForCollection ]
    on: Warning
    do: [ :ex |
      Transcript
        cr;
        show: ex description.
      ex resume ]

  4. Then do a backup and restore ... you can use GemTools to do the restore,
      but then you should read the SysAdmin docs[1] for instructions to do the restore
      (I've enclosed link to 3.2 docs, but the procedure and commands should pretty
      much be the same, but it's best to look up the docs for your GemStone version[2]
      and follow those instructions)

As I mentioned earlier, it will probably take a while for each of these operations to complete (object log will be fast and the backup will be fast, if the mfc tosses out the majority of your data) and it is likely that the repository will grow some more during the process (hard to predict this one, tho).

Step 1 will touch every session and every continuation so it is hard to say what percent of the objects are going to be touched (the expensive part), still there are likely to be a lot of those puppies and they will have to be read from disk into the SPC ...

Step 3. is going scan all of the live objects and again it hard to predict exactly how expensive it will be ...

Dale

[1] http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm
[2] http://gemtalksystems.com/techsupport/resources/

On 3/25/15 10:18 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL. 

  I’m not running the maintenance VM and I haven’t been doing regular mark for collects. 

  I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup? 

  I’ll also try the ObjectLog trick. 

  I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual. 

  
Best, 

Larry


  
On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Lawrence,

Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?

Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...

Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...

The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...

Dale

On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22528 megabytes.
    Repository has grown to 22528 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22544 megabytes.
    Repository has grown to 22544 megabytes.

--- 03/08/2015 03:31:45 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22560 megabytes.
    Repository has grown to 22560 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22576 megabytes.
    Repository has grown to 22576 megabytes.

--- 03/10/2015 03:19:34 AM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22592 megabytes.
    Repository has grown to 22592 megabytes.

--- 03/10/2015 03:46:39 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22608 megabytes.
    Repository has grown to 22608 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22624 megabytes.
    Repository has grown to 22624 megabytes.


My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list
Hmm, I scanned the code in WACache>>gemstoneReap and saw that a commit was being down every 100 entries, but presumably you blew up while building the collection of entries to expire ... sorry about that ... There should be a a stack dump in the gem log (depending upon where you ran the expressions where we can verify my assumption.

It looks like we need to replace WACache>>gemstoneReap with the following:

gemstoneReap
  "Iterate through the cache and remove objects that have expired."

  "In GemStone, this method is performed by a separate maintenance VM,
     so we are already in transaction (assumed to be running in #autoBegin
     transactionMode) and do not have to worry about acquiring the TransactionMutex.
    Since we are using reducedConflict dictionaries in the first place, we will remove the keys
    and values from the existing dictionaries without using the mutex."

  | expired count platform |
  expired := UserGlobals at: #'ExpiryCleanup' put: OrderedCollection new.
  platform := GRPlatform current.
  platform doCommitTransaction.
  count := 0.
  objectsByKey
    associationsDo: [ :assoc |
      (self expiryPolicy isExpiredUpdating: assoc value key: assoc key)
        ifTrue: [
          self notifyRemoved: assoc value key: assoc key.
          count := count + 1.
          expired add: assoc.
          count \\ 100 == 0
            ifTrue: [ platform doCommitTransaction ].
          count \\ 1000 == 0
            ifTrue: [ Transcript cr; show: 'Scan progress: ' , count printString.] ] ].
  Transcript cr; show: 'finished scan: ' , count printString.
  count := 0.
  (UserGlobals at: #'ExpiryCleanup' )
    do: [ :assoc |
      count := count + 1.
      objectsByKey removeKey: assoc key ifAbsent: [].
      keysByObject removeKey: assoc value ifAbsent: [  ].
      count \\ 100 == 0
        ifTrue: [ platform doCommitTransaction ].
      count \\ 1000 == 0
        ifTrue: [ Transcript cr; show: 'Expire progress: ' , count printString.]].
  platform doCommitTransaction.
  UserGlobals removeKey: #'ExpiryCleanup'.
  platform doCommitTransaction.
  ^ expired size

This implementation should be more resistant to an out of memory condition and I've got some logging in  there as well ... the `Transcript show:` should show up in the gem.log and/or Transcript...

I haven't tested this but if there's a problem in the scan loop it will fail quickly. If there's a problem in the expire loop, you can skip the scan loop, and just run the expire loop ...

Sorry about that ... hopefully the second time will be the charm ...

Dale

On 3/25/15 2:36 PM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, Step 1 ran for hours and halted in the debugger with this error: 

Error: VM temporary object memory is full 
, almost out of memory, too many markSweeps since last successful scavenge


What do I do now? 

Best, 

  Larry 



On Mar 25, 2015, at 2:15 PM, Dale Henrichs <[hidden email]> wrote:

Okay here's the sequence of steps that I think you should take:

  1. expire all of your sessions:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransactions

  2. initalize your object log

  3. run MFC

  [
  System abortTransaction.
  SystemRepository markForCollection ]
    on: Warning
    do: [ :ex |
      Transcript
        cr;
        show: ex description.
      ex resume ]

  4. Then do a backup and restore ... you can use GemTools to do the restore,
      but then you should read the SysAdmin docs[1] for instructions to do the restore
      (I've enclosed link to 3.2 docs, but the procedure and commands should pretty
      much be the same, but it's best to look up the docs for your GemStone version[2]
      and follow those instructions)

As I mentioned earlier, it will probably take a while for each of these operations to complete (object log will be fast and the backup will be fast, if the mfc tosses out the majority of your data) and it is likely that the repository will grow some more during the process (hard to predict this one, tho).

Step 1 will touch every session and every continuation so it is hard to say what percent of the objects are going to be touched (the expensive part), still there are likely to be a lot of those puppies and they will have to be read from disk into the SPC ...

Step 3. is going scan all of the live objects and again it hard to predict exactly how expensive it will be ...

Dale

[1] http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm
[2] http://gemtalksystems.com/techsupport/resources/

On 3/25/15 10:18 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL. 

  I’m not running the maintenance VM and I haven’t been doing regular mark for collects. 

  I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup? 

  I’ll also try the ObjectLog trick. 

  I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual. 

  
Best, 

Larry


  
On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Lawrence,

Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?

Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...

Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...

The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...

Dale

On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22528 megabytes.
    Repository has grown to 22528 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22544 megabytes.
    Repository has grown to 22544 megabytes.

--- 03/08/2015 03:31:45 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22560 megabytes.
    Repository has grown to 22560 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22576 megabytes.
    Repository has grown to 22576 megabytes.

--- 03/10/2015 03:19:34 AM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22592 megabytes.
    Repository has grown to 22592 megabytes.

--- 03/10/2015 03:46:39 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22608 megabytes.
    Repository has grown to 22608 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22624 megabytes.
    Repository has grown to 22624 megabytes.


My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass





_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list
Hello Dale, 

  Well, I replaced the WACache>>gemstoneReap method and ran the code from before but it just shows Scan Progress: 0 over and over again. 

  I let it run a few hours but had to kill it as the computer is in my bedroom and I can’t sleep if it makes noise all night. 

  Do I try again tomorrow? 

  best, 

  Larry

 
On Mar 25, 2015, at 6:23 PM, Dale Henrichs <[hidden email]> wrote:

Hmm, I scanned the code in WACache>>gemstoneReap and saw that a commit was being down every 100 entries, but presumably you blew up while building the collection of entries to expire ... sorry about that ... There should be a a stack dump in the gem log (depending upon where you ran the expressions where we can verify my assumption.

It looks like we need to replace WACache>>gemstoneReap with the following:

gemstoneReap
  "Iterate through the cache and remove objects that have expired."

  "In GemStone, this method is performed by a separate maintenance VM,
     so we are already in transaction (assumed to be running in #autoBegin
     transactionMode) and do not have to worry about acquiring the TransactionMutex.
    Since we are using reducedConflict dictionaries in the first place, we will remove the keys
    and values from the existing dictionaries without using the mutex."

  | expired count platform |
  expired := UserGlobals at: #'ExpiryCleanup' put: OrderedCollection new.
  platform := GRPlatform current.
  platform doCommitTransaction.
  count := 0.
  objectsByKey
    associationsDo: [ :assoc |
      (self expiryPolicy isExpiredUpdating: assoc value key: assoc key)
        ifTrue: [
          self notifyRemoved: assoc value key: assoc key.
          count := count + 1.
          expired add: assoc.
          count \\ 100 == 0
            ifTrue: [ platform doCommitTransaction ].
          count \\ 1000 == 0
            ifTrue: [ Transcript cr; show: 'Scan progress: ' , count printString.] ] ].
  Transcript cr; show: 'finished scan: ' , count printString.
  count := 0.
  (UserGlobals at: #'ExpiryCleanup' )
    do: [ :assoc |
      count := count + 1.
      objectsByKey removeKey: assoc key ifAbsent: [].
      keysByObject removeKey: assoc value ifAbsent: [  ].
      count \\ 100 == 0
        ifTrue: [ platform doCommitTransaction ].
      count \\ 1000 == 0
        ifTrue: [ Transcript cr; show: 'Expire progress: ' , count printString.]].
  platform doCommitTransaction.
  UserGlobals removeKey: #'ExpiryCleanup'.
  platform doCommitTransaction.
  ^ expired size

This implementation should be more resistant to an out of memory condition and I've got some logging in  there as well ... the `Transcript show:` should show up in the gem.log and/or Transcript...

I haven't tested this but if there's a problem in the scan loop it will fail quickly. If there's a problem in the expire loop, you can skip the scan loop, and just run the expire loop ...

Sorry about that ... hopefully the second time will be the charm ...

Dale

On 3/25/15 2:36 PM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, Step 1 ran for hours and halted in the debugger with this error: 

Error: VM temporary object memory is full 
, almost out of memory, too many markSweeps since last successful scavenge


What do I do now? 

Best, 

  Larry 



On Mar 25, 2015, at 2:15 PM, Dale Henrichs <[hidden email]> wrote:

Okay here's the sequence of steps that I think you should take:

  1. expire all of your sessions:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransactions

  2. initalize your object log

  3. run MFC

  [
  System abortTransaction.
  SystemRepository markForCollection ]
    on: Warning
    do: [ :ex |
      Transcript
        cr;
        show: ex description.
      ex resume ]

  4. Then do a backup and restore ... you can use GemTools to do the restore,
      but then you should read the SysAdmin docs[1] for instructions to do the restore
      (I've enclosed link to 3.2 docs, but the procedure and commands should pretty
      much be the same, but it's best to look up the docs for your GemStone version[2]
      and follow those instructions)

As I mentioned earlier, it will probably take a while for each of these operations to complete (object log will be fast and the backup will be fast, if the mfc tosses out the majority of your data) and it is likely that the repository will grow some more during the process (hard to predict this one, tho).

Step 1 will touch every session and every continuation so it is hard to say what percent of the objects are going to be touched (the expensive part), still there are likely to be a lot of those puppies and they will have to be read from disk into the SPC ...

Step 3. is going scan all of the live objects and again it hard to predict exactly how expensive it will be ...

Dale

[1] http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm
[2] http://gemtalksystems.com/techsupport/resources/

On 3/25/15 10:18 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL. 

  I’m not running the maintenance VM and I haven’t been doing regular mark for collects. 

  I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup? 

  I’ll also try the ObjectLog trick. 

  I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual. 

  
Best, 

Larry


  
On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Lawrence,

Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?

Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...

Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...

The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...

Dale

On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22528 megabytes.
    Repository has grown to 22528 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22544 megabytes.
    Repository has grown to 22544 megabytes.

--- 03/08/2015 03:31:45 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22560 megabytes.
    Repository has grown to 22560 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22576 megabytes.
    Repository has grown to 22576 megabytes.

--- 03/10/2015 03:19:34 AM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22592 megabytes.
    Repository has grown to 22592 megabytes.

--- 03/10/2015 03:46:39 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22608 megabytes.
    Repository has grown to 22608 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22624 megabytes.
    Repository has grown to 22624 megabytes.


My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass






_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list
Hmmm, I think we need to set a breakpoint in there ... I might be tired, but I can't see how that can happen ... but then that's what a debugger is for ...

Are you running this expression from GemTools? If so the `Transcript show:` might be awfully expensive (lot's o round trips over the WAN) ... you might use GsFile gciLogServer: and then tail the gem log file on the server to see progress ...

Need to find out where the 0 is coming from first...

Dale

On 3/25/15 7:34 PM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, I replaced the WACache>>gemstoneReap method and ran the code from before but it just shows Scan Progress: 0 over and over again. 

  I let it run a few hours but had to kill it as the computer is in my bedroom and I can’t sleep if it makes noise all night. 

  Do I try again tomorrow? 

  best, 

  Larry

 
On Mar 25, 2015, at 6:23 PM, Dale Henrichs <[hidden email]> wrote:

Hmm, I scanned the code in WACache>>gemstoneReap and saw that a commit was being down every 100 entries, but presumably you blew up while building the collection of entries to expire ... sorry about that ... There should be a a stack dump in the gem log (depending upon where you ran the expressions where we can verify my assumption.

It looks like we need to replace WACache>>gemstoneReap with the following:

gemstoneReap
  "Iterate through the cache and remove objects that have expired."

  "In GemStone, this method is performed by a separate maintenance VM,
     so we are already in transaction (assumed to be running in #autoBegin
     transactionMode) and do not have to worry about acquiring the TransactionMutex.
    Since we are using reducedConflict dictionaries in the first place, we will remove the keys
    and values from the existing dictionaries without using the mutex."

  | expired count platform |
  expired := UserGlobals at: #'ExpiryCleanup' put: OrderedCollection new.
  platform := GRPlatform current.
  platform doCommitTransaction.
  count := 0.
  objectsByKey
    associationsDo: [ :assoc |
      (self expiryPolicy isExpiredUpdating: assoc value key: assoc key)
        ifTrue: [
          self notifyRemoved: assoc value key: assoc key.
          count := count + 1.
          expired add: assoc.
          count \\ 100 == 0
            ifTrue: [ platform doCommitTransaction ].
          count \\ 1000 == 0
            ifTrue: [ Transcript cr; show: 'Scan progress: ' , count printString.] ] ].
  Transcript cr; show: 'finished scan: ' , count printString.
  count := 0.
  (UserGlobals at: #'ExpiryCleanup' )
    do: [ :assoc |
      count := count + 1.
      objectsByKey removeKey: assoc key ifAbsent: [].
      keysByObject removeKey: assoc value ifAbsent: [  ].
      count \\ 100 == 0
        ifTrue: [ platform doCommitTransaction ].
      count \\ 1000 == 0
        ifTrue: [ Transcript cr; show: 'Expire progress: ' , count printString.]].
  platform doCommitTransaction.
  UserGlobals removeKey: #'ExpiryCleanup'.
  platform doCommitTransaction.
  ^ expired size

This implementation should be more resistant to an out of memory condition and I've got some logging in  there as well ... the `Transcript show:` should show up in the gem.log and/or Transcript...

I haven't tested this but if there's a problem in the scan loop it will fail quickly. If there's a problem in the expire loop, you can skip the scan loop, and just run the expire loop ...

Sorry about that ... hopefully the second time will be the charm ...

Dale

On 3/25/15 2:36 PM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, Step 1 ran for hours and halted in the debugger with this error: 

Error: VM temporary object memory is full 
, almost out of memory, too many markSweeps since last successful scavenge


What do I do now? 

Best, 

  Larry 



On Mar 25, 2015, at 2:15 PM, Dale Henrichs <[hidden email]> wrote:

Okay here's the sequence of steps that I think you should take:

  1. expire all of your sessions:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransactions

  2. initalize your object log

  3. run MFC

  [
  System abortTransaction.
  SystemRepository markForCollection ]
    on: Warning
    do: [ :ex |
      Transcript
        cr;
        show: ex description.
      ex resume ]

  4. Then do a backup and restore ... you can use GemTools to do the restore,
      but then you should read the SysAdmin docs[1] for instructions to do the restore
      (I've enclosed link to 3.2 docs, but the procedure and commands should pretty
      much be the same, but it's best to look up the docs for your GemStone version[2]
      and follow those instructions)

As I mentioned earlier, it will probably take a while for each of these operations to complete (object log will be fast and the backup will be fast, if the mfc tosses out the majority of your data) and it is likely that the repository will grow some more during the process (hard to predict this one, tho).

Step 1 will touch every session and every continuation so it is hard to say what percent of the objects are going to be touched (the expensive part), still there are likely to be a lot of those puppies and they will have to be read from disk into the SPC ...

Step 3. is going scan all of the live objects and again it hard to predict exactly how expensive it will be ...

Dale

[1] http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm
[2] http://gemtalksystems.com/techsupport/resources/

On 3/25/15 10:18 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL. 

  I’m not running the maintenance VM and I haven’t been doing regular mark for collects. 

  I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup? 

  I’ll also try the ObjectLog trick. 

  I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual. 

  
Best, 

Larry


  
On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Lawrence,

Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?

Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...

Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...

The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...

Dale

On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22528 megabytes.
    Repository has grown to 22528 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22544 megabytes.
    Repository has grown to 22544 megabytes.

--- 03/08/2015 03:31:45 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22560 megabytes.
    Repository has grown to 22560 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22576 megabytes.
    Repository has grown to 22576 megabytes.

--- 03/10/2015 03:19:34 AM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22592 megabytes.
    Repository has grown to 22592 megabytes.

--- 03/10/2015 03:46:39 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22608 megabytes.
    Repository has grown to 22608 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22624 megabytes.
    Repository has grown to 22624 megabytes.


My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass







_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list
Well, I put in Transcript calls to show count all the time, not just at mod 1000..and I made it all the way through, except it does’t seem to want to come back to me. The image is frozen.

Is it hung in the final commit? 

Here is a screen shot of the last part of the display. Yes I’m running it in GemTools. I don’t care if it’s slow as long as it works. 



Larry

On Mar 26, 2015, at 1:01 AM, Dale Henrichs <[hidden email]> wrote:

Hmmm, I think we need to set a breakpoint in there ... I might be tired, but I can't see how that can happen ... but then that's what a debugger is for ...

Are you running this expression from GemTools? If so the `Transcript show:` might be awfully expensive (lot's o round trips over the WAN) ... you might use GsFile gciLogServer: and then tail the gem log file on the server to see progress ...

Need to find out where the 0 is coming from first...

Dale

On 3/25/15 7:34 PM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, I replaced the WACache>>gemstoneReap method and ran the code from before but it just shows Scan Progress: 0 over and over again. 

  I let it run a few hours but had to kill it as the computer is in my bedroom and I can’t sleep if it makes noise all night. 

  Do I try again tomorrow? 

  best, 

  Larry

 
On Mar 25, 2015, at 6:23 PM, Dale Henrichs <[hidden email]> wrote:

Hmm, I scanned the code in WACache>>gemstoneReap and saw that a commit was being down every 100 entries, but presumably you blew up while building the collection of entries to expire ... sorry about that ... There should be a a stack dump in the gem log (depending upon where you ran the expressions where we can verify my assumption.

It looks like we need to replace WACache>>gemstoneReap with the following:

gemstoneReap
  "Iterate through the cache and remove objects that have expired."

  "In GemStone, this method is performed by a separate maintenance VM,
     so we are already in transaction (assumed to be running in #autoBegin
     transactionMode) and do not have to worry about acquiring the TransactionMutex.
    Since we are using reducedConflict dictionaries in the first place, we will remove the keys
    and values from the existing dictionaries without using the mutex."

  | expired count platform |
  expired := UserGlobals at: #'ExpiryCleanup' put: OrderedCollection new.
  platform := GRPlatform current.
  platform doCommitTransaction.
  count := 0.
  objectsByKey
    associationsDo: [ :assoc |
      (self expiryPolicy isExpiredUpdating: assoc value key: assoc key)
        ifTrue: [
          self notifyRemoved: assoc value key: assoc key.
          count := count + 1.
          expired add: assoc.
          count \\ 100 == 0
            ifTrue: [ platform doCommitTransaction ].
          count \\ 1000 == 0
            ifTrue: [ Transcript cr; show: 'Scan progress: ' , count printString.] ] ].
  Transcript cr; show: 'finished scan: ' , count printString.
  count := 0.
  (UserGlobals at: #'ExpiryCleanup' )
    do: [ :assoc |
      count := count + 1.
      objectsByKey removeKey: assoc key ifAbsent: [].
      keysByObject removeKey: assoc value ifAbsent: [  ].
      count \\ 100 == 0
        ifTrue: [ platform doCommitTransaction ].
      count \\ 1000 == 0
        ifTrue: [ Transcript cr; show: 'Expire progress: ' , count printString.]].
  platform doCommitTransaction.
  UserGlobals removeKey: #'ExpiryCleanup'.
  platform doCommitTransaction.
  ^ expired size

This implementation should be more resistant to an out of memory condition and I've got some logging in  there as well ... the `Transcript show:` should show up in the gem.log and/or Transcript...

I haven't tested this but if there's a problem in the scan loop it will fail quickly. If there's a problem in the expire loop, you can skip the scan loop, and just run the expire loop ...

Sorry about that ... hopefully the second time will be the charm ...

Dale

On 3/25/15 2:36 PM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, Step 1 ran for hours and halted in the debugger with this error: 

Error: VM temporary object memory is full 
, almost out of memory, too many markSweeps since last successful scavenge


What do I do now? 

Best, 

  Larry 



On Mar 25, 2015, at 2:15 PM, Dale Henrichs <[hidden email]> wrote:

Okay here's the sequence of steps that I think you should take:

  1. expire all of your sessions:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransactions

  2. initalize your object log

  3. run MFC

  [
  System abortTransaction.
  SystemRepository markForCollection ]
    on: Warning
    do: [ :ex |
      Transcript
        cr;
        show: ex description.
      ex resume ]

  4. Then do a backup and restore ... you can use GemTools to do the restore,
      but then you should read the SysAdmin docs[1] for instructions to do the restore
      (I've enclosed link to 3.2 docs, but the procedure and commands should pretty
      much be the same, but it's best to look up the docs for your GemStone version[2]
      and follow those instructions)

As I mentioned earlier, it will probably take a while for each of these operations to complete (object log will be fast and the backup will be fast, if the mfc tosses out the majority of your data) and it is likely that the repository will grow some more during the process (hard to predict this one, tho).

Step 1 will touch every session and every continuation so it is hard to say what percent of the objects are going to be touched (the expensive part), still there are likely to be a lot of those puppies and they will have to be read from disk into the SPC ...

Step 3. is going scan all of the live objects and again it hard to predict exactly how expensive it will be ...

Dale

[1] http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm
[2] http://gemtalksystems.com/techsupport/resources/

On 3/25/15 10:18 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL. 

  I’m not running the maintenance VM and I haven’t been doing regular mark for collects. 

  I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup? 

  I’ll also try the ObjectLog trick. 

  I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual. 

  
Best, 

Larry


  
On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Lawrence,

Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?

Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...

Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...

The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...

Dale

On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22528 megabytes.
    Repository has grown to 22528 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22544 megabytes.
    Repository has grown to 22544 megabytes.

--- 03/08/2015 03:31:45 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22560 megabytes.
    Repository has grown to 22560 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22576 megabytes.
    Repository has grown to 22576 megabytes.

--- 03/10/2015 03:19:34 AM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22592 megabytes.
    Repository has grown to 22592 megabytes.

--- 03/10/2015 03:46:39 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22608 megabytes.
    Repository has grown to 22608 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22624 megabytes.
    Repository has grown to 22624 megabytes.


My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass








_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list
Larry,

Not sure ... I guess the I'm curious if the system is in iowait or burning cpu ... top should give you that information ... I'd also be interested to know if you've run out of disk space ... check the stone log for that information ...

Based on what you find out we'll go on from there ...

Dale

On 03/27/2015 07:43 AM, Lawrence Kellogg wrote:
Well, I put in Transcript calls to show count all the time, not just at mod 1000..and I made it all the way through, except it does’t seem to want to come back to me. The image is frozen.

Is it hung in the final commit? 

Here is a screen shot of the last part of the display. Yes I’m running it in GemTools. I don’t care if it’s slow as long as it works. 



Larry

On Mar 26, 2015, at 1:01 AM, Dale Henrichs <[hidden email]> wrote:

Hmmm, I think we need to set a breakpoint in there ... I might be tired, but I can't see how that can happen ... but then that's what a debugger is for ...

Are you running this expression from GemTools? If so the `Transcript show:` might be awfully expensive (lot's o round trips over the WAN) ... you might use GsFile gciLogServer: and then tail the gem log file on the server to see progress ...

Need to find out where the 0 is coming from first...

Dale

On 3/25/15 7:34 PM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, I replaced the WACache>>gemstoneReap method and ran the code from before but it just shows Scan Progress: 0 over and over again. 

  I let it run a few hours but had to kill it as the computer is in my bedroom and I can’t sleep if it makes noise all night. 

  Do I try again tomorrow? 

  best, 

  Larry

 
On Mar 25, 2015, at 6:23 PM, Dale Henrichs <[hidden email]> wrote:

Hmm, I scanned the code in WACache>>gemstoneReap and saw that a commit was being down every 100 entries, but presumably you blew up while building the collection of entries to expire ... sorry about that ... There should be a a stack dump in the gem log (depending upon where you ran the expressions where we can verify my assumption.

It looks like we need to replace WACache>>gemstoneReap with the following:

gemstoneReap
  "Iterate through the cache and remove objects that have expired."

  "In GemStone, this method is performed by a separate maintenance VM,
     so we are already in transaction (assumed to be running in #autoBegin
     transactionMode) and do not have to worry about acquiring the TransactionMutex.
    Since we are using reducedConflict dictionaries in the first place, we will remove the keys
    and values from the existing dictionaries without using the mutex."

  | expired count platform |
  expired := UserGlobals at: #'ExpiryCleanup' put: OrderedCollection new.
  platform := GRPlatform current.
  platform doCommitTransaction.
  count := 0.
  objectsByKey
    associationsDo: [ :assoc |
      (self expiryPolicy isExpiredUpdating: assoc value key: assoc key)
        ifTrue: [
          self notifyRemoved: assoc value key: assoc key.
          count := count + 1.
          expired add: assoc.
          count \\ 100 == 0
            ifTrue: [ platform doCommitTransaction ].
          count \\ 1000 == 0
            ifTrue: [ Transcript cr; show: 'Scan progress: ' , count printString.] ] ].
  Transcript cr; show: 'finished scan: ' , count printString.
  count := 0.
  (UserGlobals at: #'ExpiryCleanup' )
    do: [ :assoc |
      count := count + 1.
      objectsByKey removeKey: assoc key ifAbsent: [].
      keysByObject removeKey: assoc value ifAbsent: [  ].
      count \\ 100 == 0
        ifTrue: [ platform doCommitTransaction ].
      count \\ 1000 == 0
        ifTrue: [ Transcript cr; show: 'Expire progress: ' , count printString.]].
  platform doCommitTransaction.
  UserGlobals removeKey: #'ExpiryCleanup'.
  platform doCommitTransaction.
  ^ expired size

This implementation should be more resistant to an out of memory condition and I've got some logging in  there as well ... the `Transcript show:` should show up in the gem.log and/or Transcript...

I haven't tested this but if there's a problem in the scan loop it will fail quickly. If there's a problem in the expire loop, you can skip the scan loop, and just run the expire loop ...

Sorry about that ... hopefully the second time will be the charm ...

Dale

On 3/25/15 2:36 PM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, Step 1 ran for hours and halted in the debugger with this error: 

Error: VM temporary object memory is full 
, almost out of memory, too many markSweeps since last successful scavenge


What do I do now? 

Best, 

  Larry 



On Mar 25, 2015, at 2:15 PM, Dale Henrichs <[hidden email]> wrote:

Okay here's the sequence of steps that I think you should take:

  1. expire all of your sessions:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransactions

  2. initalize your object log

  3. run MFC

  [
  System abortTransaction.
  SystemRepository markForCollection ]
    on: Warning
    do: [ :ex |
      Transcript
        cr;
        show: ex description.
      ex resume ]

  4. Then do a backup and restore ... you can use GemTools to do the restore,
      but then you should read the SysAdmin docs[1] for instructions to do the restore
      (I've enclosed link to 3.2 docs, but the procedure and commands should pretty
      much be the same, but it's best to look up the docs for your GemStone version[2]
      and follow those instructions)

As I mentioned earlier, it will probably take a while for each of these operations to complete (object log will be fast and the backup will be fast, if the mfc tosses out the majority of your data) and it is likely that the repository will grow some more during the process (hard to predict this one, tho).

Step 1 will touch every session and every continuation so it is hard to say what percent of the objects are going to be touched (the expensive part), still there are likely to be a lot of those puppies and they will have to be read from disk into the SPC ...

Step 3. is going scan all of the live objects and again it hard to predict exactly how expensive it will be ...

Dale

[1] http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm
[2] http://gemtalksystems.com/techsupport/resources/

On 3/25/15 10:18 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL. 

  I’m not running the maintenance VM and I haven’t been doing regular mark for collects. 

  I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup? 

  I’ll also try the ObjectLog trick. 

  I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual. 

  
Best, 

Larry


  
On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Lawrence,

Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?

Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...

Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...

The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...

Dale

On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22528 megabytes.
    Repository has grown to 22528 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22544 megabytes.
    Repository has grown to 22544 megabytes.

--- 03/08/2015 03:31:45 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22560 megabytes.
    Repository has grown to 22560 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22576 megabytes.
    Repository has grown to 22576 megabytes.

--- 03/10/2015 03:19:34 AM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22592 megabytes.
    Repository has grown to 22592 megabytes.

--- 03/10/2015 03:46:39 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22608 megabytes.
    Repository has grown to 22608 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22624 megabytes.
    Repository has grown to 22624 megabytes.


My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass









_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list

On Mar 27, 2015, at 12:08 PM, Dale Henrichs <[hidden email]> wrote:

Larry,

Not sure ... I guess the I'm curious if the system is in iowait or burning cpu ... top should give you that information ... I'd also be interested to know if you've run out of disk space ... check the stone log for that information ...

Based on what you find out we'll go on from there ...



The struggle continues, eventually, I ran out of temporary Object Memory:



Disk space as at 63%. Can I run it again? Did any of that stick?

Larry





Dale

On 03/27/2015 07:43 AM, Lawrence Kellogg wrote:
Well, I put in Transcript calls to show count all the time, not just at mod 1000..and I made it all the way through, except it does’t seem to want to come back to me. The image is frozen.

Is it hung in the final commit? 

Here is a screen shot of the last part of the display. Yes I’m running it in GemTools. I don’t care if it’s slow as long as it works. 

<Mail Attachment.png>


Larry

On Mar 26, 2015, at 1:01 AM, Dale Henrichs <[hidden email]> wrote:

Hmmm, I think we need to set a breakpoint in there ... I might be tired, but I can't see how that can happen ... but then that's what a debugger is for ...

Are you running this expression from GemTools? If so the `Transcript show:` might be awfully expensive (lot's o round trips over the WAN) ... you might use GsFile gciLogServer: and then tail the gem log file on the server to see progress ...

Need to find out where the 0 is coming from first...

Dale

On 3/25/15 7:34 PM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, I replaced the WACache>>gemstoneReap method and ran the code from before but it just shows Scan Progress: 0 over and over again. 

  I let it run a few hours but had to kill it as the computer is in my bedroom and I can’t sleep if it makes noise all night. 

  Do I try again tomorrow? 

  best, 

  Larry

 
On Mar 25, 2015, at 6:23 PM, Dale Henrichs <[hidden email]> wrote:

Hmm, I scanned the code in WACache>>gemstoneReap and saw that a commit was being down every 100 entries, but presumably you blew up while building the collection of entries to expire ... sorry about that ... There should be a a stack dump in the gem log (depending upon where you ran the expressions where we can verify my assumption.

It looks like we need to replace WACache>>gemstoneReap with the following:

gemstoneReap
  "Iterate through the cache and remove objects that have expired."

  "In GemStone, this method is performed by a separate maintenance VM,
     so we are already in transaction (assumed to be running in #autoBegin
     transactionMode) and do not have to worry about acquiring the TransactionMutex.
    Since we are using reducedConflict dictionaries in the first place, we will remove the keys
    and values from the existing dictionaries without using the mutex."

  | expired count platform |
  expired := UserGlobals at: #'ExpiryCleanup' put: OrderedCollection new.
  platform := GRPlatform current.
  platform doCommitTransaction.
  count := 0.
  objectsByKey
    associationsDo: [ :assoc |
      (self expiryPolicy isExpiredUpdating: assoc value key: assoc key)
        ifTrue: [
          self notifyRemoved: assoc value key: assoc key.
          count := count + 1.
          expired add: assoc.
          count \\ 100 == 0
            ifTrue: [ platform doCommitTransaction ].
          count \\ 1000 == 0
            ifTrue: [ Transcript cr; show: 'Scan progress: ' , count printString.] ] ].
  Transcript cr; show: 'finished scan: ' , count printString.
  count := 0.
  (UserGlobals at: #'ExpiryCleanup' )
    do: [ :assoc |
      count := count + 1.
      objectsByKey removeKey: assoc key ifAbsent: [].
      keysByObject removeKey: assoc value ifAbsent: [  ].
      count \\ 100 == 0
        ifTrue: [ platform doCommitTransaction ].
      count \\ 1000 == 0
        ifTrue: [ Transcript cr; show: 'Expire progress: ' , count printString.]].
  platform doCommitTransaction.
  UserGlobals removeKey: #'ExpiryCleanup'.
  platform doCommitTransaction.
  ^ expired size

This implementation should be more resistant to an out of memory condition and I've got some logging in  there as well ... the `Transcript show:` should show up in the gem.log and/or Transcript...

I haven't tested this but if there's a problem in the scan loop it will fail quickly. If there's a problem in the expire loop, you can skip the scan loop, and just run the expire loop ...

Sorry about that ... hopefully the second time will be the charm ...

Dale

On 3/25/15 2:36 PM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, Step 1 ran for hours and halted in the debugger with this error: 

Error: VM temporary object memory is full 
, almost out of memory, too many markSweeps since last successful scavenge


What do I do now? 

Best, 

  Larry 



On Mar 25, 2015, at 2:15 PM, Dale Henrichs <[hidden email]> wrote:

Okay here's the sequence of steps that I think you should take:

  1. expire all of your sessions:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransactions

  2. initalize your object log

  3. run MFC

  [
  System abortTransaction.
  SystemRepository markForCollection ]
    on: Warning
    do: [ :ex |
      Transcript
        cr;
        show: ex description.
      ex resume ]

  4. Then do a backup and restore ... you can use GemTools to do the restore,
      but then you should read the SysAdmin docs[1] for instructions to do the restore
      (I've enclosed link to 3.2 docs, but the procedure and commands should pretty
      much be the same, but it's best to look up the docs for your GemStone version[2]
      and follow those instructions)

As I mentioned earlier, it will probably take a while for each of these operations to complete (object log will be fast and the backup will be fast, if the mfc tosses out the majority of your data) and it is likely that the repository will grow some more during the process (hard to predict this one, tho).

Step 1 will touch every session and every continuation so it is hard to say what percent of the objects are going to be touched (the expensive part), still there are likely to be a lot of those puppies and they will have to be read from disk into the SPC ...

Step 3. is going scan all of the live objects and again it hard to predict exactly how expensive it will be ...

Dale

[1] http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm
[2] http://gemtalksystems.com/techsupport/resources/

On 3/25/15 10:18 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL. 

  I’m not running the maintenance VM and I haven’t been doing regular mark for collects. 

  I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup? 

  I’ll also try the ObjectLog trick. 

  I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual. 

  
Best, 

Larry


  
On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Lawrence,

Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?

Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...

Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...

The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...

Dale

On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22528 megabytes.
    Repository has grown to 22528 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22544 megabytes.
    Repository has grown to 22544 megabytes.

--- 03/08/2015 03:31:45 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22560 megabytes.
    Repository has grown to 22560 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22576 megabytes.
    Repository has grown to 22576 megabytes.

--- 03/10/2015 03:19:34 AM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22592 megabytes.
    Repository has grown to 22592 megabytes.

--- 03/10/2015 03:46:39 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22608 megabytes.
    Repository has grown to 22608 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22624 megabytes.
    Repository has grown to 22624 megabytes.


My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass










_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list
Larry,

Yea the bulk of the progress that you've made should be committed... should probably add a final transcript show in the gemstoneReap method ... but when your `Expire Progress` matches the preceding `finished scan` that chunk of work won't be done again ...

At this point in time, I suggest that you make a backup in case we hit a problem (like run out of extent space or something) that would make things difficult to proceed from ... this is just precautionary, but prudent ...

I am interested in what caused the out of memory so checking in the gem log for a smalltalk stack when you ran out of memory would help ... it's possible that we'll just hit the same situation right off the bat, when we start process again, so might as well try to understand that one...

Dale

On 03/27/2015 09:31 AM, Lawrence Kellogg wrote:

On Mar 27, 2015, at 12:08 PM, Dale Henrichs <[hidden email]> wrote:

Larry,

Not sure ... I guess the I'm curious if the system is in iowait or burning cpu ... top should give you that information ... I'd also be interested to know if you've run out of disk space ... check the stone log for that information ...

Based on what you find out we'll go on from there ...



The struggle continues, eventually, I ran out of temporary Object Memory:



Disk space as at 63%. Can I run it again? Did any of that stick?

Larry





Dale

On 03/27/2015 07:43 AM, Lawrence Kellogg wrote:
Well, I put in Transcript calls to show count all the time, not just at mod 1000..and I made it all the way through, except it does’t seem to want to come back to me. The image is frozen.

Is it hung in the final commit? 

Here is a screen shot of the last part of the display. Yes I’m running it in GemTools. I don’t care if it’s slow as long as it works. 

<Mail Attachment.png>


Larry

On Mar 26, 2015, at 1:01 AM, Dale Henrichs <[hidden email]> wrote:

Hmmm, I think we need to set a breakpoint in there ... I might be tired, but I can't see how that can happen ... but then that's what a debugger is for ...

Are you running this expression from GemTools? If so the `Transcript show:` might be awfully expensive (lot's o round trips over the WAN) ... you might use GsFile gciLogServer: and then tail the gem log file on the server to see progress ...

Need to find out where the 0 is coming from first...

Dale

On 3/25/15 7:34 PM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, I replaced the WACache>>gemstoneReap method and ran the code from before but it just shows Scan Progress: 0 over and over again. 

  I let it run a few hours but had to kill it as the computer is in my bedroom and I can’t sleep if it makes noise all night. 

  Do I try again tomorrow? 

  best, 

  Larry

 
On Mar 25, 2015, at 6:23 PM, Dale Henrichs <[hidden email]> wrote:

Hmm, I scanned the code in WACache>>gemstoneReap and saw that a commit was being down every 100 entries, but presumably you blew up while building the collection of entries to expire ... sorry about that ... There should be a a stack dump in the gem log (depending upon where you ran the expressions where we can verify my assumption.

It looks like we need to replace WACache>>gemstoneReap with the following:

gemstoneReap
  "Iterate through the cache and remove objects that have expired."

  "In GemStone, this method is performed by a separate maintenance VM,
     so we are already in transaction (assumed to be running in #autoBegin
     transactionMode) and do not have to worry about acquiring the TransactionMutex.
    Since we are using reducedConflict dictionaries in the first place, we will remove the keys
    and values from the existing dictionaries without using the mutex."

  | expired count platform |
  expired := UserGlobals at: #'ExpiryCleanup' put: OrderedCollection new.
  platform := GRPlatform current.
  platform doCommitTransaction.
  count := 0.
  objectsByKey
    associationsDo: [ :assoc |
      (self expiryPolicy isExpiredUpdating: assoc value key: assoc key)
        ifTrue: [
          self notifyRemoved: assoc value key: assoc key.
          count := count + 1.
          expired add: assoc.
          count \\ 100 == 0
            ifTrue: [ platform doCommitTransaction ].
          count \\ 1000 == 0
            ifTrue: [ Transcript cr; show: 'Scan progress: ' , count printString.] ] ].
  Transcript cr; show: 'finished scan: ' , count printString.
  count := 0.
  (UserGlobals at: #'ExpiryCleanup' )
    do: [ :assoc |
      count := count + 1.
      objectsByKey removeKey: assoc key ifAbsent: [].
      keysByObject removeKey: assoc value ifAbsent: [  ].
      count \\ 100 == 0
        ifTrue: [ platform doCommitTransaction ].
      count \\ 1000 == 0
        ifTrue: [ Transcript cr; show: 'Expire progress: ' , count printString.]].
  platform doCommitTransaction.
  UserGlobals removeKey: #'ExpiryCleanup'.
  platform doCommitTransaction.
  ^ expired size

This implementation should be more resistant to an out of memory condition and I've got some logging in  there as well ... the `Transcript show:` should show up in the gem.log and/or Transcript...

I haven't tested this but if there's a problem in the scan loop it will fail quickly. If there's a problem in the expire loop, you can skip the scan loop, and just run the expire loop ...

Sorry about that ... hopefully the second time will be the charm ...

Dale

On 3/25/15 2:36 PM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, Step 1 ran for hours and halted in the debugger with this error: 

Error: VM temporary object memory is full 
, almost out of memory, too many markSweeps since last successful scavenge


What do I do now? 

Best, 

  Larry 



On Mar 25, 2015, at 2:15 PM, Dale Henrichs <[hidden email]> wrote:

Okay here's the sequence of steps that I think you should take:

  1. expire all of your sessions:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransactions

  2. initalize your object log

  3. run MFC

  [
  System abortTransaction.
  SystemRepository markForCollection ]
    on: Warning
    do: [ :ex |
      Transcript
        cr;
        show: ex description.
      ex resume ]

  4. Then do a backup and restore ... you can use GemTools to do the restore,
      but then you should read the SysAdmin docs[1] for instructions to do the restore
      (I've enclosed link to 3.2 docs, but the procedure and commands should pretty
      much be the same, but it's best to look up the docs for your GemStone version[2]
      and follow those instructions)

As I mentioned earlier, it will probably take a while for each of these operations to complete (object log will be fast and the backup will be fast, if the mfc tosses out the majority of your data) and it is likely that the repository will grow some more during the process (hard to predict this one, tho).

Step 1 will touch every session and every continuation so it is hard to say what percent of the objects are going to be touched (the expensive part), still there are likely to be a lot of those puppies and they will have to be read from disk into the SPC ...

Step 3. is going scan all of the live objects and again it hard to predict exactly how expensive it will be ...

Dale

[1] http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm
[2] http://gemtalksystems.com/techsupport/resources/

On 3/25/15 10:18 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL. 

  I’m not running the maintenance VM and I haven’t been doing regular mark for collects. 

  I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup? 

  I’ll also try the ObjectLog trick. 

  I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual. 

  
Best, 

Larry


  
On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Lawrence,

Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?

Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...

Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...

The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...

Dale

On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22528 megabytes.
    Repository has grown to 22528 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22544 megabytes.
    Repository has grown to 22544 megabytes.

--- 03/08/2015 03:31:45 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22560 megabytes.
    Repository has grown to 22560 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22576 megabytes.
    Repository has grown to 22576 megabytes.

--- 03/10/2015 03:19:34 AM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22592 megabytes.
    Repository has grown to 22592 megabytes.

--- 03/10/2015 03:46:39 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22608 megabytes.
    Repository has grown to 22608 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22624 megabytes.
    Repository has grown to 22624 megabytes.


My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass











_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list

On Mar 27, 2015, at 12:48 PM, Dale Henrichs <[hidden email]> wrote:

Larry,

Yea the bulk of the progress that you've made should be committed... should probably add a final transcript show in the gemstoneReap method ... but when your `Expire Progress` matches the preceding `finished scan` that chunk of work won't be done again ...

At this point in time, I suggest that you make a backup in case we hit a problem (like run out of extent space or something) that would make things difficult to proceed from ... this is just precautionary, but prudent ...

I am interested in what caused the out of memory so checking in the gem log for a smalltalk stack when you ran out of memory would help ... it's possible that we'll just hit the same situation right off the bat, when we start process again, so might as well try to understand that one...


Well, the beginning of a very long stack dump is this: 

GsMethod 1926657,   Object >> _doesNotUnderstand:
 GsMethod 10662913,   metaSortedCollection >> sortBlock:
 GsMethod 1754881,   Behavior >> new
 GsMethod 9573889,   SortedCollection >> sortBlock:
 GsMethod 1924097,   Object >> size
 GsMethod 3381249,   SmallInteger >> >
 GsMethod 2242817,   metaKeyValueDictionary >> new
 GsMethod 2243841,   metaKeyValueDictionary >> new:
 GsMethod 2057729,   metaInteger >> _selectedPrimeGreaterThan:
 GsMethod 1924609,   Object >> at:
 GsMethod 3378689,   SmallInteger >> //
 GsMethod 3378945,   SmallInteger >> <=
 GsMethod 1753857,   Behavior >> _basicNew:
 GsMethod 2257665,   KeyValueDictionary >> _initializeWithoutClear:
 GsMethod 3181569,   metaSystem >> _sessionStateAt:put:
 GsMethod 3206913,   metaSystem >> myUserProfile
 GsMethod 3294977,   SymbolDictionary >> at:ifAbsent:
 GsMethod 1534721,   ComplexBlock >> value
 GsMethod 7693313,   metaGsSession >> currentSession
 GsMethod 7723777,   GsCurrentSession >> objectNamed:
 GsMethod 3275521,   SymbolList >> resolveSymbol:
 GsMethod 1795329,   Association >> value
 GsMethod 7592193,   metaGsPackagePolicy >> globalName
 GsMethod 3300353,   SymbolDictionary >> at:otherwise:
 GsMethod 3297281,   SymbolDictionary >> associationAt:otherwise:
 GsMethod 7660545,   GsPackagePolicy >> enabled
 GsMethod 7640833,   metaSessionTemps >> current
 GsMethod 3182081,   metaSystem >> _sessionStateAt:
 GsMethod 7657217,   GsPackagePolicy >> sessionMethodDictionaryGlobalName
 GsMethod 3299585,   SymbolDictionary >> at:put:
 GsMethod 3293953,   SymbolDictionary >> _validatePrivilegeOld:new:
 GsMethod 1789185,   metaAssociation >> newWithKey:value:
 GsMethod 3307777,   SymbolAssociation >> key:value:
 GsMethod 3308033,   SymbolAssociation >> key:
 GsMethod 1796865,   Association >> value:
 GsMethod 3300097,   SymbolDictionary >> _at:put:
 GsMethod 3299073,   SymbolDictionary >> _validatePrivilegeGeneric:
 GsMethod 3140097,   metaSystem >> _protectedMode
 GsMethod 3215873,   metaSystem >> _zeroArgPrim:
 GsMethod 2278145,   KeyValueDictionary >> _at:put:
 GsMethod 2230273,   IdentityDictionary >> hashFunction:
 GsMethod 1910529,   Object >> identityHash
 GsMethod 2272513,   KeyValueDictionary >> tableSize
 GsMethod 3377665,   SmallInteger >> \\
 GsMethod 2263297,   KeyValueDictionary >> keyAt:
 GsMethod 1923585,   Object >> _at:
 GsMethod 2267649,   KeyValueDictionary >> valueAt:
 GsMethod 1916417,   Object >> _basicAt:
 GsMethod 1699329,   Boolean >> |
 GsMethod 2276609,   KeyValueDictionary >> atHash:putKey:
 GsMethod 1922561,   Object >> _basicAt:put:
 GsMethod 2221569,   IdentityDictionary >> atHash:putValue:
 GsMethod 2256129,   KeyValueDictionary >> atHash:putValue:
 GsMethod 7661313,   GsPackagePolicy >> sessionMethodDictionary
 GsMethod 1900033,   Object >> class
 GsMethod 2278401,   KeyValueDictionary >> size
 GsMethod 7652097,   GsPackagePolicy >> symbolList
 GsMethod 7723521,   GsCurrentSession >> symbolList
 GsMethod 3395329,   SequenceableCollection >> reverseDo:
 GsMethod 1942785,   Number >> downTo:do:
 GsMethod 1943041,   Number >> downTo:by:do:

which ends with this:


 GsMethod 119023873,   GRObject >> initialize
 GsMethod 178011137,   WAAttributeSearchContext >> findAttributeAndSelectAncestorsOf:
 GsMethod 178009601,   WAAttributeSearchContext >> key
 GsMethod 177473793,   WAConfiguration >> localAttributeAt:ifPresent:
 GsMethod 177920257,   WAUserConfiguration >> localAttributeAt:ifAbsent:
 GsMethod 177924865,   WAUserConfiguration >> parents
 GsMethod 177361153,   WASystemConfiguration >> localAttributeAt:ifAbsent:
 GsMethod 177361409,   WASystemConfiguration >> attributes
 GsMethod 177362945,   WASystemConfiguration >> description
 GsMethod 177375233,   WAConfigurationDescription >> attributes
 GsMethod 177473281,   WAConfiguration >> expressionAt:ifPresent:
 GsMethod 177920769,   WAUserConfiguration >> expressionAt:ifAbsent:
 GsMethod 178011649,   WAAttributeSearchContext >> isAttributeLocalOn:
 GsMethod 177471233,   WAConfiguration >> inheritedValueForContext:
 GsMethod 178010881,   WAAttributeSearchContext >> isAttributeInheritedOn:
 GsMethod 178009857,   WAAttributeSearchContext >> at:put:
(vmGc  Instances counts for generation code
     GsMethod                                id:     98817    1025 instances   320176 bytes
 vmGc ---)
-----------------------------------------------------
GemStone: Error         Fatal
VM temporary object memory is full
, almost out of memory, too many markSweeps since last successful
scavenge
Error Category: 231169 [GemStone] Number: 4067 Arg Count: 1 Context : 20
Arg 1:   20

[Info]: Logging out at 03/27/2015 04:11:47 PM UTC


*****************************************************
****** GemStone Abnormal Shutdown at 03/27/2015 04:11:48 PM UTC
*****************************************************
-----------------------------------------------------
GemStone: Error         Fatal
VM temporary object memory is full
, almost out of memory, too many markSweeps since last successful
scavenge
Error Category: 231169 [GemStone] Number: 4067 Arg Count: 1 Context : 20
Arg 1:   20

                     


I’ll make a backup and try again.


Larry



Dale

On 03/27/2015 09:31 AM, Lawrence Kellogg wrote:

On Mar 27, 2015, at 12:08 PM, Dale Henrichs <[hidden email]> wrote:

Larry,

Not sure ... I guess the I'm curious if the system is in iowait or burning cpu ... top should give you that information ... I'd also be interested to know if you've run out of disk space ... check the stone log for that information ...

Based on what you find out we'll go on from there ...



The struggle continues, eventually, I ran out of temporary Object Memory:

<Mail Attachment.png>


Disk space as at 63%. Can I run it again? Did any of that stick?

Larry





Dale

On 03/27/2015 07:43 AM, Lawrence Kellogg wrote:
Well, I put in Transcript calls to show count all the time, not just at mod 1000..and I made it all the way through, except it does’t seem to want to come back to me. The image is frozen.

Is it hung in the final commit? 

Here is a screen shot of the last part of the display. Yes I’m running it in GemTools. I don’t care if it’s slow as long as it works. 

<Mail Attachment.png>


Larry

On Mar 26, 2015, at 1:01 AM, Dale Henrichs <[hidden email]> wrote:

Hmmm, I think we need to set a breakpoint in there ... I might be tired, but I can't see how that can happen ... but then that's what a debugger is for ...

Are you running this expression from GemTools? If so the `Transcript show:` might be awfully expensive (lot's o round trips over the WAN) ... you might use GsFile gciLogServer: and then tail the gem log file on the server to see progress ...

Need to find out where the 0 is coming from first...

Dale

On 3/25/15 7:34 PM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, I replaced the WACache>>gemstoneReap method and ran the code from before but it just shows Scan Progress: 0 over and over again. 

  I let it run a few hours but had to kill it as the computer is in my bedroom and I can’t sleep if it makes noise all night. 

  Do I try again tomorrow? 

  best, 

  Larry

 
On Mar 25, 2015, at 6:23 PM, Dale Henrichs <[hidden email]> wrote:

Hmm, I scanned the code in WACache>>gemstoneReap and saw that a commit was being down every 100 entries, but presumably you blew up while building the collection of entries to expire ... sorry about that ... There should be a a stack dump in the gem log (depending upon where you ran the expressions where we can verify my assumption.

It looks like we need to replace WACache>>gemstoneReap with the following:

gemstoneReap
  "Iterate through the cache and remove objects that have expired."

  "In GemStone, this method is performed by a separate maintenance VM,
     so we are already in transaction (assumed to be running in #autoBegin
     transactionMode) and do not have to worry about acquiring the TransactionMutex.
    Since we are using reducedConflict dictionaries in the first place, we will remove the keys
    and values from the existing dictionaries without using the mutex."

  | expired count platform |
  expired := UserGlobals at: #'ExpiryCleanup' put: OrderedCollection new.
  platform := GRPlatform current.
  platform doCommitTransaction.
  count := 0.
  objectsByKey
    associationsDo: [ :assoc |
      (self expiryPolicy isExpiredUpdating: assoc value key: assoc key)
        ifTrue: [
          self notifyRemoved: assoc value key: assoc key.
          count := count + 1.
          expired add: assoc.
          count \\ 100 == 0
            ifTrue: [ platform doCommitTransaction ].
          count \\ 1000 == 0
            ifTrue: [ Transcript cr; show: 'Scan progress: ' , count printString.] ] ].
  Transcript cr; show: 'finished scan: ' , count printString.
  count := 0.
  (UserGlobals at: #'ExpiryCleanup' )
    do: [ :assoc |
      count := count + 1.
      objectsByKey removeKey: assoc key ifAbsent: [].
      keysByObject removeKey: assoc value ifAbsent: [  ].
      count \\ 100 == 0
        ifTrue: [ platform doCommitTransaction ].
      count \\ 1000 == 0
        ifTrue: [ Transcript cr; show: 'Expire progress: ' , count printString.]].
  platform doCommitTransaction.
  UserGlobals removeKey: #'ExpiryCleanup'.
  platform doCommitTransaction.
  ^ expired size

This implementation should be more resistant to an out of memory condition and I've got some logging in  there as well ... the `Transcript show:` should show up in the gem.log and/or Transcript...

I haven't tested this but if there's a problem in the scan loop it will fail quickly. If there's a problem in the expire loop, you can skip the scan loop, and just run the expire loop ...

Sorry about that ... hopefully the second time will be the charm ...

Dale

On 3/25/15 2:36 PM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, Step 1 ran for hours and halted in the debugger with this error: 

Error: VM temporary object memory is full 
, almost out of memory, too many markSweeps since last successful scavenge


What do I do now? 

Best, 

  Larry 



On Mar 25, 2015, at 2:15 PM, Dale Henrichs <[hidden email]> wrote:

Okay here's the sequence of steps that I think you should take:

  1. expire all of your sessions:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransactions

  2. initalize your object log

  3. run MFC

  [
  System abortTransaction.
  SystemRepository markForCollection ]
    on: Warning
    do: [ :ex |
      Transcript
        cr;
        show: ex description.
      ex resume ]

  4. Then do a backup and restore ... you can use GemTools to do the restore,
      but then you should read the SysAdmin docs[1] for instructions to do the restore
      (I've enclosed link to 3.2 docs, but the procedure and commands should pretty
      much be the same, but it's best to look up the docs for your GemStone version[2]
      and follow those instructions)

As I mentioned earlier, it will probably take a while for each of these operations to complete (object log will be fast and the backup will be fast, if the mfc tosses out the majority of your data) and it is likely that the repository will grow some more during the process (hard to predict this one, tho).

Step 1 will touch every session and every continuation so it is hard to say what percent of the objects are going to be touched (the expensive part), still there are likely to be a lot of those puppies and they will have to be read from disk into the SPC ...

Step 3. is going scan all of the live objects and again it hard to predict exactly how expensive it will be ...

Dale

[1] http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm
[2] http://gemtalksystems.com/techsupport/resources/

On 3/25/15 10:18 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL. 

  I’m not running the maintenance VM and I haven’t been doing regular mark for collects. 

  I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup? 

  I’ll also try the ObjectLog trick. 

  I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual. 

  
Best, 

Larry


  
On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Lawrence,

Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?

Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...

Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...

The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...

Dale

On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22528 megabytes.
    Repository has grown to 22528 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22544 megabytes.
    Repository has grown to 22544 megabytes.

--- 03/08/2015 03:31:45 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22560 megabytes.
    Repository has grown to 22560 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22576 megabytes.
    Repository has grown to 22576 megabytes.

--- 03/10/2015 03:19:34 AM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22592 megabytes.
    Repository has grown to 22592 megabytes.

--- 03/10/2015 03:46:39 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22608 megabytes.
    Repository has grown to 22608 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22624 megabytes.
    Repository has grown to 22624 megabytes.


My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass












_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list
Larry,

To get my bearings, it would be useful to see the chunk of the stack where WACache>>gemstoneReap is involved ... so we can understand where we blowing out memory...

Dale

On 03/27/2015 10:04 AM, Lawrence Kellogg wrote:

On Mar 27, 2015, at 12:48 PM, Dale Henrichs <[hidden email]> wrote:

Larry,

Yea the bulk of the progress that you've made should be committed... should probably add a final transcript show in the gemstoneReap method ... but when your `Expire Progress` matches the preceding `finished scan` that chunk of work won't be done again ...

At this point in time, I suggest that you make a backup in case we hit a problem (like run out of extent space or something) that would make things difficult to proceed from ... this is just precautionary, but prudent ...

I am interested in what caused the out of memory so checking in the gem log for a smalltalk stack when you ran out of memory would help ... it's possible that we'll just hit the same situation right off the bat, when we start process again, so might as well try to understand that one...


Well, the beginning of a very long stack dump is this: 

GsMethod 1926657,   Object >> _doesNotUnderstand:
 GsMethod 10662913,   metaSortedCollection >> sortBlock:
 GsMethod 1754881,   Behavior >> new
 GsMethod 9573889,   SortedCollection >> sortBlock:
 GsMethod 1924097,   Object >> size
 GsMethod 3381249,   SmallInteger >> >
 GsMethod 2242817,   metaKeyValueDictionary >> new
 GsMethod 2243841,   metaKeyValueDictionary >> new:
 GsMethod 2057729,   metaInteger >> _selectedPrimeGreaterThan:
 GsMethod 1924609,   Object >> at:
 GsMethod 3378689,   SmallInteger >> //
 GsMethod 3378945,   SmallInteger >> <=
 GsMethod 1753857,   Behavior >> _basicNew:
 GsMethod 2257665,   KeyValueDictionary >> _initializeWithoutClear:
 GsMethod 3181569,   metaSystem >> _sessionStateAt:put:
 GsMethod 3206913,   metaSystem >> myUserProfile
 GsMethod 3294977,   SymbolDictionary >> at:ifAbsent:
 GsMethod 1534721,   ComplexBlock >> value
 GsMethod 7693313,   metaGsSession >> currentSession
 GsMethod 7723777,   GsCurrentSession >> objectNamed:
 GsMethod 3275521,   SymbolList >> resolveSymbol:
 GsMethod 1795329,   Association >> value
 GsMethod 7592193,   metaGsPackagePolicy >> globalName
 GsMethod 3300353,   SymbolDictionary >> at:otherwise:
 GsMethod 3297281,   SymbolDictionary >> associationAt:otherwise:
 GsMethod 7660545,   GsPackagePolicy >> enabled
 GsMethod 7640833,   metaSessionTemps >> current
 GsMethod 3182081,   metaSystem >> _sessionStateAt:
 GsMethod 7657217,   GsPackagePolicy >> sessionMethodDictionaryGlobalName
 GsMethod 3299585,   SymbolDictionary >> at:put:
 GsMethod 3293953,   SymbolDictionary >> _validatePrivilegeOld:new:
 GsMethod 1789185,   metaAssociation >> newWithKey:value:
 GsMethod 3307777,   SymbolAssociation >> key:value:
 GsMethod 3308033,   SymbolAssociation >> key:
 GsMethod 1796865,   Association >> value:
 GsMethod 3300097,   SymbolDictionary >> _at:put:
 GsMethod 3299073,   SymbolDictionary >> _validatePrivilegeGeneric:
 GsMethod 3140097,   metaSystem >> _protectedMode
 GsMethod 3215873,   metaSystem >> _zeroArgPrim:
 GsMethod 2278145,   KeyValueDictionary >> _at:put:
 GsMethod 2230273,   IdentityDictionary >> hashFunction:
 GsMethod 1910529,   Object >> identityHash
 GsMethod 2272513,   KeyValueDictionary >> tableSize
 GsMethod 3377665,   SmallInteger >> \\
 GsMethod 2263297,   KeyValueDictionary >> keyAt:
 GsMethod 1923585,   Object >> _at:
 GsMethod 2267649,   KeyValueDictionary >> valueAt:
 GsMethod 1916417,   Object >> _basicAt:
 GsMethod 1699329,   Boolean >> |
 GsMethod 2276609,   KeyValueDictionary >> atHash:putKey:
 GsMethod 1922561,   Object >> _basicAt:put:
 GsMethod 2221569,   IdentityDictionary >> atHash:putValue:
 GsMethod 2256129,   KeyValueDictionary >> atHash:putValue:
 GsMethod 7661313,   GsPackagePolicy >> sessionMethodDictionary
 GsMethod 1900033,   Object >> class
 GsMethod 2278401,   KeyValueDictionary >> size
 GsMethod 7652097,   GsPackagePolicy >> symbolList
 GsMethod 7723521,   GsCurrentSession >> symbolList
 GsMethod 3395329,   SequenceableCollection >> reverseDo:
 GsMethod 1942785,   Number >> downTo:do:
 GsMethod 1943041,   Number >> downTo:by:do:

which ends with this:


 GsMethod 119023873,   GRObject >> initialize
 GsMethod 178011137,   WAAttributeSearchContext >> findAttributeAndSelectAncestorsOf:
 GsMethod 178009601,   WAAttributeSearchContext >> key
 GsMethod 177473793,   WAConfiguration >> localAttributeAt:ifPresent:
 GsMethod 177920257,   WAUserConfiguration >> localAttributeAt:ifAbsent:
 GsMethod 177924865,   WAUserConfiguration >> parents
 GsMethod 177361153,   WASystemConfiguration >> localAttributeAt:ifAbsent:
 GsMethod 177361409,   WASystemConfiguration >> attributes
 GsMethod 177362945,   WASystemConfiguration >> description
 GsMethod 177375233,   WAConfigurationDescription >> attributes
 GsMethod 177473281,   WAConfiguration >> expressionAt:ifPresent:
 GsMethod 177920769,   WAUserConfiguration >> expressionAt:ifAbsent:
 GsMethod 178011649,   WAAttributeSearchContext >> isAttributeLocalOn:
 GsMethod 177471233,   WAConfiguration >> inheritedValueForContext:
 GsMethod 178010881,   WAAttributeSearchContext >> isAttributeInheritedOn:
 GsMethod 178009857,   WAAttributeSearchContext >> at:put:
(vmGc  Instances counts for generation code
     GsMethod                                id:     98817    1025 instances   320176 bytes
 vmGc ---)
-----------------------------------------------------
GemStone: Error         Fatal
VM temporary object memory is full
, almost out of memory, too many markSweeps since last successful
scavenge
Error Category: 231169 [GemStone] Number: 4067 Arg Count: 1 Context : 20
Arg 1:   20

[Info]: Logging out at 03/27/2015 04:11:47 PM UTC


*****************************************************
****** GemStone Abnormal Shutdown at 03/27/2015 04:11:48 PM UTC
*****************************************************
-----------------------------------------------------
GemStone: Error         Fatal
VM temporary object memory is full
, almost out of memory, too many markSweeps since last successful
scavenge
Error Category: 231169 [GemStone] Number: 4067 Arg Count: 1 Context : 20
Arg 1:   20

                     


I’ll make a backup and try again.


Larry



Dale

On 03/27/2015 09:31 AM, Lawrence Kellogg wrote:

On Mar 27, 2015, at 12:08 PM, Dale Henrichs <[hidden email]> wrote:

Larry,

Not sure ... I guess the I'm curious if the system is in iowait or burning cpu ... top should give you that information ... I'd also be interested to know if you've run out of disk space ... check the stone log for that information ...

Based on what you find out we'll go on from there ...



The struggle continues, eventually, I ran out of temporary Object Memory:

<Mail Attachment.png>


Disk space as at 63%. Can I run it again? Did any of that stick?

Larry





Dale

On 03/27/2015 07:43 AM, Lawrence Kellogg wrote:
Well, I put in Transcript calls to show count all the time, not just at mod 1000..and I made it all the way through, except it does’t seem to want to come back to me. The image is frozen.

Is it hung in the final commit? 

Here is a screen shot of the last part of the display. Yes I’m running it in GemTools. I don’t care if it’s slow as long as it works. 

<Mail Attachment.png>


Larry

On Mar 26, 2015, at 1:01 AM, Dale Henrichs <[hidden email]> wrote:

Hmmm, I think we need to set a breakpoint in there ... I might be tired, but I can't see how that can happen ... but then that's what a debugger is for ...

Are you running this expression from GemTools? If so the `Transcript show:` might be awfully expensive (lot's o round trips over the WAN) ... you might use GsFile gciLogServer: and then tail the gem log file on the server to see progress ...

Need to find out where the 0 is coming from first...

Dale

On 3/25/15 7:34 PM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, I replaced the WACache>>gemstoneReap method and ran the code from before but it just shows Scan Progress: 0 over and over again. 

  I let it run a few hours but had to kill it as the computer is in my bedroom and I can’t sleep if it makes noise all night. 

  Do I try again tomorrow? 

  best, 

  Larry

 
On Mar 25, 2015, at 6:23 PM, Dale Henrichs <[hidden email]> wrote:

Hmm, I scanned the code in WACache>>gemstoneReap and saw that a commit was being down every 100 entries, but presumably you blew up while building the collection of entries to expire ... sorry about that ... There should be a a stack dump in the gem log (depending upon where you ran the expressions where we can verify my assumption.

It looks like we need to replace WACache>>gemstoneReap with the following:

gemstoneReap
  "Iterate through the cache and remove objects that have expired."

  "In GemStone, this method is performed by a separate maintenance VM,
     so we are already in transaction (assumed to be running in #autoBegin
     transactionMode) and do not have to worry about acquiring the TransactionMutex.
    Since we are using reducedConflict dictionaries in the first place, we will remove the keys
    and values from the existing dictionaries without using the mutex."

  | expired count platform |
  expired := UserGlobals at: #'ExpiryCleanup' put: OrderedCollection new.
  platform := GRPlatform current.
  platform doCommitTransaction.
  count := 0.
  objectsByKey
    associationsDo: [ :assoc |
      (self expiryPolicy isExpiredUpdating: assoc value key: assoc key)
        ifTrue: [
          self notifyRemoved: assoc value key: assoc key.
          count := count + 1.
          expired add: assoc.
          count \\ 100 == 0
            ifTrue: [ platform doCommitTransaction ].
          count \\ 1000 == 0
            ifTrue: [ Transcript cr; show: 'Scan progress: ' , count printString.] ] ].
  Transcript cr; show: 'finished scan: ' , count printString.
  count := 0.
  (UserGlobals at: #'ExpiryCleanup' )
    do: [ :assoc |
      count := count + 1.
      objectsByKey removeKey: assoc key ifAbsent: [].
      keysByObject removeKey: assoc value ifAbsent: [  ].
      count \\ 100 == 0
        ifTrue: [ platform doCommitTransaction ].
      count \\ 1000 == 0
        ifTrue: [ Transcript cr; show: 'Expire progress: ' , count printString.]].
  platform doCommitTransaction.
  UserGlobals removeKey: #'ExpiryCleanup'.
  platform doCommitTransaction.
  ^ expired size

This implementation should be more resistant to an out of memory condition and I've got some logging in  there as well ... the `Transcript show:` should show up in the gem.log and/or Transcript...

I haven't tested this but if there's a problem in the scan loop it will fail quickly. If there's a problem in the expire loop, you can skip the scan loop, and just run the expire loop ...

Sorry about that ... hopefully the second time will be the charm ...

Dale

On 3/25/15 2:36 PM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, Step 1 ran for hours and halted in the debugger with this error: 

Error: VM temporary object memory is full 
, almost out of memory, too many markSweeps since last successful scavenge


What do I do now? 

Best, 

  Larry 



On Mar 25, 2015, at 2:15 PM, Dale Henrichs <[hidden email]> wrote:

Okay here's the sequence of steps that I think you should take:

  1. expire all of your sessions:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransactions

  2. initalize your object log

  3. run MFC

  [
  System abortTransaction.
  SystemRepository markForCollection ]
    on: Warning
    do: [ :ex |
      Transcript
        cr;
        show: ex description.
      ex resume ]

  4. Then do a backup and restore ... you can use GemTools to do the restore,
      but then you should read the SysAdmin docs[1] for instructions to do the restore
      (I've enclosed link to 3.2 docs, but the procedure and commands should pretty
      much be the same, but it's best to look up the docs for your GemStone version[2]
      and follow those instructions)

As I mentioned earlier, it will probably take a while for each of these operations to complete (object log will be fast and the backup will be fast, if the mfc tosses out the majority of your data) and it is likely that the repository will grow some more during the process (hard to predict this one, tho).

Step 1 will touch every session and every continuation so it is hard to say what percent of the objects are going to be touched (the expensive part), still there are likely to be a lot of those puppies and they will have to be read from disk into the SPC ...

Step 3. is going scan all of the live objects and again it hard to predict exactly how expensive it will be ...

Dale

[1] http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm
[2] http://gemtalksystems.com/techsupport/resources/

On 3/25/15 10:18 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL. 

  I’m not running the maintenance VM and I haven’t been doing regular mark for collects. 

  I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup? 

  I’ll also try the ObjectLog trick. 

  I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual. 

  
Best, 

Larry


  
On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Lawrence,

Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?

Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...

Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...

The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...

Dale

On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22528 megabytes.
    Repository has grown to 22528 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22544 megabytes.
    Repository has grown to 22544 megabytes.

--- 03/08/2015 03:31:45 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22560 megabytes.
    Repository has grown to 22560 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22576 megabytes.
    Repository has grown to 22576 megabytes.

--- 03/10/2015 03:19:34 AM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22592 megabytes.
    Repository has grown to 22592 megabytes.

--- 03/10/2015 03:46:39 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22608 megabytes.
    Repository has grown to 22608 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22624 megabytes.
    Repository has grown to 22624 megabytes.


My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass













_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list

On Mar 27, 2015, at 1:24 PM, Dale Henrichs <[hidden email]> wrote:

Larry,

To get my bearings, it would be useful to see the chunk of the stack where WACache>>gemstoneReap is involved ... so we can understand where we blowing out memory...


Ok, here is the full stack: (it is impressive)

 GsMethod 1926657,   Object >> _doesNotUnderstand:
 GsMethod 10662913,   metaSortedCollection >> sortBlock:
 GsMethod 1754881,   Behavior >> new
 GsMethod 9573889,   SortedCollection >> sortBlock:
 GsMethod 1924097,   Object >> size
 GsMethod 3381249,   SmallInteger >> >
 GsMethod 2242817,   metaKeyValueDictionary >> new
 GsMethod 2243841,   metaKeyValueDictionary >> new:
 GsMethod 2057729,   metaInteger >> _selectedPrimeGreaterThan:
 GsMethod 1924609,   Object >> at:
 GsMethod 3378689,   SmallInteger >> //
 GsMethod 3378945,   SmallInteger >> <=
 GsMethod 1753857,   Behavior >> _basicNew:
 GsMethod 2257665,   KeyValueDictionary >> _initializeWithoutClear:
 GsMethod 3181569,   metaSystem >> _sessionStateAt:put:
 GsMethod 3206913,   metaSystem >> myUserProfile
 GsMethod 3294977,   SymbolDictionary >> at:ifAbsent:
 GsMethod 1534721,   ComplexBlock >> value
 GsMethod 7693313,   metaGsSession >> currentSession
 GsMethod 7723777,   GsCurrentSession >> objectNamed:
 GsMethod 3275521,   SymbolList >> resolveSymbol:
 GsMethod 1795329,   Association >> value
 GsMethod 7592193,   metaGsPackagePolicy >> globalName
 GsMethod 3300353,   SymbolDictionary >> at:otherwise:
 GsMethod 3297281,   SymbolDictionary >> associationAt:otherwise:
 GsMethod 7660545,   GsPackagePolicy >> enabled
 GsMethod 7640833,   metaSessionTemps >> current
 GsMethod 3182081,   metaSystem >> _sessionStateAt:
 GsMethod 7657217,   GsPackagePolicy >> sessionMethodDictionaryGlobalName
 GsMethod 3299585,   SymbolDictionary >> at:put:
 GsMethod 3293953,   SymbolDictionary >> _validatePrivilegeOld:new:
 GsMethod 1789185,   metaAssociation >> newWithKey:value:
 GsMethod 3307777,   SymbolAssociation >> key:value:
 GsMethod 3308033,   SymbolAssociation >> key:
 GsMethod 1796865,   Association >> value:
 GsMethod 3300097,   SymbolDictionary >> _at:put:
 GsMethod 3299073,   SymbolDictionary >> _validatePrivilegeGeneric:
 GsMethod 3140097,   metaSystem >> _protectedMode
 GsMethod 3215873,   metaSystem >> _zeroArgPrim:
 GsMethod 2278145,   KeyValueDictionary >> _at:put:
 GsMethod 2230273,   IdentityDictionary >> hashFunction:
 GsMethod 1910529,   Object >> identityHash
 GsMethod 2272513,   KeyValueDictionary >> tableSize
 GsMethod 3377665,   SmallInteger >> \\
 GsMethod 2263297,   KeyValueDictionary >> keyAt:
 GsMethod 1923585,   Object >> _at:
 GsMethod 2267649,   KeyValueDictionary >> valueAt:
 GsMethod 1916417,   Object >> _basicAt:
 GsMethod 1699329,   Boolean >> |
 GsMethod 2276609,   KeyValueDictionary >> atHash:putKey:
 GsMethod 1922561,   Object >> _basicAt:put:
 GsMethod 2221569,   IdentityDictionary >> atHash:putValue:
 GsMethod 2256129,   KeyValueDictionary >> atHash:putValue:
 GsMethod 7661313,   GsPackagePolicy >> sessionMethodDictionary
 GsMethod 1900033,   Object >> class
 GsMethod 2278401,   KeyValueDictionary >> size
 GsMethod 7652097,   GsPackagePolicy >> symbolList
 GsMethod 7723521,   GsCurrentSession >> symbolList
 GsMethod 3395329,   SequenceableCollection >> reverseDo:
 GsMethod 1942785,   Number >> downTo:do:
 GsMethod 1943041,   Number >> downTo:by:do:
 GsMethod 1536001,   ComplexBlock >> value:
 GsMethod 7639809,   metaGsPackage >> globalName
 GsMethod 7706113,   GsPackage >> sessionMethods
 GsMethod 1790977,   Association >> key
 GsMethod 1533697,   ComplexBlock >> value:value:
 GsMethod 2990337,   GsMethodDictionary >> at:otherwise:
 GsMethod 2990081,   GsMethodDictionary >> at:ifAbsent:
 GsMethod 7699457,   GsSessionMethodDictionary >> hashFunction:
 GsMethod 2991873,   GsMethodDictionary >> hashFunction:
 GsMethod 1910273,   Object >> basicIdentityHash
 GsMethod 1829889,   AbstractDictionary >> _reportKeyNotFound:with:
 GsMethod 2989057,   GsMethodDictionary >> keysAndValuesDo:
 GsMethod 1913089,   Object >> _basicSize
 GsMethod 7591425,   metaGsPackagePolicy >> current
 GsMethod 20491265,   metaSystemLoginNotification >> sessionStart
 GsMethod 20491009,   metaSystemLoginNotification >> subscriptions
 GsMethod 3219713,   metaSystem >> rcValueCacheAt:for:otherwise:
 GsMethod 3220481,   metaSystem >> _rcValueCache
 GsMethod 20422913,   metaObject >> new
 GsMethod 269799681,   Object >> initialize
 GsMethod 3211521,   metaSystem >> _addRootObjectToRcReadSet:
 GsMethod 3223809,   metaSystem >> _addToRcReadSet:includingAllNodes:
 GsMethod 3224321,   metaSystem >> _addEntireObjectToRcReadSet:
 GsMethod 1553409,   Collection >> isEmpty
 GsMethod 2105089,   IdentityBag >> addAll:
 GsMethod 1878273,   Object >> _isRcIdentityBag
 GsMethod 2124801,   IdentityBag >> _addAll:
 GsMethod 3219201,   metaSystem >> rcValueCacheAt:put:for:
 GsMethod 3220993,   metaSystem >> rcValueCache
 GsMethod 2278913,   KeyValueDictionary >> at:otherwise:
 GsMethod 3032065,   IdentityKeyValueDictionary >> hashFunction:
 GsMethod 2270721,   KeyValueDictionary >> at:put:
 GsMethod 1804545,   Array >> addLast:
 GsMethod 2116609,   IdentityBag >> _at:
 GsMethod 3387905,   SimpleBlock >> value:
 GsMethod 20742145,   SmalltalkProxy >> at:ifAbsent:
 GsMethod 1920001,   Object >> isNil
 GsMethod 3074305,   UserProfile >> symbolList
 GsMethod 3266561,   SymbolList >> objectNamed:
 GsMethod 3206657,   metaSystem >> session
 GsMethod 3380225,   SmallInteger >> bitXor:
 GsMethod 2217985,   IdentityDictionary >> at:ifAbsent:
 GsMethod 2282241,   KeyValueDictionary >> _at:otherwise:
 GsMethod 3386625,   SimpleBlock >> value
 GsMethod 19090945,   Object >> copy
 GsMethod 3486209,   Object >> _basicCopy
 GsMethod 19081985,   Object >> postCopy
 GsMethod 2223361,   IdentityDictionary >> at:put:
 GsMethod 1795585,   Association >> key:value:
 GsMethod 3488513,   Object >> segment
 GsMethod 3487745,   Object >> assignToSegment:
 GsMethod 3488001,   Object >> _changeToSegment:
 GsMethod 2228225,   IdentityDictionary >> compareKey:with:
 GsMethod 3033089,   IdentityKeyValueDictionary >> collisionBucketClass
 GsMethod 2199041,   metaIdentityCollisionBucket >> new
 GsMethod 1846273,   metaAbstractCollisionBucket >> new:
 GsMethod 1754625,   Behavior >> new:
 GsMethod 1867265,   AbstractCollisionBucket >> initialize
 GsMethod 1538561,   CollisionBucket >> keyValueDictionary:
 GsMethod 2206977,   IdentityCollisionBucket >> at:put:keyValDict_coll:
 GsMethod 2201857,   IdentityCollisionBucket >> binarySearchForInsertKey:
 GsMethod 3376897,   SmallInteger >> <
 GsMethod 2206209,   IdentityCollisionBucket >> insertAt:key:value:
 GsMethod 3401729,   SequenceableCollection >> insertAll:at:
 GsMethod 1863937,   AbstractCollisionBucket >> size
 GsMethod 1864193,   AbstractCollisionBucket >> at:otherwise:
 GsMethod 2202881,   IdentityCollisionBucket >> searchForKey:
 GsMethod 1914113,   Object >> at:put:
 GsMethod 2266369,   KeyValueDictionary >> tableSize:
 GsMethod 1890305,   Object >> basicAt:
 GsMethod 1538817,   CollisionBucket >> _removeAll
 GsMethod 1907201,   Object >> size:
 GsMethod 1920257,   Object >> _basicSize:
 GsMethod 2281217,   KeyValueDictionary >> _copyNilIntoSelfFrom:to:
 GsMethod 1887745,   Object >> respondsTo:
 GsMethod 15058433,   Behavior >> whichClassIncludesSelector:
 GsMethod 3316737,   metaSymbol >> _existingWithAll:
 GsMethod 7655169,   GsPackagePolicy >> includesSelector:for:
 GsMethod 1826305,   AbstractDictionary >> includesKey:
 GsMethod 3380737,   SmallInteger >> printString
 GsMethod 3377409,   SmallInteger >> asString
 GsMethod 3346945,   String >> ,
 GsMethod 3348993,   String >> size
 GsMethod 1986817,   Magnitude >> min:
 GsMethod 19069185,   CharacterCollection >> copyFrom:to:
 GsMethod 1608961,   metaCharacterCollection >> _newString:
 GsMethod 3345409,   String >> copyFrom:to:into:startingAt:
 GsMethod 10828289,   metaDictionary >> new
 GsMethod 10828801,   metaDictionary >> new:
 GsMethod 1747201,   Behavior >> basicNew:
 GsMethod 9665537,   Dictionary >> count:
 GsMethod 9657857,   Dictionary >> tableSize:
 GsMethod 9665793,   Dictionary >> emptySlotHint:
 GsMethod 9666049,   Dictionary >> numEmptySlots:
 GsMethod 9663233,   Dictionary >> at:put:
 GsMethod 1842945,   AbstractDictionary >> hashFunction:
 GsMethod 1647105,   CharacterCollection >> hash
 GsMethod 9658369,   Dictionary >> tableSize
 GsMethod 9666305,   Dictionary >> associationAtOrNil:hash:
 GsMethod 10829313,   metaDictionary >> associationClass
 GsMethod 9660161,   Dictionary >> addNewAssociation:hash:
 GsMethod 1890049,   Object >> basicAt:put:
 GsMethod 2271489,   KeyValueDictionary >> keys
 GsMethod 2283265,   KeyValueDictionary >> keysDo:
 GsMethod 2275841,   KeyValueDictionary >> keysAndValuesDo:
 GsMethod 2113281,   IdentityBag >> add:
 GsMethod 20446209,   Collection >> do:separatedBy:
 GsMethod 2127617,   IdentityBag >> do:
 GsMethod 37907201,   Class >> proxyClassName
 GsMethod 47120641,   metaOBGemStonePlatform >> gsParserResolve:forClass:
 GsMethod 47118849,   metaOBGemStonePlatform >> hasInterned:ifTrue:
 GsMethod 19115521,   metaSymbol >> hasInterned:ifTrue:
 GsMethod 79149569,   Object >> ifNotNil:
 GsMethod 19138305,   BlockClosure >> valueWithPossibleArgs:
 GsMethod 19137793,   BlockClosure >> numArgs
 GsMethod 2303233,   ExecutableBlock >> numberArgs
 GsMethod 9005057,   ExecutableBlock >> on:do:
 GsMethod 10062081,   ExceptionHandler >> try:on:do:
 GsMethod 11896833,   metaExceptionA >> errorCategoryToCatch
 GsMethod 11897089,   metaExceptionA >> errorNumberToCatch
 GsMethod 2315265,   metaException >> category:number:do:
 GsMethod 2324481,   Exception >> handler:
 GsMethod 10065409,   ExceptionHandler >> doTryBlock:
 GsMethod 19170305,   UndefinedObject >> ifNil:
 GsMethod 20741889,   SmalltalkProxy >> at:
 GsMethod 19082753,   Object >> ifNil:
 GsMethod 11896577,   metaExceptionA >> signal
 GsMethod 11895041,   metaExceptionA >> signal:
 GsMethod 11895297,   metaExceptionA >> new
 GsMethod 1747457,   Behavior >> basicNew
 GsMethod 1755137,   Behavior >> _basicNew
 GsMethod 10052865,   ExceptionA >> initialize
 GsMethod 10050817,   ExceptionA >> signal:
 GsMethod 10057473,   ExceptionA >> signal
 GsMethod 10055169,   ExceptionA >> gsNumber
 GsMethod 11898113,   metaExceptionA >> errorNumberToSignal
 GsMethod 20480001,   ExceptionA >> gsCategory
 GsMethod 1919489,   Object >> notNil
 GsMethod 2318081,   metaException >> _exceptionFor:signalDictionary:
 GsMethod 20418817,   metaError >> handlesCategory:number:
 GsMethod 20510209,   metaExceptionA >> handles:
 GsMethod 1699841,   Boolean >> not
 GsMethod 10050561,   ExceptionA >> _defaultAction
 GsMethod 11669505,   Notification >> isResumable
 GsMethod 7174145,   metaOrderedCollection >> new:
 GsMethod 9663489,   Dictionary >> at:ifAbsent:
 GsMethod 9661441,   Dictionary >> associationAt:otherwise:
 GsMethod 269800193,   metaSequenceableCollection >> streamContents:
 GsMethod 3437057,   metaPositionableStream >> on:
 GsMethod 3441409,   PositionableStream >> _initStreamWith:
 GsMethod 19049473,   Behavior >> selectorsAndMethodsDo:
 GsMethod 15063041,   Behavior >> _methodDict
 GsMethod 1900289,   Object >> hash
 GsMethod 1911297,   Object >> =
 GsMethod 15066113,   Behavior >> pragmasForMethod:
 GsMethod 7653121,   GsPackagePolicy >> packagesDo:
 GsMethod 1547777,   Collection >> do:
 GsMethod 3326465,   Symbol >> asSymbol
 GsMethod 3321345,   Symbol >> =
 GsMethod 3038209,   WriteStream >> nextPut:
 GsMethod 3038977,   WriteStream >> _nextPut:
 GsMethod 3038465,   WriteStream >> contents
 GsMethod 19132673,   SequenceableCollection >> copyFrom:to:
 GsMethod 7177729,   OrderedCollection >> add:
 GsMethod 3349249,   String >> =
 GsMethod 9660417,   Dictionary >> findEmptySlot
 GsMethod 9659393,   Dictionary >> rebuild
 GsMethod 1885953,   Object >> yourself
 GsMethod 1907713,   Object >> perform:with:
 GsMethod 3388929,   metaSequenceableCollection >> with:
 GsMethod 19092737,   Object >> value
 GsMethod 7180289,   OrderedCollection >> addAll:
 GsMethod 3397121,   SequenceableCollection >> ,
 GsMethod 28537857,   String >> setBlessingInMetacelloConfig:
 GsMethod 28531713,   String >> setBlessingInMetacelloVersion:
 GsMethod 28774145,   BlockClosure >> setProject:withInMetacelloConfig:
 GsMethod 28535297,   String >> addToMetacelloRepositories:
 GsMethod 28532737,   String >> asMetacelloVersionNumber
 GsMethod 19066881,   CharacterCollection >> findTokens:
 GsMethod 19076609,   Object >> isCharacter
 GsMethod 19071489,   CharacterCollection >> skipDelimiters:startingAt:
 GsMethod 3405569,   SequenceableCollection >> detect:ifNone:
 GsMethod 3340289,   String >> at:
 GsMethod 1664257,   Character >> =
 GsMethod 19070721,   CharacterCollection >> findDelimiters:startingAt:
 GsMethod 19124481,   SequenceableCollection >> first
 GsMethod 164897537,   CharacterCollection >> asNumber
 GsMethod 79154945,   SequenceableCollection >> readStream
 GsMethod 164902401,   metaNumber >> readFrom:
 GsMethod 19091201,   Object >> isString
 GsMethod 79196417,   SmallInteger >> isLarge
 GsMethod 19141889,   Character >> charCode
 GsMethod 10063361,   ExceptionHandler >> return:
 GsMethod 19119873,   SequenceableCollection >> allButFirst
 GsMethod 19132929,   SequenceableCollection >> allButFirst:
 GsMethod 28523265,   Collection >> setLoadsInMetacelloProject:
 GsMethod 1554945,   Collection >> asArray
 GsMethod 1540353,   metaCollection >> withAll:
 GsMethod 1814529,   Array >> addAll:
 GsMethod 28537089,   String >> setLoadsInMetacelloProject:
 GsMethod 28520961,   Collection >> setIncludesInMetacelloPackage:
 GsMethod 9659137,   Dictionary >> associationsDo:
 GsMethod 9656833,   Dictionary >> add:
 GsMethod 28538113,   String >> setDescriptionInMetacelloConfig:
 GsMethod 28534785,   String >> setDescriptionInMetacelloVersion:
 GsMethod 28537601,   String >> setAuthorInMetacelloConfig:
 GsMethod 28532225,   String >> setAuthorInMetacelloVersion:
 GsMethod 28538369,   String >> setTimestampInMetacelloConfig:
 GsMethod 28535809,   String >> setTimestampInMetacelloVersion:
 GsMethod 28535553,   String >> setProject:withInMetacelloConfig:
 GsMethod 1886465,   Object >> species
 GsMethod 20503297,   Dictionary >> copy
 GsMethod 20462081,   metaAssociation >> key:value:
 GsMethod 10803201,   metaSet >> new
 GsMethod 9650433,   Set >> initialize:
 GsMethod 9650689,   Set >> add:
 GsMethod 1787905,   Behavior >> varyingConstraint
 GsMethod 2277377,   KeyValueDictionary >> at:ifAbsent:
 GsMethod 118801153,   ExceptionSet >> handlesCategory:number:
 GsMethod 20508673,   metaExceptionA >> category:number:arguments:
 GsMethod 9659649,   Dictionary >> valuesDo:
 GsMethod 19118849,   SequenceableCollection >> sort:
 GsMethod 19128833,   SequenceableCollection >> mergeSortFrom:to:by:
 GsMethod 172938241,   Object >> assert:
 GsMethod 19132161,   SequenceableCollection >> clone
 GsMethod 19129089,   SequenceableCollection >> mergeSortFrom:to:src:dst:by:
 GsMethod 19128577,   SequenceableCollection >> mergeFirst:middle:last:into:by:
 GsMethod 3385857,   SimpleBlock >> value:value:
 GsMethod 7180033,   OrderedCollection >> addFirst:
 GsMethod 3401217,   SequenceableCollection >> insertObject:at:
 GsMethod 19067137,   CharacterCollection >> isString
 GsMethod 28547073,   Integer >> metacelloVersionComponentLessThan:
 GsMethod 28547585,   Integer >> metacelloIntegerLessThanSelf:
 GsMethod 3396353,   SequenceableCollection >> replaceFrom:to:with:startingAt:
 GsMethod 28533505,   String >> metacelloVersionComponentLessThan:
 GsMethod 28547329,   Integer >> metacelloStringLessThanSelf:
 GsMethod 1937921,   Number >> _retry:coercing:
 GsMethod 28534529,   String >> metacelloIntegerLessThanSelf:
 GsMethod 3394817,   SequenceableCollection >> select:
 GsMethod 1886209,   Object >> speciesForSelect
 GsMethod 3401985,   SequenceableCollection >> includes:
 GsMethod 1807361,   Array >> add:
 GsMethod 1602305,   Class >> name
 GsMethod 3326209,   Symbol >> asString
 GsMethod 3328001,   metaString >> withAll:
 GsMethod 19086593,   Object >> instVarNamed:put:
 GsMethod 3350017,   String >> asSymbol
 GsMethod 3315457,   metaSymbol >> withAll:
 GsMethod 2280449,   KeyValueDictionary >> compareKey:with:
 GsMethod 2265345,   KeyValueDictionary >> collisionBucketClass
 GsMethod 1845505,   metaAbstractCollisionBucket >> new
 GsMethod 1866753,   AbstractCollisionBucket >> at:put:keyValDict_coll:
 GsMethod 1859585,   AbstractCollisionBucket >> tableSize
 GsMethod 1868033,   AbstractCollisionBucket >> keyAt:
 GsMethod 1864961,   AbstractCollisionBucket >> compareKey:with:
 GsMethod 1862401,   AbstractCollisionBucket >> at:putKey:
 GsMethod 1862657,   AbstractCollisionBucket >> at:putValue:
 GsMethod 1863681,   AbstractCollisionBucket >> at:ifAbsent:
 GsMethod 1859329,   AbstractCollisionBucket >> searchForKey:
 GsMethod 1862913,   AbstractCollisionBucket >> _reportKeyNotFound:with:
 GsMethod 2267393,   KeyValueDictionary >> rebuildTable:
 GsMethod 9651969,   Set >> do:
 GsMethod 1549313,   Collection >> addAll:
 GsMethod 20437761,   Collection >> removeAllFoundIn:
 GsMethod 1862145,   AbstractCollisionBucket >> removeKey:ifAbsent:
 GsMethod 1868289,   AbstractCollisionBucket >> valueAt:
 GsMethod 3395073,   SequenceableCollection >> collect:
 GsMethod 1951489,   Number >> abs
 GsMethod 28779009,   BlockClosure >> setPackage:withInMetacelloConfig:
 GsMethod 28537345,   String >> setRequiresInMetacelloPackage:
 GsMethod 28520705,   Collection >> setRequiresInMetacelloPackage:
 GsMethod 120885505,   Symbol >> setPostLoadDoItInMetacelloSpec:
 GsMethod 28535041,   String >> setPackage:withInMetacelloConfig:
 GsMethod 120885761,   Symbol >> setPreLoadDoItInMetacelloSpec:
 GsMethod 3345665,   String >> <=
 GsMethod 3344641,   String >> >
 GsMethod 28534017,   String >> metacelloStringLessThanSelf:
 GsMethod 3348737,   String >> <
 GsMethod 1907969,   Object >> perform:
 GsMethod 19135233,   SequenceableCollection >> detectMax:
 GsMethod 3038721,   WriteStream >> nextPutAll:
 GsMethod 3343617,   String >> addAll:
 GsMethod 3340801,   String >> at:put:
 GsMethod 19070209,   CharacterCollection >> beginsWith:
 GsMethod 19130113,   SequenceableCollection >> last
 GsMethod 19123457,   SequenceableCollection >> copyUpToLast:
 GsMethod 19120385,   SequenceableCollection >> lastIndexOf:ifAbsent:
 GsMethod 19128321,   SequenceableCollection >> lastIndexOf:startingAt:ifAbsent:
 GsMethod 19123201,   SequenceableCollection >> first:
 GsMethod 19134465,   SequenceableCollection >> copyAfterLast:
 GsMethod 19131393,   SequenceableCollection >> copyUpTo:
 GsMethod 19134209,   SequenceableCollection >> copyAfter:
 GsMethod 328041985,   CharacterCollection >> asInteger
 GsMethod 19096833,   Number >> isZero
 GsMethod 1702913,   Boolean >> &
 GsMethod 164906241,   metaInteger >> readFrom:
 GsMethod 269801729,   metaInteger >> readFrom:base:
 GsMethod 2180609,   UnorderedCollection >> copy
 GsMethod 2180353,   UnorderedCollection >> _finishShallowCopy
 GsMethod 3167489,   metaSystem >> _configurationAt:isStone:kind:
 GsMethod 3408897,   SequenceableCollection >> =
 GsMethod 9650945,   Set >> includes:
 GsMethod 19050497,   Behavior >> allSelectors
 GsMethod 15063297,   Behavior >> selectors
 GsMethod 7654401,   GsPackagePolicy >> selectorsFor:into:
 GsMethod 1763585,   Behavior >> superClass
 GsMethod 20444929,   Collection >> asSortedArray
 GsMethod 1555457,   Collection >> asSortedCollection
 GsMethod 10663425,   metaSortedCollection >> withAll:
 GsMethod 10663681,   metaSortedCollection >> new
 GsMethod 9570561,   SortedCollection >> addAll:
 GsMethod 9570817,   SortedCollection >> add:
 GsMethod 9570049,   SortedCollection >> _findIndex:
 GsMethod 9574657,   SortedCollection >> _insertAll:at:
 GsMethod 1839617,   AbstractDictionary >> at:ifAbsentPut:
 GsMethod 2224385,   IdentityDictionary >> at:otherwise:
 GsMethod 42221825,   OBBrowser >> announcer
 GsMethod 42220289,   OBBrowser >> navigationPanel
 GsMethod 47279361,   OBPanel >> isNavigation
 GsMethod 47492353,   OBColumnPanel >> isNavigation
 GsMethod 47181057,   OBNode >> metaNode:
 GsMethod 499691777,   OBColumn >> listHolder
 GsMethod 41703937,   OBList >> fan
 GsMethod 47186433,   OBNode >> defaultChildNodes
 GsMethod 47070977,   OBMetaNode >> defaultChildrenForNode:
 GsMethod 47073281,   OBMetaNode >> defaultEdgesForNode:
 GsMethod 47074561,   OBMetaNode >> defaultEdges
 GsMethod 47078145,   OBMetaNode >> edges
 GsMethod 47022337,   OBMetaEdge >> isDefaultNavigation
 GsMethod 47023873,   OBMetaEdge >> navigation
 GsMethod 47800065,   OBDefaultEdgeNavigation >> isDefaultNavigation
 GsMethod 1556993,   Collection >> inject:into:
 GsMethod 120883969,   SequenceableCollection >> gather:
 GsMethod 47023617,   OBMetaEdge >> nodesForParent:
 GsMethod 47015425,   OBEnvironmentNode >> categories
 GsMethod 20743425,   SmalltalkProxy >> systemOrganization
 GsMethod 19323905,   SystemOrganizer >> categories
 GsMethod 19323649,   SystemOrganizer >> categoryDict
 GsMethod 18999297,   metaClassOrganizer >> cachedOrganizer
 GsMethod 2110977,   IdentityBag >> includesIdentical:
 GsMethod 2234881,   IdentityDictionary >> keysAndValuesDo:
 GsMethod 1778177,   Behavior >> isBehavior
 GsMethod 1597441,   Class >> isMeta
 GsMethod 3398145,   SequenceableCollection >> removeFrom:to:
 GsMethod 1545985,   Collection >> _sortPaths:directions:persistentRoot:
 GsMethod 1627649,   CharacterCollection >> asArrayOfPathTerms
 GsMethod 1553921,   Collection >> _indexObjectsFor:
 GsMethod 11426817,   metaPathEvaluator >> new
 GsMethod 10425089,   PathEvaluator >> initialize
 GsMethod 10419457,   PathEvaluator >> nsc:
 GsMethod 10428161,   PathEvaluator >> asMostSpecificType
 GsMethod 10431489,   PathEvaluator >> _calculateIvOffsets
 GsMethod 1755905,   Behavior >> elementConstraint
 GsMethod 10423297,   PathEvaluator >> _calculateIvOffsetsOnKind:
 GsMethod 10420225,   PathEvaluator >> _getConstraintsStartingOnKind:
 GsMethod 1750785,   Behavior >> _ivOffsetOf:
 GsMethod 10421249,   PathEvaluator >> _sortOn:directions:persistentRoot:
 GsMethod 12794113,   metaPathSorter >> on:directions:
 GsMethod 12500481,   PathSorter >> sortNodeClassForSortLevel:
 GsMethod 10430721,   PathEvaluator >> lastElementClass
 GsMethod 10424321,   PathEvaluator >> isIndexOnRootNsc
 GsMethod 1736193,   Behavior >> sortNodeClass
 GsMethod 11227649,   metaRangeEqualityIndex >> isBasicClass:
 GsMethod 1740033,   Behavior >> isSpecial
 GsMethod 1767937,   Behavior >> _isSpecial
 GsMethod 11294209,   metaIndexManager >> current
 GsMethod 10423041,   PathEvaluator >> nscRoot
 GsMethod 11193601,   metaSortNode >> new:
 GsMethod 9523457,   BtreeNode >> numElements:
 GsMethod 10297345,   SortNode >> totalElements:
 GsMethod 10297601,   SortNode >> sortLevel:
 GsMethod 10298113,   SortNode >> pathSorter:
 GsMethod 12503553,   PathSorter >> sortNodes:
 GsMethod 12503809,   PathSorter >> directions:
 GsMethod 12501505,   PathSorter >> incompletes:
 GsMethod 11293441,   metaIndexManager >> autoCommit
 GsMethod 10352385,   IndexManager >> autoCommit
 GsMethod 10355457,   IndexManager >> sessionAutoCommit
 GsMethod 10421761,   PathEvaluator >> _sortOn:using:persistentRoot:
 GsMethod 12502529,   PathSorter >> sortNodes
 GsMethod 12503041,   PathSorter >> incompletes
 GsMethod 2150401,   UnorderedCollection >> _asIdentityBag
 GsMethod 10300673,   SortNode >> sortLevel
 GsMethod 10350593,   IndexManager >> commitIndexMaintenance:at:
 GsMethod 10355201,   IndexManager >> shouldCommit
 GsMethod 12502273,   PathSorter >> sortInto:startingAt:
 GsMethod 10887169,   SortNodeArray >> sortInto:startingAt:incompletes:
 GsMethod 10298625,   SortNode >> pathSorter
 GsMethod 12502785,   PathSorter >> directions
 GsMethod 10885633,   SortNodeArray >> indexManager
 GsMethod 1950721,   Number >> even
 GsMethod 3362049,   StringKeyValueDictionary >> at:otherwise:
 GsMethod 3362817,   StringKeyValueDictionary >> _stringAt:caseSensitive:otherwise:
 GsMethod 3361793,   StringKeyValueDictionary >> at:put:
 GsMethod 3362561,   StringKeyValueDictionary >> _stringAt:put:caseSensitive:
 GsMethod 9595905,   ClassOrganizer >> categories
 GsMethod 1805057,   Array >> _addAllFromNsc:
 GsMethod 1557761,   Collection >> sortAscending
 GsMethod 1547521,   Collection >> sortAscending:persistentRoot:
 GsMethod 1546497,   Collection >> sortWith:persistentRoot:
 GsMethod 1624321,   CharacterCollection >> speciesForConversion
 GsMethod 1554433,   Collection >> _findRangeIndexWithPath:
 GsMethod 1777409,   Behavior >> inheritsFrom:
 GsMethod 1785857,   Behavior >> isSubclassOf:
 GsMethod 10583553,   metaBtreeNode >> entrySize
 GsMethod 10583297,   metaBtreeNode >> maxNumberOfElements
 GsMethod 3394305,   SequenceableCollection >> _asCollectionForSorting
 GsMethod 12506113,   PathSorter >> _addObject:inNodes:hasSecondarySort:
 GsMethod 10297089,   SortNode >> isFull
 GsMethod 10297857,   SortNode >> _at:put:forBtree:hasSecondarySort:
 GsMethod 10298369,   SortNode >> at:put:hasSecondarySort:
 GsMethod 10299137,   SortNode >> _insertKey:value:atIndex:hasSecondarySort:
 GsMethod 9519617,   BtreeNode >> _lastIndex
 GsMethod 9520897,   BtreeNode >> entrySize
 GsMethod 1811201,   Array >> _insertAt:from:fromStart:fromEnd:numToMoveDown:
 GsMethod 9486849,   BtreeLeafNode >> _insertEncryptionFor:value:startingAt:
 GsMethod 9524225,   BtreeNode >> _binarySearchCoveringKey:totalOrder:
 GsMethod 3378177,   SmallInteger >> quo:
 GsMethod 9524993,   BtreeNode >> _compareKey:lessThanEntryAt:
 GsMethod 9527041,   BtreeNode >> _compare:key:lessThanEntryAt:
 GsMethod 12946433,   metaBtreeComparisonForSort >> compareKey:lessThan:
 GsMethod 1622785,   CharacterCollection >> _idxForSortLessThan:
 GsMethod 1621249,   CharacterCollection >> _idxForSortCharacterCollectionLessThanSelf:
 GsMethod 3336193,   String >> _idxPrimCompareLessThan:
 GsMethod 9521153,   BtreeNode >> _compareKey:greaterThanEntryAt:
 GsMethod 9523201,   BtreeNode >> _compare:key:greaterThanEntryAt:
 GsMethod 12945921,   metaBtreeComparisonForSort >> compareKey:greaterThan:
 GsMethod 1623297,   CharacterCollection >> _idxForSortGreaterThan:
 GsMethod 1621761,   CharacterCollection >> _idxForSortCharacterCollectionGreaterThanSelf:
 GsMethod 3335937,   String >> _idxPrimCompareGreaterThan:
 GsMethod 10299649,   SortNode >> sortInto:startingAt:
 GsMethod 1810433,   Array >> speciesForCollect
 GsMethod 48335361,   metaOBClassCategoryNode >> on:inEnvironment:
 GsMethod 42266369,   OBClassCategoryNode >> setName:environment:
 GsMethod 47180033,   OBNode >> navigation:
 GsMethod 19168257,   UndefinedObject >> ifNotNil:
 GsMethod 19078401,   Object >> changed:
 GsMethod 19084545,   Object >> dependents
 GsMethod 19085569,   Object >> myDependents
 GsMethod 19088129,   Object >> platformSupport
 GsMethod 26930945,   metaMCPlatformSupport >> dependentsFor:
 GsMethod 26918657,   metaMCPlatformSupport >> transientGlobalNamed:
 GsMethod 26917889,   metaMCPlatformSupport >> transientGlobals
 GsMethod 499486721,   OBList >> position
 GsMethod 19113729,   Symbol >> isUnary
 GsMethod 1648129,   CharacterCollection >> isInfix
 GsMethod 3343361,   String >> includes:
 GsMethod 3344897,   String >> includesValue:
 GsMethod 20441601,   Collection >> isCollection
 GsMethod 19164929,   Class >> allSubclasses
 GsMethod 21613569,   CachingClassOrganizer >> allSubclassesOf:
 GsMethod 41690625,   OBInteractionRequest >> defaultAction
 GsMethod 3299841,   SymbolDictionary >> at:
 GsMethod 44753665,   ClientForwarder >> doesNotUnderstand:
 GsMethod 44752897,   ClientForwarder >> perform:withArguments:
 GsMethod 3223297,   metaSystem >> _signalGciError:args:signalDictionary:
 GsMethod 1877761,   Object >> _gsReturnTos
 GsMethod 120902145,   JadeServer >> stringFromCollection:
 GsMethod 120902401,   JadeServer >> stringFromCollection:filter:
 GsMethod 1651457,   metaCharacter >> tab
 GsMethod 1650177,   metaCharacter >> withValue:
 GsMethod 3365633,   Stream >> tab
 GsMethod 48126721,   OBUpdateConcentrator >> dependentsMap
 GsMethod 19119617,   SequenceableCollection >> reversed
 GsMethod 48386561,   OBColumn >> selectedNode
 GsMethod 41699841,   OBList >> selectedNode
 GsMethod 41699329,   OBList >> selection
 GsMethod 47347969,   OBPosition >> index
 GsMethod 47870977,   OBFan >> nodeAt:
 GsMethod 47873025,   OBFan >> children
 GsMethod 19122433,   SequenceableCollection >> at:ifAbsent:
 GsMethod 1990401,   Magnitude >> between:and:
 GsMethod 19169025,   UndefinedObject >> ifNotNilDo:
 GsMethod 19170049,   UndefinedObject >> ifNil:ifNotNil:
 GsMethod 41671937,   JadeServer >> columnInfoFor:
 GsMethod 48391681,   OBColumn >> columnInfoFor:
 GsMethod 48386817,   OBColumn >> list
 GsMethod 41700097,   OBList >> list
 GsMethod 47873281,   OBFan >> list
 GsMethod 47867649,   OBFan >> displayStringForChild:
 GsMethod 47866881,   OBFan >> parent
 GsMethod 47183873,   OBNode >> displayStringForChild:
 GsMethod 47070465,   OBMetaNode >> displayStringForParent:child:
 GsMethod 47184129,   OBNode >> displayString
 GsMethod 47073025,   OBMetaNode >> displayStringForNode:
 GsMethod 47074817,   OBMetaNode >> displaySelector
 GsMethod 42267137,   OBClassCategoryNode >> name
 GsMethod 47317761,   OBFilter >> displayString:forNode:
 GsMethod 48398081,   OBColumn >> iconAt:
 GsMethod 41706241,   OBList >> iconAt:
 GsMethod 47866625,   OBFan >> iconAt:
 GsMethod 47189761,   OBNode >> icon
 GsMethod 47075073,   OBMetaNode >> iconForNode:
 GsMethod 47318785,   OBFilter >> icon:forNode:
 GsMethod 3091457,   UndefinedObject >> asString
 GsMethod 48386049,   OBColumn >> selection
 GsMethod 3388673,   metaSequenceableCollection >> with:with:with:
 GsMethod 48226049,   OBAnnouncer >> announce:
 GsMethod 38021889,   Behavior >> asAnnouncement
 GsMethod 21033217,   ActionSequence >> valueWithArguments:
 GsMethod 21048321,   MessageSend >> valueWithArguments:
 GsMethod 21046785,   MessageSend >> collectArguments:
 GsMethod 21047553,   MessageSend >> arguments
 GsMethod 1626113,   CharacterCollection >> numArgs
 GsMethod 1551873,   Collection >> identicalOccurrencesOf:
 GsMethod 1906689,   Object >> perform:withArguments:
 GsMethod 15062529,   Behavior >> _rawMethodDict
 GsMethod 47493889,   OBColumnPanel >> currentNode
 GsMethod 19144961,   Integer >> to:
 GsMethod 7209473,   metaInterval >> from:to:
 GsMethod 7258369,   Interval >> from:to:by:
 GsMethod 20419841,   metaString >> tab
 GsMethod 48384769,   OBColumn >> isEmpty
 GsMethod 41701377,   OBList >> isEmpty
 GsMethod 47868161,   OBFan >> isEmpty
 GsMethod 20426497,   Boolean >> and:and:
 GsMethod 20425985,   Boolean >> and:and:and:
 GsMethod 20436993,   Collection >> groupBy:having:
 GsMethod 1831681,   AbstractDictionary >> at:
 GsMethod 19127041,   SequenceableCollection >> do:separatedBy:
 GsMethod 499689985,   OBCommand >> clientExecute
 GsMethod 48127489,   OBUpdateConcentrator >> recordUpdatesDuring:
 GsMethod 48128257,   OBUpdateConcentrator >> canRecordUpdates
 GsMethod 19056129,   Behavior >> compiledMethodAt:ifAbsent:
 GsMethod 7654913,   GsPackagePolicy >> compiledMethodAt:for:
 GsMethod 7704065,   GsPackage >> compiledMethodAt:for:
 GsMethod 79167233,   CharacterCollection >> endsWith:
 GsMethod 19065089,   CharacterCollection >> includesSubstring:caseSensitive:
 GsMethod 19134721,   SequenceableCollection >> copyWithFirst:
 GsMethod 19073537,   metaCollection >> ofSize:
 GsMethod 37907457,   Class >> asNode
 GsMethod 48735233,   metaOBClassNode >> on:
 GsMethod 47156737,   OBClassNode >> setClass:
 GsMethod 15064065,   Behavior >> theNonMetaClass
 GsMethod 42253057,   OBClassAwareNode >> theClass:
 GsMethod 19082497,   Object >> canDiscardEdits
 GsMethod 19121409,   SequenceableCollection >> do:without:
 GsMethod 37924097,   Object >> asAnnouncement
 GsMethod 48394241,   OBColumn >> refreshAndSignal:
 GsMethod 41704449,   OBList >> refresh
 GsMethod 47868673,   OBFan >> refresh
 GsMethod 1909761,   Object >> ~=
 GsMethod 47185921,   OBNode >> =
 GsMethod 48116225,   OBDependentProxy >> update:
 GsMethod 48116481,   OBDependentProxy >> concentrator
 GsMethod 48129537,   OBUpdateConcentrator >> update:for:
 GsMethod 48128513,   OBUpdateConcentrator >> shouldRecordUpdates
 GsMethod 44752385,   ClientForwarder >> update:
 GsMethod 44754177,   ClientForwarder >> perform:with:
 GsMethod 47155713,   OBClassNode >> =
 GsMethod 3241473,   metaSystem >> signal:args:signalDictionary:
 GsMethod 500186881,   metaExceptionA >> gsNumber:arguments:
 GsMethod 42254849,   OBClassAwareNode >> theClass
 GsMethod 40764929,   OBModalFilter >> edgesFrom:forNode:
 GsMethod 40765953,   OBModalFilter >> selection
 GsMethod 47319041,   OBFilter >> edgesFrom:forNode:
 GsMethod 42266113,   OBClassCategoryNode >> rawClasses
 GsMethod 42268673,   OBClassCategoryNode >> classNames
 GsMethod 19325953,   SystemOrganizer >> listAtCategoryNamed:
 GsMethod 47319809,   OBFilter >> nodesFrom:forNode:
 GsMethod 41314561,   OBClassSortFilter >> nodesFrom:forNode:
 GsMethod 1550081,   Collection >> asOrderedCollection
 GsMethod 164894721,   Collection >> ifEmpty:
 GsMethod 41314305,   OBClassSortFilter >> initializeResults
 GsMethod 19054849,   Behavior >> withAllSuperclasses
 GsMethod 1778433,   Behavior >> allSuperClasses
 GsMethod 3399425,   SequenceableCollection >> removeAllSuchThat:
 GsMethod 20438785,   Collection >> contents
 GsMethod 1890817,   Object >> ->
 GsMethod 41313537,   OBClassSortFilter >> buildIndentIndex
 GsMethod 47155969,   OBClassNode >> hash
 GsMethod 41313793,   OBClassSortFilter >> collectSortedNodes
 GsMethod 48285953,   metaOBAllMethodCategoryNode >> on:
 GsMethod 47257601,   metaOBMethodCategoryNode >> on:inClass:
 GsMethod 40824065,   OBMethodCategoryNode >> setName:class:
 GsMethod 500121857,   OBClassNode >> categories
 GsMethod 15059969,   Behavior >> sortedCategoryNames
 GsMethod 15060225,   Behavior >> categoryNames
 GsMethod 10803713,   metaSet >> new:
 GsMethod 7662337,   GsPackagePolicy >> categoryNamesFor:into:
 GsMethod 7708929,   GsPackage >> categoryNamesFor:into:
 GsMethod 19083265,   Object >> ifNotNilDo:
 GsMethod 19076097,   Object >> ifNil:ifNotNilDo:
 GsMethod 19120897,   SequenceableCollection >> keysAndValuesDo:
 GsMethod 172937985,   Object >> ifNotNil:ifNil:
 GsMethod 47318529,   OBFilter >> displayString:forParent:child:
 GsMethod 41313281,   OBClassSortFilter >> displayString:forParent:child:
 GsMethod 3396609,   SequenceableCollection >> replaceFrom:to:with:
 GsMethod 19057665,   CharacterCollection >> isText
 GsMethod 19168513,   UndefinedObject >> ifNil:ifNotNilDo:
 GsMethod 3322881,   Symbol >> ,
 GsMethod 3399169,   SequenceableCollection >> removeAtIndex:
 GsMethod 40824577,   OBMethodCategoryNode >> =
 GsMethod 40826625,   OBMethodCategoryNode >> name
 GsMethod 42235137,   OBAllMethodCategoryNode >> methodReferences
 GsMethod 21080833,   MethodReference >> setStandardClass:methodSymbol:
 GsMethod 499580673,   MethodReference >> asNode
 GsMethod 48762625,   metaOBMethodNode >> on:
 GsMethod 47337729,   OBMethodNode >> setReference:
 GsMethod 21082881,   MethodReference >> methodSymbol
 GsMethod 21081857,   MethodReference >> actualClass
 GsMethod 47337985,   OBMethodNode >> setSelector:class:
 GsMethod 47331073,   OBMethodNode >> displayString
 GsMethod 12153345,   metaMessageNotUnderstood >> errorCategoryToCatch
 GsMethod 12153601,   metaMessageNotUnderstood >> errorNumberToCatch
 GsMethod 12153857,   metaMessageNotUnderstood >> errorNumberToSignal
 GsMethod 38023681,   Behavior >> includesLocalSelector:
 GsMethod 15061505,   Behavior >> includesSelector:
 GsMethod 41747457,   OBInheritanceFilter >> icon:forNode:
 GsMethod 47328513,   OBMethodNode >> selector
 GsMethod 19045121,   Behavior >> lookupSelector:
 GsMethod 3010305,   GsMethod >> _allBreakpoints
 GsMethod 3009793,   GsMethod >> _selectorPool
 GsMethod 3008257,   GsMethod >> literals
 GsMethod 20449025,   UnorderedCollection >> ,
 GsMethod 2093057,   IdentitySet >> addAll:
 GsMethod 47339521,   OBMethodNode >> isOverride
 GsMethod 19047425,   Behavior >> allSuperclassesDo:
 GsMethod 47331841,   OBMethodNode >> isOverridden
 GsMethod 19163393,   Class >> allSubclassesDo:
 GsMethod 19045889,   Behavior >> environment
 GsMethod 37908225,   Class >> ogDefinition
 GsMethod 24999937,   Class >> _ogDefinitionInContext:
 GsMethod 25000193,   Class >> _ogDefinitionInContext:className:superClassName:
 GsMethod 19045377,   Behavior >> isTransientDB
 GsMethod 3353089,   String >> addLast:
 GsMethod 19154177,   Class >> _sortedClassVarNames
 GsMethod 19048449,   Behavior >> classVarNames
 GsMethod 3302657,   SymbolDictionary >> keys
 GsMethod 2176001,   UnorderedCollection >> select:
 GsMethod 2179329,   UnorderedCollection >> _select:
 GsMethod 10616577,   metaNscBuilder >> for:max:
 GsMethod 10616065,   metaNscBuilder >> maxSize
 GsMethod 9542913,   NscBuilder >> counter:
 GsMethod 9543169,   NscBuilder >> finalResult:
 GsMethod 9544193,   NscBuilder >> nsc:
 GsMethod 9542657,   NscBuilder >> completeBag
 GsMethod 9543425,   NscBuilder >> _completeNsc
 GsMethod 1770753,   Behavior >> sharedPools
 GsMethod 1640193,   CharacterCollection >> printString
 GsMethod 1639937,   CharacterCollection >> printStringWithMaxSize:
 GsMethod 10130945,   metaPrintStream >> printingOn:maxSize:
 GsMethod 9776641,   PrintStream >> _initStreamWith:maxSize:
 GsMethod 3339777,   String >> printOn:
 GsMethod 9779457,   PrintStream >> _nextPut:
 GsMethod 3353345,   String >> indexOf:startingAt:
 GsMethod 9779201,   PrintStream >> nextPutAll:
 GsMethod 9775873,   PrintStream >> isFull
 GsMethod 9778177,   PrintStream >> contents
 GsMethod 47336961,   OBMethodNode >> definition
 GsMethod 47329025,   OBMethodNode >> selection
 GsMethod 47326721,   OBMethodNode >> source
 GsMethod 19044097,   Behavior >> sourceCodeAt:ifAbsent:
 GsMethod 15063809,   Behavior >> sourceCodeAt:
 GsMethod 3020033,   GsMethod >> _sourceString
 GsMethod 49304321,   metaOBMethodDefinition >> selection:source:inClass:
 GsMethod 49304577,   metaOBMethodDefinition >> selection:source:inCategory:inClass:
 GsMethod 499463681,   OBMethodDefinition >> setClass:category:source:selection:
 GsMethod 47268097,   OBMethodDefinition >> text
 GsMethod 1763073,   Behavior >> allClassVarNames
 GsMethod 3402241,   SequenceableCollection >> insert:at:
 GsMethod 1889793,   Object >> _validateClass:
 GsMethod 1761281,   Behavior >> allSharedPools
 GsMethod 79246337,   SmalltalkProxy >> bindingOf:
 GsMethod 20741377,   SmalltalkProxy >> associationAt:
 GsMethod 19056641,   CharacterCollection >> translated
 GsMethod 19122945,   SequenceableCollection >> second
 GsMethod 19125761,   SequenceableCollection >> checkedAt:
 GsMethod 47852545,   OBDefinitionPanel >> getDefinition
 GsMethod 42222593,   OBBrowser >> currentNode
 GsMethod 19056385,   metaBehavior >> parseSelector:for:
 GsMethod 1947393,   Number >> negative
 GsMethod 15057665,   Behavior >> compiledMethodAt:
 GsMethod 20444161,   Collection >> anyOne
 GsMethod 20439041,   Collection >> emptyCheck
 GsMethod 19124993,   SequenceableCollection >> atAll:
 GsMethod 19100929,   String >> findSelector
 GsMethod 19069953,   CharacterCollection >> withBlanksTrimmed
 GsMethod 19048705,   Behavior >> whichCategoryIncludesSelector:
 GsMethod 15057409,   Behavior >> categoryOfSelector:
 GsMethod 326735873,   Behavior >> compileMethod:category:using:
 GsMethod 15065601,   Behavior >> stampForMethod:
 GsMethod 269875713,   Behavior >> _compileMethod:category:using:
 GsMethod 15059201,   Behavior >> compileMethod:dictionaries:category:
 GsMethod 15056641,   Behavior >> changeStamp
 GsMethod 15064577,   Behavior >> authorInitials
 GsMethod 15065345,   Behavior >> setStamp:forMethod:
 GsMethod 19055361,   Behavior >> createPragmaFrom:for:
 GsMethod 15065089,   Behavior >> setPragmas:forMethod:
 GsMethod 20442881,   Collection >> isEmptyOrNil
 GsMethod 19064065,   CharacterCollection >> findString:startingAt:caseSensitive:
 GsMethod 47335937,   OBMethodNode >> =
 GsMethod 3230465,   metaSystem >> commitTransaction
 GsMethod 3222785,   metaSystem >> _commit:
 GsMethod 3159553,   metaSystem >> _commitCoordinator
 GsMethod 7713281,   TransactionBoundaryDefaultPolicy >> commit:
 GsMethod 3222529,   metaSystem >> _localCommit:
 GsMethod 3223041,   metaSystem >> _processDeferredGciUpdates
 GsMethod 3223553,   metaSystem >> _pendingCommitAbort
 GsMethod 3222017,   metaSystem >> __commit:
 GsMethod 3226881,   metaSystem >> _primitiveCommit:
 GsMethod 20641793,   SessionMethodTransactionBoundaryPolicy >> transactionBoundary:
 GsMethod 20637441,   SessionMethodTransactionBoundaryPolicy >> cachedClassChangeCount
 GsMethod 20637697,   SessionMethodTransactionBoundaryPolicy >> cacheDictionary
 GsMethod 20638209,   SessionMethodTransactionBoundaryPolicy >> cacheName
 GsMethod 20637953,   SessionMethodTransactionBoundaryPolicy >> classChangeCountCacheName
 GsMethod 18998785,   metaClassOrganizer >> _resetCachedOrganizer
 GsMethod 20638465,   SessionMethodTransactionBoundaryPolicy >> cachedSessionMethodChangeCount
 GsMethod 20638721,   SessionMethodTransactionBoundaryPolicy >> sessionMethodChangeCountCacheName
 GsMethod 48076545,   OBCodeBrowser >> signalRefresh
 GsMethod 42214913,   OBBrowser >> signalRefresh
 GsMethod 47853569,   OBDefinitionPanel >> refresh:
 GsMethod 48397057,   OBColumn >> refresh:
 GsMethod 41674497,   JadeServer >> doIt:in:
 GsMethod 499568641,   JadeServer >> evaluate:inContext:
 GsMethod 3340545,   String >> evaluateInContext:symbolList:
 GsMethod 3341057,   String >> _compileInContext:symbolList:
 GsMethod 3026177,   GsMethod >> _executeInContext:
 GsMethod 3019265,   GsMethod >> _primitiveExecuteInContext:
 GsMethod 1,  classId 20 selectorId 0
 GsMethod 21088257,   metaTranscriptProxy >> cr
 GsMethod 21087233,   metaTranscriptProxy >> proxyTranscript
 GsMethod 11804161,   metaDateAndTimeANSI >> now
 GsMethod 20524289,   DateAndTimeANSI >> initializeAsNow
 GsMethod 12067329,   metaDateAndTime >> secondsSince2001
 GsMethod 3188737,   metaSystem >> _timeGmtFloat
 GsMethod 7279105,   SmallDouble >> -
 GsMethod 3382529,   SmallInteger >> asFloat
 GsMethod 11974913,   metaTimeZone >> current
 GsMethod 11971073,   TimeZoneInfo >> transitionAtUTC:
 GsMethod 10005505,   DateAndTimeANSI >> asPosixSeconds
 GsMethod 7276033,   SmallDouble >> +
 GsMethod 11970305,   TimeZoneInfo >> detectLastTransition:
 GsMethod 10023169,   TimeZoneTransition >> transitionTimeUTC
 GsMethod 2075649,   Integer >> <=
 GsMethod 3377921,   SmallInteger >> _generality
 GsMethod 7287809,   SmallDouble >> _generality
 GsMethod 7274753,   SmallDouble >> _coerce:
 GsMethod 7281665,   SmallDouble >> <=
 GsMethod 10024193,   TimeZoneTransition >> offsetFromUTC
 GsMethod 10002177,   DateAndTimeANSI >> printString
 GsMethod 13488129,   Object >> printString
 GsMethod 10131457,   metaPrintStream >> printingOn:
 GsMethod 10011137,   DateAndTimeANSI >> printOn:
 GsMethod 10002689,   DateAndTimeANSI >> printAnsiOn:
 GsMethod 10011905,   DateAndTimeANSI >> asFloatParts
 GsMethod 10012161,   DateAndTimeANSI >> partsFrom:
 GsMethod 1951745,   Number >> \\
 GsMethod 1953281,   Number >> //
 GsMethod 7278593,   SmallDouble >> /
 GsMethod 7287553,   SmallDouble >> floor
 GsMethod 269790721,   DateAndTimeANSI >> print:on:
 GsMethod 1950209,   Number >> printOn:
 GsMethod 9776129,   PrintStream >> nextPut:
 GsMethod 79193345,   Number >> _printAsDateAndTimeANSISecondsOn:
 GsMethod 7277569,   SmallDouble >> =
 GsMethod 7283713,   SmallDouble >> asFloat
 GsMethod 7275265,   SmallDouble >> asStringLocaleC
 GsMethod 7278081,   SmallDouble >> _mathPrim:
 GsMethod 10008577,   DateAndTimeANSI >> offset
 GsMethod 11818497,   metaDuration >> seconds:
 GsMethod 10016257,   Duration >> _seconds:
 GsMethod 10016513,   Duration >> printOnHoursMinutes:
 GsMethod 10019329,   Duration >> hours
 GsMethod 1952769,   Number >> negated
 GsMethod 1936897,   Number >> sign
 GsMethod 10017025,   Duration >> minutes
 GsMethod 9778689,   PrintStream >> _collection
 GsMethod 79247105,   metaTranscriptProxy >> show:
 GsMethod 79243009,   metaObjectLogEntry >> transcript:object:
 GsMethod 20839169,   ObjectLogEntry >> initialize
 GsMethod 3214337,   metaSystem >> gemVersionReport
 GsMethod 3204609,   metaSystem >> _serverVersionReport:
 GsMethod 20839425,   ObjectLogEntry >> timestamp
 GsMethod 20839937,   ObjectLogEntry >> label:object:
 GsMethod 79243265,   metaObjectLogEntry >> transcript
 GsMethod 20839681,   ObjectLogEntry >> priority:
 GsMethod 20842497,   ObjectLogEntry >> addToLog
 GsMethod 22586369,   metaObjectLogEntry >> objectQueue
 GsMethod 12454913,   RcQueue >> add:
 GsMethod 12460033,   RcQueue >> _getSessionComponentFor:
 GsMethod 12452609,   RcQueue >> maxSessionId
 GsMethod 1892865,   Object >> _rcAt:
 GsMethod 11579137,   RcQueueElement >> sequenceNumber
 GsMethod 11637761,   metaRcQueueSessionComponent >> speciesOfElements
 GsMethod 12927745,   metaRcQueueEntry >> newWithValue:sequenceNumber:inSegmentOf:
 GsMethod 12449793,   RcQueue >> _findFirstValidQueueElementForSession:
 GsMethod 179615745,   metaWABasicDevelopment >> reapSeasideCache
 GsMethod 178838529,   metaWADispatcher >> default
 GsMethod 3398657,   SequenceableCollection >> removeFirst
 GsMethod 191317761,   WARequestHandler >> isApplication
 GsMethod 179091969,   WADispatcher >> handlers
 GsMethod 179616513,   metaWABasicDevelopment >> reapHandlerCache:dispatchers:
 GsMethod 202641409,   WAApplication >> isApplication
 GsMethod 176151809,   WARegistry >> cache
 GsMethod 119004929,   metaGRPlatform >> current
 GsMethod 499656961,   GRGemStonePlatform >> doCommitTransaction
 GsMethod 3225857,   metaSystem >> inTransaction
 GsMethod 119189249,   SystemCommitTransaction >> defaultAction
 GsMethod 3151105,   metaSystem >> _commitPrintingDiagnostics
 GsMethod 1923073,   Object >> halt
 GsMethod 19074561,   Object >> halt:
 GsMethod 22476545,   metaControlInterrupt >> signal:
 GsMethod 21788417,   ControlInterrupt >> signal
 GsMethod 79226881,   Halt >> defaultAction
 GsMethod 10380033,   RcKeyValueDictionary >> associationsDo:
 GsMethod 10381825,   RcKeyValueDictionary >> keysAndValuesDo:
 GsMethod 10383873,   RcKeyValueDictionary >> _doCollisionBuckets:
 GsMethod 9549057,   RcCollisionBucket >> keyAt:
 GsMethod 9549313,   RcCollisionBucket >> valueAt:
 GsMethod 3033857,   IdentityKeyValueDictionary >> compareKey:with:
 GsMethod 41680129,   JadeServer >> printIt:in:
 GsMethod 1,  classId 20 selectorId 0
 GsMethod 1926913,   Object >> doesNotUnderstand:
 GsMethod 48056577,   metaOTIndexedVariableNode >> on:index:
 GsMethod 42066689,   OTIndexedVariableNode >> setIndex:
 GsMethod 47982081,   OTDictionaryInspectorNode >> name
 GsMethod 47980801,   OTDictionaryInspectorNode >> key
 GsMethod 37922305,   Object >> inspectorName
 GsMethod 47981313,   OTDictionaryInspectorNode >> =
 GsMethod 46414337,   OTCmdOutDebugger >> execute
 GsMethod 40810497,   OGStackFrame >> =
 GsMethod 44751361,   ComplexBlock >> blockInspectorNodes
 GsMethod 37938689,   ExecutableBlock >> basicInspectorNodes
 GsMethod 499592449,   ComplexBlock >> derivedInspectorNodes
 GsMethod 499736833,   ExecutableBlock >> derivedInspectorNodes
 GsMethod 1,  classId 20 selectorId 0
 GsMethod 105770960385,   WACache >> gemstoneReap
 GsMethod 499446017,   OTCmdProceedDebugger >> execute
 GsMethod 49870849,   metaOGDebuggerProceedRequest >> signal:
 GsMethod 47216385,   OGDebuggerProceedRequest >> debugger:
 GsMethod 47216641,   OGDebuggerProceedRequest >> handleWith:
 GsMethod 47117313,   metaOBGemStonePlatform >> handleDebuggerProceedRequest:
 GsMethod 47216129,   OGDebuggerProceedRequest >> debugger
 GsMethod 1635329,   CharacterCollection >> copyReplaceAll:with:
 GsMethod 1634561,   CharacterCollection >> findString:startingAt:
 GsMethod 3337729,   String >> removeFrom:to:
 GsMethod 3343105,   String >> insertAll:at:
 GsMethod 19142913,   metaCharacter >> separators
 GsMethod 19143937,   metaCharacter >> value:
 GsMethod 1548289,   Collection >> occurrencesOf:
 GsMethod 47969537,   OTDefinitionPanel >> doItReceiver
 GsMethod 41678849,   JadeServer >> inspectIt:in:
 GsMethod 41674753,   JadeServer >> reportCompileErrors:
 GsMethod 47116801,   metaOBGemStonePlatform >> gsParserResolvePartial:forClass:
 GsMethod 19049729,   Behavior >> classPool
 GsMethod 1772289,   Behavior >> _classVars
 GsMethod 19043073,   AbstractDictionary >> hasBindingThatBeginsWith:
 GsMethod 38012929,   SmalltalkProxy >> hasBindingThatBeginsWith:
 GsMethod 47905793,   OTContextTempVariableNode >> =
 GsMethod 47906305,   OTContextTempVariableNode >> value
 GsMethod 40816385,   OGStackFrame >> printOn:
 GsMethod 1,  classId 20 selectorId 0
 GsMethod 105770745345,   WACache >> gemstoneReap
 GsMethod 176972545,   WACache >> expiryPolicy
 GsMethod 499561473,   WARcLastAccessExpiryPolicy >> isExpiredUpdating:key:
 GsMethod 10382337,   RcKeyValueDictionary >> at:ifAbsent:
 GsMethod 10386433,   RcKeyValueDictionary >> hashFunction:
 GsMethod 176140289,   WARcLastAccessExpiryPolicy >> timeout
 GsMethod 177983489,   WACachePlugin >> configuration
 GsMethod 177472001,   WAConfiguration >> at:
 GsMethod 177474305,   WAConfiguration >> getSearchContextFor:
 GsMethod 177472513,   WAConfiguration >> retrieveSearchContext:ifAbsentPut:
 GsMethod 178009345,   WAAttributeSearchContext >> attribute
 GsMethod 177473537,   WAConfiguration >> valueForContext:
 GsMethod 178010113,   WAAttributeSearchContext >> at:ifPresent:
 GsMethod 178011393,   WAAttributeSearchContext >> cachedValues
 GsMethod 19042817,   AbstractDictionary >> at:ifPresent:
 GsMethod 177908737,   WAAttribute >> default
 GsMethod 177431297,   WARcLastAccessEntry >> isExpiredUpdating:
 GsMethod 177431553,   WARcLastAccessEntry >> isExpired:updating:
 GsMethod 177430529,   WARcLastAccessEntry >> counter
 GsMethod 12494849,   RcCounter >> value
 GsMethod 12492545,   RcCounter >> _calculateValue
 GsMethod 10954241,   RcCounterElement >> value
 GsMethod 19117569,   metaTime >> totalSeconds
 GsMethod 11819265,   metaDuration >> zero
 GsMethod 11806209,   metaDateAndTimeANSI >> secondsUTC:offset:
 GsMethod 20523777,   DateAndTimeANSI >> _secondsUTC:offset:
 GsMethod 10017793,   Duration >> asSeconds
 GsMethod 10005249,   DateAndTimeANSI >> asSeconds
 GsMethod 1949953,   Number >> asInteger
 GsMethod 7286529,   SmallDouble >> truncated
 GsMethod 176976129,   WACache >> notifyRemoved:key:
 GsMethod 176969985,   WACache >> pluginsDo:
 GsMethod 176139777,   WARcLastAccessExpiryPolicy >> removed:key:
 GsMethod 10379777,   RcKeyValueDictionary >> removeKey:ifAbsent:
 GsMethod 10382849,   RcKeyValueDictionary >> _logRemoveKey:inCollisionBucket:
 GsMethod 9552897,   LogEntry >> receiver:
 GsMethod 9553153,   LogEntry >> selector:
 GsMethod 9552641,   LogEntry >> argArray:
 GsMethod 3218689,   metaSystem >> redoLog
 GsMethod 10738945,   metaRedoLog >> new
 GsMethod 9613313,   RedoLog >> initialize
 GsMethod 3372545,   metaSmallInteger >> maximumValue
 GsMethod 2277889,   KeyValueDictionary >> collisionLimit:
 GsMethod 9611521,   RedoLog >> addLogEntry:forConflictObject:
 GsMethod 9551617,   LogEntry >> receiver
 GsMethod 176974081,   WACache >> reapingStrategy
 GsMethod 177983745,   WACachePlugin >> removed:key:
 GsMethod 176977409,   WACache >> missStrategy
 GsMethod 176976641,   WACache >> removalAction
 GsMethod 177967617,   WANotifyRemovalAction >> removed:key:
 GsMethod 178559489,   WARequestHandler >> unregistered
 GsMethod 1,  classId 20 selectorId 0
 GsMethod 105770596353,   WACache >> gemstoneReap
 GsMethod 1,  classId 20 selectorId 0
 GsMethod 105767327233,   WACache >> gemstoneReap
 GsMethod 119025921,   metaGRObject >> new
 GsMethod 177432065,   WARcLastAccessEntry >> initialize
 GsMethod 12780033,   metaRcCounter >> new:
 GsMethod 12495873,   RcCounter >> initialize
 GsMethod 12493825,   RcCounter >> _getSessionElementFor:
 GsMethod 11641345,   metaRcCounterElement >> new
 GsMethod 10954497,   RcCounterElement >> initialize
 GsMethod 10954753,   RcCounterElement >> value:
 GsMethod 9611009,   RedoLog >> addConflictObject:for:
 GsMethod 12492289,   RcCounter >> increment
 GsMethod 12494081,   RcCounter >> _privateIncrementBy:
 GsMethod 12493569,   RcCounter >> _thisSessionIndex
 GsMethod 10953985,   RcCounterElement >> incrementValueBy:
 GsMethod 178284801,   WADocumentHandler >> hash
 GsMethod 178284289,   WADocumentHandler >> document
 GsMethod 177157121,   WAMimeDocument >> hash
 GsMethod 177155329,   WAMimeDocument >> mimeType
 GsMethod 177332481,   WAMimeType >> hash
 GsMethod 177332737,   WAMimeType >> main
 GsMethod 177331713,   WAMimeType >> sub
 GsMethod 177156353,   WAMimeDocument >> fileName
 GsMethod 177157377,   WAMimeDocument >> contents
 GsMethod 178284545,   WADocumentHandler >> =
 GsMethod 177156865,   WAMimeDocument >> =
 GsMethod 177331969,   WAMimeType >> =
 GsMethod 3294721,   SymbolDictionary >> removeKey:
 GsMethod 2217729,   IdentityDictionary >> removeKey:
 GsMethod 2273537,   KeyValueDictionary >> _removeKey:ifAbsent:
 GsMethod 2202625,   IdentityCollisionBucket >> removeKey:ifAbsent:
 GsMethod 202641153,   WAApplication >> sessionsDo:
 GsMethod 176154881,   WARegistry >> handlersDo:
 GsMethod 176155393,   WARegistry >> keysAndHandlersDo:
 GsMethod 176971777,   WACache >> keysAndValuesDo:
 GsMethod 191315713,   WARequestHandler >> isSession
 GsMethod 178564097,   WARequestHandler >> isDispatcher
 GsMethod 3035137,   IdentityKeyValueDictionary >> rebuildTable:
 GsMethod 202211841,   WASession >> isSession
 GsMethod 202211329,   WASession >> continuations
 GsMethod 177481729,   metaWAAttributeSearchContext >> key:target:
 GsMethod 178010369,   WAAttributeSearchContext >> initializeWithKey:
 GsMethod 119023873,   GRObject >> initialize
 GsMethod 178011137,   WAAttributeSearchContext >> findAttributeAndSelectAncestorsOf:
 GsMethod 178009601,   WAAttributeSearchContext >> key
 GsMethod 177473793,   WAConfiguration >> localAttributeAt:ifPresent:
 GsMethod 177920257,   WAUserConfiguration >> localAttributeAt:ifAbsent:
 GsMethod 177924865,   WAUserConfiguration >> parents
 GsMethod 177361153,   WASystemConfiguration >> localAttributeAt:ifAbsent:
 GsMethod 177361409,   WASystemConfiguration >> attributes
 GsMethod 177362945,   WASystemConfiguration >> description
 GsMethod 177375233,   WAConfigurationDescription >> attributes
 GsMethod 177473281,   WAConfiguration >> expressionAt:ifPresent:
 GsMethod 177920769,   WAUserConfiguration >> expressionAt:ifAbsent:
 GsMethod 178011649,   WAAttributeSearchContext >> isAttributeLocalOn:
 GsMethod 177471233,   WAConfiguration >> inheritedValueForContext:
 GsMethod 178010881,   WAAttributeSearchContext >> isAttributeInheritedOn:
 GsMethod 178009857,   WAAttributeSearchContext >> at:put:
(vmGc  Instances counts for generation code
     GsMethod                                id:     98817    1025 instances   320176 bytes
 vmGc ---)
-----------------------------------------------------
GemStone: Error         Fatal
VM temporary object memory is full
, almost out of memory, too many markSweeps since last successful
scavenge
Error Category: 231169 [GemStone] Number: 4067 Arg Count: 1 Context : 20
Arg 1:   20

[Info]: Logging out at 03/27/2015 04:11:47 PM UTC


*****************************************************
****** GemStone Abnormal Shutdown at 03/27/2015 04:11:48 PM UTC
*****************************************************
-----------------------------------------------------
GemStone: Error         Fatal
VM temporary object memory is full
, almost out of memory, too many markSweeps since last successful
scavenge
Error Category: 231169 [GemStone] Number: 4067 Arg Count: 1 Context : 20
Arg 1:   20
~                                                                                                                                                                                                      
Type  :qu


Dale

On 03/27/2015 10:04 AM, Lawrence Kellogg wrote:

On Mar 27, 2015, at 12:48 PM, Dale Henrichs <[hidden email]> wrote:

Larry,

Yea the bulk of the progress that you've made should be committed... should probably add a final transcript show in the gemstoneReap method ... but when your `Expire Progress` matches the preceding `finished scan` that chunk of work won't be done again ...

At this point in time, I suggest that you make a backup in case we hit a problem (like run out of extent space or something) that would make things difficult to proceed from ... this is just precautionary, but prudent ...

I am interested in what caused the out of memory so checking in the gem log for a smalltalk stack when you ran out of memory would help ... it's possible that we'll just hit the same situation right off the bat, when we start process again, so might as well try to understand that one...


Well, the beginning of a very long stack dump is this: 

GsMethod 1926657,   Object >> _doesNotUnderstand:
 GsMethod 10662913,   metaSortedCollection >> sortBlock:
 GsMethod 1754881,   Behavior >> new
 GsMethod 9573889,   SortedCollection >> sortBlock:
 GsMethod 1924097,   Object >> size
 GsMethod 3381249,   SmallInteger >> >
 GsMethod 2242817,   metaKeyValueDictionary >> new
 GsMethod 2243841,   metaKeyValueDictionary >> new:
 GsMethod 2057729,   metaInteger >> _selectedPrimeGreaterThan:
 GsMethod 1924609,   Object >> at:
 GsMethod 3378689,   SmallInteger >> //
 GsMethod 3378945,   SmallInteger >> <=
 GsMethod 1753857,   Behavior >> _basicNew:
 GsMethod 2257665,   KeyValueDictionary >> _initializeWithoutClear:
 GsMethod 3181569,   metaSystem >> _sessionStateAt:put:
 GsMethod 3206913,   metaSystem >> myUserProfile
 GsMethod 3294977,   SymbolDictionary >> at:ifAbsent:
 GsMethod 1534721,   ComplexBlock >> value
 GsMethod 7693313,   metaGsSession >> currentSession
 GsMethod 7723777,   GsCurrentSession >> objectNamed:
 GsMethod 3275521,   SymbolList >> resolveSymbol:
 GsMethod 1795329,   Association >> value
 GsMethod 7592193,   metaGsPackagePolicy >> globalName
 GsMethod 3300353,   SymbolDictionary >> at:otherwise:
 GsMethod 3297281,   SymbolDictionary >> associationAt:otherwise:
 GsMethod 7660545,   GsPackagePolicy >> enabled
 GsMethod 7640833,   metaSessionTemps >> current
 GsMethod 3182081,   metaSystem >> _sessionStateAt:
 GsMethod 7657217,   GsPackagePolicy >> sessionMethodDictionaryGlobalName
 GsMethod 3299585,   SymbolDictionary >> at:put:
 GsMethod 3293953,   SymbolDictionary >> _validatePrivilegeOld:new:
 GsMethod 1789185,   metaAssociation >> newWithKey:value:
 GsMethod 3307777,   SymbolAssociation >> key:value:
 GsMethod 3308033,   SymbolAssociation >> key:
 GsMethod 1796865,   Association >> value:
 GsMethod 3300097,   SymbolDictionary >> _at:put:
 GsMethod 3299073,   SymbolDictionary >> _validatePrivilegeGeneric:
 GsMethod 3140097,   metaSystem >> _protectedMode
 GsMethod 3215873,   metaSystem >> _zeroArgPrim:
 GsMethod 2278145,   KeyValueDictionary >> _at:put:
 GsMethod 2230273,   IdentityDictionary >> hashFunction:
 GsMethod 1910529,   Object >> identityHash
 GsMethod 2272513,   KeyValueDictionary >> tableSize
 GsMethod 3377665,   SmallInteger >> \\
 GsMethod 2263297,   KeyValueDictionary >> keyAt:
 GsMethod 1923585,   Object >> _at:
 GsMethod 2267649,   KeyValueDictionary >> valueAt:
 GsMethod 1916417,   Object >> _basicAt:
 GsMethod 1699329,   Boolean >> |
 GsMethod 2276609,   KeyValueDictionary >> atHash:putKey:
 GsMethod 1922561,   Object >> _basicAt:put:
 GsMethod 2221569,   IdentityDictionary >> atHash:putValue:
 GsMethod 2256129,   KeyValueDictionary >> atHash:putValue:
 GsMethod 7661313,   GsPackagePolicy >> sessionMethodDictionary
 GsMethod 1900033,   Object >> class
 GsMethod 2278401,   KeyValueDictionary >> size
 GsMethod 7652097,   GsPackagePolicy >> symbolList
 GsMethod 7723521,   GsCurrentSession >> symbolList
 GsMethod 3395329,   SequenceableCollection >> reverseDo:
 GsMethod 1942785,   Number >> downTo:do:
 GsMethod 1943041,   Number >> downTo:by:do:

which ends with this:


 GsMethod 119023873,   GRObject >> initialize
 GsMethod 178011137,   WAAttributeSearchContext >> findAttributeAndSelectAncestorsOf:
 GsMethod 178009601,   WAAttributeSearchContext >> key
 GsMethod 177473793,   WAConfiguration >> localAttributeAt:ifPresent:
 GsMethod 177920257,   WAUserConfiguration >> localAttributeAt:ifAbsent:
 GsMethod 177924865,   WAUserConfiguration >> parents
 GsMethod 177361153,   WASystemConfiguration >> localAttributeAt:ifAbsent:
 GsMethod 177361409,   WASystemConfiguration >> attributes
 GsMethod 177362945,   WASystemConfiguration >> description
 GsMethod 177375233,   WAConfigurationDescription >> attributes
 GsMethod 177473281,   WAConfiguration >> expressionAt:ifPresent:
 GsMethod 177920769,   WAUserConfiguration >> expressionAt:ifAbsent:
 GsMethod 178011649,   WAAttributeSearchContext >> isAttributeLocalOn:
 GsMethod 177471233,   WAConfiguration >> inheritedValueForContext:
 GsMethod 178010881,   WAAttributeSearchContext >> isAttributeInheritedOn:
 GsMethod 178009857,   WAAttributeSearchContext >> at:put:
(vmGc  Instances counts for generation code
     GsMethod                                id:     98817    1025 instances   320176 bytes
 vmGc ---)
-----------------------------------------------------
GemStone: Error         Fatal
VM temporary object memory is full
, almost out of memory, too many markSweeps since last successful
scavenge
Error Category: 231169 [GemStone] Number: 4067 Arg Count: 1 Context : 20
Arg 1:   20

[Info]: Logging out at 03/27/2015 04:11:47 PM UTC


*****************************************************
****** GemStone Abnormal Shutdown at 03/27/2015 04:11:48 PM UTC
*****************************************************
-----------------------------------------------------
GemStone: Error         Fatal
VM temporary object memory is full
, almost out of memory, too many markSweeps since last successful
scavenge
Error Category: 231169 [GemStone] Number: 4067 Arg Count: 1 Context : 20
Arg 1:   20
                     

I’ll make a backup and try again.

Larry



Dale

On 03/27/2015 09:31 AM, Lawrence Kellogg wrote:

On Mar 27, 2015, at 12:08 PM, Dale Henrichs <[hidden email]> wrote:

Larry,

Not sure ... I guess the I'm curious if the system is in iowait or burning cpu ... top should give you that information ... I'd also be interested to know if you've run out of disk space ... check the stone log for that information ...

Based on what you find out we'll go on from there ...



The struggle continues, eventually, I ran out of temporary Object Memory:

<Mail Attachment.png>


Disk space as at 63%. Can I run it again? Did any of that stick?

Larry





Dale

On 03/27/2015 07:43 AM, Lawrence Kellogg wrote:
Well, I put in Transcript calls to show count all the time, not just at mod 1000..and I made it all the way through, except it does’t seem to want to come back to me. The image is frozen.

Is it hung in the final commit? 

Here is a screen shot of the last part of the display. Yes I’m running it in GemTools. I don’t care if it’s slow as long as it works. 

<Mail Attachment.png>


Larry

On Mar 26, 2015, at 1:01 AM, Dale Henrichs <[hidden email]> wrote:

Hmmm, I think we need to set a breakpoint in there ... I might be tired, but I can't see how that can happen ... but then that's what a debugger is for ...

Are you running this expression from GemTools? If so the `Transcript show:` might be awfully expensive (lot's o round trips over the WAN) ... you might use GsFile gciLogServer: and then tail the gem log file on the server to see progress ...

Need to find out where the 0 is coming from first...

Dale

On 3/25/15 7:34 PM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, I replaced the WACache>>gemstoneReap method and ran the code from before but it just shows Scan Progress: 0 over and over again. 

  I let it run a few hours but had to kill it as the computer is in my bedroom and I can’t sleep if it makes noise all night. 

  Do I try again tomorrow? 

  best, 

  Larry

 
On Mar 25, 2015, at 6:23 PM, Dale Henrichs <[hidden email]> wrote:

Hmm, I scanned the code in WACache>>gemstoneReap and saw that a commit was being down every 100 entries, but presumably you blew up while building the collection of entries to expire ... sorry about that ... There should be a a stack dump in the gem log (depending upon where you ran the expressions where we can verify my assumption.

It looks like we need to replace WACache>>gemstoneReap with the following:

gemstoneReap
  "Iterate through the cache and remove objects that have expired."

  "In GemStone, this method is performed by a separate maintenance VM,
     so we are already in transaction (assumed to be running in #autoBegin
     transactionMode) and do not have to worry about acquiring the TransactionMutex.
    Since we are using reducedConflict dictionaries in the first place, we will remove the keys
    and values from the existing dictionaries without using the mutex."

  | expired count platform |
  expired := UserGlobals at: #'ExpiryCleanup' put: OrderedCollection new.
  platform := GRPlatform current.
  platform doCommitTransaction.
  count := 0.
  objectsByKey
    associationsDo: [ :assoc |
      (self expiryPolicy isExpiredUpdating: assoc value key: assoc key)
        ifTrue: [
          self notifyRemoved: assoc value key: assoc key.
          count := count + 1.
          expired add: assoc.
          count \\ 100 == 0
            ifTrue: [ platform doCommitTransaction ].
          count \\ 1000 == 0
            ifTrue: [ Transcript cr; show: 'Scan progress: ' , count printString.] ] ].
  Transcript cr; show: 'finished scan: ' , count printString.
  count := 0.
  (UserGlobals at: #'ExpiryCleanup' )
    do: [ :assoc |
      count := count + 1.
      objectsByKey removeKey: assoc key ifAbsent: [].
      keysByObject removeKey: assoc value ifAbsent: [  ].
      count \\ 100 == 0
        ifTrue: [ platform doCommitTransaction ].
      count \\ 1000 == 0
        ifTrue: [ Transcript cr; show: 'Expire progress: ' , count printString.]].
  platform doCommitTransaction.
  UserGlobals removeKey: #'ExpiryCleanup'.
  platform doCommitTransaction.
  ^ expired size

This implementation should be more resistant to an out of memory condition and I've got some logging in  there as well ... the `Transcript show:` should show up in the gem.log and/or Transcript...

I haven't tested this but if there's a problem in the scan loop it will fail quickly. If there's a problem in the expire loop, you can skip the scan loop, and just run the expire loop ...

Sorry about that ... hopefully the second time will be the charm ...

Dale

On 3/25/15 2:36 PM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, Step 1 ran for hours and halted in the debugger with this error: 

Error: VM temporary object memory is full 
, almost out of memory, too many markSweeps since last successful scavenge


What do I do now? 

Best, 

  Larry 



On Mar 25, 2015, at 2:15 PM, Dale Henrichs <[hidden email]> wrote:

Okay here's the sequence of steps that I think you should take:

  1. expire all of your sessions:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransactions

  2. initalize your object log

  3. run MFC

  [
  System abortTransaction.
  SystemRepository markForCollection ]
    on: Warning
    do: [ :ex |
      Transcript
        cr;
        show: ex description.
      ex resume ]

  4. Then do a backup and restore ... you can use GemTools to do the restore,
      but then you should read the SysAdmin docs[1] for instructions to do the restore
      (I've enclosed link to 3.2 docs, but the procedure and commands should pretty
      much be the same, but it's best to look up the docs for your GemStone version[2]
      and follow those instructions)

As I mentioned earlier, it will probably take a while for each of these operations to complete (object log will be fast and the backup will be fast, if the mfc tosses out the majority of your data) and it is likely that the repository will grow some more during the process (hard to predict this one, tho).

Step 1 will touch every session and every continuation so it is hard to say what percent of the objects are going to be touched (the expensive part), still there are likely to be a lot of those puppies and they will have to be read from disk into the SPC ...

Step 3. is going scan all of the live objects and again it hard to predict exactly how expensive it will be ...

Dale

[1] http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm
[2] http://gemtalksystems.com/techsupport/resources/

On 3/25/15 10:18 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL. 

  I’m not running the maintenance VM and I haven’t been doing regular mark for collects. 

  I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup? 

  I’ll also try the ObjectLog trick. 

  I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual. 

  
Best, 

Larry


  
On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Lawrence,

Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?

Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...

Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...

The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...

Dale

On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22528 megabytes.
    Repository has grown to 22528 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22544 megabytes.
    Repository has grown to 22544 megabytes.

--- 03/08/2015 03:31:45 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22560 megabytes.
    Repository has grown to 22560 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22576 megabytes.
    Repository has grown to 22576 megabytes.

--- 03/10/2015 03:19:34 AM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22592 megabytes.
    Repository has grown to 22592 megabytes.

--- 03/10/2015 03:46:39 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22608 megabytes.
    Repository has grown to 22608 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22624 megabytes.
    Repository has grown to 22624 megabytes.


My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass














_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list
Larry,

I have not been able to grok the whole thing (still seems to be missing chunks of the stack), but it does look like the out of memory may be occurring when a debugger is coming up ... I see OGStackFrame on the stack ... so perhaps wrapping an Error handler around the invocation with a `Transcript cr; show: ex description`  will at least help us understand what might be triggering the debugger ...

Dale

On 03/27/2015 10:37 AM, Lawrence Kellogg wrote:

On Mar 27, 2015, at 1:24 PM, Dale Henrichs <[hidden email]> wrote:

Larry,

To get my bearings, it would be useful to see the chunk of the stack where WACache>>gemstoneReap is involved ... so we can understand where we blowing out memory...


Ok, here is the full stack: (it is impressive)

 GsMethod 1926657,   Object >> _doesNotUnderstand:
 GsMethod 10662913,   metaSortedCollection >> sortBlock:
 GsMethod 1754881,   Behavior >> new
 GsMethod 9573889,   SortedCollection >> sortBlock:
 GsMethod 1924097,   Object >> size
 GsMethod 3381249,   SmallInteger >> >
 GsMethod 2242817,   metaKeyValueDictionary >> new
 GsMethod 2243841,   metaKeyValueDictionary >> new:
 GsMethod 2057729,   metaInteger >> _selectedPrimeGreaterThan:
 GsMethod 1924609,   Object >> at:
 GsMethod 3378689,   SmallInteger >> //
 GsMethod 3378945,   SmallInteger >> <=
 GsMethod 1753857,   Behavior >> _basicNew:
 GsMethod 2257665,   KeyValueDictionary >> _initializeWithoutClear:
 GsMethod 3181569,   metaSystem >> _sessionStateAt:put:
 GsMethod 3206913,   metaSystem >> myUserProfile
 GsMethod 3294977,   SymbolDictionary >> at:ifAbsent:
 GsMethod 1534721,   ComplexBlock >> value
 GsMethod 7693313,   metaGsSession >> currentSession
 GsMethod 7723777,   GsCurrentSession >> objectNamed:
 GsMethod 3275521,   SymbolList >> resolveSymbol:
 GsMethod 1795329,   Association >> value
 GsMethod 7592193,   metaGsPackagePolicy >> globalName
 GsMethod 3300353,   SymbolDictionary >> at:otherwise:
 GsMethod 3297281,   SymbolDictionary >> associationAt:otherwise:
 GsMethod 7660545,   GsPackagePolicy >> enabled
 GsMethod 7640833,   metaSessionTemps >> current
 GsMethod 3182081,   metaSystem >> _sessionStateAt:
 GsMethod 7657217,   GsPackagePolicy >> sessionMethodDictionaryGlobalName
 GsMethod 3299585,   SymbolDictionary >> at:put:
 GsMethod 3293953,   SymbolDictionary >> _validatePrivilegeOld:new:
 GsMethod 1789185,   metaAssociation >> newWithKey:value:
 GsMethod 3307777,   SymbolAssociation >> key:value:
 GsMethod 3308033,   SymbolAssociation >> key:
 GsMethod 1796865,   Association >> value:
 GsMethod 3300097,   SymbolDictionary >> _at:put:
 GsMethod 3299073,   SymbolDictionary >> _validatePrivilegeGeneric:
 GsMethod 3140097,   metaSystem >> _protectedMode
 GsMethod 3215873,   metaSystem >> _zeroArgPrim:
 GsMethod 2278145,   KeyValueDictionary >> _at:put:
 GsMethod 2230273,   IdentityDictionary >> hashFunction:
 GsMethod 1910529,   Object >> identityHash
 GsMethod 2272513,   KeyValueDictionary >> tableSize
 GsMethod 3377665,   SmallInteger >> \\
 GsMethod 2263297,   KeyValueDictionary >> keyAt:
 GsMethod 1923585,   Object >> _at:
 GsMethod 2267649,   KeyValueDictionary >> valueAt:
 GsMethod 1916417,   Object >> _basicAt:
 GsMethod 1699329,   Boolean >> |
 GsMethod 2276609,   KeyValueDictionary >> atHash:putKey:
 GsMethod 1922561,   Object >> _basicAt:put:
 GsMethod 2221569,   IdentityDictionary >> atHash:putValue:
 GsMethod 2256129,   KeyValueDictionary >> atHash:putValue:
 GsMethod 7661313,   GsPackagePolicy >> sessionMethodDictionary
 GsMethod 1900033,   Object >> class
 GsMethod 2278401,   KeyValueDictionary >> size
 GsMethod 7652097,   GsPackagePolicy >> symbolList
 GsMethod 7723521,   GsCurrentSession >> symbolList
 GsMethod 3395329,   SequenceableCollection >> reverseDo:
 GsMethod 1942785,   Number >> downTo:do:
 GsMethod 1943041,   Number >> downTo:by:do:
 GsMethod 1536001,   ComplexBlock >> value:
 GsMethod 7639809,   metaGsPackage >> globalName
 GsMethod 7706113,   GsPackage >> sessionMethods
 GsMethod 1790977,   Association >> key
 GsMethod 1533697,   ComplexBlock >> value:value:
 GsMethod 2990337,   GsMethodDictionary >> at:otherwise:
 GsMethod 2990081,   GsMethodDictionary >> at:ifAbsent:
 GsMethod 7699457,   GsSessionMethodDictionary >> hashFunction:
 GsMethod 2991873,   GsMethodDictionary >> hashFunction:
 GsMethod 1910273,   Object >> basicIdentityHash
 GsMethod 1829889,   AbstractDictionary >> _reportKeyNotFound:with:
 GsMethod 2989057,   GsMethodDictionary >> keysAndValuesDo:
 GsMethod 1913089,   Object >> _basicSize
 GsMethod 7591425,   metaGsPackagePolicy >> current
 GsMethod 20491265,   metaSystemLoginNotification >> sessionStart
 GsMethod 20491009,   metaSystemLoginNotification >> subscriptions
 GsMethod 3219713,   metaSystem >> rcValueCacheAt:for:otherwise:
 GsMethod 3220481,   metaSystem >> _rcValueCache
 GsMethod 20422913,   metaObject >> new
 GsMethod 269799681,   Object >> initialize
 GsMethod 3211521,   metaSystem >> _addRootObjectToRcReadSet:
 GsMethod 3223809,   metaSystem >> _addToRcReadSet:includingAllNodes:
 GsMethod 3224321,   metaSystem >> _addEntireObjectToRcReadSet:
 GsMethod 1553409,   Collection >> isEmpty
 GsMethod 2105089,   IdentityBag >> addAll:
 GsMethod 1878273,   Object >> _isRcIdentityBag
 GsMethod 2124801,   IdentityBag >> _addAll:
 GsMethod 3219201,   metaSystem >> rcValueCacheAt:put:for:
 GsMethod 3220993,   metaSystem >> rcValueCache
 GsMethod 2278913,   KeyValueDictionary >> at:otherwise:
 GsMethod 3032065,   IdentityKeyValueDictionary >> hashFunction:
 GsMethod 2270721,   KeyValueDictionary >> at:put:
 GsMethod 1804545,   Array >> addLast:
 GsMethod 2116609,   IdentityBag >> _at:
 GsMethod 3387905,   SimpleBlock >> value:
 GsMethod 20742145,   SmalltalkProxy >> at:ifAbsent:
 GsMethod 1920001,   Object >> isNil
 GsMethod 3074305,   UserProfile >> symbolList
 GsMethod 3266561,   SymbolList >> objectNamed:
 GsMethod 3206657,   metaSystem >> session
 GsMethod 3380225,   SmallInteger >> bitXor:
 GsMethod 2217985,   IdentityDictionary >> at:ifAbsent:
 GsMethod 2282241,   KeyValueDictionary >> _at:otherwise:
 GsMethod 3386625,   SimpleBlock >> value
 GsMethod 19090945,   Object >> copy
 GsMethod 3486209,   Object >> _basicCopy
 GsMethod 19081985,   Object >> postCopy
 GsMethod 2223361,   IdentityDictionary >> at:put:
 GsMethod 1795585,   Association >> key:value:
 GsMethod 3488513,   Object >> segment
 GsMethod 3487745,   Object >> assignToSegment:
 GsMethod 3488001,   Object >> _changeToSegment:
 GsMethod 2228225,   IdentityDictionary >> compareKey:with:
 GsMethod 3033089,   IdentityKeyValueDictionary >> collisionBucketClass
 GsMethod 2199041,   metaIdentityCollisionBucket >> new
 GsMethod 1846273,   metaAbstractCollisionBucket >> new:
 GsMethod 1754625,   Behavior >> new:
 GsMethod 1867265,   AbstractCollisionBucket >> initialize
 GsMethod 1538561,   CollisionBucket >> keyValueDictionary:
 GsMethod 2206977,   IdentityCollisionBucket >> at:put:keyValDict_coll:
 GsMethod 2201857,   IdentityCollisionBucket >> binarySearchForInsertKey:
 GsMethod 3376897,   SmallInteger >> <
 GsMethod 2206209,   IdentityCollisionBucket >> insertAt:key:value:
 GsMethod 3401729,   SequenceableCollection >> insertAll:at:
 GsMethod 1863937,   AbstractCollisionBucket >> size
 GsMethod 1864193,   AbstractCollisionBucket >> at:otherwise:
 GsMethod 2202881,   IdentityCollisionBucket >> searchForKey:
 GsMethod 1914113,   Object >> at:put:
 GsMethod 2266369,   KeyValueDictionary >> tableSize:
 GsMethod 1890305,   Object >> basicAt:
 GsMethod 1538817,   CollisionBucket >> _removeAll
 GsMethod 1907201,   Object >> size:
 GsMethod 1920257,   Object >> _basicSize:
 GsMethod 2281217,   KeyValueDictionary >> _copyNilIntoSelfFrom:to:
 GsMethod 1887745,   Object >> respondsTo:
 GsMethod 15058433,   Behavior >> whichClassIncludesSelector:
 GsMethod 3316737,   metaSymbol >> _existingWithAll:
 GsMethod 7655169,   GsPackagePolicy >> includesSelector:for:
 GsMethod 1826305,   AbstractDictionary >> includesKey:
 GsMethod 3380737,   SmallInteger >> printString
 GsMethod 3377409,   SmallInteger >> asString
 GsMethod 3346945,   String >> ,
 GsMethod 3348993,   String >> size
 GsMethod 1986817,   Magnitude >> min:
 GsMethod 19069185,   CharacterCollection >> copyFrom:to:
 GsMethod 1608961,   metaCharacterCollection >> _newString:
 GsMethod 3345409,   String >> copyFrom:to:into:startingAt:
 GsMethod 10828289,   metaDictionary >> new
 GsMethod 10828801,   metaDictionary >> new:
 GsMethod 1747201,   Behavior >> basicNew:
 GsMethod 9665537,   Dictionary >> count:
 GsMethod 9657857,   Dictionary >> tableSize:
 GsMethod 9665793,   Dictionary >> emptySlotHint:
 GsMethod 9666049,   Dictionary >> numEmptySlots:
 GsMethod 9663233,   Dictionary >> at:put:
 GsMethod 1842945,   AbstractDictionary >> hashFunction:
 GsMethod 1647105,   CharacterCollection >> hash
 GsMethod 9658369,   Dictionary >> tableSize
 GsMethod 9666305,   Dictionary >> associationAtOrNil:hash:
 GsMethod 10829313,   metaDictionary >> associationClass
 GsMethod 9660161,   Dictionary >> addNewAssociation:hash:
 GsMethod 1890049,   Object >> basicAt:put:
 GsMethod 2271489,   KeyValueDictionary >> keys
 GsMethod 2283265,   KeyValueDictionary >> keysDo:
 GsMethod 2275841,   KeyValueDictionary >> keysAndValuesDo:
 GsMethod 2113281,   IdentityBag >> add:
 GsMethod 20446209,   Collection >> do:separatedBy:
 GsMethod 2127617,   IdentityBag >> do:
 GsMethod 37907201,   Class >> proxyClassName
 GsMethod 47120641,   metaOBGemStonePlatform >> gsParserResolve:forClass:
 GsMethod 47118849,   metaOBGemStonePlatform >> hasInterned:ifTrue:
 GsMethod 19115521,   metaSymbol >> hasInterned:ifTrue:
 GsMethod 79149569,   Object >> ifNotNil:
 GsMethod 19138305,   BlockClosure >> valueWithPossibleArgs:
 GsMethod 19137793,   BlockClosure >> numArgs
 GsMethod 2303233,   ExecutableBlock >> numberArgs
 GsMethod 9005057,   ExecutableBlock >> on:do:
 GsMethod 10062081,   ExceptionHandler >> try:on:do:
 GsMethod 11896833,   metaExceptionA >> errorCategoryToCatch
 GsMethod 11897089,   metaExceptionA >> errorNumberToCatch
 GsMethod 2315265,   metaException >> category:number:do:
 GsMethod 2324481,   Exception >> handler:
 GsMethod 10065409,   ExceptionHandler >> doTryBlock:
 GsMethod 19170305,   UndefinedObject >> ifNil:
 GsMethod 20741889,   SmalltalkProxy >> at:
 GsMethod 19082753,   Object >> ifNil:
 GsMethod 11896577,   metaExceptionA >> signal
 GsMethod 11895041,   metaExceptionA >> signal:
 GsMethod 11895297,   metaExceptionA >> new
 GsMethod 1747457,   Behavior >> basicNew
 GsMethod 1755137,   Behavior >> _basicNew
 GsMethod 10052865,   ExceptionA >> initialize
 GsMethod 10050817,   ExceptionA >> signal:
 GsMethod 10057473,   ExceptionA >> signal
 GsMethod 10055169,   ExceptionA >> gsNumber
 GsMethod 11898113,   metaExceptionA >> errorNumberToSignal
 GsMethod 20480001,   ExceptionA >> gsCategory
 GsMethod 1919489,   Object >> notNil
 GsMethod 2318081,   metaException >> _exceptionFor:signalDictionary:
 GsMethod 20418817,   metaError >> handlesCategory:number:
 GsMethod 20510209,   metaExceptionA >> handles:
 GsMethod 1699841,   Boolean >> not
 GsMethod 10050561,   ExceptionA >> _defaultAction
 GsMethod 11669505,   Notification >> isResumable
 GsMethod 7174145,   metaOrderedCollection >> new:
 GsMethod 9663489,   Dictionary >> at:ifAbsent:
 GsMethod 9661441,   Dictionary >> associationAt:otherwise:
 GsMethod 269800193,   metaSequenceableCollection >> streamContents:
 GsMethod 3437057,   metaPositionableStream >> on:
 GsMethod 3441409,   PositionableStream >> _initStreamWith:
 GsMethod 19049473,   Behavior >> selectorsAndMethodsDo:
 GsMethod 15063041,   Behavior >> _methodDict
 GsMethod 1900289,   Object >> hash
 GsMethod 1911297,   Object >> =
 GsMethod 15066113,   Behavior >> pragmasForMethod:
 GsMethod 7653121,   GsPackagePolicy >> packagesDo:
 GsMethod 1547777,   Collection >> do:
 GsMethod 3326465,   Symbol >> asSymbol
 GsMethod 3321345,   Symbol >> =
 GsMethod 3038209,   WriteStream >> nextPut:
 GsMethod 3038977,   WriteStream >> _nextPut:
 GsMethod 3038465,   WriteStream >> contents
 GsMethod 19132673,   SequenceableCollection >> copyFrom:to:
 GsMethod 7177729,   OrderedCollection >> add:
 GsMethod 3349249,   String >> =
 GsMethod 9660417,   Dictionary >> findEmptySlot
 GsMethod 9659393,   Dictionary >> rebuild
 GsMethod 1885953,   Object >> yourself
 GsMethod 1907713,   Object >> perform:with:
 GsMethod 3388929,   metaSequenceableCollection >> with:
 GsMethod 19092737,   Object >> value
 GsMethod 7180289,   OrderedCollection >> addAll:
 GsMethod 3397121,   SequenceableCollection >> ,
 GsMethod 28537857,   String >> setBlessingInMetacelloConfig:
 GsMethod 28531713,   String >> setBlessingInMetacelloVersion:
 GsMethod 28774145,   BlockClosure >> setProject:withInMetacelloConfig:
 GsMethod 28535297,   String >> addToMetacelloRepositories:
 GsMethod 28532737,   String >> asMetacelloVersionNumber
 GsMethod 19066881,   CharacterCollection >> findTokens:
 GsMethod 19076609,   Object >> isCharacter
 GsMethod 19071489,   CharacterCollection >> skipDelimiters:startingAt:
 GsMethod 3405569,   SequenceableCollection >> detect:ifNone:
 GsMethod 3340289,   String >> at:
 GsMethod 1664257,   Character >> =
 GsMethod 19070721,   CharacterCollection >> findDelimiters:startingAt:
 GsMethod 19124481,   SequenceableCollection >> first
 GsMethod 164897537,   CharacterCollection >> asNumber
 GsMethod 79154945,   SequenceableCollection >> readStream
 GsMethod 164902401,   metaNumber >> readFrom:
 GsMethod 19091201,   Object >> isString
 GsMethod 79196417,   SmallInteger >> isLarge
 GsMethod 19141889,   Character >> charCode
 GsMethod 10063361,   ExceptionHandler >> return:
 GsMethod 19119873,   SequenceableCollection >> allButFirst
 GsMethod 19132929,   SequenceableCollection >> allButFirst:
 GsMethod 28523265,   Collection >> setLoadsInMetacelloProject:
 GsMethod 1554945,   Collection >> asArray
 GsMethod 1540353,   metaCollection >> withAll:
 GsMethod 1814529,   Array >> addAll:
 GsMethod 28537089,   String >> setLoadsInMetacelloProject:
 GsMethod 28520961,   Collection >> setIncludesInMetacelloPackage:
 GsMethod 9659137,   Dictionary >> associationsDo:
 GsMethod 9656833,   Dictionary >> add:
 GsMethod 28538113,   String >> setDescriptionInMetacelloConfig:
 GsMethod 28534785,   String >> setDescriptionInMetacelloVersion:
 GsMethod 28537601,   String >> setAuthorInMetacelloConfig:
 GsMethod 28532225,   String >> setAuthorInMetacelloVersion:
 GsMethod 28538369,   String >> setTimestampInMetacelloConfig:
 GsMethod 28535809,   String >> setTimestampInMetacelloVersion:
 GsMethod 28535553,   String >> setProject:withInMetacelloConfig:
 GsMethod 1886465,   Object >> species
 GsMethod 20503297,   Dictionary >> copy
 GsMethod 20462081,   metaAssociation >> key:value:
 GsMethod 10803201,   metaSet >> new
 GsMethod 9650433,   Set >> initialize:
 GsMethod 9650689,   Set >> add:
 GsMethod 1787905,   Behavior >> varyingConstraint
 GsMethod 2277377,   KeyValueDictionary >> at:ifAbsent:
 GsMethod 118801153,   ExceptionSet >> handlesCategory:number:
 GsMethod 20508673,   metaExceptionA >> category:number:arguments:
 GsMethod 9659649,   Dictionary >> valuesDo:
 GsMethod 19118849,   SequenceableCollection >> sort:
 GsMethod 19128833,   SequenceableCollection >> mergeSortFrom:to:by:
 GsMethod 172938241,   Object >> assert:
 GsMethod 19132161,   SequenceableCollection >> clone
 GsMethod 19129089,   SequenceableCollection >> mergeSortFrom:to:src:dst:by:
 GsMethod 19128577,   SequenceableCollection >> mergeFirst:middle:last:into:by:
 GsMethod 3385857,   SimpleBlock >> value:value:
 GsMethod 7180033,   OrderedCollection >> addFirst:
 GsMethod 3401217,   SequenceableCollection >> insertObject:at:
 GsMethod 19067137,   CharacterCollection >> isString
 GsMethod 28547073,   Integer >> metacelloVersionComponentLessThan:
 GsMethod 28547585,   Integer >> metacelloIntegerLessThanSelf:
 GsMethod 3396353,   SequenceableCollection >> replaceFrom:to:with:startingAt:
 GsMethod 28533505,   String >> metacelloVersionComponentLessThan:
 GsMethod 28547329,   Integer >> metacelloStringLessThanSelf:
 GsMethod 1937921,   Number >> _retry:coercing:
 GsMethod 28534529,   String >> metacelloIntegerLessThanSelf:
 GsMethod 3394817,   SequenceableCollection >> select:
 GsMethod 1886209,   Object >> speciesForSelect
 GsMethod 3401985,   SequenceableCollection >> includes:
 GsMethod 1807361,   Array >> add:
 GsMethod 1602305,   Class >> name
 GsMethod 3326209,   Symbol >> asString
 GsMethod 3328001,   metaString >> withAll:
 GsMethod 19086593,   Object >> instVarNamed:put:
 GsMethod 3350017,   String >> asSymbol
 GsMethod 3315457,   metaSymbol >> withAll:
 GsMethod 2280449,   KeyValueDictionary >> compareKey:with:
 GsMethod 2265345,   KeyValueDictionary >> collisionBucketClass
 GsMethod 1845505,   metaAbstractCollisionBucket >> new
 GsMethod 1866753,   AbstractCollisionBucket >> at:put:keyValDict_coll:
 GsMethod 1859585,   AbstractCollisionBucket >> tableSize
 GsMethod 1868033,   AbstractCollisionBucket >> keyAt:
 GsMethod 1864961,   AbstractCollisionBucket >> compareKey:with:
 GsMethod 1862401,   AbstractCollisionBucket >> at:putKey:
 GsMethod 1862657,   AbstractCollisionBucket >> at:putValue:
 GsMethod 1863681,   AbstractCollisionBucket >> at:ifAbsent:
 GsMethod 1859329,   AbstractCollisionBucket >> searchForKey:
 GsMethod 1862913,   AbstractCollisionBucket >> _reportKeyNotFound:with:
 GsMethod 2267393,   KeyValueDictionary >> rebuildTable:
 GsMethod 9651969,   Set >> do:
 GsMethod 1549313,   Collection >> addAll:
 GsMethod 20437761,   Collection >> removeAllFoundIn:
 GsMethod 1862145,   AbstractCollisionBucket >> removeKey:ifAbsent:
 GsMethod 1868289,   AbstractCollisionBucket >> valueAt:
 GsMethod 3395073,   SequenceableCollection >> collect:
 GsMethod 1951489,   Number >> abs
 GsMethod 28779009,   BlockClosure >> setPackage:withInMetacelloConfig:
 GsMethod 28537345,   String >> setRequiresInMetacelloPackage:
 GsMethod 28520705,   Collection >> setRequiresInMetacelloPackage:
 GsMethod 120885505,   Symbol >> setPostLoadDoItInMetacelloSpec:
 GsMethod 28535041,   String >> setPackage:withInMetacelloConfig:
 GsMethod 120885761,   Symbol >> setPreLoadDoItInMetacelloSpec:
 GsMethod 3345665,   String >> <=
 GsMethod 3344641,   String >> >
 GsMethod 28534017,   String >> metacelloStringLessThanSelf:
 GsMethod 3348737,   String >> <
 GsMethod 1907969,   Object >> perform:
 GsMethod 19135233,   SequenceableCollection >> detectMax:
 GsMethod 3038721,   WriteStream >> nextPutAll:
 GsMethod 3343617,   String >> addAll:
 GsMethod 3340801,   String >> at:put:
 GsMethod 19070209,   CharacterCollection >> beginsWith:
 GsMethod 19130113,   SequenceableCollection >> last
 GsMethod 19123457,   SequenceableCollection >> copyUpToLast:
 GsMethod 19120385,   SequenceableCollection >> lastIndexOf:ifAbsent:
 GsMethod 19128321,   SequenceableCollection >> lastIndexOf:startingAt:ifAbsent:
 GsMethod 19123201,   SequenceableCollection >> first:
 GsMethod 19134465,   SequenceableCollection >> copyAfterLast:
 GsMethod 19131393,   SequenceableCollection >> copyUpTo:
 GsMethod 19134209,   SequenceableCollection >> copyAfter:
 GsMethod 328041985,   CharacterCollection >> asInteger
 GsMethod 19096833,   Number >> isZero
 GsMethod 1702913,   Boolean >> &
 GsMethod 164906241,   metaInteger >> readFrom:
 GsMethod 269801729,   metaInteger >> readFrom:base:
 GsMethod 2180609,   UnorderedCollection >> copy
 GsMethod 2180353,   UnorderedCollection >> _finishShallowCopy
 GsMethod 3167489,   metaSystem >> _configurationAt:isStone:kind:
 GsMethod 3408897,   SequenceableCollection >> =
 GsMethod 9650945,   Set >> includes:
 GsMethod 19050497,   Behavior >> allSelectors
 GsMethod 15063297,   Behavior >> selectors
 GsMethod 7654401,   GsPackagePolicy >> selectorsFor:into:
 GsMethod 1763585,   Behavior >> superClass
 GsMethod 20444929,   Collection >> asSortedArray
 GsMethod 1555457,   Collection >> asSortedCollection
 GsMethod 10663425,   metaSortedCollection >> withAll:
 GsMethod 10663681,   metaSortedCollection >> new
 GsMethod 9570561,   SortedCollection >> addAll:
 GsMethod 9570817,   SortedCollection >> add:
 GsMethod 9570049,   SortedCollection >> _findIndex:
 GsMethod 9574657,   SortedCollection >> _insertAll:at:
 GsMethod 1839617,   AbstractDictionary >> at:ifAbsentPut:
 GsMethod 2224385,   IdentityDictionary >> at:otherwise:
 GsMethod 42221825,   OBBrowser >> announcer
 GsMethod 42220289,   OBBrowser >> navigationPanel
 GsMethod 47279361,   OBPanel >> isNavigation
 GsMethod 47492353,   OBColumnPanel >> isNavigation
 GsMethod 47181057,   OBNode >> metaNode:
 GsMethod 499691777,   OBColumn >> listHolder
 GsMethod 41703937,   OBList >> fan
 GsMethod 47186433,   OBNode >> defaultChildNodes
 GsMethod 47070977,   OBMetaNode >> defaultChildrenForNode:
 GsMethod 47073281,   OBMetaNode >> defaultEdgesForNode:
 GsMethod 47074561,   OBMetaNode >> defaultEdges
 GsMethod 47078145,   OBMetaNode >> edges
 GsMethod 47022337,   OBMetaEdge >> isDefaultNavigation
 GsMethod 47023873,   OBMetaEdge >> navigation
 GsMethod 47800065,   OBDefaultEdgeNavigation >> isDefaultNavigation
 GsMethod 1556993,   Collection >> inject:into:
 GsMethod 120883969,   SequenceableCollection >> gather:
 GsMethod 47023617,   OBMetaEdge >> nodesForParent:
 GsMethod 47015425,   OBEnvironmentNode >> categories
 GsMethod 20743425,   SmalltalkProxy >> systemOrganization
 GsMethod 19323905,   SystemOrganizer >> categories
 GsMethod 19323649,   SystemOrganizer >> categoryDict
 GsMethod 18999297,   metaClassOrganizer >> cachedOrganizer
 GsMethod 2110977,   IdentityBag >> includesIdentical:
 GsMethod 2234881,   IdentityDictionary >> keysAndValuesDo:
 GsMethod 1778177,   Behavior >> isBehavior
 GsMethod 1597441,   Class >> isMeta
 GsMethod 3398145,   SequenceableCollection >> removeFrom:to:
 GsMethod 1545985,   Collection >> _sortPaths:directions:persistentRoot:
 GsMethod 1627649,   CharacterCollection >> asArrayOfPathTerms
 GsMethod 1553921,   Collection >> _indexObjectsFor:
 GsMethod 11426817,   metaPathEvaluator >> new
 GsMethod 10425089,   PathEvaluator >> initialize
 GsMethod 10419457,   PathEvaluator >> nsc:
 GsMethod 10428161,   PathEvaluator >> asMostSpecificType
 GsMethod 10431489,   PathEvaluator >> _calculateIvOffsets
 GsMethod 1755905,   Behavior >> elementConstraint
 GsMethod 10423297,   PathEvaluator >> _calculateIvOffsetsOnKind:
 GsMethod 10420225,   PathEvaluator >> _getConstraintsStartingOnKind:
 GsMethod 1750785,   Behavior >> _ivOffsetOf:
 GsMethod 10421249,   PathEvaluator >> _sortOn:directions:persistentRoot:
 GsMethod 12794113,   metaPathSorter >> on:directions:
 GsMethod 12500481,   PathSorter >> sortNodeClassForSortLevel:
 GsMethod 10430721,   PathEvaluator >> lastElementClass
 GsMethod 10424321,   PathEvaluator >> isIndexOnRootNsc
 GsMethod 1736193,   Behavior >> sortNodeClass
 GsMethod 11227649,   metaRangeEqualityIndex >> isBasicClass:
 GsMethod 1740033,   Behavior >> isSpecial
 GsMethod 1767937,   Behavior >> _isSpecial
 GsMethod 11294209,   metaIndexManager >> current
 GsMethod 10423041,   PathEvaluator >> nscRoot
 GsMethod 11193601,   metaSortNode >> new:
 GsMethod 9523457,   BtreeNode >> numElements:
 GsMethod 10297345,   SortNode >> totalElements:
 GsMethod 10297601,   SortNode >> sortLevel:
 GsMethod 10298113,   SortNode >> pathSorter:
 GsMethod 12503553,   PathSorter >> sortNodes:
 GsMethod 12503809,   PathSorter >> directions:
 GsMethod 12501505,   PathSorter >> incompletes:
 GsMethod 11293441,   metaIndexManager >> autoCommit
 GsMethod 10352385,   IndexManager >> autoCommit
 GsMethod 10355457,   IndexManager >> sessionAutoCommit
 GsMethod 10421761,   PathEvaluator >> _sortOn:using:persistentRoot:
 GsMethod 12502529,   PathSorter >> sortNodes
 GsMethod 12503041,   PathSorter >> incompletes
 GsMethod 2150401,   UnorderedCollection >> _asIdentityBag
 GsMethod 10300673,   SortNode >> sortLevel
 GsMethod 10350593,   IndexManager >> commitIndexMaintenance:at:
 GsMethod 10355201,   IndexManager >> shouldCommit
 GsMethod 12502273,   PathSorter >> sortInto:startingAt:
 GsMethod 10887169,   SortNodeArray >> sortInto:startingAt:incompletes:
 GsMethod 10298625,   SortNode >> pathSorter
 GsMethod 12502785,   PathSorter >> directions
 GsMethod 10885633,   SortNodeArray >> indexManager
 GsMethod 1950721,   Number >> even
 GsMethod 3362049,   StringKeyValueDictionary >> at:otherwise:
 GsMethod 3362817,   StringKeyValueDictionary >> _stringAt:caseSensitive:otherwise:
 GsMethod 3361793,   StringKeyValueDictionary >> at:put:
 GsMethod 3362561,   StringKeyValueDictionary >> _stringAt:put:caseSensitive:
 GsMethod 9595905,   ClassOrganizer >> categories
 GsMethod 1805057,   Array >> _addAllFromNsc:
 GsMethod 1557761,   Collection >> sortAscending
 GsMethod 1547521,   Collection >> sortAscending:persistentRoot:
 GsMethod 1546497,   Collection >> sortWith:persistentRoot:
 GsMethod 1624321,   CharacterCollection >> speciesForConversion
 GsMethod 1554433,   Collection >> _findRangeIndexWithPath:
 GsMethod 1777409,   Behavior >> inheritsFrom:
 GsMethod 1785857,   Behavior >> isSubclassOf:
 GsMethod 10583553,   metaBtreeNode >> entrySize
 GsMethod 10583297,   metaBtreeNode >> maxNumberOfElements
 GsMethod 3394305,   SequenceableCollection >> _asCollectionForSorting
 GsMethod 12506113,   PathSorter >> _addObject:inNodes:hasSecondarySort:
 GsMethod 10297089,   SortNode >> isFull
 GsMethod 10297857,   SortNode >> _at:put:forBtree:hasSecondarySort:
 GsMethod 10298369,   SortNode >> at:put:hasSecondarySort:
 GsMethod 10299137,   SortNode >> _insertKey:value:atIndex:hasSecondarySort:
 GsMethod 9519617,   BtreeNode >> _lastIndex
 GsMethod 9520897,   BtreeNode >> entrySize
 GsMethod 1811201,   Array >> _insertAt:from:fromStart:fromEnd:numToMoveDown:
 GsMethod 9486849,   BtreeLeafNode >> _insertEncryptionFor:value:startingAt:
 GsMethod 9524225,   BtreeNode >> _binarySearchCoveringKey:totalOrder:
 GsMethod 3378177,   SmallInteger >> quo:
 GsMethod 9524993,   BtreeNode >> _compareKey:lessThanEntryAt:
 GsMethod 9527041,   BtreeNode >> _compare:key:lessThanEntryAt:
 GsMethod 12946433,   metaBtreeComparisonForSort >> compareKey:lessThan:
 GsMethod 1622785,   CharacterCollection >> _idxForSortLessThan:
 GsMethod 1621249,   CharacterCollection >> _idxForSortCharacterCollectionLessThanSelf:
 GsMethod 3336193,   String >> _idxPrimCompareLessThan:
 GsMethod 9521153,   BtreeNode >> _compareKey:greaterThanEntryAt:
 GsMethod 9523201,   BtreeNode >> _compare:key:greaterThanEntryAt:
 GsMethod 12945921,   metaBtreeComparisonForSort >> compareKey:greaterThan:
 GsMethod 1623297,   CharacterCollection >> _idxForSortGreaterThan:
 GsMethod 1621761,   CharacterCollection >> _idxForSortCharacterCollectionGreaterThanSelf:
 GsMethod 3335937,   String >> _idxPrimCompareGreaterThan:
 GsMethod 10299649,   SortNode >> sortInto:startingAt:
 GsMethod 1810433,   Array >> speciesForCollect
 GsMethod 48335361,   metaOBClassCategoryNode >> on:inEnvironment:
 GsMethod 42266369,   OBClassCategoryNode >> setName:environment:
 GsMethod 47180033,   OBNode >> navigation:
 GsMethod 19168257,   UndefinedObject >> ifNotNil:
 GsMethod 19078401,   Object >> changed:
 GsMethod 19084545,   Object >> dependents
 GsMethod 19085569,   Object >> myDependents
 GsMethod 19088129,   Object >> platformSupport
 GsMethod 26930945,   metaMCPlatformSupport >> dependentsFor:
 GsMethod 26918657,   metaMCPlatformSupport >> transientGlobalNamed:
 GsMethod 26917889,   metaMCPlatformSupport >> transientGlobals
 GsMethod 499486721,   OBList >> position
 GsMethod 19113729,   Symbol >> isUnary
 GsMethod 1648129,   CharacterCollection >> isInfix
 GsMethod 3343361,   String >> includes:
 GsMethod 3344897,   String >> includesValue:
 GsMethod 20441601,   Collection >> isCollection
 GsMethod 19164929,   Class >> allSubclasses
 GsMethod 21613569,   CachingClassOrganizer >> allSubclassesOf:
 GsMethod 41690625,   OBInteractionRequest >> defaultAction
 GsMethod 3299841,   SymbolDictionary >> at:
 GsMethod 44753665,   ClientForwarder >> doesNotUnderstand:
 GsMethod 44752897,   ClientForwarder >> perform:withArguments:
 GsMethod 3223297,   metaSystem >> _signalGciError:args:signalDictionary:
 GsMethod 1877761,   Object >> _gsReturnTos
 GsMethod 120902145,   JadeServer >> stringFromCollection:
 GsMethod 120902401,   JadeServer >> stringFromCollection:filter:
 GsMethod 1651457,   metaCharacter >> tab
 GsMethod 1650177,   metaCharacter >> withValue:
 GsMethod 3365633,   Stream >> tab
 GsMethod 48126721,   OBUpdateConcentrator >> dependentsMap
 GsMethod 19119617,   SequenceableCollection >> reversed
 GsMethod 48386561,   OBColumn >> selectedNode
 GsMethod 41699841,   OBList >> selectedNode
 GsMethod 41699329,   OBList >> selection
 GsMethod 47347969,   OBPosition >> index
 GsMethod 47870977,   OBFan >> nodeAt:
 GsMethod 47873025,   OBFan >> children
 GsMethod 19122433,   SequenceableCollection >> at:ifAbsent:
 GsMethod 1990401,   Magnitude >> between:and:
 GsMethod 19169025,   UndefinedObject >> ifNotNilDo:
 GsMethod 19170049,   UndefinedObject >> ifNil:ifNotNil:
 GsMethod 41671937,   JadeServer >> columnInfoFor:
 GsMethod 48391681,   OBColumn >> columnInfoFor:
 GsMethod 48386817,   OBColumn >> list
 GsMethod 41700097,   OBList >> list
 GsMethod 47873281,   OBFan >> list
 GsMethod 47867649,   OBFan >> displayStringForChild:
 GsMethod 47866881,   OBFan >> parent
 GsMethod 47183873,   OBNode >> displayStringForChild:
 GsMethod 47070465,   OBMetaNode >> displayStringForParent:child:
 GsMethod 47184129,   OBNode >> displayString
 GsMethod 47073025,   OBMetaNode >> displayStringForNode:
 GsMethod 47074817,   OBMetaNode >> displaySelector
 GsMethod 42267137,   OBClassCategoryNode >> name
 GsMethod 47317761,   OBFilter >> displayString:forNode:
 GsMethod 48398081,   OBColumn >> iconAt:
 GsMethod 41706241,   OBList >> iconAt:
 GsMethod 47866625,   OBFan >> iconAt:
 GsMethod 47189761,   OBNode >> icon
 GsMethod 47075073,   OBMetaNode >> iconForNode:
 GsMethod 47318785,   OBFilter >> icon:forNode:
 GsMethod 3091457,   UndefinedObject >> asString
 GsMethod 48386049,   OBColumn >> selection
 GsMethod 3388673,   metaSequenceableCollection >> with:with:with:
 GsMethod 48226049,   OBAnnouncer >> announce:
 GsMethod 38021889,   Behavior >> asAnnouncement
 GsMethod 21033217,   ActionSequence >> valueWithArguments:
 GsMethod 21048321,   MessageSend >> valueWithArguments:
 GsMethod 21046785,   MessageSend >> collectArguments:
 GsMethod 21047553,   MessageSend >> arguments
 GsMethod 1626113,   CharacterCollection >> numArgs
 GsMethod 1551873,   Collection >> identicalOccurrencesOf:
 GsMethod 1906689,   Object >> perform:withArguments:
 GsMethod 15062529,   Behavior >> _rawMethodDict
 GsMethod 47493889,   OBColumnPanel >> currentNode
 GsMethod 19144961,   Integer >> to:
 GsMethod 7209473,   metaInterval >> from:to:
 GsMethod 7258369,   Interval >> from:to:by:
 GsMethod 20419841,   metaString >> tab
 GsMethod 48384769,   OBColumn >> isEmpty
 GsMethod 41701377,   OBList >> isEmpty
 GsMethod 47868161,   OBFan >> isEmpty
 GsMethod 20426497,   Boolean >> and:and:
 GsMethod 20425985,   Boolean >> and:and:and:
 GsMethod 20436993,   Collection >> groupBy:having:
 GsMethod 1831681,   AbstractDictionary >> at:
 GsMethod 19127041,   SequenceableCollection >> do:separatedBy:
 GsMethod 499689985,   OBCommand >> clientExecute
 GsMethod 48127489,   OBUpdateConcentrator >> recordUpdatesDuring:
 GsMethod 48128257,   OBUpdateConcentrator >> canRecordUpdates
 GsMethod 19056129,   Behavior >> compiledMethodAt:ifAbsent:
 GsMethod 7654913,   GsPackagePolicy >> compiledMethodAt:for:
 GsMethod 7704065,   GsPackage >> compiledMethodAt:for:
 GsMethod 79167233,   CharacterCollection >> endsWith:
 GsMethod 19065089,   CharacterCollection >> includesSubstring:caseSensitive:
 GsMethod 19134721,   SequenceableCollection >> copyWithFirst:
 GsMethod 19073537,   metaCollection >> ofSize:
 GsMethod 37907457,   Class >> asNode
 GsMethod 48735233,   metaOBClassNode >> on:
 GsMethod 47156737,   OBClassNode >> setClass:
 GsMethod 15064065,   Behavior >> theNonMetaClass
 GsMethod 42253057,   OBClassAwareNode >> theClass:
 GsMethod 19082497,   Object >> canDiscardEdits
 GsMethod 19121409,   SequenceableCollection >> do:without:
 GsMethod 37924097,   Object >> asAnnouncement
 GsMethod 48394241,   OBColumn >> refreshAndSignal:
 GsMethod 41704449,   OBList >> refresh
 GsMethod 47868673,   OBFan >> refresh
 GsMethod 1909761,   Object >> ~=
 GsMethod 47185921,   OBNode >> =
 GsMethod 48116225,   OBDependentProxy >> update:
 GsMethod 48116481,   OBDependentProxy >> concentrator
 GsMethod 48129537,   OBUpdateConcentrator >> update:for:
 GsMethod 48128513,   OBUpdateConcentrator >> shouldRecordUpdates
 GsMethod 44752385,   ClientForwarder >> update:
 GsMethod 44754177,   ClientForwarder >> perform:with:
 GsMethod 47155713,   OBClassNode >> =
 GsMethod 3241473,   metaSystem >> signal:args:signalDictionary:
 GsMethod 500186881,   metaExceptionA >> gsNumber:arguments:
 GsMethod 42254849,   OBClassAwareNode >> theClass
 GsMethod 40764929,   OBModalFilter >> edgesFrom:forNode:
 GsMethod 40765953,   OBModalFilter >> selection
 GsMethod 47319041,   OBFilter >> edgesFrom:forNode:
 GsMethod 42266113,   OBClassCategoryNode >> rawClasses
 GsMethod 42268673,   OBClassCategoryNode >> classNames
 GsMethod 19325953,   SystemOrganizer >> listAtCategoryNamed:
 GsMethod 47319809,   OBFilter >> nodesFrom:forNode:
 GsMethod 41314561,   OBClassSortFilter >> nodesFrom:forNode:
 GsMethod 1550081,   Collection >> asOrderedCollection
 GsMethod 164894721,   Collection >> ifEmpty:
 GsMethod 41314305,   OBClassSortFilter >> initializeResults
 GsMethod 19054849,   Behavior >> withAllSuperclasses
 GsMethod 1778433,   Behavior >> allSuperClasses
 GsMethod 3399425,   SequenceableCollection >> removeAllSuchThat:
 GsMethod 20438785,   Collection >> contents
 GsMethod 1890817,   Object >> ->
 GsMethod 41313537,   OBClassSortFilter >> buildIndentIndex
 GsMethod 47155969,   OBClassNode >> hash
 GsMethod 41313793,   OBClassSortFilter >> collectSortedNodes
 GsMethod 48285953,   metaOBAllMethodCategoryNode >> on:
 GsMethod 47257601,   metaOBMethodCategoryNode >> on:inClass:
 GsMethod 40824065,   OBMethodCategoryNode >> setName:class:
 GsMethod 500121857,   OBClassNode >> categories
 GsMethod 15059969,   Behavior >> sortedCategoryNames
 GsMethod 15060225,   Behavior >> categoryNames
 GsMethod 10803713,   metaSet >> new:
 GsMethod 7662337,   GsPackagePolicy >> categoryNamesFor:into:
 GsMethod 7708929,   GsPackage >> categoryNamesFor:into:
 GsMethod 19083265,   Object >> ifNotNilDo:
 GsMethod 19076097,   Object >> ifNil:ifNotNilDo:
 GsMethod 19120897,   SequenceableCollection >> keysAndValuesDo:
 GsMethod 172937985,   Object >> ifNotNil:ifNil:
 GsMethod 47318529,   OBFilter >> displayString:forParent:child:
 GsMethod 41313281,   OBClassSortFilter >> displayString:forParent:child:
 GsMethod 3396609,   SequenceableCollection >> replaceFrom:to:with:
 GsMethod 19057665,   CharacterCollection >> isText
 GsMethod 19168513,   UndefinedObject >> ifNil:ifNotNilDo:
 GsMethod 3322881,   Symbol >> ,
 GsMethod 3399169,   SequenceableCollection >> removeAtIndex:
 GsMethod 40824577,   OBMethodCategoryNode >> =
 GsMethod 40826625,   OBMethodCategoryNode >> name
 GsMethod 42235137,   OBAllMethodCategoryNode >> methodReferences
 GsMethod 21080833,   MethodReference >> setStandardClass:methodSymbol:
 GsMethod 499580673,   MethodReference >> asNode
 GsMethod 48762625,   metaOBMethodNode >> on:
 GsMethod 47337729,   OBMethodNode >> setReference:
 GsMethod 21082881,   MethodReference >> methodSymbol
 GsMethod 21081857,   MethodReference >> actualClass
 GsMethod 47337985,   OBMethodNode >> setSelector:class:
 GsMethod 47331073,   OBMethodNode >> displayString
 GsMethod 12153345,   metaMessageNotUnderstood >> errorCategoryToCatch
 GsMethod 12153601,   metaMessageNotUnderstood >> errorNumberToCatch
 GsMethod 12153857,   metaMessageNotUnderstood >> errorNumberToSignal
 GsMethod 38023681,   Behavior >> includesLocalSelector:
 GsMethod 15061505,   Behavior >> includesSelector:
 GsMethod 41747457,   OBInheritanceFilter >> icon:forNode:
 GsMethod 47328513,   OBMethodNode >> selector
 GsMethod 19045121,   Behavior >> lookupSelector:
 GsMethod 3010305,   GsMethod >> _allBreakpoints
 GsMethod 3009793,   GsMethod >> _selectorPool
 GsMethod 3008257,   GsMethod >> literals
 GsMethod 20449025,   UnorderedCollection >> ,
 GsMethod 2093057,   IdentitySet >> addAll:
 GsMethod 47339521,   OBMethodNode >> isOverride
 GsMethod 19047425,   Behavior >> allSuperclassesDo:
 GsMethod 47331841,   OBMethodNode >> isOverridden
 GsMethod 19163393,   Class >> allSubclassesDo:
 GsMethod 19045889,   Behavior >> environment
 GsMethod 37908225,   Class >> ogDefinition
 GsMethod 24999937,   Class >> _ogDefinitionInContext:
 GsMethod 25000193,   Class >> _ogDefinitionInContext:className:superClassName:
 GsMethod 19045377,   Behavior >> isTransientDB
 GsMethod 3353089,   String >> addLast:
 GsMethod 19154177,   Class >> _sortedClassVarNames
 GsMethod 19048449,   Behavior >> classVarNames
 GsMethod 3302657,   SymbolDictionary >> keys
 GsMethod 2176001,   UnorderedCollection >> select:
 GsMethod 2179329,   UnorderedCollection >> _select:
 GsMethod 10616577,   metaNscBuilder >> for:max:
 GsMethod 10616065,   metaNscBuilder >> maxSize
 GsMethod 9542913,   NscBuilder >> counter:
 GsMethod 9543169,   NscBuilder >> finalResult:
 GsMethod 9544193,   NscBuilder >> nsc:
 GsMethod 9542657,   NscBuilder >> completeBag
 GsMethod 9543425,   NscBuilder >> _completeNsc
 GsMethod 1770753,   Behavior >> sharedPools
 GsMethod 1640193,   CharacterCollection >> printString
 GsMethod 1639937,   CharacterCollection >> printStringWithMaxSize:
 GsMethod 10130945,   metaPrintStream >> printingOn:maxSize:
 GsMethod 9776641,   PrintStream >> _initStreamWith:maxSize:
 GsMethod 3339777,   String >> printOn:
 GsMethod 9779457,   PrintStream >> _nextPut:
 GsMethod 3353345,   String >> indexOf:startingAt:
 GsMethod 9779201,   PrintStream >> nextPutAll:
 GsMethod 9775873,   PrintStream >> isFull
 GsMethod 9778177,   PrintStream >> contents
 GsMethod 47336961,   OBMethodNode >> definition
 GsMethod 47329025,   OBMethodNode >> selection
 GsMethod 47326721,   OBMethodNode >> source
 GsMethod 19044097,   Behavior >> sourceCodeAt:ifAbsent:
 GsMethod 15063809,   Behavior >> sourceCodeAt:
 GsMethod 3020033,   GsMethod >> _sourceString
 GsMethod 49304321,   metaOBMethodDefinition >> selection:source:inClass:
 GsMethod 49304577,   metaOBMethodDefinition >> selection:source:inCategory:inClass:
 GsMethod 499463681,   OBMethodDefinition >> setClass:category:source:selection:
 GsMethod 47268097,   OBMethodDefinition >> text
 GsMethod 1763073,   Behavior >> allClassVarNames
 GsMethod 3402241,   SequenceableCollection >> insert:at:
 GsMethod 1889793,   Object >> _validateClass:
 GsMethod 1761281,   Behavior >> allSharedPools
 GsMethod 79246337,   SmalltalkProxy >> bindingOf:
 GsMethod 20741377,   SmalltalkProxy >> associationAt:
 GsMethod 19056641,   CharacterCollection >> translated
 GsMethod 19122945,   SequenceableCollection >> second
 GsMethod 19125761,   SequenceableCollection >> checkedAt:
 GsMethod 47852545,   OBDefinitionPanel >> getDefinition
 GsMethod 42222593,   OBBrowser >> currentNode
 GsMethod 19056385,   metaBehavior >> parseSelector:for:
 GsMethod 1947393,   Number >> negative
 GsMethod 15057665,   Behavior >> compiledMethodAt:
 GsMethod 20444161,   Collection >> anyOne
 GsMethod 20439041,   Collection >> emptyCheck
 GsMethod 19124993,   SequenceableCollection >> atAll:
 GsMethod 19100929,   String >> findSelector
 GsMethod 19069953,   CharacterCollection >> withBlanksTrimmed
 GsMethod 19048705,   Behavior >> whichCategoryIncludesSelector:
 GsMethod 15057409,   Behavior >> categoryOfSelector:
 GsMethod 326735873,   Behavior >> compileMethod:category:using:
 GsMethod 15065601,   Behavior >> stampForMethod:
 GsMethod 269875713,   Behavior >> _compileMethod:category:using:
 GsMethod 15059201,   Behavior >> compileMethod:dictionaries:category:
 GsMethod 15056641,   Behavior >> changeStamp
 GsMethod 15064577,   Behavior >> authorInitials
 GsMethod 15065345,   Behavior >> setStamp:forMethod:
 GsMethod 19055361,   Behavior >> createPragmaFrom:for:
 GsMethod 15065089,   Behavior >> setPragmas:forMethod:
 GsMethod 20442881,   Collection >> isEmptyOrNil
 GsMethod 19064065,   CharacterCollection >> findString:startingAt:caseSensitive:
 GsMethod 47335937,   OBMethodNode >> =
 GsMethod 3230465,   metaSystem >> commitTransaction
 GsMethod 3222785,   metaSystem >> _commit:
 GsMethod 3159553,   metaSystem >> _commitCoordinator
 GsMethod 7713281,   TransactionBoundaryDefaultPolicy >> commit:
 GsMethod 3222529,   metaSystem >> _localCommit:
 GsMethod 3223041,   metaSystem >> _processDeferredGciUpdates
 GsMethod 3223553,   metaSystem >> _pendingCommitAbort
 GsMethod 3222017,   metaSystem >> __commit:
 GsMethod 3226881,   metaSystem >> _primitiveCommit:
 GsMethod 20641793,   SessionMethodTransactionBoundaryPolicy >> transactionBoundary:
 GsMethod 20637441,   SessionMethodTransactionBoundaryPolicy >> cachedClassChangeCount
 GsMethod 20637697,   SessionMethodTransactionBoundaryPolicy >> cacheDictionary
 GsMethod 20638209,   SessionMethodTransactionBoundaryPolicy >> cacheName
 GsMethod 20637953,   SessionMethodTransactionBoundaryPolicy >> classChangeCountCacheName
 GsMethod 18998785,   metaClassOrganizer >> _resetCachedOrganizer
 GsMethod 20638465,   SessionMethodTransactionBoundaryPolicy >> cachedSessionMethodChangeCount
 GsMethod 20638721,   SessionMethodTransactionBoundaryPolicy >> sessionMethodChangeCountCacheName
 GsMethod 48076545,   OBCodeBrowser >> signalRefresh
 GsMethod 42214913,   OBBrowser >> signalRefresh
 GsMethod 47853569,   OBDefinitionPanel >> refresh:
 GsMethod 48397057,   OBColumn >> refresh:
 GsMethod 41674497,   JadeServer >> doIt:in:
 GsMethod 499568641,   JadeServer >> evaluate:inContext:
 GsMethod 3340545,   String >> evaluateInContext:symbolList:
 GsMethod 3341057,   String >> _compileInContext:symbolList:
 GsMethod 3026177,   GsMethod >> _executeInContext:
 GsMethod 3019265,   GsMethod >> _primitiveExecuteInContext:
 GsMethod 1,  classId 20 selectorId 0
 GsMethod 21088257,   metaTranscriptProxy >> cr
 GsMethod 21087233,   metaTranscriptProxy >> proxyTranscript
 GsMethod 11804161,   metaDateAndTimeANSI >> now
 GsMethod 20524289,   DateAndTimeANSI >> initializeAsNow
 GsMethod 12067329,   metaDateAndTime >> secondsSince2001
 GsMethod 3188737,   metaSystem >> _timeGmtFloat
 GsMethod 7279105,   SmallDouble >> -
 GsMethod 3382529,   SmallInteger >> asFloat
 GsMethod 11974913,   metaTimeZone >> current
 GsMethod 11971073,   TimeZoneInfo >> transitionAtUTC:
 GsMethod 10005505,   DateAndTimeANSI >> asPosixSeconds
 GsMethod 7276033,   SmallDouble >> +
 GsMethod 11970305,   TimeZoneInfo >> detectLastTransition:
 GsMethod 10023169,   TimeZoneTransition >> transitionTimeUTC
 GsMethod 2075649,   Integer >> <=
 GsMethod 3377921,   SmallInteger >> _generality
 GsMethod 7287809,   SmallDouble >> _generality
 GsMethod 7274753,   SmallDouble >> _coerce:
 GsMethod 7281665,   SmallDouble >> <=
 GsMethod 10024193,   TimeZoneTransition >> offsetFromUTC
 GsMethod 10002177,   DateAndTimeANSI >> printString
 GsMethod 13488129,   Object >> printString
 GsMethod 10131457,   metaPrintStream >> printingOn:
 GsMethod 10011137,   DateAndTimeANSI >> printOn:
 GsMethod 10002689,   DateAndTimeANSI >> printAnsiOn:
 GsMethod 10011905,   DateAndTimeANSI >> asFloatParts
 GsMethod 10012161,   DateAndTimeANSI >> partsFrom:
 GsMethod 1951745,   Number >> \\
 GsMethod 1953281,   Number >> //
 GsMethod 7278593,   SmallDouble >> /
 GsMethod 7287553,   SmallDouble >> floor
 GsMethod 269790721,   DateAndTimeANSI >> print:on:
 GsMethod 1950209,   Number >> printOn:
 GsMethod 9776129,   PrintStream >> nextPut:
 GsMethod 79193345,   Number >> _printAsDateAndTimeANSISecondsOn:
 GsMethod 7277569,   SmallDouble >> =
 GsMethod 7283713,   SmallDouble >> asFloat
 GsMethod 7275265,   SmallDouble >> asStringLocaleC
 GsMethod 7278081,   SmallDouble >> _mathPrim:
 GsMethod 10008577,   DateAndTimeANSI >> offset
 GsMethod 11818497,   metaDuration >> seconds:
 GsMethod 10016257,   Duration >> _seconds:
 GsMethod 10016513,   Duration >> printOnHoursMinutes:
 GsMethod 10019329,   Duration >> hours
 GsMethod 1952769,   Number >> negated
 GsMethod 1936897,   Number >> sign
 GsMethod 10017025,   Duration >> minutes
 GsMethod 9778689,   PrintStream >> _collection
 GsMethod 79247105,   metaTranscriptProxy >> show:
 GsMethod 79243009,   metaObjectLogEntry >> transcript:object:
 GsMethod 20839169,   ObjectLogEntry >> initialize
 GsMethod 3214337,   metaSystem >> gemVersionReport
 GsMethod 3204609,   metaSystem >> _serverVersionReport:
 GsMethod 20839425,   ObjectLogEntry >> timestamp
 GsMethod 20839937,   ObjectLogEntry >> label:object:
 GsMethod 79243265,   metaObjectLogEntry >> transcript
 GsMethod 20839681,   ObjectLogEntry >> priority:
 GsMethod 20842497,   ObjectLogEntry >> addToLog
 GsMethod 22586369,   metaObjectLogEntry >> objectQueue
 GsMethod 12454913,   RcQueue >> add:
 GsMethod 12460033,   RcQueue >> _getSessionComponentFor:
 GsMethod 12452609,   RcQueue >> maxSessionId
 GsMethod 1892865,   Object >> _rcAt:
 GsMethod 11579137,   RcQueueElement >> sequenceNumber
 GsMethod 11637761,   metaRcQueueSessionComponent >> speciesOfElements
 GsMethod 12927745,   metaRcQueueEntry >> newWithValue:sequenceNumber:inSegmentOf:
 GsMethod 12449793,   RcQueue >> _findFirstValidQueueElementForSession:
 GsMethod 179615745,   metaWABasicDevelopment >> reapSeasideCache
 GsMethod 178838529,   metaWADispatcher >> default
 GsMethod 3398657,   SequenceableCollection >> removeFirst
 GsMethod 191317761,   WARequestHandler >> isApplication
 GsMethod 179091969,   WADispatcher >> handlers
 GsMethod 179616513,   metaWABasicDevelopment >> reapHandlerCache:dispatchers:
 GsMethod 202641409,   WAApplication >> isApplication
 GsMethod 176151809,   WARegistry >> cache
 GsMethod 119004929,   metaGRPlatform >> current
 GsMethod 499656961,   GRGemStonePlatform >> doCommitTransaction
 GsMethod 3225857,   metaSystem >> inTransaction
 GsMethod 119189249,   SystemCommitTransaction >> defaultAction
 GsMethod 3151105,   metaSystem >> _commitPrintingDiagnostics
 GsMethod 1923073,   Object >> halt
 GsMethod 19074561,   Object >> halt:
 GsMethod 22476545,   metaControlInterrupt >> signal:
 GsMethod 21788417,   ControlInterrupt >> signal
 GsMethod 79226881,   Halt >> defaultAction
 GsMethod 10380033,   RcKeyValueDictionary >> associationsDo:
 GsMethod 10381825,   RcKeyValueDictionary >> keysAndValuesDo:
 GsMethod 10383873,   RcKeyValueDictionary >> _doCollisionBuckets:
 GsMethod 9549057,   RcCollisionBucket >> keyAt:
 GsMethod 9549313,   RcCollisionBucket >> valueAt:
 GsMethod 3033857,   IdentityKeyValueDictionary >> compareKey:with:
 GsMethod 41680129,   JadeServer >> printIt:in:
 GsMethod 1,  classId 20 selectorId 0
 GsMethod 1926913,   Object >> doesNotUnderstand:
 GsMethod 48056577,   metaOTIndexedVariableNode >> on:index:
 GsMethod 42066689,   OTIndexedVariableNode >> setIndex:
 GsMethod 47982081,   OTDictionaryInspectorNode >> name
 GsMethod 47980801,   OTDictionaryInspectorNode >> key
 GsMethod 37922305,   Object >> inspectorName
 GsMethod 47981313,   OTDictionaryInspectorNode >> =
 GsMethod 46414337,   OTCmdOutDebugger >> execute
 GsMethod 40810497,   OGStackFrame >> =
 GsMethod 44751361,   ComplexBlock >> blockInspectorNodes
 GsMethod 37938689,   ExecutableBlock >> basicInspectorNodes
 GsMethod 499592449,   ComplexBlock >> derivedInspectorNodes
 GsMethod 499736833,   ExecutableBlock >> derivedInspectorNodes
 GsMethod 1,  classId 20 selectorId 0
 GsMethod 105770960385,   WACache >> gemstoneReap
 GsMethod 499446017,   OTCmdProceedDebugger >> execute
 GsMethod 49870849,   metaOGDebuggerProceedRequest >> signal:
 GsMethod 47216385,   OGDebuggerProceedRequest >> debugger:
 GsMethod 47216641,   OGDebuggerProceedRequest >> handleWith:
 GsMethod 47117313,   metaOBGemStonePlatform >> handleDebuggerProceedRequest:
 GsMethod 47216129,   OGDebuggerProceedRequest >> debugger
 GsMethod 1635329,   CharacterCollection >> copyReplaceAll:with:
 GsMethod 1634561,   CharacterCollection >> findString:startingAt:
 GsMethod 3337729,   String >> removeFrom:to:
 GsMethod 3343105,   String >> insertAll:at:
 GsMethod 19142913,   metaCharacter >> separators
 GsMethod 19143937,   metaCharacter >> value:
 GsMethod 1548289,   Collection >> occurrencesOf:
 GsMethod 47969537,   OTDefinitionPanel >> doItReceiver
 GsMethod 41678849,   JadeServer >> inspectIt:in:
 GsMethod 41674753,   JadeServer >> reportCompileErrors:
 GsMethod 47116801,   metaOBGemStonePlatform >> gsParserResolvePartial:forClass:
 GsMethod 19049729,   Behavior >> classPool
 GsMethod 1772289,   Behavior >> _classVars
 GsMethod 19043073,   AbstractDictionary >> hasBindingThatBeginsWith:
 GsMethod 38012929,   SmalltalkProxy >> hasBindingThatBeginsWith:
 GsMethod 47905793,   OTContextTempVariableNode >> =
 GsMethod 47906305,   OTContextTempVariableNode >> value
 GsMethod 40816385,   OGStackFrame >> printOn:
 GsMethod 1,  classId 20 selectorId 0
 GsMethod 105770745345,   WACache >> gemstoneReap
 GsMethod 176972545,   WACache >> expiryPolicy
 GsMethod 499561473,   WARcLastAccessExpiryPolicy >> isExpiredUpdating:key:
 GsMethod 10382337,   RcKeyValueDictionary >> at:ifAbsent:
 GsMethod 10386433,   RcKeyValueDictionary >> hashFunction:
 GsMethod 176140289,   WARcLastAccessExpiryPolicy >> timeout
 GsMethod 177983489,   WACachePlugin >> configuration
 GsMethod 177472001,   WAConfiguration >> at:
 GsMethod 177474305,   WAConfiguration >> getSearchContextFor:
 GsMethod 177472513,   WAConfiguration >> retrieveSearchContext:ifAbsentPut:
 GsMethod 178009345,   WAAttributeSearchContext >> attribute
 GsMethod 177473537,   WAConfiguration >> valueForContext:
 GsMethod 178010113,   WAAttributeSearchContext >> at:ifPresent:
 GsMethod 178011393,   WAAttributeSearchContext >> cachedValues
 GsMethod 19042817,   AbstractDictionary >> at:ifPresent:
 GsMethod 177908737,   WAAttribute >> default
 GsMethod 177431297,   WARcLastAccessEntry >> isExpiredUpdating:
 GsMethod 177431553,   WARcLastAccessEntry >> isExpired:updating:
 GsMethod 177430529,   WARcLastAccessEntry >> counter
 GsMethod 12494849,   RcCounter >> value
 GsMethod 12492545,   RcCounter >> _calculateValue
 GsMethod 10954241,   RcCounterElement >> value
 GsMethod 19117569,   metaTime >> totalSeconds
 GsMethod 11819265,   metaDuration >> zero
 GsMethod 11806209,   metaDateAndTimeANSI >> secondsUTC:offset:
 GsMethod 20523777,   DateAndTimeANSI >> _secondsUTC:offset:
 GsMethod 10017793,   Duration >> asSeconds
 GsMethod 10005249,   DateAndTimeANSI >> asSeconds
 GsMethod 1949953,   Number >> asInteger
 GsMethod 7286529,   SmallDouble >> truncated
 GsMethod 176976129,   WACache >> notifyRemoved:key:
 GsMethod 176969985,   WACache >> pluginsDo:
 GsMethod 176139777,   WARcLastAccessExpiryPolicy >> removed:key:
 GsMethod 10379777,   RcKeyValueDictionary >> removeKey:ifAbsent:
 GsMethod 10382849,   RcKeyValueDictionary >> _logRemoveKey:inCollisionBucket:
 GsMethod 9552897,   LogEntry >> receiver:
 GsMethod 9553153,   LogEntry >> selector:
 GsMethod 9552641,   LogEntry >> argArray:
 GsMethod 3218689,   metaSystem >> redoLog
 GsMethod 10738945,   metaRedoLog >> new
 GsMethod 9613313,   RedoLog >> initialize
 GsMethod 3372545,   metaSmallInteger >> maximumValue
 GsMethod 2277889,   KeyValueDictionary >> collisionLimit:
 GsMethod 9611521,   RedoLog >> addLogEntry:forConflictObject:
 GsMethod 9551617,   LogEntry >> receiver
 GsMethod 176974081,   WACache >> reapingStrategy
 GsMethod 177983745,   WACachePlugin >> removed:key:
 GsMethod 176977409,   WACache >> missStrategy
 GsMethod 176976641,   WACache >> removalAction
 GsMethod 177967617,   WANotifyRemovalAction >> removed:key:
 GsMethod 178559489,   WARequestHandler >> unregistered
 GsMethod 1,  classId 20 selectorId 0
 GsMethod 105770596353,   WACache >> gemstoneReap
 GsMethod 1,  classId 20 selectorId 0
 GsMethod 105767327233,   WACache >> gemstoneReap
 GsMethod 119025921,   metaGRObject >> new
 GsMethod 177432065,   WARcLastAccessEntry >> initialize
 GsMethod 12780033,   metaRcCounter >> new:
 GsMethod 12495873,   RcCounter >> initialize
 GsMethod 12493825,   RcCounter >> _getSessionElementFor:
 GsMethod 11641345,   metaRcCounterElement >> new
 GsMethod 10954497,   RcCounterElement >> initialize
 GsMethod 10954753,   RcCounterElement >> value:
 GsMethod 9611009,   RedoLog >> addConflictObject:for:
 GsMethod 12492289,   RcCounter >> increment
 GsMethod 12494081,   RcCounter >> _privateIncrementBy:
 GsMethod 12493569,   RcCounter >> _thisSessionIndex
 GsMethod 10953985,   RcCounterElement >> incrementValueBy:
 GsMethod 178284801,   WADocumentHandler >> hash
 GsMethod 178284289,   WADocumentHandler >> document
 GsMethod 177157121,   WAMimeDocument >> hash
 GsMethod 177155329,   WAMimeDocument >> mimeType
 GsMethod 177332481,   WAMimeType >> hash
 GsMethod 177332737,   WAMimeType >> main
 GsMethod 177331713,   WAMimeType >> sub
 GsMethod 177156353,   WAMimeDocument >> fileName
 GsMethod 177157377,   WAMimeDocument >> contents
 GsMethod 178284545,   WADocumentHandler >> =
 GsMethod 177156865,   WAMimeDocument >> =
 GsMethod 177331969,   WAMimeType >> =
 GsMethod 3294721,   SymbolDictionary >> removeKey:
 GsMethod 2217729,   IdentityDictionary >> removeKey:
 GsMethod 2273537,   KeyValueDictionary >> _removeKey:ifAbsent:
 GsMethod 2202625,   IdentityCollisionBucket >> removeKey:ifAbsent:
 GsMethod 202641153,   WAApplication >> sessionsDo:
 GsMethod 176154881,   WARegistry >> handlersDo:
 GsMethod 176155393,   WARegistry >> keysAndHandlersDo:
 GsMethod 176971777,   WACache >> keysAndValuesDo:
 GsMethod 191315713,   WARequestHandler >> isSession
 GsMethod 178564097,   WARequestHandler >> isDispatcher
 GsMethod 3035137,   IdentityKeyValueDictionary >> rebuildTable:
 GsMethod 202211841,   WASession >> isSession
 GsMethod 202211329,   WASession >> continuations
 GsMethod 177481729,   metaWAAttributeSearchContext >> key:target:
 GsMethod 178010369,   WAAttributeSearchContext >> initializeWithKey:
 GsMethod 119023873,   GRObject >> initialize
 GsMethod 178011137,   WAAttributeSearchContext >> findAttributeAndSelectAncestorsOf:
 GsMethod 178009601,   WAAttributeSearchContext >> key
 GsMethod 177473793,   WAConfiguration >> localAttributeAt:ifPresent:
 GsMethod 177920257,   WAUserConfiguration >> localAttributeAt:ifAbsent:
 GsMethod 177924865,   WAUserConfiguration >> parents
 GsMethod 177361153,   WASystemConfiguration >> localAttributeAt:ifAbsent:
 GsMethod 177361409,   WASystemConfiguration >> attributes
 GsMethod 177362945,   WASystemConfiguration >> description
 GsMethod 177375233,   WAConfigurationDescription >> attributes
 GsMethod 177473281,   WAConfiguration >> expressionAt:ifPresent:
 GsMethod 177920769,   WAUserConfiguration >> expressionAt:ifAbsent:
 GsMethod 178011649,   WAAttributeSearchContext >> isAttributeLocalOn:
 GsMethod 177471233,   WAConfiguration >> inheritedValueForContext:
 GsMethod 178010881,   WAAttributeSearchContext >> isAttributeInheritedOn:
 GsMethod 178009857,   WAAttributeSearchContext >> at:put:
(vmGc  Instances counts for generation code
     GsMethod                                id:     98817    1025 instances   320176 bytes
 vmGc ---)
-----------------------------------------------------
GemStone: Error         Fatal
VM temporary object memory is full
, almost out of memory, too many markSweeps since last successful
scavenge
Error Category: 231169 [GemStone] Number: 4067 Arg Count: 1 Context : 20
Arg 1:   20

[Info]: Logging out at 03/27/2015 04:11:47 PM UTC


*****************************************************
****** GemStone Abnormal Shutdown at 03/27/2015 04:11:48 PM UTC
*****************************************************
-----------------------------------------------------
GemStone: Error         Fatal
VM temporary object memory is full
, almost out of memory, too many markSweeps since last successful
scavenge
Error Category: 231169 [GemStone] Number: 4067 Arg Count: 1 Context : 20
Arg 1:   20
~                                                                                                                                                                                                      
Type  :qu


Dale

On 03/27/2015 10:04 AM, Lawrence Kellogg wrote:

On Mar 27, 2015, at 12:48 PM, Dale Henrichs <[hidden email]> wrote:

Larry,

Yea the bulk of the progress that you've made should be committed... should probably add a final transcript show in the gemstoneReap method ... but when your `Expire Progress` matches the preceding `finished scan` that chunk of work won't be done again ...

At this point in time, I suggest that you make a backup in case we hit a problem (like run out of extent space or something) that would make things difficult to proceed from ... this is just precautionary, but prudent ...

I am interested in what caused the out of memory so checking in the gem log for a smalltalk stack when you ran out of memory would help ... it's possible that we'll just hit the same situation right off the bat, when we start process again, so might as well try to understand that one...


Well, the beginning of a very long stack dump is this: 

GsMethod 1926657,   Object >> _doesNotUnderstand:
 GsMethod 10662913,   metaSortedCollection >> sortBlock:
 GsMethod 1754881,   Behavior >> new
 GsMethod 9573889,   SortedCollection >> sortBlock:
 GsMethod 1924097,   Object >> size
 GsMethod 3381249,   SmallInteger >> >
 GsMethod 2242817,   metaKeyValueDictionary >> new
 GsMethod 2243841,   metaKeyValueDictionary >> new:
 GsMethod 2057729,   metaInteger >> _selectedPrimeGreaterThan:
 GsMethod 1924609,   Object >> at:
 GsMethod 3378689,   SmallInteger >> //
 GsMethod 3378945,   SmallInteger >> <=
 GsMethod 1753857,   Behavior >> _basicNew:
 GsMethod 2257665,   KeyValueDictionary >> _initializeWithoutClear:
 GsMethod 3181569,   metaSystem >> _sessionStateAt:put:
 GsMethod 3206913,   metaSystem >> myUserProfile
 GsMethod 3294977,   SymbolDictionary >> at:ifAbsent:
 GsMethod 1534721,   ComplexBlock >> value
 GsMethod 7693313,   metaGsSession >> currentSession
 GsMethod 7723777,   GsCurrentSession >> objectNamed:
 GsMethod 3275521,   SymbolList >> resolveSymbol:
 GsMethod 1795329,   Association >> value
 GsMethod 7592193,   metaGsPackagePolicy >> globalName
 GsMethod 3300353,   SymbolDictionary >> at:otherwise:
 GsMethod 3297281,   SymbolDictionary >> associationAt:otherwise:
 GsMethod 7660545,   GsPackagePolicy >> enabled
 GsMethod 7640833,   metaSessionTemps >> current
 GsMethod 3182081,   metaSystem >> _sessionStateAt:
 GsMethod 7657217,   GsPackagePolicy >> sessionMethodDictionaryGlobalName
 GsMethod 3299585,   SymbolDictionary >> at:put:
 GsMethod 3293953,   SymbolDictionary >> _validatePrivilegeOld:new:
 GsMethod 1789185,   metaAssociation >> newWithKey:value:
 GsMethod 3307777,   SymbolAssociation >> key:value:
 GsMethod 3308033,   SymbolAssociation >> key:
 GsMethod 1796865,   Association >> value:
 GsMethod 3300097,   SymbolDictionary >> _at:put:
 GsMethod 3299073,   SymbolDictionary >> _validatePrivilegeGeneric:
 GsMethod 3140097,   metaSystem >> _protectedMode
 GsMethod 3215873,   metaSystem >> _zeroArgPrim:
 GsMethod 2278145,   KeyValueDictionary >> _at:put:
 GsMethod 2230273,   IdentityDictionary >> hashFunction:
 GsMethod 1910529,   Object >> identityHash
 GsMethod 2272513,   KeyValueDictionary >> tableSize
 GsMethod 3377665,   SmallInteger >> \\
 GsMethod 2263297,   KeyValueDictionary >> keyAt:
 GsMethod 1923585,   Object >> _at:
 GsMethod 2267649,   KeyValueDictionary >> valueAt:
 GsMethod 1916417,   Object >> _basicAt:
 GsMethod 1699329,   Boolean >> |
 GsMethod 2276609,   KeyValueDictionary >> atHash:putKey:
 GsMethod 1922561,   Object >> _basicAt:put:
 GsMethod 2221569,   IdentityDictionary >> atHash:putValue:
 GsMethod 2256129,   KeyValueDictionary >> atHash:putValue:
 GsMethod 7661313,   GsPackagePolicy >> sessionMethodDictionary
 GsMethod 1900033,   Object >> class
 GsMethod 2278401,   KeyValueDictionary >> size
 GsMethod 7652097,   GsPackagePolicy >> symbolList
 GsMethod 7723521,   GsCurrentSession >> symbolList
 GsMethod 3395329,   SequenceableCollection >> reverseDo:
 GsMethod 1942785,   Number >> downTo:do:
 GsMethod 1943041,   Number >> downTo:by:do:

which ends with this:


 GsMethod 119023873,   GRObject >> initialize
 GsMethod 178011137,   WAAttributeSearchContext >> findAttributeAndSelectAncestorsOf:
 GsMethod 178009601,   WAAttributeSearchContext >> key
 GsMethod 177473793,   WAConfiguration >> localAttributeAt:ifPresent:
 GsMethod 177920257,   WAUserConfiguration >> localAttributeAt:ifAbsent:
 GsMethod 177924865,   WAUserConfiguration >> parents
 GsMethod 177361153,   WASystemConfiguration >> localAttributeAt:ifAbsent:
 GsMethod 177361409,   WASystemConfiguration >> attributes
 GsMethod 177362945,   WASystemConfiguration >> description
 GsMethod 177375233,   WAConfigurationDescription >> attributes
 GsMethod 177473281,   WAConfiguration >> expressionAt:ifPresent:
 GsMethod 177920769,   WAUserConfiguration >> expressionAt:ifAbsent:
 GsMethod 178011649,   WAAttributeSearchContext >> isAttributeLocalOn:
 GsMethod 177471233,   WAConfiguration >> inheritedValueForContext:
 GsMethod 178010881,   WAAttributeSearchContext >> isAttributeInheritedOn:
 GsMethod 178009857,   WAAttributeSearchContext >> at:put:
(vmGc  Instances counts for generation code
     GsMethod                                id:     98817    1025 instances   320176 bytes
 vmGc ---)
-----------------------------------------------------
GemStone: Error         Fatal
VM temporary object memory is full
, almost out of memory, too many markSweeps since last successful
scavenge
Error Category: 231169 [GemStone] Number: 4067 Arg Count: 1 Context : 20
Arg 1:   20

[Info]: Logging out at 03/27/2015 04:11:47 PM UTC


*****************************************************
****** GemStone Abnormal Shutdown at 03/27/2015 04:11:48 PM UTC
*****************************************************
-----------------------------------------------------
GemStone: Error         Fatal
VM temporary object memory is full
, almost out of memory, too many markSweeps since last successful
scavenge
Error Category: 231169 [GemStone] Number: 4067 Arg Count: 1 Context : 20
Arg 1:   20
                     

I’ll make a backup and try again.

Larry



Dale

On 03/27/2015 09:31 AM, Lawrence Kellogg wrote:

On Mar 27, 2015, at 12:08 PM, Dale Henrichs <[hidden email]> wrote:

Larry,

Not sure ... I guess the I'm curious if the system is in iowait or burning cpu ... top should give you that information ... I'd also be interested to know if you've run out of disk space ... check the stone log for that information ...

Based on what you find out we'll go on from there ...



The struggle continues, eventually, I ran out of temporary Object Memory:

<Mail Attachment.png>


Disk space as at 63%. Can I run it again? Did any of that stick?

Larry





Dale

On 03/27/2015 07:43 AM, Lawrence Kellogg wrote:
Well, I put in Transcript calls to show count all the time, not just at mod 1000..and I made it all the way through, except it does’t seem to want to come back to me. The image is frozen.

Is it hung in the final commit? 

Here is a screen shot of the last part of the display. Yes I’m running it in GemTools. I don’t care if it’s slow as long as it works. 

<Mail Attachment.png>


Larry

On Mar 26, 2015, at 1:01 AM, Dale Henrichs <[hidden email]> wrote:

Hmmm, I think we need to set a breakpoint in there ... I might be tired, but I can't see how that can happen ... but then that's what a debugger is for ...

Are you running this expression from GemTools? If so the `Transcript show:` might be awfully expensive (lot's o round trips over the WAN) ... you might use GsFile gciLogServer: and then tail the gem log file on the server to see progress ...

Need to find out where the 0 is coming from first...

Dale

On 3/25/15 7:34 PM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, I replaced the WACache>>gemstoneReap method and ran the code from before but it just shows Scan Progress: 0 over and over again. 

  I let it run a few hours but had to kill it as the computer is in my bedroom and I can’t sleep if it makes noise all night. 

  Do I try again tomorrow? 

  best, 

  Larry

 
On Mar 25, 2015, at 6:23 PM, Dale Henrichs <[hidden email]> wrote:

Hmm, I scanned the code in WACache>>gemstoneReap and saw that a commit was being down every 100 entries, but presumably you blew up while building the collection of entries to expire ... sorry about that ... There should be a a stack dump in the gem log (depending upon where you ran the expressions where we can verify my assumption.

It looks like we need to replace WACache>>gemstoneReap with the following:

gemstoneReap
  "Iterate through the cache and remove objects that have expired."

  "In GemStone, this method is performed by a separate maintenance VM,
     so we are already in transaction (assumed to be running in #autoBegin
     transactionMode) and do not have to worry about acquiring the TransactionMutex.
    Since we are using reducedConflict dictionaries in the first place, we will remove the keys
    and values from the existing dictionaries without using the mutex."

  | expired count platform |
  expired := UserGlobals at: #'ExpiryCleanup' put: OrderedCollection new.
  platform := GRPlatform current.
  platform doCommitTransaction.
  count := 0.
  objectsByKey
    associationsDo: [ :assoc |
      (self expiryPolicy isExpiredUpdating: assoc value key: assoc key)
        ifTrue: [
          self notifyRemoved: assoc value key: assoc key.
          count := count + 1.
          expired add: assoc.
          count \\ 100 == 0
            ifTrue: [ platform doCommitTransaction ].
          count \\ 1000 == 0
            ifTrue: [ Transcript cr; show: 'Scan progress: ' , count printString.] ] ].
  Transcript cr; show: 'finished scan: ' , count printString.
  count := 0.
  (UserGlobals at: #'ExpiryCleanup' )
    do: [ :assoc |
      count := count + 1.
      objectsByKey removeKey: assoc key ifAbsent: [].
      keysByObject removeKey: assoc value ifAbsent: [  ].
      count \\ 100 == 0
        ifTrue: [ platform doCommitTransaction ].
      count \\ 1000 == 0
        ifTrue: [ Transcript cr; show: 'Expire progress: ' , count printString.]].
  platform doCommitTransaction.
  UserGlobals removeKey: #'ExpiryCleanup'.
  platform doCommitTransaction.
  ^ expired size

This implementation should be more resistant to an out of memory condition and I've got some logging in  there as well ... the `Transcript show:` should show up in the gem.log and/or Transcript...

I haven't tested this but if there's a problem in the scan loop it will fail quickly. If there's a problem in the expire loop, you can skip the scan loop, and just run the expire loop ...

Sorry about that ... hopefully the second time will be the charm ...

Dale

On 3/25/15 2:36 PM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, Step 1 ran for hours and halted in the debugger with this error: 

Error: VM temporary object memory is full 
, almost out of memory, too many markSweeps since last successful scavenge


What do I do now? 

Best, 

  Larry 



On Mar 25, 2015, at 2:15 PM, Dale Henrichs <[hidden email]> wrote:

Okay here's the sequence of steps that I think you should take:

  1. expire all of your sessions:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransactions

  2. initalize your object log

  3. run MFC

  [
  System abortTransaction.
  SystemRepository markForCollection ]
    on: Warning
    do: [ :ex |
      Transcript
        cr;
        show: ex description.
      ex resume ]

  4. Then do a backup and restore ... you can use GemTools to do the restore,
      but then you should read the SysAdmin docs[1] for instructions to do the restore
      (I've enclosed link to 3.2 docs, but the procedure and commands should pretty
      much be the same, but it's best to look up the docs for your GemStone version[2]
      and follow those instructions)

As I mentioned earlier, it will probably take a while for each of these operations to complete (object log will be fast and the backup will be fast, if the mfc tosses out the majority of your data) and it is likely that the repository will grow some more during the process (hard to predict this one, tho).

Step 1 will touch every session and every continuation so it is hard to say what percent of the objects are going to be touched (the expensive part), still there are likely to be a lot of those puppies and they will have to be read from disk into the SPC ...

Step 3. is going scan all of the live objects and again it hard to predict exactly how expensive it will be ...

Dale

[1] http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm
[2] http://gemtalksystems.com/techsupport/resources/

On 3/25/15 10:18 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL. 

  I’m not running the maintenance VM and I haven’t been doing regular mark for collects. 

  I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup? 

  I’ll also try the ObjectLog trick. 

  I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual. 

  
Best, 

Larry


  
On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Lawrence,

Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?

Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...

Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...

The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...

Dale

On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22528 megabytes.
    Repository has grown to 22528 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22544 megabytes.
    Repository has grown to 22544 megabytes.

--- 03/08/2015 03:31:45 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22560 megabytes.
    Repository has grown to 22560 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22576 megabytes.
    Repository has grown to 22576 megabytes.

--- 03/10/2015 03:19:34 AM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22592 megabytes.
    Repository has grown to 22592 megabytes.

--- 03/10/2015 03:46:39 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22608 megabytes.
    Repository has grown to 22608 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22624 megabytes.
    Repository has grown to 22624 megabytes.


My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass















_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
123