Large image acting strangely ...

My normal image runs about 120Mb  with 2.8 million objects.
This image shows
    242.84 Mbytes total;    220.04 Mbytes used,    22.8 Mbytes free.
and "numOopsNumBytes" shows
    #(5,335,909 218,775,720)
(the commas are mine so I can read it).

This image just "freezes" for periods from 2 to 5 seconds.  Nothing works
(Ctrl/\ ignored,  Shift+Ctrl/Y ignored) -- until it comes back again.
Some actions I normally do which take a fraction of a second (opening a
particular window)
can now freeze (same as a above) for 10 to 20 seconds.

Since I cannot interrupt it during this "freeze", I don't know what its
doing.
TimeProfiler does not seem to tell me anything much other than that it
spent lots of time doing
a fair number of unrelated things -- which seems strange.

Anyone seen this kind of behavior? have any suggestions??

--
Dennis Smith                         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              sip:[hidden email]
Canada         http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

I observed a similar behavior having lots of background processes running
which always - on intend - waited for 0.1 sec before doing something. But too many
didn't end properly so they kept running and froze my image.

Karl
 


__________________________________________________________
Karl Breith
AREVA NP GmbH
Methods & Codes (FDEEC-G)

Freyeslebenstraße 1
91058 Erlangen
Phone: +49 (0) 9131 18-97393
Fax: +49 (0) 9131 18-94045
mail to:  [hidden email]
An AREVA and Siemens company
 
Vorsitzender des Aufsichtsrats: Bertrand Durrande - Geschäftsführer: Dr. Ralf Güldner, Rüdiger Steuerlein
Sitz der Gesellschaft: Erlangen - Registergericht: Fürth, HRB 7817 - www.areva-np.com
 
Wichtiger Hinweis: Diese E-Mail kann Betriebs- oder Geschäftsgeheimnisse, bzw. sonstige vertrauliche Informationen enthalten. Sollten Sie diese E-Mail irrtümlich erhalten haben, ist Ihnen eine Kenntnisnahme des Inhalts, eine Vervielfältigung oder Weitergabe der E-Mail ausdrücklich untersagt. Bitte benachrichtigen Sie uns und vernichten Sie die empfangene E-Mail. Vielen Dank.
 
Important Note: This e-mail may contain trade secrets or privileged, undisclosed or otherwise confidential information. If you have received this e-mail in error, you are hereby notified that any review, copying or distribution of it is strictly prohibited. Please inform us immediately and destroy the original transmittal. Thank you for your cooperation.


-----Ursprüngliche Nachricht-----
Von: Dennis Smith [mailto:[hidden email]]
Gesendet: Dienstag, 17. April 2007 14:41
An: VWNC,
Betreff: Large image acting strangely ...

My normal image runs about 120Mb  with 2.8 million objects.
This image shows
    242.84 Mbytes total;    220.04 Mbytes used,    22.8 Mbytes free.
and "numOopsNumBytes" shows
    #(5,335,909 218,775,720)
(the commas are mine so I can read it).

This image just "freezes" for periods from 2 to 5 seconds.  Nothing works (Ctrl/\ ignored,  Shift+Ctrl/Y ignored) -- until it comes back again.
Some actions I normally do which take a fraction of a second (opening a particular window) can now freeze (same as a above) for 10 to 20 seconds.

Since I cannot interrupt it during this "freeze", I don't know what its doing.
TimeProfiler does not seem to tell me anything much other than that it spent lots of time doing a fair number of unrelated things -- which seems strange.

Anyone seen this kind of behavior? have any suggestions??

--
Dennis Smith                         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              sip:[hidden email]
Canada         http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

Look at growthRegimeUpperBound and memoryUpperBound (in
ObjectMemory), and see if they are set in such a way that your image
gets into a memory thrash situation.

At 08:41 AM 4/17/2007, you wrote:
<Talk Small and Carry a Big Class Library>
James Robertson, Product Manager, Cincom Smalltalk
http://www.cincomsmalltalk.com/blog/blogView

James Robertson wrote:
> Look at growthRegimeUpperBound and memoryUpperBound (in ObjectMemory),
> and see if they are set in such a way that your image gets into a
> memory thrash situation.
Sorry, should have shown those.   They memory policy stuff is
    400Mb    upper
    360Mb   growth regume upper
    18Mb   free
Memory used is 244Mb


--
Dennis Smith                         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              sip:[hidden email]
Canada         http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

Process monitor shows nothing strange.
I will keep looking into this and report back if I find anything
interesting.

BREITH Karl-Albert FDEEC (AREVA NP GmbH) wrote:
--
Dennis Smith                         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              sip:[hidden email]
Canada         http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

Dennis

It could be either doing a garbage collect or a memory
compaction, both can freeze the image.

Terry
 
===========================================================
Terry Raymond       Smalltalk Professional Debug Package
Crafted Smalltalk
80 Lazywood Ln.
Tiverton, RI  02878
(401) 624-4517      [hidden email]
<http://www.craftedsmalltalk.com>
===========================================================

I thought that too -- still do even more after more testing because I put breakpoints at a number of places
in the code its going through and the "delays" are random (by nothing when it got to each BP).
HOWEVER, I dumped to the transcript at all the GC and compact methods I could find in ObjectMemory (class side)
and nothing showed.  Also, no cursor changes.  Perhaps I missed something??

Terry Raymond wrote:

Dennis

It could be either doing a garbage collect or a memory
compaction, both can freeze the image.

Terry
 
===========================================================
Terry Raymond       Smalltalk Professional Debug Package
Crafted Smalltalk
80 Lazywood Ln.
Tiverton, RI  02878
(401) 624-4517      [hidden email]
<http://www.craftedsmalltalk.com>
===========================================================
  

-----Original Message-----
From: Dennis Smith [[hidden email]]
Sent: Tuesday, April 17, 2007 8:41 AM
To: VWNC,
Subject: Large image acting strangely ...

My normal image runs about 120Mb  with 2.8 million objects.
This image shows
    242.84 Mbytes total;    220.04 Mbytes used,    22.8 Mbytes free.
and "numOopsNumBytes" shows
    #(5,335,909 218,775,720)
(the commas are mine so I can read it).

This image just "freezes" for periods from 2 to 5 seconds.  Nothing works
(Ctrl/\ ignored,  Shift+Ctrl/Y ignored) -- until it comes back again.
Some actions I normally do which take a fraction of a second (opening a
particular window)
can now freeze (same as a above) for 10 to 20 seconds.

Since I cannot interrupt it during this "freeze", I don't know what its
doing.
TimeProfiler does not seem to tell me anything much other than that it
spent lots of time doing
a fair number of unrelated things -- which seems strange.

Anyone seen this kind of behavior? have any suggestions??

--
Dennis Smith                 		         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              [hidden email]
Canada			         http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP
    

-- 
Dennis Smith                 		         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              [hidden email]
Canada			         http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

Hmm - along the process lines brought up by Karl, any chance you
could be seeing new objects getting created and tenured too quickly -
i.e., is old space growing steadily?

At 08:49 AM 4/17/2007, you wrote:


<Talk Small and Carry a Big Class Library>
James Robertson, Product Manager, Cincom Smalltalk
http://www.cincomsmalltalk.com/blog/blogView

Dennis Smith wrote:

I thought that too -- still do even more after more testing because I put breakpoints at a number of places
in the code its going through and the "delays" are random (by nothing when it got to each BP).
HOWEVER, I dumped to the transcript at all the GC and compact methods I could find in ObjectMemory (class side)
and nothing showed.  Also, no cursor changes.  Perhaps I missed something??

I just noticed that TimeProfiler shows some GC stuff at the top.
Its showing, on average, 30 scavanges and 1 incGC.

Terry Raymond wrote:

Dennis

It could be either doing a garbage collect or a memory
compaction, both can freeze the image.

Terry
 
===========================================================
Terry Raymond       Smalltalk Professional Debug Package
Crafted Smalltalk
80 Lazywood Ln.
Tiverton, RI  02878
(401) 624-4517      [hidden email]
<http://www.craftedsmalltalk.com>
===========================================================
  

-----Original Message-----
From: Dennis Smith [[hidden email]]
Sent: Tuesday, April 17, 2007 8:41 AM
To: VWNC,
Subject: Large image acting strangely ...

My normal image runs about 120Mb  with 2.8 million objects.
This image shows
    242.84 Mbytes total;    220.04 Mbytes used,    22.8 Mbytes free.
and "numOopsNumBytes" shows
    #(5,335,909 218,775,720)
(the commas are mine so I can read it).

This image just "freezes" for periods from 2 to 5 seconds.  Nothing works
(Ctrl/\ ignored,  Shift+Ctrl/Y ignored) -- until it comes back again.
Some actions I normally do which take a fraction of a second (opening a
particular window)
can now freeze (same as a above) for 10 to 20 seconds.

Since I cannot interrupt it during this "freeze", I don't know what its
doing.
TimeProfiler does not seem to tell me anything much other than that it
spent lots of time doing
a fair number of unrelated things -- which seems strange.

Anyone seen this kind of behavior? have any suggestions??

--
Dennis Smith                 		         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              [hidden email]
Canada			         http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP
    

-- 
Dennis Smith                 		         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              [hidden email]
Canada			         http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

-- 
Dennis Smith                 		         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              [hidden email]
Canada			         http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

James Robertson wrote:
> Hmm - along the process lines brought up by Karl, any chance you could
> be seeing new objects getting created and tenured too quickly - i.e.,
> is old space growing steadily?
Maybe -- how do I look at that??
I also noted that when its in this "frozen" state, I cannot move the
current window -- in fact when I move the cursor
up to the title bar, it does not change, still stays as hour-glass
(which I put it at when I started the operation).

--
Dennis Smith                         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              sip:[hidden email]
Canada         http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

It doesn't sound like a normal sort of lockup to me.

The way to test for the garbage thing is to use the MultiTimeProfiler to begin with - this will show a disproportionate number of methods in the timer takign a substantial amount of time doing things related to garbage collection.

If that proves to be right, you can use the MultiAllocProfiler to see what is begin made in a period of time and why.

But I kinda doubt that's your problem... it sort of sounds more like a heap memory leak. Do you use any C interfaces?

Cheers,
Michael

Thanks -- it did not show much
    100.0 All Processes
       100 Primitives
looks a bit strange.
Anyway, I think I found about 90% of the problem.  I have two items of size 300,000.
One is a WeakDictionary and the other a WeakKeyedRegistry.

A bit of explanation perhaps.  We normally run against a Gemstone database and in a few places we use
the oop of the object, either asking the object for its oop or asking for the object belonging to a saved oop.
So when we run (for testing or development purposes) without Gemstone, we need a similar capability.
Previously I kept an OrderedCollection and searched it -- worked well for small numbers of business objects.
In THIS case I am testing a payroll system for a client with about 5000 employees and about 60,000 detail objects
created during a payrun.  Wanted to do it without Gemstone because of initial load time of the base objects,
so the WeekXxx stuff got a workout.

As a quick test I emptied those Weak-thingy's and my 20 seconds becamse 1.5 seconds.

Thanks for everyone's help.  I will look harder the the Weak-thingy's -- the problem with anything else of course
is that the objects come and go all the time.
In the final analysis, this is NOT a real problem -- I was concerned about what would happen with a large number of
objects when connected to Gemstone, but my WeakThingy's don't exist then.


Michael Lucas-Smith wrote:

It doesn't sound like a normal sort of lockup to me.

The way to test for the garbage thing is to use the MultiTimeProfiler to begin with - this will show a disproportionate number of methods in the timer takign a substantial amount of time doing things related to garbage collection.

If that proves to be right, you can use the MultiAllocProfiler to see what is begin made in a period of time and why.

But I kinda doubt that's your problem... it sort of sounds more like a heap memory leak. Do you use any C interfaces?

Cheers,
Michael

On 18/04/07, Dennis Smith <[hidden email]> wrote:

James Robertson wrote:
> Hmm - along the process lines brought up by Karl, any chance you could
> be seeing new objects getting created and tenured too quickly - i.e.,
> is old space growing steadily?
Maybe -- how do I look at that??
I also noted that when its in this "frozen" state, I cannot move the
current window -- in fact when I move the cursor
up to the title bar, it does not change, still stays as hour-glass
(which I put it at when I started the operation).

>
> At 08:49 AM 4/17/2007, you wrote:
>
>
>> James Robertson wrote:
>>> Look at growthRegimeUpperBound and memoryUpperBound (in
>>> ObjectMemory), and see if they are set in such a way that your image
>>> gets into a memory thrash situation.
>> Sorry, should have shown those.   They memory policy stuff is
>>    400Mb    upper
>>    360Mb   growth regume upper
>>    18Mb   free
>> Memory used is 244Mb
>>
>>
>>>
>>> At 08:41 AM 4/17/2007, you wrote:
>>>> My normal image runs about 120Mb  with 2.8 million objects.
>>>> This image shows
>>>>    242.84 Mbytes total;    220.04 Mbytes used,    22.8 Mbytes free.
>>>> and "numOopsNumBytes" shows
>>>>    #(5,335,909 218,775,720)
>>>> (the commas are mine so I can read it).
>>>>
>>>> This image just "freezes" for periods from 2 to 5 seconds.  Nothing
>>>> works
>>>> (Ctrl/\ ignored,  Shift+Ctrl/Y ignored) -- until it comes back again.
>>>> Some actions I normally do which take a fraction of a second
>>>> (opening a particular window)
>>>> can now freeze (same as a above) for 10 to 20 seconds.
>>>>
>>>> Since I cannot interrupt it during this "freeze", I don't know what
>>>> its doing.
>>>> TimeProfiler does not seem to tell me anything much other than that
>>>> it spent lots of time doing
>>>> a fair number of unrelated things -- which seems strange.
>>>>
>>>> Anyone seen this kind of behavior? have any suggestions??
>>>>
>>>> --
>>>> Dennis Smith                                     +1 416.798.7948
>>>> Cherniak Software Development Corporation   Fax: +1 416.798.0948
>>>> 509-2001 Sheppard Avenue East         [hidden email]
>>>> Toronto, ON M2J 4Z8              [hidden email]
>>>> Canada                           http://www.CherniakSoftware.com
>>>> Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP
>>>
>>> <Talk Small and Carry a Big Class Library>
>>> James Robertson, Product Manager, Cincom Smalltalk
>>> http://www.cincomsmalltalk.com/blog/blogView
>>
>> --
>> Dennis Smith                                     +1 416.798.7948
>> Cherniak Software Development Corporation   Fax: +1 416.798.0948
>> 509-2001 Sheppard Avenue East        [hidden email]
>> Toronto, ON M2J 4Z8               [hidden email]
>> Canada                           http://www.CherniakSoftware.com
>> Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP
>
> <Talk Small and Carry a Big Class Library>
> James Robertson, Product Manager, Cincom Smalltalk
> http://www.cincomsmalltalk.com/blog/blogView
>

--
Dennis Smith                                     +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              [hidden email]
Canada                           http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

-- 
Dennis Smith                 		         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              [hidden email]
Canada			         http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

I will look over your notes below - particularly on gemstone.
I did narrow down my problems to 200,000+ Ephemerons in the
WeakKeyedRegistry.  I got rid of those using only a single WeakArray of about 800,000 elements
(many empty because I converted the table from the Ephemerons).
This now runs at a reasonable speed.  So it was the VM presumably locking up looking at all those
Ephemerons.

Paul Baumann wrote:

Hashing collections become inefficient and revert to linear searches once you exceed hash limits (16K in VW). The freeze is in growing large inefficient hashed collections. Consider how many of those objects are stored in large hashed collections.

 

Since you mentioned GemStone, here are some GBS performance improvements related to hash limitations that may or may not apply to the GBS version you are using:

 

GBS has an incomplete set of performance improvements. The GsToSt cache uses oop value in place of a hash value but the caches are still constrained to 16K.

 

GbxGsToStDictionary class>>maximumHashValue
 "Integer keys hash to themselves--they have no limits."

 "These ICE changes to #maximumHashValue and #selectedPrimes provide a typical 29% general performance improvement
  by allowing caches to grow within the new oop range of hash values. -plb 20070327"

 ^SmallInteger maxVal

 

GbxGsToStDictionary class>>selectedPrimes
 "These ICE changes to #maximumHashValue and #selectedPrimes provide a typical 29% general performance improvement
  by allowing caches to grow within the new oop range of hash values. -plb 20070327"

 ^#(1009 10007 26003 200001 300001 400001)

 

 

Assigning the session semaphore to the cache (as it was prior to about GBS 5.2) and then using a condition to avoid traversing the semaphore results in a 30% general performance boost:

 

GbsSession>>sessionProtect
 "ICE extension for performance improvements. -plb 20070328"
 ^sessionProtect

GbxGsToStDictionary>>at: aKey otherwise: otherwiseObject
 "ICE extension for performance improvements. -plb 20070328"

 ^accessProtect isActiveProcessInCriticalRegion
  ifTrue: [self basicAt: aKey otherwise: otherwiseObject ]
  ifFalse: [self readCritical: [ self basicAt: aKey otherwise: otherwiseObject ]].

GbxGsToStDictionary>>owner: aSession
 "ICE extension for performance improvements. -plb 20070328"

 super owner: aSession.
 accessProtect := aSession sessionProtect.

 

I've also achieved a *steady* 9-22% cache performance increase by tuning cache buckets, but that is more than just a few methods. In short, I avoided empty-slot traversal costs and fixed a GBS bug preventing bucket pre-growth.

 

I also got a 30% general performance improvement in code that may frequent use of #asLocalObjectCopy by using a more efficient collection in place of Dictionary for a traversal keeper. Newer GBS versions remove efficient #asLocalObjectCopy though.

 

You mentioned WeakArray performance so I'll ask what version of GBS you are using. The most recent version of GBS I saw was starting to implement a replacement for GsToSt cache that stores in a b-tree and would have many WeakArray slots to finalize. I don't expect the cache saturation to be favorable with that approach and finalization is one way it would show.

 

Paul Baumann 
IntercontinentalExchange | ICE
[hidden email]

 

---

From: Dennis Smith [[hidden email]]
Sent: Tuesday, April 17, 2007 1:37 PM
To: vwnc >> "VWNC, "
Subject: Re: Large image acting strangely ...

Thanks -- it did not show much
    100.0 All Processes
       100 Primitives
looks a bit strange.
Anyway, I think I found about 90% of the problem.  I have two items of size 300,000.
One is a WeakDictionary and the other a WeakKeyedRegistry.

A bit of explanation perhaps.  We normally run against a Gemstone database and in a few places we use
the oop of the object, either asking the object for its oop or asking for the object belonging to a saved oop.
So when we run (for testing or development purposes) without Gemstone, we need a similar capability.
Previously I kept an OrderedCollection and searched it -- worked well for small numbers of business objects.
In THIS case I am testing a payroll system for a client with about 5000 employees and about 60,000 detail objects
created during a payrun.  Wanted to do it without Gemstone because of initial load time of the base objects,
so the WeekXxx stuff got a workout.

As a quick test I emptied those Weak-thingy's and my 20 seconds becamse 1.5 seconds.

Thanks for everyone's help.  I will look harder the the Weak-thingy's -- the problem with anything else of course
is that the objects come and go all the time.
In the final analysis, this is NOT a real problem -- I was concerned about what would happen with a large number of
objects when connected to Gemstone, but my WeakThingy's don't exist then.


Michael Lucas-Smith wrote:

It doesn't sound like a normal sort of lockup to me.

The way to test for the garbage thing is to use the MultiTimeProfiler to begin with - this will show a disproportionate number of methods in the timer takign a substantial amount of time doing things related to garbage collection.

If that proves to be right, you can use the MultiAllocProfiler to see what is begin made in a period of time and why.

But I kinda doubt that's your problem... it sort of sounds more like a heap memory leak. Do you use any C interfaces?

Cheers,
Michael

On 18/04/07, Dennis Smith <[hidden email]> wrote:

James Robertson wrote:
> Hmm - along the process lines brought up by Karl, any chance you could
> be seeing new objects getting created and tenured too quickly - i.e.,
> is old space growing steadily?
Maybe -- how do I look at that??
I also noted that when its in this "frozen" state, I cannot move the
current window -- in fact when I move the cursor
up to the title bar, it does not change, still stays as hour-glass
(which I put it at when I started the operation).

>
> At 08:49 AM 4/17/2007, you wrote:
>
>
>> James Robertson wrote:
>>> Look at growthRegimeUpperBound and memoryUpperBound (in
>>> ObjectMemory), and see if they are set in such a way that your image
>>> gets into a memory thrash situation.
>> Sorry, should have shown those.   They memory policy stuff is
>>    400Mb    upper
>>    360Mb   growth regume upper
>>    18Mb   free
>> Memory used is 244Mb
>>
>>
>>>
>>> At 08:41 AM 4/17/2007, you wrote:
>>>> My normal image runs about 120Mb  with 2.8 million objects.
>>>> This image shows
>>>>    242.84 Mbytes total;    220.04 Mbytes used,    22.8 Mbytes free.
>>>> and "numOopsNumBytes" shows
>>>>    #(5,335,909 218,775,720)
>>>> (the commas are mine so I can read it).
>>>>
>>>> This image just "freezes" for periods from 2 to 5 seconds.  Nothing
>>>> works
>>>> (Ctrl/\ ignored,  Shift+Ctrl/Y ignored) -- until it comes back again.
>>>> Some actions I normally do which take a fraction of a second
>>>> (opening a particular window)
>>>> can now freeze (same as a above) for 10 to 20 seconds.
>>>>
>>>> Since I cannot interrupt it during this "freeze", I don't know what
>>>> its doing.
>>>> TimeProfiler does not seem to tell me anything much other than that
>>>> it spent lots of time doing
>>>> a fair number of unrelated things -- which seems strange.
>>>>
>>>> Anyone seen this kind of behavior? have any suggestions??
>>>>
>>>> --
>>>> Dennis Smith                                     +1 416.798.7948
>>>> Cherniak Software Development Corporation   Fax: +1 416.798.0948
>>>> 509-2001 Sheppard Avenue East         [hidden email]
>>>> Toronto, ON M2J 4Z8              [hidden email]
>>>> Canada                           http://www.CherniakSoftware.com
>>>> Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP
>>>
>>> <Talk Small and Carry a Big Class Library>
>>> James Robertson, Product Manager, Cincom Smalltalk
>>> http://www.cincomsmalltalk.com/blog/blogView
>>
>> --
>> Dennis Smith                                     +1 416.798.7948
>> Cherniak Software Development Corporation   Fax: +1 416.798.0948
>> 509-2001 Sheppard Avenue East        [hidden email]
>> Toronto, ON M2J 4Z8               [hidden email]
>> Canada                           http://www.CherniakSoftware.com
>> Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP
>
> <Talk Small and Carry a Big Class Library>
> James Robertson, Product Manager, Cincom Smalltalk
> http://www.cincomsmalltalk.com/blog/blogView
>

--
Dennis Smith                                     +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              [hidden email]
Canada                           http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

-- 
Dennis Smith                 		         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              [hidden email]
Canada			         http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

 

 

 

---

This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange, Inc. (ICE), its subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.

-- 
Dennis Smith                 		         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              [hidden email]
Canada			         http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

I've seen something similar where I had a large weak dictionary with sequential integer keys. That could get into pathological situations as it removed elements, and it was happening a high priority process, so it essentially froze the image.

At 03:35 PM 4/17/2007, Dennis Smith wrote:

I will look over your notes below - particularly on gemstone.
I did narrow down my problems to 200,000+ Ephemerons in the
WeakKeyedRegistry.  I got rid of those using only a single WeakArray of about 800,000 elements
(many empty because I converted the table from the Ephemerons).
This now runs at a reasonable speed.  So it was the VM presumably locking up looking at all those
Ephemerons.

Paul Baumann wrote:

Hashing collections become inefficient and revert to linear searches once you exceed hash limits (16K in VW). The freeze is in growing large inefficient hashed collections. Consider how many of those objects are stored in large hashed collections.
 
Since you mentioned GemStone, here are some GBS performance improvements related to hash limitations that may or may not apply to the GBS version you are using:
 
GBS has an incomplete set of performance improvements. The GsToSt cache uses oop value in place of a hash value but the caches are still constrained to 16K.
 
GbxGsToStDictionary class>>maximumHashValue
 "Integer keys hash to themselves--they have no limits."
 "These ICE changes to #maximumHashValue and #selectedPrimes provide a typical 29% general performance improvement
  by allowing caches to grow within the new oop range of hash values. -plb 20070327"
 ^SmallInteger maxVal
 
GbxGsToStDictionary class>>selectedPrimes
 "These ICE changes to #maximumHashValue and #selectedPrimes provide a typical 29% general performance improvement
  by allowing caches to grow within the new oop range of hash values. -plb 20070327"
 ^#(1009 10007 26003 200001 300001 400001)
 
 
Assigning the session semaphore to the cache (as it was prior to about GBS 5.2) and then using a condition to avoid traversing the semaphore results in a 30% general performance boost:
 
GbsSession>>sessionProtect
 "ICE extension for performance improvements. -plb 20070328"
 ^sessionProtect
GbxGsToStDictionary>>at: aKey otherwise: otherwiseObject
 "ICE extension for performance improvements. -plb 20070328"
 ^accessProtect isActiveProcessInCriticalRegion
  ifTrue: [self basicAt: aKey otherwise: otherwiseObject ]
  ifFalse: [self readCritical: [ self basicAt: aKey otherwise: otherwiseObject ]].
GbxGsToStDictionary>>owner: aSession
 "ICE extension for performance improvements. -plb 20070328"
 super owner: aSession.
 accessProtect := aSession sessionProtect.
 
I've also achieved a *steady* 9-22% cache performance increase by tuning cache buckets, but that is more than just a few methods. In short, I avoided empty-slot traversal costs and fixed a GBS bug preventing bucket pre-growth.
 
I also got a 30% general performance improvement in code that may frequent use of #asLocalObjectCopy by using a more efficient collection in place of Dictionary for a traversal keeper. Newer GBS versions remove efficient #asLocalObjectCopy though.
 
You mentioned WeakArray performance so I'll ask what version of GBS you are using. The most recent version of GBS I saw was starting to implement a replacement for GsToSt cache that stores in a b-tree and would have many WeakArray slots to finalize. I don't expect the cache saturation to be favorable with that approach and finalization is one way it would show.
 
Paul Baumann
IntercontinentalExchange | ICE
[hidden email]
 

---

From: Dennis Smith [[hidden email]]
Sent: Tuesday, April 17, 2007 1:37 PM
To: vwnc >> "VWNC, "
Subject: Re: Large image acting strangely ...

Thanks -- it did not show much
    100.0 All Processes
       100 Primitives
looks a bit strange.
Anyway, I think I found about 90% of the problem.  I have two items of size 300,000.
One is a WeakDictionary and the other a WeakKeyedRegistry.

A bit of explanation perhaps.  We normally run against a Gemstone database and in a few places we use
the oop of the object, either asking the object for its oop or asking for the object belonging to a saved oop.
So when we run (for testing or development purposes) without Gemstone, we need a similar capability.
Previously I kept an OrderedCollection and searched it -- worked well for small numbers of business objects.
In THIS case I am testing a payroll system for a client with about 5000 employees and about 60,000 detail objects
created during a payrun.  Wanted to do it without Gemstone because of initial load time of the base objects,
so the WeekXxx stuff got a workout.

As a quick test I emptied those Weak-thingy's and my 20 seconds becamse 1.5 seconds.

Thanks for everyone's help.  I will look harder the the Weak-thingy's -- the problem with anything else of course
is that the objects come and go all the time.
In the final analysis, this is NOT a real problem -- I was concerned about what would happen with a large number of
objects when connected to Gemstone, but my WeakThingy's don't exist then.


Michael Lucas-Smith wrote:

It doesn't sound like a normal sort of lockup to me.

The way to test for the garbage thing is to use the MultiTimeProfiler to begin with - this will show a disproportionate number of methods in the timer takign a substantial amount of time doing things related to garbage collection.

If that proves to be right, you can use the MultiAllocProfiler to see what is begin made in a period of time and why.

But I kinda doubt that's your problem... it sort of sounds more like a heap memory leak. Do you use any C interfaces?

Cheers,
Michael

On 18/04/07, Dennis Smith <[hidden email]> wrote:

James Robertson wrote:

> Hmm - along the process lines brought up by Karl, any chance you could

> be seeing new objects getting created and tenured too quickly - i.e.,

> is old space growing steadily?

Maybe -- how do I look at that??

I also noted that when its in this "frozen" state, I cannot move the

current window -- in fact when I move the cursor

up to the title bar, it does not change, still stays as hour-glass

(which I put it at when I started the operation).

>

> At 08:49 AM 4/17/2007, you wrote:

>

>

>> James Robertson wrote:

>>> Look at growthRegimeUpperBound and memoryUpperBound (in

>>> ObjectMemory), and see if they are set in such a way that your image

>>> gets into a memory thrash situation.

>> Sorry, should have shown those.   They memory policy stuff is

>>    400Mb    upper

>>    360Mb   growth regume upper

>>    18Mb   free

>> Memory used is 244Mb

>>

>>

>>>

>>> At 08:41 AM 4/17/2007, you wrote:

>>>> My normal image runs about 120Mb  with 2.8 million objects.

>>>> This image shows

>>>>    242.84 Mbytes total;    220.04 Mbytes used,    22.8 Mbytes free.

>>>> and "numOopsNumBytes" shows

>>>>    #(5,335,909 218,775,720)

>>>> (the commas are mine so I can read it).

>>>>

>>>> This image just "freezes" for periods from 2 to 5 seconds.  Nothing

>>>> works

>>>> (Ctrl/\ ignored,  Shift+Ctrl/Y ignored) -- until it comes back again.

>>>> Some actions I normally do which take a fraction of a second

>>>> (opening a particular window)

>>>> can now freeze (same as a above) for 10 to 20 seconds.

>>>>

>>>> Since I cannot interrupt it during this "freeze", I don't know what

>>>> its doing.

>>>> TimeProfiler does not seem to tell me anything much other than that

>>>> it spent lots of time doing

>>>> a fair number of unrelated things -- which seems strange.

>>>>

>>>> Anyone seen this kind of behavior? have any suggestions??

>>>>

>>>> --

>>>> Dennis Smith                                     +1 416.798.7948

>>>> Cherniak Software Development Corporation   Fax: +1 416.798.0948

>>>> 509-2001 Sheppard Avenue East         [hidden email]

>>>> Toronto, ON M2J 4Z8              [hidden email]

>>>> Canada                           http://www.CherniakSoftware.com

>>>> Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

>>>

>>> <Talk Small and Carry a Big Class Library>

>>> James Robertson, Product Manager, Cincom Smalltalk

>>> http://www.cincomsmalltalk.com/blog/blogView

>>

>> --

>> Dennis Smith                                     +1 416.798.7948

>> Cherniak Software Development Corporation   Fax: +1 416.798.0948

>> 509-2001 Sheppard Avenue East        [hidden email]

>> Toronto, ON M2J 4Z8               [hidden email]

>> Canada                           http://www.CherniakSoftware.com

>> Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

>

> <Talk Small and Carry a Big Class Library>

> James Robertson, Product Manager, Cincom Smalltalk

> http://www.cincomsmalltalk.com/blog/blogView

>

--

Dennis Smith                                     +1 416.798.7948

Cherniak Software Development Corporation   Fax: +1 416.798.0948

509-2001 Sheppard Avenue East        [hidden email]

Toronto, ON M2J 4Z8              [hidden email]

Canada                           http://www.CherniakSoftware.com

Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

-- 
Dennis
Smith                
   
        
         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1
416.798.0948
509-2001 Sheppard Avenue East       
[hidden email]
Toronto, ON M2J
4Z8             
[hidden email]
Canada  
        
        
        

http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the
DVP

 

 

 

This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange, Inc. (ICE), its subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.

-- 
Dennis
Smith                
   
        
         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1
416.798.0948
509-2001 Sheppard Avenue East       
[hidden email]
Toronto, ON M2J
4Z8             
[hidden email]
Canada  
        
        
        

http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the
DVP

Dennis Smith wrote:
> I did narrow down my problems to 200,000+ Ephemerons in the
> WeakKeyedRegistry.  I got rid of those using only a single WeakArray of
> about 800,000 elements
> (many empty because I converted the table from the Ephemerons).
> This now runs at a reasonable speed.  So it was the VM presumably
> locking up looking at all those
> Ephemerons.
>

That's very interesting. Eliot once told me that he expected large
numbers of ephemerons to perform much better than one big WeakArray.

If anyone understands whey the reverse might be tru, I'd like to
understand it better.

Regards,

-Martin

Paul Baumann wrote:
Hi Paul,

I believe Dennis is using GBS 7.1 or later, which no longer uses the
GbxGsToStDictionary class.

I wouldn't think you'd need to modify #at:otherwise:. The
GbxRecursionLock does the same check to see if it needs to enter the
critical section, so you should get essentially the same performance
without it.

I suppose this is getting a bit off-topic for VWNC. If you want to
discuss further, I wouldn't mind moving this to the GemStone customer forum.

Regards,

-Martin

Martin McClure wrote: Wow -- have you got all the clients memorized?  I would have guessed 6.x
something, but its 7.1.1 -- guess
its been too long since I loaded a new one :)
--
Dennis Smith                         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              sip:[hidden email]
Canada         http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

Martin,

The condition avoids BlockClosure costs. The performance difference
matters in the caches--which is why the conditions used to exist in that
code.

Also note (again) the inefficiency of the code below:

GbsSession>>cacheCritical: aBlock
        ^GBSM stObjectCache writeCritical: [
                | gsCache |
                gsCache := gsObjectCache.
                gsCache isNil
                        ifTrue: aBlock
                        ifFalse: [ gsCache writeCritical: aBlock ]]

Paul Baumann

-----Original Message-----
From: Martin McClure [mailto:[hidden email]]
> I wouldn't think you'd need to modify #at:otherwise:. The
GbxRecursionLock does the same
> check to see if it needs to enter the critical section, so you should
get essentially
> the same performance without it.
 
 
 
--------------------------------------------------------
This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange, Inc. (ICE), its subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.  
 

Dennis Smith wrote:
> Thanks -- it did not show much
>    100.0 All Processes
>       100 Primitives
> looks a bit strange.
> Anyway, I think I found about 90% of the problem.  I have two items of
> size 300,000.
> One is a WeakDictionary and the other a WeakKeyedRegistry.
>

Hi Dennis,

I had something similar with DotNETConnect. When the object registry gets
large, garbage collection took a large part of the time. I found increasing
the size of newspace can help a lot. I think I did something like
ObjectMemory sizesAtStartup: #(10.0 10.0 1.0 1.0 1.0 10.0 1.0)
to make some tests run 3 times faster.

The problem is that WeakDictionary does a self rehash at the end of finalizeElements.
In rehash a new WeakDictionary with all the surviving elements is created
(in your case with 300000 keys, values and exectors). With a big enough WeakDictionary,
at that time newspace is full again, i.e. it is time for the next scavenge.

HTH

Ralf

--
Ralf Propach, [hidden email]
Tel: +49 231 975 99 38   Fax: +49 231 975 99 20
Georg Heeg eK (Dortmund)
Handelsregister: Amtsgericht Dortmund  A 12812