Smalltalk › Squeak › Squeak - Dev

Squeak hang at full cpu - help

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

7 messages Options

Alain rastoul

Squeak hang at full cpu - help

Hi Everybody,
I have a standard squeak 4.1 with seaside 3.0 preloaded under Windows XP
(seaside 3.0 rc OneClick Image) with FFI, ODBC packages installed and a very
memory intensive application who load data from a MSSQL server and hash it
into dictionaries.
It is running in a process started at systemBackgroundPriority, and is
launched with a MessageTally spyAllOn:.
After running a few minutes it hangs up at full CPU, and I can't see what is
going on, everything
is hung (no Ctrl., no mouse)
When I open the debug console and dump processes I see only few processes
(MessageTally>spyEvery:on:,
handleTimerEvent, userInterruptWatcher, lowSpaceWatcher) but no UI process
and no app process.
I tryed to put halts in code but it doesn't change anything.
I noticed that it occurs when the memory usage is about 99% (I see it with a
MemoryUsageMorph ticking),
but not allways at the same time, it can load 10k, 20k or 30k rows before
hanging.
If I remove the MessageTally, it ran several times without hanging.
I don't see any direct relation between MessageTally and hanging the VM
(allocations / GC ?)
I will try to generate and compile the vm with MSVC if I can (I can't test
my app under linux)

What is the way go with this kind of problem ?
Any help, tip for debugging, advice, idea, remark (...) are welcome.
Thanks in advance

Regards,
Alain

Levente Uzonyi-2

Re: Squeak hang at full cpu - help

On Thu, 21 Oct 2010, Alain_Rastoul wrote:

> Hi Everybody,
> I have a standard squeak 4.1 with seaside 3.0 preloaded under Windows XP
> (seaside 3.0 rc OneClick Image) with FFI, ODBC packages installed and a very
> memory intensive application who load data from a MSSQL server and hash it
> into dictionaries.
> It is running in a process started at systemBackgroundPriority, and is
> launched with a MessageTally spyAllOn:.
> After running a few minutes it hangs up at full CPU, and I can't see what is
> going on, everything
> is hung (no Ctrl., no mouse)
> When I open the debug console and dump processes I see only few processes
> (MessageTally>spyEvery:on:,
> handleTimerEvent, userInterruptWatcher, lowSpaceWatcher) but no UI process
> and no app process.
> I tryed to put halts in code but it doesn't change anything.
> I noticed that it occurs when the memory usage is about 99% (I see it with a
> MemoryUsageMorph ticking),
> but not allways at the same time, it can load 10k, 20k or 30k rows before
> hanging.
> If I remove the MessageTally, it ran several times without hanging.
> I don't see any direct relation between MessageTally and hanging the VM
> (allocations / GC ?)
> I will try to generate and compile the vm with MSVC if I can (I can't test
> my app under linux)
>
> What is the way go with this kind of problem ?
> Any help, tip for debugging, advice, idea, remark (...) are welcome.
> Thanks in advance

I guess you're running out of memory. What's the memory usage of Squeak
when the image hangs?

Levente

>
> Regards,
> Alain
>
>
>
>
>

Alain rastoul

Re: Squeak hang at full cpu - help

Thanks Levente, but. . . no, wrong guess , sorry :-)

Instead of memory intensive I should have said keeping memory about 95/99 %
used all the time until hang.

When squeak hangs it is about 60, 80, 90 or 100Mo , not always the same, but
not that much because it is a small test database (less than 500 mo in
mssql) and in most databases a lot of columns have redundant values - ie bad
selectivity.
I've been thinking of something wrong with FFI/ODBC, but another test I just
made without it have the same problem (initially with Cog but I reported it
in my squeak image) : I load a 1Gb bulk export file of the table of another
database into one big dictionary.
With squeak It hangs in few minutes at about 60 mo , in Cog it finishes in
30 minutes at about 220 mo.

I'm trying to compile the vm (ok for now) and run it under debug but gdb is
not ok at this time, I have problems with my mingw toolchain (MSVC didn't
want to compile perhaps due to the express version I have), I 'm not sure it
will be useful and I would prefer a "smalltalk" way to search.

I don't understand why everything seems to be ok without MessageTally ?
It seems abnormal to me - am I wrong? - to see only few processes in the
debug call stack ?
Another symptom that makes me think processes are terminated is that in the
debug output console a lot of event buffer overflow messages are displayed
when I move the mouse , I suppose because the UI process has vanished and
does not peek them.
I tryed to intercept process termination but it didn't work, I will make
another try with dump to a file it could work if they are "cleanly"
terminated

Thank you BTW for your response, any ideas are very welcome

Alain

"Levente Uzonyi" <[hidden email]> a écrit dans le message de news:
[hidden email]...

> On Thu, 21 Oct 2010, Alain_Rastoul wrote:
>
>> Hi Everybody,
>> I have a standard squeak 4.1 with seaside 3.0 preloaded under Windows XP
>> (seaside 3.0 rc OneClick Image) with FFI, ODBC packages installed and a
>> very
>> memory intensive application who load data from a MSSQL server and hash
>> it
>> into dictionaries.
>> It is running in a process started at systemBackgroundPriority, and is
>> launched with a MessageTally spyAllOn:.
>> After running a few minutes it hangs up at full CPU, and I can't see what
>> is
>> going on, everything
>> is hung (no Ctrl., no mouse)
>> When I open the debug console and dump processes I see only few processes
>> (MessageTally>spyEvery:on:,
>> handleTimerEvent, userInterruptWatcher, lowSpaceWatcher) but no UI
>> process
>> and no app process.
>> I tryed to put halts in code but it doesn't change anything.
>> I noticed that it occurs when the memory usage is about 99% (I see it
>> with a
>> MemoryUsageMorph ticking),
>> but not allways at the same time, it can load 10k, 20k or 30k rows before
>> hanging.
>> If I remove the MessageTally, it ran several times without hanging.
>> I don't see any direct relation between MessageTally and hanging the VM
>> (allocations / GC ?)
>> I will try to generate and compile the vm with MSVC if I can (I can't
>> test
>> my app under linux)
>>
>> What is the way go with this kind of problem ?
>> Any help, tip for debugging, advice, idea, remark (...) are welcome.
>> Thanks in advance
>
> I guess you're running out of memory. What's the memory usage of Squeak
> when the image hangs?
>
>
> Levente
>
>>
>> Regards,
>> Alain
>>
>>
>>
>>
>>
>
>

Alain rastoul

Re: Squeak hang at full cpu - help

Another precision, I have removed all gc tuning values I used before having
this problem, so the vm has all the default values for gc, I don't know if
it is gc or memory related.

"Alain_Rastoul" <[hidden email]> a écrit dans le message de news:
i9sch8$kuh$[hidden email]...

> Thanks Levente, but. . . no, wrong guess , sorry :-)
>
> Instead of memory intensive I should have said keeping memory about 95/99
> % used all the time until hang.
>
> When squeak hangs it is about 60, 80, 90 or 100Mo , not always the same,
> but not that much because it is a small test database (less than 500 mo in
> mssql) and in most databases a lot of columns have redundant values - ie
> bad selectivity.
> I've been thinking of something wrong with FFI/ODBC, but another test I
> just made without it have the same problem (initially with Cog but I
> reported it in my squeak image) : I load a 1Gb bulk export file of the
> table of another database into one big dictionary.
> With squeak It hangs in few minutes at about 60 mo , in Cog it finishes in
> 30 minutes at about 220 mo.
>
> I'm trying to compile the vm (ok for now) and run it under debug but gdb
> is not ok at this time, I have problems with my mingw toolchain (MSVC
> didn't want to compile perhaps due to the express version I have), I 'm
> not sure it will be useful and I would prefer a "smalltalk" way to search.
>
> I don't understand why everything seems to be ok without MessageTally ?
> It seems abnormal to me - am I wrong? - to see only few processes in the
> debug call stack ?
> Another symptom that makes me think processes are terminated is that in
> the debug output console a lot of event buffer overflow messages are
> displayed when I move the mouse , I suppose because the UI process has
> vanished and does not peek them.
> I tryed to intercept process termination but it didn't work, I will make
> another try with dump to a file it could work if they are "cleanly"
> terminated
>
> Thank you BTW for your response, any ideas are very welcome
>
> Alain
>
> "Levente Uzonyi" <[hidden email]> a écrit dans le message de news:
> [hidden email]...
>> On Thu, 21 Oct 2010, Alain_Rastoul wrote:
>>
>>> Hi Everybody,
>>> I have a standard squeak 4.1 with seaside 3.0 preloaded under Windows XP
>>> (seaside 3.0 rc OneClick Image) with FFI, ODBC packages installed and a
>>> very
>>> memory intensive application who load data from a MSSQL server and hash
>>> it
>>> into dictionaries.
>>> It is running in a process started at systemBackgroundPriority, and is
>>> launched with a MessageTally spyAllOn:.
>>> After running a few minutes it hangs up at full CPU, and I can't see
>>> what is
>>> going on, everything
>>> is hung (no Ctrl., no mouse)
>>> When I open the debug console and dump processes I see only few
>>> processes
>>> (MessageTally>spyEvery:on:,
>>> handleTimerEvent, userInterruptWatcher, lowSpaceWatcher) but no UI
>>> process
>>> and no app process.
>>> I tryed to put halts in code but it doesn't change anything.
>>> I noticed that it occurs when the memory usage is about 99% (I see it
>>> with a
>>> MemoryUsageMorph ticking),
>>> but not allways at the same time, it can load 10k, 20k or 30k rows
>>> before
>>> hanging.
>>> If I remove the MessageTally, it ran several times without hanging.
>>> I don't see any direct relation between MessageTally and hanging the VM
>>> (allocations / GC ?)
>>> I will try to generate and compile the vm with MSVC if I can (I can't
>>> test
>>> my app under linux)
>>>
>>> What is the way go with this kind of problem ?
>>> Any help, tip for debugging, advice, idea, remark (...) are welcome.
>>> Thanks in advance
>>
>> I guess you're running out of memory. What's the memory usage of Squeak
>> when the image hangs?
>>
>>
>> Levente
>>
>>>
>>> Regards,
>>> Alain
>>>
>>>
>>>
>>>
>>>
>>
>>
>
>
>
>
>

johnmci

Re: Squeak hang at full cpu - help

Ok, if you don't have

Smalltalk setGCBiasToGrowGCLimit: "some magic number in megabytes"
Smalltalk setGCBiasToGrow: 1.

Then you can enter into a situation (unlikely) where the amount of free space is just less than enough to
make the image grow, but enough to make the software do a full compaction. Think of it as a grey zone you
approach if you can pass thru it promptly the image will grow, if your movement is just not correct then then
you wander in the grey zone for days... The setGCBiasToGrow: alter that behavior so we pound thru the grey
zone until we've allocated setGCBiasToGrowGCLimit: where we must do a full compaction (expensive).

However what's more likely is that you actually run out of memory, this is a I believe on windows 512MB?
Other platforms may cap by default at 512MB or 1GB depending on the VM and it's default settings.
For non-windows platforms you can use the instructions "found somewhere?" to alter the default setting.

What happens in this case is when you approach the final bits of free memory via the hard upper cap the
VM will enter a endless cycle of doing GC work and finds just enough free memory measured in kilobytes to
to make another GC cycle attempt. If you are lucky this pattern will fail by the failure to allocated a needed
method context and the VM will crash.

I can only suppose that MessageTally requires memory to run, is it statically bound? Or will it's memory
needs grow forever?

On 2010-10-22, at 10:30 AM, Alain_Rastoul wrote:

> Another precision, I have removed all gc tuning values I used before having
> this problem, so the vm has all the default values for gc, I don't know if
> it is gc or memory related.

--
===========================================================================
John M. McIntosh <[hidden email]> Twitter: squeaker68882
Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com
===========================================================================

smime.p7s (5K) Download Attachment

Alain rastoul

Re: Re: Squeak hang at full cpu - help

Hi John,

Thanks for the tips.
My VM is a standard squeak VM (limited to 512 M, but this should be ok at
least for the first test)

I tryed :
Smalltalk setGCBiasToGrowGCLimit: 16*1024*1024.
Smalltalk setGCBiasToGrow: 1.

and my image still hang. (tryed different numbers -biggers and smaller-
without success)

I saw your posts about GC and GCMonitor and need to understand the GC logic
to ivestigate.
(unfortunatly I must stop right now :-( and can't go further today and
tomorrow).

Thanks again

"John M McIntosh" <[hidden email]> a écrit dans le message
de news: [hidden email]...
>
>

Alain rastoul

Re: Re: Squeak hang at full cpu - help

Hi,
In case it could help someone else,
I finally found that setting vmParamterAt: 24 to 256 MB seems to
have resolved my problem

"parameter 24 : memory threshold above which shrinking object memory (rw)
originally 8*1024*1024"
SmalltalkImage current vmParameterAt: 24 put: 256*1024*1024.

Alain

"Alain_Rastoul" <[hidden email]> a écrit dans le message de news:
i9stcj$5j5$[hidden email]...

> Hi John,
>
> Thanks for the tips.
> My VM is a standard squeak VM (limited to 512 M, but this should be ok at
> least for the first test)
>
> I tryed :
> Smalltalk setGCBiasToGrowGCLimit: 16*1024*1024.
> Smalltalk setGCBiasToGrow: 1.
>
> and my image still hang. (tryed different numbers -biggers and smaller-
> without success)
>
> I saw your posts about GC and GCMonitor and need to understand the GC
> logic to ivestigate.
> (unfortunatly I must stop right now :-( and can't go further today and
> tomorrow).
>
> Thanks again
>
>
> "John M McIntosh" <[hidden email]> a écrit dans le
> message de news:
> [hidden email]...
>>
>>
>
>
>
>
>