Smalltalk › Cincom › VisualWorks

[vwnc] Debugging Memory Growth

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

12 messages Options

Mike Hales

[vwnc] Debugging Memory Growth

In my application, under certain circumstances I see slow memory growth of the image until eventually it runs out of memory and crashes. Does anyone have a good set of steps they use to troubleshoot this kind of problem? I don't know where to start looking for the cruft that is building up. Are there any good tools in the public store that help with this?

Thanks,

Mike

Mike Hales
Engineering Manager
KnowledgeScape
www.kscape.com

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Boris Popov, DeepCove Labs (SNN)

Re: [vwnc] Debugging Memory Growth

Mike,

Load AT System Analysis and try,

SystemAnalyzer showSystemHistogram

You can set your app to log the results at an interval and watch the changes over time with,

systemHistogramString
"SystemAnalyzer systemHistogramString"

| hist sc sa |
ObjectMemory garbageCollect.
sa := self new.
hist := sa tallyObjectsSatisfying: [:obj | true].
sc := SortedCollection sortBlock: [:i1 :i2 | i1 totalBytes > i2 totalBytes].
sc addAll: hist.
^String streamContents: [:ws | sa reportFrom: sc on: ws].

-Boris

From: [hidden email] [mailto:[hidden email]] On Behalf Of Mike Hales
Sent: Friday, May 22, 2009 10:21 AM
To: [hidden email]
Subject: [vwnc] Debugging Memory Growth

Thanks,

Mike

Mike Hales
Engineering Manager
KnowledgeScape
www.kscape.com

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Travis Griggs-3

Re: [vwnc] Debugging Memory Growth

In reply to this post by Mike Hales

On May 22, 2009, at 10:21 AM, Mike Hales wrote:

> In my application, under certain circumstances I see slow memory
> growth of the image until eventually it runs out of memory and
> crashes. Does anyone have a good set of steps they use to
> troubleshoot this kind of problem? I don't know where to start
> looking for the cruft that is building up. Are there any good tools
> in the public store that help with this?

Are you using any external resources (DLLCC things)?

--
Travis Griggs
Objologist
"The best way to know you have a mind is to change it" -Judge Pierre
Leval

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Terry Raymond

Re: [vwnc] Debugging Memory Growth

In reply to this post by Mike Hales

Mike

Look at the CraftedSmalltalk memory policy in the goodies.

It sounds like the running the data compaction in the idle loop

will go a long way to fix the problem.

Terry

===========================================================
Terry Raymond
Crafted Smalltalk
80 Lazywood Ln.
Tiverton, RI 02878
(401) 624-4517 [hidden email]
<http://www.craftedsmalltalk.com>
===========================================================

From: [hidden email] [mailto:[hidden email]] On Behalf Of Mike Hales
Sent: Friday, May 22, 2009 1:21 PM
To: [hidden email]
Subject: [vwnc] Debugging Memory Growth

Thanks,

Mike

Mike Hales
Engineering Manager
KnowledgeScape
www.kscape.com

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Mike Hales

Re: [vwnc] Debugging Memory Growth

In reply to this post by Travis Griggs-3

Awesome, thanks for the pointers. Between the SystemAnalysis and the CraftedMemoryPolicy I should be able to sort it out. I am using DLLCC stuff, but have been very careful there to release when done, or use gcMalloc to make sure I don't hang on to things I don't need. But that did remind me of some code I should check, thanks. One thing of note for the maintainers of the AT System Analysis package, in my image (7.6) when using SystemAnalyzer class>>showSystemHistogram, it basically runs on forever, and never returns. This method uses the ObjectMemory class>>someObject and Object>>nextObject enumeration scheme. For me, this never finishes, (maybe it needs to be run at priority 100?). To fix the problem I just changed SystemAnalyzer>>tallyObjectsSatisfying: to use ObjectMemory class>>allObjectsDo: which gets all the objects in one array by going through a different primitive, which does work for me. A fileout is attached.

Mike

Mike Hales
Engineering Manager
KnowledgeScape
www.kscape.com

On Fri, May 22, 2009 at 1:08 PM, Travis Griggs <[hidden email]> wrote:

On May 22, 2009, at 10:21 AM, Mike Hales wrote:

In my application, under certain circumstances I see slow memory growth of the image until eventually it runs out of memory and crashes. Does anyone have a good set of steps they use to troubleshoot this kind of problem? I don't know where to start looking for the cruft that is building up. Are there any good tools in the public store that help with this?

Are you using any external resources (DLLCC things)?

--
Travis Griggs
Objologist
"The best way to know you have a mind is to change it" -Judge Pierre Leval

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

SystemAnalyzer-tallyObjectsSatisfying.st (1K) Download Attachment

Terry Raymond

Re: [vwnc] Debugging Memory Growth

Mike

When I read your message I assumed it meant that the memory

growth occurred while your app was running. The CraftedMemoryPolicy

will help if you do a global GC and your memory usage is reduced.

If doing global GC has no effect then the memory policy will not help

and you need to figure out what is referencing your objects.

From: [hidden email] [mailto:[hidden email]] On Behalf Of Mike Hales
Sent: Friday, May 22, 2009 4:55 PM
To: [hidden email]
Subject: Re: [vwnc] Debugging Memory Growth

Mike

Mike Hales
Engineering Manager
KnowledgeScape
www.kscape.com

On Fri, May 22, 2009 at 1:08 PM, Travis Griggs <[hidden email]> wrote:

On May 22, 2009, at 10:21 AM, Mike Hales wrote:

Are you using any external resources (DLLCC things)?

--
Travis Griggs
Objologist
"The best way to know you have a mind is to change it" -Judge Pierre Leval

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Boris Popov, DeepCove Labs (SNN)

Re: [vwnc] Debugging Memory Growth

In reply to this post by Mike Hales

Ah, yes, I have an identical override here and recall reporting this some years ago...

-Boris

--
+1.604.689.0322
DeepCove Labs Ltd.
4th floor 595 Howe Street
Vancouver, Canada V6C 2T5
http://tinyurl.com/r7uw4

[hidden email]

CONFIDENTIALITY NOTICE

This email is intended only for the persons named in the message header. Unless otherwise indicated, it contains information that is private and confidential. If you have received it in error, please notify the sender and delete the entire message including any attachments.

Thank you.

From: [hidden email] [mailto:[hidden email]] On Behalf Of Mike Hales
Sent: Friday, May 22, 2009 1:55 PM
To: [hidden email]
Subject: Re: [vwnc] Debugging Memory Growth

Mike

Mike Hales
Engineering Manager
KnowledgeScape
www.kscape.com

On Fri, May 22, 2009 at 1:08 PM, Travis Griggs <[hidden email]> wrote:

On May 22, 2009, at 10:21 AM, Mike Hales wrote:

In my application, under certain circumstances I see slow memory growth of the image until eventually it runs out of memory and crashes. Does anyone have a good set of steps they use to troubleshoot this kind of problem? I don't know where to start looking for the cruft that is building up. Are there any good tools in the public store that help with this?

Are you using any external resources (DLLCC things)?

--
Travis Griggs
Objologist
"The best way to know you have a mind is to change it" -Judge Pierre Leval

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Andres Valloud-6

Re: [vwnc] Debugging Memory Growth

In reply to this post by Mike Hales

Watch out, if there are subclasses of nil in the image, you may end up sending nextObject to a proxy. That may either not work outright, or trigger unwanted reification of other objects. For 7.7, we fixed this issue by using a mirror API to get the next object.

From: [hidden email] [mailto:[hidden email]] On Behalf Of Mike Hales
Sent: Friday, May 22, 2009 1:55 PM
To: [hidden email]
Subject: Re: [vwnc] Debugging Memory Growth

Mike

Mike Hales
Engineering Manager
KnowledgeScape
www.kscape.com

On Fri, May 22, 2009 at 1:08 PM, Travis Griggs <[hidden email]> wrote:

On May 22, 2009, at 10:21 AM, Mike Hales wrote:

In my application, under certain circumstances I see slow memory growth of the image until eventually it runs out of memory and crashes. Does anyone have a good set of steps they use to troubleshoot this kind of problem? I don't know where to start looking for the cruft that is building up. Are there any good tools in the public store that help with this?

Are you using any external resources (DLLCC things)?

--
Travis Griggs
Objologist
"The best way to know you have a mind is to change it" -Judge Pierre Leval

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

giorgiof

Re: [vwnc] Debugging Memory Growth

In reply to this post by Mike Hales

Hi,
I'm experiencing a similar situation, my memory grows from 70 mb to 500 mb (looking at the memory usage on the Windows Task Manager), and it grows almost instantaneously.
The memory is never reclaimed back, but this is what I see on on the Windows Task Manager.
If I look at the image, using both Memory Monitor and SystemAnalizer showSystemHistogram, I don't see a big difference in Smalltalk memory between the state where TaskManager show an image of around 70 Mb and an when it shous one of around 500 Mb. SystemAnalizer showSystemHistogram just show a small grow, not a tenfold one!
It's like if I'm allocating memory outside of the reaching of smalltalk itself!! (the growing seems connected to big select from DB, but the strange is, if I do select again and again, memory never grows more than that first burst of 500 mb. And this seems strange, I'm loading new data... )
I'm Using VW 7.6 with Postgres exdi on windows.

Any Idea on how trubleshooting this? Or where I missed the point?

Thanks a lot
Giorgio

On Fri, May 22, 2009 at 7:21 PM, Mike Hales <[hidden email]> wrote:

In my application, under certain circumstances I see slow memory growth of the image until eventually it runs out of memory and crashes. Does anyone have a good set of steps they use to troubleshoot this kind of problem? I don't know where to start looking for the cruft that is building up. Are there any good tools in the public store that help with this?

Thanks,

Mike

Mike Hales
Engineering Manager
KnowledgeScape
www.kscape.com

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

SystemAnalyzer showSystemHistogram
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Michael Lucas-Smith-2

Re: [vwnc] Debugging Memory Growth

Any memory allocated on the heap needs to be reclaimed with #free. Often
developers will use #gcMalloc instead of #malloc or #calloc so that
it'll reclaim itself when nothing in Smalltalk references it any more.

It sounds like you've found a memory leak somewhere, you just need to
track it down to the particular allocation that is causing the problem.

Cheers,
Michael

giorgio ferraris wrote:

> Hi,
> I'm experiencing a similar situation, my memory grows from 70 mb to
> 500 mb (looking at the memory usage on the Windows Task Manager), and
> it grows almost instantaneously.
> The memory is never reclaimed back, but this is what I see on on the
> Windows Task Manager.
> If I look at the image, using both Memory Monitor and SystemAnalizer
> showSystemHistogram, I don't see a big difference in Smalltalk memory
> between the state where TaskManager show an image of around 70 Mb and
> an when it shous one of around 500 Mb. SystemAnalizer
> showSystemHistogram just show a small grow, not a tenfold one!
> It's like if I'm allocating memory outside of the reaching of
> smalltalk itself!! (the growing seems connected to big select from DB,
> but the strange is, if I do select again and again, memory never grows
> more than that first burst of 500 mb. And this seems strange, I'm
> loading new data... )
> I'm Using VW 7.6 with Postgres exdi on windows.
>
>
> Any Idea on how trubleshooting this? Or where I missed the point?
>
> Thanks a lot
> Giorgio
>
>
> On Fri, May 22, 2009 at 7:21 PM, Mike Hales <[hidden email]
> <mailto:[hidden email]>> wrote:
>
> In my application, under certain circumstances I see slow memory
> growth of the image until eventually it runs out of memory and
> crashes. Does anyone have a good set of steps they use to
> troubleshoot this kind of problem? I don't know where to start
> looking for the cruft that is building up. Are there any good
> tools in the public store that help with this?
>
> Thanks,
>
> Mike
>
> Mike Hales
> Engineering Manager
> KnowledgeScape
> www.kscape.com <http://www.kscape.com>
>
> _______________________________________________
> vwnc mailing list
> [hidden email] <mailto:[hidden email]>
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>
> SystemAnalyzer showSystemHistogram
> ------------------------------------------------------------------------
>
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Karsten Kusche

Re: [vwnc] Debugging Memory Growth

In reply to this post by giorgiof

Hi Giorgio,

what i've also found helpful in this area is the Class Report from Advanced Tools. Sometimes you might just reference a lot of objects in your image so that they don't get garbage collected.

Simply load All Advanced Tools from the Parcel Manager and go to Tools->Advanced->Class reports in the Launcher. Then right-click on the list of classes and select Add All. On the right side of the window select space and run a class report on the instance size. This will list all the objects that are present in your image, maybe this helps you find a clue which objects are still in your image.

Kind Regards
Karsten

giorgio ferraris wrote:

Hi,
I'm experiencing a similar situation, my memory grows from 70 mb to 500 mb (looking at the memory usage on the Windows Task Manager), and it grows almost instantaneously.
The memory is never reclaimed back, but this is what I see on on the Windows Task Manager.
If I look at the image, using both Memory Monitor and SystemAnalizer showSystemHistogram, I don't see a big difference in Smalltalk memory between the state where TaskManager show an image of around 70 Mb and an when it shous one of around 500 Mb. SystemAnalizer showSystemHistogram just show a small grow, not a tenfold one!
It's like if I'm allocating memory outside of the reaching of smalltalk itself!! (the growing seems connected to big select from DB, but the strange is, if I do select again and again, memory never grows more than that first burst of 500 mb. And this seems strange, I'm loading new data... )
I'm Using VW 7.6 with Postgres exdi on windows.

Any Idea on how trubleshooting this? Or where I missed the point?

Thanks a lot
Giorgio

On Fri, May 22, 2009 at 7:21 PM, Mike Hales <[hidden email]> wrote:

In my application, under certain circumstances I see slow memory growth of the image until eventually it runs out of memory and crashes. Does anyone have a good set of steps they use to troubleshoot this kind of problem? I don't know where to start looking for the cruft that is building up. Are there any good tools in the public store that help with this?

Thanks,

Mike

Mike Hales
Engineering Manager
KnowledgeScape
www.kscape.com

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

SystemAnalyzer showSystemHistogram
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
  

--
Karsten Kusche - Dipl.Inf. - [hidden email]
Tel: +49 3496 21 43 29
Georg Heeg eK - Köthen
Handelsregister: Amtsgericht Dortmund A 12812

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

giorgiof

Re: [vwnc] Debugging Memory Growth

Hi,
thanks to all for the suggestions.
I try to see if I can find some light.
Ciao
Giorgio

On Tue, Jun 16, 2009 at 9:26 AM, Karsten <[hidden email]> wrote:

Hi Giorgio,

what i've also found helpful in this area is the Class Report from Advanced Tools. Sometimes you might just reference a lot of objects in your image so that they don't get garbage collected.

Simply load All Advanced Tools from the Parcel Manager and go to Tools->Advanced->Class reports in the Launcher. Then right-click on the list of classes and select Add All. On the right side of the window select space and run a class report on the instance size. This will list all the objects that are present in your image, maybe this helps you find a clue which objects are still in your image.

Kind Regards
Karsten

giorgio ferraris wrote:
Hi,
I'm experiencing a similar situation, my memory grows from 70 mb to 500 mb (looking at the memory usage on the Windows Task Manager), and it grows almost instantaneously.
The memory is never reclaimed back, but this is what I see on on the Windows Task Manager.
If I look at the image, using both Memory Monitor and SystemAnalizer showSystemHistogram, I don't see a big difference in Smalltalk memory between the state where TaskManager show an image of around 70 Mb and an when it shous one of around 500 Mb. SystemAnalizer showSystemHistogram just show a small grow, not a tenfold one!
It's like if I'm allocating memory outside of the reaching of smalltalk itself!! (the growing seems connected to big select from DB, but the strange is, if I do select again and again, memory never grows more than that first burst of 500 mb. And this seems strange, I'm loading new data... )
I'm Using VW 7.6 with Postgres exdi on windows.

Any Idea on how trubleshooting this? Or where I missed the point?

Thanks a lot
Giorgio

On Fri, May 22, 2009 at 7:21 PM, Mike Hales <[hidden email]> wrote:

In my application, under certain circumstances I see slow memory growth of the image until eventually it runs out of memory and crashes. Does anyone have a good set of steps they use to troubleshoot this kind of problem? I don't know where to start looking for the cruft that is building up. Are there any good tools in the public store that help with this?

Thanks,

Mike

Mike Hales
Engineering Manager
KnowledgeScape
www.kscape.com

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

SystemAnalyzer showSystemHistogram
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
  
--
Karsten Kusche - Dipl.Inf. - [hidden email]
Tel: +49 3496 21 43 29
Georg Heeg eK - Köthen
Handelsregister: Amtsgericht Dortmund A 12812

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc