Smalltalk › Cincom › VisualWorks

Detect cause of "No Space Left."?

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

26 messages Options

Steve Cline

Detect cause of "No Space Left."?

I am debugging an error and am stuck. I can recreate it in more than half the attempts. Two client processes send the same command to an image and cause the Process Monitor to open up, with the title "Process Monitor Emergency: No Space Left.". If I select Monitor / Proceed all and Close from the menu the tasks complete successfully.

Is there a way to tell what is taking up the space, or intercept the opening of the process monitor to try and detect what is causing the condition? I can make the error go away by bumping up the memoryUpperBound value, but I want to know who the guilty parties are.

Thanks, Steve

Andres Valloud-6

Re: Detect cause of "No Space Left."?

Which version?

On 1/6/2012 3:13 PM, Steve Cline wrote:

> I am debugging an error and am stuck. I can recreate it in more than half
> the attempts. Two client processes send the same command to an image and
> cause the Process Monitor to open up, with the title "Process Monitor
> Emergency: No Space Left.". If I select Monitor / Proceed all and Close
> from the menu the tasks complete successfully.
>
> Is there a way to tell what is taking up the space, or intercept the opening
> of the process monitor to try and detect what is causing the condition? I
> can make the error go away by bumping up the memoryUpperBound value, but I
> want to know who the guilty parties are.
>
> Thanks, Steve
>
> --
> View this message in context: http://forum.world.st/Detect-cause-of-No-Space-Left-tp4271258p4271258.html
> Sent from the VisualWorks mailing list archive at Nabble.com.
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Steve Cline

Re: Detect cause of "No Space Left."?

Sorry, VW7.8 running on Windows Server 2008 R2 Service Pack 1, 64-bit

Andres Valloud-6

Re: Detect cause of "No Space Left."?

Ok, I'm assuming you're running 32 bit VW on that machine. That dialog
comes up when the memory policy thinks the image is at risk of causing
the VM to shut it down due to lack of memory. Here's a lengthy
explanation that certainly doesn't fit on the margin of this email:

http://www.youtube.com/watch?v=d4p1d1PprvA

Given that, the task is to figure out why the memory policy brought up
the emergency notifier. How much memory is used when the dialog comes
up (lower right corner)? What's the memory upper bound? Did you
reconfigure the memory policy in any way? If you close and proceed
(without doing a GC), how much memory is used by the image? And how
much memory is used after doing a GC?

On 1/7/2012 5:11 AM, Steve Cline wrote:

> Sorry, VW7.8 running on Windows Server 2008 R2 Service Pack 1, 64-bit
>
> --
> View this message in context: http://forum.world.st/Detect-cause-of-No-Space-Left-tp4271258p4272873.html
> Sent from the VisualWorks mailing list archive at Nabble.com.
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Steve Cline

Re: Detect cause of "No Space Left."?

It is a 64 bit environment, but that should not be the most relevant factor - some experimentation with bringing up the process monitor and looking at the memory size shows that each time I run a particular operation with a different user I grow the image by roughly 150 MB. A manual GC gets rid of most of that, residual from the calculations that are made over a large network of nodes. What I am hoping for are commands to either "instance: tell me the size and count of you and all your instance variables recursively" or "class: give me the number of your instances and the amount of space they take up", or something similar. It is complicated somewhat in that the image is a server and can be handling multiple requests simultaneously. I suspect but would like to be able to prove that I am creating the same objects over and over.

Mark Roberts

Re: Detect cause of "No Space Left."?

What kind of server are you running?

If, by chance, it's the Web Toolkit, there are settings for startup
memory sizes and low memory actions (open the Settings from the Server
Console). These parameters are discussed in the "Memory Configuration"
section of the Web Server Config Guide (PDF in the /doc directory).

Regards,

M. Roberts
Cincom Systems, Inc.

On 1/10/2012 10:58 PM, Steve Cline wrote:

> It is a 64 bit environment, but that should not be the most relevant factor -
> some experimentation with bringing up the process monitor and looking at the
> memory size shows that each time I run a particular operation with a
> different user I grow the image by roughly 150 MB. A manual GC gets rid of
> most of that, residual from the calculations that are made over a large
> network of nodes. What I am hoping for are commands to either "instance:
> tell me the size and count of you and all your instance variables
> recursively" or "class: give me the number of your instances and the amount
> of space they take up", or something similar. It is complicated somewhat in
> that the image is a server and can be handling multiple requests
> simultaneously. I suspect but would like to be able to prove that I am
> creating the same objects over and over.
>
> --
> View this message in context: http://forum.world.st/Detect-cause-of-No-Space-Left-tp4271258p4282024.html
> Sent from the VisualWorks mailing list archive at Nabble.com.
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Niall Ross

Re: Detect cause of "No Space Left."?

In reply to this post by Steve Cline

Dear Steve,

>What I am hoping for are commands to either "instance:
>tell me the size and count of you and all your instance variables
>recursively" or "class: give me the number of your instances and the amount
>of space they take up", or something similar.
>
>
Is the 'Space' tab of the ClassReporter the kind of thing you are
thinking about (in AT System Analysis - manual is last chapter of
doc/ToolGuide.pdf)? Set the filter to show classes you're interested
in, select all or some of them in the pane below, then choose 'Instance
size' on the Space tab and Run.

HTH
Niall Ross

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Steve Cline

Re: Detect cause of "No Space Left."?

Thanks Niall -

I would call that "a great start" :-).

What I am wishing for is a method that I could send to a collection that would produce something like the "Space"/"Instance size" report for the objects in the collection - I have so many instances of these classes and they take up this much space. Even just the "this much space" would be useful.

Andres Valloud-4

Re: Detect cause of "No Space Left."?

In reply to this post by Steve Cline

I need to know if it is a 64 bit image or a 32 bit image running on that
64 bit OS. Also, see the email I just sent out regarding the memory
monitor, I need to see that CSV log to see what's going on with your
memory emergency. Finally, did you change the memory policy or
sizesAtStartup?

On 1/10/12 5:58 , Steve Cline wrote:

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Steve Cline

Re: Detect cause of "No Space Left."?

Sorry, misunderstood the question - it is a 64-bit image running on a 64-bit environment. If I increase the memoryUpperBound to 1500 MB, I see that 3 simultaneous requests force memory up to 1061 MB, well over the normal operating value of 750MB, explaining *why* I get the no space left error.

What I am hoping for is something I can use on one of the requests to get a quick report of what it generated, or that I can send to objects (in my case, collections of transitory objects) and see how big they are. Right now I can get a rough idea by bossing them out and looking at the file size, but was hoping for something more elegant, finer grained, and with less leavings that need to be cleaned up :-)

Steve Cline

Re: Detect cause of "No Space Left."?

To be clear - in the last paragraph I mean "in the debugger", though a log could be acceptable

Andres Valloud-4

Re: Detect cause of "No Space Left."?

In reply to this post by Steve Cline

Here's a way to track down the problem. Load "All Advanced Tools",
start the multi allocation profiler on the server like this

MultiAllocationProfiler startProfiling

then run the three requests, then

MultiAllocationProfiler stopProfiling

and look at the results. You can use the [MAP] button on the
MemoryMonitor tool to start / stop the multi allocation profiler
(assuming All Advanced Tools is loaded).

On 1/11/12 9:07 , Steve Cline wrote:

> Sorry, misunderstood the question - it is a 64-bit image running on a 64-bit
> environment. If I increase the memoryUpperBound to 1500 MB, I see that 3
> simultaneous requests force memory up to 1061 MB, well over the normal
> operating value of 750MB, explaining *why* I get the no space left error.
>
> What I am hoping for is something I can use on one of the requests to get a
> quick report of what it generated, or that I can send to objects (in my
> case, collections of transitory objects) and see how big they are. Right
> now I can get a rough idea by bossing them out and looking at the file size,
> but was hoping for something more elegant, finer grained, and with less
> leavings that need to be cleaned up :-)
>
> --
> View this message in context: http://forum.world.st/Detect-cause-of-No-Space-Left-tp4271258p4286115.html
> Sent from the VisualWorks mailing list archive at Nabble.com.
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Steve Cline

Re: Detect cause of "No Space Left."?

I tried the MultiAllocationProfiler but it was not as helpful as I hoped. Here is the result:

It seems to tell me what methods were excercised that created new objects, but nothing about the class or count of the objects created, or their relationships. Am I misunderstanding the usefulness of this tool?

Also, there is no MAP button on the Memory Monitor:

(hope images uploaded - I can not see them)

Thanks, Steve

Andres Valloud-6

Re: Detect cause of "No Space Left."?

Wow... over 500mb of allocations, and 25% of that is sets?

Also, load "MemoryMonitor", not "Memory Monitor".

On 1/24/2012 12:30 PM, Steve Cline wrote:

> I tried the MultiAllocationProfiler but it was not as helpful as I hoped.
> Here is the result:
>
> http://forum.world.st/file/n4325129/Allocation1-24-2012.png
>
> It seems to tell me what methods were excercised that created new objects,
> but nothing about the class or count of the objects created, or their
> relationships. Am I misunderstanding the usefulness of this tool?
>
> Also, there is no MAP button on the Memory Monitor:
>
> http://forum.world.st/file/n4325129/MM_1-24-2012.png
>
> (hope images uploaded - I can not see them)
>
> Thanks, Steve
>
> --
> View this message in context: http://forum.world.st/Detect-cause-of-No-Space-Left-tp4271258p4325129.html
> Sent from the VisualWorks mailing list archive at Nabble.com.
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Andres Valloud-6

Re: Detect cause of "No Space Left."?

In the No Space Left notifier, you will see a list processes. I'd look
around and see which process ended up causing the emergency. Is it
because of a huge allocation that fails (e.g.: Set new: largeValue)?
What about the low space process, how did it think there was no space left?

Oh, another thing... if you have a lot of rather large objects, you may
get into address space fragmentation... I'd like to believe it shouldn't
happen in 64 bit environments, but I'd look into the sizes of active old
space segments and the free space in them. Perhaps the VM is denied
memory growth for some reason, yet it can't find enough contiguous space
for a given allocation.

On 1/24/2012 12:39 PM, Andres Valloud wrote:

> Wow... over 500mb of allocations, and 25% of that is sets?
>
> Also, load "MemoryMonitor", not "Memory Monitor".
>
> On 1/24/2012 12:30 PM, Steve Cline wrote:
>> I tried the MultiAllocationProfiler but it was not as helpful as I hoped.
>> Here is the result:
>>
>> http://forum.world.st/file/n4325129/Allocation1-24-2012.png
>>
>> It seems to tell me what methods were excercised that created new objects,
>> but nothing about the class or count of the objects created, or their
>> relationships. Am I misunderstanding the usefulness of this tool?
>>
>> Also, there is no MAP button on the Memory Monitor:
>>
>> http://forum.world.st/file/n4325129/MM_1-24-2012.png
>>
>> (hope images uploaded - I can not see them)
>>
>> Thanks, Steve
>>
>> --
>> View this message in context: http://forum.world.st/Detect-cause-of-No-Space-Left-tp4271258p4325129.html
>> Sent from the VisualWorks mailing list archive at Nabble.com.
>> _______________________________________________
>> vwnc mailing list
>> [hidden email]
>> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>>
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Andres Valloud-6

Re: Detect cause of "No Space Left."?

If you suspect fragmentation, go to targetPreferredGrowthIncrement in
LargeGrainMemoryPolicy and replace the divisor of 24 with e.g.: 12. Do
you still see the problem?

On 1/24/2012 12:44 PM, Andres Valloud wrote:

> In the No Space Left notifier, you will see a list processes. I'd look
> around and see which process ended up causing the emergency. Is it
> because of a huge allocation that fails (e.g.: Set new: largeValue)?
> What about the low space process, how did it think there was no space left?
>
> Oh, another thing... if you have a lot of rather large objects, you may
> get into address space fragmentation... I'd like to believe it shouldn't
> happen in 64 bit environments, but I'd look into the sizes of active old
> space segments and the free space in them. Perhaps the VM is denied
> memory growth for some reason, yet it can't find enough contiguous space
> for a given allocation.
>
> On 1/24/2012 12:39 PM, Andres Valloud wrote:
>> Wow... over 500mb of allocations, and 25% of that is sets?
>>
>> Also, load "MemoryMonitor", not "Memory Monitor".
>>
>> On 1/24/2012 12:30 PM, Steve Cline wrote:
>>> I tried the MultiAllocationProfiler but it was not as helpful as I hoped.
>>> Here is the result:
>>>
>>> http://forum.world.st/file/n4325129/Allocation1-24-2012.png
>>>
>>> It seems to tell me what methods were excercised that created new objects,
>>> but nothing about the class or count of the objects created, or their
>>> relationships. Am I misunderstanding the usefulness of this tool?
>>>
>>> Also, there is no MAP button on the Memory Monitor:
>>>
>>> http://forum.world.st/file/n4325129/MM_1-24-2012.png
>>>
>>> (hope images uploaded - I can not see them)
>>>
>>> Thanks, Steve
>>>
>>> --
>>> View this message in context: http://forum.world.st/Detect-cause-of-No-Space-Left-tp4271258p4325129.html
>>> Sent from the VisualWorks mailing list archive at Nabble.com.
>>> _______________________________________________
>>> vwnc mailing list
>>> [hidden email]
>>> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>>>
>> _______________________________________________
>> vwnc mailing list
>> [hidden email]
>> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>>
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Steve Cline

Re: Detect cause of "No Space Left."?

In reply to this post by Andres Valloud-6

If I swipe 'MemoryMonitor open', I get the window I posted earlier - is there some other way to get a MemoryMonitor that does not call itself 'Memory Monitor' in the tool menu or it's header bar?

Steve Cline

Re: Detect cause of "No Space Left."?

In reply to this post by Andres Valloud-6

As I mentioned ealier, I accept that the No Space error happens because I am making too much stuff. What I want is something that would tell me, for example, that between the start and stop of monitoring I created (X instances of class A, with a size of N), and (Y instances of class B, with a size of M), and so on.

I am not as interested in 'this is the creation method that failed because it is the one that pushed me over some arbitrary boundary', but rather 'here is the breakdown of the stuff that gets created when you execute this code'. I suspect, but would like to prove, that my problem could be addressed by avoiding recreating the same objects (caching is an obvious solution, but there may be others - but I really want to know what objects get created)

Randy Coulman

Re: Detect cause of "No Space Left."?

I've had good luck with the Class Report (part of Advanced Tools, I believe). I run a class report, copy and paste the results to a text file, then run the suspect code, then run another class report. I then sort the two reports by class name and run a diff on the two reports.

Randy

On Tue, Jan 24, 2012 at 2:38 PM, Steve Cline <[hidden email]> wrote:

As I mentioned ealier, I accept that the No Space error happens because I am
making too much stuff. What I want is something that would tell me, for
example, that between the start and stop of monitoring I created (X
instances of class A, with a size of N), and (Y instances of class B, with a
size of M), and so on.

I am not as interested in 'this is the creation method that failed because
it is the one that pushed me over some arbitrary boundary', but rather 'here
is the breakdown of the stuff that gets created when you execute this code'.
I suspect, but would like to prove, that my problem could be addressed by
avoiding recreating the same objects (caching is an obvious solution, but
there may be others - but I really want to know what objects get created)

--
View this message in context: http://forum.world.st/Detect-cause-of-No-Space-Left-tp4271258p4325588.html
Sent from the VisualWorks mailing list archive at Nabble.com.
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

--
Randy Coulman
[hidden email]

Twitter: @randycoulman

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Steve Cline

Re: Detect cause of "No Space Left."?

Randy, Thanks for the tip! This looks like what I am looking for. It would be nice if I could get output as a csv file (never satisfied, I know!), but this seems to give me the data I need