What are we leaking in the www.squeak.org process?
Locked
What are we leaking in the www.squeak.org process?
 
|
For some time now I've been bugging Janko about the memory usage of the
process that serves http://www.squeak.org/ . I've gathered a bit more info today and I'd like to see if anyone here has any thoughts. First while there is some variation most of the Squeak processes that serve squeak.org sites (source.squeak.org, map.squeak.org, wiki.squeak.org) have a more or less constant and reasonable memory usage. Let me clarify that what I am talking about here is RSS (Resident Set Size) as reported by ps/top on Linux. Swiki as the oldest of the group is the clear winner at just under 30MB, squeakmap is around 47MB, and source.squeak.org is around 59MB. In contrast immediately after starting it www.squeak.org is using about 70MB and it grows from there. Now a bit more than an hour since the last restart it is already up to about 85MB. I have seen it over 300MB before, and it commonly uses 140-250MB. It doesn't simply grow and grow, it does shrink occasionally. First, there is a known issue with Swazoo that it does not close sockets correctly and Janko has addressed this by having Swazoo automatically restarted every 15 minutes. So using netstat for example I can watch the socket count for the process continually grow and then it drops to zero and this repeats. In contrast for all the other process (all running some variation of Kom/Comanche I believe) I'm hard pressed to even catch a single open socket at any time (not counting the listener that is). Secondly, since I've been complaining about this issue for a long time Janko has running a little morph that shows current memory usage. What this shows is the values of SmalltalkImage current vmParameterAt: 2 and 3. These numbers are quite stable. For example at this moment these are 68.3MB and 71.0MB respectively. Whereas ps shows the RSS at 85752KB. Where is the discrepancy here? My current theory is that the vmParameters are not reflecting incidental memory use by the VM and plugins or that we are simply misinterpreting the values. Is it possible that Swazoo is failing to release something that is causing the VM to use more memory than it should? Note that the same VM is used by all of these processes. It is 3.8a-1 #1 Sun May 1 19:46:46 EDT 2005 gcc 3.3.5 Yes, I know it's old, but 'if it ain't broke, don't fix it' has been the motto and with the possible exception of Swazoo/www.squeak.org it has been working wonderfully for a long time now. Your thoughts would be greatly appreciated, Ken |
signature.asc (197 bytes)
Re: What are we leaking in the www.squeak.org process?
|
|
An update as things may not be quite as I originally described them. It
seems the values reported by vmParameters 2 and 3 are not as constant as I had been lead to believe. I noticed that just a short while after sending the image the RSS had jumped up to 165800KB. I still had VNC open on the image so I checked what is shown there and I see 145.6MB/147.7MB so there is still a discrepancy, but not perhaps as wide as I indicated. I also did not mention that in addition to restarting Swazoo every 15 minutes there is other cleanup that is done once a day related to statistics gathering that goes on. My understanding is that this cleanup is meant to return things very close to the starting state. Perhaps Janko can comment on that more fully. Ken On Thu, 2010-10-14 at 16:55 -0500, Ken Causey wrote: > For some time now I've been bugging Janko about the memory usage of the > process that serves http://www.squeak.org/ . I've gathered a bit more > info today and I'd like to see if anyone here has any thoughts. > > First while there is some variation most of the Squeak processes that > serve squeak.org sites (source.squeak.org, map.squeak.org, > wiki.squeak.org) have a more or less constant and reasonable memory > usage. Let me clarify that what I am talking about here is RSS > (Resident Set Size) as reported by ps/top on Linux. Swiki as the oldest > of the group is the clear winner at just under 30MB, squeakmap is around > 47MB, and source.squeak.org is around 59MB. > > In contrast immediately after starting it www.squeak.org is using about > 70MB and it grows from there. Now a bit more than an hour since the > last restart it is already up to about 85MB. I have seen it over 300MB > before, and it commonly uses 140-250MB. It doesn't simply grow and > grow, it does shrink occasionally. > > First, there is a known issue with Swazoo that it does not close sockets > correctly and Janko has addressed this by having Swazoo automatically > restarted every 15 minutes. So using netstat for example I can watch > the socket count for the process continually grow and then it drops to > zero and this repeats. In contrast for all the other process (all > running some variation of Kom/Comanche I believe) I'm hard pressed to > even catch a single open socket at any time (not counting the listener > that is). > > Secondly, since I've been complaining about this issue for a long time > Janko has running a little morph that shows current memory usage. What > this shows is the values of SmalltalkImage current vmParameterAt: 2 and > 3. These numbers are quite stable. For example at this moment these > are 68.3MB and 71.0MB respectively. Whereas ps shows the RSS at > 85752KB. Where is the discrepancy here? > > My current theory is that the vmParameters are not reflecting incidental > memory use by the VM and plugins or that we are simply misinterpreting > the values. Is it possible that Swazoo is failing to release something > that is causing the VM to use more memory than it should? > > Note that the same VM is used by all of these processes. It is > > 3.8a-1 #1 Sun May 1 19:46:46 EDT 2005 gcc 3.3.5 > > Yes, I know it's old, but 'if it ain't broke, don't fix it' has been the > motto and with the possible exception of Swazoo/www.squeak.org it has > been working wonderfully for a long time now. > > Your thoughts would be greatly appreciated, > > Ken |
Re: [Webteam] Re: [squeak-dev] What are we leaking in the www.squeak.org process?
|
|
Another update with more data conflicting somewhat with my last report:
Shown by ps: 214560KB RSS Shown by vmParameters (2,3): 64.6MB/67.4MB A discrepancy of some 150MB. Ken On Thu, 2010-10-14 at 17:31 -0500, Ken Causey wrote: > An update as things may not be quite as I originally described them. It > seems the values reported by vmParameters 2 and 3 are not as constant as > I had been lead to believe. I noticed that just a short while after > sending the image the RSS had jumped up to 165800KB. I still had VNC > open on the image so I checked what is shown there and I see > 145.6MB/147.7MB so there is still a discrepancy, but not perhaps as wide > as I indicated. > > I also did not mention that in addition to restarting Swazoo every 15 > minutes there is other cleanup that is done once a day related to > statistics gathering that goes on. My understanding is that this > cleanup is meant to return things very close to the starting state. > Perhaps Janko can comment on that more fully. > > Ken > > On Thu, 2010-10-14 at 16:55 -0500, Ken Causey wrote: > > For some time now I've been bugging Janko about the memory usage of the > > process that serves http://www.squeak.org/ . I've gathered a bit more > > info today and I'd like to see if anyone here has any thoughts. > > > > First while there is some variation most of the Squeak processes that > > serve squeak.org sites (source.squeak.org, map.squeak.org, > > wiki.squeak.org) have a more or less constant and reasonable memory > > usage. Let me clarify that what I am talking about here is RSS > > (Resident Set Size) as reported by ps/top on Linux. Swiki as the oldest > > of the group is the clear winner at just under 30MB, squeakmap is around > > 47MB, and source.squeak.org is around 59MB. > > > > In contrast immediately after starting it www.squeak.org is using about > > 70MB and it grows from there. Now a bit more than an hour since the > > last restart it is already up to about 85MB. I have seen it over 300MB > > before, and it commonly uses 140-250MB. It doesn't simply grow and > > grow, it does shrink occasionally. > > > > First, there is a known issue with Swazoo that it does not close sockets > > correctly and Janko has addressed this by having Swazoo automatically > > restarted every 15 minutes. So using netstat for example I can watch > > the socket count for the process continually grow and then it drops to > > zero and this repeats. In contrast for all the other process (all > > running some variation of Kom/Comanche I believe) I'm hard pressed to > > even catch a single open socket at any time (not counting the listener > > that is). > > > > Secondly, since I've been complaining about this issue for a long time > > Janko has running a little morph that shows current memory usage. What > > this shows is the values of SmalltalkImage current vmParameterAt: 2 and > > 3. These numbers are quite stable. For example at this moment these > > are 68.3MB and 71.0MB respectively. Whereas ps shows the RSS at > > 85752KB. Where is the discrepancy here? > > > > My current theory is that the vmParameters are not reflecting incidental > > memory use by the VM and plugins or that we are simply misinterpreting > > the values. Is it possible that Swazoo is failing to release something > > that is causing the VM to use more memory than it should? > > > > Note that the same VM is used by all of these processes. It is > > > > 3.8a-1 #1 Sun May 1 19:46:46 EDT 2005 gcc 3.3.5 > > > > Yes, I know it's old, but 'if it ain't broke, don't fix it' has been the > > motto and with the possible exception of Swazoo/www.squeak.org it has > > been working wonderfully for a long time now. > > > > Your thoughts would be greatly appreciated, > > > > Ken > > _______________________________________________ > Webteam mailing list > [hidden email] > http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/webteam |
Re: [Webteam] Re: [squeak-dev] What are we leaking in the www.squeak.org process?
|
|
RE: [Webteam] Re: [squeak-dev] What are we leaking in the www.squeak.org process?
|
|
RE: [Webteam] Re: [squeak-dev] What are we leaking in the www.squeak.org process?
|
|
On Thu, 2010-10-14 at 19:00 -0700, Ken Causey wrote:
> > -------- Original Message -------- > > Subject: Re: [Webteam] Re: [squeak-dev] What are we leaking in the > > www.squeak.org process? > > From: Igor Stasenko <[hidden email]> > > Date: Thu, October 14, 2010 8:45 pm > > To: The general-purpose Squeak developers list > > <[hidden email]> > > > > > > On 15 October 2010 04:10, Ken Causey <[hidden email]> wrote: > > > Another update with more data conflicting somewhat with my last report: > > > > > > Shown by ps: 214560KB RSS > > > Shown by vmParameters (2,3): 64.6MB/67.4MB > > > > > > A discrepancy of some 150MB. > > > > My understanding is, that Squeak VM cannot shrink a memory which were > > commited (due to peak growth), > > to give it back to OS. > > > > So, if at some peak, VM comitted 150 MB, and OS sees it as 150M, > > while VM after GC actually using just 64M. > > No, I don't believe this is true. The RSS does go both down as well as > up. I thought I made that clear in my original comments. > > Although I suppose it is possible that drops in the RSS figure are due > to paging out to swap. I'm not sure how to really check that. > > Ken Shown by ps: 167344KB RSS Shown by vmParameters (2,3): 62.9MB/65.8MB The process has not been restarted but the RSS has dropped. Ken > > > > > > > > Ken > > > > > > On Thu, 2010-10-14 at 17:31 -0500, Ken Causey wrote: > > >> An update as things may not be quite as I originally described them. It > > >> seems the values reported by vmParameters 2 and 3 are not as constant as > > >> I had been lead to believe. I noticed that just a short while after > > >> sending the image the RSS had jumped up to 165800KB. I still had VNC > > >> open on the image so I checked what is shown there and I see > > >> 145.6MB/147.7MB so there is still a discrepancy, but not perhaps as wide > > >> as I indicated. > > >> > > >> I also did not mention that in addition to restarting Swazoo every 15 > > >> minutes there is other cleanup that is done once a day related to > > >> statistics gathering that goes on. My understanding is that this > > >> cleanup is meant to return things very close to the starting state. > > >> Perhaps Janko can comment on that more fully. > > >> > > >> Ken > > >> > > >> On Thu, 2010-10-14 at 16:55 -0500, Ken Causey wrote: > > >> > For some time now I've been bugging Janko about the memory usage of the > > >> > process that serves http://www.squeak.org/ . I've gathered a bit more > > >> > info today and I'd like to see if anyone here has any thoughts. > > >> > > > >> > First while there is some variation most of the Squeak processes that > > >> > serve squeak.org sites (source.squeak.org, map.squeak.org, > > >> > wiki.squeak.org) have a more or less constant and reasonable memory > > >> > usage. Let me clarify that what I am talking about here is RSS > > >> > (Resident Set Size) as reported by ps/top on Linux. Swiki as the oldest > > >> > of the group is the clear winner at just under 30MB, squeakmap is around > > >> > 47MB, and source.squeak.org is around 59MB. > > >> > > > >> > In contrast immediately after starting it www.squeak.org is using about > > >> > 70MB and it grows from there. Now a bit more than an hour since the > > >> > last restart it is already up to about 85MB. I have seen it over 300MB > > >> > before, and it commonly uses 140-250MB. It doesn't simply grow and > > >> > grow, it does shrink occasionally. > > >> > > > >> > First, there is a known issue with Swazoo that it does not close sockets > > >> > correctly and Janko has addressed this by having Swazoo automatically > > >> > restarted every 15 minutes. So using netstat for example I can watch > > >> > the socket count for the process continually grow and then it drops to > > >> > zero and this repeats. In contrast for all the other process (all > > >> > running some variation of Kom/Comanche I believe) I'm hard pressed to > > >> > even catch a single open socket at any time (not counting the listener > > >> > that is). > > >> > > > >> > Secondly, since I've been complaining about this issue for a long time > > >> > Janko has running a little morph that shows current memory usage. What > > >> > this shows is the values of SmalltalkImage current vmParameterAt: 2 and > > >> > 3. These numbers are quite stable. For example at this moment these > > >> > are 68.3MB and 71.0MB respectively. Whereas ps shows the RSS at > > >> > 85752KB. Where is the discrepancy here? > > >> > > > >> > My current theory is that the vmParameters are not reflecting incidental > > >> > memory use by the VM and plugins or that we are simply misinterpreting > > >> > the values. Is it possible that Swazoo is failing to release something > > >> > that is causing the VM to use more memory than it should? > > >> > > > >> > Note that the same VM is used by all of these processes. It is > > >> > > > >> > 3.8a-1 #1 Sun May 1 19:46:46 EDT 2005 gcc 3.3.5 > > >> > > > >> > Yes, I know it's old, but 'if it ain't broke, don't fix it' has been the > > >> > motto and with the possible exception of Swazoo/www.squeak.org it has > > >> > been working wonderfully for a long time now. > > >> > > > >> > Your thoughts would be greatly appreciated, > > >> > > > >> > Ken > > >> > > >> _______________________________________________ > > >> Webteam mailing list > > >> [hidden email] > > >> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/webteam > > > > > > > > > > > > > > > > > > > > > > > -- > > Best regards, > > Igor Stasenko AKA sig. > > > |
Re: vsz instead of rss maybe? (was: What are we leaking in the www.squeak.org process?)
|
|
On Fri, 2010-10-15 at 16:20 -0700, Tom Rushworth wrote:
> Hi Ken, > > I'm no "ps" expert, but isn't the resident set size the wrong thing to > be looking at? My understanding is that rss is the amount of physical > (real) memory currently paged in (resident) for a process, and that > will go up and down as the system load changes. If I understand > correctly, you should be looking at something like vsize. I'm also > not running on Linux at the moment, but if I do a "man ps" on my Mac, > it gives me the command option for adding fields to the output (in > this case "-O") and a list of keywords (with short description) for > things to add. That would mean the the right number to look at would > be in the "vsz" column of the command "ps -O vsize". The linux > equivalent must surely exist for ps, and possibly for top as well. > > Mind you, the VSZ numbers on my machine don't make a lot of sense to > me, but they might on your linux box. (The Mac man page says the VSZ > is in Kbytes, but the numbers I actually get are ridiculous: > > @ ps -O vsize > PID VSZ TT STAT TIME COMMAND > 873 2435468 s000 S 0:00.03 -bash > 1845 600772 s000 S+ 0:01.53 /usr/local/bin/vile Shared/main.m > ..... > I'm not too sure I believe 600Mbytes for a simple vi-like edit of a 20 > line file:).) often request more memory than they actually use and Squeak is a prime example here as, by default on Linux, it requests (mmaps I think) a full gigabyte of RAM, which is more or less what you always see for VSZ for squeak process, that doesn't mean it is actually using this, just that it has requested a map with theoretically that much space. The VSZ value gives no useful information for my purposes. I agree that RSS is not ideal either, but it's closer than VSZ and I can clearly see that as RSS goes up the free space reported by free goes down. So in some sense it is a a relevant figure to me since we tend to have very little free RAM these days on the server and we are well into swap. Ken |
Re: vsz instead of rss maybe? (was: What are we leaking in the www.squeak.org process?)
|
|
Re: vsz instead of rss maybe? (was: What are we leaking in the www.squeak.org process?)
|
|
On Fri, 2010-10-15 at 17:21 -0700, Tom Rushworth wrote:
> I see. Perhaps it's time to dig into the VM and see if you can't make > the GC cough up info about how much space it thinks the VM is really > using. Not nearly as easy as a "ps" though... Which is what SmalltalkImage (as of 3.9.1-7075, the www.squeak.org image, the class name may vary)>>vmParameterAt: seems to provide, quote: ...[the argument] is a positive integer corresponding to one of the VM's internal parameter/metric registers. Answer with the current value of that register. Fail if parameterIndex has no corresponding register. VM parameters are numbered as follows: 1 end of old-space (0-based, read-only) 2 end of young-space (read-only) 3 end of memory (read-only) 4 allocationCount (read-only) 5 allocations between GCs (read-write) 6 survivor count tenuring threshold (read-write) 7 full GCs since startup (read-only) 8 total milliseconds in full GCs since startup (read-only) 9 incremental GCs since startup (read-only) 10 total milliseconds in incremental GCs since startup (read-only) 11 tenures of surving objects since startup (read-only) 12-20 specific to the translating VM 21 root table size (read-only) 22 root table overflows since startup (read-only) 23 bytes of extra memory to reserve for VM buffers, plugins, etc. 24 memory threshold above which shrinking object memory (rw) 25 memory headroom when growing object memory (rw) 26 interruptChecksEveryNms - force an ioProcessEvents every N milliseconds, in case the image is not calling getNextEvent often (rw) 27 number of times mark loop iterated for current IGC/FGC (read-only) includes ALL marking 28 number of times sweep loop iterated for current IGC/FGC (read-only) 29 number of times make forward loop iterated for current IGC/FGC (read-only) 30 number of times compact move loop iterated for current IGC/FGC (read-only) 31 number of grow memory requests (read-only) 32 number of shrink memory requests (read-only) 33 number of root table entries used for current IGC/FGC (read-only) 34 number of allocations done before current IGC/FGC (read-only) 35 number of survivor objects after current IGC/FGC (read-only) 36 millisecond clock when current IGC/FGC completed (read-only) 37 number of marked objects for Roots of the world, not including Root Table entries for current IGC/FGC (read-only) 38 milliseconds taken by current IGC (read-only) 39 Number of finalization signals for Weak Objects pending when current IGC/FGC completed (read-only) 40 VM word size - 4 or 8 (read-only) this function maps to primitive 254. In this thread I have been quoting figures reported by parameters 2 and 3 with, I have assumed, 3 having some relevance to the RSS. > Which means that part of the information your really want may be paged > out and so not counted :(. Sure, but that's not entirely relevant to my question, at least one of them anyway, which is why there is so much discrepancy between what from the image seems to be the memory usage and what the operating system reports it as. The answer of course is likely, at least in part, that the value of vmParameter 3 is being misinterpreted and that this is at best part of the right number, perhaps combined with the values of some of the other parameters. I started this thread looking for pointers to what could be the cause of the unexpected amount of memory being used by the website process. At the same time this lead to other curious things and perhaps it would be best to leave those out of the discussion for now in preference to the issue at hand as to what we can due to decrease the resource cost of the website process serving www.squeak.org. > > > > Ken > > > > -- > Tom Rushworth |
Re: What are we leaking in the www.squeak.org process?
|
|
In reply to this post by Ken Causey-3
To add a bit more data to this. I surveyed the rest of the VM
Parameters. (1-25, above 25 I was getting primitive failures and I don't believe any of those are relevant, let me know if I am mistaken.) And I can't find any data there that explains the discrepancy reported between VM Parameter 3 (end of memory) and the resident set size of the process as reported by the operating system. Ignoring those entries that are clearly counts of GC iterations and GC runtimes and such all of the rest of the values that seem like they might be related to memory usage summed up does not come close the current discrepancy of about 70MB. I would really really appreciate some guidance here. Ken On Fri, 2010-10-15 at 15:05 -0500, Ken Causey wrote: > On Thu, 2010-10-14 at 19:00 -0700, Ken Causey wrote: > > > -------- Original Message -------- > > > Subject: Re: [Webteam] Re: [squeak-dev] What are we leaking in the > > > www.squeak.org process? > > > From: Igor Stasenko <[hidden email]> > > > Date: Thu, October 14, 2010 8:45 pm > > > To: The general-purpose Squeak developers list > > > <[hidden email]> > > > > > > > > > On 15 October 2010 04:10, Ken Causey <[hidden email]> wrote: > > > > Another update with more data conflicting somewhat with my last report: > > > > > > > > Shown by ps: 214560KB RSS > > > > Shown by vmParameters (2,3): 64.6MB/67.4MB > > > > > > > > A discrepancy of some 150MB. > > > > > > My understanding is, that Squeak VM cannot shrink a memory which were > > > commited (due to peak growth), > > > to give it back to OS. > > > > > > So, if at some peak, VM comitted 150 MB, and OS sees it as 150M, > > > while VM after GC actually using just 64M. > > > > No, I don't believe this is true. The RSS does go both down as well as > > up. I thought I made that clear in my original comments. > > > > Although I suppose it is possible that drops in the RSS figure are due > > to paging out to swap. I'm not sure how to really check that. > > > > Ken > > For example, another data point: > > Shown by ps: 167344KB RSS > Shown by vmParameters (2,3): 62.9MB/65.8MB > > The process has not been restarted but the RSS has dropped. > > Ken > > > > > > > > > > > > Ken > > > > > > > > On Thu, 2010-10-14 at 17:31 -0500, Ken Causey wrote: > > > >> An update as things may not be quite as I originally described them. It > > > >> seems the values reported by vmParameters 2 and 3 are not as constant as > > > >> I had been lead to believe. I noticed that just a short while after > > > >> sending the image the RSS had jumped up to 165800KB. I still had VNC > > > >> open on the image so I checked what is shown there and I see > > > >> 145.6MB/147.7MB so there is still a discrepancy, but not perhaps as wide > > > >> as I indicated. > > > >> > > > >> I also did not mention that in addition to restarting Swazoo every 15 > > > >> minutes there is other cleanup that is done once a day related to > > > >> statistics gathering that goes on. My understanding is that this > > > >> cleanup is meant to return things very close to the starting state. > > > >> Perhaps Janko can comment on that more fully. > > > >> > > > >> Ken > > > >> > > > >> On Thu, 2010-10-14 at 16:55 -0500, Ken Causey wrote: > > > >> > For some time now I've been bugging Janko about the memory usage of the > > > >> > process that serves http://www.squeak.org/ . I've gathered a bit more > > > >> > info today and I'd like to see if anyone here has any thoughts. > > > >> > > > > >> > First while there is some variation most of the Squeak processes that > > > >> > serve squeak.org sites (source.squeak.org, map.squeak.org, > > > >> > wiki.squeak.org) have a more or less constant and reasonable memory > > > >> > usage. Let me clarify that what I am talking about here is RSS > > > >> > (Resident Set Size) as reported by ps/top on Linux. Swiki as the oldest > > > >> > of the group is the clear winner at just under 30MB, squeakmap is around > > > >> > 47MB, and source.squeak.org is around 59MB. > > > >> > > > > >> > In contrast immediately after starting it www.squeak.org is using about > > > >> > 70MB and it grows from there. Now a bit more than an hour since the > > > >> > last restart it is already up to about 85MB. I have seen it over 300MB > > > >> > before, and it commonly uses 140-250MB. It doesn't simply grow and > > > >> > grow, it does shrink occasionally. > > > >> > > > > >> > First, there is a known issue with Swazoo that it does not close sockets > > > >> > correctly and Janko has addressed this by having Swazoo automatically > > > >> > restarted every 15 minutes. So using netstat for example I can watch > > > >> > the socket count for the process continually grow and then it drops to > > > >> > zero and this repeats. In contrast for all the other process (all > > > >> > running some variation of Kom/Comanche I believe) I'm hard pressed to > > > >> > even catch a single open socket at any time (not counting the listener > > > >> > that is). > > > >> > > > > >> > Secondly, since I've been complaining about this issue for a long time > > > >> > Janko has running a little morph that shows current memory usage. What > > > >> > this shows is the values of SmalltalkImage current vmParameterAt: 2 and > > > >> > 3. These numbers are quite stable. For example at this moment these > > > >> > are 68.3MB and 71.0MB respectively. Whereas ps shows the RSS at > > > >> > 85752KB. Where is the discrepancy here? > > > >> > > > > >> > My current theory is that the vmParameters are not reflecting incidental > > > >> > memory use by the VM and plugins or that we are simply misinterpreting > > > >> > the values. Is it possible that Swazoo is failing to release something > > > >> > that is causing the VM to use more memory than it should? > > > >> > > > > >> > Note that the same VM is used by all of these processes. It is > > > >> > > > > >> > 3.8a-1 #1 Sun May 1 19:46:46 EDT 2005 gcc 3.3.5 > > > >> > > > > >> > Yes, I know it's old, but 'if it ain't broke, don't fix it' has been the > > > >> > motto and with the possible exception of Swazoo/www.squeak.org it has > > > >> > been working wonderfully for a long time now. > > > >> > > > > >> > Your thoughts would be greatly appreciated, > > > >> > > > > >> > Ken > > > >> > > > >> _______________________________________________ > > > >> Webteam mailing list > > > >> [hidden email] > > > >> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/webteam > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Best regards, > > > Igor Stasenko AKA sig. > > > > > > > > _______________________________________________ > Webteam mailing list > [hidden email] > http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/webteam |
Re: What are we leaking in the www.squeak.org process?
|
|
In reply to this post by Ken Causey-3
To add a bit more data to this. I surveyed the rest of the VM
Parameters. (1-25, above 25 I was getting primitive failures and I don't believe any of those are relevant, let me know if I am mistaken.) And I can't find any data there that explains the discrepancy reported between VM Parameter 3 (end of memory) and the resident set size of the process as reported by the operating system. Ignoring those entries that are clearly counts of GC iterations and GC runtimes and such all of the rest of the values that seem like they might be related to memory usage summed up does not come close the current discrepancy of about 70MB. I would really really appreciate some guidance here. Ken On Fri, 2010-10-15 at 15:05 -0500, Ken Causey wrote: > On Thu, 2010-10-14 at 19:00 -0700, Ken Causey wrote: > > > -------- Original Message -------- > > > Subject: Re: [Webteam] Re: [squeak-dev] What are we leaking in the > > > www.squeak.org process? > > > From: Igor Stasenko <[hidden email]> > > > Date: Thu, October 14, 2010 8:45 pm > > > To: The general-purpose Squeak developers list > > > <[hidden email]> > > > > > > > > > On 15 October 2010 04:10, Ken Causey <[hidden email]> wrote: > > > > Another update with more data conflicting somewhat with my last report: > > > > > > > > Shown by ps: 214560KB RSS > > > > Shown by vmParameters (2,3): 64.6MB/67.4MB > > > > > > > > A discrepancy of some 150MB. > > > > > > My understanding is, that Squeak VM cannot shrink a memory which were > > > commited (due to peak growth), > > > to give it back to OS. > > > > > > So, if at some peak, VM comitted 150 MB, and OS sees it as 150M, > > > while VM after GC actually using just 64M. > > > > No, I don't believe this is true. The RSS does go both down as well as > > up. I thought I made that clear in my original comments. > > > > Although I suppose it is possible that drops in the RSS figure are due > > to paging out to swap. I'm not sure how to really check that. > > > > Ken > > For example, another data point: > > Shown by ps: 167344KB RSS > Shown by vmParameters (2,3): 62.9MB/65.8MB > > The process has not been restarted but the RSS has dropped. > > Ken > > > > > > > > > > > > Ken > > > > > > > > On Thu, 2010-10-14 at 17:31 -0500, Ken Causey wrote: > > > >> An update as things may not be quite as I originally described them. It > > > >> seems the values reported by vmParameters 2 and 3 are not as constant as > > > >> I had been lead to believe. I noticed that just a short while after > > > >> sending the image the RSS had jumped up to 165800KB. I still had VNC > > > >> open on the image so I checked what is shown there and I see > > > >> 145.6MB/147.7MB so there is still a discrepancy, but not perhaps as wide > > > >> as I indicated. > > > >> > > > >> I also did not mention that in addition to restarting Swazoo every 15 > > > >> minutes there is other cleanup that is done once a day related to > > > >> statistics gathering that goes on. My understanding is that this > > > >> cleanup is meant to return things very close to the starting state. > > > >> Perhaps Janko can comment on that more fully. > > > >> > > > >> Ken > > > >> > > > >> On Thu, 2010-10-14 at 16:55 -0500, Ken Causey wrote: > > > >> > For some time now I've been bugging Janko about the memory usage of the > > > >> > process that serves http://www.squeak.org/ . I've gathered a bit more > > > >> > info today and I'd like to see if anyone here has any thoughts. > > > >> > > > > >> > First while there is some variation most of the Squeak processes that > > > >> > serve squeak.org sites (source.squeak.org, map.squeak.org, > > > >> > wiki.squeak.org) have a more or less constant and reasonable memory > > > >> > usage. Let me clarify that what I am talking about here is RSS > > > >> > (Resident Set Size) as reported by ps/top on Linux. Swiki as the oldest > > > >> > of the group is the clear winner at just under 30MB, squeakmap is around > > > >> > 47MB, and source.squeak.org is around 59MB. > > > >> > > > > >> > In contrast immediately after starting it www.squeak.org is using about > > > >> > 70MB and it grows from there. Now a bit more than an hour since the > > > >> > last restart it is already up to about 85MB. I have seen it over 300MB > > > >> > before, and it commonly uses 140-250MB. It doesn't simply grow and > > > >> > grow, it does shrink occasionally. > > > >> > > > > >> > First, there is a known issue with Swazoo that it does not close sockets > > > >> > correctly and Janko has addressed this by having Swazoo automatically > > > >> > restarted every 15 minutes. So using netstat for example I can watch > > > >> > the socket count for the process continually grow and then it drops to > > > >> > zero and this repeats. In contrast for all the other process (all > > > >> > running some variation of Kom/Comanche I believe) I'm hard pressed to > > > >> > even catch a single open socket at any time (not counting the listener > > > >> > that is). > > > >> > > > > >> > Secondly, since I've been complaining about this issue for a long time > > > >> > Janko has running a little morph that shows current memory usage. What > > > >> > this shows is the values of SmalltalkImage current vmParameterAt: 2 and > > > >> > 3. These numbers are quite stable. For example at this moment these > > > >> > are 68.3MB and 71.0MB respectively. Whereas ps shows the RSS at > > > >> > 85752KB. Where is the discrepancy here? > > > >> > > > > >> > My current theory is that the vmParameters are not reflecting incidental > > > >> > memory use by the VM and plugins or that we are simply misinterpreting > > > >> > the values. Is it possible that Swazoo is failing to release something > > > >> > that is causing the VM to use more memory than it should? > > > >> > > > > >> > Note that the same VM is used by all of these processes. It is > > > >> > > > > >> > 3.8a-1 #1 Sun May 1 19:46:46 EDT 2005 gcc 3.3.5 > > > >> > > > > >> > Yes, I know it's old, but 'if it ain't broke, don't fix it' has been the > > > >> > motto and with the possible exception of Swazoo/www.squeak.org it has > > > >> > been working wonderfully for a long time now. > > > >> > > > > >> > Your thoughts would be greatly appreciated, > > > >> > > > > >> > Ken > > > >> > > > >> _______________________________________________ > > > >> Webteam mailing list > > > >> [hidden email] > > > >> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/webteam > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Best regards, > > > Igor Stasenko AKA sig. > > > > > > > > _______________________________________________ > Webteam mailing list > [hidden email] > http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/webteam |
Re: What are we leaking in the www.squeak.org process?
|
|
Re: What are we leaking in the www.squeak.org process?
|
|
On Wed, 2010-10-20 at 00:16 -0700, Colin Putney wrote:
> On Tue, Oct 19, 2010 at 10:35 PM, Ken Causey <[hidden email]> wrote: > > > > I would really really appreciate some guidance here. > > I'd assume that the VM is working fine, and something in the image is > what's using the memory. Try looking at counts of instances by class > and see if anything sticks out, then get a hold of one of them and > trace the references back and see where it's being referenced from. > Look at running Processes and see if there are any with odd things on > the stack that could reference a large tree of objects. Look at > Morphic: explore the World, the deferred messages queue and the ui > process. > > Also look at the VM process from outside, via the OS. How many file > handles does it have open? What files, sockets, pipes etc do they > refer to? What shared libraries are loaded? Can you profile it and see > where it's spending most of its processing time? How do all these > things differ between a VM process that's just started, and one that's > been running a while and is using a lot of memory? > > Just a few suggestions. Good luck! > > Colin been going on for a long time now and the low hanging fruit has already been picked. Janko Mivsek has investigated to the best of his ability and dooesn't believe there is any garbage that is not being dereferenced and not collected. This is very strongly reinforced by the usage of a morphic tool call 'Memory Usage' which shows the values of VM parameters 2 and 3. With the exception of short lived peaks, the values shown here are stable in the range of 60-70MB at all times for weeks and months of time. I don't have any reason to think there is a problem with the VM, it is the same VM most of the Squeak processes have been using on our server for some time now. But the rest all have very consistent solid memory usage from the operating system perspective. This is not the case for the website process which is using the very same VM. So I agree that something in the image is probably wrong, or perhaps Janko has succeeded in tickling a bug that the others have not. Regarding looking at the VM process from outside. Please read my long message more carefully, I realize it's not formatted terribly well, sorry about that. But I mention another issue which is known regarding failures to properly close sockets. But as mentioned this is cleared up every 15 minutes, which I can see from the operating system side. But this cleanup is not reflected in any way in the memory usage. This is a difference compared to other Squeak processes we are running and maybe the problem is related to this, but it's non-obvious and help is needed to understand why, if that's the case. I'm no lsof expert, but a quick look at the output there doesn't indicate anything surprising to me, certainly no great number of files or other file-like items open. Near the end of the message you seem to imply some sort of issue with the problem getting worse over time. Apparently I was less than clear on this. The problem with the memory issue can happen within a matter of minutes of startup, certainly within an hour or two. And it doesn't get particularly worse with time. The fundamental issue is simply that for no good reason that we have been able to discover the website process uses several times as much memory as any other of our Squeak-based services. At this point the thing that clearly differs for this process is the discrepancy between what the VM reports regarding memory usage and that reported by the operating system. The amount of this discrepancy also varies seemingly in step with the RSS reported by the operating system in contrast to the reported memory usage by the VM which is comparatively constant. Ken |
Re[2]: [squeak-dev] What are we leaking in the www.squeak.org process?
|
|
Re[3]: [squeak-dev] What are we leaking in the www.squeak.org process?
|
|
Re: What are we leaking in the www.squeak.org process?
|
|
In reply to this post by Ken Causey-3
Ahhhh.
All the source code exists, so let's look. Since WikiServer had to run with 50MB and people wanted to upload/download 50MB documents via the seaside server, I've had to chase this issue at length. So first you must figure out what binary and when it was created because the source code does change from time to time. Second you should review the documentation you possibly could find at. http://isqueak.org/sqAllocateMemory http://isqueak.org/sqGrowMemoryBy http://isqueak.org/sqShrinkMemoryBy In looking at the unix code base (current) which might no have relationship to whatever old binary you are running you will find sqUnixMemory.c -- dynamic memory management Assuming you are using the MMAP logic which is choose or not based on startup parms, then it does heap= mmap(0, heapLimit, MAP_PROT, MAP_FLAGS, devZero, 0) Please note the /* Note: * * The code allows memory to be overallocated; i.e., the initial * block is reserved via mmap() and then the unused portion * munmap()ped from the top end. This is INHERENTLY DANGEROUS since * malloc() may randomly map new memory in the block we "reserved" * and subsequently unmap()ped. Enabling this causes crashes in * Croquet, which makes heavy use of the FFI and thus calls malloc() * all over the place. * * For this reason, overallocateMemory is DISABLED by default. * * The upshot of all this is that Squeak will claim (and hold on to) * ALL of the available virtual memory (or at least 75% of it) when * it starts up. If you can't live with that, use the -memory * option to allocate a fixed size heap. */ But this of course thru bitter experiences has been a moving target and your binary might have different behaviour, but let's assume you run the current one which will never give memory once seized. So if you take the default of allocating 512MB of memory then the Squeak VM gets 512MB+ Now for the end of memory pointer we just move that pointer within the block of mmap. Also past the end of squeak VM oops memory we allocate memory for the CG to do the compacting run, but the numbers reported by the VM parms just indicate where the end of oops memory is, not the amount of mmap memory, which is a constant set by default or your override. Now from the Operating systems viewpoint and I'll only talk about BSD related ones, linux may or may not be the same you would have to ask, is that you might ask for 2 trillion bytes but if you only use 200MB then that is what will show. You must touch the memory to actually make the operating system understand you aren't lying. Hence on my mac here it cheerfully says I've 184.98 GB of virtual memory allocated, but lies are lies given it only has a 100GB hard disk. Mmm actually it only has < 3GB of memory actually used. So where DOES the memory monster come from? Well from my past experiences with WikiServer I know that the HTTP servers are setup to SUCK in complete files. Got a 100MB MC file? Well you will need 100MB or 200MB to make that pass thru the image. I note you can configure the various http servers to say they should do streaming, but I found the various authors actually have never done an end to end test so even if they promise to stream which you set on the HTTP server, they accidentally do things like x := "stream contents foobar" Although in the summer of 09 I did manage to fix both http servers and feed results back to authors so that I could upload or download a 50MB file to wikiserver. To test this you should configure a VM with a hard upper memory limit. Likely via a command line parm asking for just a 50MB block of memory? Then pass thru a MC project that requires 50MB file and see where in the http server or seaside or mc that it's making an in-memory copy of the entire file. I note once the squeak VM has touched the 200MB the operating system will say that 200MB are used by the app, even if the VM now is only using 40MB. The other 160MB (actually less) is now idle.... The operating system virtual memory logic will at some point if needed apply the rules about recently used pages to decide to push those pages to some backing store in order to free up the RAM page for use by someone who has more immediate needs. Noting that on this mac, the Squeak VM running says 38.9 MB in use, 286.9MB virtual memory in use. Yet I know I asked for 512MB so it doesn't even report the fact it should be 512+ correctly either. But this might be an optimization and it's decided that it could supply 286.9-38.9 immediately Mmm foo := ByteArray new: 200*1024*1024 Ya, 236.1 mb, 414.9mb Virtual memory. 203.01 GB total... I'd go check your httpserver configuration for streaming, then see who is sucking in entire MC files in a go.. On 2010-10-19, at 10:35 PM, Ken Causey wrote: > To add a bit more data to this. I surveyed the rest of the VM > Parameters. (1-25, above 25 I was getting primitive failures and I > don't believe any of those are relevant, let me know if I am mistaken.) > And I can't find any data there that explains the discrepancy reported > between VM Parameter 3 (end of memory) and the resident set size of the > process as reported by the operating system. > > Ignoring those entries that are clearly counts of GC iterations and GC > runtimes and such all of the rest of the values that seem like they > might be related to memory usage summed up does not come close the > current discrepancy of about 70MB. > > I would really really appreciate some guidance here. > > Ken =========================================================================== John M. McIntosh <[hidden email]> Twitter: squeaker68882 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com =========================================================================== |
Re: What are we leaking in the www.squeak.org process?
|
|
Thank you John, Herbert and Colin,
Janko and I will be going over your emails in the next couple of days and seeing if we can find anything based on the information provided. Janko will probably reply in time, perhaps with further questions. None of this directly addresses my question regarding the large and non-constant discrepancy between the memory usage reported by the operating system (RSS) and the information the VM gives us about memory usage. That is, except possibly for John's post which does suggest some things but I can't see how they apply at the moment, Janko may think differently though. To be clearer, let's remember I'm talking about the process behind http://www.squeak.org/ . Yes, technically that is a wiki on top of AidaWeb/Swazoo. That said, there is no significant editing going on right now (none in fact, as far as I'm aware) and certainly no large files being added. Also, as far as I'm aware, there are no particularly large files downloaded from the site that a served through Squeak. I could be wrong about this and I've asked Janko to consider that. So at the moment I'm at a loss to see how we can apply much of this, but I'm not that knowledgeable about AidaWeb/Swazoo and in particular the code for www.squeak.org and Janko will hopefully find some information that helps us. Thanks, Ken |
Re: What are we leaking in the www.squeak.org process?
|
|
Re: What are we leaking in the www.squeak.org process?
|
|
Janko and I have been doing some fiddling around and monitoring today
and I begin to develop a theory. First, meet some images: Image 1: SqueakMap (map.squeak.org). This is our oldest Squeak service image with the possible exception of wiki.squeak.org. This is a 3.8 image running on a 3.8 VM. SqueakMap saves it's data to a separate file and snapshots of the image are only made when source code changes occur, aka manually. Image 2: SqueakSource (source.squeak.org). This is a 3.11 image running on a 3.11 VM. This image saves it's data by snapshoting hourly. Image 3: www.squeak.org. This is a 3.9 VM that has been running for some time on the same 3.8 VM as SqueakMap but we just a short while ago decided to try it on the 3.11 VM SqueakSource is using and it is running fine so far. So for a few hours now I have been monitoring the memory usage and GC behavior of these three images. The short story is that SqueakMap is almost completely rock solid but that the SqueakSource image, despite my earlier claims to the contrary, appears to suffer a much reduced form of the problem we are observing on www.squeak.org. To reiterate: The problem is that the resident set size of the process appears to continually grow. The operating system will occasionally swap out some of this giving the impression that the RSS has dropped but I'm almost certain that the total cost (RAM+swap) of the process is the same and continuing to increase. Our available RAM and swap on the server is quite limited with all the various services we are running. What I'm observing is a discrepancy between what the VM claims is the end of memory and the RSS. When any of these images start this discrepancy is quite small, maybe a megabyte. But whenever a GC occurs the end of memory figure drops, but the RSS goes up. So from the images side it all appears copacetic: we cleaned up the garbage and we are more or less back at our base memory usage level. But as far as the operating system is concerned nothing has happened, in fact more memory has been claimed. So my theory is that somehow when GC occurs on the linux VM (3.8 VM and 3.11 VM) the freed up memory is not released to the operating system but it seems that new objects go in a different memory location from the operating system's perspective therefore using additional memory on top of what was used before the GC. I only really observe this on full GCs. When few if any full GCs occur, there is no real problem. But when you do things like snapshot the image hourly, this all adds up to several megabytes a day of lost RAM. This may not be quite right, I'm still observing and of course this theory is built out of black box observation and little else. Feel free to set me straight. Ken |
| Free forum by Nabble | Edit this page |