Understanding temp memory used seaside gems and 'ps' output
Locked
Understanding temp memory used seaside gems and 'ps' output
 
|
Hi guys,
I run this command on my server $ ps -e -o pid,rss,comm= | sort -n -k 2 And processes used the most memory (are at the very end of the above script) are topaz processes: 26570 569256 topaz 26537 617288 topaz 2560 622048 topaz 2132 635224 gem 26603 669584 topaz 2564 726456 topaz 1865 731244 pgsvrmain 2563 786204 topaz 2895 939380 gem 2869 955488 pgsvrmain Of course, those PIDs (2563, 2564 etc) are exactly those of my seaside gems. 786204 KB (pid 2563) is around 786.204MB. $ ps -fea | grep 2563 myuser 2563 1 0 Jun15 ? 00:03:42 /opt/gemstone/product/bin/topaz -l -e /opt/gemstone/product/seaside/etc/seaside30.conf -I /xxx/sites/yyy/gemstone/.topazini Note that in /opt/gemstone/product/seaside/etc/seaside30.conf I define: GEM_TEMPOBJ_CACHE_SIZE = 700000; So.. first question is why process 2563 is showing more than 700MB (786MB in this case). Anyway...if I go and cycle over each of my seaside gems and I print the results of "System _tempObjSpacePercentUsed" for each gem, I usually get between 5% and 20%, which would mean between 35MB to 140MB. This is FAR from 786MB reported by PS. So.... I am misunderstanding something? yes...my OS is having little memory free and I am trying to see if I am not holding unnecessary memory... Thanks in advance for any clarification. _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Understanding temp memory used seaside gems and 'ps' output
|
|
On Fri, Jun 19, 2015 at 5:32 PM, Richard Sargent <[hidden email]> wrote:
Hi Richard, Yes, I did that. In fact, before sending the email, I tried with vsz rather than rss and for the same process it showed me 3.5GB!!! much more than the 700MB. So that confused me too haha.
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Understanding temp memory used seaside gems and 'ps' output
|
|
In reply to this post by GLASS mailing list
On 06/19/2015 01:22 PM, Mariano
Martinez Peck via Glass wrote:
Well, the GEM_TEMPOBJ_CACHE_SIZE is not the sole factor in controlling the size of your process....there are a handful of other memory controls that drive memory consumption: GEM_TEMPOBJ_MESPACE_SIZE GEM_TEMPOBJ_OOPMAP_SIZE GEM_TEMPOBJ_POMGEN_SIZE GEM_TEMPOBJ_SCOPES_SIZE With the configuration parameters you can control the sizes of each of the memory regions, but by default these spaces are sized as a function of GEM_TEMPOBJ_CACHE_SIZE. How big is your SPC ... I think that the active pages in the SPC get factored into the the memory consumption reported for gems ... I assume that you have allocated a generaout amount of swap space? With enough swap space, your system will continue running (slowly) if you end up consuming all of available memory ... Dale _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Understanding temp memory used seaside gems and 'ps' output
|
|
On Mon, Jun 22, 2015 at 1:54 PM, Dale Henrichs via Glass <[hidden email]> wrote:
Oh, thanks for such a detail. Very interesting.
It is about 1GB for those stones. But I think that you nailed "the active pages in the SPC get factored into the the memory consumption reported for gems" because I think the PS output and how linux works etc is not very very accurate so it could be what you say.
yes, I have plenty. $ cat /proc/meminfo | grep Swap SwapCached: 3632 kB SwapTotal: 17039352 kB SwapFree: 16996596 kB And yes, the system continues to work, but I noticed some random "slow pauses" in the app of around 15 to 30 seconds and when I check "free -m" I notice I am full with memory, so that's why I start to analyze these things a bit. Thanks!
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Understanding temp memory used seaside gems and 'ps' output
|
|
On 06/22/2015 10:02 AM, Mariano
Martinez Peck wrote:
Yeah paging can be expensive, but if you start seeing these kinds of pauses that can't be blamed on swapping, there are other things you might do ... the main one I'm thinking about is that Linux will prioritize disk reads over disk writes and that means that a commit (disk write for tranlog) can be delayed in favor of disk reads (disk swapping or SPC page reads). In this case you can move the tranlogs to a separate LUN reserved for your tranlogs ... Dale _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Understanding temp memory used seaside gems and 'ps' output
|
|
In reply to this post by GLASS mailing list
On 06/19/2015 01:22 PM, Mariano Martinez Peck via Glass wrote:
> > *So.. first question is why process 2563 is showing more than 700MB > (786MB in this case).* > > Anyway...if I go and cycle over each of my seaside gems and I print the > results of *"System _tempObjSpacePercentUsed" for each gem, I usually > get between 5% and 20%*, which would mean between 35MB to 140MB. This is > FAR from 786MB reported by PS. So.... I am misunderstanding something? > > yes...my OS is having little memory free and I am trying to see if I am > not holding unnecessary memory... > Linux is very precise about what it reports, but not very obvious, since it avoids using memory whenever it can get away with not using memory. /proc/meminfo gives the best information about the entire system, but you have to read a bunch of documentation (and sometimes kernel code) to understand what all the lines mean. As I understand it, the VSZ of a process is all of the virtual memory pages that has been allocated by the process. This might be the total of everything that shows up in /proc/<pid>/maps. This will include memory that is being used, and also memory that is shared, and memory that is swapped out, but also things that have never been used, and perhaps will never be used, and never consume actual memory. For instance, every gem will probably report the entire shared page cache as part of its VSZ. The RSS of a process is the subset of VSZ that is actually mapped to real memory at a point in time. This includes shared memory, so every gem reports some portion of the SPC in its RSS. Pages in the SPC are mapped to the process on demand (this is a Linux kernel thing, not a GemStone thing) and never unmapped once mapped. So the RSS of a gem will be larger or smaller depending on which page frames in the SPC that it has ever touched ever since it was launched. So the RSS, by itself, doesn't tell you much. The amount of free memory in the system doesn't tell you much either. Any Linux system that has been running a while will have low free memory, unless some process recently exited that was using a lot of anonymous memory (memory which is not backed by a file). This is because when Linux reads things from files into memory, it keeps the memory version of that file around until there is demand for memory, since freeing this memory is cheap and easy, but re-reading the contents from a file is expensive, and someone might want that file again. So much for things that don't tell you much. What *does* tell you useful things? Recent kernels have a field in /proc/meminfo called "MemAvailable" which is an estimate of how much you could allocate before actually running out. At this moment my desktop system is reporting 1.8G free, but 18G available. If you don't have the MemAvailable stat in your kernel, it's approximately MemFree + Active(file) + Inactive(file). The iotop tool can tell you what processes are doing disk I/O, and when they're doing it, and whether and how much swapping is going on. Using a small amount of swap space is not a problem, and occasional small reads and writes to swap are not a problem. This indicates that the system is swapping out things that hardly ever get used, freeing up memory for more frequently-accessed stuff. But frequent swap in/out activity is an indicator of memory pressure, so if you see that you should look at decreasing memory demand or adding memory to the system. If you see pauses that correspond to disk I/O that is not swap I/O, then things like Dale's hint apply. Some of the Linux I/O schedulers do prioritize reads over writes. Others don't, and which scheduler you choose can make a difference. Some schedulers may be tunable. There are ways to tell which I/O schedulers your kernel supports, but I'll let you look that up if you think you need it. :-) Regards, -Martin _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
«
Return to GLASS
|
1 view|%1 views
| Free forum by Nabble | Edit this page |