GC on Gems only fired when temp space is over?
Locked
GC on Gems only fired when temp space is over?
 
|
Hi guys,
I wanted to know if the following assumption is correct... Imagine I have a stone/gems that I do not use frequently. There is 1 stone and 4 seaside gems. I do understand that when local temp memory of a gem is running out of space, a scavenge is done so that young objects can die, then move survivors, maybe flash out to disk persistent objects not referenced anymore etc etc. Now.. imagine that I have a .. .700MB temp object space. And since I have 4 gems, I may randomly use a different one for each request (I have no session affinity). So imagine that now my temp space for my gems are all around 500MB. The memory is not yet filled and there is yet some more space. The question is...imagine I do not use this stone/gems for some time...a couple of days. The question is...are these 500MB of memory retained by the gems until I use it again and I run out of space? Isn't the scavenge run every some time besides the "run on running out of memory that would trigger a scavenge"? What things could I do here to improve this scenario? 1) make temp space smaller..but that would impact when I DO use it 2) maybe use session affinity so that at least I fill up less gems? (if there are little concurrent sessions) thanks in advance? _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: GC on Gems only fired when temp space is over?
|
|
On 06/19/2015 02:28 PM, Mariano Martinez Peck via Glass wrote: > Hi guys, > > I wanted to know if the following assumption is correct... Imagine I > have a stone/gems that I do not use frequently. There is 1 stone and 4 > seaside gems. I do understand that when local temp memory of a gem is > running out of space, a scavenge is done so that young objects can > die, then move survivors, maybe flash out to disk persistent objects > not referenced anymore etc etc. Now.. imagine that I have a .. .700MB > temp object space. And since I have 4 gems, I may randomly use a > different one for each request (I have no session affinity). So > imagine that now my temp space for my gems are all around 500MB. The > memory is not yet filled and there is yet some more space. > > The question is...imagine I do not use this stone/gems for some > time...a couple of days. The question is...are these 500MB of memory > retained by the gems until I use it again and I run out of space? > Isn't the scavenge run every some time besides the "run on running out > of memory that would trigger a scavenge"? is possible for idle gems to consume "too much memory" > > What things could I do here to improve this scenario? > > 1) make temp space smaller..but that would impact when I DO use it It is a good question as to why you need such a large TOC ... if your working set occasionally balloons to require that much space then you need that much space ... > 2) maybe use session affinity so that at least I fill up less gems? > (if there are little concurrent sessions) > You could schedule a periodic scavenge and mark sweep in your seaside vms ... Once an hour or so would make sure that your idle seaside gems are using the bare minimum of memory ... it may tak a mark sweep for the vm to return memory pages ... BTW, I assume that you are running on linux. On the mac the full memory footprint is allocated in memory ... Dale _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: GC on Gems only fired when temp space is over?
|
|
On Mon, Jun 22, 2015 at 2:16 PM, Dale Henrichs via Glass <[hidden email]> wrote:
OK, that's what I imagined.
Well... this is a finance app...so most of the times, when they are using it, I normally need a high TOC.. By default I am using 60/70 % of SPC. But...there might not be many concurrent users at some point, so yes, I would like to minimize memory consumed by idle gems.
How can I do this if above you said "scavenges and mark sweep are only triggered under memory pressure" ? Once an hour or so would make sure that your idle seaside gems are using the bare minimum of memory ... it may tak a mark sweep for the vm to return memory pages ... BTW, I assume that you are running on linux. On the mac the full memory footprint is allocated in memory ... I would love to do that. Any idea how? I also saw a thread of Otto today with a (maybe?) related topic. Do you think what he does could help me too? Thanks in advance! Dale _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: GC on Gems only fired when temp space is over?
|
|
On 06/22/2015 10:36 AM, Mariano
Martinez Peck wrote:
These are the methods to use ... and as I mentioned in my last post, the "sigabort process" would be a good place to this processing. Here are the methods to manually trigger scavenges and mark sweeps ... the last one is probably the one you want: System class>>_vmMarkSweep System class>>_generationScavenge System class>>_generationScavenge_vmMarkSweep Haven't looked into Otto's issue yet ... it's next up:) Dale _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: GC on Gems only fired when temp space is over?
|
|
On Mon, Jun 22, 2015 at 3:09 PM, Dale Henrichs <[hidden email]> wrote:
Dale, Thanks for such methods! Very useful. Now..I have a couple of questions :) In another email you said: -------- We really try to make sure that idle gems stay idle and will only wake up when needed ... but there is at least one restless process that wakes up every 30 seconds to facilitate the handling of SigAborts ... the process itself doesn't touch much memory, but if a SigAbort is required then the gem will also do an abort and page in a bit more memory, but an abort is pretty inexpensive for a previously idle process ... BTW, this little "sigabort process" would be a spot where you could add the logic for running the periodic scavenge and mark sweep:) -------- The main question here is....are you that confident that #_generationScavenge_vmMarkSweep would be so fast that calling it every 30 seconds won't impact much the system performance? I was thinking about calling it like...half an hour hahahaha. Anyway, if I watch the code: Transcript cr; show: 'New style SigAbort hanlder'. TransactionBacklog addDefaultHandler: [ :ex | "Run the abort in a lowPriority process, since we must acquire the transactionMutex." Transcript cr; show: 'handled sigabort: ', DateAndTime now printString. [ GRPlatform current transactionMutex critical: [ GRPlatform current doAbortTransaction ]. TransactionBacklog enableSignalling ] forkAt: Processor lowestPriority. ex resume ]. RepositoryViewLost addDefaultHandler: [ :ex | GRPlatform current logError: ex description title: 'Lost OT' shouldCommit: false. ex pass ]. TransactionBacklog enableSignalling. it is not clear to me how can I hook. Also..I was thinking to include it as part of seaside maintenance VM (a new task) to run every 30 minutes. But then the other question I had is that from the code/comment of #_generationScavenge_vmMarkSweep it is not clear to me if it is performed only in the current gem or in all gems. It seems to me in current gem. So..... I guess I cannot call this from maintenance VM, can I? Thanks in advance, _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: GC on Gems only fired when temp space is over?
|
|
On 06/22/2015 01:22 PM, Mariano
Martinez Peck wrote:
No I meant that you could put code into that process that ran the scavenge/marksweep once an hour or day or half hour or whatever ... every 30 seconds is probably too often ... the scavenge and marksweep _are_ very quick, but you only really need to run them often enough to keep idle gems from wasting swap space ... The real timing will be up to you as far as how long you think a gem can sit idle before it should drop any extra memory that it's using ... That isn't the spot I was thinking of ... Here's the process that I was thinking could be piggy-backed: "This thread is needed to handle the SigAbort exception, when the primary thread is blocked. Assuming default 60 second STN_GEM_ABORT_TIMEOUT, wake up at 30 second intervals." [ [ true ] whileTrue: [ (Delay forSeconds: 30) wait ]. ] forkAt: Processor lowestPriority. The sigabort will only be triggered when smalltalk code is running ... if all of the processes in the vm are blocked on a socket read, the sigabort won't get through ... so this guy is there to make sure that the vm will wake up and check for a sigabort often enough ...
The scavenge and sweep are only run in the current vm ... so each vm would have to have this code added ... Dale _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: GC on Gems only fired when temp space is over?
|
|
Thank you SO MUCH Dale! So this is what I did for all seaside gems in case someone else is interested: "This thread is needed to handle the SigAbort exception, when the primary thread is blocked on an accept. Assuming default 60 second STN_GEM_ABORT_TIMEOUT, wake up at 30 second intervals." [ | count minutesToForceGemGC | count := 0. minutesToForceGemGC := 30. [ true ] whileTrue: [ (Delay forSeconds: 30) wait. count := count + 1. (count \\\ (minutesToForceGemGC * 2)) = 0 ifTrue: [ System _generationScavenge_vmMarkSweep. Transcript show: 'Gem GC forced'; cr. count := 0. ]. ]. ] forkAt: Processor lowestPriority. And yeah, it seems to work :) And yeah, the GC seems to be super fast :) Good job! Also.... similar to this great idea, do you have something else in mind I can do to minimize memory used when system is mostly "idle" ? I ask because I did not noticed much of a change if I shutdown all seaside gems and start them again. However.....if I shut down all stones and start them again (with all seaside gems again), then the memory used decreases A LOT. While this may be obvious, it is not fully clear to me why. As a first thought I would guess the main difference is the SPC. However, SPC is a segment/shared and that memory is reserved from the very beginning. So I don't expect that to change wether I reset the stone or not. So...if it is not the SPC.... they are not the seaside gems, they are not the admin/reclaim gc (those use very little temp space), what else could be such a difference if memory consumption when I reset the stone? Thanks in advance, On Mon, Jun 22, 2015 at 8:49 PM, Dale Henrichs <[hidden email]> wrote:
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: GC on Gems only fired when temp space is over?
|
|
On 06/23/2015 06:51 AM, Mariano
Martinez Peck wrote:
Cool, I'll look at adding this logic to the GemServer scripts .... Have you noticed any differences in the `ps` output? Also do you see the memory use grow over time .. even with an idle system? It is very likely that the SPC is the culprit ... the shrpcmonitor regularly scans pages in the SPC, while it doesn't touch all of the pages, it touches maybe half so that could account for differences between a freshly started system and one that has run for awhile... off hand I don't know how often it does it's scan ... I think you should be able to plot memory consumption as compared to other things going on in the system by starting up that statmonitor (use the -A option to statmonitor to collect system stats) ... Dale _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: GC on Gems only fired when temp space is over?
|
|
On Tue, Jun 23, 2015 at 1:33 PM, Dale Henrichs <[hidden email]> wrote:
Yes, I was about to recommend you that, since the GC seems super fast for a.. 700MB temp space, and considering it runs every .. 30 mins ,,is more than fine!
I am not sure if it increases with a fully fully idle system, but it does increase with barely (only a couple of requests a day) idle system.
I just did a "ps -aux" after a bit of usage (the barely usage) and then the comparison after stoping and starting stone again and all seaside gems. By far, the biggest difference was pgsvrmain which ps showed me like 300MB and after stone restart was nothing.... The second process that decreased quite a bit is pagemanager. Also.. the reclaimgcgem decreased quite a bit too. Is there a way to decrease those if the system is mostly "idle" ? anything interesting here? Thanks in advance!
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: GC on Gems only fired when temp space is over?
|
|
On 06/23/2015 06:02 PM, Mariano
Martinez Peck wrote:
I've submitted Issue #34 for this[1]. [1] https://github.com/GsDevKit/gsApplicationTools/issues/34
I imagine that the memory consumption here is largely related to pages in the SPC as those three processes do most of their work in shared memory ... I've pinged engineering to see if anyone has additional information... Dale _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: GC on Gems only fired when temp space is over?
|
|
On Wed, Jun 24, 2015 at 1:57 PM, Dale Henrichs <[hidden email]> wrote:
Cool!
Ok, thanks. I think this could be very useful for many people. Imagine you have an app that you deploy for different clients using different stones. They could run in same server. And each client could have different timezone. So maybe a store is mostly idle for 8 hours, while others are very busy and vice-versa. If not automatic, at least an API to flush such things. So then I can define a cron that say... from 8pm to 8am, flush all pages out of memory (or whatever that means). In other words...I can keep the system use less resources if "mostly being idle" and then if some poor guy wants to use, then ok, he may pay some extra performance. Thanks! _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: GC on Gems only fired when temp space is over?
|
|
In reply to this post by GLASS mailing list
On 06/23/2015 06:51 AM, Mariano Martinez Peck via Glass wrote:
> > I ask because I did not noticed much of a change if I shutdown all > seaside gems and start them again. However.....if I shut down all stones > and start them again (with all seaside gems again), then the memory used > decreases A LOT. While this may be obvious, it is not fully clear to me > why. As a first thought I would guess the main difference is the SPC. > However, SPC is a segment/shared and that memory is reserved from the > very beginning. So I don't expect that to change wether I reset the > stone or not. > > So...if it is not the SPC.... they are not the seaside gems, they are > not the admin/reclaim gc (those use very little temp space), what else > could be such a difference if memory consumption when I reset the stone? With reference to my recent reply on a different thread... If you're judging memory use by how much free memory the system has, you probably shouldn't. /proc/meminfo's MemAvailable or equivalent is a better indicator of whether your system is under memory pressure. That and how much swap activity is going on. It's normal to see free memory increase when you shut the stone down -- you've freed the entire SPC, among other things. When you shut down a gem, only the private memory that was used by that gem is freed, and most of the memory in a gem's RSS is shared with other processes or is cached file memory. Bottom line, stopping and starting a stone when your system is not under serious memory pressure just throws a bunch of stuff out of memory that you then have to read back in from disk on demand, slowing your system down. Regards, -Martin _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
«
Return to GLASS
|
1 view|%1 views
| Free forum by Nabble | Edit this page |