[Glass] Garbage Collection , maintenance gem and typical seaside app
Locked
[Glass] Garbage Collection , maintenance gem and typical seaside app
 
|
Hi GemStoners,
I tried not to pay attention that much in the first steps on my GemStone project. But now, it is time to investigate this. I have read all the chapters about GC, but I have a few questions.
From what I understand, I am running one admingcgem and one reclaimgcgem which start automatically with the stone. Since my repository/app is still small, I think that's enough for the moment.
Now, what what I understand, the reclaim part is worthless if there is no "mark" part. I do know I have #markForCollection, but the first question is: Does GemStone ever run internally/automatically a full mark pass? Like in Pharo..like besides having #garbageCollection, Pharo itself sometimes run the GC. So...GemStone does that or if I don't invoke it (as a sysadmin), it will never be?
GLASS comes with nice seaside scripts to start/stop gems and also a maintenance gem which calls WAGemStoneMaintenanceTask. This class, from what I can see, it does seaside session expiration and MFC. So....second question... if I have this VM running...then I wouldn't need to explicitly run a MFC as part of a cron job, right?
I have been facing/reading a problem in which some seaside gems (or maintenance VM) could hold into objects preventing them from being GCed. As a workaround, Dale suggested to reset gems every in a while or increase the GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE=90; which in my case it is already 90. So ok...imagine I have a daily script / cron that resets all seaside gems and maintenance gem. That's all I would have to do right? because soon after (less than 1 hours), my maintenance gem will run MFC and then at some point the reclaim GC will do their work. I am correct?
Even if that's try, if I run the "reset" of the seaside gems at a strange hour (like 4 am), I may want to even force a MFC and a #reclaimAll, right? Even if I know they would happen automatically in the future, I may prefer to pay its cost/time at 4 am. Would this make sense?
Finally...what's your opinion about epochGC for a typical Seaside app? I know there is a lot of trade-offs and it depends on the app...but as a general option, I found it strange to be disabled by default in GLASS.
Thanks for the clarification, Mariano http://marianopeck.wordpress.com _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: [Glass] Garbage Collection , maintenance gem and typical seaside app
|
|
Hi Mariano,
Good questions. In addition to reading all the chapters I’d suggest that you watch a few videos on GC (#19-22) at https://programminggems.wordpress.com/2010/02/05/scaling-objects-videos/. Further comments below... On May 8, 2014, at 8:24 AM, Mariano Martinez Peck <[hidden email]> wrote:
The reclaim gem(s) also consolidate objects from sparse pages, especially when objects are modified and the old versions are no longer referenced, so the reclaim part is not “worthless”!
No, this is resource intensive and left to the discretion of the system administrator.
_______________________________________________ _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
> GLASS comes with nice seaside scripts to start/stop gems and also a
> maintenance gem which calls WAGemStoneMaintenanceTask. This class, from what > I can see, it does seaside session expiration and MFC. So....second > question... if I have this VM running...then I wouldn't need to explicitly > run a MFC as part of a cron job, right? > > > Right. Sorry, I lied. The default maintenance30 implementation does an MFC every hour. We disabled that in our apps and switched the epoch GC on. I think the MFC is *extremely* heavyweight and will influence your application when it runs on a DB with some data. I can't really give a measure of how much "some" is, in other words when the DB is small enough where the MGC will not influence it that much. I understand that the purpose of the epoch GC is to be a process that runs in the background with loads of fancy parameters to throttle it. A seaside app in my understanding is an ideal candidate app for the epoch GC. We bypassed WAGemStoneMaintenanceTask >> maintenanceTaskMarkForCollect such a long time ago that I forgot about it. To me, it does not make sense to implement that complexity in maintenanceTaskMarkForCollect if there is a much more sophisticated and efficient tool around (epoch GC). _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
The MFC can be heavy-weight but is designed to be tunable so that it can
also be very light-weight at the expense of taking more time to complete. This is done by tuning the number of threads used by the MFC and how much CPU those threads are allowed to consume. For example, in 3.x, running the MFC with 1 thread and a cpu limit of 10% should have minimal impact on the system: SystemRepository markForCollectionWithMaxThreads: 1 waitForLock: 60 percentCpuActiveLimit: 10 Of course you have to be careful not to limit resources too much else the MFC may never finish. Norm On 5/8/14, 9:08, Otto Behrens wrote: >> GLASS comes with nice seaside scripts to start/stop gems and also a >> maintenance gem which calls WAGemStoneMaintenanceTask. This class, from what >> I can see, it does seaside session expiration and MFC. So....second >> question... if I have this VM running...then I wouldn't need to explicitly >> run a MFC as part of a cron job, right? >> >> >> Right. > Sorry, I lied. The default maintenance30 implementation does an MFC > every hour. We disabled that in our apps and switched the epoch GC on. > I think the MFC is *extremely* heavyweight and will influence your > application when it runs on a DB with some data. I can't really give a > measure of how much "some" is, in other words when the DB is small > enough where the MGC will not influence it that much. > > I understand that the purpose of the epoch GC is to be a process that > runs in the background with loads of fancy parameters to throttle it. > A seaside app in my understanding is an ideal candidate app for the > epoch GC. We bypassed WAGemStoneMaintenanceTask >> > maintenanceTaskMarkForCollect such a long time ago that I forgot about > it. To me, it does not make sense to implement that complexity in > maintenanceTaskMarkForCollect if there is a much more sophisticated > and efficient tool around (epoch GC). > _______________________________________________ > Glass mailing list > [hidden email] > http://lists.gemtalksystems.com/mailman/listinfo/glass _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: [Glass] Garbage Collection , maintenance gem and typical seaside app
|
|
In reply to this post by Mariano Martinez Peck
On Thu, May 8, 2014 at 8:24 AM, Mariano Martinez Peck <[hidden email]> wrote:
The whole GLASS setup has been aimed at _allowing_ you to defer these decisions until you need to:)
Agree with James ...
"Need to.." no, however, running an MFC once an hour can have a negative impact on performance while it is running especially if your repository size is significantly larger than your SPC. During an MFC "all" of the pages containing live objects are loaded into the SPC, which can end up pre-empting pages that your "normally running" application will have to load back into the SPC ... if you or your users notice that the system responding sluggishly during MFC, then it is time to consider moving the MFC from the maintenance vm to a separate cron job, so that you can control when the mfc runs and avoid having the MFC run during peak hours.
Technically the mfc runs immediately upon starting the maintenance vm (and you can change that if you want).
When you are running an MFC once an hour an epochGC is not needed. If you start running the MFC less frequently, then you want to keep an eye on repository growth (especially if using a limited repository sized key file or have your extents on disks without a lot of free space) ... If the repository is growing larger than you'd like between MFCs and you do not want to run the MFC more frequently, then the epochGC is the solution for you ...
Dale
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
In reply to this post by nrgiii
Thanks, so the MFC have become quite fancy. Very interesting.
So when do I use the MFC (eg from cron / maintenance process) and when do I use the epoch GC? On Thu, May 8, 2014 at 6:19 PM, Norm Green <[hidden email]> wrote: > The MFC can be heavy-weight but is designed to be tunable so that it can > also be very light-weight at the expense of taking more time to complete. > This is done by tuning the number of threads used by the MFC and how much > CPU those threads are allowed to consume. > > For example, in 3.x, running the MFC with 1 thread and a cpu limit of 10% > should have minimal impact on the system: > > SystemRepository markForCollectionWithMaxThreads: 1 waitForLock: 60 > percentCpuActiveLimit: 10 > > Of course you have to be careful not to limit resources too much else the > MFC may never finish. > > Norm > > > > > On 5/8/14, 9:08, Otto Behrens wrote: >>> >>> GLASS comes with nice seaside scripts to start/stop gems and also a >>> maintenance gem which calls WAGemStoneMaintenanceTask. This class, from >>> what >>> I can see, it does seaside session expiration and MFC. So....second >>> question... if I have this VM running...then I wouldn't need to >>> explicitly >>> run a MFC as part of a cron job, right? >>> >>> >>> Right. >> >> Sorry, I lied. The default maintenance30 implementation does an MFC >> every hour. We disabled that in our apps and switched the epoch GC on. >> I think the MFC is *extremely* heavyweight and will influence your >> application when it runs on a DB with some data. I can't really give a >> measure of how much "some" is, in other words when the DB is small >> enough where the MGC will not influence it that much. >> >> I understand that the purpose of the epoch GC is to be a process that >> runs in the background with loads of fancy parameters to throttle it. >> A seaside app in my understanding is an ideal candidate app for the >> epoch GC. We bypassed WAGemStoneMaintenanceTask >> >> maintenanceTaskMarkForCollect such a long time ago that I forgot about >> it. To me, it does not make sense to implement that complexity in >> maintenanceTaskMarkForCollect if there is a much more sophisticated >> and efficient tool around (epoch GC). >> _______________________________________________ >> Glass mailing list >> [hidden email] >> http://lists.gemtalksystems.com/mailman/listinfo/glass > > > _______________________________________________ > Glass mailing list > [hidden email] > http://lists.gemtalksystems.com/mailman/listinfo/glass Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
Well, it all depends on the usage pattern of the database. Epoch GC is
designed to cleanup short lived objects. MFC is the global cleanup. Running epochs hourly or daily and running MFC during low times (weekends or middle of the night) are common practices. Norm On 5/8/14, 9:32, Otto Behrens wrote: > Thanks, so the MFC have become quite fancy. Very interesting. > > So when do I use the MFC (eg from cron / maintenance process) and when > do I use the epoch GC? > > On Thu, May 8, 2014 at 6:19 PM, Norm Green > <[hidden email]> wrote: >> The MFC can be heavy-weight but is designed to be tunable so that it can >> also be very light-weight at the expense of taking more time to complete. >> This is done by tuning the number of threads used by the MFC and how much >> CPU those threads are allowed to consume. >> >> For example, in 3.x, running the MFC with 1 thread and a cpu limit of 10% >> should have minimal impact on the system: >> >> SystemRepository markForCollectionWithMaxThreads: 1 waitForLock: 60 >> percentCpuActiveLimit: 10 >> >> Of course you have to be careful not to limit resources too much else the >> MFC may never finish. >> >> Norm >> >> >> >> >> On 5/8/14, 9:08, Otto Behrens wrote: >>>> GLASS comes with nice seaside scripts to start/stop gems and also a >>>> maintenance gem which calls WAGemStoneMaintenanceTask. This class, from >>>> what >>>> I can see, it does seaside session expiration and MFC. So....second >>>> question... if I have this VM running...then I wouldn't need to >>>> explicitly >>>> run a MFC as part of a cron job, right? >>>> >>>> >>>> Right. >>> Sorry, I lied. The default maintenance30 implementation does an MFC >>> every hour. We disabled that in our apps and switched the epoch GC on. >>> I think the MFC is *extremely* heavyweight and will influence your >>> application when it runs on a DB with some data. I can't really give a >>> measure of how much "some" is, in other words when the DB is small >>> enough where the MGC will not influence it that much. >>> >>> I understand that the purpose of the epoch GC is to be a process that >>> runs in the background with loads of fancy parameters to throttle it. >>> A seaside app in my understanding is an ideal candidate app for the >>> epoch GC. We bypassed WAGemStoneMaintenanceTask >> >>> maintenanceTaskMarkForCollect such a long time ago that I forgot about >>> it. To me, it does not make sense to implement that complexity in >>> maintenanceTaskMarkForCollect if there is a much more sophisticated >>> and efficient tool around (epoch GC). >>> _______________________________________________ >>> Glass mailing list >>> [hidden email] >>> http://lists.gemtalksystems.com/mailman/listinfo/glass >> >> _______________________________________________ >> Glass mailing list >> [hidden email] >> http://lists.gemtalksystems.com/mailman/listinfo/glass _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: [Glass] Garbage Collection , maintenance gem and typical seaside app
|
|
In reply to this post by nrgiii
Norm makes a good point. If you do tweak the MFC to run at a lower rate, you should consider moving the MFC into a job separate from the maintenance vm so that the session expiration jobs can continue to run at regular rates ...
Dale On Thu, May 8, 2014 at 9:19 AM, Norm Green <[hidden email]> wrote: The MFC can be heavy-weight but is designed to be tunable so that it can also be very light-weight at the expense of taking more time to complete. This is done by tuning the number of threads used by the MFC and how much CPU those threads are allowed to consume. _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: [Glass] Garbage Collection , maintenance gem and typical seaside app
|
|
In reply to this post by nrgiii
For Seaside we know that there is a lot of persistent session state that expires every 10 minutes (or whatever your expiry is) and is naturally short-lived. This is all about managing repository size and we know that there can be significant repository growth due to session state (the magnitude of the growth is dependent upon the number of requests per second)... so the frequency of the epoch should be based upon how much repository growth you are willing to live with ...
Epoch gc is not 100% effective in cleaning up short-lived objects. Those objects whose lifetimes span the epoch boundaries are not going to be collected by epoch gc, so the longer the period between epochs the higher your effective rate.
For SS3 I think I run an MFC twice a day and run an epoch every hour and for that site, the repository size stays pretty steady state ... As a final note, you want to keep an eye on your Object Log ... if you have a lot of continuations in your object log, you end keeping an even larger number of objects alive ... it is a good practice to remove the continuations from your object log on a regular basis or simply clear the object log on a regular basis...
Dale On Thu, May 8, 2014 at 9:37 AM, Norm Green <[hidden email]> wrote: Well, it all depends on the usage pattern of the database. Epoch GC is designed to cleanup short lived objects. MFC is the global cleanup. _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: [Glass] Garbage Collection , maintenance gem and typical seaside app
|
|
In reply to this post by nrgiii
On Thu, May 8, 2014 at 1:37 PM, Norm Green <[hidden email]> wrote: Well, it all depends on the usage pattern of the database. Epoch GC is designed to cleanup short lived objects. MFC is the global cleanup. This sounds very much like the "incremental GC" of Pharo and the full one (#garbageCollect). Running epochs hourly or daily and running MFC during low times (weekends or middle of the night) are common practices. Indeed, after having read all this thread, I think that enabling epoch and running MFC at the middle of the night or weekends looks like the path I will follow next (when the MFC per hour of the maintenance VM starts to bother the app users).
Thanks for all the suggestions. Norm Mariano http://marianopeck.wordpress.com _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: [Glass] Garbage Collection , maintenance gem and typical seaside app
|
|
In reply to this post by Mariano Martinez Peck
How do you do that Otto? I tried a simple script like this: [ [
System beginTransaction. System stopUserSessions. SystemRepository markForCollection.
SystemRepository reclaimAll. System commitTransaction.
System resumeLogins. ] on: Halt, Warning do: [:ex | ex resume] ] on: Error do: [:ex2 | ex2 |
System resumeLogins. ex2 pass. ] But #markForCollection complains it cannot log in. So how can I let markForCollection to work yet disable/kill other sessions? Thanks, _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
>> We do this, yes. We run stopUserSessions as well while this is running.
>> > > How do you do that Otto? I tried a simple script like this: > > > [ > [ > System beginTransaction. > System stopUserSessions. > SystemRepository markForCollection. > SystemRepository reclaimAll. > System commitTransaction. > System resumeLogins. > > ] on: Halt, Warning do: [:ex | ex resume] > ] on: Error do: [:ex2 | ex2 | > System resumeLogins. > ex2 pass. > ] > > But #markForCollection complains it cannot log in. So how can I let > markForCollection to work yet disable/kill other sessions? Firstly, we make sure that our "service processes" (Fast CGI, maintenance, others) are stopped. Then we run stopUserSessions in a separate topaz session to make sure nobody else is logged in. We then start a new topaz session to do the markForCollection & reclaimAll. There are race conditions where a process can slip in (via cron or something else), so I don't think what we have is perfect. I also think we should be calling suspendLogins just before MFC & reclaimAll, which is supposed to suspend other logins (besides our current process). Sorry, things changed a bit between versions and I'm not sure what the best way is to get this right. My intention is to make sure as few processes are logged in as possible so that the MFC is as efficient and complete as possible (sessions logged in may just misbehave and hold onto things that they are not supposed to). Sessions logged in concurrently have to vote before objects are promoted to possibly dead. We do this once a day and take a full backup after this. _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
|
|
In reply to this post by Dale Henrichs-3
Thanks, Dale.
> For Seaside we know that there is a lot of persistent session state that > expires every 10 minutes (or whatever your expiry is) and is naturally > short-lived. > > This is all about managing repository size and we know that there can be > significant repository growth due to session state (the magnitude of the > growth is dependent upon the number of requests per second)... so the > frequency of the epoch should be based upon how much repository growth you > are willing to live with ... > > Epoch gc is not 100% effective in cleaning up short-lived objects. Those > objects whose lifetimes span the epoch boundaries are not going to be > collected by epoch gc, so the longer the period between epochs the higher > your effective rate. Yes, I think this is the most sensible setup since the expired session state objects are short lived. We find that it is effective enough as it cleans out most of the session state garbage. > For SS3 I think I run an MFC twice a day and run an epoch every hour and for > that site, the repository size stays pretty steady state ... > > As a final note, you want to keep an eye on your Object Log ... if you have > a lot of continuations in your object log, you end keeping an even larger > number of objects alive ... it is a good practice to remove the > continuations from your object log on a regular basis or simply clear the > object log on a regular basis... Thanks, will have a look at a few of these things. _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: [Glass] Garbage Collection , maintenance gem and typical seaside app
|
|
In reply to this post by otto
Otto, While the MFC is designed to be able to do it's job without having to shut down parts of the system, you do have a point that with all of the "service processes" shutdown, you will get maximum yield from the MFC and none of the dead should be voted down ...
If you are required to keep your system on-line 24x7 then you do have to live with a little slop with regards to garbage collection... Dale On Fri, May 9, 2014 at 12:29 AM, Otto Behrens <[hidden email]> wrote:
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
«
Return to GLASS
|
1 view|%1 views
| Free forum by Nabble | Edit this page |