[Glass] Garbage Collection , maintenance gem and typical seaside app

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

[Glass] Garbage Collection , maintenance gem and typical seaside app

Mariano Martinez Peck
Hi GemStoners,

I tried not to pay attention that much in the first steps on my GemStone project. But now, it is time to investigate this. I have read all the chapters about GC, but I have a few questions. 

From what I understand, I am running one admingcgem and one reclaimgcgem which start automatically with the stone. Since my repository/app is still small, I think that's enough for the moment. 

Now, what what I understand, the reclaim part is worthless if there is no "mark" part. I do know I have #markForCollection, but the first question is:  Does GemStone ever run internally/automatically a full mark pass? Like in Pharo..like besides having #garbageCollection, Pharo itself sometimes run the GC. So...GemStone does that or if I don't invoke it (as a sysadmin), it will never be? 

GLASS comes with nice seaside scripts to start/stop gems and also a maintenance gem which calls WAGemStoneMaintenanceTask. This class, from what I can see, it does seaside session expiration and MFC. So....second question... if I have this VM running...then I wouldn't need to explicitly run a MFC as part of a cron job, right?

I have been facing/reading a problem in which some seaside gems (or maintenance VM) could hold into objects preventing them from being GCed. As a workaround, Dale suggested to reset gems every in a while or increase the  GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE=90; which in my case it is already 90. So ok...imagine I have a daily script / cron that resets all seaside gems and maintenance gem. That's all I would have to do right? because soon after (less than 1 hours), my maintenance gem will run MFC and then at some point the reclaim GC will do their work. I am correct?
Even if that's try, if I run the "reset" of the seaside gems at a strange hour (like 4 am), I may want to even force a MFC and a #reclaimAll, right? Even if I know they would happen automatically in the future, I may prefer to pay its cost/time at 4 am. Would this make sense?

Finally...what's your opinion about epochGC for a typical Seaside app? I know there is a lot of trade-offs and it depends on the app...but as a general option, I found it strange to be disabled by default in GLASS. 

Thanks for the clarification, 


--
Mariano
http://marianopeck.wordpress.com

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Garbage Collection , maintenance gem and typical seaside app

James Foster-9
Hi Mariano,

Good questions. In addition to reading all the chapters I’d suggest that you watch a few videos on GC (#19-22) at https://programminggems.wordpress.com/2010/02/05/scaling-objects-videos/. Further comments below...

On May 8, 2014, at 8:24 AM, Mariano Martinez Peck <[hidden email]> wrote:

Hi GemStoners,

I tried not to pay attention that much in the first steps on my GemStone project. But now, it is time to investigate this. I have read all the chapters about GC, but I have a few questions. 

From what I understand, I am running one admingcgem and one reclaimgcgem which start automatically with the stone. Since my repository/app is still small, I think that's enough for the moment. 

Now, what what I understand, the reclaim part is worthless if there is no "mark" part.

The reclaim gem(s) also consolidate objects from sparse pages, especially when objects are modified and the old versions are no longer referenced, so the reclaim part is not “worthless”!

I do know I have #markForCollection, but the first question is:  Does GemStone ever run internally/automatically a full mark pass?

No, this is resource intensive and left to the discretion of the system administrator.

Like in Pharo..like besides having #garbageCollection, Pharo itself sometimes run the GC. So...GemStone does that or if I don't invoke it (as a sysadmin), it will never be? 

It will never be invoked automatically.

GLASS comes with nice seaside scripts to start/stop gems and also a maintenance gem which calls WAGemStoneMaintenanceTask. This class, from what I can see, it does seaside session expiration and MFC. So....second question... if I have this VM running...then I wouldn't need to explicitly run a MFC as part of a cron job, right?

Right.

I have been facing/reading a problem in which some seaside gems (or maintenance VM) could hold into objects preventing them from being GCed. As a workaround, Dale suggested to reset gems every in a while or increase the  GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE=90; which in my case it is already 90. So ok...imagine I have a daily script / cron that resets all seaside gems and maintenance gem. That's all I would have to do right? because soon after (less than 1 hours), my maintenance gem will run MFC and then at some point the reclaim GC will do their work. I am correct?
Even if that's try, if I run the "reset" of the seaside gems at a strange hour (like 4 am), I may want to even force a MFC and a #reclaimAll, right? Even if I know they would happen automatically in the future, I may prefer to pay its cost/time at 4 am. Would this make sense?

Yes, this is a common approach.

Finally...what's your opinion about epochGC for a typical Seaside app? I know there is a lot of trade-offs and it depends on the app...but as a general option, I found it strange to be disabled by default in GLASS. 

By default the MFC runs frequently, so there is no need for the EpochGC.

Thanks for the clarification, 


--
Mariano
http://marianopeck.wordpress.com
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Garbage Collection , maintenance gem and typical seaside app

otto
> GLASS comes with nice seaside scripts to start/stop gems and also a
> maintenance gem which calls WAGemStoneMaintenanceTask. This class, from what
> I can see, it does seaside session expiration and MFC. So....second
> question... if I have this VM running...then I wouldn't need to explicitly
> run a MFC as part of a cron job, right?
>
>
> Right.

Sorry, I lied. The default maintenance30 implementation does an MFC
every hour. We disabled that in our apps and switched the epoch GC on.
I think the MFC is *extremely* heavyweight and will influence your
application when it runs on a DB with some data. I can't really give a
measure of how much "some" is, in other words when the DB is small
enough where the MGC will not influence it that much.

I understand that the purpose of the epoch GC is to be a process that
runs in the background with loads of fancy parameters to throttle it.
A seaside app in my understanding is an ideal candidate app for the
epoch GC. We bypassed WAGemStoneMaintenanceTask >>
maintenanceTaskMarkForCollect such a long time ago that I forgot about
it. To me, it does not make sense to implement that complexity in
maintenanceTaskMarkForCollect if there is a much more sophisticated
and efficient tool around (epoch GC).
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Garbage Collection , maintenance gem and typical seaside app

nrgiii
The MFC can be heavy-weight but is designed to be tunable so that it can
also be very light-weight at the expense of taking more time to
complete.  This is done by tuning the number of threads used by the MFC
and how much CPU those threads are allowed to consume.

For example, in 3.x, running the MFC with 1 thread and a cpu limit of
10% should have minimal impact on the system:

SystemRepository markForCollectionWithMaxThreads: 1 waitForLock: 60
percentCpuActiveLimit: 10

Of course you have to be careful not to limit resources too much else
the MFC may never finish.

Norm



On 5/8/14, 9:08, Otto Behrens wrote:

>> GLASS comes with nice seaside scripts to start/stop gems and also a
>> maintenance gem which calls WAGemStoneMaintenanceTask. This class, from what
>> I can see, it does seaside session expiration and MFC. So....second
>> question... if I have this VM running...then I wouldn't need to explicitly
>> run a MFC as part of a cron job, right?
>>
>>
>> Right.
> Sorry, I lied. The default maintenance30 implementation does an MFC
> every hour. We disabled that in our apps and switched the epoch GC on.
> I think the MFC is *extremely* heavyweight and will influence your
> application when it runs on a DB with some data. I can't really give a
> measure of how much "some" is, in other words when the DB is small
> enough where the MGC will not influence it that much.
>
> I understand that the purpose of the epoch GC is to be a process that
> runs in the background with loads of fancy parameters to throttle it.
> A seaside app in my understanding is an ideal candidate app for the
> epoch GC. We bypassed WAGemStoneMaintenanceTask >>
> maintenanceTaskMarkForCollect such a long time ago that I forgot about
> it. To me, it does not make sense to implement that complexity in
> maintenanceTaskMarkForCollect if there is a much more sophisticated
> and efficient tool around (epoch GC).
> _______________________________________________
> Glass mailing list
> [hidden email]
> http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Garbage Collection , maintenance gem and typical seaside app

Dale Henrichs-3
In reply to this post by Mariano Martinez Peck



On Thu, May 8, 2014 at 8:24 AM, Mariano Martinez Peck <[hidden email]> wrote:
Hi GemStoners,

I tried not to pay attention that much in the first steps on my GemStone project. But now, it is time to investigate this. I have read all the chapters about GC, but I have a few questions. 

The whole GLASS setup has been aimed at _allowing_ you to defer these decisions until you need to:)
 

From what I understand, I am running one admingcgem and one reclaimgcgem which start automatically with the stone. Since my repository/app is still small, I think that's enough for the moment. 

Now, what what I understand, the reclaim part is worthless if there is no "mark" part. I do know I have #markForCollection, but the first question is:  Does GemStone ever run internally/automatically a full mark pass? Like in Pharo..like besides having #garbageCollection, Pharo itself sometimes run the GC. So...GemStone does that or if I don't invoke it (as a sysadmin), it will never be? 

Agree with James ...   

GLASS comes with nice seaside scripts to start/stop gems and also a maintenance gem which calls WAGemStoneMaintenanceTask. This class, from what I can see, it does seaside session expiration and MFC. So....second question... if I have this VM running...then I wouldn't need to explicitly run a MFC as part of a cron job, right?

"Need to.." no, however, running an MFC once an hour can have a negative impact on performance while it is running especially if your repository size is significantly larger  than your SPC. During an MFC "all" of the pages containing live objects are loaded into the SPC, which can end up pre-empting pages that your "normally running" application will have to load back into the SPC ... if you or your users notice that the system responding sluggishly during MFC, then it is time to consider moving the MFC from the maintenance vm to a separate cron job, so that you can control when the mfc runs and avoid having the MFC run during peak hours.

I have been facing/reading a problem in which some seaside gems (or maintenance VM) could hold into objects preventing them from being GCed. As a workaround, Dale suggested to reset gems every in a while or increase the  GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE=90; which in my case it is already 90. So ok...imagine I have a daily script / cron that resets all seaside gems and maintenance gem. That's all I would have to do right? because soon after (less than 1 hours), my maintenance gem will run MFC and then at some point the reclaim GC will do their work. I am correct?
Even if that's try, if I run the "reset" of the seaside gems at a strange hour (like 4 am), I may want to even force a MFC and a #reclaimAll, right? Even if I know they would happen automatically in the future, I may prefer to pay its cost/time at 4 am. Would this make sense?

Technically the mfc runs immediately upon starting the maintenance vm (and you can change that if you want).
 

Finally...what's your opinion about epochGC for a typical Seaside app? I know there is a lot of trade-offs and it depends on the app...but as a general option, I found it strange to be disabled by default in GLASS. 

When you are running an MFC once an hour an epochGC is not needed. If you start running the MFC less frequently, then you want to keep an eye on repository growth (especially if using a limited repository sized key file or have your extents on disks without a lot of free space) ... If the repository is growing larger than you'd like between MFCs and you do not want to run the MFC more frequently, then the epochGC is the solution for you ...

Dale

Thanks for the clarification, 


--
Mariano
http://marianopeck.wordpress.com

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass



_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Garbage Collection , maintenance gem and typical seaside app

otto
In reply to this post by nrgiii
Thanks, so the MFC have become quite fancy. Very interesting.

So when do I use the MFC (eg from cron / maintenance process) and when
do I use the epoch GC?

On Thu, May 8, 2014 at 6:19 PM, Norm Green
<[hidden email]> wrote:

> The MFC can be heavy-weight but is designed to be tunable so that it can
> also be very light-weight at the expense of taking more time to complete.
> This is done by tuning the number of threads used by the MFC and how much
> CPU those threads are allowed to consume.
>
> For example, in 3.x, running the MFC with 1 thread and a cpu limit of 10%
> should have minimal impact on the system:
>
> SystemRepository markForCollectionWithMaxThreads: 1 waitForLock: 60
> percentCpuActiveLimit: 10
>
> Of course you have to be careful not to limit resources too much else the
> MFC may never finish.
>
> Norm
>
>
>
>
> On 5/8/14, 9:08, Otto Behrens wrote:
>>>
>>> GLASS comes with nice seaside scripts to start/stop gems and also a
>>> maintenance gem which calls WAGemStoneMaintenanceTask. This class, from
>>> what
>>> I can see, it does seaside session expiration and MFC. So....second
>>> question... if I have this VM running...then I wouldn't need to
>>> explicitly
>>> run a MFC as part of a cron job, right?
>>>
>>>
>>> Right.
>>
>> Sorry, I lied. The default maintenance30 implementation does an MFC
>> every hour. We disabled that in our apps and switched the epoch GC on.
>> I think the MFC is *extremely* heavyweight and will influence your
>> application when it runs on a DB with some data. I can't really give a
>> measure of how much "some" is, in other words when the DB is small
>> enough where the MGC will not influence it that much.
>>
>> I understand that the purpose of the epoch GC is to be a process that
>> runs in the background with loads of fancy parameters to throttle it.
>> A seaside app in my understanding is an ideal candidate app for the
>> epoch GC. We bypassed WAGemStoneMaintenanceTask >>
>> maintenanceTaskMarkForCollect such a long time ago that I forgot about
>> it. To me, it does not make sense to implement that complexity in
>> maintenanceTaskMarkForCollect if there is a much more sophisticated
>> and efficient tool around (epoch GC).
>> _______________________________________________
>> Glass mailing list
>> [hidden email]
>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>
>
> _______________________________________________
> Glass mailing list
> [hidden email]
> http://lists.gemtalksystems.com/mailman/listinfo/glass
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Garbage Collection , maintenance gem and typical seaside app

nrgiii
Well, it all depends on the usage pattern of the database.  Epoch GC is
designed to cleanup short lived objects.  MFC is the global cleanup.

Running epochs hourly or daily and running MFC during low times
(weekends or middle of the night) are common practices.

Norm

On 5/8/14, 9:32, Otto Behrens wrote:

> Thanks, so the MFC have become quite fancy. Very interesting.
>
> So when do I use the MFC (eg from cron / maintenance process) and when
> do I use the epoch GC?
>
> On Thu, May 8, 2014 at 6:19 PM, Norm Green
> <[hidden email]> wrote:
>> The MFC can be heavy-weight but is designed to be tunable so that it can
>> also be very light-weight at the expense of taking more time to complete.
>> This is done by tuning the number of threads used by the MFC and how much
>> CPU those threads are allowed to consume.
>>
>> For example, in 3.x, running the MFC with 1 thread and a cpu limit of 10%
>> should have minimal impact on the system:
>>
>> SystemRepository markForCollectionWithMaxThreads: 1 waitForLock: 60
>> percentCpuActiveLimit: 10
>>
>> Of course you have to be careful not to limit resources too much else the
>> MFC may never finish.
>>
>> Norm
>>
>>
>>
>>
>> On 5/8/14, 9:08, Otto Behrens wrote:
>>>> GLASS comes with nice seaside scripts to start/stop gems and also a
>>>> maintenance gem which calls WAGemStoneMaintenanceTask. This class, from
>>>> what
>>>> I can see, it does seaside session expiration and MFC. So....second
>>>> question... if I have this VM running...then I wouldn't need to
>>>> explicitly
>>>> run a MFC as part of a cron job, right?
>>>>
>>>>
>>>> Right.
>>> Sorry, I lied. The default maintenance30 implementation does an MFC
>>> every hour. We disabled that in our apps and switched the epoch GC on.
>>> I think the MFC is *extremely* heavyweight and will influence your
>>> application when it runs on a DB with some data. I can't really give a
>>> measure of how much "some" is, in other words when the DB is small
>>> enough where the MGC will not influence it that much.
>>>
>>> I understand that the purpose of the epoch GC is to be a process that
>>> runs in the background with loads of fancy parameters to throttle it.
>>> A seaside app in my understanding is an ideal candidate app for the
>>> epoch GC. We bypassed WAGemStoneMaintenanceTask >>
>>> maintenanceTaskMarkForCollect such a long time ago that I forgot about
>>> it. To me, it does not make sense to implement that complexity in
>>> maintenanceTaskMarkForCollect if there is a much more sophisticated
>>> and efficient tool around (epoch GC).
>>> _______________________________________________
>>> Glass mailing list
>>> [hidden email]
>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>
>> _______________________________________________
>> Glass mailing list
>> [hidden email]
>> http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Garbage Collection , maintenance gem and typical seaside app

Dale Henrichs-3
In reply to this post by nrgiii
Norm makes a good point. 

If you do tweak the MFC to run at a lower rate, you should consider moving the MFC into a job separate from the maintenance vm so that the session expiration jobs can continue to run at regular rates ...

Dale


On Thu, May 8, 2014 at 9:19 AM, Norm Green <[hidden email]> wrote:
The MFC can be heavy-weight but is designed to be tunable so that it can also be very light-weight at the expense of taking more time to complete.  This is done by tuning the number of threads used by the MFC and how much CPU those threads are allowed to consume.

For example, in 3.x, running the MFC with 1 thread and a cpu limit of 10% should have minimal impact on the system:

SystemRepository markForCollectionWithMaxThreads: 1 waitForLock: 60 percentCpuActiveLimit: 10

Of course you have to be careful not to limit resources too much else the MFC may never finish.

Norm




On 5/8/14, 9:08, Otto Behrens wrote:
GLASS comes with nice seaside scripts to start/stop gems and also a
maintenance gem which calls WAGemStoneMaintenanceTask. This class, from what
I can see, it does seaside session expiration and MFC. So....second
question... if I have this VM running...then I wouldn't need to explicitly
run a MFC as part of a cron job, right?


Right.
Sorry, I lied. The default maintenance30 implementation does an MFC
every hour. We disabled that in our apps and switched the epoch GC on.
I think the MFC is *extremely* heavyweight and will influence your
application when it runs on a DB with some data. I can't really give a
measure of how much "some" is, in other words when the DB is small
enough where the MGC will not influence it that much.

I understand that the purpose of the epoch GC is to be a process that
runs in the background with loads of fancy parameters to throttle it.
A seaside app in my understanding is an ideal candidate app for the
epoch GC. We bypassed WAGemStoneMaintenanceTask >>
maintenanceTaskMarkForCollect such a long time ago that I forgot about
it. To me, it does not make sense to implement that complexity in
maintenanceTaskMarkForCollect if there is a much more sophisticated
and efficient tool around (epoch GC).
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Garbage Collection , maintenance gem and typical seaside app

Dale Henrichs-3
In reply to this post by nrgiii
For Seaside we know that there is a lot of persistent session state that expires every 10 minutes (or whatever your expiry is) and is naturally short-lived. 

This is all about managing repository size and we know that there can be significant repository growth due to session state (the magnitude of the growth is dependent upon the number of requests per second)... so the frequency of the epoch should be based upon how much repository growth you are willing to live with ... 

Epoch gc is not 100% effective in cleaning up short-lived objects. Those objects whose lifetimes span the epoch boundaries are not going to be collected by epoch gc, so the longer the period between epochs the higher your effective rate.

For SS3 I think I run an MFC twice a day and run an epoch every hour and for that site, the repository size stays pretty steady state ...

As a final note, you want to keep an eye on your Object Log ... if you have a lot of continuations in your object log, you end keeping an even larger number of objects alive ... it is a good practice to  remove the continuations from your object log on a regular basis or simply clear the object log on a regular basis...

Dale


On Thu, May 8, 2014 at 9:37 AM, Norm Green <[hidden email]> wrote:
Well, it all depends on the usage pattern of the database.  Epoch GC is designed to cleanup short lived objects.  MFC is the global cleanup.

Running epochs hourly or daily and running MFC during low times (weekends or middle of the night) are common practices.

Norm


On 5/8/14, 9:32, Otto Behrens wrote:
Thanks, so the MFC have become quite fancy. Very interesting.

So when do I use the MFC (eg from cron / maintenance process) and when
do I use the epoch GC?

On Thu, May 8, 2014 at 6:19 PM, Norm Green
<[hidden email]> wrote:
The MFC can be heavy-weight but is designed to be tunable so that it can
also be very light-weight at the expense of taking more time to complete.
This is done by tuning the number of threads used by the MFC and how much
CPU those threads are allowed to consume.

For example, in 3.x, running the MFC with 1 thread and a cpu limit of 10%
should have minimal impact on the system:

SystemRepository markForCollectionWithMaxThreads: 1 waitForLock: 60
percentCpuActiveLimit: 10

Of course you have to be careful not to limit resources too much else the
MFC may never finish.

Norm




On 5/8/14, 9:08, Otto Behrens wrote:
GLASS comes with nice seaside scripts to start/stop gems and also a
maintenance gem which calls WAGemStoneMaintenanceTask. This class, from
what
I can see, it does seaside session expiration and MFC. So....second
question... if I have this VM running...then I wouldn't need to
explicitly
run a MFC as part of a cron job, right?


Right.
Sorry, I lied. The default maintenance30 implementation does an MFC
every hour. We disabled that in our apps and switched the epoch GC on.
I think the MFC is *extremely* heavyweight and will influence your
application when it runs on a DB with some data. I can't really give a
measure of how much "some" is, in other words when the DB is small
enough where the MGC will not influence it that much.

I understand that the purpose of the epoch GC is to be a process that
runs in the background with loads of fancy parameters to throttle it.
A seaside app in my understanding is an ideal candidate app for the
epoch GC. We bypassed WAGemStoneMaintenanceTask >>
maintenanceTaskMarkForCollect such a long time ago that I forgot about
it. To me, it does not make sense to implement that complexity in
maintenanceTaskMarkForCollect if there is a much more sophisticated
and efficient tool around (epoch GC).
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Garbage Collection , maintenance gem and typical seaside app

Mariano Martinez Peck
In reply to this post by nrgiii



On Thu, May 8, 2014 at 1:37 PM, Norm Green <[hidden email]> wrote:
Well, it all depends on the usage pattern of the database.  Epoch GC is designed to cleanup short lived objects.  MFC is the global cleanup.


This sounds very much like the "incremental GC" of Pharo and the full one (#garbageCollect). 
 
Running epochs hourly or daily and running MFC during low times (weekends or middle of the night) are common practices.


Indeed, after having read all this thread, I think that enabling epoch and running MFC at the middle of the night or weekends looks like the path I will follow next (when the MFC per hour of the maintenance VM starts to bother the app users). 

Thanks for all the suggestions. 
 


Norm


On 5/8/14, 9:32, Otto Behrens wrote:
Thanks, so the MFC have become quite fancy. Very interesting.

So when do I use the MFC (eg from cron / maintenance process) and when
do I use the epoch GC?

On Thu, May 8, 2014 at 6:19 PM, Norm Green
<[hidden email]> wrote:
The MFC can be heavy-weight but is designed to be tunable so that it can
also be very light-weight at the expense of taking more time to complete.
This is done by tuning the number of threads used by the MFC and how much
CPU those threads are allowed to consume.

For example, in 3.x, running the MFC with 1 thread and a cpu limit of 10%
should have minimal impact on the system:

SystemRepository markForCollectionWithMaxThreads: 1 waitForLock: 60
percentCpuActiveLimit: 10

Of course you have to be careful not to limit resources too much else the
MFC may never finish.

Norm




On 5/8/14, 9:08, Otto Behrens wrote:
GLASS comes with nice seaside scripts to start/stop gems and also a
maintenance gem which calls WAGemStoneMaintenanceTask. This class, from
what
I can see, it does seaside session expiration and MFC. So....second
question... if I have this VM running...then I wouldn't need to
explicitly
run a MFC as part of a cron job, right?


Right.
Sorry, I lied. The default maintenance30 implementation does an MFC
every hour. We disabled that in our apps and switched the epoch GC on.
I think the MFC is *extremely* heavyweight and will influence your
application when it runs on a DB with some data. I can't really give a
measure of how much "some" is, in other words when the DB is small
enough where the MGC will not influence it that much.

I understand that the purpose of the epoch GC is to be a process that
runs in the background with loads of fancy parameters to throttle it.
A seaside app in my understanding is an ideal candidate app for the
epoch GC. We bypassed WAGemStoneMaintenanceTask >>
maintenanceTaskMarkForCollect such a long time ago that I forgot about
it. To me, it does not make sense to implement that complexity in
maintenanceTaskMarkForCollect if there is a much more sophisticated
and efficient tool around (epoch GC).
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass



--
Mariano
http://marianopeck.wordpress.com

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Garbage Collection , maintenance gem and typical seaside app

Mariano Martinez Peck
In reply to this post by Mariano Martinez Peck


> Even if that's try, if I run the "reset" of the seaside gems at a strange
> hour (like 4 am), I may want to even force a MFC and a #reclaimAll, right?
> Even if I know they would happen automatically in the future, I may prefer
> to pay its cost/time at 4 am. Would this make sense?

We do this, yes. We run stopUserSessions as well while this is running.


How do you do that Otto?  I tried a simple script like this:


[
[
System beginTransaction.
System stopUserSessions.
SystemRepository markForCollection.
SystemRepository reclaimAll.
System commitTransaction.
System resumeLogins.

] on: Halt,  Warning do: [:ex | ex resume]
] on: Error do: [:ex2 | ex2 | 
System resumeLogins.
ex2 pass.
]

But #markForCollection complains it cannot log in. So how can I let markForCollection to work yet disable/kill other sessions?

Thanks, 
 

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Garbage Collection , maintenance gem and typical seaside app

otto
>> We do this, yes. We run stopUserSessions as well while this is running.
>>
>
> How do you do that Otto?  I tried a simple script like this:
>
>
> [
> [
> System beginTransaction.
> System stopUserSessions.
> SystemRepository markForCollection.
> SystemRepository reclaimAll.
> System commitTransaction.
> System resumeLogins.
>
> ] on: Halt,  Warning do: [:ex | ex resume]
> ] on: Error do: [:ex2 | ex2 |
> System resumeLogins.
> ex2 pass.
> ]
>
> But #markForCollection complains it cannot log in. So how can I let
> markForCollection to work yet disable/kill other sessions?

Firstly, we make sure that our "service processes" (Fast CGI,
maintenance, others) are stopped. Then we run stopUserSessions in a
separate topaz session to make sure nobody else is logged in. We then
start a new topaz session to do the markForCollection & reclaimAll.

There are race conditions where a process can slip in (via cron or
something else), so I don't think what we have is perfect. I also
think we should be calling suspendLogins just before MFC & reclaimAll,
which is supposed to suspend other logins (besides our current
process).

Sorry, things changed a bit between versions and I'm not sure what the
best way is to get this right. My intention is to make sure as few
processes are logged in as possible so that the MFC is as efficient
and complete as possible (sessions logged in may just misbehave and
hold onto things that they are not supposed to). Sessions logged in
concurrently have to vote before objects are promoted to possibly
dead.

We do this once a day and take a full backup after this.
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Garbage Collection , maintenance gem and typical seaside app

otto
In reply to this post by Dale Henrichs-3
Thanks, Dale.

> For Seaside we know that there is a lot of persistent session state that
> expires every 10 minutes (or whatever your expiry is) and is naturally
> short-lived.
>
> This is all about managing repository size and we know that there can be
> significant repository growth due to session state (the magnitude of the
> growth is dependent upon the number of requests per second)... so the
> frequency of the epoch should be based upon how much repository growth you
> are willing to live with ...
>
> Epoch gc is not 100% effective in cleaning up short-lived objects. Those
> objects whose lifetimes span the epoch boundaries are not going to be
> collected by epoch gc, so the longer the period between epochs the higher
> your effective rate.

Yes, I think this is the most sensible setup since the expired session
state objects are short lived. We find that it is effective enough as
it cleans out most of the session state garbage.

> For SS3 I think I run an MFC twice a day and run an epoch every hour and for
> that site, the repository size stays pretty steady state ...
>
> As a final note, you want to keep an eye on your Object Log ... if you have
> a lot of continuations in your object log, you end keeping an even larger
> number of objects alive ... it is a good practice to  remove the
> continuations from your object log on a regular basis or simply clear the
> object log on a regular basis...

Thanks, will have a look at a few of these things.
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] Garbage Collection , maintenance gem and typical seaside app

Dale Henrichs-3
In reply to this post by otto
Otto,

While the MFC is designed to be able to do it's job without having to shut down parts of the system, you do have a point that with all of the "service processes" shutdown, you will get maximum yield from the MFC and none of the dead should be voted down ... 

If you are required to keep your system on-line 24x7 then you do have to live with a little slop with regards to garbage collection...

Dale


On Fri, May 9, 2014 at 12:29 AM, Otto Behrens <[hidden email]> wrote:
>> We do this, yes. We run stopUserSessions as well while this is running.
>>
>
> How do you do that Otto?  I tried a simple script like this:
>
>
> [
> [
> System beginTransaction.
> System stopUserSessions.
> SystemRepository markForCollection.
> SystemRepository reclaimAll.
> System commitTransaction.
> System resumeLogins.
>
> ] on: Halt,  Warning do: [:ex | ex resume]
> ] on: Error do: [:ex2 | ex2 |
> System resumeLogins.
> ex2 pass.
> ]
>
> But #markForCollection complains it cannot log in. So how can I let
> markForCollection to work yet disable/kill other sessions?

Firstly, we make sure that our "service processes" (Fast CGI,
maintenance, others) are stopped. Then we run stopUserSessions in a
separate topaz session to make sure nobody else is logged in. We then
start a new topaz session to do the markForCollection & reclaimAll.

There are race conditions where a process can slip in (via cron or
something else), so I don't think what we have is perfect. I also
think we should be calling suspendLogins just before MFC & reclaimAll,
which is supposed to suspend other logins (besides our current
process).

Sorry, things changed a bit between versions and I'm not sure what the
best way is to get this right. My intention is to make sure as few
processes are logged in as possible so that the MFC is as efficient
and complete as possible (sessions logged in may just misbehave and
hold onto things that they are not supposed to). Sessions logged in
concurrently have to vote before objects are promoted to possibly
dead.

We do this once a day and take a full backup after this.
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass