Reduce SPC pressure by keeping Seaside session state outside of the repository?

Hello,

Sorry for the long email, but the bottom line is: would it make sense to keep seaside session state as gem session state? And would that be feasible?

I've been monitoring the behavior of the SPC over our daily runs, especially with respect to free frames and objects that got faulted-in from disk when the SPC started throwing objects out under pressure from the free frame list. In short: I _think_ I notice that under pressure from the persistent seaside session state, that domain model objects have to be re-read from disk. In other words: domain model objects are preempted and thrown out of the shared page cache.

I did the monitoring using the hidden set that is created as mentioned in System>>_enableObjectsRead. This method only shows the objects that are faulted into VM memory (and not the SPC) but the given combination of seeing disk access happening during the launching of a new seaside session, the fact that the statmonit graph shows a flattening line for free frames and the fact that the contents of the hidden set lead me to think that way. I assume a new Seaside session does not need references to past Seaside sessions and therefore any objects being read from disk should be related to the objects it needs from the domain model. (Note that no 'FrameFromFindFree is occurring at all).

But I'm a quite amazed at the fact that the SPC appears to be throwing out my domain model objects. They are outnumbered by the amount of objects of the Seaside session state (see table below) but they should be referenced quite often and be kept in the VM memory (and the SPC). So... my guts are probably still saying this should not happen... but nevertheless:

When I check the VM for the list of objects in its POM memory, I always notice a huge number of the following classes. The second column is the number of instances and the third column is bytesize.

WAValueHolder 382375 12236000
Association 321599 12863960
IdentityCollisionBucket 95072 7014208
RcCounterElement 50182 1605824

After these top 4, we can find the instances of our application components and only then come the domain model objects.

My assumption now is that when I would keep Seaside session state as gem session state, that the SPC would not be stressed and can keep the domain model in memory. A lot less disk access would be needed, essentially eliminating a bottleneck. The only assumption is that the front-end load balancer should employ session affinity.

Now, I guess this has been thought over before? Would this make sense?

Johan

2012/2/20 Johan Brichau <[hidden email]>:
Form a Seaside PoV session affinity is definitely doable. An easy
solution is mod_proxy_balancer [1] and the Seaside-Cluster module [2]
(you have to take an older version like pmm.9, the newer ones require
Seaside 3.1). You can also write something yourself, modify the action
URL and add a rewrite rule.

 [1] http://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html
 [2] http://www.squeaksource.com/ajp

Cheers
Philippe

Johan,

I explore the idea of not persisting session state in this post[1]. Because of the GemStone transaction model (the whole vm changes state on an abort), a GemStone vm can only handle one concurrent seaside request. If you can afford to allocate one vm per active user, then you have a chance to make the one session/vm work. I've got code lying around somewhere that allows a pool of vms to be used to service sets of users, but you have to be able to start enough vms to handle all of the concurrent users.

If you can guarantee that your request processing will not take too long (causing unreasonable delays for the queued up sessions), then you can try to use session affinity and keep session state in the vm... Depending upon how many sessions are sharing a vm and how long your session timeouts are, you will probably end up having to worry about running out of temp obj space.

Using your framework if all state modification occurring in a request is in a protected block then you might think about handling multiple requests concurrently, but remember a single abort in one request will nuke all modified persistent objects so you have to do processing like we do for the maintenance vm:

  acquire transaction mutex
  abort
  do your work
  commit
  release mutex

and outside the protected you cannot safely modify persistent state or perform any aborts or commits ...

Of course we have left the realm of transparent persistence:)...

Coming back into the realm of transparent persistence...

Another technique to employ is to use RESTful requests where ever possible. If you use Seaside-REST, you can handle requests with 0 seaside session state intermixed with session state requests ... This is what is done for ss3...If you have read only requests you can even avoid the commits on request handling boundaries...Maybe the ajax requests can be routed to a RESTful API and safe a bunch of session state...

Another technique is that you might be able to tune your cache. I'd have to analyze a statmon file where you are under pressure, basically I'd be looking at the size of your local dirty pages count and global dirty pages count ... Depending upon what I see there might be tuning tricks that will help with the pressure ... can't promise anything until I look at the statmon files.

Another technique would be to talk to Monty Williams about the costs for upgrading the size of the SPC ... I'm an engineer not a salesman, but I do know that that option exists.

Finally, we could take a look at the session state that is being shoved to disk and see if there are ways to minimize the amount of session state that is being created ...

I think that the first option is to get a second opinion on the statmon interpretation and share the file with me and your interpretation ... I'm not claiming you are wrong, but it is worth having me take a look....perhaps your POM space is not large enough for your application ... if what you claim is correct that domain objects that aren't changing are still being flushed from the vm, then some tuning of the vm is in order...

Dale

[1] http://gemstonesoup.wordpress.com/2008/08/19/1-session-per-vm-another-scaling-alternative/

----- Original Message -----
| From: "Johan Brichau" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Monday, February 20, 2012 7:19:49 AM
| Subject: [GS/SS Beta] Reduce SPC pressure by keeping Seaside session state outside of the repository?
|
| Hello,
|
| Sorry for the long email, but the bottom line is: would it make sense
| to keep seaside session state as gem session state? And would that
| be feasible?
|
| I've been monitoring the behavior of the SPC over our daily runs,
| especially with respect to free frames and objects that got
| faulted-in from disk when the SPC started throwing objects out under
| pressure from the free frame list. In short: I _think_ I notice that
| under pressure from the persistent seaside session state, that
| domain model objects have to be re-read from disk. In other words:
| domain model objects are preempted and thrown out of the shared page
| cache.
|
| I did the monitoring using the hidden set that is created as
| mentioned in System>>_enableObjectsRead. This method only shows the
| objects that are faulted into VM memory (and not the SPC) but the
| given combination of seeing disk access happening during the
| launching of a new seaside session, the fact that the statmonit
| graph shows a flattening line for free frames and the fact that the
| contents of the hidden set lead me to think that way. I assume a new
| Seaside session does not need references to past Seaside sessions
| and therefore any objects being read from disk should be related to
| the objects it needs from the domain model. (Note that no
| 'FrameFromFindFree is occurring at all).
|
| But I'm a quite amazed at the fact that the SPC appears to be
| throwing out my domain model objects. They are outnumbered by the
| amount of objects of the Seaside session state (see table below) but
| they should be referenced quite often and be kept in the VM memory
| (and the SPC). So... my guts are probably still saying this should
| not happen... but nevertheless:
|
| When I check the VM for the list of objects in its POM memory, I
| always notice a huge number of the following classes. The second
| column is the number of instances and the third column is bytesize.
|
| WAValueHolder 382375 12236000
| Association 321599 12863960
| IdentityCollisionBucket 95072 7014208
| RcCounterElement 50182 1605824
|
| After these top 4, we can find the instances of our application
| components and only then come the domain model objects.
|
| My assumption now is that when I would keep Seaside session state as
| gem session state, that the SPC would not be stressed and can keep
| the domain model in memory. A lot less disk access would be needed,
| essentially eliminating a bottleneck. The only assumption is that
| the front-end load balancer should employ session affinity.
|
| Now, I guess this has been thought over before? Would this make
| sense?
|
| Johan

Hi Dale,

> I explore the idea of not persisting session state in this post[1]. Because of the GemStone transaction model (the whole vm changes state on an abort), a GemStone vm can only handle one concurrent seaside request. If you can afford to allocate one vm per active user, then you have a chance to make the one session/vm work. I've got code lying around somewhere that allows a pool of vms to be used to service sets of users, but you have to be able to start enough vms to handle all of the concurrent users.

I'm not sure I mean to have a vm per user, but maybe I'm overseeing something.
Since I have been storing session state (using _sessionStateAt:put methods) in a VM that remains across an abort boundary, I was primarily thinking of using that technique to store the session state (i.e. WACache & WASnapshot instances,...). We could store state for many different sessions, just like it's done when they are stored in the persistent root? But I have not yet looked at how that part of Seaside works though, so I might hit something that does not work.

> If you can guarantee that your request processing will not take too long (causing unreasonable delays for the queued up sessions), then you can try to use session affinity and keep session state in the vm...

Actually, we are already using session affinity. The reason is that when using round-trip load balancing, we saw how ajax requests that were issued 'in parallel' ended up being processed in parallel by different vms, which eventually means they got blocked by the session lock. This eventually gives worse response times.

> Depending upon how many sessions are sharing a vm and how long your session timeouts are, you will probably end up having to worry about running out of temp obj space.

_that_ would obviously be something to worry about. I guess the only way to know is to hack something together ;-)

> Another technique is that you might be able to tune your cache. I'd have to analyze a statmon file where you are under pressure, basically I'd be looking at the size of your local dirty pages count and global dirty pages count ... Depending upon what I see there might be tuning tricks that will help with the pressure ... can't promise anything until I look at the statmon files.

I guess that cannot hurt ;-) I will send you one in a separate email. Thanks!

> Finally, we could take a look at the session state that is being shoved to disk and see if there are ways to minimize the amount of session state that is being created ...

Indeed. I'm actually exploring that option too. Obviously, Yesplan has a quite large set of components and state. But since we are not using any back-forward functionality, all the snapshot creation is something we do not seem to need. My guess is that this accounts for a very large portion of the committed/changed objects and hence, a load of i/o. I have subclasses of the WARenderContinuation & WAActionContinuation that skip this phase and all seems to run very well. Now I just need to test if it helps with reducing session state.

Well, I'm only starting out hacking and exploring on this... let's see what happens ;-)

Johan