Workaround for "Too many retries: 11" [was: session lock denied]

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Workaround for "Too many retries: 11" [was: session lock denied]

Carla F. Griggio
Although the "session lock denied" thread was pretty old now... I'd like to add something for posterity, as Sean says :P

I've been having the same problem (a "Too many retries: 11" error comes up when the user clicks too fast repeatedly on links), but with FastCGI, not Swazoo. Maybe you can find a similar solution for Swazoo.

Dale recommended me to look for methods with the word "retries" :P And I found:

For FastCGI:
FSSeasideHandler>>answerResponderRole:

For Hyper:
SeasideHTTPService>>process: aRequest

In those methods there's a delay until retrying. You can adjust the miliseconds to wait until retrying the request. The default is 10ms, I bumped it to 100ms and so far is working OK (I'm trying to "double click" on links I know that take a little longer to load, and I don't get the "Too many retries: 11" error anymore).

It was very difficult to "google" this one... so I hope this is useful for someone else in the future :P

On Fri, May 21, 2010 at 5:03 PM, Sean Allen <[hidden email]> wrote:
On Thu, May 20, 2010 at 7:22 PM, Dale Henrichs <[hidden email]> wrote:
> Sean Allen wrote:
>>
>> So for posterity...
>>
>> we had code that was taking too long to execute and then, the front
>> end server tried proxying to another
>> swazoo and we got the error.
>
> "front end server" is that Apache? Can't you tell the "front-end server" to
> wait longer before trying again or better yet to not timeout at all?

i should be more clear on that for posterity.

user clicks on link. it takes a long time, gem has session locked.
user gets antsy, clicks again.
nginx proxies to a different gem ( as the one that has the session is busy )
second gem hits the session lock error.

Reply | Threaded
Open this post in threaded view
|

Re: Workaround for "Too many retries: 11" [was: session lock denied]

Johan Brichau-2
Hi Carla,

Are you still getting request retries or are these gone as well?

I am asking because the too many retries only really happens after quite some time (I think it's only at the 10th retry). Although we did not have this "too many retries", we did have a lot of retries happening as well in the past. To prevent such retries, we are now using sticky sessions: the front-end webserver forwards all requests for the same Seaside session to the same Seaside gem. If a user double clicks, or if the webpage is doing multiple ajax requests at the same time, they are queued to the same gem and hence, do not cause commit conflicts on the Seaside session.

The big problem with the round-robin dispatching, is that a single user can block all Seaside gems because they are all having commit conflicts on the Seaside session and keep retrying requests. We have noticed significant speedup for ajax requests that were sent in parallel (because we avoid the commit conflicts and resulting retries).

There is, however, still an issue I am seeing under load: http://code.google.com/p/glassdb/issues/detail?id=303 

I just thought it might be useful to know. I have been meaning to write some blog post on this as well, but time is not my friend ;-)

Johan

On 07 Nov 2011, at 17:50, Carla F. Griggio wrote:

> Although the "session lock denied" thread was pretty old now... I'd like to add something for posterity, as Sean says :P
>
> I've been having the same problem (a "Too many retries: 11" error comes up when the user clicks too fast repeatedly on links), but with FastCGI, not Swazoo. Maybe you can find a similar solution for Swazoo.
>
> Dale recommended me to look for methods with the word "retries" :P And I found:
>
> For FastCGI:
> FSSeasideHandler>>answerResponderRole:
>
> For Hyper:
> SeasideHTTPService>>process: aRequest
>
> In those methods there's a delay until retrying. You can adjust the miliseconds to wait until retrying the request. The default is 10ms, I bumped it to 100ms and so far is working OK (I'm trying to "double click" on links I know that take a little longer to load, and I don't get the "Too many retries: 11" error anymore).
>
> It was very difficult to "google" this one... so I hope this is useful for someone else in the future :P
>
> On Fri, May 21, 2010 at 5:03 PM, Sean Allen <[hidden email]> wrote:
> On Thu, May 20, 2010 at 7:22 PM, Dale Henrichs <[hidden email]> wrote:
> > Sean Allen wrote:
> >>
> >> So for posterity...
> >>
> >> we had code that was taking too long to execute and then, the front
> >> end server tried proxying to another
> >> swazoo and we got the error.
> >
> > "front end server" is that Apache? Can't you tell the "front-end server" to
> > wait longer before trying again or better yet to not timeout at all?
>
> i should be more clear on that for posterity.
>
> user clicks on link. it takes a long time, gem has session locked.
> user gets antsy, clicks again.
> nginx proxies to a different gem ( as the one that has the session is busy )
> second gem hits the session lock error.
>

Reply | Threaded
Open this post in threaded view
|

Re: Workaround for "Too many retries: 11" [was: session lock denied]

Carla F. Griggio
Hi Johan!
So far, I notice that the Too many retries problem has been reduced, but not gone for good. When heavy tasks are being done and the request take longer to respond, sometimes it appears again.
Now I'm tracking this behaviour to adjust the milliseconds to wait until retrying the request, to see if I can reduce a little more the Too many retries error.

From a usability point of view, as I bumped up the delay before retrying, when a request is retried the user sees that the page is still loading, instead of suddenly viewing the Too many retries: 11 white page. So, although it can still be a little annoying, they can associate it with "pages taking long to load". I think that way is a little bit less annoying for them.

What you say sounds very interesting, I'm looking forward to read that blog post!!

Cheers,
Carla.

On Tue, Nov 8, 2011 at 7:37 AM, Johan Brichau <[hidden email]> wrote:
Hi Carla,

Are you still getting request retries or are these gone as well?

I am asking because the too many retries only really happens after quite some time (I think it's only at the 10th retry). Although we did not have this "too many retries", we did have a lot of retries happening as well in the past. To prevent such retries, we are now using sticky sessions: the front-end webserver forwards all requests for the same Seaside session to the same Seaside gem. If a user double clicks, or if the webpage is doing multiple ajax requests at the same time, they are queued to the same gem and hence, do not cause commit conflicts on the Seaside session.

The big problem with the round-robin dispatching, is that a single user can block all Seaside gems because they are all having commit conflicts on the Seaside session and keep retrying requests. We have noticed significant speedup for ajax requests that were sent in parallel (because we avoid the commit conflicts and resulting retries).

There is, however, still an issue I am seeing under load: http://code.google.com/p/glassdb/issues/detail?id=303

I just thought it might be useful to know. I have been meaning to write some blog post on this as well, but time is not my friend ;-)

Johan

On 07 Nov 2011, at 17:50, Carla F. Griggio wrote:

> Although the "session lock denied" thread was pretty old now... I'd like to add something for posterity, as Sean says :P
>
> I've been having the same problem (a "Too many retries: 11" error comes up when the user clicks too fast repeatedly on links), but with FastCGI, not Swazoo. Maybe you can find a similar solution for Swazoo.
>
> Dale recommended me to look for methods with the word "retries" :P And I found:
>
> For FastCGI:
> FSSeasideHandler>>answerResponderRole:
>
> For Hyper:
> SeasideHTTPService>>process: aRequest
>
> In those methods there's a delay until retrying. You can adjust the miliseconds to wait until retrying the request. The default is 10ms, I bumped it to 100ms and so far is working OK (I'm trying to "double click" on links I know that take a little longer to load, and I don't get the "Too many retries: 11" error anymore).
>
> It was very difficult to "google" this one... so I hope this is useful for someone else in the future :P
>
> On Fri, May 21, 2010 at 5:03 PM, Sean Allen <[hidden email]> wrote:
> On Thu, May 20, 2010 at 7:22 PM, Dale Henrichs <[hidden email]> wrote:
> > Sean Allen wrote:
> >>
> >> So for posterity...
> >>
> >> we had code that was taking too long to execute and then, the front
> >> end server tried proxying to another
> >> swazoo and we got the error.
> >
> > "front end server" is that Apache? Can't you tell the "front-end server" to
> > wait longer before trying again or better yet to not timeout at all?
>
> i should be more clear on that for posterity.
>
> user clicks on link. it takes a long time, gem has session locked.
> user gets antsy, clicks again.
> nginx proxies to a different gem ( as the one that has the session is busy )
> second gem hits the session lock error.
>


Reply | Threaded
Open this post in threaded view
|

Re: Workaround for "Too many retries: 11" [was: session lock denied]

Stern, Richard
Hello!
 
We experienced similar problems in the past - an AJAX call took some time, users had impression that "nothing happens", got impatient and clicked on the same item again - thus causing the parallel requests for the same session and consequently the too may retries error. This is actually a quite common problem with AJAX applications in general (and not just Seaside ones).
 
Root of the problem in most cases is that users are not aware that there is already a call in progress. They need some feedback on that - so we took that as starting point rather than trying to solve at server side. A very efficient solution in our case was to
 
1. give some visual feedback that the system is working (progressor, hourglass etc.)
2. block any further inputs by having the web page overlaid with a div etc. catching any click- (and if necessary key-)events
 
Adding such actions to the AJAX calls is usually not that complicated as most AJAX frameworks offer hooks for that (e.g. for Prototype framework in Seaside see category "events" of class PTAjax). The harder part (at least for a Smalltalk programmer) is the implementation of the visual feedback and the "event blocker" in Javascript.
 
Kind regards
 
Richard


Von: [hidden email] [mailto:[hidden email]] Im Auftrag von Carla F. Griggio
Gesendet: Montag, 14. November 2011 16:54
An: GemStone Seaside beta discussion
Betreff: Re: [GS/SS Beta] Workaround for "Too many retries: 11" [was: session lock denied]

Hi Johan!
So far, I notice that the Too many retries problem has been reduced, but not gone for good. When heavy tasks are being done and the request take longer to respond, sometimes it appears again.
Now I'm tracking this behaviour to adjust the milliseconds to wait until retrying the request, to see if I can reduce a little more the Too many retries error.

From a usability point of view, as I bumped up the delay before retrying, when a request is retried the user sees that the page is still loading, instead of suddenly viewing the Too many retries: 11 white page. So, although it can still be a little annoying, they can associate it with "pages taking long to load". I think that way is a little bit less annoying for them.

What you say sounds very interesting, I'm looking forward to read that blog post!!

Cheers,
Carla.

On Tue, Nov 8, 2011 at 7:37 AM, Johan Brichau <[hidden email]> wrote:
Hi Carla,

Are you still getting request retries or are these gone as well?

I am asking because the too many retries only really happens after quite some time (I think it's only at the 10th retry). Although we did not have this "too many retries", we did have a lot of retries happening as well in the past. To prevent such retries, we are now using sticky sessions: the front-end webserver forwards all requests for the same Seaside session to the same Seaside gem. If a user double clicks, or if the webpage is doing multiple ajax requests at the same time, they are queued to the same gem and hence, do not cause commit conflicts on the Seaside session.

The big problem with the round-robin dispatching, is that a single user can block all Seaside gems because they are all having commit conflicts on the Seaside session and keep retrying requests. We have noticed significant speedup for ajax requests that were sent in parallel (because we avoid the commit conflicts and resulting retries).

There is, however, still an issue I am seeing under load: http://code.google.com/p/glassdb/issues/detail?id=303

I just thought it might be useful to know. I have been meaning to write some blog post on this as well, but time is not my friend ;-)

Johan

On 07 Nov 2011, at 17:50, Carla F. Griggio wrote:

> Although the "session lock denied" thread was pretty old now... I'd like to add something for posterity, as Sean says :P
>
> I've been having the same problem (a "Too many retries: 11" error comes up when the user clicks too fast repeatedly on links), but with FastCGI, not Swazoo. Maybe you can find a similar solution for Swazoo.
>
> Dale recommended me to look for methods with the word "retries" :P And I found:
>
> For FastCGI:
> FSSeasideHandler>>answerResponderRole:
>
> For Hyper:
> SeasideHTTPService>>process: aRequest
>
> In those methods there's a delay until retrying. You can adjust the miliseconds to wait until retrying the request. The default is 10ms, I bumped it to 100ms and so far is working OK (I'm trying to "double click" on links I know that take a little longer to load, and I don't get the "Too many retries: 11" error anymore).
>
> It was very difficult to "google" this one... so I hope this is useful for someone else in the future :P
>
> On Fri, May 21, 2010 at 5:03 PM, Sean Allen <[hidden email]> wrote:
> On Thu, May 20, 2010 at 7:22 PM, Dale Henrichs <[hidden email]> wrote:
> > Sean Allen wrote:
> >>
> >> So for posterity...
> >>
> >> we had code that was taking too long to execute and then, the front
> >> end server tried proxying to another
> >> swazoo and we got the error.
> >
> > "front end server" is that Apache? Can't you tell the "front-end server" to
> > wait longer before trying again or better yet to not timeout at all?
>
> i should be more clear on that for posterity.
>
> user clicks on link. it takes a long time, gem has session locked.
> user gets antsy, clicks again.
> nginx proxies to a different gem ( as the one that has the session is busy )
> second gem hits the session lock error.
>