[Glass] How to deal with timeouts, fastcgi and what is the expected behavior?

Hi guys, 

I have a strange situation with timeouts under nginx/fastCGI and I am not sure what is expected. I am executing a Seaside WATask, where one of its method takes (for sure) more then the timeout of nginx/fastCGI. My piece of nginx setup is something like this:

location @seasidemariano {

include fastcgi_params;

fastcgi_param REQUEST_URI $uri?$args; 

  fastcgi_pass seasidemariano;

fastcgi_connect_timeout      180;

      fastcgi_send_timeout         180;

      fastcgi_read_timeout         180;

  fastcgi_next_upstream error invalid_header timeout http_500;

}

So...as you can see I have a timeout of 180 and I tell to go to the next upstream (gem) in any error, including timeout. Now...say I have this method being executed and it takes more than 180 seconds. What happens is that the user gets a Nginx 504 Gateway Time-out in the browser. Ok. But... I have some questions:

1) what happens with that gem that was executing the task (the one that took more than 180)? is the execution finished even if the nginx give a timeout and pass the reuqest to the next gem? Or the gem execution is aborted?  Why I ask? Because...I out a log to a file inside my method...and it looks like if the method were called 3 times rather than 1. And from a domain point of view.... it is not good that such a method is executed 3 times...

2) If I put a larger timeout...say 1500 ... it works correct..the method is executed only once, no timeout. Same if I use swazoo. So it is something to do with the timeouts and fastCGI for sure. 

3) 3 times...why? It seems because I have 3 gems. I did an experiment, and I set only 2 gems to nginx fastcgi. And yes, the method was executed only 2 times rather than 3. 

So....how do people normally deal with this? Of course, the immediate workaround seems to increase the timeout...but it seems risky to me, thinking that if for some reason (like GC running or whatever) one particular request takes more than the timeout, then my "backend code" could be run more than once...

Thanks in advance, 

--
Mariano
http://marianopeck.wordpress.com

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

On Fri, Jul 18, 2014 at 7:36 AM, Mariano Martinez Peck <[hidden email]> wrote:

Hi guys, 

I have a strange situation with timeouts under nginx/fastCGI and I am not sure what is expected. I am executing a Seaside WATask, where one of its method takes (for sure) more then the timeout of nginx/fastCGI. My piece of nginx setup is something like this:

location @seasidemariano {

include fastcgi_params;

fastcgi_param REQUEST_URI $uri?$args; 

  fastcgi_pass seasidemariano;

fastcgi_connect_timeout      180;

      fastcgi_send_timeout         180;

      fastcgi_read_timeout         180;

  fastcgi_next_upstream error invalid_header timeout http_500;

}

So...as you can see I have a timeout of 180 and I tell to go to the next upstream (gem) in any error, including timeout. Now...say I have this method being executed and it takes more than 180 seconds. What happens is that the user gets a Nginx 504 Gateway Time-out in the browser. Ok. But... I have some questions:

1) what happens with that gem that was executing the task (the one that took more than 180)? is the execution finished even if the nginx give a timeout and pass the reuqest to the next gem? Or the gem execution is aborted?  Why I ask? Because...I out a log to a file inside my method...and it looks like if the method were called 3 times rather than 1. And from a domain point of view.... it is not good that such a method is executed 3 times...

It does sound like nginx is redispatching the http request on timeout... 

2) If I put a larger timeout...say 1500 ... it works correct..the method is executed only once, no timeout. Same if I use swazoo. So it is something to do with the timeouts and fastCGI for sure. 

In general I try to avoid timeouts ... it seems that timeouts fire more often because the system is slow than for any other reason and the standard answer: increase the timeout ...

So I guess I would wonder why the operation is taking so long ... if the operation is slow because the system is overloaded, then a longer timeout _is_ called for, but then what is a good value for the timeout ..

I guess the real question to ask is what is the purpose of the timeout? 

You might want the gem itself to decide to terminate a request if it is "taking too long" then you don't need  a timeout at the nginx level?

3) 3 times...why? It seems because I have 3 gems. I did an experiment, and I set only 2 gems to nginx fastcgi. And yes, the method was executed only 2 times rather than 3. 

It does sound like nginx is sending the request again upon a timeout ... could that be?

 

So....how do people normally deal with this? Of course, the immediate workaround seems to increase the timeout...but it seems risky to me, thinking that if for some reason (like GC running or whatever) one particular request takes more than the timeout, then my "backend code" could be run more than once...

Thanks in advance, 

--
Mariano
http://marianopeck.wordpress.com

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

On Fri, Jul 18, 2014 at 12:08 PM, Dale Henrichs <[hidden email]> wrote:

On Fri, Jul 18, 2014 at 7:36 AM, Mariano Martinez Peck <[hidden email]> wrote:

Hi guys, 

I have a strange situation with timeouts under nginx/fastCGI and I am not sure what is expected. I am executing a Seaside WATask, where one of its method takes (for sure) more then the timeout of nginx/fastCGI. My piece of nginx setup is something like this:

location @seasidemariano {

include fastcgi_params;

fastcgi_param REQUEST_URI $uri?$args; 

  fastcgi_pass seasidemariano;

fastcgi_connect_timeout      180;

      fastcgi_send_timeout         180;

      fastcgi_read_timeout         180;

  fastcgi_next_upstream error invalid_header timeout http_500;

}

So...as you can see I have a timeout of 180 and I tell to go to the next upstream (gem) in any error, including timeout. Now...say I have this method being executed and it takes more than 180 seconds. What happens is that the user gets a Nginx 504 Gateway Time-out in the browser. Ok. But... I have some questions:

1) what happens with that gem that was executing the task (the one that took more than 180)? is the execution finished even if the nginx give a timeout and pass the reuqest to the next gem? Or the gem execution is aborted?  Why I ask? Because...I out a log to a file inside my method...and it looks like if the method were called 3 times rather than 1. And from a domain point of view.... it is not good that such a method is executed 3 times...

It does sound like nginx is redispatching the http request on timeout... 

Exactly. And it should, as my configuration is: 

fastcgi_next_upstream error invalid_header timeout http_500;

So yes...upon a gem timeout, nginx forwards the request to the next gem. 

 

2) If I put a larger timeout...say 1500 ... it works correct..the method is executed only once, no timeout. Same if I use swazoo. So it is something to do with the timeouts and fastCGI for sure. 

In general I try to avoid timeouts ... it seems that timeouts fire more often because the system is slow than for any other reason and the standard answer: increase the timeout ...

So I guess I would wonder why the operation is taking so long ... if the operation is slow because the system is overloaded, then a longer timeout _is_ called for, but then what is a good value for the timeout ..

The operation takes long because I need to call a HTTPS api (using Zinc to a local nginx tunnel) many times where I need to post a XML and I also get a large XML response. The time it takes..depends on how many "items" have been selected. So it is hard to estimate how much it would take. 

 

I guess the real question to ask is what is the purpose of the timeout? 

If a gem went down, I would like nginx to forward request to the other (available gems).

 

You might want the gem itself to decide to terminate a request if it is "taking too long" then you don't need  a timeout at the nginx level?

3) 3 times...why? It seems because I have 3 gems. I did an experiment, and I set only 2 gems to nginx fastcgi. And yes, the method was executed only 2 times rather than 3. 

It does sound like nginx is sending the request again upon a timeout ... could that be?

Yes, it is that. But I don't know how to properly solve both things... be able to have large timeout (like for this scenario), yet manage the scenario of gems going down. Imagine I don't care and I put a timeout of 1 hour (to say something). Then.. imagine I have a gem down and then a web user that connects to the site.  nginx might assign that gem that gem that went down. Therefore... it would have the web user waiting in the browser for 1 hour until nginx answers for a timeout....Is this correct?

The biggest issue is that nginx thinks the gem timeouted and then fordwads to the next gem. However...the gem was not dead...it was simply too busy with a time consuming request  ;) There is no way I can make the gem answer to nginx "I am fine, don't worry, just busy, continue to next gem" hahaha ? 

Probably the real real solution is the service VM as you and Paul pointed out several times. But i didn't have time to take a look to it yet :(

 

 

So....how do people normally deal with this? Of course, the immediate workaround seems to increase the timeout...but it seems risky to me, thinking that if for some reason (like GC running or whatever) one particular request takes more than the timeout, then my "backend code" could be run more than once...

Thanks in advance, 

--
Mariano
http://marianopeck.wordpress.com

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

--
Mariano
http://marianopeck.wordpress.com

On Fri, Jul 18, 2014 at 12:54 PM, Mariano Martinez Peck <[hidden email]> wrote:

On Fri, Jul 18, 2014 at 12:08 PM, Dale Henrichs <[hidden email]> wrote:

On Fri, Jul 18, 2014 at 7:36 AM, Mariano Martinez Peck <[hidden email]> wrote:

Hi guys, 

I have a strange situation with timeouts under nginx/fastCGI and I am not sure what is expected. I am executing a Seaside WATask, where one of its method takes (for sure) more then the timeout of nginx/fastCGI. My piece of nginx setup is something like this:

location @seasidemariano {

include fastcgi_params;

fastcgi_param REQUEST_URI $uri?$args; 

  fastcgi_pass seasidemariano;

fastcgi_connect_timeout      180;

      fastcgi_send_timeout         180;

      fastcgi_read_timeout         180;

  fastcgi_next_upstream error invalid_header timeout http_500;

}

So...as you can see I have a timeout of 180 and I tell to go to the next upstream (gem) in any error, including timeout. Now...say I have this method being executed and it takes more than 180 seconds. What happens is that the user gets a Nginx 504 Gateway Time-out in the browser. Ok. But... I have some questions:

1) what happens with that gem that was executing the task (the one that took more than 180)? is the execution finished even if the nginx give a timeout and pass the reuqest to the next gem? Or the gem execution is aborted?  Why I ask? Because...I out a log to a file inside my method...and it looks like if the method were called 3 times rather than 1. And from a domain point of view.... it is not good that such a method is executed 3 times...

It does sound like nginx is redispatching the http request on timeout... 

Exactly. And it should, as my configuration is: 

fastcgi_next_upstream error invalid_header timeout http_500;

So yes...upon a gem timeout, nginx forwards the request to the next gem. 

 

2) If I put a larger timeout...say 1500 ... it works correct..the method is executed only once, no timeout. Same if I use swazoo. So it is something to do with the timeouts and fastCGI for sure. 

In general I try to avoid timeouts ... it seems that timeouts fire more often because the system is slow than for any other reason and the standard answer: increase the timeout ...

So I guess I would wonder why the operation is taking so long ... if the operation is slow because the system is overloaded, then a longer timeout _is_ called for, but then what is a good value for the timeout ..

The operation takes long because I need to call a HTTPS api (using Zinc to a local nginx tunnel) many times where I need to post a XML and I also get a large XML response. The time it takes..depends on how many "items" have been selected. So it is hard to estimate how much it would take. 

 

I guess the real question to ask is what is the purpose of the timeout? 

If a gem went down, I would like nginx to forward request to the other (available gems).

 

You might want the gem itself to decide to terminate a request if it is "taking too long" then you don't need  a timeout at the nginx level?

3) 3 times...why? It seems because I have 3 gems. I did an experiment, and I set only 2 gems to nginx fastcgi. And yes, the method was executed only 2 times rather than 3. 

It does sound like nginx is sending the request again upon a timeout ... could that be?

Yes, it is that. But I don't know how to properly solve both things... be able to have large timeout (like for this scenario), yet manage the scenario of gems going down. Imagine I don't care and I put a timeout of 1 hour (to say something). Then.. imagine I have a gem down and then a web user that connects to the site.  nginx might assign that gem that gem that went down. Therefore... it would have the web user waiting in the browser for 1 hour until nginx answers for a timeout....Is this correct?

The biggest issue is that nginx thinks the gem timeouted and then fordwads to the next gem. However...the gem was not dead...it was simply too busy with a time consuming request  ;) There is no way I can make the gem answer to nginx "I am fine, don't worry, just busy, continue to next gem" hahaha ? 

If we assume that when a gem goes down it is normally that the process is aborted (rather than an unresponding http server), then it would be nice if nignx could check if the process is alive (using PID or whatever) in order to forward the request to another upstream rather than checking via http timeout ... 

 

Probably the real real solution is the service VM as you and Paul pointed out several times. But i didn't have time to take a look to it yet :(

 

 

So....how do people normally deal with this? Of course, the immediate workaround seems to increase the timeout...but it seems risky to me, thinking that if for some reason (like GC running or whatever) one particular request takes more than the timeout, then my "backend code" could be run more than once...

Thanks in advance, 

--
Mariano
http://marianopeck.wordpress.com

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

--
Mariano
http://marianopeck.wordpress.com

--
Mariano
http://marianopeck.wordpress.com

On Fri, Jul 18, 2014 at 12:08 PM, Dale Henrichs <[hidden email]> wrote:

On Fri, Jul 18, 2014 at 7:36 AM, Mariano Martinez Peck <[hidden email]> wrote:

Hi guys, 

I have a strange situation with timeouts under nginx/fastCGI and I am not sure what is expected. I am executing a Seaside WATask, where one of its method takes (for sure) more then the timeout of nginx/fastCGI. My piece of nginx setup is something like this:

location @seasidemariano {

include fastcgi_params;

fastcgi_param REQUEST_URI $uri?$args; 

  fastcgi_pass seasidemariano;

fastcgi_connect_timeout      180;

      fastcgi_send_timeout         180;

      fastcgi_read_timeout         180;

  fastcgi_next_upstream error invalid_header timeout http_500;

}

So...as you can see I have a timeout of 180 and I tell to go to the next upstream (gem) in any error, including timeout. Now...say I have this method being executed and it takes more than 180 seconds. What happens is that the user gets a Nginx 504 Gateway Time-out in the browser. Ok. But... I have some questions:

1) what happens with that gem that was executing the task (the one that took more than 180)? is the execution finished even if the nginx give a timeout and pass the reuqest to the next gem? Or the gem execution is aborted?  Why I ask? Because...I out a log to a file inside my method...and it looks like if the method were called 3 times rather than 1. And from a domain point of view.... it is not good that such a method is executed 3 times...

It does sound like nginx is redispatching the http request on timeout... 

Exactly. And it should, as my configuration is: 

fastcgi_next_upstream error invalid_header timeout http_500;

So yes...upon a gem timeout, nginx forwards the request to the next gem. 

 

2) If I put a larger timeout...say 1500 ... it works correct..the method is executed only once, no timeout. Same if I use swazoo. So it is something to do with the timeouts and fastCGI for sure. 

In general I try to avoid timeouts ... it seems that timeouts fire more often because the system is slow than for any other reason and the standard answer: increase the timeout ...

So I guess I would wonder why the operation is taking so long ... if the operation is slow because the system is overloaded, then a longer timeout _is_ called for, but then what is a good value for the timeout ..

The operation takes long because I need to call a HTTPS api (using Zinc to a local nginx tunnel) many times where I need to post a XML and I also get a large XML response. The time it takes..depends on how many "items" have been selected. So it is hard to estimate how much it would take. 

 

I guess the real question to ask is what is the purpose of the timeout? 

If a gem went down, I would like nginx to forward request to the other (available gems).

 

You might want the gem itself to decide to terminate a request if it is "taking too long" then you don't need  a timeout at the nginx level?

3) 3 times...why? It seems because I have 3 gems. I did an experiment, and I set only 2 gems to nginx fastcgi. And yes, the method was executed only 2 times rather than 3. 

It does sound like nginx is sending the request again upon a timeout ... could that be?

Yes, it is that. But I don't know how to properly solve both things... be able to have large timeout (like for this scenario), yet manage the scenario of gems going down. Imagine I don't care and I put a timeout of 1 hour (to say something). Then.. imagine I have a gem down and then a web user that connects to the site.  nginx might assign that gem that gem that went down. Therefore... it would have the web user waiting in the browser for 1 hour until nginx answers for a timeout....Is this correct?

The biggest issue is that nginx thinks the gem timeouted and then fordwads to the next gem. However...the gem was not dead...it was simply too busy with a time consuming request  ;) There is no way I can make the gem answer to nginx "I am fine, don't worry, just busy, continue to next gem" hahaha ? 

Probably the real real solution is the service VM as you and Paul pointed out several times. But i didn't have time to take a look to it yet :(

Maybe I can let the "error" option but remove "timeout". That way, maybe, it will only forward upon a gem crash and not timeout. I will try this and let you know.

In the meanwhile...do you know how does nginx does the assignment of a request to the particular upstream?  it is a plain round robin or smarter? Imagine I have some gems that are idle and some that are busy (already processing a request). Would nginx simply forward to the next one (when a request arrives) or is it smart to look for those that are idle?