Making Server Application ready for deployment: Error reporting

So we are now slowly getting into the stage where we need to dive into the dirty details of application deployment. There is so much to do that has nothing to do with the application per se that we need to prepare these things early enough - like an internal test phase on test servers.

One of the things to do is establish tools and mechanics to learn about and analyse errors. An important part here is to write out information about errors. In a production environment it will not be efficient to display the stack trace on a web page (although it's far better than nothing) or simply have the application ignore it.

So how do people running VA ST servers handle errors (this question applies not only to Seaside applications) ?
Do you write stack dumps or traces? Is the overhead of using dumps as compared to stack traces worth the effort in your experience?
How do people poll for errors on the server, assuming that not every user will write a support ticket? Do you simply watch some directory for new trace/dump files?

I am grateful for any comments on this topic

Joachim

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To view this discussion on the web visit https://groups.google.com/d/msg/va-smalltalk/-/9AZtCfAeJqQJ.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/va-smalltalk?hl=en.

Hello Joachim,

I've worked with several headless server apps running Web Connection
or servicing requests from Websphere MQ.

If the application encounters an error or exception that it catches
and handles, we generally write a stack trace to the log and trigger a
notification to our production on-call group or person. The
notification method has been an e-mail, sent via eMan, an
event written to the system log, or an HTTP post to an homegrown
monitoring system. System logs are monitored by a standard event
monitoring app which is usually used on systems running both Smalltalk
and non-Smalltalk apps.

We have a runtime debugging option to write a full stack dump in
addition to the stack trace for use in the rare case where we have a
recurring error of some sort we can't figure out from the stack trace.
This tends to get used in QA systems much more than production.

If the app encounters an error that causes a stack dump or core file, we
have a korn shell script (on UNIX) that periodically queries the
directory for new files. It then uses the same notification method as
above to notify on-call. Because you want the image to stay up and
processing client requests, you usually want to catch exceptions at
a high enough level in Smalltalk that you can handle it within the
image in your error handling code rather than relying on generating a
stack dump and restarting the image.

Doug Swartz


Wednesday, April 11, 2012, 9:41:13 AM, you wrote:

> So we are now slowly getting into the stage where we need to dive
> into the dirty details of application deployment. There is so much
> to do that has nothing to do with the application per se that we
> need to prepare these things early enough - like an internal test phase on test servers.

> One of the things to do is establish tools and mechanics to learn
> about and analyse errors. An important part here is to write out
> information about errors. In a production environment it will not be
> efficient to display the stack trace on a web page (although it's
> far better than nothing) or simply have the application ignore it.

> So how do people running VA ST servers handle errors (this question
> applies not only to Seaside applications) ?
> Do you write stack dumps or traces? Is the overhead of using dumps
> as compared to stack traces worth the effort in your experience?
> How do people poll for errors on the server, assuming that not
> every user will write a support ticket? Do you simply watch some
> directory for new trace/dump files?

> I am grateful for any comments on this topic

> Joachim





--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/va-smalltalk?hl=en.

Hi Doug,

thanks for your description.
So you're using almost the full arsenal of options, maybe excluding online remote debugging...

We want to keep the images up and running continuously, so catching exceptions and informing support surely is the way to go.
I was wondering if dumps really help in many situations. On one hand, it really is frustrating to see you have no chance to debug your code in a headless server (unless you go the remote debugging way) and logs tend to get very long and hard to follow once your image really works hard. But on the other, we've been using stack traces to find bugs in lots of traditional fat clients for years now and most of the times they were good enough to find a problem quite fast. The problem with dumps, as far as I can tell from my little experience with them is that there is a lot of stuff needed to really debug them and they're still limited as compared to a live debugger...

So I guess we'll start  with an infrastructure around logs and stack traces (not one giant stack trace file, but one for each uncaught exception, maybe combined with a "Mail to support" button on an error page) and see how far we'll get with it.

Again, thanks for your time

Joachim

Am Donnerstag, 12. April 2012 12:22:18 UTC+2 schrieb dswartz:

Hello Joachim,

I've worked with several headless server apps running Web Connection
or servicing requests from Websphere MQ.

If the application encounters an error or exception that it catches
and handles, we generally write a stack trace to the log and trigger a
notification to our production on-call group or person. The
notification method has been an e-mail, sent via eMan, an
event written to the system log, or an HTTP post to an homegrown
monitoring system. System logs are monitored by a standard event
monitoring app which is usually used on systems running both Smalltalk
and non-Smalltalk apps.

We have a runtime debugging option to write a full stack dump in
addition to the stack trace for use in the rare case where we have a
recurring error of some sort we can't figure out from the stack trace.
This tends to get used in QA systems much more than production.

If the app encounters an error that causes a stack dump or core file, we
have a korn shell script (on UNIX) that periodically queries the
directory for new files. It then uses the same notification method as
above to notify on-call. Because you want the image to stay up and
processing client requests, you usually want to catch exceptions at
a high enough level in Smalltalk that you can handle it within the
image in your error handling code rather than relying on generating a
stack dump and restarting the image.

Doug Swartz

Wednesday, April 11, 2012, 9:41:13 AM, you wrote:

> So we are now slowly getting into the stage where we need to dive

> into the dirty details of application deployment. There is so much
> to do that has nothing to do with the application per se that we
> need to prepare these things early enough - like an internal test phase on test servers.

> One of the things to do is establish tools and mechanics to learn
> about and analyse errors. An important part here is to write out
> information about errors. In a production environment it will not be
> efficient to display the stack trace on a web page (although it's
> far better than nothing) or simply have the application ignore it.

> So how do people running VA ST servers handle errors (this question
> applies not only to Seaside applications) ?
> Do you write stack dumps or traces? Is the overhead of using dumps
> as compared to stack traces worth the effort in your experience?
> How do people poll for errors on the server, assuming that not
> every user will write a support ticket? Do you simply watch some
> directory for new trace/dump files?

> I am grateful for any comments on this topic

> Joachim

--

Hello Joachim,

In my experience, stack dumps are rarely needed. It's almost
always possible to figure out what is going on from a stack trace.

We usually restarted server images every day, so the length of a
common log file for trace info and any encountered stack traces was
almost never a problem. Of course, it is important to not generate
lots of exceptions that cause stack traces to be written :)

Doug


Thursday, April 12, 2012, 6:29:41 AM, you wrote:

> Hi Doug,

> thanks for your description.
> So you're using almost the full arsenal of options, maybe excluding online remote debugging...

> So I guess we'll start  with an infrastructure around logs and
> stack traces (not one giant stack trace file, but one for each
> uncaught exception, maybe combined with a "Mail to support" button
> on an error page) and see how far we'll get with it.

> Again, thanks for your time

> Joachim




















--
Best regards,
 Douglas                            mailto:[hidden email]

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/va-smalltalk?hl=en.