Image taking up 100% CPU

Hi list,

A couple months ago I deployed an Iliad application in a Debian VPS, which has been working great except for the fact that about every two weeks the image takes up 100% of the CPU and, of course, the app becomes unusable.

Does anyone have a method for finding out what's going on? I've been trying to debug the problem, but as I don't know what triggers it, it's pretty impossible to detect...

I've checked the running threads and nothing seems unusual. I've also profiled the app for a whole day of real usage with end users and not a single method took more time than what I expected, so I'm completely lost here.

Any help will be appreciated! :)

Cheers,

Bernat.

--
Bernat Romagosa.

Not really, the image is not being saved regularly... all data is stored in a neo4j database, so there's no need to save the image.

It seems like it may have something to do with users uploading files in some particular moments, but I'm using Göran's file upload mechanism, in which files are streamed straight to disk, so there shouldn't be any image clogging problems even for huge files...

2013/9/24 Mariano Martinez Peck <[hidden email]>

Bernat, could this be triggered when you SAVE the image? If true, do you happen to have RFB running?

On Tue, Sep 24, 2013 at 4:45 AM, Bernat Romagosa <[hidden email]> wrote:

Hi list,

A couple months ago I deployed an Iliad application in a Debian VPS, which has been working great except for the fact that about every two weeks the image takes up 100% of the CPU and, of course, the app becomes unusable.

Does anyone have a method for finding out what's going on? I've been trying to debug the problem, but as I don't know what triggers it, it's pretty impossible to detect...

I've checked the running threads and nothing seems unusual. I've also profiled the app for a whole day of real usage with end users and not a single method took more time than what I expected, so I'm completely lost here.

Any help will be appreciated! :)

Cheers,

Bernat.

--
Bernat Romagosa.

--
Mariano
http://marianopeck.wordpress.com

--
Bernat Romagosa.

Hi Bernat,

I remember once having similar problems on Swazoo. But on image restart.
If Swazoo was not stopped before image shutdown, image went into 100%
CPU after restart. Something like that. It seems therefore that this is
sockets related problem

What if you stop your web server and start again?

Best regards
Janko

Dne 25. 09. 2013 10:03, piše Bernat Romagosa:

> Forgot to answer that I'm not using RFB. I had sooo many problems with

> it that since last year I've been using xpra <http://xpra.org/> instead,

> which lets you capture running X windows remotely over SSH. It's kind of
> like screen (not even remotely so powerful) but for GUIs.
>
> Cheers,
> Bernat.
>
>
> 2013/9/25 Bernat Romagosa <[hidden email]

> <mailto:[hidden email]>>

>
>     Not really, the image is not being saved regularly... all data is
>     stored in a neo4j database, so there's no need to save the image.
>
>     It seems like it may have something to do with users uploading files
>     in some particular moments, but I'm using Göran's file upload
>     mechanism, in which files are streamed straight to disk, so there
>     shouldn't be any image clogging problems even for huge files...
>
>     2013/9/24 Mariano Martinez Peck <[hidden email]

>     <mailto:[hidden email]>>

>
>         Bernat, could this be triggered when you SAVE the image? If
>         true, do you happen to have RFB running?
>
>
>         On Tue, Sep 24, 2013 at 4:45 AM, Bernat Romagosa
>         <[hidden email]

>         <mailto:[hidden email]>> wrote:
>
>             Hi list,
>
>             A couple months ago I deployed an Iliad application in a
>             Debian VPS, which has been working great except for the fact
>             that about every two weeks the image takes up 100% of the
>             CPU and, of course, the app becomes unusable.
>
>             Does anyone have a method for finding out what's going on?
>             I've been trying to debug the problem, but as I don't know
>             what triggers it, it's pretty impossible to detect...
>
>             I've checked the running threads and nothing seems unusual.
>             I've also profiled the app for a whole day of real usage
>             with end users and not a single method took more time than
>             what I expected, so I'm completely lost here.
>
>             Any help will be appreciated! :)

--
Janko Mivšek
Aida/Web
Smalltalk Web Application Server
http://www.aidaweb.si

Which image and vm ?

On Sep 25, 2013, at 3:24 AM, Bernat Romagosa <[hidden email]> wrote:

Hmmm something weird just happened. I restarted Komanche, then when I saved the image and the CPU rushed to 100%!

2013/9/25 Janko Mivšek <[hidden email]>

Hi Bernat,

I remember once having similar problems on Swazoo. But on image restart.
If Swazoo was not stopped before image shutdown, image went into 100%
CPU after restart. Something like that. It seems therefore that this is
sockets related problem

What if you stop your web server and start again?

Best regards
Janko

Dne 25. 09. 2013 10:03, piše Bernat Romagosa:

> Forgot to answer that I'm not using RFB. I had sooo many problems with

> it that since last year I've been using xpra <http://xpra.org/> instead,

> which lets you capture running X windows remotely over SSH. It's kind of
> like screen (not even remotely so powerful) but for GUIs.
>
> Cheers,
> Bernat.
>
>
> 2013/9/25 Bernat Romagosa <[hidden email]

> <mailto:[hidden email]>>

>
>     Not really, the image is not being saved regularly... all data is
>     stored in a neo4j database, so there's no need to save the image.
>
>     It seems like it may have something to do with users uploading files
>     in some particular moments, but I'm using Göran's file upload
>     mechanism, in which files are streamed straight to disk, so there
>     shouldn't be any image clogging problems even for huge files...
>
>     2013/9/24 Mariano Martinez Peck <[hidden email]

>     <mailto:[hidden email]>>

>
>         Bernat, could this be triggered when you SAVE the image? If
>         true, do you happen to have RFB running?
>
>
>         On Tue, Sep 24, 2013 at 4:45 AM, Bernat Romagosa
>         <[hidden email]

>         <mailto:[hidden email]>> wrote:
>
>             Hi list,
>
>             A couple months ago I deployed an Iliad application in a
>             Debian VPS, which has been working great except for the fact
>             that about every two weeks the image takes up 100% of the
>             CPU and, of course, the app becomes unusable.
>
>             Does anyone have a method for finding out what's going on?
>             I've been trying to debug the problem, but as I don't know
>             what triggers it, it's pretty impossible to detect...
>
>             I've checked the running threads and nothing seems unusual.
>             I've also profiled the app for a whole day of real usage
>             with end users and not a single method took more time than
>             what I expected, so I'm completely lost here.
>
>             Any help will be appreciated! :)

--
Janko Mivšek
Aida/Web
Smalltalk Web Application Server
http://www.aidaweb.si

--
Bernat Romagosa.

It's a Pharo 1.4 image on this VM:

'CoInterpreter VMMaker-oscog-EstebanLorenzano.161 uuid: 8e0c22c3-b48d-4d8d-a7f9-8a75dc246f28 Jul 18 2012, StackToRegisterMappingCogit VMMaker-oscog-EstebanLorenzano.161 uuid: 8e0c22c3-b48d-4d8d-a7f9-8a75dc246f28 Jul 18 2012, https://git.gitorious.org/cogvm/blessed.git Commit: 5151310c41b08f55b70e9c6250711cb1f3672ce7 Date: 2012-07-18 14:06:15 +0200 By: Camillo Bruni <[hidden email]>'

(On a Debian VPS)

2013/9/25 Paul DeBruicker <[hidden email]>

Which image and vm ?

On Sep 25, 2013, at 3:24 AM, Bernat Romagosa <[hidden email]> wrote:

Hmmm something weird just happened. I restarted Komanche, then when I saved the image and the CPU rushed to 100%!

2013/9/25 Janko Mivšek <[hidden email]>

Hi Bernat,

I remember once having similar problems on Swazoo. But on image restart.
If Swazoo was not stopped before image shutdown, image went into 100%
CPU after restart. Something like that. It seems therefore that this is
sockets related problem

What if you stop your web server and start again?

Best regards
Janko

Dne 25. 09. 2013 10:03, piše Bernat Romagosa:

> Forgot to answer that I'm not using RFB. I had sooo many problems with

> it that since last year I've been using xpra <http://xpra.org/> instead,

> which lets you capture running X windows remotely over SSH. It's kind of
> like screen (not even remotely so powerful) but for GUIs.
>
> Cheers,
> Bernat.
>
>
> 2013/9/25 Bernat Romagosa <[hidden email]

> <mailto:[hidden email]>>

>
>     Not really, the image is not being saved regularly... all data is
>     stored in a neo4j database, so there's no need to save the image.
>
>     It seems like it may have something to do with users uploading files
>     in some particular moments, but I'm using Göran's file upload
>     mechanism, in which files are streamed straight to disk, so there
>     shouldn't be any image clogging problems even for huge files...
>
>     2013/9/24 Mariano Martinez Peck <[hidden email]

>     <mailto:[hidden email]>>

>
>         Bernat, could this be triggered when you SAVE the image? If
>         true, do you happen to have RFB running?
>
>
>         On Tue, Sep 24, 2013 at 4:45 AM, Bernat Romagosa
>         <[hidden email]

>         <mailto:[hidden email]>> wrote:
>
>             Hi list,
>
>             A couple months ago I deployed an Iliad application in a
>             Debian VPS, which has been working great except for the fact
>             that about every two weeks the image takes up 100% of the
>             CPU and, of course, the app becomes unusable.
>
>             Does anyone have a method for finding out what's going on?
>             I've been trying to debug the problem, but as I don't know
>             what triggers it, it's pretty impossible to detect...
>
>             I've checked the running threads and nothing seems unusual.
>             I've also profiled the app for a whole day of real usage
>             with end users and not a single method took more time than
>             what I expected, so I'm completely lost here.
>
>             Any help will be appreciated! :)

--
Janko Mivšek
Aida/Web
Smalltalk Web Application Server
http://www.aidaweb.si

--
Bernat Romagosa.

--
Bernat Romagosa.