More on stability problems

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

More on stability problems

Vincent Girard-Reydet
Hi,

I'de like to push the discussion on Squeak image's stability & Seaside
to the front again.
We have a squeak image that is running in production:

server: Linux Debian sarge
Squeak VM: 3.9-8 (get from squeaksource.com)
Image: Squeak 3.9 #7067
Seaside: Seaside2.6b1-mb.124 (seaside running on por 8080)

the VM is started as follows:
squeak -nodisplay -mmap 550m myimage.image &

I tried a simple test: launching siege to
http://myhost.mydomain.com:8080/seaside/.

After up to 21 successful hits, the Komanche server seams to fall. No
more response to HTTP queries. I had a REPLServer running on port 23 and
opened in Telnet, it is frozen too. No way to another connection to it
either. I had also a RFB server opened, I saw no error window raise.
There is also no SqueakDebug.log.
The only way to restore the connection is to restart the image.

I performed the same test with other seaside-powered sites
(www.seasidehosting.st, www.squeaksource.com), and they respond perfectly.

Cold anyone provide me a valid configuration known to be stable with
Seaside & Squeak ? I'm even ready to use a Windows server if it can
provide me with the desired stablility.

Thanks for your help !

Vincent
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
Reply | Threaded
Open this post in threaded view
|

Re: More on stability problems

Lukas Renggli
> I performed the same test with other seaside-powered sites
> (www.seasidehosting.st, www.squeaksource.com), and they respond perfectly.
>
> Cold anyone provide me a valid configuration known to be stable with
> Seaside & Squeak ? I'm even ready to use a Windows server if it can
> provide me with the desired stablility.

www.seasidehosting.st:
- Linux 2.6.8-2-686-smp
- Squeak VM 3.7-7
- Squeak 3.8-6665
- Seaside 2.7a1-lr.137

www.squeaksource.com:
- OS X Server 10.4.8
- Squeak VM 3.9-8
- Squeak 3.7
- Seaside 2-lr.94 (this is old)

I could give you at least 10 more working configurations.

Cheers,
Lukas

--
Lukas Renggli
http://www.lukas-renggli.ch
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
Reply | Threaded
Open this post in threaded view
|

Re: More on stability problems

stephane ducasse
please do this is important that we collect these data.

Stef

On 22 déc. 06, at 09:48, Lukas Renggli wrote:

>> I performed the same test with other seaside-powered sites
>> (www.seasidehosting.st, www.squeaksource.com), and they respond  
>> perfectly.
>>
>> Cold anyone provide me a valid configuration known to be stable with
>> Seaside & Squeak ? I'm even ready to use a Windows server if it can
>> provide me with the desired stablility.
>
> www.seasidehosting.st:
> - Linux 2.6.8-2-686-smp
> - Squeak VM 3.7-7
> - Squeak 3.8-6665
> - Seaside 2.7a1-lr.137
>
> www.squeaksource.com:
> - OS X Server 10.4.8
> - Squeak VM 3.9-8
> - Squeak 3.7
> - Seaside 2-lr.94 (this is old)
>
> I could give you at least 10 more working configurations.
>
> Cheers,
> Lukas
>
> --
> Lukas Renggli
> http://www.lukas-renggli.ch
> _______________________________________________
> Seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
Reply | Threaded
Open this post in threaded view
|

Re: More on stability problems

Adrian Lienhard
In reply to this post by Vincent Girard-Reydet
Hi Vincent,

Its hard to tell what the problem is without more information.

I suggest you try to gather more data points. For example:
- don't start headless, see whether UI still responds, try to open a  
debugger on the running process (alt-.)
- if above does not work, attach to the process with gdb and print  
the call stacks and a backtrace
- what is the CPU and memory usage of the Squeak process?

You could also prepare an image which people can download.

We have some other working configurations than the ones Lukas  
mentioned. I've recently investigated the other VM problem which you  
(and we) work around with the -mmap parameter. Apparently the problem  
occurs only on a 2.6 Linux kernel but not on 2.4 (will post more  
details later). Anyway, the problem you are experiencing, seems to be  
something different.

Adrian



On Dec 22, 2006, at 09:37 , Vincent Girard-Reydet wrote:

> Hi,
>
> I'de like to push the discussion on Squeak image's stability &  
> Seaside to the front again.
> We have a squeak image that is running in production:
>
> server: Linux Debian sarge
> Squeak VM: 3.9-8 (get from squeaksource.com)
> Image: Squeak 3.9 #7067
> Seaside: Seaside2.6b1-mb.124 (seaside running on por 8080)
>
> the VM is started as follows:
> squeak -nodisplay -mmap 550m myimage.image &
>
> I tried a simple test: launching siege to http://
> myhost.mydomain.com:8080/seaside/.
>
> After up to 21 successful hits, the Komanche server seams to fall.  
> No more response to HTTP queries. I had a REPLServer running on  
> port 23 and opened in Telnet, it is frozen too. No way to another  
> connection to it either. I had also a RFB server opened, I saw no  
> error window raise. There is also no SqueakDebug.log.
> The only way to restore the connection is to restart the image.
>
> I performed the same test with other seaside-powered sites  
> (www.seasidehosting.st, www.squeaksource.com), and they respond  
> perfectly.
>
> Cold anyone provide me a valid configuration known to be stable  
> with Seaside & Squeak ? I'm even ready to use a Windows server if  
> it can provide me with the desired stablility.
>
> Thanks for your help !
>
> Vincent
> _______________________________________________
> Seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
Reply | Threaded
Open this post in threaded view
|

Re: More on stability problems

Vincent Girard-Reydet
Adrian,

Unfortunaltely I can't do UI because I have no local console to my server.
What I can tell at the moment the problme appears:
   - CPU usage: peak at 99.9% when teh bug appears, then between 88.2
and 89.6 %
   - virt. memory usage: 502M (no change with when no problem)
   - backtrace:
#0  0x0806fdf7 in updatePointersInRangeFromto (memStart=1148,
     memEnd=-1705236052)
     at /var/squeak/Squeak-3.9-8/platforms/unix/src/vm/interp.c:22582
#1  0x0805e8b0 in incCompBody ()
     at /var/squeak/Squeak-3.9-8/platforms/unix/src/vm/interp.c:5047
#2  0x0805ecc6 in incrementalGC ()
     at /var/squeak/Squeak-3.9-8/platforms/unix/src/vm/interp.c:5362
#3  0x0806f924 in sufficientSpaceAfterGC (minFree=200096)
     at /var/squeak/Squeak-3.9-8/platforms/unix/src/vm/interp.c:22256
#4  0x0805be0f in allocateheaderSizeh1h2h3doFillwith (byteSize=92,
hdrSize=1,
     baseHeader=350282588, classOop=-1729481856, extendedSize=-1729481856,
     doFill=0, fillWord=0)
     at /var/squeak/Squeak-3.9-8/platforms/unix/src/vm/interp.c:2443
#5  0x0805f583 in instantiateContextsizeInBytes (classPointer=-1724896820,
     sizeInBytes=92)
     at /var/squeak/Squeak-3.9-8/platforms/unix/src/vm/interp.c:5850
#6  0x0805bc82 in allocateOrRecycleContext (needsLarge=1148)
     at /var/squeak/Squeak-3.9-8/platforms/unix/src/vm/interp.c:2401
#7  0x08061058 in interpret ()
     at /var/squeak/Squeak-3.9-8/platforms/unix/src/vm/interp.c:7783
#8  0x0805b3c8 in main (argc=5, argv=0xbfa3a524, envp=0x98ea3380)
     at /var/squeak/Squeak-3.9-8/platforms/unix/vm/sqUnixMain.c:1388


I start my seaside as follows:

| ma seaside |
seaside := WAKomEncoded39 default.
ma := ModuleAssembly core.
ma alias: '/seaside' to: [ma addPlug: [:request | seaside process:
request]].
documentRoot := '/var/squeak/DocumentRoot/'
ma documentRoot: FileDirectory default directoryNamed: 'DocumentRoot')
fullName.
ma serveFiles.
(HttpService startOn: aPort named: 'httpd') plug: ma rootModule

Vincent

Adrian Lienhard wrote:

> Hi Vincent,
>
> Its hard to tell what the problem is without more information.
>
> I suggest you try to gather more data points. For example:
> - don't start headless, see whether UI still responds, try to open a
> debugger on the running process (alt-.)
> - if above does not work, attach to the process with gdb and print the
> call stacks and a backtrace
> - what is the CPU and memory usage of the Squeak process?
>
> You could also prepare an image which people can download.
>
> We have some other working configurations than the ones Lukas mentioned.
> I've recently investigated the other VM problem which you (and we) work
> around with the -mmap parameter. Apparently the problem occurs only on a
> 2.6 Linux kernel but not on 2.4 (will post more details later). Anyway,
> the problem you are experiencing, seems to be something different.
>
> Adrian
>
>
>
> On Dec 22, 2006, at 09:37 , Vincent Girard-Reydet wrote:
>
>> Hi,
>>
>> I'de like to push the discussion on Squeak image's stability & Seaside
>> to the front again.
>> We have a squeak image that is running in production:
>>
>> server: Linux Debian sarge
>> Squeak VM: 3.9-8 (get from squeaksource.com)
>> Image: Squeak 3.9 #7067
>> Seaside: Seaside2.6b1-mb.124 (seaside running on por 8080)
>>
>> the VM is started as follows:
>> squeak -nodisplay -mmap 550m myimage.image &
>>
>> I tried a simple test: launching siege to
>> http://myhost.mydomain.com:8080/seaside/.
>>
>> After up to 21 successful hits, the Komanche server seams to fall. No
>> more response to HTTP queries. I had a REPLServer running on port 23
>> and opened in Telnet, it is frozen too. No way to another connection
>> to it either. I had also a RFB server opened, I saw no error window
>> raise. There is also no SqueakDebug.log.
>> The only way to restore the connection is to restart the image.
>>
>> I performed the same test with other seaside-powered sites
>> (www.seasidehosting.st, www.squeaksource.com), and they respond
>> perfectly.
>>
>> Cold anyone provide me a valid configuration known to be stable with
>> Seaside & Squeak ? I'm even ready to use a Windows server if it can
>> provide me with the desired stablility.
>>
>> Thanks for your help !
>>
>> Vincent
>> _______________________________________________
>> Seaside mailing list
>> [hidden email]
>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>
> _______________________________________________
> Seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
Reply | Threaded
Open this post in threaded view
|

Re: More on stability problems

Vincent Girard-Reydet
In reply to this post by Lukas Renggli
Lukas,

I tried the first config (from seasidehosting.st) and it works fine!
Thanks a lot!
So it means there's a problem with my previous config.

I'll try to investigate it further. Anyone interested, I can provide an
image that exposes the problem.

Vincent


Lukas Renggli wrote:

>> I performed the same test with other seaside-powered sites
>> (www.seasidehosting.st, www.squeaksource.com), and they respond
>> perfectly.
>>
>> Cold anyone provide me a valid configuration known to be stable with
>> Seaside & Squeak ? I'm even ready to use a Windows server if it can
>> provide me with the desired stablility.
>
> www.seasidehosting.st:
> - Linux 2.6.8-2-686-smp
> - Squeak VM 3.7-7
> - Squeak 3.8-6665
> - Seaside 2.7a1-lr.137
>
> www.squeaksource.com:
> - OS X Server 10.4.8
> - Squeak VM 3.9-8
> - Squeak 3.7
> - Seaside 2-lr.94 (this is old)
>
> I could give you at least 10 more working configurations.
>
> Cheers,
> Lukas
>

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
Reply | Threaded
Open this post in threaded view
|

Re: More on stability problems

Lukas Renggli
In reply to this post by Vincent Girard-Reydet
> I start my seaside as follows:
>
> | ma seaside |
> seaside := WAKomEncoded39 default.
> ma := ModuleAssembly core.
> ma alias: '/seaside' to: [ma addPlug: [:request | seaside process:
> request]].
> documentRoot := '/var/squeak/DocumentRoot/'
> ma documentRoot: FileDirectory default directoryNamed: 'DocumentRoot')
> fullName.
> ma serveFiles.
> (HttpService startOn: aPort named: 'httpd') plug: ma rootModule

Just a guess, but it might help to reduce the problem area:

- is the problem reproduceable if you start the server using
WAKomEncoded39>>#startOn:?

- is the problem reproduceable if you start the server using WAKom>>#startOn:?

Cheers,
Lukas

--
Lukas Renggli
http://www.lukas-renggli.ch
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
Reply | Threaded
Open this post in threaded view
|

Re: More on stability problems

Adrian Lienhard
In reply to this post by Vincent Girard-Reydet
Hi Vincent,

On Dec 22, 2006, at 11:43 , Vincent Girard-Reydet wrote:

> Adrian,
>
> Unfortunaltely I can't do UI because I have no local console to my  
> server.

and e.g., with VNC?

> What I can tell at the moment the problme appears:
>   - CPU usage: peak at 99.9% when teh bug appears, then between  
> 88.2 and 89.6 %
>   - virt. memory usage: 502M (no change with when no problem)

and RES memory? i.e., did you already hit the max heap size which you  
set with -mmap?

>   - backtrace:
> #0  0x0806fdf7 in updatePointersInRangeFromto (memStart=1148,
>     memEnd=-1705236052)
>     at /var/squeak/Squeak-3.9-8/platforms/unix/src/vm/interp.c:22582
> #1  0x0805e8b0 in incCompBody ()
>     at /var/squeak/Squeak-3.9-8/platforms/unix/src/vm/interp.c:5047
> #2  0x0805ecc6 in incrementalGC ()
>     at /var/squeak/Squeak-3.9-8/platforms/unix/src/vm/interp.c:5362
> #3  0x0806f924 in sufficientSpaceAfterGC (minFree=200096)
>     at /var/squeak/Squeak-3.9-8/platforms/unix/src/vm/interp.c:22256
> #4  0x0805be0f in allocateheaderSizeh1h2h3doFillwith (byteSize=92,  
> hdrSize=1,
>     baseHeader=350282588, classOop=-1729481856,  
> extendedSize=-1729481856,
>     doFill=0, fillWord=0)
>     at /var/squeak/Squeak-3.9-8/platforms/unix/src/vm/interp.c:2443
> #5  0x0805f583 in instantiateContextsizeInBytes  
> (classPointer=-1724896820,
>     sizeInBytes=92)
>     at /var/squeak/Squeak-3.9-8/platforms/unix/src/vm/interp.c:5850
> #6  0x0805bc82 in allocateOrRecycleContext (needsLarge=1148)
>     at /var/squeak/Squeak-3.9-8/platforms/unix/src/vm/interp.c:2401
> #7  0x08061058 in interpret ()
>     at /var/squeak/Squeak-3.9-8/platforms/unix/src/vm/interp.c:7783
> #8  0x0805b3c8 in main (argc=5, argv=0xbfa3a524, envp=0x98ea3380)
>     at /var/squeak/Squeak-3.9-8/platforms/unix/vm/sqUnixMain.c:1388

looks familiar to me... see http://bugs.impara.de/view.php?id=4709,  
however, as said before, I could never reproduce the problem when  
using -mmap, so maybe it has nothing to do with that. If you have a  
chance to test this on linux kernel <= 2.4 (e.g., Debian woody).

Could you post the output of printAllCallStacks() and printCallStack
() (or something like that)?

And yes, if you could post an image, I'll have a look.

Adrian

>
>
> I start my seaside as follows:
>
> | ma seaside |
> seaside := WAKomEncoded39 default.
> ma := ModuleAssembly core.
> ma alias: '/seaside' to: [ma addPlug: [:request | seaside process:  
> request]].
> documentRoot := '/var/squeak/DocumentRoot/'
> ma documentRoot: FileDirectory default directoryNamed:  
> 'DocumentRoot') fullName.
> ma serveFiles.
> (HttpService startOn: aPort named: 'httpd') plug: ma rootModule
>
> Vincent
>
> Adrian Lienhard wrote:
>> Hi Vincent,
>> Its hard to tell what the problem is without more information.
>> I suggest you try to gather more data points. For example:
>> - don't start headless, see whether UI still responds, try to open  
>> a debugger on the running process (alt-.)
>> - if above does not work, attach to the process with gdb and print  
>> the call stacks and a backtrace
>> - what is the CPU and memory usage of the Squeak process?
>> You could also prepare an image which people can download.
>> We have some other working configurations than the ones Lukas  
>> mentioned. I've recently investigated the other VM problem which  
>> you (and we) work around with the -mmap parameter. Apparently the  
>> problem occurs only on a 2.6 Linux kernel but not on 2.4 (will  
>> post more details later). Anyway, the problem you are  
>> experiencing, seems to be something different.
>> Adrian
>> On Dec 22, 2006, at 09:37 , Vincent Girard-Reydet wrote:
>>> Hi,
>>>
>>> I'de like to push the discussion on Squeak image's stability &  
>>> Seaside to the front again.
>>> We have a squeak image that is running in production:
>>>
>>> server: Linux Debian sarge
>>> Squeak VM: 3.9-8 (get from squeaksource.com)
>>> Image: Squeak 3.9 #7067
>>> Seaside: Seaside2.6b1-mb.124 (seaside running on por 8080)
>>>
>>> the VM is started as follows:
>>> squeak -nodisplay -mmap 550m myimage.image &
>>>
>>> I tried a simple test: launching siege to http://
>>> myhost.mydomain.com:8080/seaside/.
>>>
>>> After up to 21 successful hits, the Komanche server seams to  
>>> fall. No more response to HTTP queries. I had a REPLServer  
>>> running on port 23 and opened in Telnet, it is frozen too. No way  
>>> to another connection to it either. I had also a RFB server  
>>> opened, I saw no error window raise. There is also no  
>>> SqueakDebug.log.
>>> The only way to restore the connection is to restart the image.
>>>
>>> I performed the same test with other seaside-powered sites  
>>> (www.seasidehosting.st, www.squeaksource.com), and they respond  
>>> perfectly.
>>>
>>> Cold anyone provide me a valid configuration known to be stable  
>>> with Seaside & Squeak ? I'm even ready to use a Windows server if  
>>> it can provide me with the desired stablility.
>>>
>>> Thanks for your help !
>>>
>>> Vincent
>>> _______________________________________________
>>> Seaside mailing list
>>> [hidden email]
>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>> _______________________________________________
>> Seaside mailing list
>> [hidden email]
>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>
> _______________________________________________
> Seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
Reply | Threaded
Open this post in threaded view
|

Re: More on stability problems

Bryce Kampjes
In reply to this post by Vincent Girard-Reydet
Vincent Girard-Reydet writes:
 > Adrian,
 >
 > Unfortunaltely I can't do UI because I have no local console to my server.
 > What I can tell at the moment the problme appears:
 >    - CPU usage: peak at 99.9% when teh bug appears, then between 88.2
 > and 89.6 %
 >    - virt. memory usage: 502M (no change with when no problem)
 >    - backtrace:
 > #0  0x0806fdf7 in updatePointersInRangeFromto (memStart=1148,
 >      memEnd=-1705236052)

Given that backtrace I'd try playing around with the -memory option.
The values for memStart and memEnd look suspicious to me. I'd suspect
that you've run into a signed arithmetic with pointers bug.

That can happen if your object memory crosses the 2G point as the
values below will be positive and the values above negative.

Bryce
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside