[Glass] out of resources

Hi,

With GS 3.1 we're running out of semaphores and shared memory because
the system resources are not freed up.

When we run ipcs, we get lists of shared memory segments (about 1GB
each) that report no attached processes. When we use ipcrm -m <id>,
the memory is free.

ipcs -s shows a long list of semaphore arrays. When using ipcs -s -i
<array id>, we see that the referenced processes are dead.

This happens on our jenkins machines where we start & stop GS a lot.
(Jobs running tests restore from a built GS backup.) We think we are
using stopstone properly (with waitstone to make sure it is stopped,
etc.). We need to investigate properly and make sure the jenkins jobs
do this in the way we are expecting.

I was hoping someone can give us some ideas on this. Perhaps there's a
GS flag or a OS setup that we're missing. Your insights are
appreciated.

Thanks
Otto
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

Thanks for the response

> The way that we use shared memory resources, we have things setup so that
> the resources are deallocated when the last process detaches from the cache.
>
> So I would look around the system for rogue stoned, topaz or gem processes
> that may be hung/refusing to quit ... if you find some hanging around note
> their process ids and track down their log file...there should be
> information there as to why they are hung ...

There are no rogue stoned, topaz, gem or shared cache monitor
processes hanging around. The output from ipcs -m tells me that there
are no processes attached. eg:

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x6a0180a5 819200     wonka      660        1059602432 0
0x16018585 1146881    wonka      660        1059602432 0
0x950180a5 163844     wonka      660        1059602432 0
0xfd0180a5 458757     wonka      660        1059602432 0

The command

ipcrm -m 819200

frees up the memory instantly (the docs says that it frees up only
after the last process detatches), which is consistent with the nattch
count of 0.

> Another thing is if you 'kill -9' the shrpc monitor process, the shared
> memory segment will not be told to clean up when the last process detaches
> and the shared memore/semaphores will be left around.

No shrpc process around

> While not recommended, it is "safe" to kill -9 the stoned process as a last
> resort, you will lose any transactions that are in progress and will need to
> restore from tranlogs on restart, but kill -9 on stoned should not corrupt
> the db  ... the shrpc monitor process is really the only process that it is
> not "safe" to use kill -9 on but even then the db will not be corrupted,
> like killing the stone, you will lose any transactions in progress AND you
> will leave shared memory resources around to be cleaned up manually ...

AFAIK we are not killing processes. But we should have a very close
look to make sure.

What I can conclude from your response is that it is unexpected then
that these resources are not freed up?

Thanks
Otto