Trouble upgrading 3.1.0.1 to 3.1.0.2

I tried to upgrade my gemstone version today in order to get rid of the memory leak problem. Are there upgrade instructions for this? I read the release notes and these point to the old installation guide. I'm not sure if the steps in the installation guide need to be repeated. I mean the "upgradeImage…".

Anyway I don't get that far. I did shutdown the stone, unpacked to 3.1.0.2 release and adjusted permissions and environment settings. But trying to start the stone failed soon. I attach the logs I've got in the startup process.

kernel setting didn't change. They are

kernel.shmmax=2147483648
kernel.shmall=2097152

There are only netldi lock files in /opt/gemstone/locks when trying to start. And there are no *pcmon* logs.

Any help is welcomed,

Norbert

Norbert,

The logs do look like there is a shared page cache configuration problem. Do you have another stone running? Is there any left-over shared memory? The linux command 'ipcs' should give you a list of shared resources. Is this a machine that you can reboot easily? If so, then that would be a clean way to start. You could change the configuration on the stone to use less shared page cache (just as a temporary measure to test the theory that there is a configuration issue). Are you able to restart back to the 3.1.0.1 system without error? Can you increase the kernel.shm* settings (and reboot) to see if that helps?

James

On Apr 14, 2013, at 7:10 AM, Norbert Hartl <[hidden email]> wrote:

I'd imagine that you've got another stone (or two or three) running or the SPC cache for old stones were not cleanly shut down and as James suggests `ipcs -a` should give you a pretty good picture. If there are additional memory and semaphore segments in use `ipcrm` is the command to clean them up...

I wouldn't rule out the possibility that 3.1.0.2 would take a bit more shared memory than 3.1.0.1 (I recall discussions about allocating more shared memory for some additional data structures, but don't recall if that was about 3.2 or not). Nonetheless I wouldn't think 3.1.0.2 would be _significantly larger_ (check with the devs on Monday), but it could be significant if you'd sized your shared memroy on your system to _just fit_.

A rogue SPC is the most likely culprit... if not then James' mail pretty much covers all of the bases ...

Dale

----- Original Message -----
| From: "Norbert Hartl" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Sunday, April 14, 2013 5:10:21 AM
| Subject: [GS/SS Beta] Trouble upgrading 3.1.0.1 to 3.1.0.2
|
| I tried to upgrade my gemstone version today in order to get rid of the
| memory leak problem. Are there upgrade instructions for this? I read the
| release notes and these point to the old installation guide. I'm not sure if
| the steps in the installation guide need to be repeated. I mean the
| "upgradeImage…".
|
| Anyway I don't get that far. I did shutdown the stone, unpacked to 3.1.0.2
| release and adjusted permissions and environment settings. But trying to
| start the stone failed soon. I attach the logs I've got in the startup
| process.
|
| kernel setting didn't change. They are
|
| kernel.shmmax=2147483648
| kernel.shmall=2097152
|
| There are only netldi lock files in /opt/gemstone/locks when trying to start.
| And there are no *pcmon* logs.
|
| Any help is welcomed,
|
| Norbert
|
|

Norbert,

Following the stale SPC thread just a bit more ... you may have a topaz and gem process that did not shutdown correctly and linux will keep the shared memory segment around as long as processes are attached .... killing the roque process should clean up a rogue SPC ...

My math may be rusty on Sunday morning (and I haven't read all of the relevant man pages), but you are using a 2G SPC for the stone and your shmmax is suspiciously close to 2G:). So it is worth pointing out that a 2G SPC may take more that 2G of ram ... the formula is actually pretty complicated for calculating the exact amount of RAM that will be used, but I'm pretty certain that when you ask for a 2G SPC more than 2G of shared memory will be allocated ...

With the shmmax so close to the size of SPC requested, any small change in shared memory allocation between 3.1.0.1 and 3.1.0.2 could have triggered the failure ...

So if you don't have any rogue SPCs sitting around you may want to either bump the shmmax a bit or reduce your SHR_PAGE_CACHE_SIZE_KB a bit...

Dale

----- Original Message -----
| From: "Norbert Hartl" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Sunday, April 14, 2013 5:10:21 AM
| Subject: [GS/SS Beta] Trouble upgrading 3.1.0.1 to 3.1.0.2
|
| I tried to upgrade my gemstone version today in order to get rid of the
| memory leak problem. Are there upgrade instructions for this? I read the
| release notes and these point to the old installation guide. I'm not sure if
| the steps in the installation guide need to be repeated. I mean the
| "upgradeImage…".
|
| Anyway I don't get that far. I did shutdown the stone, unpacked to 3.1.0.2
| release and adjusted permissions and environment settings. But trying to
| start the stone failed soon. I attach the logs I've got in the startup
| process.
|
| kernel setting didn't change. They are
|
| kernel.shmmax=2147483648
| kernel.shmall=2097152
|
| There are only netldi lock files in /opt/gemstone/locks when trying to start.
| And there are no *pcmon* logs.
|
| Any help is welcomed,
|
| Norbert
|
|

Am 14.04.2013 um 16:02 schrieb James Foster <[hidden email]>:

> Norbert,
>
> The logs do look like there is a shared page cache configuration problem. Do you have another stone running? Is there any left-over shared memory? The linux command 'ipcs' should give you a list of shared resources. Is this a machine that you can reboot easily? If so, then that would be a clean way to start. You could change the configuration on the stone to use less shared page cache (just as a temporary measure to test the theory that there is a configuration issue). Are you able to restart back to the 3.1.0.1 system without error? Can you increase the kernel.shm* settings (and reboot) to see if that helps?
>
Thanks James. I forgot to write all of the details. I don't have another stone running and of course I tried rebooting the machine. I could easily go back to 3.1.0.1. At that point I wrote the mail. I just didn't think of the possibility that the shared memory setup could be the culprit. I reduced the SPC size and it works. Thanks. The weird thing is that I can't remember setting the SPC that high because it is only a test server :)

Norbert

Am 14.04.2013 um 18:35 schrieb Dale Henrichs <[hidden email]>:
Sure, that was it.

thanks,

Norbert