[Glass] shrpcmonitor fails to start because lock file doesn't exits

Mariano,

I think these lines should be diagnostic:

  |   GemStone could not retrieve the IPC identifier associated with the memory |

  |   key -704639710.  shmget() error = errno=28,ENOSPC, There is no space left |

  | on the device (or, in fcntl(), there are no more record locks).             |

Well hidden, but I think it is telling you that you don't have enough shared memory on the system where you are starting your second stone ... if you have plenty of RAM on the system you probably need to bump up the size of shared memory 

Dale

On Thu, Jan 23, 2014 at 7:46 AM, Mariano Martinez Peck <[hidden email]> wrote:

Hi guys,

I continue trying to make my mult-site with multi-stone approach and now I am dealing with another problem. I am trying to build the second stone but I get an error when starting the shrpmonitor.

Before I starting this stone, I made sure nothing from this stone was in gslist, and no lock file for this stone was in /opt/gemstone/locks.

The error says

 _____________________________________________________________________________

|             GemStone/S64 Object-Oriented Data Management System             |

|                   Copyright (C) GemTalk Systems 1986-2013                   |

|                            All rights reserved.                             |

+-----------------------------------------------------------------------------+

|    PROGRAM: SHRPCMON, GemStone SharedPageCache Monitor                      |

|    VERSION: 3.1.0.5, Thu Nov 14 13:28:38 2013                               |

|      BUILD: gss64_3_1_0_x_branch-31878                                      |

|  BUILT FOR: x86-64 (Linux)                                                  |

|       MODE: 64 bit                                                          |

| RUNNING ON: 2-CPU centosserver x86_64 (Linux 2.6.32-358.2.1.el6.x86_64 #1 SMP

| Wed Mar 13 00:26:49 UTC 2013) 7870MB                                        |

| PROCESS ID: 7156      DATE: 01/23/14 10:32:44 EST                           |

|   USER IDS: REAL=debristesting (506) EFFECTIVE=debristesting (506)          |

|_____________________________________________________________________________|

|   GemStone could not retrieve the IPC identifier associated with the memory |

|   key -704639710.  shmget() error = errno=28,ENOSPC, There is no space left |

| on the device (or, in fcntl(), there are no more record locks).             |

|                                                                             |

file not found: /opt/gemstone/locks/XXX~8190d706af803f99..LCK  GemStone could not attach to the shared page cache.

  [SpcMon trace]: ... cache creation failed ...

--- 01/23/14 10:32:44.744 EST :Starting shrpcmonitor shutdown

--- 01/23/14 10:32:44.744 EST :Waiting for crashed slot recovery thread to shutdown...Done.

--- 01/23/14 10:32:44.744 EST :Waiting for clean slot recovery thread to shutdown...Done.

--- 01/23/14 10:32:44.744 EST :Waiting for stats thread to shutdown...Done.

--- 01/23/14 10:32:44.744 EST :All threads have stopped.

  The Shared Page Cache Monitor is shutting down.

Now...in /opt/gemstone/locks I do have a file 'XXX~8190d706af803f99' but not 'XXX~8190d706af803f99..LCK'

Any idea what can be the problem?

Thanks in advance, 

--
Mariano
http://marianopeck.wordpress.com

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

And now I have also more questions...then thing is like this: I may have several stones running. I want to assign to them the max possible SPC, that is, 2GB. Are those 2GB took from the beginning or on demand? Say I have 6GB of RAM...does it mean I can have (almost) 3 stones only? Or I can have many ones and if I have less RAM, then even if the SPC is 2GB max..they will be smaller...?  

Thanks Dale!

 

Dale

On Thu, Jan 23, 2014 at 7:46 AM, Mariano Martinez Peck <[hidden email]> wrote:

Hi guys,

I continue trying to make my mult-site with multi-stone approach and now I am dealing with another problem. I am trying to build the second stone but I get an error when starting the shrpmonitor.

Before I starting this stone, I made sure nothing from this stone was in gslist, and no lock file for this stone was in /opt/gemstone/locks.

The error says

 _____________________________________________________________________________

|             GemStone/S64 Object-Oriented Data Management System             |

|                   Copyright (C) GemTalk Systems 1986-2013                   |

|                            All rights reserved.                             |

+-----------------------------------------------------------------------------+

|    PROGRAM: SHRPCMON, GemStone SharedPageCache Monitor                      |

|    VERSION: 3.1.0.5, Thu Nov 14 13:28:38 2013                               |

|      BUILD: gss64_3_1_0_x_branch-31878                                      |

|  BUILT FOR: x86-64 (Linux)                                                  |

|       MODE: 64 bit                                                          |

| RUNNING ON: 2-CPU centosserver x86_64 (Linux 2.6.32-358.2.1.el6.x86_64 #1 SMP

| Wed Mar 13 00:26:49 UTC 2013) 7870MB                                        |

| PROCESS ID: 7156      DATE: 01/23/14 10:32:44 EST                           |

|   USER IDS: REAL=debristesting (506) EFFECTIVE=debristesting (506)          |

|_____________________________________________________________________________|

|   GemStone could not retrieve the IPC identifier associated with the memory |

|   key -704639710.  shmget() error = errno=28,ENOSPC, There is no space left |

| on the device (or, in fcntl(), there are no more record locks).             |

|                                                                             |

file not found: /opt/gemstone/locks/XXX~8190d706af803f99..LCK  GemStone could not attach to the shared page cache.

  [SpcMon trace]: ... cache creation failed ...

--- 01/23/14 10:32:44.744 EST :Starting shrpcmonitor shutdown

--- 01/23/14 10:32:44.744 EST :Waiting for crashed slot recovery thread to shutdown...Done.

--- 01/23/14 10:32:44.744 EST :Waiting for clean slot recovery thread to shutdown...Done.

--- 01/23/14 10:32:44.744 EST :Waiting for stats thread to shutdown...Done.

--- 01/23/14 10:32:44.744 EST :All threads have stopped.

  The Shared Page Cache Monitor is shutting down.

Now...in /opt/gemstone/locks I do have a file 'XXX~8190d706af803f99' but not 'XXX~8190d706af803f99..LCK'

Any idea what can be the problem?

Thanks in advance, 

--
Mariano
http://marianopeck.wordpress.com

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

--
Mariano
http://marianopeck.wordpress.com

On Thu, Jan 23, 2014 at 12:50 PM, Dale Henrichs <[hidden email]> wrote:

Mariano,

I think these lines should be diagnostic:

  |   GemStone could not retrieve the IPC identifier associated with the memory |

  |   key -704639710.  shmget() error = errno=28,ENOSPC, There is no space left |

  | on the device (or, in fcntl(), there are no more record locks).             |

Well hidden, but I think it is telling you that you don't have enough shared memory on the system where you are starting your second stone ... if you have plenty of RAM on the system you probably need to bump up the size of shared memory 

Dale, I feel a bit stupid...I saw that, but I ignored it thinking it was impossible. Also, I found some logs from gemstone that tells me something is wrong but it actually isn't so I wasn't very trustful ;)  I can point you to these places once I find them again if you want.

Anyway....I don't understand why that error. My virtualmachine (virtualbox machine) has 8GB assigned. I read in the admin guide that assiging a 0.75 of such for the shared memory is a good idea. So, 6GB is approx. that. So I have:

$ sudo sysctl -A | grep kernel.shm

kernel.shmmax = 6442450944

kernel.shmall = 1572864

kernel.shmmni = 4096

kernel.shm_rmid_forced = 0

Which from what I understand, it is more than enough right?  Also:

$ free -m

             total       used       free     shared    buffers     cached

Mem:          7870       4679       3191          0        163       2561

-/+ buffers/cache:       1954       5916

Swap:        12223          0      12223

so I have 3GB free of RAM and 12 GB of swap space. 

$ sudo ipcs -m

0x93000d26 163845     stone1 660        <a href="tel:2156756992" value="+12156756992" target="_blank">2156756992 15

0xb000130c 196614     stone2 660        <a href="tel:2156756992" value="+12156756992" target="_blank">2156756992 0

Here it seems both stones are as assigned the 2GB shared memory.

so...what do I miss?

And now I have also more questions...then thing is like this: I may have several stones running. I want to assign to them the max possible SPC, that is, 2GB. Are those 2GB took from the beginning or on demand? Say I have 6GB of RAM...does it mean I can have (almost) 3 stones only? Or I can have many ones and if I have less RAM, then even if the SPC is 2GB max..they will be smaller...?