Starting a repository fails after shutdown of server.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Starting a repository fails after shutdown of server.

Gemstone/S mailing list
Hello,

I'm having an extrange error while tryng to start a repository.

Here is the important part of "repository.log":
*
========================================================================
    Now starting GemStone monitor.

Write to /proc/6578/oom_score_adj failed with EACCES , linux user does not
have CAP_SYS_RESOURCE
No server process protection from OOM killer

_____________________________________________________________________________
|     SESSION CONFIGURATION: The maximum number of concurrent sessions is
40. |
|_____________________________________________________________________________|

    Attaching the Shared Cache using Stone name: Newstone


WARNING: CONCURRENCY_MODE is not a known GemStone option.
    Started free frame page server, processId 6587
    Successfully started 1 free frame page servers.

    -------------------------------------------------------
    Summary of Configured Transaction Logs
      Directory   0:
        configured name $repositoryDir/tranlogs
        expanded name /mnt/disk3/New/tranlogs/
        configuredSize 100 MB
      Directory   1:
        configured name $repositoryDir/tranlogs
        expanded name /mnt/disk3/New/tranlogs/
        configuredSize 100 MB
    -------------------------------------------------------
    Started AIO page server, processId 6590

--- 09/21/17 11:54:22 -03 ---
    Extent #0
    -----------
    Filename = !#dbf!/mnt/disk3/New/data/extent_01.dbf
    Maximum size = NONE
    File size = 70 Mbytes = 4480 pages
    Space available = 48 Mbytes = 3109 pages

    Totals
    ------
    Repository Size = 70 Mbytes = 4480 pages
    Free Space = 48 Mbytes = 3109 pages
    ---------------------------------------------------
    In extent 0, maxSize is inconsistent.

    Repository was not shutdown cleanly, recovery needed.
    Starting recovery of extent sizes
    changing the maximum size from UNLIMITED MB to 3500 MB for extent:
      !#dbf!/mnt/disk3/New/data/extent_01.dbf    Finished recovery of extent
size changes

 
    Repository startup statistics:
        Pages Need Reclaiming =1
        Free Oops=9144
        Oop Number High Water Mark=130800
        Possible Dead Objects=0
        Dead Objects=0
        Epoch Transaction Count=0
        Epoch New Objects Union=0
        Epoch Written Objects Union=0
        Epoch DependencyMap Objects Union=0

    Repository startup is from checkpoint = (fileId 0, blockId 32305)

   SearchForMostRecentLog did not find any tranlogs

 :: (wildcard) found in listening addresses, ignoring other addresses
created listening socket for :: on :: port 39755

    Opened page manager thread log
/mnt/disk3/New/logs/repository_6578pagemanager.log

    Creating a new transaction log.
       directoryId 0, fileId 1,
       filename = /mnt/disk3/New/tranlogs/tranlog1.dbf

--- 09/21/17 11:54:24 -03 ---
    Starting AF_INET reconnect to AIO pgsvrs
AioServerReconnect pgsvrPid 6590 old fd 12 new fd -1
reconnect failed(B) for AIO pgsvr pid 6590

--- 09/21/17 11:56:25 -03 ---
RDbfAioServersReconnect failed

    Terminating stone.
*

As I understand, the problem is related to de page server manager, but I
tried with different configurations on the machine but couldn't resolve this
problem.

This problem started recently, after shutdown of the machine.

I am pretty sure that there is some configuration that I'm missing, but I
didn't find the problem yet.

The OS is Linux CentOS 6 and I'm using gemstone 3.2.8.1.

Could you help me please? Which configurations should i check?

Thanks in advance!

Ezequiel.




--
Sent from: http://forum.world.st/Gemstone-S-f1461796.html
_______________________________________________
GemStone-Smalltalk mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk
Reply | Threaded
Open this post in threaded view
|

Re: Starting a repository fails after shutdown of server.

Gemstone/S mailing list


On Fri, Sep 22, 2017 at 4:23 PM, brianstone via GemStone-Smalltalk <[hidden email]> wrote:
Hello,

I'm having an extrange error while tryng to start a repository.

Here is the important part of "repository.log":
*
========================================================================
    Now starting GemStone monitor.

Write to /proc/6578/oom_score_adj failed with EACCES , linux user does not
have CAP_SYS_RESOURCE
No server process protection from OOM killer

_____________________________________________________________________________
|     SESSION CONFIGURATION: The maximum number of concurrent sessions is
40. |
|_____________________________________________________________________________|

    Attaching the Shared Cache using Stone name: Newstone


WARNING: CONCURRENCY_MODE is not a known GemStone option.
    Started free frame page server, processId 6587
    Successfully started 1 free frame page servers.

    -------------------------------------------------------
    Summary of Configured Transaction Logs
      Directory   0:
        configured name $repositoryDir/tranlogs
        expanded name /mnt/disk3/New/tranlogs/
        configuredSize 100 MB
      Directory   1:
        configured name $repositoryDir/tranlogs
        expanded name /mnt/disk3/New/tranlogs/
        configuredSize 100 MB
    -------------------------------------------------------
    Started AIO page server, processId 6590

--- 09/21/17 11:54:22 -03 ---
    Extent #0
    -----------
    Filename = !#dbf!/mnt/disk3/New/data/extent_01.dbf
    Maximum size = NONE
    File size = 70 Mbytes = 4480 pages
    Space available = 48 Mbytes = 3109 pages

    Totals
    ------
    Repository Size = 70 Mbytes = 4480 pages
    Free Space = 48 Mbytes = 3109 pages
    ---------------------------------------------------
    In extent 0, maxSize is inconsistent.

    Repository was not shutdown cleanly, recovery needed.
    Starting recovery of extent sizes
    changing the maximum size from UNLIMITED MB to 3500 MB for extent:
      !#dbf!/mnt/disk3/New/data/extent_01.dbf    Finished recovery of extent
size changes


    Repository startup statistics:
        Pages Need Reclaiming =1
        Free Oops=9144
        Oop Number High Water Mark=130800
        Possible Dead Objects=0
        Dead Objects=0
        Epoch Transaction Count=0
        Epoch New Objects Union=0
        Epoch Written Objects Union=0
        Epoch DependencyMap Objects Union=0

    Repository startup is from checkpoint = (fileId 0, blockId 32305)

   SearchForMostRecentLog did not find any tranlogs

 :: (wildcard) found in listening addresses, ignoring other addresses
created listening socket for :: on :: port 39755

    Opened page manager thread log
/mnt/disk3/New/logs/repository_6578pagemanager.log

    Creating a new transaction log.
       directoryId 0, fileId 1,
       filename = /mnt/disk3/New/tranlogs/tranlog1.dbf

--- 09/21/17 11:54:24 -03 ---
    Starting AF_INET reconnect to AIO pgsvrs
AioServerReconnect pgsvrPid 6590 old fd 12 new fd -1
reconnect failed(B) for AIO pgsvr pid 6590

--- 09/21/17 11:56:25 -03 ---
RDbfAioServersReconnect failed

    Terminating stone.
*

As I understand, the problem is related to de page server manager, but I
tried with different configurations on the machine but couldn't resolve this
problem.


Yeah, the issue does not seem to be in the stone process itself. You should send the rest of the logs. 
Maybe there is something useful in XXXpcmon.log

 
This problem started recently, after shutdown of the machine.

I am pretty sure that there is some configuration that I'm missing, but I
didn't find the problem yet.

The OS is Linux CentOS 6 and I'm using gemstone 3.2.8.1.

Could you help me please? Which configurations should i check?

Thanks in advance!

Ezequiel.




--
Sent from: http://forum.world.st/Gemstone-S-f1461796.html
_______________________________________________
GemStone-Smalltalk mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk



--

_______________________________________________
GemStone-Smalltalk mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk
Reply | Threaded
Open this post in threaded view
|

Re: Starting a repository fails after shutdown of server.

Gemstone/S mailing list
It's true, sorry.

Here is *pcmon.log:

*

_____________________________________________________________________________
|                             GemStone Child Task                            
|
|                                                                            
|
|    VERSION: 3.2.8.1, Fri Aug 28 08:43:23 2015                              
|
|      BUILD: gss64_3_2_x_branch-37291                                      
|
|  BUILT FOR: x86-64 (Linux)                                                
|
|       MODE: 64 bit                                                        
|
| RUNNING ON: 12-CPU SRVR23 x86_64 (Linux 2.6.32-642.15.1.el6.x86_64 #1 SMP
Fri
| Feb 24 14:31:22 UTC 2017) 32059MB                                          
|
| PROCESS ID: 6580      DATE: 09/21/17 11:54:20 -03                          
|
|   USER IDS: REAL=gemst643281 (1045) EFFECTIVE=gemst643281 (1045)
LOGIN=gemst643281
| (1045)                                                                    
|
|    COMMAND: /usr/local/gemstone643281/sys/startshrpcmon
Newstone~7663a27bab8c7a96
| 312500 51 1 5000 7000 1 1900 0 432 0 60                                    
|
|_____________________________________________________________________________|
startshrpcmon.sh GEMSTONE: /usr/local/gemstone643281
SharedPageCache Monitor Summary:
   the hostname is:    SRVR23
   GEMSTONE is:        "/usr/local/gemstone643281"
   shrpcmon arguments are:
      SharedPageCache Monitor name  Newstone~7663a27bab8c7a96.
      Number of pages               312500.
      Number of processes           51.
      Cache locked in memory        1.
      SpinLockCount                 5000.
      TargetFreeFrameCount          7000.
      Number of free frame servers  1.
      Number of shared counters     1900.
      Port number                   0.
      Permissions                   432.
      Large Page Policy             0.
      Stone/cachepgsvr connect timeout 60.

_____________________________________________________________________________
|             GemStone/S64 Object-Oriented Data Management System            
|
|                   Copyright (C) GemTalk Systems 1986-2015                  
|
|                            All rights reserved.                            
|
+-----------------------------------------------------------------------------+
|    PROGRAM: SHRPCMON, GemStone SharedPageCache Monitor                    
|
|    VERSION: 3.2.8.1, Fri Aug 28 08:43:23 2015                              
|
|      BUILD: gss64_3_2_x_branch-37291                                      
|
|  BUILT FOR: x86-64 (Linux)                                                
|
|       MODE: 64 bit                                                        
|
| RUNNING ON: 12-CPU SRVR23 x86_64 (Linux 2.6.32-642.15.1.el6.x86_64 #1 SMP
Fri
| Feb 24 14:31:22 UTC 2017) 32059MB                                          
|
| PROCESS ID: 6580      DATE: 09/21/17 11:54:20 -03                          
|
|   USER IDS: REAL=gemst643281 (1045) EFFECTIVE=gemst643281 (1045)
LOGIN=gemst643281
| (1045)                                                                    
|
|_____________________________________________________________________________|

Cache config is 312500 pages = 4882MB,  total is 5232MB,  overhead 7% of
configured size
[Info]: Building the free frame list using 1 cache region.
  WARNING:  Shared cache could not be locked in memory because
            the memory available is insufficient. (ENOMEM)
Shared memory region number 1 has id = 360452
The shared semaphore array has id = 327682


--- 09/21/17 11:54:21.258 -03 cache creation took 1 seconds

created listening socket for localhost6 on ::1 port 38772
synthesizing   127.0.0.1 , attempting listen on IPv4 localhost
created listening socket for localhost on ::ffff:127.0.0.1 port 38772

--- 09/21/17 11:54:21.273 -03 Listening for clients on port 38772


Write to /proc/6580/oom_score_adj failed with EACCES , linux user does not
have CAP_SYS_RESOURCE
No server process protection from OOM killer
timeout for stone/cache pgsvr connect is 20 seconds
[09/21/17 11:56:25.819 -03]: Client died: Slot    1, PID    6578,
LostOtFlags    0, sessionId 0 Name Newstone
--- 09/21/17 11:56:25.820 -03 Starting shrpcmonitor shutdown
--- 09/21/17 11:56:25.820 -03 Waiting for crashed slot recovery thread to
shutdown...Done.
--- 09/21/17 11:56:25.821 -03 Waiting for clean slot recovery thread to
shutdown...Done.
--- 09/21/17 11:56:25.821 -03 Waiting for stats thread to shutdown...Done.
--- 09/21/17 11:56:25.890 -03 All threads have stopped.
  The Shared Page Cache Monitor is shutting down.
*

And here is *pagemanager.log:
*

_____________________________________________________________________________
|                          Stone Page Manager Thread                        
|
|                                                                            
|
|    VERSION: 3.2.8.1, Fri Aug 28 08:43:23 2015                              
|
|      BUILD: gss64_3_2_x_branch-37291                                      
|
|  BUILT FOR: x86-64 (Linux)                                                
|
|       MODE: 64 bit                                                        
|
| RUNNING ON: 12-CPU SRVR23 x86_64 (Linux 2.6.32-642.15.1.el6.x86_64 #1 SMP
Fri
| Feb 24 14:31:22 UTC 2017) 32059MB                                          
|
| PROCESS ID: 6578      DATE: 09/21/17 11:54:22 -03                          
|
|   USER IDS: REAL=gemst643281 (1045) EFFECTIVE=gemst643281 (1045)
LOGIN=gemst643281
| (1045)                                                                    
|
|_____________________________________________________________________________|

[Info]: Initial Page Manager Settings:
  StnPageMgrRemoveMinPages:        40
  StnPageMgrRemoveMaxPages:     16384
  PageMgrCompressionEnabled:    FALSE
  PageMgrPrintTimeoutThreshold:     5
  StnRemoteCachePgsvrTimeout:       0
  RemoteSharedPageCacheMax:       255
*


I'll be waiting for your comments.

Thanks



--
Sent from: http://forum.world.st/Gemstone-S-f1461796.html
_______________________________________________
GemStone-Smalltalk mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk
Reply | Threaded
Open this post in threaded view
|

Re: Starting a repository fails after shutdown of server.

Gemstone/S mailing list
That's weird...no error in pcmon either. Can you check and send us the logs of pgsvr ??

Thanks

On Mon, Sep 25, 2017 at 11:41 AM, brianstone via GemStone-Smalltalk <[hidden email]> wrote:
It's true, sorry.

Here is *pcmon.log:

*

_____________________________________________________________________________
|                             GemStone Child Task
|
|
|
|    VERSION: 3.2.8.1, Fri Aug 28 08:43:23 2015
|
|      BUILD: gss64_3_2_x_branch-37291
|
|  BUILT FOR: x86-64 (Linux)
|
|       MODE: 64 bit
|
| RUNNING ON: 12-CPU SRVR23 x86_64 (Linux 2.6.32-642.15.1.el6.x86_64 #1 SMP
Fri
| Feb 24 14:31:22 UTC 2017) 32059MB
|
| PROCESS ID: 6580      DATE: 09/21/17 11:54:20 -03
|
|   USER IDS: REAL=gemst643281 (1045) EFFECTIVE=gemst643281 (1045)
LOGIN=gemst643281
| (1045)
|
|    COMMAND: /usr/local/gemstone643281/sys/startshrpcmon
Newstone~7663a27bab8c7a96
| 312500 51 1 5000 7000 1 1900 0 432 0 60
|
|_____________________________________________________________________________|
startshrpcmon.sh GEMSTONE: /usr/local/gemstone643281
SharedPageCache Monitor Summary:
   the hostname is:    SRVR23
   GEMSTONE is:        "/usr/local/gemstone643281"
   shrpcmon arguments are:
      SharedPageCache Monitor name  Newstone~7663a27bab8c7a96.
      Number of pages               312500.
      Number of processes           51.
      Cache locked in memory        1.
      SpinLockCount                 5000.
      TargetFreeFrameCount          7000.
      Number of free frame servers  1.
      Number of shared counters     1900.
      Port number                   0.
      Permissions                   432.
      Large Page Policy             0.
      Stone/cachepgsvr connect timeout 60.

_____________________________________________________________________________
|             GemStone/S64 Object-Oriented Data Management System
|
|                   Copyright (C) GemTalk Systems 1986-2015
|
|                            All rights reserved.
|
+-----------------------------------------------------------------------------+
|    PROGRAM: SHRPCMON, GemStone SharedPageCache Monitor
|
|    VERSION: 3.2.8.1, Fri Aug 28 08:43:23 2015
|
|      BUILD: gss64_3_2_x_branch-37291
|
|  BUILT FOR: x86-64 (Linux)
|
|       MODE: 64 bit
|
| RUNNING ON: 12-CPU SRVR23 x86_64 (Linux 2.6.32-642.15.1.el6.x86_64 #1 SMP
Fri
| Feb 24 14:31:22 UTC 2017) 32059MB
|
| PROCESS ID: 6580      DATE: 09/21/17 11:54:20 -03
|
|   USER IDS: REAL=gemst643281 (1045) EFFECTIVE=gemst643281 (1045)
LOGIN=gemst643281
| (1045)
|
|_____________________________________________________________________________|

Cache config is 312500 pages = 4882MB,  total is 5232MB,  overhead 7% of
configured size
[Info]: Building the free frame list using 1 cache region.
  WARNING:  Shared cache could not be locked in memory because
            the memory available is insufficient. (ENOMEM)
Shared memory region number 1 has id = 360452
The shared semaphore array has id = 327682


--- 09/21/17 11:54:21.258 -03 cache creation took 1 seconds

created listening socket for localhost6 on ::1 port 38772
synthesizing   127.0.0.1 , attempting listen on IPv4 localhost
created listening socket for localhost on ::ffff:127.0.0.1 port 38772

--- 09/21/17 11:54:21.273 -03 Listening for clients on port 38772


Write to /proc/6580/oom_score_adj failed with EACCES , linux user does not
have CAP_SYS_RESOURCE
No server process protection from OOM killer
timeout for stone/cache pgsvr connect is 20 seconds
[09/21/17 11:56:25.819 -03]: Client died: Slot    1, PID    6578,
LostOtFlags    0, sessionId 0 Name Newstone
--- 09/21/17 11:56:25.820 -03 Starting shrpcmonitor shutdown
--- 09/21/17 11:56:25.820 -03 Waiting for crashed slot recovery thread to
shutdown...Done.
--- 09/21/17 11:56:25.821 -03 Waiting for clean slot recovery thread to
shutdown...Done.
--- 09/21/17 11:56:25.821 -03 Waiting for stats thread to shutdown...Done.
--- 09/21/17 11:56:25.890 -03 All threads have stopped.
  The Shared Page Cache Monitor is shutting down.
*

And here is *pagemanager.log:
*

_____________________________________________________________________________
|                          Stone Page Manager Thread
|
|
|
|    VERSION: 3.2.8.1, Fri Aug 28 08:43:23 2015
|
|      BUILD: gss64_3_2_x_branch-37291
|
|  BUILT FOR: x86-64 (Linux)
|
|       MODE: 64 bit
|
| RUNNING ON: 12-CPU SRVR23 x86_64 (Linux 2.6.32-642.15.1.el6.x86_64 #1 SMP
Fri
| Feb 24 14:31:22 UTC 2017) 32059MB
|
| PROCESS ID: 6578      DATE: 09/21/17 11:54:22 -03
|
|   USER IDS: REAL=gemst643281 (1045) EFFECTIVE=gemst643281 (1045)
LOGIN=gemst643281
| (1045)
|
|_____________________________________________________________________________|

[Info]: Initial Page Manager Settings:
  StnPageMgrRemoveMinPages:        40
  StnPageMgrRemoveMaxPages:     16384
  PageMgrCompressionEnabled:    FALSE
  PageMgrPrintTimeoutThreshold:     5
  StnRemoteCachePgsvrTimeout:       0
  RemoteSharedPageCacheMax:       255
*


I'll be waiting for your comments.

Thanks



--
Sent from: http://forum.world.st/Gemstone-S-f1461796.html
_______________________________________________
GemStone-Smalltalk mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk



--

_______________________________________________
GemStone-Smalltalk mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk
Reply | Threaded
Open this post in threaded view
|

Re: Starting a repository fails after shutdown of server.

Gemstone/S mailing list
In reply to this post by Gemstone/S mailing list
In particular, see the following from the stone log:

--- 09/21/17 11:54:24 -03 ---
   Starting AF_INET reconnect to AIO pgsvrs
AioServerReconnect pgsvrPid 6590 old fd 12 new fd -1
reconnect failed(B) for AIO pgsvr pid 6590

Is there a log file for PID 6590?

On Sep 25, 2017, at 7:41 AM, brianstone via GemStone-Smalltalk <[hidden email]> wrote:

It's true, sorry.

Here is *pcmon.log:

*

_____________________________________________________________________________
|                             GemStone Child Task                            
|
|                                                                            
|
|    VERSION: 3.2.8.1, Fri Aug 28 08:43:23 2015                              
|
|      BUILD: gss64_3_2_x_branch-37291                                       
|
|  BUILT FOR: x86-64 (Linux)                                                 
|
|       MODE: 64 bit                                                         
|
| RUNNING ON: 12-CPU SRVR23 x86_64 (Linux 2.6.32-642.15.1.el6.x86_64 #1 SMP
Fri
| Feb 24 14:31:22 UTC 2017) 32059MB                                          
|
| PROCESS ID: 6580      DATE: 09/21/17 11:54:20 -03                          
|
|   USER IDS: REAL=gemst643281 (1045) EFFECTIVE=gemst643281 (1045)
LOGIN=gemst643281
| (1045)                                                                     
|
|    COMMAND: /usr/local/gemstone643281/sys/startshrpcmon
Newstone~7663a27bab8c7a96
| 312500 51 1 5000 7000 1 1900 0 432 0 60                                    
|
|_____________________________________________________________________________|
startshrpcmon.sh GEMSTONE: /usr/local/gemstone643281
SharedPageCache Monitor Summary:
  the hostname is:    SRVR23
  GEMSTONE is:        "/usr/local/gemstone643281"
  shrpcmon arguments are:
     SharedPageCache Monitor name  Newstone~7663a27bab8c7a96.
     Number of pages               312500.
     Number of processes           51.
     Cache locked in memory        1.
     SpinLockCount                 5000.
     TargetFreeFrameCount          7000.
     Number of free frame servers  1.
     Number of shared counters     1900.
     Port number                   0.
     Permissions                   432.
     Large Page Policy             0.
     Stone/cachepgsvr connect timeout 60.

_____________________________________________________________________________
|             GemStone/S64 Object-Oriented Data Management System            
|
|                   Copyright (C) GemTalk Systems 1986-2015                  
|
|                            All rights reserved.                            
|
+-----------------------------------------------------------------------------+
|    PROGRAM: SHRPCMON, GemStone SharedPageCache Monitor                     
|
|    VERSION: 3.2.8.1, Fri Aug 28 08:43:23 2015                              
|
|      BUILD: gss64_3_2_x_branch-37291                                       
|
|  BUILT FOR: x86-64 (Linux)                                                 
|
|       MODE: 64 bit                                                         
|
| RUNNING ON: 12-CPU SRVR23 x86_64 (Linux 2.6.32-642.15.1.el6.x86_64 #1 SMP
Fri
| Feb 24 14:31:22 UTC 2017) 32059MB                                          
|
| PROCESS ID: 6580      DATE: 09/21/17 11:54:20 -03                          
|
|   USER IDS: REAL=gemst643281 (1045) EFFECTIVE=gemst643281 (1045)
LOGIN=gemst643281
| (1045)                                                                     
|
|_____________________________________________________________________________|

Cache config is 312500 pages = 4882MB,  total is 5232MB,  overhead 7% of
configured size
[Info]: Building the free frame list using 1 cache region.
 WARNING:  Shared cache could not be locked in memory because
           the memory available is insufficient. (ENOMEM)
Shared memory region number 1 has id = 360452
The shared semaphore array has id = 327682


--- 09/21/17 11:54:21.258 -03 cache creation took 1 seconds

created listening socket for localhost6 on ::1 port 38772
synthesizing   127.0.0.1 , attempting listen on IPv4 localhost
created listening socket for localhost on ::ffff:127.0.0.1 port 38772

--- 09/21/17 11:54:21.273 -03 Listening for clients on port 38772


Write to /proc/6580/oom_score_adj failed with EACCES , linux user does not
have CAP_SYS_RESOURCE
No server process protection from OOM killer
timeout for stone/cache pgsvr connect is 20 seconds
[09/21/17 11:56:25.819 -03]: Client died: Slot    1, PID    6578,
LostOtFlags    0, sessionId 0 Name Newstone
--- 09/21/17 11:56:25.820 -03 Starting shrpcmonitor shutdown
--- 09/21/17 11:56:25.820 -03 Waiting for crashed slot recovery thread to
shutdown...Done.
--- 09/21/17 11:56:25.821 -03 Waiting for clean slot recovery thread to
shutdown...Done.
--- 09/21/17 11:56:25.821 -03 Waiting for stats thread to shutdown...Done.
--- 09/21/17 11:56:25.890 -03 All threads have stopped.
 The Shared Page Cache Monitor is shutting down.
*

And here is *pagemanager.log:
*

_____________________________________________________________________________
|                          Stone Page Manager Thread                         
|
|                                                                            
|
|    VERSION: 3.2.8.1, Fri Aug 28 08:43:23 2015                              
|
|      BUILD: gss64_3_2_x_branch-37291                                       
|
|  BUILT FOR: x86-64 (Linux)                                                 
|
|       MODE: 64 bit                                                         
|
| RUNNING ON: 12-CPU SRVR23 x86_64 (Linux 2.6.32-642.15.1.el6.x86_64 #1 SMP
Fri
| Feb 24 14:31:22 UTC 2017) 32059MB                                          
|
| PROCESS ID: 6578      DATE: 09/21/17 11:54:22 -03                          
|
|   USER IDS: REAL=gemst643281 (1045) EFFECTIVE=gemst643281 (1045)
LOGIN=gemst643281
| (1045)                                                                     
|
|_____________________________________________________________________________|

[Info]: Initial Page Manager Settings:
 StnPageMgrRemoveMinPages:        40
 StnPageMgrRemoveMaxPages:     16384
 PageMgrCompressionEnabled:    FALSE
 PageMgrPrintTimeoutThreshold:     5
 StnRemoteCachePgsvrTimeout:       0
 RemoteSharedPageCacheMax:       255
*


I'll be waiting for your comments.

Thanks



--
Sent from: http://forum.world.st/Gemstone-S-f1461796.html
_______________________________________________
GemStone-Smalltalk mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk


_______________________________________________
GemStone-Smalltalk mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk
Reply | Threaded
Open this post in threaded view
|

Re: Starting a repository fails after shutdown of server.

Gemstone/S mailing list
Hi James,

There was not a log for that PID.
I was reading the SysAdminGuide and there says that the log is deleted
automatically in case of normal exit.
In this case is not a normal exit but anyway log files are deleted.

So, I modified the file runpgsrvrmain to keep log files (uncommenting
"export GEMSTONE_KEEP_LOG=1") and tried to start again the repository.

Here is the results:

*An extract from repository.log*

    Starting AF_INET reconnect to AIO pgsvrs
    Recovery took 0.028 seconds
    Waiting for Recovery Reader thread to stop
AioServerReconnect pgsvrPid 7940 old fd 12 new fd -1
reconnect failed(B) for AIO pgsvr pid 7940

--- 09/25/17 12:59:18 -03 ---
RDbfAioServersReconnect failed

    Terminating stone.

*Log file for PID 7940 in file named "repository_7940pgsvraio.log"*

_____________________________________________________________________________
|                             GemStone Child Task                            
|
|                                                                            
|
|    VERSION: 3.2.8.1, Fri Aug 28 08:43:23 2015                              
|
|      BUILD: gss64_3_2_x_branch-37291                                      
|
|  BUILT FOR: x86-64 (Linux)                                                
|
|       MODE: 64 bit                                                        
|
| RUNNING ON: 12-CPU SRVR23 x86_64 (Linux 2.6.32-642.15.1.el6.x86_64 #1 SMP
Fri
| Feb 24 14:31:22 UTC 2017) 32059MB                                          
|
| PROCESS ID: 7940      DATE: 09/25/17 12:57:17 -03                          
|
|   USER IDS: REAL=gemst643281 (1045) EFFECTIVE=gemst643281 (1045)
LOGIN=gemst643281
| (1045)                                                                    
|
|    COMMAND: /usr/local/gemstone643281/sys/runpgsvrmain TCP 13 90          
|
|_____________________________________________________________________________|
runpgsvr[Info]:     Description of arguments:
   the hostname is:     SRVR23
   GEMSTONE is:         /usr/local/gemstone643281
   pgsvr arguments are: TCP 13 90

_____________________________________________________________________________
|             GemStone/S64 Object-Oriented Data Management System            
|
|                   Copyright (C) GemTalk Systems 1986-2015                  
|
|                            All rights reserved.                            
|
+-----------------------------------------------------------------------------+
|    PROGRAM: PGSVRSHR, GemStone Networked DBF I/O Service (shared library)  
|
|    VERSION: 3.2.8.1, Fri Aug 28 08:43:23 2015                              
|
|      BUILD: gss64_3_2_x_branch-37291                                      
|
|  BUILT FOR: x86-64 (Linux)                                                
|
|       MODE: 64 bit                                                        
|
| RUNNING ON: 12-CPU SRVR23 x86_64 (Linux 2.6.32-642.15.1.el6.x86_64 #1 SMP
Fri
| Feb 24 14:31:22 UTC 2017) 32059MB                                          
|
| PROCESS ID: 7940      DATE: 09/25/2017 12:57:17 -03                        
|
|   USER IDS: REAL=gemst643281 (1045) EFFECTIVE=gemst643281 (1045)
LOGIN=gemst643281
| (1045)                                                                    
|
|_____________________________________________________________________________|


command line is:
/usr/local/gemstone643281/sys/pgsvrmain TCP 13 90

The hostname is SRVR23
createNetConnection: SocketFamily_UNIX
Network connection has been inherited.
Entering Service Loop
[Info]: ClientPid: 7928
[Info]: Client SessionId: -2
[Info]: Client Host: SRVR23
[Info]: My cache slot:  3
[Info]: My cache name: AioPgsvr3
Write to /proc/7940/oom_score_adj failed with EACCES , linux user does not
have CAP_SYS_RESOURCE
No server process protection from OOM killer
--- 09/25/2017 12:59:18.753 -03 Connection failure detected:


--- 09/25/2017 12:59:18.753 -03 entering pgsShrExit
mainThread  Detaching cache.--- 09/25/2017 12:59:18.753 -03 [Info]:
Detaching Shared Page Cache.
--- 09/25/2017 12:59:18.791 -03 mainThread: pgsShrExit with status: 0


*Additionally heres is the content of another file named
"repository_7937pgsvrff.log"*


_____________________________________________________________________________
|                             GemStone Child Task                            
|
|                                                                            
|
|    VERSION: 3.2.8.1, Fri Aug 28 08:43:23 2015                              
|
|      BUILD: gss64_3_2_x_branch-37291                                      
|
|  BUILT FOR: x86-64 (Linux)                                                
|
|       MODE: 64 bit                                                        
|
| RUNNING ON: 12-CPU SRVR23 x86_64 (Linux 2.6.32-642.15.1.el6.x86_64 #1 SMP
Fri
| Feb 24 14:31:22 UTC 2017) 32059MB                                          
|
| PROCESS ID: 7937      DATE: 09/25/17 12:57:17 -03                          
|
|   USER IDS: REAL=gemst643281 (1045) EFFECTIVE=gemst643281 (1045)
LOGIN=gemst643281
| (1045)                                                                    
|
|    COMMAND: /usr/local/gemstone643281/sys/runpgsvrmain
Newstone~7663a27bab8c7a96
| 0 1 -1 TCP 10 90                                                          
|
|_____________________________________________________________________________|
runpgsvr[Info]:     Description of arguments:
   the hostname is:     SRVR23
   GEMSTONE is:         /usr/local/gemstone643281
   pgsvr arguments are: Newstone~7663a27bab8c7a96 0 1 -1 TCP 10 90

_____________________________________________________________________________
|             GemStone/S64 Object-Oriented Data Management System            
|
|                   Copyright (C) GemTalk Systems 1986-2015                  
|
|                            All rights reserved.                            
|
+-----------------------------------------------------------------------------+
|    PROGRAM: PGSVRSHR, GemStone Networked DBF I/O Service (shared library)  
|
|    VERSION: 3.2.8.1, Fri Aug 28 08:43:23 2015                              
|
|      BUILD: gss64_3_2_x_branch-37291                                      
|
|  BUILT FOR: x86-64 (Linux)                                                
|
|       MODE: 64 bit                                                        
|
| RUNNING ON: 12-CPU SRVR23 x86_64 (Linux 2.6.32-642.15.1.el6.x86_64 #1 SMP
Fri
| Feb 24 14:31:22 UTC 2017) 32059MB                                          
|
| PROCESS ID: 7937      DATE: 09/25/2017 12:57:17 -03                        
|
|   USER IDS: REAL=gemst643281 (1045) EFFECTIVE=gemst643281 (1045)
LOGIN=gemst643281
| (1045)                                                                    
|
|_____________________________________________________________________________|


command line is:
/usr/local/gemstone643281/sys/pgsvrmain Newstone~7663a27bab8c7a96 0 1 -1 TCP
10 90

The hostname is SRVR23
Write to /proc/7937/oom_score_adj failed with EACCES , linux user does not
have CAP_SYS_RESOURCE
No server process protection from OOM killer

    Free Frame Page Server startup was successful.
    Target Free Frame Limit is 7000
    Entering Free List Service Loop.
createNetConnection: SocketFamily_UNIX
Network connection has been inherited.
Entering Service Loop
[Info]: ClientPid: 7928
--- 09/25/2017 12:59:18.753 -03 Connection failure detected:


--- 09/25/2017 12:59:18.753 -03 entering pgsShrExit
mainThread  Detaching cache.--- 09/25/2017 12:59:18.753 -03 [Info]:
Detaching Shared Page Cache.
--- 09/25/2017 12:59:18.753 -03 mainThread: pgsShrExit with status: 0






--
Sent from: http://forum.world.st/Gemstone-S-f1461796.html
_______________________________________________
GemStone-Smalltalk mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk
Reply | Threaded
Open this post in threaded view
|

Re: Starting a repository fails after shutdown of server.

Gemstone/S mailing list
Hi Ezequiel,

From the initial stone log we see that a listening socket was created, the AIO pgsvrs were started, but the reconnect failed:

created listening socket for :: on :: port 39755
...
--- 09/21/17 11:54:24 -03 ---
   Starting AF_INET reconnect to AIO pgsvrs
AioServerReconnect pgsvrPid 6590 old fd 12 new fd -1
reconnect failed(B) for AIO pgsvr pid 6590

--- 09/21/17 11:56:25 -03 ---
RDbfAioServersReconnect failed

Note the 121-second gap and the ‘-1’ value for the new fd (file descriptor or socket ID), indicating an error.

From the later page server log we see the process start date/time and the connection failure date/time:

| PROCESS ID: 7937      DATE: 09/25/2017 12:57:17 -03       
--- 09/25/2017 12:59:18.753 -03 Connection failure detected:

Note again a 121-second gap. 

As part of stone startup, the stone spawns additional processes and waits for them to connect back to itself. The system waits 120 seconds for the connection to complete and then reports a failure. So, the page server is unable to open a socket to the stone (that is, a networking problem).

Is this machine heavily overloaded, such that a new process would fail to run in two minutes? Have you every run GemStone successfully on this host? What sort of internal firewall does it have?

During the two-minute delay in the startup, it would be interesting to see if you can communicate with the stone using another process. From another shell try the following
$ gslist -cvl
If the shell doesn’t recognize gslist, then you don’t have $GEMSTONE/bin in your path. If it doesn’t find any servers, then you haven’t started the NetLDI process yet and it would be good to do so (if only to show that gslist is working!). If it finds the stone and reports that the stone is in ‘startup’ mode, then gslist has successfully communicated with the stone, suggesting that internal networking is not completely broken. If gslist recognizes that there is a stone, but can’t communicate with it, then we’ve confirmed an internal networking problem.

To further debug networking problems, export GEMSTONE_SOCKET_DEBUG=1 (as you did with GEMSTONE_KEEP_LOG; good job reading the manual!), and then provide full logs on the stone and on a failing page server process.

James Foster

On Sep 25, 2017, at 10:59 AM, brianstone via GemStone-Smalltalk <[hidden email]> wrote:

Hi James,

There was not a log for that PID.
I was reading the SysAdminGuide and there says that the log is deleted
automatically in case of normal exit.
In this case is not a normal exit but anyway log files are deleted.

So, I modified the file runpgsrvrmain to keep log files (uncommenting
"export GEMSTONE_KEEP_LOG=1") and tried to start again the repository.

Here is the results:

*An extract from repository.log*

   Starting AF_INET reconnect to AIO pgsvrs
   Recovery took 0.028 seconds
   Waiting for Recovery Reader thread to stop
AioServerReconnect pgsvrPid 7940 old fd 12 new fd -1
reconnect failed(B) for AIO pgsvr pid 7940

--- 09/25/17 12:59:18 -03 ---
RDbfAioServersReconnect failed

   Terminating stone.

*Log file for PID 7940 in file named "repository_7940pgsvraio.log"*

_____________________________________________________________________________
|                             GemStone Child Task                            
|
|                                                                            
|
|    VERSION: 3.2.8.1, Fri Aug 28 08:43:23 2015                              
|
|      BUILD: gss64_3_2_x_branch-37291                                       
|
|  BUILT FOR: x86-64 (Linux)                                                 
|
|       MODE: 64 bit                                                         
|
| RUNNING ON: 12-CPU SRVR23 x86_64 (Linux 2.6.32-642.15.1.el6.x86_64 #1 SMP
Fri
| Feb 24 14:31:22 UTC 2017) 32059MB                                          
|
| PROCESS ID: 7940      DATE: 09/25/17 12:57:17 -03                          
|
|   USER IDS: REAL=gemst643281 (1045) EFFECTIVE=gemst643281 (1045)
LOGIN=gemst643281
| (1045)                                                                     
|
|    COMMAND: /usr/local/gemstone643281/sys/runpgsvrmain TCP 13 90           
|
|_____________________________________________________________________________|
runpgsvr[Info]:     Description of arguments:
  the hostname is:     SRVR23
  GEMSTONE is:         /usr/local/gemstone643281
  pgsvr arguments are: TCP 13 90

_____________________________________________________________________________
|             GemStone/S64 Object-Oriented Data Management System            
|
|                   Copyright (C) GemTalk Systems 1986-2015                  
|
|                            All rights reserved.                            
|
+-----------------------------------------------------------------------------+
|    PROGRAM: PGSVRSHR, GemStone Networked DBF I/O Service (shared library)  
|
|    VERSION: 3.2.8.1, Fri Aug 28 08:43:23 2015                              
|
|      BUILD: gss64_3_2_x_branch-37291                                       
|
|  BUILT FOR: x86-64 (Linux)                                                 
|
|       MODE: 64 bit                                                         
|
| RUNNING ON: 12-CPU SRVR23 x86_64 (Linux 2.6.32-642.15.1.el6.x86_64 #1 SMP
Fri
| Feb 24 14:31:22 UTC 2017) 32059MB                                          
|
| PROCESS ID: 7940      DATE: 09/25/2017 12:57:17 -03                        
|
|   USER IDS: REAL=gemst643281 (1045) EFFECTIVE=gemst643281 (1045)
LOGIN=gemst643281
| (1045)                                                                     
|
|_____________________________________________________________________________|


command line is:
/usr/local/gemstone643281/sys/pgsvrmain TCP 13 90

The hostname is SRVR23
createNetConnection: SocketFamily_UNIX
Network connection has been inherited.
Entering Service Loop
[Info]: ClientPid: 7928
[Info]: Client SessionId: -2
[Info]: Client Host: SRVR23
[Info]: My cache slot:  3
[Info]: My cache name: AioPgsvr3
Write to /proc/7940/oom_score_adj failed with EACCES , linux user does not
have CAP_SYS_RESOURCE
No server process protection from OOM killer
--- 09/25/2017 12:59:18.753 -03 Connection failure detected:


--- 09/25/2017 12:59:18.753 -03 entering pgsShrExit
mainThread  Detaching cache.--- 09/25/2017 12:59:18.753 -03 [Info]:
Detaching Shared Page Cache.
--- 09/25/2017 12:59:18.791 -03 mainThread: pgsShrExit with status: 0


*Additionally heres is the content of another file named
"repository_7937pgsvrff.log"*


_____________________________________________________________________________
|                             GemStone Child Task                            
|
|                                                                            
|
|    VERSION: 3.2.8.1, Fri Aug 28 08:43:23 2015                              
|
|      BUILD: gss64_3_2_x_branch-37291                                       
|
|  BUILT FOR: x86-64 (Linux)                                                 
|
|       MODE: 64 bit                                                         
|
| RUNNING ON: 12-CPU SRVR23 x86_64 (Linux 2.6.32-642.15.1.el6.x86_64 #1 SMP
Fri
| Feb 24 14:31:22 UTC 2017) 32059MB                                          
|
| PROCESS ID: 7937      DATE: 09/25/17 12:57:17 -03                          
|
|   USER IDS: REAL=gemst643281 (1045) EFFECTIVE=gemst643281 (1045)
LOGIN=gemst643281
| (1045)                                                                     
|
|    COMMAND: /usr/local/gemstone643281/sys/runpgsvrmain
Newstone~7663a27bab8c7a96
| 0 1 -1 TCP 10 90                                                           
|
|_____________________________________________________________________________|
runpgsvr[Info]:     Description of arguments:
  the hostname is:     SRVR23
  GEMSTONE is:         /usr/local/gemstone643281
  pgsvr arguments are: Newstone~7663a27bab8c7a96 0 1 -1 TCP 10 90

_____________________________________________________________________________
|             GemStone/S64 Object-Oriented Data Management System            
|
|                   Copyright (C) GemTalk Systems 1986-2015                  
|
|                            All rights reserved.                            
|
+-----------------------------------------------------------------------------+
|    PROGRAM: PGSVRSHR, GemStone Networked DBF I/O Service (shared library)  
|
|    VERSION: 3.2.8.1, Fri Aug 28 08:43:23 2015                              
|
|      BUILD: gss64_3_2_x_branch-37291                                       
|
|  BUILT FOR: x86-64 (Linux)                                                 
|
|       MODE: 64 bit                                                         
|
| RUNNING ON: 12-CPU SRVR23 x86_64 (Linux 2.6.32-642.15.1.el6.x86_64 #1 SMP
Fri
| Feb 24 14:31:22 UTC 2017) 32059MB                                          
|
| PROCESS ID: 7937      DATE: 09/25/2017 12:57:17 -03                        
|
|   USER IDS: REAL=gemst643281 (1045) EFFECTIVE=gemst643281 (1045)
LOGIN=gemst643281
| (1045)                                                                     
|
|_____________________________________________________________________________|


command line is:
/usr/local/gemstone643281/sys/pgsvrmain Newstone~7663a27bab8c7a96 0 1 -1 TCP
10 90

The hostname is SRVR23
Write to /proc/7937/oom_score_adj failed with EACCES , linux user does not
have CAP_SYS_RESOURCE
No server process protection from OOM killer

   Free Frame Page Server startup was successful.
   Target Free Frame Limit is 7000
   Entering Free List Service Loop.
createNetConnection: SocketFamily_UNIX
Network connection has been inherited.
Entering Service Loop
[Info]: ClientPid: 7928
--- 09/25/2017 12:59:18.753 -03 Connection failure detected:


--- 09/25/2017 12:59:18.753 -03 entering pgsShrExit
mainThread  Detaching cache.--- 09/25/2017 12:59:18.753 -03 [Info]:
Detaching Shared Page Cache.
--- 09/25/2017 12:59:18.753 -03 mainThread: pgsShrExit with status: 0






--
Sent from: http://forum.world.st/Gemstone-S-f1461796.html
_______________________________________________
GemStone-Smalltalk mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk


_______________________________________________
GemStone-Smalltalk mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk
Reply | Threaded
Open this post in threaded view
|

Re: Starting a repository fails after shutdown of server.

Gemstone/S mailing list
James,

Here are all the log files obtained using "GEMSTONE_SOCKET_DEBUG=1"

logs.zip <http://forum.world.st/file/t370666/logs.zip>  

On the other hand, during the startup I've tried many times with gslist to
see the result.
Using gslist-l

gslist -l
Status       Version    Owner    Pid   Port   Started     Type       Name
-------     --------- --------- ----- ----- ------------ ------      ----
exists      3.2.8.1   gemst643281  9878 37052 Sep 25 15:48 cache      
Newstone~7663a27bab8c7a96
exists      3.2.8.1   gemst643281  9876 38044 Sep 25 15:48 Stone      
Newstone
exists      3.2.8.1   gemst643281  7912 50387 Sep 25 12:56 Netldi      
gs64ldi-3281

Using gslist -clv

gslist -clv
Status       Version    Owner    Pid   Port   Started     Type       Name
-------     --------- --------- ----- ----- ------------ ------      ----
  OK        3.2.8.1   gemst643281  9878 37052 Sep 25 15:48 cache      
Newstone~7663a27bab8c7a96
frozen      3.2.8.1   gemst643281  9876 38044 Sep 25 15:48 Stone      
Newstone
  OK        3.2.8.1   gemst643281  7912 50387 Sep 25 12:56 Netldi    
gs64ldi-3281


Based on the logs, I think that you are right and is a networking problem.

Regarding your questions:

This server belongs to the company which I work for. It's often used to keep
some internal web sites running  during a short period, for example for
internal contests. Since I'm the administrator of the server and I have full
control over it, some time ago I decided to install and test Gemstone.
Currently is not running any site, and is unused, so the server is not
overloaded.
Recently this machine have been moved to a different network which have a
more restrictive firewall, but I requested to enable traffic through the
netldi port (I thought it will be enough).

Summarizing, yes, I was able to run Gemstone before the network change, but
I thought that enabling netldi port should be enough.

What are the networking requisites needed to start Gemstone?

Hope this info can help you to deveal the problem.

Ezequiel




--
Sent from: http://forum.world.st/Gemstone-S-f1461796.html
_______________________________________________
GemStone-Smalltalk mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk
Reply | Threaded
Open this post in threaded view
|

Re: Starting a repository fails after shutdown of server.

Gemstone/S mailing list


On Mon, Sep 25, 2017 at 4:41 PM, brianstone via GemStone-Smalltalk <[hidden email]> wrote:
James,

Here are all the log files obtained using "GEMSTONE_SOCKET_DEBUG=1"

logs.zip <http://forum.world.st/file/t370666/logs.zip>

On the other hand, during the startup I've tried many times with gslist to
see the result.
Using gslist-l

gslist -l
Status       Version    Owner    Pid   Port   Started     Type       Name
-------     --------- --------- ----- ----- ------------ ------      ----
exists      3.2.8.1   gemst643281  9878 37052 Sep 25 15:48 cache
Newstone~7663a27bab8c7a96
exists      3.2.8.1   gemst643281  9876 38044 Sep 25 15:48 Stone
Newstone
exists      3.2.8.1   gemst643281  7912 50387 Sep 25 12:56 Netldi
gs64ldi-3281

Using gslist -clv

gslist -clv
Status       Version    Owner    Pid   Port   Started     Type       Name
-------     --------- --------- ----- ----- ------------ ------      ----
  OK        3.2.8.1   gemst643281  9878 37052 Sep 25 15:48 cache
Newstone~7663a27bab8c7a96
frozen      3.2.8.1   gemst643281  9876 38044 Sep 25 15:48 Stone
Newstone
  OK        3.2.8.1   gemst643281  7912 50387 Sep 25 12:56 Netldi
gs64ldi-3281


Based on the logs, I think that you are right and is a networking problem.

Regarding your questions:

This server belongs to the company which I work for. It's often used to keep
some internal web sites running  during a short period, for example for
internal contests. Since I'm the administrator of the server and I have full
control over it, some time ago I decided to install and test Gemstone.
Currently is not running any site, and is unused, so the server is not
overloaded.
Recently this machine have been moved to a different network which have a
more restrictive firewall, but I requested to enable traffic through the
netldi port (I thought it will be enough).

Summarizing, yes, I was able to run Gemstone before the network change, but
I thought that enabling netldi port should be enough.



I think that in the past (not sure which versions), GemStone would use 1 or 2 more ports aside from the netldi one (netldi+1 and  netldi+2 or something like that). I think in latest Gemstone, netldi was able to use only one part for all traffics. I suspect trying to start netldi with -p option, like this:

$GEMSTONE/bin/startnetldi -P 50377 -p50378:50379 -g -a $USER
 
may help (of course, be sure to open 50378 and 50379 in the firewall).

at least not as a definitive workaround but for troubleshooting. 

Cheers,


What are the networking requisites needed to start Gemstone?

Hope this info can help you to deveal the problem.

Ezequiel




--
Sent from: http://forum.world.st/Gemstone-S-f1461796.html
_______________________________________________
GemStone-Smalltalk mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk



--

_______________________________________________
GemStone-Smalltalk mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk
Reply | Threaded
Open this post in threaded view
|

Re: Starting a repository fails after shutdown of server.

Gemstone/S mailing list
Mariano,

You are right about the NetLDI ports and changes with respect to a RPC client communicating with the NetLDI and the new Gem, but Ezequiel is dealing with an AIO page server talking to the stone, and this is a different port.

James

On Sep 25, 2017, at 1:07 PM, Mariano Martinez Peck via GemStone-Smalltalk <[hidden email]> wrote:



On Mon, Sep 25, 2017 at 4:41 PM, brianstone via GemStone-Smalltalk <[hidden email]>wrote:
James,

Here are all the log files obtained using "GEMSTONE_SOCKET_DEBUG=1"

logs.zip <http://forum.world.st/file/t370666/logs.zip>

On the other hand, during the startup I've tried many times with gslist to
see the result.
Using gslist-l

gslist -l
Status       Version    Owner    Pid   Port   Started     Type       Name
-------     --------- --------- ----- ----- ------------ ------      ----
exists      3.2.8.1   gemst643281  9878 37052 Sep 25 15:48 cache
Newstone~7663a27bab8c7a96
exists      3.2.8.1   gemst643281  9876 38044 Sep 25 15:48 Stone
Newstone
exists      3.2.8.1   gemst643281  7912 50387 Sep 25 12:56 Netldi
gs64ldi-3281

Using gslist -clv

gslist -clv
Status       Version    Owner    Pid   Port   Started     Type       Name
-------     --------- --------- ----- ----- ------------ ------      ----
  OK        3.2.8.1   gemst643281  9878 37052 Sep 25 15:48 cache
Newstone~7663a27bab8c7a96
frozen      3.2.8.1   gemst643281  9876 38044 Sep 25 15:48 Stone
Newstone
  OK        3.2.8.1   gemst643281  7912 50387 Sep 25 12:56 Netldi
gs64ldi-3281


Based on the logs, I think that you are right and is a networking problem.

Regarding your questions:

This server belongs to the company which I work for. It's often used to keep
some internal web sites running  during a short period, for example for
internal contests. Since I'm the administrator of the server and I have full
control over it, some time ago I decided to install and test Gemstone.
Currently is not running any site, and is unused, so the server is not
overloaded.
Recently this machine have been moved to a different network which have a
more restrictive firewall, but I requested to enable traffic through the
netldi port (I thought it will be enough).

Summarizing, yes, I was able to run Gemstone before the network change, but
I thought that enabling netldi port should be enough.



I think that in the past (not sure which versions), GemStone would use 1 or 2 more ports aside from the netldi one (netldi+1 and  netldi+2 or something like that). I think in latest Gemstone, netldi was able to use only one part for all traffics. I suspect trying to start netldi with -p option, like this:

$GEMSTONE/bin/startnetldi -P 50377 -p50378:50379 -g -a $USER
 
may help (of course, be sure to open 50378 and 50379 in the firewall).

at least not as a definitive workaround but for troubleshooting. 

Cheers,


What are the networking requisites needed to start Gemstone?

Hope this info can help you to deveal the problem.

Ezequiel




--
Sent from: http://forum.world.st/Gemstone-S-f1461796.html
_______________________________________________
GemStone-Smalltalk mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk



-- 
_______________________________________________
GemStone-Smalltalk mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk


_______________________________________________
GemStone-Smalltalk mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk
Reply | Threaded
Open this post in threaded view
|

Re: Starting a repository fails after shutdown of server.

Gemstone/S mailing list
In reply to this post by Gemstone/S mailing list
Ezequiel,

In general, a firewall will restrict communications between hosts, but you can also configure a firewall to restrict communications between processes on a single host. If your machine is configured to limit communications between processes that are started by the same user, then you need to enable some additional ports.

By default, the various processes listen on a random port (as assigned by the OS). You can use various configurations to specify the actual port used. This is typically more important with respect to the NetLDI process, since it is called from external processes, but it can be an issue for other processes as well.

In the System Administration Guide, search for ‘WELL_KNOWN’ and then configure your system to use well-known ports for the Stone and the Shared Page Cache Monitor. Then enable those ports in your internal firewall and try starting the system again and let us know the result.

James Foster


> On Sep 25, 2017, at 12:41 PM, brianstone via GemStone-Smalltalk <[hidden email]> wrote:
>
> James,
>
> Here are all the log files obtained using "GEMSTONE_SOCKET_DEBUG=1"
>
> logs.zip <http://forum.world.st/file/t370666/logs.zip>  
>
> On the other hand, during the startup I've tried many times with gslist to
> see the result.
> Using gslist-l
>
> gslist -l
> Status       Version    Owner    Pid   Port   Started     Type       Name
> -------     --------- --------- ----- ----- ------------ ------      ----
> exists      3.2.8.1   gemst643281  9878 37052 Sep 25 15:48 cache      
> Newstone~7663a27bab8c7a96
> exists      3.2.8.1   gemst643281  9876 38044 Sep 25 15:48 Stone      
> Newstone
> exists      3.2.8.1   gemst643281  7912 50387 Sep 25 12:56 Netldi      
> gs64ldi-3281
>
> Using gslist -clv
>
> gslist -clv
> Status       Version    Owner    Pid   Port   Started     Type       Name
> -------     --------- --------- ----- ----- ------------ ------      ----
>  OK        3.2.8.1   gemst643281  9878 37052 Sep 25 15:48 cache      
> Newstone~7663a27bab8c7a96
> frozen      3.2.8.1   gemst643281  9876 38044 Sep 25 15:48 Stone      
> Newstone
>  OK        3.2.8.1   gemst643281  7912 50387 Sep 25 12:56 Netldi    
> gs64ldi-3281
>
>
> Based on the logs, I think that you are right and is a networking problem.
>
> Regarding your questions:
>
> This server belongs to the company which I work for. It's often used to keep
> some internal web sites running  during a short period, for example for
> internal contests. Since I'm the administrator of the server and I have full
> control over it, some time ago I decided to install and test Gemstone.
> Currently is not running any site, and is unused, so the server is not
> overloaded.
> Recently this machine have been moved to a different network which have a
> more restrictive firewall, but I requested to enable traffic through the
> netldi port (I thought it will be enough).
>
> Summarizing, yes, I was able to run Gemstone before the network change, but
> I thought that enabling netldi port should be enough.
>
> What are the networking requisites needed to start Gemstone?
>
> Hope this info can help you to deveal the problem.
>
> Ezequiel
>
>
>
>
> --
> Sent from: http://forum.world.st/Gemstone-S-f1461796.html
> _______________________________________________
> GemStone-Smalltalk mailing list
> [hidden email]
> http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk

_______________________________________________
GemStone-Smalltalk mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk
Reply | Threaded
Open this post in threaded view
|

Re: Starting a repository fails after shutdown of server.

Gemstone/S mailing list
Hi,

I followed James's advice and configured the ports for the Stone and the
Shared Page Cache Monitor.

I'm using this configuration:

STN_WELL_KNOWN_PORT_NUMBER = 51001;
SHR_WELL_KNOWN_PORT_NUMBER = 51002;

Regarding an internall firewall, I tested by deactivating any firewall in
the machine but anyway the problem still there.

On my last test I have activated again iptables and configured to allow
traffic through these ports(51001 and 51002). After that, I tried again to
start Gemstone, and during the startup I tested the following commands:

*netstat -uptona | grep 51001*
tcp        0      0 :::51001                    :::*                      
LISTEN      30145/stoned        off (0.00/0/0)
tcp      160      0 ::1:51001                   ::1:47588                  
ESTABLISHED 30145/stoned        off (0.00/0/0)
tcp        0      0 ::1:47588                   ::1:51001                  
ESTABLISHED 30157/pgsvrmain     off (0.00/0/0)

*netstat -uptona | grep 51002*
tcp        0      0 ::ffff:127.0.0.1:51002      :::*                      
LISTEN      30147/shrpcmonitor  off (0.00/0/0)
tcp        0      0 ::1:51002                   :::*                      
LISTEN      30147/shrpcmonitor  off (0.00/0/0)
tcp        0      0 ::1:45972                   ::1:51002                  
ESTABLISHED 30157/pgsvrmain     off (0.00/0/0)
tcp        0      0 ::1:45970                   ::1:51002                  
ESTABLISHED 30154/pgsvrmain     off (0.00/0/0)
tcp        0      0 ::1:45968                   ::1:51002                  
ESTABLISHED 30145/stoned        off (0.00/0/0)
tcp        0      0 ::1:51002                   ::1:45974                  
ESTABLISHED 30147/shrpcmonitor  off (0.00/0/0)
tcp        0      0 ::1:51002                   ::1:45970                  
ESTABLISHED 30147/shrpcmonitor  off (0.00/0/0)
tcp        0      0 ::1:51002                   ::1:45968                  
ESTABLISHED 30147/shrpcmonitor  off (0.00/0/0)
tcp        0      0 ::1:51002                   ::1:45972                  
ESTABLISHED 30147/shrpcmonitor  off (0.00/0/0)
tcp        0      0 ::1:45974                   ::1:51002                  
ESTABLISHED 30145/stoned        off (0.00/0/0)


*nmap -p1-52000 localhost*

Starting Nmap 5.51 ( http://nmap.org ) at 2017-09-28 11:49 -03
Nmap scan report for localhost (127.0.0.1)
Host is up (0.0000040s latency).
rDNS record for 127.0.0.1: SRVR23
Not shown: 51992 closed ports
PORT      STATE SERVICE
22/tcp    open  ssh
25/tcp    open  smtp
111/tcp   open  rpcbind
631/tcp   open  ipp
5989/tcp  open  wbem-https
27017/tcp open  unknown
51001/tcp open  unknown
51002/tcp open  unknown

Nmap done: 1 IP address (1 host up) scanned in 1.04 seconds


In the result of 'netstat' shows that there is an entablished connection.
For me, that is a clearly proof that the connection is not a problem. I'm
right?

I'm suspecting about a configuration with file descriptors. Can it be a
problem?

Thanks in advance.

Ezequiel



--
Sent from: http://forum.world.st/Gemstone-S-f1461796.html
_______________________________________________
GemStone-Smalltalk mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk
Reply | Threaded
Open this post in threaded view
|

Re: Starting a repository fails after shutdown of server.

Gemstone/S mailing list
Hi Ezequiel,

It appears from your observations below and the earlier log files (with extra socket logging) that you are right that at least some connections are succeeding. Thus, it is probably not a firewall issue. After further review of the logs it does seem that there is a problem with how the AIO page server is connecting to the stone. In one of the logs we see a response of SOCK_EINPROGRESS but there is never a full connection. Without looking deeper, we aren’t really sure why the failure happens.

Is there a reason you are using version 3.2.8.1 and not something more recent? That was released in August 2015, so is over two years old. Could you try starting a stone for the most recent version, say 3.3.6 or even 3.4.0-Alpha5 (see http://downloads.gemtalksystems.com/pub/GemStone64/)? I think we could get more traction in investigating this on a newer version.

Regards,

James Foster

On Sep 28, 2017, at 8:21 AM, brianstone via GemStone-Smalltalk <[hidden email]> wrote:

Hi,

I followed James's advice and configured the ports for the Stone and the
Shared Page Cache Monitor.

I'm using this configuration:

STN_WELL_KNOWN_PORT_NUMBER = 51001;
SHR_WELL_KNOWN_PORT_NUMBER = 51002;

Regarding an internall firewall, I tested by deactivating any firewall in
the machine but anyway the problem still there.

On my last test I have activated again iptables and configured to allow
traffic through these ports(51001 and 51002). After that, I tried again to
start Gemstone, and during the startup I tested the following commands:

*netstat -uptona | grep 51001*
tcp        0      0 :::51001                    :::*                       
LISTEN      30145/stoned        off (0.00/0/0)
tcp      160      0 ::1:51001                   ::1:47588                  
ESTABLISHED 30145/stoned        off (0.00/0/0)
tcp        0      0 ::1:47588                   ::1:51001                  
ESTABLISHED 30157/pgsvrmain     off (0.00/0/0)

*netstat -uptona | grep 51002*
tcp        0      0 ::ffff:127.0.0.1:51002      :::*                       
LISTEN      30147/shrpcmonitor  off (0.00/0/0)
tcp        0      0 ::1:51002                   :::*                       
LISTEN      30147/shrpcmonitor  off (0.00/0/0)
tcp        0      0 ::1:45972                   ::1:51002                  
ESTABLISHED 30157/pgsvrmain     off (0.00/0/0)
tcp        0      0 ::1:45970                   ::1:51002                  
ESTABLISHED 30154/pgsvrmain     off (0.00/0/0)
tcp        0      0 ::1:45968                   ::1:51002                  
ESTABLISHED 30145/stoned        off (0.00/0/0)
tcp        0      0 ::1:51002                   ::1:45974                  
ESTABLISHED 30147/shrpcmonitor  off (0.00/0/0)
tcp        0      0 ::1:51002                   ::1:45970                  
ESTABLISHED 30147/shrpcmonitor  off (0.00/0/0)
tcp        0      0 ::1:51002                   ::1:45968                  
ESTABLISHED 30147/shrpcmonitor  off (0.00/0/0)
tcp        0      0 ::1:51002                   ::1:45972                  
ESTABLISHED 30147/shrpcmonitor  off (0.00/0/0)
tcp        0      0 ::1:45974                   ::1:51002                  
ESTABLISHED 30145/stoned        off (0.00/0/0)


*nmap -p1-52000 localhost*

Starting Nmap 5.51 ( http://nmap.org ) at 2017-09-28 11:49 -03
Nmap scan report for localhost (127.0.0.1)
Host is up (0.0000040s latency).
rDNS record for 127.0.0.1: SRVR23
Not shown: 51992 closed ports
PORT      STATE SERVICE
22/tcp    open  ssh
25/tcp    open  smtp
111/tcp   open  rpcbind
631/tcp   open  ipp
5989/tcp  open  wbem-https
27017/tcp open  unknown
51001/tcp open  unknown
51002/tcp open  unknown

Nmap done: 1 IP address (1 host up) scanned in 1.04 seconds


In the result of 'netstat' shows that there is an entablished connection.
For me, that is a clearly proof that the connection is not a problem. I'm
right?

I'm suspecting about a configuration with file descriptors. Can it be a
problem?

Thanks in advance.

Ezequiel



--
Sent from: http://forum.world.st/Gemstone-S-f1461796.html
_______________________________________________
GemStone-Smalltalk mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk


_______________________________________________
GemStone-Smalltalk mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk
Reply | Threaded
Open this post in threaded view
|

Re: Starting a repository fails after shutdown of server.

Gemstone/S mailing list
Hi James,

I finally found the problem.

There was missing a line in the hosts file.

I was looking at this part on the pcmon.log

*created listening socket for localhost6 on ::1 port 41793
synthesizing   127.0.0.1 , attempting listen on IPv4 localhost
created listening socket for localhost on ::ffff:127.0.0.1 port 41793*

Since I'm not an expert on this, I found strange that there is created a
socket in localhost6 and one in localhost (as I understand IPV4).

So, after a while googling, I noticed that in general, other hosts files in
Linux systems, have this line:
*::1         localhost localhost.localdomain localhost6
localhost6.localdomain6*

So, I added that line to my host file and then Gemstone started succesfully
:-)

The reason of this behavior seems to be the fact that now I don't have a DNS
in the new network of the server or something like that.

You are free to add any comment you consider relevant,

Thanks you for all your help James,

Ezequiel




--
Sent from: http://forum.world.st/Gemstone-S-f1461796.html
_______________________________________________
GemStone-Smalltalk mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk
Reply | Threaded
Open this post in threaded view
|

Re: Starting a repository fails after shutdown of server.

Gemstone/S mailing list
Hi Ezequiel,

I’m glad you got the problem resolved. I’ll report this to the engineering team to see if the failure can be made more explicit.

James


> On Oct 2, 2017, at 7:44 AM, brianstone via GemStone-Smalltalk <[hidden email]> wrote:
>
> Hi James,
>
> I finally found the problem.
>
> There was missing a line in the hosts file.
>
> I was looking at this part on the pcmon.log
>
> *created listening socket for localhost6 on ::1 port 41793
> synthesizing   127.0.0.1 , attempting listen on IPv4 localhost
> created listening socket for localhost on ::ffff:127.0.0.1 port 41793*
>
> Since I'm not an expert on this, I found strange that there is created a
> socket in localhost6 and one in localhost (as I understand IPV4).
>
> So, after a while googling, I noticed that in general, other hosts files in
> Linux systems, have this line:
> *::1         localhost localhost.localdomain localhost6
> localhost6.localdomain6*
>
> So, I added that line to my host file and then Gemstone started succesfully
> :-)
>
> The reason of this behavior seems to be the fact that now I don't have a DNS
> in the new network of the server or something like that.
>
> You are free to add any comment you consider relevant,
>
> Thanks you for all your help James,
>
> Ezequiel
>
>
>
>
> --
> Sent from: http://forum.world.st/Gemstone-S-f1461796.html
> _______________________________________________
> GemStone-Smalltalk mailing list
> [hidden email]
> http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk

_______________________________________________
GemStone-Smalltalk mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/gemstone-smalltalk