netldi / gemtools connection error

> Hi Dale,
>
>> Let's see, I've got a testLogin command in tODE that gives me better diagnostics ...
> I need to catch up on tODE ;)
> I’ll see to get there so I can send you the info of that command.
>
>> So think I'd like to see the error message that is associated with the failed login  ... I assume that you are getting a client-side stack, so extracting the errMsg (I think) from the GcIErrSType instance would perhaps give some clue …
> Client-side, the error is a Gci31xErrSType64 instance with the following messageText:
>
> 'The connection to the Stone Repository monitor was refused:
> , '
>
>> Also a look at the netldi logfile and the stone log file might shed additional light …
> I am attaching those logs files here.
>
> btw: the Linux firewall (iptables) is turned off.
> Google compute engine vm instances have 2 ips: an internal and an external ip. I do suspect this has something to do with it but I’m not sure where Gemstone / netldi goes wrong with that.
>
> thanks for all the help
> Johan
>
>

> On 01 Mar 2016, at 23:53, Dale Henrichs <[hidden email]> wrote:
>
> Okay ... we see that you did get a gem log, so that means that you are connecting to the netldi and getting the gem launched, but you are not connecting to the stone for some reason ... we ought to print some of that info to the gem log, but we don't currently (I'm submitting a bug:).
>
> The stone was started as `seaside`, so I'm curious what you GemTools login information looks like just to be sure ...
>
> Also I am wondering if  `/bin/hostname` and `/etc/hosts` are consistent...`/bin/hostname` needs to be in `/etc/hosts` ...
>
> Finally, we can get more information about what might be going wrong by stopping the netldi and setting the following env var and then restarting the netldi:
>
>  export GEMSTONE_SOCKET_DEBUG=1
>
> This time when you attempt to login from GemTools, you should get a bunch of information dumped to the gem log ... a fresh set of stone/gem/netldi logs would then help us figure things out ...
>
> Dale
>
> On 03/01/2016 12:17 AM, Johan Brichau wrote:
>> Hi Dale,
>>
>>> Let's see, I've got a testLogin command in tODE that gives me better diagnostics ...
>> I need to catch up on tODE ;)
>> I’ll see to get there so I can send you the info of that command.
>>
>>> So think I'd like to see the error message that is associated with the failed login  ... I assume that you are getting a client-side stack, so extracting the errMsg (I think) from the GcIErrSType instance would perhaps give some clue …
>> Client-side, the error is a Gci31xErrSType64 instance with the following messageText:
>>
>> 'The connection to the Stone Repository monitor was refused:
>> , '
>>
>>> Also a look at the netldi logfile and the stone log file might shed additional light …
>> I am attaching those logs files here.
>>
>> btw: the Linux firewall (iptables) is turned off.
>> Google compute engine vm instances have 2 ips: an internal and an external ip. I do suspect this has something to do with it but I’m not sure where Gemstone / netldi goes wrong with that.
>>
>> thanks for all the help
>> Johan
>>
>>
>

> On 03 Mar 2016, at 20:34, Johan Brichau <[hidden email]> wrote:
>
> Hi Dale,
>
> With the GEMSTONE_SOCKET_DEBUG set to 1, I get the following log in the netldi and gem log files.
> The VM instance I have is installed with a CentOS 6 but I did not find a /etc/hostname (I do have these on other centos 6 machines, not on google). I added the hostname dev2.yesplan.be there and also added it elsewhere.
>
> Do I read correctly a connection to my client machine is being tried on port 0 (which fails)?
>
> thx
> Johan
>
>
> Internal tables are configured for 1359 concurrent clients.
> Server socket gs64ldi (protocol TCP) is created, with a delay of 30 seconds.
> Summary of netldi parameters:
>   The host name is 'dev2.yesplan.be'.
> GEMSTONE is: '/opt/gemstone/product'.
> GEMSTONE_NRS_ALL is: '#dir:$GEMSTONE_DATADIR#log:/$GEMSTONE_LOGDIR/%N%P.log'.
>   Clients are not authenticated.
>   Process creation MUST be through services.dat.
> Pool of ports is 50378:50378 .
>   Created processes belong to the account named 'johan'.
>   The default directory for log files is '#dir:$GEMSTONE_DATADIR#log:/$GEMSTONE_LOGDIR/%N%P.log'.
>   The command line is:
>     /opt/gemstone/GemStone64Bit3.1.0.6-x86_64.Linux/sys/netldid -g -n -ajohan \
>      -p50378:50378 -l /opt/gemstone/log/gs64ldi.log gs64ldi
> Entering Service Loop
> SocketAccept success: listeningSocket 6 resultSocket 5
> setSocketNonBlocking fd 5 val 1 status 1
> setSocketCloseOnExec fd 5 val 1 status 1  
> setSocketNonBlocking fd 5 val 0 status 1
> SocketAccept success: listeningSocket 6 resultSocket 5
> setSocketNonBlocking fd 5 val 1 status 1
> setSocketCloseOnExec fd 5 val 1 status 1  
> --- 03/03/2016 07:55:31.329 UTC :SocketCreate socket(10, 1) (AF_INET6, TCP) new sock 8
> setSocketNonBlocking fd 8 val 0 status 1
> setBufferSize fd 8 SNDBUF changed old 16384 new 35328
> setBufferSize fd 8 RCVBUF no change old 87380 new 35328
> setSocketCloseOnExec fd 8 val 1 status 1  
> --- 03/03/2016 07:55:31.329 UTC :SocketBind_ ok , sock 8, AF_INET6  port 0 addr ::1
> setSocketNonBlocking fd 8 val 1 status 1
> --- 03/03/2016 07:55:31.329 UTC :SocketListen ok  , sock 8, AF_INET6  port 44050 addr ::1
> AF_INET4 getaddrinfo(94-224-141-192.access.telenet.be, NULL, flags 0x0 ...)  -- OK
> --- 03/03/2016 07:55:31.331 UTC :SocketCreate socket(2, 2) (AF_INET, UDP) new sock 9
> --- 03/03/2016 07:55:31.332 UTC :SocketBind_ failed , sock 9, AF_INET, port 0 addr 94.224.141.192 errno 99
> SocketBind_ failed: bind(9,94.224.141.192, port 0) failed with errno=99,EADDRNOTAVAIL, Cannot assign requested addressn
>
>
>
> The gemnetlogobject file says:
>
> --- 03/03/2016 07:55:31.348 UTC :SocketCreate socket(10, 1) (AF_INET6, TCP) new sock 5
> setSocketNonBlocking fd 5 val 0 status 1
> setBufferSize fd 5 SNDBUF changed old 16384 new 35328
> setBufferSize fd 5 RCVBUF no change old 87380 new 35328
> setSocketCloseOnExec fd 5 val 1 status 1  
> --- 03/03/2016 07:55:31.348 UTC :SocketBind_ ok , sock 5, AF_INET6  port 50378 addr ::
> setSocketNonBlocking fd 5 val 1 status 1
> --- 03/03/2016 07:55:31.348 UTC :SocketListen ok  , sock 5, AF_INET6  port 50378 addr ::
> AF_INET6 getaddrinfo(::1, 44050, flags 0x40c ...)  -- OK
> --- 03/03/2016 07:55:31.348 UTC :SocketCreate socket(10, 1) (AF_INET6, TCP) new sock 6
> setSocketNonBlocking fd 6 val 1 status 1
> setBufferSize fd 6 SNDBUF changed old 16384 new 35328
> setBufferSize fd 6 RCVBUF no change old 87380 new 35328
> setSocketCloseOnExec fd 6 val 1 status 1  
> --- 03/03/2016 07:55:31.348 UTC :starting SocketConnect , sock 6, AF_INET6  port 44050 addr ::1
> --- 03/03/2016 07:55:31.349 UTC :SocketConnect: sock 6 SOCK_EINPROGRESS , success
> --- 03/03/2016 07:55:31.349 UTC :sock 6 EVENTRES_EVENT success
> setSocketNonBlocking fd 6 val 1 status 1
> setSocketCloseOnExec fd 6 val 1 status 1  
> setSocketNonBlocking fd 6 val 0 status 1
> SocketAccept success: listeningSocket 5 resultSocket 6
> setSocketNonBlocking fd 6 val 1 status 1
> setSocketCloseOnExec fd 6 val 1 status 1  
> SocketAccept: listeningSocket 5 errno 11 , Resource temporarily unavailable
> SocketAccept success: listeningSocket 5 resultSocket 7
> setSocketNonBlocking fd 7 val 1 status 1
> setSocketCloseOnExec fd 7 val 1 status 1  
> setSocketNonBlocking fd 5 val 0 status 1
> cmdLine=
> [Info]: RPC client/gem/minimum GCI levels = 851/851/851
> [Info]: Client PID: 26647
> AF_INET4 getaddrinfo(dev2.yesplan.be, 46179, flags 0x0 ...)  -- OK
> --- 03/03/2016 07:55:31.924 UTC :SocketCreate socket(2, 1) (AF_INET, TCP) new sock 5
> setSocketNonBlocking fd 5 val 1 status 1
> setBufferSize fd 5 SNDBUF changed old 16384 new 35328
> setBufferSize fd 5 RCVBUF no change old 87380 new 35328
> setSocketCloseOnExec fd 5 val 1 status 1  
> --- 03/03/2016 07:55:31.924 UTC :starting SocketConnect , sock 5, AF_INET, port 46179 addr 23.251.142.191
> --- 03/03/2016 07:55:31.924 UTC :SocketConnect: sock 5 SOCK_EINPROGRESS , success
> --- 03/03/2016 07:56:31.925 UTC :sock 5 EVENTRES_TIMEOUT
> --- 03/03/2016 07:56:31.925 UTC :NetSBasicTimedConnect IB failed,  
> AF_INET6 getaddrinfo(dev2.yesplan.be, 46179, flags 0x8 ...)  -- OK
> --- 03/03/2016 07:56:31.925 UTC :SocketCreate socket(10, 1) (AF_INET6, TCP) new sock 5
> setSocketNonBlocking fd 5 val 1 status 1
> setBufferSize fd 5 SNDBUF changed old 16384 new 35328
> setBufferSize fd 5 RCVBUF no change old 87380 new 35328
> setSocketCloseOnExec fd 5 val 1 status 1  
> --- 03/03/2016 07:56:31.925 UTC :starting SocketConnect , sock 5, AF_INET6  port 46179 addr ::ffff:23.251.142.191
> --- 03/03/2016 07:56:31.925 UTC :SocketConnect: sock 5 SOCK_EINPROGRESS , success
> --- 03/03/2016 07:57:31.933 UTC :sock 5 EVENTRES_TIMEOUT
> --- 03/03/2016 07:57:31.933 UTC :NetSBasicTimedConnect IB failed,  
> Error connecting to stone:
> [Info]: Logging out at 03/03/2016 07:57:31 UTC
> -----------------------------------------------------
> GemStone: Error         Fatal
> The connection to the Stone Repository monitor was refused:
> ,
> Error Category: 231169 [GemStone] Number: 4136  Arg Count: 1 Context : 20 exception : 20
> Arg 1:   20
> setSocketNonBlocking fd 6 val 0 status 1
> setSocketNonBlocking fd 7 val 0 status 1
> *****************************************************
> ****** GemStone Abnormal Shutdown at 03/03/2016 07:57:32 UTC
> *****************************************************
> -----------------------------------------------------
> GemStone: Error         Fatal
> The connection to the Stone Repository monitor was refused:
> ,
> Error Category: 231169 [GemStone] Number: 4136  Arg Count: 1 Context : 20 exception : 20
> Arg 1:   20
>
>> On 01 Mar 2016, at 23:53, Dale Henrichs <[hidden email]> wrote:
>>
>> Okay ... we see that you did get a gem log, so that means that you are connecting to the netldi and getting the gem launched, but you are not connecting to the stone for some reason ... we ought to print some of that info to the gem log, but we don't currently (I'm submitting a bug:).
>>
>> The stone was started as `seaside`, so I'm curious what you GemTools login information looks like just to be sure ...
>>
>> Also I am wondering if  `/bin/hostname` and `/etc/hosts` are consistent...`/bin/hostname` needs to be in `/etc/hosts` ...
>>
>> Finally, we can get more information about what might be going wrong by stopping the netldi and setting the following env var and then restarting the netldi:
>>
>> export GEMSTONE_SOCKET_DEBUG=1
>>
>> This time when you attempt to login from GemTools, you should get a bunch of information dumped to the gem log ... a fresh set of stone/gem/netldi logs would then help us figure things out ...
>>
>> Dale
>>
>> On 03/01/2016 12:17 AM, Johan Brichau wrote:
>>> Hi Dale,
>>>
>>>> Let's see, I've got a testLogin command in tODE that gives me better diagnostics ...
>>> I need to catch up on tODE ;)
>>> I’ll see to get there so I can send you the info of that command.
>>>
>>>> So think I'd like to see the error message that is associated with the failed login  ... I assume that you are getting a client-side stack, so extracting the errMsg (I think) from the GcIErrSType instance would perhaps give some clue …
>>> Client-side, the error is a Gci31xErrSType64 instance with the following messageText:
>>>
>>> 'The connection to the Stone Repository monitor was refused:
>>> , '
>>>
>>>> Also a look at the netldi logfile and the stone log file might shed additional light …
>>> I am attaching those logs files here.
>>>
>>> btw: the Linux firewall (iptables) is turned off.
>>> Google compute engine vm instances have 2 ips: an internal and an external ip. I do suspect this has something to do with it but I’m not sure where Gemstone / netldi goes wrong with that.
>>>
>>> thanks for all the help
>>> Johan
>>>
>>>
>>
>

> On 03 Mar 2016, at 20:59, Johan Brichau <[hidden email]> wrote:
>
> Dale,
>
> I should have tried this before… I opened up a very wide range of ports in the firewall since I saw that some other port numbers than 50377 and 50378 were being mentioned in the log files.
> Also, when I compare this log with a successful connection log on another machine, I see the same ‘error’ at the end of netldi log.
>
> So: why does netldi not respect the ‘pool of ports’ ?
> I start netldi as follows:
>
> $GEMSTONE/bin/startnetldi -p 50378:50378 -g -a $USER -n -l $GEMSTONE_LOGDIR/gs64ldi.log
>
> cheers
> Johan
>
>> On 03 Mar 2016, at 20:34, Johan Brichau <[hidden email]> wrote:
>>
>> Hi Dale,
>>
>> With the GEMSTONE_SOCKET_DEBUG set to 1, I get the following log in the netldi and gem log files.
>> The VM instance I have is installed with a CentOS 6 but I did not find a /etc/hostname (I do have these on other centos 6 machines, not on google). I added the hostname dev2.yesplan.be there and also added it elsewhere.
>>
>> Do I read correctly a connection to my client machine is being tried on port 0 (which fails)?
>>
>> thx
>> Johan
>>
>>
>> Internal tables are configured for 1359 concurrent clients.
>> Server socket gs64ldi (protocol TCP) is created, with a delay of 30 seconds.
>> Summary of netldi parameters:
>>  The host name is 'dev2.yesplan.be'.
>> GEMSTONE is: '/opt/gemstone/product'.
>> GEMSTONE_NRS_ALL is: '#dir:$GEMSTONE_DATADIR#log:/$GEMSTONE_LOGDIR/%N%P.log'.
>>  Clients are not authenticated.
>>  Process creation MUST be through services.dat.
>> Pool of ports is 50378:50378 .
>>  Created processes belong to the account named 'johan'.
>>  The default directory for log files is '#dir:$GEMSTONE_DATADIR#log:/$GEMSTONE_LOGDIR/%N%P.log'.
>>  The command line is:
>>    /opt/gemstone/GemStone64Bit3.1.0.6-x86_64.Linux/sys/netldid -g -n -ajohan \
>>     -p50378:50378 -l /opt/gemstone/log/gs64ldi.log gs64ldi
>> Entering Service Loop
>> SocketAccept success: listeningSocket 6 resultSocket 5
>> setSocketNonBlocking fd 5 val 1 status 1
>> setSocketCloseOnExec fd 5 val 1 status 1  
>> setSocketNonBlocking fd 5 val 0 status 1
>> SocketAccept success: listeningSocket 6 resultSocket 5
>> setSocketNonBlocking fd 5 val 1 status 1
>> setSocketCloseOnExec fd 5 val 1 status 1  
>> --- 03/03/2016 07:55:31.329 UTC :SocketCreate socket(10, 1) (AF_INET6, TCP) new sock 8
>> setSocketNonBlocking fd 8 val 0 status 1
>> setBufferSize fd 8 SNDBUF changed old 16384 new 35328
>> setBufferSize fd 8 RCVBUF no change old 87380 new 35328
>> setSocketCloseOnExec fd 8 val 1 status 1  
>> --- 03/03/2016 07:55:31.329 UTC :SocketBind_ ok , sock 8, AF_INET6  port 0 addr ::1
>> setSocketNonBlocking fd 8 val 1 status 1
>> --- 03/03/2016 07:55:31.329 UTC :SocketListen ok  , sock 8, AF_INET6  port 44050 addr ::1
>> AF_INET4 getaddrinfo(94-224-141-192.access.telenet.be, NULL, flags 0x0 ...)  -- OK
>> --- 03/03/2016 07:55:31.331 UTC :SocketCreate socket(2, 2) (AF_INET, UDP) new sock 9
>> --- 03/03/2016 07:55:31.332 UTC :SocketBind_ failed , sock 9, AF_INET, port 0 addr 94.224.141.192 errno 99
>> SocketBind_ failed: bind(9,94.224.141.192, port 0) failed with errno=99,EADDRNOTAVAIL, Cannot assign requested addressn
>>
>>
>>
>> The gemnetlogobject file says:
>>
>> --- 03/03/2016 07:55:31.348 UTC :SocketCreate socket(10, 1) (AF_INET6, TCP) new sock 5
>> setSocketNonBlocking fd 5 val 0 status 1
>> setBufferSize fd 5 SNDBUF changed old 16384 new 35328
>> setBufferSize fd 5 RCVBUF no change old 87380 new 35328
>> setSocketCloseOnExec fd 5 val 1 status 1  
>> --- 03/03/2016 07:55:31.348 UTC :SocketBind_ ok , sock 5, AF_INET6  port 50378 addr ::
>> setSocketNonBlocking fd 5 val 1 status 1
>> --- 03/03/2016 07:55:31.348 UTC :SocketListen ok  , sock 5, AF_INET6  port 50378 addr ::
>> AF_INET6 getaddrinfo(::1, 44050, flags 0x40c ...)  -- OK
>> --- 03/03/2016 07:55:31.348 UTC :SocketCreate socket(10, 1) (AF_INET6, TCP) new sock 6
>> setSocketNonBlocking fd 6 val 1 status 1
>> setBufferSize fd 6 SNDBUF changed old 16384 new 35328
>> setBufferSize fd 6 RCVBUF no change old 87380 new 35328
>> setSocketCloseOnExec fd 6 val 1 status 1  
>> --- 03/03/2016 07:55:31.348 UTC :starting SocketConnect , sock 6, AF_INET6  port 44050 addr ::1
>> --- 03/03/2016 07:55:31.349 UTC :SocketConnect: sock 6 SOCK_EINPROGRESS , success
>> --- 03/03/2016 07:55:31.349 UTC :sock 6 EVENTRES_EVENT success
>> setSocketNonBlocking fd 6 val 1 status 1
>> setSocketCloseOnExec fd 6 val 1 status 1  
>> setSocketNonBlocking fd 6 val 0 status 1
>> SocketAccept success: listeningSocket 5 resultSocket 6
>> setSocketNonBlocking fd 6 val 1 status 1
>> setSocketCloseOnExec fd 6 val 1 status 1  
>> SocketAccept: listeningSocket 5 errno 11 , Resource temporarily unavailable
>> SocketAccept success: listeningSocket 5 resultSocket 7
>> setSocketNonBlocking fd 7 val 1 status 1
>> setSocketCloseOnExec fd 7 val 1 status 1  
>> setSocketNonBlocking fd 5 val 0 status 1
>> cmdLine=
>> [Info]: RPC client/gem/minimum GCI levels = 851/851/851
>> [Info]: Client PID: 26647
>> AF_INET4 getaddrinfo(dev2.yesplan.be, 46179, flags 0x0 ...)  -- OK
>> --- 03/03/2016 07:55:31.924 UTC :SocketCreate socket(2, 1) (AF_INET, TCP) new sock 5
>> setSocketNonBlocking fd 5 val 1 status 1
>> setBufferSize fd 5 SNDBUF changed old 16384 new 35328
>> setBufferSize fd 5 RCVBUF no change old 87380 new 35328
>> setSocketCloseOnExec fd 5 val 1 status 1  
>> --- 03/03/2016 07:55:31.924 UTC :starting SocketConnect , sock 5, AF_INET, port 46179 addr 23.251.142.191
>> --- 03/03/2016 07:55:31.924 UTC :SocketConnect: sock 5 SOCK_EINPROGRESS , success
>> --- 03/03/2016 07:56:31.925 UTC :sock 5 EVENTRES_TIMEOUT
>> --- 03/03/2016 07:56:31.925 UTC :NetSBasicTimedConnect IB failed,  
>> AF_INET6 getaddrinfo(dev2.yesplan.be, 46179, flags 0x8 ...)  -- OK
>> --- 03/03/2016 07:56:31.925 UTC :SocketCreate socket(10, 1) (AF_INET6, TCP) new sock 5
>> setSocketNonBlocking fd 5 val 1 status 1
>> setBufferSize fd 5 SNDBUF changed old 16384 new 35328
>> setBufferSize fd 5 RCVBUF no change old 87380 new 35328
>> setSocketCloseOnExec fd 5 val 1 status 1  
>> --- 03/03/2016 07:56:31.925 UTC :starting SocketConnect , sock 5, AF_INET6  port 46179 addr ::ffff:23.251.142.191
>> --- 03/03/2016 07:56:31.925 UTC :SocketConnect: sock 5 SOCK_EINPROGRESS , success
>> --- 03/03/2016 07:57:31.933 UTC :sock 5 EVENTRES_TIMEOUT
>> --- 03/03/2016 07:57:31.933 UTC :NetSBasicTimedConnect IB failed,  
>> Error connecting to stone:
>> [Info]: Logging out at 03/03/2016 07:57:31 UTC
>> -----------------------------------------------------
>> GemStone: Error         Fatal
>> The connection to the Stone Repository monitor was refused:
>> ,
>> Error Category: 231169 [GemStone] Number: 4136  Arg Count: 1 Context : 20 exception : 20
>> Arg 1:   20
>> setSocketNonBlocking fd 6 val 0 status 1
>> setSocketNonBlocking fd 7 val 0 status 1
>> *****************************************************
>> ****** GemStone Abnormal Shutdown at 03/03/2016 07:57:32 UTC
>> *****************************************************
>> -----------------------------------------------------
>> GemStone: Error         Fatal
>> The connection to the Stone Repository monitor was refused:
>> ,
>> Error Category: 231169 [GemStone] Number: 4136  Arg Count: 1 Context : 20 exception : 20
>> Arg 1:   20
>>
>>> On 01 Mar 2016, at 23:53, Dale Henrichs <[hidden email]> wrote:
>>>
>>> Okay ... we see that you did get a gem log, so that means that you are connecting to the netldi and getting the gem launched, but you are not connecting to the stone for some reason ... we ought to print some of that info to the gem log, but we don't currently (I'm submitting a bug:).
>>>
>>> The stone was started as `seaside`, so I'm curious what you GemTools login information looks like just to be sure ...
>>>
>>> Also I am wondering if  `/bin/hostname` and `/etc/hosts` are consistent...`/bin/hostname` needs to be in `/etc/hosts` ...
>>>
>>> Finally, we can get more information about what might be going wrong by stopping the netldi and setting the following env var and then restarting the netldi:
>>>
>>> export GEMSTONE_SOCKET_DEBUG=1
>>>
>>> This time when you attempt to login from GemTools, you should get a bunch of information dumped to the gem log ... a fresh set of stone/gem/netldi logs would then help us figure things out ...
>>>
>>> Dale
>>>
>>> On 03/01/2016 12:17 AM, Johan Brichau wrote:
>>>> Hi Dale,
>>>>
>>>>> Let's see, I've got a testLogin command in tODE that gives me better diagnostics ...
>>>> I need to catch up on tODE ;)
>>>> I’ll see to get there so I can send you the info of that command.
>>>>
>>>>> So think I'd like to see the error message that is associated with the failed login  ... I assume that you are getting a client-side stack, so extracting the errMsg (I think) from the GcIErrSType instance would perhaps give some clue …
>>>> Client-side, the error is a Gci31xErrSType64 instance with the following messageText:
>>>>
>>>> 'The connection to the Stone Repository monitor was refused:
>>>> , '
>>>>
>>>>> Also a look at the netldi logfile and the stone log file might shed additional light …
>>>> I am attaching those logs files here.
>>>>
>>>> btw: the Linux firewall (iptables) is turned off.
>>>> Google compute engine vm instances have 2 ips: an internal and an external ip. I do suspect this has something to do with it but I’m not sure where Gemstone / netldi goes wrong with that.
>>>>
>>>> thanks for all the help
>>>> Johan
>>>>
>>>>
>>>
>>
>

> Dale,
>
> Forgot to clarify: if I open up the port range from 40000 to 60000, gemtools connects just fine.
> We’re using a 3.1.0.6 behind a firewall elsewhere and we do not run into these issues, so there must be something specific going on here.
> IP6? the private/public IP or hostname at Google?
>
> Johan
>
>> On 03 Mar 2016, at 20:59, Johan Brichau <[hidden email]> wrote:
>>
>> Dale,
>>
>> I should have tried this before… I opened up a very wide range of ports in the firewall since I saw that some other port numbers than 50377 and 50378 were being mentioned in the log files.
>> Also, when I compare this log with a successful connection log on another machine, I see the same ‘error’ at the end of netldi log.
>>
>> So: why does netldi not respect the ‘pool of ports’ ?
>> I start netldi as follows:
>>
>> $GEMSTONE/bin/startnetldi -p 50378:50378 -g -a $USER -n -l $GEMSTONE_LOGDIR/gs64ldi.log
>>
>> cheers
>> Johan
>>
>>> On 03 Mar 2016, at 20:34, Johan Brichau <[hidden email]> wrote:
>>>
>>> Hi Dale,
>>>
>>> With the GEMSTONE_SOCKET_DEBUG set to 1, I get the following log in the netldi and gem log files.
>>> The VM instance I have is installed with a CentOS 6 but I did not find a /etc/hostname (I do have these on other centos 6 machines, not on google). I added the hostname dev2.yesplan.be there and also added it elsewhere.
>>>
>>> Do I read correctly a connection to my client machine is being tried on port 0 (which fails)?
>>>
>>> thx
>>> Johan
>>>
>>>
>>> Internal tables are configured for 1359 concurrent clients.
>>> Server socket gs64ldi (protocol TCP) is created, with a delay of 30 seconds.
>>> Summary of netldi parameters:
>>>   The host name is 'dev2.yesplan.be'.
>>> GEMSTONE is: '/opt/gemstone/product'.
>>> GEMSTONE_NRS_ALL is: '#dir:$GEMSTONE_DATADIR#log:/$GEMSTONE_LOGDIR/%N%P.log'.
>>>   Clients are not authenticated.
>>>   Process creation MUST be through services.dat.
>>> Pool of ports is 50378:50378 .
>>>   Created processes belong to the account named 'johan'.
>>>   The default directory for log files is '#dir:$GEMSTONE_DATADIR#log:/$GEMSTONE_LOGDIR/%N%P.log'.
>>>   The command line is:
>>>     /opt/gemstone/GemStone64Bit3.1.0.6-x86_64.Linux/sys/netldid -g -n -ajohan \
>>>      -p50378:50378 -l /opt/gemstone/log/gs64ldi.log gs64ldi
>>> Entering Service Loop
>>> SocketAccept success: listeningSocket 6 resultSocket 5
>>> setSocketNonBlocking fd 5 val 1 status 1
>>> setSocketCloseOnExec fd 5 val 1 status 1
>>> setSocketNonBlocking fd 5 val 0 status 1
>>> SocketAccept success: listeningSocket 6 resultSocket 5
>>> setSocketNonBlocking fd 5 val 1 status 1
>>> setSocketCloseOnExec fd 5 val 1 status 1
>>> --- 03/03/2016 07:55:31.329 UTC :SocketCreate socket(10, 1) (AF_INET6, TCP) new sock 8
>>> setSocketNonBlocking fd 8 val 0 status 1
>>> setBufferSize fd 8 SNDBUF changed old 16384 new 35328
>>> setBufferSize fd 8 RCVBUF no change old 87380 new 35328
>>> setSocketCloseOnExec fd 8 val 1 status 1
>>> --- 03/03/2016 07:55:31.329 UTC :SocketBind_ ok , sock 8, AF_INET6  port 0 addr ::1
>>> setSocketNonBlocking fd 8 val 1 status 1
>>> --- 03/03/2016 07:55:31.329 UTC :SocketListen ok  , sock 8, AF_INET6  port 44050 addr ::1
>>> AF_INET4 getaddrinfo(94-224-141-192.access.telenet.be, NULL, flags 0x0 ...)  -- OK
>>> --- 03/03/2016 07:55:31.331 UTC :SocketCreate socket(2, 2) (AF_INET, UDP) new sock 9
>>> --- 03/03/2016 07:55:31.332 UTC :SocketBind_ failed , sock 9, AF_INET, port 0 addr 94.224.141.192 errno 99
>>> SocketBind_ failed: bind(9,94.224.141.192, port 0) failed with errno=99,EADDRNOTAVAIL, Cannot assign requested addressn
>>>
>>>
>>>
>>> The gemnetlogobject file says:
>>>
>>> --- 03/03/2016 07:55:31.348 UTC :SocketCreate socket(10, 1) (AF_INET6, TCP) new sock 5
>>> setSocketNonBlocking fd 5 val 0 status 1
>>> setBufferSize fd 5 SNDBUF changed old 16384 new 35328
>>> setBufferSize fd 5 RCVBUF no change old 87380 new 35328
>>> setSocketCloseOnExec fd 5 val 1 status 1
>>> --- 03/03/2016 07:55:31.348 UTC :SocketBind_ ok , sock 5, AF_INET6  port 50378 addr ::
>>> setSocketNonBlocking fd 5 val 1 status 1
>>> --- 03/03/2016 07:55:31.348 UTC :SocketListen ok  , sock 5, AF_INET6  port 50378 addr ::
>>> AF_INET6 getaddrinfo(::1, 44050, flags 0x40c ...)  -- OK
>>> --- 03/03/2016 07:55:31.348 UTC :SocketCreate socket(10, 1) (AF_INET6, TCP) new sock 6
>>> setSocketNonBlocking fd 6 val 1 status 1
>>> setBufferSize fd 6 SNDBUF changed old 16384 new 35328
>>> setBufferSize fd 6 RCVBUF no change old 87380 new 35328
>>> setSocketCloseOnExec fd 6 val 1 status 1
>>> --- 03/03/2016 07:55:31.348 UTC :starting SocketConnect , sock 6, AF_INET6  port 44050 addr ::1
>>> --- 03/03/2016 07:55:31.349 UTC :SocketConnect: sock 6 SOCK_EINPROGRESS , success
>>> --- 03/03/2016 07:55:31.349 UTC :sock 6 EVENTRES_EVENT success
>>> setSocketNonBlocking fd 6 val 1 status 1
>>> setSocketCloseOnExec fd 6 val 1 status 1
>>> setSocketNonBlocking fd 6 val 0 status 1
>>> SocketAccept success: listeningSocket 5 resultSocket 6
>>> setSocketNonBlocking fd 6 val 1 status 1
>>> setSocketCloseOnExec fd 6 val 1 status 1
>>> SocketAccept: listeningSocket 5 errno 11 , Resource temporarily unavailable
>>> SocketAccept success: listeningSocket 5 resultSocket 7
>>> setSocketNonBlocking fd 7 val 1 status 1
>>> setSocketCloseOnExec fd 7 val 1 status 1
>>> setSocketNonBlocking fd 5 val 0 status 1
>>> cmdLine=
>>> [Info]: RPC client/gem/minimum GCI levels = 851/851/851
>>> [Info]: Client PID: 26647
>>> AF_INET4 getaddrinfo(dev2.yesplan.be, 46179, flags 0x0 ...)  -- OK
>>> --- 03/03/2016 07:55:31.924 UTC :SocketCreate socket(2, 1) (AF_INET, TCP) new sock 5
>>> setSocketNonBlocking fd 5 val 1 status 1
>>> setBufferSize fd 5 SNDBUF changed old 16384 new 35328
>>> setBufferSize fd 5 RCVBUF no change old 87380 new 35328
>>> setSocketCloseOnExec fd 5 val 1 status 1
>>> --- 03/03/2016 07:55:31.924 UTC :starting SocketConnect , sock 5, AF_INET, port 46179 addr 23.251.142.191
>>> --- 03/03/2016 07:55:31.924 UTC :SocketConnect: sock 5 SOCK_EINPROGRESS , success
>>> --- 03/03/2016 07:56:31.925 UTC :sock 5 EVENTRES_TIMEOUT
>>> --- 03/03/2016 07:56:31.925 UTC :NetSBasicTimedConnect IB failed,
>>> AF_INET6 getaddrinfo(dev2.yesplan.be, 46179, flags 0x8 ...)  -- OK
>>> --- 03/03/2016 07:56:31.925 UTC :SocketCreate socket(10, 1) (AF_INET6, TCP) new sock 5
>>> setSocketNonBlocking fd 5 val 1 status 1
>>> setBufferSize fd 5 SNDBUF changed old 16384 new 35328
>>> setBufferSize fd 5 RCVBUF no change old 87380 new 35328
>>> setSocketCloseOnExec fd 5 val 1 status 1
>>> --- 03/03/2016 07:56:31.925 UTC :starting SocketConnect , sock 5, AF_INET6  port 46179 addr ::ffff:23.251.142.191
>>> --- 03/03/2016 07:56:31.925 UTC :SocketConnect: sock 5 SOCK_EINPROGRESS , success
>>> --- 03/03/2016 07:57:31.933 UTC :sock 5 EVENTRES_TIMEOUT
>>> --- 03/03/2016 07:57:31.933 UTC :NetSBasicTimedConnect IB failed,
>>> Error connecting to stone:
>>> [Info]: Logging out at 03/03/2016 07:57:31 UTC
>>> -----------------------------------------------------
>>> GemStone: Error         Fatal
>>> The connection to the Stone Repository monitor was refused:
>>> ,
>>> Error Category: 231169 [GemStone] Number: 4136  Arg Count: 1 Context : 20 exception : 20
>>> Arg 1:   20
>>> setSocketNonBlocking fd 6 val 0 status 1
>>> setSocketNonBlocking fd 7 val 0 status 1
>>> *****************************************************
>>> ****** GemStone Abnormal Shutdown at 03/03/2016 07:57:32 UTC
>>> *****************************************************
>>> -----------------------------------------------------
>>> GemStone: Error         Fatal
>>> The connection to the Stone Repository monitor was refused:
>>> ,
>>> Error Category: 231169 [GemStone] Number: 4136  Arg Count: 1 Context : 20 exception : 20
>>> Arg 1:   20
>>>
>>>> On 01 Mar 2016, at 23:53, Dale Henrichs <[hidden email]> wrote:
>>>>
>>>> Okay ... we see that you did get a gem log, so that means that you are connecting to the netldi and getting the gem launched, but you are not connecting to the stone for some reason ... we ought to print some of that info to the gem log, but we don't currently (I'm submitting a bug:).
>>>>
>>>> The stone was started as `seaside`, so I'm curious what you GemTools login information looks like just to be sure ...
>>>>
>>>> Also I am wondering if  `/bin/hostname` and `/etc/hosts` are consistent...`/bin/hostname` needs to be in `/etc/hosts` ...
>>>>
>>>> Finally, we can get more information about what might be going wrong by stopping the netldi and setting the following env var and then restarting the netldi:
>>>>
>>>> export GEMSTONE_SOCKET_DEBUG=1
>>>>
>>>> This time when you attempt to login from GemTools, you should get a bunch of information dumped to the gem log ... a fresh set of stone/gem/netldi logs would then help us figure things out ...
>>>>
>>>> Dale
>>>>
>>>> On 03/01/2016 12:17 AM, Johan Brichau wrote:
>>>>> Hi Dale,
>>>>>
>>>>>> Let's see, I've got a testLogin command in tODE that gives me better diagnostics ...
>>>>> I need to catch up on tODE ;)
>>>>> I’ll see to get there so I can send you the info of that command.
>>>>>
>>>>>> So think I'd like to see the error message that is associated with the failed login  ... I assume that you are getting a client-side stack, so extracting the errMsg (I think) from the GcIErrSType instance would perhaps give some clue …
>>>>> Client-side, the error is a Gci31xErrSType64 instance with the following messageText:
>>>>>
>>>>> 'The connection to the Stone Repository monitor was refused:
>>>>> , '
>>>>>
>>>>>> Also a look at the netldi logfile and the stone log file might shed additional light …
>>>>> I am attaching those logs files here.
>>>>>
>>>>> btw: the Linux firewall (iptables) is turned off.
>>>>> Google compute engine vm instances have 2 ips: an internal and an external ip. I do suspect this has something to do with it but I’m not sure where Gemstone / netldi goes wrong with that.
>>>>>
>>>>> thanks for all the help
>>>>> Johan
>>>>>
>>>>>