[vwnc] Inactive Opentalk Broker
Locked
 
|
We have a Opentalk setup to communicate with support images in remote locations (transfer patches, check status, etc) that stops responding every two hours.
The six remote client locations poll the server at configured one to five minute intervals. The server records the client connection and compares it to the expected interval. If the time of the last contact is within the expected interval, the client image is considered active. Every ten minutes a process on the server checks to make sure that at least one client is considered active. If none are active, the Broker is stopped and a new one started.
Here is the interesting part: the server ends up restarting the Broker every two hours. Consistently. After one hour and fifty minutes, none of the client images get a response from the server. Once a new RequestBroker is created, the client images are able to connect again. Stopping and starting the same Broker instance did not work.
In a different setup, we use image to image communication to dispatch Seaside sessions across a pool of images. In those cases, the dispatcher image polls the Seaside images every 15 seconds. We've had no RequestBroker communication issues with the dispatcher images.
Any suggestions as to what could be causing the server RequestBroker to stop responding? Both the server and client Broker instances are created using... Opentalk.RequestBroker newStstTcpAt: ((IPSocketAddress hostAddress: hostAddress port: port))
...with their own hostAddresses and port numbers. And the client code stops and starts its broker for each connection. We check for Opentalk.OtECommunicationFailure exceptions, and while the server is not responding, we see a 'Remote invocation timeout!' error.
The restart works: from an application point of view, we don't really notice the client support image disconnect. But I'd really like to understand why this is happening. Thanks for any help, Bob Nemec Cherniak Software _______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Bob Nemec
|
|
|
"Bob N."<[hidden email]> wrote:
> Date: September 10, 2009 2:30:46 PM > Any suggestions as to what could be causing the server RequestBroker to stop > responding? > > Both the server and client Broker instances are created using... > Opentalk.RequestBroker newStstTcpAt: ((IPSocketAddress hostAddress: > hostAddress port: port)) > ...with their own hostAddresses and port numbers. > > And the client code stops and starts its broker for each connection. > We check for Opentalk.OtECommunicationFailure exceptions, and while the > server is not responding, we see a 'Remote invocation timeout!' error. If you get a timeout, it suggests that there was no error transmitting the ping to the server at least from the client point of view. The client just didn't receive the response from the ping. Any chance the server is failing during processing of the ping ? Or getting stuck there on a semaphore or something ? Try logging on the server: serverBroker sendAllEventsTo: EventPrinter new Any errors on the server side would be suppressed by default because those are all happening in background processes. If you want those pop up in with a notifier you can try serverBroker passErrors Hopefully that will help to narrow down the problem scope. HTH, Martin _______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc |
Re: [vwnc] Inactive Opentalk Broker
|
|
FWIW, we have just used one of our trial client ports (from 5i to 7.6) to implement headless clients bulk operations. We noticed that Opentalk seemed to saturate when it gets too many connections. For example, our limit was in the upwards neighborhood of 40 clients for a single broker. We found, however, that we could reorganize into a hierarchy of one master controller, some number and depth of intermediate server controllers, and n clients per server. Clients talked to their immediate upline server, which in turn talked to the master controller, forwarding client requests. This limited the total connections to a controller’s broker. This approach easily ran many clients for many hours at a time with enough work to kill the CPU’s on the server boxes without any problems in communications.
Cheers!
Tom Hawker -------------------------- Senior Framework Developer -------------------------- Home +1 (408) 274-4128 Office +1 (408) 576-6591 Mobile +1 (408) 835-3643
-----Original Message-----
"Bob N."<[hidden email]> wrote: > Date: September 10, 2009 2:30:46 PM > Any suggestions as to what could be causing the server RequestBroker to stop > responding? > > Both the server and client Broker instances are created using... > Opentalk.RequestBroker newStstTcpAt: ((IPSocketAddress hostAddress: > hostAddress port: port)) > ...with their own hostAddresses and port numbers. > > And the client code stops and starts its broker for each connection. > We check for Opentalk.OtECommunicationFailure exceptions, and while the > server is not responding, we see a 'Remote invocation timeout!' error.
If you get a timeout, it suggests that there was no error transmitting the ping to the server at least from the client point of view. The client just didn't receive the response from the ping. Any chance the server is failing during processing of the ping ? Or getting stuck there on a semaphore or something ?
Try logging on the server:
serverBroker sendAllEventsTo: EventPrinter new
Any errors on the server side would be suppressed by default because those are all happening in background processes. If you want those pop up in with a notifier you can try
serverBroker passErrors
Hopefully that will help to narrow down the problem scope.
HTH,
Martin _______________________________________________ vwnc mailing list http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
_______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc |
Re: [vwnc] Inactive Opentalk Broker
|
|
I’d also recommend browsing through official 7.6 resolutions and
upgrading to the latest 7.6c VM; for example, our Seaside-Opentalk broker
issues were resolved by, 95743 The Windows VM crashes during socket communication. 96010 Short listener backlog on the server causes TCP resets
on Windows. http://www.cincomsmalltalk.com/ResolutionsApplication/CincomResolutionsPortal -Boris -- +1.604.689.0322 DeepCove Labs Ltd. 4th floor 595 Howe Street Vancouver, Canada V6C 2T5 From:
[hidden email] [mailto:[hidden email]] On Behalf Of [hidden email] FWIW, we have just used one of our trial client ports
(from 5i to 7.6) to implement headless clients bulk operations. We
noticed that Opentalk seemed to saturate when it gets too many
connections. For example, our limit was in the upwards neighborhood of 40
clients for a single broker. We found, however, that we could reorganize
into a hierarchy of one master controller, some number and depth of
intermediate server controllers, and n clients per server. Clients talked
to their immediate upline server, which in turn talked to the master
controller, forwarding client requests. This limited the total
connections to a controller’s broker. This approach easily ran many
clients for many hours at a time with enough work to kill the CPU’s on the
server boxes without any problems in communications. Cheers! Tom Hawker -------------------------- Senior Framework Developer -------------------------- Home +1 (408) 274-4128 Office +1 (408) 576-6591 Mobile +1 (408) 835-3643 -----Original Message----- "Bob N."<[hidden email]> wrote: > Date: September 10, 2009 2:30:46 PM > Any suggestions as to what could be causing the
server RequestBroker to stop > responding? > > Both the server and client Broker instances are
created using... > Opentalk.RequestBroker
newStstTcpAt: ((IPSocketAddress hostAddress: > hostAddress port: port)) > ...with their own hostAddresses and port numbers. > > And the client code stops and starts its broker for
each connection. > We check for Opentalk.OtECommunicationFailure
exceptions, and while the > server is not responding, we see a 'Remote
invocation timeout!' error. If you get a timeout, it suggests that there was no error
transmitting the ping to the server at least from the client point of view. The
client just didn't receive the response from the ping. Any chance the server is
failing during processing of the ping ? Or getting stuck there on a semaphore
or something ? Try logging on the server: serverBroker
sendAllEventsTo: EventPrinter new Any errors on the server side would be suppressed by
default because those are all happening in background processes. If you want
those pop up in with a notifier you can try serverBroker passErrors Hopefully that will help to narrow down the problem
scope. HTH, Martin _______________________________________________ vwnc mailing list http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
_______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc |
| Free forum by Nabble | Edit this page |