Multiple GS sessions in Polycephaly drone

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Multiple GS sessions in Polycephaly drone

Bob Nemec
Hello,
I'm looking for some diagnostic advice. Our Seaside application runs in VW, using several images, each with several GS sessions. One 'dispatcher' image communicates to each application image with OpenTalk for status and control, and redirects initial logins. All works fine.

I wanted a simpler way to manage this multi-image setup, so I wrote a version that uses Polycephaly. All works: each drone is able to log in to GemStone and serve a Seaside interface on its own port. 

Our web server setup allows for multiple shared GS sessions, and it works fine in the separate image configuration.  It also works fine if I spawn a DebugMachine instead of a ViritualMachine.  However, if I use a VirutalMachine, logging in the second GS session locks up the Seaside interface and I get an 'Connection can not reopen!' on the ExternalWriteStream for that drone.

Any suggestions on how to diagnose this?  

It's probably a unique mix of Polycephaly and GS; it would be cool to hear of another setup.

FWIW: I have... 
GbsConfiguration current confirm: false.
GbsConfiguration current verbose: false.

Thanks for any help,

Bob Nemec
HTS Engineering

--
GemStone-Smalltalk mailing list
Unsubscribe: send e-mail with the word "unsubscribe" in body (without quotes) to:
  [hidden email]
Archive: http://forum.world.st/Gemstone-Customers-f1461796.html
Bob Nemec
Reply | Threaded
Open this post in threaded view
|

Re: [vwnc] Multiple GS sessions in Polycephaly drone

Michael Lucas-Smith-2
Have you tried using Poly2 instead? There's issues in Poly1 of stdout getting reused by other interfaces accidentally and breaking the poly communication. Besides that, I'm not sure - debug just keeps things headful, so if there's some code that needs the system to be headful and it's not when you run it in normal mode, perhaps that's the problem.

Michael

On 31/05/2013, at 12:25 AM, [hidden email] wrote:

Hello,
I'm looking for some diagnostic advice. Our Seaside application runs in VW, using several images, each with several GS sessions. One 'dispatcher' image communicates to each application image with OpenTalk for status and control, and redirects initial logins. All works fine.

I wanted a simpler way to manage this multi-image setup, so I wrote a version that uses Polycephaly. All works: each drone is able to log in to GemStone and serve a Seaside interface on its own port. 

Our web server setup allows for multiple shared GS sessions, and it works fine in the separate image configuration.  It also works fine if I spawn a DebugMachine instead of a ViritualMachine.  However, if I use a VirutalMachine, logging in the second GS session locks up the Seaside interface and I get an 'Connection can not reopen!' on the ExternalWriteStream for that drone.

Any suggestions on how to diagnose this?  

It's probably a unique mix of Polycephaly and GS; it would be cool to hear of another setup.

FWIW: I have... 
GbsConfiguration current confirm: false.
GbsConfiguration current verbose: false.

Thanks for any help,

Bob Nemec
HTS Engineering
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc


--
GemStone-Smalltalk mailing list
Unsubscribe: send e-mail with the word "unsubscribe" in body (without quotes) to:
  [hidden email]
Archive: http://forum.world.st/Gemstone-Customers-f1461796.html
Reply | Threaded
Open this post in threaded view
|

Re: [vwnc] Multiple GS sessions in Polycephaly drone

Bob Nemec
Michael,
Using Polycephaly2 fixed the problem,

Thank you,
Bob


From: Michael Lucas-Smith <[hidden email]>
To: [hidden email]
Cc: "[hidden email]" <[hidden email]>; "[hidden email]" <[hidden email]>
Sent: Thursday, May 30, 2013 10:59:48 AM
Subject: Re: [vwnc] Multiple GS sessions in Polycephaly drone

Have you tried using Poly2 instead? There's issues in Poly1 of stdout getting reused by other interfaces accidentally and breaking the poly communication. Besides that, I'm not sure - debug just keeps things headful, so if there's some code that needs the system to be headful and it's not when you run it in normal mode, perhaps that's the problem.

Michael

On 31/05/2013, at 12:25 AM, [hidden email] wrote:

Hello,
I'm looking for some diagnostic advice. Our Seaside application runs in VW, using several images, each with several GS sessions. One 'dispatcher' image communicates to each application image with OpenTalk for status and control, and redirects initial logins. All works fine.

I wanted a simpler way to manage this multi-image setup, so I wrote a version that uses Polycephaly. All works: each drone is able to log in to GemStone and serve a Seaside interface on its own port. 

Our web server setup allows for multiple shared GS sessions, and it works fine in the separate image configuration.  It also works fine if I spawn a DebugMachine instead of a ViritualMachine.  However, if I use a VirutalMachine, logging in the second GS session locks up the Seaside interface and I get an 'Connection can not reopen!' on the ExternalWriteStream for that drone.

Any suggestions on how to diagnose this?  

It's probably a unique mix of Polycephaly and GS; it would be cool to hear of another setup.

FWIW: I have... 
GbsConfiguration current confirm: false.
GbsConfiguration current verbose: false.

Thanks for any help,

Bob Nemec
HTS Engineering
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc




--
GemStone-Smalltalk mailing list
Unsubscribe: send e-mail with the word "unsubscribe" in body (without quotes) to:
  [hidden email]
Archive: http://forum.world.st/Gemstone-Customers-f1461796.html
Bob Nemec
Reply | Threaded
Open this post in threaded view
|

Re: Multiple GS sessions in Polycephaly drone

Paul Baumann
In reply to this post by Bob Nemec

Hi Bob,

 

Suspect a deadlock with two semaphores acquired in a different sequence by two processes. I’ve found and avoided several of these over the years. I recall at least one would happen with code that is also used during debugging.

 

A trick for diagnosing is to have a background process that does a “GbsConfiguration dumpAllProcessStacks” every 30 seconds or so. All process stacks get written to a file that you can review. Review the most recent file after you notice a hang. Look for two processes each holding a semaphore and waiting on the semaphore held by the other. If the GBS version you use still has bugs like mentioned above then this trick may not work very well for you, but it is the approach that I use once the tools are fixed to allow it.

 

logging in the second GS session locks up the Seaside interface

 

You might want to refactor your code so that a background process handling requests is separate from the process that handles UI events. If the background process needs to do something UI related then have that happen within a #uiEventFor: block. It may be that Polycephaly is already giving you the UI separation but that your code needs #uiEventFor: for when a background process does something that involves a UI change.

 

It's probably a unique mix of Polycephaly and GS; it would be cool to hear of another setup.

 

This is probably overkill for your needs, but…

 

If you have a multiple-producers-but-one-consumer situation then you might think along the lines of a request-queue approach where a queue in VW is serviced in a background process that waits on a semaphore signal. You might think of it like an RcQueue for VW. An add to the queue would #signal the semaphore. The processing of each request involves a semaphore #wait. The queue acts to isolate and sequence the processing of requests in a uniform manner through GS. There are many ways you might want to notify when a request has been processed. You may need to protect code from changes in the queue collection (usually done with a second semaphore or a queue design like the EsQueue that you’d remember from VAST).

 

Paul Baumann

 

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of [hidden email]
Sent: Thursday, May 30, 2013 10:26
To: [hidden email]; [hidden email]
Subject: [GemStone-Smalltalk] Multiple GS sessions in Polycephaly drone

 

Hello,

I'm looking for some diagnostic advice. Our Seaside application runs in VW, using several images, each with several GS sessions. One 'dispatcher' image communicates to each application image with OpenTalk for status and control, and redirects initial logins. All works fine.

 

I wanted a simpler way to manage this multi-image setup, so I wrote a version that uses Polycephaly. All works: each drone is able to log in to GemStone and serve a Seaside interface on its own port. 

 

Our web server setup allows for multiple shared GS sessions, and it works fine in the separate image configuration.  It also works fine if I spawn a DebugMachine instead of a ViritualMachine.  However, if I use a VirutalMachine, logging in the second GS session locks up the Seaside interface and I get an 'Connection can not reopen!' on the ExternalWriteStream for that drone.

 

Any suggestions on how to diagnose this?  

 

It's probably a unique mix of Polycephaly and GS; it would be cool to hear of another setup.

 

FWIW: I have... 

GbsConfiguration current confirm: false.

GbsConfiguration current verbose: false.

 

Thanks for any help,

 

Bob Nemec

HTS Engineering



This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange, Inc. (ICE), its subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.

--
GemStone-Smalltalk mailing list
Unsubscribe: send e-mail with the word "unsubscribe" in body (without quotes) to:
  [hidden email]
Archive: http://forum.world.st/Gemstone-Customers-f1461796.html
Reply | Threaded
Open this post in threaded view
|

Re: Multiple GS sessions in Polycephaly drone

Bob Nemec
Thanks Paul,
Polycephaly2 solved my multi-GS session problem, but thanks for the diagnostic comments.  I had not used dumpAllProcessStacks before; nice.

I've avoided at lot of trouble with the shared sessions by only passing simple objects around (Array, String, Integer down, one XML string back). 
Even with that I had a problem when I passed a hard coded array:  #(('stuff')) is bad... Array with: (Array with: 'stuff') ...was OK.

Bob


From: Paul Baumann <[hidden email]>
To: "'[hidden email]'" <[hidden email]>; "[hidden email]" <[hidden email]>; "[hidden email]" <[hidden email]>
Sent: Thursday, May 30, 2013 12:51:42 PM
Subject: RE: [GemStone-Smalltalk] Multiple GS sessions in Polycephaly drone

Hi Bob,
 
Suspect a deadlock with two semaphores acquired in a different sequence by two processes. I’ve found and avoided several of these over the years. I recall at least one would happen with code that is also used during debugging.
 
A trick for diagnosing is to have a background process that does a “GbsConfiguration dumpAllProcessStacks” every 30 seconds or so. All process stacks get written to a file that you can review. Review the most recent file after you notice a hang. Look for two processes each holding a semaphore and waiting on the semaphore held by the other. If the GBS version you use still has bugs like mentioned above then this trick may not work very well for you, but it is the approach that I use once the tools are fixed to allow it.
 
logging in the second GS session locks up the Seaside interface
 
You might want to refactor your code so that a background process handling requests is separate from the process that handles UI events. If the background process needs to do something UI related then have that happen within a #uiEventFor: block. It may be that Polycephaly is already giving you the UI separation but that your code needs #uiEventFor: for when a background process does something that involves a UI change.
 
It's probably a unique mix of Polycephaly and GS; it would be cool to hear of another setup.
 
This is probably overkill for your needs, but…
 
If you have a multiple-producers-but-one-consumer situation then you might think along the lines of a request-queue approach where a queue in VW is serviced in a background process that waits on a semaphore signal. You might think of it like an RcQueue for VW. An add to the queue would #signal the semaphore. The processing of each request involves a semaphore #wait. The queue acts to isolate and sequence the processing of requests in a uniform manner through GS. There are many ways you might want to notify when a request has been processed. You may need to protect code from changes in the queue collection (usually done with a second semaphore or a queue design like the EsQueue that you’d remember from VAST).
 
Paul Baumann
 
 
From: [hidden email] [mailto:[hidden email]] On Behalf Of [hidden email]
Sent: Thursday, May 30, 2013 10:26
To: [hidden email]; [hidden email]
Subject: [GemStone-Smalltalk] Multiple GS sessions in Polycephaly drone
 
Hello,
I'm looking for some diagnostic advice. Our Seaside application runs in VW, using several images, each with several GS sessions. One 'dispatcher' image communicates to each application image with OpenTalk for status and control, and redirects initial logins. All works fine.
 
I wanted a simpler way to manage this multi-image setup, so I wrote a version that uses Polycephaly. All works: each drone is able to log in to GemStone and serve a Seaside interface on its own port. 
 
Our web server setup allows for multiple shared GS sessions, and it works fine in the separate image configuration.  It also works fine if I spawn a DebugMachine instead of a ViritualMachine.  However, if I use a VirutalMachine, logging in the second GS session locks up the Seaside interface and I get an 'Connection can not reopen!' on the ExternalWriteStream for that drone.
 
Any suggestions on how to diagnose this?  
 
It's probably a unique mix of Polycephaly and GS; it would be cool to hear of another setup.
 
FWIW: I have... 
GbsConfiguration current confirm: false.
GbsConfiguration current verbose: false.
 
Thanks for any help,
 
Bob Nemec
HTS Engineering


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange, Inc. (ICE), its subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.



--
GemStone-Smalltalk mailing list
Unsubscribe: send e-mail with the word "unsubscribe" in body (without quotes) to:
  [hidden email]
Archive: http://forum.world.st/Gemstone-Customers-f1461796.html
Bob Nemec
Reply | Threaded
Open this post in threaded view
|

Re: Multiple GS sessions in Polycephaly drone

Paul Baumann

Hi Bob,

 

It is interesting that you are having trouble replicating arrays that are method literals and yet not strings that are often method literals too. That may suggest a problem related to managing persistent objects that are in the export set for more than one session. It seems you can avoid the problem by passing transient objects that another session would not also have. You might want GemTalk/vmware/GemStone/Brokat/GemStone/Servio to track that down.

 

It is eerily close to the block replication problem that has randomly affected us for a couple years now. In that case, GS and VW get out of agreement about the export status of the oop that had earlier represented the VW decompiled source for a block flushed to GS. GS thinks the oop is unused and assigns it to a different object that gets replicated to VW and VW uses the object (for the oop) that is still found in the GBS cache. The errors are easy to identify because the stack dumps show VW decompiled source for a block that ends up in an object graph. Finding the problem is a challenge because the bug happens sometime earlier. After years of trying to help track that down it was better to just avoid any replication and flushing of blocks. That workaround helps, but it is a challenge to find the patterns in existing code and also prevent re-introduction.

 

I’m glad Poly2 helped.

 

Regards,

 

Paul Baumann

 

 

From: [hidden email] [mailto:[hidden email]]
Sent: Friday, May 31, 2013 15:35
To: Paul Baumann; [hidden email]
Subject: Re: [GemStone-Smalltalk] Multiple GS sessions in Polycephaly drone

 

Thanks Paul,

Polycephaly2 solved my multi-GS session problem, but thanks for the diagnostic comments.  I had not used dumpAllProcessStacks before; nice.

 

I've avoided at lot of trouble with the shared sessions by only passing simple objects around (Array, String, Integer down, one XML string back). 

Even with that I had a problem when I passed a hard coded array:  #(('stuff')) is bad... Array with: (Array with: 'stuff') ...was OK.

 

Bob

 


From: Paul Baumann <[hidden email]>
To: "'[hidden email]'" <[hidden email]>; "[hidden email]" <[hidden email]>; "[hidden email]" <[hidden email]>
Sent: Thursday, May 30, 2013 12:51:42 PM
Subject: RE: [GemStone-Smalltalk] Multiple GS sessions in Polycephaly drone

 

Hi Bob,

 

Suspect a deadlock with two semaphores acquired in a different sequence by two processes. I’ve found and avoided several of these over the years. I recall at least one would happen with code that is also used during debugging.

 

A trick for diagnosing is to have a background process that does a “GbsConfiguration dumpAllProcessStacks” every 30 seconds or so. All process stacks get written to a file that you can review. Review the most recent file after you notice a hang. Look for two processes each holding a semaphore and waiting on the semaphore held by the other. If the GBS version you use still has bugs like mentioned above then this trick may not work very well for you, but it is the approach that I use once the tools are fixed to allow it.

 

logging in the second GS session locks up the Seaside interface

 

You might want to refactor your code so that a background process handling requests is separate from the process that handles UI events. If the background process needs to do something UI related then have that happen within a #uiEventFor: block. It may be that Polycephaly is already giving you the UI separation but that your code needs #uiEventFor: for when a background process does something that involves a UI change.

 

It's probably a unique mix of Polycephaly and GS; it would be cool to hear of another setup.

 

This is probably overkill for your needs, but…

 

If you have a multiple-producers-but-one-consumer situation then you might think along the lines of a request-queue approach where a queue in VW is serviced in a background process that waits on a semaphore signal. You might think of it like an RcQueue for VW. An add to the queue would #signal the semaphore. The processing of each request involves a semaphore #wait. The queue acts to isolate and sequence the processing of requests in a uniform manner through GS. There are many ways you might want to notify when a request has been processed. You may need to protect code from changes in the queue collection (usually done with a second semaphore or a queue design like the EsQueue that you’d remember from VAST).

 

Paul Baumann

 

 

From: [hidden email] [[hidden email]] On Behalf Of [hidden email]
Sent: Thursday, May 30, 2013 10:26
To: [hidden email]; [hidden email]
Subject: [GemStone-Smalltalk] Multiple GS sessions in Polycephaly drone

 

Hello,

I'm looking for some diagnostic advice. Our Seaside application runs in VW, using several images, each with several GS sessions. One 'dispatcher' image communicates to each application image with OpenTalk for status and control, and redirects initial logins. All works fine.

 

I wanted a simpler way to manage this multi-image setup, so I wrote a version that uses Polycephaly. All works: each drone is able to log in to GemStone and serve a Seaside interface on its own port. 

 

Our web server setup allows for multiple shared GS sessions, and it works fine in the separate image configuration.  It also works fine if I spawn a DebugMachine instead of a ViritualMachine.  However, if I use a VirutalMachine, logging in the second GS session locks up the Seaside interface and I get an 'Connection can not reopen!' on the ExternalWriteStream for that drone.

 

Any suggestions on how to diagnose this?  

 

It's probably a unique mix of Polycephaly and GS; it would be cool to hear of another setup.

 

FWIW: I have... 

GbsConfiguration current confirm: false.

GbsConfiguration current verbose: false.

 

Thanks for any help,

 

Bob Nemec

HTS Engineering

 


This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange, Inc. (ICE), its subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.

 



This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange, Inc. (ICE), its subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.

--
GemStone-Smalltalk mailing list
Unsubscribe: send e-mail with the word "unsubscribe" in body (without quotes) to:
  [hidden email]
Archive: http://forum.world.st/Gemstone-Customers-f1461796.html