Has anyone been able to implement the producer-consumer pattern?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Has anyone been able to implement the producer-consumer pattern?

Joshua Gephart
I realize that a SharedQueue can be used in a single OS process Smalltalk VM to implement the producer-consumer pattern as is explained on Bluebook page 265.  Also, I realize Semaphore's can not be shared between multiple OS processes in Gemstone.  I am looking for a SharedQueue equivalent that works between Gemstone sessions.  I'd prefer not to poll.

Would anyone be generous enough to post simple code to implement producer-consumer that can span processes/hosts in Gemstone DB?  Implementing in Topaz sessions would be sufficient.

Here is what I have currently:

On producer host:
  topaz
  printit
    UserGlobals at:#eventQueue put: RcQueue new
  %
  commit
  printit
    eventQueue add: anObject
  %
  commit

On consumer host:
  topaz
  commit
  printit
    eventQueue remove
  %

I am looking for a "SharedQueue next" equivalent that suspends the current process while the queue is empty.

Thanks for your help,

Joshua Gephart





--
GemStone-Smalltalk mailing list
Unsubscribe: send e-mail with the word "unsubscribe" in body (without quotes) to:
  [hidden email]
Archive: http://forum.world.st/Gemstone-Customers-f1461796.html
Reply | Threaded
Open this post in threaded view
|

Re: Has anyone been able to implement the producer-consumer pattern?

Ken Treis
Hi Joshua,

RcQueue will get you close. See the Programming Guide for a brief run-down on how to use it.

Because of the transactional nature of GemStone, I think you'll need to poll. Your consumer has a consistent view of the repository state within a transaction, and you need to commit or abort to get an updated view of the queue.

--
Ken Treis
Miriam Technologies, Inc.
(866) 652-2040 x221

On Feb 6, 2013, at 8:33 AM, Joshua Gephart wrote:

I realize that a SharedQueue can be used in a single OS process Smalltalk VM to implement the producer-consumer pattern as is explained on Bluebook page 265.  Also, I realize Semaphore's can not be shared between multiple OS processes in Gemstone.  I am looking for a SharedQueue equivalent that works between Gemstone sessions.  I'd prefer not to poll.

Would anyone be generous enough to post simple code to implement producer-consumer that can span processes/hosts in Gemstone DB?  Implementing in Topaz sessions would be sufficient.

Here is what I have currently:

On producer host:
  topaz
  printit
    UserGlobals at:#eventQueue put: RcQueue new
  %
  commit
  printit
    eventQueue add: anObject
  %
  commit

On consumer host:
  topaz
  commit
  printit
    eventQueue remove
  %

I am looking for a "SharedQueue next" equivalent that suspends the current process while the queue is empty.

Thanks for your help,

Joshua Gephart




--
GemStone-Smalltalk mailing list
Unsubscribe: send e-mail with the word "unsubscribe" in body (without quotes) to:
 [hidden email]
Archive: http://forum.world.st/Gemstone-Customers-f1461796.html


--
GemStone-Smalltalk mailing list
Unsubscribe: send e-mail with the word "unsubscribe" in body (without quotes) to:
  [hidden email]
Archive: http://forum.world.st/Gemstone-Customers-f1461796.html
Reply | Threaded
Open this post in threaded view
|

Re: Has anyone been able to implement the producer-consumer pattern?

Joshua Gephart
Using polling I have:

On producer host:

Unchanged from original post..

On consumer host:

topaz 1> input event_consumer.gs
topaz 1> printit
  | event logFile |
  logFile := GsFile openWrite: '/tmp/event_consumer.log'. 
  logFile nextPutAll: 'waiting for events..' ; cr.
  [1 = 1] whileTrue: [ 
    System continueTransaction.
    event := eventQueue remove.
    (event = nil) ifFalse: [ 
      logFile nextPutAll: event printString ; cr.
    ].
    Delay waitForSeconds: 1.
  ].
  GsFile closeAll.
%
joshua@consumer:/tmp$ tail -f event_consumer.log 
waiting for events..
10



Ken, I used "System continueTransaction" to update my view.

Bruce, I'll take a go at gem-to-gem signalling.

Thank you for your input guys,

Joshua

On Thu, Feb 7, 2013 at 12:34 AM, Bruce Badger <[hidden email]> wrote:

Would gem-to-gem signalling help here?

Sent from my phone.  Please forgive brevity.

On 6 Feb 2013 17:11, "Ken Treis" <[hidden email]> wrote:
Hi Joshua,

RcQueue will get you close. See the Programming Guide for a brief run-down on how to use it.

Because of the transactional nature of GemStone, I think you'll need to poll. Your consumer has a consistent view of the repository state within a transaction, and you need to commit or abort to get an updated view of the queue.

--
Ken Treis
Miriam Technologies, Inc.
<a href="tel:%28866%29%20652-2040%20x221" value="+18666522040" target="_blank">(866) 652-2040 x221

On Feb 6, 2013, at 8:33 AM, Joshua Gephart wrote:

I realize that a SharedQueue can be used in a single OS process Smalltalk VM to implement the producer-consumer pattern as is explained on Bluebook page 265.  Also, I realize Semaphore's can not be shared between multiple OS processes in Gemstone.  I am looking for a SharedQueue equivalent that works between Gemstone sessions.  I'd prefer not to poll.

Would anyone be generous enough to post simple code to implement producer-consumer that can span processes/hosts in Gemstone DB?  Implementing in Topaz sessions would be sufficient.

Here is what I have currently:

On producer host:
  topaz
  printit
    UserGlobals at:#eventQueue put: RcQueue new
  %
  commit
  printit
    eventQueue add: anObject
  %
  commit

On consumer host:
  topaz
  commit
  printit
    eventQueue remove
  %

I am looking for a "SharedQueue next" equivalent that suspends the current process while the queue is empty.

Thanks for your help,

Joshua Gephart




--
GemStone-Smalltalk mailing list
Unsubscribe: send e-mail with the word "unsubscribe" in body (without quotes) to:
 [hidden email]
Archive: http://forum.world.st/Gemstone-Customers-f1461796.html


--
GemStone-Smalltalk mailing list
Unsubscribe: send e-mail with the word "unsubscribe" in body (without quotes) to:
  [hidden email]
Archive: http://forum.world.st/Gemstone-Customers-f1461796.html



--
GemStone-Smalltalk mailing list
Unsubscribe: send e-mail with the word "unsubscribe" in body (without quotes) to:
  [hidden email]
Archive: http://forum.world.st/Gemstone-Customers-f1461796.html
Reply | Threaded
Open this post in threaded view
|

Re: Has anyone been able to implement the producer-consumer pattern?

Norm Green-2
In reply to this post by Joshua Gephart
I have a class that does that called FileSemaphore.  It uses user actions, so a shared library is required and the queue adder and remover must both be on the same host with the current implementation.  Also the current implementation uses a UNIX FIFO queue, so it only supports 1 adder and 1 remover.  But this limitation could be removed by using a socket instead.  RcQueue support multiple adders and 1 consumer.

The remover calls:

aFileSemaphore>>waitForMilliseconds: anInt

to wait for the adder to add something to the queue.  The result indicates if a signal was received from the sender or the method timed out.



The adder sends:

aFileSemaphore>>signal

to signal the remover.  The signal can be sent before or after the commit which adds something to the RcQueue finishes.  Signaling before the commit might give lower latency, but it also means the remover will have to poll and abort a few times for the object to show up in the queue while the adder's commit finishes.  Signaling after the commit returns successfully avoids this problem.

The code is partially in C++ and partially in Smalltalk.  The C++ part must be compiled into a user action shared library.

It should also be possible to implement this same pattern entirely in Smalltalk using GsSocket to communicate between the sessions.

Norm





From: "Joshua Gephart" <[hidden email]>
To: [hidden email]
Sent: Wednesday, February 6, 2013 8:33:12 AM
Subject: [GemStone-Smalltalk] Has anyone been able to implement the        producer-consumer pattern?

I realize that a SharedQueue can be used in a single OS process Smalltalk VM to implement the producer-consumer pattern as is explained on Bluebook page 265.  Also, I realize Semaphore's can not be shared between multiple OS processes in Gemstone.  I am looking for a SharedQueue equivalent that works between Gemstone sessions.  I'd prefer not to poll.

Would anyone be generous enough to post simple code to implement producer-consumer that can span processes/hosts in Gemstone DB?  Implementing in Topaz sessions would be sufficient.

Here is what I have currently:

On producer host:
  topaz
  printit
    UserGlobals at:#eventQueue put: RcQueue new
  %
  commit
  printit
    eventQueue add: anObject
  %
  commit

On consumer host:
  topaz
  commit
  printit
    eventQueue remove
  %

I am looking for a "SharedQueue next" equivalent that suspends the current process while the queue is empty.

Thanks for your help,

Joshua Gephart





--
GemStone-Smalltalk mailing list
Unsubscribe: send e-mail with the word "unsubscribe" in body (without quotes) to:
  [hidden email]
Archive: http://forum.world.st/Gemstone-Customers-f1461796.html


--
GemStone-Smalltalk mailing list
Unsubscribe: send e-mail with the word "unsubscribe" in body (without quotes) to:
  [hidden email]
Archive: http://forum.world.st/Gemstone-Customers-f1461796.html