ZnClient>>#close & socket semaphores

I ran into an out-of-external semaphores error when testing the Tropo
API. It uses ZnEasy to post data to their api servers.  I know I can
raise the semaphore limit by running:

Smalltalk vm maxExternalSemaphoresSilently: 3000.

then saving.  But I wondered if changing ZnClient>>#close to this:

close
        "Close any HTTP network connection that I might have kept open."

        connection
                ifNotNil: [
                        connection close: self isOneShot.
                        connection := nil.
                        state := #closed ].
        lastUsed := nil

And then adding these two methods:

ZdcAbstractSocketStream>>#close:
close: andDestroy
        "Close the stream, flush if necessary"

        self flush.
        socket isNil
                ifTrue: [ ^ self ].
        self socketClose: andDestroy.
        socket := nil

ZdcAbstractSocketStream>>#socketClose: andDestroy
        andDestroy
                ifTrue: [ socket closeAndDestroy ]
                ifFalse: [ socket close ]


might enable ZnClient to clean up after it issues 'one shot' requests in
a safe manner.

I can submit a change if so ( an a test! ).

Thanks
Paul

Paul,

On 15 May 2012, at 20:50, Paul DeBruicker wrote:

> I ran into an out-of-external semaphores error when testing the Tropo API. It uses ZnEasy to post data to their api servers.

Any idea at what rate you are posting, just a ballpark figure ?

I am assuming you are on Pharo 1.3 or 1.4. But since you are talking about ZdcAbstractSocketStream you have Zodiac loaded as well. Are you using HTTPS then ? Because #close: does not exist on regular SocketStreams.

If you are talking about regular HTTP, then you must have loaded the somewhat older/experimental ZdcNetworkingUtils, and are not using regular SocketStreams but ZdcOptimizedSocketStreams. Which is brave since this is something I am not doing myself ;-)

Anyway, the close and/or destroy discussion puzzles me as well. When running all unit Zn tests I sometimes do see some cleanup happening in delayed fashion under finalization. I have not yet had problems with that.

Note that ZnClient under #beOneShot behavior asks the other side to close as well.

Socket>> closeAndDestroy does a wait of up to 20 seconds...

Anyway, it is a good issue, but we should understand it better first I think.

Sven

--
Sven Van Caekenberghe
http://stfx.eu
Smalltalk is the Red Pill

Sven Van Caekenberghe wrote

Paul,

On 15 May 2012, at 20:50, Paul DeBruicker wrote:

> I ran into an out-of-external semaphores error when testing the Tropo API. It uses ZnEasy to post data to their api servers.

Any idea at what rate you are posting, just a ballpark figure ?

Maybe 50 POSTs in half an hour.  Just messing with sending and receiving SMS messages.  Using the ZnMultiThreadedServer to handle responses from their API.  

Also - since I sent this I changed from ZnEasy to just ZnClient.  Maybe keeping the the client around in the instance will cut down on creating/keeping semaphores.  But I still think its a relevant topic.  

Sven Van Caekenberghe wrote

I am assuming you are on Pharo 1.3 or 1.4. But since you are talking about ZdcAbstractSocketStream you have Zodiac loaded as well. Are you using HTTPS then ? Because #close: does not exist on regular SocketStreams.

Pharo 1.3 right now.  Zodiac is loaded in the image but this is just an HTTP post for testing purposes to 'http://api.tropo.com' .  I'm making another pass over the Stripe API later today and will be using Zodiac/SqueakSSL for HTTPS for that.    For production I plan to route all SSL through nginx/apache.  

Sven Van Caekenberghe wrote

If you are talking about regular HTTP, then you must have loaded the somewhat older/experimental ZdcNetworkingUtils, and are not using regular SocketStreams but ZdcOptimizedSocketStreams. Which is brave since this is something I am not doing myself ;-)

I have these versions loaded: Zodiac-Core-SvenVanCaekenberghe.17 & Zinc-HTTP-SvenVanCaekenberghe.275

As far as I know I've done nothing to actively choose anything other than the defaults.  

Sven Van Caekenberghe wrote

Anyway, the close and/or destroy discussion puzzles me as well. When running all unit Zn tests I sometimes do see some cleanup happening in delayed fashion under finalization. I have not yet had problems with that.

Note that ZnClient under #beOneShot behavior asks the other side to close as well.

Socket>> closeAndDestroy does a wait of up to 20 seconds...

Socket>>#closeAndDestroy only waits if its still connected. Otherwise it 'detroys' straight away and gets rid of the semaphore.  From what I can see 'close' doesn't dump the semaphore.  

On 15 May 2012, at 22:48, Paul DeBruicker wrote:

That is not fast enough to cause trouble I would think, maybe something else prevents the sockets from closing.

> Using the ZnMultiThreadedServer to handle responses from their API.

OK, these could also keep on lingering too long (nowadays its is the managing server that is the default, check the collection inside).

>  Also - since I sent this I changed from ZnEasy to just ZnClient.  Maybe
> keeping the the client around in the instance will cut down on
> creating/keeping semaphores.  But I still think its a relevant topic.  

ZnEasy uses ZnClient internally, so it comes down to the same thing, but yes reusing a client to access a server can be a good idea, mulitthreaded warnings apply.

Stripe is interesting, I want to hear how it goes.

What does ZnNetworkingUtils default socketStreamClass return ?

Yes, but potentially it will wait, and as a busy server trying to cleanup/conserve resources, you don't want that.

Sven

Sven - Sorry for the delay in getting back to you about this and I know Igor has submitted a new method for automatically expanding the semaphore table so the issue may be moot but I thought I'd respond below anyway.  

Sven Van Caekenberghe wrote

On 15 May 2012, at 22:48, Paul DeBruicker wrote:

> Sven Van Caekenberghe wrote
>>
>> Paul,
>>
>> On 15 May 2012, at 20:50, Paul DeBruicker wrote:
>>
>>> I ran into an out-of-external semaphores error when testing the Tropo
>>> API. It uses ZnEasy to post data to their api servers.
>>
>> Any idea at what rate you are posting, just a ballpark figure ?
>
> Maybe 50 POSTs in half an hour.  Just messing with sending and receiving SMS
> messages.  

That is not fast enough to cause trouble I would think, maybe something else prevents the sockets from closing.

I was also using the ZnStaticFileServerDelegate so there were also instances of FileStream about.  In fact at this time I'm guessing that that may be the culprit because it happened again yesterday while not using any of the ZnClient functionality. When the error was raised I did inspect the ExternalSemaphoreTable unprotectedExternalObjects and a significant portion of the slots in the array were nil.   Maybe I'm looking at the wrong thing.  

Sven Van Caekenberghe wrote

Stripe is interesting, I want to hear how it goes.

The package works fine and I added the ability to receive and log events from them using their WebHooks functionality all using Zinc/Zodiac + SSLPlugin. So I think it implements most of their  API.   Also can run it behind nginx as needed.  

Sven Van Caekenberghe wrote

> Sven Van Caekenberghe wrote
>>
>> If you are talking about regular HTTP, then you must have loaded the
>> somewhat older/experimental ZdcNetworkingUtils, and are not using regular
>> SocketStreams but ZdcOptimizedSocketStreams. Which is brave since this is
>> something I am not doing myself ;-)
>
> I have these versions loaded: Zodiac-Core-SvenVanCaekenberghe.17 &
> Zinc-HTTP-SvenVanCaekenberghe.275
>
> As far as I know I've done nothing to actively choose anything other than the defaults.  

What does ZnNetworkingUtils default socketStreamClass return ?

 ZdcSocketStream


Thanks again and I really should just make the external table bigger than what it is now so this issue doesn't come up as often.  

Paul