leaking semaphores

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

leaking semaphores

mmimica
Hi!

I am continuing my search of why does my remote headless image run out of semaphores eventually.

I have found the culprit this time. It is most definitively related to RFBServer. I am using version 38 from http://source.lukas-renggli.ch/unsorted.
Every time I disconnect from the image with my VNC client (I click the disconnect icon) the count of leaked semaphores increases by 3. Since there is 3 semaphores per socket, I guess this is related to sockets.

Now what do I do? What do I check?

I am running the a Pharo image and Cog I got from CI server yesterday.

This is the script I use to count leaked semaphores:

| semaphores arr res |
arr := ExternalSemaphoreTable unprotectedExternalObjects.
semaphores := arr reject: #isNil.

res := semaphores collect: [:sema | sema pointersTo reject: [:ptr | ptr == arr or: [ptr == semaphores ] ] ].
res := res select: [ :each | each isEmpty ].
res size.
 

--
Milan Mimica
http://sparklet.sf.net
Reply | Threaded
Open this post in threaded view
|

Re: leaking semaphores

Igor Stasenko
I find it funny, that your "interference" is actually root of all problems :)

I don't know much about RFBServer , the only thing i know is that it is quite
old (not in terms that it rusty, but in terms of it existence).
Can't say what happens, but probably on terminating connection it does some
nasty/incompatible things.

On 29 February 2012 23:08, Milan Mimica <[hidden email]> wrote:

> Hi!
>
> I am continuing my search of why does my remote headless image run out of
> semaphores eventually.
>
> I have found the culprit this time. It is most definitively related to
> RFBServer. I am using version 38
> from http://source.lukas-renggli.ch/unsorted.
> Every time I disconnect from the image with my VNC client (I click the
> disconnect icon) the count of leaked semaphores increases by 3. Since there
> is 3 semaphores per socket, I guess this is related to sockets.
>
> Now what do I do? What do I check?
>
> I am running the a Pharo image and Cog I got from CI server yesterday.
>
> This is the script I use to count leaked semaphores:
>
> | semaphores arr res |
> arr := ExternalSemaphoreTable unprotectedExternalObjects.
> semaphores := arr reject: #isNil.
>
> res := semaphores collect: [:sema | sema pointersTo reject: [:ptr | ptr ==
> arr or: [ptr == semaphores ] ] ].
> res := res select: [ :each | each isEmpty ].
> res size.
>
>
> --
> Milan Mimica
> http://sparklet.sf.net



--
Best regards,
Igor Stasenko.

Reply | Threaded
Open this post in threaded view
|

Re: leaking semaphores

Henrik Sperre Johansen
In reply to this post by mmimica

On Feb 29, 2012, at 10:08 PM, Milan Mimica wrote:

Now what do I do? What do I check?

Modify RFBSocket class >> #acceptFrom: to read:
^RFBSocketInstances add: (super acceptFrom: aSocket)

super new works for Socket, since it avoids calling the actual #new implementation in Socket, same is not true its subclasses call super…

Note to tracker:
Might be a good idea to "bulletproof" Socket>>acceptFrom:, by either
1) having it unregister existing handle/semaphores (The "I hope you weren't using that!" approach), or 
2) raise an error in that case. (The "Surely you jest, madame?" approach)

As someone rarely using sockets, I would prefer 2 but have no strong opinion on the matter, what do the rest of you think?

Cheers,
Henry

Reply | Threaded
Open this post in threaded view
|

Re: leaking semaphores

Henrik Sperre Johansen
In reply to this post by mmimica
On Mar 1, 2012, at 2:11 AM, Henrik Johansen wrote:

Note to tracker:


Cheers,
Henry
Reply | Threaded
Open this post in threaded view
|

Re: leaking semaphores

Guido Stepken

Sooner or later Pharo Smalltalkers will have to read into this paper:

http://www.lemis.com/grog/SMPng/USENIX/paper.ascii

Why? Pharo is a OS on top of another OS. The same principles apply here. Without considering these principles or algorithms (no more or as little semaphores as possible, replaced by wait_queues with adjustable priorities), Pharo will suffer those problems.

tnx 4 understanding, Guido Stepken

Am 01.03.2012 02:23 schrieb "Henrik Johansen" <[hidden email]>:
On Mar 1, 2012, at 2:11 AM, Henrik Johansen wrote:

Note to tracker:


Cheers,
Henry
Reply | Threaded
Open this post in threaded view
|

Re: leaking semaphores

Henrik Sperre Johansen

On Mar 1, 2012, at 8:38 AM, Guido Stepken wrote:

Sooner or later Pharo Smalltalkers will have to read into this paper:

http://www.lemis.com/grog/SMPng/USENIX/paper.ascii

Why? Pharo is a OS on top of another OS. The same principles apply here. Without considering these principles or algorithms (no more or as little semaphores as possible, replaced by wait_queues with adjustable priorities), Pharo will suffer those problems.

tnx 4 understanding, Guido Stepken


I could enumerate the ways you are off target in this context, but that would be wasting my time.

thx 4 the laugh, Henry

p.s. The paper itself is indeed quite good and worth reading :D
Reply | Threaded
Open this post in threaded view
|

Re: leaking semaphores

Guido Stepken

As long as GC, recompile, (see blocked UI during updates), ffi database drivers completely block Smalltalk execution or do not access external databases smoothly in parallel (also see async I/O, MP) i see only one laugh here - that's indeed you!

Have fun!

Guido Stepken

Am 01.03.2012 09:43 schrieb "Henrik Johansen" <[hidden email]>:

On Mar 1, 2012, at 8:38 AM, Guido Stepken wrote:

Sooner or later Pharo Smalltalkers will have to read into this paper:

http://www.lemis.com/grog/SMPng/USENIX/paper.ascii

Why? Pharo is a OS on top of another OS. The same principles apply here. Without considering these principles or algorithms (no more or as little semaphores as possible, replaced by wait_queues with adjustable priorities), Pharo will suffer those problems.

tnx 4 understanding, Guido Stepken


I could enumerate the ways you are off target in this context, but that would be wasting my time.

thx 4 the laugh, Henry

p.s. The paper itself is indeed quite good and worth reading :D
Reply | Threaded
Open this post in threaded view
|

Re: leaking semaphores

Henrik Sperre Johansen

On Mar 1, 2012, at 10:03 AM, Guido Stepken wrote:

As long as GC, recompile, (see blocked UI during updates), ffi database drivers completely block Smalltalk execution or do not access external databases smoothly in parallel (also see async I/O, MP) i see only one laugh here - that's indeed you!

Have fun!

Guido Stepken



Aaah, the good ol' "If you can't hit with a rifle, switch to a shotgun" approach, haven't seen that in a while.

Have fun!
Henry
Reply | Threaded
Open this post in threaded view
|

Re: leaking semaphores

Guido Stepken

No. I just tried to teach Pharo to do several things in parallel: Seaching stings in several 1 GB files, opening and reading several databases at once in parallel, handing results over sockets to several clients. Try it yourself.
Whatever you do with Pharo, you won't get any smooth IO, even if you use COGMT. Smalltalk code is not yet adapted, it seems. Same with eventhandlers. Try editing while copying and filtering a file in background. Happy waiting!

I had much success with Smalltalk MT, LUAJit, medium success with node.js  ...

Don't get me wrong, but business develops to realtime filtering of concurrent information streams coming from different sources (twitter, facebook, mail, irc, p2p, databases, homepages, rss, XML, Soap ) and that intelligently presented in realtime on my Desktop.

We did intensive testing of several interpreters.

I personally really prefer Pharo, because it offers a outstanding, very complex construction kit for doing marvellous things, beginning from "abstract connectors" that combine filters with editors with gui, web, streams, databases, over to very good IDE up to certain (decreasing) compatibility to squeak and its mass of "toy apps".

I think, that, when a deeper understanding of how to acheive parallelism, realtime, concurrency comes to Smalltalk programmers (not just starting coroutines and hoping, the kernel gets it right, claiming everything is multithreaded), Pharo very soon could be business ready! :-)

And it is good to understand, *why* Wikipedia now uses LUAJit (a highly dynamic language, like Smalltalk), Blizzard (World of Warcraft), Steam (realtime control of complex game worlds) use LUAJit, EVE uses Stackless Python with continuations. Smalltalk would be very well suited, but its implementations are missing important features, realtime qualities, real concurrency, not just one at the surface, claiming apps being "multithreaded".

Seaside, e.g. immediately drops to under 1/second, at even 10 simulaneous clients, when data comes from SQL or even OODB, means case of cache misses.
Pharo becomes unresponsive.

For me, good architectures can be recognized, when they smoothly go on serving, even at 50 times overload, still keeping small memory footprint. Compare to Linux at high loads (sshd, telnetd not answering) and FreeBSD (you can still login, even if 50.000 clients connected).

You might still laugh at me... i don't.

Have fun, Guido Stepken

Am 01.03.2012 10:29 schrieb "Henrik Johansen" <[hidden email]>:

On Mar 1, 2012, at 10:03 AM, Guido Stepken wrote:

As long as GC, recompile, (see blocked UI during updates), ffi database drivers completely block Smalltalk execution or do not access external databases smoothly in parallel (also see async I/O, MP) i see only one laugh here - that's indeed you!

Have fun!

Guido Stepken



Aaah, the good ol' "If you can't hit with a rifle, switch to a shotgun" approach, haven't seen that in a while.

Have fun!
Henry
Reply | Threaded
Open this post in threaded view
|

Re: leaking semaphores

Igor Stasenko
On 1 March 2012 11:41, Guido Stepken <[hidden email]> wrote:

> No. I just tried to teach Pharo to do several things in parallel: Seaching
> stings in several 1 GB files, opening and reading several databases at once
> in parallel, handing results over sockets to several clients. Try it
> yourself.
> Whatever you do with Pharo, you won't get any smooth IO, even if you use
> COGMT. Smalltalk code is not yet adapted, it seems. Same with eventhandlers.
> Try editing while copying and filtering a file in background. Happy waiting!
>
> I had much success with Smalltalk MT, LUAJit, medium success with node.js
> ...
>
> Don't get me wrong, but business develops to realtime filtering of
> concurrent information streams coming from different sources (twitter,
> facebook, mail, irc, p2p, databases, homepages, rss, XML, Soap ) and that
> intelligently presented in realtime on my Desktop.
>
> We did intensive testing of several interpreters.
>
> I personally really prefer Pharo, because it offers a outstanding, very
> complex construction kit for doing marvellous things, beginning from
> "abstract connectors" that combine filters with editors with gui, web,
> streams, databases, over to very good IDE up to certain (decreasing)
> compatibility to squeak and its mass of "toy apps".
>
> I think, that, when a deeper understanding of how to acheive parallelism,
> realtime, concurrency comes to Smalltalk programmers (not just starting
> coroutines and hoping, the kernel gets it right, claiming everything is
> multithreaded), Pharo very soon could be business ready! :-)
>
> And it is good to understand, *why* Wikipedia now uses LUAJit (a highly
> dynamic language, like Smalltalk), Blizzard (World of Warcraft), Steam
> (realtime control of complex game worlds) use LUAJit, EVE uses Stackless
> Python with continuations. Smalltalk would be very well suited, but its
> implementations are missing important features, realtime qualities, real
> concurrency, not just one at the surface, claiming apps being
> "multithreaded".
>
> Seaside, e.g. immediately drops to under 1/second, at even 10 simulaneous
> clients, when data comes from SQL or even OODB, means case of cache misses.
> Pharo becomes unresponsive.
>
> For me, good architectures can be recognized, when they smoothly go on
> serving, even at 50 times overload, still keeping small memory footprint.
> Compare to Linux at high loads (sshd, telnetd not answering) and FreeBSD
> (you can still login, even if 50.000 clients connected).
>
> You might still laugh at me... i don't.
>

Finally something worth replying.
Back in 2007 i did a HydraVM, which were capable of running multiple
interpreters in parallel on separate native threads.
It was a proof-of-concept. Showing that things are doable. What is
hard to do is to make everything in image side to be instantly aware
of new feature
and make use of it. You need to develop tools around it, like
inter-image communication, remote debugging, and last but not least
figuring out how you can spawn a dozens of micro-images
which are not full-blown multimegabyte image as we usually having, but
contain only some small algorithm, which will be running on given
thread.
There's also a RoarVM, which attacking the same problem but at
slightly different angle than HydraVM.

So, indeed, there's nothing to laugh at.
The sad thing is that you have to rewrite a lot in kernel to use new
architecture in vm, and there's no way back: no way how it can be
backwards compatible.
In case if you insist of backward compatibility, the adoption will
never be as good as design based on new concepts.
It also requires a lot of study, in multiple areas as well as
evaluating different approaches.
Another sad thing is that you just trolling, thinking that only you
know how to do things right and treating people here either blind or
stupid idiots.

> Have fun, Guido Stepken
>
> Am 01.03.2012 10:29 schrieb "Henrik Johansen"
> <[hidden email]>:
>
>>
>> On Mar 1, 2012, at 10:03 AM, Guido Stepken wrote:
>>
>> As long as GC, recompile, (see blocked UI during updates), ffi database
>> drivers completely block Smalltalk execution or do not access external
>> databases smoothly in parallel (also see async I/O, MP) i see only one laugh
>> here - that's indeed you!
>>
>> Have fun!
>>
>> Guido Stepken
>>
>>
>>
>> Aaah, the good ol' "If you can't hit with a rifle, switch to a shotgun"
>> approach, haven't seen that in a while.
>>
>> Have fun!
>> Henry



--
Best regards,
Igor Stasenko.

Reply | Threaded
Open this post in threaded view
|

Re: leaking semaphores

Henrik Sperre Johansen
In reply to this post by Guido Stepken

On Mar 1, 2012, at 11:41 AM, Guido Stepken wrote:


You might still laugh at me... i don't.



Not at you, just your way of trying to provoke a response.

"Someone mentioned semaphores! Better link this article and tell them how it should be done!" 
without even checking if it's ontopic to the actual discussion.

If you did, you'd find the use of semaphores in this case actually being close to what the paper describes as a compromise between mutex and spin lock used for interrupt processing.

"What, he didn't bite, FIRE ALL CANNONS WITH EVERYTHING I THINK IS WRONG"

Yeah, as if that makes it more likely I'd be interested in derailing the conversation further.
(Other than with snide replies like this)

TLDR; Bug diagnosed/fix proposed, thanks to Milan for a good reproducible test case. 
Could someone contact Lukas and see if he can update the package?

Cheers,
Henry
Reply | Threaded
Open this post in threaded view
|

Re: leaking semaphores

mmimica
In reply to this post by Henrik Sperre Johansen
On 1 March 2012 02:11, Henrik Johansen <[hidden email]> wrote:

Modify RFBSocket class >> #acceptFrom: to read:
^RFBSocketInstances add: (super acceptFrom: aSocket)

Congratulations! You have just turned Pharo from so-so to awesome.


--
Milan Mimica
http://sparklet.sf.net
Reply | Threaded
Open this post in threaded view
|

Re: leaking semaphores

Igor Stasenko
On 1 March 2012 18:20, Milan Mimica <[hidden email]> wrote:
> On 1 March 2012 02:11, Henrik Johansen <[hidden email]> wrote:
>>
>>
>> Modify RFBSocket class >> #acceptFrom: to read:
>> ^RFBSocketInstances add: (super acceptFrom: aSocket)
>
>
> Congratulations! You have just turned Pharo from so-so to awesome.
>
With one line of code? No, you must be joking! :)



--
Best regards,
Igor Stasenko.

Reply | Threaded
Open this post in threaded view
|

Re: leaking semaphores

Henrik Sperre Johansen

On Mar 1, 2012, at 7:28 PM, Igor Stasenko wrote:

> On 1 March 2012 18:20, Milan Mimica <[hidden email]> wrote:
>> On 1 March 2012 02:11, Henrik Johansen <[hidden email]> wrote:
>>>
>>>
>>> Modify RFBSocket class >> #acceptFrom: to read:
>>> ^RFBSocketInstances add: (super acceptFrom: aSocket)
>>
>>
>> Congratulations! You have just turned Pharo from so-so to awesome.
>>
> With one line of code? No, you must be joking! :)

Not an entire line of code, -1 word :)

Cheers,
Henry