Issue 5965 in pharo: Can't allocate semaphores? VirtualMachine>>maxExternalSemaphores: aSize | Not enough space for external objects, set a larger size at startup!

Status: New
Owner: ----
Labels: Type-Bug

New issue 5965 by [hidden email]: Can't allocate semaphores?  
VirtualMachine>>maxExternalSemaphores: aSize | Not enough space for  
external objects, set a larger size at startup!
http://code.google.com/p/pharo/issues/detail?id=5965

Steps to reproduce:
1. Using a web app for a while brings a sockets problem (basically it can't  
open a new one)
2. you'll see that after a while you get an exception like this: Not enough  
space for external objects, set a larger size at startup!
3. in VirtualMachine>>maxExternalSemaphores: aSize


it was observed with this VM for OSX:

Smalltalk vm version
'CoInterpreter VMMaker-oscog-EstebanLorenzano.155 uuid:  
c53d1f34-009e-425c-9f88-398f8d39c5e8 Apr 15 2012,  
StackToRegisterMappingCogit VMMaker-oscog-EstebanLorenzano.155 uuid:  
c53d1f34-009e-425c-9f88-398f8d39c5e8 Apr 15 2012,  
https://git.gitorious.org/cogvm/blessed.git Commit:  
043d256c0d39b78e04db8608c0ebfbf096223a3f Date: Sun Apr 15 14:18:12 2012  
+0200 By: Guille <[hidden email]>'

and this one for linux:
'CoInterpreter VMMaker.oscog-eem.154 uuid:  
5cbb57c7-0a54-4b7e-848c-1f292759f1fa Mar  3 2012,  
StackToRegisterMappingCogit VMMaker.oscog-eem.154 uuid:  
5cbb57c7-0a54-4b7e-848c-1f292759f1fa Mar  3 2012, r2540  
http://www.squeakvm.org/svn/squeak/branches/Cog'


and this one for linux too:
'Croquet Closure Cog VM [CoInterpreter VMMaker.oscog-eem.154] r2540  
http://www.squeakvm.org/svn/squeak/branches/Cog'

Pharo1.4a
Latest update: #14225


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #1 on issue 5965 by [hidden email]: Can't allocate  
semaphores? VirtualMachine>>maxExternalSemaphores: aSize | Not enough space  
for external objects, set a larger size at startup!
http://code.google.com/p/pharo/issues/detail?id=5965

Sebastian,

Did you read all the method comments in VirtualMachine concerning  
maxExternalSemaphores ?
There is a limit and you can change it, if you cross it, you get an  
exception.
These kind of resources are limited anyway.

What would you otherwise suggest ?

The other, more important question is: why are you consuming so many  
resource ? It could be normal due to load, or there could be a leak  
somewhere. The leak can be your fault: not cleaning up, or a system fault.

A simple, reproduceable piece of code creating the problem as you see it  
always helps.

Sven


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Updates:
        Status: Accepted
        Labels: Milestone-1.4 Milestone-2.0 Importance-High

Comment #2 on issue 5965 by [hidden email]: Can't allocate semaphores?  
VirtualMachine>>maxExternalSemaphores: aSize | Not enough space for  
external objects, set a larger size at startup!
http://code.google.com/p/pharo/issues/detail?id=5965

(No comment was entered for this change.)


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #3 on issue 5965 by [hidden email]: Can't allocate semaphores?  
VirtualMachine>>maxExternalSemaphores: aSize | Not enough space for  
external objects, set a larger size at startup!
http://code.google.com/p/pharo/issues/detail?id=5965

This is not a pharo bug.
It's either:
a) Web App not releasing sockets in a timely manner (framework/application  
issue)
b) Limit not set high enough for the usage scenario (user fix -> increase  
the limit as the message suggests)

The alternative (and behaviour in Pharos < 1.4 with Cog VM's) would be  
Sockets silently starting to always wait for the max timeout, since their  
semaphores never get signalled by the VM.

See http://code.google.com/p/pharo/issues/detail?id=4505


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #4 on issue 5965 by [hidden email]: Can't allocate  
semaphores? VirtualMachine>>maxExternalSemaphores: aSize | Not enough space  
for external objects, set a larger size at startup!
http://code.google.com/p/pharo/issues/detail?id=5965

@Sven,

does 260 sounds like a lot of resources to you?

I'll try to get more info on the issue


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #5 on issue 5965 by [hidden email]: Can't allocate  
semaphores? VirtualMachine>>maxExternalSemaphores: aSize | Not enough space  
for external objects, set a larger size at startup!
http://code.google.com/p/pharo/issues/detail?id=5965

Yeah, on my main VM it is about the same (I think 256), a bit low, as each  
socket uses 3 (three!) semaphores...


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #6 on issue 5965 by [hidden email]: Can't allocate semaphores?  
VirtualMachine>>maxExternalSemaphores: aSize | Not enough space for  
external objects, set a larger size at startup!
http://code.google.com/p/pharo/issues/detail?id=5965

Yes, 256 as a default might be a bit... spartan.

One consideration, if the limit were to be increased in the development  
image, short of using the commandline switch (and I haven't tested it would  
work giving a parameter lower than stored in image header) added in Cog  
2499 (http://forum.world.st/new-Cog-VM-available-at-td3891099.html),  
there'd be no going back to a smaller size.

Each entry is an oop of 4 bytes, so the default 256 is 1KiB of the VM's  
memory space.
On modern machines not the end of the world to increase it abit, but could  
be a needless limitation for deployment on limited hardware where Sockets  
will not be used.

Please note it rounds to the next higher power of two, so for example
Smalltalk vm maxSemaphoresSilently: 1024
would increase the limit to 2048.

PS: There's a limit to what is sensible irregardless of memory concerns wrt  
Sockets, when testing large amounts Windows crapped out after a while (1k?  
10k? can't remember...) due to restrictions in internal Windows resources  
used by the socket implementation  :)


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #7 on issue 5965 by [hidden email]: Can't allocate  
semaphores? VirtualMachine>>maxExternalSemaphores: aSize | Not enough space  
for external objects, set a larger size at startup!
http://code.google.com/p/pharo/issues/detail?id=5965

For the record, when it happens this:

Socket allInstances size

evaluates to:

90




_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Updates:
        Cc: [hidden email]

Comment #8 on issue 5965 by [hidden email]: Can't allocate semaphores?  
VirtualMachine>>maxExternalSemaphores: aSize | Not enough space for  
external objects, set a larger size at startup!
http://code.google.com/p/pharo/issues/detail?id=5965

The Eliot's design of sqExternalSemaphores.c assumes that there potentially  
very high number of semaphore signals between interrupt checks. Otherwise i  
don't see
why not keep using Squeak VM's implementation (a bit modified of course,  
but without introducing a limit).
In Cog it keeps separate counters for every external semaphore in private  
table, and hence it needs to know the limit of max entries in semaphore  
table.

While before, in Squeak VM, it was just a simple list of 512 entries, and  
upon signal
it was just appended new item to the list (with semaphore number to signal).
Upon interrupt check this table were flushed.
Frankly, i cannot see how anything in our VM&plugins can produce so much  
signals between two interrupt checks (which happen every millisecond IIRC),  
and i cannot foresee such high rate in future (the CPU clock rate won't  
grow in the future, and for manycore systems, which can produce so many  
signals, we will need completely different VM(s) anyways ).



_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #9 on issue 5965 by [hidden email]: Can't allocate  
semaphores? VirtualMachine>>maxExternalSemaphores: aSize | Not enough space  
for external objects, set a larger size at startup!
http://code.google.com/p/pharo/issues/detail?id=5965

The issue is not a high rate of signals.  The issue is thread-safety.  
Signals can be delivered from other threads.  My design is thread-safe and  
will not lose a signal if delivered form another thrwad.  The Squeak VM  
implementation is *not* thread0safe.  It can lose signals.  Lose one signal  
and your application may become unresponsive.


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #10 on issue 5965 by [hidden email]: Can't allocate semaphores?  
VirtualMachine>>maxExternalSemaphores: aSize | Not enough space for  
external objects, set a larger size at startup!
http://code.google.com/p/pharo/issues/detail?id=5965

Thanks for clarification, Eliot.
Can i rewrite it to be thread-safe and have no hard limit on registered  
semaphores?


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #11 on issue 5965 by [hidden email]: Can't allocate  
semaphores? VirtualMachine>>maxExternalSemaphores: aSize | Not enough space  
for external objects, set a larger size at startup!
http://code.google.com/p/pharo/issues/detail?id=5965

Yes.  That would be ideal.  That was the step I was too lazy to take.  
Writing the table management so that it can grow is difficult.  It is a  
lock-free data structure.  Locking to allow growing is not an option.


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #12 on issue 5965 by [hidden email]: Can't allocate semaphores?  
VirtualMachine>>maxExternalSemaphores: aSize | Not enough space for  
external objects, set a larger size at startup!
http://code.google.com/p/pharo/issues/detail?id=5965

Do you think that signals table needs to be dynamically allocated?
A static limit , like in squeak before - 512 entries looks fine as to me.
It means  max 512 signals between two interrupt checks.
Under what conditions we can exceed such already big number?


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #13 on issue 5965 by [hidden email]: Can't allocate  
semaphores? VirtualMachine>>maxExternalSemaphores: aSize | Not enough space  
for external objects, set a larger size at startup!
http://code.google.com/p/pharo/issues/detail?id=5965

That's where we came in.  My compromise was to allow resizing the table at  
start-up or with a VM parameter   I thought this whole thread was about  
avoiding that and allowing it to dynamically resize.  In which case one has  
to bite the bullet and write a lock-free growth mechanism.


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #14 on issue 5965 by [hidden email]: Can't allocate semaphores?  
VirtualMachine>>maxExternalSemaphores: aSize | Not enough space for  
external objects, set a larger size at startup!
http://code.google.com/p/pharo/issues/detail?id=5965

Oh.. by saying rewrite, i dind't meant i would use any dynamic relocation :)
Just a static-sized queue with 512 entries, which filled up by signalers  
and flushed by interrupt checker.
Why we need sophisticated growth dances, if we can avoid it?
Especially, since you introduced CAS, the implementation is straightforward.


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #15 on issue 5965 by [hidden email]: Can't allocate semaphores?  
VirtualMachine>>maxExternalSemaphores: aSize | Not enough space for  
external objects, set a larger size at startup!
http://code.google.com/p/pharo/issues/detail?id=5965

Here the code.
Compiles & runs just fine.

You can try something like:

coll := (1 to: 10000) collect: [:e | Semaphore new ].
coll do: [:each |
        Smalltalk registerExternalObject: each
        ].

"do some networking.. open/close files here etc"

coll do: [:each |
        Smalltalk unregisterExternalObject: each
        ].


Eliot, can you please review the code?
And do you have some tests to test this stuff?
If anyone else wanna try/test it, let me know your platform, i can make a  
binary.

Attachments:
        sqExternalSemaphores.c  7.7 KB


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #16 on issue 5965 by [hidden email]: Can't allocate semaphores?  
VirtualMachine>>maxExternalSemaphores: aSize | Not enough space for  
external objects, set a larger size at startup!
http://code.google.com/p/pharo/issues/detail?id=5965

here is more polished version, this one is more nicer when using dynamic  
queue max size.
for static max queue size, nothing has changed.

Attachments:
        sqExternalSemaphores.c  7.9 KB


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Updates:
        Labels: -Milestone-1.4

Comment #17 on issue 5965 by [hidden email]: Can't allocate  
semaphores? VirtualMachine>>maxExternalSemaphores: aSize | Not enough space  
for external objects, set a larger size at startup!
http://code.google.com/p/pharo/issues/detail?id=5965

Should we move this to the Cog bug tracker?


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #18 on issue 5965 by [hidden email]: Can't allocate  
semaphores? VirtualMachine>>maxExternalSemaphores: aSize | Not enough space  
for external objects, set a larger size at startup!
http://code.google.com/p/pharo/issues/detail?id=5965

Move issue to Cog tracker


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Updates:
        Status: Moved

Comment #19 on issue 5965 by [hidden email]: Can't allocate  
semaphores? VirtualMachine>>maxExternalSemaphores: aSize | Not enough space  
for external objects, set a larger size at startup!
http://code.google.com/p/pharo/issues/detail?id=5965

Moved to: Issue cog:102


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker