Problem with fork
Locked
 
|
Hi folks! I am doing some benchmarks and I having problems with fork (I think).
I have this method collect: collect |semaphores tr| semaphores := Array new: 10. tr := ThreadSafeTranscript new. tr open. 1 to: 10 do: [ :index | semaphores at: index put: Semaphore forMutualExclusion ]. 1 to: 10 do: [:i | [ tr show: 'one fork'; cr. (semaphores at: i) signal. ] fork ]. semaphores do: [:each | each wait ]. tr show: 'all forks proccesed'; cr. What I want is that the method collect returns when ALL of the forks created inside, are finished. Obviously in my real code, I don't do a transcript show: but another thing. I would expect something like this: one fork one fork one fork one fork one fork one fork one fork one fork one fork one fork all forks proccesed But the output is: all forks proccesed one fork one fork one fork one forkone fork one fork one fork one forkone fork one fork one fork one forkone fork So, I guess it isn't working :( I don't know too much about forks so any help is welcome! can be a problem with ThreadSafeTranscript ? If so, how could I test if my code is working as expected ? Thanks, Mariano _______________________________________________ Pharo-project mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project |
|
|
Look at forMutualExclusion implementation, and you'll see it's set up
with one excess signal.
So the first thing to wait for it, doesn't have to wait, ie. your semaphores do:[]... falls straight through and 'all forks processed' gets displayed before any of the forks are run. Substitute with new, and you SHOULD get the behaviour you expect. However, show: is NOT threadsafe, it releases access lock between nextPutAll and endEntry, thus you often end up with several threads writing to stream, then endEntry displaying them multiple times If you change nextPutAll: back to show: in the code below, this is apparent (at least it was on my machine...) |semaphores tr| semaphores := Array new: 10. tr := ThreadSafeTranscript new. tr open. 1 to: 10 do: [ :index | semaphores at: index put: Semaphore new ]. 1 to: 10 do: [:i | [ tr nextPutAll: i printString, ' fork '; cr. (semaphores at: i) signal. ] fork ]. semaphores do: [:each | each wait ]. tr show: 'all forks proccesed'; cr. Cheers, Henry On 01.07.2009 16:41, Mariano Martinez Peck wrote: Hi folks! I am doing some benchmarks and I having problems with fork (I think). _______________________________________________ Pharo-project mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project |
|
|
Also, neither is cascading sends in a thread (like the ts show: xxx; cr
in your example)...
For things like that, you can use ThreadSafeTranscript with: [ :aStream | ], or if you prefer Transcript protocols, implement a method doing the same but with self as parameter. - This will have some overhead though, seeing as how you'll usually be reentering the critical section once for each call. Something like: TreadSafeTranscript >> doUninterruptably: aBlock accessSemaphore critical: [aBlock value: self] and you'll write: 1 to: 10 do: [:i | [ tr doUninterruptably: [:transcript | transcript show: i printString, ' fork ' ; cr.]. (semaphores at: i) signal. ] fork. Cheers, Henry On 01.07.2009 19:45, Henrik Sperre Johansen wrote: Look at forMutualExclusion implementation, and you'll see it's set up with one excess signal. _______________________________________________ Pharo-project mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project |
|
|
In reply to this post by Mariano Martinez Peck
Hi all,
this is my first post here, just joined this group so let me do a quick introduction. I have 10+ years of experience doing full-time development with VisualWorks (creating trading platforms for the European energy exchanges). Also I have near-zero experience with using Squeak or how its development process works. I'll slowly cut my teeth, starting by interacting on this mailing list. Multi threading is something I have spent a lot of quality time with, so I want to share some thoughts on the following: |semaphores tr| semaphores := Array new: 10. tr := ThreadSafeTranscript new. tr open. 1 to: 10 do: [ :index | semaphores at: index put: Semaphore new ]. 1 to: 10 do: [:i | [ tr nextPutAll: i printString, ' fork '; cr. (semaphores at: i) signal. ] fork ]. semaphores do: [:each | each wait ]. tr show: 'all forks proccesed'; cr. I have seen this pattern often (allocating a semaphore for every forked process), I usually interpret this as a signal that such code is still in its first 'make it work/make it right' stages. What a lot of people don't realize is that at its heart a semapore is a thread-safe counter/register (and if you look at the hierarchy it is implemented on you wouldn't guess that either, since the hierarchy stresses the implementation part that manages waiting processes rather than the counter aspect). So trying to take the code snippet toward 'make it abstract' territory this could be refactored to lean more on the counter aspect of semaphores and use only a single semaphore: |count sem tr| tr := ThreadSafeTranscript new. tr open. count := 10. sem := Semaphore new. 1 to: count do: [:i | [ tr nextPutAll: (i printString, ' fork\') withCRs. sem signal. ] fork]. count timesRepeat: [sem wait]. tr show: 'all forks proccesed'; cr. Now the above is about as far as you can go with the current Squeak and VisualWorks implementations so you can take it as a simple refactoring advise. However I want to press on a bit more (and go a bit off-topic for this list ;-) because I feel it still has a big problem: we need to maintain a 'count' and pass that between the two loops in the above example. In the current example this is not much of a problem but in more complex applications where the forking is done by yet other forked processes we will need to make 'count' thread-safe as well -- I find this very ugly, because you will need an extra semaphore just to make the original semaphore work as required. Furthermore you cannot add new forked processes once the second loop has started running. So here is an experiment I did a couple of years ago with VisualWorks: I altered the VM (just one line of its source ;-) so it would react properly to semaphores that have negative values in the 'excessSignals' instance variable, and I added a method #unsignal to Semaphore that would decrease the value of that ivar. In my experiments that yielded many opportunities to simplify multiprocessing code (not only for thread synchronization but also for passing around counts in a thread-safe register!). In the above code that would allow us to 'pre-load' the semaphore at the place where the threads are created with as result that the 'count' variable can be removed and the bottom loop can be removed too: |count sem tr| tr := ThreadSafeTranscript new. tr open. sem := Semaphore forMutualExclusion. "We need one excessSignal to balance the #wait below" 1 to: 10 do: [:i | sem unsignal. "outside the forked code" [ tr nextPutAll: (i printString, ' fork\') withCRs. sem signal. "balance the unsignal" ] fork]. sem wait. "no loop, no need to know the count!" tr show: 'all forks proccesed'; cr. Above was just a simple refactoring, but look at how I needed it: |sem tr| tr := ThreadSafeTranscript new. tr open. sem := Semaphore new. "no excessSignal this time" "set up a monitoring system first(!)" [ sem wait. tr show: 'all forks proccesed'; cr ] fork. "then create jobs (in my case I had only a single first job that would recursively create more jobs, not shown here)" 1 to: 10 do: [:i | sem unsignal. [ tr nextPutAll: (i printString, ' fork\') withCRs. sem signal. ] fork]. "Now that we are sure at least one job is entered balance the #wait we started out with" sem signal. Since we elided 'count' I can move the code that relied on it up in front of the thread creation code, I very much like this flavor of decoupling. I guess this illustrates that Semaphore is stuck in the 'make it work/ make it right' phase for thirty years now, and that moving it into 'make it abstract' territory will make lots of hairy multi-threading code much simpler to express... (And for those thinking this through: yes I did implement a thread- safe #add: and #valueWithReset on semaphore too ;-) I hope I didn't bore y'all and stray to far off-topic, but I did want to share this bit of insight I gained by tinkering with the semaphore implementation: semaphores are thread-safe counters at their heart. Cheers, Reinout ------- PS: big congrats with the license cleaning milestone, this is what finally pulled me into this project :-) _______________________________________________ Pharo-project mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project |
|
|
On Jul 3, 2009, at 13:35 , Reinout Heeck wrote:
[...] > PS: big congrats with the license cleaning milestone, this is what > finally pulled me into this project :-) Thanks, and welcome! Adrian ___________________ http://www.adrian-lienhard.ch/ _______________________________________________ Pharo-project mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project |
|
|
In reply to this post by Reinout Heeck
Hi all, this is my first post here, just joined this group so let me do a quick introduction. I have 10+ years of experience doing full-time development with VisualWorks (creating trading platforms for the European energy exchanges). Also I have near-zero experience with using Squeak or how its development process works. I'll slowly cut my teeth, starting by interacting on this mailing list. Multi threading is something I have spent a lot of quality time with, so I want to share some thoughts on the following: |semaphores tr| semaphores := Array new: 10. tr := ThreadSafeTranscript new. tr open. 1 to: 10 do: [:i | [ (semaphores at: i) signal. I have seen this pattern often (allocating a semaphore for every] fork ]. semaphores do: [:each | each wait ]. tr show: 'all forks proccesed'; cr. forked process), I usually interpret this as a signal that such code is still in its first 'make it work/make it right' stages. What a lot of people don't realize is that at its heart a semapore is a thread-safe counter/register (and if you look at the hierarchy it is implemented on you wouldn't guess that either, since the hierarchy stresses the implementation part that manages waiting processes rather than the counter aspect). So trying to take the code snippet toward 'make it abstract' territory this could be refactored to lean more on the counter aspect of semaphores and use only a single semaphore: |count sem tr| tr := ThreadSafeTranscript new. count := 10.tr open. sem := Semaphore new. 1 to: count do: [:i | [ tr nextPutAll: (i printString, ' fork\') withCRs. sem signal. ] fork]. count timesRepeat: [sem wait]. tr show: 'all forks proccesed'; cr. Now the above is about as far as you can go with the current Squeakand VisualWorks implementations so you can take it as a simple refactoring advise. However I want to press on a bit more (and go a bit off-topic for this list ;-) because I feel it still has a big problem: we need to maintain a 'count' and pass that between the two loops in the above example. In the current example this is not much of a problem but in more complex applications where the forking is done by yet other forked processes we will need to make 'count' thread-safe as well -- I find this very ugly, because you will need an extra semaphore just to make the original semaphore work as required. Furthermore you cannot add new forked processes once the second loop has started running. So here is an experiment I did a couple of years ago with VisualWorks: I altered the VM (just one line of its source ;-) so it would react properly to semaphores that have negative values in the 'excessSignals' instance variable, and I added a method #unsignal to Semaphore that would decrease the value of that ivar. In my experiments that yielded many opportunities to simplify multiprocessing code (not only for thread synchronization but also for passing around counts in a thread-safe register!). In the above code that would allow us to 'pre-load' the semaphore at the place where the threads are created with as result that the 'count' variable can be removed and the bottom loop can be removed too: |count sem tr| tr := ThreadSafeTranscript new. sem := Semaphore forMutualExclusion. "We need one excessSignal totr open. balance the #wait below" 1 to: 10 do: [:i | [ tr nextPutAll: (i printString, ' fork\') withCRs. sem signal. "balance the unsignal" ] fork]. sem wait. "no loop, no need to know the count!" tr show: 'all forks proccesed'; cr. Above was just a simple refactoring, but look at how I needed it:|sem tr| tr := ThreadSafeTranscript new. sem := Semaphore new. "no excessSignal this time"tr open. "set up a monitoring system first(!)" [ sem wait. tr show: 'all forks proccesed'; cr ] fork."then create jobs (in my case I had only a single first job that would recursively create more jobs, not shown here)" 1 to: 10 do: [:i | [ tr nextPutAll: (i printString, ' fork\') withCRs. sem signal. ] fork]. "Now that we are sure at least one job is entered balance the #wait we started out with" sem signal. Since we elided 'count' I can move the code that relied on it up in front of the thread creation code, I very much like this flavor of decoupling. I guess this illustrates that Semaphore is stuck in the 'make it work/ make it right' phase for thirty years now, and that moving it into 'make it abstract' territory will make lots of hairy multi-threading code much simpler to express... (And for those thinking this through: yes I did implement a thread- safe #add: and #valueWithReset on semaphore too ;-) I hope I didn't bore y'all and stray to far off-topic, but I did want to share this bit of insight I gained by tinkering with the semaphore implementation: semaphores are thread-safe counters at their heart. Cheers, Reinout ------- PS: big congrats with the license cleaning milestone, this is what finally pulled me into this project :-) _______________________________________________ Pharo-project mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project _______________________________________________ Pharo-project mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project |
|
|
In reply to this post by Reinout Heeck
> Also I have near-zero experience with using Squeak or how its
> development process works. I'll slowly cut my teeth, starting by > interacting on this mailing list. Welcome! Alexandre -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. _______________________________________________ Pharo-project mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project |
|
|
In reply to this post by Reinout Heeck
Reinout: I forgot to thank you for the explanation and the time to do it :)
I was very useful for me. I even change my code to your proposal. Thanks!!! Mariano On Fri, Jul 3, 2009 at 10:35 AM, Reinout Heeck <[hidden email]> wrote: Hi all, _______________________________________________ Pharo-project mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project |
«
Return to Pharo Smalltalk Developers
|
1 view|%1 views
| Free forum by Nabble | Edit this page |