The Trunk: Kernel-mt.1009.mcz

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

The Trunk: Kernel-mt.1009.mcz

commits-2
Marcel Taeumel uploaded a new version of Kernel to project The Trunk:
http://source.squeak.org/trunk/Kernel-mt.1009.mcz

==================== Summary ====================

Name: Kernel-mt.1009
Author: mt
Time: 2 April 2016, 12:23:20.21041 pm
UUID: bd1b45d5-cb4d-41b0-890b-c8f70de70b1c
Ancestors: Kernel-ul.1008

Recent CogVMs do not yield preempted processes. Fixes the behavior where "[ [] repeat] fork" could not be interrupted anymore via CMD+. and hence locked the image.

=============== Diff against Kernel-ul.1008 ===============

Item was changed:
  ----- Method: ProcessorScheduler>>preemptedProcess (in category 'accessing') -----
  preemptedProcess
  "Return the process that the currently active process just preempted."
+
+ self activeProcess priority to: 1 by: -1 do: [:priority |
+ (quiescentProcessLists at: priority) ifNotEmpty: [:list |
+ ^ Smalltalk processPreemptionYields
+ ifTrue: [list last]
+ ifFalse: [list first]]].
+ ^ nil
+
- | list |
- activeProcess priority to: 1 by: -1 do:[:priority|
- list := quiescentProcessLists at: priority.
- list isEmpty ifFalse:[^list last].
- ].
- ^nil
-
  "Processor preemptedProcess"!


Reply | Threaded
Open this post in threaded view
|

Re: The Trunk: Kernel-mt.1009.mcz

Eliot Miranda-2
Hi Marcel,


> On Apr 2, 2016, at 8:50 AM, [hidden email] wrote:
>
> Marcel Taeumel uploaded a new version of Kernel to project The Trunk:
> http://source.squeak.org/trunk/Kernel-mt.1009.mcz
>
> ==================== Summary ====================
>
> Name: Kernel-mt.1009
> Author: mt
> Time: 2 April 2016, 12:23:20.21041 pm
> UUID: bd1b45d5-cb4d-41b0-890b-c8f70de70b1c
> Ancestors: Kernel-ul.1008
>
> Recent CogVMs do not yield preempted processes. Fixes the behavior where "[ [] repeat] fork" could not be interrupted anymore via CMD+. and hence locked the image.

Just to be clear, this is not a VM change.  The VM supports either yielding on preemption (the original behaviour) or preemption not yielding (the new, IMO correct, behaviour), based on a flag stored in the image header.  So your comment should have read

Squeak 5, as of Kernel-eem.??? does not yield preempted processes. Fixes the behavior where "[ [] repeat] fork" could not be interrupted anymore via CMD+. and hence locked the image.

I'll figure out which version when "away from my phone" ;-)

> =============== Diff against Kernel-ul.1008 ===============
>
> Item was changed:
>  ----- Method: ProcessorScheduler>>preemptedProcess (in category 'accessing') -----
>  preemptedProcess
>      "Return the process that the currently active process just preempted."
> +    
> +    self activeProcess priority to: 1 by: -1 do: [:priority |
> +        (quiescentProcessLists at: priority) ifNotEmpty: [:list |
> +            ^ Smalltalk processPreemptionYields
> +                ifTrue: [list last]
> +                ifFalse: [list first]]].
> +    ^ nil
> +    
> -    | list |
> -    activeProcess priority to: 1 by: -1 do:[:priority|
> -        list := quiescentProcessLists at: priority.
> -        list isEmpty ifFalse:[^list last].
> -    ].
> -    ^nil
> -
>      "Processor preemptedProcess"!
>
>

Reply | Threaded
Open this post in threaded view
|

Re: The Trunk: Kernel-mt.1009.mcz

marcel.taeumel
Hi Eliot,

thanks for clarification. Unfortunately, I cannot update the commit message -- can I?

Best,
Marcel
Reply | Threaded
Open this post in threaded view
|

Re: The Trunk: Kernel-mt.1009.mcz

Eliot Miranda-2
Hi Marcel,


> On Apr 2, 2016, at 9:15 AM, marcel.taeumel <[hidden email]> wrote:
>
> Hi Eliot,
>
> thanks for clarification. Unfortunately, I cannot update the commit message
> -- can I?

Not easily.  But it's not important.  I only wanted to point out that it's not a VM change, but a system setting/mode change.

> Best,
> Marcel

_,,,^..^,,,_ (phone)

>
> --
> View this message in context: http://forum.world.st/The-Trunk-Kernel-mt-1009-mcz-tp4887889p4887941.html
> Sent from the Squeak - Dev mailing list archive at Nabble.com.
>

Reply | Threaded
Open this post in threaded view
|

Re: The Trunk: Kernel-mt.1009.mcz

Chris Muller-3
In reply to this post by marcel.taeumel
> thanks for clarification. Unfortunately, I cannot update the commit message
> -- can I?

Yes, you can load the prior version, file-in your change, re-save it
as +1 higher with the correct comment.

Reply | Threaded
Open this post in threaded view
|

Re: The Trunk: Kernel-mt.1009.mcz

Bert Freudenberg
In reply to this post by Eliot Miranda-2
On 02.04.2016, at 18:54, Eliot Miranda <[hidden email]> wrote:
>
> Hi Marcel,
> [...]
> I only wanted to point out that it's not a VM change, but a system setting/mode change.

Can it be queried and changed at runtime?

- Bert -


smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: The Trunk: Kernel-mt.1009.mcz

Bert Freudenberg

> On 03.04.2016, at 15:49, Bert Freudenberg <[hidden email]> wrote:
>
> On 02.04.2016, at 18:54, Eliot Miranda <[hidden email]> wrote:
>>
>> Hi Marcel,
>> [...]
>> I only wanted to point out that it's not a VM change, but a system setting/mode change.
>
> Can it be queried and changed at runtime?

Ah, it can. Just found processPreemptionYields.

- Bert -





smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: The Trunk: Kernel-mt.1009.mcz

Eliot Miranda-2
In reply to this post by Bert Freudenberg
Hi Bert,

> On Apr 3, 2016, at 6:49 AM, Bert Freudenberg <[hidden email]> wrote:
>
>> On 02.04.2016, at 18:54, Eliot Miranda <[hidden email]> wrote:
>>
>> Hi Marcel,
>> [...]
>> I only wanted to point out that it's not a VM change, but a system setting/mode change.
>
> Can it be queried and changed at runtime?

Yes.  There are even a setter and a getter in trunk.  See SmalltalkImage>>#processPreemptionYields[:].

>
> - Bert -

Reply | Threaded
Open this post in threaded view
|

Re: The Trunk: Kernel-mt.1009.mcz

marcel.taeumel
We might expose this as a preference?

Best,
Marcel
Reply | Threaded
Open this post in threaded view
|

Re: The Trunk: Kernel-mt.1009.mcz

Eliot Miranda-2
Hi Marcel,

> On Apr 3, 2016, at 11:33 PM, marcel.taeumel <[hidden email]> wrote:
>
> We might expose this as a preference?

I think it has too deep and pervasive effects to be a preference.  Basically we're still fixing the system after making this change (but the change is important for concurrency).  Making it a preference would cause a lot of uncertainty in debugging.  Please let's keep it the way it is.

>
> Best,
> Marcel
>
>
>
> --
> View this message in context: http://forum.world.st/The-Trunk-Kernel-mt-1009-mcz-tp4887889p4888130.html
> Sent from the Squeak - Dev mailing list archive at Nabble.com.
>

Reply | Threaded
Open this post in threaded view
|

Re: The Trunk: Kernel-mt.1009.mcz

marcel.taeumel
Hi Eliot,

with "Smalltalk processPreemptionYields == false", is there a Smalltalk way to implement ProcessorScheduler >> #yield?

At the moment, it looks like this:

| semaphore |
<primitive: 167>
semaphore := Semaphore new.
[semaphore signal] fork.
semaphore wait.

And another question: How to "yield" a process (i.e. put back in line) that is not currently running but runnable?

Best,
Marcel
Reply | Threaded
Open this post in threaded view
|

Re: The Trunk: Kernel-mt.1009.mcz

Levente Uzonyi
The line

  [] forkAt: Processor activePriority + 1

should interrupt the current process without any Semaphores involved.
Except for the case when the process has the highest possible priority,
but yielding from that process, assuming there's only one, wouldn't make
any sense anyway.

Levente

On Tue, 5 Apr 2016, marcel.taeumel wrote:

> Hi Eliot,
>
> with "Smalltalk processPreemptionYields == false", is there a Smalltalk way
> to implement ProcessorScheduler >> #yield?
>
> At the moment, it looks like this:
>
> | semaphore |
> <primitive: 167>
> semaphore := Semaphore new.
> [semaphore signal] fork.
> semaphore wait.
>
> And another question: How to "yield" a process (i.e. put back in line) that
> is not currently running but runnable?
>
> Best,
> Marcel
>
>
>
> --
> View this message in context: http://forum.world.st/The-Trunk-Kernel-mt-1009-mcz-tp4887889p4888435.html
> Sent from the Squeak - Dev mailing list archive at Nabble.com.
>
>

Reply | Threaded
Open this post in threaded view
|

Re: The Trunk: Kernel-mt.1009.mcz

Eliot Miranda-2
In reply to this post by marcel.taeumel
Hi Marcel,

On Tue, Apr 5, 2016 at 9:27 AM, marcel.taeumel <[hidden email]> wrote:
Hi Eliot,

with "Smalltalk processPreemptionYields == false", is there a Smalltalk way
to implement ProcessorScheduler >> #yield?

At the moment, it looks like this:

| semaphore |
<primitive: 167>
semaphore := Semaphore new.
[semaphore signal] fork.
semaphore wait.

So that works, right?  Remember what the setting does:

If  Smalltalk processPreemptionYields == false then when preempted by a higher-priority process, the current process stays at the head of its run queue.  But when a process waits on a semaphore it is removed from its run queue.  When a process resumes it always gets added to the back of its run queue.  So in the above nothing changes.  Either the primitive puts the calling process to the back of its run queue, or (if the primitive is not implemented):

1. the fork creates a new process, adding it to the end of the active process's run queue
2. the wait in "semaphore wait" removes the active process from its run queue and adds it to the semaphore, so the active process is now not runnable, waiting on the semaphore, which
3. allows the next process in the run queue (if any) to run, and eventually
4. allows the newly forked process to run, and
5. the signal in "semaphore signal" removes the process from the semaphore and adds it to the back of the run queue, so
6. all processes at the same priority level as the process that called yield have run

Note that all the primitive does is to circumvent having to create a semaphore and create and schedule a process, and do a signal and a wait to move a process to the back of its run queue.  This is worth while because most of the time a process's run queue is empty, it being the only runnable process at that priority.

Test it:
| yielded |
yielded := false.
[yielded := true] fork.
yielded


| yielded |
yielded := false.
[yielded := true] fork.
Processor yield.
yielded


OK, so there's clearly a lot of confusion about process preemption yielding.  I hope that the following two examples show what's different between the two regimes.  In the first example we merely create two processes at a lower priority than the active process, at a priority where there are no other processes:

| run priority process1 process2 |
run := true.
"find an empty priority level at a priority lower than the active process"
priority := Processor activePriority - 1.
[(Processor waitingProcessesAt: priority) isEmpty] whileFalse:
[priority := priority - 1].

"Create two processes at that priority"
process1 := [[run] whileTrue] forkAt: priority.
process2 := [[run] whileTrue] forkAt: priority.

"Check that their order in the list is the same as the order in which they were created"
self assert: (Processor waitingProcessesAt: priority) first == process1.
self assert: (Processor waitingProcessesAt: priority) last == process2.

"set the flag to zero and block on a delay, allowing the processes to run to termination"
run := false.
(Delay forMilliseconds: 50) wait.

"Check that they have indeed terminated"
self assert: (Processor waitingProcessesAt: priority) isEmpty


So far so good.  Now let's preempt process1 while it is running, by waiting on a delay without setting run to false:

| run priority process1 process2 |
run := true.
"find an empty priority level at a priority lower than the active process"
priority := Processor activePriority - 1.
[(Processor waitingProcessesAt: priority) isEmpty] whileFalse:
[priority := priority - 1].

"Create two processes at that priority"
process1 := [[run] whileTrue] forkAt: priority.
process2 := [[run] whileTrue] forkAt: priority.

"Check that their order in the list is the same as the order in which they were created"
self assert: (Processor waitingProcessesAt: priority) first == process1.
self assert: (Processor waitingProcessesAt: priority) last == process2.

"Now block on a delay, allowing the first one to run, spinning in its loop.
When the delay ends the current process will preempt process1, because process1 is at a lower priority."
(Delay forMilliseconds: 50) wait.

Smalltalk processPreemptionYields
ifTrue: "If process preemption yields, process1 will get sent to the back of the run queue (which is wrong; no process explicitly yielded)"
[self assert: (Processor waitingProcessesAt: priority) first == process2.
self assert: (Processor waitingProcessesAt: priority) last == process1]
ifFalse: "If process preemption doesn't yield, the processes retain their order (yay! we indeed have cooperative scheduling within a priority)."
[self assert: (Processor waitingProcessesAt: priority) first == process1.
self assert: (Processor waitingProcessesAt: priority) last == process2].

"set the flag to zero and block on a delay, allowing the processes to run to termination"
run := false.
(Delay forMilliseconds: 50) wait.
"Check that they have indeed terminated"
self assert: (Processor waitingProcessesAt: priority) isEmpty

Run the above after trying both "Smalltalk processPreemptionYields: false" and "Smalltalk processPreemptionYields: true"

Does the above clarify things?  Nothing changes within priorities.  What the setting controls is what happens when a process is preempted by a higher-priority process.  The old code did an implicit yield of the preempted process, destroying the guarantee of cooperative scheduling within a priority.


I've attached the two examples above in a workspace text (in which "Processor waitingProcessesAt: priority" is cached in a variable).


And another question: How to "yield" a process (i.e. put back in line) that
is not currently running but runnable?

>From a higher priority process one can directly manipulate the lists of lower priority processes.  So, e.g. if in the above we've created process1 and process2 and they're on the list, we can do things like

        (Processor waitingProcessesAt: priority) addLast: (Processor waitingProcessesAt: priority) removeFirst

Again, this is easier if Smalltalk processPreemptionYields is false.  You know that the only thing that can change the order of processes in a list is either cooperative scheduling within those processes or explicit manipulation of the list by higher priority processes.


Do you have a good model of the runnable processes?  The active process is always not on its list because it is running.  There may be runnable, but not running, processes at its priority which will be on the list at the active process's priority.  All other runnable processes are on lists at lower priorities.  Each priority has one list.  Resuming a process (either by resume or by aSemaphore signal) adds a previously not runnable process to the end of the list at its priority (unless it is the highest-priority runnable process, in which case it will become the active process).  Suspending a process, or causing it to wait on a semaphore, removes the process from its list (unless it is the active process, in which case it does not need to be removed).

This is interesting and important.  Having implemented the Blue Book VM and worked on the innards of the scheduler for more years than I'd care to remember I have a very clear picture of the scheduler; it's simple, and well designed.  But Marcel, you're not sure of a number of things, which makes me think you don't have that model in your head, and given that you're an extremely strong Smalltalker, that implies that many other Smalltalk programmers probably don't have that model in their heads either.  What can we do to explain the scheduler more clearly to Smalltalk programmers as opposed to VM implementors?


Best,
Marcel

--
View this message in context: http://forum.world.st/The-Trunk-Kernel-mt-1009-mcz-tp4887889p4888435.html
Sent from the Squeak - Dev mailing list archive at Nabble.com.

_,,,^..^,,,_
best, Eliot



PreemptionTutorialWorkspace.text (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: The Trunk: Kernel-mt.1009.mcz

Eliot Miranda-2
In reply to this post by Levente Uzonyi


On Tue, Apr 5, 2016 at 10:44 AM, Levente Uzonyi <[hidden email]> wrote:
The line

        [] forkAt: Processor activePriority + 1

should interrupt the current process without any Semaphores involved.

Yes, but if Smalltalk processPremeptionYields is false, this will not cause the current process to yield (which is to be desired).
 
Except for the case when the process has the highest possible priority, but yielding from that process, assuming there's only one, wouldn't make any sense anyway.

Levente

Thank you, Levente, that has shown me to write a much simpler example.  

The following doit should answer false:

| yielded |
yielded := false.
[yielded := true] fork.
yielded

But this should answer true:

| yielded |
yielded := false.
[yielded := true] fork.
Processor yield.
yielded

Now try the following with processPreemptionYields either true or false:

| yielded |
yielded := false.
[yielded := true] fork.
[] forkAt: Processor activePriority + 1.
yielded

This will answer what Smalltalk processPreemptionYields adds, i.e.

| yielded |
yielded := false.
[yielded := true] fork.
[] forkAt: Processor activePriority + 1.
self assert: yielded = Smalltalk processPreemptionYields



Extra examples attached...

On Tue, 5 Apr 2016, marcel.taeumel wrote:

Hi Eliot,

with "Smalltalk processPreemptionYields == false", is there a Smalltalk way
to implement ProcessorScheduler >> #yield?

At the moment, it looks like this:

| semaphore |
<primitive: 167>
semaphore := Semaphore new.
[semaphore signal] fork.
semaphore wait.

And another question: How to "yield" a process (i.e. put back in line) that
is not currently running but runnable?

Best,
Marcel

--
View this message in context: http://forum.world.st/The-Trunk-Kernel-mt-1009-mcz-tp4887889p4888435.html
Sent from the Squeak - Dev mailing list archive at Nabble.com.


HTH
_,,,^..^,,,_
best, Eliot



PreemptionTutorialWorkspace.text (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: The Trunk: Kernel-mt.1009.mcz

marcel.taeumel
Thank you, Eliot. I was mistaken that "[ ] fork" (resp. Process >> #resume) adds it to the front, when it actually adds it to the back of the queue.

Best,
Marcel