Tasks of the heartbeat code

Dear Eliot,

I tried to understand the basic load of the VM and I have two questions for the heartbeat
code. Is it right that there are two concerns for the code?


1.) Call forceInterruptCheckFromHeartbeat periodically

2.) Update the clock as seen by the image


1st is useful to interrupt Processes like […] repeat with a pre-emptive scheduler? On
a different note do backwards jumps generate interrupt checks too?


2nd is this an optimization to not have to generate the clock when the primitive is being
used?

thanks
        holger

Hi Holger,

Yes

> 2nd is this an optimization to not have to generate the clock when the primitive is being
> used?

Nearly.

You are correct for #1.  The main use is 1.  To cause a breakout from machine code at regular intervals (default every 2ms).  Backward jumps also perform the test.  Using counters instead does not work well; counters are expensive to update in every send and the frequency of breakout varies wildly depending on whether one is calling code containing slow primitives (eg LargeIntegerArithmetic) or not.

For 2, getting the heartbeat to update the clock is for delay expiry.  Fetching the time is slow, and delays need only have about a millisecond resolution.  So having the heartbeat update the clock saves time when seeing if the active delay has fired.  At first it was used also to update the time seen through the time primitives but Dave had me change it.  This gives you much better (microsecond) clock resolution at the cost of a slower primitive.  That's probably a good default choice.  It does lessen the case for doing this at all though.  That's an experiment that I'm interested in the results of.

There is another use of the heartbeat in the threaded FFI.  When a call out releases the VM the heartbeat is used to detect that the VM is free and that another thread should attempt to run it.

Sent from my iPhone
>
> thanks
>    holger

> On 13 Sep 2015, at 19:53, Eliot Miranda <[hidden email]> wrote:
>


> Hi Holger,

Hi!

thank you for the super quick reply!


> You are correct for #1.  The main use is 1.  To cause a breakout from machine code at regular intervals (default every 2ms).  Backward jumps also perform the test.  Using counters instead does not work well; counters are expensive to update in every send and the frequency of breakout varies wildly depending on whether one is calling code containing slow primitives (eg LargeIntegerArithmetic) or not.
>
> For 2, getting the heartbeat to update the clock is for delay expiry.  Fetching the time is slow, and delays need only have about a millisecond resolution.  So having the heartbeat update the clock saves time when seeing if the active delay has fired.  At first it was used also to update the time seen through the time primitives but Dave had me change it.  This gives you much better (microsecond) clock resolution at the cost of a slower primitive.  That's probably a good default choice.  It does lessen the case for doing this at all though. That's an experiment that I'm interested in the results of.


can you think of a strategy to stop the timer when nothing is running inside the
image? I see getNextWakeupUsecs in ioRelinquishProcessorForMicroseconds
but that does’t seem to disable the interval timer? E.g. if I look at the mac vm
and ioRelinquishProcessorForMicroseconds it doesn’t appear to stop the itimer?


I am looking at this from the aioPoll/SIGIO point of view. I was surprised how close
we are to have a system that doesn’t consume CPU when nothing is running. Maybe
there is something I am missing (I don’t know win32) but if the IdleProcess starts to
run the system could:

        * Stop the heartbeat timer
        * Have the mac variant of ioRelinquishProcessorForMicroseconds for unix
          and  get the time the next Delay expires.
        * Set the heartbeat to expire at this time?
        * Yield execution and wait for the SIGIO (or the timer)?
        * On return to the image call aioPoll if the SIGIO occurred?


Is there any general objection to this?



kind regards
        holger

 

> On 13 Sep 2015, at 22:11, John McIntosh <[hidden email]> wrote:

Hi!


> I looked at this six years back and altered the code base to pause the heart beat thread  until we know when we have to wakeup .
>
> Results were inclusive, so more testing is welcome.
>
> (a) how much overhead is there to pause and wakeup the thread? If you sleep for 2 ms, is it worth it?
> (b) because of the morphic polling look we are woken every 16-20 milliseconds.
> (c) There is a race? between what the VM thinks the next wakeup time, and what the ioRelinquishProcessorForMicroseconds is expected.
> Sometimes you are asked to sleep, yet the internal VM timer says you should wakeup to service a task right now or a millisecond in the past. Yet the VM had decided there was no runable tasks just microseconds ago.

Thank you for the insight, so depends on the usecase. I don’t know Morphic/WorldState
enough to judge if the polling/delay loop is necessary. The usecase I look at is for an image
that is mostly waiting for socket i/o (e.g. incoming REST call, TCP command) and the “idle”
load is kind of a issue (running 10 images might already fill one CPU core).

Let’s hope i find enough time over the next couple of weeks to work on it for the Unix VM



> (d) Do mouse/trackpad/keyboard/socket/FD events actually awake the paused program?


For the X11 code SIGIO is enabled for the XDisplay connection so an event would lead to a
wake-up.

 

Hi Holger,

I don't want to put you off the path of improving things, but did you see the socket plugin patch that lets you start an image from a socket event using systemd?

see:
http://forum.world.st/DEV-Systemd-socket-activation-patch-for-the-VM-td4767030.html

Holger Freyther wrote

> On 13 Sep 2015, at 22:11, John McIntosh <[hidden email]> wrote:

Hi!


> I looked at this six years back and altered the code base to pause the heart beat thread  until we know when we have to wakeup .
>
> Results were inclusive, so more testing is welcome.
>
> (a) how much overhead is there to pause and wakeup the thread? If you sleep for 2 ms, is it worth it?
> (b) because of the morphic polling look we are woken every 16-20 milliseconds.
> (c) There is a race? between what the VM thinks the next wakeup time, and what the ioRelinquishProcessorForMicroseconds is expected.
> Sometimes you are asked to sleep, yet the internal VM timer says you should wakeup to service a task right now or a millisecond in the past. Yet the VM had decided there was no runable tasks just microseconds ago.

Thank you for the insight, so depends on the usecase. I don’t know Morphic/WorldState
enough to judge if the polling/delay loop is necessary. The usecase I look at is for an image
that is mostly waiting for socket i/o (e.g. incoming REST call, TCP command) and the “idle”
load is kind of a issue (running 10 images might already fill one CPU core).

Let’s hope i find enough time over the next couple of weeks to work on it for the Unix VM



> (d) Do mouse/trackpad/keyboard/socket/FD events actually awake the paused program?


For the X11 code SIGIO is enabled for the XDisplay connection so an event would lead to a
wake-up.

> On 14 Sep 2015, at 22:43, Eliot Miranda <[hidden email]> wrote:
>
> Hi Holger,


Dear Eliot,

time flies but I finally made a bit of progress.



> You've read what John has said.  I think the easiest thing to do is not to stop and start the heartbeat thread, but merely to set a flag when entering ioRelinquishProcessorForMicroseconds, clearing it on exit, and have the heartbeat test this flag before doing anything that could disturb the VM thread(s).  The overhead of the heartbeat thread is very low.  It doesn't call select or anything like that.  All it does is spin, calling nanosleep, polling the clock and then setting a variable.  For example, avoiding polling the clock when in ioRelinquishProcessorForMicroseconds might save a few cycles.

I have done my current experiments in uild.linux32x86/squeak.cog.v3/build.itimerheartbeat. I don't know if that
is the closest that the pharovm is using but the issue is that it creates work when the system would be idle and
as such "cost" (either your cloud platform charges you for CPU usage, or it takes resources from your cluster,
etc).

I have done it using the X11 plugin:

1.) Don't disable SIGIO for the X11 handle:

@@ -6723,7 +6775,7 @@
   aioHandle(browserPipes[0], npHandler, AIO_RX);
  }
       isConnectedToXServer= 1;
-      aioEnable(stXfd, 0, AIO_EXT);
+      aioEnable(stXfd, 0, 0);
       aioHandle(stXfd, xHandler, AIO_RX);
     }
   return 0;

2.) Hack (needs to use CAS or such) ioProcessEvents. So we remember when a
SIGIO has been received and instead of just trying to poll every nextPollUsecs we
poll if we know it will succeed.

--- ../../src/vm/gcc3x-cointerp.c (revision 3546)
+++ ../../src/vm/gcc3x-cointerp.c (working copy)
@@ -39583,6 +39583,7 @@
 }
 
 
+int doDispatch = 0;
 /* Check for possible interrupts and handle one if necessary.
  Answer if a context switch has occurred. */
 
@@ -39661,15 +39662,13 @@
  switched = 1;
  }
  }
- if (((now = ioUTCMicroseconds())) >= GIV(nextPollUsecs)) {
+
+ if (doDispatch) {
+ doDispatch = 0;
  GIV(statIOProcessEvents) += 1;
  ioProcessEvents();
-
- /* msecs to wait before next call to ioProcessEvents.  Note that strictly
-   speaking we might need to update 'now' at this point since
-   ioProcessEvents could take a very long time on some platforms */
- GIV(nextPollUsecs) = now + 20000;
  }
+ now = ioUTCMicroseconds();
 
  if (GIV(interruptPending)) {
 


--- vm/aio.c (revision 3546)
+++ vm/aio.c (working copy)
@@ -143,10 +143,17 @@
 #endif
 
 /* initialise asynchronous i/o */
+void sigio_signal(int sig)
+{
+  extern void forceInterruptCheck(int); /* not really, but hey */
+  extern int doDispatch;
+  doDispatch = 1;
+  forceInterruptCheck(sig);
+}
 
+
 void aioInit(void)
 {
-  extern void forceInterruptCheck(int); /* not really, but hey */
 
   FD_ZERO(&fdMask);
   FD_ZERO(&rdMask);
@@ -155,7 +162,7 @@
   FD_ZERO(&xdMask);
   maxFd= 0;
   signal(SIGPIPE, SIG_IGN);
-  signal(SIGIO,   forceInterruptCheck);
+  signal(SIGIO,   sigio_signal);
 }


3.) When the relinquish primitive is called

--- vm-display-X11/sqUnixX11.c (revision 3546)
+++ vm-display-X11/sqUnixX11.c (working copy)
@@ -4828,8 +4828,38 @@
 
 static sqInt display_ioRelinquishProcessorForMicroseconds(sqInt microSeconds)
 {
-  aioSleepForUsecs(handleEvents() ? 0 : microSeconds);
+  sigset_t set;
+  sigemptyset(&set);
+
+  extern usqLong getNextWakeupUsecs();
+  if (getNextWakeupUsecs() == 0) {
+   sigsuspend(&set);
+  } else {
+ usqLong nextWakeUp = getNextWakeupUsecs();
+ usqLong now = ioUTCMicroseconds();
+ struct itimerval value;
+
+ sqLong deltaMicro = nextWakeUp - now;
+ if (deltaMicro < 1)
+ deltaMicro = 1;
+ /* don't re-run automatically */
+ value.it_interval.tv_sec = value.it_interval.tv_usec = 0;
+ value.it_value.tv_sec = deltaMicro / 1000000;
+ value.it_value.tv_usec = deltaMicro % 1000000;
+ setitimer(ITIMER_REAL, &value, &value);
+ sigsuspend(&set);
+ value.it_value = value.it_interval;
+ setitimer(ITIMER_REAL, &value, NULL);
+
+ extern int doDispatch;
+ if (doDispatch) {
+ doDispatch = 0;
+ aioPoll(0);
+ }
+  }
+
   return 0;
+#endif
 }


let me follow up with the "result" of my test.

holger

> On 25 Dec 2015, at 21:40, Holger Freyther <[hidden email]> wrote:
>
>
> I have done it using the X11 plugin:
>
> 1.) Don't disable SIGIO for the X11 handle:
> 2.) Hack (needs to use CAS or such) ioProcessEvents. So we remember when a
> SIGIO has been received and instead of just trying to poll every nextPollUsecs we
> poll if we know it will succeed.

I think these two changes make sense by itself. For platforms we have SIGIO we
should enable it unconditionally and rely on it. The question is if we want to use
atomic operations to have "doDispatch" change to 1 when it is '0' and atomically
change it '1' to '0'?


> 3.) When the relinquish primitive is called


> +  extern usqLong getNextWakeupUsecs();
> +  if (getNextWakeupUsecs() == 0) {
> +   sigsuspend(&set);


This _never_ happens. It means that at least one part of the system is driven by
delays. That is a bit unfortunate for network systems (e.g. unless I use RFB I do
not need to have Morph update)


> +  } else {
>
> + setitimer(ITIMER_REAL, &value, NULL);

This is something I don't understand. The interactive responsiveness is really terrible
when I stop the heart beat timer. I assume that primitiveRelinquishProcessor will not
lead to a context switch so it takes longer until a new task will be selected.

Eliot do you have the full picture between VM and Image here?  In theory after doing
1st and 2nd the VM should now more quickly call ioProcessEvents and as it is not
sleeping longer than the time to the next wake up time the interactive performance should
not have changed.


kind regards
        holger

> On 25 Dec 2015, at 22:03, Holger Freyther <[hidden email]> wrote:
>
>
> This _never_ happens. It means that at least one part of the system is driven by
> delays. That is a bit unfortunate for network systems (e.g. unless I use RFB I do
> not need to have Morph update)

        "And signal when the next request is due. We sleep at most 1sec here
        as a soft busy-loop so that we don't accidentally miss signals."

Both Pharo and Squeak seem to do that. This looks a bit odd and I wonder if anybody
remembers if there have been issues with the wakeup in the past?

kind regards and I wish all of you a happy holiday season

        holger



PS: With the Cogit/StackInterpreter (not spur) I seem to end up having the wrong
receiver in top of stack. I don't know if that is a Pharo/Opal issue or something with
the VM itself.

> On 25 Dec 2015, at 21:40, Holger Freyther <[hidden email]> wrote:

Dear Eliot,

I was thinking what to do to make the below race free? I think I will:

* Block the SIGIO
* Memory barrier?
* Check if an event is already queued and then restore mask and return
* Set itimer
* sigsuspend
* Unblock SIGIO
* dispatch

Does this look correct? At the same time I think the per display backend
relinquish code can be replaced with a generic unix one? Maybe even
separating waiting for a timeout and waiting for I/O could be done and
creates a better internal interface?

kind regards
        holger

> On 25 Dec 2015, at 21:40, Holger Freyther <[hidden email]> wrote:

Dear Eliot,



could you please help me with getting the above right and help me to get it into the
VMMaker? I wonder which atomic interface to use between the aioPoll code and
the VMMaker without breaking the Windows/OSX build right now.

1.) I think I don't want the time based polling but only if something has happened

2.) In the SIGIO signal handler I will need to register an event. I'm wondering if I
want to use:
        __sync_fetch_and_add to count the number of SIGIOs not handled?
        otherwise the classic loop with __sync_bool_compare_and_swap?

3.) In the VMMaker and only if the SIGIO interface is present. I think I need to use
sqCompareAndSwap to try to read the number of events?

4.) Which variables (and addresses) can be used and where would I keep them?

kind regards
        holger

> On 01 Jan 2016, at 18:33, Holger Freyther <[hidden email]> wrote:
>


Good Evening Eliot, all,

I had to wait quite some time for my next flight and looked a bit at the more modern builtin atomic support.


I think this still holds true. If we miss a SIGIO event I want it to fail miserably instead of being lucky by a periodic fix-up. :)


> 2.) In the SIGIO signal handler I will need to register an event. I'm wondering if I
> want to use:
> __sync_fetch_and_add to count the number of SIGIOs not handled?
> otherwise the classic loop with __sync_bool_compare_and_swap?

Unless I am completely tired I think we are fine with an atomic store of '1'

void sigIO_handler(int sig)
{
        __atomic_store_n(&doDispatch, 1, __ATOMIC_RELAXED)
}



> 3.) In the VMMaker and only if the SIGIO interface is present. I think I need to use
> sqCompareAndSwap to try to read the number of events?

A simple exchange seems to do?

void check_event()
{
        uint32_t should_dispatch;
        should_dispatch = __atomic_exchange_n(&doDispatch, 0, __ATOMIC_RELAXED);
        if (should_dispatch)
                ioProcessEvents();
}



The integration into VMMaker and display interface is probably going to be more involved. I will try to hack together a "current" idle vm with these primitives.

kind regards
        holger