TCP/IP communication (under VA) is "slow" in some cases

Hi Guys,

In another topic (.Net-Smalltalk IPC), my good friend Marten suggested that "TCP/IP communication (under VA) is "slow" in some cases".  I have another idea as to why this may be the case.

I don't think the problem is with sockets so much as with the VA Smalltalk Delay class code.  Delay, as part of its dispatching function, asks the OS to interrupt it every InterruptPeriod milliseconds.  The default value for InterruptPeriod is 100 milliseconds (1/10th of a second).  This is fine for most things and as far as I know no one ever changes InterruptPeriod but me (if anyone else out there does, please let us know about your experience).

Marten uses Totally Objects SocketSet as do I.  It and I guess most other socket code when asked to read from a socket will look for data.  If the requested amount of data is present, it is returned and things keep running.  If the requested data is not present, a delay is issued to wait a little while to see if the data shows up.  Because of the way delays are handled, the smallest delay is going to be 1/10th of a second.  The requested data could show up 1 millisecond later but the socket won't know about it until it gets back from the delay.  This can result in processing being as slow as 10 messages per second through the socket.

I set InterruptPeriod to 10 milliseconds.  I really only get a delay as short as about 14-15 milliseconds because of the way th VA Smalltalk VM (windows) handles timer interrupt calls to the OS.  I'm not sure what is going on in the VM because we can't see the C code (and I'm not a C programmer anyway) but the Squeak VM can get down to 1 millisecond delays.  So, I think the problem is with the VA VM and not windows.

I have experienced this slow through put first hand.  When going from VA version 5.5.2 to 8.0.3, one of my servers that was doing okay with the 100 millisecond delay experienced a drop in through put.  If anything I would have expected an increase in through put as there were changes in v8.0.3 that should have been an improvement.  I concluded that the code that was handling the data obtained from the socket was now coming back for more "sooner" than before and data wasn't available where as before things took a little longer and data was available.  Now, with the slighter faster v8.0.3 system, a delay was required to wait for more data.  Things went from a state where data showing up and being expected were in-sync to one where 100 millisecond delays were being thrown all the time and through put suffered.

I would like to see a few changes in this area.

1) Changing the VM to allow/handle/whatever 1 millisecond delays would be good.

2) Changing the delay code to calculate the interrupt time based upon the current time and when the next (smallest) delay should end.  This could be calculated to not be larger than 100 milliseconds or whatever InterruptPeriod is set to.

These changes would allow developers to issue delays smaller than 100 milliseconds (as small as 1ms) without changing InterruptPeriod and without having the OS interrupt the VM more often that needed.

Note: I have seen places in the base code where delays of 50 ms are issued.  Clearly someone wanted the short delay but they don't get it.

Lou

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/va-smalltalk.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Bob,

The default configurations (as #serverTransportConfiguration in SstHttpCommunications) has Nagle's algorithm disabled. Fancy to share the other tweaks you made? ;-)

Cheers,
Adriaan.

--

You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/va-smalltalk.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Marten,

Ok, when we are at this topic. The Delay has never worked for me over the last 15 years.

When we use

Delay forMilliseconds: msMilliseconds

we actually get :

Delay forMilliseconds: (msMilliseconds + interruptPeriod).

As Lou mentioned: the interruptPeriod is per default 100ms. Therefore if you write:

Delay forMilliseconds: 100

you get

Delay forMilliseconds: 200.

This isn't really true, it just seems like it is.

Delays are typically defined as "Delay forMilliseconds: 50" but they can also be defined as "Delay untilMilliseconds: aMillisecondClockValue".  For  "Delay forMilliseconds: 50" a Delay is instantiated and #delayTime: is set to the millisecond value.  For "Delay untilMilliseconds: aMillisecondClockValue"  a Delay is instantiated and #resumptionTime: is set to aMillisecondClockValue.

When #wait is sent, if delayTime is set, resumptionTime is set to the current millisecond clock value (Time millisecondClockValue) plus delayTime.  The delay is added to a list of delayed tasks (sorted by low to high by resumptionTime).  The OS (via the VM) is asked to interrupt us in 100 milliseconds (I'm not 100% sure about this as I can't see what the VM does and different OSs may deal with the interrupt requests in different ways).  The current process (the one associated with this delay) is suspended.  The Smalltalk dispatcher will then dispatch the next ready to run process based upon its priority.  If no process is ready to run the we go to an idle process that relinquishes the CPU to the OS (goes to sleep).

When the interrupt triggers, the list of delayed tasks is checked to see if any of their resumptionTime(s) have past and they are made ready to run.

The problem is as I have described it.  If one does a delay for 20 milliseconds, its  resumptionTime will be set to 20 milliseconds in the future but the OS won't trigger the interrupt until 100 milliseconds in the future, by then the delay is 80 milliseconds past due.

As I also said, for reasons I don't understand, the VM can't seem to get the OS to trigger interrupts more often that every 14-15 milliseconds.  So, even if the above situation is fixed and I think that is easy to do, we still can't get better resolution than about 15 milliseconds.

Try this:

| delay times |
delay := Delay forMilliseconds: 1.
times := OrderedCollection new.
times add: Time millisecondClockValue.
100 timesRepeat: [
delay wait.
times add: Time millisecondClockValue.
].
times inspect.
times last - times first.

Then do this:

Delay interruptPeriod: 10.

and repeat above code.  I get 10000 and 1560 respectively.  And when:

Delay interruptPeriod: 1.

I still get 1560.

So we can't delay for short amounts of time, less than 100 milliseconds, without changing the delay interrupt period and even then we can't get below 15 milliseconds.

For low volume - high CPU tasks this isn't very bad but for high volume - low CPU tasks it can be a killer.

For one case, I'm going to simplify it here, I had two servers (NT services) where server A send server B work via TCP/IP sockets.  Server A would wait until server B sent back the result before sending server B more work.  This was running on a single CPU VMWare VM windows server (because it is often recommended that only one CPU be defined for a VM as it wouldn't be dispatched until both CPUs are available).  This configuration forced server B to wait on a read on the socket.  Server A might return more work in say 5 milliseconds but server B wouldn't know about the data for another 95 milliseconds (assuming I hadn't changed the interrupt period).  When I changed the interrupt period things got better but still can't do better than the 15 millisecond rate.

Lou

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/va-smalltalk.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Bob,

 

What's your OS?  If it's Windows, that makes sense, as the system timer for, e.g. Windows 7 is 15.6 mS.  Older versions of Windows are worse.  You can actually get extremely accurate timers on Windows by using performance counters, but unless they use that technique in the VM, you won't get that accuracy by using Delay.

I'm using windows 7.  Similar tests with Squeak on windows 7 get delays down to 1 millisecond.

But the more interesting question is why socket-related code has explicit delays.  That sounds strange.

I'm using Totally Objects SocketSet http://www.totallyobjects.com/Sockset.htm.  I have had to learn way more than I ever wanted to about communications/TCP/IP/sockets than I ever wanted to and SocketSet eased that pain somewhat (probably a lot and I highly recommend it).  It would do a delay (all be it with a bug or two that I fixed) when needing to wait for data.  I guess it could have been built to trigger an event when the data showed up but that's not what it does.

I'm not sure event driven sockets would solve the problem because sooner or later the program may relinquish the CPU to the OS and unless data showing up for a socket will wake up the program it would still wait until the 100 milliseconds is up.

Lou

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/va-smalltalk.
For more options, visit https://groups.google.com/groups/opt_out.