[squeak-dev] Delay time question

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
47 messages Options
123
Reply | Threaded
Open this post in threaded view
|

[squeak-dev] Re: Delay time question

Andreas.Raab
Robert F. Scheer wrote:
> I forgot to be very clear about this in the last post, there is one main
> reason why C would have no trouble keeping these loops all serviced.
>
> The reads would block.  While waiting for the first byte of a packet to
> come in for the next cycle, no processing cycles would be consumed.  No
> delays inside loops would be needed.  

I see. Well, if you'd be willing to throw a couple of native threads at
the problem to service these lines, you could deal with the problem
easily simply by writing the blocking read in C and have that (once it
reads the data) signal a semaphore in Squeak which makes the associated
Squeak process wake up and retrieve and process the data you just read.

Cheers,
   - Andreas

Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Delay time question

Robert F. Scheer-2
In reply to this post by Andreas.Raab

On Thu, 2008-02-28 at 00:18 -0800, Andreas Raab wrote:

> Robert F. Scheer wrote:
> > Picture 5 serial lines.  Each has a packet of a few to a few dozen bytes
> > arriving every 20ms although they are not synchronous.  The data arrives
> > on time within a few microseconds every cycle and it must be captured.  
> >
> > I was trying to use delays of around 15ms to prevent the Squeak i/o
> > processes from continuously tying up compute cycles trying to read the
> > serial lines when no data was expected.  Read a packet, delay 15ms, then
> > go into a tighter loop trying every 1ms until the packet was read.  
> >
> > Now, I know this is a bad scheme.  Even a 1ms programmed delay could
> > become longer than 20ms sometimes.  So let's get rid of all programmed
> > delays inside the serial i/o loops.  
>
> Ah. Now we are talking. What you want is the ability for a serial port
> to issue an interrupt / signal a semaphore so that your waiting process
> gets woken up when something happens. The way to do this is by fixing
> the (very, very naive) serial port primitives to add something that
> allows you to signal a semaphore upon data availability. There are
> examples for this kind of thing in other plugins (like the socket or
> async file plugin).
>
> Also, I vaguely remember that John had built a better serial port plugin
> (don't remember where I saw this but I'm sure he'll chime in) which may
> very well support this out of the box.
>
> > I probably don't understand the primitive serial methods well enough.
> > They seem not to block while awaiting input so that requires
> > continuously looping to read.  Is there a blocking method?
>
> There isn't. Since Squeak doesn't use native threads internally (instead
> it uses so-called "green threads") a blocking read would block the
> entire VM and completely defeat the purpose of a blocking read. That's
> why all such primitives are designed to simply signal a semaphore so
> that the process in squeak just sits there doing something like:
>
>    dataArrived := Semaphore new.
>    self registerSemaphoreForSignallingData: dataArrived.
>    [true] whileTrue:[
>       dataArrived wait.
>       self processData.
>    ].
>
> etc.
>
> Cheers,
>    - Andreas
>
>

Yes, that's what I need.  Unfortunately being a Squeak newbie and having
one month till competition, this looks like an insurmountable gap to me
atm.  But if it existed, I'd be in heaven.

Thanks for the analysis.

- Robert


Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Delay time question

Robert F. Scheer-2
In reply to this post by Andreas.Raab

On Thu, 2008-02-28 at 00:26 -0800, Andreas Raab wrote:

> Robert F. Scheer wrote:
> > I forgot to be very clear about this in the last post, there is one main
> > reason why C would have no trouble keeping these loops all serviced.
> >
> > The reads would block.  While waiting for the first byte of a packet to
> > come in for the next cycle, no processing cycles would be consumed.  No
> > delays inside loops would be needed.  
>
> I see. Well, if you'd be willing to throw a couple of native threads at
> the problem to service these lines, you could deal with the problem
> easily simply by writing the blocking read in C and have that (once it
> reads the data) signal a semaphore in Squeak which makes the associated
> Squeak process wake up and retrieve and process the data you just read.
>
> Cheers,
>    - Andreas
>
>

I can do it this way, sure.  Just have to figure out how to do the
signalling and interprocess communication.  Would you use sockets for
the ipc?


Reply | Threaded
Open this post in threaded view
|

[squeak-dev] Re: Delay time question

Andreas.Raab
Robert F. Scheer wrote:
> I can do it this way, sure.  Just have to figure out how to do the
> signalling and interprocess communication.  Would you use sockets for
> the ipc?

Yes. It's the easiest way to get things going given your time
constraints. If you had more time I would recommend learning enough
slang and vm plugin coding to signal a semaphore directly since this
would take out one additional set of hops in the middle.

Cheers,
   - Andreas


Reply | Threaded
Open this post in threaded view
|

[squeak-dev] Re: Delay time question

johnmci
In reply to this post by Andreas.Raab
Yawn, there has to be a reason why most of the VM maintainers are up  
at midnight, well assuming it's midnight for Igor.
But I'm too tired now, so will respond tomorrow.

On Feb 27, 2008, at 11:58 PM, Andreas Raab wrote:

> John M McIntosh wrote:
>> I can't speak for Windows works but the magic on unix systems  
>> happens in ioRelinquishProcessorForMicroseconds
>
> But none of it contradicts what I said earlier - if platformSleep  
> uses the system's default timer (which it should) then the accuracy  
> of C code will be no better than the accuracy of Squeak. And if it  
> doesn't, well then the code is broken and you should fix it. It sure  
> as hell works on Windows (perhaps I can shame you into fixing it? ;-)
>
> Cheers,
>  - Andreas

--
=
=
=
========================================================================
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
=
=
=
========================================================================



Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Delay time question

Jon Hylands
In reply to this post by Robert F. Scheer-2
On Thu, 28 Feb 2008 02:00:38 -0800, "Robert F. Scheer"
<[hidden email]> wrote:

> I can do it this way, sure.  Just have to figure out how to do the
> signalling and interprocess communication.  Would you use sockets for
> the ipc?

The other thing you can try (to do this all in Squeak), which is what I
normally do, is this:

- use a higher priority process in Squeak (forkAt: Processor
userInterruptPriority) to run your serial loop

        [
                [done] whileFalse: [
                        data := serialPort readBytes.
                        (data notNil and: [data notEmpty])
                                ifTrue: [sharedQueue nextPut: data].
                        (Delay forMilliseconds: 1) wait]] forkAt: Processor
userInterruptPriority named: 'Serial-Reader'.

This way the normal processing loop just waits on a SharedQueue (basically
waiting on a semaphore), and it becomes a blocking read. You can either
feed all of your raw data into one shared queue (appropriately tagged) or
have different ones for different things.

Later,
Jon

--------------------------------------------------------------
   Jon Hylands      [hidden email]      http://www.huv.com/jon

  Project: Micro Raptor (Small Biped Velociraptor Robot)
           http://www.huv.com/blog

Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Delay time question

David T. Lewis
In reply to this post by Robert F. Scheer-2
On Thu, Feb 28, 2008 at 01:53:26AM -0800, Robert F. Scheer wrote:

>
> On Thu, 2008-02-28 at 00:18 -0800, Andreas Raab wrote:
> > Robert F. Scheer wrote:
> >
> > > I probably don't understand the primitive serial methods well enough.
> > > They seem not to block while awaiting input so that requires
> > > continuously looping to read.  Is there a blocking method?
> >
> > There isn't. Since Squeak doesn't use native threads internally (instead
> > it uses so-called "green threads") a blocking read would block the
> > entire VM and completely defeat the purpose of a blocking read. That's
> > why all such primitives are designed to simply signal a semaphore so
> > that the process in squeak just sits there doing something like:
> >
> >    dataArrived := Semaphore new.
> >    self registerSemaphoreForSignallingData: dataArrived.
> >    [true] whileTrue:[
> >       dataArrived wait.
> >       self processData.
> >    ].
>
> Yes, that's what I need.  Unfortunately being a Squeak newbie and having
> one month till competition, this looks like an insurmountable gap to me
> atm.  But if it existed, I'd be in heaven.

Robert,

I don't have time to read catch up with this thread, but FYI the
kind of event driven input that Andreas describes is implemented in
AioPlugin and OSProcess. CommandShell uses this heavily with pipes. I
do not know if it will work with serial lines (/dev/ttyXX) but I can't
think of any reason that it wouldn't.

With that sort of approach, you would use one Squeak process to service
each inbound serial connection, and each process would wait on a semaphore,
so no polling required. The Linux serial drivers handle the IO buffering
etc, so it might work fine.

Sorry I don't have any performance numbers to quote, and I've never
tried it on /dev/ttyXX.

HTH,
Dave
 

Reply | Threaded
Open this post in threaded view
|

[squeak-dev] Re: Delay time question

johnmci
In reply to this post by Andreas.Raab

On Feb 27, 2008, at 11:58 PM, Andreas Raab wrote:

> John M McIntosh wrote:
>> I can't speak for Windows works but the magic on unix systems  
>> happens in ioRelinquishProcessorForMicroseconds
>
> But none of it contradicts what I said earlier - if platformSleep  
> uses the system's default timer (which it should) then the accuracy  
> of C code will be no better than the accuracy of Squeak. And if it  
> doesn't, well then the code is broken and you should fix it. It sure  
> as hell works on Windows (perhaps I can shame you into fixing it? ;-)

Oh, no not me, you see years ago we used Open Transport for tcp/ip  
which was interrupt driven and then the non-portable thread sleep logic.

a SortedCollection(895->1 99->2 5->3 1->4)  (2003) likely data would  
be better today on machines being much faster.

Then one weekend I convince Ian he had to change the unix socket code  
so that it would actually run my Socket SUnits. It helped
that was 9 hours west of him because he kinda zoned out on me after 36  
hours or so of coding. Well plus he discovered a bug in the NetBSD tcp/
ip stack
which didn't help things and we had to document that for the BSD team.

After that I switch to the unix socket code, and later had to migrate  
to the unix aioPoll sleep logic because of Socket accept issues, which  
as you see gives jittery results.  Fixing that requires more work from  
Ian...

--
=
=
=
========================================================================
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
=
=
=
========================================================================



Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Delay time question

johnmci
In reply to this post by Andreas.Raab
I'm curious/lazy here, I stuck this pattern in the Morphic event  
control loop years back to achieve the 60 fps morphic update rate.
However as someone finally discovered last year this had sideeffect,  
on an MC server it would stop responding because it was discovered
the higher priority UI loop would run the calculation of waitTime  
which was <= 0  Then it would not run the Delay which hung any
incoming lower priority MC queries, and as we know, swirling the mouse  
restored sanity.

On a (Delay forMilliseconds: x) where x is <= 0 does it let other  
lower priority processes run for a bit?

On Feb 27, 2008, at 11:38 PM, Andreas Raab wrote:

> I am not sure why your architecture has to change dramatically as a  
> result. If your main problem is to make sure that that a loop is  
> done every 20msecs and you need to adjust for statistical delay  
> variations, you can simply compute the actual time the wait took and  
> do something like:
>
> nextDesiredTick := Time millisecondClockValue + 20.
> [true] whileTrue:[
>   self doControl.
>   "note: for very, very long doControl the waitTime could be negative.
>    you have to decide whether it's better to skip or to run an extra
>    doControl in this case."
>   waitTime := nextDesiredTick - Time millisecondClockValue.
>   (Delay forMilliseconds: waitTime) wait.
>   "Next desired tick is twenty msecs from last one"
>   nextDesiredTick := nextDesiredTick + 20.
> ].

--
=
=
=
========================================================================
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
=
=
=
========================================================================



Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Delay time question

johnmci
In reply to this post by Andreas.Raab

On Feb 28, 2008, at 12:18 AM, Andreas Raab wrote:

> Also, I vaguely remember that John had built a better serial port  
> plugin (don't remember where I saw this but I'm sure he'll chime in)  
> which may very well support this out of the box.

That would be the http://map.squeak.org/package/36c286ee-c0af-4853-98a4-42ea9540f571
It fully exposes all the BSD api for dealing with serial devices. It  
was done on i386 FreeBSD and powerPC os-x
so I don't think there are any endian issues, well porting issues of  
course.

See the SVN squeak macintosh plugins folder for the  
SerialExtendedUnixPlugin xcode project.

On my ftp or idisk in the experimental folder there should be
EnhancedSerialPorts-JMM.11.cs.gz
and
SerialExtendedUnixPlugin.bundle.zip

--
=
=
=
========================================================================
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
=
=
=
========================================================================



Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Delay time question

johnmci
In reply to this post by Jon Hylands
A more extensive, obsolete code sample
http://www.smalltalkconsulting.com/papers/tipsAndThoughts/source/JMM-Network-Server.1.cs

On Feb 28, 2008, at 3:55 AM, Jon Hylands wrote:

> On Thu, 28 Feb 2008 02:00:38 -0800, "Robert F. Scheer"
> <[hidden email]> wrote:
>
>> I can do it this way, sure.  Just have to figure out how to do the
>> signalling and interprocess communication.  Would you use sockets for
>> the ipc?
>
> The other thing you can try (to do this all in Squeak), which is  
> what I
> normally do, is this:
>
> - use a higher priority process in Squeak (forkAt: Processor
> userInterruptPriority) to run your serial loop

--
=
=
=
========================================================================
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
=
=
=
========================================================================



Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Delay time question

Robert F. Scheer-2
In reply to this post by Jon Hylands

On Thu, 2008-02-28 at 06:55 -0500, Jon Hylands wrote:

> The other thing you can try (to do this all in Squeak), which is what I
> normally do, is this:
>
> - use a higher priority process in Squeak (forkAt: Processor
> userInterruptPriority) to run your serial loop
>
> [
> [done] whileFalse: [
> data := serialPort readBytes.
> (data notNil and: [data notEmpty])
> ifTrue: [sharedQueue nextPut: data].
> (Delay forMilliseconds: 1) wait]] forkAt: Processor
> userInterruptPriority named: 'Serial-Reader'.
>
> Later,
> Jon
>

I tested the higher priority fork against a simple user loop and the
forked loop came out the loser.  

        [
        delay := Delay forMilliseconds: 1.
        bag1 := Bag new.
        1000 timesRepeat:[bag1 add: [delay wait] timeToRun].
        ]
        forkAt: Processor userInterruptPriority
       
        bag1 sortedCounts
"a SortedCollection(746->1 86->16 69->14 65->15 32->13 1->2 1->22)"

        delay := Delay forMilliseconds: 1.
        bag2 := Bag new.
        1000 timesRepeat:[bag2 add: [delay wait] timeToRun].

        bag2 sortedCounts
"a SortedCollection(982->1 6->16 5->13 4->14 1->15 1->6 1->7)"
       
I then ran both loops at the same time to see how they'd interact and
got an interesting result.

        delay1 := Delay forMilliseconds: 1.
        bag1 := Bag new.
        delay2 := Delay forMilliseconds: 1.
        bag2 := Bag new.
        [
        1000 timesRepeat:[bag1 add: [delay1 wait] timeToRun].
        ]
        forkAt: Processor userInterruptPriority.
        1000 timesRepeat:[bag2 add: [delay2 wait] timeToRun].

        bag1 sortedCounts "a SortedCollection(914->1 39->16 17->15 14->13
14->14 1->4 1->6)"
       
        bag2 sortedCounts "a SortedCollection(914->1 39->16 17->15 14->13
14->14 1->5 1->6)"

My conclusion is not to use 1ms delays anywhere in the time critical
loops of the robot.  I'm off to look at the interprocess semaphore
methods that David and Andreas have alluded to.  It looks like C serial
handlers are the way to go atm.

Thanks for all the suggestions everyone.

- Robert



Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Delay time question

Bert Freudenberg
On Feb 28, 2008, at 20:26 , Robert F. Scheer wrote:
> My conclusion is not to use 1ms delays anywhere in the time critical
> loops of the robot.  I'm off to look at the interprocess semaphore
> methods that David and Andreas have alluded to.  It looks like C  
> serial
> handlers are the way to go atm.

Well, I had satisfactory results reading from a pipe using AsyncFile  
(had to add a utility method though):

        http://tinlizzie.org/updates/olpc/updates/0995sugar-bf.cs

In this code, #readChar blocks until data is actually available - it  
might work to just open /dev/tty... and start reading in a process,  
similar to what I did in #fetchCommandsFrom:.

- Bert -



Reply | Threaded
Open this post in threaded view
|

[squeak-dev] re: Delay time question

ccrraaiigg
In reply to this post by Andreas.Raab

Hi--

      For what it's worth, the Center for Contemporary Research in Music
and Acoustics at Stanford University (CCRMA) offers packages to make
RedHat and Fedora Core Linux suitable for very-low-latency multimedia
use[1]. I've seen demos of it and it works.


-C

[1] http://ccrma.stanford.edu/planetccrma/software



Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] re: Delay time question

Robert F. Scheer-2

On Thu, 2008-02-28 at 12:59 -0800, Craig Latta wrote:

> Hi--
>
>       For what it's worth, the Center for Contemporary Research in Music
> and Acoustics at Stanford University (CCRMA) offers packages to make
> RedHat and Fedora Core Linux suitable for very-low-latency multimedia
> use[1]. I've seen demos of it and it works.
>
>
> -C
>
> [1] http://ccrma.stanford.edu/planetccrma/software

And also for Ubuntu there's:

sudo apt-get install linux-rt

which I'm testing today.

- Robert


Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] re: Delay time question

Robert F. Scheer-2

On Thu, 2008-02-28 at 14:51 -0800, Robert F. Scheer wrote:

> On Thu, 2008-02-28 at 12:59 -0800, Craig Latta wrote:
> > Hi--
> >
> >       For what it's worth, the Center for Contemporary Research in Music
> > and Acoustics at Stanford University (CCRMA) offers packages to make
> > RedHat and Fedora Core Linux suitable for very-low-latency multimedia
> > use[1]. I've seen demos of it and it works.
> >
> >
> > -C
> >
> > [1] http://ccrma.stanford.edu/planetccrma/software
>
> And also for Ubuntu there's:
>
> sudo apt-get install linux-rt
>
> which I'm testing today.
>
> - Robert
>
>
>

I did a new install of ubuntu and squeak and compared linux-rt with the
standard kernel.

The delay bag test results were so poor on the -rt kernel that there's
no need to trouble with posting them from the test computer.  Half the
delays were 4ms.  The rest were basically worse.  Some up to 45ms.

The stock kernel was >98% at 1ms and the rest up to 16ms.  

- Robert


Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Delay time question

Robert F. Scheer-2
In reply to this post by Bert Freudenberg

On Thu, 2008-02-28 at 20:26 +0100, Bert Freudenberg wrote:

> On Feb 28, 2008, at 20:26 , Robert F. Scheer wrote:
> > My conclusion is not to use 1ms delays anywhere in the time critical
> > loops of the robot.  I'm off to look at the interprocess semaphore
> > methods that David and Andreas have alluded to.  It looks like C  
> > serial
> > handlers are the way to go atm.
>
> Well, I had satisfactory results reading from a pipe using AsyncFile  
> (had to add a utility method though):
>
> http://tinlizzie.org/updates/olpc/updates/0995sugar-bf.cs
>
> In this code, #readChar blocks until data is actually available - it  
> might work to just open /dev/tty... and start reading in a process,  
> similar to what I did in #fetchCommandsFrom:.
>
> - Bert -
>
>
>

This is tantalizingly close to working.  I can open, close and write
something to serial ports as pipes with the AsyncFile methods but can't
successfully read.  As Jon Hylands pointed out, it's probably because
they have no ability to negotiate baud rates and other minor serial port
stuff.  

The exciting part was opening '/dev/ttyUSB0' instead of having to
symlink ttySx to ttyUSBx and pretending the USB port is a tty.

But at this point, I don't see how to get it working with serial lines
unless I put C handlers in the way and use maybe RAM disk shared files
as transfer pipes.

- Robert


Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Delay time question

David T. Lewis
On Thu, Feb 28, 2008 at 09:11:28PM -0800, Robert F. Scheer wrote:
>
> This is tantalizingly close to working.  I can open, close and write
> something to serial ports as pipes with the AsyncFile methods but can't
> successfully read.  As Jon Hylands pointed out, it's probably because
> they have no ability to negotiate baud rates and other minor serial port
> stuff.  

I assume that you are familiar with the unix "stty" command for setting
serial port parameters, but I'll mention it just in case.

Dave


Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Delay time question

Robert F. Scheer-2

On Sat, 2008-03-01 at 09:59 -0500, David T. Lewis wrote:

> On Thu, Feb 28, 2008 at 09:11:28PM -0800, Robert F. Scheer wrote:
> >
> > This is tantalizingly close to working.  I can open, close and write
> > something to serial ports as pipes with the AsyncFile methods but can't
> > successfully read.  As Jon Hylands pointed out, it's probably because
> > they have no ability to negotiate baud rates and other minor serial port
> > stuff.  
>
> I assume that you are familiar with the unix "stty" command for setting
> serial port parameters, but I'll mention it just in case.
>
> Dave
>
>

I was not but am now and thanks for the thoughtful followup.  If it
works, you've saved Squeak!  Well, for me at least.

- Robert


Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Delay time question

Robert F. Scheer-2
In reply to this post by David T. Lewis
David,

You're a life-saver!  AysncFile methods have no trouble reading and
writing from a properly configured serial port.  Just set up the port
properly with stty and it's no sweat.

It's especially useful to be able to designate /dev/ttyUSBx for a serial
port that is connected through USB instead of the old RS232 ports.

I'll do some further tests of timing and blocking but just wanted to say
THANKS!!

- Robert

On Sat, 2008-03-01 at 09:59 -0500, David T. Lewis wrote:

> On Thu, Feb 28, 2008 at 09:11:28PM -0800, Robert F. Scheer wrote:
> >
> > This is tantalizingly close to working.  I can open, close and write
> > something to serial ports as pipes with the AsyncFile methods but can't
> > successfully read.  As Jon Hylands pointed out, it's probably because
> > they have no ability to negotiate baud rates and other minor serial port
> > stuff.  
>
> I assume that you are familiar with the unix "stty" command for setting
> serial port parameters, but I'll mention it just in case.
>
> Dave
>
>
>


123