How to create a background process for a long running task?

Hi:

I would like to create a background process that continuously index all the source code in the image. The process will proactively search for changes and not depend on any event mechanism to inform it of changes. The indexer has also to be ready to response to user queries at random times.

I tried to learn from ObjectMemory class>>idleLoop and BackgroundSearchDriver. The former seems to depend on VM support. The latter seems to be far more interactive than required. How should I create such a background process?

Thanks in advance,
Aik-Siong Koh

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Askoh

[my long running code] forkAt: Processor systemRockBottomPriority.

Look at the senders of #systemRockBottomPriority, #systemBackgroundPriority,
and #userBackgroundPriority. If you want your code to run at the very bottom
then use #systemRockBottomPriority.

If you look at ProcessorScheduler class side #initialize you will see the
initial priority
values.

Also, read the AppDevGuide "Process Control" section.

Terry

===========================================================
Terry Raymond
Crafted Smalltalk
80 Lazywood Ln.
Tiverton, RI  02878
(401) 624-4517      [hidden email]
===========================================================


> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On
> Behalf Of askoh
> Sent: Wednesday, April 02, 2014 11:05 PM
> To: [hidden email]
> Subject: [vwnc] How to create a background process for a long running
task?
>
> Hi:
>
> I would like to create a background process that continuously index all
the
> source code in the image. The process will proactively search for changes
and


_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Thanks for responding. What troubles me is the use of Semaphones in
ObjectMemory class>>idleLoop
        [IdleLoopSemaphore wait.
        CurrentMemoryPolicy idleLoopAction.
        IdleLoopSemaphore initSignals] repeat
and
BackgroundSearchDriver>>search
        | results |
        self triggerEvent: #searching.
        results := searchBlock value: patternHolder value.
        self triggerEvent: #stopped.
        mutex critical: [resultHolder value: results]

I am beginning to see that BackgroundSearchDriver is what I am looking for with a lower priority.

I have looked at Searchlight and am modifying it to add pagerank. I notice that the database does not catch one or more of fileins, parcel loads or store loads.

I want the indexer to be more like a web crawler working totally independently and unobstrusively.

Thanks again,
Aik-Siong Koh

Hi askoh,

The "long running code" would have a loop of some kind. What you do in that loop is important. If you #yield then control may return either immediately or never, depending on what else is going on.  If you Delay then control will be predictably returned even if higher priority processes are in the middle of something; the priority then is only relevant for when the first delay happens if the work after the delay ever yields. If you use semaphores then process priority is useful but it is up to you to manage signaling. Keep in mind that the priority of a process can be changed later too.

Here is pseudo-code to show some tricks:

SourceIndexer class>>startBackgroundProcess
        | newProcess |
        newProcess := [
                [
                        "Predictable timing while allowing lower priority work."
                        (Delay forSeconds: 30) wait.
                        "Avoid having delay interrupt something more important
                         while hoping that control eventually returns."
                        Processor yield.
                        "Should I still be running?"
                        ReindexSourceProcess == Processor activeProcess
                ] whileTrue: [
                        | safeRef |
                        "You figure this out, gets something that can answer a unit of work."
                        safeRef := ReindexSourceWorkStream := self reindexSource_newWorkStream.
                        [
                                "Give more important work a chance (or immediately resume)."
                                Processor yield.
                                "Does stream still have work to do?"
                                safeRef == ReindexSourceWorkStream and: [safeRef atEnd not]
                        ] whileTrue: [
                                (SourceIndexer reindexPortion: safeRef next) == #interrupted
                                        ifTrue: [safeRef setToEnd].
                        ].
                ].
        ] newProcess.
        newProcess name: 'SourceIndexer startBackgroundProcess'.
        ReindexSourceProcess := newProcess.
        ReindexSourceWorkStream := nil.
        newProcess resume.
        Processor yield.
        newProcess priority: Processor userBackgroundPriority.

SourceIndexer class>>stop
        "Allow process to shut itself down."

        ReindexSourceProcess := nil.
        ReindexSourceStream := nil.

SourceIndexer class>>start
        "Run this whenever new work might be available."

        | safeRef |
        ((safeRef := ReindexSourceProcess) isNil or: [safeRef isTerminated])
                ifTrue: [self startBackgroundProcess]
                ifFalse: [ReindexSourceWorkStream := nil].

Consider hacking (SystemUtils class>>modifySystem:) to #start the source indexer after each system modification. The way I showed the code allows low cost interruption. How you break up work and implement #reindexPortion: will affect how responsive the windows are while indexing is going on. One way to tune this further could be to try to identify what work doesn't need to be reindexed.

A process at low priority may not get any attention while other higher priority processes are active.  Likewise, a high priority process that only yields (no Delay or semaphore wait) can starve attention from patient (yielding) lower priority processes. How well the example works using #yield depends on how well managed the rest of your processes are. A multi-process semaphore solution (with a Delay for timing) can do what #yield based solutions can't; however, they are far more tricky and usually not necessary.

Paul Baumann


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Terry Raymond
Sent: Thursday, April 03, 2014 07:16
To: 'askoh'; [hidden email]
Subject: Re: [vwnc] How to create a background process for a long running task?

Askoh

[my long running code] forkAt: Processor systemRockBottomPriority.

Look at the senders of #systemRockBottomPriority, #systemBackgroundPriority, and #userBackgroundPriority. If you want your code to run at the very bottom then use #systemRockBottomPriority.

If you look at ProcessorScheduler class side #initialize you will see the initial priority values.

Also, read the AppDevGuide "Process Control" section.

Terry

===========================================================
Terry Raymond
Crafted Smalltalk
80 Lazywood Ln.
Tiverton, RI  02878
(401) 624-4517      [hidden email] ===========================================================


> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On
> Behalf Of askoh
> Sent: Wednesday, April 02, 2014 11:05 PM
> To: [hidden email]
> Subject: [vwnc] How to create a background process for a long running
task?
>
> Hi:
>
> I would like to create a background process that continuously index all
the
> source code in the image. The process will proactively search for changes
and


_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of IntercontinentalExchange, Inc. (ICE), its subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

hi askoh

In application i am working on now I have a background process which
waits on the background for some even to occur and then responds to it.
To accomplish this I have a shared queue of events which this background
process is waiting for:

couple of code snippets:

initialize
     queue := SharedQueue new.
     process := [[self process] repeat] newProcess.
     process priority: Processor userBackgroundPriority- 6.
     process name: self class name asString.
     process resume

=============
process
     | commandset command arg aStr |
     commandset := queue next.
     command := (commandset at: 1) asLowercase.
     arg := (commandset at: 2) asLowercase.
     command = 'open' ifTrue: [self open: arg].
     command = 'system'
         ifTrue:
             [(aStr := arg readStream) throughAll: 'open'.
             aStr atEnd ifFalse: [self open: aStr upToEnd trimBlanks]].
=================================
addCommand: aCommandset

     queue nextPut: aCommandset.
=======================

So in my case some user action sends the #addCommand:  ,
and the   processor does whatever needs to be done  depending on the
command type, and then waits on the queue again.
you can modify your  tools  to add the changes to the queue, which will
be dealt later when the environment is not busy.

--
Mark

askoh wrote:
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc