Refreshing a list view during heavy processing? GC?

I am running some code that does heavy database processing in a development
image.  I have a list view that shows some records, and I am iterating over
the list view moving the selection as a process a bunch for records in other
tables that are keyed off the selected record in the list view.  It works
great, however sometimes the list view stops refreshing, and then I can't
see the processing status (which record is selected).  I have a 3 second
delay in-between records (to ensure GC), I also added an invalidate before
the delay to try to ensure that the view would be repainted.  I found that
seemed to work, but when I let it run overnight and look at it the next day
it is totally white and never repaints again.  I went so far as to make a
logging Delay subclass, and it appears that the Delay is working.

I have distilled the phenomenon into some sample code included in this
message (see bellow).  I run the code from a workspace, and open the
SysInternals Dbgview tool.  I then run the processing, and I can see the
list selection changing.  Now if I drag the dbgview window over any part of
the list it ceases to repaint, and the selection does not even redraw
however I continue to see that new items are being processed in dbgview.
This seems to mirror what happens in my db processor.  When I do a
Ctrl-Break the list is finally redrawn and I can see the appropriate
selection.

What is going on here?  Could the GC be exceeding the duration of the delay,
causing the repaints to be deferred?  In my real application I am using a
delay of 3 seconds between records, I really don't want to increase that
value.  How can I ensure a complete GC and UI repaint?  If I could do those
two things I would not even need a delay.  In my DB processor I am still not
sure that complete GC's are always happening as I sometimes get errors about
exceeding the maximum number of DB handles for a few rows in the list, and
then it starts working again.

Sample Code
==================
"Setup"
col := OrderedCollection withAll: (1 to: 2000).
lp := ListPresenter showOn: (ListModel on: col).
lp topShell extent: 50@400.
"Start Processing"
lp selectionByIndex: 1.
[lp selectionByIndex < lp model size]
 whileTrue: [
  lp view invalidate.
  (Delay forMilliseconds: 500) wait.
  KernelLibrary default outputDebugString: lp selection displayString.
  (lp selection *10000 factorial) displayString.
  lp selectionByIndex: lp selectionByIndex + 1].
==================

Chris

Chris

Can I first summarise the general issue for the benefit of others: When
running batch-style processing through DB Connection one can exhaust DB
handles for those DBs which impose a limit on such things. The cause of this
is that, unless explicitly released, the handles allocated by the DB package
are free'd through finalization. Finalization is a background priority task,
and is dependent on sufficient cycles being available both to identify
candicates for finalization (in the GC), and to allow the low priority
finalization process to run. This is exacerbated because the statement
objects are weakly referenced from the connection objects, meaning that
identification of finalization candidates from the statement population
requires a full GC cycle. In addition finalization of connections must be
delayed until all statements opened on the connection have themselves been
finalized.

As far as the DB Connection is concerned the best way to avoid this problem
is to:
a) minimize statement usage by using parameterised statements for inserts,
etc.
b) explicity free (with #free) any DB connections, statements, etc, that are
no longer needed.

However this is not always convenient so...

You wrote in message news:bj80tp$gfuif$[hidden email]...
> I am running some code that does heavy database processing in a
development
> image.  I have a list view that shows some records, ...
>...sometimes the list view stops refreshing, and then I can't
> see the processing status (which record is selected).  I have a 3 second
> delay in-between records (to ensure GC), I also added an invalidate before
> the delay to try to ensure that the view would be repainted.  I found that
> seemed to work, but when I let it run overnight and look at it the next
day
> it is totally white and never repaints again.  I went so far as to make a
> logging Delay subclass, and it appears that the Delay is working.
>
> I have distilled the phenomenon into some sample code included in this
> message (see bellow).  I run the code from a workspace, and open the
> SysInternals Dbgview tool.  I then run the processing, and I can see the
> list selection changing.  Now if I drag the dbgview window over any part
of
> the list it ceases to repaint, and the selection does not even redraw
> however I continue to see that new items are being processed in dbgview.
> This seems to mirror what happens in my db processor.  When I do a
> Ctrl-Break the list is finally redrawn and I can see the appropriate
> selection.
>

Well in the case of your example the the code is being run in the foreground
process, and therefore the Delay would not make time available for
repainting the views, but instead would just block the UI process and
prevent it from doing anything. The fact that the view is updating at all is
because the ListView performs some screen updates synchronously (like
repainting items on selection changes). I note that when the list appears it
doesn't paint the list contents, just the selected item (although you can
change this by using #update instead of #invalidate).  In fact I don't think
the list has ever been able to receive a WM_PAINT because the UI process
never returns to the message queue. I would guess that when you switch away
to another app and drag it over the list view that Windows then judges the
app. to be non-responsive (which it is, since it isn't processing any
messages in the queue), and this somehow causes even the synchronous
repaints to do nothing.

To "fix" this example you could replace the delay with: 'SessionManager
inputState pumpMessages'. You won't need the #invalidate.

One thing to be aware of is that #pumpMessages could potentially process
further user input, so it is a good idea to disable your application window.
If you are kicking the process off from a menu/toolbar command, then an easy
way to do this is to mark it as 'Modal' on the Styles tab in the menu
editor's command menu item dialog, or the #isModalCommand aspect.

> What is going on here?  Could the GC be exceeding the duration of the
delay,
> causing the repaints to be deferred?

Well, if you are doing the batch processing on the main UI thread, then the
same would apply. You have to make sure that the Windows input queue is
serviced.

>...In my real application I am using a
> delay of 3 seconds between records, I really don't want to increase that
> value.  How can I ensure a complete GC and UI repaint?

Replace the Delay with:

MemoryManager current collectGarbage; administerLastRites.    "Do GC and
finalize - do this more than once if you "
SessionManager inputState pumpMessages.
"Keep UI painted"

>...  If I could do those
> two things I would not even need a delay.  In my DB processor I am still
not
> sure that complete GC's are always happening as I sometimes get errors
about
> exceeding the maximum number of DB handles for a few rows in the list, and
> then it starts working again.

Well that would be because the Delay isn't necessarily a reliable way of
ensure that GC and finalization keeps up with your handle consumption if the
limits are low (e.g Oracle).

Hope this helps, if not then ProgressDialog may make interesting browsing...

Regards

Blair

"Christopher J. Demers" <[hidden email]> wrote in
message news:bj80tp$gfuif$[hidden email]...
...
> I have distilled the phenomenon into some sample code included in this
> message (see bellow).  I run the code from a workspace, and open the
> SysInternals Dbgview tool.  I then run the processing, and I can see the
> list selection changing.  Now if I drag the dbgview window over any part
of
> the list it ceases to repaint, and the selection does not even redraw
> however I continue to see that new items are being processed in dbgview.
> This seems to mirror what happens in my db processor.  When I do a
> Ctrl-Break the list is finally redrawn and I can see the appropriate
> selection.
...

After seeing a previous posting from Blair I tried sending update to the
view.  This had the same problem as before.  Then I tried forking it at
userBackground priority, and it seemed to update the display.  I guess I
might just have to fork.  I am still curious about the GC issues, maybe
forking will improve that as well.  Is forking the recommended way of doing
this?  I have a vague recollection of problems with error trapping and
forking from a previous project.  Hopefully that won't be an issue here.

Chris

"Blair McGlashan" <[hidden email]> wrote in message
news:bj88jc$gh5ru$[hidden email]... ...

Thanks, this looks like it will do exactly what I need.  It fixes my sample,
and now I will pound on it in my real tool.  Forcing a GC (potentially even
more than once) feels a lot better than using the delay and will probably be
faster in most situations anyway.

As always thanks for the great support.
Chris