Possible Dolphi R4 memory leak using ODBC

I am using Dolphin ODBC classes.  This is Dolphin Release 4.0 PL1.
The database is SQL Server 2000 Personal Edition.
Running Windows 2000 Pro SP1 with 768Meg memory and AMD 1100MHZ
There are 30GB available space for sql server data.
The database is running on the same machine as dolphin.  I have not yet
tested this with the db over a network.

I have defined a simple table with one integer field.
The following Dolphin code is executed in a workspace with the Windows task
manager
open displaying memory usage.

| c |

c := (DBConnection new) dsn: 'database containing test table'; uid: 'valid
userid'; pwd: 'correct password'; connect.

1 to: 100000000 do: [:count |
 c exec: 'INSERT INTO TEST VALUES (' , count printString, ')'.
].

Memory usage quickly goes up and up and after awhile there is NO memory
left.

This also happens in my real application.  Unfortunately when I place halt
statements in real application, the act of going into debug mode frees up
the memory leak.  This is suggestive of some things but I haven't tracked
down the answer.

Other points:

1.  This memory leak is happening with prepared statements as well.

2.  When this code is running it seems impossible to break into the Dolphin
debugger with CTRL-Break

3.  I don't think that this is a SQL server or ODBC problem because when
Dolphin is terminated in the task manager, memory instantly frees up.
However, this doesn't entirely rule out those possibilities.

Eric

You wrote in message news:[hidden email]...
That suggests a garbage collection issue, specifically the collector is not
getting sufficient time to run. To work around this trying inserting a pause
in the loop every now and then, or explicitly invoke a full GC, i.e:

1 to: 100000000 do: [:count |
  c exec: 'INSERT INTO TEST VALUES (' , count printString, ')'.
    c\\1000 == 0 ifTrue: [Processor sleep: 10 "or MemoryManager current
collectGarbage"]].

> 2.  When this code is running it seems impossible to break into the
Dolphin
> debugger with CTRL-Break

It seems that way because the test for the break key is performed only after
many thousand full message sends (and/or backwards loop jumps). This can
mean that a loop which spends a lot of time executing lengthy primitives or
external calls (the insert is going to take significant time relative to
message sending) will take a long time to break. If you wait long enough,
however, it will break. Normally this is not an issue in application code
because one tends to execute rather more processing in the loop, bumping up
the message send count.

The test interval is calculated based on the speed of the machine, and on a
fast machine like that it will be very large. Evaluate the following to see
how large:

    InputState classPool at: 'SamplingInterval'

On a P400 it will be around 120k. That sounds like a lot, but will normally
equate to a small fraction of a second.

The test interval can be reduced for an individual run by evaluating an
expression such as "SessionManager inputState primSampleInterval: <N>", this
will start to have a noticeable performance impact if the interval, N, is
made too small. To make a change to the sampling interval stick, you'll need
to modify (or comment out) the expression used to calculate it in
InputState>>onStartup.

Regards

Blair

Blair,
FWIW Preliminary test suggests that garbage collect approach solved the
problem.  Processor sleep did not.

Thanks for you help

"Blair McGlashan" <[hidden email]> wrote in message
news:93f32s$a16gj$[hidden email]... halt
> > statements in real application, the act of going into debug mode frees
up
> > the memory leak.  This is suggestive of some things but I haven't
tracked
> > down the answer.
>
> That suggests a garbage collection issue, specifically the collector is
not
> getting sufficient time to run. To work around this trying inserting a
pause after
> many thousand full message sends (and/or backwards loop jumps). This can
> mean that a loop which spends a lot of time executing lengthy primitives
or
> external calls (the insert is going to take significant time relative to
> message sending) will take a long time to break. If you wait long enough,
> however, it will break. Normally this is not an issue in application code
> because one tends to execute rather more processing in the loop, bumping
up
> the message send count.
>
> The test interval is calculated based on the speed of the machine, and on
a
> fast machine like that it will be very large. Evaluate the following to
see
> how large:
>
>     InputState classPool at: 'SamplingInterval'
>
> On a P400 it will be around 120k. That sounds like a lot, but will
normally
> equate to a small fraction of a second.
>
> The test interval can be reduced for an individual run by evaluating an
> expression such as "SessionManager inputState primSampleInterval: <N>",
this
> will start to have a noticeable performance impact if the interval, N, is
> made too small. To make a change to the sampling interval stick, you'll
need
> to modify (or comment out) the expression used to calculate it in
> InputState>>onStartup.
>
> Regards
>
> Blair
>
>
>

I am processing roughly 25 million records via parameterized odbc calls.
One thing I notice is that Dolphin seems to keep the CPU busy 100 percent of
the time when doing this work.
When I move this to a production machine I will need to get CPU utilization
down a bit<g>.

Is Processor sleep: xxx  the right approach to free up some cpu time for
other tasks.  What is the best way to do this.

Thanks again for your help


"Blair McGlashan" <[hidden email]> wrote in message
news:93f32s$a16gj$[hidden email]... halt
> > statements in real application, the act of going into debug mode frees
up
> > the memory leak.  This is suggestive of some things but I haven't
tracked
> > down the answer.
>
> That suggests a garbage collection issue, specifically the collector is
not
> getting sufficient time to run. To work around this trying inserting a
pause after
> many thousand full message sends (and/or backwards loop jumps). This can
> mean that a loop which spends a lot of time executing lengthy primitives
or
> external calls (the insert is going to take significant time relative to
> message sending) will take a long time to break. If you wait long enough,
> however, it will break. Normally this is not an issue in application code
> because one tends to execute rather more processing in the loop, bumping
up
> the message send count.
>
> The test interval is calculated based on the speed of the machine, and on
a
> fast machine like that it will be very large. Evaluate the following to
see
> how large:
>
>     InputState classPool at: 'SamplingInterval'
>
> On a P400 it will be around 120k. That sounds like a lot, but will
normally
> equate to a small fraction of a second.
>
> The test interval can be reduced for an individual run by evaluating an
> expression such as "SessionManager inputState primSampleInterval: <N>",
this
> will start to have a noticeable performance impact if the interval, N, is
> made too small. To make a change to the sampling interval stick, you'll
need
> to modify (or comment out) the expression used to calculate it in
> InputState>>onStartup.
>
> Regards
>
> Blair
>
>
>

Eric

You wrote in message news:[hidden email]...
> I am processing roughly 25 million records via parameterized odbc calls.

Wow.

> One thing I notice is that Dolphin seems to keep the CPU busy 100 percent
of
> the time when doing this work.

In many ways that is a positive indication that things are working well.

> When I move this to a production machine I will need to get CPU
utilization
> down a bit<g>.

Indeed, although you might find it either isn't an issue (Dolphin is
generally quite friendly, perhaps too friendly, to other apps, and yields
freely), or that lowering the priority of the task (presuming it runs on
Win2k or NT) is appropriate.

>
> Is Processor sleep: xxx  the right approach to free up some cpu time for
> other tasks.  What is the best way to do this.

That would be the simplest approach. If you increase the delay in the sleep
I put forward as one alternative to the GC issue, then you may find it kills
both birds. On the other hand intensive background jobs (it sounds like a
batch job) really ought to be run at a lower priority, and that way you will
make the best use of available CPU horsepower without negatively impacting
other applications too much.

An application can lower its own priority programmatically with a call to
SetThreadPriority(GetCurrentThread(), THREAD_PRIORITY_BELOW_NORMAL). The api
call needs to be added to KernelLibrary though.

Regards

Blair