Multi-column ListView display artifact with DATE objects from ADO...

I experienced a strange display artifact with ListViews a while ago.  I only
recently had a chance to put together a demonstration of the problem.  The
data is loaded from an MS Access database via ADO.  It is then placed into a
LookupTable.  A collection of LookupTables is the model of the ListView.
The elements of the LookupTable are the columns.  The first column always
has data in it, and looks fine.  The second column sometimes has data in it.
When it has data it looks fine.  When it does not have data, it sometimes
displays as garbage characters.  The third column sometimes has data in it.
It always looks fine.  I think the display anomaly is caused by an
interaction between the third column and the second column.  Actually I
think I know what the cause of the problem is, but I don't understand some
of the subtleties.

The problem seems to be that the third column is a DATE object returned by
ADO.  But I wonder why this sometimes messes up the second column.  The
issue I have experienced seems to indicate that it is not safe to hold on to
DATE objects, and that they should be converted to TimeStamps.  When I
convert DATE objects to TimeStamps or save and reload them from an STB file
I see no gibberish.  I wonder if it would make sense for the ADO wrappers to
coerce DATE objects into TimeStamp objects, or perhaps that would be too
great a performance hit.

Interested parties can see a screen capture here
http://www.mitchellscientific.com/temp/GiberishInListView.jpg (36k) and can
get code and data needed to reproduce the issue here
http://www.mitchellscientific.com/temp/DolphTest1Data.zip (210k) .  The text
in the database has been shrouded to random characters.  The gibberish is
obvious as it is usually extended ASCII characters.  The workspace code is
just intended to demonstrate this issue.  I did see the issue in a normal
application.

Thoughts, and reflections welcome.

Chris

Chris,

> Thoughts, and reflections welcome.

Just an observation, there's probably a deeper cause...

It looks a bit like the database is answering a pointer to an null
string and then, occasionally, reusing that memory for something else
(probably the DATE for the next field).  If you change the LookupTable
creation code to ....

recSet do:
    [:each |
    | lt |
    lt := LookupTable new.
    col add: lt.
    each do: [:eachField | lt at: eachField name asSymbol put: eachField
value].
    ((lt at: #MitSciStatus) notNil and: [(lt at: #MitSciStatus)
isEmpty])
        ifTrue: [lt at: #MitSciStatus put: String new]].

.... so that you immediately replace the String referenced by the
pointer that (I assume) the DB answers with a new one then the garbage
disappears.

--
Ian

Use the Reply-To address to contact me.
Mail sent to the From address is ignored.

Chris

You wrote in message news:c2m45k$1tnps4$[hidden email]...
> I experienced a strange display artifact with ListViews a while ago. ...
>
> The problem seems to be that the third column is a DATE object returned by
> ADO.  But I wonder why this sometimes messes up the second column.  The
> issue I have experienced seems to indicate that it is not safe to hold on
to
> DATE objects, and that they should be converted to TimeStamps.  When I
> convert DATE objects to TimeStamps or save and reload them from an STB
file
> I see no gibberish.  I wonder if it would make sense for the ADO wrappers
to
> coerce DATE objects into TimeStamp objects, or perhaps that would be too
> great a performance hit.
> ....

Chris, I'm glad you found this. This would appear to be a bug in the driver
to which the VARIANT class' #value primitive is sensitive. DATE is a
red-herring, however. What is actually happening is that the String created
by the VARIANT>>value primitive is not correctly null-terminated. The
implementation of LVITEM>>textInBuffer: relies on the String it is passed
being null-terminated (which Dolphin String's should be, even though the
null is not normally visible to Smalltalk), since it uses a C library call
(strncpy) to populate the text buffer supplied by the ListView. If the
string is not null terminated, then junk will be copied into the buffer,
even though the String appears to be empty in Smalltalk.

So how does the non-null terminated String instance get created? Well
without have investigated properly yet I am speculating that it is created
by the VARIANT>>value primitive from a BSTR value that is not itself
null-terminated. According to MSDN "[BSTRs] are zero-terminated, and in most
cases they can be treated just like OLECHAR* strings". The VM assumes that
this will be true, and so although it uses the SysStringLen function and
determines that the BSTR is empty, it still expects to be able to copy a
null out of it into the Smalltalk string it creates. In this case it would
appear that the DB driver has somehow created an invalid BSTR that is of
zero length but which does not have a null terminator.

The correct fix for this () is to make the VM insensitive to the presence or
absence of a null-terminator in the BSTR, but in the meantime you can
workaround the problem by commenting out the use of primitive 172 in the
VARIANT>>value method. This will slow down access to the value of a variant
somewhat, but it probably won't be that significant in most cases (it slows
down the population of the ListView by about 10% in your example code).

Regards

Blair

Chris

You wrote in message news:c2m45k$1tnps4$[hidden email]...
> I experienced a strange display artifact with ListViews a while ago. ...
>
> The problem seems to be that the third column is a DATE object returned by
> ADO.  But I wonder why this sometimes messes up the second column.  The
> issue I have experienced seems to indicate that it is not safe to hold on
to
> DATE objects, and that they should be converted to TimeStamps.  When I
> convert DATE objects to TimeStamps or save and reload them from an STB
file
> I see no gibberish.  I wonder if it would make sense for the ADO wrappers
to
> coerce DATE objects into TimeStamp objects, or perhaps that would be too
> great a performance hit.
> ....

Chris, I'm glad you found this. This would appear to be a bug in the driver
to which the VARIANT class' #value primitive is sensitive. DATE is a
red-herring, however. What is actually happening is that the String created
by the VARIANT>>value primitive is not correctly null-terminated. The
implementation of LVITEM>>textInBuffer: relies on the String it is passed
being null-terminated (which Dolphin String's should be, even though the
null is not normally visible to Smalltalk), since it uses a C library call
(strncpy) to populate the text buffer supplied by the ListView. If the
string is not null terminated, then junk will be copied into the buffer,
even though the String appears to be empty in Smalltalk.

So how does the non-null terminated String instance get created? Well
without have investigated properly yet I am speculating that it is created
by the VARIANT>>value primitive from a BSTR value that is not itself
null-terminated. According to MSDN "[BSTRs] are zero-terminated, and in most
cases they can be treated just like OLECHAR* strings". The VM assumes that
this will be true, and so although it uses the SysStringLen function and
determines that the BSTR is empty, it still expects to be able to copy a
null out of it into the Smalltalk string it creates. In this case it would
appear that the DB driver has somehow created an invalid BSTR that is of
zero length but which does not have a null terminator.

The correct fix for this (#1509) is to make the VM insensitive to the
presence or
absence of a null-terminator in the BSTR, but in the meantime you can
workaround the problem by commenting out the use of primitive 172 in the
VARIANT>>value method. This will slow down access to the value of a variant
somewhat, but it probably won't be that significant in most cases (it slows
down the population of the ListView by about 10% in your example code).

Regards

Blair

"Blair McGlashan" <[hidden email]> wrote in message
news:c2nde5$1vdm3e$[hidden email]...
> Chris
>
> You wrote in message news:c2m45k$1tnps4$[hidden email]...
> > I experienced a strange display artifact with ListViews a while ago. ...
> >
> > The problem seems to be that the third column is a DATE object returned
by
> > ADO.  But I wonder why this sometimes messes up the second column.  The
> > issue I have experienced seems to indicate that it is not safe to hold
on
...
> Chris, I'm glad you found this. This would appear to be a bug in the
driver
> to which the VARIANT class' #value primitive is sensitive. DATE is a
> red-herring, however. What is actually happening is that the String
created
> by the VARIANT>>value primitive is not correctly null-terminated. The
...

Thanks Ian and Blair.  This really helps to explain the situation.  The
multiple personality (external/internal) of strings is what threw me.  The
string seemed to contain the junk characters but I could not see them in the
Dolphin Strings.  Your explanation makes perfect sense.

Chris