Re: memory and VM issues (interp.c part 1 of 2 attached)

> Ok, well my code and  Andreas' code were at one point separate.
> I've attached Andreas' original changeset that dates from january and  
> you should be
> able to apply to a SVN 1100 pre 64 bit vmmaker setup. That should  
> tell us if the
> problem exists with his code.
>
> Don't apply my GCInstrumentJMMImprovementsAR.1.cs.
>
>
> On 31-Jul-05, at 7:43 PM, David T. Lewis wrote:
>
> > On Fri, Jul 29, 2005 at 01:03:31PM -0700, Tim Rowledge wrote:
> >
> >> Could you try a 3.8-6665 image, SVN revision 1100, VMMaker38b4 combo?
> >> That is the last pre-64bit-fiddling build and should work ok. One  
> >> thing
> >> we're a bit concerned with is the changes to GC/weak objects that got
> >> put in later.
> >>
> >

> I've been trying to isolate this problem on my <red herring> 32 bit  
> Intel Linux
> </red herring> system by looking for what changed in the VMM code  
> and the SVN
> platforms tree, testing the resulting VMs with the 32 bit image from
> squeak.hpl.hp.com/squeak64/dist3/Squeak32-3.8g-6548.image.tar.gz.
>
> The results so far can be summarized as:
>
>
>> The problem was introduced in VMM code as opposed to platforms code.
>> It was introduced in the change from VMMaker-tpr.21 to VMMaker-tpr.
>> 22. The VMMaker-tpr.22
>> change set contains "Merge in johnmci & ar's GC instrumentation  
>> and weak pointer changes".
>>
>> Platforms version             VMMaker version Result
>> -----------------             --------------- ------
>> squeak-svn-source-SVN1200     VMMaker-tpr.21  OK
>> squeak-svn-source-SVN1200     VMMaker-tpr.22  NFG
>>
>
> But I discovered (entirely by accident) that I overlooked a rather  
> large
> dimension of the problem. The bad (?) VM created from squeak-svn-
> source-SVN1200
> and VMMaker-tpr.22 works just fine with an image that does not have  
> the 64 bit
> changes installed. So now comparing the results from a  
> Squeak32-3.8g-6548.image
> from the original ./dist3 distribution (image with 64 bit changes  
> loaded) versus
> a "stock" Squeak image such as Squeak3.7-5989-basic.image or  
> Squeak3.9a-6472.image,
> we have this:
>
> Platforms version               VMMaker version  
> Image                           Result
> -----------------               ---------------  
> -----                           ------
> squeak-svn-source-SVN1200       VMMaker-tpr.21  dist3/
> Squeak32-3.8g-6548.image  OK
> squeak-svn-source-SVN1200       VMMaker-tpr.22  dist3/
> Squeak32-3.8g-6548.image  NFG
> squeak-svn-source-SVN1200       VMMaker-tpr.21  Squeak3.7-5989-
> basic.image      OK
> squeak-svn-source-SVN1200       VMMaker-tpr.22  Squeak3.7-5989-
> basic.image      OK
> squeak-svn-source-SVN1200       VMMaker-tpr.21  
> Squeak3.9a-6472.image           OK
> squeak-svn-source-SVN1200       VMMaker-tpr.22  
> Squeak3.9a-6472.image           OK
>
> Evidently the combination of the changes introduce in VMMaker-tpr.
> 22 with the
> changes that were added to the Squeak32-3.8g-6548 image in ./dist3,  
> and maybe
> or maybe not in combination with a 32 bit Intel Linux platform, are  
> what cause
> the problem to be present.
>
> I have not tried any other experiments yet, but I thought I should  
> report this
> in case anyone is losing sleep over the problem right now.
>
> My next steps (but not tonight) will be to try adding some or all of
> VMM38b4-64bit-image1-ikp.1.cs, VMM38b4-64bit-image2-ikp.1.cs, and
> System-Tracing.2.cs to a "stock" image and see if one of them induces
> the problem when running on a "bad" VM built from squeak-svn-source-
> SVN1200
> and VMMaker-tpr.22.
>
> Dave
>
>
>

>
> To confirm consider
> !ObjectMemory methodsFor: 'finalization' stamp: 'ar 1/18/2005 16:39'!
> finalizeReference: oop
>      "During sweep phase we have encountered a weak reference.
>      Check if  its object has gone away (or is about to) and if so,  
> signal a
>      semaphore. "
>      "Do *not* inline this in sweepPhase - it is quite an unlikely
>      case to run into a weak reference"
>      | weakOop oopGone chunk firstField lastField |
>      self inline: false.
>      firstField _ BaseHeaderSize + ((self nonWeakFieldsOf: oop) << 2).
>      lastField _ self lastPointerOf: oop.
>      firstField to: lastField by: 4 do: [:i |
>
>
> where we do a to:by: with 4, or the shift (<<) by 2.
>
> The VMMaker in beta that later shipped, what is the version?. change  
> that I believe to
> !ObjectMemory methodsFor: 'finalization' stamp: 'JMM 4/13/2005 20:54'!
> finalizeReference: oop
>      "During sweep phase we have encountered a weak reference.
>      Check if  its object has gone away (or is about to) and if so,  
> signal a
>      semaphore. "
>      "Do *not* inline this in sweepPhase - it is quite an unlikely
>      case to run into a weak reference"
>      | weakOop oopGone chunk firstField lastField |
>      self inline: false.
>      firstField _ BaseHeaderSize + ((self nonWeakFieldsOf: oop) <<  
> ShiftForWord).
>      lastField _ self lastPointerOf: oop.
>      firstField to: lastField by: BytesPerWord do: [:i |
>
>
> where as  you see we use the ShiftForWord and the  BytesPerWord
>
> So if you are building a 64bit VM, and if the code you have says      
> firstField to: lastField by: 4 do: [:i |
> then you are hosed. If the code does have the BytesPerWord, it is  
> possible lurking in the changeset a missing change where we need to
> use some nemonic versus 2 or 4 or some other magic number that is  
> affected when you go from 32 to 64.
>
> On 3-Aug-05, at 6:49 PM, David T. Lewis wrote:
>
> > I've been trying to isolate this problem on my <red herring> 32 bit  
> > Intel Linux
> > </red herring> system by looking for what changed in the VMM code  
> > and the SVN
> > platforms tree, testing the resulting VMs with the 32 bit image from
> > squeak.hpl.hp.com/squeak64/dist3/Squeak32-3.8g-6548.image.tar.gz.
> >
> > The results so far can be summarized as:
> >
> >
> >> The problem was introduced in VMM code as opposed to platforms code.
> >> It was introduced in the change from VMMaker-tpr.21 to VMMaker-tpr.
> >> 22. The VMMaker-tpr.22
> >> change set contains "Merge in johnmci & ar's GC instrumentation  
> >> and weak pointer changes".
> >>
> >> Platforms version             VMMaker version Result
> >> -----------------             --------------- ------
> >> squeak-svn-source-SVN1200     VMMaker-tpr.21  OK
> >> squeak-svn-source-SVN1200     VMMaker-tpr.22  NFG
> >>
> >
> > But I discovered (entirely by accident) that I overlooked a rather  
> > large
> > dimension of the problem. The bad (?) VM created from squeak-svn-
> > source-SVN1200
> > and VMMaker-tpr.22 works just fine with an image that does not have  
> > the 64 bit
> > changes installed. So now comparing the results from a  
> > Squeak32-3.8g-6548.image
> > from the original ./dist3 distribution (image with 64 bit changes  
> > loaded) versus
> > a "stock" Squeak image such as Squeak3.7-5989-basic.image or  
> > Squeak3.9a-6472.image,
> > we have this:
> >
> > Platforms version               VMMaker version  
> > Image                           Result
> > -----------------               ---------------  
> > -----                           ------
> > squeak-svn-source-SVN1200       VMMaker-tpr.21  dist3/
> > Squeak32-3.8g-6548.image  OK
> > squeak-svn-source-SVN1200       VMMaker-tpr.22  dist3/
> > Squeak32-3.8g-6548.image  NFG
> > squeak-svn-source-SVN1200       VMMaker-tpr.21  Squeak3.7-5989-
> > basic.image      OK
> > squeak-svn-source-SVN1200       VMMaker-tpr.22  Squeak3.7-5989-
> > basic.image      OK
> > squeak-svn-source-SVN1200       VMMaker-tpr.21  
> > Squeak3.9a-6472.image           OK
> > squeak-svn-source-SVN1200       VMMaker-tpr.22  
> > Squeak3.9a-6472.image           OK
> >
> > Evidently the combination of the changes introduce in VMMaker-tpr.
> > 22 with the
> > changes that were added to the Squeak32-3.8g-6548 image in ./dist3,  
> > and maybe
> > or maybe not in combination with a 32 bit Intel Linux platform, are  
> > what cause
> > the problem to be present.
> >
> > I have not tried any other experiments yet, but I thought I should  
> > report this
> > in case anyone is losing sleep over the problem right now.
> >
> > My next steps (but not tonight) will be to try adding some or all of
> > VMM38b4-64bit-image1-ikp.1.cs, VMM38b4-64bit-image2-ikp.1.cs, and
> > System-Tracing.2.cs to a "stock" image and see if one of them induces
> > the problem when running on a "bad" VM built from squeak-svn-source-
> > SVN1200
> > and VMMaker-tpr.22.
> >
> > Dave
> >
> >
> >
>
> --
> ========================================================================
> ===
> John M. McIntosh <[hidden email]> 1-800-477-2659
> Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
> ========================================================================
> ===

> On Wed, Aug 03, 2005 at 11:00:12PM -0700, John M McIntosh wrote:
>
>> I think if you look the johnmci & ar's GC instrumentation and weak
>> pointer changes do not consider the
>> width 32 versus 64 and originally used hard coded numbers to indicate
>> the size of a word or header.
>> Later this was changed to use constants.
>>
>
> Yes, that's right. But I've tested VMM versions with and without
> the constants (Tim put these in in the very next version of VMM as
> I recall), so that does not appear to be related to the problem.
> Also note that I'm building 32 bit VMs on 32 bit hardware for all
> the results I've reported.
>
> So to summarize: The problem first appears as of VMMaker-tpr.22, which
> adds the johnmci & ar's GC instrumentation and weak pointer changes.
> The combination of VMMaker-tpr.21 plus ar's changes alone are not
> sufficient to trigger the problem.  The problem shows up when running
> an image from ./dist3 (i.e. an image that has had the 64 bit image
> changes added), but not on other images such as Squeak3.9a-6472.
> All tests were done on 32 bit Intel Linux.
>
> Dave
>
>
>

> Ok, perhaps you can summarize.
>
> Are you saying if you take a 3.8 base image then apply a tpr.21 and  
> build a VM that it works fine with the 3.8 base image?
>
> And If you take Ian's ./dist3 image and run that then it fails using  
> the new VM?
>
> I will note that the call to get VMMparameter data has different  
> slots from the one Ian originally submitted.
> You should check that the ./dist3 image has
>
> wordSize
>      "Answer the size (in bytes) of an object pointer."
>      "Smalltalk wordSize"
>
>      ^[SmalltalkImage current vmParameterAt: 40] on: Error do: [4]
>
>
> primitiveVMParameter
>      "Behaviour depends on argument count:
>          0 args:    return an Array of VM parameter values;
>          1 arg:    return the indicated VM parameter;
>          2 args:    set the VM indicated parameter.
>      VM parameters are numbered as follows:
>          1    end of old-space (0-based, read-only)
>          2    end of young-space (read-only)
>          3    end of memory (read-only)
>          4    allocationCount (read-only)
>          5    allocations between GCs (read-write)
>          6    survivor count tenuring threshold (read-write)
>          7    full GCs since startup (read-only)
>          8    total milliseconds in full GCs since startup (read-only)
>          9    incremental GCs since startup (read-only)
>          10    total milliseconds in incremental GCs since startup  
> (read-only)
>          11    tenures of surving objects since startup (read-only)
>          12-20 specific to the translating VM
>          21    root table size (read-only)
>          22    root table overflows since startup (read-only)
>          23    bytes of extra memory to reserve for VM buffers,  
> plugins, etc.
>          24    memory threshold above which shrinking object memory (rw)
>          25    memory headroom when growing object memory (rw)
>          26  interruptChecksEveryNms - force an ioProcessEvents every  
> N milliseconds, in case the image  is not calling getNextEvent often  
> (rw)
>          27    number of times mark loop iterated for current IGC/FGC  
> (read-only) includes ALL marking
>          28    number of times sweep loop iterated  for current IGC/
> FGC (read-only)
>          29    number of times make forward loop iterated for current  
> IGC/FGC (read-only)
>          30    number of times compact move loop iterated for current  
> IGC/FGC (read-only)
>          31    number of grow memory requests (read-only)
>          32    number of shrink memory requests (read-only)
>          33    number of root table entries used for current IGC/FGC  
> (read-only)
>          34    number of allocations done before current IGC/FGC  
> (read-only)
>          35    number of survivor objects after current IGC/FGC (read-
> only)
>          36  millisecond clock when current IGC/FGC completed (read-
> only)
>          37  number of marked objects for Roots of the world, not  
> including Root Table entries for current IGC/FGC (read-only)
>          38  milliseconds taken by current IGC  (read-only)
>          39  Number of finalization signals for Weak Objects pending  
> when current IGC/FGC completed (read-only)
>          40 BytesPerWord for this image
>
>
>
> On 4-Aug-05, at 4:20 AM, David T. Lewis wrote:
>
> > On Wed, Aug 03, 2005 at 11:00:12PM -0700, John M McIntosh wrote:
> >
> >> I think if you look the johnmci & ar's GC instrumentation and weak
> >> pointer changes do not consider the
> >> width 32 versus 64 and originally used hard coded numbers to indicate
> >> the size of a word or header.
> >> Later this was changed to use constants.
> >>
> >
> > Yes, that's right. But I've tested VMM versions with and without
> > the constants (Tim put these in in the very next version of VMM as
> > I recall), so that does not appear to be related to the problem.
> > Also note that I'm building 32 bit VMs on 32 bit hardware for all
> > the results I've reported.
> >
> > So to summarize: The problem first appears as of VMMaker-tpr.22, which
> > adds the johnmci & ar's GC instrumentation and weak pointer changes.
> > The combination of VMMaker-tpr.21 plus ar's changes alone are not
> > sufficient to trigger the problem.  The problem shows up when running
> > an image from ./dist3 (i.e. an image that has had the 64 bit image
> > changes added), but not on other images such as Squeak3.9a-6472.
> > All tests were done on 32 bit Intel Linux.
> >
> > Dave
> >
> >
> >
>
> --
> ========================================================================
> ===
> John M. McIntosh <[hidden email]> 1-800-477-2659
> Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
> ========================================================================
> ===

> Ok, perhaps you can summarize.
>
> Are you saying if you take a 3.8 base image then apply a tpr.21 and  
> build a VM that it works fine with the 3.8 base image?
>
> And If you take Ian's ./dist3 image and run that then it fails using  
> the new VM?
>
> I will note that the call to get VMMparameter data has different  
> slots from the one Ian originally submitted.
> You should check that the ./dist3 image has
>
> wordSize
>      "Answer the size (in bytes) of an object pointer."
>      "Smalltalk wordSize"
>
>      ^[SmalltalkImage current vmParameterAt: 40] on: Error do: [4]
>
>
> primitiveVMParameter
>      "Behaviour depends on argument count:
>          0 args:    return an Array of VM parameter values;
>          1 arg:    return the indicated VM parameter;
>          2 args:    set the VM indicated parameter.
>      VM parameters are numbered as follows:
>          1    end of old-space (0-based, read-only)
>          2    end of young-space (read-only)
>          3    end of memory (read-only)
>          4    allocationCount (read-only)
>          5    allocations between GCs (read-write)
>          6    survivor count tenuring threshold (read-write)
>          7    full GCs since startup (read-only)
>          8    total milliseconds in full GCs since startup (read-only)
>          9    incremental GCs since startup (read-only)
>          10    total milliseconds in incremental GCs since startup  
> (read-only)
>          11    tenures of surving objects since startup (read-only)
>          12-20 specific to the translating VM
>          21    root table size (read-only)
>          22    root table overflows since startup (read-only)
>          23    bytes of extra memory to reserve for VM buffers,  
> plugins, etc.
>          24    memory threshold above which shrinking object memory (rw)
>          25    memory headroom when growing object memory (rw)
>          26  interruptChecksEveryNms - force an ioProcessEvents every  
> N milliseconds, in case the image  is not calling getNextEvent often  
> (rw)
>          27    number of times mark loop iterated for current IGC/FGC  
> (read-only) includes ALL marking
>          28    number of times sweep loop iterated  for current IGC/
> FGC (read-only)
>          29    number of times make forward loop iterated for current  
> IGC/FGC (read-only)
>          30    number of times compact move loop iterated for current  
> IGC/FGC (read-only)
>          31    number of grow memory requests (read-only)
>          32    number of shrink memory requests (read-only)
>          33    number of root table entries used for current IGC/FGC  
> (read-only)
>          34    number of allocations done before current IGC/FGC  
> (read-only)
>          35    number of survivor objects after current IGC/FGC (read-
> only)
>          36  millisecond clock when current IGC/FGC completed (read-
> only)
>          37  number of marked objects for Roots of the world, not  
> including Root Table entries for current IGC/FGC (read-only)
>          38  milliseconds taken by current IGC  (read-only)
>          39  Number of finalization signals for Weak Objects pending  
> when current IGC/FGC completed (read-only)
>          40 BytesPerWord for this image
>
>
>
> On 4-Aug-05, at 4:20 AM, David T. Lewis wrote:
>
> > On Wed, Aug 03, 2005 at 11:00:12PM -0700, John M McIntosh wrote:
> >
> >> I think if you look the johnmci & ar's GC instrumentation and weak
> >> pointer changes do not consider the
> >> width 32 versus 64 and originally used hard coded numbers to indicate
> >> the size of a word or header.
> >> Later this was changed to use constants.
> >>
> >
> > Yes, that's right. But I've tested VMM versions with and without
> > the constants (Tim put these in in the very next version of VMM as
> > I recall), so that does not appear to be related to the problem.
> > Also note that I'm building 32 bit VMs on 32 bit hardware for all
> > the results I've reported.
> >
> > So to summarize: The problem first appears as of VMMaker-tpr.22, which
> > adds the johnmci & ar's GC instrumentation and weak pointer changes.
> > The combination of VMMaker-tpr.21 plus ar's changes alone are not
> > sufficient to trigger the problem.  The problem shows up when running
> > an image from ./dist3 (i.e. an image that has had the 64 bit image
> > changes added), but not on other images such as Squeak3.9a-6472.
> > All tests were done on 32 bit Intel Linux.
> >
> > Dave
> >
> >
> >
>
> --
> ========================================================================
> ===
> John M. McIntosh <[hidden email]> 1-800-477-2659
> Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
> ========================================================================
> ===

>
> Bingo. But good grief, Charlie Brown.
>
> The image looks at vmParameterAt: 27, but  
> Interpreter>>primitiveVMParameter puts the parameter in slot 40.
>
> The assignment of slot 27 in the image is from the ikp changes of  
> 3/26/2005 (i.e. the ./dist3 image, or the VMM38b4-64bit-image1-ikp.
> 1.cs change set).
>
> The actual primitive puts the Smalltalk word size in VM parameter  
> 40. This would be from any version of VMM since VMMaker-tpr.22.  
> Prior to this (VMMaker-tpr.21 and earlier), the slot assignment is 27.
>
> If an image running on a recent VM asks for the value of  
> vmParameterAt: 27, it is going to get the "number of mark loop  
> iterated for current IGC/FGC (read-only) includes ALL marking".  
> This is an interesting value, but perhaps not an optimal thing to  
> use as the wordSize for object memory.
>
> If I change SystemDictionary>>wordSize to use vmParameter at: 40,  
> then all is well.
>
> So there must be some image side update to Dan's work that is  
> supposed to go along with the change from VMMaker-tpr.21 to VMMaker-
> tpr.22. Actually it's just the small update to  
> SystemDictionary>>wordSize, but who knows if I'm missing something  
> else. My guess would be that it's something that was included in  
> the image side part of your (John's) object memory instrumentation  
> that just has not been coordinated with the VMM side?
>
> Thanks to Tim and John for the help; this lays the mystery to rest.
>
> Dave
>
>
> On Thu, Aug 04, 2005 at 09:50:18AM -0700, John M McIntosh wrote:
>
>> Ok, perhaps you can summarize.
>>
>> Are you saying if you take a 3.8 base image then apply a tpr.21 and
>> build a VM that it works fine with the 3.8 base image?
>>
>> And If you take Ian's ./dist3 image and run that then it fails using
>> the new VM?
>>
>> I will note that the call to get VMMparameter data has different
>> slots from the one Ian originally submitted.
>> You should check that the ./dist3 image has
>>
>> wordSize
>>      "Answer the size (in bytes) of an object pointer."
>>      "Smalltalk wordSize"
>>
>>      ^[SmalltalkImage current vmParameterAt: 40] on: Error do: [4]
>>
>>
>> primitiveVMParameter
>>      "Behaviour depends on argument count:
>>          0 args:    return an Array of VM parameter values;
>>          1 arg:    return the indicated VM parameter;
>>          2 args:    set the VM indicated parameter.
>>      VM parameters are numbered as follows:
>>          1    end of old-space (0-based, read-only)
>>          2    end of young-space (read-only)
>>          3    end of memory (read-only)
>>          4    allocationCount (read-only)
>>          5    allocations between GCs (read-write)
>>          6    survivor count tenuring threshold (read-write)
>>          7    full GCs since startup (read-only)
>>          8    total milliseconds in full GCs since startup (read-
>> only)
>>          9    incremental GCs since startup (read-only)
>>          10    total milliseconds in incremental GCs since startup
>> (read-only)
>>          11    tenures of surving objects since startup (read-only)
>>          12-20 specific to the translating VM
>>          21    root table size (read-only)
>>          22    root table overflows since startup (read-only)
>>          23    bytes of extra memory to reserve for VM buffers,
>> plugins, etc.
>>          24    memory threshold above which shrinking object  
>> memory (rw)
>>          25    memory headroom when growing object memory (rw)
>>          26  interruptChecksEveryNms - force an ioProcessEvents every
>> N milliseconds, in case the image  is not calling getNextEvent often
>> (rw)
>>          27    number of times mark loop iterated for current IGC/FGC
>> (read-only) includes ALL marking
>>          28    number of times sweep loop iterated  for current IGC/
>> FGC (read-only)
>>          29    number of times make forward loop iterated for current
>> IGC/FGC (read-only)
>>          30    number of times compact move loop iterated for current
>> IGC/FGC (read-only)
>>          31    number of grow memory requests (read-only)
>>          32    number of shrink memory requests (read-only)
>>          33    number of root table entries used for current IGC/FGC
>> (read-only)
>>          34    number of allocations done before current IGC/FGC
>> (read-only)
>>          35    number of survivor objects after current IGC/FGC  
>> (read-
>> only)
>>          36  millisecond clock when current IGC/FGC completed (read-
>> only)
>>          37  number of marked objects for Roots of the world, not
>> including Root Table entries for current IGC/FGC (read-only)
>>          38  milliseconds taken by current IGC  (read-only)
>>          39  Number of finalization signals for Weak Objects pending
>> when current IGC/FGC completed (read-only)
>>          40 BytesPerWord for this image
>>
>>
>>
>> On 4-Aug-05, at 4:20 AM, David T. Lewis wrote:
>>
>>
>>> On Wed, Aug 03, 2005 at 11:00:12PM -0700, John M McIntosh wrote:
>>>
>>>
>>>> I think if you look the johnmci & ar's GC instrumentation and weak
>>>> pointer changes do not consider the
>>>> width 32 versus 64 and originally used hard coded numbers to  
>>>> indicate
>>>> the size of a word or header.
>>>> Later this was changed to use constants.
>>>>
>>>>
>>>
>>> Yes, that's right. But I've tested VMM versions with and without
>>> the constants (Tim put these in in the very next version of VMM as
>>> I recall), so that does not appear to be related to the problem.
>>> Also note that I'm building 32 bit VMs on 32 bit hardware for all
>>> the results I've reported.
>>>
>>> So to summarize: The problem first appears as of VMMaker-tpr.22,  
>>> which
>>> adds the johnmci & ar's GC instrumentation and weak pointer changes.
>>> The combination of VMMaker-tpr.21 plus ar's changes alone are not
>>> sufficient to trigger the problem.  The problem shows up when  
>>> running
>>> an image from ./dist3 (i.e. an image that has had the 64 bit image
>>> changes added), but not on other images such as Squeak3.9a-6472.
>>> All tests were done on 32 bit Intel Linux.
>>>
>>> Dave
>>>
>>>
>>>
>>>
>>
>> --
>> =====================================================================
>> ===
>> ===
>> John M. McIntosh <[hidden email]> 1-800-477-2659
>> Corporate Smalltalk Consulting Ltd.  http://
>> www.smalltalkconsulting.com
>> =====================================================================
>> ===
>> ===
>>
>
>