[5.2a] styled text in a class comment spawns endless subscript out-of-bounds

Sorry:  I should have said my hand-renaming the class in the .cs *may* be the cause of what you found, but not that it necessarily *is* the cause of what you found.

Thanks,

Tim

On Jul 4, 2018, at 10:59 PM, Tim Johnson wrote:

Hi Bob & David,

Thanks for all your help with this.

The runs are short because in my original fileOut, the class was named MoodleClient, but I hand-edited the .cs to rename it FooClient and to remove all my proprietary methods I'm not yet ready to share.

I did try to explain this in my original post which is now in the distant realm of the thread.  :)

Best,

Tim

On Jul 4, 2018, at 3:55 PM, Bob Arning wrote:

A little more...

This problem may ultimately be due to something broken in the text being displayed. If I look at the funky class comment, it has these runs

a RunArray runs: #(1 35 18 20 3) values: {#() . {a TextFontChange font: 1} . #() . {a TextEmphasis code: 1} . #()}

The total length of the run sizes is 77, yet the string is only 70 bytes long.

If I copy that text out and paste into a workspace, the runs now look like:

a RunArray runs: #(1 35 18 16) values: {#() . {a TextFontChange font: 1} . #() . {a TextEmphasis code: 1}}

and the size of the runs matches the size of the string at 70. I wonder if other occurrences of this problem might have a similar cause.

On 7/4/18 4:20 PM, David T. Lewis wrote:

On Wed, Jul 04, 2018 at 03:46:16PM -0400, Bob Arning wrote:

Here is a side-by-side comparison with the styled version on the right 
and the same text unstyled on the left. Note the difference in runStopIndex

I see it now. Thanks!

Dave

On 7/4/18 2:49 PM, David T. Lewis wrote:

I tried reverting #scanCharactersFrom:to:in:rightX:stopConditions:kern:
to the earlier version (cmm 6/12/2010 11:52), but I still get failures
with cursor and mouse in the class comment text when browsing the FooClient
class. There must be something else going on here, and it is certainly
not obvious to me why it is happening only with the FooClient test
example.

Dave

On Wed, Jul 04, 2018 at 02:10:00PM -0400, Bob Arning wrote:

perhaps this will help:

This version does not seem to exhibit the error and it limits stopIndex
to the size of the string

!CharacterScanner methodsFor: 'scanning' stamp: 'cmm 6/12/2010 11:52'!
scanCharactersFrom: startIndex to: stopIndex in: sourceString rightX:
rightX stopConditions: stops kern: kernDelta

?????? | startEncoding selector |
?????? (sourceString isByteString) ifTrue: [^ self
basicScanCharactersFrom: startIndex *to: (stopIndex min: sourceString
size)* in: sourceString rightX: rightX stopConditions: stops kern:
kernDelta.].

?????? (sourceString isWideString) ifTrue: [
?????? ?????? startIndex > stopIndex ifTrue: [lastIndex := stopIndex. ^
stops endOfRun].
?????? ?????? startEncoding :=?? (sourceString at: startIndex) 
leadingChar.
?????? ?????? selector := EncodedCharSet scanSelectorAt: startEncoding.
?????? ?????? ^ self perform: selector withArguments: (Array with:
startIndex with: stopIndex with: sourceString with: rightX with: stops
with: kernDelta).
?????? ].

?????? ^ stops endOfRun
! !

This version fails and does not limit the stopIndex.

!CharacterScanner methodsFor: 'scanning' stamp: 'nice 10/22/2013 20:49'!
scanCharactersFrom: startIndex to: stopIndex in: sourceString rightX:
rightX
?????? ^sourceString scanCharactersFrom: startIndex *to: stopIndex* with:
self rightX: rightX font: font! !


On 7/4/18 11:53 AM, David T. Lewis wrote:

On Wed, Jul 04, 2018 at 02:35:59PM +0200, karl ramberg wrote:

On Wed, Jul 4, 2018 at 11:10 AM K K Subbu [hidden email] wrote:

On Tuesday 03 July 2018 04:40 AM, Tim Johnson wrote:

1. File-in the .cs as attached*.
2. In a system browser, click on the class, then click on the [?] 
button
to view class comment.
3. Notice the styled text in the comment.
4. Click around the comment pane a bit.
5. Eventually, see "Error: subscript is out of bounds: 71" appear over
and over until you interrupt.

In step 4, it is sufficient to click once near the bottom of the pane
when the cursor changes from I-bar to arrow and then wait. This will
trigger the error, but the error window triggers itself recursively
around 50 times before stopping and displaying the windows.

I added an assert to catch it before recursion:

    self assert: lastIndex < sourceString size.

The assert window itself throws up another error, but that is a topic
for another thread ;-).

Regards .. Subbu

Hi,
See class comment i NewParagraph for a hint of what happens.
A empty lineSpan 71 to: 70 is added to the end of the paragraph, when 
you
double click below the text.
When the character scanner looks for characters outside of the string at
index 71 it cause a recursion.

Best,
Karl

Good tiip to look at the NewParagraph class comment, there is apparently
some left over messiness in here somewhere.

Another way to trigger the bug using the test case with FooClient is
to position the cursor in the comment pane and use keyboard right cursor
to move to the end of the comment. This causes the same bounds error but
takes a different path through the code leading to the error.

Looking through the stack in a debugger after triggering the error with
the mouse, here are a few notes (but no conclusions):

On mouseUp: we enter Editor>>selectWord then call
Editor>>selectWordLeftDelimiters:rightDelimiters:
which has this:

	"Select the whole text when clicking before first or after last
	character"
	(here > string size or: [here < 2]) ifTrue: [^self selectFrom: 1 to:
	string size].

where string is the 70 character String derived from the Text.

At this point we are trying to select from 1 to 70 (the whole string for
the entire text) in TextEditor>>selectFrom:to:

This calls Editor>>selectInvisiblyFrom:to: and here the stop index is
set past the size of the string:

  selectInvisiblyFrom: start to: stop
     "Select the designated characters, inclusive.  Make no visual
     changes."

     self markIndex: start pointIndex: stop + 1

From this point we are trying to go from index 1 to index 71 over a 
string
of size 70, and eventually end up with the out of bounds exception.

A clue that this might be accumulation of kludges: Looking at senders of
selectInvisiblyFrom:to: (such as TextEditor>>addText:event:) shows a 
number
of methods that subtract 1 from the stop index when before calling the
method
that adds 1 back to the stop index. This certainly does not have a good
smell.

It is not clear to me why selectInvisiblyFrom:to: would be adding one to
the
stop index, but I see that the method stamp is jmv 5/27/2011, which
suggests
to me that we should take a look at Cuis to figure out how this is 
intended
to function. Cuis does not show any cases of senders of
selectInvisiblyFrom:to:
that subtract 1 from the stop index, and the Cuis implementation of
selectInvisiblyFrom:to: is that same as Squeak.

Dave

It's good to remember what startIndex/startBlock/selectionStart and stopIndex/stopBlock/selectionStop are used for. If you have a paragraph with two characters 'ab'

...and the 'a' is selected, then this is what the paragraph is thinking

selectionStart:     a CharacterBlock with index 1 and character $a and rectangle 3@0 corner: [hidden email]
 in a TextLine 1 to: 2
selectionStop:     a CharacterBlock with index 2 and character $b and rectangle 11@0 corner: [hidden email]
 in a TextLine 1 to: 2

...if the cursor is between 'a' and 'b' then this happens

selectionStart:     a CharacterBlock with index 2 and character $b and rectangle 11@0 corner: [hidden email]
 in a TextLine 1 to: 2
selectionStop:     a CharacterBlock with index 2 and character $b and rectangle 11@0 corner: [hidden email]
 in a TextLine 1 to: 2

So, in the first case, it can highlight from the topLeft of selectionStart to the bottomLeft of selectionStop. In the second case, since the two are equal, it just displays the cursor along the left edge of either.

In either case, the stopIndex is one more than last selected character and

On Thu, Jul 05, 2018 at 02:51:58PM +0530, K K Subbu wrote:

On Wednesday 04 July 2018 09:23 PM, David T. Lewis wrote:

A clue that this might be accumulation of kludges: Looking at senders of
selectInvisiblyFrom:to: (such as TextEditor>>addText:event:) shows a number
of methods that subtract 1 from the stop index when before calling the 
method
that adds 1 back to the stop index. This certainly does not have a good 
smell.

Looks like different intervals for dealing with selections and 
substrings. I see that the names used are start/stop instead of 
startIndex/stopIndex. I found this consistent pattern in all methods 
that deal with selections.

  stop = (stopIndex - 1)
  stopIndex = (stop + 1)

Regular methods that deal with substrings like 
AbstractFont>>#widthOString... and BitBlt>>#primDisplayString... and the 
scanner methods iterate over aString with:

   startIndex to: stopIndex

while Editor>>selectionInterval extracts selections with startIndex to: 
(stopIndex-1).

Regards .. Subbu

Ah yes, that makes sense now. Thank you for the explanation.

Dave
 

A useful step would be to check this:

(Text allInstances select: [ :e | e size ~= e runs size]) explore

and see if it finds anything. If people did it periodically (like before and after loading new code from somewhere) we might know if/how these anomalies might be created.

Hi all,

I'm sorry if it seems like I may have wasted a dozen hours of your time on a simple problem of a hand-edited changeset.

However, I honorably swear that (a) this problem was happening to me before FooClient.cs even existed, and (b) this has happened to me in various images maybe 3-6 times over the past few years.  And I'm not too exotic with my images:  I just use Monticello and fileIn/fileOut.

I notice (off-hand) that Monticello actually strips formatting from class comments either during export or import, and changesets do not.

I would like to be able to provide instructions for getting a class into the problem condition, but I haven't found them yet.  I can provide my original class as a changeset without modification, but not for a few days, as I am away from that computer.  

I am trying to reproduce the situation without the need to share a changeset.  This may involve finding the exact source of the styled text which I pasted into the class comment.  (I am trying this in stock 5.1 image, in case improvements to the boilerplate text could have something to do with this.)

My efforts to construct a test case are being hampered, however.  I'll post separately.

Thanks again,

Tim

On Jul 5, 2018, at 4:17 AM, Bob Arning wrote:

Yes, it is the cause and the editor could be a bit more robust in the face of such puzzles. :-)

On 7/5/18 2:01 AM, Tim Johnson wrote:

Sorry:  I should have said my hand-renaming the class in the .cs *may* be the cause of what you found, but not that it necessarily *is* the cause of what you found.

Thanks,

Tim

On Jul 4, 2018, at 10:59 PM, Tim Johnson wrote:

Hi Bob & David,

Thanks for all your help with this.

The runs are short because in my original fileOut, the class was named MoodleClient, but I hand-edited the .cs to rename it FooClient and to remove all my proprietary methods I'm not yet ready to share.

I did try to explain this in my original post which is now in the distant realm of the thread.  :)

Best,

Tim

On Jul 4, 2018, at 3:55 PM, Bob Arning wrote:

A little more...

This problem may ultimately be due to something broken in the text being displayed. If I look at the funky class comment, it has these runs

a RunArray runs: #(1 35 18 20 3) values: {#() . {a TextFontChange font: 1} . #() . {a TextEmphasis code: 1} . #()}

The total length of the run sizes is 77, yet the string is only 70 bytes long.

If I copy that text out and paste into a workspace, the runs now look like:

a RunArray runs: #(1 35 18 16) values: {#() . {a TextFontChange font: 1} . #() . {a TextEmphasis code: 1}}

and the size of the runs matches the size of the string at 70. I wonder if other occurrences of this problem might have a similar cause.

On 7/4/18 4:20 PM, David T. Lewis wrote:

On Wed, Jul 04, 2018 at 03:46:16PM -0400, Bob Arning wrote:

Here is a side-by-side comparison with the styled version on the right 
and the same text unstyled on the left. Note the difference in runStopIndex

I see it now. Thanks!

Dave

On 7/4/18 2:49 PM, David T. Lewis wrote:

I tried reverting #scanCharactersFrom:to:in:rightX:stopConditions:kern:
to the earlier version (cmm 6/12/2010 11:52), but I still get failures
with cursor and mouse in the class comment text when browsing the FooClient
class. There must be something else going on here, and it is certainly
not obvious to me why it is happening only with the FooClient test
example.

Dave

On Wed, Jul 04, 2018 at 02:10:00PM -0400, Bob Arning wrote:

perhaps this will help:

This version does not seem to exhibit the error and it limits stopIndex
to the size of the string

!CharacterScanner methodsFor: 'scanning' stamp: 'cmm 6/12/2010 11:52'!
scanCharactersFrom: startIndex to: stopIndex in: sourceString rightX:
rightX stopConditions: stops kern: kernDelta

?????? | startEncoding selector |
?????? (sourceString isByteString) ifTrue: [^ self
basicScanCharactersFrom: startIndex *to: (stopIndex min: sourceString
size)* in: sourceString rightX: rightX stopConditions: stops kern:
kernDelta.].

?????? (sourceString isWideString) ifTrue: [
?????? ?????? startIndex > stopIndex ifTrue: [lastIndex := stopIndex. ^
stops endOfRun].
?????? ?????? startEncoding :=?? (sourceString at: startIndex) 
leadingChar.
?????? ?????? selector := EncodedCharSet scanSelectorAt: startEncoding.
?????? ?????? ^ self perform: selector withArguments: (Array with:
startIndex with: stopIndex with: sourceString with: rightX with: stops
with: kernDelta).
?????? ].

?????? ^ stops endOfRun
! !

This version fails and does not limit the stopIndex.

!CharacterScanner methodsFor: 'scanning' stamp: 'nice 10/22/2013 20:49'!
scanCharactersFrom: startIndex to: stopIndex in: sourceString rightX:
rightX
?????? ^sourceString scanCharactersFrom: startIndex *to: stopIndex* with:
self rightX: rightX font: font! !


On 7/4/18 11:53 AM, David T. Lewis wrote:

On Wed, Jul 04, 2018 at 02:35:59PM +0200, karl ramberg wrote:

On Wed, Jul 4, 2018 at 11:10 AM K K Subbu [hidden email] wrote:

On Tuesday 03 July 2018 04:40 AM, Tim Johnson wrote:

1. File-in the .cs as attached*.
2. In a system browser, click on the class, then click on the [?] 
button
to view class comment.
3. Notice the styled text in the comment.
4. Click around the comment pane a bit.
5. Eventually, see "Error: subscript is out of bounds: 71" appear over
and over until you interrupt.

In step 4, it is sufficient to click once near the bottom of the pane
when the cursor changes from I-bar to arrow and then wait. This will
trigger the error, but the error window triggers itself recursively
around 50 times before stopping and displaying the windows.

I added an assert to catch it before recursion:

    self assert: lastIndex < sourceString size.

The assert window itself throws up another error, but that is a topic
for another thread ;-).

Regards .. Subbu

Hi,
See class comment i NewParagraph for a hint of what happens.
A empty lineSpan 71 to: 70 is added to the end of the paragraph, when 
you
double click below the text.
When the character scanner looks for characters outside of the string at
index 71 it cause a recursion.

Best,
Karl

Good tiip to look at the NewParagraph class comment, there is apparently
some left over messiness in here somewhere.

Another way to trigger the bug using the test case with FooClient is
to position the cursor in the comment pane and use keyboard right cursor
to move to the end of the comment. This causes the same bounds error but
takes a different path through the code leading to the error.

Looking through the stack in a debugger after triggering the error with
the mouse, here are a few notes (but no conclusions):

On mouseUp: we enter Editor>>selectWord then call
Editor>>selectWordLeftDelimiters:rightDelimiters:
which has this:

	"Select the whole text when clicking before first or after last
	character"
	(here > string size or: [here < 2]) ifTrue: [^self selectFrom: 1 to:
	string size].

where string is the 70 character String derived from the Text.

At this point we are trying to select from 1 to 70 (the whole string for
the entire text) in TextEditor>>selectFrom:to:

This calls Editor>>selectInvisiblyFrom:to: and here the stop index is
set past the size of the string:

  selectInvisiblyFrom: start to: stop
     "Select the designated characters, inclusive.  Make no visual
     changes."

     self markIndex: start pointIndex: stop + 1

From this point we are trying to go from index 1 to index 71 over a 
string
of size 70, and eventually end up with the out of bounds exception.

A clue that this might be accumulation of kludges: Looking at senders of
selectInvisiblyFrom:to: (such as TextEditor>>addText:event:) shows a 
number
of methods that subtract 1 from the stop index when before calling the
method
that adds 1 back to the stop index. This certainly does not have a good
smell.

It is not clear to me why selectInvisiblyFrom:to: would be adding one to
the
stop index, but I see that the method stamp is jmv 5/27/2011, which
suggests
to me that we should take a look at Cuis to figure out how this is 
intended
to function. Cuis does not show any cases of senders of
selectInvisiblyFrom:to:
that subtract 1 from the stop index, and the Cuis implementation of
selectInvisiblyFrom:to: is that same as Squeak.

Dave

Well, that drops styling altogether if it is not exactly right. You could try to patch up the runs with:

aRunArray size > aString size ifTrue: [aRunArray := aRunArray copy from: 1 to aString size].

aRunArray size < aString size ifTrue: [aRunArray  addLast: {} times: aString size - aRunArray size].

On Thursday 05 July 2018 10:09 PM, Bob Arning wrote:

A useful step would be to check this:

(Text allInstances select: [ :e | e size ~= e runs size]) explore

and see if it finds anything. If people did it periodically (like before and after loading new code from somewhere) we might know if/how these anomalies might be created.

Text drops bad runs silently in:

setString: aString setRunsChecking: aRunArray
    ...
    aRunArray size = aString size ifFalse: [^ aString asText].
    runs := aRunArray

This method is called from PositionableStream>>#nextChunkText but not from #nextChunkTextFromStream: methods of converter used by MultiByteFileStream.

Should these methods be changed to check runs? or should we catch the error in MultiByteStream>>#nextChunkText post conversion?

Regards .. Subbu