[5.2a] styled text in a class comment spawns endless subscript out-of-bounds

perhaps this will help:

This version does not seem to exhibit the error and it limits stopIndex to the size of the string

!CharacterScanner methodsFor: 'scanning' stamp: 'cmm 6/12/2010 11:52'!
scanCharactersFrom: startIndex to: stopIndex in: sourceString rightX: rightX stopConditions: stops kern: kernDelta

    | startEncoding selector |
    (sourceString isByteString) ifTrue: [^ self basicScanCharactersFrom: startIndex to: (stopIndex min: sourceString size) in: sourceString rightX: rightX stopConditions: stops kern: kernDelta.].

    (sourceString isWideString) ifTrue: [
        startIndex > stopIndex ifTrue: [lastIndex := stopIndex. ^ stops endOfRun].
        startEncoding :=  (sourceString at: startIndex) leadingChar.
        selector := EncodedCharSet scanSelectorAt: startEncoding.
        ^ self perform: selector withArguments: (Array with: startIndex with: stopIndex with: sourceString with: rightX with: stops with: kernDelta).
    ].
   
    ^ stops endOfRun
! !

This version fails and does not limit the stopIndex.

!CharacterScanner methodsFor: 'scanning' stamp: 'nice 10/22/2013 20:49'!
scanCharactersFrom: startIndex to: stopIndex in: sourceString rightX: rightX
    ^sourceString scanCharactersFrom: startIndex to: stopIndex with: self rightX: rightX font: font! !

On Wed, Jul 04, 2018 at 02:35:59PM +0200, karl ramberg wrote:

On Wed, Jul 4, 2018 at 11:10 AM K K Subbu [hidden email] wrote:

On Tuesday 03 July 2018 04:40 AM, Tim Johnson wrote:

1. File-in the .cs as attached*.
2. In a system browser, click on the class, then click on the [?] button
to view class comment.
3. Notice the styled text in the comment.
4. Click around the comment pane a bit.
5. Eventually, see "Error: subscript is out of bounds: 71" appear over
and over until you interrupt.

In step 4, it is sufficient to click once near the bottom of the pane
when the cursor changes from I-bar to arrow and then wait. This will
trigger the error, but the error window triggers itself recursively
around 50 times before stopping and displaying the windows.

I added an assert to catch it before recursion:

    self assert: lastIndex < sourceString size.

The assert window itself throws up another error, but that is a topic
for another thread ;-).

Regards .. Subbu

Hi,
See class comment i NewParagraph for a hint of what happens.
A empty lineSpan 71 to: 70 is added to the end of the paragraph, when you
double click below the text.
When the character scanner looks for characters outside of the string at
index 71 it cause a recursion.

Best,
Karl

Good tiip to look at the NewParagraph class comment, there is apparently
some left over messiness in here somewhere.

Another way to trigger the bug using the test case with FooClient is
to position the cursor in the comment pane and use keyboard right cursor
to move to the end of the comment. This causes the same bounds error but
takes a different path through the code leading to the error.

Looking through the stack in a debugger after triggering the error with
the mouse, here are a few notes (but no conclusions):

On mouseUp: we enter Editor>>selectWord then call Editor>>selectWordLeftDelimiters:rightDelimiters:
which has this:

	"Select the whole text when clicking before first or after last character"
	(here > string size or: [here < 2]) ifTrue: [^self selectFrom: 1 to: string size].

where string is the 70 character String derived from the Text.

At this point we are trying to select from 1 to 70 (the whole string for
the entire text) in TextEditor>>selectFrom:to:

This calls Editor>>selectInvisiblyFrom:to: and here the stop index is
set past the size of the string:

  selectInvisiblyFrom: start to: stop
     "Select the designated characters, inclusive.  Make no visual changes."

     self markIndex: start pointIndex: stop + 1

From this point we are trying to go from index 1 to index 71 over a string
of size 70, and eventually end up with the out of bounds exception.

A clue that this might be accumulation of kludges: Looking at senders of
selectInvisiblyFrom:to: (such as TextEditor>>addText:event:) shows a number
of methods that subtract 1 from the stop index when before calling the method
that adds 1 back to the stop index. This certainly does not have a good smell.

It is not clear to me why selectInvisiblyFrom:to: would be adding one to the
stop index, but I see that the method stamp is jmv 5/27/2011, which suggests
to me that we should take a look at Cuis to figure out how this is intended
to function. Cuis does not show any cases of senders of selectInvisiblyFrom:to:
that subtract 1 from the stop index, and the Cuis implementation of
selectInvisiblyFrom:to: is that same as Squeak.

Dave

It's only happening sometimes because of the styled text:

   runLength := text runLengthFor: line first.
    lineStop := characterIndex    "scanning for index"
        ifNil: [ line last ].            "scanning for point"
    runStopIndex := lastIndex + (runLength - 1) min: lineStop.

In unstyled text, runLength is zero and so runStopIndex is effectively decremented

In styled text, runStopIndex remains one past the end of the string

===================================

Reverting to the earlier version may not work (I was running that in squeak 4) due to other changes, but this does seem to work:

    ^sourceString scanCharactersFrom: startIndex to: (stopIndex min: sourceString size) with: self rightX: rightX font: font

I tried reverting #scanCharactersFrom:to:in:rightX:stopConditions:kern:
to the earlier version (cmm 6/12/2010 11:52), but I still get failures
with cursor and mouse in the class comment text when browsing the FooClient
class. There must be something else going on here, and it is certainly
not obvious to me why it is happening only with the FooClient test
example.

Dave

On Wed, Jul 04, 2018 at 02:10:00PM -0400, Bob Arning wrote:

perhaps this will help:

This version does not seem to exhibit the error and it limits stopIndex 
to the size of the string

!CharacterScanner methodsFor: 'scanning' stamp: 'cmm 6/12/2010 11:52'!
scanCharactersFrom: startIndex to: stopIndex in: sourceString rightX: 
rightX stopConditions: stops kern: kernDelta

?????? | startEncoding selector |
?????? (sourceString isByteString) ifTrue: [^ self 
basicScanCharactersFrom: startIndex *to: (stopIndex min: sourceString 
size)* in: sourceString rightX: rightX stopConditions: stops kern: 
kernDelta.].

?????? (sourceString isWideString) ifTrue: [
?????? ?????? startIndex > stopIndex ifTrue: [lastIndex := stopIndex. ^ 
stops endOfRun].
?????? ?????? startEncoding :=?? (sourceString at: startIndex) leadingChar.
?????? ?????? selector := EncodedCharSet scanSelectorAt: startEncoding.
?????? ?????? ^ self perform: selector withArguments: (Array with: 
startIndex with: stopIndex with: sourceString with: rightX with: stops 
with: kernDelta).
?????? ].

?????? ^ stops endOfRun
! !

This version fails and does not limit the stopIndex.

!CharacterScanner methodsFor: 'scanning' stamp: 'nice 10/22/2013 20:49'!
scanCharactersFrom: startIndex to: stopIndex in: sourceString rightX: 
rightX
?????? ^sourceString scanCharactersFrom: startIndex *to: stopIndex* with: 
self rightX: rightX font: font! !


On 7/4/18 11:53 AM, David T. Lewis wrote:

On Wed, Jul 04, 2018 at 02:35:59PM +0200, karl ramberg wrote:

On Wed, Jul 4, 2018 at 11:10 AM K K Subbu [hidden email] wrote:

On Tuesday 03 July 2018 04:40 AM, Tim Johnson wrote:

1. File-in the .cs as attached*.
2. In a system browser, click on the class, then click on the [?] button
to view class comment.
3. Notice the styled text in the comment.
4. Click around the comment pane a bit.
5. Eventually, see "Error: subscript is out of bounds: 71" appear over
and over until you interrupt.

In step 4, it is sufficient to click once near the bottom of the pane
when the cursor changes from I-bar to arrow and then wait. This will
trigger the error, but the error window triggers itself recursively
around 50 times before stopping and displaying the windows.

I added an assert to catch it before recursion:

    self assert: lastIndex < sourceString size.

The assert window itself throws up another error, but that is a topic
for another thread ;-).

Regards .. Subbu

Hi,
See class comment i NewParagraph for a hint of what happens.
A empty lineSpan 71 to: 70 is added to the end of the paragraph, when you
double click below the text.
When the character scanner looks for characters outside of the string at
index 71 it cause a recursion.

Best,
Karl

Good tiip to look at the NewParagraph class comment, there is apparently
some left over messiness in here somewhere.

Another way to trigger the bug using the test case with FooClient is
to position the cursor in the comment pane and use keyboard right cursor
to move to the end of the comment. This causes the same bounds error but
takes a different path through the code leading to the error.

Looking through the stack in a debugger after triggering the error with
the mouse, here are a few notes (but no conclusions):

On mouseUp: we enter Editor>>selectWord then call 
Editor>>selectWordLeftDelimiters:rightDelimiters:
which has this:

	"Select the whole text when clicking before first or after last 
	character"
	(here > string size or: [here < 2]) ifTrue: [^self selectFrom: 1 to: 
	string size].

where string is the 70 character String derived from the Text.

At this point we are trying to select from 1 to 70 (the whole string for
the entire text) in TextEditor>>selectFrom:to:

This calls Editor>>selectInvisiblyFrom:to: and here the stop index is
set past the size of the string:

  selectInvisiblyFrom: start to: stop
     "Select the designated characters, inclusive.  Make no visual 
     changes."

     self markIndex: start pointIndex: stop + 1

From this point we are trying to go from index 1 to index 71 over a string
of size 70, and eventually end up with the out of bounds exception.

A clue that this might be accumulation of kludges: Looking at senders of
selectInvisiblyFrom:to: (such as TextEditor>>addText:event:) shows a number
of methods that subtract 1 from the stop index when before calling the 
method
that adds 1 back to the stop index. This certainly does not have a good 
smell.

It is not clear to me why selectInvisiblyFrom:to: would be adding one to 
the
stop index, but I see that the method stamp is jmv 5/27/2011, which 
suggests
to me that we should take a look at Cuis to figure out how this is intended
to function. Cuis does not show any cases of senders of 
selectInvisiblyFrom:to:
that subtract 1 from the stop index, and the Cuis implementation of
selectInvisiblyFrom:to: is that same as Squeak.

Dave

Here is a side-by-side comparison with the styled version on the right and the same text unstyled on the left. Note the difference in runStopIndex

I tried reverting #scanCharactersFrom:to:in:rightX:stopConditions:kern:
to the earlier version (cmm 6/12/2010 11:52), but I still get failures
with cursor and mouse in the class comment text when browsing the FooClient
class. There must be something else going on here, and it is certainly
not obvious to me why it is happening only with the FooClient test
example.

Dave

On Wed, Jul 04, 2018 at 02:10:00PM -0400, Bob Arning wrote:

perhaps this will help:

This version does not seem to exhibit the error and it limits stopIndex 
to the size of the string

!CharacterScanner methodsFor: 'scanning' stamp: 'cmm 6/12/2010 11:52'!
scanCharactersFrom: startIndex to: stopIndex in: sourceString rightX: 
rightX stopConditions: stops kern: kernDelta

?????? | startEncoding selector |
?????? (sourceString isByteString) ifTrue: [^ self 
basicScanCharactersFrom: startIndex *to: (stopIndex min: sourceString 
size)* in: sourceString rightX: rightX stopConditions: stops kern: 
kernDelta.].

?????? (sourceString isWideString) ifTrue: [
?????? ?????? startIndex > stopIndex ifTrue: [lastIndex := stopIndex. ^ 
stops endOfRun].
?????? ?????? startEncoding :=?? (sourceString at: startIndex) leadingChar.
?????? ?????? selector := EncodedCharSet scanSelectorAt: startEncoding.
?????? ?????? ^ self perform: selector withArguments: (Array with: 
startIndex with: stopIndex with: sourceString with: rightX with: stops 
with: kernDelta).
?????? ].

?????? ^ stops endOfRun
! !

This version fails and does not limit the stopIndex.

!CharacterScanner methodsFor: 'scanning' stamp: 'nice 10/22/2013 20:49'!
scanCharactersFrom: startIndex to: stopIndex in: sourceString rightX: 
rightX
?????? ^sourceString scanCharactersFrom: startIndex *to: stopIndex* with: 
self rightX: rightX font: font! !


On 7/4/18 11:53 AM, David T. Lewis wrote:

On Wed, Jul 04, 2018 at 02:35:59PM +0200, karl ramberg wrote:

On Wed, Jul 4, 2018 at 11:10 AM K K Subbu [hidden email] wrote:

On Tuesday 03 July 2018 04:40 AM, Tim Johnson wrote:

1. File-in the .cs as attached*.
2. In a system browser, click on the class, then click on the [?] button
to view class comment.
3. Notice the styled text in the comment.
4. Click around the comment pane a bit.
5. Eventually, see "Error: subscript is out of bounds: 71" appear over
and over until you interrupt.

In step 4, it is sufficient to click once near the bottom of the pane
when the cursor changes from I-bar to arrow and then wait. This will
trigger the error, but the error window triggers itself recursively
around 50 times before stopping and displaying the windows.

I added an assert to catch it before recursion:

    self assert: lastIndex < sourceString size.

The assert window itself throws up another error, but that is a topic
for another thread ;-).

Regards .. Subbu

Hi,
See class comment i NewParagraph for a hint of what happens.
A empty lineSpan 71 to: 70 is added to the end of the paragraph, when you
double click below the text.
When the character scanner looks for characters outside of the string at
index 71 it cause a recursion.

Best,
Karl

Good tiip to look at the NewParagraph class comment, there is apparently
some left over messiness in here somewhere.

Another way to trigger the bug using the test case with FooClient is
to position the cursor in the comment pane and use keyboard right cursor
to move to the end of the comment. This causes the same bounds error but
takes a different path through the code leading to the error.

Looking through the stack in a debugger after triggering the error with
the mouse, here are a few notes (but no conclusions):

On mouseUp: we enter Editor>>selectWord then call 
Editor>>selectWordLeftDelimiters:rightDelimiters:
which has this:

	"Select the whole text when clicking before first or after last 
	character"
	(here > string size or: [here < 2]) ifTrue: [^self selectFrom: 1 to: 
	string size].

where string is the 70 character String derived from the Text.

At this point we are trying to select from 1 to 70 (the whole string for
the entire text) in TextEditor>>selectFrom:to:

This calls Editor>>selectInvisiblyFrom:to: and here the stop index is
set past the size of the string:

  selectInvisiblyFrom: start to: stop
     "Select the designated characters, inclusive.  Make no visual 
     changes."

     self markIndex: start pointIndex: stop + 1

From this point we are trying to go from index 1 to index 71 over a string
of size 70, and eventually end up with the out of bounds exception.

A clue that this might be accumulation of kludges: Looking at senders of
selectInvisiblyFrom:to: (such as TextEditor>>addText:event:) shows a number
of methods that subtract 1 from the stop index when before calling the 
method
that adds 1 back to the stop index. This certainly does not have a good 
smell.

It is not clear to me why selectInvisiblyFrom:to: would be adding one to 
the
stop index, but I see that the method stamp is jmv 5/27/2011, which 
suggests
to me that we should take a look at Cuis to figure out how this is intended
to function. Cuis does not show any cases of senders of 
selectInvisiblyFrom:to:
that subtract 1 from the stop index, and the Cuis implementation of
selectInvisiblyFrom:to: is that same as Squeak.

Dave

A little more...

This problem may ultimately be due to something broken in the text being displayed. If I look at the funky class comment, it has these runs

a RunArray runs: #(1 35 18 20 3) values: {#() . {a TextFontChange font: 1} . #() . {a TextEmphasis code: 1} . #()}

The total length of the run sizes is 77, yet the string is only 70 bytes long.

If I copy that text out and paste into a workspace, the runs now look like:

a RunArray runs: #(1 35 18 16) values: {#() . {a TextFontChange font: 1} . #() . {a TextEmphasis code: 1}}

and the size of the runs matches the size of the string at 70. I wonder if other occurrences of this problem might have a similar cause.

On Wed, Jul 04, 2018 at 03:46:16PM -0400, Bob Arning wrote:

Here is a side-by-side comparison with the styled version on the right 
and the same text unstyled on the left. Note the difference in runStopIndex

I see it now. Thanks!

Dave

On 7/4/18 2:49 PM, David T. Lewis wrote:

I tried reverting #scanCharactersFrom:to:in:rightX:stopConditions:kern:
to the earlier version (cmm 6/12/2010 11:52), but I still get failures
with cursor and mouse in the class comment text when browsing the FooClient
class. There must be something else going on here, and it is certainly
not obvious to me why it is happening only with the FooClient test
example.

Dave

On Wed, Jul 04, 2018 at 02:10:00PM -0400, Bob Arning wrote:

perhaps this will help:

This version does not seem to exhibit the error and it limits stopIndex
to the size of the string

!CharacterScanner methodsFor: 'scanning' stamp: 'cmm 6/12/2010 11:52'!
scanCharactersFrom: startIndex to: stopIndex in: sourceString rightX:
rightX stopConditions: stops kern: kernDelta

?????? | startEncoding selector |
?????? (sourceString isByteString) ifTrue: [^ self
basicScanCharactersFrom: startIndex *to: (stopIndex min: sourceString
size)* in: sourceString rightX: rightX stopConditions: stops kern:
kernDelta.].

?????? (sourceString isWideString) ifTrue: [
?????? ?????? startIndex > stopIndex ifTrue: [lastIndex := stopIndex. ^
stops endOfRun].
?????? ?????? startEncoding :=?? (sourceString at: startIndex) 
leadingChar.
?????? ?????? selector := EncodedCharSet scanSelectorAt: startEncoding.
?????? ?????? ^ self perform: selector withArguments: (Array with:
startIndex with: stopIndex with: sourceString with: rightX with: stops
with: kernDelta).
?????? ].

?????? ^ stops endOfRun
! !

This version fails and does not limit the stopIndex.

!CharacterScanner methodsFor: 'scanning' stamp: 'nice 10/22/2013 20:49'!
scanCharactersFrom: startIndex to: stopIndex in: sourceString rightX:
rightX
?????? ^sourceString scanCharactersFrom: startIndex *to: stopIndex* with:
self rightX: rightX font: font! !


On 7/4/18 11:53 AM, David T. Lewis wrote:

On Wed, Jul 04, 2018 at 02:35:59PM +0200, karl ramberg wrote:

On Wed, Jul 4, 2018 at 11:10 AM K K Subbu [hidden email] wrote:

On Tuesday 03 July 2018 04:40 AM, Tim Johnson wrote:

1. File-in the .cs as attached*.
2. In a system browser, click on the class, then click on the [?] 
button
to view class comment.
3. Notice the styled text in the comment.
4. Click around the comment pane a bit.
5. Eventually, see "Error: subscript is out of bounds: 71" appear over
and over until you interrupt.

In step 4, it is sufficient to click once near the bottom of the pane
when the cursor changes from I-bar to arrow and then wait. This will
trigger the error, but the error window triggers itself recursively
around 50 times before stopping and displaying the windows.

I added an assert to catch it before recursion:

    self assert: lastIndex < sourceString size.

The assert window itself throws up another error, but that is a topic
for another thread ;-).

Regards .. Subbu

Hi,
See class comment i NewParagraph for a hint of what happens.
A empty lineSpan 71 to: 70 is added to the end of the paragraph, when 
you
double click below the text.
When the character scanner looks for characters outside of the string at
index 71 it cause a recursion.

Best,
Karl

Good tiip to look at the NewParagraph class comment, there is apparently
some left over messiness in here somewhere.

Another way to trigger the bug using the test case with FooClient is
to position the cursor in the comment pane and use keyboard right cursor
to move to the end of the comment. This causes the same bounds error but
takes a different path through the code leading to the error.

Looking through the stack in a debugger after triggering the error with
the mouse, here are a few notes (but no conclusions):

On mouseUp: we enter Editor>>selectWord then call
Editor>>selectWordLeftDelimiters:rightDelimiters:
which has this:

	"Select the whole text when clicking before first or after last
	character"
	(here > string size or: [here < 2]) ifTrue: [^self selectFrom: 1 to:
	string size].

where string is the 70 character String derived from the Text.

At this point we are trying to select from 1 to 70 (the whole string for
the entire text) in TextEditor>>selectFrom:to:

This calls Editor>>selectInvisiblyFrom:to: and here the stop index is
set past the size of the string:

  selectInvisiblyFrom: start to: stop
     "Select the designated characters, inclusive.  Make no visual
     changes."

     self markIndex: start pointIndex: stop + 1

From this point we are trying to go from index 1 to index 71 over a 
string
of size 70, and eventually end up with the out of bounds exception.

A clue that this might be accumulation of kludges: Looking at senders of
selectInvisiblyFrom:to: (such as TextEditor>>addText:event:) shows a 
number
of methods that subtract 1 from the stop index when before calling the
method
that adds 1 back to the stop index. This certainly does not have a good
smell.

It is not clear to me why selectInvisiblyFrom:to: would be adding one to
the
stop index, but I see that the method stamp is jmv 5/27/2011, which
suggests
to me that we should take a look at Cuis to figure out how this is 
intended
to function. Cuis does not show any cases of senders of
selectInvisiblyFrom:to:
that subtract 1 from the stop index, and the Cuis implementation of
selectInvisiblyFrom:to: is that same as Squeak.

Dave