Encoding problem

> We noticed because the text area used to edit pier documents
> became unusable.
>
> Stephan
>
>
>
> _______________________________________________
> seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

>
> On 20 May 2014, at 12:47, Stephan Eggermont <[hidden email]> wrote:
>
>> We have an issue in default Pier with rendering/encoding.
>> Did I miss setting encoding somewhere?
>>
>> If we enter in WAEncodingFunctionalTest the following three characters:
>> ->ë
>> the text input field removes the -.
>
> I can confirm the problem. This is really weird.
>
> Apparently you need a non-ascii character preceded by an xml-unsafe one:
>
> abc<é
> aaa&ç
>
> And when the characters are printed one by one wrapped in a span they all show !
>
>> We noticed because the text area used to edit pier documents
>> became unusable.
>>
>> Stephan
>>
>>
>>
>> _______________________________________________
>> seaside mailing list
>> [hidden email]
>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>
> _______________________________________________
> seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

> What version of Seaside are you guys running?
>
> On 20 May 2014, at 13:19, Sven Van Caekenberghe <[hidden email]> wrote:
>
>>
>> On 20 May 2014, at 12:47, Stephan Eggermont <[hidden email]> wrote:
>>
>>> We have an issue in default Pier with rendering/encoding.
>>> Did I miss setting encoding somewhere?
>>>
>>> If we enter in WAEncodingFunctionalTest the following three characters:
>>> ->ë
>>> the text input field removes the -.
>>
>> I can confirm the problem. This is really weird.
>>
>> Apparently you need a non-ascii character preceded by an xml-unsafe one:
>>
>> abc<é
>> aaa&ç
>>
>> And when the characters are printed one by one wrapped in a span they all show !
>>
>>> We noticed because the text area used to edit pier documents
>>> became unusable.
>>>
>>> Stephan
>>>
>>>
>>>
>>> _______________________________________________
>>> seaside mailing list
>>> [hidden email]
>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>>
>> _______________________________________________
>> seaside mailing list
>> [hidden email]
>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>
> _______________________________________________
> seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

>
> Johan Brichau-2 wrote
>> What version of Seaside are you guys running?
>>
>> On 20 May 2014, at 13:19, Sven Van Caekenberghe &lt;
>
>> sven@
>
>> &gt; wrote:
>>
>>>
>>> On 20 May 2014, at 12:47, Stephan Eggermont &lt;
>
>> stephan@
>
>> &gt; wrote:
>>>
>>>> We have an issue in default Pier with rendering/encoding.
>>>> Did I miss setting encoding somewhere?
>>>>
>>>> If we enter in WAEncodingFunctionalTest the following three characters:
>>>> ->ë
>>>> the text input field removes the -.
>>>
>>> I can confirm the problem. This is really weird.
>>>
>>> Apparently you need a non-ascii character preceded by an xml-unsafe one:
>>>
>>> abc
>> <é
>>>
>> aaa&ç
>>>
>>> And when the characters are printed one by one wrapped in a span they all
>>> show !
>>>
>>>> We noticed because the text area used to edit pier documents
>>>> became unusable.
>>>>
>>>> Stephan
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> seaside mailing list
>>>>
>
>> seaside@.squeakfoundation
>
>>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>>>
>>> _______________________________________________
>>> seaside mailing list
>>>
>
>> seaside@.squeakfoundation
>
>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>>
>> _______________________________________________
>> seaside mailing list
>
>> seaside@.squeakfoundation
>
>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>
>
>
>
>
> --
> View this message in context: http://forum.world.st/Encoding-problem-tp4759665p4759711.html
> Sent from the Seaside General mailing list archive at Nabble.com.
> _______________________________________________
> seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

>
> On 20 May 2014, at 17:19, Paul DeBruicker <[hidden email]> wrote:
>
>> On Pharo 1.4, with Seaside 3.1, there is no error.
>>
>> I have not tried with Pier loaded.  Should I?
>
> No, it happens without Pier being loaded.
>
>>
>> Johan Brichau-2 wrote
>>> What version of Seaside are you guys running?
>>>
>>> On 20 May 2014, at 13:19, Sven Van Caekenberghe &lt;
>>
>>> sven@
>>
>>> &gt; wrote:
>>>
>>>>
>>>> On 20 May 2014, at 12:47, Stephan Eggermont &lt;
>>
>>> stephan@
>>
>>> &gt; wrote:
>>>>
>>>>> We have an issue in default Pier with rendering/encoding.
>>>>> Did I miss setting encoding somewhere?
>>>>>
>>>>> If we enter in WAEncodingFunctionalTest the following three characters:
>>>>> ->ë
>>>>> the text input field removes the -.
>>>>
>>>> I can confirm the problem. This is really weird.
>>>>
>>>> Apparently you need a non-ascii character preceded by an xml-unsafe one:
>>>>
>>>> abc
>>> <é
>>>>
>>> aaa&ç
>>>>
>>>> And when the characters are printed one by one wrapped in a span they all
>>>> show !
>>>>
>>>>> We noticed because the text area used to edit pier documents
>>>>> became unusable.
>>>>>
>>>>> Stephan
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> seaside mailing list
>>>>>
>>
>>> seaside@.squeakfoundation
>>
>>>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>>>>
>>>> _______________________________________________
>>>> seaside mailing list
>>>>
>>
>>> seaside@.squeakfoundation
>>
>>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>>>
>>> _______________________________________________
>>> seaside mailing list
>>
>>> seaside@.squeakfoundation
>>
>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>>
>>
>>
>>
>>
>> --
>> View this message in context: http://forum.world.st/Encoding-problem-tp4759665p4759711.html
>> Sent from the Seaside General mailing list archive at Nabble.com.
>> _______________________________________________
>> seaside mailing list
>> [hidden email]
>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>
> _______________________________________________
> seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

> Thanks for reporting, I created an issue for it: https://code.google.com/p/seaside/issues/detail?id=792
>
> There is something going wrong in the WAPharoXmlEncoder in combination with a GRPharoUtf8EncodedStream:
>
> |encoder stream |
> stream := GRPharoUtf8CodecStream on: (WriteStream on: String new) converter: UTF8TextConverter new.
> encoder := WAPharoXmlEncoder on: stream.
> encoder nextPutAllFast: '->ë'.
> stream contents
>
> The above works correctly if the stream is a normal WriteStream
>
> Getting too late now to dive into it...
>
> Johan
>
> On 20 May 2014, at 19:36, Sven Van Caekenberghe <[hidden email]> wrote:
>
>>
>> On 20 May 2014, at 17:19, Paul DeBruicker <[hidden email]> wrote:
>>
>>> On Pharo 1.4, with Seaside 3.1, there is no error.
>>>
>>> I have not tried with Pier loaded.  Should I?
>>
>> No, it happens without Pier being loaded.
>>
>>>
>>> Johan Brichau-2 wrote
>>>> What version of Seaside are you guys running?
>>>>
>>>> On 20 May 2014, at 13:19, Sven Van Caekenberghe &lt;
>>>
>>>> sven@
>>>
>>>> &gt; wrote:
>>>>
>>>>>
>>>>> On 20 May 2014, at 12:47, Stephan Eggermont &lt;
>>>
>>>> stephan@
>>>
>>>> &gt; wrote:
>>>>>
>>>>>> We have an issue in default Pier with rendering/encoding.
>>>>>> Did I miss setting encoding somewhere?
>>>>>>
>>>>>> If we enter in WAEncodingFunctionalTest the following three characters:
>>>>>> ->ë
>>>>>> the text input field removes the -.
>>>>>
>>>>> I can confirm the problem. This is really weird.
>>>>>
>>>>> Apparently you need a non-ascii character preceded by an xml-unsafe one:
>>>>>
>>>>> abc
>>>> <é
>>>>>
>>>> aaa&ç
>>>>>
>>>>> And when the characters are printed one by one wrapped in a span they all
>>>>> show !
>>>>>
>>>>>> We noticed because the text area used to edit pier documents
>>>>>> became unusable.
>>>>>>
>>>>>> Stephan
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> seaside mailing list
>>>>>>
>>>
>>>> seaside@.squeakfoundation
>>>
>>>>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>>>>>
>>>>> _______________________________________________
>>>>> seaside mailing list
>>>>>
>>>
>>>> seaside@.squeakfoundation
>>>
>>>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>>>>
>>>> _______________________________________________
>>>> seaside mailing list
>>>
>>>> seaside@.squeakfoundation
>>>
>>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>>>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context: http://forum.world.st/Encoding-problem-tp4759665p4759711.html
>>> Sent from the Seaside General mailing list archive at Nabble.com.
>>> _______________________________________________
>>> seaside mailing list
>>> [hidden email]
>>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>>
>> _______________________________________________
>> seaside mailing list
>> [hidden email]
>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>
> _______________________________________________
> seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

> The error is in
>
> GRPharoUtf8CodecStream>>#greaseNext:putAllFast:startingAt:
>
> the first escape clause in the whileFalse computes a wrong first argument to #greaseNext:putAllFast:startingAt:
>
> however, it is a bit hard to say what the correct one should be, I will try with
>
> startIndex + anInteger - lastIndex
>
> I hope there are enough tests.

>
> On 20 May 2014, at 23:15, Sven Van Caekenberghe <[hidden email]> wrote:
>
>> The error is in
>>
>> GRPharoUtf8CodecStream>>#greaseNext:putAllFast:startingAt:
>>
>> the first escape clause in the whileFalse computes a wrong first argument to #greaseNext:putAllFast:startingAt:
>>
>> however, it is a bit hard to say what the correct one should be, I will try with
>>
>> startIndex + anInteger - lastIndex
>>
>> I hope there are enough tests.
>
> I added an extra assert
>
> self assert: 'ab', umlaut next: 1 startingAt: 1 gives:  'a'.
>
> to GRPharoCodecTest>>#testGreaseNextPutAllStartingAt so that it now reads:
>
>   | umlaut encodedUmlaut |
> umlaut := String with: (Character value: 228).
> encodedUmlaut := String with: (Character value: 195) with: (Character value: 164).
> self assert: 'ab' next: 1 startingAt: 1 gives:  'a'.
> self assert: 'a', umlaut, 'b' next: 1 startingAt: 1 gives:  'a'.
> self assert: 'ab', umlaut next: 1 startingAt: 1 gives:  'a'.
> self assert: 'a', umlaut, 'b' next: 2 startingAt: 1 gives:  'a', encodedUmlaut.
> self assert: 'a', umlaut, 'b' next: 1 startingAt: 2 gives:  encodedUmlaut.
> self assert: 'a', umlaut, 'b' next: 2 startingAt: 2 gives:  encodedUmlaut, 'b'.
> self assert: 'a', umlaut, umlaut next: 2 startingAt: 2 gives:  encodedUmlaut, encodedUmlaut.
> self assert: 'ab', umlaut, 'b', umlaut next: 3 startingAt: 2 gives:  'b', encodedUmlaut, 'b'
>
> it fails before the proposed change, it succeeds after it - and the functional test now passes.
>
> Final code of GRPharoUtf8CodecStream>>#greaseNext:putAllFast:startingAt:
>
> | lastIndex nextIndex |
> lastIndex := startIndex.
> nextIndex := ByteString findFirstInString: aByteString inSet: Latin1ToUtf8Map startingAt: lastIndex.
> nextIndex = 0 ifTrue: [ ^ stream greaseNext: anInteger putAll: aByteString startingAt: startIndex ].
> [
> nextIndex >= (startIndex + anInteger) ifTrue: [
> ^ stream greaseNext: startIndex + anInteger - lastIndex putAll: aByteString startingAt: lastIndex ].
> nextIndex > lastIndex ifTrue: [
> stream greaseNext: nextIndex - lastIndex putAll: aByteString startingAt: lastIndex ].
> stream nextPutAll: (Latin1ToUtf8Encodings at: (aByteString byteAt: nextIndex) + 1).
> lastIndex := nextIndex + 1.
> nextIndex := ByteString findFirstInString: aByteString inSet: Latin1ToUtf8Map startingAt: lastIndex.
> (nextIndex = 0 or: [ nextIndex >= (startIndex + anInteger) ]) ] whileFalse.
> lastIndex >= (startIndex + anInteger) ifFalse: [
> stream greaseNext: startIndex + anInteger - lastIndex putAll: aByteString startingAt: lastIndex ]
>
> But this certainly needs another pair of eyes.
>
> Sven
>
> _______________________________________________
> seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

>
> On 20 May 2014, at 23:15, Sven Van Caekenberghe <[hidden email]> wrote:
>
>> The error is in
>>
>> GRPharoUtf8CodecStream>>#greaseNext:putAllFast:startingAt:
>>
>> the first escape clause in the whileFalse computes a wrong first argument to #greaseNext:putAllFast:startingAt:
>>
>> however, it is a bit hard to say what the correct one should be, I will try with
>>
>> startIndex + anInteger - lastIndex
>>
>> I hope there are enough tests.
>
> I added an extra assert
>
>         self assert: 'ab', umlaut next: 1 startingAt: 1 gives:  'a'.
>
> to GRPharoCodecTest>>#testGreaseNextPutAllStartingAt so that it now reads:
>
>         | umlaut encodedUmlaut |
>         umlaut := String with: (Character value: 228).
>         encodedUmlaut := String with: (Character value: 195) with: (Character value: 164).
>         self assert: 'ab' next: 1 startingAt: 1 gives:  'a'.
>         self assert: 'a', umlaut, 'b' next: 1 startingAt: 1 gives:  'a'.
>         self assert: 'ab', umlaut next: 1 startingAt: 1 gives:  'a'.
>         self assert: 'a', umlaut, 'b' next: 2 startingAt: 1 gives:  'a', encodedUmlaut.
>         self assert: 'a', umlaut, 'b' next: 1 startingAt: 2 gives:  encodedUmlaut.
>         self assert: 'a', umlaut, 'b' next: 2 startingAt: 2 gives:  encodedUmlaut, 'b'.
>         self assert: 'a', umlaut, umlaut next: 2 startingAt: 2 gives:  encodedUmlaut, encodedUmlaut.
>         self assert: 'ab', umlaut, 'b', umlaut next: 3 startingAt: 2 gives:  'b', encodedUmlaut, 'b'
>
> it fails before the proposed change, it succeeds after it - and the functional test now passes.