Non-deterministic ombu errors

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Non-deterministic ombu errors

gcotelli

Hi,

I'm having some non-deterministic errors in the Travis builds, the error is always something like:

OmFileStoreReadingError: Reading entry from File @ pharo-local/ombu-sessions/Pharo.4p8buz6mww48y1noxrolgwkl1.ombu@17432579: Invalid utf8 input detected

If I retried the build it works without any change. I think it happens during the code loading phase.

Is anyone experiencing this issue?

Here's an extract of the travis log:

OmFileStoreReadingError: Reading entry from File @ pharo-local/ombu-sessions/Pharo.4p8buz6mww48y1noxrolgwkl1.ombu@17432579: Invalid utf8 input detected

[ :error |

(OmFileStoreReadingError

readingError: error

on: self fileReference

position: readStream position) signal ] in [ :readStream |

[ ^ aBlockClosure value: readStream ascii ]

on: Error

do: [ :error |

(OmFileStoreReadingError

readingError: error

on: self fileReference

position: readStream position) signal ] ] in OmBlockFileStore(OmFileStore)>>readEntriesWith: in Block: [ :error | ...

BlockClosure>>cull:

Context>>evaluateSignal:

Context>>handleSignal:

UTF8InvalidText(Exception)>>signal

UTF8InvalidText(Exception)>>signal:

UTF8TextConverter>>errorMalformedInput

UTF8TextConverter>>nextFromStream:

MultiByteFileStream>>next

MultiByteFileStream(PositionableStream)>>match:

MultiByteFileStream(PositionableStream)>>upToAll:

OmSTONEntryReader>>nextEntryPositionIfFound:ifNone:

OmBlockFileStore>>refreshNewBlocksFrom:

[ :readStream | self refreshNewBlocksFrom: readStream ] in OmBlockFileStore>>refreshEntryPositionsByLocalNameStartingAt:since: in Block: [ :readStream | self refreshNewBlocksFrom: readStr...etc...

[ ^ aBlockClosure value: readStream ascii ] in [ :readStream |

[ ^ aBlockClosure value: readStream ascii ]

on: Error

do: [ :error |

(OmFileStoreReadingError

readingError: error

on: self fileReference

position: readStream position) signal ] ] in OmBlockFileStore(OmFileStore)>>readEntriesWith: in Block: [ ^ aBlockClosure value: readStream ascii ]

BlockClosure>>on:do:

[ :readStream |

[ ^ aBlockClosure value: readStream ascii ]

on: Error

do: [ :error |

(OmFileStoreReadingError

readingError: error

on: self fileReference

position: readStream position) signal ] ] in OmBlockFileStore(OmFileStore)>>readEntriesWith: in Block: [ :readStream | ...

[ aBlock value: stream ] in FileReference(AbstractFileReference)>>readStreamDo: in Block: [ aBlock value: stream ]

BlockClosure>>ensure:

FileReference(AbstractFileReference)>>readStreamDo:

OmBlockFileStore(OmFileStore)>>readEntriesWith:

OmBlockFileStore>>refreshEntryPositionsByLocalNameStartingAt:since:

[ | initialPosition initialLocalName |

self entryBuffer isEmpty

ifTrue: [ ^ self ].

fileReference

writeStreamDo: [ :fileStream |

fileStream setToEnd.

initialPosition := fileStream position.

initialLocalName := self entryBuffer first value.

ZnBufferedWriteStream

on: fileStream

do: [ :aWriteStream |

| anEntryWriter |

anEntryWriter := self newEntryWriter.

[ self entryBuffer isEmpty ]

whileFalse: [ | next entry |

next := self entryBuffer removeFirst.

entry := next key.

"Write entry to file"

anEntryWriter on: aWriteStream nextEntryPut: entry ] ].

"In Linux it was necessary to explicitely flush the file stream"

fileStream flush.

lastStreamPosition := fileStream size ].

self

refreshEntryPositionsByLocalNameStartingAt: initialPosition

since: initialLocalName ] in OmBlockFileStore(OmFileStore)>>flushEntryBuffer in Block: [ | initialPosition initialLocalName |...

[ caught := true.

self wait.

blockValue := mutuallyExcludedBlock value ] in Semaphore>>critical: in Block: [ caught := true....

BlockClosure>>ensure:

Semaphore>>critical:

OmBlockFileStore(OmFileStore)>>critical:

OmBlockFileStore(OmFileStore)>>flushEntryBuffer

OmDeferrer>>sendMessage

[ self sendMessage ] in OmDeferrer>>flush in Block: [ self sendMessage ]

Reply | Threaded
Open this post in threaded view
|

Re: Non-deterministic ombu errors

Jan Blizničenko
Hello

I am having very similar errors from time to time when manually loading code
into Pharo from filetree (gitfiletree), but only in one of my projects which
contains non-ASCII characters like ěščřžýáíéůúťď. For the very same reason
(not able to find why sometimes the error is there and sometimes is not), I
did not report it yet, but it is quite annoying. Retrying loading the
package helps every time.

Jan


gcotelli wrote

> Hi,
>
> I'm having some non-deterministic errors in the Travis builds, the error
> is
> always something like:
>
> *OmFileStoreReadingError: Reading entry from File @
> pharo-local/ombu-sessions/Pharo.4p8buz6mww48y1noxrolgwkl1.ombu@17432579:
> Invalid utf8 input detected*
>
> If I retried the build it works without any change. I think it happens
> during the code loading phase.
>
> Is anyone experiencing this issue?
>
> Here's an extract of the travis log:





--
Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html

Reply | Threaded
Open this post in threaded view
|

Re: Non-deterministic ombu errors

Stephane Ducasse-3
Hi jan

let us know if you have a reproducible case so that we fix it.

Stef

On Sat, Sep 23, 2017 at 10:42 PM, Jan Blizničenko <[hidden email]> wrote:

> Hello
>
> I am having very similar errors from time to time when manually loading code
> into Pharo from filetree (gitfiletree), but only in one of my projects which
> contains non-ASCII characters like ěščřžýáíéůúťď. For the very same reason
> (not able to find why sometimes the error is there and sometimes is not), I
> did not report it yet, but it is quite annoying. Retrying loading the
> package helps every time.
>
> Jan
>
>
> gcotelli wrote
>> Hi,
>>
>> I'm having some non-deterministic errors in the Travis builds, the error
>> is
>> always something like:
>>
>> *OmFileStoreReadingError: Reading entry from File @
>> pharo-local/ombu-sessions/Pharo.4p8buz6mww48y1noxrolgwkl1.ombu@17432579:
>> Invalid utf8 input detected*
>>
>> If I retried the build it works without any change. I think it happens
>> during the code loading phase.
>>
>> Is anyone experiencing this issue?
>>
>> Here's an extract of the travis log:
>
>
>
>
>
> --
> Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html
>

Reply | Threaded
Open this post in threaded view
|

Re: Non-deterministic ombu errors

Sven Van Caekenberghe-2
In reply to this post by Jan Blizničenko

> On 23 Sep 2017, at 22:42, Jan Blizničenko <[hidden email]> wrote:
>
> Hello
>
> I am having very similar errors from time to time when manually loading code
> into Pharo from filetree (gitfiletree), but only in one of my projects which
> contains non-ASCII characters like ěščřžýáíéůúťď. For the very same reason
> (not able to find why sometimes the error is there and sometimes is not), I
> did not report it yet, but it is quite annoying. Retrying loading the
> package helps every time.

I think was reported before. I believe Ombu somewhere takes an arbitrary chunk of bytes out of an UTF-8 encoded file (like the last 100 of something like that) and then starts reading that. With UTF-8, a variable length encoding, that can be dangerous because you could end up in the middle of a character, not between encoded character boundaries. This would also explain the randomness, as it is highly dependent on the contents.

ZnUTF8Encoder>>#backOnStream: can be used to make sure you are really at the start of an encoded character.

> Jan
>
>
> gcotelli wrote
>> Hi,
>>
>> I'm having some non-deterministic errors in the Travis builds, the error
>> is
>> always something like:
>>
>> *OmFileStoreReadingError: Reading entry from File @
>> pharo-local/ombu-sessions/Pharo.4p8buz6mww48y1noxrolgwkl1.ombu@17432579:
>> Invalid utf8 input detected*
>>
>> If I retried the build it works without any change. I think it happens
>> during the code loading phase.
>>
>> Is anyone experiencing this issue?
>>
>> Here's an extract of the travis log:
>
>
>
>
>
> --
> Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html


Reply | Threaded
Open this post in threaded view
|

Re: Non-deterministic ombu errors

tinchodias


On Sun, Sep 24, 2017 at 5:49 AM, Sven Van Caekenberghe <[hidden email]> wrote:

> On 23 Sep 2017, at 22:42, Jan Blizničenko <[hidden email]> wrote:
>
> Hello
>
> I am having very similar errors from time to time when manually loading code
> into Pharo from filetree (gitfiletree), but only in one of my projects which
> contains non-ASCII characters like ěščřžýáíéůúťď. For the very same reason
> (not able to find why sometimes the error is there and sometimes is not), I
> did not report it yet, but it is quite annoying. Retrying loading the
> package helps every time.

I think was reported before. I believe Ombu somewhere takes an arbitrary chunk of bytes out of an UTF-8 encoded file (like the last 100 of something like that) and then starts reading that. With UTF-8, a variable length encoding, that can be dangerous because you could end up in the middle of a character, not between encoded character boundaries. This would also explain the randomness, as it is highly dependent on the contents.

ZnUTF8Encoder>>#backOnStream: can be used to make sure you are really at the start of an encoded character.


Hello

This issue was reported as:

https://pharo.fogbugz.com/f/cases/20112/Epicea-can-fail-while-reading-non-ascii-blocks

My apologies for the bug... I introduced that code as an i/o optimization but wasn't properly tested.

Thnks Sven, but I finally used stream's #basicNext (as Henrik suggested in fogbugz) which skips forward.


Martin

Reply | Threaded
Open this post in threaded view
|

Re: Non-deterministic ombu errors

Stephane Ducasse-3
tx martin.

On Mon, Sep 25, 2017 at 1:07 AM, Martin Dias <[hidden email]> wrote:

>
>
> On Sun, Sep 24, 2017 at 5:49 AM, Sven Van Caekenberghe <[hidden email]> wrote:
>>
>>
>> > On 23 Sep 2017, at 22:42, Jan Blizničenko <[hidden email]> wrote:
>> >
>> > Hello
>> >
>> > I am having very similar errors from time to time when manually loading
>> > code
>> > into Pharo from filetree (gitfiletree), but only in one of my projects
>> > which
>> > contains non-ASCII characters like ěščřžýáíéůúťď. For the very same
>> > reason
>> > (not able to find why sometimes the error is there and sometimes is
>> > not), I
>> > did not report it yet, but it is quite annoying. Retrying loading the
>> > package helps every time.
>>
>> I think was reported before. I believe Ombu somewhere takes an arbitrary
>> chunk of bytes out of an UTF-8 encoded file (like the last 100 of something
>> like that) and then starts reading that. With UTF-8, a variable length
>> encoding, that can be dangerous because you could end up in the middle of a
>> character, not between encoded character boundaries. This would also explain
>> the randomness, as it is highly dependent on the contents.
>>
>> ZnUTF8Encoder>>#backOnStream: can be used to make sure you are really at
>> the start of an encoded character.
>
>
>
> Hello
>
> This issue was reported as:
>
> https://pharo.fogbugz.com/f/cases/20112/Epicea-can-fail-while-reading-non-ascii-blocks
>
> My apologies for the bug... I introduced that code as an i/o optimization
> but wasn't properly tested.
>
> Thnks Sven, but I finally used stream's #basicNext (as Henrik suggested in
> fogbugz) which skips forward.
>
>
> Martin
>

Reply | Threaded
Open this post in threaded view
|

Re: Non-deterministic ombu errors

Jan Blizničenko
In reply to this post by tinchodias
I'm glad it has known reason then, thank you for looking into it, all I was
able to find out was that it relates to non-ascii characters and that build
on our production server fails because of that :)

Jan


tinchodias wrote
> On Sun, Sep 24, 2017 at 5:49 AM, Sven Van Caekenberghe &lt;

> sven@

> &gt; wrote:
>
>>
>> > On 23 Sep 2017, at 22:42, Jan Blizničenko &lt;

> bliznjan@.cvut

> &gt; wrote:
>> >
>> > Hello
>> >
>> > I am having very similar errors from time to time when manually loading
>> code
>> > into Pharo from filetree (gitfiletree), but only in one of my projects
>> which
>> > contains non-ASCII characters like ěščřžýáíéůúťď. For the very same
>> reason
>> > (not able to find why sometimes the error is there and sometimes is
>> not), I
>> > did not report it yet, but it is quite annoying. Retrying loading the
>> > package helps every time.
>>
>> I think was reported before. I believe Ombu somewhere takes an arbitrary
>> chunk of bytes out of an UTF-8 encoded file (like the last 100 of
>> something
>> like that) and then starts reading that. With UTF-8, a variable length
>> encoding, that can be dangerous because you could end up in the middle of
>> a
>> character, not between encoded character boundaries. This would also
>> explain the randomness, as it is highly dependent on the contents.
>>
>> ZnUTF8Encoder>>#backOnStream: can be used to make sure you are really at
>> the start of an encoded character.
>>
>
>
> Hello
>
> This issue was reported as:
>
> https://pharo.fogbugz.com/f/cases/20112/Epicea-can-fail-while-reading-non-ascii-blocks
>
> My apologies for the bug... I introduced that code as an i/o optimization
> but wasn't properly tested.
>
> Thnks Sven, but I finally used stream's #basicNext (as Henrik suggested in
> fogbugz) which skips forward.
>
>
> Martin





--
Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html

Reply | Threaded
Open this post in threaded view
|

Re: Non-deterministic ombu errors

tinchodias
Now I do not know the best way for you to get the new fixed version. It's too late to include it in P6.1, I guess, so... either add epicea into metacello configuration or wait until P7.


On Fri, Sep 29, 2017 at 5:34 AM, Jan Blizničenko <[hidden email]> wrote:
I'm glad it has known reason then, thank you for looking into it, all I was
able to find out was that it relates to non-ascii characters and that build
on our production server fails because of that :)

Jan


tinchodias wrote
> On Sun, Sep 24, 2017 at 5:49 AM, Sven Van Caekenberghe &lt;

> sven@

> &gt; wrote:
>
>>
>> > On 23 Sep 2017, at 22:42, Jan Blizničenko &lt;

> bliznjan@.cvut

> &gt; wrote:
>> >
>> > Hello
>> >
>> > I am having very similar errors from time to time when manually loading
>> code
>> > into Pharo from filetree (gitfiletree), but only in one of my projects
>> which
>> > contains non-ASCII characters like ěščřžýáíéůúťď. For the very same
>> reason
>> > (not able to find why sometimes the error is there and sometimes is
>> not), I
>> > did not report it yet, but it is quite annoying. Retrying loading the
>> > package helps every time.
>>
>> I think was reported before. I believe Ombu somewhere takes an arbitrary
>> chunk of bytes out of an UTF-8 encoded file (like the last 100 of
>> something
>> like that) and then starts reading that. With UTF-8, a variable length
>> encoding, that can be dangerous because you could end up in the middle of
>> a
>> character, not between encoded character boundaries. This would also
>> explain the randomness, as it is highly dependent on the contents.
>>
>> ZnUTF8Encoder>>#backOnStream: can be used to make sure you are really at
>> the start of an encoded character.
>>
>
>
> Hello
>
> This issue was reported as:
>
> https://pharo.fogbugz.com/f/cases/20112/Epicea-can-fail-while-reading-non-ascii-blocks
>
> My apologies for the bug... I introduced that code as an i/o optimization
> but wasn't properly tested.
>
> Thnks Sven, but I finally used stream's #basicNext (as Henrik suggested in
> fogbugz) which skips forward.
>
>
> Martin






Reply | Threaded
Open this post in threaded view
|

Re: Non-deterministic ombu errors

Stephane Ducasse-3
Martin

if you publish a fix for 6.1 we will port it back because this is important.

Stef

On Fri, Sep 29, 2017 at 3:55 PM, Martin Dias <[hidden email]> wrote:

> Now I do not know the best way for you to get the new fixed version. It's
> too late to include it in P6.1, I guess, so... either add epicea into
> metacello configuration or wait until P7.
>
>
>
> On Fri, Sep 29, 2017 at 5:34 AM, Jan Blizničenko <[hidden email]>
> wrote:
>>
>> I'm glad it has known reason then, thank you for looking into it, all I
>> was
>> able to find out was that it relates to non-ascii characters and that
>> build
>> on our production server fails because of that :)
>>
>> Jan
>>
>>
>> tinchodias wrote
>> > On Sun, Sep 24, 2017 at 5:49 AM, Sven Van Caekenberghe &lt;
>>
>> > sven@
>>
>> > &gt; wrote:
>> >
>> >>
>> >> > On 23 Sep 2017, at 22:42, Jan Blizničenko &lt;
>>
>> > bliznjan@.cvut
>>
>> > &gt; wrote:
>> >> >
>> >> > Hello
>> >> >
>> >> > I am having very similar errors from time to time when manually
>> >> > loading
>> >> code
>> >> > into Pharo from filetree (gitfiletree), but only in one of my
>> >> > projects
>> >> which
>> >> > contains non-ASCII characters like ěščřžýáíéůúťď. For the very same
>> >> reason
>> >> > (not able to find why sometimes the error is there and sometimes is
>> >> not), I
>> >> > did not report it yet, but it is quite annoying. Retrying loading the
>> >> > package helps every time.
>> >>
>> >> I think was reported before. I believe Ombu somewhere takes an
>> >> arbitrary
>> >> chunk of bytes out of an UTF-8 encoded file (like the last 100 of
>> >> something
>> >> like that) and then starts reading that. With UTF-8, a variable length
>> >> encoding, that can be dangerous because you could end up in the middle
>> >> of
>> >> a
>> >> character, not between encoded character boundaries. This would also
>> >> explain the randomness, as it is highly dependent on the contents.
>> >>
>> >> ZnUTF8Encoder>>#backOnStream: can be used to make sure you are really
>> >> at
>> >> the start of an encoded character.
>> >>
>> >
>> >
>> > Hello
>> >
>> > This issue was reported as:
>> >
>> >
>> > https://pharo.fogbugz.com/f/cases/20112/Epicea-can-fail-while-reading-non-ascii-blocks
>> >
>> > My apologies for the bug... I introduced that code as an i/o
>> > optimization
>> > but wasn't properly tested.
>> >
>> > Thnks Sven, but I finally used stream's #basicNext (as Henrik suggested
>> > in
>> > fogbugz) which skips forward.
>> >
>> >
>> > Martin
>>
>>
>>
>>
>>
>> --
>> Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Non-deterministic ombu errors

gcotelli
Yes, please backport the changes to Pharo 6.1 because the travis builds are failing a lot with this kind of errors when you have a project with non-ASCII characters.

Probably it makes no sense to enable Epicea in the images used by SmalltalkCI. Is there some way to disable it?

On Mon, Oct 2, 2017 at 10:18 AM, Stephane Ducasse <[hidden email]> wrote:
Martin

if you publish a fix for 6.1 we will port it back because this is important.

Stef

On Fri, Sep 29, 2017 at 3:55 PM, Martin Dias <[hidden email]> wrote:
> Now I do not know the best way for you to get the new fixed version. It's
> too late to include it in P6.1, I guess, so... either add epicea into
> metacello configuration or wait until P7.
>
>
>
> On Fri, Sep 29, 2017 at 5:34 AM, Jan Blizničenko <[hidden email]>
> wrote:
>>
>> I'm glad it has known reason then, thank you for looking into it, all I
>> was
>> able to find out was that it relates to non-ascii characters and that
>> build
>> on our production server fails because of that :)
>>
>> Jan
>>
>>
>> tinchodias wrote
>> > On Sun, Sep 24, 2017 at 5:49 AM, Sven Van Caekenberghe &lt;
>>
>> > sven@
>>
>> > &gt; wrote:
>> >
>> >>
>> >> > On 23 Sep 2017, at 22:42, Jan Blizničenko &lt;
>>
>> > bliznjan@.cvut
>>
>> > &gt; wrote:
>> >> >
>> >> > Hello
>> >> >
>> >> > I am having very similar errors from time to time when manually
>> >> > loading
>> >> code
>> >> > into Pharo from filetree (gitfiletree), but only in one of my
>> >> > projects
>> >> which
>> >> > contains non-ASCII characters like ěščřžýáíéůúťď. For the very same
>> >> reason
>> >> > (not able to find why sometimes the error is there and sometimes is
>> >> not), I
>> >> > did not report it yet, but it is quite annoying. Retrying loading the
>> >> > package helps every time.
>> >>
>> >> I think was reported before. I believe Ombu somewhere takes an
>> >> arbitrary
>> >> chunk of bytes out of an UTF-8 encoded file (like the last 100 of
>> >> something
>> >> like that) and then starts reading that. With UTF-8, a variable length
>> >> encoding, that can be dangerous because you could end up in the middle
>> >> of
>> >> a
>> >> character, not between encoded character boundaries. This would also
>> >> explain the randomness, as it is highly dependent on the contents.
>> >>
>> >> ZnUTF8Encoder>>#backOnStream: can be used to make sure you are really
>> >> at
>> >> the start of an encoded character.
>> >>
>> >
>> >
>> > Hello
>> >
>> > This issue was reported as:
>> >
>> >
>> > https://pharo.fogbugz.com/f/cases/20112/Epicea-can-fail-while-reading-non-ascii-blocks
>> >
>> > My apologies for the bug... I introduced that code as an i/o
>> > optimization
>> > but wasn't properly tested.
>> >
>> > Thnks Sven, but I finally used stream's #basicNext (as Henrik suggested
>> > in
>> > fogbugz) which skips forward.
>> >
>> >
>> > Martin
>>
>>
>>
>>
>>
>> --
>> Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html
>>
>


Reply | Threaded
Open this post in threaded view
|

Re: Non-deterministic ombu errors

tinchodias
Hi

On Tue, Oct 3, 2017 at 11:40 AM, Gabriel Cotelli <[hidden email]> wrote:
Yes, please backport the changes to Pharo 6.1 because the travis builds are failing a lot with this kind of errors when you have a project with non-ASCII characters.

Ok
 

Probably it makes no sense to enable Epicea in the images used by SmalltalkCI. Is there some way to disable it?

Sure, you can evaluate: EpMonitor reset


 


On Mon, Oct 2, 2017 at 10:18 AM, Stephane Ducasse <[hidden email]> wrote:
Martin

if you publish a fix for 6.1 we will port it back because this is important.

Stef

On Fri, Sep 29, 2017 at 3:55 PM, Martin Dias <[hidden email]> wrote:
> Now I do not know the best way for you to get the new fixed version. It's
> too late to include it in P6.1, I guess, so... either add epicea into
> metacello configuration or wait until P7.
>
>
>
> On Fri, Sep 29, 2017 at 5:34 AM, Jan Blizničenko <[hidden email]>
> wrote:
>>
>> I'm glad it has known reason then, thank you for looking into it, all I
>> was
>> able to find out was that it relates to non-ascii characters and that
>> build
>> on our production server fails because of that :)
>>
>> Jan
>>
>>
>> tinchodias wrote
>> > On Sun, Sep 24, 2017 at 5:49 AM, Sven Van Caekenberghe &lt;
>>
>> > sven@
>>
>> > &gt; wrote:
>> >
>> >>
>> >> > On 23 Sep 2017, at 22:42, Jan Blizničenko &lt;
>>
>> > bliznjan@.cvut
>>
>> > &gt; wrote:
>> >> >
>> >> > Hello
>> >> >
>> >> > I am having very similar errors from time to time when manually
>> >> > loading
>> >> code
>> >> > into Pharo from filetree (gitfiletree), but only in one of my
>> >> > projects
>> >> which
>> >> > contains non-ASCII characters like ěščřžýáíéůúťď. For the very same
>> >> reason
>> >> > (not able to find why sometimes the error is there and sometimes is
>> >> not), I
>> >> > did not report it yet, but it is quite annoying. Retrying loading the
>> >> > package helps every time.
>> >>
>> >> I think was reported before. I believe Ombu somewhere takes an
>> >> arbitrary
>> >> chunk of bytes out of an UTF-8 encoded file (like the last 100 of
>> >> something
>> >> like that) and then starts reading that. With UTF-8, a variable length
>> >> encoding, that can be dangerous because you could end up in the middle
>> >> of
>> >> a
>> >> character, not between encoded character boundaries. This would also
>> >> explain the randomness, as it is highly dependent on the contents.
>> >>
>> >> ZnUTF8Encoder>>#backOnStream: can be used to make sure you are really
>> >> at
>> >> the start of an encoded character.
>> >>
>> >
>> >
>> > Hello
>> >
>> > This issue was reported as:
>> >
>> >
>> > https://pharo.fogbugz.com/f/cases/20112/Epicea-can-fail-while-reading-non-ascii-blocks
>> >
>> > My apologies for the bug... I introduced that code as an i/o
>> > optimization
>> > but wasn't properly tested.
>> >
>> > Thnks Sven, but I finally used stream's #basicNext (as Henrik suggested
>> > in
>> > fogbugz) which skips forward.
>> >
>> >
>> > Martin
>>
>>
>>
>>
>>
>> --
>> Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html
>>
>



Reply | Threaded
Open this post in threaded view
|

Re: Non-deterministic ombu errors

tinchodias
I've put ConfigurationOfEpicea 8.2.4 in p6 inbox. 
Martín

On Tue, Oct 3, 2017 at 1:40 PM, Martin Dias <[hidden email]> wrote:
Hi

On Tue, Oct 3, 2017 at 11:40 AM, Gabriel Cotelli <[hidden email]> wrote:
Yes, please backport the changes to Pharo 6.1 because the travis builds are failing a lot with this kind of errors when you have a project with non-ASCII characters.

Ok
 

Probably it makes no sense to enable Epicea in the images used by SmalltalkCI. Is there some way to disable it?

Sure, you can evaluate: EpMonitor reset


 


On Mon, Oct 2, 2017 at 10:18 AM, Stephane Ducasse <[hidden email]> wrote:
Martin

if you publish a fix for 6.1 we will port it back because this is important.

Stef

On Fri, Sep 29, 2017 at 3:55 PM, Martin Dias <[hidden email]> wrote:
> Now I do not know the best way for you to get the new fixed version. It's
> too late to include it in P6.1, I guess, so... either add epicea into
> metacello configuration or wait until P7.
>
>
>
> On Fri, Sep 29, 2017 at 5:34 AM, Jan Blizničenko <[hidden email]>
> wrote:
>>
>> I'm glad it has known reason then, thank you for looking into it, all I
>> was
>> able to find out was that it relates to non-ascii characters and that
>> build
>> on our production server fails because of that :)
>>
>> Jan
>>
>>
>> tinchodias wrote
>> > On Sun, Sep 24, 2017 at 5:49 AM, Sven Van Caekenberghe &lt;
>>
>> > sven@
>>
>> > &gt; wrote:
>> >
>> >>
>> >> > On 23 Sep 2017, at 22:42, Jan Blizničenko &lt;
>>
>> > bliznjan@.cvut
>>
>> > &gt; wrote:
>> >> >
>> >> > Hello
>> >> >
>> >> > I am having very similar errors from time to time when manually
>> >> > loading
>> >> code
>> >> > into Pharo from filetree (gitfiletree), but only in one of my
>> >> > projects
>> >> which
>> >> > contains non-ASCII characters like ěščřžýáíéůúťď. For the very same
>> >> reason
>> >> > (not able to find why sometimes the error is there and sometimes is
>> >> not), I
>> >> > did not report it yet, but it is quite annoying. Retrying loading the
>> >> > package helps every time.
>> >>
>> >> I think was reported before. I believe Ombu somewhere takes an
>> >> arbitrary
>> >> chunk of bytes out of an UTF-8 encoded file (like the last 100 of
>> >> something
>> >> like that) and then starts reading that. With UTF-8, a variable length
>> >> encoding, that can be dangerous because you could end up in the middle
>> >> of
>> >> a
>> >> character, not between encoded character boundaries. This would also
>> >> explain the randomness, as it is highly dependent on the contents.
>> >>
>> >> ZnUTF8Encoder>>#backOnStream: can be used to make sure you are really
>> >> at
>> >> the start of an encoded character.
>> >>
>> >
>> >
>> > Hello
>> >
>> > This issue was reported as:
>> >
>> >
>> > https://pharo.fogbugz.com/f/cases/20112/Epicea-can-fail-while-reading-non-ascii-blocks
>> >
>> > My apologies for the bug... I introduced that code as an i/o
>> > optimization
>> > but wasn't properly tested.
>> >
>> > Thnks Sven, but I finally used stream's #basicNext (as Henrik suggested
>> > in
>> > fogbugz) which skips forward.
>> >
>> >
>> > Martin
>>
>>
>>
>>
>>
>> --
>> Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html
>>
>




Reply | Threaded
Open this post in threaded view
|

Re: Non-deterministic ombu errors

Stephane Ducasse-3
Tx martin I added
https://pharo.fogbugz.com/f/cases/20492/Update-EPICEA

Do you have a bug entry for the 7.0 integration or should I create one?

Stef

On Tue, Oct 3, 2017 at 7:58 PM, Martin Dias <[hidden email]> wrote:

> I've put ConfigurationOfEpicea 8.2.4 in p6 inbox.
> Martín
>
> On Tue, Oct 3, 2017 at 1:40 PM, Martin Dias <[hidden email]> wrote:
>>
>> Hi
>>
>> On Tue, Oct 3, 2017 at 11:40 AM, Gabriel Cotelli <[hidden email]>
>> wrote:
>>>
>>> Yes, please backport the changes to Pharo 6.1 because the travis builds
>>> are failing a lot with this kind of errors when you have a project with
>>> non-ASCII characters.
>>
>>
>> Ok
>>
>>>
>>>
>>> Probably it makes no sense to enable Epicea in the images used by
>>> SmalltalkCI. Is there some way to disable it?
>>
>>
>> Sure, you can evaluate: EpMonitor reset
>>
>>
>>
>>>
>>>
>>>
>>> On Mon, Oct 2, 2017 at 10:18 AM, Stephane Ducasse
>>> <[hidden email]> wrote:
>>>>
>>>> Martin
>>>>
>>>> if you publish a fix for 6.1 we will port it back because this is
>>>> important.
>>>>
>>>> Stef
>>>>
>>>> On Fri, Sep 29, 2017 at 3:55 PM, Martin Dias <[hidden email]>
>>>> wrote:
>>>> > Now I do not know the best way for you to get the new fixed version.
>>>> > It's
>>>> > too late to include it in P6.1, I guess, so... either add epicea into
>>>> > metacello configuration or wait until P7.
>>>> >
>>>> >
>>>> >
>>>> > On Fri, Sep 29, 2017 at 5:34 AM, Jan Blizničenko
>>>> > <[hidden email]>
>>>> > wrote:
>>>> >>
>>>> >> I'm glad it has known reason then, thank you for looking into it, all
>>>> >> I
>>>> >> was
>>>> >> able to find out was that it relates to non-ascii characters and that
>>>> >> build
>>>> >> on our production server fails because of that :)
>>>> >>
>>>> >> Jan
>>>> >>
>>>> >>
>>>> >> tinchodias wrote
>>>> >> > On Sun, Sep 24, 2017 at 5:49 AM, Sven Van Caekenberghe &lt;
>>>> >>
>>>> >> > sven@
>>>> >>
>>>> >> > &gt; wrote:
>>>> >> >
>>>> >> >>
>>>> >> >> > On 23 Sep 2017, at 22:42, Jan Blizničenko &lt;
>>>> >>
>>>> >> > bliznjan@.cvut
>>>> >>
>>>> >> > &gt; wrote:
>>>> >> >> >
>>>> >> >> > Hello
>>>> >> >> >
>>>> >> >> > I am having very similar errors from time to time when manually
>>>> >> >> > loading
>>>> >> >> code
>>>> >> >> > into Pharo from filetree (gitfiletree), but only in one of my
>>>> >> >> > projects
>>>> >> >> which
>>>> >> >> > contains non-ASCII characters like ěščřžýáíéůúťď. For the very
>>>> >> >> > same
>>>> >> >> reason
>>>> >> >> > (not able to find why sometimes the error is there and sometimes
>>>> >> >> > is
>>>> >> >> not), I
>>>> >> >> > did not report it yet, but it is quite annoying. Retrying
>>>> >> >> > loading the
>>>> >> >> > package helps every time.
>>>> >> >>
>>>> >> >> I think was reported before. I believe Ombu somewhere takes an
>>>> >> >> arbitrary
>>>> >> >> chunk of bytes out of an UTF-8 encoded file (like the last 100 of
>>>> >> >> something
>>>> >> >> like that) and then starts reading that. With UTF-8, a variable
>>>> >> >> length
>>>> >> >> encoding, that can be dangerous because you could end up in the
>>>> >> >> middle
>>>> >> >> of
>>>> >> >> a
>>>> >> >> character, not between encoded character boundaries. This would
>>>> >> >> also
>>>> >> >> explain the randomness, as it is highly dependent on the contents.
>>>> >> >>
>>>> >> >> ZnUTF8Encoder>>#backOnStream: can be used to make sure you are
>>>> >> >> really
>>>> >> >> at
>>>> >> >> the start of an encoded character.
>>>> >> >>
>>>> >> >
>>>> >> >
>>>> >> > Hello
>>>> >> >
>>>> >> > This issue was reported as:
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > https://pharo.fogbugz.com/f/cases/20112/Epicea-can-fail-while-reading-non-ascii-blocks
>>>> >> >
>>>> >> > My apologies for the bug... I introduced that code as an i/o
>>>> >> > optimization
>>>> >> > but wasn't properly tested.
>>>> >> >
>>>> >> > Thnks Sven, but I finally used stream's #basicNext (as Henrik
>>>> >> > suggested
>>>> >> > in
>>>> >> > fogbugz) which skips forward.
>>>> >> >
>>>> >> >
>>>> >> > Martin
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Sent from:
>>>> >> http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html
>>>> >>
>>>> >
>>>>
>>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Non-deterministic ombu errors

tinchodias
Ok but for 6.0 there was already https://pharo.fogbugz.com/f/cases/20112/Epicea-can-fail-while-reading-non-ascii-blocks

The same test failed and then I didn't investigate why that test fails only in ci. (Reported by Marcus).

Yes, please create one for 7.0, I still didnt read how is the new integration process.

El 5 oct. 2017 7:11 AM, "Stephane Ducasse" <[hidden email]> escribió:
Tx martin I added
https://pharo.fogbugz.com/f/cases/20492/Update-EPICEA

Do you have a bug entry for the 7.0 integration or should I create one?

Stef

On Tue, Oct 3, 2017 at 7:58 PM, Martin Dias <[hidden email]> wrote:
> I've put ConfigurationOfEpicea 8.2.4 in p6 inbox.
> Martín
>
> On Tue, Oct 3, 2017 at 1:40 PM, Martin Dias <[hidden email]> wrote:
>>
>> Hi
>>
>> On Tue, Oct 3, 2017 at 11:40 AM, Gabriel Cotelli <[hidden email]>
>> wrote:
>>>
>>> Yes, please backport the changes to Pharo 6.1 because the travis builds
>>> are failing a lot with this kind of errors when you have a project with
>>> non-ASCII characters.
>>
>>
>> Ok
>>
>>>
>>>
>>> Probably it makes no sense to enable Epicea in the images used by
>>> SmalltalkCI. Is there some way to disable it?
>>
>>
>> Sure, you can evaluate: EpMonitor reset
>>
>>
>>
>>>
>>>
>>>
>>> On Mon, Oct 2, 2017 at 10:18 AM, Stephane Ducasse
>>> <[hidden email]> wrote:
>>>>
>>>> Martin
>>>>
>>>> if you publish a fix for 6.1 we will port it back because this is
>>>> important.
>>>>
>>>> Stef
>>>>
>>>> On Fri, Sep 29, 2017 at 3:55 PM, Martin Dias <[hidden email]>
>>>> wrote:
>>>> > Now I do not know the best way for you to get the new fixed version.
>>>> > It's
>>>> > too late to include it in P6.1, I guess, so... either add epicea into
>>>> > metacello configuration or wait until P7.
>>>> >
>>>> >
>>>> >
>>>> > On Fri, Sep 29, 2017 at 5:34 AM, Jan Blizničenko
>>>> > <[hidden email]>
>>>> > wrote:
>>>> >>
>>>> >> I'm glad it has known reason then, thank you for looking into it, all
>>>> >> I
>>>> >> was
>>>> >> able to find out was that it relates to non-ascii characters and that
>>>> >> build
>>>> >> on our production server fails because of that :)
>>>> >>
>>>> >> Jan
>>>> >>
>>>> >>
>>>> >> tinchodias wrote
>>>> >> > On Sun, Sep 24, 2017 at 5:49 AM, Sven Van Caekenberghe &lt;
>>>> >>
>>>> >> > sven@
>>>> >>
>>>> >> > &gt; wrote:
>>>> >> >
>>>> >> >>
>>>> >> >> > On 23 Sep 2017, at 22:42, Jan Blizničenko &lt;
>>>> >>
>>>> >> > bliznjan@.cvut
>>>> >>
>>>> >> > &gt; wrote:
>>>> >> >> >
>>>> >> >> > Hello
>>>> >> >> >
>>>> >> >> > I am having very similar errors from time to time when manually
>>>> >> >> > loading
>>>> >> >> code
>>>> >> >> > into Pharo from filetree (gitfiletree), but only in one of my
>>>> >> >> > projects
>>>> >> >> which
>>>> >> >> > contains non-ASCII characters like ěščřžýáíéůúťď. For the very
>>>> >> >> > same
>>>> >> >> reason
>>>> >> >> > (not able to find why sometimes the error is there and sometimes
>>>> >> >> > is
>>>> >> >> not), I
>>>> >> >> > did not report it yet, but it is quite annoying. Retrying
>>>> >> >> > loading the
>>>> >> >> > package helps every time.
>>>> >> >>
>>>> >> >> I think was reported before. I believe Ombu somewhere takes an
>>>> >> >> arbitrary
>>>> >> >> chunk of bytes out of an UTF-8 encoded file (like the last 100 of
>>>> >> >> something
>>>> >> >> like that) and then starts reading that. With UTF-8, a variable
>>>> >> >> length
>>>> >> >> encoding, that can be dangerous because you could end up in the
>>>> >> >> middle
>>>> >> >> of
>>>> >> >> a
>>>> >> >> character, not between encoded character boundaries. This would
>>>> >> >> also
>>>> >> >> explain the randomness, as it is highly dependent on the contents.
>>>> >> >>
>>>> >> >> ZnUTF8Encoder>>#backOnStream: can be used to make sure you are
>>>> >> >> really
>>>> >> >> at
>>>> >> >> the start of an encoded character.
>>>> >> >>
>>>> >> >
>>>> >> >
>>>> >> > Hello
>>>> >> >
>>>> >> > This issue was reported as:
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > https://pharo.fogbugz.com/f/cases/20112/Epicea-can-fail-while-reading-non-ascii-blocks
>>>> >> >
>>>> >> > My apologies for the bug... I introduced that code as an i/o
>>>> >> > optimization
>>>> >> > but wasn't properly tested.
>>>> >> >
>>>> >> > Thnks Sven, but I finally used stream's #basicNext (as Henrik
>>>> >> > suggested
>>>> >> > in
>>>> >> > fogbugz) which skips forward.
>>>> >> >
>>>> >> >
>>>> >> > Martin
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Sent from:
>>>> >> http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html
>>>> >>
>>>> >
>>>>
>>>
>>
>