Non-deterministic ombu errors
Locked
 
|
Hi, I'm having some non-deterministic errors in the Travis builds, the error is always something like: OmFileStoreReadingError: Reading entry from File @ pharo-local/ombu-sessions/Pharo.4p8buz6mww48y1noxrolgwkl1.ombu@17432579: Invalid utf8 input detected If I retried the build it works without any change. I think it happens during the code loading phase. Is anyone experiencing this issue? Here's an extract of the travis log: OmFileStoreReadingError: Reading entry from File @ pharo-local/ombu-sessions/Pharo.4p8buz6mww48y1noxrolgwkl1.ombu@17432579: Invalid utf8 input detected [ :error | (OmFileStoreReadingError readingError: error on: self fileReference position: readStream position) signal ] in [ :readStream | [ ^ aBlockClosure value: readStream ascii ] on: Error do: [ :error | (OmFileStoreReadingError readingError: error on: self fileReference position: readStream position) signal ] ] in OmBlockFileStore(OmFileStore)>>readEntriesWith: in Block: [ :error | ... BlockClosure>>cull: Context>>evaluateSignal: Context>>handleSignal: UTF8InvalidText(Exception)>>signal UTF8InvalidText(Exception)>>signal: UTF8TextConverter>>errorMalformedInput UTF8TextConverter>>nextFromStream: MultiByteFileStream>>next MultiByteFileStream(PositionableStream)>>match: MultiByteFileStream(PositionableStream)>>upToAll: OmSTONEntryReader>>nextEntryPositionIfFound:ifNone: OmBlockFileStore>>refreshNewBlocksFrom: [ :readStream | self refreshNewBlocksFrom: readStream ] in OmBlockFileStore>>refreshEntryPositionsByLocalNameStartingAt:since: in Block: [ :readStream | self refreshNewBlocksFrom: readStr...etc... [ ^ aBlockClosure value: readStream ascii ] in [ :readStream | [ ^ aBlockClosure value: readStream ascii ] on: Error do: [ :error | (OmFileStoreReadingError readingError: error on: self fileReference position: readStream position) signal ] ] in OmBlockFileStore(OmFileStore)>>readEntriesWith: in Block: [ ^ aBlockClosure value: readStream ascii ] BlockClosure>>on:do: [ :readStream | [ ^ aBlockClosure value: readStream ascii ] on: Error do: [ :error | (OmFileStoreReadingError readingError: error on: self fileReference position: readStream position) signal ] ] in OmBlockFileStore(OmFileStore)>>readEntriesWith: in Block: [ :readStream | ... [ aBlock value: stream ] in FileReference(AbstractFileReference)>>readStreamDo: in Block: [ aBlock value: stream ] BlockClosure>>ensure: FileReference(AbstractFileReference)>>readStreamDo: OmBlockFileStore(OmFileStore)>>readEntriesWith: OmBlockFileStore>>refreshEntryPositionsByLocalNameStartingAt:since: [ | initialPosition initialLocalName | self entryBuffer isEmpty ifTrue: [ ^ self ]. fileReference writeStreamDo: [ :fileStream | fileStream setToEnd. initialPosition := fileStream position. initialLocalName := self entryBuffer first value. ZnBufferedWriteStream on: fileStream do: [ :aWriteStream | | anEntryWriter | anEntryWriter := self newEntryWriter. [ self entryBuffer isEmpty ] whileFalse: [ | next entry | next := self entryBuffer removeFirst. entry := next key. "Write entry to file" anEntryWriter on: aWriteStream nextEntryPut: entry ] ]. "In Linux it was necessary to explicitely flush the file stream" fileStream flush. lastStreamPosition := fileStream size ]. self refreshEntryPositionsByLocalNameStartingAt: initialPosition since: initialLocalName ] in OmBlockFileStore(OmFileStore)>>flushEntryBuffer in Block: [ | initialPosition initialLocalName |... [ caught := true. self wait. blockValue := mutuallyExcludedBlock value ] in Semaphore>>critical: in Block: [ caught := true.... BlockClosure>>ensure: Semaphore>>critical: OmBlockFileStore(OmFileStore)>>critical: OmBlockFileStore(OmFileStore)>>flushEntryBuffer OmDeferrer>>sendMessage [ self sendMessage ] in OmDeferrer>>flush in Block: [ self sendMessage ] |
Re: Non-deterministic ombu errors
|
|
Hello
I am having very similar errors from time to time when manually loading code into Pharo from filetree (gitfiletree), but only in one of my projects which contains non-ASCII characters like ěščřžýáíéůúťď. For the very same reason (not able to find why sometimes the error is there and sometimes is not), I did not report it yet, but it is quite annoying. Retrying loading the package helps every time. Jan gcotelli wrote > Hi, > > I'm having some non-deterministic errors in the Travis builds, the error > is > always something like: > > *OmFileStoreReadingError: Reading entry from File @ > pharo-local/ombu-sessions/Pharo.4p8buz6mww48y1noxrolgwkl1.ombu@17432579: > Invalid utf8 input detected* > > If I retried the build it works without any change. I think it happens > during the code loading phase. > > Is anyone experiencing this issue? > > Here's an extract of the travis log: -- Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html |
Re: Non-deterministic ombu errors
|
|
Hi jan
let us know if you have a reproducible case so that we fix it. Stef On Sat, Sep 23, 2017 at 10:42 PM, Jan Blizničenko <[hidden email]> wrote: > Hello > > I am having very similar errors from time to time when manually loading code > into Pharo from filetree (gitfiletree), but only in one of my projects which > contains non-ASCII characters like ěščřžýáíéůúťď. For the very same reason > (not able to find why sometimes the error is there and sometimes is not), I > did not report it yet, but it is quite annoying. Retrying loading the > package helps every time. > > Jan > > > gcotelli wrote >> Hi, >> >> I'm having some non-deterministic errors in the Travis builds, the error >> is >> always something like: >> >> *OmFileStoreReadingError: Reading entry from File @ >> pharo-local/ombu-sessions/Pharo.4p8buz6mww48y1noxrolgwkl1.ombu@17432579: >> Invalid utf8 input detected* >> >> If I retried the build it works without any change. I think it happens >> during the code loading phase. >> >> Is anyone experiencing this issue? >> >> Here's an extract of the travis log: > > > > > > -- > Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html > |
Re: Non-deterministic ombu errors
|
|
In reply to this post by Jan Blizničenko
> On 23 Sep 2017, at 22:42, Jan Blizničenko <[hidden email]> wrote: > > Hello > > I am having very similar errors from time to time when manually loading code > into Pharo from filetree (gitfiletree), but only in one of my projects which > contains non-ASCII characters like ěščřžýáíéůúťď. For the very same reason > (not able to find why sometimes the error is there and sometimes is not), I > did not report it yet, but it is quite annoying. Retrying loading the > package helps every time. I think was reported before. I believe Ombu somewhere takes an arbitrary chunk of bytes out of an UTF-8 encoded file (like the last 100 of something like that) and then starts reading that. With UTF-8, a variable length encoding, that can be dangerous because you could end up in the middle of a character, not between encoded character boundaries. This would also explain the randomness, as it is highly dependent on the contents. ZnUTF8Encoder>>#backOnStream: can be used to make sure you are really at the start of an encoded character. > Jan > > > gcotelli wrote >> Hi, >> >> I'm having some non-deterministic errors in the Travis builds, the error >> is >> always something like: >> >> *OmFileStoreReadingError: Reading entry from File @ >> pharo-local/ombu-sessions/Pharo.4p8buz6mww48y1noxrolgwkl1.ombu@17432579: >> Invalid utf8 input detected* >> >> If I retried the build it works without any change. I think it happens >> during the code loading phase. >> >> Is anyone experiencing this issue? >> >> Here's an extract of the travis log: > > > > > > -- > Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html |
|
|
On Sun, Sep 24, 2017 at 5:49 AM, Sven Van Caekenberghe <[hidden email]> wrote:
This issue was reported as: https://pharo.fogbugz.com/f/cases/20112/Epicea-can-fail-while-reading-non-ascii-blocks My apologies for the bug... I introduced that code as an i/o optimization but wasn't properly tested. Thnks Sven, but I finally used stream's #basicNext (as Henrik suggested in fogbugz) which skips forward. Martin |
Re: Non-deterministic ombu errors
|
|
tx martin.
On Mon, Sep 25, 2017 at 1:07 AM, Martin Dias <[hidden email]> wrote: > > > On Sun, Sep 24, 2017 at 5:49 AM, Sven Van Caekenberghe <[hidden email]> wrote: >> >> >> > On 23 Sep 2017, at 22:42, Jan Blizničenko <[hidden email]> wrote: >> > >> > Hello >> > >> > I am having very similar errors from time to time when manually loading >> > code >> > into Pharo from filetree (gitfiletree), but only in one of my projects >> > which >> > contains non-ASCII characters like ěščřžýáíéůúťď. For the very same >> > reason >> > (not able to find why sometimes the error is there and sometimes is >> > not), I >> > did not report it yet, but it is quite annoying. Retrying loading the >> > package helps every time. >> >> I think was reported before. I believe Ombu somewhere takes an arbitrary >> chunk of bytes out of an UTF-8 encoded file (like the last 100 of something >> like that) and then starts reading that. With UTF-8, a variable length >> encoding, that can be dangerous because you could end up in the middle of a >> character, not between encoded character boundaries. This would also explain >> the randomness, as it is highly dependent on the contents. >> >> ZnUTF8Encoder>>#backOnStream: can be used to make sure you are really at >> the start of an encoded character. > > > > Hello > > This issue was reported as: > > https://pharo.fogbugz.com/f/cases/20112/Epicea-can-fail-while-reading-non-ascii-blocks > > My apologies for the bug... I introduced that code as an i/o optimization > but wasn't properly tested. > > Thnks Sven, but I finally used stream's #basicNext (as Henrik suggested in > fogbugz) which skips forward. > > > Martin > |
Re: Non-deterministic ombu errors
|
|
In reply to this post by tinchodias
I'm glad it has known reason then, thank you for looking into it, all I was
able to find out was that it relates to non-ascii characters and that build on our production server fails because of that :) Jan tinchodias wrote > On Sun, Sep 24, 2017 at 5:49 AM, Sven Van Caekenberghe < > sven@ > > wrote: > >> >> > On 23 Sep 2017, at 22:42, Jan Blizničenko < > bliznjan@.cvut > > wrote: >> > >> > Hello >> > >> > I am having very similar errors from time to time when manually loading >> code >> > into Pharo from filetree (gitfiletree), but only in one of my projects >> which >> > contains non-ASCII characters like ěščřžýáíéůúťď. For the very same >> reason >> > (not able to find why sometimes the error is there and sometimes is >> not), I >> > did not report it yet, but it is quite annoying. Retrying loading the >> > package helps every time. >> >> I think was reported before. I believe Ombu somewhere takes an arbitrary >> chunk of bytes out of an UTF-8 encoded file (like the last 100 of >> something >> like that) and then starts reading that. With UTF-8, a variable length >> encoding, that can be dangerous because you could end up in the middle of >> a >> character, not between encoded character boundaries. This would also >> explain the randomness, as it is highly dependent on the contents. >> >> ZnUTF8Encoder>>#backOnStream: can be used to make sure you are really at >> the start of an encoded character. >> > > > Hello > > This issue was reported as: > > https://pharo.fogbugz.com/f/cases/20112/Epicea-can-fail-while-reading-non-ascii-blocks > > My apologies for the bug... I introduced that code as an i/o optimization > but wasn't properly tested. > > Thnks Sven, but I finally used stream's #basicNext (as Henrik suggested in > fogbugz) which skips forward. > > > Martin -- Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html |
|
|
Now I do not know the best way for you to get the new fixed version. It's too late to include it in P6.1, I guess, so... either add epicea into metacello configuration or wait until P7.
On Fri, Sep 29, 2017 at 5:34 AM, Jan Blizničenko <[hidden email]> wrote: I'm glad it has known reason then, thank you for looking into it, all I was |
Re: Non-deterministic ombu errors
|
|
Martin
if you publish a fix for 6.1 we will port it back because this is important. Stef On Fri, Sep 29, 2017 at 3:55 PM, Martin Dias <[hidden email]> wrote: > Now I do not know the best way for you to get the new fixed version. It's > too late to include it in P6.1, I guess, so... either add epicea into > metacello configuration or wait until P7. > > > > On Fri, Sep 29, 2017 at 5:34 AM, Jan Blizničenko <[hidden email]> > wrote: >> >> I'm glad it has known reason then, thank you for looking into it, all I >> was >> able to find out was that it relates to non-ascii characters and that >> build >> on our production server fails because of that :) >> >> Jan >> >> >> tinchodias wrote >> > On Sun, Sep 24, 2017 at 5:49 AM, Sven Van Caekenberghe < >> >> > sven@ >> >> > > wrote: >> > >> >> >> >> > On 23 Sep 2017, at 22:42, Jan Blizničenko < >> >> > bliznjan@.cvut >> >> > > wrote: >> >> > >> >> > Hello >> >> > >> >> > I am having very similar errors from time to time when manually >> >> > loading >> >> code >> >> > into Pharo from filetree (gitfiletree), but only in one of my >> >> > projects >> >> which >> >> > contains non-ASCII characters like ěščřžýáíéůúťď. For the very same >> >> reason >> >> > (not able to find why sometimes the error is there and sometimes is >> >> not), I >> >> > did not report it yet, but it is quite annoying. Retrying loading the >> >> > package helps every time. >> >> >> >> I think was reported before. I believe Ombu somewhere takes an >> >> arbitrary >> >> chunk of bytes out of an UTF-8 encoded file (like the last 100 of >> >> something >> >> like that) and then starts reading that. With UTF-8, a variable length >> >> encoding, that can be dangerous because you could end up in the middle >> >> of >> >> a >> >> character, not between encoded character boundaries. This would also >> >> explain the randomness, as it is highly dependent on the contents. >> >> >> >> ZnUTF8Encoder>>#backOnStream: can be used to make sure you are really >> >> at >> >> the start of an encoded character. >> >> >> > >> > >> > Hello >> > >> > This issue was reported as: >> > >> > >> > https://pharo.fogbugz.com/f/cases/20112/Epicea-can-fail-while-reading-non-ascii-blocks >> > >> > My apologies for the bug... I introduced that code as an i/o >> > optimization >> > but wasn't properly tested. >> > >> > Thnks Sven, but I finally used stream's #basicNext (as Henrik suggested >> > in >> > fogbugz) which skips forward. >> > >> > >> > Martin >> >> >> >> >> >> -- >> Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html >> > |
|
|
Yes, please backport the changes to Pharo 6.1 because the travis builds are failing a lot with this kind of errors when you have a project with non-ASCII characters.
Probably it makes no sense to enable Epicea in the images used by SmalltalkCI. Is there some way to disable it? On Mon, Oct 2, 2017 at 10:18 AM, Stephane Ducasse <[hidden email]> wrote: Martin |
|
|
Hi
On Tue, Oct 3, 2017 at 11:40 AM, Gabriel Cotelli <[hidden email]> wrote:
Ok
Sure, you can evaluate: EpMonitor reset
|
|
|
I've put ConfigurationOfEpicea 8.2.4 in p6 inbox. Martín On Tue, Oct 3, 2017 at 1:40 PM, Martin Dias <[hidden email]> wrote:
|
Re: Non-deterministic ombu errors
|
|
Tx martin I added
https://pharo.fogbugz.com/f/cases/20492/Update-EPICEA Do you have a bug entry for the 7.0 integration or should I create one? Stef On Tue, Oct 3, 2017 at 7:58 PM, Martin Dias <[hidden email]> wrote: > I've put ConfigurationOfEpicea 8.2.4 in p6 inbox. > Martín > > On Tue, Oct 3, 2017 at 1:40 PM, Martin Dias <[hidden email]> wrote: >> >> Hi >> >> On Tue, Oct 3, 2017 at 11:40 AM, Gabriel Cotelli <[hidden email]> >> wrote: >>> >>> Yes, please backport the changes to Pharo 6.1 because the travis builds >>> are failing a lot with this kind of errors when you have a project with >>> non-ASCII characters. >> >> >> Ok >> >>> >>> >>> Probably it makes no sense to enable Epicea in the images used by >>> SmalltalkCI. Is there some way to disable it? >> >> >> Sure, you can evaluate: EpMonitor reset >> >> >> >>> >>> >>> >>> On Mon, Oct 2, 2017 at 10:18 AM, Stephane Ducasse >>> <[hidden email]> wrote: >>>> >>>> Martin >>>> >>>> if you publish a fix for 6.1 we will port it back because this is >>>> important. >>>> >>>> Stef >>>> >>>> On Fri, Sep 29, 2017 at 3:55 PM, Martin Dias <[hidden email]> >>>> wrote: >>>> > Now I do not know the best way for you to get the new fixed version. >>>> > It's >>>> > too late to include it in P6.1, I guess, so... either add epicea into >>>> > metacello configuration or wait until P7. >>>> > >>>> > >>>> > >>>> > On Fri, Sep 29, 2017 at 5:34 AM, Jan Blizničenko >>>> > <[hidden email]> >>>> > wrote: >>>> >> >>>> >> I'm glad it has known reason then, thank you for looking into it, all >>>> >> I >>>> >> was >>>> >> able to find out was that it relates to non-ascii characters and that >>>> >> build >>>> >> on our production server fails because of that :) >>>> >> >>>> >> Jan >>>> >> >>>> >> >>>> >> tinchodias wrote >>>> >> > On Sun, Sep 24, 2017 at 5:49 AM, Sven Van Caekenberghe < >>>> >> >>>> >> > sven@ >>>> >> >>>> >> > > wrote: >>>> >> > >>>> >> >> >>>> >> >> > On 23 Sep 2017, at 22:42, Jan Blizničenko < >>>> >> >>>> >> > bliznjan@.cvut >>>> >> >>>> >> > > wrote: >>>> >> >> > >>>> >> >> > Hello >>>> >> >> > >>>> >> >> > I am having very similar errors from time to time when manually >>>> >> >> > loading >>>> >> >> code >>>> >> >> > into Pharo from filetree (gitfiletree), but only in one of my >>>> >> >> > projects >>>> >> >> which >>>> >> >> > contains non-ASCII characters like ěščřžýáíéůúťď. For the very >>>> >> >> > same >>>> >> >> reason >>>> >> >> > (not able to find why sometimes the error is there and sometimes >>>> >> >> > is >>>> >> >> not), I >>>> >> >> > did not report it yet, but it is quite annoying. Retrying >>>> >> >> > loading the >>>> >> >> > package helps every time. >>>> >> >> >>>> >> >> I think was reported before. I believe Ombu somewhere takes an >>>> >> >> arbitrary >>>> >> >> chunk of bytes out of an UTF-8 encoded file (like the last 100 of >>>> >> >> something >>>> >> >> like that) and then starts reading that. With UTF-8, a variable >>>> >> >> length >>>> >> >> encoding, that can be dangerous because you could end up in the >>>> >> >> middle >>>> >> >> of >>>> >> >> a >>>> >> >> character, not between encoded character boundaries. This would >>>> >> >> also >>>> >> >> explain the randomness, as it is highly dependent on the contents. >>>> >> >> >>>> >> >> ZnUTF8Encoder>>#backOnStream: can be used to make sure you are >>>> >> >> really >>>> >> >> at >>>> >> >> the start of an encoded character. >>>> >> >> >>>> >> > >>>> >> > >>>> >> > Hello >>>> >> > >>>> >> > This issue was reported as: >>>> >> > >>>> >> > >>>> >> > >>>> >> > https://pharo.fogbugz.com/f/cases/20112/Epicea-can-fail-while-reading-non-ascii-blocks >>>> >> > >>>> >> > My apologies for the bug... I introduced that code as an i/o >>>> >> > optimization >>>> >> > but wasn't properly tested. >>>> >> > >>>> >> > Thnks Sven, but I finally used stream's #basicNext (as Henrik >>>> >> > suggested >>>> >> > in >>>> >> > fogbugz) which skips forward. >>>> >> > >>>> >> > >>>> >> > Martin >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> -- >>>> >> Sent from: >>>> >> http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html >>>> >> >>>> > >>>> >>> >> > |
|
|
Ok but for 6.0 there was already https://pharo.fogbugz.com/f/cases/20112/Epicea-can-fail-while-reading-non-ascii-blocks The same test failed and then I didn't investigate why that test fails only in ci. (Reported by Marcus). Yes, please create one for 7.0, I still didnt read how is the new integration process. El 5 oct. 2017 7:11 AM, "Stephane Ducasse" <[hidden email]> escribió: Tx martin I added |
«
Return to Pharo Smalltalk Developers
|
1 view|%1 views
| Free forum by Nabble | Edit this page |