failing tests in Pharo 7

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

failing tests in Pharo 7

Ben Coman
It great having a CI system to check all our tests a passing,
but I thought I'd go old school and run them from the TestRunner GUI
and found in a fresh latest Pharo 7 there are "128 failures, 63 errors"

This one is interesting...
ArrayTest>>test0FixtureAsSetForIdentityMultiplinessTest
| anElement res |
self elementsCopyNonIdenticalWithoutEqualElements.
anElement := self elementsCopyNonIdenticalWithoutEqualElements anyOne.
self deny: anElement copy == anElement.  "<<<<FAILS"

where anElement ==> SmallFloat64 (1.5)
so the deny fails.

It seems that the test setUp is wrong since #elementsCopyNonIdenticalWithoutEqualElements 
does not return a collection (of elements for which copy is not identical ) without equal elements:"

This seems like a permanent error rather than a transient one.
How does this slip through CI testing?
and what can be done about cleaning this up?

cheers -ben

P.S. With the loss of method version info is impossible to guess when this may have been introduced, to guess which Issue it was related to, to guess how I might approach fixing this myself.
Reply | Threaded
Open this post in threaded view
|

Re: failing tests in Pharo 7

Ben Coman


On 6 January 2018 at 19:09, Ben Coman <[hidden email]> wrote:
It great having a CI system to check all our tests a passing,
but I thought I'd go old school and run them from the TestRunner GUI
and found in a fresh latest Pharo 7 there are "128 failures, 63 errors"

So that was Pharo64bit.
Pharo32bit is better with only has only "1 failures, 20 errors".

So what is the CI dependency of the 32 bit and 64 bit Images?
Do they both run tests independently?

cheers -ben
 

This one is interesting...
ArrayTest>>test0FixtureAsSetForIdentityMultiplinessTest
| anElement res |
self elementsCopyNonIdenticalWithoutEqualElements.
anElement := self elementsCopyNonIdenticalWithoutEqualElements anyOne.
self deny: anElement copy == anElement.  "<<<<FAILS"

where anElement ==> SmallFloat64 (1.5)
so the deny fails.

It seems that the test setUp is wrong since #elementsCopyNonIdenticalWithoutEqualElements 
does not return a collection (of elements for which copy is not identical ) without equal elements:"

This seems like a permanent error rather than a transient one.
How does this slip through CI testing?
and what can be done about cleaning this up?

cheers -ben

P.S. With the loss of method version info is impossible to guess when this may have been introduced, to guess which Issue it was related to, to guess how I might approach fixing this myself.

Reply | Threaded
Open this post in threaded view
|

Re: failing tests in Pharo 7

alistairgrant
In reply to this post by Ben Coman
Hi Ben,

On 6 January 2018 at 12:09, Ben Coman <[hidden email]> wrote:

> It great having a CI system to check all our tests a passing,
> but I thought I'd go old school and run them from the TestRunner GUI
> and found in a fresh latest Pharo 7 there are "128 failures, 63 errors"
>
> This one is interesting...
> ArrayTest>>test0FixtureAsSetForIdentityMultiplinessTest
> | anElement res |
> self elementsCopyNonIdenticalWithoutEqualElements.
> anElement := self elementsCopyNonIdenticalWithoutEqualElements anyOne.
> self deny: anElement copy == anElement.  "<<<<FAILS"
>
> where anElement ==> SmallFloat64 (1.5)
> so the deny fails.

Which platform and image?

Pharo 7.0
Build information:
Pharo-7.0+alpha.build.410.sha.aa40385cd6fee69bb51d4a3afcacfb5f91eed206
(32 Bit)
Ubuntu 16.04

14915 run, 14859 passes, 31 skipped, 50 expected failures, 5 failures,
1 errors, 0 unexpected passes
Failures:
ReleaseTest>>#testMethodsWithUnboundGlobals
TonelWriterTest>>#testWriteSnapshot
ReleaseTest>>#testUnpackagedClasses
ReleaseTest>>#testNoEmptyPackages
ReleaseTest>>#testThatAllMethodsArePackaged

Errors:
ZnClientTests>>#testRedirect


Cheers,
Alistair



> It seems that the test setUp is wrong since
> #elementsCopyNonIdenticalWithoutEqualElements
> does not return a collection (of elements for which copy is not identical )
> without equal elements:"
>
> This seems like a permanent error rather than a transient one.
> How does this slip through CI testing?
> and what can be done about cleaning this up?
>
> cheers -ben
>
> P.S. With the loss of method version info is impossible to guess when this
> may have been introduced, to guess which Issue it was related to, to guess
> how I might approach fixing this myself.

Reply | Threaded
Open this post in threaded view
|

Re: failing tests in Pharo 7

Sven Van Caekenberghe-2


> On 6 Jan 2018, at 20:08, Alistair Grant <[hidden email]> wrote:
>
> Hi Ben,
>
> On 6 January 2018 at 12:09, Ben Coman <[hidden email]> wrote:
>> It great having a CI system to check all our tests a passing,
>> but I thought I'd go old school and run them from the TestRunner GUI
>> and found in a fresh latest Pharo 7 there are "128 failures, 63 errors"
>>
>> This one is interesting...
>> ArrayTest>>test0FixtureAsSetForIdentityMultiplinessTest
>> | anElement res |
>> self elementsCopyNonIdenticalWithoutEqualElements.
>> anElement := self elementsCopyNonIdenticalWithoutEqualElements anyOne.
>> self deny: anElement copy == anElement.  "<<<<FAILS"
>>
>> where anElement ==> SmallFloat64 (1.5)
>> so the deny fails.
>
> Which platform and image?
>
> Pharo 7.0
> Build information:
> Pharo-7.0+alpha.build.410.sha.aa40385cd6fee69bb51d4a3afcacfb5f91eed206
> (32 Bit)
> Ubuntu 16.04
>
> 14915 run, 14859 passes, 31 skipped, 50 expected failures, 5 failures,
> 1 errors, 0 unexpected passes
> Failures:
> ReleaseTest>>#testMethodsWithUnboundGlobals
> TonelWriterTest>>#testWriteSnapshot
> ReleaseTest>>#testUnpackagedClasses
> ReleaseTest>>#testNoEmptyPackages
> ReleaseTest>>#testThatAllMethodsArePackaged
>
> Errors:
> ZnClientTests>>#testRedirect

Hmm, strange. Does it failed each time ?

It does not fail for me (macOS). Furthermore, all Zinc tests are green https://ci.inria.fr/pharo-contribution/job/ZincHTTPComponents/

> Cheers,
> Alistair
>
>
>
>> It seems that the test setUp is wrong since
>> #elementsCopyNonIdenticalWithoutEqualElements
>> does not return a collection (of elements for which copy is not identical )
>> without equal elements:"
>>
>> This seems like a permanent error rather than a transient one.
>> How does this slip through CI testing?
>> and what can be done about cleaning this up?
>>
>> cheers -ben
>>
>> P.S. With the loss of method version info is impossible to guess when this
>> may have been introduced, to guess which Issue it was related to, to guess
>> how I might approach fixing this myself.


Reply | Threaded
Open this post in threaded view
|

Re: failing tests in Pharo 7

Ben Coman
In reply to this post by alistairgrant


On 7 January 2018 at 03:08, Alistair Grant <[hidden email]> wrote:
Hi Ben,

On 6 January 2018 at 12:09, Ben Coman <[hidden email]> wrote:
> It great having a CI system to check all our tests a passing,
> but I thought I'd go old school and run them from the TestRunner GUI
> and found in a fresh latest Pharo 7 there are "128 failures, 63 errors"
>
> This one is interesting...
> ArrayTest>>test0FixtureAsSetForIdentityMultiplinessTest
> | anElement res |
> self elementsCopyNonIdenticalWithoutEqualElements.
> anElement := self elementsCopyNonIdenticalWithoutEqualElements anyOne.
> self deny: anElement copy == anElement.  "<<<<FAILS"
>
> where anElement ==> SmallFloat64 (1.5)
> so the deny fails.

Which platform and image?

I guess its not a platform issue, and the SmallFloat64 was sufficient hint, but sorry I should be more explicit.
Ubuntu 16.04 64bit
Build information: Pharo-7.0+alpha.build.409.sha.bb4eaaf976e3fb148b33b6d87598022b77329768 (64 Bit)
Virtual Machine: Pharo64/lib/pharo/5.0-201712211450/pharo

The root cause seems to be...

in 32-bit Image...
  x := 1.0.
  x copy == x.
==> false

in 64-bit Image...
  x := 1.0.
  x copy == x.
==> true

which I presume has always been the case and is expected,
but such floats are no longer are suitable for tests like...
    test0FixtureAsSetForIdentityMultiplinessTest
         "a collection (of elements for which copy is not identical ) [...]"

If someone can suggest the preferred element type to substitute for the floats
I can do the leg work to push this through.


btw I'm curious, how are these steps interrelated in the CI build?
a. build 32-bit image
b. build 64-bit image
c. test 32-bit image
d. test 64-bit image


Pharo 7.0
Build information:
Pharo-7.0+alpha.build.410.sha.aa40385cd6fee69bb51d4a3afcacfb5f91eed206
(32 Bit)
Ubuntu 16.04



14915 run, 14859 passes, 31 skipped, 50 expected failures, 5 failures,
1 errors, 0 unexpected passes
Failures:
ReleaseTest>>#testMethodsWithUnboundGlobals
TonelWriterTest>>#testWriteSnapshot
ReleaseTest>>#testUnpackagedClasses
ReleaseTest>>#testNoEmptyPackages
ReleaseTest>>#testThatAllMethodsArePackaged

I get about the same in 32-bit.  I'm too worried about the release tests,
although being totally clear would be really nice.
 
 
Errors:
ZnClientTests>>#testRedirect
 

On 7 January 2018 at 05:55, Sven Van Caekenberghe <[hidden email]> wrote:

Hmm, strange. Does it failed each time ?

It does not fail for me (macOS). Furthermore, all Zinc tests are green https://ci.inria.fr/pharo-contribution/job/ZincHTTPComponents/

ZnClientTests>>#testRedirect   doesn't error for me, 
but   ZnHTTPSTests>>#testGForceInria  failed yesterday and today five out of five tries.
The error is  TestTookTooMuchTime

I see that test does...
(client := ZnClient new) get: 'http://gforge.inria.fr/frs/?group_id=1299'.
and that url opens okay in Chrome web browser, so I don't know what to make of it.


In Pharo-7.0.0-alpha.build.410.sha.aa40385.arch.32bit
I also get a bunch of errors for SystemDependenciesTest 
that seem to have the same root cause...
    KeyNotFound: key #ManifestManifestResourcesTests not found in SystemDictionary


cheers -ben
  
Reply | Threaded
Open this post in threaded view
|

Re: failing tests in Pharo 7

alistairgrant
In reply to this post by Sven Van Caekenberghe-2
Hi Sven.

On 6 January 2018 at 22:55, Sven Van Caekenberghe <[hidden email]> wrote:

>
>
>> On 6 Jan 2018, at 20:08, Alistair Grant <[hidden email]> wrote:
>>
>> Hi Ben,
>>
>> On 6 January 2018 at 12:09, Ben Coman <[hidden email]> wrote:
>>> It great having a CI system to check all our tests a passing,
>>> but I thought I'd go old school and run them from the TestRunner GUI
>>> and found in a fresh latest Pharo 7 there are "128 failures, 63 errors"
>>>
>>> This one is interesting...
>>> ArrayTest>>test0FixtureAsSetForIdentityMultiplinessTest
>>> | anElement res |
>>> self elementsCopyNonIdenticalWithoutEqualElements.
>>> anElement := self elementsCopyNonIdenticalWithoutEqualElements anyOne.
>>> self deny: anElement copy == anElement.  "<<<<FAILS"
>>>
>>> where anElement ==> SmallFloat64 (1.5)
>>> so the deny fails.
>>
>> Which platform and image?
>>
>> Pharo 7.0
>> Build information:
>> Pharo-7.0+alpha.build.410.sha.aa40385cd6fee69bb51d4a3afcacfb5f91eed206
>> (32 Bit)
>> Ubuntu 16.04
>>
>> 14915 run, 14859 passes, 31 skipped, 50 expected failures, 5 failures,
>> 1 errors, 0 unexpected passes
>> Failures:
>> ReleaseTest>>#testMethodsWithUnboundGlobals
>> TonelWriterTest>>#testWriteSnapshot
>> ReleaseTest>>#testUnpackagedClasses
>> ReleaseTest>>#testNoEmptyPackages
>> ReleaseTest>>#testThatAllMethodsArePackaged
>>
>> Errors:
>> ZnClientTests>>#testRedirect
>
> Hmm, strange. Does it failed each time ?
>
> It does not fail for me (macOS). Furthermore, all Zinc tests are green https://ci.inria.fr/pharo-contribution/job/ZincHTTPComponents/

It's failing quite rarely.  I think we just got (un)lucky this time.
#testRedirect connects to http://zn.stfx.eu, so it could well be an
intermittent internet connection issue.

What would we lose by restructuring the tests to use an internal
server?  My initial reaction is that an increase in test reliability
would outweigh any loss of testing.  And we could create manually run
tests for anything that can't be covered by the internal server tests.


Cheers,
Alistair





>> Cheers,
>> Alistair
>>
>>
>>
>>> It seems that the test setUp is wrong since
>>> #elementsCopyNonIdenticalWithoutEqualElements
>>> does not return a collection (of elements for which copy is not identical )
>>> without equal elements:"
>>>
>>> This seems like a permanent error rather than a transient one.
>>> How does this slip through CI testing?
>>> and what can be done about cleaning this up?
>>>
>>> cheers -ben
>>>
>>> P.S. With the loss of method version info is impossible to guess when this
>>> may have been introduced, to guess which Issue it was related to, to guess
>>> how I might approach fixing this myself.
>
>

Reply | Threaded
Open this post in threaded view
|

Re: failing tests in Pharo 7

Sven Van Caekenberghe-2


> On 7 Jan 2018, at 10:30, Alistair Grant <[hidden email]> wrote:
>
> Hi Sven.
>
> On 6 January 2018 at 22:55, Sven Van Caekenberghe <[hidden email]> wrote:
>>
>>
>>> On 6 Jan 2018, at 20:08, Alistair Grant <[hidden email]> wrote:
>>>
>>> Hi Ben,
>>>
>>> On 6 January 2018 at 12:09, Ben Coman <[hidden email]> wrote:
>>>> It great having a CI system to check all our tests a passing,
>>>> but I thought I'd go old school and run them from the TestRunner GUI
>>>> and found in a fresh latest Pharo 7 there are "128 failures, 63 errors"
>>>>
>>>> This one is interesting...
>>>> ArrayTest>>test0FixtureAsSetForIdentityMultiplinessTest
>>>> | anElement res |
>>>> self elementsCopyNonIdenticalWithoutEqualElements.
>>>> anElement := self elementsCopyNonIdenticalWithoutEqualElements anyOne.
>>>> self deny: anElement copy == anElement.  "<<<<FAILS"
>>>>
>>>> where anElement ==> SmallFloat64 (1.5)
>>>> so the deny fails.
>>>
>>> Which platform and image?
>>>
>>> Pharo 7.0
>>> Build information:
>>> Pharo-7.0+alpha.build.410.sha.aa40385cd6fee69bb51d4a3afcacfb5f91eed206
>>> (32 Bit)
>>> Ubuntu 16.04
>>>
>>> 14915 run, 14859 passes, 31 skipped, 50 expected failures, 5 failures,
>>> 1 errors, 0 unexpected passes
>>> Failures:
>>> ReleaseTest>>#testMethodsWithUnboundGlobals
>>> TonelWriterTest>>#testWriteSnapshot
>>> ReleaseTest>>#testUnpackagedClasses
>>> ReleaseTest>>#testNoEmptyPackages
>>> ReleaseTest>>#testThatAllMethodsArePackaged
>>>
>>> Errors:
>>> ZnClientTests>>#testRedirect
>>
>> Hmm, strange. Does it failed each time ?
>>
>> It does not fail for me (macOS). Furthermore, all Zinc tests are green https://ci.inria.fr/pharo-contribution/job/ZincHTTPComponents/
>
> It's failing quite rarely.  I think we just got (un)lucky this time.
> #testRedirect connects to http://zn.stfx.eu, so it could well be an
> intermittent internet connection issue.
>
> What would we lose by restructuring the tests to use an internal
> server?  My initial reaction is that an increase in test reliability
> would outweigh any loss of testing.  And we could create manually run
> tests for anything that can't be covered by the internal server tests.

I know, but most (functional) tests *ARE* internal (are running against a server created just for the test). The remaining are there because HTTP *IS* a network communication protocol where it is important to test against servers 'out there in the wild'. It would be too limiting to exclude the networking (with its connectivity and timing issues) and the fact that there are hundreds of server and TLS/HTTPS implementations out there.

Splitting the tests in two group might be a first step.

> Cheers,
> Alistair
>
>
>
>
>
>>> Cheers,
>>> Alistair
>>>
>>>
>>>
>>>> It seems that the test setUp is wrong since
>>>> #elementsCopyNonIdenticalWithoutEqualElements
>>>> does not return a collection (of elements for which copy is not identical )
>>>> without equal elements:"
>>>>
>>>> This seems like a permanent error rather than a transient one.
>>>> How does this slip through CI testing?
>>>> and what can be done about cleaning this up?
>>>>
>>>> cheers -ben
>>>>
>>>> P.S. With the loss of method version info is impossible to guess when this
>>>> may have been introduced, to guess which Issue it was related to, to guess
>>>> how I might approach fixing this myself.


Reply | Threaded
Open this post in threaded view
|

Re: failing tests in Pharo 7

Eliot Miranda-2
In reply to this post by alistairgrant
Hi All,

> On Jan 7, 2018, at 1:30 AM, Alistair Grant <[hidden email]> wrote:
>
> Hi Sven.
>
>> On 6 January 2018 at 22:55, Sven Van Caekenberghe <[hidden email]> wrote:
>>
>>
>>> On 6 Jan 2018, at 20:08, Alistair Grant <[hidden email]> wrote:
>>>
>>> Hi Ben,
>>>
>>>> On 6 January 2018 at 12:09, Ben Coman <[hidden email]> wrote:
>>>> It great having a CI system to check all our tests a passing,
>>>> but I thought I'd go old school and run them from the TestRunner GUI
>>>> and found in a fresh latest Pharo 7 there are "128 failures, 63 errors"
>>>>
>>>> This one is interesting...
>>>> ArrayTest>>test0FixtureAsSetForIdentityMultiplinessTest
>>>> | anElement res |
>>>> self elementsCopyNonIdenticalWithoutEqualElements.
>>>> anElement := self elementsCopyNonIdenticalWithoutEqualElements anyOne.
>>>> self deny: anElement copy == anElement.  "<<<<FAILS"
>>>>
>>>> where anElement ==> SmallFloat64 (1.5)
>>>> so the deny fails.

Shouldn't the test distinguish objects whose class answers true for isImmediateClass?  In 64-bits SmallIntegers, Characters /and/ SmallFloat64s are all immediate so will always be #== to any instance of the same value.  In 32-bits there are no instances of SmallFloat64.  In 64-bits any float whose exponent falls within the single precision range (which excludes Inf & NaN) will be represented as a SmallFloat64.

>>>
>>> Which platform and image?
>>>
>>> Pharo 7.0
>>> Build information:
>>> Pharo-7.0+alpha.build.410.sha.aa40385cd6fee69bb51d4a3afcacfb5f91eed206
>>> (32 Bit)
>>> Ubuntu 16.04
>>>
>>> 14915 run, 14859 passes, 31 skipped, 50 expected failures, 5 failures,
>>> 1 errors, 0 unexpected passes
>>> Failures:
>>> ReleaseTest>>#testMethodsWithUnboundGlobals
>>> TonelWriterTest>>#testWriteSnapshot
>>> ReleaseTest>>#testUnpackagedClasses
>>> ReleaseTest>>#testNoEmptyPackages
>>> ReleaseTest>>#testThatAllMethodsArePackaged
>>>
>>> Errors:
>>> ZnClientTests>>#testRedirect
>>
>> Hmm, strange. Does it failed each time ?
>>
>> It does not fail for me (macOS). Furthermore, all Zinc tests are green https://ci.inria.fr/pharo-contribution/job/ZincHTTPComponents/
>
> It's failing quite rarely.  I think we just got (un)lucky this time.
> #testRedirect connects to http://zn.stfx.eu, so it could well be an
> intermittent internet connection issue.
>
> What would we lose by restructuring the tests to use an internal
> server?  My initial reaction is that an increase in test reliability
> would outweigh any loss of testing.  And we could create manually run
> tests for anything that can't be covered by the internal server tests.
>
>
> Cheers,
> Alistair
>
>
>
>
>
>>> Cheers,
>>> Alistair
>>>
>>>
>>>
>>>> It seems that the test setUp is wrong since
>>>> #elementsCopyNonIdenticalWithoutEqualElements
>>>> does not return a collection (of elements for which copy is not identical )
>>>> without equal elements:"
>>>>
>>>> This seems like a permanent error rather than a transient one.
>>>> How does this slip through CI testing?
>>>> and what can be done about cleaning this up?
>>>>
>>>> cheers -ben
>>>>
>>>> P.S. With the loss of method version info is impossible to guess when this
>>>> may have been introduced, to guess which Issue it was related to, to guess
>>>> how I might approach fixing this myself.
>>
>>
>