SqueakSSLTest failure/errors on macOS?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
bpi
Reply | Threaded
Open this post in threaded view
|

SqueakSSLTest failure/errors on macOS?

bpi
Hi,

I have one failure in #testFacebookAPI and three errors in #testSSLSockets, #testSocketAccept, and #testSocketAccept in SqueakSSLTest on macOS Mojave 10.14.5.

I tried the latest trunk images 18618 with the release VMs and the latest VMs on bintray, both 32bit and 64bit:

Croquet Closure Cog[Spur] VM [CoInterpreterPrimitives VMMaker.oscog-eem.2530] 5.0.201906130203
Mac OS X built on Jun 13 2019 02:12:03 GMT Compiler: 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)
platform sources revision VM: 201906130203 https://github.com/OpenSmalltalk/opensmalltalk-vm.git Date: Wed Jun 12 19:03:53 2019 CommitHash: a85579a1 Plugins: 201906130203 https://github.com/OpenSmalltalk/opensmalltalk-vm.git
CoInterpreter VMMaker.oscog-eem.2530 uuid: 4d90ede0-0700-4d15-8173-2aaf2360b7d1 Jun 13 2019
StackToRegisterMappingCogit VMMaker.oscog-eem.2530 uuid: 4d90ede0-0700-4d15-8173-2aaf2360b7d1 Jun 13 2019

I would like to know if other Mac users can confirm these. If yes, does anyone know anything about these problems?

I assume the tests on macOS are not run during the trunk build, right?

Cheers,
Bernhard

Reply | Threaded
Open this post in threaded view
|

Re: SqueakSSLTest failure/errors on macOS?

Tobias Pape
Hi

> On 20.06.2019, at 19:53, Bernhard Pieber <[hidden email]> wrote:
>
> Hi,
>
> I have one failure in #testFacebookAPI and three errors in #testSSLSockets, #testSocketAccept, and #testSocketAccept in SqueakSSLTest on macOS Mojave 10.14.5.
>
> I tried the latest trunk images 18618 with the release VMs and the latest VMs on bintray, both 32bit and 64bit:
>
> Croquet Closure Cog[Spur] VM [CoInterpreterPrimitives VMMaker.oscog-eem.2530] 5.0.201906130203
> Mac OS X built on Jun 13 2019 02:12:03 GMT Compiler: 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)
> platform sources revision VM: 201906130203 https://github.com/OpenSmalltalk/opensmalltalk-vm.git Date: Wed Jun 12 19:03:53 2019 CommitHash: a85579a1 Plugins: 201906130203 https://github.com/OpenSmalltalk/opensmalltalk-vm.git
> CoInterpreter VMMaker.oscog-eem.2530 uuid: 4d90ede0-0700-4d15-8173-2aaf2360b7d1 Jun 13 2019
> StackToRegisterMappingCogit VMMaker.oscog-eem.2530 uuid: 4d90ede0-0700-4d15-8173-2aaf2360b7d1 Jun 13 2019
>
> I would like to know if other Mac users can confirm these. If yes, does anyone know anything about these problems?

The failure/errors have been there since I have looked at all that code.
I have no idea why they do not work. They use the SecureSocket variant of SqueakSSL, the rest of the system uses the SecureSocketStream IIRC, and that works…

>
> I assume the tests on macOS are not run during the trunk build, right?

I am not sure. It seems that these are not part of the failed tests on travis:

        https://travis-ci.org/squeak-smalltalk/squeak-app/jobs/548131159#L2326

But Really, I have no idea…

Best regards
        -Tobias


>
> Cheers,
> Bernhard
>



bpi
Reply | Threaded
Open this post in threaded view
|

Re: SqueakSSLTest failure/errors on macOS?

bpi
Thank you for taking the time to answer. At least I know I am not the only one.

Your link to the travis job log helped me find the travis config:
https://travis-ci.org/squeak-smalltalk/squeak-app/jobs/548131159/config

According to this the build seems to be run on macOS 10.11. So it is strange that the test failures don't show up in the job log. Maybe they succeed on travis. However, that is hard to imagine.

Anyone else on a Mac for whom the SqueakSSLTest succeeds?

- Bernhard

> Am 20.06.2019 um 20:07 schrieb Tobias Pape <[hidden email]>:
>
> Hi
>
>> On 20.06.2019, at 19:53, Bernhard Pieber <[hidden email]> wrote:
>>
>> Hi,
>>
>> I have one failure in #testFacebookAPI and three errors in #testSSLSockets, #testSocketAccept, and #testSocketAccept in SqueakSSLTest on macOS Mojave 10.14.5.
>>
>> I tried the latest trunk images 18618 with the release VMs and the latest VMs on bintray, both 32bit and 64bit:
>>
>> Croquet Closure Cog[Spur] VM [CoInterpreterPrimitives VMMaker.oscog-eem.2530] 5.0.201906130203
>> Mac OS X built on Jun 13 2019 02:12:03 GMT Compiler: 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)
>> platform sources revision VM: 201906130203 https://github.com/OpenSmalltalk/opensmalltalk-vm.git Date: Wed Jun 12 19:03:53 2019 CommitHash: a85579a1 Plugins: 201906130203 https://github.com/OpenSmalltalk/opensmalltalk-vm.git
>> CoInterpreter VMMaker.oscog-eem.2530 uuid: 4d90ede0-0700-4d15-8173-2aaf2360b7d1 Jun 13 2019
>> StackToRegisterMappingCogit VMMaker.oscog-eem.2530 uuid: 4d90ede0-0700-4d15-8173-2aaf2360b7d1 Jun 13 2019
>>
>> I would like to know if other Mac users can confirm these. If yes, does anyone know anything about these problems?
>
> The failure/errors have been there since I have looked at all that code.
> I have no idea why they do not work. They use the SecureSocket variant of SqueakSSL, the rest of the system uses the SecureSocketStream IIRC, and that works…
>
>>
>> I assume the tests on macOS are not run during the trunk build, right?
>
> I am not sure. It seems that these are not part of the failed tests on travis:
>
> https://travis-ci.org/squeak-smalltalk/squeak-app/jobs/548131159#L2326
>
> But Really, I have no idea…
>
> Best regards
> -Tobias
>
>
>>
>> Cheers,
>> Bernhard


Reply | Threaded
Open this post in threaded view
|

Re: SqueakSSLTest failure/errors on macOS?

fniephaus


On Fri, 21 Jun 2019 at 11:45 am, Bernhard Pieber <[hidden email]> wrote:
Thank you for taking the time to answer. At least I know I am not the only one.

Your link to the travis job log helped me find the travis config:
https://travis-ci.org/squeak-smalltalk/squeak-app/jobs/548131159/config

The SUnit tests are being executed by smalltalkCI and in its config, SqueakSSLTest is excluded [1].
BTW: I think builds succeed, even if tests fail. Sometimes, the VM even segfaults during testing.

Fabio




According to this the build seems to be run on macOS 10.11. So it is strange that the test failures don't show up in the job log. Maybe they succeed on travis. However, that is hard to imagine.

Anyone else on a Mac for whom the SqueakSSLTest succeeds?

- Bernhard

> Am 20.06.2019 um 20:07 schrieb Tobias Pape <[hidden email]>:
>
> Hi
>
>> On 20.06.2019, at 19:53, Bernhard Pieber <[hidden email]> wrote:
>>
>> Hi,
>>
>> I have one failure in #testFacebookAPI and three errors in #testSSLSockets, #testSocketAccept, and #testSocketAccept in SqueakSSLTest on macOS Mojave 10.14.5.
>>
>> I tried the latest trunk images 18618 with the release VMs and the latest VMs on bintray, both 32bit and 64bit:
>>
>> Croquet Closure Cog[Spur] VM [CoInterpreterPrimitives VMMaker.oscog-eem.2530] 5.0.201906130203
>> Mac OS X built on Jun 13 2019 02:12:03 GMT Compiler: 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)
>> platform sources revision VM: 201906130203 https://github.com/OpenSmalltalk/opensmalltalk-vm.git Date: Wed Jun 12 19:03:53 2019 CommitHash: a85579a1 Plugins: 201906130203 https://github.com/OpenSmalltalk/opensmalltalk-vm.git
>> CoInterpreter VMMaker.oscog-eem.2530 uuid: 4d90ede0-0700-4d15-8173-2aaf2360b7d1 Jun 13 2019
>> StackToRegisterMappingCogit VMMaker.oscog-eem.2530 uuid: 4d90ede0-0700-4d15-8173-2aaf2360b7d1 Jun 13 2019
>>
>> I would like to know if other Mac users can confirm these. If yes, does anyone know anything about these problems?
>
> The failure/errors have been there since I have looked at all that code.
> I have no idea why they do not work. They use the SecureSocket variant of SqueakSSL, the rest of the system uses the SecureSocketStream IIRC, and that works…
>
>>
>> I assume the tests on macOS are not run during the trunk build, right?
>
> I am not sure. It seems that these are not part of the failed tests on travis:
>
>       https://travis-ci.org/squeak-smalltalk/squeak-app/jobs/548131159#L2326
>
> But Really, I have no idea…
>
> Best regards
>       -Tobias
>
>
>>
>> Cheers,
>> Bernhard




bpi
Reply | Threaded
Open this post in threaded view
|

Re: SqueakSSLTest failure/errors on macOS?

bpi
Thanks for the clarification. I did not know there was a separate mechanism for ignoring tests in smalltalkCI beside #expectedFailures in the image. Do you happen to know what is the rationale for that? Especially when the build succeeds anyway.

It is a pity that the trunk is not green all the time. I wonder how difficult it would be to include information on failing tests in the download packages say as a known_test_failures.txt alongside the image file. It might help to increase the urge to fix them. Besides it would make it easier to find out whether one has introduced a regression.

- Bernhard

> Am 21.06.2019 um 15:05 schrieb Fabio Niephaus <[hidden email]>:
>
>
>
> On Fri, 21 Jun 2019 at 11:45 am, Bernhard Pieber <[hidden email]> wrote:
> Thank you for taking the time to answer. At least I know I am not the only one.
>
> Your link to the travis job log helped me find the travis config:
> https://travis-ci.org/squeak-smalltalk/squeak-app/jobs/548131159/config
>
> The SUnit tests are being executed by smalltalkCI and in its config, SqueakSSLTest is excluded [1].
> BTW: I think builds succeed, even if tests fail. Sometimes, the VM even segfaults during testing.
>
> Fabio
>
> [1] https://github.com/squeak-smalltalk/squeak-app/blob/squeak-trunk/smalltalk-ci/Squeak-trunk-64.ston#L10
>
>
>
> According to this the build seems to be run on macOS 10.11. So it is strange that the test failures don't show up in the job log. Maybe they succeed on travis. However, that is hard to imagine.
>
> Anyone else on a Mac for whom the SqueakSSLTest succeeds?
>
> - Bernhard
>
> > Am 20.06.2019 um 20:07 schrieb Tobias Pape <[hidden email]>:
> >
> > Hi
> >
> >> On 20.06.2019, at 19:53, Bernhard Pieber <[hidden email]> wrote:
> >>
> >> Hi,
> >>
> >> I have one failure in #testFacebookAPI and three errors in #testSSLSockets, #testSocketAccept, and #testSocketAccept in SqueakSSLTest on macOS Mojave 10.14.5.
> >>
> >> I tried the latest trunk images 18618 with the release VMs and the latest VMs on bintray, both 32bit and 64bit:
> >>
> >> Croquet Closure Cog[Spur] VM [CoInterpreterPrimitives VMMaker.oscog-eem.2530] 5.0.201906130203
> >> Mac OS X built on Jun 13 2019 02:12:03 GMT Compiler: 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)
> >> platform sources revision VM: 201906130203 https://github.com/OpenSmalltalk/opensmalltalk-vm.git Date: Wed Jun 12 19:03:53 2019 CommitHash: a85579a1 Plugins: 201906130203 https://github.com/OpenSmalltalk/opensmalltalk-vm.git
> >> CoInterpreter VMMaker.oscog-eem.2530 uuid: 4d90ede0-0700-4d15-8173-2aaf2360b7d1 Jun 13 2019
> >> StackToRegisterMappingCogit VMMaker.oscog-eem.2530 uuid: 4d90ede0-0700-4d15-8173-2aaf2360b7d1 Jun 13 2019
> >>
> >> I would like to know if other Mac users can confirm these. If yes, does anyone know anything about these problems?
> >
> > The failure/errors have been there since I have looked at all that code.
> > I have no idea why they do not work. They use the SecureSocket variant of SqueakSSL, the rest of the system uses the SecureSocketStream IIRC, and that works…
> >
> >>
> >> I assume the tests on macOS are not run during the trunk build, right?
> >
> > I am not sure. It seems that these are not part of the failed tests on travis:
> >
> >       https://travis-ci.org/squeak-smalltalk/squeak-app/jobs/548131159#L2326
> >
> > But Really, I have no idea…
> >
> > Best regards
> >       -Tobias
> >
> >
> >>
> >> Cheers,
> >> Bernhard
>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: SqueakSSLTest failure/errors on macOS?

fniephaus


On Fri, 21 Jun 2019 at 4:20 pm, Bernhard Pieber <[hidden email]> wrote:
Thanks for the clarification. I did not know there was a separate mechanism for ignoring tests in smalltalkCI beside #expectedFailures in the image. Do you happen to know what is the rationale for that? Especially when the build succeeds anyway.

During the preparation of, I think, the Squeak 5.1 release, we wanted to fix all tests, so we set up the CI to give us a todo list. Eventually, I wanted to let trunk builds depend on the test results, so that we'd always get notified if test break again. Unfortunately, we couldn't fix all tests in time for the release and some still fail. We currently run all builds on macOS, so that we can sign the app bundles. We probably need something like this for Windows too at some point. But as far as I can remember, SqueakSSLTest have never passed on macOS.



It is a pity that the trunk is not green all the time. I wonder how difficult it would be to include information on failing tests in the download packages say as a known_test_failures.txt alongside the image file. It might help to increase the urge to fix them. Besides it would make it easier to find out whether one has introduced a regression.

I agree. You could programmatically download and parse the last X build results using Travis' REST API. That's why we haven't uploaded them anywhere. I'd rather spend some time fixing bugs that are failing in the latest trunk and then flip the switch, so that we get notified if someone breaks something. Some test are documenting feature requests or VM-level bugs that cannot be fixed in the image. So we probably still need to ignore some in the smalltalkCI config. Also, sometimes builds timeout due to fragile network handling or VM segfaults. Some hardening would also be necessary I guess.

Fabio



- Bernhard

> Am 21.06.2019 um 15:05 schrieb Fabio Niephaus <[hidden email]>:
>
>
>
> On Fri, 21 Jun 2019 at 11:45 am, Bernhard Pieber <[hidden email]> wrote:
> Thank you for taking the time to answer. At least I know I am not the only one.
>
> Your link to the travis job log helped me find the travis config:
> https://travis-ci.org/squeak-smalltalk/squeak-app/jobs/548131159/config
>
> The SUnit tests are being executed by smalltalkCI and in its config, SqueakSSLTest is excluded [1].
> BTW: I think builds succeed, even if tests fail. Sometimes, the VM even segfaults during testing.
>
> Fabio
>
> [1] https://github.com/squeak-smalltalk/squeak-app/blob/squeak-trunk/smalltalk-ci/Squeak-trunk-64.ston#L10
>
>
>
> According to this the build seems to be run on macOS 10.11. So it is strange that the test failures don't show up in the job log. Maybe they succeed on travis. However, that is hard to imagine.
>
> Anyone else on a Mac for whom the SqueakSSLTest succeeds?
>
> - Bernhard
>
> > Am 20.06.2019 um 20:07 schrieb Tobias Pape <[hidden email]>:
> >
> > Hi
> >
> >> On 20.06.2019, at 19:53, Bernhard Pieber <[hidden email]> wrote:
> >>
> >> Hi,
> >>
> >> I have one failure in #testFacebookAPI and three errors in #testSSLSockets, #testSocketAccept, and #testSocketAccept in SqueakSSLTest on macOS Mojave 10.14.5.
> >>
> >> I tried the latest trunk images 18618 with the release VMs and the latest VMs on bintray, both 32bit and 64bit:
> >>
> >> Croquet Closure Cog[Spur] VM [CoInterpreterPrimitives VMMaker.oscog-eem.2530] 5.0.201906130203
> >> Mac OS X built on Jun 13 2019 02:12:03 GMT Compiler: 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)
> >> platform sources revision VM: 201906130203 https://github.com/OpenSmalltalk/opensmalltalk-vm.git Date: Wed Jun 12 19:03:53 2019 CommitHash: a85579a1 Plugins: 201906130203 https://github.com/OpenSmalltalk/opensmalltalk-vm.git
> >> CoInterpreter VMMaker.oscog-eem.2530 uuid: 4d90ede0-0700-4d15-8173-2aaf2360b7d1 Jun 13 2019
> >> StackToRegisterMappingCogit VMMaker.oscog-eem.2530 uuid: 4d90ede0-0700-4d15-8173-2aaf2360b7d1 Jun 13 2019
> >>
> >> I would like to know if other Mac users can confirm these. If yes, does anyone know anything about these problems?
> >
> > The failure/errors have been there since I have looked at all that code.
> > I have no idea why they do not work. They use the SecureSocket variant of SqueakSSL, the rest of the system uses the SecureSocketStream IIRC, and that works…
> >
> >>
> >> I assume the tests on macOS are not run during the trunk build, right?
> >
> > I am not sure. It seems that these are not part of the failed tests on travis:
> >
> >       https://travis-ci.org/squeak-smalltalk/squeak-app/jobs/548131159#L2326
> >
> > But Really, I have no idea…
> >
> > Best regards
> >       -Tobias
> >
> >
> >>
> >> Cheers,
> >> Bernhard
>
>
>




Reply | Threaded
Open this post in threaded view
|

Re: SqueakSSLTest failure/errors on macOS?

Chris Muller-3
In reply to this post by bpi
Hi Bernhard,

> Thanks for the clarification. I did not know there was a separate mechanism for ignoring tests in smalltalkCI beside #expectedFailures in the image. Do you happen to know what is the rationale for that? Especially when the build succeeds anyway.

The three-state Status Indicator ( (green | yellow | red) = (okay |
warning | error) ) is one of the most widely-studied and important
outputs of many applications.  Unfortunately for SUnit, having tests
that are known not to pass on certain platforms basically destroys
that Status output.

> It is a pity that the trunk is not green all the time.

Until we fix our use of #expectedFailures, it'll never report green.
The primary requirement of SUnit is supposed to report the status of
the software, and relieve users from needing to dig into the details
just to ascertain that.  In fact, that digging is actually 95% of the
work of running the tests oneself, the other 5% being the single click
launch of the Suite.

So, that external patch file specifying internal class names is a
desperate hack trying to make up for #expectedFailures not doing what
it's supposed to.  We really should fix our SUnit.

> I wonder how difficult it would be to include information on failing tests in the download packages say as a known_test_failures.txt alongside the image file.

+1.  But this doesn't fix the quick output Status, so we should still
fix SUnit, too.

> It might help to increase the urge to fix them.

I think we will always have some #expectedFailures, though.  It seems
to be a useful and necessary feature (except for how it subverts the
green Status, of course).

> Besides it would make it easier to find out whether one has introduced a regression.

Exactly.

Although Kent Beck's original SUnit article [1] didn't anticipate the
need for an #expectedFailures mechanism, he did, under "Philosophy",
clearly define a "failure":

         "A failure is an anticipated problem. When you write tests,
you check for expected results. If you get a different answer, that is
a failure."

Additionally, check out the original 2003 implementation of
TestResult>>#passed still in trunk:

        passed
              ^ self expectedPasses, self expectedDefects

There it is.  "passed" includes the #expectedDefects, which are the
'errors' + 'failures' which are expected to fail, so this original
part of the implementation, too, is consistent with Beck's definition,
above.  It also shows my own hack fix (SUnit-cmm.116.mcz in the Inbox)
is not a complete and/or proper fix, but it has the correct intention.

We definitely should find a consensus on a fix for this.

Best,
  Chris

[1] - http://swing.fit.cvut.cz/projects/stx/doc/online/english/tools/misc/testfram.htm

Reply | Threaded
Open this post in threaded view
|

Re: SqueakSSLTest failure/errors on macOS?

marcel.taeumel
Hi all,

I think that SUnit's traffic light would be more reliable if we (1) introduce #skippedTests (being as flexible as #expectedFailures), (2) migrate all expected failures to that new form of skipped tests, and (3) deprecate the idea of expected failures in SUnit. That is, #skippedTests *should not run* because on certain platforms, they don't make sense. In all other cases, tests are just tests and are expected to pass. Consequently, there would be no arguing about how to handle "unexpected passes" in tools anymore. The traffic light would be fine.

Best,
Marcel

Am 22.06.2019 00:24:45 schrieb Chris Muller <[hidden email]>:

Hi Bernhard,

> Thanks for the clarification. I did not know there was a separate mechanism for ignoring tests in smalltalkCI beside #expectedFailures in the image. Do you happen to know what is the rationale for that? Especially when the build succeeds anyway.

The three-state Status Indicator ( (green | yellow | red) = (okay |
warning | error) ) is one of the most widely-studied and important
outputs of many applications. Unfortunately for SUnit, having tests
that are known not to pass on certain platforms basically destroys
that Status output.

> It is a pity that the trunk is not green all the time.

Until we fix our use of #expectedFailures, it'll never report green.
The primary requirement of SUnit is supposed to report the status of
the software, and relieve users from needing to dig into the details
just to ascertain that. In fact, that digging is actually 95% of the
work of running the tests oneself, the other 5% being the single click
launch of the Suite.

So, that external patch file specifying internal class names is a
desperate hack trying to make up for #expectedFailures not doing what
it's supposed to. We really should fix our SUnit.

> I wonder how difficult it would be to include information on failing tests in the download packages say as a known_test_failures.txt alongside the image file.

+1. But this doesn't fix the quick output Status, so we should still
fix SUnit, too.

> It might help to increase the urge to fix them.

I think we will always have some #expectedFailures, though. It seems
to be a useful and necessary feature (except for how it subverts the
green Status, of course).

> Besides it would make it easier to find out whether one has introduced a regression.

Exactly.

Although Kent Beck's original SUnit article [1] didn't anticipate the
need for an #expectedFailures mechanism, he did, under "Philosophy",
clearly define a "failure":

"A failure is an anticipated problem. When you write tests,
you check for expected results. If you get a different answer, that is
a failure."

Additionally, check out the original 2003 implementation of
TestResult>>#passed still in trunk:

passed
^ self expectedPasses, self expectedDefects

There it is. "passed" includes the #expectedDefects, which are the
'errors' + 'failures' which are expected to fail, so this original
part of the implementation, too, is consistent with Beck's definition,
above. It also shows my own hack fix (SUnit-cmm.116.mcz in the Inbox)
is not a complete and/or proper fix, but it has the correct intention.

We definitely should find a consensus on a fix for this.

Best,
Chris

[1] - http://swing.fit.cvut.cz/projects/stx/doc/online/english/tools/misc/testfram.htm



Reply | Threaded
Open this post in threaded view
|

Re: SqueakSSLTest failure/errors on macOS?

Chris Muller-3
Hey Marcel,

Consider future readers of the #test.. methods which are skipped in this way -- all the code there with no conditions on its execution.  Then they run the Suite and it reports all green, woo hoo!  Even if they knew about #skippedTests going in, it's easy to imagine them forgetting to cross-check their code reading against that list, so the #test.. methods they just read in the browser were implicitly conveyed to them by the system as "passing" when, in fact, they didn't even run.

SUnit derives its value by efficient and correct conveyance of the status of the software.

I'm sure we'll figure something out.

Best,
  Chris


On Mon, Jun 24, 2019 at 1:12 AM Marcel Taeumel <[hidden email]> wrote:
Hi all,

I think that SUnit's traffic light would be more reliable if we (1) introduce #skippedTests (being as flexible as #expectedFailures), (2) migrate all expected failures to that new form of skipped tests, and (3) deprecate the idea of expected failures in SUnit. That is, #skippedTests *should not run* because on certain platforms, they don't make sense. In all other cases, tests are just tests and are expected to pass. Consequently, there would be no arguing about how to handle "unexpected passes" in tools anymore. The traffic light would be fine.

Best,
Marcel

Am 22.06.2019 00:24:45 schrieb Chris Muller <[hidden email]>:

Hi Bernhard,

> Thanks for the clarification. I did not know there was a separate mechanism for ignoring tests in smalltalkCI beside #expectedFailures in the image. Do you happen to know what is the rationale for that? Especially when the build succeeds anyway.

The three-state Status Indicator ( (green | yellow | red) = (okay |
warning | error) ) is one of the most widely-studied and important
outputs of many applications. Unfortunately for SUnit, having tests
that are known not to pass on certain platforms basically destroys
that Status output.

> It is a pity that the trunk is not green all the time.

Until we fix our use of #expectedFailures, it'll never report green.
The primary requirement of SUnit is supposed to report the status of
the software, and relieve users from needing to dig into the details
just to ascertain that. In fact, that digging is actually 95% of the
work of running the tests oneself, the other 5% being the single click
launch of the Suite.

So, that external patch file specifying internal class names is a
desperate hack trying to make up for #expectedFailures not doing what
it's supposed to. We really should fix our SUnit.

> I wonder how difficult it would be to include information on failing tests in the download packages say as a known_test_failures.txt alongside the image file.

+1. But this doesn't fix the quick output Status, so we should still
fix SUnit, too.

> It might help to increase the urge to fix them.

I think we will always have some #expectedFailures, though. It seems
to be a useful and necessary feature (except for how it subverts the
green Status, of course).

> Besides it would make it easier to find out whether one has introduced a regression.

Exactly.

Although Kent Beck's original SUnit article [1] didn't anticipate the
need for an #expectedFailures mechanism, he did, under "Philosophy",
clearly define a "failure":

"A failure is an anticipated problem. When you write tests,
you check for expected results. If you get a different answer, that is
a failure."

Additionally, check out the original 2003 implementation of
TestResult>>#passed still in trunk:

passed
^ self expectedPasses, self expectedDefects

There it is. "passed" includes the #expectedDefects, which are the
'errors' + 'failures' which are expected to fail, so this original
part of the implementation, too, is consistent with Beck's definition,
above. It also shows my own hack fix (SUnit-cmm.116.mcz in the Inbox)
is not a complete and/or proper fix, but it has the correct intention.

We definitely should find a consensus on a fix for this.

Best,
Chris

[1] - http://swing.fit.cvut.cz/projects/stx/doc/online/english/tools/misc/testfram.htm




Reply | Threaded
Open this post in threaded view
|

Re: SqueakSSLTest failure/errors on macOS?

marcel.taeumel
Hi Chris,

so are you looking for information more close to an individaul test method or test result? I see two complementary approaches here: a) re-design SUnit test representations (i.e., the GUI) and b) re-design SUnit test specifications (i.e., the code).

Best,
Marcel

Am 25.06.2019 03:30:14 schrieb Chris Muller <[hidden email]>:

Hey Marcel,

Consider future readers of the #test.. methods which are skipped in this way -- all the code there with no conditions on its execution.  Then they run the Suite and it reports all green, woo hoo!  Even if they knew about #skippedTests going in, it's easy to imagine them forgetting to cross-check their code reading against that list, so the #test.. methods they just read in the browser were implicitly conveyed to them by the system as "passing" when, in fact, they didn't even run.

SUnit derives its value by efficient and correct conveyance of the status of the software.

I'm sure we'll figure something out.

Best,
  Chris


On Mon, Jun 24, 2019 at 1:12 AM Marcel Taeumel <[hidden email]> wrote:
Hi all,

I think that SUnit's traffic light would be more reliable if we (1) introduce #skippedTests (being as flexible as #expectedFailures), (2) migrate all expected failures to that new form of skipped tests, and (3) deprecate the idea of expected failures in SUnit. That is, #skippedTests *should not run* because on certain platforms, they don't make sense. In all other cases, tests are just tests and are expected to pass. Consequently, there would be no arguing about how to handle "unexpected passes" in tools anymore. The traffic light would be fine.

Best,
Marcel

Am 22.06.2019 00:24:45 schrieb Chris Muller <[hidden email]>:

Hi Bernhard,

> Thanks for the clarification. I did not know there was a separate mechanism for ignoring tests in smalltalkCI beside #expectedFailures in the image. Do you happen to know what is the rationale for that? Especially when the build succeeds anyway.

The three-state Status Indicator ( (green | yellow | red) = (okay |
warning | error) ) is one of the most widely-studied and important
outputs of many applications. Unfortunately for SUnit, having tests
that are known not to pass on certain platforms basically destroys
that Status output.

> It is a pity that the trunk is not green all the time.

Until we fix our use of #expectedFailures, it'll never report green.
The primary requirement of SUnit is supposed to report the status of
the software, and relieve users from needing to dig into the details
just to ascertain that. In fact, that digging is actually 95% of the
work of running the tests oneself, the other 5% being the single click
launch of the Suite.

So, that external patch file specifying internal class names is a
desperate hack trying to make up for #expectedFailures not doing what
it's supposed to. We really should fix our SUnit.

> I wonder how difficult it would be to include information on failing tests in the download packages say as a known_test_failures.txt alongside the image file.

+1. But this doesn't fix the quick output Status, so we should still
fix SUnit, too.

> It might help to increase the urge to fix them.

I think we will always have some #expectedFailures, though. It seems
to be a useful and necessary feature (except for how it subverts the
green Status, of course).

> Besides it would make it easier to find out whether one has introduced a regression.

Exactly.

Although Kent Beck's original SUnit article [1] didn't anticipate the
need for an #expectedFailures mechanism, he did, under "Philosophy",
clearly define a "failure":

"A failure is an anticipated problem. When you write tests,
you check for expected results. If you get a different answer, that is
a failure."

Additionally, check out the original 2003 implementation of
TestResult>>#passed still in trunk:

passed
^ self expectedPasses, self expectedDefects

There it is. "passed" includes the #expectedDefects, which are the
'errors' + 'failures' which are expected to fail, so this original
part of the implementation, too, is consistent with Beck's definition,
above. It also shows my own hack fix (SUnit-cmm.116.mcz in the Inbox)
is not a complete and/or proper fix, but it has the correct intention.

We definitely should find a consensus on a fix for this.

Best,
Chris

[1] - http://swing.fit.cvut.cz/projects/stx/doc/online/english/tools/misc/testfram.htm