SqueakSSLTest failure/errors on macOS?
Locked
 
|
Hi,
I have one failure in #testFacebookAPI and three errors in #testSSLSockets, #testSocketAccept, and #testSocketAccept in SqueakSSLTest on macOS Mojave 10.14.5. I tried the latest trunk images 18618 with the release VMs and the latest VMs on bintray, both 32bit and 64bit: Croquet Closure Cog[Spur] VM [CoInterpreterPrimitives VMMaker.oscog-eem.2530] 5.0.201906130203 Mac OS X built on Jun 13 2019 02:12:03 GMT Compiler: 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42) platform sources revision VM: 201906130203 https://github.com/OpenSmalltalk/opensmalltalk-vm.git Date: Wed Jun 12 19:03:53 2019 CommitHash: a85579a1 Plugins: 201906130203 https://github.com/OpenSmalltalk/opensmalltalk-vm.git CoInterpreter VMMaker.oscog-eem.2530 uuid: 4d90ede0-0700-4d15-8173-2aaf2360b7d1 Jun 13 2019 StackToRegisterMappingCogit VMMaker.oscog-eem.2530 uuid: 4d90ede0-0700-4d15-8173-2aaf2360b7d1 Jun 13 2019 I would like to know if other Mac users can confirm these. If yes, does anyone know anything about these problems? I assume the tests on macOS are not run during the trunk build, right? Cheers, Bernhard |
Re: SqueakSSLTest failure/errors on macOS?
|
|
Hi
> On 20.06.2019, at 19:53, Bernhard Pieber <[hidden email]> wrote: > > Hi, > > I have one failure in #testFacebookAPI and three errors in #testSSLSockets, #testSocketAccept, and #testSocketAccept in SqueakSSLTest on macOS Mojave 10.14.5. > > I tried the latest trunk images 18618 with the release VMs and the latest VMs on bintray, both 32bit and 64bit: > > Croquet Closure Cog[Spur] VM [CoInterpreterPrimitives VMMaker.oscog-eem.2530] 5.0.201906130203 > Mac OS X built on Jun 13 2019 02:12:03 GMT Compiler: 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42) > platform sources revision VM: 201906130203 https://github.com/OpenSmalltalk/opensmalltalk-vm.git Date: Wed Jun 12 19:03:53 2019 CommitHash: a85579a1 Plugins: 201906130203 https://github.com/OpenSmalltalk/opensmalltalk-vm.git > CoInterpreter VMMaker.oscog-eem.2530 uuid: 4d90ede0-0700-4d15-8173-2aaf2360b7d1 Jun 13 2019 > StackToRegisterMappingCogit VMMaker.oscog-eem.2530 uuid: 4d90ede0-0700-4d15-8173-2aaf2360b7d1 Jun 13 2019 > > I would like to know if other Mac users can confirm these. If yes, does anyone know anything about these problems? The failure/errors have been there since I have looked at all that code. I have no idea why they do not work. They use the SecureSocket variant of SqueakSSL, the rest of the system uses the SecureSocketStream IIRC, and that works… > > I assume the tests on macOS are not run during the trunk build, right? I am not sure. It seems that these are not part of the failed tests on travis: https://travis-ci.org/squeak-smalltalk/squeak-app/jobs/548131159#L2326 But Really, I have no idea… Best regards -Tobias > > Cheers, > Bernhard > |
|
|
Thank you for taking the time to answer. At least I know I am not the only one.
Your link to the travis job log helped me find the travis config: https://travis-ci.org/squeak-smalltalk/squeak-app/jobs/548131159/config According to this the build seems to be run on macOS 10.11. So it is strange that the test failures don't show up in the job log. Maybe they succeed on travis. However, that is hard to imagine. Anyone else on a Mac for whom the SqueakSSLTest succeeds? - Bernhard > Am 20.06.2019 um 20:07 schrieb Tobias Pape <[hidden email]>: > > Hi > >> On 20.06.2019, at 19:53, Bernhard Pieber <[hidden email]> wrote: >> >> Hi, >> >> I have one failure in #testFacebookAPI and three errors in #testSSLSockets, #testSocketAccept, and #testSocketAccept in SqueakSSLTest on macOS Mojave 10.14.5. >> >> I tried the latest trunk images 18618 with the release VMs and the latest VMs on bintray, both 32bit and 64bit: >> >> Croquet Closure Cog[Spur] VM [CoInterpreterPrimitives VMMaker.oscog-eem.2530] 5.0.201906130203 >> Mac OS X built on Jun 13 2019 02:12:03 GMT Compiler: 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42) >> platform sources revision VM: 201906130203 https://github.com/OpenSmalltalk/opensmalltalk-vm.git Date: Wed Jun 12 19:03:53 2019 CommitHash: a85579a1 Plugins: 201906130203 https://github.com/OpenSmalltalk/opensmalltalk-vm.git >> CoInterpreter VMMaker.oscog-eem.2530 uuid: 4d90ede0-0700-4d15-8173-2aaf2360b7d1 Jun 13 2019 >> StackToRegisterMappingCogit VMMaker.oscog-eem.2530 uuid: 4d90ede0-0700-4d15-8173-2aaf2360b7d1 Jun 13 2019 >> >> I would like to know if other Mac users can confirm these. If yes, does anyone know anything about these problems? > > The failure/errors have been there since I have looked at all that code. > I have no idea why they do not work. They use the SecureSocket variant of SqueakSSL, the rest of the system uses the SecureSocketStream IIRC, and that works… > >> >> I assume the tests on macOS are not run during the trunk build, right? > > I am not sure. It seems that these are not part of the failed tests on travis: > > https://travis-ci.org/squeak-smalltalk/squeak-app/jobs/548131159#L2326 > > But Really, I have no idea… > > Best regards > -Tobias > > >> >> Cheers, >> Bernhard |
|
On Fri, 21 Jun 2019 at 11:45 am, Bernhard Pieber <[hidden email]> wrote: Thank you for taking the time to answer. At least I know I am not the only one. The SUnit tests are being executed by smalltalkCI and in its config, SqueakSSLTest is excluded [1]. BTW: I think builds succeed, even if tests fail. Sometimes, the VM even segfaults during testing. Fabio
|
|
|
Thanks for the clarification. I did not know there was a separate mechanism for ignoring tests in smalltalkCI beside #expectedFailures in the image. Do you happen to know what is the rationale for that? Especially when the build succeeds anyway.
It is a pity that the trunk is not green all the time. I wonder how difficult it would be to include information on failing tests in the download packages say as a known_test_failures.txt alongside the image file. It might help to increase the urge to fix them. Besides it would make it easier to find out whether one has introduced a regression. - Bernhard > Am 21.06.2019 um 15:05 schrieb Fabio Niephaus <[hidden email]>: > > > > On Fri, 21 Jun 2019 at 11:45 am, Bernhard Pieber <[hidden email]> wrote: > Thank you for taking the time to answer. At least I know I am not the only one. > > Your link to the travis job log helped me find the travis config: > https://travis-ci.org/squeak-smalltalk/squeak-app/jobs/548131159/config > > The SUnit tests are being executed by smalltalkCI and in its config, SqueakSSLTest is excluded [1]. > BTW: I think builds succeed, even if tests fail. Sometimes, the VM even segfaults during testing. > > Fabio > > [1] https://github.com/squeak-smalltalk/squeak-app/blob/squeak-trunk/smalltalk-ci/Squeak-trunk-64.ston#L10 > > > > According to this the build seems to be run on macOS 10.11. So it is strange that the test failures don't show up in the job log. Maybe they succeed on travis. However, that is hard to imagine. > > Anyone else on a Mac for whom the SqueakSSLTest succeeds? > > - Bernhard > > > Am 20.06.2019 um 20:07 schrieb Tobias Pape <[hidden email]>: > > > > Hi > > > >> On 20.06.2019, at 19:53, Bernhard Pieber <[hidden email]> wrote: > >> > >> Hi, > >> > >> I have one failure in #testFacebookAPI and three errors in #testSSLSockets, #testSocketAccept, and #testSocketAccept in SqueakSSLTest on macOS Mojave 10.14.5. > >> > >> I tried the latest trunk images 18618 with the release VMs and the latest VMs on bintray, both 32bit and 64bit: > >> > >> Croquet Closure Cog[Spur] VM [CoInterpreterPrimitives VMMaker.oscog-eem.2530] 5.0.201906130203 > >> Mac OS X built on Jun 13 2019 02:12:03 GMT Compiler: 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42) > >> platform sources revision VM: 201906130203 https://github.com/OpenSmalltalk/opensmalltalk-vm.git Date: Wed Jun 12 19:03:53 2019 CommitHash: a85579a1 Plugins: 201906130203 https://github.com/OpenSmalltalk/opensmalltalk-vm.git > >> CoInterpreter VMMaker.oscog-eem.2530 uuid: 4d90ede0-0700-4d15-8173-2aaf2360b7d1 Jun 13 2019 > >> StackToRegisterMappingCogit VMMaker.oscog-eem.2530 uuid: 4d90ede0-0700-4d15-8173-2aaf2360b7d1 Jun 13 2019 > >> > >> I would like to know if other Mac users can confirm these. If yes, does anyone know anything about these problems? > > > > The failure/errors have been there since I have looked at all that code. > > I have no idea why they do not work. They use the SecureSocket variant of SqueakSSL, the rest of the system uses the SecureSocketStream IIRC, and that works… > > > >> > >> I assume the tests on macOS are not run during the trunk build, right? > > > > I am not sure. It seems that these are not part of the failed tests on travis: > > > > https://travis-ci.org/squeak-smalltalk/squeak-app/jobs/548131159#L2326 > > > > But Really, I have no idea… > > > > Best regards > > -Tobias > > > > > >> > >> Cheers, > >> Bernhard > > > |
|
|
On Fri, 21 Jun 2019 at 4:20 pm, Bernhard Pieber <[hidden email]> wrote: Thanks for the clarification. I did not know there was a separate mechanism for ignoring tests in smalltalkCI beside #expectedFailures in the image. Do you happen to know what is the rationale for that? Especially when the build succeeds anyway. During the preparation of, I think, the Squeak 5.1 release, we wanted to fix all tests, so we set up the CI to give us a todo list. Eventually, I wanted to let trunk builds depend on the test results, so that we'd always get notified if test break again. Unfortunately, we couldn't fix all tests in time for the release and some still fail. We currently run all builds on macOS, so that we can sign the app bundles. We probably need something like this for Windows too at some point. But as far as I can remember, SqueakSSLTest have never passed on macOS.
I agree. You could programmatically download and parse the last X build results using Travis' REST API. That's why we haven't uploaded them anywhere. I'd rather spend some time fixing bugs that are failing in the latest trunk and then flip the switch, so that we get notified if someone breaks something. Some test are documenting feature requests or VM-level bugs that cannot be fixed in the image. So we probably still need to ignore some in the smalltalkCI config. Also, sometimes builds timeout due to fragile network handling or VM segfaults. Some hardening would also be necessary I guess. Fabio
|
Re: SqueakSSLTest failure/errors on macOS?
|
|
In reply to this post by bpi
Hi Bernhard,
> Thanks for the clarification. I did not know there was a separate mechanism for ignoring tests in smalltalkCI beside #expectedFailures in the image. Do you happen to know what is the rationale for that? Especially when the build succeeds anyway. The three-state Status Indicator ( (green | yellow | red) = (okay | warning | error) ) is one of the most widely-studied and important outputs of many applications. Unfortunately for SUnit, having tests that are known not to pass on certain platforms basically destroys that Status output. > It is a pity that the trunk is not green all the time. Until we fix our use of #expectedFailures, it'll never report green. The primary requirement of SUnit is supposed to report the status of the software, and relieve users from needing to dig into the details just to ascertain that. In fact, that digging is actually 95% of the work of running the tests oneself, the other 5% being the single click launch of the Suite. So, that external patch file specifying internal class names is a desperate hack trying to make up for #expectedFailures not doing what it's supposed to. We really should fix our SUnit. > I wonder how difficult it would be to include information on failing tests in the download packages say as a known_test_failures.txt alongside the image file. +1. But this doesn't fix the quick output Status, so we should still fix SUnit, too. > It might help to increase the urge to fix them. I think we will always have some #expectedFailures, though. It seems to be a useful and necessary feature (except for how it subverts the green Status, of course). > Besides it would make it easier to find out whether one has introduced a regression. Exactly. Although Kent Beck's original SUnit article [1] didn't anticipate the need for an #expectedFailures mechanism, he did, under "Philosophy", clearly define a "failure": "A failure is an anticipated problem. When you write tests, you check for expected results. If you get a different answer, that is a failure." Additionally, check out the original 2003 implementation of TestResult>>#passed still in trunk: passed ^ self expectedPasses, self expectedDefects There it is. "passed" includes the #expectedDefects, which are the 'errors' + 'failures' which are expected to fail, so this original part of the implementation, too, is consistent with Beck's definition, above. It also shows my own hack fix (SUnit-cmm.116.mcz in the Inbox) is not a complete and/or proper fix, but it has the correct intention. We definitely should find a consensus on a fix for this. Best, Chris [1] - http://swing.fit.cvut.cz/projects/stx/doc/online/english/tools/misc/testfram.htm |
Re: SqueakSSLTest failure/errors on macOS?
|
Hi all, I think that SUnit's traffic light would be more reliable if we (1) introduce #skippedTests (being as flexible as #expectedFailures), (2) migrate all expected failures to that new form of skipped tests, and (3) deprecate the idea of expected failures in SUnit. That is, #skippedTests *should not run* because on certain platforms, they don't make sense. In all other cases, tests are just tests and are expected to pass. Consequently, there would be no arguing about how to handle "unexpected passes" in tools anymore. The traffic light would be fine. Best, Marcel
|
Re: SqueakSSLTest failure/errors on macOS?
|
|
Hey Marcel, Consider future readers of the #test.. methods which are skipped in this way -- all the code there with no conditions on its execution. Then they run the Suite and it reports all green, woo hoo! Even if they knew about #skippedTests going in, it's easy to imagine them forgetting to cross-check their code reading against that list, so the #test.. methods they just read in the browser were implicitly conveyed to them by the system as "passing" when, in fact, they didn't even run. SUnit derives its value by efficient and correct conveyance of the status of the software. I'm sure we'll figure something out. Best, Chris On Mon, Jun 24, 2019 at 1:12 AM Marcel Taeumel <[hidden email]> wrote:
|
Re: SqueakSSLTest failure/errors on macOS?
|
|
Hi Chris, so are you looking for information more close to an individaul test method or test result? I see two complementary approaches here: a) re-design SUnit test representations (i.e., the GUI) and b) re-design SUnit test specifications (i.e., the code). Best, Marcel
|
«
Return to Squeak - Dev
|
1 view|%1 views
| Free forum by Nabble | Edit this page |