[CRON] Fixed: squeak-smalltalk/squeak-app#16
Locked
[CRON] Fixed: squeak-smalltalk/squeak-app#16
 
|
|
Fabio Niephaus
Re: [CRON] Fixed: squeak-smalltalk/squeak-app#16
|
|
Do we really want to get exit code 0 before every single test passes? Von: Squeak-dev <[hidden email]> im Auftrag von Travis CI <[hidden email]>
Gesendet: Freitag, 17. April 2020 15:41:44 An: [hidden email] Betreff: [squeak-dev] [CRON] Fixed: squeak-smalltalk/squeak-app#1648 (squeak-trunk - b063518)
Carpe Squeak!
|
Re: [CRON] Fixed: squeak-smalltalk/squeak-app#16
|
On Fri, Apr 17, 2020 at 4:10 PM Thiede, Christoph <[hidden email]> wrote:
Sorry, but I don't understand your question. Maybe because not all tests were passing [1]?
|
Re: [CRON] Fixed: squeak-smalltalk/squeak-app#16
|
|
Sorry for the imprecise question. What I find a bit confusing is the following: Not all tests are passing, but the Travis job still ends up with exit code 0 so we get the mail "build was fixed", and the badge of the repo becomes green. Wouldn't it be more intuitive if the mails sent to the list would keep reporting "still failing" until we have a green bar in the Trunk?
This is the relevant place in the code (in the function check_test_status): Couldn't we remove this edge case and run check_test_status *after* the deployment stuff in prepare.sh?
Best, Christoph
Von: Squeak-dev <[hidden email]> im Auftrag von Fabio Niephaus <[hidden email]>
Gesendet: Freitag, 17. April 2020 16:25:40 An: The general-purpose Squeak developers list Betreff: Re: [squeak-dev] [CRON] Fixed: squeak-smalltalk/squeak-app#1648 (squeak-trunk - b063518) On Fri, Apr 17, 2020 at 4:10 PM Thiede, Christoph <[hidden email]> wrote:
Sorry, but I don't understand your question. Maybe because not all tests were passing [1]?
Carpe Squeak!
|
Re: [CRON] Fixed: squeak-smalltalk/squeak-app#16
|
Hmmm... in any case, we want to new bundles to appear on files.squeak.org/trunk right? That's why it is called "ALPHA". :-) Best, Marcel
|
Re: [CRON] Fixed: squeak-smalltalk/squeak-app#16
|
|
> Hmmm... in any case, we want to new bundles to appear on files.squeak.org/trunk right?
Of course, but that does not mean that we have to handle the build as successful, does it? We can also upload the new bundles and then let the build fail.
Best, Christoph Von: Squeak-dev <[hidden email]> im Auftrag von Taeumel, Marcel
Gesendet: Freitag, 17. April 2020 17:27:29 An: gettimothy via Squeak-dev Betreff: Re: [squeak-dev] [CRON] Fixed: squeak-smalltalk/squeak-app#1648 (squeak-trunk - b063518)
Hmmm... in any case, we want to new bundles to appear on files.squeak.org/trunk right? That's why it is called "ALPHA". :-)
Best,
Marcel
Carpe Squeak!
|
Re: [CRON] Fixed: squeak-smalltalk/squeak-app#16
|
|
In reply to this post by Christoph Thiede
On Fri, Apr 17, 2020 at 5:22 PM Thiede, Christoph <[hidden email]> wrote:
We've discussed this a few times and decided that we don't want a daily reminder that things are broken. In an ideal world, we'd fix all tests and remove the hack you mentioned. We'd then get an error notification if someone broke something, hoping that person will fix it. In the past, however, people have broken trunk builds with this hack in place in many different other ways (e.g. documenting VM crashes with tests ;), not accepting Apple developer agreements, ...), so ignoring the outcome was "good enough". Fabio
|
Re: [CRON] Fixed: squeak-smalltalk/squeak-app#16
|
|
In reply to this post by Christoph Thiede
On Fri, Apr 17, 2020 at 5:53 PM Thiede, Christoph <[hidden email]> wrote:
Or we could fix (or ignore) the four failing tests and move on:  Fabio
|
Re: [CRON] Fixed: squeak-smalltalk/squeak-app#16
|
|
Note that we're already excluding some test cases: On Fri, Apr 17, 2020 at 5:57 PM Fabio Niephaus <[hidden email]> wrote:
|
Re: [CRON] Fixed: squeak-smalltalk/squeak-app#16
|
|
Thanks for your arguments! Yeah, we should really take CI more seriously, a red bar shouldn't be something you see every day. > Note that we're already excluding some test cases:
> https://github.com/squeak-smalltalk/squeak-app/blob/b063518c61711ab7a0ca89ffa3d2494925fac823/smalltalk-ci/Squeak64-trunk.ston#L8-L12
Hm, for this purpose of "expected failures, don't run them at all" I think it could be helpful if smalltalkCI printed these tests into the output log.
Example:
What do you think? :-)
Best,
Christoph
Von: Squeak-dev <[hidden email]> im Auftrag von Fabio Niephaus <[hidden email]>
Gesendet: Freitag, 17. April 2020 17:58 Uhr An: The general-purpose Squeak developers list Betreff: Re: [squeak-dev] [CRON] Fixed: squeak-smalltalk/squeak-app#1648 (squeak-trunk - b063518) Note that we're already excluding some test cases:
On Fri, Apr 17, 2020 at 5:57 PM Fabio Niephaus <[hidden email]> wrote:
Carpe Squeak!
|
Skipping tests (was: [CRON] Fixed: squeak-smalltalk/squeak-app#16
|
|
By the way, this also reminds me of some previous ideas about skipping tests in SUnit.
SqueakSheet contains an implementation of TestInconclusive and a <skip> pragma. Leon chose a similar approach in the PragmaTestCase package.
Best,
Christoph
Von: Squeak-dev <[hidden email]> im Auftrag von Thiede, Christoph
Gesendet: Samstag, 18. April 2020 15:22 Uhr An: The general-purpose Squeak developers list Betreff: Re: [squeak-dev] [CRON] Fixed: squeak-smalltalk/squeak-app#1648 (squeak-trunk - b063518) Thanks for your arguments! Yeah, we should really take CI more seriously, a red bar shouldn't be something you see every day. > Note that we're already excluding some test cases:
> https://github.com/squeak-smalltalk/squeak-app/blob/b063518c61711ab7a0ca89ffa3d2494925fac823/smalltalk-ci/Squeak64-trunk.ston#L8-L12
Hm, for this purpose of "expected failures, don't run them at all" I think it could be helpful if smalltalkCI printed these tests into the output log.
Example:
What do you think? :-)
Best,
Christoph
Von: Squeak-dev <[hidden email]> im Auftrag von Fabio Niephaus <[hidden email]>
Gesendet: Freitag, 17. April 2020 17:58 Uhr An: The general-purpose Squeak developers list Betreff: Re: [squeak-dev] [CRON] Fixed: squeak-smalltalk/squeak-app#1648 (squeak-trunk - b063518) Note that we're already excluding some test cases:
On Fri, Apr 17, 2020 at 5:57 PM Fabio Niephaus <[hidden email]> wrote:
Carpe Squeak!
|
Re: [CRON] Fixed: squeak-smalltalk/squeak-app#16
|
|
In reply to this post by Christoph Thiede
+1 If you try to look up since when something fails you should not be fooled by skipped tests. Thiede, Christoph <[hidden email]> schrieb am Sa., 18. Apr. 2020, 15:22:
|
image.png (25K)
Re: Skipping tests (was: [CRON] Fixed: squeak-smalltalk/squeak-app#16
|
|
In reply to this post by Christoph Thiede
Hmm... could be another can-o-worms. :-) Skipping is not the same as an expected failure. Could as well be a platform-specific circumstance that makes a test not suitable. Best, Marcel
|
Re: Skipping tests (was: [CRON] Fixed: squeak-smalltalk/squeak-app#16
|
|
JUnit has assumptions for the "not suitable" case:
https://junit.org/junit5/docs/5.0.0/api/org/junit/jupiter/api/Assumptions.html That would match "platform-specific circumstance" and TestInconclusive. What about the other suggestion to use pragmas? I'd like that (more than listing selectors in another method). But I think it is only practical for the "disable" case because otherwise you would have to put the conditions for "suitable" in an expression in the pragma and would have to evaluate that in the runner ... Am Mo., 20. Apr. 2020 um 17:09 Uhr schrieb Marcel Taeumel <[hidden email]>:
|
Re: Skipping tests (was: [CRON] Fixed: squeak-smalltalk/squeak-app#16
|
|
When I find myself swimming in "solutions", I like to take a step back and refocus on the problem for guidance. What am I _really_ wanting, and why? For me, it's about #1 -- a meaningful, overall status universalis (e.g., red, yellow, green) #2 -- ability to defer broken tests until later so their individual status won't affect the overall output of #1 (2b -- see / edit / run the list of deferred tests) #3 -- "notification" of when a deferred test is fixed, but still in the deferred list It really boils down to making the "overall status" useful by deciding which tests I want to decouple from it, combined with helping me keep that list maintained in the similar way as it informs me about broken tests -- by flagging those tests in some way ("unexpected success" or whatever -- I care a lot less about what words we use in the selectors as much as the ability to meet the above requirements). - Chris On Mon, Apr 20, 2020 at 1:22 PM Jakob Reschke <[hidden email]> wrote:
|
Re: Skipping tests (was: [CRON] Fixed: squeak-smalltalk/squeak-app#16
|
|
> overall status universalis (e.g., red, yellow, green) I think this list is not yet exhaustive, I would like to add a fourth status (grey) for single tests that are inconclusive.
Jakob, thanks for the inspirations! How about the following implementation of suitability in SUnit?
And maybe class-wise, too:
For other scenarios where suitability is not obvious, we could use an inconclusive message like:
Or maybe like this?
And
if we had the #suitable: pragma, we could also use this notation for skipping a test:
Christoph
Von: Squeak-dev <[hidden email]> im Auftrag von Chris Muller <[hidden email]>
Gesendet: Dienstag, 21. April 2020 00:53 Uhr An: The general-purpose Squeak developers list Betreff: Re: [squeak-dev] Skipping tests (was: [CRON] Fixed: squeak-smalltalk/squeak-app#1648 (squeak-trunk - b063518)) When I find myself swimming in "solutions", I like to take a step back and refocus on the problem for guidance. What am I _really_ wanting, and why? For me, it's about
#1 -- a meaningful, overall status universalis (e.g., red, yellow, green)
#2 -- ability to defer broken tests until later so their individual status won't affect the overall output of #1
(2b -- see / edit / run the list of deferred tests)
#3 -- "notification" of when a deferred test is fixed, but still in the deferred list
It really boils down to making the "overall status" useful by deciding which tests I want to decouple from it, combined with helping me keep that list maintained in the similar way as it informs me about broken tests -- by flagging those tests in some
way ("unexpected success" or whatever -- I care a lot less about what words we use in the selectors as much as the ability to meet the above requirements).
- Chris
On Mon, Apr 20, 2020 at 1:22 PM Jakob Reschke <[hidden email]> wrote:
Carpe Squeak!
|
Re: Skipping tests (was: [CRON] Fixed: squeak-smalltalk/squeak-app#16
|
|
On Sun, Apr 26, 2020 at 12:35 PM Thiede, Christoph <[hidden email]> wrote:
-1. Red, yellow, green is universally understood. The colors are supposed to denote the action required, nothing more. Green=none, yellow=investigate, red=action required. "Inconclusive" would be yellow. Errors and Failures should both be red, IMO. If there were a gray, it should mean nothing other than "Off" or "no status available". Maybe that's what you meant..? - Chris
|
Re: Skipping tests (was: [CRON] Fixed: squeak-smalltalk/squeak-app#16
|
|
I don't think we need an extra status; not even sure how to measure "inconclusive". The traffic light in Squeak's SUnit has already yellow as an extra color compared to related unit testing implementations. Well, I do like to be able to distinguish between assertion failures and other errors at a glance. However, red always feels like bugs that are easier to fix. That's paradoxical, isn't it? Best, Marcel
|
| Free forum by Nabble | Edit this page |
squeak-app/prepare_image.sh
Lines 90 to 94 in b063518