SUnit Time out
Locked
12
12
 
|
2010/6/4 Hernan Wilkinson <[hidden email]>:
> I dont think it is a necessary... I mean, if you are doing TDD, you write > the test and you run it inmediatly, if it takes too much time to run (that > is, more than 2 seconds :-)) then you press ctrl+ . and problem solved... Pretend you are Stephane or Andreas and trying to integrate a bunch of tests that you did not write. How do you know whether the test programmer cared about shortness ? How do you know whether the test is expected to be long or there is a regression somewhere causing an infinite loop ? From integrator point of view, best strategy is to reject any long test. If you are a producer and want to produce a long test, then you MUST declare it as such. Of course, that breaks a few legacy tests... > I'm saying this because the should:notTakeMoreThan: has some interesting > implementation details, like running the test in another process and > synchronizing them, etc... so running all the test would create a process > for each test, etc. tacking more time unnecessarily in most of the cases... > unless the implementation takes care of only doing this special "feature" > when needed (for example, when you have that pragma) but the, if you have to > write the pragma why not just send the message #should:notTakeMoreThan: and > problem solved? > Your problem, probably, Andreas and Steph problem, I don't think so. Implementation details should be discussed. also the threshold is arbitrary and hardware dependent. Maybe we could use the Dorado as unit of time ;) but IMO, this feature is worth. Nicolas > On Sat, May 29, 2010 at 3:42 PM, Stéphane Ducasse > <[hidden email]> wrote: >> >> For me the point is that I have the impression that this is an interesting >> feature to have >> when we have a test server. Like that you make sure that you do not have >> tests with infinite recursion >> now this is also true that on a test server you do not really care if your >> tests take 30s or 2 min. >> >> Stef >> >> > We already have #should:notTakeMoreThan: and friends in TestCase. The >> > complete TestCase can be protected by overriding #runCase, the >> > individual test by wrapping the code of the test method. >> > >> > Lukas >> > >> > On 29 May 2010 16:43, Schwab,Wilhelm K <[hidden email]> wrote: >> >> Stef, >> >> >> >> Time to expose some of my ignorance (don't worry, I have plenty more >> >> waiting where I found this): what is the tag concept? That sounds very >> >> Tweak-ish, and I am a real believer in doing things "with the language, not >> >> TO the language" whenever possible. That is not to say that frameworks are >> >> bad; in fact, it means that frameworks are good, language extensions are >> >> anywhere from suspect to evil. >> >> >> >> I have some code that I am still porting, but the basic idea is to be >> >> able to write >> >> >> >> [ >> >> "code that might not complete" >> >> >> >> ] tryForSeconds:10 onTimeOut:[ >> >> >> >> ]. >> >> >> >> With a robust capability to do such things, it is probably not >> >> necessary (or even appropriate) for TestCase to enforce timeouts. The >> >> timeout block can simply raise an exception or assert false, and there is no >> >> need to disable timeouts where they do not belong. >> >> >> >> Bill >> >> >> >> ________________________________________ >> >> From: [hidden email] >> >> [[hidden email]] On Behalf Of Michael Roberts >> >> [[hidden email]] >> >> Sent: Saturday, May 29, 2010 6:17 AM >> >> To: [hidden email] >> >> Subject: Re: [Pharo-project] SUnit Time out >> >> >> >> yes think it's a good idea. I'm not sure the granularity that's >> >> required though. >> >> >> >> mike >> >> >> >> On Saturday, May 29, 2010, stephane ducasse <[hidden email]> >> >> wrote: >> >>> Hi guys >> >>> >> >>> in Squeak andreas introduced the idea of test time out >> >>> Do you think that this is interesting? >> >>> >> >>> Stef >> >>> >> >>> SUnit >> >>> ----- >> >>> All test cases now have an associated timeout after which the test is >> >>> considered failed. The purpose of the timeout is to catch issues like >> >>> infinite loops, unexpected user input etc. in automated test environments. >> >>> Timeouts can be set on an individual test basis using the <timeout: seconds> >> >>> tag or for an entire test case by implementing the #defaultTimeout method. >> >>> _______________________________________________ >> >>> Pharo-project mailing list >> >>> [hidden email] >> >>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> >>> >> >> >> >> _______________________________________________ >> >> Pharo-project mailing list >> >> [hidden email] >> >> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> >> >> >> _______________________________________________ >> >> Pharo-project mailing list >> >> [hidden email] >> >> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> >> >> > >> > >> > >> > -- >> > Lukas Renggli >> > www.lukas-renggli.ch >> > >> > _______________________________________________ >> > Pharo-project mailing list >> > [hidden email] >> > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> >> >> _______________________________________________ >> Pharo-project mailing list >> [hidden email] >> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project > > > _______________________________________________ > Pharo-project mailing list > [hidden email] > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project > _______________________________________________ Pharo-project mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project |
|
|
On Fri, Jun 4, 2010 at 2:32 PM, Nicolas Cellier <[hidden email]> wrote: 2010/6/4 Hernan Wilkinson <[hidden email]>: So, the problem is not development, it is integration... so, it is not related to TDD but how to run tests when integrating In that case, if you are integrating the first thing you have to do is to verify that the change, package, etc you are about to integrate "runs", that is, the integrator should run all the tests before integrating if not after integrating when a test fails he will not know if that is due to an integration error or an error in the original change set. Therefore if you have to run the test in the change "original" environment you will know how long they will take after running them and also how long they should take after integrating them. Maybe changing the integration procedure avoids the need for this "feature", that could impact when developing because of a need when integrating... I worked in a system that had around 23.000 tests and when integrating never feel the necessity for something like this... but when we started to have so many tests running all of them before integration was taking more time than expected (even thoug the 23.000 tests run in about 10 minutes or so), so we decided to change our procedure in such a way that when a programmer versioned a change, with that change we added the test running information so the integrator did not need to run the test but only see how they ran before closing the version... the time to run the test could be added to that information and make the automatic running of test when integrating take those times into account. If you are a producer and want to produce a long test, then you MUST hmm well again, I thought we were talking about development, but I think that there are more developers that integrators and you spend more time developing than integrating so, I still think that the solution should not affect development but concentrate on the integration part of the development process Bye, Hernan.
_______________________________________________ Pharo-project mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project |
|
|
2010/6/5 Hernan Wilkinson <[hidden email]>:
> > > On Fri, Jun 4, 2010 at 2:32 PM, Nicolas Cellier > <[hidden email]> wrote: >> >> 2010/6/4 Hernan Wilkinson <[hidden email]>: >> > I dont think it is a necessary... I mean, if you are doing TDD, you >> > write >> > the test and you run it inmediatly, if it takes too much time to run >> > (that >> > is, more than 2 seconds :-)) then you press ctrl+ . and problem >> > solved... >> >> Pretend you are Stephane or Andreas and trying to integrate a bunch of >> tests that you did not write. >> How do you know whether the test programmer cared about shortness ? >> How do you know whether the test is expected to be long or there is a >> regression somewhere causing an infinite loop ? >> >> From integrator point of view, best strategy is to reject any long test. > > So, the problem is not development, it is integration... so, it is not > related to TDD but how to run tests when integrating > In that case, if you are integrating the first thing you have to do is to > verify that the change, package, etc you are about to integrate "runs", that > is, the integrator should run all the tests before integrating if not after > integrating when a test fails he will not know if that is due to an > integration error or an error in the original change set. Therefore if you > have to run the test in the change "original" environment you will know how > long they will take after running them and also how long they should take > after integrating them. Yes but I doubt Andreas and Steph would make that effort. They are not even aware of "original" environment. Either the test runs in a reasonable time slot, or has an explicit long timeout statement. > Maybe changing the integration procedure avoids the need for this "feature", > that could impact when developing because of a need when integrating... > I worked in a system that had around 23.000 tests and when integrating never > feel the necessity for something like this... but when we started to have so > many tests running all of them before integration was taking more time than > expected (even thoug the 23.000 tests run in about 10 minutes or so), so we > decided to change our procedure in such a way that when a programmer > versioned a change, with that change we added the test running information > so the integrator did not need to run the test but only see how they ran > before closing the version... the time to run the test could be added to > that information and make the automatic running of test when integrating > take those times into account. > You did not hit the problem probably because you have a well managed environment/application. Andreas added this feature on pragmatic basis after hitting the problem. Automatically generated meta information could be a solution, but far more complex than Andreas simple hack. > >> If you are a producer and want to produce a long test, then you MUST >> declare it as such. >> >> Of course, that breaks a few legacy tests... >> >> > I'm saying this because the should:notTakeMoreThan: has some interesting >> > implementation details, like running the test in another process and >> > synchronizing them, etc... so running all the test would create a >> > process >> > for each test, etc. tacking more time unnecessarily in most of the >> > cases... >> > unless the implementation takes care of only doing this special >> > "feature" >> > when needed (for example, when you have that pragma) but the, if you >> > have to >> > write the pragma why not just send the message #should:notTakeMoreThan: >> > and >> > problem solved? >> > >> >> Your problem, probably, Andreas and Steph problem, I don't think so. > > hmm well again, I thought we were talking about development, but I think > that there are more developers that integrators and you spend more time > developing than integrating so, I still think that the solution should not > affect development but concentrate on the integration part of the > development process > > Bye, > Hernan. Maybe... I have the feeling that if we go toward a modular image with kind of automated testing we will hit the integration problem more and more. Nicolas >> >> Implementation details should be discussed. >> also the threshold is arbitrary and hardware dependent. >> Maybe we could use the Dorado as unit of time ;) >> but IMO, this feature is worth. >> >> Nicolas >> >> > On Sat, May 29, 2010 at 3:42 PM, Stéphane Ducasse >> > <[hidden email]> wrote: >> >> >> >> For me the point is that I have the impression that this is an >> >> interesting >> >> feature to have >> >> when we have a test server. Like that you make sure that you do not >> >> have >> >> tests with infinite recursion >> >> now this is also true that on a test server you do not really care if >> >> your >> >> tests take 30s or 2 min. >> >> >> >> Stef >> >> >> >> > We already have #should:notTakeMoreThan: and friends in TestCase. The >> >> > complete TestCase can be protected by overriding #runCase, the >> >> > individual test by wrapping the code of the test method. >> >> > >> >> > Lukas >> >> > >> >> > On 29 May 2010 16:43, Schwab,Wilhelm K <[hidden email]> wrote: >> >> >> Stef, >> >> >> >> >> >> Time to expose some of my ignorance (don't worry, I have plenty more >> >> >> waiting where I found this): what is the tag concept? That sounds >> >> >> very >> >> >> Tweak-ish, and I am a real believer in doing things "with the >> >> >> language, not >> >> >> TO the language" whenever possible. That is not to say that >> >> >> frameworks are >> >> >> bad; in fact, it means that frameworks are good, language extensions >> >> >> are >> >> >> anywhere from suspect to evil. >> >> >> >> >> >> I have some code that I am still porting, but the basic idea is to >> >> >> be >> >> >> able to write >> >> >> >> >> >> [ >> >> >> "code that might not complete" >> >> >> >> >> >> ] tryForSeconds:10 onTimeOut:[ >> >> >> >> >> >> ]. >> >> >> >> >> >> With a robust capability to do such things, it is probably not >> >> >> necessary (or even appropriate) for TestCase to enforce timeouts. >> >> >> The >> >> >> timeout block can simply raise an exception or assert false, and >> >> >> there is no >> >> >> need to disable timeouts where they do not belong. >> >> >> >> >> >> Bill >> >> >> >> >> >> ________________________________________ >> >> >> From: [hidden email] >> >> >> [[hidden email]] On Behalf Of Michael >> >> >> Roberts >> >> >> [[hidden email]] >> >> >> Sent: Saturday, May 29, 2010 6:17 AM >> >> >> To: [hidden email] >> >> >> Subject: Re: [Pharo-project] SUnit Time out >> >> >> >> >> >> yes think it's a good idea. I'm not sure the granularity that's >> >> >> required though. >> >> >> >> >> >> mike >> >> >> >> >> >> On Saturday, May 29, 2010, stephane ducasse >> >> >> <[hidden email]> >> >> >> wrote: >> >> >>> Hi guys >> >> >>> >> >> >>> in Squeak andreas introduced the idea of test time out >> >> >>> Do you think that this is interesting? >> >> >>> >> >> >>> Stef >> >> >>> >> >> >>> SUnit >> >> >>> ----- >> >> >>> All test cases now have an associated timeout after which the test >> >> >>> is >> >> >>> considered failed. The purpose of the timeout is to catch issues >> >> >>> like >> >> >>> infinite loops, unexpected user input etc. in automated test >> >> >>> environments. >> >> >>> Timeouts can be set on an individual test basis using the <timeout: >> >> >>> seconds> >> >> >>> tag or for an entire test case by implementing the #defaultTimeout >> >> >>> method. >> >> >>> _______________________________________________ >> >> >>> Pharo-project mailing list >> >> >>> [hidden email] >> >> >>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> >> >>> >> >> >> >> >> >> _______________________________________________ >> >> >> Pharo-project mailing list >> >> >> [hidden email] >> >> >> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> >> >> >> >> >> _______________________________________________ >> >> >> Pharo-project mailing list >> >> >> [hidden email] >> >> >> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> >> >> >> >> > >> >> > >> >> > >> >> > -- >> >> > Lukas Renggli >> >> > www.lukas-renggli.ch >> >> > >> >> > _______________________________________________ >> >> > Pharo-project mailing list >> >> > [hidden email] >> >> > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> >> >> >> >> >> _______________________________________________ >> >> Pharo-project mailing list >> >> [hidden email] >> >> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> > >> > >> > _______________________________________________ >> > Pharo-project mailing list >> > [hidden email] >> > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> > >> >> _______________________________________________ >> Pharo-project mailing list >> [hidden email] >> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project > > > _______________________________________________ > Pharo-project mailing list > [hidden email] > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project > _______________________________________________ Pharo-project mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project |
Re: [squeak-dev] Re: SUnit Time out
|
|
In reply to this post by stephane ducasse
I meant to CC the Pharo list on this. The patch I mention is
http://source.squeak.org/inbox/SUnit-dtl.79.mcz. Dave ----- Forwarded message from "David T. Lewis" <[hidden email]> ----- Delivered-To: [hidden email] Date: Sun, 6 Jun 2010 16:19:54 -0400 From: "David T. Lewis" <[hidden email]> To: The general-purpose Squeak developers list <[hidden email]> Subject: Re: [squeak-dev] Re: [Pharo-project] SUnit Time out On Tue, Jun 01, 2010 at 09:36:48PM -0700, Andreas Raab wrote: > > These changes are largely intended for automated integration testing. I > am hoping to automate the tests for community supported packages to a > point where there will be no user in front of the system. I've run into one issue for externally supported packages that need to work on older images. The <timeout: 30> method annotation works very well, but is not supported on all images. I put SUnit-dtl.79 in the inbox as a possible solution. Dave ----- End forwarded message ----- _______________________________________________ Pharo-project mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project |
«
Return to Pharo Smalltalk Developers
|
1 view|%1 views
| Free forum by Nabble | Edit this page |