What is a recommend way to run SUnit tests as part of an automated image build? We build our VW development images daily from the latest published Store code. Once built, the image runs our full suite of SUnit tests, using code like this... block := [:eachResult | "write results to a write stream" ]. [ TestCase when: Result send: #value: to: block. self runTests. ] ensure: [TestCase unsubscribe: block] ...where the write stream is to a log file. This works most of the time, but sometimes I get failed results that I cannot recreate. They all seem related to resources not being properly set up when running the tests. All the tests run
correctly when when triggered from the UI, and they run correctly individually. I've searched the archive for batch, automated, scripted, etc. + 'SUnit', but no luck. Any
suggestions? ... hopefully it's not something well documented that I keep missing ;-) Thanks, Bob Nemec HTS Engineering _______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Bob Nemec
|
Bob,
TestCase when: Result send: #value: to: block. And So It Goes Sames ______________________________________________________________________ Samuel S. Shuster [|] _______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc |
In reply to this post by Bob Nemec
Look at bundle "Assessments" in the Public Store repository. It's fully
scriptable, and the self-tests have examples of how to do that. On 1/2/2013 8:11 AM, [hidden email] wrote: > What is a recommend way to run SUnit tests as part of an automated image > build? > > We build our VW development images daily from the latest published Store > code. > Once built, the image runs our full suite of SUnit tests, using code > like this... > > block := [:eachResult | "write results to a write stream" ]. > [ > TestCase when: Result send: #value: to: block. > self runTests. > ] > ensure: [TestCase unsubscribe: block] > > ...where the write stream is to a log file. > > This works most of the time, but sometimes I get failed results that I > cannot recreate. > They all seem related to resources not being properly set up when > running the tests. > All the tests run correctly when when triggered from the UI, and they > run correctly individually. > > I've searched the archive for batch, automated, scripted, etc. + > 'SUnit', but no luck. > > Any suggestions? ... hopefully it's not something well documented that I > keep missing ;-) > > Thanks, > Bob Nemec > HTS Engineering > vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc |
In reply to this post by Bob Nemec
For SUnitToo (which it looks like you're using), there are at least two packages in the public Store repository for this.
One is TestRunner, which we wrote and use all the time. Our latest version isn't likely in public Store yet, but if you're interested and have trouble with what's there, let me know, and I'll make time to get that up-to-date.
The other is Holger's JenkinsUnitTester. I've not used it, but I believe it does essentially the same job. Both tools are capable of producing JUnit-compatible output for use in continuous integration tools like Jenkins and CruiseControl.
I know that TestRunner doesn't handle the unavailable test resource exception yet, but I will be adding that soon. I'm not sure about JenkinsUnitTester. Randy
On Wed, Jan 2, 2013 at 8:11 AM, <[hidden email]> wrote:
Randy Coulman [hidden email] Twitter: @randycoulman
_______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc |
In reply to this post by Andres Valloud-6
Thank you all... cool that there is more than one option (Assessments & TestRunner ... 'JenkinsUnitTester' is new to me). Is there a Jenkins integration option for Assessments? I'll also have to see which one works for running our tests on GemStone. We do that now, answering a string that gets added to our overall test
results. Currently I can only run some framework tests, since our GS repositories have existing domain models. Our next step is to build test resources in an empty GS repository and run the tests with full GS transactions (including tests for change conflicts and performance). I'm really close to starting with a new visual.im and extent0.dbf, loading everything from Store to VW, updating GS, and running both VW and GS tests, with eventual integration with Jenkins. Now, if only I could get all my source code to be stored in Git... ;-) Bob From: Andres Valloud <[hidden email]> To: "[hidden email]" <[hidden email]> Sent: Wednesday, January 2, 2013 4:17:27 PM Subject: Re: [vwnc] Running SUnit batch tools Look at bundle "Assessments" in the Public Store repository. It's fully scriptable, and the self-tests have examples of how to do that. On 1/2/2013 8:11 AM, [hidden email] wrote: > What is a recommend way to run SUnit tests as part of an automated image > build? > > We build our VW development images daily from the latest published Store > code. > Once built, the image runs our full suite of SUnit tests, using code > like this... > > block := [:eachResult | "write results to a write stream" ]. > [ > TestCase when: Result send: #value: to: block. > self runTests. > ] > ensure: [TestCase unsubscribe: block] > > ...where the write stream is to a log file. > > This works most of the time, but sometimes I get failed results that I > cannot recreate. > They all seem related to resources not being properly set up when > running the tests. > All the tests run correctly when when triggered from the UI, and they > run correctly individually. > > I've searched the archive for batch, automated, scripted, etc. + > 'SUnit', but no luck. > > Any suggestions? ... hopefully it's not something well documented that I > keep missing ;-) > > Thanks, > Bob Nemec > HTS Engineering > vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc _______________________________________________ vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Bob Nemec
|
I'm not aware of anyone having integrated Assessments with Jenkins. It
can't be that hard to do though. I'm familiar with Assessments but not with Jenkins. What would be needed to integrate them? On 1/2/2013 1:48 PM, [hidden email] wrote: > Thank you all... cool that there is more than one option (Assessments > & TestRunner ... 'JenkinsUnitTester' is new to me). > > Is there a Jenkins integration option for Assessments? > > I'll also have to see which one works for running our tests on GemStone. > We do that now, answering a string that gets added to our overall test > results. > Currently I can only run some framework tests, since our GS repositories > have existing domain models. > Our next step is to build test resources in an empty GS repository and > run the tests with full GS transactions (including tests for change > conflicts and performance). > > I'm really close to starting with a new visual.im and extent0.dbf, > loading everything from Store to VW, updating GS, and running both VW > and GS tests, with eventual integration with Jenkins. > > > Now, if only I could get all my source code to be stored in Git... ;-) > Bob > > ------------------------------------------------------------------------ > *From:* Andres Valloud <[hidden email]> > *To:* "[hidden email]" <[hidden email]> > *Sent:* Wednesday, January 2, 2013 4:17:27 PM > *Subject:* Re: [vwnc] Running SUnit batch tools > > Look at bundle "Assessments" in the Public Store repository. It's fully > scriptable, and the self-tests have examples of how to do that. > > On 1/2/2013 8:11 AM, [hidden email] <mailto:[hidden email]> wrote: > > What is a recommend way to run SUnit tests as part of an automated image > > build? > > > > We build our VW development images daily from the latest published Store > > code. > > Once built, the image runs our full suite of SUnit tests, using code > > like this... > > > > block := [:eachResult | "write results to a write stream" ]. > > [ > > TestCase when: Result send: #value: to: block. > > self runTests. > > ] > > ensure: [TestCase unsubscribe: block] > > > > ...where the write stream is to a log file. > > > > This works most of the time, but sometimes I get failed results that I > > cannot recreate. > > They all seem related to resources not being properly set up when > > running the tests. > > All the tests run correctly when when triggered from the UI, and they > > run correctly individually. > > > > I've searched the archive for batch, automated, scripted, etc. + > > 'SUnit', but no luck. > > > > Any suggestions? ... hopefully it's not something well documented that I > > keep missing ;-) > > > > Thanks, > > Bob Nemec > > HTS Engineering > > > _______________________________________________ > vwnc mailing list > [hidden email] <mailto:[hidden email]> > http://lists.cs.uiuc.edu/mailman/listinfo/vwnc > > vwnc mailing list [hidden email] http://lists.cs.uiuc.edu/mailman/listinfo/vwnc |
In reply to this post by Bob Nemec
SUNitToo deliberately runs tests in random order, which might explain this behavior, if one test modifies the shared resources. Other issues could be external resources and/or timing, e.g. timeouts when creating the test resources. |
Free forum by Nabble | Edit this page |