Running SUnit batch tools

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Running SUnit batch tools

Bob Nemec
What is a recommend way to run SUnit tests as part of an automated image build?

We build our VW development images daily from the latest published Store code. 
Once built, the image runs our full suite of SUnit tests, using code like this...

block := [:eachResult | "write results to a write stream" ].
[
TestCase when: Result send: #value: to: block.
self runTests.
]
ensure: [TestCase unsubscribe: block]

...where the write stream is to a log file.

This works most of the time, but sometimes I get failed results that I cannot recreate.
They all seem related to resources not being properly set up when running the tests.
All the tests run correctly when when triggered from the UI, and they run correctly individually.

I've searched the archive for batch, automated, scripted, etc. + 'SUnit', but no luck.

Any suggestions? ... hopefully it's not something well documented that I keep missing ;-)

Thanks,
Bob Nemec
HTS Engineering


_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Bob Nemec
Reply | Threaded
Open this post in threaded view
|

Re: Running SUnit batch tools

Samuel S. Shuster-2
Bob,

TestCase when: Result send: #value: to: block.

Ah, that is SUnitToo (not plain old SUnit). Travis Griggs wrote that, but isn't doing much Smalltalk anymore. Perhaps Niall who is the shepherd of "Plain Old" can help.

                                And So It Goes
                                     Sames
______________________________________________________________________

Samuel S. Shuster [|]





_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Reply | Threaded
Open this post in threaded view
|

Re: Running SUnit batch tools

Andres Valloud-6
In reply to this post by Bob Nemec
Look at bundle "Assessments" in the Public Store repository.  It's fully
scriptable, and the self-tests have examples of how to do that.

On 1/2/2013 8:11 AM, [hidden email] wrote:

> What is a recommend way to run SUnit tests as part of an automated image
> build?
>
> We build our VW development images daily from the latest published Store
> code.
> Once built, the image runs our full suite of SUnit tests, using code
> like this...
>
> block := [:eachResult | "write results to a write stream" ].
> [
> TestCase when: Result send: #value: to: block.
> self runTests.
> ]
> ensure: [TestCase unsubscribe: block]
>
> ...where the write stream is to a log file.
>
> This works most of the time, but sometimes I get failed results that I
> cannot recreate.
> They all seem related to resources not being properly set up when
> running the tests.
> All the tests run correctly when when triggered from the UI, and they
> run correctly individually.
>
> I've searched the archive for batch, automated, scripted, etc. +
> 'SUnit', but no luck.
>
> Any suggestions? ... hopefully it's not something well documented that I
> keep missing ;-)
>
> Thanks,
> Bob Nemec
> HTS Engineering
>
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Reply | Threaded
Open this post in threaded view
|

Re: Running SUnit batch tools

Randy Coulman
In reply to this post by Bob Nemec
For SUnitToo (which it looks like you're using), there are at least two packages in the public Store repository for this.

One is TestRunner, which we wrote and use all the time.  Our latest version isn't likely in public Store yet, but if you're interested and have trouble with what's there, let me know, and I'll make time to get that up-to-date.

The other is Holger's JenkinsUnitTester.  I've not used it, but I believe it does essentially the same job.

Both tools are capable of producing JUnit-compatible output for use in continuous integration tools like Jenkins and CruiseControl.

I know that TestRunner doesn't handle the unavailable test resource exception yet, but I will be adding that soon.  I'm not sure about JenkinsUnitTester.

Randy

On Wed, Jan 2, 2013 at 8:11 AM, <[hidden email]> wrote:
What is a recommend way to run SUnit tests as part of an automated image build?

We build our VW development images daily from the latest published Store code. 
Once built, the image runs our full suite of SUnit tests, using code like this...

block := [:eachResult | "write results to a write stream" ].
[
TestCase when: Result send: #value: to: block.
self runTests.
]
ensure: [TestCase unsubscribe: block]

...where the write stream is to a log file.

This works most of the time, but sometimes I get failed results that I cannot recreate.
They all seem related to resources not being properly set up when running the tests.
All the tests run correctly when when triggered from the UI, and they run correctly individually.

I've searched the archive for batch, automated, scripted, etc. + 'SUnit', but no luck.

Any suggestions? ... hopefully it's not something well documented that I keep missing ;-)

Thanks,
Bob Nemec
HTS Engineering


_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc




--
Randy Coulman
[hidden email]
Twitter: @randycoulman

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Reply | Threaded
Open this post in threaded view
|

Re: Running SUnit batch tools

Bob Nemec
In reply to this post by Andres Valloud-6
Thank you all... cool that there is more than one option (Assessments & TestRunner ... 'JenkinsUnitTester' is new to me). 

Is there a Jenkins integration option for Assessments?

I'll also have to see which one works for running our tests on GemStone. We do that now, answering a string that gets added to our overall test results.
Currently I can only run some framework tests, since our GS repositories have existing domain models. 
Our next step is to build test resources in an empty GS repository and run the tests with full GS transactions (including tests for change conflicts and performance).

I'm really close to starting with a new visual.im and extent0.dbf, loading everything from Store to VW, updating GS, and running both VW and GS tests, with eventual integration with Jenkins.


Now, if only I could get all my source code to be stored in Git... ;-) 
Bob


From: Andres Valloud <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Wednesday, January 2, 2013 4:17:27 PM
Subject: Re: [vwnc] Running SUnit batch tools

Look at bundle "Assessments" in the Public Store repository.  It's fully
scriptable, and the self-tests have examples of how to do that.

On 1/2/2013 8:11 AM, [hidden email] wrote:

> What is a recommend way to run SUnit tests as part of an automated image
> build?
>
> We build our VW development images daily from the latest published Store
> code.
> Once built, the image runs our full suite of SUnit tests, using code
> like this...
>
> block := [:eachResult | "write results to a write stream" ].
> [
> TestCase when: Result send: #value: to: block.
> self runTests.
> ]
> ensure: [TestCase unsubscribe: block]
>
> ...where the write stream is to a log file.
>
> This works most of the time, but sometimes I get failed results that I
> cannot recreate.
> They all seem related to resources not being properly set up when
> running the tests.
> All the tests run correctly when when triggered from the UI, and they
> run correctly individually.
>
> I've searched the archive for batch, automated, scripted, etc. +
> 'SUnit', but no luck.
>
> Any suggestions? ... hopefully it's not something well documented that I
> keep missing ;-)
>
> Thanks,
> Bob Nemec
> HTS Engineering
>
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc



_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Bob Nemec
Reply | Threaded
Open this post in threaded view
|

Re: Running SUnit batch tools

Andres Valloud-6
I'm not aware of anyone having integrated Assessments with Jenkins.  It
can't be that hard to do though.  I'm familiar with Assessments but not
with Jenkins.  What would be needed to integrate them?

On 1/2/2013 1:48 PM, [hidden email] wrote:

> Thank you all... cool that there is more than one option (Assessments
> & TestRunner ... 'JenkinsUnitTester' is new to me).
>
> Is there a Jenkins integration option for Assessments?
>
> I'll also have to see which one works for running our tests on GemStone.
> We do that now, answering a string that gets added to our overall test
> results.
> Currently I can only run some framework tests, since our GS repositories
> have existing domain models.
> Our next step is to build test resources in an empty GS repository and
> run the tests with full GS transactions (including tests for change
> conflicts and performance).
>
> I'm really close to starting with a new visual.im and extent0.dbf,
> loading everything from Store to VW, updating GS, and running both VW
> and GS tests, with eventual integration with Jenkins.
>
>
> Now, if only I could get all my source code to be stored in Git... ;-)
> Bob
>
> ------------------------------------------------------------------------
> *From:* Andres Valloud <[hidden email]>
> *To:* "[hidden email]" <[hidden email]>
> *Sent:* Wednesday, January 2, 2013 4:17:27 PM
> *Subject:* Re: [vwnc] Running SUnit batch tools
>
> Look at bundle "Assessments" in the Public Store repository.  It's fully
> scriptable, and the self-tests have examples of how to do that.
>
> On 1/2/2013 8:11 AM, [hidden email] <mailto:[hidden email]> wrote:
>  > What is a recommend way to run SUnit tests as part of an automated image
>  > build?
>  >
>  > We build our VW development images daily from the latest published Store
>  > code.
>  > Once built, the image runs our full suite of SUnit tests, using code
>  > like this...
>  >
>  > block := [:eachResult | "write results to a write stream" ].
>  > [
>  > TestCase when: Result send: #value: to: block.
>  > self runTests.
>  > ]
>  > ensure: [TestCase unsubscribe: block]
>  >
>  > ...where the write stream is to a log file.
>  >
>  > This works most of the time, but sometimes I get failed results that I
>  > cannot recreate.
>  > They all seem related to resources not being properly set up when
>  > running the tests.
>  > All the tests run correctly when when triggered from the UI, and they
>  > run correctly individually.
>  >
>  > I've searched the archive for batch, automated, scripted, etc. +
>  > 'SUnit', but no luck.
>  >
>  > Any suggestions? ... hopefully it's not something well documented that I
>  > keep missing ;-)
>  >
>  > Thanks,
>  > Bob Nemec
>  > HTS Engineering
>  >
> _______________________________________________
> vwnc mailing list
> [hidden email] <mailto:[hidden email]>
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>
>
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Reply | Threaded
Open this post in threaded view
|

Re: Running SUnit batch tools

Holger Kleinsorgen
In reply to this post by Bob Nemec
Bob Nemec wrote
What is a recommend way to run SUnit tests as part of an automated image build?

We build our VW development images daily from the latest published Store code. 
Once built, the image runs our full suite of SUnit tests, using code like this...

....

This works most of the time, but sometimes I get failed results that I cannot recreate.
They all seem related to resources not being properly set up when running the tests.
All the tests run correctly when when triggered from the UI, and they run correctly individually.
SUNitToo deliberately runs tests in random order, which might explain this behavior, if one test modifies the shared resources.
Other issues could be external resources and/or timing, e.g. timeouts when creating the test resources.