Did we drop PNG support?

I hear what you're saying, Juan.

Counter arguments:

a) You're right, automated tests will not catch every possible bug. But they'll catch some, sometimes, and that's more information than no information.

b) I think your statement here misses the point. The point is that someone is *alerted* right away, generally by email, which is never a bad thing.

And I disagree with your statement that "people starts being less careful about quality, believing that the CI server can compensate for it." Obviously the CI server isn't going to fix bugs. IMHO, implementing CI is by definition being *more* careful about quality, not less.

Let me put it another way: people won't run the tests. Even if you pay them, they probably won't run the tests. Even if you convince them that they're probably going to hell if they don't run the tests, people still won't run the tests before they check in. I'm passionate about testing, but you know when the last time I ran the tests was? It was right before the last time we shipped a major version.

Running the tests as often as every commit is never a bad thing; having to do it manually though, is a pain in the ass.

To be clear: I know how much you care about quality. I know that you read every code submission very carefully, and you're keen to reject code that doesn't meet your standards for quality, which are very high. I've even see you rewrite bits of submissions to improve them before including them in the system. You're right about one thing, too, which is that no amount of automation can come close to the work of a dedicated, passionate, and able human being.

It's still worth doing, when the time is right. Eventually -- we should hope -- the number of submissions will become burdensome. Automated testing can reduce some of that burden, and let us know when something has become provably broken, even if us volunteers are too busy surviving to pay close attention that day.

Juan does have a point here. He's controlling checkins to the core system right now, and there aren't yet so many submissions that he has to deputize other people to be able to merge submissions. I think it will be when we reach the point where the number of submissions exceed Juan's available time that we will very seriously need to think about CI. I'm also glad to hear that he works in an image with the standard non-core packages loaded. That alone greatly assuages the particular concern I voiced.

One last advantage of CI: the CI system can build multiple image targets. This means it can cook us a core image and it can also cook us a standard image. The latter is advantageous because it's probably better that the easy-to-grab image we work in has "stuff we might break" loaded already; it's arguably burdensome to load all the external packages before beginning to work (I, for example, don't.)

Anyway, these are just my views, and they aren't carved on stone tablets.

Cheers,

Casey

On Fri, Dec 20, 2013 at 10:15 AM, Juan Vuletich <[hidden email]> wrote:

On 12/20/2013 1:16 PM, Frank Shearar wrote:

On every commit, CI runs your entire test suite. When things fail, you
immediately know which commit broke the test, which gives you much
less code (hopefully) to analyse, to find the breakage. CI is all
about closing the feedback loop.

Further, in GitHub's case, you can use Travis CI to handle the CI
part. Because the test suite runs against every commit, a pull request
will be validated (or not), and Travis CI can update the pull
request's commit status. This gives you, the reviewer of the pull
request, a green/red light, telling you that the PR doesn't break
anything.

(For the seriously pedantic, the thing you _really_ want is to merge
the PR into the master and run the test suite on that. _That_ green
light should inform the commit status of the PR. Alas, no one actually
does that.)

frank

But all this assumes that:
a) Tests will catch every possible bug
b) Immediately after a test starts failing, someone will review the code

And people starts being less careful about quality, believing that the CI server can compensate for it.

(Do I need to say names?)

Cheers,
Juan Vuletich

_______________________________________________
Cuis mailing list
[hidden email]
http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org

_______________________________________________
Cuis mailing list
[hidden email]
http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org

I hear what you're saying, Juan.

Counter arguments:

a) You're right, automated tests will not catch every possible bug. But they'll catch some, sometimes, and that's more information than no information.

b) I think your statement here misses the point. The point is that someone is *alerted* right away, generally by email, which is never a bad thing.

And I disagree with your statement that "people starts being less careful about quality, believing that the CI server can compensate for it." Obviously the CI server isn't going to fix bugs. IMHO, implementing CI is by definition being *more* careful about quality, not less.

Let me put it another way: people won't run the tests. Even if you pay them, they probably won't run the tests. Even if you convince them that they're probably going to hell if they don't run the tests, people still won't run the tests before they check in. I'm passionate about testing, but you know when the last time I ran the tests was? It was right before the last time we shipped a major version.

Running the tests as often as every commit is never a bad thing; having to do it manually though, is a pain in the ass.

To be clear: I know how much you care about quality. I know that you read every code submission very carefully, and you're keen to reject code that doesn't meet your standards for quality, which are very high. I've even see you rewrite bits of submissions to improve them before including them in the system. You're right about one thing, too, which is that no amount of automation can come close to the work of a dedicated, passionate, and able human being.

It's still worth doing, when the time is right. Eventually -- we should hope -- the number of submissions will become burdensome. Automated testing can reduce some of that burden, and let us know when something has become provably broken, even if us volunteers are too busy surviving to pay close attention that day.

Juan does have a point here. He's controlling checkins to the core system right now, and there aren't yet so many submissions that he has to deputize other people to be able to merge submissions. I think it will be when we reach the point where the number of submissions exceed Juan's available time that we will very seriously need to think about CI. I'm also glad to hear that he works in an image with the standard non-core packages loaded. That alone greatly assuages the particular concern I voiced.

One last advantage of CI: the CI system can build multiple image targets. This means it can cook us a core image and it can also cook us a standard image. The latter is advantageous because it's probably better that the easy-to-grab image we work in has "stuff we might break" loaded already; it's arguably burdensome to load all the external packages before beginning to work (I, for example, don't.)

Anyway, these are just my views, and they aren't carved on stone tablets.

Cheers,

Casey

On Fri, Dec 20, 2013 at 10:15 AM, Juan Vuletich <[hidden email]> wrote:

On 12/20/2013 1:16 PM, Frank Shearar wrote:

On every commit, CI runs your entire test suite. When things fail, you
immediately know which commit broke the test, which gives you much
less code (hopefully) to analyse, to find the breakage. CI is all
about closing the feedback loop.

Further, in GitHub's case, you can use Travis CI to handle the CI
part. Because the test suite runs against every commit, a pull request
will be validated (or not), and Travis CI can update the pull
request's commit status. This gives you, the reviewer of the pull
request, a green/red light, telling you that the PR doesn't break
anything.

(For the seriously pedantic, the thing you _really_ want is to merge
the PR into the master and run the test suite on that. _That_ green
light should inform the commit status of the PR. Alas, no one actually
does that.)

frank

But all this assumes that:
a) Tests will catch every possible bug
b) Immediately after a test starts failing, someone will review the code

And people starts being less careful about quality, believing that the CI server can compensate for it.

(Do I need to say names?)

Cheers,
Juan Vuletich

_______________________________________________
Cuis mailing list
[hidden email]
http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org

_______________________________________________
Cuis mailing list
[hidden email]
http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org