Smalltalk › Pharo › Pharo Smalltalk Developers

fuelized test failures

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

18 messages Options

Camillo Bruni-3

Jul 20, 2012; 4:09pm

fuelized test failures

1828 posts

From now on all the test-failures are serialized on the build server.

This allows us to quickly find the cause of a failing test:
- download the .fuel file for the failing test
- materialize the .fuel in the image and debug the code
| aContext |
aContext := FLMaterializer materializeFromFileNamed: 'PharoDebug.fuel'.
Debugger openContext: aContext label: 'This is the new debugger!' contents: nil

The serialized stack trace is very limited (generally only 2 contexts) but they
are enough to get a decent view on what the variables are

Tudor Girba-2

Jul 21, 2012; 1:29pm

Re: fuelized test failures

7411 posts

Wow! This is truly science fiction.

Cheers,
Doru

On 20 Jul 2012, at 18:09, Camillo Bruni wrote:

> From now on all the test-failures are serialized on the build server.
>
> This allows us to quickly find the cause of a failing test:
> - download the .fuel file for the failing test
> - materialize the .fuel in the image and debug the code
> | aContext |
> aContext := FLMaterializer materializeFromFileNamed: 'PharoDebug.fuel'.
> Debugger openContext: aContext label: 'This is the new debugger!' contents: nil
>
> The serialized stack trace is very limited (generally only 2 contexts) but they
> are enough to get a decent view on what the variables are

--
www.tudorgirba.com

"In a world where everything is moving ever faster,
one might have better chances to win by moving slower."

Mariano Martinez Peck

Jul 21, 2012; 5:29pm

Re: fuelized test failures

9001 posts

On Sat, Jul 21, 2012 at 3:29 PM, Tudor Girba <[hidden email]> wrote:

Wow! This is truly science fiction.

hehehe yes, but there are still some problems when serializing contexs/debuggers. I would like to clarify the possible problems.

When you serialize a context, we serialize all the context state together with its sender, and the sender has a sender, and a sender...so you end up serializing the whole list of contexts. That means that:

1) the object graph that you serialize, and hence, the stream size can be REALLY large depending on what the contexts have. Sometimes a context end up in the UI, so you end up serializing lots of morphs, colors, forms, etc. So..if everything is fine, it should be a couple hundred or thousands KB in the file. If it start to be in MB...then you may be serializing too much.

2) the same problem of 1) is that not only the graph is too big, but also that you incorporate objects (mostly when these are objects from the UI) that CHANGE while being serialized. This will cause Fuel to throw an error saying the graph has changed during serialization.

3) if 2) happens, then depending where you trigger the fuel serialization, you may end up in a loop. For example, say you want to serialize each error with Fuel. So you change SmalltalkImage>>logError:inContext: to write the context with Fuel. Now, if 2) happens and Fuel throws an error, you will try to log that again, causing again the serialization.... infinitive loop.

So...some workarounds you may try to do (still, not sure if will help in all cases):

1) Deep copy the context before serializing it.

2) If you want to serialize particular contexts (for example, particular domain exceptions) then you may know WHERE to hook to make some instVars transient and therefore avoid serializing things you don't want and that may cause 2).

3) Copy the context, and actually serialize a PART of it.

Cheers,

Cheers,
Doru

On 20 Jul 2012, at 18:09, Camillo Bruni wrote:

> From now on all the test-failures are serialized on the build server.
>
> This allows us to quickly find the cause of a failing test:
> - download the .fuel file for the failing test
> - materialize the .fuel in the image and debug the code
> | aContext |
> aContext := FLMaterializer materializeFromFileNamed: 'PharoDebug.fuel'.
> Debugger openContext: aContext label: 'This is the new debugger!' contents: nil
>
> The serialized stack trace is very limited (generally only 2 contexts) but they
> are enough to get a decent view on what the variables are

--
www.tudorgirba.com

"In a world where everything is moving ever faster,
one might have better chances to win by moving slower."

... [show rest of quote]

--
Mariano
http://marianopeck.wordpress.com

Stéphane Ducasse

Jul 22, 2012; 6:59am

Re: fuelized test failures

17193 posts

In reply to this post by Camillo Bruni-3

I love that scenario :)
This is great that powerful tools let us imagine and build solutions that would not be possible
before. Power to imagination…
It fits perfectly with one of the motto of Pharo: Inventing the future.

On Jul 20, 2012, at 6:09 PM, Camillo Bruni wrote:

Marcus Denker-4

Jul 22, 2012; 9:11am

Re: fuelized test failures

7582 posts

On Jul 22, 2012, at 10:00 AM, Stéphane Ducasse wrote:

> I love that scenario :)
> This is great that powerful tools let us imagine and build solutions that would not be possible
> before. Power to imagination…
>
And it is an example for a building block that helps to build things. Things that are *impossible*
without it.

Another way to view Fuel is that of a scientific experiment: We empirically study the existing
(ImageSegment). Without pre-concived notions to replace it! (We actually thought ImageSegments
would turn out to be the greatest thing ever and just needed some documentation/more understandable
implementation).
-> You study the existing
-> You claim that you can do better
-> You do better.

And then the next step is *extremeley* importnat:

-> You prove that it is better for real by *replacing* ALL the existing subsystems that do the same.

That last part is very essential: in the end, you often see that it is actually not that easy.

So *replacing* is important. The other thing important is to realize that it enables things that are impossible
without.

There are people that claim that instead of Pharo, what we should have done is to just do our research on
top of an unchanged Squeak. (While not even trying to improve Squeak itself).

But I really wonder... just project Pharo some more years in the Future. Will that what we will be able to do
even be *thinkable* in the context of an unchanged Squeak 3.8?

Marcus

--
Marcus Denker -- http://marcusdenker.de

Stéphane Ducasse

Jul 22, 2012; 11:00am

Re: fuelized test failures

17193 posts

nice explanation. Indeed I really liked IS but fuel is the way to go :)

On Jul 22, 2012, at 11:11 AM, Marcus Denker wrote:

>
> On Jul 22, 2012, at 10:00 AM, Stéphane Ducasse wrote:
>
>> I love that scenario :)
>> This is great that powerful tools let us imagine and build solutions that would not be possible
>> before. Power to imagination…
>>
> And it is an example for a building block that helps to build things. Things that are *impossible*
> without it.
>
> Another way to view Fuel is that of a scientific experiment: We empirically study the existing
> (ImageSegment). Without pre-concived notions to replace it! (We actually thought ImageSegments
> would turn out to be the greatest thing ever and just needed some documentation/more understandable
> implementation).
> -> You study the existing
> -> You claim that you can do better
> -> You do better.
>
> And then the next step is *extremeley* importnat:
>
> -> You prove that it is better for real by *replacing* ALL the existing subsystems that do the same.
>
> That last part is very essential: in the end, you often see that it is actually not that easy.
>
> So *replacing* is important. The other thing important is to realize that it enables things that are impossible
> without.
>
> There are people that claim that instead of Pharo, what we should have done is to just do our research on
> top of an unchanged Squeak. (While not even trying to improve Squeak itself).
>
> But I really wonder... just project Pharo some more years in the Future. Will that what we will be able to do
> even be *thinkable* in the context of an unchanged Squeak 3.8?
>
> Marcus
>
> --
> Marcus Denker -- http://marcusdenker.de
>
>

... [show rest of quote]

Mariano Martinez Peck

Jul 23, 2012; 8:22am

Re: fuelized test failures

9001 posts

In reply to this post by Camillo Bruni-3

On Fri, Jul 20, 2012 at 6:09 PM, Camillo Bruni <[hidden email]> wrote:

From now on all the test-failures are serialized on the build server.

This allows us to quickly find the cause of a failing test:
- download the .fuel file for the failing test

Where can I find these .fuel of the tests?

- materialize the .fuel in the image and debug the code
| aContext |
aContext := FLMaterializer materializeFromFileNamed: 'PharoDebug.fuel'.
Debugger openContext: aContext label: 'This is the new debugger!' contents: nil

The serialized stack trace is very limited (generally only 2 contexts) but they
are enough to get a decent view on what the variables are

--
Mariano
http://marianopeck.wordpress.com

Pavel Krivanek-3

Jul 23, 2012; 8:26am

Re: fuelized test failures

1314 posts

in the Workspace of the job

-- Pavel

On Mon, Jul 23, 2012 at 10:22 AM, Mariano Martinez Peck
<[hidden email]> wrote:

>
>
> On Fri, Jul 20, 2012 at 6:09 PM, Camillo Bruni <[hidden email]>
> wrote:
>>
>> From now on all the test-failures are serialized on the build server.
>>
>> This allows us to quickly find the cause of a failing test:
>> - download the .fuel file for the failing test
>
>
>
> Where can I find these .fuel of the tests?
>
>
>>
>> - materialize the .fuel in the image and debug the code
>> | aContext |
>> aContext := FLMaterializer materializeFromFileNamed:
>> 'PharoDebug.fuel'.
>> Debugger openContext: aContext label: 'This is the new debugger!'
>> contents: nil
>>
>> The serialized stack trace is very limited (generally only 2 contexts) but
>> they
>> are enough to get a decent view on what the variables are
>
>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com
>

... [show rest of quote]

Mariano Martinez Peck

Jul 23, 2012; 10:07am

Re: fuelized test failures

9001 posts

In reply to this post by Marcus Denker-4

On Sun, Jul 22, 2012 at 11:11 AM, Marcus Denker <[hidden email]> wrote:

On Jul 22, 2012, at 10:00 AM, Stéphane Ducasse wrote:

> I love that scenario :)
> This is great that powerful tools let us imagine and build solutions that would not be possible
> before. Power to imagination…
>

And it is an example for a building block that helps to build things. Things that are *impossible*
without it.

Another way to view Fuel is that of a scientific experiment: We empirically study the existing
(ImageSegment). Without pre-concived notions to replace it! (We actually thought ImageSegments
would turn out to be the greatest thing ever and just needed some documentation/more understandable
implementation).

Indeed. I remember my first months in Douai for my PhD making drawings to understand the IS primitives :)

And then Martin arrived hahaha.

-> You study the existing
-> You claim that you can do better
-> You do better.

And then the next step is *extremeley* importnat:

-> You prove that it is better for real by *replacing* ALL the existing subsystems that do the same.

That last part is very essential: in the end, you often see that it is actually not that easy.

That's a real good point. In Spanish I would say "Los pingos se ven en la cancha"

So *replacing* is important. The other thing important is to realize that it enables things that are impossible
without.

Fuel is getting more and more users. Even in BioInformatics:

http://biosmalltalk.blogspot.fr/2012/07/custom-serialization-of-big-dna-or.html

http://rmod.lille.inria.fr/web/pier/software/Fuel/Software-using-Fuel

Cheers,

--
Mariano
http://marianopeck.wordpress.com

Mariano Martinez Peck

Jul 24, 2012; 2:29pm

Re: fuelized test failures

9001 posts

In reply to this post by Camillo Bruni-3

On Fri, Jul 20, 2012 at 6:09 PM, Camillo Bruni <[hidden email]> wrote:

From now on all the test-failures are serialized on the build server.

This allows us to quickly find the cause of a failing test:
- download the .fuel file for the failing test
- materialize the .fuel in the image and debug the code
| aContext |
aContext := FLMaterializer materializeFromFileNamed: 'PharoDebug.fuel'.
Debugger openContext: aContext label: 'This is the new debugger!' contents: nil

does it really work for you?? I am trying to materialize, for example, https://ci.lille.inria.fr/pharo/view/Pharo%202.0/job/pharo-2.0-tests/ws/Architecture/32/OS/mac/Pharo-2.0-AfterRunningTests/ClassHierarchyTest-testMethodDuplication.fuel

and I get:

FLClassNotFound: Materialization error. Class named #HDTestReport not found

That happens because during serialization the graph somehow reached the class HDTestReport (probably a sender at some point in the stack). But now, when materializing, such class is not present. So...is there in the workspace an image with the jenkins stuff?

Anyway, the idea was to include this directly in Pharo, wasn't it?

Thanks

The serialized stack trace is very limited (generally only 2 contexts) but they
are enough to get a decent view on what the variables are

--
Mariano
http://marianopeck.wordpress.com

Camillo Bruni-3

Jul 24, 2012; 2:35pm

Re: fuelized test failures

1828 posts

On 2012-07-24, at 16:29, Mariano Martinez Peck wrote:

> On Fri, Jul 20, 2012 at 6:09 PM, Camillo Bruni <[hidden email]>wrote:
>
>> From now on all the test-failures are serialized on the build server.
>>
>> This allows us to quickly find the cause of a failing test:
>> - download the .fuel file for the failing test
>> - materialize the .fuel in the image and debug the code
>> | aContext |
>> aContext := FLMaterializer materializeFromFileNamed:
>> 'PharoDebug.fuel'.
>> Debugger openContext: aContext label: 'This is the new debugger!'
>> contents: nil
>>
>>
>
> does it really work for you?? I am trying to materialize, for example,
> https://ci.lille.inria.fr/pharo/view/Pharo%202.0/job/pharo-2.0-tests/ws/Architecture/32/OS/mac/Pharo-2.0-AfterRunningTests/ClassHierarchyTest-testMethodDuplication.fuel
>
> and I get:
>
> FLClassNotFound: Materialization error. Class named #HDTestReport not found
>
> That happens because during serialization the graph somehow reached the
> class HDTestReport (probably a sender at some point in the stack). But now,
> when materializing, such class is not present. So...is there in the
> workspace an image with the jenkins stuff?
> Anyway, the idea was to include this directly in Pharo, wasn't it?

... [show rest of quote]

I suggest you use the image in the same directory, cause we install
some Hudson tools (which IMO should be included by default). I don't
really like that we have to load stuff for running the test, meaning
that this might affect the outcome of the results...

So for now either take the following image:
https://ci.lille.inria.fr/pharo/view/Pharo%202.0/job/pharo-2.0-tests/Architecture=32,OS=mac/lastSuccessfulBuild/artifact/Pharo-2.0-AfterRunningTests.zip

or manually install the hudson tools:
"Load Hudson Build Tools"
Gofer new
url: 'http://ss3.gemstone.com/ss/CISupport';
package: 'HudsonBuildTools20';
load.

then everything should work I think.
best
cami

Mariano Martinez Peck

Jul 24, 2012; 5:57pm

Re: fuelized test failures

9001 posts

On Tue, Jul 24, 2012 at 4:35 PM, Camillo Bruni <[hidden email]> wrote:

On 2012-07-24, at 16:29, Mariano Martinez Peck wrote:

> On Fri, Jul 20, 2012 at 6:09 PM, Camillo Bruni <[hidden email]>wrote:
>
>> From now on all the test-failures are serialized on the build server.
>>
>> This allows us to quickly find the cause of a failing test:
>> - download the .fuel file for the failing test
>> - materialize the .fuel in the image and debug the code
>> | aContext |
>> aContext := FLMaterializer materializeFromFileNamed:
>> 'PharoDebug.fuel'.
>> Debugger openContext: aContext label: 'This is the new debugger!'
>> contents: nil
>>
>>
>
> does it really work for you?? I am trying to materialize, for example,
> https://ci.lille.inria.fr/pharo/view/Pharo%202.0/job/pharo-2.0-tests/ws/Architecture/32/OS/mac/Pharo-2.0-AfterRunningTests/ClassHierarchyTest-testMethodDuplication.fuel
>
> and I get:
>
> FLClassNotFound: Materialization error. Class named #HDTestReport not found
>
> That happens because during serialization the graph somehow reached the
> class HDTestReport (probably a sender at some point in the stack). But now,
> when materializing, such class is not present. So...is there in the
> workspace an image with the jenkins stuff?
> Anyway, the idea was to include this directly in Pharo, wasn't it?

I suggest you use the image in the same directory, cause we install
some Hudson tools (which IMO should be included by default).

... [show rest of quote]

+1 I remember there was an issue for that...I tried to find it but I fail.

Do you remember?

I don't
really like that we have to load stuff for running the test, meaning
that this might affect the outcome of the results...

So for now either take the following image:
https://ci.lille.inria.fr/pharo/view/Pharo%202.0/job/pharo-2.0-tests/Architecture=32,OS=mac/lastSuccessfulBuild/artifact/Pharo-2.0-AfterRunningTests.zip

ok, thanks!

or manually install the hudson tools:
"Load Hudson Build Tools"
Gofer new
url: 'http://ss3.gemstone.com/ss/CISupport';
package: 'HudsonBuildTools20';
load.

then everything should work I think.

yes, it works. And it is really cool :)

Thanks!

--
Mariano
http://marianopeck.wordpress.com

Mariano Martinez Peck

Jul 24, 2012; 6:54pm

Re: fuelized test failures

9001 posts

THere is something I don't understand. If I see: https://ci.lille.inria.fr/pharo/view/Pharo%202.0/job/pharo-2.0-tests/Architecture=32,OS=mac/ws/Pharo-2.0-AfterRunningTests/

it seems it is independent of a particular build. What does it mean? that we only have the .fuel of the last build? if this is the case, then it is not very helpful at all.

Thanks!

On Tue, Jul 24, 2012 at 7:57 PM, Mariano Martinez Peck <[hidden email]> wrote:

On Tue, Jul 24, 2012 at 4:35 PM, Camillo Bruni <[hidden email]> wrote:

On 2012-07-24, at 16:29, Mariano Martinez Peck wrote:

> On Fri, Jul 20, 2012 at 6:09 PM, Camillo Bruni <[hidden email]>wrote:
>
>> From now on all the test-failures are serialized on the build server.
>>
>> This allows us to quickly find the cause of a failing test:
>> - download the .fuel file for the failing test
>> - materialize the .fuel in the image and debug the code
>> | aContext |
>> aContext := FLMaterializer materializeFromFileNamed:
>> 'PharoDebug.fuel'.
>> Debugger openContext: aContext label: 'This is the new debugger!'
>> contents: nil
>>
>>
>
> does it really work for you?? I am trying to materialize, for example,
> https://ci.lille.inria.fr/pharo/view/Pharo%202.0/job/pharo-2.0-tests/ws/Architecture/32/OS/mac/Pharo-2.0-AfterRunningTests/ClassHierarchyTest-testMethodDuplication.fuel
>
> and I get:
>
> FLClassNotFound: Materialization error. Class named #HDTestReport not found
>
> That happens because during serialization the graph somehow reached the
> class HDTestReport (probably a sender at some point in the stack). But now,
> when materializing, such class is not present. So...is there in the
> workspace an image with the jenkins stuff?
> Anyway, the idea was to include this directly in Pharo, wasn't it?

I suggest you use the image in the same directory, cause we install
some Hudson tools (which IMO should be included by default).
... [show rest of quote]

+1 I remember there was an issue for that...I tried to find it but I fail.
Do you remember?

I don't
really like that we have to load stuff for running the test, meaning
that this might affect the outcome of the results...

So for now either take the following image:
https://ci.lille.inria.fr/pharo/view/Pharo%202.0/job/pharo-2.0-tests/Architecture=32,OS=mac/lastSuccessfulBuild/artifact/Pharo-2.0-AfterRunningTests.zip

ok, thanks!

or manually install the hudson tools:
"Load Hudson Build Tools"
Gofer new
url: 'http://ss3.gemstone.com/ss/CISupport';
package: 'HudsonBuildTools20';
load.

then everything should work I think.

yes, it works. And it is really cool :)

Thanks!

--
Mariano
http://marianopeck.wordpress.com

... [show rest of quote]

--
Mariano
http://marianopeck.wordpress.com

Sean P. DeNigris

Jul 24, 2012; 8:09pm

Re: fuelized test failures

Administrator

5777 posts

Mariano Martinez Peck wrote

we only have the .fuel of the last build?

Why don't we zip any fuel files for failing tests in the artifact. This way, anyone can download and reconstruct the system. Also, the workspace is only available when we are logged in.

Cheers,
Sean

Cheers,
Sean

Mariano Martinez Peck

Jul 24, 2012; 9:10pm

Re: fuelized test failures

9001 posts

On Tue, Jul 24, 2012 at 10:09 PM, Sean P. DeNigris <[hidden email]> wrote:

Mariano Martinez Peck wrote

>
> we only have the .fuel of the last build?

Why don't we zip any fuel files for failing tests in the artifact.

I don't know much about Jenkins, but if that allows me to have the .fuel files for each version, then please go ahead :)

Now...besides zipping all together, I would also like to have the individual .fuel files.

CAmi/marcus anyone can take a look to this?

This way,
anyone can download and reconstruct the system. Also, the workspace is only
available when we are logged in.

Cheers,
Sean

--
View this message in context: http://forum.world.st/fuelized-test-failures-tp4640954p4641360.html
Sent from the Pharo Smalltalk mailing list archive at Nabble.com.

--
Mariano
http://marianopeck.wordpress.com

Sean P. DeNigris

Jul 24, 2012; 10:49pm

Re: fuelized test failures

Administrator

5777 posts

Mariano Martinez Peck wrote

but if that allows me to have the .fuel
files for each version, then please go ahead :)

The zip artifact now includes all the fuel files. I didn't add the fuel files as separate artifacts because it seems there's no way to specify a target folder for artifacts without a plugin.

Cheers,
Sean

Camillo Bruni-3

Jul 25, 2012; 4:45am

Re: fuelized test failures

1828 posts

On 2012-07-25, at 00:50, Sean P. DeNigris wrote:

>
> Mariano Martinez Peck wrote
>>
>> but if that allows me to have the .fuel
>> files for each version, then please go ahead :)
>>
>
> The zip artifact now includes all the fuel files. I didn't add the fuel
> files as separate artifacts because it seems there's no way to specify a
> target folder for artifacts without a plugin.

perfect guys!

Mariano Martinez Peck

Jul 25, 2012; 6:59pm

Re: fuelized test failures

9001 posts

And here I wrote what we are doing so that others know :)

http://marianopeck.wordpress.com/2012/07/25/reviving-ci-test-failures-in-local-machine/

Thanks Cami and Sean. You rock.

Cheers,

On Wed, Jul 25, 2012 at 6:45 AM, Camillo Bruni <[hidden email]> wrote:

On 2012-07-25, at 00:50, Sean P. DeNigris wrote:

>
> Mariano Martinez Peck wrote
>>
>> but if that allows me to have the .fuel
>> files for each version, then please go ahead :)
>>
>
> The zip artifact now includes all the fuel files. I didn't add the fuel
> files as separate artifacts because it seems there's no way to specify a
> target folder for artifacts without a plugin.

perfect guys!

... [show rest of quote]

--
Mariano
http://marianopeck.wordpress.com