fuelized test failures

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

fuelized test failures

Camillo Bruni-3
From now on all the test-failures are serialized on the build server.

This allows us to quickly find the cause of a failing test:
- download the .fuel file for the failing test
- materialize the .fuel in the image and debug the code
        | aContext |
        aContext := FLMaterializer materializeFromFileNamed: 'PharoDebug.fuel'.
        Debugger openContext: aContext label:  'This is the new debugger!' contents: nil

The serialized stack trace is very limited (generally only 2 contexts) but they
are enough to get a decent view on what the variables are
Reply | Threaded
Open this post in threaded view
|

Re: fuelized test failures

Tudor Girba-2
Wow! This is truly science fiction.

Cheers,
Doru


On 20 Jul 2012, at 18:09, Camillo Bruni wrote:

> From now on all the test-failures are serialized on the build server.
>
> This allows us to quickly find the cause of a failing test:
> - download the .fuel file for the failing test
> - materialize the .fuel in the image and debug the code
> | aContext |
> aContext := FLMaterializer materializeFromFileNamed: 'PharoDebug.fuel'.
> Debugger openContext: aContext label:  'This is the new debugger!' contents: nil
>
> The serialized stack trace is very limited (generally only 2 contexts) but they
> are enough to get a decent view on what the variables are

--
www.tudorgirba.com

"In a world where everything is moving ever faster,
one might have better chances to win by moving slower."




Reply | Threaded
Open this post in threaded view
|

Re: fuelized test failures

Mariano Martinez Peck


On Sat, Jul 21, 2012 at 3:29 PM, Tudor Girba <[hidden email]> wrote:
Wow! This is truly science fiction.


hehehe yes, but there are still some problems when serializing contexs/debuggers. I would like to clarify the possible problems. 

When you serialize a context, we serialize all the context state together with its sender, and the sender has a sender, and a sender...so you end up serializing the whole list of contexts. That means that:

1) the object graph that you serialize, and hence, the stream size can be REALLY large depending on what the contexts have. Sometimes a context end up in the UI, so you end up serializing lots of morphs, colors, forms, etc. So..if everything is fine, it should be a couple hundred or thousands KB in the file. If it start to be in MB...then you may be serializing too much.  
2) the same problem of 1) is that not only the graph is too big, but also that you incorporate objects (mostly when these are objects from the UI) that CHANGE while being serialized. This will cause Fuel to throw an error saying the graph has changed during serialization. 
3)  if 2) happens, then depending where you trigger the fuel serialization, you may end up in a loop. For example, say you want to serialize each error with Fuel. So you change  SmalltalkImage>>logError:inContext:  to write the context with Fuel. Now, if 2) happens and Fuel throws an error, you will try to log that again, causing again the serialization.... infinitive loop. 

So...some workarounds you may try to do (still, not sure if will help in all cases):

1) Deep copy the context before serializing it. 
2) If you want to serialize particular contexts (for example, particular domain exceptions) then you may know WHERE to hook to make some instVars transient and therefore avoid serializing things you don't want and that may cause 2). 
3) Copy the context, and actually serialize a PART of it.

Cheers,

 
Cheers,
Doru


On 20 Jul 2012, at 18:09, Camillo Bruni wrote:

> From now on all the test-failures are serialized on the build server.
>
> This allows us to quickly find the cause of a failing test:
> - download the .fuel file for the failing test
> - materialize the .fuel in the image and debug the code
>       | aContext |
>       aContext := FLMaterializer materializeFromFileNamed: 'PharoDebug.fuel'.
>       Debugger openContext: aContext label:  'This is the new debugger!' contents: nil
>
> The serialized stack trace is very limited (generally only 2 contexts) but they
> are enough to get a decent view on what the variables are

--
www.tudorgirba.com

"In a world where everything is moving ever faster,
one might have better chances to win by moving slower."







--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: fuelized test failures

Stéphane Ducasse
In reply to this post by Camillo Bruni-3
I love that scenario :)
This is great that powerful tools let us imagine and build solutions that would not be possible
before. Power to imagination…
It fits perfectly with one of the motto of Pharo: Inventing the future.

On Jul 20, 2012, at 6:09 PM, Camillo Bruni wrote:

> From now on all the test-failures are serialized on the build server.
>
> This allows us to quickly find the cause of a failing test:
> - download the .fuel file for the failing test
> - materialize the .fuel in the image and debug the code
> | aContext |
> aContext := FLMaterializer materializeFromFileNamed: 'PharoDebug.fuel'.
> Debugger openContext: aContext label:  'This is the new debugger!' contents: nil
>
> The serialized stack trace is very limited (generally only 2 contexts) but they
> are enough to get a decent view on what the variables are


Reply | Threaded
Open this post in threaded view
|

Re: fuelized test failures

Marcus Denker-4

On Jul 22, 2012, at 10:00 AM, Stéphane Ducasse wrote:

> I love that scenario :)
> This is great that powerful tools let us imagine and build solutions that would not be possible
> before. Power to imagination…
>
And it is an example for a building block that helps to build things. Things that are *impossible*
without it.

Another way to view Fuel is that of a scientific experiment: We empirically study the existing
(ImageSegment). Without pre-concived notions to replace it! (We actually thought ImageSegments
would turn out to be the greatest thing ever and just needed some documentation/more understandable
implementation).
-> You study the existing
-> You claim that you can do better
-> You do better.

And then the next step is *extremeley* importnat:

-> You prove that it is better for real by *replacing* ALL the existing subsystems that do the same.

That last part is very essential: in the end, you often see that it is actually not that easy.

So *replacing* is important. The other thing important is to realize that it enables things that are impossible
without.

There are people that claim that instead of Pharo, what we should have done is to just do our research on
top of an unchanged Squeak. (While not even trying to improve Squeak itself).

But I really wonder... just project Pharo some more years in the Future. Will that what we will be able to do
even be *thinkable* in the context of an unchanged Squeak 3.8?

        Marcus

--
Marcus Denker -- http://marcusdenker.de


Reply | Threaded
Open this post in threaded view
|

Re: fuelized test failures

Stéphane Ducasse
nice explanation. Indeed I really liked IS but fuel is the way to go :)

On Jul 22, 2012, at 11:11 AM, Marcus Denker wrote:

>
> On Jul 22, 2012, at 10:00 AM, Stéphane Ducasse wrote:
>
>> I love that scenario :)
>> This is great that powerful tools let us imagine and build solutions that would not be possible
>> before. Power to imagination…
>>
> And it is an example for a building block that helps to build things. Things that are *impossible*
> without it.
>
> Another way to view Fuel is that of a scientific experiment: We empirically study the existing
> (ImageSegment). Without pre-concived notions to replace it! (We actually thought ImageSegments
> would turn out to be the greatest thing ever and just needed some documentation/more understandable
> implementation).
> -> You study the existing
> -> You claim that you can do better
> -> You do better.
>
> And then the next step is *extremeley* importnat:
>
> -> You prove that it is better for real by *replacing* ALL the existing subsystems that do the same.
>
> That last part is very essential: in the end, you often see that it is actually not that easy.
>
> So *replacing* is important. The other thing important is to realize that it enables things that are impossible
> without.
>
> There are people that claim that instead of Pharo, what we should have done is to just do our research on
> top of an unchanged Squeak. (While not even trying to improve Squeak itself).
>
> But I really wonder... just project Pharo some more years in the Future. Will that what we will be able to do
> even be *thinkable* in the context of an unchanged Squeak 3.8?
>
> Marcus
>
> --
> Marcus Denker -- http://marcusdenker.de
>
>


Reply | Threaded
Open this post in threaded view
|

Re: fuelized test failures

Mariano Martinez Peck
In reply to this post by Camillo Bruni-3


On Fri, Jul 20, 2012 at 6:09 PM, Camillo Bruni <[hidden email]> wrote:
From now on all the test-failures are serialized on the build server.

This allows us to quickly find the cause of a failing test:
- download the .fuel file for the failing test


Where can I find these .fuel of the tests? 

 
- materialize the .fuel in the image and debug the code
        | aContext |
        aContext := FLMaterializer materializeFromFileNamed: 'PharoDebug.fuel'.
        Debugger openContext: aContext label:  'This is the new debugger!' contents: nil

The serialized stack trace is very limited (generally only 2 contexts) but they
are enough to get a decent view on what the variables are



--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: fuelized test failures

Pavel Krivanek-3
in the Workspace of the job

-- Pavel

On Mon, Jul 23, 2012 at 10:22 AM, Mariano Martinez Peck
<[hidden email]> wrote:

>
>
> On Fri, Jul 20, 2012 at 6:09 PM, Camillo Bruni <[hidden email]>
> wrote:
>>
>> From now on all the test-failures are serialized on the build server.
>>
>> This allows us to quickly find the cause of a failing test:
>> - download the .fuel file for the failing test
>
>
>
> Where can I find these .fuel of the tests?
>
>
>>
>> - materialize the .fuel in the image and debug the code
>>         | aContext |
>>         aContext := FLMaterializer materializeFromFileNamed:
>> 'PharoDebug.fuel'.
>>         Debugger openContext: aContext label:  'This is the new debugger!'
>> contents: nil
>>
>> The serialized stack trace is very limited (generally only 2 contexts) but
>> they
>> are enough to get a decent view on what the variables are
>
>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com
>

Reply | Threaded
Open this post in threaded view
|

Re: fuelized test failures

Mariano Martinez Peck
In reply to this post by Marcus Denker-4


On Sun, Jul 22, 2012 at 11:11 AM, Marcus Denker <[hidden email]> wrote:

On Jul 22, 2012, at 10:00 AM, Stéphane Ducasse wrote:

> I love that scenario :)
> This is great that powerful tools let us imagine and build solutions that would not be possible
> before. Power to imagination…
>
And it is an example for a building block that helps to build things. Things that are *impossible*
without it.

Another way to view Fuel is that of a scientific experiment: We empirically study the existing
(ImageSegment). Without pre-concived notions to replace it! (We actually thought ImageSegments
would turn out to be the greatest thing ever and just needed some documentation/more understandable
implementation).

Indeed. I remember my first months in Douai for my PhD making drawings to understand the IS primitives :)
And then Martin arrived hahaha. 

 
-> You study the existing
-> You claim that you can do better
-> You do better.

And then the next step is *extremeley* importnat:

-> You prove that it is better for real by *replacing* ALL the existing subsystems that do the same.

That last part is very essential: in the end, you often see that it is actually not that easy.

That's a real good point. In Spanish I would say "Los pingos se ven en la cancha"
 

So *replacing* is important. The other thing important is to realize that it enables things that are impossible
without.


+1
Fuel is getting more and more users. Even in BioInformatics: 


Cheers,

--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: fuelized test failures

Mariano Martinez Peck
In reply to this post by Camillo Bruni-3


On Fri, Jul 20, 2012 at 6:09 PM, Camillo Bruni <[hidden email]> wrote:
From now on all the test-failures are serialized on the build server.

This allows us to quickly find the cause of a failing test:
- download the .fuel file for the failing test
- materialize the .fuel in the image and debug the code
        | aContext |
        aContext := FLMaterializer materializeFromFileNamed: 'PharoDebug.fuel'.
        Debugger openContext: aContext label:  'This is the new debugger!' contents: nil




and I get:

FLClassNotFound: Materialization error. Class named #HDTestReport not found

That happens because during serialization the graph somehow reached the class HDTestReport (probably a sender at some point in the stack). But now, when materializing, such class is not present. So...is there in the workspace an image with the jenkins stuff?
Anyway, the idea was to include this directly in Pharo, wasn't it?

Thanks

 
The serialized stack trace is very limited (generally only 2 contexts) but they
are enough to get a decent view on what the variables are



--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: fuelized test failures

Camillo Bruni-3

On 2012-07-24, at 16:29, Mariano Martinez Peck wrote:

> On Fri, Jul 20, 2012 at 6:09 PM, Camillo Bruni <[hidden email]>wrote:
>
>> From now on all the test-failures are serialized on the build server.
>>
>> This allows us to quickly find the cause of a failing test:
>> - download the .fuel file for the failing test
>> - materialize the .fuel in the image and debug the code
>>        | aContext |
>>        aContext := FLMaterializer materializeFromFileNamed:
>> 'PharoDebug.fuel'.
>>        Debugger openContext: aContext label:  'This is the new debugger!'
>> contents: nil
>>
>>
>
> does it really work for you?? I am trying to materialize, for example,
> https://ci.lille.inria.fr/pharo/view/Pharo%202.0/job/pharo-2.0-tests/ws/Architecture/32/OS/mac/Pharo-2.0-AfterRunningTests/ClassHierarchyTest-testMethodDuplication.fuel
>
> and I get:
>
> FLClassNotFound: Materialization error. Class named #HDTestReport not found
>
> That happens because during serialization the graph somehow reached the
> class HDTestReport (probably a sender at some point in the stack). But now,
> when materializing, such class is not present. So...is there in the
> workspace an image with the jenkins stuff?
> Anyway, the idea was to include this directly in Pharo, wasn't it?

I suggest you use the image in the same directory, cause we install
some Hudson tools (which IMO should be included by default). I don't
really like that we have to load stuff for running the test, meaning
that this might affect the outcome of the results...

So for now either take the following image:
        https://ci.lille.inria.fr/pharo/view/Pharo%202.0/job/pharo-2.0-tests/Architecture=32,OS=mac/lastSuccessfulBuild/artifact/Pharo-2.0-AfterRunningTests.zip

or manually install the hudson tools:
"Load Hudson Build Tools"
Gofer new
        url: 'http://ss3.gemstone.com/ss/CISupport';
        package: 'HudsonBuildTools20';
        load.

then everything should work I think.
best
cami
Reply | Threaded
Open this post in threaded view
|

Re: fuelized test failures

Mariano Martinez Peck


On Tue, Jul 24, 2012 at 4:35 PM, Camillo Bruni <[hidden email]> wrote:

On 2012-07-24, at 16:29, Mariano Martinez Peck wrote:

> On Fri, Jul 20, 2012 at 6:09 PM, Camillo Bruni <[hidden email]>wrote:
>
>> From now on all the test-failures are serialized on the build server.
>>
>> This allows us to quickly find the cause of a failing test:
>> - download the .fuel file for the failing test
>> - materialize the .fuel in the image and debug the code
>>        | aContext |
>>        aContext := FLMaterializer materializeFromFileNamed:
>> 'PharoDebug.fuel'.
>>        Debugger openContext: aContext label:  'This is the new debugger!'
>> contents: nil
>>
>>
>
> does it really work for you?? I am trying to materialize, for example,
> https://ci.lille.inria.fr/pharo/view/Pharo%202.0/job/pharo-2.0-tests/ws/Architecture/32/OS/mac/Pharo-2.0-AfterRunningTests/ClassHierarchyTest-testMethodDuplication.fuel
>
> and I get:
>
> FLClassNotFound: Materialization error. Class named #HDTestReport not found
>
> That happens because during serialization the graph somehow reached the
> class HDTestReport (probably a sender at some point in the stack). But now,
> when materializing, such class is not present. So...is there in the
> workspace an image with the jenkins stuff?
> Anyway, the idea was to include this directly in Pharo, wasn't it?

I suggest you use the image in the same directory, cause we install
some Hudson tools (which IMO should be included by default).

+1  I remember there was an issue for that...I tried to find it but I fail. 
Do you remember?
 
I don't
really like that we have to load stuff for running the test, meaning
that this might affect the outcome of the results...

So for now either take the following image:
        https://ci.lille.inria.fr/pharo/view/Pharo%202.0/job/pharo-2.0-tests/Architecture=32,OS=mac/lastSuccessfulBuild/artifact/Pharo-2.0-AfterRunningTests.zip


ok, thanks!
 
or manually install the hudson tools:
"Load Hudson Build Tools"
Gofer new
        url: 'http://ss3.gemstone.com/ss/CISupport';
        package: 'HudsonBuildTools20';
        load.

then everything should work I think.


yes, it works. And it is really cool :)

Thanks! 

--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: fuelized test failures

Mariano Martinez Peck
THere is something I don't understand. If I see: https://ci.lille.inria.fr/pharo/view/Pharo%202.0/job/pharo-2.0-tests/Architecture=32,OS=mac/ws/Pharo-2.0-AfterRunningTests/
it seems it is independent of a particular build. What does it mean? that we only have the .fuel of the last build? if this is the case, then it is not very helpful at all.

Thanks!

On Tue, Jul 24, 2012 at 7:57 PM, Mariano Martinez Peck <[hidden email]> wrote:


On Tue, Jul 24, 2012 at 4:35 PM, Camillo Bruni <[hidden email]> wrote:

On 2012-07-24, at 16:29, Mariano Martinez Peck wrote:

> On Fri, Jul 20, 2012 at 6:09 PM, Camillo Bruni <[hidden email]>wrote:
>
>> From now on all the test-failures are serialized on the build server.
>>
>> This allows us to quickly find the cause of a failing test:
>> - download the .fuel file for the failing test
>> - materialize the .fuel in the image and debug the code
>>        | aContext |
>>        aContext := FLMaterializer materializeFromFileNamed:
>> 'PharoDebug.fuel'.
>>        Debugger openContext: aContext label:  'This is the new debugger!'
>> contents: nil
>>
>>
>
> does it really work for you?? I am trying to materialize, for example,
> https://ci.lille.inria.fr/pharo/view/Pharo%202.0/job/pharo-2.0-tests/ws/Architecture/32/OS/mac/Pharo-2.0-AfterRunningTests/ClassHierarchyTest-testMethodDuplication.fuel
>
> and I get:
>
> FLClassNotFound: Materialization error. Class named #HDTestReport not found
>
> That happens because during serialization the graph somehow reached the
> class HDTestReport (probably a sender at some point in the stack). But now,
> when materializing, such class is not present. So...is there in the
> workspace an image with the jenkins stuff?
> Anyway, the idea was to include this directly in Pharo, wasn't it?

I suggest you use the image in the same directory, cause we install
some Hudson tools (which IMO should be included by default).

+1  I remember there was an issue for that...I tried to find it but I fail. 
Do you remember?
 
I don't
really like that we have to load stuff for running the test, meaning
that this might affect the outcome of the results...

So for now either take the following image:
        https://ci.lille.inria.fr/pharo/view/Pharo%202.0/job/pharo-2.0-tests/Architecture=32,OS=mac/lastSuccessfulBuild/artifact/Pharo-2.0-AfterRunningTests.zip


ok, thanks!
 
or manually install the hudson tools:
"Load Hudson Build Tools"
Gofer new
        url: 'http://ss3.gemstone.com/ss/CISupport';
        package: 'HudsonBuildTools20';
        load.

then everything should work I think.


yes, it works. And it is really cool :)

Thanks! 

--
Mariano
http://marianopeck.wordpress.com




--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: fuelized test failures

Sean P. DeNigris
Administrator
Mariano Martinez Peck wrote
we only have the .fuel of the last build?
Why don't we zip any fuel files for failing tests in the artifact. This way, anyone can download and reconstruct the system. Also, the workspace is only available when we are logged in.

Cheers,
Sean
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: fuelized test failures

Mariano Martinez Peck


On Tue, Jul 24, 2012 at 10:09 PM, Sean P. DeNigris <[hidden email]> wrote:

Mariano Martinez Peck wrote
>
> we only have the .fuel of the last build?

Why don't we zip any fuel files for failing tests in the artifact.

I don't know much about Jenkins, but if that allows me to have the .fuel files for each version, then please go ahead :)
Now...besides zipping all together, I would also like to have the individual .fuel files. 
CAmi/marcus anyone can take a look to this? 

This way,
anyone can download and reconstruct the system. Also, the workspace is only
available when we are logged in.


+1
 
Cheers,
Sean



--
View this message in context: http://forum.world.st/fuelized-test-failures-tp4640954p4641360.html
Sent from the Pharo Smalltalk mailing list archive at Nabble.com.




--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: fuelized test failures

Sean P. DeNigris
Administrator
Mariano Martinez Peck wrote
but if that allows me to have the .fuel
files for each version, then please go ahead :)
The zip artifact now includes all the fuel files. I didn't add the fuel files as separate artifacts because it seems there's no way to specify a target folder for artifacts without a plugin.
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: fuelized test failures

Camillo Bruni-3

On 2012-07-25, at 00:50, Sean P. DeNigris wrote:

>
> Mariano Martinez Peck wrote
>>
>> but if that allows me to have the .fuel
>> files for each version, then please go ahead :)
>>
>
> The zip artifact now includes all the fuel files. I didn't add the fuel
> files as separate artifacts because it seems there's no way to specify a
> target folder for artifacts without a plugin.

perfect guys!

Reply | Threaded
Open this post in threaded view
|

Re: fuelized test failures

Mariano Martinez Peck
And here I wrote what we are doing so that others know :)

Thanks Cami and Sean. You rock.

Cheers,

On Wed, Jul 25, 2012 at 6:45 AM, Camillo Bruni <[hidden email]> wrote:

On 2012-07-25, at 00:50, Sean P. DeNigris wrote:

>
> Mariano Martinez Peck wrote
>>
>> but if that allows me to have the .fuel
>> files for each version, then please go ahead :)
>>
>
> The zip artifact now includes all the fuel files. I didn't add the fuel
> files as separate artifacts because it seems there's no way to specify a
> target folder for artifacts without a plugin.

perfect guys!




--
Mariano
http://marianopeck.wordpress.com