What is the craziest bug you ever face

Fixing a race condition in handling open sockets when forking an image. At first I had no clue where the problem could come from, then I spent a lot of time guessing at the conditions (of course, being a race condition there was no means to force the specific condition but I didn't know that yet).

Over all I spent about 6 weeks on this bug and finally fixed it by creating a new primitive to handle that specific case. I'm not sure what tools could have helped me as this was a rather specific problem (OSProcess). But the hardest problems in my experience are usually concurrency / asynchrony (e.g. race conditions) or bugs in libraries (where you always assume that you must have made a mistake, never the library).

Max

> On 9 Mar 2017, at 12:36, Stephane Ducasse <[hidden email]> wrote:
>
> Hi guys
>
> During the DSU workshop we were brainstorming about what are the most difficult bugs we faced and what are the conceptual tools that would have helped you.
>
> Stef

Hi guys

During the DSU workshop we were brainstorming about what are the most difficult bugs we faced and what are the conceptual tools that would have helped you.

Stef

> On 9 Mar 2017, at 12:36, Stephane Ducasse <[hidden email]> wrote:
>
> Hi guys
>
> During the DSU workshop we were brainstorming about what are the most difficult bugs we faced and what are the conceptual tools that would have helped you.

Tracking down a problem where a header file was like this

struct touch_screen_event {
#ifdef SOME_FLAG
        ... other fields
#endif
        int x;
        int y;
        int pressure;
};

The touchscreen library was compile with -DSOME_FLAG but the code using that library didn't have the flag set. This means the code using the touchscreen events read x/y from the wrong offset in memory. The example work of the touchscreen library worked while the real user didn't. It would have helped to embed struct sizes and offsets into the shared library to find differences at link time.

---

Keyboard handling in kdrive (an Xserver for embedded/mobile usage): After plugging/unplugging USB into the device, the keyboard started to generate wrong keycodes. In Linux (depending on your keyboard mode) every key event is represented as a byte(?). This worked for a long time but then keyboards started to have more keys. So a special key value is used to indicate that a multi byte sequence will follow. As it turns out plugging/unplugging generated a multi-byte keyboard event...

Not sure what would have helped? :)

Stef,

One of the things that I do all of the time when trying to debug particularly difficult problems is to keep a "bug notebook".  My "bug notebook" is a text file where I record information gleaned from an inspector intermixed with comments, code and observations...

I've attached an example from a couple of days ago (bug_notebook.txt).

The particular problem I am working on involves btree nodes being improperly updated during index object updates. Each btree node is basically a 2000+ element Array ... so there is a lot of data to deal with .

In the notebook, I extract the interesting entries from the btree node and record the original set of values and then record the updated set of values and in this particular case I was able to "see" that after the update I had 4 entries that were not properly deleted ...

In addition to recording the interesting chunks of large data structures, I often end up with chunks of stacks from the debugger, copies of a inspector panes in my "bug notebooks" and chunks of code ...

Over the last several weeks I have been tackling different sets of bugs in this same area and I have been keeping a time-stamped log of the work that I've done along the way (analysis_notebook.txt) ... in this notebook I've bee recording test results and branch/SHA information so that I can easily reference different versions of methods as I work through fixing bugs and managing regressions ... There's  a section in the notebook where I've record performance information, so that when I have finished work on this set of bugs, I can evaluate how I've impacted performance ... Finally at the very bottom of the notebook, I've recorded a simple todo-list ...

The  notebooks are on a shared disk so that I can access and update the "bug notebook" when I am working from home on a mac or at work from a linux box ... 

I rely on the fact that I am working with text and not images ... since I like to copy and paste as well as search for particular strings in the data from the inspector ... in the bug_notebook  I searched for the `victim` string in the text output  from an inspector ...

I have come to really depend upon bug notebooks when addressing difficult bugs and I think that support for creating and maintaining bug notebooks would be a valuable addition the the debugging arsenal...

Dale

Thanks you all.

The idea is that we want to see how we cn improve our debugging arsenal. 

So it is important that your scenario give use some hints. 

It is difficult to convey what we are really looking for :)

Hi guys

During the DSU workshop we were brainstorming about what are the most difficult bugs we faced and what are the conceptual tools that would have helped you.

Stef

--

Using Opera's mail client: http://www.opera.com/mail/

I actually have a whole class of bugs and elements of a very Pharo-y solution.

I do a lot of work in a logic programming framework. Being a logic programming framework, it does not execute the code you write in a start-to-finish manner, but executes a few lines here, then jumps around and executes a few lines there. As a result, if you are trying to trace the evolution of a particular branch in the search tree, you can only follow one or two operations at a time in the debugger, then that branch gets put in the queue and you may have to step through tens or hundreds of other searches before you get back to it (if you can even recognize which branch you were following when it comes around again). This makes debugging basically impossible in every other language this framework has been ported to, and the standard strategy is just to stare at the code. Pharo's debugger was already a step up, but it still wasn't nearly up to the task.

I had been trying to get a particularly complex program to work for several days with no success, at which point I decided to investigate the moldable debugger framework. It took me several more days of going through the debugger code in different versions and emailing with its developers to get something working, which just speaks to the state that framework and documentation happened to be in at the time, but when I did get it working I found the bug in a few minutes. I essentially set up a visual depiction of the search tree where I could click on a branch and have the framework run until that branch came up again to the front of the queue. This meant I could trace execution linearly by giving myself a way to get a handle on the control flow via a visual interface. I have since changed my framework and broken the debugger code, but I definitely intend to try to get that rigged up again when the logic system is in a more polished form.

Admittedly, many programs probably do not have the problem that their control flow is so complex that a normal linear debugger is too difficult to understand, but on some level the ability to specify a debugging control flow outside the execution of the program itself seems like a pretty powerful idea, and something Pharo should be very well suited for with its interactivity/reflectivity. Maybe there would be more complex, logic-based programs if they weren't so hard to write!

Evan

I should add that users of the logic framework from non-smalltalk languages were very impressed by the potential.

Evan

Stef,

One of the things that I do all of the time when trying to debug particularly difficult problems is to keep a "bug notebook".  My "bug notebook" is a text file where I record information gleaned from an inspector intermixed with comments, code and observations...

I've attached an example from a couple of days ago (bug_notebook.txt).

The particular problem I am working on involves btree nodes being improperly updated during index object updates. Each btree node is basically a 2000+ element Array ... so there is a lot of data to deal with .

In the notebook, I extract the interesting entries from the btree node and record the original set of values and then record the updated set of values and in this particular case I was able to "see" that after the update I had 4 entries that were not properly deleted ...

In addition to recording the interesting chunks of large data structures, I often end up with chunks of stacks from the debugger, copies of a inspector panes in my "bug notebooks" and chunks of code ...

Over the last several weeks I have been tackling different sets of bugs in this same area and I have been keeping a time-stamped log of the work that I've done along the way (analysis_notebook.txt) ... in this notebook I've bee recording test results and branch/SHA information so that I can easily reference different versions of methods as I work through fixing bugs and managing regressions ... There's  a section in the notebook where I've record performance information, so that when I have finished work on this set of bugs, I can evaluate how I've impacted performance ... Finally at the very bottom of the notebook, I've recorded a simple todo-list ...

The  notebooks are on a shared disk so that I can access and update the "bug notebook" when I am working from home on a mac or at work from a linux box ... 

I rely on the fact that I am working with text and not images ... since I like to copy and paste as well as search for particular strings in the data from the inspector ... in the bug_notebook  I searched for the `victim` string in the text output  from an inspector ...

I have come to really depend upon bug notebooks when addressing difficult bugs and I think that support for creating and maintaining bug notebooks would be a valuable addition the the debugging arsenal...

Dale

On 03/09/2017 10:50 AM, stepharong wrote:

Thanks you all.

The idea is that we want to see how we cn improve our debugging arsenal. 

So it is important that your scenario give use some hints. 

It is difficult to convey what we are really looking for :)

Hi guys

During the DSU workshop we were brainstorming about what are the most difficult bugs we faced and what are the conceptual tools that would have helped you.

Stef

--

Using Opera's mail client: http://www.opera.com/mail/

On Thu, 09 Mar 2017 20:53:48 +0100, Evan Donahue <[hidden email]>  
wrote:

did you think about adapting the debugger to your execution semantics?
... I read that you did it
Excellent!
and broken the debugger code, but I definitely intend to try to get that

--
Using Opera's mail client: http://www.opera.com/mail/

The attached screnshot  illustrates why I like having text instead of pngs ... being able to highlight the association value (-5.00->5.00) is very useful when identifying the entries in a btree associated with the association (the entries at 13, 16,19, and 22)...

Stef,

One of the things that I do all of the time when trying to debug particularly difficult problems is to keep a "bug notebook".  My "bug notebook" is a text file where I record information gleaned from an inspector intermixed with comments, code and observations...

I've attached an example from a couple of days ago (bug_notebook.txt).

The particular problem I am working on involves btree nodes being improperly updated during index object updates. Each btree node is basically a 2000+ element Array ... so there is a lot of data to deal with .

In the notebook, I extract the interesting entries from the btree node and record the original set of values and then record the updated set of values and in this particular case I was able to "see" that after the update I had 4 entries that were not properly deleted ...

In addition to recording the interesting chunks of large data structures, I often end up with chunks of stacks from the debugger, copies of a inspector panes in my "bug notebooks" and chunks of code ...

Over the last several weeks I have been tackling different sets of bugs in this same area and I have been keeping a time-stamped log of the work that I've done along the way (analysis_notebook.txt) ... in this notebook I've bee recording test results and branch/SHA information so that I can easily reference different versions of methods as I work through fixing bugs and managing regressions ... There's  a section in the notebook where I've record performance information, so that when I have finished work on this set of bugs, I can evaluate how I've impacted performance ... Finally at the very bottom of the notebook, I've recorded a simple todo-list ...

The  notebooks are on a shared disk so that I can access and update the "bug notebook" when I am working from home on a mac or at work from a linux box ... 

I rely on the fact that I am working with text and not images ... since I like to copy and paste as well as search for particular strings in the data from the inspector ... in the bug_notebook  I searched for the `victim` string in the text output  from an inspector ...

I have come to really depend upon bug notebooks when addressing difficult bugs and I think that support for creating and maintaining bug notebooks would be a valuable addition the the debugging arsenal...

Dale

On 03/09/2017 10:50 AM, stepharong wrote:

Thanks you all.

The idea is that we want to see how we cn improve our debugging arsenal. 

So it is important that your scenario give use some hints. 

It is difficult to convey what we are really looking for :)

Hi guys

During the DSU workshop we were brainstorming about what are the most difficult bugs we faced and what are the conceptual tools that would have helped you.

Stef

--

Using Opera's mail client: http://www.opera.com/mail/

The attached screnshot  illustrates why I like having text instead of pngs ... being able to highlight the association value (-5.00->5.00) is very useful when identifying the entries in a btree associated with the association (the entries at 13, 16,19, and 22)...

On 03/09/2017 11:34 AM, Dale Henrichs wrote:

Stef,

One of the things that I do all of the time when trying to debug particularly difficult problems is to keep a "bug notebook".  My "bug notebook" is a text file where I record information gleaned from an inspector intermixed with comments, code and observations...

I've attached an example from a couple of days ago (bug_notebook.txt).

The particular problem I am working on involves btree nodes being improperly updated during index object updates. Each btree node is basically a 2000+ element Array ... so there is a lot of data to deal with .

In the notebook, I extract the interesting entries from the btree node and record the original set of values and then record the updated set of values and in this particular case I was able to "see" that after the update I had 4 entries that were not properly deleted ...

In addition to recording the interesting chunks of large data structures, I often end up with chunks of stacks from the debugger, copies of a inspector panes in my "bug notebooks" and chunks of code ...

Over the last several weeks I have been tackling different sets of bugs in this same area and I have been keeping a time-stamped log of the work that I've done along the way (analysis_notebook.txt) ... in this notebook I've bee recording test results and branch/SHA information so that I can easily reference different versions of methods as I work through fixing bugs and managing regressions ... There's  a section in the notebook where I've record performance information, so that when I have finished work on this set of bugs, I can evaluate how I've impacted performance ... Finally at the very bottom of the notebook, I've recorded a simple todo-list ...

The  notebooks are on a shared disk so that I can access and update the "bug notebook" when I am working from home on a mac or at work from a linux box ... 

I rely on the fact that I am working with text and not images ... since I like to copy and paste as well as search for particular strings in the data from the inspector ... in the bug_notebook  I searched for the `victim` string in the text output  from an inspector ...

I have come to really depend upon bug notebooks when addressing difficult bugs and I think that support for creating and maintaining bug notebooks would be a valuable addition the the debugging arsenal...

Dale

On 03/09/2017 10:50 AM, stepharong wrote:

Thanks you all.

The idea is that we want to see how we cn improve our debugging arsenal. 

So it is important that your scenario give use some hints. 

It is difficult to convey what we are really looking for :)

Hi guys

During the DSU workshop we were brainstorming about what are the most difficult bugs we faced and what are the conceptual tools that would have helped you.

Stef

--

Using Opera's mail client: http://www.opera.com/mail/

Yeah, that sounds right. The debugger needed to be able to step through lines of code like normal (to debug the logic framework) as well as to step through search steps (to debug programs written in the framework) defined in terms of the system semantics (since bugs could be in either program or framework at this point). It also needed to recognize the semantics not only of the framework, but of the debugging interaction, because I needed it to step through the search space until it arrived at a particular leaf of the search frontier. The framework itself basically does a breadth-first-search, and it doesn't care when a particular frontier node will be visited again, but if I am tracing the search in one particular part of the space for the very specific purposes of debugging, only then do I need the debugger to recognize the semantics of when we are expanding a child of a given frontier node.

I also played with the idea of actually changing the search order while debugging, which would change the semantics of the actual running program to make it turn out a certain way so that I could observe rare behavior. The analogy would be manually setting a random number in the debugger, except that it might take more than a single instance variable assignment to change the structure of a search tree.

Evan

Hi,

On the issue of developers log/notebook, I also have mine about Grafoscopio and other ideas (which is, of course, an interactive notebook, made in Grafoscopio). That allows me to take notes and trying to reproduce stuff that is not yet into the system as part of a package or create a story and different contextual information about my dev process to communicate it to another people. Here is an screenshot of how it looks:

I have attached the notebook in ston format also :-).

Cheers,

Offray

Stef,

One of the things that I do all of the time when trying to debug particularly difficult problems is to keep a "bug notebook".  My "bug notebook" is a text file where I record information gleaned from an inspector intermixed with comments, code and observations...

I've attached an example from a couple of days ago (bug_notebook.txt).

The particular problem I am working on involves btree nodes being improperly updated during index object updates. Each btree node is basically a 2000+ element Array ... so there is a lot of data to deal with .

In the notebook, I extract the interesting entries from the btree node and record the original set of values and then record the updated set of values and in this particular case I was able to "see" that after the update I had 4 entries that were not properly deleted ...

In addition to recording the interesting chunks of large data structures, I often end up with chunks of stacks from the debugger, copies of a inspector panes in my "bug notebooks" and chunks of code ...

Over the last several weeks I have been tackling different sets of bugs in this same area and I have been keeping a time-stamped log of the work that I've done along the way (analysis_notebook.txt) ... in this notebook I've bee recording test results and branch/SHA information so that I can easily reference different versions of methods as I work through fixing bugs and managing regressions ... There's  a section in the notebook where I've record performance information, so that when I have finished work on this set of bugs, I can evaluate how I've impacted performance ... Finally at the very bottom of the notebook, I've recorded a simple todo-list ...

The  notebooks are on a shared disk so that I can access and update the "bug notebook" when I am working from home on a mac or at work from a linux box ... 

I rely on the fact that I am working with text and not images ... since I like to copy and paste as well as search for particular strings in the data from the inspector ... in the bug_notebook  I searched for the `victim` string in the text output  from an inspector ...

I have come to really depend upon bug notebooks when addressing difficult bugs and I think that support for creating and maintaining bug notebooks would be a valuable addition the the debugging arsenal...

Dale

On 03/09/2017 10:50 AM, stepharong wrote:

Thanks you all.

The idea is that we want to see how we cn improve our debugging arsenal. 

So it is important that your scenario give use some hints. 

It is difficult to convey what we are really looking for :)

Hi guys

During the DSU workshop we were brainstorming about what are the most difficult bugs we faced and what are the conceptual tools that would have helped you.

Stef

--

Using Opera's mail client: http://www.opera.com/mail/

Hi Stephane,
bugs i found difficult to debug are eg (1) red cross of death bugs in
morphs, (2) bugs that are not visible any more because they are caught
and dealt with in the wrong place and then produce wrong results instead
of errors, and (3) bugs in iterative algos that work eg with #while and
that pop up eg in the 1000th iteration, where i first have to add a
counter to know that it happens in the 1000th iteration and then have to
add a #halt if counter=999 to see what happens then in the debugger.
werner

On 03/09/2017 12:36 PM, Stephane Ducasse wrote:
> Hi guys
>
> During the DSU workshop we were brainstorming about what are the most
> difficult bugs we faced and what are the conceptual tools that would
> have helped you.
>
> Stef

and occasionally memory leaks
werner

On 03/10/2017 04:02 PM, werner kassens wrote: