Unstable test: Tests.Monticello.MCSnapshotBrowserTest.testClassSelected

> This guy, and associated/similar tests, has/have become unstable.
> Prior to build #339 they just worked. From #340 onwards they sometimes
> pass, sometimes fail.
>
> According to the manifests we publish as part of each build the change
> from 339 to 340 is just
>
> < Files (cmm.119)
> ---
>> Files (cmm.120)
>
> which seems completely unrelated!
>
> If it weren't for the fact that everything in the test happened in the
> context of the main World thread (whatever its proper name is), I'd
> think it was a race: the tests consist of
> * perform a UI action
> * check that the UI's showing what we expect - that there's a list
> morph somewhere in the UI whose contents match some list.
>
> I've tried debugging these, and they're annoyingly non-deterministic.
> I've often found that rerunning the test - run it til it fails, rerun
> the failing assertion - makes the test pass!
>
> Any thoughts?

> On Mon, 27 May 2013, Frank Shearar wrote:
>
>> This guy, and associated/similar tests, has/have become unstable.
>> Prior to build #339 they just worked. From #340 onwards they sometimes
>> pass, sometimes fail.
>>
>> According to the manifests we publish as part of each build the change
>> from 339 to 340 is just
>>
>> < Files (cmm.119)
>> ---
>>>
>>> Files (cmm.120)
>>
>>
>> which seems completely unrelated!
>>
>> If it weren't for the fact that everything in the test happened in the
>> context of the main World thread (whatever its proper name is), I'd
>> think it was a race: the tests consist of
>> * perform a UI action
>> * check that the UI's showing what we expect - that there's a list
>> morph somewhere in the UI whose contents match some list.
>>
>> I've tried debugging these, and they're annoyingly non-deterministic.
>> I've often found that rerunning the test - run it til it fails, rerun
>> the failing assertion - makes the test pass!
>>
>> Any thoughts?
>
>
> I think it's because the test highly depends on the fact that it's being run
> from Morphic and how some morphs (PluggableListMorph) works (or used to
> work).
> I think the randomness comes from the fact that PluggableListMorph's new
> preselection feature uses a future, which delays the actual selection.

>
>
>
> On Mon, May 27, 2013 at 11:44 AM, Frank Shearar <[hidden email]>
> wrote:
>
>>
>> I don't like the way the tests are written anyway though: I don't like
>> unit tests crawling through the UI. I'd rather see tests that checked
>> the instvars containing the class list etc.
>
>
> These days, the way to do it would be to use ToolBuilder-SUnit. It provides
> trivial widget implementations that don't do anything but expose their
> state. So you can make assertions about what the UI is displaying, and
> easily simulate clicks etc, without having to deal with actual morphs.