Working On Windmill in Launchpad This Week
This week I've started work on getting our Windmill tests re-enabled. William Grant had to disable them in our automatic runs, since they were failing spuriously too often. William pushed them off to a non-blocking Jenkins run, which is good for those of us curious about them, but as Björn noted way back, you have to block on test failures or tests will bit rot and be useless. So this week, I've set myself the task of getting these tests going again.
For those who don't know, Windmill is what we use on Launchpad for browser XHR integration testing and to run our suite of YUI tests. Windmill is like Selenium, for those who know that tool. I have a love/hate relationship with Windmill. I love what it promises to offer and hate it when it doesn't live up, which can be frequently.
I've spent a lot of time getting to know how Windmill works in an attempt to make it stable for our use. Failures in Launchpad usually come in two forms:
- Windmill not playing well within our Zope test runner
- People not knowing how to write good Windmill tests
We've made signicant progress on writing better tests, though some weak tests still exist and people make innocent mistakes. The main pain lately has been that Windmill is not playing well with our test runner -- i.e. hanging test runs or tests that fail for non-obvious reasons. My task this week is to sort out why this is happening so regularly now and craft some fixes. In order to share knowledge (and hopefully get someone on Launchpad working on Windmill along with me) I'll blog everyday this week about my progress.
Yesterday was not too fruitful. I spent most of the day working on the first of the tests that has rotted. I have to have a good test suite before I can suss out the other behavior. This test is failing due to string comparisons that now fail due to recent recipe page changes. I'll go into greater detail in a subsequent post about how to fix this test -- once I know! :-) -- but for now, I'm trying to work out if the test is broken and should be changed or if the behavior is broken and should be fixed.
My gut says it's the test that's wrong, but I should know for sure here shortly.
Until the next Windmill post then!
Posted by deryck on March 15, 2011
