July 17, 2019 // By Josh Wencl
It’s Friday afternoon, I’m looking into a test that has been failing every other build. It’s been doing this for weeks, the developers have been calling it a “flaky test”. The webdriver keeps hitting its timeout while the page loads. I increased the timeout from 15 seconds to 25 seconds.
I think many people can commiserate with this, with flaky tests or fragile test code.
Its not a “flaky test” its a test that’s doing its job.
Depending on your application, a long load time is a death sentence, worse than typos or a bad user experience. According to Kissmetrics 40% of visitors to a website will leave if the load time is greater than 3 seconds.
“Fragile” tests do a good job of exposing defects and problems, but testers are often more interested in making their automation work end-to-end and fixing a failing test than they are in understanding why their test is failing in the first place.
Test failed, run it again.
AzureDevops even has a feature in their builds where you can “re-run” failed tests within a certain threshold.
What do you call a test that passes once out of every 3 test runs?
A failure.
As testers we have to get into the mindset that these kinds of errors are often indicative of a fragile system and not fragile automation.
So not only is writing too much defensive programming into your test code a waste of time, it’s actively hiding defects.
The only thing worse than a type 1 error (a false positive) is a type 2 error (a false negative).