March 1, 2016 // By Bill Roske
I hear it all the time. “Test automation, especially at the User Interface(UI) level, is fragile. Test everything at a lower (business logic) level or wait until the rate of change slows down.” In this blog I want to take a closer look at these statements.
I HATE generalizations, and the idea that “[UI] test automation… is fragile” is a gross generalization. Why would UI automation have such a reputation? As I see it, there are (at least) 2 reasons: poor test framework design and lack of communication.
Poor Test Framework Design:
Test automation is a software development function. Automated tests need to be designed and developed as if they are a durable asset; just like the software they are testing. UI Automation can suffer from a couple of perceived instabilities: Timing, navigation, and cost of change. I say “perceived" because these can be all but eliminated, with the right design.
Timing issues come into play when a test cannot reliably access elements in the UI, because the test code is looking for elements before the application under test is ready for interaction. Care must be taken to wait for the right moment to move forward. Navigation issues come into play when the automated tests are not designed to handle unexpected redirects or pop-ups. The cost of updating tests can be high if a proper modular design is not employed. One where code specific to a particular function is not centralized and reused. All these issues can be avoided through the proper design of the tests and frameworks (and potentially input into the design and implementation of the software under test).
Lack of Communication:
The other thing I hear a great deal is: “The app is always changing and we cannot keep up.” Or, “We get surprised by UI changes that break our tests.” Whether you are a waterfall shop, or an Agile organization, there is no excuse… for this excuse. If you are reviewing requirement and/or design changes, you must be aware of the changes coming. In waterfall, you can see the changes coming a long way off and have the time to make sure you understand the UI (and therefore automation) implications. If you are an agile team, then your testers are part of the backlog grooming, iteration planning, and standups. This is where discussions around the extent of changes should happen. More to the point, developers should know the impact of UI changes on the automated tests and their teammates.
No Right Answer:
Don’t get me wrong. Many times, if we can, we want to perform a bulk of automated testing below the UI. Why? NOT because it is fragile, but because it is SLOW! Waiting for application pages to render and finding identifiers or XPaths in the application, or the DOM, can be time-consuming. Testing the business logic without the UI can greatly speed up test development and execution. We are also going to run the tests a LOT, especially in an agile SDLC, so every little savings helps.
Given a well-designed approach, test automation can be fast, reliable, and cost effective at many levels. There is no one right answer for what level to automate the testing of your application. However, using the “fragile” excuse to not test the UI, or to put off automation, is the WRONG answer!
If you’d like to contact us directly, email us or call us at 877-277-1044.