This article by one of our QAT practice’s principal consultants, Andy Tinkham was originally published on SoftwareTestPro.com. We’re republishing it here as it illustrates part of our
QAT practice’s philosophy. We’ll have more posts around our practice philosophy in the future, but for
now, we hope you enjoy this post that demonstrates one way in which our QAT practice members work
smarter, not harder, and how they prioritize some of their testing activities to balance deep examination
of a system with rapid information to give back to the rest of the development team.
Quick attacks are test techniques in which the tester doesn't have to spend a lot of time preparing
for, planning, or executing the test. Instead, the tester can sit down with an application (even
one they’ve never seen before!) and very quickly start finding important bugs. Many of these
tests draw on patterns of data input to the system like February 29th in leap years (or in non-
leap years to test for error handling) or special characters in strings, for example. Others will
be based on common patterns of failure such as a failure to account for special cases in input
verification routines. Several prominent testers have published lists of these attacks including
Elisabeth Hendrickson (with her Test Heuristics Cheat Sheet), Matt Heusser (several articles
and blog posts on the subject), and James Whittaker (along with Hugh Thompson and
Mike Andrews) (the books How to Break Software, How to Break Web Software, and How to
Break Software Security).
Quick attacks capture the imagination of many software testers - who among us wouldn't
love to have a broad arsenal of techniques that they can use to sit down at an application and
quickly start finding important bugs in that application? Even so, these attacks have caused
some controversy; people have argued that focusing on quick attacks trivializes the work of
smart, brain-engaged testing and reduces the need to spend time understanding the application
being tested. Certainly, these are important aspects of testing, and a testing approach focused
entirely around quick attacks would likely leave many gaps in the test coverage of an application.
However, it can be equally important for testers to quickly provide feedback on an application.
Quick attacks are one way that testers can achieve this goal.
Another area where quick attacks can excel is in breaking up longer, mentally-draining activities.
Our brains tend to be able to focus on one task for only a finite amount of time and many testing
tasks can take longer than this time period. When tasks take longer than our attention span, we
can become less focused on the task at hand as our levels of mental fatigue increase. Ultimately,
focus can be lost enough that we end up missing bugs we might otherwise have found. If a good
stopping point is present in the longer task, testers may find that stopping their main task and
executing a quick attack or two serves to clear away some of that fatigue. They can then return to
their main task with a rejuvenated ability to think, focus, and find bugs.
A third area where quick attacks shine is their ability to allow testers to build out new tests based
on their past experiences. Many quick attacks arise from places where testers have seen problems
occur in the past – a text field where the developers didn’t think to handle special characters (or
characters that aren’t necessarily special in normal usage but are in the context of the program
like single quotes being injected into SQL queries where they then terminate a string early).
Testers build up these experiences as they do their jobs, and sharing these attacks that have found
bugs in the past allows us to collectively take advantage of more experiences than each of us
personally has had.
However, if quick attacks have value but focusing on them exclusively leaves out critical areas
of testing, this leads to a question - what is the right balance for a test team between employing
quick attacks and longer, more planned out testing? The answer boils down to one of risk - for
every task we do as testers there are a number of other tasks that we could have done in the same
time and now may no longer have time to do. Each of these other tasks has some probability
of finding a bug, and each potential bug that could be found has aspects that make it more or
less serious and more or less likely to be fixed. For quick attacks, there is generally only a short
amount of time required to execute the test. Each test also has a risk that it is designed to find.
That risk can also be identified prior to the test being executed. For example, the application
might break when I give it a non-ASCII character, or perhaps I can cause an error condition that
doesn’t get handled gracefully by the app. The same is true for other testing tasks as well. If
these risks are explicitly identified they can then be compared, and the highest likelihood, highest
severity risks can be addressed first. After addressing those risks, additional considerations such
as remaining available chunks of time or areas of particular concern can also be addressed. Quick
attacks are useful to fill in around these tasks as they do not take a lot of time, allowing them to
be interspersed with other testing tasks without a large impact on the overall schedule.
Ultimately, quick attacks are a powerful tool in a tester’s toolbox. They are quick to perform and
don’t require extensive amounts of setup or preparation, but they can uncover important defects
and provide rapid feedback to the rest of the team. For more information on specific attacks,
check out the references below or (shameless plug) my upcoming hands-on track session at
STPCon Fall 2012 in Miami!