July 30, 2020 // By Bill Roske
For those of you not familiar with the “pesticide effect”, here it goes: When you use the same pesticide over and over, you kill off all the bugs that are susceptible to that pesticide. This leaves less competition for the bugs that are NOT affected by that pesticide. The more you use it, the less effective it is at protecting your crop. You can see how this might be related to software testing, and regression tests in particular. In 1990, Boris Beizer referred to this as the “Pesticide Paradox” in his book “Software Testing Techniques” (1990. ISBN-13: 978-0442206727).
In the mid-90s, I was spearheading a quality improvement effort at my employer. My team and I packaged our tests so that developers could run our tests before they integrated changes they had made. When I asked other testing managers to do the same, one of them said “Absolutely NOT! If I give the developers my tests, they will change their code so that the tests pass. That will just accelerate the pesticide effect!” It was then that I realized, we had 2 different goals. His was to find bugs. His success was dependent on the failure of the developers to meet requirements. The developers’ success meant he failed. He viewed the pesticide effect as a bad thing. This is an adversarial relationship that (IMHO) is unhealthy. On the other hand, I viewed this “pesticide effect” as a good thing. My job wasn’t (and isn’t) just to find bugs, it was (and is) to help make the software better. The sooner the software behaves the way we expect it to, the better. Pesticide effect? BRING IT ON!.
So, this begs the question: In the “Modern age” of agile software development and testing, is that still a problem? I submit it is less of an issue these days. And, with all due respect to Mr. Beizer, if we as software quality engineers were doing our jobs right, it wasn’t that big an issue back in 1990! Let me explain.
The goal of testing:
Let’s look at the goal of the software quality engineer, and specifically regression testing. Is the goal to find bugs? Well, fundamentally yes, but NO! That is our goal in as much as a carpenter’s goal is to drive nails! As I mentioned in the story above, the goal of a software quality engineer is much more than that. Our tests need to validate that the software works as intended. If they are correctly written, and they pass… that’s a win. And my success is NOT at the expense of the developers. Now, if there are bugs, I want to find them. Success is not just “regression tests are green at all costs.” That is the subject of another blog post. But my goal IS to find bugs so that they can be fixed. To eliminate them, and keep them gone. That’s what the whole team wants.
Coverage:
If we do not adequately cover the expected behavior of the software then there will be bugs that our tests will not find. If those defects adversely affect the expected behavior of the app, then we are missing critical coverage. I’m not sure that qualifies as the pesticide effect. It’s a bit like spraying only half your field. What would you expect? So, we do need to make sure we cover the critical business value of the application. That is simply doing our jobs properly.
The value of passing tests:
This leads us to some big questions! Do tests that always pass have zero value? Do regression tests become useless? My answer is “of COURSE NOT!” The tests represent a validation of intended behavior. As new behavior is added, or underlying components (operating systems, libraries) are updated, we need to know that the delivered business value continues to work properly. Do we need to add tests for new functionality? Of course. But that does not deter from the “insurance value” of the existing tests.
The goal will always be to deliver software that works. That means finding AND FIXING bugs. If we are:
- updating tests as functionality changes
- adding tests as functionality is added
- running the tests anytime there is ANY change in the environment
and our tests aren’t finding new bugs, that’s a WIN.
Pesticide effect? BRING IT ON!