October 30, 2019 // By Bill Roske
Rube Goldberg was an American cartoonist and engineer (among other things). He is probably best known for his cartoons depicting extremely complex systems to perform a simple task. They even put one on a postage stamp!
There are TONS of videos on YouTube showing complex Rube Goldberg Machines. Having built one or two myself (dominoes, books, and balloons…oh, my!) I know that you really need to build in pieces, with very controlled hand-off between segments.
So, what does that have to do with End=to-End testing? EVERYTHING!
In today’s complex systems there are often multiple subsystems, developed (and tested) by different teams, that contribute to the completion of a single business processes. When we talk about End-to-End testing, we are tempted to line up ALL subsystems and the entire business process into a big, long scenario, and then coordinate the execution for one end to the other (hence the name). But those scenarios can be complex and time-consuming! And, just like Rube Goldberg machines, if something goes wrong, you may have to reset the whole thing and start over! So, how can we define, develop and execute End-to-End tests that encompass multiple systems and teams without creating a Rube Goldberg machine?
It’s a (relatively) simple process:
- Break the scenario into segments, whenever it crosses program boundaries or data is at rest
- Define contracts between the segments
- Build the capability to validate the data at the end of a segment and set data to the expected state at the beginning of a segment.
Now we have separate test segments that may be run in any order, as they do not depend on the previous segment to complete (and pass). The “tooling” work in step 3, if done in a single utility method or program, guarantees that contract changes will be honored by downstream segments. Interestingly enough, if scenarios are specified up-front then segment tests that “participate” in an end-to-end scenario, by definition, cover functional testing of the subsystem. This means that End-to-End validation is not a separate test process. It is, rather, simply an additional requirement on our function test development.
Let’s look at an example. Let’s say we have a veterinarian medical supply company and we need an end-to-end test for the process of adding a new veterinarian clinic to our system. The clinic’s information is entered into an enrollment system. A validation system verifies FDA identification number, business license and credit information provided before the clinic (and DVM) are accepted as a customer. Once that process is complete, the customer is assigned a customer ID and is able to order supplies for delivery to their clinic. Our End-to-End scenario seeks to validate the process from enrollment of a (valid) new clinic customer through payment of their first invoice. We choose this scenario because we want to know that this process still works before we release a new version to production.
Testing this process as a single End-to-End scenario would not only be time-consuming but an error late in the process could require the restart of the whole scenario, necessitating re-execution of steps by teams up-stream.
To make the process less “Goldberg-ian”, we identify 5 separate segments (step 1), based on the subsystems: Enrollment, Verifications, Orders, Billing, and Remittance. The sub-systems are developed/tested by 5 agile teams, but each hands information off to the next.
There must be a contract between these subsystems, allowing them to be developed independently (step 2). We can (and should) verify that the output at the end of a test is valid to the contract, whether it is stored in a static file or sent via an API call to the next system. Testers on each team are likely doing this already. However, with separate teams it is possible that the producing team has their own code to validate the output, and the consuming team has their own code to generate inputs.
The “magic” (if it is magic) is in using the same code to validate the output against the contract AND to generate data that conforms to it (step 3). This ensures downstream systems are using the same spec, and do not require the existence or proper functioning of upstream systems in order to validate that they can process valid inputs properly. This is a process advocated for microservice architectures for APIs. (see https://microservices.io/patterns/testing/service-integration-contract-test.html), but it need not be limited to microservices. The pattern can be applied to any API producer/consumer relationship.
But, what about that “Data at rest” thing in Step 1? Some of these systems may require multiple state changes before moving on to the next subsystem. Billing receives and processes order invoices from Order Processing throughout the month and then issues a single itemized bill at the end of the month. The Customer’s account status is updated in the database, and is “at rest” between orders. These are opportunities to separate these steps into separate tests. This requires we specify what an account is expected to look like as different order types are sent through (an internal contract between the functions of receiving order invoices and generating billing. If we could insert a customer account in a specific state, then we can test the outgoing bills even if our input processing code is broken. Here again, having code (whether external to the production code, or interfaces in the code implemented for test-ability) that can validate the customer records in billing after input AND produce a customer account with multiple complex orders, allows us to test separate processing steps in the subsystem without requiring all functionality to work.
We have identified 6 segments: 5 subsystems with one broken into 2 processes with data at rest between steps. We’ve identified the contracts between each segment. We can implement code to validate AND generate data to the contract. This allows each segment of the End-to-End process to be tested in isolation, without invalidating the integrity of the end-to-end scenario. In effect, there is something very satisfying in seeing a complex scenario work from end to end, but the cost to get there is high! I’ll leave that satisfaction to the whimsical videos of Rube Goldberg machines!