Is the Pareto Principle good enough?
Especially for non-QA stakeholders, it might be tempting to simply apply the good old Pareto Principle. It would mean that software testing should be stopped as early as 20% into the process, as you must have found 80% of the issues by that point. That would sure save quite a lot of money and speed up releases, right?
Unfortunately, things are not as simple as that. Depending on the industry, there may be simply too many (potential) critical issues to cut QA effort by 80%. Reducing expenses and releasing features faster might be tempting, but banking software testing implies catching anything remotely unsecure. Then there is a separate matter of regression: you canāt just let devs do one round of fixes and trust it didnāt cause some new bugs. Alas, the Pareto Principle is too lean for software QA.
Even though the 80/20 rule canāt be applied to software QA, there is still a need to balance resources and outcome ā hopefully, a relatively issue-free software product. How do you do that in QA? The answer is adding exit criteria in testing plan.
Want to dive a bit deeper into this? Our latest video could be just the thing for you.
Defining text exit criteria
The purpose of exit criteria in a test plan is to guide you as a checklist would. Just like the lack of particularly vital and/or loved products makes you go to the grocery store, reaching all exit criteria is a stop time in testing of software. Letās look at some of them.
Time
There comes a time when itās time to stop testing. Modern Agile development is fueled by very short iterations, which means you have a fixed deadline that canāt be moved. Yes, a feature that is still too raw to go to production will be postponed. As far as sprint planning goes, you will still have to stop the quality assurance effort it for now.
Test Budget
This is the simplest exit criteria for system testing. If you donāt have the means to carry on, you will have to be content with what issues the QA team found so far.
Requirements coverage
Itās one thing to stop the QA work because you didnāt have the time to go deep enough through absolutely everything. We have discussed why it is unavoidable and makes sense to do. On the other hand, your effort should be wide enough to at least have an idea of how all key pieces of software are performing. Achieving full requirements coverage is a good reason to move on.
Test coverage
Unlike requirements coverage, thereās no need to push for the 100% mark. Still, the absolute majority of code that is planned to go into production should be covered with test cases, preferably wrapped into test scenarios for smoother quality assurance. Automated testing tools or solutions that help you manage them, such as aqua, are a great help as well.
Defect severity
Different companies use different scales at which they grade defects. There is also an entirely separate conversation about the colloquial vs factual difference between severity and priority. For the purposes of this article, letās go with the following classification of defect severity:
Defect Severity
1
Critical
2
Major
3
Minor
4
Low
So, when to stop testing? Simple: when you fixed all Critical and Major defects. There are both software development and client relation reasons not to make the new version of your product more unstable than the previous one. Resolving all defects of the two highest severity types gives you that.
If you are looking for more QA strategy input, look no further than template. We applied 20 years of our experience in QA to create a simple, easy-to-tailor template that breaks down defect handling and much more.Ā
Get a testing strategy template that enables us to release 2 times faster
Further metrics
Quality Assurance teams track a lot of metrics to analyse the state of the product, their progress on the upcoming release, and the general productivity and success of the team. These metrics can also be used to define testing exit criteria as well. Some of them include:
- Threshold of open defects (any severity)
- Defect rate percentage
- Test case pass/failure rate
Functional testing success
Although the test case pass rate is not required to be 100%, all functional tests should be green before a new version of the product goes live. It doesnāt matter whether some things are a bit wonky, since thatās what Minor and Low severity defects are for. All key features, however, still should work ā even if not user scenarios do.
A good example here would be QA testing in Insurance. One of the primary insurance models means automatic coverage at partner clinics with documentation handled by a health facility and no actual money exchanging hands. There are also plans that provide partial or full reimbursement at non-partner clinics when the client has to file an insurance claim.
Filing an insurance claim is a key function of insurance companies software, and you canāt release a new version if clients canāt apply for reimbursement. Your QA specialists, however, could learn that the app fails to fill out claim data based on the photo of a bill, but users can still enter everything manually and send the claim. As long as the key functionality ā actually sending the claim and getting reimbursed ā is still present, your team can release a new version of the software.
Go / No Go meeting
Last but not least, thereās a Go / No Go meeting where all the tech people decide whether youāre ready to release the new version. If the previous exit criteria indicate that things are ready on the QA side, thatās when you stop.
Test Exit Criteria Checklist
Here are sample exit criteria ā feel free to exclude some, add more, or change values:
Conclusion
Test exit criteria help you find the right balance between covering requirements and not overextending your effort. Use the ideas above, apply your productās specific needs, and consider regulatory requirements to define exit criteria that suit.
One way to not spend too much time on QA is artificial intelligence. aquaās AI copilot can make entire tests to quickly cover edge cases and/or push for near-100% test coverage. The test case management functionality in aqua is great for any degree of AI integration, too.
AI testing to go from feasible to maximum test coverage
