Is my regression suite too big to run for each release?
Will reducing the regression cycles reduce the time to market?
Am I still seeing test escapes in spite of running complete regression cycle?
Is my regression suite complete?
Is it worth spending time in identifying the tests for regression suites?
If any of the above questions bother you, then the inputs given below might help you.
Organizations spend millions of dollars to regress their code before each release. Various processes and methods are adopted to come up with the regression suite. Predicting by various statistical methods might be a right approach to find the right set and area of regression but without using statistical methods if we want to come up with the optimal set of tests, what do we do?
Before we answer the question posed above, let’s see how are regression tests identified?
With the experience of working on test projects from different industries like Banking & Finance, High-tech, Retail, Energy, Telecommunication, Social Media and Healthcare, and also with the experience of working on projects of varied complexity, we found the following methods are used to find the regression suites.
- Important features and tests: All the important tests cases and features are added to the regression suite. There are a set of tests that are not added to the suite considering them to be unimportant. This approach could lead to a failure at the customer location. Time to market is not high with this approach.
- All functional tests: This is exhaustive testing but has some serious implications. As the product grows, the regression suite will grow too. Any failure in regression test will translate to increased cycles and slower time to market.
- Experience: The experienced and senior personnel in the team would decide what should go into the regression suite into each release. They usually look at the bug fixes and decide what tests should be executed per release. This method usually yields decent results but is still prone to test escapes.
All of the above stated approaches have their own drawbacks.
Going back to the original question that was asked in the beginning of this article,
Without using statistical methods if we want to come up with the optimal set of tests, what do we do?
How is an optimal set of regression suite identified that will have relatively low risk and have lower time to market?
To get the right set of regression tests into a suite, we need just not experience of the tester but some crucial information about the fixes in the release and guidance from developers.
Let’s take an example to understand this.
Assume there are n number of different modules that use a #define value of a variable called NUM_DIRECTORS. Say the value is 8. The value is used at multiple places and at one point the developer thought the value should be 6 instead of 8. He changes the value of the #define to 6. But there is another developer who hard codes the value to 8 in his code.
This creates a regression in the code.
The tester architect cannot find this kind of regressions, as he would not know where in the code this particular #define value is used. A wise developer would ask the tester to test all the areas that could have used the #define value directly or as a hard coded value.
Regression Test Identification Process
- Test architects with tremendous experience in the product cannot alone decide what tests are to be selected into regression suite. They requests the developers to give their insights into the areas to be tested.
- The developer who has fixed and checked in the code can give the area that needs to be regressed based on the complexity of fix, the confidence the developer has on the fix and the criticality of the code path.
- Based on the fix and the areas suggested by the developers, test architects can use their experience to find the correct set of tests.
Conclusion: It is suggested that the regression test suite identification process should be a collaborative effort between the developers and test engineering teams. The quality time spent in identifying the optimal set would help reduce the time to market and also improve the quality of the product by reducing the test escapes.
Regression Planner: Nexii Labs has developed a tool called Regression Planner that will help test architects to plan the regression test suite. This tool is used to collaboratively identify the tests by test and development teams, notify, review, communicate, assign and finally move them to Quality Center for execution.
Please send inquiries about the Regression planner to email@example.com.