There were 4 phases that describe the COBOL testing team’s journey which was the most complete.

  1. Automated testing after cycle 1 test cases are executed manually. This is where the automated tests were created and executed after the first cycle of manual testing of all test cases. The assertions were made manually by running the submission of claim requests and then manually verifying the response and then saving it for comparison with a comparison tool.
  2. Automated testing during cycle 1 test case execution. This is where the automated tests were created and executed after the all the test cases were written. The test execution phase was reduced as a result. There was no manual test case execution but this phase would uncover defects in the test case calculations.
  3. Automated testing during test case creation. At some point, the test cases were still being written after the code was delivered to QA. Testers volunteered to run the tests as they wrote them. This allowed them to uncover defects in the test case calcuations before they would have been multiplied by copy and pasting. This further reduced the test execution phase time by eliminating test case related defects. The test planning phase though did take longer as the team started doing hourly batches of planning and executing.
  4. Automated testing during code development. In order to maximise the benefit of 3 above, when it became easy enough to run the tests in the lowest dev environment, the testers started working with the developers. This reduced developer bug fix time as we saw in the 6 month release that needed 9 months. At this point, my testers suggested that we just need a fixed length QA execution phase to run regressions provided there were improvements to setting up the data in QA.

How to determine when is it easy enough for developers to run the tests? I’d say there were a few factors, here’s some that I recall:

  1. QA test case defects need to be reduced. Creating incorrect test cases slow down the developers. Even if you don’t have calculations using a model based testing tool help improve the quality of the test cases themselves. Find as many causes for incorrect test case and try to eliminate them, you’re looking for common causes like the copy-paste because of floating point rounding errors.
  2. Start with a change that’s a production bug fix rather than a whole feature. There’s usually fewer people involved because the functionality is most likely better understood than something which is brand new.
  3. Instead of trying to automate all the tests, try just one test case with as few fields and steps as possible, then progressively add more fields and steps. By the time you’re ready to try automating tests for a new feature, maybe just focus on the bugs for that feature. I pick bugs because that’s probably where the developers have a weakness in their testing practice. There’d be no point in trying to automate tests for parts of the code the developers have no bugs.
  4. Don’t attempt to match any existing dev frameworks. Most likely they weren’t auto-generated and the general absence of patterns might make it hard to generate test cases that fit into such a framework. In time if the developers feel they want to migrate away from legacy testing practices, that’s up to them. In general the automated test cases should be disposable so they can be generated only when needed and shouldn’t add additional maintenance burden on the developers.