During the 4.5 years I was leading my team and working with the COBOL developers to run the QA tests in their development environments, they were never slowed down. Even the development managers of those developers could see no indication of any slow down and they did nothing to speed it up either. The code might have come into QA a little later but the developers saved significant time not bug fixing.

In fact, it actually sped up the development process. In my second last year there was a 6 month release. The dev team wanted 4-5 months to code and my team had estimated 4-5 months to execute the tests. This meant that we needed almost 9 months to do the work but only had 6. Thankfully by this point in time my team could recognise that this was the same problem as constructing the Empire State building.

The solution to which was to overlap phases which we did. In the end, the developers got the 4-5 months they asked for. My testers wrote and tested the code along side the developers as early as the second month in their development environment. The code was delivered into QA in the 5th month and soon after all the tests were run within a day and we then ran the regression. We were done somewhere in the middle of the 6th month.

Even though the tests were run in a day by the time the code got to QA I should add some notes

  1. The process of creating the setup data took days, because of all the new Java web-app code that had to be tested first and was buggy.
  2. One developer took longer to deliver the code to QA but still took less time overall because she had no bug fixes. More on this below.
  3. The regression for some reason was not run in the lower environments even though it could have been, I can’t recall why it wasn’t done.

If you were a fly on the wall watching the developers and testers, what would you see? I’ll go over the interaction between my tester and the developer who delivered the code later. For this release I had setup Slack conversations between pairs of developers and testers and included myself. I did this so that if they needed me to tweak the XText or Maven plug-ins, then I could do so ASAP. This was the closest I could get to seeing what was happening on the shop floor. What I saw was an interaction that organically came to be over months before this release. Twice a day, the tester would make the next small batch of tests, and the developer would prepare the next chunk of code. Then they’d run it together in seconds (it was automated) and make a plan for the next half day. Each half day was like a floor in the Empire State building being completed.

So how was this possible? The math didn’t add up for people. Let’s say a developer needs 10 days to code and fix bugs and a tester needs 5 days to write tests and 5 days to execute them. Now I want the developer to run the QA tests, how much work does the developer now have? So far people have given me one of these two answers

  1. 15 days. They just add the work to the developers estimate
  2. More than 10 days. They recognise that if the tests are run, there will be fewer bugs but there’s still a cost to automating the tests which is key to making this work.

The actual answer is that it works out to be less than 10 days, here’s how that’s possible

  1. All the tests if not more than 95% were automated so that means the test execution time actually goes down to seconds. If that’s the case, why wouldn’t the time for QA execution be 0 too? Well that’s because if the code is buggy, automated tests fail and then someone has to spend time checking them, logging defects etc. Also if there’s no bug with the code, the testers have to redo the math for the test cases, it’s what I referred to the re-planning phase.
  2. If the test are automated to run in a dev environment, surely that would need time to automate them by developers? The QA tests ran in an environment where the front-end apps were available but those weren’t on the COBOL servers. That should have increased the work for developers to make that automation right? Well not in our case, since the test automation was automatically generated from the ubiquitous language so that the test data could be extracted and injected into the backend database. This took me a day or two to make the mapping code the first time.
  3. Finally the defect fixing time all but disappeared because the tests were run right after the code was written in most cases except one (the dev practiced red-green-refactor). The developers would detect bugs while the code was fresh in their mind so it didn’t take long to find the issue and fix it. Bugs were still being created but the cost to fix them was much lower than finding them weeks later in QA.

I think I’d be on the right track if the following are true

  1. The overall time to execute test cases in QA goes down.
  2. Defect counts go down but planning time goes up because defect discussions happen earlier.
  3. QA tests executed automatically in lower environments goes up.
  4. Developers spend less time fixing defects because they’re just contract or integration defects and not complex logic ones.
  5. Developers can take more time to get the code to QA and nobody panics.
  6. More developers can pair up to help each other finish the work on time. This is important for monolithic builds.