Part 1 can be found here.
Keeping it within constraints
I experimented with a ton of approaches to reach my objective. I knew that going and fix 20,000 tests would not scale, so there had to be some form of evaluation. Here are some ideas I tried:
- Fix the failed tests.
- Fix the ones that have found defects in the past.
- Remove tests that have not found defect in the past.
- Remove tests for features that are not high priority.
- Replace tests with a new designed tests.
I reviewed my findings with some team members to get feedback. The problem with these techniques is the effort of evaluating this many tests. It could work, but they depended on my resources. This was tricky, the more resources I would need, the more I ran the risk of getting into time consuming negotiations with management. I decided to create my own constraints that would seem “reasonable” to my product unit. After some not-entirely-scientific calculations I came up with:
- Due date: End of March 2010. This coincides with the release of Dev10, which made it easy to justify.
- Resources: 3 persons. This decision fit within a bigger initiative that we were driving in the team. The basic idea was that this group would provide key services that will allow the rest of the team to really focus on product development.
Around this time, I decided to become a manager lead. My group, named Engineering Services, would consist of 3 team members and one of our charters was the Test Bed goals that I previously described. This is not our only responsibility, but it is up to us to manage our time properly so that this project is a success.
It may sound backwards for some, but by arbitrarily choosing constraints that were easily “agreeable”, I got no push back and now had clear constraints that I could work with to come up with an actual strategy. Of course, it helps that I have a manager that believes in me.
Deciding on a Strategy
It was November 2009, which meant I had 5 months to turn things around and I had to juggle this deliverable with at least 3 others (won’t go into them): hand off our test bed to the servicing team, driving test passes, take ownership of tests.
I concluded that my first priority was to create a “base line”. The idea was to create a small test bed that was well understood and robust that we could use to make intelligent decisions about our coverage going forward. I entertained the idea of creating it from scratch, a compact super integrated test bed, but I abandoned it in December after a costing exercise. Instead, we would use some of our current assets to create the base line.
To understand this next part I need to talk about numbers. Our test bed is divided into 3 priorities, what each priority meant was never clearly defined, which means that it is totally subjective to what the tester thought what Pri 0, 1 or 2 meant. We knew that a human at some point made the decision of how to prioritize their tests, but we didn’t really understand how that decision was made. The break out was:
- 2,000 Pri 0
- 10,000 Pri 1
- 8,000 Pri 2
2,000 looked manageable, so the strategy became: the Pri 0’s will become our base line, once we get these tests to be stable and robust, we can analyze whether more tests are needed. This was a touchy subject, it read like we were going to start fresh by only keeping Pri 0 tests and discarding 18,000 tests. That made people nervous. However, this is only step #1, once we got to a stable state we would evaluate if more tests are needed in any area and surgically move tests from the 18,000 to our new test bed. The point was to protect our test bed, from now on, the only thing that goes in there are things that product development absolutely needs and that we approve.
Thanks to Leslie for trusting in my methods and to Drew who helped me bounce ideas with him.
Next: Details and Making it Happen.