Testing in Chaos

At Delver we work in Blitzkrieg mode: we work hard, develop fast and improve our asskicking search engine almost on a daily basis. Working like this is very exciting, I can sometime feel the imaginary wind in my face, but from QA perspective I have grown to realize that we have a problem testing efficiently in this development environment. Here's a list of the problems and questions I'm dealing with:

• Sometimes the builds we get in QA should have never have left dev. The builds contain critical bugs which render the build untestable.
• I'm not always sure how thoroughly we should test each build. Through testing of a new build is currently ~20 man hours for "core", previously tested, features. That's not including new features.
  
• I don't know when I'll get a new build. The longer we delay the build, the more time I have to test it as more features are added.
• What amount of testing should be performed when configuration changes are done in Production?
• If I test a build thoroughly on my lab, do I have to test it with the same amount of thoroughness when it is uploaded to Production? Where should I focus my testing efforts? Lab? production? Both?
  

With all these problems I sometimes feel I'm losing control of the process and that I react intuitively instead of in a planned manner. I want to test in the most efficient manner without changing the R&D flow. I want the best coverage for the minimal effort and I want to invest resources where needed, i.e. in new features and risky areas and reduce the amount of routine testing performed by my team. How am I going to tackle these problems:

• Develop a set of Sanity tests which cover all aspects of our application without going too deep into any of them. For example: Make sure Search feature works, but don't test special characters or long strings.
• These Sanity tests should be performed by developers before handing out new build to QA.
• These Sanity tests should be performed by Operations team when they change a hardware configuration and want to make sure they didn't break anything.
• We will automate these tests and instruct Operations and dev how to run them.

• We have a rather impressive set of Regression tests, most of them automatic. I'll divide them into two levels:
• Level I (Core): The Core of Regression tests. Deeper than Sanity tests, faster than Full Regression. All automatic tests are Core but not all Core are automatic. When do we test Core tests?
  
• When a build is not Production candidate, or
  
• When only minor changes were made since last Build
• Level II (Full): All Regression tests. When do we test Full Regression
• When a build is a Production Candidate, and
• Major changes/features are implemented in this build

To summarize, here is the procedure we'll use regarding new builds and configuration changes in delver:

• Every new build should pass a Sanity test by dev before it is ready for QA.
• When Operation changes configuration or install minor fixes which did not pass QA, they should also run Sanity tests.
• QA will run Core tests on new builds unless the new build is both Production candidate and major changes were performed since last build. In that case, Full Regression tests will be performed.
  
• New features will tested thoroughly on the build they become available. In later builds they will be tested either as part of Core or Full Regression tests.

I'm still not sure about how much resources should I invest in testing in Production versus Lab, but that's a matter for another post. Sometime in 2009, probably.