QA Feature owner - when testers become project managers

In our R&D organization, new features are usually led by developers. While managing the new feature, the leading developer is assigned the title Feature Owner for that feature. She delegates some work to other developers while performing some coding tasks personally. The larger the feature, more time is spent on managing and coordinating efforts and in smaller ones the FO spends more time coding. The interesting issue (for me) is the interaction with the testing team.

Most features are multi tiered: there's a backend layer, some application layer and a UI. Tests for each layer are different, of course: Greybox testing for backend layer, SQL scripts for database related tests, UI Automation and manual tests for the UI, etc'. The problem is, most testers don't feel equally comfortable with each testing methodology so what tester do I sign for each feature? If I assign a manual tester to a "mostly UI" feature does it mean we want test the backend? My coder can't handle the UI automation suite, maybe I'll just skip automatic UI test?

The other alternative is assigning several testers for each feature, based on their skills. This ensures that each tester is assigned tasks she's good at but it also poses other problems:

• The overhead of testers learning new features is now multiplied by number of testers involved in testing the feature
• Dispatching work between the testers and writing a test plan can not be done by the development FO, as she does not have the testing know how. Also, who is to know all the bugs related to the feature? Each assigned tester? Too messy and getting a quality overview is hard.
  
• Why not assign the QA team leader for dispatching tasks and writing a test plan? Because most of the time she will not do the actual testing on the feature, so we have a paradox: the QA person who studied the feature and designed the tests does not test it and the people who test it did not study it thoroughly (overhead, remember?)

The solution we are practicing now in my testing team is assigning a QA Feature Owner (QFO) to the development Feature Owner (FO). The assigned QFO is usually the person who's skills are the most relevant to the feature, but she needn't posses all the required testing skills: the QFO will assign tasks to other testers in the team in areas in which she lacks or simply does not have time to run or design tests.

This is how it works: upon reading the spec, the QFO designs a test plan, a document detailing the general testing strategy. In this document risks are identified and then tests are written as a sketch: a few words describing the general purpose of each test. We find it convenient to start by designing Greybox (GB) tests, then Automatic UI tests (AUI) and finally manual tests.

It doesn't matter if the QFO is a programmer or not when designing GB tests. Using Data Driven Testing methodologies most GB test cases are list of methods and what they are supposed to do and attached CSV files which are no more than input/output files based on the original spec. AUI tests are simply stories in English ("go here, click there") which can also be augmented beforehand using CSV files. Test cases are later reviewed by other team members to validate their testing logic and enrich the tests in areas in which the QFO is weaker. For example, a manual tester can review the manual tests written by a QFO who has strong programming background and suggest additional tests. Test cases will later be examined by the FO and QFO together, in order to ensure full coverage.

Once all test cases are approved by the QFO and feature and test coding begins, the QFO starts following the testing project. The QFO is responsible for ensuring that tests are run but not necessarily by the QFO personally: testing tasks can be assigned to other testers as long as the QFO remains the testing team's focal point for the feature. The QFO does not need to be the one reporting all defects but must know all defects opened which are relevant to her feature. The QFO is responsible for the testing of the feature but the actual tests may be performed by others.

We have started implementing this methodology only a few weeks ago and it remains to be seen whether we benefit from it on the long run. I don't take the fact that we're asking hands-on testers to become project managers for granted and believe some will adapt to it better than others. Others will resent the time spent in meetings and reporting and will feel more at home getting their hands dirty with testing. I have no intention of forcing any tester to become a QFO, I consider it a perk, not a requirement. I do, however, believe that tester assuming the role of QFO will enjoy the challenge of managing a testing project, interacting with more team members from the R&D than they are used to and experience new testing methodologies. I believe a good QFO is a potential testing leader.

QAing in a startup

After 8 years in big companies like Mercury (R.I.P) and Symantec I have started working for Delver, a startup company, in late 2007. After 15 months of starting and running the QA team I am wiser about QAing in a startup company.

1. Testing is Testing is Testing. Good QA is about methodology: be methodic in your risk analysis and in your tests. This principal doesn't correlate to the size of the company. In Delver we won't write a test case to a feature if it's not documented. I don't settle for oral briefing or email overview. No document, no test case. Easy, really. On the same principal, no testing without a test case, even for small features.
2. Speed over quality. In a startup company the emphasis is on fast development with speed being first priority, quality second. This perspective dictates more investment in speeding up the testing process. We used the following methods:
   
-Automation for testing relatively stable components
-Testing new components outside the main build if the build is not ready
-Issuing "non release" build for focusing on delicate features

3. QA freedom/No safety net. Another aspect of speed oriented, developer dominated organization is the freedom for QA. Since the focus is usually on the developers the QA team in Delver enjoys more professional latitude than what I have experienced on larger organizations. The flip side is the lack of testing peers and no organizational testing history. There is no safety net for me in Delver and when I make a methodical mistake usually there will be no one to correct me. This is an exhilarating feeling, but it's also a scary one. Like all good thrills, I guess.
4. Smaller budgets. Big, commercial testing software is expensive. Startups don't like "expensive". Startups like "cheap" or even better: "free". On one hand this seems like a major limitation (oh, who am I fooling. It is a major limitation.) but on the other hand it's also a challenge and a cool one. Finding a 300$ alternative for a 80K$ load testing suite is a personal victory and shows me that I can overcome seemingly seemingly impassable hurdles (financial, for example). In the future I'll dedicate a post to cheap and free tools we use in Delver.
   
5. No bureaucracy. Well, this one is obvious but it's still cool. When I need to ask something I can go to the guy and ask him. No email to another office, no phone calls to another continent. Go to the guy. Ask. End of story. If I want to hire a candidate I can give him a contract a day or two since the first interview. If I need a new license I simply go to the room of the guy who signs the checks and ask for the money. Simple. Fast.
6. More flexibility in reporting bugs. I don't open a bug for every problem we encounter. Since the company is a small one, when I'm not sure I can simply go to the room of the developer who is responsible for the code I found the problem in and verify whether it's a bug or not. This can be done in larger organizations as well but sometimes the relevant developer is not in the same building. Or town. Or continent.
   

Half baked versions are good for you!

Often, we find ourselves in a situation in which a feature is ready but the version as a whole is not: It can take anywhere between a day or two to a week until all the developers have finished working on a build and all integration tests are concluded, but during this time some features can be tested. So we came up with a method for speeding up the testing process: testing incomplete builds.

The idea is simple. When a feature is ready, we get an incomplete version. This version is installed on a separate, local, system and only the specific feature (or features) is tested. We don't test other components of the application and when bugs are found there we ignore them. This way, by focusing on specific, new, features, we are able to detect bugs before the release candidate version is released to the QA. Once we get the official version, we are more familiar with the new components and they are usually more robust as the most glaring bugs were purged earlier.

Testing in this method helps us test more and more often, but one must bear in mind that testing this way can't replace thorough testing later when the full version is tested. Some bugs only rear their ugly heads in more complex environments and need to be detected separately.

Testing in Chaos

At Delver we work in Blitzkrieg mode: we work hard, develop fast and improve our asskicking search engine almost on a daily basis. Working like this is very exciting, I can sometime feel the imaginary wind in my face, but from QA perspective I have grown to realize that we have a problem testing efficiently in this development environment. Here's a list of the problems and questions I'm dealing with:

• Sometimes the builds we get in QA should have never have left dev. The builds contain critical bugs which render the build untestable.
• I'm not always sure how thoroughly we should test each build. Through testing of a new build is currently ~20 man hours for "core", previously tested, features. That's not including new features.
  
• I don't know when I'll get a new build. The longer we delay the build, the more time I have to test it as more features are added.
• What amount of testing should be performed when configuration changes are done in Production?
• If I test a build thoroughly on my lab, do I have to test it with the same amount of thoroughness when it is uploaded to Production? Where should I focus my testing efforts? Lab? production? Both?
  

With all these problems I sometimes feel I'm losing control of the process and that I react intuitively instead of in a planned manner. I want to test in the most efficient manner without changing the R&D flow. I want the best coverage for the minimal effort and I want to invest resources where needed, i.e. in new features and risky areas and reduce the amount of routine testing performed by my team. How am I going to tackle these problems:

• Develop a set of Sanity tests which cover all aspects of our application without going too deep into any of them. For example: Make sure Search feature works, but don't test special characters or long strings.
• These Sanity tests should be performed by developers before handing out new build to QA.
• These Sanity tests should be performed by Operations team when they change a hardware configuration and want to make sure they didn't break anything.
• We will automate these tests and instruct Operations and dev how to run them.

• We have a rather impressive set of Regression tests, most of them automatic. I'll divide them into two levels:
• Level I (Core): The Core of Regression tests. Deeper than Sanity tests, faster than Full Regression. All automatic tests are Core but not all Core are automatic. When do we test Core tests?
  
• When a build is not Production candidate, or
  
• When only minor changes were made since last Build
• Level II (Full): All Regression tests. When do we test Full Regression
• When a build is a Production Candidate, and
• Major changes/features are implemented in this build

To summarize, here is the procedure we'll use regarding new builds and configuration changes in delver:

• Every new build should pass a Sanity test by dev before it is ready for QA.
• When Operation changes configuration or install minor fixes which did not pass QA, they should also run Sanity tests.
• QA will run Core tests on new builds unless the new build is both Production candidate and major changes were performed since last build. In that case, Full Regression tests will be performed.
  
• New features will tested thoroughly on the build they become available. In later builds they will be tested either as part of Core or Full Regression tests.

I'm still not sure about how much resources should I invest in testing in Production versus Lab, but that's a matter for another post. Sometime in 2009, probably.

Performance testing mistakes (I've made)

I've been testing for Performance for eight years now, more or less. When I started testing, I didn't even know I'm supposed to test for "Performance", I was simply the- dude-who-knew-VUGen. Neither me, nor my manager cared much about performance testing methodologies: I'd simply record a script, run it with multiple users and report my findings. This is a very bad attitude to run performance tests, as I have found out in several heated QA-dev meetings. The worst part was seeing my credibility deteriorate, and my hard earned findings doubted. Over the time I managed to pinpoint my mistakes and win my credibility back, but the process was long and painful. In this post I'll discuss some of the biggest performance methodology mistakes I have committed over the years.


1. Configuration problems. This one was the hardest on me. You run a test, you find what you think is valuable performance issue, you call some developers and have them investigate your system, only to find out that you ran out of disk space. Or the debug level was to high. Or the memory wasn't properly configured.

Always verify your system before a test. Read all the Read Me's, even the ones no other QA tester reads. Same goes for Best Practices. In one of the companies I worked for, we found a DB configuration document no one even knew existed, and we configured our system properly. Prepare a checklist a verify it before each test. Misconfiguration will bite you. Hard.


2. Mixing conclusions with findings. I love monitors. I disperse them generously across the applications and servers and monitor all relevant process (and some which aren't). In the end of the day, I sometimes have 100-200 monitors per server. While this is not a problem in itself, it became one when I didn't know what to report.

When asked to present my conclusions, I used to report anything which held a trend: Working Sets increasing, Context Switch peaks, Threads amount changing. The graphs looked impressive and I felt like I have a lot of "meat" in my reports. Problem started when people started asking: "well, what does it mean?" , "isn't it normal?" or even "is it good or bad?"

In many cases, I had no idea. I figured that other R&D personnel knew what those stats meant even when I didn't. Well, they didn't either.

Many trends are cyclic by nature (resource consumption is increasing and then decreasing), some reach a flat plateau after increasing and some rise and fall like saw teeth. In most cases, Monitors of hardware usage only give you hints of the problem and rarely the problem proper. There are some exceptions to this rule: high CPU consumption over a prolonged period of time is bad in most cases, for example.

The findings are important, as they help know your application better and know it baseline. If you're already familiar with specific monitors (I/O of process x, for example) following its trend can help you detect when an application is misbehaving, but in most cases monitors results are best left out of your conclusions section and kept in the findings baseline. Report only things which are definitely bad and you're sure about.


3. Fighting your previous war. There's a saying about the army that it's always preparing for the previous war instead of the next one. While it's always fun to make jokes about the army, most people handle the current project similar to the way they handled the previous one. I once had my engineers spend a week trying to locate memory leaks which weren't there, only because the previous product I tested suffered from memory leaks. In another time, it took me some time to adjust to the fact that the load on the system I tested is not caused by user activity, but by amount of data flowing into the system. Bye-bye 100 concurrent users load test.

Performance problems on different systems are caused by different factors affect different components. Study the system you test, learn what hurts it and how does it express its pain. Design your tests after you are familiar with the system and try to verify with yourself whether you're testing what you're testing because you're used to it or because it's relevant to the current product. Fight the current war, not the previous one. Or you can join the army if you like the uniform, though.

A subsection of this issue would be to test what's easy for you test. Nice try, but no. Don't.

4. Convoluted reports (no one reads). I'm not intimidated by putting together a 200 pages summary document, I actually take a perverse pleasure in it. It had taken me some time to understand no one really reads them if you simply throw the reports in their faces (or in their inbox, for that matter). Having a 10 MB, 200 pages monster uncoil before your eyes is a scary sight when you're unprepared and the natural reaction of most people I sent the report to was to close it and "read it later".

So I changed the format of my reporting and now my reports get the attention they deserve (most of the time, anyway). R&D managers are like ADHD children: they have short attention span and they like bright and shiny things, so in order to get their attention I compose a mail they can easily digest. The secret is to put nice colorful graphs and simple explanations summarizing your most important conclusions in the mail you send to the relevant people. No more than 2-3 issues. Put the rest of the report online somewhere and attach a link to it. Most people will ignore it, so you better make sure that executive summary count. Colorful graphs, simple explanations and no more than 2-3 conclusions. Easy.

What's QA?

I had a not-too-bright manager once, who told me that the job of the QA was to ensure the quality of the product. I told him that I can't do that. I can ensure the quality of the miniatures I paint, the salad I cut or, to some extent, the manner of my daughters. I can't ensure the quality of code I don't write, it's that simple. Ensuring the quality of the software is not my job.

I often see QA engineers who feel personally offended that software is released with (some) open bugs, or that their bugs are re-categorized as enhancements. I always tell them that being offended by the way our bugs are treated is not our job.

In the organization I currently work, I encounter Dev personnel design and run tests. I was surprised and a little offended at first, but I soon realized there's no reason for that. Hell, even testing is not my job (though I'd better be good at doing it).

So what's my job, actually? It's not eating these donuts.

In Delver, we have a white board in the meeting room. My job is to go to that board once in a few weeks and write the product's status. I give a smiley to a component which is ready for production, so-so face to a component which still has some bugs before it's ready and a frowning face to a component with critical issues. That's my job as a QA manager. Go to the board and paint some faces with a dry erase marker.

Because the role of QA is not to ensure the quality of the software, not to fight for bugs and not even to test the product (yes, I'm being a wise ass here). The role of QA in organization is to be able to reflect the quality of the software. In order to do that we have to test or help the dev personnel test. We must be efficient, resourceful and clever in our tests. We must know the requirements and understand how the users use the software. But in the end of the day, our job is to reflect the product's status.

Are there critical bugs you feel are ignored? Publish them. Make sure everyone who should know about them, knows. Should you care about known defects the software is released with? of course, you're part of the team. Is it your job to stop the version? No. Your job is to make sure that the people who make the decision about shipping it know everything they should know.

Here we go

I test software for roughly eight years. I learned a lot and I still do. I have started this blog in order for me to help others, talk about the stuff I like and rant about the stuff I dislike. I believe I'll focus on Performance testing which I love and QA in Startups, which is the challenge I'm currently facing. I intend to keep this blog mostly professional, but I can't help it if I mention my family. my daughters are simply too cute, I apologize in advance.