Tuesday, October 18, 2011

Evaluating workers, skills Vs traits

How do we evaluate our employees? We know the traits we value in employees, but which traits are more important and which are less? Through gut feeling we may feel that trait A is more important than trait B, but how does each of them translate into day to day performance?

In this post I will discuss a rather cool method we have developed for determining which traits are the most important for an employee to posses, how to evaluate employees and more. The first steps are rather obvious, but starting from the third step it's getting rather cool. Trust me.

Some time ago, I had a conversation with one of my Team Leaders, regarding the needed skills and traits for a QFO (QA Feature Owner). We made a list of the traits we valued the most and wrote them on a whiteboard but then the calender reared its ugly head and we were forced to stop and vowed to resume the session "soon".

"Soon" turned out to be some six months later. When we met again, were no longer two people in the room but six, as I asked other leads from my team and two HR to join the party. After a long discussion we decided that our way for evaluating our employees will consist on the following phases:

1. At first we decided what are the needed skills the QFOs needed. The skills we decided our QFOs need are:
  • Task planning
  • Task duration assessment
  • Synching other interfaces
  • Reporting
  • Following up on features
  • Context switching
  • Task prioritization
  • Problem solving
  • Specification document analysis
  • Writing organized test documents
  • Effective test coverage
  • Good test prioritization
  • Bug hunter
  • Organized bugs
  • Through bug analysis
  • Bug prioritization
  • Etc' (this post is not about the skills required to be a good QFO)

2. After we were happy with the skills needed, we set about writing down the traits we thought good QFOs should posses. The list we came up with was something like this:
  • Fire and forget
  • Broad vision
  • Self driven
  • Self improving
  • Passionate about the product
  • passionate about QA
  • Technical skills
  • Organized and thorough
  • Businesslike (practical, focuses on getting things done instead of ego games)
  • Critical thinker
  • Creative

So now we have two lists, skills and traits. Now we put them in an excel table, with traits being columns and skills being rows:


3. And now the cool stuff begins. In the next phase we marked each box in which a trait and a skill intersected. For example, Task Planning requires a QFO to have a Fire and forget attitude, posses a Broad Vision of the team's goals and be Self Driven. So we marked those intersection boxes with 1's. Self Improving, on the other hand, is not especially important for Task Planning so we left that box empty. We proceeded to fill the table until it looked something like this:


4. In the next phase, we summed (in the last row) the number time each trait was used:



Looking at the last row, we see that the three most important traits for a QFO in our organization are being Organized and Thorough, possessing a Broad Vision regarding the company's goals and being a Critical Thinker. We know that they are important because they affect the largest amount of skills we defined as important for a QFO's daily work.

Next post I'll do more fine tuning and show how to use this method for evaluating employees and giving them more effective feedback.












































































































































































Tuesday, October 4, 2011

The role of QA in a Continuous Development environment

One of the challenges we, as a QA organization, face in a Continuous Development environment, is that the "old" definitions of QA are no longer valid for us. We are not the "gate keepers". We do not give a "green light" or "Go-no-go" signals. Why? Because if we want to deploy a new version with new features on a daily basis in production, we need to move fast and repetitive dev and QA cycles are a huge overhead.

In the following bullets I have summarized the challenges our R&D organization faces and the way we in the QA team should tackle those challenges.

  • What do we want to achieve?

o We want to release features as soon as they are ready, regardless of other features’ readiness.

o We want to keep the time between the developer starts coding and the feature is deployed in Prod to a minimum.

  • · How are we going to keep the development cycle short while releasing on a daily basis?

o Reduce the amount of “links” in the release chain – only one person (FO, development Feature Owner) will be needed to take a version to Production (no dev->QA->dev->QA->OPS->Prod, but instead only dev->Prod)

o Reduce the overhead of Dev/QA interaction. This overhead is bearable when releasing a version every two weeks (if the QFO (aka tester in some organizations) is stuck she can do something else) but becomes unbearable when releasing on a daily basis (if the QFO is stuck, no QA work is done and as result the whole process is stuck).

  • · How are we going to reduce the number of links in the chain and reduce dev/QA friction?

o QFOs (testers) will no longer be a “required phase” for releasing the version. Instead, QFOs will now specialize in providing tools for the development Feature Owners to verify that the released version is of high quality. If the QFOs provide high quality tools and the FO uses them properly, the version will reach production in high quality.

  • · What tools will the QA provide for the FOs?

o Critical review of Spec and Design

o Test cases

o Automatic tests (where the QFO is able to code automation)

o Exploratory tests once the feature reaches production

  • · Who does the actual testing? It depends:

o Automation can be written either by the development Feature Owner or QA Feature Owner, depending on availability and proficiency in automation coding.

o High priority manual tests are usually performed by FO. QFO assists when needed.

o Low priority tests are QFO domain

o When a feature reaches production, after it is deployed, QA run exploratory tests on it, to see if we missed anything

  • · Who owns the quality?

o FO owns the quality of each release. The FO bases her assessments of the release quality based on the tools provided by QA.

§ Why can’t QA own the quality of each release? Because if the QA is to own it, we will need time testing it and this is the exact overhead we’re trying to cut out.

o QA owns the general quality of the application. They are required to reflect the quality of the application to decision makers and help them focus the bug fixing resources on the most painful areas.


Our CEO, having seen the above bullets in an email thread, challenged me and asked: "so if you guys are not doing the testing and not the owners of quality, why do we need QA?". Fair question, to which I gave the following reply:

Why do we need QA?

  • (Most) developers don’t write good tests.

o Developers tend to think positively instead of negatively. The tests that they will write will make sure that the feature works where used properly, but will not find where it breaks.

o QFOs usually have broader view of the system and see interactions between components developers tend to neglect when designing tests.

  • QA still owns the quality of the application end to end.

o QA is responsible for examing all the exiting problems in the system and generating relevant heat maps that will serve decision makers when deciding resource balancing between developing new features and fixing bugs.

o Even if developers test their features perfectly without QA assistance, QA is needed to reflect whether the experience is good. Are the response times reasonable? Is it easy (not only possible) to register? Is the newsfeed interesting (instead of merely working)?

o QA is still the team to ask “what is the quality of the our offering?”

  • I believe that QFO can also take ownership of automating the tests, as we see in some cases. Developers will code the feature and QFOs will code the tests and hand them to the developers.