We Test For Quality But Who Cares?

Nicely Written by eric Jacobson

The most challenging presentation I saw at Stareast was by Google Senior Test Engineer, Goranka Bjedov. She makes the case that the world is heading toward developing software without testing for quality and that this practice may not be a bad thing. Scary but true!

First, Goranka pigeon-holed testing into two categories; productivity and quality. Her definitions (per my notes) are as follows:

Productivity Testing – Making sure programmers don’t break code (e.g., unit tests). Testing things consumed by machines. Anything consumed by machines is easy to automate. These tests are cheap, fast, well-defined. The problems failed tests expose do not require deep analysis.

Quality Testing – Testing things consumed by humans. Anything consumed by humans is not easy to automate and therefore difficult to test. Expensive. Tests become more flaky as the system becomes more complex. The right tests are not clear. Failed tests require deep analysis. These tests take longer.

With the promise of quicker software delivery, productivity testing has become more important than quality testing. Wake up, the world is already adapting in several ways.

For example, at Google, they know hardware and infrastructure will always fail. Instead of wasting time with exhaustive tests, their solution is to manage risk (e.g., build in seamless failovers and backups) and shield the user from the failures.

Goranka also countered that in cases where poor quality is seemingly not optional (e.g., medical software) users have already adapted by not relying on it. She claims users in hospitals, for example, know not to trust someone’s life in a piece of software. Instead, they monitor the patient as a human and understand that software is fallible.

These are excellent points, IMO, and I would have been satisfied contemplating a future where my job no longer existed...but hold on!

Goranka asked us to do a little exercise. She asked us to determine the rule used to generate these three sequences by writing five additional sequences of our own:

-25, -5, 15, 35, …

2, 4, 6, 8, …

0, 3, 6, 9, …

I don't want to give away her rule, but you can still try it on your own.

After surveying the audience, she pointed out that developers tend to write confirmatory tests more than testers, who tend to write more negative tests. Thus, perhaps testers do play an important role. She also questioned how much productivity tests actually tell us about the system as a whole. Her answer? …they tell us nothing.

In the end, she left us with this thought…

If you think (non-programmer) testers are important, you better start doing something about it.

Nicely Written by eric Jacobson