Context Driven Testing

What is Context-Driven Testing?

The context-driven school of software testing advocates continuous and creative evaluation of testing opportunities in light of the potential information revealed and the value of that information to the organization right now. The context-driven school of software testing advocates testing in a way that conforms to the context of the project, as opposed to testing in a way that follows some fixed notion of "best practice". It declares that the human part of that context is most important, and that good testing is ultimately a matter of skill, not procedure.

KentBeck has criticized BobBinder's writing on software testing because it only focusses on benefits without regard to costs. Cost-benefit analysis is central to context-driven testing.

Context-driven testing could arguably be called agile testing because the principles it recommends are analogous to those suggested in the AgileManifesto. Specifically, context-driven testers would answer each of the following questions in the affirmative:

Do we put more value in individuals and their interactions over processes or tools?
Do we put more value in seeing working software over documentation?
Do we put more value in collaborating with our customers (including development, which context-driven testers see as an important customer of testing services) over negotiating contracts (and specifications).
Do we put more value in responding to change over following the plan?

However, no context-driven tester would want to be limited to the few testing techniques recommended by specific agile methodologies like ExtremeProgramming. (See discussion below for more on this point.)

Context-driven software testing is a set of values about test methodology. It is not itself a test technique. To be a context-driven tester is to approach each testing situation as if it were unique in important ways, and to develop the skills to react to situations with a broad and deep awareness of problems in projects and possible testing-related solutions to those problems.

The Seven Basic Principles of the Context-Driven School

The value of any practice depends on its context.
There are good practices in context, but there are no best practices.
People, working together, are the most important part of any project's context.
Projects unfold over time in ways that are often not predictable.
The product is a solution. If the problem isn't solved, the product doesn't work.
Good software testing is a challenging intellectual process.
Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.

The Development of the Context-Driven School

The context-driven school was founded by CemKaner, BrianMarick, JamesBach, and BretPettichord. The school was founded on November 21, 1999, when CemKaner, JamesBach, and BrianMarick were discussing the possibility of writing a testing book together. A few days later, they established the context-driven testing mailing list on Yahoogroups (software-testing). Two years later, the first self-described context-driven testing book (Lessons Learned in Software Testing: A Context-Driven Approach) was published by Cem, James, and BretPettichord. The context-driven testing manifesto was also distributed at that time.

Although the school was declared in 1999, its direct roots can be traced back to the eighties, when James and Cem were independently trying and failing to make prevailing "best practices" of testing work in the world of commercial mass-market software products. Independently, they developed test methodologies that were agile and practitioner-centered. In 1995, James and Cem met and began to collaborate with each other and with like-minded people to find better ways of analyzing and discussing the relative value of technical practices.

CemKaner and BrianLawrence? began hosting the Los Altos Workshops on Software Testing in 1997. These brought many testers together to share experiences and learn how to share ideas across barriers of project size, market, technology, and many other aspects of project context. In many ways, this was really the beginning of school of context-driven testing. Early participants in LAWST workshops included JamesBach, BrianMarick, BretPettichord, DougHoffman?, ElisabethHendrickson, NoelNyman?, JohannaRothman? ...

Many context-driven testers have looked to philsophy of science and epistemology for insight into how to think about software testing. One example of this is interpreting KarlPopper 's verification principle for testing scientific theories as a way of thinking about testing software.

Early Context-Driven Writings

In the years between 1992 and 1999 show a development of the ideas of context-driven testing in various writings.

CemKaner published Testing Computer Software in 1988, saying that testers need to work effectively in the common situation that individuals rather than specifications control the quality of the product and "the programmers don't, won't, and don't have to follow the rules." Prior to its publication, most testing literature said that testers could only be effective if they programmers followed the waterfall method, providing detailed specifications. This book was welcomed by testers who had worked with high-performance teams that had succeeded because they focused on speed and working code over extensive planning and documentation. Kaner supplemented TCS in 1988 with Test Planning for Consumer Software (http://www.kaner.com/pdfs/TestPlanningConsumerSoftware.pdf) that specifically urged testers to advocate for evolutionary development (which XP follows) over waterfall, to minimize test-related documentation, and to focus their quality vision from customer needs in rather than code or process out.

JamesBach wrote "Process Evolution in a Mad World" in '93, "The Immaturity of the CMM" in '94, and a series of other articles in '97 and '98 as part of his Software Realities column in IEEE Computer. "Good Practice Hunting", written in '99, was published shortly before the school was declared, and was inspired by the discussions at LAWST. All these articles may be found at http://www.satisfice.com. Many of these articles were seen as defining the validity of an attitude towards development that later became labelled "agile."

BrianMarick wrote "Classic Testing Mistakes."

More Principles

It is not a testers' job to demand documentation. Good testers work with the information they have, use unofficial documents, and are specific when requesting additional information.
Testing is a information-providing service, not a "quality assurance" function.
The value of testing is determined by whether it provides useful and timely information (about the quality of the software).
There are no guarentees. We don't hide behind the test plan. We learn how to improve testing as we test.
A tester is a customer advocate. Testers try to understand the customer position and make the best case when they feel it isn't being address.

Testing Techniques

Testing techniques developed in the Context-Driven Testing School include:

Further Reading

See the following links for more information:

This wiki page is being developed in DocumentMode. If you have factual corrections or additions, please make them above. If you have differences of opinion or interpretation with what is stated above, please comment below.

''It is tempting to call context-driven testing "agile testing" because the principles it recommends are in some ways analogous to those suggested in the AgileManifesto. However, agile development is a particular set of values that belong to one kind of context. Context-driven testing is broader than that.'' -- James Bach

I am wondering what context you are claiming that AgilePrinciples are restricted to? So far it seems that most of its successes have been on small projects, but isn't the same true of ContextDrivenTesting? Have we actually gotten a chance to deploy ContextDrivenTesting on a large scale?

I am also confused because you've described your early work with Cem as "agile and practitioner-centered". So you've been tempted to call it agile as well. Do you regret this?

Sometimes it is helpful to distinguish between testing on an agile development project and testing with an agile mind-set (on any kind of project). Is this the point you are trying to make?

-- BretPettichord

- Agile development is about agility. It may be worth pursuing in contexts where people consider agility to be a high priority. - Context-driven methodology is about solving problems that actually exist, here and now. It's worth pursuing in any context where people believe in solving actual problems.

In any context where agility is a high priority, context-driven and agile may coincide. However, context-driven methodology also applies where agility is not valued.

Two cases where context-driven methodology is not worth pursuing might be:

- if you are under the direct supervision and control of someone else who takes responsibility for the quality of your work.

- if you are in a context so well understood and so stable that it would be silly to continually question context and re-relate that to practice.

My work has been focused a lot on agility -- both in the dictionary sense and in the sense of the Agile Manifesto, but not in the sense of many of the testing practices of Extreme Programming, which are sometimes called agile, but maybe ought to be called "Agile" with a capital A.

-- JamesBach

So would it be fair to say that Context-Driven Testing and XP Testing could both be called "agile testing"?

Are AgileProcesses "about" agility, or is agility an emergent property of the agile values? Notice that the word agility is not in the definition of AgileProcesses:

We value

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

In other words, what changes, if any, do you want to make the main text above? Do you still think that it is controversial whether ContextDrivenTesting should be called agile?

-- BretPettichord

The wording of the article is okay with me, as it stands. I agree that CDT could "arguably" be called agile (we do advocate a certain agility of mind). CDT certainly becomes agile as needed. I also like the caveat about agility that is included in the article.

That the word agility is not in the definition of agile processes is completely unremarkable. Isn't it customary not to use a term in its own definition? In discussions I had with certain delegates to the Agile Manifesto meeting before that meeting occurred, it seemed clear to me that they were using agile as a descriptive term, not merely as some kind of marketing trope. When I read the Agile Manifesto, it seems to me that the description of Agile principles correspond both with an intent to be agile, in the dictionary sense, and with principles that will cause agility to emerge.

"Responding to change" is pretty close to the dictionary definition of agile, as I read it.

-- JamesBach

Let me suggest alternative answer to the question, "What is context-driven testing?"

Context-driven testers choose their testing objectives, techniques, and deliverables (including test documentation) by looking first to the details of the specific situation, including the desires of the stakeholders who commissioned the testing. The essence of context-driven testing is project-appropriate application of skill and judgment. The Context-Driven School of testing places this approach to testing within a humanistic social and ethical framework.

Contrasting context-driven with context-aware testing.

Many testers think of their approach as context-driven because they take contextual factors into account as they do their work. Here are a few examples that might illustrate the differences between context-driven and context-aware:

Context-driven testers reject the notion of best practices, because they present certain practices as appropriate independent of context. Of course it is widely accepted that any "best practice" might be inapplicable under some circumstances. However, when someone looks to best practices first and to project-specific factors second, that may be context-aware, but not context-driven.

Similarly, some people create standards, like IEEE Standard 829 for test documentation, because they think that it is useful to have a standard to lay out what is generally the right thing to do. This is not unusual, nor disreputable, but it is not context-driven. Standard 829 starts with a vision of good documentation and encourages the tester to modify what is created based on the needs of the stakeholders. Context-driven testing starts with the requirements of the stakeholders and the practical constraints and opportunities of the project. The standard provides implementation-level suggestions rather than prescriptions.

Some testers advocate favored life-cycle models, favored organizational models, or favored artifacts. Consider for example, the V-model, the mutually suspicious separation between programming and testing groups, and the demand that all code delivered to testers come with detailed specifications. Context-driven testing has no room for this advocacy. Testers get what they get, and skilled testers have to know how to cope with what comes their way. Of course, we can and should explain tradeoffs to people, make it clear what makes us more efficient and more effective, but ultimately, testing is a service to stakeholders who make the broader project management decisions. Yes, of course, some demands are unreasonable and we refuse them, such as demands that the tester falsify records, make false claims about the product or the testing, or work unreasonable hours. And yes, of course, some demands are absurd because they call for the impossible, such as assessing conformance of a product with contractually-specified characteristics without access to the contract or its specifications. But these are specific situations. The fact that we need a specification if we are to assess conformance with that specification does not mean that we always need specifications. We start from the project's needs, not from the process preferences.

Contrasting context-driven with context-oblivious, context-specific, and context-imperial testing.

To say "context-driven" is to distinguish this approach to testing from context-oblivious, context-specific, or context-imperial approaches:

Context-oblivious testing is done without a thought for the match between testing practices and testing problems. This is common among testers who are just learning the craft, or are merely copying what they've seen other testers do.
Context-specific testing applies an approach that is optimized for a specific setting or problem, without room for adjustment in the event that the context changes. This is common in organizations with longstanding projects and teams, wherein the testers may not have worked in more than one organization. For example, one test group might develop expertise with military software, another group with games. In the specific situation, a context-specific tester and a context-driven tester might test their software in exactly the same way. However, the context-specific tester knows only how to work within her or his one development context (MilSpec) (or games), and s/he is not aware of the degree to which skilled testing will be different across contexts.
Context-imperial testing insists on changing the project or the business in order to fit the testers' own standardized concept of "best" or "professional" practice, instead of designing or adapting practices to fit the project. The context-imperial approach is common among consultants who know testing primarily from reading books, or whose practical experience was context-specific, or who are trying to appeal to a market that believes its approach to development is the one true way.

Contrasting context-driven with agile testing.

Agile development models advocate for a customer-responsive, waste-minimizing, humanistic approach to software development and so does context-driven testing. However, context-driven testing is not inherently part of the Agile development movement. For example, Agile development generally advocates for extensive use of unit tests. Context-driven testers will modify how they test if they know that unit testing was done well. Many (probably most) context-driven testers will recommend unit testing as a way to make later system testing much more efficient. However, if the development team doesn't create reusable test suites, the context-driven tester will suggest testing approaches that don't expect or rely on successful unit tests. Similarly, Agile developers often recommend an evolutionary or spiral life cycle model with minimal documentation that is developed as needed. Many (perhaps most) context-driven testers would be particularly comfortable working within this life cycle, but it is no less context-driven to create extensively-documented tests within a waterfall project that creates big documentation up front.

Ultimately, context-driven testing is about doing the best we can with what we get. There might not be such a thing as Agile Testing (in the sense used by the agile development community) in the absence of effective unit testing, but there can certainly be context-driven testing.

Reasonable people can advocate for standards-driven testing. Or for the idea that testing activities should be routinized to the extent that they can be delegated to less expensive and less skilled people who apply the routine directions. Or for the idea that the biggest return on investment today lies in improving those testing practices intimately tied to writing the code. These are all widely espoused views. However, even if their proponents emphasize the need to tailor these views to the specific situation, these views reflect fundamentally different starting points from context-driven testing.

/s/

Cem Kaner, J.D., Ph.D.

James Bach

CategoryTesting

I think also, where the original piece states - "Specifically, context-driven testers would answer each of the following questions in the affirmative: ", it's off the mark. Context-driven testers would say 'It depends', and then describe the factors upon which it depends, tradeoffs and relevant experiences. See this post - http://www.satisfice.com/blog/archives/462

Jared Quinert