Acceptance Test

A formal test conducted to determine whether or not a system satisfies its acceptance criteria and to enable the customer to determine whether or not to accept the system.

Originally called FunctionalTests because each acceptance test tries to test the functionality of a user story.

Acceptance tests are different from UnitTests in that UnitTests are modeled and written by the developer of each class, while the acceptance test is at least modeled and possibly even written by the customer. ...Hence the even-newer name, CustomerTest.

Proposed by KentBeck in a poll on the XpMailingList at the beginning of April, 2000. Approved by the committee. Now the standard term in XP.

Q. Who is the committee?

Acceptance Test or Acceptance Criteria - Example form:

A [named user role] can [select/operate] [feature/function] so that [output] is [visible/complete/etc.] and leads to a Yes/No decision for user and developer

The Acceptance Test or Criteria should be more specific then the User Story description, and better define the customer's conditions of satisfaction. The Criteria should not include things like validation, boundary conditions, or "if then" statements.

You wrote this definition in such an abstract way, that it inspires to transfer it to public transparent negotiation processes in <-> ExtremeOpenBusiness. As in XOB all participant agents (could include AI processes) are constantly testing each other's "offers" to accept or reject, we get a sequence of Yes/No decisions for selecting visible features of each agent's offers. In a mature system the agents exchange and share their accepted resources, which are always in developement. Yes/No triggers the effort of what gets more development energy. (Take this note only as a memo-marker ;-) -- FridemarPache

Could the AcceptanceTest be considered a formal requirement specification? At least it is formal in the sense that it is unambiguous.

An AcceptanceTest is formal only in the sense that it satisfies a contractual obligation between the customer and the developer. For more information, please see CustomerTest.

I have a question. How exactly do customers write acceptance tests? Does this mean that they need to be programmers? Do programmers translate their conceptual tests into automated testing code? How is it done? This is a very important question for me since it will make-or-break a proposal to start ExtremeProgramming at our company. -- RobHarwood

I'd suggest getting your customer to get as close to a formal specification (i.e. code) as she can, then take over from there. It's a good enough approximation of "the customer writes the test" to start with. I've had some success using ATs (Acceptance Tests) (though they turn out to be vastly more difficult to do right than UTs (Unit Tests)) which I wrote myself after asking the customer, "If it was you doing a manual acceptance test, how would you verify that this story has been completed, after I claimed it was completed ?" and attempting to get as close to that in my code as I could. There might be other, better ways. When you find a better way, please share. -- LaurentBossavit

Thank you, and I will! :-) Anyone else have any examples?

Taking Laurent's approach one step further, in the case of some of the web apps I've worked on, it has been possible to quite straightforwardly automate a lot of the UserStories using HttpUnit. It's quite common for clients to describe bits of functionality like 'On the home page, there's a button which links to the feedback form which has <some fields>, and when they submit the user gets a message saying...' (simplistic, obviously, but the pattern holds for more complex things).

And now that I think of it, any sort of server (or even pseudo-server) app ought to be AcceptanceTest-able in the same way, by automating the client interaction. So as someone points out below, your users don't need to be able to program, they just need to describe the things they want to be able to do. Or am I missing something?

Gosh, I am astonished that you have customers who can (almost) program and understand the intricacies of testing. Certainly on the XP projects I have worked on, we have mainly looked for a full-time "Customer" as somebody who understood the business being supported and is authorized to prioritize functionality. We then get them to work closely with a professional programmer, who starts an on-going dialogue with them to tease out appropriate acceptance tests for stories and translate them into code. Furthermore, at least in my experience, the acceptance tests tend to be continually negotiated and refined throughout development as requirements (i.e., stories) become clearer and the adequacies/inadequacies of the current acceptance tests are revealed. -- Anthony Lauder

There seems to be a misunderstanding; we are in violent agreement. The suggestion above isn't that the customer should (almost) be able to program. The customer will know how the system must behave for the story to be considered "done", since at the very least they will eventually have to use the system !

One of the DeveloperBillOfRights is that we have a right to precise, detailed specifications. So we are justified in asking the customer to provide us with a very formal, very detailed description of what they would do - if you sat them in front of a screen and said "we're done implementing, check this out" - and this description is the acceptance test.

There is one step that isn't necessarily taken by the customer, that of transcribing the acceptance test into code. Development can do that, although it will necessarily affect velocity. Dedicated QA coders could do it, although I wouldn't want to miss out on the design-time benefits of coding my own ATs.

Yes, I think we are in agreement. My main point, though, is that since acceptance tests form the "spec" for a story, they emerge from on-going dialogue between the customer and the developers as the story unfolds. That is, at least as I have experienced it, since a story is only a "promise for conversation", requirements are refined throughout an episode, wherein we continually refine acceptance tests as we go along. Sorry about any misunderstandings - please feel free to delete my comments. -- Anthony Lauder

The TgpMethodology tries to answer this by working with BusinessProfessionals, which are the customer inside the development. You can't expect your user to be a programmer, and you can't be yourself a customer, because you don't have the customer's point of view. However, you can have analysts who know (better than the developr and the customer) what to software should do. Since OrganicTesting lets you write tests in a declarative way, the BusinessProfessionals should be able to write them, and the customers should be able to understand them. -Moddy Te'eni

ThankYou both! This is great info. Does anyone have a different experience with AcceptanceTests?

I am a bit lost about this AcceptanceTest thing. I do understand acceptance tests for things like long filter pipelines or protocol implementations, but I really don't see how you make acceptance tests for interactive programs. I'm not concerned about the technical difficulties, mind you, I do believe you can always make a GUI / curses emulator and have it drive the test. It's the semantical difficulties I'm concerned about.

Usually, the interactive part of the program is, at least in my projects, the most changing part of the program. I think that specifying exactly ("the 'follow' button should be at location (100,100) and the 'borrow' button in location (150,100)" or even something tamer like "the 'frobnize' menu item should bring up a dialog which has a 'do the frobnization' button to begin the frobnization") what the user interface should be like is pure madness. But if the user interface is not specified in this way, how do I test it? What kind of test can I write that will pass if I change the menu items to have more descriptive names? And regroup them? And add a new language to the program, if it's i18n'd with, say, gettext? What about checking that a diagram was rendered correctly, so that the test passes even after I've added drop shadows and gradients to the diagram? When I add AI to organise my pie char so that "similar" items are put in adjacent sectors? And, what is more, how do I write a test that will not pass when the user interface is sloppy and hard to use, but will pass when it's slick, easy and clear?

Do you really have customers that write precise specifications? That sounds like pure stone age to me - are you not shifting BigDesignUpFront to your customer? How much work hours does the customer spend to write their specs, how much do you use to code acceptance tests for changed specs?

I'm not against acceptance tests but I don't see how you could have a comprehensive acceptance test set for any piece of software that is supposed to deal with people (instead of other programs) and be developed iteratively. I'd love it if somebody would clarify these things to me. I suspect you're not supposed to write acceptance tests at all for things like the above, but FunctionalTest and this page make it sound as if you are.

-- PanuKalliokoski

If you GUI system is famous, you may find a special test package, like HttpUnit for web applications. But what if we have to create the GUI from scratch in the application (a game actually)? Does it worth the cost if we build a test framework by ourselves?

We did a lot of AcceptanceTesting for WebApplications? using CanooWebTest. -- DierkKoenig

Sorry, I forgot to mention earlier that - at least in my experience - sometimes a subset of the customer's AcceptanceTests are not necessarily automated at all (unlike UnitTests), so there isn't much programming to do. Rather, the user sometimes just wants to sit at a keyboard and actually use the system to see it working in practice. Of course, you would build up a pile of UnitTests yourself when implementing the required functionality. -- Anthony Lauder

I've had clients with all sorts of difference acceptance criteria for software. The most common I've found, in Europe and New Zealand, is for my QA team to have written functional tests and passed the system as being of a high enough quality to be released to the client, and then for the client to run their own set of acceptance tests on the software to prove to themselves that it meets their requirements. Our formal workflow is from development (which includes unit testing) to QA (who have been involved with the project at quite a detailed level from the outset) to UAT (UserAcceptanceTesting?).

[Please see QaIsNotQc.]

However, I've also had clients who have asked me to propose requirements because they don't really understand the context in which they want their software deployed. This is most common with eCommerce applications where the client isn't really familiar with the web, but has a non-web product that can have an 'e' placed before it.

In this model how can the client write the acceptance tests? Do I, in effect, become the client? I certainly know more about the software product than the actual client does. Or perhaps the client is a role fulfilled by two people in this case, both the actual client, and myself.

This raises an issue, though. Will I be as effective in my role as customer as a real customer would be?

-- BryanDollery

I have a lot of experience in Acceptance Testing flight simulators. Most customers expect us to write the Acceptance Tests (which we call FQT - Formal Qualification Testing, they agree on the tests and watch us do the tests, which can take from 1 to 3 weeks. In addition they have a few days of "Free Flight" in which they exercise the simulator's capabilities. This, of course is a big headache ... what do you do when the customers' pilot says "this doesn't fly quite right" .... ??? Still, it is usually the customers final acceptance of the system.

In this area, the customers' specifications are often quite short, say "I need a flight simulator to train in Emergency Procedures and Instrument Fight Rles for the following aircraft ..." I am expected to analyze the requirements and write the requirements myself.

Perhaps the biggest problem here is that the Customer is not the User: the Users are pilots while the customer could be American Airlines or the Chinese Air Force --staff personnel that pay the money :)

All in all, using XP in this area is very hard. I have found the best method is to fake it. On the one side we have a formal contract with the Customer, BigDesignUpFront and all that, and on the other hand I try and act as Customer and steer the project to where I think it should go. I write the stories, write the Acceptance Tests, prioritise, etcetera. The Customer sees none of this.

-- NissimHadar

IMHO, most of the questions posted on this page can be answered by a tool called FIT, FrameworkForIntegratedTest.

This tool was created by WardCunningham.

It allows any customer, with no programming experience, to design AcceptanceTests with any editor capable of saving documents in HTML.

It requires some programming from your development team, very much like an xUnit test.

The execution of tests with FIT can be fully automated (with Ant, NAnt or any similar scripting tool).

The customer can combine tests in the HTML document into a bigger, composite test, without the need to reprogram the original, parcial test programs.

All information on FIT and the tool itself (it is open source) can be found in http://fit.c2.com/

--GastonNusimovich

GrigoriMelnik, GerardMeszaros and JonBach? provide a good discussion of the acceptance process and a collection of thinking models in their new book "Acceptance Test Engineering Guide: How to Decide if Software is Ready for You and Your Customers" (available for preview and commenting at http://testingguidance.codeplex.com/)

CategoryTesting | CategoryCustomer