Acceptance Test Examples

See AcceptanceTests.

Please list concrete examples of real-life AcceptanceTests here. If you have several related to the same project in a WikiPage, include a link to that page here. Thank you!

Some examples:

An example: a sonet switch must do an APS switchover in 50ms. There are many scenarios under which this must be true and the setups are very complex.

Another example: when equipment X is pulled, alarms Y and events Z are generated.

This list of this kind of things goes on and on. A customer could care less about any of your unit tests. They want the things they care about to work, that's why it's an "acceptance" test. --Someone

So when we say that the customer is 'writing acceptance tests', they're not actually writing any kind of code; they're writing plain English and the developers must translate the plain English into an automated test?

How do the developers/customers know that the tests the developers translate actually test the things they wanted to test?
If the developers write poor tests (based on poor understanding, I guess), doesn't this give the customer a false sense of security, since the features aren't fully tested?

It just depends on the test. The timing tests are validated with hp test equipment. No code involved at all. And for the rest, of course, if they do a poor job they will get poor results. The degree of acceptance testing varies greatly by customer. Some will have very elaborate tests with many people working on it, others customers will do very little. --Someone

I'm sorry, I was unclear. I was talking about the kinds of automated AcceptanceTests used in ExtremeProgramming. I understand (all too well) that on a typical project customers can be extremely flaky about 'acceptance' and whether or not they try out the release. I was trying to get a handle on the ExtremeProgramming practice of having the customer write automated AcceptanceTests that get run overnight to track the team's progress.

The rule is that a UserStory gets Development full points if it passes its AcceptanceTest. What results from that rule in any given context depends on that context; some customers won't be able to read code at all and will trust that developers write correct tests, others might be able to read the tests, others might have a QA team writing "adversarial" tests, yet others might prefer to write tests themselves; what tools or languages are available for writing ATs will influence those factors.

I believe it doesn't matter all that much; the salient feature of an AcceptanceTest is that, as code, it can't lie. Whoever writes the AT must be honest, because the test is a formal specification and conformance can be formally verified. Honest errors in the ATs will be corrected, either as a result of the customer noticing something amiss when the system is in production, or preferably as a result of somebody noticing a vagueness in the spec when they sit down to write, or implement, the test. The nice, trust-building aspect of ATs is that such things are not "bugs", they can only be recognized for what they are - breakdowns in communication. Accepting that the solution is to communicate more follows naturally.

Great, thanks! That clarifies a lot about how AcceptanceTests get from the customer's head to the machine. How about a concrete example of code that is an AcceptanceTest (i.e. it was specified by a real-life customer)? I'm really curious to see one. I've seen quite a few UnitTests so far, and even some good UserStoryExamples, but the AcceptanceTest seems to be an elusive hand-wavy kind of thing. The feedback from UserStoryExamples seems to indicate that I'm not alone. Just one example would be great. This would be a great way for PromotingXp through education. -- RobHarwood

Here are some real AT methods from ProjectCanon. Delete at will if someone has better and/or more succint examples to post. Using JavaUnit, HttpUnit, XpathLanguage, plus some supporting code. The longwinded comment with the story in it isn't actually in the code, I'm just including it so you know what we're testing. Even the story card isn't that verbose; we talked about what the code had to do, and the conversation went into the AT.

 /***
  Story "User Login" (2 pts)
  Story : after creating a user, the system will know that you are
  that user when you login with that user's id and password; if you
  are not authenticated, or if you supply a bad id/password pair, or
  other error cases, the login page is displayed. If a CMS folder is
  marked as requiring authentication, access to any page under that
  folder will result in an authentication check.
  ***/

 public void testNotLoggedIn() throws Throwable {
  testNotLoggedIn("/");
  testNotLoggedIn("/edit.cms");
  testNotLoggedIn("/does/not/exist.cms");
 }

 public void testNotLoggedIn(String page) throws Throwable {
  endSession();
  connectTo(page);
  assertLoginError("mustlogin");
  assert(getForm("login") != null);
 }

 public void testIncorrectLogin() throws Throwable {
  login("/","stoopid","stoopid");
  assertLoginError("badlogin");
 }

 public void testIncorrectPassword() throws Throwable {
  login("/","userone","two");
  assertLoginError("badpassword");
 }

 public void assertLoginError(String code) throws Throwable {
  assertExists("//head");
  assertExists("//head/meta[@class='error']");
  assertExists("//head/meta[@class='error'][@category='authentication']");
  assertExists("//head/meta[@class='error'][@category='authentication'][@code='"+code+"']");
 }

 public void login(String page, String login, String password) throws Throwable {
  endSession();
  connectTo(page);
  WebRequest? toLogin = getForm("login").getRequest();
  toLogin.setParameter("username",login);
  toLogin.setParameter("userpass",password);
  getResponse(toLogin);
 }

 public void testCorrectLogin() throws Throwable {
  testCorrectLogin("userone","one","User One");
  makeNewUser("usertwo","two","User Two");
  testCorrectLogin("usertwo","two","User Two");
 }

 public void testLoginExpires() throws Throwable {
  testLoginExpires("userone","one","User One");
 }

 public void testCorrectLogin(String login, String password, String realName) throws Throwable {  
  login("/",login,password);
  assertDoesntExist("/head/meta[@class='error']");
  assertExtract("//span[@class='userinfo']/text()",realName);
  // And it persists in the session...
  connectTo("/");
  assertDoesntExist("/head/meta[@class='error']");
  assertExtract("//span[@class='userinfo']/text()",realName);
 }

 public void testLoginExpires(String login, String password, String realName) throws Throwable {
  testCorrectLogin(login,password,realName);
  endSession();
  connectTo("/");
  assertLoginError("mustlogin");
 }

This is a very nice and neat TestUnit?. Thanks for sharing it ! -- StephaneMor?

The same tests for non-programmers (like customers, testers): CanooWebTest

I'm sorry, I was unclear. I was talking about the kinds of automated AcceptanceTests used in ExtremeProgramming.

Well, i'm dealing with real customers, not idealized customers working with idealized programmers. Most customers will not have the talent or resource to do anything like what you are suggesting. The ones that do then great. We deal with 20+ customers simultaneously. Every one of them wants something different.

From my understanding, the customer doesn't actually write the ATs, they just specify them. The programmers translate that into the ATs.

I have previously used Ant scripts for doing the AcceptanceTests. The customer specifies the input and output in simple format like (just tabular data) and I wrote the Ant tasks to read the input, set up the task and compare the output with expected.

The only hassle was that Ant would quit after first failure, so I had to not throw the BuildException? but log it and every target would have a "reporter" task that will go through the exception log and report failures to Ant.

The advantage with Ant is I do not have to deal with command parsing or the parameter files. - VM

CategoryExtremeProgramming | CategoryExtremeProgrammingExamples | CategoryTesting