Web Acceptance Tests

So you've got an application that uses the web. How do you write AcceptanceTests for it?

Maybe-viable methods include:

Testing the HTML output.

Pros:

• No change to app -- just test it as it is.
• Doesn't depend on rendered layout.

Cons:

• Doesn't let you test visual things. For example, how would you write a test for some fancy JavaScript doo-dads?
• Changes in HTML layout might break the tests. I doubt this is a serious problem, though.

Testing an intermediate representation, such as XML.

Pros:

• Doesn't depend on rendered layout, or HTML layout.

Cons:

• Requires you to have a mechanism to spit out an intermediate representation.
• Doesn't give you the ability to test rendered stuff (same as HTML above).

Using Tcl windowing scripts (such as WinTclSend)

Pros:

• Browser-independent.
• Generically useful. If you learn how to script a web browser through simulating mouse clicks and keyboard input, you'll be able to script any GUI app.

Cons:

• Unless you know how to avoid it (which you can, often, by using keystrokes instead of mouse-clicks), you can end up coupling yourself to exact pixel values, which are prone to change.

Using WindowsScriptingHost control over the InternetExplorer object.

Pros:

• Might be easier than a Tcl solution.

Cons:

• Only tests one browser. You'll have to do something else for Netscape/Mozilla/Opera/etc.
• Only runs on MS OS's.

---

Note: If you have actually used one of these methods, please add your experiences above.

---

What about using something like JMeter? http://jakarta.apache.org/jmeter/ It's primarily designed as a load testing tool, but is it a good or bad idea to use it for AcceptanceTests?

---

How I've been doing this lately:

• Give up the idea of writing web tests.
• Move as much as possible into another library and test that, making the web application the absolute thinnest shell around that library that I can. (Generally more possible then it may seem, if you think about it enough.)
• Optionally, use the JavaScript unit test library for any JavaScript you may need tests for.

Trying to test web applications in toto has proven too difficult to be worthwhile, so shrink it as far as possible until it's so thin that while you may not be happy leaving it "untested" you can at least be reasonably confident about it.

It may not sound XP, but XP is pragmatic, right? In my experience, the cost of doing unit tests on the part of the web application that can't be shoved into an independent library far exceeds the benefits; generally the biggest problems left over at that point are browser layout differences anyhow and your Unit Tests can't really catch those in a meaningful fashion.

This implies the use of an environment where you can create code that runs in both a web server and a separate test suite. This eliminates some environments from consideration, such as old-style ASP code based on Visual Basic Script. I think you mostly have to just face the fact that there is no reasonable way to test ASP code, because the environment is inimical to this style of working.

We're over 10 years into the web revolution now; if someone was going to figure out how to write useful and flexible testing code for web applications, I think somebody would have done it by now. Instead, what we have either doesn't work, or is so difficult, tedious, and fragile to use that nobody wants to use it (which is just another form of not working). It's time to throw in the towel.

---

CategoryTesting