Customer Test Framework

I am having trouble finding the tool I need for writing automated customer (acceptance / functional / ... ) tests for our web development project. I want to be able to write automated tests in (legible) xml, to simulate user interactions with the web application, manipulate the (relational) database, and xml configuration files. Ideally we'd like 'scripting' type features like conditional logic and variables.

The idea is that just before implementing a user story, we sit with the customer and write the tests for it. Think 'pair test development' - developer and customer both at the screen.

Of the many testing frameworks available, so far HttpUnit and Avignon are possibilities, HttpUnit does everything I need except that the tests are specified in Java. Avignon does almost everything I need, including specifying the tests in xml, but the only problem is that although avignon is open source (great), it does not seem to be a 'community' type of project, it is a proprietary tool for which the source has been released under GPL. It is not immediately clear whether it is still in active development, how I could submit patches etc.

Does anyone know of any other similar frameworks I could check out?

Might it be a good idea to initiate a sourceforge project, perhaps starting with the avignon code? Anyone interesting in contributing to such a project?

-- MatthewReeve

Have you looked at FrameworkForIntegratedTest?

Yes. I have gone back and checked this out again, and I'm still failing to have the 'aha' moment. I don't understand the desire to express acceptance test as tables.

When I write a test, I want to simulate complex sequences of user interactions actions with the system, and make assertions about how the system responds. The simulated interactions need to be informed by the configuration of the system, and may contain complex conditionals and flow-control. HttpUnit does all of this, and provides all the power of Java, but it will be difficult for the customer to read. I want a format for my tests that the customer can read, but that retain most or all of the expressive power of the Java language statements. A table just doesn't seem intuitive to me for expressing this kind of logic.

To illustrate further: I see an automated customer test as similar in essence to an ant build script - they are both just convenient ways of automating a sequence of actions that we need to do often. The only difference between an ant build script and a customer test (in my mind), is that ant script simulates the actions of a developer building a piece of software, and a customer test simulates the actions of a customer/tester/developer testing a piece of software. XML seems like a natural format for expressing this kind of thing, but I can't say the same for tables. I'm sure it's /possible/ to express any program (script, test, etc) as a table, or set of tables, but I don't see why you might want to (sorry Ward!). My brain just doesn't parse the tables as easily as an XML document (could be due to the fact that I spend a good portion of my life editing XML documents!).

I am happy to accept that I am misunderstanding something; please help me to see the light. :-)

If your customer is able to read and write XML, then expressing tests in XML may be just fine for you. But while XML is more expressive than tables, the nice thing about tables is that they are more familiar to most people. The table output is easy to scan because it is largely identical to the input except that it is color-coded to show discrepancies between actual and expected results. Bear in mind that FrameworkForIntegratedTest supports different kinds of tables: the ColumnFixture in which field columns contain input values and method columns contain results; the ActionFixture in which rows represent a sequence of actions and queries; and the RowFixture in which rows represent objects in a collection, showing the result of some kind of filtering. -- JonReid

Have you looked at FIT. I have been reading Test-Driven Development in Microsoft.NET by James W. Newkirk and Alexi A. Vorontsov. (Founders of NUNit) That is what they recommend. It is on SourceForge. I think the table format in HTML would be easier for a Customer to understand than XML.

Mark Dockal Jr.