Canoo Web Test

CanooWebTest is an OpenSource tool that uses Ant and HttpUnit to implement functional testing of web applications.

The home page can be found at

http://webtest.canoo.com/

I've checked this tool out at the homepage and it seems pretty neat. Anybody out there using it? I am also a bit curious why XML is used to script the tests. The only advantage I can see is (as they mention on the site) that one might be able to generate system documentation from the test cases using XSLT. But besides that XML is quite a clumsy tool for scripting, as far I can see.

It works by adding WebTest?-specific tasks to Ant, and Ant scripts are XML-based. It would be nice to have a simple GUI tool for generating the test scripts, so that we could give it to our 'on-site customer' to generate tests for the site. --StuartBarker

Having the tool out now for almost 2 years (7/22/03), I feel encouraged to share some experiences here.

Usage: We have about 8000 downloads of the build.zip, 4000 in the last 6 month. We assume that on average each download results in one installation. This assumption is backed by the load increase of the documentation access. More than 200 users subscribed for the mailinglist, a number of developers contributed to this OpenSource effort. The size of projects that use CanooWebTest range from a-few-person-month up to a several-hundred-person-year. The largest TestSuite reported so far exceeds 30000 (!) steps and still runs in an acceptable amount of time.
OpenSource: The number of contributors is fairly low as we impose some rigid rules to ensure quality. Before granting write access to the repository, contributors must prove their understanding of the concepts by sending one contribution along with the UnitTests, functional selftests and documentation. However, it seems that quite a lot of users make local adaptions to the code.
CruiseControl: The CruiseControl running for this project has proven to be of highest value. Along with the ANT scripts for automation and the Unit and functional testing, it has saved us numerous times from inconsistent states of the build. For an OpenSource project I would never start without it.
Concept: The most important decision in this project was about what language the user should face for writing tests. We felt that this language needs to provide the possibility to eliminate duplication. That means we need a concept of modules that in turn need parameters or variables as language constructs. We found that Ant would provide all this and a little more.
TestFirst: Having failed painfully with "record-replay" tools multiple times, we wanted CanooWebTest to support writing tests before the implementation. It turned out that this had an unexpected benefit: it was much easier to tell expected/asserted behaviour from accidential behaviour.
TestCoverage: In addition to the CruiseControl and integrated in it CanooWebTest measures the coverage of its selftest. You can explore its TestCoverage online since it is reported with every build. We aim for over 99% coverage.

Features

Ease of Writing Tests: Well, it is hard to claim that writing Ant scripts is actually easy for the average user. However, there is a lot of support that we get from XML-tools without coding a single line. That includes syntax highlighting and code completion based on the DTD. On the mailinglist an intranet back-office project reported that WebTests were written by the end user, i.e. the secretary staff.
Reporting: This feature turned out to be crucial. All executed Test steps get logged in XML format and optionally visualized via XSLT. There is also an option to store all intermediate results (i.e. web pages) for later browsing. We found that this visualization of the result also makes an excellent "storybook", a complete description of the asserted behaviour. BTW: this style of reporting also allows for easy integration with CruiseControl as results can get reported with every new build. The CanooWebTest MavenPlugin makes use of this feature.
Build integration: Being an Ant task, build integration is for free. It is no effort to integrate any suite of WebTests into your build such that the tests are guarantied to be called after each build, optionally stopping the build in case of error or failure.
Life Cycle: As tests are text files, they can live with your code undergoing the same source control. The same is true for test results. You get a full history of what version of your code ran what suite of tests producing what results.
Eat your own DogFood: CanooWebTest is not only tested via UnitTests but also via AcceptanceTests. The tool used for acceptance testing is again CanooWebTest itself :-) This also includes different usage szenarios, build and CruiseControl integration and reporting. You can see this "live" at the distribution site.
Ease of extending: It is easy to extend CanooWebTest or to adapt its behaviour. You can do that without touching the existing codebase. The usual Ant mechanism of writing and using "tasks" is used here. A sample extension (and - yes - a sample tests for this extension) comes with the distribution.

-DierkKoenig

CategorySoftwareTool CategoryTesting