Testing Tools

Funny that we had a contractor spend all his time and energy to build a 'custom' test tool for our project. It was some kind of a fancy, GUI-based tool. Obviously, it quickly became obsolete as we moved from one release to the next.

It was dumped.

Will any day recommend a command-line, easily executable test scripts that can be frequently run and results analyzed.

List of various testing tools

TestingFrameworks
Rational makes some good tools, but they're very expensive.
JTest. This does QA metrics and generates tests for every public+protected method for some basic things (like passing in nulls). Can run from the command line to test, retest, partially test etc. It was easy enough to use but to benefit fully you need to use it from the start of the project; keeping on the right side of its QA rules is like const-correctness.
ab (apache bench) is useful for some quick web stress tests
daiquiri will replay webserver logs in common log format; I found this really useful for reproducing and testing problems I had with a live system. No analysis of results provided though. http://www.cnds.jhu.edu/~jesus/projects/daiquiri.tar.gz
JMeter is an apache GUI tool which out of the box does web stress testing but is extensible so you can build your own tools on top of it. http://jakarta.apache.org/jmeter/index.html
eValid is yet another web testing tool. This one basically automates IE. It records your actions in scripts which you can edit; so its kindof like a cut down WinRunner/Rational Test
'script' in unix will record the commands you type in. Really useful for converting what you did to a shell script after the fact. (link to man page: http://linux.ctyme.com/man/man1972.htm)
DejaGnu - The FSF test tool, uses TCL as its scripting language. Designed to front-end testing which you can then do in any way you like, producing standardised output for analysis.

* ...

I am confused, is the testing tool meant to be used to write tests, or to execute tests? Is it used prior to work on the next release, or just before it is released? Are there commercial testing tools?

A question also arises from this, since the tool was dumped, is the contractor free to utilize the concepts and methods he developed in a product he would develop outside of the company? Or does the dumping squelch further development and forbid his use of it since the contractor probably signed some sort of restrictive contract?

I added some of the tools in above. Although eValid was crap for stress testing, its style of test tool seemed particularly good; record your actions and play them back, but also allow you to edit the generated script of actions that would be taken. A 'standard' test log from these tools (in the same way that there is a common log format for webservers) would be really handy, as it would allow you to plug & play analysis tools. There is actually a spec for this kind of thing, DejaGnu is the only tool I know that follows it! (http://www.fsf.org/software/dejagnu/manual/posix.html)

(see CapturePlayback for a list of eValid-like tools, none of which follow a standard log format either...)