Automated Tests Prove Progress

Should the tests that prove progress be AutomatedTests?

I don't think there is any choice. If the tests aren't automated, eventually no one will execute them (even if they are automated it can be a chore). When testing stops, the project goes to hell. It hurts when we do this.

There is a substantial price to automated tests. It's worth it. Pay it. -- KentBeck

So all tests for user stories can be automated? If they can't, it ain't a story.

So, do you apply the TestingFramework to represent the AcceptanceTests, or in what kind of framework do you run the tests?

A server can use something much like the TestingFramework xUnit. For GUI-oriented code I don't know of an outstanding solution. I'm trying to experiment with writing UnitTest's for a trading system (using TimeTravel), but I find myself writing gui code without tests all the time (one of the drawbacks of ExtremeProgrammingForOne is that there is no one there to stop me). The UnitTest's for the domain model help a lot, but I'm still falling back on PokeAndPeek far too much. -- KentBeck

I've used 3rd party tools for GUI testing, like SQA Test (now owned by Rational) and Microsoft MS Test. It's generally best to use a tool that can check and interact with the widgits you're using rather than just screen bitmaps -- so your tests will be more robust. It's tedious, but can be a real lifesaver. -- JeffGrigg

How do you automatically test the UserStory "The shuttle engines shall turn off 3 minutes after ignition"?

(Behind me asking this is a confusion in my mind abouthow much Xp is "just" about software. There's hardware, working procedures policies, and much more involved in the solution of many business problems. Sometimes part of DoTheSimplestThingThatCouldPossiblyWork is not doing anything in the software at all, but changing the other bits so that the functionality is no longer required in the software. I'm sure this is the wrong place to discuss this point - if someone wants to point me in the right direction I'd be greatful). -- PaulHudson

It's a fine place to start discussion of this point. This page first arose (as I remember) because I questioned the use of 'automated tests' (now 'repeatable tests') in Article 2 of the SoftwareManagementManifesto - with exactly this kind of issue in mind. Although the context there is a software project, the truly professional team always needs to be mindful of ways to meet certain customer objectives (which should still be testable) without writing software.

"Sometimes part of DoTheSimplestThingThatCouldPossiblyWork is not doing anything in the software at all, but changing the other bits so that the functionality is no longer required in the software" deserves a page of its own if this point is not already made forcably enough on Wiki. TomGilb and I would certainly co-author it with you! --> see SimplifyTheRequirements great WikiName, great points, what's the impact on testing?

It all depends on what a UserStory is allowed to be. OfficialXpPersonnel? -- RichardDrake

A UserStory can be a story describing any function the system must perform. I look forward to hearing how to remove "the shuttle engines shall turn off 3 minutes after ignition" from the software requirements. Supplying only 3 minutes of fuel won't work: I happen to know there's another story about getting out of orbit.

You forget:

The booster rockets run out of fuel and are dropped.
The main fuel tank empties and is dropped.
Another fuel tank, in the shuttle, is used for manuvering and to slow to reentry speed.

(Air friction and gravity undo most of what the first two did. ;-)

What do you mean by "tests that prove progress" ?

At first I jumped to the conclusion that we were talking about tests that somehow checked to make sure the system was actually getting *better* (smaller, faster, handles more data more efficiently, ... something like that). Some sort of time-dependent test that, instead of checking that the program could handle the full 500 GB in the UserStory, today only checked 0.5 GB, but next week it checked 5.0 GB, etc. ...

Are we really talking about the "normal" XP process of: AutomatedTest ContinuousTesting TestFirst ?

Those tests don't really prove "progress". But they are incredibly helpful in proving that the latest modifications to the program don't cause "NegativeProgress?".

CategoryAutomated CategoryTesting