Coding Java Unit Exception Tests

Please make a summary of 1 or 2 good frameworks and explain them. Do not review x frameworks: it is not a newsgroup discussion (otherwise it will be endless). (VladimirBossicard)

MikeDuffy? asked me to comment on an on-going thread on the JavaUnit mailing list. The thread was about coding tests that expect exceptions. Here were my thoughts.

There has been a lot of talk about how to implement test cases that check that an exception is thrown. Several code patterns have been proposed. I would like to propose the simplest pattern (A) as this:

 public void testIllegalArguments() throws Exception {
  try {
MyClass myClass = new MyClass(null);
fail("Created a MyClass with a null foozle?!");
  }
  catch (IllegalArgumentException expected) {
assertTrue("Exception message doesn't even mention 'foozle'?!",
expected.getMessage().indexOf("foozle") >= 0);
  }
 }

Here is an explanation.

We add a fail() message after calling the method that throws our expected exception. This signals that the test case has gone too far: "If you got here, you're dead." Notice the meaningful message. It indicates what the test did that was wrong without getting too technical. There may be other kinds of illegal arguments to the MyClass constructor and you may wish to roll a few tests - each testing a different kind of illegal argument - into one test method. That's a style issue and you can do what works well for you.

We catch the expected exception, calling it "success", because that makes it clear to the reader that catching this exception is a success - or at least probably a success.

Inside the "catch" block you should check that the exception instance is as you'd expect. This test requires the exception detail message to mention "foozle", which is the name of the illegal argument. If there is nothing to check, leave this empty.

(As an aside, when you catch your own homegrown exceptions, do take the time to add useful information to the exception object. A "detail message" is good, but you can do better. I use a pattern for writing exceptions that, although commonsense, doesn't appear to be popular. Still, it's quite useful.)

Finally, the test itself throws Exception. If MyClass' constructor throws any other kind of exception, then this test registers as an "error" instead of a "failure". This is logical, since if the constructor throws some other kind of exception, then likely something very strange is going on. This could be more than a mere test failure, but rather some other strange RuntimeException indicating that the operating environment has gone bad: disk full, that kind of thing.

There have been other suggestions. Let me address them here.

This pattern (B) has been proposed:

 try {
 //tested code starts here.
 ...
 fail( "should have thrown exception TestedException?" ) {
 catch ( TestedException? e ) { return; }
 catch ( Exception e ) { fail( "thrown wrong exception: "+e ); }

Notice the "return" inside the catch block of the expected exception. This is superfluous and I'm not sure it communicates anything useful. Not only that - if we have more than one such test inside a single test method, the "return" statement must be removed. In that case, never add it. I have suggested using the exception local variable name "success" to draw the reader's attention to the fact that the test expects the exception to be caught - that is represents a successful test.

Also notice the block catching all exceptions and reporting a test failure on them. I used to do this, but have found that it's extra code for no real gain. Not only that, but after seeing a few exceptions thrown that were symptoms of a bad operating environment, I decided these should be errors and not failures. For this reason, I prefer to propagate the exceptions up to the JUnit framework and let the framework handle them.

We have also heard about the ExceptionTestCase? extension. KentBeck is right: it's overkill, so you're not likely to use it. The acid test is this: given time, would you write it yourself? If so, use it; if not, then don't.

Finally, please note that these tests are very important. Often, programmers leave "error handling" to the end; something which always - ALWAYS - comes back to haunt them at least once. Think back to a long, painful debugging session after which you screamed, "Why didn't they just throw a DiskFullException??!?!! Then I would have known what was going on right away!" Don't hurt yourself. Throw good exceptions. Pretend you're your client - imagine the code you'd like to write in their place.

 try {
 doSomeCrazyStuff();
 }
 catch (YouBlewYourHeadOffException? oops) {
 logger.out("Accidentally blew off our heads. Ouch. (Detail: " + oops.getMessage() + ")");
 }

Can it get any more clear than that?

Happy coding.

From reviewing the argument on the JUnit mailing list, it seems the major reason for catching the exception so that you can fail is that you can diagnose from the failure message what went wrong. This seems to me to be a little spurious; if you are using unit tests properly, you know what went wrong to cause the failure: it's the last thing you just did. If that's not enough information to explain the failure, then odds are you're going to have to dig to find out what went wrong. At this point, the proposed pattern becomes harmful; you don't have the stack trace, so you don't know where the Exception was thrown. If you have a few steps inside your try/catch block, you don't even know what step caused the problem.

By contrast, if you just declare the unit test to throw Exception, any unhandled exceptions will propagate out to the framework. They are then caught as errors, which result in test failure. Furthermore, all the details of the exception are available as well; class, message, and stack trace. In the default textui for JUnit, stack traces are displayed in full. If more details are required, then an alternative test runner can be used. As an example, the test runner included in the Jakarta Ant project displays class, message, and stack trace. -- RobertWatkins

Just to clarify what I think I've read above: The body of tests for a given class should also act as documentation on how to use - and not use - the target class. So, for tests representing the HappyPath (i.e. an exception should never be thrown), ignore any thrown exception. In Java, this means you just declare that the test method "throws Exception".

The documentation (aka tests) must also include instructions to a developer on how things can go wrong, and what happens when they do go wrong. In this case, you do want to trap the exception in your unit test, failing if it isn't thrown. The try-catch block clarifies your intent on how you want developers to treat the failure.

I think pattern A above is perfect. Pattern B is fine, but trapping the wrong exception is unnecessary. If your code is correct, other exception types will not normally be thrown, other than in unusual situations, in which case JUnit will trap them.

Writing unit tests for exceptions drives me toward UsingSpecificExceptionTypes, instead of toward writing ugly code that tests against some sort of specific error string. -- JeffLangr

I have been using the following pattern:

Exception thrown = null

try{
 x.method(badData);
}catch(Exception e){
thrown = e;
}

Assert.assertNotNull(thrown);
//assertion will fail if thrown is still null after try/catch

One can obviously test the exception type with more accuracy (checking e.getMessage() is handy too, to make sure it's carping how you want it to)

-- McClainLooney

When testing exceptions in unit-tests, I've sometimes heard the exceptions affectionately called 'expections'. I think this word is brilliantly self-explanatory and easily memorable. This type of testing could then be called UnitTestingExpections?. -- PaulRuane

NUnit provides an attribute called ExpectedException? to signify that the code under test should throw.

Instead of:

assertTrue("Exception message doesn't even mention 'foozle'?!",
success.getMessage().indexOf("foozle") >= 0);

I use this:

assertExceptionContains("some regexp", expectedException);

Reason: this is much more readable, and since I need to quote my test cases in bug reports, occasionally even show them to management types, it's damn important for me :)

// elsewhere in a framework base class,
// which is subclass of junit.framework.TestCase itself
public void assertExceptionContains(String expectedPattern, Throwable exception) {
assertMatches(
SOME_CHARACTERS + expectedPattern + SOME_CHARACTERS,
exception.getMessage());
}

-- AlexeyVerkhovsky

CategoryTesting: TestingFramework for Java. CategoryJava. JavaUnit. JavaUnitBestPractices

CategoryException