Organizing Test Cases

There is some discussion on how to organize test cases (in Java, mostly ;-). Everybody is welcome to make his/her suggestions here.

[OrganizingTestCasesInJava??]

(For "A Very Crude C++ Method", see OrganizingTestCasesAveryCrudeCppMethod. ;-)

What works fine in our environment is the following setting:

1) We make the following directory structure for the project X:

X classes production test

2) 'production' holds all the java files of your production code, 'test' holds all the sources of your test cases and 'classes' holds all the class files. That means that you have to use the -d option of your compiler to create the class files in the 'classes' dir.

3) The clou: if we want to test "com.x.y.Sample" (in the production dir) we create the test case as "com.x.y.SampleTest" (in the test dir). Sure, we have these "AllTests?" classes in each package.

Using this setup, we have separated test and production code and still can test features that have package-wide visibility. If we want to eliminate the class files of the test cases, we just delete the whole 'classes' dir and recompile the 'production' dir. (Or delete all *Test.class files if you feel confident in this feature)

The same idea can be used to deal with javadoc.

-- DierkKoenig

Another way of organizing code, which I find quite useful, is:

X code test

"code" contains all of the application code;

"test" contains all of the unit tests

the CLASSPATH includes both X/code and X/test

That way, I still have physically separated test and application code; the test classes for a particular package are still in the same package as the application code; and to omit all of the compiled test classes, I simply omit the test directory.

Similarly, by telling Javadoc about only one hierarchy, I don't generate documentation for the test code at all.

(Credit where due: this method was actually proposed in one of the JUnit articles, or in the documentation, or somewhere. I didn't think of it myself.)

-- BrettNeumeier

I found the package thing to be pretty straightforward. I created a second CodeWarrior project called JeraWorks_test, and add all my UnitTests to that. It has the main JeraWorks project as a subproject, so I'm always running my tests against the production JeraWorks.jar. For a class "com.jera.foo.Bar", I create a class "com.jera.foo.test.Bar_test" to test it. I use the "_test" suffix because I normally use ParcStyleBiCapitalization? for class names, so the "_" character shows up only in the names of test classes.

What I'm struggling with now is nesting and ordering of tests. How do I make sure that I run the tests in the right order, based on dependencies? I'm starting to suspect that the answer will be DoTheSimplestThingThatCouldPossiblyWork right now, and RefactorMercilessly as the test suite grows more complicated. -- JohnBrewer

Doesn't this package strategy make it difficult to ship only the production code? My intuition went with this structure also, as I didn't have to worry about maintaining parallel package hierarchies (i.e. each package has a subpackage .test). Unfortunately, this mixes tests up with production code and there is no good way in java to conditionally include packages (i.e. .test only include when some TEST switch is present).

-- RobertDiFalco

Robert, if you always put tests for a package in a .test subpackage, perhaps you can just do a "find . -type d -name test -exec rm -rf {} \;" from the base directory of the code tree, which will delete all directories named "test".

''If you use the code/test/classes structure: In the test version, include both 'code' and 'test' in the compilation path (classpath/sourcepath). For the release version, recursively delete files in the 'classes' directory and then recompile without 'test' in the compilation path. This is fairly easy to do when you are using some kind of build tool like Ant.

-- BrianSmith

Why not ShipTheTests? -- DanilSuits

My "Aha!" for the day is that how you order your UnitTests doesn't really matter. I thought I needed to run my low-level tests first. That way, a failure in one of those low-level classes would be caught in the class itself, instead of in a higher-level dependent class.

What I just realized is that because in XP you run UnitTests after each small change to the codebase, you can always isolate the cause of the test failure to the small change made since the last time you ran the UnitTests. So it looks like "YouArentGonnaNeedIt" applies to unit test ordering as well. -- JohnBrewer

We started with the different packages approach as described above but changed our strategy to simply placing test classes (as well as test helper classes that are often required for testing hooks in frameworks) into the same package as the production code. A simple naming convention allows us to easily identify and remove test classes and test helper classes for production release builds. The reason why we changed our strategy was the following: We wanted to keep test classes and test helper classes together (it doesn´t make a lot of sense to separate test classes but leave helper classes with the production code) and wanted to be able to test protected methods too (which are only visible inside the same package). -- PeterMaier

From: JavaUnit I'm clear on JUnit usage, but there's some debate at the start of my new project on the best way to organize JUnit tests for a LARGE project:

Two suggestions:

(a) Put all the test fixtures into a new package each time

com.some.project.subsystem
com.some.project.subsystem.tests

Disadvantages

Can only test interface; not package visible code.
Have to disentangle test packages when packaging up for customer
Test code is distant from source code

(b) Put all test fixtures in the same package, but in a different dir

$SRC/com/some/project/subsystem/ $TESTS/com/some/project/subsystem

CLASSPATH = $SRC:$TESTS

Advantages

Can test package visible methods, limited WhiteBox? testing.
Can package customer loads by simply working with $SRC and not $TESTS

Disadvantages

Might confuse IDEs, in particular VisualAge, our proposed solution.
Some people have expressed a concern that this isn't legal or moral Java practice. This is a view I don't hold myself but as someone without Java experience I cannot respond easily.

My project is veering towards (a) currently, but I feel that (b) is a better idea. Any comments?

-- David Kennedy, kennedyd@nortelnetworks.com

I would have to say (b). I've seen it a few places now, I've also tried both with my own small projects, and I think it's just plan easier to find the class I'm looking for. As for the disadvantages, I would do a test of the IDE in question, that should remove that fear. I've been using Java for a while now, and at first I felt like I was doing a BadThing, but when those directories start filling up with files, it's nice to have the tests and code separated. I also like to compile both trees into the same build directory (it also makes sure you are using that -d option which can help you detect dependencies in your classes). Now to be fair I should mention that the first disadvantage of (a) can be overcome with jdk1.3 reflection. In fact, AndreasHeilwagen has written a JUnit extension which does this for you. (http://www.extreme-java.de/) -- ErikMeade

Here is a possible package naming approach that I have seen, but that I haven't seen discussed here. It helps to know I have a VisualAge bias, and that I am not a big believer in package-visible features.

production code: com.company.package.name... test code: test.company.package.name... -OR- test code: com.company.test.package.name...

I like pushing the test package element as high up the naming structure as I can get it.

I implement the JavaUnit TestCase in a StaticNested class of the tested class.

See JunitWithForte and http://innertesters.netbeans.org/junit-style.html

Using an InnerClass has the advantage that the UnitTest is staring you in the face as you code. It also links the UnitTest's JavaDoc to the JavaDoc of the tested class.

What are the drawbacks?

-- JoeBowbeer

One disadvantage of putting tests in the same package as your production code can be package visibility. If you're just doing internal use only unit tests, this doesn't matter, and as noted, is more convenient!

But if you're trying to test the programmatic access to your tool by end users, this can give you a false sense of testing, since most end users won't (or shouldn't) be putting their code into your packagenames. I.e. you can merrily write your test in org.your.packagename and run it, and all's well. Then when a user tries to program to your public API's (from org.users.packagename), they discover that they don't have access to such-and-such important method - since it's package visibility, not public.

Just something to be aware of if you're testing the external access to functionality in your tool.

-- curcuru@apache.org

Anyone see a disadvantage to organizing testcases in a parallel structure like so:

$SRC/com/some/project/subsystem/ $TESTS/com/some/project/subsystem/tests/

This structure seems to me to provide a balance between giving tests access to the package and maintaining separability for easier cleanup.

--grimace78@yahoo.com

did you see the above discussion this looks very similar to the "b" discussed above. what is the difference between it, and suffixing "tests" at the end of the $TESTS... structure? - JonMadison

We have run into this exact discussion this morning on the above, grimace78@yahoo.com -- and I now see the subtle difference (for anyone else who didn't immediately notice it)--the package structure reflects a), but the directory structure reflects b). The only disadvantage i currently see is that the test classes are "distant" from the source classes. This may be annoying when working from the command line or from an IDE (like eclipse, where we'd have to have both directories exploded) - JonMadison

See UnitTestFileStructure, OrganizingAcceptanceTestCases