Optimizing Unit Tests

I just finished a large application that had thousands of UnitTests. Unfortunately, it took hours to run all the tests, so immediate complete testing wasn't possible. Also, it was often the combination of development changes that caused tests to fail. But commitment to the process can help regain confidence in the extreme testing. -- KieranBarry?

They must run in a reasonable amount of time. I remember once on C3 our UnitTests were taking 25 minutes to run. When KentBeck came in he was appalled. He immediately sat down and trimmed and optimized our tests back down to one minute. The VcapsProject has been through this twice already in 6 months. As soon as the UnitTests take over 10 minutes to run you are in big big trouble and immediately need to optimize or remove redundant tests. And they are not just for releasing, you must be unafraid to just fire them off while you are working. -- DonWells

Experience report: In our project we have more than 900 UnitTests. In total they need between 4 and 5 minutes to finish. Recently we have started to split them into smaller parts, about three or four. We were also able to reduce the time needed for each of smaller packages down to 30 or 40 seconds.

Of help was that the UnitTesting tool we are using (CsUnit) displays a list of all successful tests along with the time needed to run each of them. This way we were able to identify tests which took too much time; we focused our optimization efforts on them, thus avoiding a waste of time. If you are not using CsUnit you can achieve a similar effect by running your UnitTesting tool under a profiler.

We, too, start working on the test performance when running all of them takes more than approximately 10 minutes. -- ManfredLange

10 minutes! That's way too long. The tests need to run every time you build. Your programmers are not going to stand for an extra 10 minutes added to every build. Our test suite completes in 30 seconds (and that's still too slow; I need to find the slow-poke and fix it). Also, extensive reporting can really slow you down. It often takes longer to time a test and print that information to a report than it took to run the test in the first place. -- PaulKrause

One thing that's quite nice is that you can set up ant targets that only run the tests in a particular package or group of packages. This makes it easier to work with large numbers of tests as you can just re-run the tests for the piece of code you're working on and it's dependent packages. Of course you have to remember to periodically run the ant target that tests the whole codebase. This is definitely riskier than testing everything after each change (as you may break something in another package without noticing it) but sometimes it's the least worst choice. -- AdewaleOshineye

On one project (doing partial-XP)we have just over 1,100 unit tests that take just under 10 minutes to run. We feel that is way too long. The ATs take 1.75 hours and we feel that is too long also but we are not as alarmed about ATs taking a long time. On another project (doing full-XP with a smaller group of people), they have 700 unit tests that run in about 1 and a half minutes! We feel like that project has really "made it." On the first project, we are planning a special effort starting next week (late-March 2003) to tune both the UTs and ATs. Our goal is to get the 1,100 UTs to run in around 3 minutes and to get the 290 ATs to run in around 45 minutes. This would make sure that the full Cruise Control process (with CVS log and update commands and full build, deploy war, stop/start Tomcat, etc...) would take just at an hour. I don't know if we will make those goals but here are the steps we will take.

Refactoring to make sure that no logic is happening inside of GUI classes (Java Swing). After awhile, it is easy to get lazy and allow some logic to get back into GUI classes.
Very aggressive use of mock objects.
Once And Only Once - make sure we are testing things only once. If we are building up a test and we do some setup work and there is a pre-assert that verifies the setup work, only do that in one test (assuming the setup is the same).
For ATs, if we find a situation where the functionality of an AT written earlier is covered in a later AT, we will break up the earlier AT into two, one with non-dup tests and the other with the dup'd tests. We will name the AT with the dup'd tests with a naming convention (xxxxx_DUP_AT) and exclude them from the Ant junit target. Then, we will have another target that doesn't exclude them. We will have Cruise Control run the normal target (with excluded DUPs) all the time and the other target with the DUPs included once a week. If after a few months, those DUP tests don't ever fail without the later test failing, we will remove those tests completely. We always can go back to them in CVS if needed.
Re-using setup's. For our ATs, since it is a full end-to-end test, we read a spreadsheet (provided by the customer) and write the data to the database, then run the test, then run a cleanup that knows how to surgically get our data out. We have started using the oneTimeSetup() idea (JUnit's TestSetup?) for ATs to provide a driver class that set's up a large amount of data (expensive) (named xxxxOneTimeSetupAT), and then runs multiple AT classes (we name them xxxxxxATimpl). We will increase the use of that pattern as it makes multi-minute setups and teardowns happen only once for a set of AT classes.
Profiling where necessary. If after the above steps we still see things that take a long time and we can't figure it out, we will profile the code. However, we will probably use AspectJ's debug example as the profiler as it will give a lot less noise than a true profiler. -- MikeCorum

CategoryTesting