Can You Have Too Many Unit Tests
Can You Have Too Many Unit TestsThis was one of the discussions at the TriangleXpUsersGroup. Since the participants in the discussion were self selecting it should come as no surprise that the unanimous consensus was "Yes".
There were two main ways in which people observed they had too many UnitTests.
The use of MockObjects was also discussed, but the approach needs to be employed carefully because it can end up tying your implementation so closely to your tests, that again, just reordering code within a class can result in a test needing to be rewritten.
There was some discussion around the use of code coverage metrics, but recognition that being driven by this resulted in writing unit tests for getters/setters just to get the extra coverage points.
If you can't use a specific code coverage metric, how do you know how many tests is enough? An approach recommended was write tests first, and then write just enough code to make the tests pass. This should result in you testing everything important. Another approach outlined was focusing on the interfaces between components and systems - a more black box approach to testing. And writing tests based on scenarios, which sounds closer to functional testing.
When you find your unit tests are taking 2+ hours to run (discouraging you from running them all the time) it's time to refactor or delete tests. The first candidate suggested was delete tests of getters/setters. But judicious use of an approach like MockObjects might also be needed to stub out remote system behavior. In particular if your tests are slow because of lots of database calls, this might be the only way.
Another cause of tests running slowly can be that you are subclassing tests where similar functionality was recognized. This will result in the tests in the superclass being run multiple times, and should be avoided.
In trying to identify unnecessary tests it was suggested that it would be useful to tell how many times a particular line or method had been executed. In general this would help where developer A wrote tests for component A and developer B wrote tests for component B, but component B actually exercises all the methods on component A, so you have duplicate tests. A tool call NoUnit http://nounit.sourceforge.net/ was brought up, although no-one had experience with it.
There was discussion about how many tests people generally had for a given amount of production code. One example given was 30,000 lines of production code with about 5,000 lines of test code comprising 500 tests. In this case, because of heavy dependence upon a database, the tests would run for over 2 hours. An example was given of a system that KenAuer worked on where 2000 tests ran in 12 minutes. -- AlexChapman
If your tests run slow because they're accessing the database, then you need MockObjects. Further, without using MockObjects for the database, I think you're doing AcceptanceTesting, rather than UnitTesting.
Testing accessors? No; that's pretty much a waste of time.
You don't have to test everything. You only have to TestEverythingThatCouldPossiblyBreak.
I haven't counted lines recently, but 5k of test for 30k of production code seems quite unbalanced to me: You don't have enough tests. You need UnitTests, not more AcceptanceTesting. I'd expect test and production code to be much closer to the same size. -- JeffGrigg
EditText of this page
(last edited June 28, 2004)
or FindPage with title or text search