Code Coverage

Naively, code coverage is the fraction or percentage of code paths executed by some test or test suite of a program. Normally measured by a tool which executes the test and logs the lines of code the test "touches" while running. At its most basic, every conditional statement creates a "branch" defining two unique code paths, and theoretically, both "branches" of each condition must be executed by the test suite in order for the developer to be certain that the code behaves correctly in each condition.

Code coverage is often used as a metric to determine the efficacy of UnitTests. Low coverage typically means that developers are not writing adequate unit tests; there are many code paths in the system which may possibly behave incorrectly, but because no unit test results in execution of that path, the developer (and other stakeholders) can't be confident in the correctness of that code either way. Code coverage can also identify SpeculativeCoding?; code written because it *might* be needed in some hypothetical situation, which the documented requirements (from which the tests were ostensibly developed) do not mention.

The ideal goal is 100% code coverage; the unit test suite executes each and every line of code in the codebase. In most non-trivial programs, this is infeasible:

Most languages have "boilerplate" code in source files that are necessary for compilation but are not represented in the compilation product. A naive coverage tool may compute the coverage percentage based on all non-whitespace lines of code.
Unit tests, as a rule, should not make "trivial" assertions (the term is subjective but typically includes assertions that cannot be false by definition, code that is impossible to change such that it would fail the assertion, etc). Code which can be proven correct by inspection and which cannot possibly be changed to be incorrect is typically not unit-tested.
Unit tests, as a rule, may not produce side effects (changes to state of production objects or persistent data outside the scope of the object under test) as a result of their execution. Thus, lines of code that require an external dependency that cannot be mocked or stubbed cannot be unit tested. Lines of code that call built-in file-reading/file-writing methods, such as in .NET's System.IO namespace, are classic examples.
Unit tests do not and cannot be run in the context of the full runtime of the final product, such as the hosting environment of a web application. As such, certain code intended to interface directly with that runtime often cannot be meaningfully unit-tested, and test coverage instead normally starts at the level of code that can be programmatically invoked (for instance, calling event handler methods directly instead of attempting to pass a message through a "mock" runtime that would result in the event being raised).

An important concept to understand is that CoverageIsNotExercise?. The fact that a line of code (or some part of it in the case of compound statements) is being executed does not necessarily mean that the expected result of that execution is properly asserted, nor that all expected permutations of all code paths are executed and assertions made. As a result, code may have passing unit tests with 100% coverage and still be full of defects, because the unit tests do not adequately "exercise" the code to assert it behaves correctly in all expected situations.

(Better descriptions welcomed; meanwhile see (eg) http://www.testingcraft.com/coverage.html.)

See "Introduction to Code Coverage" article by Lasse Koskela at http://www.javaranch.com/newsletter/200401/IntroToCodeCoverage.html

See: CodeCoverageTools, TddCodeCoverage ("What CodeCoverage do you see on TestDrivenDevelopment projects?")

Isn't it natural to use code coverage tools to see how well our Unit Tests have done? Yet today in Smalltalk, Unit Tests are very popular and code coverage seems almost unheard of. There is no equivalent to SUnit. What's going on? Is the current situation one we should be happy with?

Sigh. I've long harbored the suspicion that if you added up all of the things you'd have to do to avoid being "surely unprofessional", you would a) never sleep, and b) never ship anything to anyone other than the federal government. --DaveSmith

I changed the tone of the opening comment.

I think there is something startling and probably sub-optimal about this absence. Many people didn't use unit tests until they were advocated here. Similarly I suspect they don't use code coverage tools through ignorance rather then a conscious decision about what is professional/successful. -- DaveHarris

Not out of ignorance, no. I've long noticed how long it takes testers who are using code coverage tools to get anything tested. I've also noticed how quickly I can get code tested and out there. I've also noticed that their code, despite being so carefully tested, is not higher quality than mine. It could be that coverage analysis would make my tests better. But it could also be that the benefit of coverage analysis would not pay for the cost. I could have spent that time refactoring my code so that it is clean, talking to the customer so I understand the problem, and talking to the other programmers so I understand the system. These things, I think, have a very high payoff. A technique such as code coverage must have a very high payoff indeed to be more worthwhile.

Also, properly factored OOP code tends to be pretty simple and doesn't have that many branches. I won't claim that every line of code I write is like that (don't I wish!), but it turns out to be pretty easy to make tests that adequately exercise all the branches when there aren't that many branches to begin with. No fancy tools needed. -- WayneConrad

Maybe the ignorance is mostly mine.

I have been thinking about branches. To test a switch statement properly we need to ensure that the test data includes every case. If we replace a switch statement with a polymorphic message send, do we now have to ensure that the test data includes an instance of every class?

I think the answer is possibly "no", on the grounds that the message dispatch is managed automatically and presumably flawlessly, so doesn't need to be tested, where-as the switch code is written out by hand. However, it still bothers me a little that we have interactions that are "less tested" in the OO case. This may be a weakness of OO rather than a strength. We are relying on Liskov Substitution instead, so it depends on how confident we are that our classes are truly substitutable. -- DaveHarris

I think that's exactly right. -- WayneConrad

When you test a switch statement, you not only want to test the "manual dispatch" code. You also want to check that the code for every case is correct. Now if you use methods instead, the first part disappears because the compiler does the job for you. You still want to check that every case (that is, implementation of the method) is correct. Therefore you need to ensure that the test data includes enough instances to run code in every overriding of the method. So I don't see why this second version would be "less tested". Am I missing something? -- DanielBonniot

I don't know about professional, but there seems to be an obvious fit between the two tool categories. If they aren't being used together (I don't know that they aren't) it might be because the "obvious" turns out, in practice, not to be true.

Suggestions of coverage tools which integrate well with xUnit, even commercial ones, but preferably with an eval version readily available, would be welcome. I for one am both an enthusiastic JavaUnit user and eager to look into whether code coverage tools might improve the results I get from unit tests, for instance by pointing out portions of my code that don't get tested but where things might break. -- LaurentBossavit

My experience with CodeCoverage tools has been negative (using mostly IBM's PureCoverage?). It seemed mostly like the number was something about which the PointyHairedBoss could pat themselves on the back of it was over 95% and ruthlessly attack the developers if it was under. They didn't know how to read the report, just that one number. They had no clude about getters/setters and why they weren't tested. They didn't know that if a class isn't touched at all during your unit test suite in PureCoverage? it wasn't aggregated into the results (so you could literally get 100% by having no tests).

Overall, the CodeCoverage reports had little impact on whether the developers actually wrote good unit tests that actually covered a good majority of the relevant cases, only CodeReviews? could do that. I'd say don't bother with CodeCoverage, it's not worth your time.

[Descriptions of tools moved to CodeCoverageTools.]