Design By Contract Assertions Vs Unit Tests Vs Types

Tests and Assertions are similar in that

Both are part of specification.
Both serve as "executable comments" that must be consistent for compiles to go through.
They are helpful during design, but in different ways. See below.

Tests and Assertions are different in that (overstating a little)

Usually, assertions have a smaller span, because an assertion in a method must depend only on the object and method parameters, otherwise you get too many dependencies.
Pre-conditions may have a wider span. For example, a routine may assume its argument is non-null. How would you write a test for that? By using a pre-condition, you effectively add a test to every point of call - even in classes that didn't exist at the time.
Assertions belong to the class, so can document class properties and assumptions that are not public. Unit tests can only check externally visible properties.
Assertions are part of executing code, so the amount of computation you can do in an assertion is limited. You can eliminate assertions from product code in the name of speed, but then you sacrifice graceful error handling.
Tests can span multiple objects; indeed tests can be carefully organized to reflect module dependencies, because tests are a sort of mini application for your code.
Assertions (Pre and Post Conditions) are a way to specify state changes in methods. Thus you get more detailed interfaces than simple type signatures.
Tests help capture requirements and expose object dependencies.
Tests only test a small number of cases. Assertions work with live data, which can be more varied and more realistic. They purport to cover infinitely many cases; they establish universal truths rather then point truths.
Assertions are visible to the compiler, so can help it produce more efficient code. For example, a pre-condition like "ptr != null" means the compiler can omit that test.

Assertions are also related to types (the notion of type here includes const modifier)

Many kinds of "consistency" and "integrity" checks in dynamic languages implement forms of type checking. Lisp often uses predicates like nil-p and cons-p to implement type checking.

''(actually, Lisp rarely does this for integrity or consistency. Normally this is more to get out of a recursion down a list, or some situation where you want to tell that you're at the end of a list. Assertions to check types in Lisp are considered "poor form"). Lisp programmers don't worry too much about checking types because all lisp objects know their own types, and the environment will raise a condition if you do something which is disallowed (say (+ 2 nil)).''

In statically type OO languages, type information is "lost" because of contravariant typing of method arguments. Type recovery mechanisms like casts then behave like assertions (especially in Java).
A type system can validate that inputs are, in some sense, whole values.
"A type checker is a poor man's program prover." Type systems are crippled by the requirement that they be verifiable automatically. Often programs have interesting properties which are true, but whose proof defeats a type-checker (perhaps because of non-locality).
In languages with LinearTypes, the type system can do some kinds of reference counting for you.
Languages with type inference can infer static types by looking at the senders of a method, constants in its body, and the methods it calls. RalphJohnson built one such system for Smalltalk. Many moons ago, I did a little project with it, showing how you can use type analysis information to allocate objects on stack instead of the heap, inline them, etc.
Simple theorem provers can do even more checking. There is a project at SRC called ExtendedStaticChecking that is exploring this. I imagine that you could infer some of the assertions for a method dependening on those it calls and its callers, and ensure that callers state logically implies callees precondition.
Invariants are essentially types; or the other way round, types are the invariants for results and parameters over all possible executions.

Comments?

-- AamodSane, DaveHarris

I added a few. I don't think anything in UnitTests quite corresponds to pre-conditions.

"The amount of computation you can do in an assertion is limited" - I don't follow this. If you replace an assertion with a unit test, how does that give you good error handling in live code? The unit test isn't even present in live code. -- DaveHarris

A: Each UnitTest is executed only once (every time you run the tests). But assertions are run every time that piece of code is executed, even if you're not "running tests." So computationally expensive assertions in frequently used modules can really slow a program down. Also, the increase in physical program size can slow execution too, due to swapping. -- JeffGrigg

You can always mix them: let the unit test define the stimulus and the assertions define the expected results. The problem is that you can't then run the unit tests on the production build. This is solved using the DUT vs reference approach (i.e. run both versions of the code with the same stimulus and compare the outputs). Placing the assertions in the code is a valuable debug aid and does not reduce the ability to use unit tests. (AcceptanceTests must, of course, remain as an independent source of expected results). -- DaveWhipp

It will be said good unit tests are a substitute for type checking and design-by-contract. This presupposes facts not yet in evidence by saying the unit tests are good. If they are not good then the whole pyramid of assumptions based on good unit tests will collapse. My experience is UnitTests are not even as good as "normal" code and they don't get maintained by new programmers as they take over the code. It's clear some people are capable of very high quality with unit tests, but on average I bet this isn't so, which means other mechanisms are needed as well. Unit tests can't be made the answer to every issue of quality and design.

Can you describe a method that if not used or "maintained by new programmers" that will be successful? I believe every method assumes its use for it to be successful.

Compiler based type checking is not maintained by developers. DBC is not maintained by developers.

Consistent use of types is maintained by developers. I have seen too many C/C++ systems where everything is passed as an int and with random changes between signed and unsigned; or char, short, int, and long; or that do not use const. I say this not as a purist, but as one who has had to track through code trying to figure out why it doesn't work under some scenarios. I can't say for certain, but I would also guess that design by contract would also fail if it were ignored by developers.

Yes, it is easy to get into a situation where UnitTests are not used by new developers. One important principle is that it should be more difficult to not run the tests than to run them. For example, I like to have a makefile where the "build exe" step has a dependency on the "unit tests passed" rule. The flow is that the makefile first creates .o files (the compiler may catch its errors at this point), and then links these with the unit tests (and dies if it fails), and then runs the unit tests (and dies if any fail). The average developer can then treat uit-test errors in the same way as any other built-time error.

The more difficult issue is to ensure the tests are maintained as change continues. I don't think there's a technical solution to this: its a question of culture. -- DaveWhipp

I think we should think about static typing (including const checking, as I added above) and asserts as specifying the conditions that should be always true in our system. Static typing and ExtendedStaticChecking are evaluated at comile time, normal asserts are evaluated at run time. The static typing rules come with the compiler and are inflexible, asserts and ExtendedStaticChecking are custom-built. We can consider here also programs for style validation and advanced error detection like lint family.

UnitTests on the other hand verify conditions that are true in each particular context of the test and work on a different level of granularity. They cannot be used for static verification as they depend heavily on the context that is not known yet.

I believe all these categories of program verifications can be generalized in a coherent and adaptive framework. Java will incorporate asserts in JDK 1.4, and the language construct can be used also for ExtendedStaticChecking - this would be definitely a step in the right direction. Another example would be if the TestingFramework 'notices' certain things are always true during the testing, and tries to assert them speculatively. -- NikolaToshev

DaveHarris wrote: I don't think anything in UnitTests quite corresponds to pre-conditions.

MockObjects are used to test preconditions in unit tests.

Could you provide more detail? It seems to me that MockObjects can verify pre-conditions for the objects you collaborate with, but not for your objects. Verifying preconditions for a method in your code should mean checking if certain conditions are always fulfilled at invocation of this method. Correct me if I am wrong.

See DoesUnitTestingMakeStaticTypingLessUseful (which might be a good home for some of the content here (?))

CategoryComparisons