Design By Contract Vs Programmer Tests

Let's start the trial by presuming DesignByContract guilty. If so, DbC will cause these bad effects:

harder to refactor or add features
harder to keep lean
harder to merge duplication

That sounds reasonable. Let's appeal to authority: http://onestepback.org/index.cgi/Tech/Programming/DbcAndTesting.html (an updated version of the original article).

Nope. This authority disagrees, so I won't appeal to him. He says "Unit tests focus on verifying that a single class functions properly. Problems between modules are not generally uncovered by unit tests. Extreme programming uses functional (or acceptance) tests to verify end to end correctness in their programs. But since functional tests only check the final result, they give little feedback about what is going on inside the program."

This is wrong because it smears "UnitTest" and "ProgrammerTest". Not the author's fault; he's following the most authoritative definition of UnitTest he can find. Then he's replacing "unit" with "class". Then he says that because BlackBox testing can't see inside the box, it also can't test boundaries.

ProgrammerTests test units of arbitrary scope; methods, classes, and class clusters. Including their interfaces, and including challenges to their contracts. If you think of a precondition, invariant, or postcondition, write it. Analyze your flow as carefully as DbC-ers do. But migrating the result out into tests just might make the code inside easier to refactor.

So, because this author was wrong about an item not on my bullet list, my bullet list must be right. Case closed. ;-)

It may be useful to reread the Post Script section of the referenced article. Here the author writes "I find that I still think of interfaces in terms of contracts with pre/post conditions, but instead using Eiffel-like assertions to define the pre/post-condition assertions, I now use unit tests to drive and define the shape of the software. Although the unit tests have a different focus than assertions (as noted in the article), I find that they are quite suitable for my needs and rarely find it necessary to supplement the coce with any additional pre/post-condition assertions." To me, it appears the author validates the usefulness of unit/programmer tests.

I found that to be a very nice tutorial about DbC, but the guy distorted much of XP to help DbC win.

I don't think the term "win" is appropriate here. The conclusion reads:

: "We see that although unit tests and Design by Contract assertions have a lot of overlap, but each has a different focus and different strong points. DbC preconditions are useful for checking inter-module correctness. Unit testing and DbC postconditions validate individual module behavior, with unit testing focusing on specific situations an DbC postconditions focusing on general behavior."

I find this pretty unbiased. There is a real difference between UT and DbC, and that's presented in the paper.

"...migrating the result out into tests..."

Hm. I don't like the migration of conditions to tests. For postconditions this is ok, in effect that's what tests usually verify. But you can't easily migrate precondition checks to test code, because you would have to insert a test dummy DX in every place where a module X is used. That's not very feasible.

-- RobertKlemme?

Not every place, only in the specific tests that you need a dummy DX. That is the approach taken by the MockObjects testing technique.

http://people.cs.uchicago.edu/~songyanf/research/contract_paper/other/EiffelUnit%20-%20Design%20by%20Contract%20and%20Unit%20Testing.htm seems to be from the same author. -- PiergiulianoBossi

I've found that TestFirstProgramming with ProgrammerTests is better than DesignByContract at helping me understand what a class does at a later date. This is because ProgrammerTests specify aspects of the class' holistic behaviour, rather than pre/post conditions on individual methods. Aspects of a class' behaviour often involve the interplay of more than one method and ProgrammerTests specify the relationships between methods. (Obviously, good names for tests is required). If you want to do the same with contracts you have to add additional instance variables and query methods to a class to keep track of which methods have been called. This pollutes the interface of the class and makes it harder to understand.

How does a batch of asserts on values help you understand a class?

A ProgrammerTest is not just a batch of asserts. It describes an aspect of a class' functionality, specifies that aspect, and then exercises the specification. At least, it should, otherwise it's not a very good ProgrammerTest. If one can't read one's ProgrammerTests that way, then there's something wrong with one's test-first process.

The tests I see are generally line after line of asserts. What is different about your tests?

Here are some of the guidelines I've discovered or learnt from other people. I probably follow more conventions, but it's late and I can't think of them...

A test should have a good name that describes what the test is specifying. I follow the convention that the name of each test (ignoring the initial "test" prefix) completes a sentence that begins "A Foo..." where Foo is the name of the class being tested. So, a fixed-size Stack might have tests "testIsCreatedEmpty", "testCanStoreUpToTheNumberOfElementsPassedToTheConstructor", "returnsStoredElementsInLIFOOrder", "throwsUnderflowExceptionIfPoppedWhenEmpty", "throwsOverflowExceptionIfElementPushedWhenFull", and so on. If, like DesignByContract specifications, my tests were related to individual methods instead of aspects of functionality, the tests would be named "testPush", "testPop", "testIsEmpty", "testIsFull" and so on, which doesn't give me any useful information.
A test should try to express a universal specification, not an isolated example. I try to write assertions as for-all or exists predicates as much as is possible. I use variable names to store expected values and give those variable names that express the universality that I am trying to specify. E.g. anyFilename, allPositiveValues, etc.
Expected values should be self describing so that they create good error messages. E.g. String anyFilename = "any filename";
All tests should follow the same format, so they are easy to skim read: set up the test fixture and preconditions, specify expectations (esp. if testing with mock objects), exercise the code under test, assert post conditions, verify outgoing calls were made correctly (if testing with mock objects).

I want my unit tests to express the same things as DesignByContract, but I want the assertions grouped in a way that helps me have a holistic understanding of what the object does, not what each method of the object does individually.

Someone else wrote: The tests should do something, exercise some functionality, then assert that the outcome was what was expected. It should be similar to how the client would actually use the class. Tests also serve as sample code on how to use the class as it was intended.

Do you have a relatively complex example? I've never seen any tests that I could deduce the original user stories from. And test code, imho, usually makes very poor example code, because it doesn't execute code how an application would. Again, a test is just a bunch of asserts.

ProgrammerTests aren't directly related to user stories, AcceptanceTests (aka CustomerTests) are. ProgrammerTests are used by the programmer to specify units of code. I don't understand the complaint that ProgrammerTests do not execute code how an application would. Why don't your ProgrammerTests execute code in a way that is useful to you? Also, DesignByContract specifications don't execute code at all.

Do you have an example? User stories define your engineering tasks. In your tasks you use tests to drive your design. Therefor you must via the tests be able to understand the original stories. Otherwise what is there to understand from tests? Tests are a series of asserts, that's not how I usually write an application.

First, user stories define user requirements. From a bunch of stories I (in my job I usually work alone because my company is very small) work out what tasks need to be done to implement the next iteration's stories. Then I pick a task and do multiple test/code/refactor phases to implement the task, where a test is a specification of some new behaviour that a class must implement.

Yes, a test is (if you're being pedantic) just a bunch of asserts. You could just as easily say that a contract is nothing but a bunch of asserts. But that's ignoring how they are used, and how they should be written to support that use. I write my tests in such a way that they read as more than just a series of unrelated boolean expressions. Similarly, in EiffelLanguage, I write contracts so that they read as more than a series of unrelated boolean expressions. However, in a programmer test those boolean expressions are organized in a way that helps me understand what the class does. In a contract, those boolean expressions are organized in a way that helps me understand what can/cannot be passed into a method and what is/is not true after it has been called, but I have found it hard to use contracts to help me understand how calls to those methods are interrelated.

In test-first programming, a test framework is, fundamentally, used as a language to express the specifications of code that you will then write. What are you doing that makes it hard to read your tests in a way that is useful to you? Why do you write your tests that way?

Again, do you have an example? My tests are useful, for testing. I have not seen any tests that are useful enough for understanding classes. You would have to read all the tests and build up the model in your head of what they mean. That's not as effective as documentation that gives you a positive descriptive statement of how a class works, its purpose, its responsibilities, and how it all works together. If you have a counter example that would be useful.

It sounds like you don't use your tests in a test-driven process, in which case, testing is all you want them to do. But how do you name them? Are those names useful to you? If not, why not?

If you are using tests in a test-first process, why don't your tests describe the purpose and responsibilities of your class? What is it about your test code that makes it difficult to use them to understand the purpose and responsibilities of the class under test?

I can put up examples, but they'll be lengthy and I don't want to waste space on the wiki. Anyway, it'll have to wait until I'm back at work.

I do a test-driven process. Do you think your tests describe the purposes and responsibilities of your class sufficiently to other people? I have not seen it. That is why concrete examples are far from a waste of space. The space is usually wasted by philosophical statements without concrete proof. That unit tests suffice for documentation and requirements is a philosophical statement.

I don't think that unit tests suffice for all documentation. They are no good for tutorial type documentation and obviously they don't work as an overview because they operate on the level of individual units or small groups of units. The same is true for specifications such as contracts. A contract is no better as documentation and, in practice, I've found it less useful than a unit test. Obviously, considering the way I work, my unit tests are good for describing my classes to me because I an the only programmer. On projects in which I've worked with others, I've found that unit tests are helpful in helping people understand others' code when they understand the test/code style of the team, when the understand the system architecture, and when they know the codebase. This is why pairing is so valuable. I expect that unit tests will not be very helpful if one has to work on a completely foreign codebase; in this case, tutorials and architectural overviews will help you get up to speed. Of course, I don't think that unit tests are good for describing requirements for the reasons I've already stated.

As I've said, I'll post an example when I can. Could you answer some of my questions in the meantime?

What in particular? I think I can agree with your above statement. My concern is how the users of the code can learn about the class, not the implementors, so we are talking about different audiences.

Picking two from the thread:

How do you name [your test cases]? Are those names useful to you? If not, why not?
What is it about your test code that makes it difficult to use them to understand the purpose and responsibilities of the class under test?

The names are useful to me. There isn't enough information in mere names to reconstruct everything you need to know about class. I can imagine trying to program the win32 interface by looking at about 100000 unit tests.

What? I never said that the names alone are enough.

You have names, values, and asserts. From that what can you figure out that you find particularly useful? Depending on the programmer I may find some constructs that weren't evident from the class, but that depends a great deal on the programmer and the amount of their effort. I worked on 100MB of lisp tests one. I wouldn't have liked to learn lisp from those tests, but I did learn some cool weird things you could do.

Names, values and asserts are all that DbC contracts give you. What is it about contracts that makes them more useful to you than ProgrammerTests, and why?

They are useful because the occur at runtime. They aren't more useful, they are just different and have different purposes. A unit test isn't a system test. I find a lot of problems occur at the system level. DBC finds these. Unit tests don't, because by definition they are only applied at the unit level. Unit tests can be considered a way of triggering DBC an addition to testing other functionality. Neither is all that useful as a form of documentation, imho.

CategoryTesting