Unit Testing Grey Areas

Here's a problem I came up with while trying to work out some UnitTests. In a nutshell, the problem is: should I put in UnitTests to verify the presence of known bugs or "grey areas" as well as customer-requested features?

Here's some (very trimmed down) Java/JUnit code to illustrate the problem:

  public class Condition 
  {
    private String symbol;
    public Condition(String symbol) 
    {
      this.symbol = symbol;
    }

    public boolean evaluate(String lhs, String rhs)
    {
      if (symbol.equals(">"))  return lhs.compareTo(rhs) > 0;
      if (symbol.equals("<"))  return lhs.compareTo(rhs) < 0;
      return false;
    }
  }

  public class ConditionTest? extends TestCase
  {
    public void testEvalLess()
    {
      Condition less = new Condition("<");

      // test the behaviour with text
      assert(!less.evaluate("aardvark","aardvark"));
      assert(less.evaluate("aardvark","zygote"));
      assert(!less.evaluate("zygote","aardvark"));

      // test the behaviour with "simple" numbers
      assert(!less.evaluate("6","6"));
      assert(less.evaluate("6","7"));
      assert(!less.evaluate("7","6"));

      // test the behaviour with "difficult" numbers
      assert(!less.evaluate("6","11"));
      assert(less.evaluate("11","6"));
    }
  }

The code is from an experimental knowledge-based system I'm putting together.

The problem is with the last section of the tests. The code at the top was produced using "do the simplest thing which could possibly work". At the moment there is no clear business requirement for "difficult" numbers. There are both string and numeric values stored in the knowledge base, but (luckily or unluckily) each variable always has the same number of digits. So the code at the top does the job. Based on "you aren't gonna need it", I haven't added any code to the Condition class to handle "difficult" numbers - it would make it a lot more complex and less efficient, even though my gut feeling is that someone will want it as soon as they put in a numeric value with a wider range.

However, that's just background. The issue is with the UnitTests. I want to be able to say that "all tests pass", so I have several choices:

• ignore the bahaviour with difficult numbers, and only test the easy cases
• put tests in which show the "proper" relationship, but commented out (as they would fail)
• put tests in which test for the current, unexpected, bahaviour (as shown above)

What is your opinion on this? Have I followed the XP guidelines correctly?

-- FrankCarver

---

My opinion is that your class does exactly what it's supposed to by not supporting those difficult numbers, so they should not be in the UnitTests. You should only test what a class is supposed to do, not what you think it might be called on to do some time in the future.

Well, on one hand, yes. But surely part of the importance of UnitTesting is to test the error cases as well as the success cases. It's the error cases which have caused all the "expensive" live-system faults that I've ever had to deal with. I think the main importance of UnitTests is that thought is put in early about all the possible conditions which may occur. Thoroughly testing small, simple classes is the best way to be sure of this.

Are you advocating that I just ignore the peculiar, non-intuitive behaviour in the case of comparing 6 and 11 above, wash my hands of it and leave it up to a future developer to sort out. That's so extreme it sounds selfish. --FC

---

If the intention of the code is that "11" test as less than "6", the test should show just that:

      assert(!less.evaluate("6","11"));
      assert(less.evaluate("11","6"));

If later the definition is changed to work as one would expect, the tests are edited to reflect the more intuitive result:

      assert(less.evaluate("6","11"));
      assert(!less.evaluate("11","6"));

This is done for at least two reasons: first, the "extra" tests serve as clear notice that the code does something non-intuitive. Second, a conscious decision was made to have it work that way. Since we decided to do it, we should decide to test it. --RonJeffries

Also, I would make "less" an instance variable and initialize it in setUp(). Then I would have three tests instead of one- testString(), testOneDigitNumber() and testLongNumber(). In the latter you could add a comment explaining the seemingly irrational behavior. OTOH, if you aren't going to sleep well at night, make the code work "right" for numbers. --KentBeck

Instead of putting in a comment explaining the non-intuitive behavior of testLongNumber(), rename that test method to something like testNonIntuitiveLongNumberBehavior. I find that I often have extra-ordinarily long test method names. A few days later, when I've completely forgotten what I was doing, I can go back to the SelfDocumentingCode in the test and, hopefully, remember what I was going on. --MarkAddleman

---

Methinks the answer doth present itself quite clearly in UnitTestAsTickler. :-) You should really read the original for proper context, but let me quote a bit of it that I think is the essence of an answer to your question.

They are speaking here about writing a test for something that they know they won't do until October, but they also don't want to forget about it come October. The proposed solution is

The first thing we did, of course, was to write a test that showed whether the bug existed. We always do that.

So now, we just put that test in the system, inside a guard clause that doesn't actually start checking until, say, October.

Is that a satisfactory answer to your question?