Doc Test

TimPeters, quoted from: http://groups.google.com/groups?selm=000001be679b%240b263e00%24a99e2299%40tim&output=gplain:

If you're like me, you've been using Python since '91, and every scheme you've come up with for testing basically sucked. Some observations:

Examples are priceless.
Examples that don't work are worse than worthless.
Examples that work, eventually turn into examples that don't.
Docstrings too often don't get written.
Docstrings that do get written rarely contain those priceless examples.
The rare written docstrings that do contain priceless examples eventually turn into rare docstrings with examples that don't work. I think this one may follow from the above ...
Module unit tests too often don't get written.
The best Python testing gets done in interactive mode, esp. trying endcases that almost never make it into a test suite because they're so tedious to code up.
The endcases that were tested interactively -- but never coded up -- also fail to work after time.

About a month ago, I tried something new: take those priceless interactive testing sessions, paste them into docstrings, and write a module to do all the rest by magic (find the examples, execute them, and verify they still work exactly as advertised).

Wow -- it turned out to be the only scheme I've ever really liked, and I like it a lot! With almost no extra work beyond what I was doing before, tests and docstrings get written now, and I'm certain the docstring examples are accurate. It's also caught an amazing number of formerly-insidious buglets in my modules, from accidental changes in endcase behavior, to hasty but inconsistent renamings.

doctest.py is attached, and it's the whole banana. Give it a try, if you like. After another month or so of ignoring your groundless complaints, I'll upload it to the python.org FTP contrib site. Note that it serves as an example of its own use, albeit an artificially strained example.

DocTest and PythonUnit are part of PythonLanguage's standard library.

This is solid gold.

DocTest should be integrated into the interpreter environment to encourage TestDrivenDevelopment.

Brief Tutorial

Python's interactive prompt is an easy way to test code that you've written. With DocTest, you can copy those interactive sessions and paste them into your source code as docstrings.

First, here's some source code to test:

 HelloWorld.py
 def Hello():
        print "Hello, World"

Our interactive session looks like this:

 >>> import HelloWorld
 >>> HelloWorld.Hello()
 "Hello, World"

Now, Cut and paste the testing session into your source code as a docstring to the function, like so:

 HelloWorld.py
 def Hello():
        """
         >>> import HelloWorld
         >>> Hello()
        "Hello, World"
        """
        print "Hello, World"

and add some standard test setup code to the bottom of the module:

 def _test():
        import doctest, HelloWorld   #The module name 
        return doctest.testmod(HelloWorld)

 if __name__ == "__main__":
        _test()

or, if you want to have exactly the same boilerplate code in each module (without having to change the text "HelloWorld" twice each time you paste it...)

 if __name__ == "__main__":
        import doctest, sys
        doctest.testmod(sys.modules[__name__])

Now, when you run the module from the command line, doctest will run the test...

 $ python HelloWorld.py

Adding a -v option will show verbose output.

Try it, it's fun.

Make sure to read the source code for a much more extensive and entertaining tutorial by TimPeters.

Note: This order is reversed from the recommended path in TestDrivenDevelopment, where the principle is to code the test first, confirm that it fails, and then fix the code to work as documented.

Discussion of DocTest and its relation to other TestingFrameworks

Just pasting in interactive testing sessions seems a very easy and intuitive way of adding tests. Maybe this could be implemented as a part of the UnitTest framework?

(personally, I've switched to doctest.py for all sorts of unit testing. Instead of running it on the module itself, I use it to verify that the test scripts for that module behave as documented. UseTheSimplestToolThatCouldPossiblyWork?. If I wanted/needed more elaborate stuff, I wouldn't be using Python in the first place ;-)

It seems to me that DocTest, of which most intended usage is code first and run it to cut/paste the result into doc strings that will verify the code thereafter, is rather geared toward CodeAndTest than TestAndCode?(or CodeUnitTestFirst); CodeAndTest is quite opposite to one of the XP principles. But It truly is a very convenient tool when you want examples and tests at one stone after you've done substantial coding.

Not at all: nothing stops you from writing the tests before you write the code. I do it all the time. Of course, if you don't know what your code will do, it's a bit harder -- but that's hardly doctest's fault. And you can always run doctest to see what happens. If the output looks right, you can turn it into a test case in seconds /F

Yes, you're definitely right. What I meant was it was possible to do either CodeAndTest or TestAndCode?, but it seemed easier to do CodeAndTest with doctest, as for me. In fact, I've been just tinkering with doctest and haven't put it into heavy practice yet, so I can't say a word of critique on it; It wasn't a criticism at all, actually -- just my first impression.

BTW, what do you do, Mr. FredrikLundh when the unit (or function) is supposed to spit out a few dozens lines of result? Make a textfile of the supposed result and put in file-to-result comparison code into the doc string? In the case, how do you maintain two separate(test file and text file) files altogether w.r.t. the location? I would happily appreciate your comments.

Contributors: a quote from TimPeters, FredrikLundh, SeanOleary, AnonymousDonors

DocTest for other ProgrammingLanguages

RubyLanguage

http://clintonforbes.blogspot.com/2007/08/doctest-for-ruby-and-rails.html

Doctest/JS for JavaScript

http://openjsan.org/doc/i/ia/ianb/doctestjs/

Contrast with DocumentationCannotCompileRunOrDoUsefulWork