Gui Testing Game Apps

The discussion on GuiTesting revolves a lot around using existing windows frameworks. This is all very well when building a windows or Web app, but my concern is how to UnitTest when you build your application out of primitive draw and blit functions.

This is not only a problem for GameApplications?, but for any application where there is no framework in place to help you out.

Any DirectX or OpenGL program will have this problem.

So far my approach has been to write tests for the ApplicationLayer, but when I hit the GUI I find it very hard to apply TestDrivenDevelopment. It's almost terrifying to write code without any tests beyond using your own eyes to see whether the surfaces, pixels and sprites are drawn as I want them.

Any insights here would be more than welcome.

You might want to make a (moderately simple) regression test framework along these lines: Use a blitter function to grab snapshots of (parts of) the screen. In the simplest case, snap the entire screen. Later on, you might want to divide up the screen into, say, 16 pieces.

When you have the screen in a "known good" state, snap a screenshot and save it to a file. Each time you run your test suite again, at that point in the test suite, snap a screenshot and compare the files. You might not need to do a diff -- often you can just make sure the number of bytes in a (compressed or vector) file is the same.

You can even do it test-first if you make your test cases simple enough that you can compose expected images "by hand" (maybe using an image-editing program like Adobe PhotoShop).

This is moderately simple? I can easily see this reaching a point where I would have a hard time defending the costs for this test suite. I have to admit I haven't tried your scheme myself, but when you have a ThreeDeeGameApp, with lots of things happening, like a RTS or a FirstPersonShooter it seems to me you would be better off using your eyes to detemine correct behaviour on the screen.

This is similar to the problem of comparing printouts from programs. Here are two solutions to this problem:

Save a text file version of the printout. Know which lines are subject to frequent change (such as datestamps) and ignore them. Compare the file sizes, number of files, number of lines in each file, and the stable lines.
Save a vector (or compressed) version of the printout. Verify that the file size is within a certain tolerance. For AutoCAD drawings, a good tolerance is 1 kilobyte plus 0.5% of the file size. YourMileageMayVary.

The thing to realise is that when unit testing your own classes, you do not need to test the behaviour of the graphics API. By assuming that the graphics API correctly translates API calls into pixels, you can unit test GUI code by checking that it makes the correct sequence of calls to the graphics API. This is easily done with MockObjects (or mock modules in a procedural API).

This approach makes testing much easier. Firstly, you don't have to capture and compare images. Secondly, you don't have to deal with the fact that some graphics APIs, such as OpenGL, don't guarantee the exact pixel values produced by an API call.

I have used this approach successfully with PyGame (isometric 3d game), PyGtk? (2d sprite game), JavaAWT and Java & SDL (2d sprite engine).

Using MockObjects makes sense to me. Now I only have to figure out how to make this work with the event driven stuff in my game loop. Would I have to make a Mock game loop of some kind then?