Unit Testing Numerical Methods

If you are developing libraries such as linear regression, many numerical methods, or implementing a NeuralNetwork, it may be impossible to generate a test case - because you don't know what the answer should be for your test data.

How do you proceed?

UnsupervisedLearning?; KohonenSelfOrganizingMaps?

But you can always create toy data, at least, even if you have no confidence that it is a good reflection of real world data, so I don't accept that it is ever "impossible to generate a test case", although it is often impossible to generate ideal test cases. There's a difference.

If you don't know what you want, then you should be happy with whatever you get. ;-)

Wouldn't you want to determine that the NeuralNetwork is actually providing refined results rather than providing the same result or even degrading? If one doesn't know how to define "better results" and what the program needs to generate better results, one might as well code return(NULL) and be done with it.

You should have test cases, though. Your algorithm shouldn't be the only way to do it; there's probably something slower that's been used before. It may be something like solving a differential equation numerically, instead of the approximation scheme you're working on. So you might not have one true answer you trust, but if you do it a few different ways, and the answers agree, you're in a good position.