Extend The Java Test Framework For Nondeterminism

This is part of JavaUnitTestChallengeSolved. -- DonWells

---

First let's write a test for our new functionality. We create a test that passes and fails alternately.

 public class InconclusiveTest? extends Test {
    static int count = 0;

    public void runTest(){
        should((count++ % 2) == 0, "this should be even");}
 }

The first thing I tried here which seemed like the simplest was to create a type of test that would run three times in a row and compare the three results. If the results were the same that was the result of the test. If the results were different then the test would report back inconclusive. I got this all coded up and found that while the tests I had been running failed almost all the time, they started to reverse that and mostly passed. So I threw the triple run code away. That would not work for a test that mostly gave false pass results.

I decided then to do what I had talked about some time ago. I extended test such that I could specify how many times a test needed to be run. If the test passed all of those times it was a pass. But if it failed once it was considered a failure. This is the new code.

    static Class testClasses [] = {GoodTest?.class, 
                                   FailTest?.class, 
                                   AbortTest?.class, 
                                   DeadLockTest?.class, 
                                   InconclusiveTest?.class};        

    public void initializeTests() {
    tests = new Test[testClasses.length];
    for (int each = 0; each < tests.length; each++){
        newTest(each);};}

void newTest(int anIndex){
    try {
        tests[anIndex] = (Test) testClasses[anIndex].newInstance();}
    catch (Exception exception) {};}

void runTests() {
    initializeTests();
    for (int each = 0; each < tests.length; each++){
        runTest(each);
        testFrame.showResults();}}

void runTest(int anIndex){
    for (int each = 0; each < tests[anIndex].numberOfTimesToRun(); each++){
        runOneTest(anIndex);
        if (!tests[anIndex].success) return;};}

void runOneTest(int anIndex) {
    newTest(anIndex);
    tests[anIndex].setResultToBeDeadlock();
    tests[anIndex].setUp();
    tests[anIndex].start();
    waitTillTestFinishesOrDeadlock(anIndex);
    tests[anIndex].stop();
    tests[anIndex].tearDown();}

We need to add a default for numberOfTimesToRun() to the Test class:

    public int numberOfTimesToRun(){
        return 1;}

We can now override this method to run any test as many times as we need to ensure that it has ample opportunity to fail. For our test, we override the method and return a 3.

This is working. Let's go back to our challenge tests. I am going to run the twenty threads test 5 times because it fails readily. TenAndOne? and OneAndTen? I am going to run 250 times because they don't fail so easily. I can run them now.

Oddly enough I catch the TenAndOne? test failing now. I try again.

Yet another result. I could probably add more iterations to TenAndOne? so that it always fails. But, experimenting a little I find the number to be about 2,000 which is excessive when I have other tests which are covering that particular bug anyway. So I will leave it at 250 till there is a reason to increase it.

---

CategoryTesting CategoryJava