Unit Testing Mars Orbiters

Are there UnUnitTestableUnits? Would a Mars Orbiter be an example of such a unit?

A failure to recognize and correct an error in a transfer of information between the Mars Climate Orbiter spacecraft team in Colorado and the mission navigation team in California led to the loss of the spacecraft last week, preliminary findings by NASA's Jet Propulsion Laboratory internal peer review indicate... The peer review preliminary findings indicate that one team used English units (e.g., inches, feet and pounds) while the other used metric units for a key spacecraft operation. This information was critical to the maneuvers required to place the spacecraft in the proper Mars orbit. -- http://mars.jpl.nasa.gov/msp98/news/mco990930.html

Or, more specifically, ftp://ftp.hq.nasa.gov/pub/pao/reports/1999/MCO_report.pdf states on Page 17:

: [...] The calculation of the thruster performance is carried out both on-board the spacecraft and on ground support system computers. Mismodeling only occurred in the ground software.
: The Software Interface Specification (SIS), used to define the format of the AMD file,specifies the units associated with the impulse bit to be Newton-seconds (N-s). Newton-seconds are the proper units for impulse (Force x Time) for metric units. The AMD software installed on the spacecraft used metric units for the computation and was correct. In the case of the ground software, the impulse bit reported to the AMD file was in English units of pounds (force)-seconds (lbf-s) rather than the metric units specified. Subsequent processing of the impulse bit values from the AMD file by the navigation software underestimated the effect of the thruster firings on the spacecraft trajectory by a factor of 4.45 (1 pound force=4.45 Newtons).
: During the first four months of the MCO cruise flight, the ground software AMD files were not used in the orbit determination process because of multiple file format errors and incorrect quaternion (spacecraft attitude data) specifications. Instead, the operations navigation team used email from the contractor to notify them when an AMD desaturation event was occurring, and they attempted to model trajectory perturbations on their own, based on this timing information. Four months were used to fix the file problems and it was not until April 1999 that the operations team could begin using the correctly formatted files. Almost immediately (within a week) it became apparent that the files contained anomalous data that was indicating underestimation of the trajectory perturbations due to desaturation events. These file format and content errors early in the cruise mission contributed to the operations navigation team not being able to quickly detect and investigate what would become the root cause.

Given that it would have been possible for both teams to work with whatever units they wanted, as long as they'd agreed on a conversion function between them, it's obviously not the fault of the measurement system.

This was a human error, a tiny glitch in the communications between two programming teams. This happens all the time in software and the only scalable solution is proper testing.

I don't think NASA tested this part of their software.

The physics of modelling a satellite going into orbit aren't that difficult, so why not build a test harness which could simulate all the inputs the satellite would receive (inertial guidance etc), and could take the output from the guidance system (start firing this thruster now, stop firing now)

Since you're in a simulation and the "player" (satellite navigation system) is a computer you can ramp up the rate at which the clock ticks. In this way NASA could have tested that the Mars Climate Orbiter navigation system worked. The level of certainty would be based on how good the model was.

How do you build a good model? Testing of course. E.g. start small, check that when the nav system says fire this thruster, check the engine management subsystem throws the switch, check you can model the body in freespace firing it's thrusters and ending up where it should. Slowly add in more complexity (but keep running all the simple tests too) e.g. gravity, change in rotational inertia as fuel is consumed. Test it can hold an orbit round a model planet, check it can change orbits etc.

Testing this kind of thing would be challenging and fun, there would be parts you'd think were not testable. But if the solution is not testable by a computer, how can you hope to solve the problem with a computer?

This has been cross-posted on http://slashdot.org/article.pl?sid=99/09/30/1437217&mode=thread and http://www.c2.com/cgi/wiki?UnitTest.

What do people think would be hard to test about putting a satellite in orbit around another planet? -- OliBye

I'm not completely familiar with the Mars Orbiter problem save that it seemed that their was a conversion problem from unit X to unit Y.

The real problem is that the actual value submitted to the satellite may not have been incorrect at all. They may have been perfectly legal values well within the domain of the problem. The detail is that Mars happened to be in the way. Unfortunately, there isn't much you can do to isolate valid, yet inappropriate data.

If the orbiter had enough savvy to know what a "Martian Orbit" was and how to put itself into one, then it could have interpreted the responses, plotted their ultimate destination, and replied "I'm sorry Dave, but I cannot do that.". But, I don't believe that current probes are that autonomous.

Overall this is an issue with any information system. Being able to pull out extreme examples of bad data and "know" it. For example, an ATM machine. I'm sure that there are several people who have accidently tried to withdraw 400 or 4000 dollars. Perfectly valid numbers, just inappropriate. -- WillHartung?

Australian Law requires banks and similar organizations to report 'unusual' transactions, which is separate from the Report All Transactions above $10000 rule. Somehow I think the politicians are naive, given that the above problem exists.

To answer the $4000 question above, the ATM and Banking By Phone systems I know of always echo the intended amount back to the user before they commit. -- NickBishop

The aerospace industry normally does unit test aggressively, to the point that every single line of code in flight control software must be tested. My point here is that the aerospace industry has a notion of quality entirely different than what we normally associate with software development. The rigorous unit testing associated with XP may be "extreme" compared to what we are used to, but I do not believe it would seem unusual to the people that developed the flight control system for the Boeing 777, for example.

Since the Mars Orbiter was developed under NASA's Faster, Cheaper paradigm, it's possible that testing was not as thorough as it might have been on previous more expensive missions. It's also possible that unit testing was plenty rigorous, but still not capable of finding this particular bug, since the bug was in the interface between two different units. So it might be that this particular incident is an example of TestsCantProveTheAbsenceOfBugs, not of any particular sloppiness on the part of NASA and the aerospace contractors that built the Mars Orbiter. -- CurtisBartley

NASA, or somebody, did goof up: the interface between this type of system is documented in an ICD (Interface Control Document) - I don't see any problem in performing an automated test of compliance to such a document. -- NissimHadar

Yes, but consider WillHartung?'s point above: The numbers going through may have been perfectly valid from a range-checking point of view, but still wrong. There's not an easy way to check that the double coming out of one system is (for example) in feet, while the double going into the other system is supposed to be in meters. If you encapsulated measurements in a class that carried along units then you could in fact test this, or better yet catch the error at compile time. However, the simplest, most straight forward way to check for this particular kind of problem in the future may be good old fashion code inspection. -- CB

Surely we can do this? First calculate an orbit (by hand!), then use it to drive the system on one side of the interface, then write a test on the other side to check the data. Whoever is writing the check will know the original orbit and will be able to make tests based on the meaning of the data. I think that semantic tests are the hardest to write but are the most useful. -- JKB