They Write The Right Stuff

FastCompany?'s article "They Write the Right Stuff" [http://web.archive.org/web/20050830190246/www.fastcompany.com/magazine/06/writestuff.html] describes the "on board shuttle group" and their main software product (420,000 lines of code). This group was (in December 1996) one of four organizations rated at the SEI CapabilityMaturityModel level 5. The article contrasts the shuttle group with ordinary software development, and lists four key points of the shuttle software process:

This group is an interesting example of BigDesignUpFront which works, although it is rather expensive. (The specifications are over 40,000 pages. The group has 260 people and an annual budget of $35 million.)

According to the article, they have a motto, it is: 

 TheSoonerYouFallBehindTheMoreTimeYouWillHaveToCatchUp

Let's just calculate what this piece of software cost: 20 years * 35 million = 700 million. Divide that by 420000 lines of code. Every line of code has cost NASA 1600 bucks. Yikes! This car is definitely a Ferrari. Our current product is at about 35$ a line (just educated guessing). (Gee, that's pretty low, unless you're doing cookie-cutter code. Typical values are around $100 (NoCitation?).)

This is a great example. When trying to get people 'round here to move on from waterfall-like practices, I've been saying over and over again: it's not that these ideas don't work, it's that they have to be made to work, and it costs a fortune. These numbers will be going into my next presentation

Please think carefully about what you are advocating. If you or your spouse or kid is being shot up there, with no air to breathe in case anything goes wrong, just how, exactly, would you prefer to coordinate the dozen subcontractors hired to construct the software? -- AlistairCockburn

What I'm saying is that if you want that kind of reliability it is going to cost you. Certainly, this expense is justified when human lives are at stake. But if you buy a word processor that today costs 100$, would you be ready to shell out roughly five grand? The article made it look as if these guys had found the philosophers stone and the rest of the industry were just reckless cowboy coders. -- ThomasMaeder (the anonymous poster)

That's right. what I'm suggesting is that the practices that are appropriate for the kinds of risk and level of funding that a space vehicle project has are inappropriate for the kind of commercial software that my company builds. -- SomeoneElse

My argument in favour of lighter-weight processes is that the heavyweight ones cost too much in most cases.

On the other hand, our ApplicationOutsourcing? practice claims that significant (~20% if I remember correctly, I try to find the exact figures) reductions in schedule and cost accrued when they forced all their projects up to CapabilityMaturityModel level 3 (and in some cases 4).

-- KeithBraithwaite (the other anonymous poster)

What seems like a lighter-weight process may be one that's just optimized in one area of the development, and pays its price in another. Processes that go light on analysis can end up paying the price in heavy system testing. The weight of a process and the cost of running that process are most often matters of intuition and subjectivity, rather than measured quantities. It's ironic that the CMM is considered heavy, whereas it's maybe (in competent hands) no more than the means to determine what is heavy and what is not. -- WaldenMathews

Please provide real-life examples of lightweight CMM.

My use of the CMM during the years 1995-1998 in managing ticker plant development for ILX Systems would be a lightweight use of the CMM. I used it informally to guide me in bolstering requirements management, configuration management and project management, which resulted in a more stable group behavior and a better product. (The above disciplines often blended into each other, rather than being discrete functions, because that's what the environment called for.) There were no assessments, no big strategic or tactical plans (although there were some small ones I wrote to keep my own head clear). Mark Paulk says the appropriate amount of documentation for a Repeatable project might fit on one side of a napkin, if it's a small enough project. The problems with WaterFall methods and CMMs is not the models, it's the failure to use common sense in scaling their use to the problem at hand. Ironically, I've found that the very folks who are the most scared of a bureaucratic process are exactly the ones who will create one when you give them the steering wheel. I wonder why that is. -- WaldenMathews

I have been living with these thoughts for a couple of years now, so they have become "obvious", slowly, over the 2-year period. The "obvious" (in hindsight) part is that heavyweight ones cost more, and can be justified on occasion; that light ones are also effective, less expensive and can justified more often.

The part that isn't clear yet ("not yet obvious") is whether heavy, rigorous methods can possibly be less expensive than lighter, more casual ones. WattsHumphrey claims they are, which seems counterintuitive, excepting when compared to sloppy development (which characterizes very many projects, I'm afraid). So the interesting question is a medium group of developers with pretty good communications (using CrystalClearMethodology, for instance), on an industry or web project, compared with a medium group of developers using PSP/TSP properly. I would like to vote for the former, but I have been hearing lately from the SEI camp lately that they claim increased productivity and reduced defects. -- AlistairCockburn

A bit of searching around NASA's Web sites found an interesting formal methods research group. See http://shemesh.larc.nasa.gov/fm/fm-why.html for their essay "Why is Formal Methods Necessary?". (This group does not seem to be related to the shuttle group.) Also, searching for "software validation" found some interesting pages like:

RichardFeynman's brief observations on the Shuttle software can be found at http://science.ksc.nasa.gov/shuttle/missions/51-l/docs/rogers-commission/Appendix-F.txt (scroll down to "Avionics"). The other sections, particularly on the main engines, are also worth reading -- I think it's a fine essay on the dangers of designing from the top down.

Recommendation seconded. Here are a few relevant sections:

It appears that there are enormous differences of opinion as to the probability of a failure with loss of vehicle and of human life. The estimates range from roughly 1 in 100 to 1 in 100,000. The higher figures come from the working engineers, and the very low figures from management. What are the causes and consequences of this lack of agreement? Since 1 part in 100,000 would imply that one could put a Shuttle up each day for 300 years expecting to lose only one, we could properly ask "What is the cause of management's fantastic faith in the machinery?"

We have also found that certification criteria used in Flight Readiness Reviews often develop a gradually decreasing strictness. The argument that the same risk was flown before without failure is often accepted as an argument for the safety of accepting it again. Because of this, obvious weaknesses are accepted again and again, sometimes without a sufficiently serious attempt to remedy them, or to delay a flight because of their continued presence.

[...]

To summarize then, the computer software checking system and attitude is of the highest quality. There appears to be no process of gradually fooling oneself while degrading standards so characteristic of the Solid Rocket Booster or Space Shuttle Main Engine safety systems. To be sure, there have been recent suggestions by management to curtail such elaborate and expensive tests as being unnecessary at this late date in Shuttle history. This must be resisted for it does not appreciate the mutual subtle influences, and sources of error generated by even small changes of one part of a program on another. There are perpetual requests for changes as new payloads and new demands and modifications are suggested by the users. Changes are expensive because they require extensive testing. The proper way to save money is to curtail the number of requested changes, not the quality of testing for each.

[...]

Finally, returning to the sensors and actuators of the avionics system, we find that the attitude to system failure and reliability is not nearly as good as for the computer system. For example, a difficulty was found with certain temperature sensors sometimes failing. Yet 18 months later the same sensors were still being used, still sometimes failing, until a launch had to be scrubbed because two of them failed at the same time. Even on a succeeding flight this unreliable sensor was used again. Again reaction control systems, the rocket jets used for reorienting and control in flight still are somewhat unreliable. There is considerable redundancy, but a long history of failures, none of which has yet been extensive enough to seriously affect flight. The action of the jets is checked by sensors, and, if they fail to fire the computers choose another jet to fire. But they are not designed to fail, and the problem should be solved.

I read the article and noticed one big thing - they have management that requires a complete rework of the schedule when a requirement is changed.

That is the key to making BigDesignUpFront work - not having a irresistable force of moving requirements and an immovable object of deadline based on original requirements. If the customer knowns that each little thing s/he asks to change will impact the schedule by X+N days, where X is the time to rework the schedule and N is the time for the change, s/he gets real careful about requesting changes. And when the schedule is actually moved out when more work is added, of course you can make the software better!

-- PeteHardie

See WhyTheShuttleIsLeftHanded

EditText of this page (last edited July 24, 2007) or FindPage with title or text search