Objective Progress Vs Subjective Paperwork

First the ground rules. The universe contains too many distinctions for any one of them to be worth more credit than any other. If we say the programming industry consists of "technicals" who write software, and "suits" who pay for and use the software, then we risk being divisive. Reality always has a finer texture.

So let's work with some temporary definitions:

Suits

management-types who understand productivity in general, but not the craft of coding in specific.

Technicals

those who actually perform the primary act of coding, but who may not comprehend the art of managing programmers.

We have a major risk of ImpedanceMismatch here. Suits who understand general ideals of productivity, but not what specific software productivity actually looks like, may then chronically request the wrong evidence. Commonly, they mistake subjective paperwork for objective progress.

Next, some few of these Technicals may be unable to manage themselves. Nothing to be ashamed of - it's just an education thing.

So to fill the gap, these Technicals learn from the kinds of management they receive. So if that management is incorrect, they may bond with ideals of productivity that aren't actually germane to its real proof. Ideals that obstruct their abilities.

To overcome these obstructions, they may then add more process. And they may feel proud of this "discipline" to chronically throw good effort after bad. An objective measurement of their practices would reveal the issues, so they feverishly defend against such pragmatic measurement, further hiding under the cloak of "process".

-- PhlIp

The problem was solved by HarlanMills circa 1970. The first publication that laid out the solution was a paper called "StructuredProgramming in a Production Environment" by F. Terry Baker, a team-leader working under Mills. Mills' success was already legendary, but very few got it right when they tried to emulate Mills. See PrinciplesOfSoftwareEngineeringManagement by TomGilb for an example of someone who figured out what Mills was talking about.

When you got to the term "StructuredProgramming" in the above paragraph, that instantly put you on the road to misunderstanding what Mills has to say. For Mills, IncrementalDevelopment was the most important component of structured programming. If you go around and poll software developers, asking "What is the most important component of Structured Programming?" none will answer "incremental development". But, this was Mills' great insight.

Incremental development involved

1. Writing a short high-level functional overview and then
2. growing the system incrementally so that initial increments of the system were available for use and/or usage-oriented testing within weeks after the start of construction.

Progress was tracked based on what percentage of the functional requirement were met at a given time by the growing system. QA was peformed by usage-oriented testing and reliability measurments on the growing system.

Incremental development enthused and organized the efforts of the programmers.

As for the suits: For any project there is a person or group in the hierarcy where the money decisions and the technical decisions come together. This is the place where projects get evaluated and continued or cancelled. Incremental development provides these folks with a way to get a handle on progress that inspires confidence in everyone. The fuctional specifications should be implemented in the order of importance from the prospective of these key suits. It is often possible to create a piece of software that is seen as valuable by these suits well before all requirement are met.

BTW: The term IncrementalDevelopment was not used in that early paper that I mentioned. They used the term top-down programming, but that term got totally garbled in transmission, so Mills abandoned it around 1980.

Baker provided an interesting solution to the paperwork issue. He pointed out that clients or management may ask for a detailed design before coding. However, he noted that these folks did not know what a detailed design was, therefore the development staff could avoid doing a detailed design and substitute a more useful document. He suggested that the "detailed design" should be a description of the development staff's understanding of the problem, it should lay out the problem domain without representing a system design. The client will be able to understand this document; they are often the experts on the problem domain so their comments on this document are helpful. (See CustomerTest.)

Baker said that this trick would not work on DepartmentOfDefense? projects because of the regulations; it was necessary to request a formal variance. But these days, the DOD regs have been changed to allow incremental development.

-- Tom Adams (reprinted without permission from the XP newsgroup)