After The Gold Rush

SteveMcConnell's latest [http://www.construx.com/stevemcc/gr.htm]

He says that software engineering should be more like other professions- CivilEngineering, the LegalProfession?, the MedicalProfession?. We should have certification exams, continuing education, and accreditation of schools of SoftwareEngineering.

I will agree the moment it appears we actually know how to engineer software. -- KentBeck

Kent- I agree that the phrase "software engineering" has a lot of baggage associated with it, and under the influence of you and Martin and others I've come to regard software development as a craft to be mastered. But doesn't your conditional agreement with McConnell's assertions rest on the definition of "engineering"? In ATGR he defines engineering as "the application of scientific principles toward practical ends". Do you not apply XP's five basic principles toward the practical end of addressing risk in software development? Is this not also what the patterns movement is all about? I have familiarized myself with both your perspective and McConnell's perspective, and I think you have a great deal of common ground. -- RandyStafford

That's certainly a reasonable definition of engineering, but that doesn't mean we know how to engineer software. In particular, McConnell seems to believe that "engineering" means "like mechanical engineering", where you carefully draw pictures of what you are going to do before you do it. I think XP is "real" software engineering, because it takes advantage of software's strengths as a medium while avoiding its weaknesses. Many folks think I'm a loony for saying this. -- KentBeck

In his books I haven't sensed that belief (carefully drawing pictures). He talks in SoftwareProjectSurvivalGuide about an incremental and iterative development process, in which "architecture design" occurs in the first iteration, and "detailed design" occurs in every iteration - and he acknowledges that expert developers take an informal approach and combine construction and design. Is that significantly different from inventing a SystemMetaphor and refactoring as you go, which has worked very well for me on my last two projects? What would it mean to "know how to engineer software"? Would it mean anything different from knowing how and when to apply principles and patterns? One of the interesting things about that definition of engineering, is the connotation of "scientific principles". What if that didn't have to connote "principles based on physical laws of nature" but could instead connote "principles shown to be valuable (really, not shown to be invalid) by the scientific method"? -- RandyStafford

MichaelJackson expresses his opinion on what SoftwareEngineering is, and why it should be called that, in SoftwareRequirementsAndSpecifications.

My reading of ATGR was that the software profession needs to strive to become more like other professions. Perhaps CivilEngineering is a RedHerring, but I don't think Kent's objects applies so strongly to the analogy with medicine or the law. How about the professional bodies attached to various arts and crafts? -- KeithBraithwaite

If 3 out of 4 software projects weren't managed into the ground, we'd be in great shape. Might even run out of things to do. It's not a problem with our tools. It's not because we don't have enough programmers. It's not a problem with professionalism, and it's not because we lack the certification-de-jeur. It's a problem with the way we manage software. -- WayneConrad

Or another way of phrasing it, it's a problem with those who would manage software development. What other profession has such a disconnect from those with expertise to those who would manage them?

Managing the teachers of public schools comes to mind. So does hospital management.

How would 'you' suggest 'we' go about managing software development? Please identify the principle faults with the "way we manage software" and how would you go about remedying the situation in a constructive fashion? -- Anonymous

I have not responded to this for a very long time because I wasn't sure how. I'm not sure I'm qualified to comment on the state of the whole industry, or even on my part of it. Especially difficult is your request that I remedy each of the things I think was wrong. I do not have cures for each of these diseases; if I did, I would be out getting $400/hour treating sick projects instead of here typing stuff into Wiki. However, I do not think that means the disease doesn't exist. If I am flat on my back sick, I don't have to know how to treat myself to know that I'm sick.

Here are some of the more remarkable management behaviors I've encountered on failed projects. The killers are:

Management by hoping: Whenever the management encountered evidence that the project was slipping features, quality, or time, no effective corrective action took place. The usual response is, "We'll work harder and catch up."
Lack of communication: Team members working in individual cubicles with as little as 30 minutes a day of contact with other programmers.

There are plenty of other causes that contributed to the failures, none quite as bad as these, but still plenty bad. Some of them were:

Guesses instead of estimates: No spikes or research of any kind allowed before giving an estimate ("we don't have the time for that, just give me a number").
Lack of awareness that costs are not simply additive: Each feature, either visible or internal, increases the cost of each other feature as well. This leads to unmanageably complex systems as too many good ideas are thrown in.
MicrosoftProjectScheduling

-- WayneConrad

That is certainly not the message I got from the book. The reason people are cowboys is because that behavior pays. McConnell looks forward to the day that behavior will not pay, but he doesn't say those people will die off. Some will, others will change. His main point is that the risky approach that lots of people take will not work as well in the future as it does now.

In fact, there are lots of software development groups that do not take a goldrush approach. Perhaps the web gets the press these days, but there are a lot of people working for phone companies and other places where reliability is more important than speed. These companies have software development methods that are slow, expensive, and very reliable. -- RalphJohnson

Those who would manage professionals should have respect for and an appreciation of the profession they hope to manage. Otherwise their activities will be counter-productive, and revert quickly to dominance games with little benefit to themselves or their organization. I am quite familiar with both the health-care and the software industry, and I have never heard of a hospital manager as ignorant of the health-care profession as the average software manager I've encountered is ignorant of how software development is actually done. I've seen the best programmers in state-of-the-art Silicon Valley companies horse-whipped by their naive managers who imagine software should spring instantaneously (and bug-free) from some paper design. The "smarter" ones learn to quickly avoid any programming responsibility, any requirement of actually making things work, because of the lack of patience in their management for any visible shortfall in the intermediate work-product. I think it is this lack of "professionalism" in software managers that has caused the programming profession to have the highest rate of overturn. The majority of those with computer-science degrees are no longer programming 5 years out of college. Why bother?

Although I agree with the idea that software management (as a profession) is the root cause of the problem, I'd hazard a guess that one major reason so many CS degreed people stop is that they are riding that Gold Rush trying to cash in on big money. They aren't drawn to software development by the love of the work; they just think they can MakeMoneyFast at it. When the management crisis hits them, they bail out. Those that remain probably hit the wall at 10 years - that's 3-5 companies, usually, and by that time they will start to despair. -- Pete Hardie

There seems also to be a trend, however, that "slow, expensive, and very reliable", is rapidly being scrubbed out of business by companies that get there sooner and cheaper, despite lower reliability. I get a sense that consumers of technology have become accustomed to the idea that software, as any new technology, is flaky, but they'd rather have it now with some quirks than highly reliable at some later date with some higher cost. Over time, bugs get fixed, overcome by events, or worked around so often that they start to seem like features. Working around deficiencies in MS products is pretty much second nature, and that experience is by no means limited to MS products.

You'd think this approach would only be a problem if it applied to software in medical devices, airplanes, automobiles, or some other life-threatening situation. Yet, the internet (possibly the flakiest domain of all) is fast becoming a critical part of many financial transactions and bugs represent an opportunity to destroy thousands or millions of livelihoods.

Rather than software development approaching a situation where buggy cowboy software becomes less common, it seems to be heading the other direction: formerly solid problem domains are increasingly subject to the pressure of reducing time-to-market, and cowboy techniques are becoming the official corporate philosophy.

On another topic for this page, the comparison to public education is intriguing. (Please forgive all the impending AmericanCulturalAssumptions.) My parents were public school teachers and often complained about school administrators enacting stupid policies surrounding the latest buzz in educational theory. I never made the connection before, but the education industry, despite all its efforts to the contrary, still doesn't have a grip on what it takes to teach, and success or failure still largely depend on the talents of the individual teachers and not on the methodology of the day. There is no secret sauce that can turn a bad teacher into a good one, no magical program that turns out well-schooled children with perfect yield. Many perceive that all of the new-fangled theories are contributing to lower performance in schools. Everyone has their ideas and theories, and direction is -mposed by elected school boards filled with people with absolutely no formal training in education. The similarities to software development are striking.

-- ChrisFay

OrphansPreferred, an article from the book, is available on-line. The draft of the second edition of AfterTheGoldRush is online at: http://www.stevemcconnell.com/gr2.htm. The book has been published under the title ProfessionalSoftwareDevelopment.

Calling ourselves "software engineers" is like janitors calling themselves "custodial engineers".

In my view, the basis of any claim to "engineering" is some kind of theoretical foundation for what we do. A civil engineer, faced with the challenge of, for example, using a new material to construct a bridge across a river, has some idea of what to measure, what to be concerned about, what to ignore, and whether or not the structure will do what's needed. In particular, the civil engineer has a reasonably deep theoretical model that he or she can use to provide safe and well-understood safety margins to critical components.

We have no comparable theoretical basis for our practice. Even the most fundamental engineering questions - how many "work group servers" do I need, how much memory to they need, what disk capacity, what CPU power, etc., etc., - not only have no answer, but lack even a theory about how to derive an answer. An "engineer" wants to measure something, transaction volume, for example, connect it to something else, latency or bandwidth, for example, look for "knees" (inflection points), and then understand what produces that inflection point.

A call center, for example, needs to guarantee a response time of 3-5 seconds across a range of loads. An "engineer" would be able to demonstrate not only that the response time is within tolerance, but that it will stay within tolerance even if the call volume triples. An engineer would be able to describe quantitatively how the parameters interact and what alternatives exist to, for example, handle increased call volume. What is the trade-off between making one local server more powerful versus adding a second? What is the performance trade-off between memory, disk, and CPU? Is the transaction rate limited by the communication channel or the system?

[Actually, the term for someone who does this sort of work is "statistician". I am in full agreement with the rest of your rant, however.] You're joking, right?

In 29 years of professional experience, I've met not even one practicing professional who can answer questions like this, I know of no curriculum that teaches it, and I know of no such theory. We know something about programming. We know something about math. We know something about information theory and knowledge modeling.

I submit that we know virtually nothing about the theory needed to call what we do "engineering".

-- TS

Complete agreement. I've been saying this for years (on this wiki too).

I beg to differ. What you are describing here is systems engineering, not software engineering. I agree with you completely except in your point that these need to be done, and that people who do them have earned the title "engineer", but I do not agree that they are required by someone called a "software engineer". I do agree with the general point that what most programmers do is definitely not deserving of the title.

I was taught how to do this analysis in my computer science education 30 years ago. If the service rates are sufficiently linear a queueing model will suffice. Otherwise, a properly validated simulation will provide the answers. If no one is much interested in these techniques then I can only assume that extra servers cost less than the people who can do the analysis. -- WardCunningham

[Indeed. Moore's Law has held back software engineering because it's become so much cheaper to say "throw some more CPUs and RAM into the VM" than to spend many hours analysing a complex and intermittent performance problem in a large heterogeneous system. We have everything we need to do "software engineering" the problem is that we've discovered that we can't afford it.]

CategoryBook