Software Development Made Heavier

Is anyone besides RonJeffries and me getting the sense that software development is getting turned into a heavier and heavier process? SteveMcConnell had some good books out, like CodeComplete and RapidDevelopment, now his SoftwareSurvivalGuide? is looking a lot like CMM.

I just started researching project management a bit more closely, most recently this intriguing site called Software Process Management Network which has the black, yellow and red books). That site gives 10 indications it is time to schedule a project review, and I couldn't even understand a number of the questions being asked!

How are the wiki readers? Are we split across the "keep it light" and "make it measurable" divide, are we throwbacks to an earlier time when one could fly by sight, or are we throwforwards of some sort?

I begin to doubt my own experiences when I read these articles and books, I sure can't figure out how to apply them on my current project.

-- AlistairCockburn

Yes - development is getting "heavier." But (in some cases) it's also getting more responsible. I like the analogy to making houses - it took a long time before we figured out how to make houses that didn't tip over, and we're still not all that good at it. It takes eight-year-old kids a few hours to build a tree house, but I wouldn't trust it enough to live in it. There's a pretty "heavy" process for making houses, especially for larger buildings (skyscrapers). It involves architects (using known practices), engineers (using known practices), and builders (more known practices). The creativity is still there, but only towards the top.

Maybe we need to be more concerned with responsibility. It doesn't bother me too much that my parents don't understand "programming," especially since I don't understand "architecture" or "brain surgery." It DOES bother me to see projects turned out based more on functionality and time-to-market when I can't use them reliably (due to bugs, crashing, etc).

It's true that the market (eg, those who sign the checks) drives most development, and hopefully the market will settle down and let the differences between "architects," "engineers," and "builders" shake out. Maybe then we can get more "responsible" code. . .

-- AustinDavid

Engineers, by nature, want to nail things down. It should all be a matter of getting the process defined. There are only a few sensible ways to design and build a bridge over water. (I am not a bridge engineer, but humor me for a moment)

In a nutshell:

The problem is defined (we want to get from land mass A to land mass B. The context (there is water separating A and B and you can see A from B - the distance is not too great to consider a man-made structure connecting the two). The forces are identified (Windy climate? Often cold? Rains most of the time? etc). The solution includes best practices and standard formulas (for stress, support, etc).

Building software is not so straightforward (yet?). We make software (processes) heavier in an attempt to nail it down.

However, most software is not like bridges. We don't live on bridges; we don't change bridges to adapt to new needs and requirements (unless we are just fixing the bridge). But, we do live in some software and we require the software to adapt to new needs and requirements. You just can't engineer liveable software. There are no well-defined processes for dictating the usability of software.

As for houses, take a look at HowBuildingsLearn and we see that even things well planned and structured must be able to adapt to changes. Change is organic; not dictated by process. The further architects got away from understanding change, the more we got to suffer working (living) in our modern skyscrapers and large office buildings. I hope software development doesn't get that heavy.

-- ToddCoram

(Is there already a "Engineering Envy" page on Wiki?) (Yes, DisciplineEnvy)

Civil engineers, the ones who build houses and skyscrapers and bridges, have three thousand years of experience, and some pretty sophisticated math that backs up their best practices. (Civil engineers can design a bridge that can support a hundred ton load because, for a given bridge design, they can calculate the load.)

Software "engineers" have roughly forty years of experience, and no mathematical foundation worth a damn. (Most of the "software metrics" used are voodoo. The work Dijkstra (EwDijkstra) and Gries and co. have been doing is a better base to build on, but at best practitioners are ignoring it, and at worst most practitioners are incapable applying it even where it can scale up to meet the problem.)

Or maybe I'm just grumpy this morning.

As I posted earlier to the QuotePage:

"Some treat the overhead of big projects as inevitable. The result of such treatment is a plethora of complicated systems to manage this overhead. All too often, however, the result of such management is merely well-organized overhead. It's still there; it's just been put in neat boxes and charts and thus it seems easy to understand. Some people revel in this overhead. They behave as if it is a GoodThing - almost as if the overhead is really the cause of productive software development, rather than something standing in the way." AndrewKoenig and BarbaraMoo, RuminationsOnCeePlusPlus

-- PaulChisholm

I, too, was very depressed by SteveMcConnell's new book. Too much process for anything smaller than, say, Microsoft Windows.

I've been involved in many software projects over the past too many years. I have been involved in many small successful projects. Most of them have been successful, I believe, because their small size allowed effective communication between developers, and between developers and customer.

I have been involved in many large projects as well. And some of them have been successful, some wildly successful. However, all the large ones seemed to slog on and on; they seemed to use up people; often they weren't much fun. The more mechanism/process they had, the less likely (it seems to me) they were to succeed. Could it be that heavy process doesn't improve communication?

My top three conclusions:

3. Small projects are more likely to succeed than large.

2. Light process is more likely to succeed than heavy.

and my number one conclusion:

1. I don't want to be on any more big heavy-process projects.

P.S. Yes, I know that IBM CleanRoom has generated some successful error-free software. Haven't been there, haven't done that, don't want the t-shirt. Has anyone tried writing the same program with a small team with small process?

-- RonJeffries

I've been writing (mostly) C code, for over 15 years. I've spent a good part of my life fixing bugs. Anything we can do to improve the quality and reliability of our software is a good thing. I'm getting too old to put up with the consequences of sloppy software development practices.

If you want to get results, then you have to use measurement. And in order to get decent measures, you have to use process. To me, processes and patterns are not all that different. A process is a pattern for accomplishing work.

Challenge yourself. Get Humphrey's PSP book, and take the PSP course.

-- KentSchnaith

(Kent, I think I started having a violent debate with you at the OOPSLA 97 ice cream social, then walked away.)

If process is a pattern for accomplishing work, then all processes are patterns, but most patterns are not processes!

Getting results does not require measurements. Certain methods of improving results (notably StatisticalQualityControl?), and most methods of managing results, require measurements. Measuring the wrong thing, and optimizing it, is a classic AntiPattern.

-- Watts S. Humphrey

Introduction to the Personal Software Process, ISBN 0201548097

A Discipline for Software Engineering, ISBN 0201546108

WattsHumphrey's PersonalSoftwareProcess, so far as I can tell, is designed to help you optimize the manageability of software development (in particular, of a series of fairly similar software development efforts, e.g., same organization, same developers, same domain). It can also help optimize other measurables that are subject to StatisticalQualityControl?.

PSP specifically doesn't tell you anything about how to develop code with fewer bugs, or in less time, or with less staff. It's a meta process for figuring that out.

PSP and other heavy software development processes seem to involve a FraternityInitiationEffect? (which is a bad thing).

-- PaulChisholm

PSP is, IMO, a good idea. I wish it weren't so student-oriented. I advocate using a ProgrammersNotebook and have for ages. A problem I had with the DoItOnaCard? approach of ExtremeProgramming was that too much history got lost. I have gone back to carrying a notebook, and the C3 team now has several other notebooks: System Performance Log, individual Support Action Logs, and a Time Tracking Log.

You do have to measure, and you do have to have process. ExtremeProgramming is a process. It stems from the notion that the process should be as simple as possible, but no simpler. EinsteinPrinciple applied to software.

-- RonJeffries

Some days, I get the feeling that several large software companies are trying to make programming so difficult that only they can do it. -- AndrewKoenig

I think a lot of this comes from a sort of funny dichotomy in management. They don't actually understand development - they don't get any of the AhHa feelings. They're very, very determined NOT to understand IT, but they also want to fix the massive project failure rate by understanding it without actually understanding it. So they concentrate on the paperwork, not on the doing because that IS something they understand. Someone, I forget who, used the phrase TheMeetingsBecomeTheProject?. In the same way, a lot of software development looses sight of the goal - ThePaperWorkBecomesTheProject. It's like the smaller the percentage of the project that actually consists of the bit between "start developing!" and "are you done yet?", the less important it is and therefore the less important it is that it's not understood.

I've found several organizations where the paperwork happens in completely divorced fashion from the actual coding. The coding gets done in an almost XP fashion down here at floor level, and over there is a bundle of people who are happily emailing each other project files and word documents. Which contain information about a fictional project that may or may not match the software produced.

I'd be happier if this was recognised by the people involved, but it rarely is. A distinction should be drawn with the sort of guerrila software development where you're doing the BigDesignUpFront because a client thinks they want it, but you're well aware this is a fib and WorkingSoftwareIsTheGoal.

-- KatieLucas

Maybe SoftwareDevelopmentMadeHeavier is just another application of the FirstLawOfBadManagement.