Only Foresight Matters

I used to always say that the best quality for a programmer is persistence, but the best quality for a SystemAnalyst is foresight (see also GuerrillaGuideToInterviewing). Of course, if you're stuck developing a whole system by yourself, then it helps to have both. Foresight is of supreme importance because it keeps you from spending an inordinate amount of effort following AntiPatterns down the wrong path.

Unfortunately, I have found it a rather difficult task to teach somebody to have foresight. It seems more like an InnateHumanTrait?, people either have it or they don't. If you are a person with foresight, then when you are in the middle of a development project and you face a decision (a fork in the road, which way should I go from here, what's the right thing to do next?), you connect to yourself (or your clients, or whomever will maintain your software) in the future, and you find out how come they are so happy with what you have done. This will then guide you to do the right thing in the present (a structure? internal documentation? a methodology or algorithm? formal documentation? a self-contained parameter-driven external procedure?).

Another perspective is that our future causes our present. In other words, we do what we are doing today because of where we will be tomorrow. Others will argue that Life is not a goal, it is only a journey. But I believe that good software development projects are both. You (may) have a goal (see TypesOfProjects) that infuses you with direction, but along the way you also need to DoTheRightThing.

But:

What, your foresight didn't tell you there was a But:? Ah, you must be one of those poor benighted souls without adequate foresight ...

Most developers on this here wiki have found a more rewarding way to look at development is as a kind of SimulatedAnnealing. Let's say your goal is to boil an egg. At first, all is chaos! Well, okay, it could be worse, you may have some joker who believes that so long as you foresee a boiled egg you're as good as fed ...

But let's say you're lucky and all you have is chaos. Wonderful! We nail part of the chaos down. First thing is, where are we going to boil our egg? In the kitchen! Great, we have a kitchen, there is a step away from chaos. Now, where can we find eggs? In the refrigerator! Ah, very comforting to have a well stocked refrigerator, there's another piece of the puzzle done. But: maybe we're out of eggs - we just discovered we need to go to the store first! Damn, why didn't we foresee that before going all the way to the kitchen? Heck, perhaps it's because we're puny humans and the task of boiling an egg has far too many variables for anyone but BigOmega to foresee ...

You understand where this goes of course. In each iteration of a development you need to reduce your uncertainty empirically. If you can't do that, all the foresight in the world won't feed you. That's the trouble with BigDesignUpFront - all foresight, no empiricism. --PeterMerel

Here are a few observations:

Hindsight is much more accurate than foresight.
Criticizing other people's failures is much easier than coming up with something good from scratch.
Things seldom turn out as you imagined. It's not that you considered a possibility, decided it was unlikely or tried to prevent it, and then it happened anyway. What usually happens is just utterly unlike what you imagined, like the way the FoundingFathers never imagined the Internet, and the early creators of the Internet never imagined CascadingStyleSheets.

What can we learn from these hard facts? You could draw the conclusion that we need to stop and think ahead more, imagine more possibilities before acting, and criticize less.

But here's another observation:

Someone tells you to think ahead. That sounds like good advice, so you try. But no matter how hard you squint and scratch your head, you either draw a blank, or you get such a rush of alternative possible futures that you don't know where to start (just think of how difficult it is to look ahead fruitfully in the GameOfChess). No matter how hard you try, you are still stuck with the three problems above.

So have you considered drawing the opposite conclusion? Since foresight doesn't work, don't rely on it. Go for early hindsight. Just try stuff, see what you can learn, and try some more stuff. Instead of trying to do things right the first time, actively invite error. Instead of avoiding criticism, criticize everything you can, and encourage others to criticize. Keep these explorations as quick and cheap as you can, and criticize in a way that encourages people to continue to invite error.

These strategies are techniques for rapid learning. They work by getting reality itself you help you out. Reality has a lot more imagination than you have. Depending on foresight, you're stuck inside your own picayune imagination, and you'll never be able to keep up.

--BenKovitz

A few more thoughts along this line:

Never say, "You should have seen this coming" or "You should have thought ahead." If they could have, they would have.
Of course you can't teach foresight. The only thing you can teach--or learn--is knowledge of the topography and causal factors of a particular domain. Armed with that knowledge, anyone can make good, educated guesses about how things will turn out. Saying, "Think ahead more" appeals to an imaginary faculty of foresight that no one has. Saying, "Here, let me show you what's going on in this game. See how this affects that? See how this other thing has very little effect?" appeals to real, amazing human cognitive powers and really helps.
Foresight is speculation. So is hindsight (at least if it involves any attempt at explaining why things happened as they did, which is necessary if you're going to apply the hindsight to your next move). Hindsight is just better-informed speculation.
Don't limit yourself to explorations where you imagine all the possible outcomes and decide in advance the conclusion you're going to draw from each. (That is, ignore the standard preaching about experimenting "scientifically".) Otherwise, you won't see all the unanticipatable insights that fall out right in front of you.
The more you think, the more wrong you get. The more you put your thinking to a test, the more corrections you get to make.

--BenKovitz

Ben, the advice above seems helpful in some situations, but aren't you lacking some balance here? You say "Things never turn out as you imagine." [emphasis mine]. On the contrary, for most of my waking hours, things turn out mostly to exactly as I imagined. The techniques you advocate have a special niche for me in the area of breaking a mental impasse, but not in the majority of my activities.

I don't think of "foresight" as knowledge of the future; I think of it as the process of extending knowledge of the past to likely futures. As such, it is a most teachable technique, and I teach it to my young children by asking them questions like "What's going to happen first? Then what?" and so on. The knowledge thusly mined is not magic, but it's also not obvious until you've developed the habit. And it's extremely helpful, especially when the parallel sense of uncertainty of prediction and shift to higher level is incorporated. To use the chess example, I would never try to plan out a series of actual moves ahead of time; that's too specific and brittle. But I would make a plan that sounds like this "In this game, I'm going to use my knights extensively in the opening." That's partly a learning strategy, but why not plan to learn?

There are two ways to go seriously awry. One is to rely too exclusively on the scanty knowledge already in mind to solve a problem that is largely foreign. The other is to neglect large bodies of past relevant experience and approach a mostly familiar problem as though it were not. But I'm preaching to the choir here; you explained this stuff so nicely in your Practical Requirements book (or was that a different BenKovitz?). I'd like to see more balanced analysis of these related phenomena.

-- WaldenMathews

Foresight is difficult: Those who use crystal balls eat broken glass.

Can insight replace it?

Sometimes you want a special kind of hindsight. You need the ability to pause, see that you missed something and go back to it. Once you see it it becomes obvious. There you are hurtling along on the freeway. You are making great progress! And you miss the turn off. EdwardDeBono argues that this error is wired into the way our brain works.

Perhaps this is where PairProgramming can make a big difference?

--DickBotting

Warren Buffet's rule : "Predicting rain does not count. Building an ark does."

A very commonly heard phrase in project reviews is : "I said so a year ago !". That's a useless remark. If you said so but did not act on it, then you were just wasting your breath.

-- MaxT

You know what, MaxT? Sometimes you can say so, write memos, publish proposals, jump up and down during meetings, and generally make as much of a pain in the neck of yourself as you want, and guess what? Jack. If you can't affect a decision made at a higher <ahem> management level then you're screwdificated. This doesn't mean that the effort was wasted, just that the actions required were not taken. If you can rub the proper noses in the proper filth as a result of them not taking your advice then sometimes it's worth it. Other times it's not. -- MartySchrader

Or indeed, decisions made by obstinate programmers, despite all advice of their peers. It's their ball and you can't play with it - shame you have to interface with it. -- Jules

CategoryPlanning