Present Empirical Data

Often when faced with someone recommending change, people ask the agent to PresentEmpiricalData.

This is rarely - I am tempted to say never - a request for empirical data. With rare exceptions, empirical data does not convince, let alone induce change.

Asking someone to PresentEmpiricalData is what sales people call an "objection", and they teach that the first objection raised usually isn't even a real one. The real meaning of an initial reaction of "PresentEmpiricalData" is, all too often, "Go away".

Therefore...

The first technique in dealing with sales objections is to ignore them. When they aren't the real objection, they tend not to be repeated, and this is often sufficient to move on to the real issue. Sometimes you have to deal more directly with the objection ... and almost invariably, when you do, you find another objection.

One way to test whether "PresentEmpiricalData" is a real objection is to invite the objector to do the experiment himself. If he answers "I already know it won't work", you can be sure that he expected no empirical data, and that if and when it was presented, he'd "already know" why it wasn't applicable to him.

Of course it would be really good to have empirical data, no denying that. Just recognize that when you get it, you won't instantly close any more sales, you'll probably just get to the next objection. Keep on drillin'.

But...

Often when faced with someone recommending change, people ask the agent to PresentEmpiricalData.

This deserves clarification - specifically, is that to be read as "recommending change in general" (there's something wrong that should be changed because going on as we are will lead to disaster) or "recommending a specific kind of change to address a specific problem" (here is how we might do things to improve the situation).

For the second kind of concern - you are recommending a specific change to address a specific problem acknowledged as such - I'd argue that empirical data by itself isn't sufficient. It needs to be presented as supporting a given set of objectives, or as a method for achieving the necessary improvements, or whatever; in short, fact without theory is worthless.

Empirical data by itself is primarily useful in justifying the first kind of concern. PresentEmpiricalData to the effect that a problem exists, and you have justified the need for change.

For that, though, you need to see things from the point of view of the people to whom you are recommending change - would they agree that your data signals a problem, according to their own definition of their objectives? (See FirstCreateTheMailbox.)

If you have been asked to solve a problem and are recommending a specific solution, and they ask you to PresentEmpiricalData - it's a stall. Brush it aside and move on to the real issues.

If you are just butting in where no-one has asked you to solve a problem, and recommending change in general "just because", PresentEmpiricalData may be a way to say Go away. But possibly it's as good a way as any other.

Further buts...

In my opinion, the above is a very black and white way of looking at the world. How do you know exactly what someone else's agenda or thoughts are? We are taught as Engineers to work with data and the Scientific Method. It is perfectly valid for someone to ask to PresentEmpiricalData. If it works, you should have rigorous EmpiricalData? to back up your claims.

We have seen great things with ExtremeProgramming. It is the pretty much the only methodology that makes sense to me after 22 years of everything from CowboyCoding to CMM. However, I think even Kent would agree that more data needs to be collected. I would disagree with what you said above. As HalClement taught me, "A theory in Science is not accepted as a theory unless there is EmpiricalData? to back it up." It should be the same for XP. Hand-waving is not acceptable.

-- sg

Please read it again, Sam, then say more about why it's black and white. Note that it says it would be nice to have data: perhaps that's not strong enough. What percentage of folks who see the data do you think will then move immediately to doing XP? And please in your reply discuss how you wound up trying XP after so long on "the other side". Thanks!

I'm not sure how many people would be "sold". But I know a lot of Engineering people make decisions on things based on this kind of data. SteveMcConnell is one of them. And there are others. I'm not saying we have to measure everything like PSP does. Gosh no. But we should be able to collect some form of EmpiricalData? and present it.

How did I come to try XP? In my twenty plus years of software development, I have seen many "heavy" processes that just did not work. I started my career working for the DOD and using DOD2167 and seeing the results. The results were that we wrote specs for months and months and did little else. The specs were completely wrong when it came time to do anything. The same thing at Raytheon with projects actually getting canceled after months of spec writing and no code. Digital was slightly better, but with their emphasis on a "Phase Review Process" at the end of each of the four "phases", it was clearly a waterfall model that broke down constantly and no-one ever read the specs anyway (we know because we put certain things in it to get a reaction!).

Throughout all of this, I always saw the customer as totally unhappy and the developers "fed up". There was no quality. I also worked with ISO9000 and CMM at various companies and it was a horrid way to do anything. Documents upon documents. There were documents just to change a line of code! It was clear to me that to survive in this software business, companies would have to adopt something much more lightweight but more rigorous than the CowboyCoding.

I found XP to be it. I had a lot of skepticism when I tried to cling onto my Rational diagrams and BigDesignUpFront. But interactions with cards and customers convinced me that XP was the better way. I try to do everything XP now. I am introducing UnitTests at my current company. Is this what you wanted?

-- sg

Yes, it is. An observation, then a question. Observation: You moved to XP, by your account, because you saw what wasn't working and were looking for something new. You chose it based on gut feel, and then on the data that really matters, your own experience. I think that's how things really happen, not decisions based on data.

Question: How do you know SteveMcConnell makes decisions based on empirical data? Because he asked for some about XP? That doesn't convince me. And I don't see a lot of empirical data in his (very wonderful) books. So ... why do you say he bases decisions on empirical data? Thanks!

We might also wonder about the adoption of pair programming, for which there is some decent evidence. (I'd observe, though, that it's not like PP showed up as 10X better in the experiments, so maybe even if we were data-driven we'd not be convinced as yet.) Still think most of us are not data-driven, but I could be wrong. Anyone got any data on that?

How do you know SteveMcConnell makes decisions based on empirical data?

You've obviously never read CodeComplete. But I find it interesting that you are using hand waving arguments about how another person feels when there are clear data points that show the opposite, both in his writing and in his own testimonial. I don't think this speaks to the weakness of empirical data in general, but more to your own weakness of discriminatory selection of premises. I understand your argument that most people think this way, but that's just more handwaving. The circularity is cutting. -- SunirShah

As for what's obvious, I have in fact read CodeComplete. As for "hand waving arguments about how another person feels", I make no arguments here whatsoever about what people feel, hand waving or otherwise. I observe that in the majority of cases where I provide evidence in response to the request for empirical data, all it does is uncover another objection. And I respectfully suggest that you review BowToYourFellowPractitioners, and adjust your tone of discourse.

Let's keep the discourse civil. I agree.

-- sg

The opening posting suggests that a request for empirical data is not always what it seems, especially when presented as a first objection. It points out that one way to find out if it is a real objection is to ignore it and see if it comes back, and it points out that another way is to suggest and experiment and see if the answer is "I already know ...". It suggests that when you do get the empirical data, you'll likely just find another objection, and that you should keep on looking for new ones. I fail to see why the posting has generated so much heat. -- RonJeffries

What I wonder about is what happens when XP advocates are presented with empirical data that contradicts their rhetoric. I was asked on UnitTestsReconsidered (specifically BugsInTheTests) to present empirical evidence of my claims, so I did (using CodeComplete no less!). The response from the XP advocates was at best confusion, sadly. I was hoping for more, like how to deal with the semantic bug problem.

Nonetheless, when presented with empirical data that disputes XP's rhetoric, XP advocates must respond by either refuting the empirical data's source, producing more empirical data to the contrary, changing their rhetoric, or correcting their methodology to fit with the new information.

Unfortunately, the XP advocates involved chose to change the rhetoric (see XpVsStandardDefinitionOfUnitTest). This is the weakest of all responses as it suggests that a) you don't know what you're doing, b) a) you don't know why you're doing it, c) all you can do is hand wave. But I know you know what you're doing - though I doubt you know why - and I definitely think you can do better than hand wave. Anecdotal evidence would be far better, for instance.

I would have been much happier with practical solutions to the problems I face. I don't care whether you call the tests drivers or contracts or whatever, there will still be BugsInTheTests. -- SunirShah

I didn't mean the above to be inflammatory. Maybe a little disappointment. I was trying to suggest constructive ways to respond, rather than the current path chosen. If you can clean it up, bonus. If you can't, delete it. By the way, I was careful to suggest it was only the local response that was flawed, and not XP itself. I like XP; I don't like "XP advocacy" in the sense that I like Linux; I don't like "Linux advocacy". -- ss

In my observation, there are TooManyVariablesForScience. Thus a different approach needs to be taken. But note that if one demands empirical evidence for A, you can also demand empirical evidence for the alternatives to A. It works both ways. In the end comparisons are usually not easy and simple (barring a show-stopping flaw), and if one expects them to be, they are usually naive.

CategoryEvidence