Xp Evaluation Framework

XpEvaluationFramework version 1.4

Well, some time has gone by and we've succeeded in attaching numbers to some of the effects of XP. We made a framework that helps measure in a consistent fashion and have completed several studies.

You can read more about XP:EF at NC State's website http://www.csc.ncsu.edu/research/tech/reports.php.

How to gather XP:EF metrics: TR-2004-18 Extreme Programming Evaluation Framework for Object-Oriented Languages Version 1.4 (ftp://ftp.ncsu.edu/pub/unity/lockers/ftp/csc_anon/tech/2004/TR-2004-18.pdf)
How to gather RUP EF metrics: TR-2005-46 Rational Unified Process Evaluation Framework v1.0

Krebs, Williams, Ho, Layman (ftp://ftp.ncsu.edu/pub/unity/lockers/ftp/csc_anon/tech/2005/TR-2005-46.pdf)

About the XP:EF framework: TR-2004-2Toward a Framework for Evaluating Extreme Programming

- Example Studies:

TR-2004-3Exploring the Use of a "Safe Subset" of Extreme Programming: An Industrial Case Study
TR-2004-8Exploring Extreme Programming in Context: An Industrial Case Study
TR-2004-23Motivations and Measurements in an Agile Case Study (ftp://ftp.ncsu.edu/pub/unity/lockers/ftp/csc_anon/tech/2004/TR-2004-23.pdf)

Also see a LucasLayman page at http://www4.ncsu.edu/~lmlayma2/xpef.html.

BillKrebs

I want to be agile, and yet get the good out of the spirit of ISO or CMM. I don't just want to pass the audit, I want to do the best practices, show that I've done them, and shown what they've bought me.I want the best of both worlds.

Why measure anything? I believe the success stories of XP, but I need numbers to convince people paying the bills to let me do XP.That's not all bad, because having the numbers may help me perfect my use of XP.

My small team at IBM has worked in conjunction with Laurie Williams, Dr. Anton, and Lucas Layman from NC State since 2002. We've come up with this measurement framework folks can reuse if they want in order to replicate our study.

The metrics to do this must themselves be lightweight. We need data on small informal teams, and they are the ones that don't have full time metrics or process specialists. I've tried to do the simplest thing that could possibly work, refactor, and iterate the metrics.The trick is measure enough stuff to be useful, but measure as little as possible and keep it simple so busy folks will actually do it.

It's called Xp Evaluation Framework, or XP:EF for short (pronounced 'ekspef'). XP:EF has three parts: Context factors, Adherence Metrics, and Outcome Metrics. It tries to be

unbiased - the framework does not assume agile is better than plan driven
concise - keep just the key metrics. Not everyone loves process and metrics as much as we do, so they don't have patience for long surveys for forms.
measurable - have concrete ways things can be measured
complete - cover the key areas of our agile development process, but in the shortest simplest set
tolerant - Admit there are several ways to do things
respectful - do not water down the vision from the creators of a given process
sound - provide proper statistics that serve the needs of academic researchers
repeatable - be specific enough on how to measure the way we did incase someone wants to replicate our study.
sensitive - allow reports to not include data that may be proprietary to the organization

We've tried to bridge the gulf between informal agilists and CMM ISO folks to find a happy middle ground. You can read about the resulting Xpef in North Carolina State University Technical Reports 18 and 20 which can be found at http://www.csc.ncsu.edu/research/tech/reports.php. I hope to work on a set of more readable power point slides in November 2003.

The goal is to synthesize great ideas on metrics from diverse sources, apply a dose of yagni and refactoring, and allow but not mandate that people use this framework. I plan to continue to use it on my projects.

Think of XpEf as open source for process improvement metrics. You can start with the context, adherence, outcome structure, measure a few common 'required' items, choose from some optional ones, and add your own custom metrics that appeal to you. The goal is to create a body of knowledge for XP to help better people justify it's use to researchers or stakeholders that (sometimes rightfully so) rely on numbers.

Please let me know if you have feedback or suggestions. BillKrebs

Bill, LaurieWilliams, LucasLayman, and PekkaAbrahamsson? have been working on the XpEf over the past year. You can find a slightly-more extensive page on the XP-EF at http://www4.ncsu.edu/~lmlayma2/xpef.html. The page contains links to all the current XpEf publications as well as tech reports detailing the entirety of the XP-EF. -LucasLayman

We also have parallel evaluation frameworks for TSP and for RUP. The idea is that Context factors and Outcome measures are largely the same, and you can plug in various adherence metrics to match the set of practices for your process.

BillKrebs