Project Lore

ProjectLore is more than just the architecture of the system. A project's lore answers all the funny questions that may have nothing to do with technical quality or the marketing profile of the system ... questions like: 'Why didn't we use the JewelRock? database when it does everything that our database can, only better' (the lore says something like: 'JewelRock? was only in beta when we went shopping and has matured faster than our database' or 'JewelRock? maintenance contracts are just too damn expensive'). The ProjectLore is usually only in the heads of the people that have been around for a long time and, when they leave, it goes with them.

Do we need any of this stuff? I think we do. For one thing it stops the new guys bugging the old guys (I must have asked the JewelRock? question a dozen times since I started my new job and only found someone who could give me an answer a few days ago - turned out to be the cost of the maintenance contract). More importantly, it is the lore, particularly from the early days, that shapes the system's architecture and, to some extent, shapes the developers and the development team. In order to really understand the architecture and the team I think you need at least some appreciation of the lore.

-- PaulDyson

See also OralTradition, CommunityLore.

Beautifully said. Now. How do you communicate lore:

Paper documents? (TechnicalMemo)
Wiki (or its poor cousin, Notes, which suffers mightily because its source code will not fit on a single sheet of A4 paper)?
Stories around a campfire?
Videotape?
Folk songs?
Interpretive dance?
Theatre?

Think carefully about short and long term costs and benefits before answering. Think also about the probability that someone with a deadline will actually use the chosen medium.

-- KentBeck (I knew it. -- RonJeffries)

Personally I would like to videotape people dancing, singing and telling stories around a campfire ... but I think that fulfils desires which have nothing to do with capturing the lore better :-). Technically, we have used memos but not to good effect (few memos actually get written but, even so, our \document hierarchy is so big and complex, and the searching mechanism so poor, that it is very hard to find anything at all which leaves you in the dark as to whether you can't find it or it's just not there). I've suggested we build (or borrow) a very small Wiki, partly as a way to allow people to take a more direct part in writing the architecture overview (see KeepItInTheirHeads). However, I don't think this is a technical problem, more a cultural one. It doesn't take anyone, even on tight deadline, very long to write a couple of paragraphs on 'why we rejected JewelRock?' ... but they have to think 'I need to do that' and people who wonder about the question need to think 'I need to look here, here and here before I go bugging people'. So whether we build a Wiki, tell stories, or construct epic poems, the people who have something to say need to remember to say it, and the people who want to know need to remember to listen. -- PaulDyson

Project lore? I have to question the value. Don't forget your Shelley:

... And on the pedestal these words appear:
"My name is Ozymandias, king of kings:
Look on my works, ye Mighty, and despair!"
Nothing beside remains. Round the decay
Of that colossal wreck, boundless and bare
The lone and level sands stretch far away.

-- PeterMerel

It's always nice to have an apt QuotationOfDoom?, but are you really suggesting that answers to questions like "Why didn't we use JewelRock?" should be lost? -- RonJeffries

Don't bring me down ... -- ElectricLightOrchestra?

Guilty as charged :-) What I mean is that software, for all our vanity about it, is impermanent. "How we got here" is a lot less valuable than "Where we are now" and "Where we can go next". That said, I actually have a good solution for this; version all your project documentation alongside the code. I'm using CvWiki to do this at HP right now, and use CvWiki's WayBackMode to explain why decisions were made when decisions were made.

So as someone who is able to do it ... it's kind of eery just going back a few months and looking around ... spooky, like wandering through a graveyard. Spooky fun, but the value remains unclear to me. Perhaps in C3 you could use it to say, "don't bother us now, we're Xtremely busy Programming; just go back in time and read this."

Actually we never say not to bother us, since Communication is one of the XP Values, and we live by oral tradition. But sometimes it'd be nice.

Most of the team write things on cards, and throw them away when done. Exceptions are:

Production Support Notebooks, logging all events;
System Performance Notebook, logging pay performance vs changes;
My personal notebooks: I write everything in them, rarely anywhere else, except of course in CRC sessions.
Project administrator tracks stories, tasks, time to complete, etc. -- RonJeffries

-- PeterMerel

On C3, ... boring ... with our very lightweight approach to documentation (Don't Do Any), I've felt the need for some history that is hard to reclaim. Examples:

Why do you use Smalltalk?
- We've answered this a million times. Should have a document.
How many stories / unit tests did you have in Iteration 5?
- We wind up wanting to know some odd fact we didn't collect.
  - (Some, perhaps most of these, are things we knew we should collect and in the press of time, do dah do dah.)
What about performance?
- Two million times.
What was your Commitment Schedule in June 97?
You never told me you needed resources.
- Put that in writing, every time.

Sometimes we wish we had kept the cards. But I wouldn't do MUCH more, because most of it would have been wasted. Maybe just throw all the slides and memos in a bin. -- RonJeffries

Surely you should follow the general XP principle ThreeStrikesAndYouAutomate.

When asked the first time, answer the question.
When asked the second time, wince at the repetition, and answer the question.
When asked the third time, assume that this will go on forever and produce a document.

Works for us. Always note down the questions as they get asked so you can see the common ones, and document them.

I've heard the 'don't worry about where we've been, just where we are and where we are going' argument a number of times and I never find it that convincing. In order to understand where we are I think you have to appreciate where we have been: 'we use object rather than relational because...' (sound technical reason, possibly captured in the architecture), 'we use this object DB rather than that one because...' (sound, totally non-technical reason, not captured anywhere).

You also need it when you look to the future. About a year ago we had to change object DBs because the one we had just didn't support Smalltalk properly (ObjectStore). So we had to look at JewelRock? again. The first thing we did was see if the maintenance contract were still too expensive - they were and we saved a lot of wasted effort evaluating the thing. This happened just before 2 out of the 3 people who knew the answer to the 'Why didn't we use JewelRock??' question left, taking the answer with them. If all three had gone before we replaced ObjectStore, who would have known to check the maintenance costs first? And, without a culture of capturing the lore, why would any of the three think to mention this before they left? -- PaulDyson

For large or long projects, should there be a standing position in the tribe for a storyteller (someone to collect and maintain ProjectLore) -- WayneCarson

Choices that were made are documented by the choices themselves. Choices that weren't made (such as 'Why didn't we use JewelRock??') are the ones that have to be documented separately, because they don't leave any kind of a DocumentationTrail?. -- CurtHagenlocher?

You betcha. TheRoadNotTraveled. Know it well.

Often, ProjectLore can't be written down because it is too embarrassing. Examples:

"That component was thrown together in a QuickAndDirty? way at 6:45 AM one morning so that we could give an impressive demo to the company president at 8:00, and we never had time to do it right."
"We didn't choose JewelRock? because the ProjectManager vetoed it, and nobody had the guts to argue with him."
"Nobody would help Jeff with his code after he said what he did during that meeting."
"We didn't exactly ask the customer for approval on that feature."
"Sandra was going through a divorce during the last few months of the project, and her work suffered."

Sometimes, the only way to learn these things is over a few beers. They will never be recorded, to protect the innocent and the guilty.

And so it is lost? A Story Teller can grab this lore and record it in a way that infuriates no one, reveals not the guilt associated with Bad Behavior, and yet captures the essence of what the driving forces actually were.

In this case the Story Teller needs to be a mixture of technical writer, engineer, and political analyst. This is a highly valuable asset to the team, but good luck finding the right person.