Defect Tracking

Here's how to use a tracking system like JitterBug or BugZilla:

Anyone associated with the project, including its end users, are permitted to add bug entries. You cannot "get in trouble" by adding the wrong one.

But, contrarily, anyone permitted to add bugs must have a unique account (their e-mail address), and instructions to use the Query system to seek their bug before adding it. A Bug can be marked as a Duplicate of another Bug, by number.

All initial entries must contain three sections, without exception: The "Steps to Reproduce", "Observed Behavior", and "Expected Behavior". A successful bug entry helps programmers reproduce the exact problem as closely as possible, and this format forces the bug report author to focus on "just the facts" and not say things like "well, the engine makes a funny noise under load..."

The bug author sets the "Severity" rating, from "Blocker" thru "Critical" to "Enhancement".

Use this life-and-limb scale:

Each bug comes with a set of attributes, such as the product and owner fields. Set them as best you can - on a large modularized project with a conjoined team there should be much overlap.

Executives review the bug stream, adjust the details, owner & severity, and then set the Priority, from P1 at highest, to P5 meaning "after the Sun burns out".

In a supermarket, eggs may come in "Medium", "Large" and "Jumbo", but your bugs' average Priority should be "P3".

It cannot be emphasized enough that the Priority rating is the executive's single most important "knob" on the team's activity. Do not direct programmers via screaming e-mails, and do not make them try to rate who is screaming loudest about what. Set the Priority such that each team of programmers sees a collated stream of feature requests, in order by BusinessValue.

For best results, give that knob to a real customer. Don't make >them< scream loudest at executives just to get things done.

Executives should not play god and add a "time estimate" to any bug report. There is simply no benefit whatsoever to this uniquely insulting and damaging practice. Anyone who estimates the time a bug should take must be prepared to fix the bug within that time estimate. And proper use of a bug tracker makes real project time estimates very easy.

Each bug has a Resolution and a Status. A bug goes from NEW and VALID to ACCEPTED to FIXED or INVALID. Programmers start by trying to reproduce the bug; if they can't they mark it WORKSFORME and take no further action until receiving clarification.

At whim a programmer may add a note to the system and set the ownership to another programmer. You cannot "get in trouble" by sending the bug to someone else, because they are expected to know how to at least send it back.

Your default Query into the Bugzilla system should display your current assigned and non-CLOSED bugs sorted by Priority, sub-sorted by Severity. Your job under this system is to walk the list from top to bottom. Don't pull items at whim, or based on what looks most fun, and never do more than one item at the same time.

After reproducing the bug and witnessing its symptoms, set the state to ASSIGNED, then research each symptom until you can write an automated test that will only fail if the symptom is alive, and only pass if the symptom is dead. See CaptureBugsWithTests

For each symptom, and afterwards for the whole bug, ask each question in this list:

Collaborate and work the system until good answers are built-into the code base.

There's an old joke about a man sawing on a tree. A passer-by comments the saw's blades are very dull. "Yessir, they are dull." "Maybe you should take time to sharpen the saw before.." "No, sir! I don't have time to SharpenTheSaw. I'm too busy cutting down this tree."

Take time to ensure your tools are adequate for the job, and as you fix make sure you leave code quality and clarity better than when you found it. Do not accept any module's current bad state as an excuse for a fix that violates good design rules, such as "short methods". That bad design is among the forces that created the bug. Try to avoid the need to ever visit this module again!

Comment the new tests and code with the bug's URL. Then integrate the fix with your team's code repository, and comment that check-in with the bug's URL.

Add a commentary to the bug report including the fix's technical details, modules affected, and answers to its question list.

Run all your modules' tests every 1 to 10 edits, and run your entire project's tests at each critical juncture during the process.

Mark the bug FIXED, and return it to either its owner or a Quality Control assignee.

The new owner will pull each daily build, then automatically and manually test that the bug is dead. If they are satisfied, they mark it CLOSED.

The bug tracking system can name and store queries. Executives should build a small suite of queries that examine things like each programmer's current load, or any orphaned or abandoned bugs.

To track, executives should count the number of bugs and feature requests fixed in each iteration (1 to 3 calendar weeks). Don't change iteration length in the middle of a project just to provide the illusion of velocity. But that number of bugs fixed is a very accurate way to predict future performance. Nobody should enter whopper bugs such as "play chess", so an even granularity of bug sizes should help provide a very stable ProjectVelocity.

In a discussion last week about the importance of defect tracking, I was referring back to the point that GeraldWeinberg makes in QualitySoftwareManagement about the distinction between failures (observable problems with the software's behaviour) and faults (the underlying problems in the code). A single fault can certainly cause multiple failures; likewise, I suppose, an observed failure might be tied to multiple faults.

I think Weinberg goes further in that he distinguishes a System Incident (an unexpected piece of behaviour in the system) that may or may not have its source in a Fault (i.e. it may be a feature and training might be the solution).

In the ideal case, your DefectTracking system would actually correlate the two, so that as you went through and marked faults resolved, it would map back to the failures that should have been resolved.

The question is, does such a DefectTracking system exist in the real world? -- BillBarnett

It would probably be sufficient to have a system that allows to mark a problem report as a duplicate of another. Less administrative overhead (i.e. you don't have to write a separate "fault description"

Perhaps we should separate these two ideas:

DefectTrackingPatterns

It makes sense that a development team might use two separate systems, one (issue tracking?) to track the status of (and record information about) failures/system incidents. Ideally, multiple incidents of the same basic issue would be logged as such (rather than as a bunch of separate incidents). External users might need visibility into this database. Once a system incident has been diagnosed as related to a fault in the code, it moves into defect tracking, whether in the same or a separate system. Defect tracking is almost an internal workflow for the development team, tracking the state of a fault (who has it been assigned to, has a fix been applied, has it been tested, approved, implemented, etc).

The trick would be to allow the system(s) to correlate between faults and their associated system incidents.

Has anyone seen a system (or set thereof) that provide this functionality? -- bb

Bugzilla allows bugs to be marked as duplicates of other bugs, putting a reference to the "first" bug in any duplicates. Often in the Mozilla project's bugzilla base you'll see entries saying "This bug has been marked as a duplicate of 65121". You can also mark bugs as "dependent" on others, such that you can't close a bug until all its dependences have been closed. Using the terminology of failure and fault from above, with diligence, it would be possible to create a "fault" bug as dependent on the bugs for each tracks "failure" incident. Not the other way around, though it might seem logical at first glance, because you want to ensure that the fault doesn't get marked resolved until all the failures are addressed, or you might discover that a failure isn't caused by the fault you thought it was, so you can move the dependency to the correct fault. --StevenNewton

[See BugTrackingSoftware] [CategoryDebugging]

EditText of this page (last edited June 5, 2005) or FindPage with title or text search