No Bug Database
No Bug DatabaseWhy do we want to save bugs anyway? Why not just design our processes so that we fix them as they are found? This would probably require that we continually test as we code and that we give bug fixing a high priority. We'd probably also need to make sure that we don't create lots of bugs as we work. We'd need clean code and regression tests for that. This doesn't sound too hard.
We maintain our bug list in a spreadsheet. Some bugs are low priority (low impact) AND hard (could take weeks to track down and fix). We can't give them the priority to fix them immediately.[We have 40 known bugs, and estimate that there is a man year in clearing them all]. If we didn't track all the bugs somewhere we might lose track of them. Bugs cleared from the bug list may then get tracked in a higher-level document that looks for patterns in bugs. We use this higher-level document to improve process. For NoBugDatabase to work well you have to be able to give bug fixing top priority and to have an excellent process for preventing and uncovering bugs already in place. -- JamesCrook
JoelSpolsky has some interesting comments on this - writing code takes known time, fixing bugs takes unknown time, so fix bugs as you go so you have a better idea if you will make deadline. He also advocates daily builds and tests.
JoelSpolsky also runs a company which sells bug databases.
If you follow his advice, you probably won't need his product. If you fix bugs as you go, the bug lists tend to be shorter.
My former boss had some very interesting comments on this - writing code takes known time, fixing bugs takes unknown time, so write code that doesn't have bugs in the first place. That's... almost his exact words, except he was more verbose. I am not making this up.
Writing bug-ridden code takes known time, writing bug-free code takes unknown time (perhaps forever).
Not if you CodeUnitTestFirst, or, alternatively, use do CleanroomSoftwareEngineering.
BugsAreJustStories, so you don't need a separate bug database during development.
Your mileage may vary. JackBolles at ThoughtWorks described his team as having a single issue tracking system containing feature tasks and bugs.
The team at EvantSolutions has a bug database (BugZilla) run in parallel with the tasks. During a run-up to a deployment release, explicit developer time was scheduled to target specific bugs.
A common question from the acolytes: "How do you serialize the problem/issue/suggestion reports from the field?"
The answer is "Running a database is well within the core competency of the civilians upstream from the OnsiteCustomer. The latter is the only gateway to the actual development team. This is a good thing, not elitism, because the customer practices WriteItOnaCard."
If the acolytes then ask "How do you store & dispense the team's own bug stream?" the answer is "Have you actually experienced a full-bore XP project?"
Have you actually experienced a XP project?
Unit testing will not find threading conditions, it will not find starvation due to unexpected loads, it will not find out of memory because of unexpected loads, it will not find weird interactions of failed hardware or mis-specified hardware, it will not find the things you actually did not think about testing, it will not find the often-crazy things users can do, it cannot find bugs in your OS and other tools in non-standard conditions, it will not find the things that you did not handle but people actually do, etc. For this you need bug tracking so customers, developers, manufacturing, QA, the field, marketing, et cetera can all share the same information.
UnitTests will not find bugs. But then, neither will your usual "bug database".
On the other hand, tests can be written to expose threading problems (see ExtremeProgrammingChallengeFourteen) or inadequate performance under high loads, to simulate faulty hardware or crazy users, and more generally to serve as an "executable bug database".
These tests are nothing compared to real-life scenarios or even those generated by good systems tests. Only someone with a very simple system finds the challenge at all useful.
I shouldn't mind being "someone with a very simple system". As long as it does what it's supposed to...
The idea of CaptureBugsWithTests isn't limited to UnitTests. If the defect lies in some strange interaction between system components, by all means capture it in a system test.
The main issue is this : if you don't have a test that exposes the bug, how are you ever going to know it's fixed?
An excellent question. That is why every time someone finds a non-trivial bug which your UnitTests would not expose, you should capture it in a RegressionTest and use that to verify that it's fixed. As a bonus, you can continue to run the RegressionTest forever and be automatically alerted if the bug ever comes back.
I think everyone would agree that there is a need for a means of communication that is more focused than email, but general enough that bugs are only a part of what is communicated. Most of the bandwidth would be consumed by feature wishes, customer service calls - due to bugs, of course, but also due to customer error -, and issues not even related to the software end of things.
Maybe people are taking issue with the name, bug database. Something more along the lines of product journal would better capture the flavor. This tool would need to be as simple to use as email (simpler? ;) - to make sure that it does get used. Something as simple even as WikiWiki. In fact, now that I think about it, WikiWiki would be a good step in the right direction. -- BenSharp
Err, no. NoBugDatabase is a dumb idea. -- WillSargent
I second that. This is probably one of the most ridiculous suggestions that I've seen on Wiki. Do any people suggesting this actually work on software that thousands of people use in various versions across multiple years in the field? Well, I do... and having NoBugDatabase in this situation would be a disaster. -- JohnPerkins
What about some sort of versionable test suite? Suppose that you are on version 5 of your product and a bug is found in version 3. Place a test in version 3 of your test suite to expose the bug. All subsequent versions of the test suite should automatically pick up the new test unless the test is manually removed. -- MarkAddleman (You are assuming that the product was developed recently under the methodology that there are already unit tests for everything.)
More talk of nirvana. Without a database...
Playing DevilsAdvocate: I don't see why bug tests need to be UnitTests. With that in mind...
A QA department often has their own methodologies for testing and their own tools. In some cases, there are additional third-party auditors that ensure quality and they will not just do whatever you tell them (as well they shouldn't).
In short, working on a large-scale project that involves geographically and philosophically and motivationally diverse groups of people amplifies complexity exponentially and demands formal organizational and communication processes beyond what I think the business process part of XP can handle. XP is great, don't get me wrong. But you can't shoehorn every project into it. There is NoSilverBullet.
-- JohnPerkins
I believe strongly in unit tests, and when I took my present job I instituted a policy of unit testing. We now have more than 330 unit tests... but we still need a bug database. -- JohnPerkins
I'd be interested to know - how many bugs in that database? Did you by any chance track that number against the number of unit tests over time?
We didn't keep statistics, but there have been very few bugs on the new code that we write, all of which has unit tests. The quality increase has been dramatic. Most bugs have been conditions we didn't know about to write unit tests for, or, mainly, integration issues with legacy code or systems. -- JohnPerkins
So, do you think you would still need a bug database if you were working on a project that had unit tests and acceptance tests right from the start?
If you read some of my other discussion on this page you will understand when I say, "Yes, definitely." It would be a smaller database in terms of number of records, but that is not the point. On very, very small projects that are driven almost entirely by engineering, limited to a single geographic location and do not have to leave an audit trail for legal or other reasons, you would be able to get away with using index cards. But those types of projects are usually internal tools or skunkworks projects. Once you start doing business with big business, you need to utiltize all the communication tools properly, one of which is a bug database. -- JohnPerkins
Having no bug database is the brutal implementation of the FixBugsFirst principle. Having a bug database helps to make more sensitive decisions but open doors for endless discussions on what priority some bugs deserve. Only a mature organization can afford such discussions. -- JeanHuguesRobert
The thought of having NoBugDatabase is probably very silly idea for most projects of any appreciable size. However, the process cults quite often turn the sensible notion of a bug database into an obscenity. The perversion of integrating the debug database with a source code management system, and forcing branches based on "defect number" has become a common abuse.
Clarify for me why this is "obscene, perverted abuse", please. According to a number of sources, these are good practices (well, maybe not the branch per defect). See for example, "Streamed Lines" http://acme.bradapp.net/branching.
Since this would be a tangent, and one that is likely to spur an entire discussion of its own, a clarification is at SourceCodeRevisionTrackingBranches.
Here are some reasons why you need a bug database:
I think that bug databases can be very useful. I work for a GamesDeveloper? and developing games is at times like developing one enormous UI where the users can be expected to do anything.
Our bug database contains not only bug descriptions, but detailed reproduction steps, screenshots, crashdump data, movies, logs, callstacks, even references to code (in the case the tester knows a little about that). Putting that data on a CrcCard would be impossible. Putting it in a Wiki would be possible, but then it's just another kind of BugDatabase. Many bugs that turn up could probably have been avoided with UnitTests, (unfortunately we don't XP yet), but a lot of them would be hard to even write an AcceptanceTest for.
After reading the comments below, I need to add that our bug database is actually managed and owned by the customer, and they prioritize and track the bugs, so you could say it's a sort of UserStory db.
It would be nice to hear more about how other teams treat and track UI and usability bugs.
On a separate issue, how does one track abstract issues in XP, such as the "fun" factor of games? We're constantly tuning code and data to make our games more "fun". Writing an AcceptanceTest for that seems quite tricky. Tracking issues and comments in a db doesn't sound all that wrong to me. -- AndreasAxelsson
RonJeffries wrote on the XpMailingList, on the utility of using a bug database for metrics over time:
What does it take in order for a database to be better than people talking with each other? I suggest that it can't ever be.
What does it take in order for a database to be a desirable support tool for people talking to each other? I suggest that the answer is "lots of data". Not the only answer but if you throw the ball in the direction of "lots of data" you'll be close to the whole answer.
To the extent that "lots of data" is central to the utility of the database, then you have to have lots of bugs, and/or lots of issues.
I am here to tell you that there is no legitimate approved authorized good way to have lots of bugs.
If you have lots of bugs, I'm sorry, but your process has been letting you down for a long time. You're screwed now, of course, and definitely need f5g Oracle to support the demand for explanations, but I adjure you by the living God not to go forward into new projects with this plan. Your defects are a plague upon the land second only to the locusts and maybe the ash borers.
If you have lots of issues, we can look a bit more deeply and perhaps more mercifully. Who owns these issues? Is the database owned by the customer, used to manage and study the myriad inputs from the world-wide customer base, so as to make good decisions about what feature to schedule next? Well, maybe OK then.
Or does the issues database belong to the developers? Is it a list of things that someone has asked them to do, and that someone now thinks is or ought to be "in the queue"? Then, again with deep regret, I'm here to tell you that your process has been letting you down. If there is a difference between what you are doing and what your stakeholders are expecting, especially if that difference is widening, then the database is not serving you.
Here's the metric to look at: Draw a graph of number of unresolved issues in the database over time. If you can't resist, graph the resolved ones on the same scale. Is the number of unresolved issues trending up? Then Something Is Wrong.
Why is that, you plaintively or angrily ask, depending on whether you are merely confused but interested or angered by the vicious attack on your favorite database which you have sworn to protect to your dying breath?
Something Is Wrong because the decision process is letting us down. Our backlog is growing at a pace beyond our ability ever to catch up. Our decision process needs to resolve those issues in a timely way. That means it needs to identify the issues which are not going to be addressed and kill them.
I would argue, and not just for the sheer joy of it though I do so love stirring the pot, that no matter how big your organization, whether it is just you, or just you and the rest of MegaCorp?, if the unresolved issues database is growing rapidly, something is wrong.
Oh sure, there need to be new things on the list of things to do. We need to collect information for our next release or our next product or the next company we plan to acquire and gut for its brain power and customer base. But to the extent that it's a Repository For Things Not Done And Never To Be, the database, whether it is the stack of cards on my desk or the rooms full of disk packs at MegaCorp?, it's a collection of things most of which are dead and which should be buried.
Yours isn't like that. Yours is a powerful communication and memory aid that you use frequently to guide your careful and effective steps into the future.
-- RonJeffries
One thing is for sure, and that is that people are imperfect, mistakes are unavoidable, unit tests are only as good as the unit tests (which may have been flawed), oversights get made, and there will always be "bugs". Calling a stack of story cards by any other name is still issue tracking. It's valid, as are other methods of issue tracking. Some issue tracking tools make it easier to share, locate, and reference issues however. I agree there is no substitute for talking to one another, but conversations cannot be searched or easily referenced later, stored in a central location, etc. The XP methodology uses stories, so use stories to track issues. Nothing wrong with that. It is still issue tracking though.
I'll stop using my Bug Database if I find a bug that turns out to be real nasty and I want to fix it right away without talking to anyone. Then I'll log it after it's well under control. Otherwise a bug report on that kind of bug before it's under control will generate too much panic. So why have a bug database? If it's just more extra reporting that doesn't serve the programmer?
Note: when it's a real bad bug, I don't want to talk to anyone. No-one can help. I am the only one who can fix it in the finite amount of time we have to spend on it. "Helpers" would slow me down. So your XP process is just so much extra baggage for me to carry as well. Where XP is desperately needed is up front, on the "design end", stop them from designing please. XP is not needed on the back end fixing bugs - no process is. Hate to break it to you. PM is such BS.
I'm surprised that nobody has mentioned compliance (as in Sarbanes-Oxley) as a reason for having a bug database. Not that it constitutes the only reason to have one, mind you... -- RonPerrella
See JitterBug, BugZilla, CaptureBugsWithTests, DefectTrackingPatterns, SheChangeDesignInTheDatabase, TaskDatabase
EditText of this page
(last edited April 15, 2014)
or FindPage with title or text search