Skip to content Skip to navigation

Connexions

You are here: Home » Content » Submitting Bug Reports

Navigation

Recently Viewed

This feature requires Javascript to be enabled.
 

Submitting Bug Reports

Module by: Warren Myers. E-mail the author

Summary: A guide to submitting good bug/error reports

Introduction

I am deeply indebted to Simon Tatham's excellent "How to Report Bugs Effectively" - most of this is in the form of direct quotations, or a condensation of what he has so aptly written. I have taken some rewording liberties to expand the focus of his work, and remove personal references, as well. I appreciate Simon's permission to adapt his work here.

"It doesn't work"

Give the creators of the tool a little credit for some basic intelligence: if the program really didn't work at all, they would probably have noticed. Since they haven't noticed, it must be working for them. Therefore, either you are doing something differently from them, or your environment is different from theirs. They need information; providing this information is the purpose of a bug report. More information is almost always better than less.

Many programs, particularly free ones, publish their list of known bugs. Check this to see if the bug you've just found is already known or not. If it's already known, it isn't worth reporting again as a new bug. However, if you think you have more information than the report in the bug list, you might want to contact the developers anyway to add to the bug report. The new information you have may enable them to fix the bug more easily/sooner.

Specific developers / companies / groups have particular ways they like bugs to be reported. If the tool comes with its own set of bug-reporting guidelines, read and follow them!

"Show me", or "Show me how to show myself"

One of the very best ways you can report a bug is by showing it directly to the developer - either in person, or via a remote support session (like webex). Demonstrate the exact thing that goes wrong. Let them watch you run the software, watch how you interact with the software, and watch what the software does in response to your inputs.

They [should] know that software like the back of their hand: which parts they trust, and which parts are most likely to have faults. Intuitively, they know what to watch for. By the time the software does something obviously wrong, they may well have already noticed something subtly wrong earlier which might give them a clue. They can observe [almost] everything the system does during the test run, and they can pick out the important bits for themselves.

This may not be enough. They may decide they need more information, and ask you to show them the same thing again. They may ask you to talk them through the procedure, so that they can reproduce the bug for themselves as many times as they want. They might try varying the procedure a few times, to see whether the problem occurs in only one case or in a family of related cases. If you're unlucky, they may need to sit down for a couple of hours with a set of development tools and really start investigating. But the most important thing is to have the programmer looking at the computer when it goes wrong. Once they can see the problem happening, they can usually take it from there and start trying to fix it.

If you have to report a bug to a programmer who can't be present in person (or via remote support tools), then you want to ensure they can reproduce the problem. If they can't, they may have to assume that there's a user error.

Report exactly what you did leading up to the error. If it's a graphical program, report exactly which buttons you pressed, and in what order you pressed them. If it's a command-line program, capture precisely what command you typed - this is where tools like `script` and `tee` can again come in handy!

Make sure the bug report has all the input you can think of. If the program reads from a file, add a copy of the file. If the program talks to another computer over a network, while you can't send a copy of that device, you can report what kind of system it is, and (if you can) what software is running on it.

"Works for me. So what goes wrong?"

If you report a long list of inputs and actions, and the developer(s) attempt to recreate the problem, but nothing goes wrong, then you haven't given them enough information. Possibly the fault doesn't show up on every computer; your system and theirs may differ in some way. Possibly you have misunderstood what the program is supposed to do, and you are both looking at exactly the same display but you think it's wrong and they know it's right.

Therefore, you must also describe what happened. Report exactly what you saw. State why you think what you saw is wrong; better still, explain exactly what you expected to see. If you say something like "and then it went wrong", you have left out a lot of very important information. If you saw error messages, then report, carefully and precisely, what they were - they are important! At this stage, the developer is not trying to fix the problem: they're trying to find it. They need to know what has gone wrong, and those error messages are the system's best effort to tell you that. Write the errors down if you have no other easy way to remember them; it's not worth reporting the program generated an error unless you can also report what the error message was.

In particular, if the error message has numbers in it, do report them! Just because you can't see any meaning in them doesn't mean there isn't any. Numbers contain all kinds of information that can be interpreted by the developers, and they are likely to contain vital clues. Numbers in error messages are there because the computer is too confused to report the error in words, but is doing the best it can to get the important information to you somehow.

At this stage, the programmer is effectively doing detective work - they don't know what's happened, and they can't get close enough to watch it happening for themselves, so they are searching for clues that might give it away: error messages, incomprehensible strings of numbers, and even unexplained delays are all just as important as fingerprints at the scene of a crime. Keep them! If you are using a Unix-like environment, the program may have produced a core dump. Core dumps are a particularly good source of clues, so don't throw them away. Make sure you put in the bug report you have such dumps; the development team may want them added to the case. Be aware that the core file typically contains a record of the complete state of the program: any "secrets" involved (maybe the program was handling a personal message, or dealing with confidential data) may be contained in the core file.

"So then I tried ..."

If you're just starting out, or even if you're a seasoned professional, why did you violate the first fundamental rule of troubleshooting: Stop, Drop, and Roll?

If you've violated that first rule, you may very well be exacerbating the problem rather than ameliorating it. There are a lot of things you might do when an error or bug comes up. Many of them make the problem worse. For example, maybe you've deleted all your Word documents by mistake, but before calling in any expert help, you tried reinstalling Word, and then running a disk defrag. Neither of those will help recover the deleted files, and between them they will probably scramble the disk to the extent that no Undelete program in the world will been able to recover anything. There might be a chance if the system is left alone.

Users like this are like a mongoose backed into a corner: with its back to the wall and seeing certain death staring it in the face, it attacks frantically, because doing something has to be better than doing nothing. This is not well adapted to the type of problems computers produce. Instead of being a mongoose, be an antelope. When an antelope is confronted with something unexpected or frightening, it freezes. It stays absolutely still and tries not to attract any attention, while it stops and thinks and works out the best thing to do. (If antelopes had a technical support line, it would be telephoning it at this point.) Then, once it has decided what the safest thing to do is, it does it.

When something goes wrong, follow the first law of troubleshooting! Immediately stop doing anything. Don't touch any buttons at all. Look at the screen and notice everything out of the ordinary, and remember it or write it down. Then perhaps start cautiously pressing "OK" or "Cancel", whichever seems safest. Try to develop a reflex reaction - if a computer does anything unexpected, freeze. If you manage to get out of the problem, whether by closing down the affected program or by rebooting the computer, a good thing to do is to try to make it happen again. Programmers like problems that they can reproduce more than once. Happy programmers fix bugs faster and more efficiently.

"I think the tachyon modulation must be wrongly polarised"

It isn't only non-programmers who produce bad bug reports. Some of the worst bug reports ever written come from programmers, and even from good programmers. Take the example of the programmer who kept finding bugs in his own code and trying to fix them. Every so often he'd hit a bug he couldn't solve, and he'd ask a colleague over to help. "What's gone wrong?" they ask. He replies by stating his current opinion of what needed to be fixed.

This worked fine if his current opinion was right - it meant he'd already done half the work and together the problem could be solved. It's efficient and useful. But quite often he was wrong. Work could go on for some time trying to figure out why some particular part of the program was producing incorrect data, and eventually the discovery is made that it wasn't, that the investigating was in a perfectly good piece of code, and that the actual problem was somewhere else. There's wasted time.

Would you do that to a doctor? "Doctor, I need a prescription for Hydroyoyodyne." People know not to say that to a doctor: you describe the symptoms, the actual discomforts and aches and pains and rashes and fevers, and you let the doctor do the diagnosis of what the problem is and what to do about it. Otherwise the doctor dismisses you as a hypochondriac or crackpot, and quite rightly so. It's the same with programmers. Providing your own diagnosis might be helpful sometimes, but always state the symptoms. The diagnosis is an optional extra, and not an alternative to giving the symptoms. Equally, sending a modification to the code to fix the problem is a useful addition to a bug report but not an adequate substitute for one.

If you are asked for extra information, don't make it up! Don't skip any diagnostic steps you're asked to perform, because they will [hopefully] lead to a solution. Using your intelligence to help the programmer is fine. Even if your deductions are wrong, the developer should be grateful that you at least tried to make their life easier. But report the symptoms as well, or you may well make their life much more difficult instead.

"That's funny, it did it a moment ago"

Say "intermittent fault" to any developer, and watch their face fall. The easy problems are the ones where performing a simple sequence of actions will cause the failure to occur. They can then repeat those actions under closely observed test conditions and watch what happens in great detail. Sadly, too many problems simply don't work that way: there will be programs that fail once a week, or fail once in a blue moon, or never fail when you try them in front of the programmer but always fail when you have a deadline coming up.

Most intermittent faults are not truly intermittent. Most of them have some logic somewhere. Some might occur when the machine is running out of memory, some might occur when another program tries to modify a critical file at the wrong moment, and some might occur only in the first half of every hour!

If you can reproduce the bug but the programmer can't, it could very well be that their system and your system are different in some way and this difference is causing the problem. For example, a program whose window curled up into a little ball in the top left corner of the screen, and sat there and sulked. But it only did it on a user's 800x600 screen; it was fine on the developer's 1024x768 monitor.

The developers will want to know anything you can find out about the problem. Try it on another machine, perhaps. Try it twice or three times and see how often it fails. If it goes wrong when you're doing serious work but not when you're trying to demonstrate it, it might be long running times or large files that make it fall over. Try to remember as much detail as you can about what you were doing to it when it did fall over, and if you see any patterns, mention them. Anything you can provide has to be some help. Even if it's only probabilistic (such as "it tends to crash more often when Emacs is running"), it might not provide direct clues to the cause of the problem, but it might help the programmer reproduce it.

Most importantly, the programmer will want to be sure of whether they're dealing with a true intermittent fault or a machine-specific fault. They will want to know lots of details about your system, so they can work out how it differs from theirs. A lot of these details will depend on the particular tool, but one thing you should definitely be ready to provide is the version number. The version number of the program itself, and the version number of the operating system, and probably the version numbers of any other programs that are involved in the problem.

"So I loaded the disk on to my Windows ..."

Writing clearly is essential in a bug report. If the programmer can't tell what you meant, you might as well not have said anything. Be specific. If you can do the same thing two different ways, state which one you used. "I selected Load" might mean "I clicked on Load" or "I pressed Alt-L". Say which you did. Sometimes it matters. Be tersely verbose - explain everything you did, but use bullet points if possible, rather than prose form. Give more information rather than less. If you say too much, the programmer can ignore some of it. If you say too little, they have to come back and ask more questions.

Be careful of pronouns. Don't use words like "it", or references like "the window", when it's unclear what they mean. Consider this: "I started FooApp. It put up a warning window. I tried to close it and it crashed." It isn't clear what the user tried to close. Did they try to close the warning window, or the whole of FooApp? It makes a difference. Instead, you could say "I started FooApp, which put up a warning window. I tried to close the warning window, and FooApp crashed." This is longer and more repetitive, but also clearer and less easy to misunderstand.

Read what you wrote. Read the report back to yourself, and see if you think it's clear. If you have listed a sequence of actions which should produce the failure, try following them yourself, to see if you missed a step.

Summary

The first aim of a bug report is to let the programmer see the failure with their own eyes. If you can't be with them to make it fail in front of them, give them detailed instructions so that they can make it fail for themselves. In case the first aim doesn't succeed, and they can't see it failing themselves, the second aim of a bug report is to describe what went wrong. Describe everything in detail. State what you saw, and also state what you expected to see. Write down all error messages, especially if they have numbers in them.

When your computer does something unexpected, freeze. Stop, Drop, and Roll. Do nothing until you're calm, and don't do anything that you think might be dangerous.

By all means try to diagnose the fault yourself if you think you can, but if you do, you should still report the symptoms as well.

Be ready to provide extra information if the asked for it. If it isn't needed, it wouldn't be asked for. The developers aren't being deliberately awkward. Have version numbers at your fingertips, because they will probably be needed.

Write clearly. Say what you mean, and make sure it can't be misinterpreted.

Above all, be precise. Programmers like precision.

Content actions

Download module as:

Add module to:

My Favorites (?)

'My Favorites' is a special kind of lens which you can use to bookmark modules and collections. 'My Favorites' can only be seen by you, and collections saved in 'My Favorites' can remember the last module you were on. You need an account to use 'My Favorites'.

| A lens I own (?)

Definition of a lens

Lenses

A lens is a custom view of the content in the repository. You can think of it as a fancy kind of list that will let you see content through the eyes of organizations and people you trust.

What is in a lens?

Lens makers point to materials (modules and collections), creating a guide that includes their own comments and descriptive tags about the content.

Who can create a lens?

Any individual member, a community, or a respected organization.

What are tags? tag icon

Tags are descriptors added by lens makers to help label content, attaching a vocabulary that is meaningful in the context of the lens.

| External bookmarks