When encountering software issues, whether in the smallest of scripts, or in enterprise tools, is to Stop, Drop, and Roll. Yes, those same three words you learned from the fireman as a child for what to do if your clothes ever catch fire.

Famous last words in most cases are, "I know what I'm doing" - you may very well, but always guess first that you don't. This is not to insult your intelligence, but rather to remind you that everyone makes mistakes!

Were you able to successfully accomplish the task at hand before? Did the script run successfully yesterday, but not today? Can someone else run this correctly and I can't?

If the answer to these, and similar questions, is "yes", then you need to find out what changed between that last time you did this and now.

If you can undo any of the changes, does the tool work again? For example, if you are nearing your quota space but you delete some files, will it then run? If so, maybe you need your quota expanded. When the network file server is back up, does it run correctly?

If the first basic checks don't yield any useful data, it's time to start looking at the the output of the program. Does it create any log files when it runs? If so, are any error messages kicked-out when you run it? Can the log setting be turned up?

If the script does not generate its own log files, it's time to start generating them on your own. Many tools have commandline options that will increase verbosity - use them in conjunction with trapping the output using tools like script and tee.

Google (or Bing, Yahoo, DuckDuckGo, etc) is your friend when trying to find answers to technical problems. After checking the man pages for a given tool (if they exist), always take any error messages found to Google and see what may be found thereon. For example, an error kicked-out by Apache Tomcat is likely to be documented just by searching the error string on Google - and probably from the official documentation / known issues / publicly-viewable bug reports (eg http://tomcat.apache.org/tomcat-6.0-doc/api/org/apache/tomcat/jni/Error.html).

Effective searching is far more than "let me Google that for you" - it is knowing both what to look for and how to evaluate the results. Weak searching is easy: type in what you're looking for and hope. Like going to eBay and searching for "baseball card" - you'll get millions of results on just that query: not the most helpful thing in the world. Try "roger maris rookie baseball card", and your resultset drops to a manageable few dozen - those are results you can pick through effectively. Similarly, searching for "OutOfMemory" will yield millions of hits, whereas "bea weblogic outofmemory java error jvm 1.4" may start to yield useful results in the top 2-5 links.

Becoming an Effective Searcher will translate over into a host of other arenas beyond mere troubleshooting: from finding photographs of an obscure HVAC part, to locating the best deals on travel, searching effectively is a skill that anyone can learn, hone, and benefit from in their daily lives.

Effective Searchers look at what they are trying to find, pick out the important parts (if it's an error message, the generic portion of the message, not the specific-to-the-instance portion), and start looking, refining as they go.

To become an Effective Searcher does take some practice, but there is no better time than the present to start. Learn your favorite search engine's advanced features. For example, with Google I can search just on apache.org by adding site:apache.org to my search. Quoting the text of what you want to find will also tend to push references of it higher.

A drawback to using search engines, though, is that since they try to index everything, you can get a lot of results that are other people asking more-or-less the same question you are trying to find the answer to. That's great if an answer was posted, but when the responses are a lot of "me too"-types, it can be frustrating.

Part of becoming an Effective Searcher is learning to identify authority in resources found: "I had a paper to write several years ago on comparing AMD’s x86-64 architecture and Intel’s IA32 architecture for the companies’ CPUs. Sources like Tom’s Hardware Guide were helpful to see real-world comparisons between the competing products, but the true sources of authority on the products were AMD and Intel themselves. I printed large chunks of the manufacturer’s technical documentation to backup conclusions I made in my paper... Authority of sources isn’t assured by just one factor – author, publisher, host, length, etc – but rather by directly linking to the data used to produce the conclusions made by that source. No resource stands on its own as an authority on any topic. In order to establish credibility, any resouce must cite where their data came from – either through some kind of bibliography in the case of a paper, or experimental results, or that the resource is maintained by the people who designed and built what they’re writing about... The means of determining authority needs to come down to the following factors: 1) is the article written in an intelligent form? 2) are the sources cited of an authoritative nature? 3) has the author written anything previously that can be considered authoritative? and 4) would someone who is a known expert in the field (perhaps a professor of the topic) agree that the source is not some crackpot?"

Another component in becoming and Effective Searcher is to learn the skill of skimming for important details - and ignoring everything else. If you don't, you'll end up like the person described in this XKCD comic!

Lastly, to become an Effective Searcher, don't be afraid to ask for help: there is likely someone sitting near you or available via instant message who can help you out, and would be happy to if you only ask them.

Linux typically has several useful debugging tools available out of the box. These include gdb (for the advanced user), strace, ps, lsof, netstat, iostat, uptime, and top. Many others exist as well, but these are the most-commonly utilized.

"User error" is among the most commonly-cited errors with software and systems: the operator did something the creators did not expect. To use a ubiquitous car analogy, it's "user error" if the driver hits the gas instead of the brake. One interesting article makes the claim that there is [almost] no such thing as "user error", and that instead it should be the developers who make tools not resilient enough to handle any user (no, a car manufacturer can't make the gas act like the brake when you "meant to stop", but maybe software developers can make their products less error-prone, or at least have them give better errors when they do have a problem).

When something has gone so awry that it has violently crashed, or even taken out its host system, it's time for some post-mortem data collection - maybe even forensic analysis.

Core dumps, log files, and even images of whole drives can be investigated during a post-mortem analysis of problems seen: as your technical acumen grows, you'll be able to investigate more parts of these prior to escalating to the tool's support or development teams.

Ideally, we would all live and work in a world where nothing ever failed, and everyone acted the way they are "supposed" to. Sadly, that world does not exist. So what can we do to help prevent issues in the first place, or respond more adeptly when they [inevitably] occur?

Some solutions are simple: add more memory to the system; increase swap space; verify storage quotas; make sure all the resources I need are available; etc. Many can be more complex.

If there is a set of "Known Issues" or release notes that come with a particular product, make sure you read and are aware of them: there is almost nothing more frustrating than finding out there is a known issue, but you didn't check the manuals first!

Asking "Why"

If you're on the administrative side of the technical world, and not just the end-user side, the other big thing to remember is to always ask "why". Why did it fail? Why did we miss the known issue? Why were we not notified a necessary resource was going to be down? Why was there no alert sent about resources nearing their limits? If you can ask (and answer) those, then you should be able to reduce the number of "why" questions you need to ask in the future - because hopefully you're solving problems before they arise.

You've completed this module, and so now you're ready to troubleshoot the most ornery problems in the most obscure corners of your system, right? Don't let me discourage you from that lofty goal: but the reality is that becoming a good troubleshooter takes time, practice, lots of exposure, practice, skimming skills, practice, and patience. Oh, and did I mention: practice!

Lots of professions require troubleshooting skills, and each has their own tricks and tips to follow: auto mechanics will check the OBDII and listen to a rattle; electricians look for wiring faults; doctors look at symptoms to come up with a diagnosis. Skills learned in one field may not always translate into another, but if you can learn the basics (which DO all transfer), then gleaning insights from others can only improve your own personal Bag O' Hatchets.