How Many Software Bugs Are Too Many?

Bugs have been part of software development since the first computers were programmed. In the early days, “bugs” were often literal – insects that became lodged inside the machine. Thankfully, today’s software bugs are an entirely different story – there’s usually no need to get up from your desk. However, you now have an entirely different problem to manage – a much more complex technology environment.

If you’re struggling with your approach to software bugs, you’re far from alone. If that’s true, how do you ship working software to your users? The answer is simple. Develop a disciplined process to systematically identify, evaluate and solve software bugs. This approach can be scaled up or down depending on the size of your team.

Ultimately, the number of software bugs you can tolerate depends on your organization’s goals and the capabilities to achieve those goals.

Set Goals For Software Quality

Before you read a single software bug report, you need to have specific goals to guide your work. Without clear goals, you will not be able to sort make meaningful progress. In my opinion, undirected bug work leads to mediocre software. That’s a strong statement but consider the user experience for a moment.

Consider a popular office productivity product like Microsoft Word or Microsoft Excel. These products have been refined and expanded for decades. Most users simply scratch the surface of what is possible. For example, some people use Excel as a database with tens of thousands of rows of data. Even then, few users extract the greatest possible value from Excel functions like PivotTables. What does this example mean for stopping software bugs?

Here’s the point: not all software features and functions are created equal. If you stop a malfunction that only matters to 1% of your users, your effort may be been wasted compared to other bugs you may have pursued instead. If you have clear goals to guide your work, you will not fall victim to this challenge.

Depending on your situation, there are two approaches to developing goals for your software development program. First, you can derive your goals based on the organization as a whole. Second, you can adopt a bottom-up approach to develop goals based on your values and understanding of the user.

Derive Software Bug Improvement Goals From The Organization

Most well-managed organizations are driven by goals. For example, a start-up may be focused on achieving a certain level of product development or revenue in order to obtain funding to continue growing.

In contrast, a larger company’s corporate goals may include profitability, partnerships, innovation and community responsibility. Regardless of your situation, you can use the following questions to discover the organization’s goals and plan your process.

Questions to ask to find your organization’s goals:

1. Ask your manager for a goals document? (Sometimes the simple approach is best)
2. Read statements published by the company’s CEO or senior leaders on what matters to them.
3. Ask yourself “what standards are important to the organization or the product?” (e.g. stay current with the latest version of iOS)

Once you have the organization’s goals in front of you, you can come up with a specific goal related to software bugs and quality. The best goals are objective and easy to measure. For example, your goal may be: “eliminate all six Tier 1 software bugs by December 1.”  This goal hints at a concept you will learn later – the importance of weighing and evaluating each software bug for importance.

How To Develop Bottom Up Goals For Software Bug Improvement

If you are running a small company as an entrepreneur, you may need to come up with goals all on your own. Fortunately, there’s no need to run out and hire a strategy consultant. A few simple questions will give you all the material you need to develop goals. Let’s get started.

1. Ask your three most valuable customers about their experience with the product: What slows them down or frustrates their experience?
2. What feature or capability would give your product the greatest leverage to grow(e.g. you may find that database crashes occur at a certain activity level)?
3. What aspect of technical performance excites you and your team the most? (i.e. appeal to your sense of professional pride)

Once you have the answers to these questions, you can define your goals for addressing software bugs. For example, you may emphasize solving all bugs that relate to the purchase process. In my opinion, software that cannot process purchases or accept payments due to bugs isn’t worth very much.

Identify and Track Software Bugs

A software bug database makes it easy to obtain a clear picture of your software bugs. For the identification process to produce value, you will need to track a few key data points.

Remember software bug reports may come in automatically though a bug report, through customer service or other channels. It’s important to gather bug reports from all sources so that you have a full picture.

Software Bug Information: Key Points To Track

Use the following checklist to check that your software bug reporting process captures all the data you need.

• Date and Time. This data helps to identify causes and patterns in your software bugs.
• Interface Data. If your product has multiple interfaces such as a desktop application and a mobile app, it is important to identify this detail clearly.
• What was the user trying to do? Depending on the situation, you may consider asking the user’s approval to take a screenshot of the problem.
• Error code. Assuming you have created meaningful error codes, tracking this data point is helpful.
• Tracking this data point helps you to identify patterns and whether you should direct users to roll back to an earlier version while your team fixes problems.
• Network Status. Your software may require a certain level of connectivity to operate successfully. Collecting information on the status of the network connection may be helpful.
• Operating System Details. The operating system plays a factor in software performance – make sure you track it in bug reports.
• This data point is critical if your software runs through a web browser.
• Anonymous User ID. In order to detect patterns, you may want to identify users in a unique, albeit depersonalized way.
• Add-on Status. Some platforms (e.g. WordPress, Shopify and Microsoft Office) have a wide variety of add-ons to customize the user experience and add functions. These add-ons may have unintended side effects on your software

Use this list as a starting point to guide your software bug report process. You may be tempted to collect dozens or hundreds of data points to support your analysis process. Remember that collecting data for the sake of data is usually wasteful.

In my opinion, it is better to collect data on a small number of variables (e.g. 5-15 data points) and analyze them deeply than collecting more information and not doing anything with that information.

What’s next after you design the information you’re seeking in your software bug report? It’s time to track and organize that data in a comprehensive fashion

Track and Organize Software Bug Data

There are two broad ways to organize software bug data: manual and automated. A manual approach is helpful if you are operating at a small scale and have a limited budget. If it all possible, always look for an opportunity to use an automated software bug tracking solution.

In either case, you will need to build a process to organize and report on your software bug data. If you are starting from scratch, use these steps to track software bug data:

1. Import your software bug data into a tracking database.
2. Summarize the software bug data by criteria such as operating system, application version, and other key data points.
3. Prepare a summary report on the past month’s software bugs. Include both quantitative data (i.e. summary statistics about the most common patterns in bug reports) and qualitative information (i.e. “Users are complaining on Twitter about iPhone crashes dozens of time each day!”)
4. Schedule a meeting with the software team to review the software bugs
5. Analyze and prioritize the software bugs that merit immediate attention.
6. At this point, you might not be sure how to prioritize your software bugs. Don’t worry. We’re going to cover that point next.

Evaluate Software Bugs

You have a database of software bugs. Now what?

It’s time to use a process to pick and choose which software bugs are worth solving. If you skip evaluating software bug reports, you will never make much progress. When you’re getting started with evaluating software bugs for importance, keep it simple. The following rules adapted from quality management will give you everything you need to get started.

1. Use Pareto’s Principle

As explained in Richard Koch’s classic business book “The 80/20 Principle,” disproportionate cause and effect is the name of the game in business. To implement this practice, organize your software bugs into categories. Now rank order the causes to find priorities. In my opinion, this is the single most important analytical tool in business to improve efficiency and profits.

2. Your organization’s goals

If your organization has a goal to become a top-ranked iOS app, that goal will inform your approach to software bug prioritization. What if you have a vague goal such as “improve software product quality.” In that case, you will need to consider the other points in this section.

3. The Executive Rule

Let’s face the hard truth. Sometimes, software and product decisions are dictated by an executive’s whim. If they have excellent judgment and experience, there is merit to this approach. Here is how you would apply this rule in practice. Ask yourself “What Does our CEO (or executive) get the most upset about? For example, they might be fixated on the purchase or check out features because they see a link to revenue. Or they may be focused on smooth Facebook integration because they are pursuing vital growth.

4. Top customer requests

If you have a handful of major customers, keeping them happy is a top priority. If that situation describes your company, then prioritize the software bugs that come from those customers. Keep in mind other considerations – such as the company’s product strategy – when applying this principle. In some cases, it makes sense to decline to solve certain problems if that work would take your company in the wrong direction.

5. Easy Wins For The Team

Solving software bugs can be thankless work – like playing a glorified version of Whack A Mole. If you are managing a team of developers, give added weight to this factor.

Consider this:

Your software developers are not robots who blindly follow orders. They are motivated by the ability to achieve and see an impact from their work. Therefore, it makes sense to include add some “easy wins” to your software bug priority list. For example – solving a simple cosmetic issue may only take a few hours of work. However, there is great satisfaction in seeing that improvement put into production.

Using these questions helps you to narrow down your list of software bugs. Instead of wondering aloud “how will I ever solve this overwhelming list of software bugs?” you will have a small list of easy to manage software bugs.

Solve Software Bugs

You have a list of high priority software bugs to fix. Now what?

It’s time to get down to work on solving those bugs and improving your product. The following management approach guides you through the process of ensuring high-quality results.

1. Assign the software bug to a specific person

Until a task is assigned to a specific person, little will be achieved. To emphasize the value of the improvement, connect the bug improvement to your goals. For example – improving the stability of the Facebook integration will drive increased revenue.

2. Follow standard testing and quality procedures for the bug solution

I’ve seen too many software bug “solutions” that end up causing other problems. That’s why all software bug fixes need to go through a full review and testing process. For low-risk improvements, you may scale down the process but it should never be omitted entirely.

3. Put the software bug solution into production

Once you have created and tested the software bug solution, it is time to put it into production. If your organization has a regular release schedule, it is best to include your bug solution into that process.

4. Check with customers and stakeholders

Solving a software bug is not enough to advance your company’s goals. In my opinion, you need to demonstrate that your software has achieved results for internal users or customers.

For example – if your VP requested a certain bug solution, inform that person when the solution is solved. Likewise, use a similar process to notify your customers. This is a fantastic way to demonstrate that you are listening to and acting on feedback.

5. Repeat the entire process next month

Like it not, the work of solving software bugs never ends. Each and every month, you need to collect information on software bugs and use the process in this article. As you become more comfortable with the process, your efficiency at solving bugs will increase.

What’s Next For You

Delivering great software requires creativity, vision, and discipline. You’ve learned the art of handling software bugs in this article. We’ve covered the fact that some software bugs are more important than others.

By strategically choosing which software bugs to focus on, your software will improve each and every year. Fundamentally, complaining about “too many software bugs” simply means that you do not have a reliable process to identify and prioritize solutions.

(Interesting to read: How to reduce the cost of bugs?)