5 Software Development Horror Stories and the Lessons You Should Learn from Them

You’ve probably heard what can go wrong in the world of custom software development. (Because trust us, if it can go wrong, it probably will.)

The stakes are high. A 2003 study commissioned by the Department of Commerce’s National Institute of Standards and Technology found that software bugs cost the US economy $59.5 billion annually. Ten years later, a Cambridge University study put the worldwide cost at $312 billion. It’s clear that the impact of bugs is growing exponentially, both domestically and worldwide.

But the truth is that it’s when those bugs are found that make all the difference.

For example, as explained in a Celerity blog post: “If a bug is found in the requirements gathering phase, the cost could be $100. If the product owner doesn’t find that bug until the QA testing phase, then the cost could be $1500. If it’s not found until production, the cost could be $10,000. And if the bug is never found, it could be secretly costing the company money and no one could be the wiser.”

The exponential factor is at work in your particular software project, not just in the economy as a whole.

Occasionally, there are bugs so costly that they are evident almost immediately. If you’re wondering what those situations look like in real-world situations, you’re in luck. We’ve pulled together some examples of real software development projects gone wrong. Really wrong.

We’ll go in chronological order and end with an ongoing situation in our home state of South Carolina. So sit back, enjoy, and hopefully learn a few lessons that could save you from some of these companies’ heartaches…

Nike & i2 Technologies

In June 2000, Nike’s new supply-and-demand software planning system from i2 Technologies ordered far thousands more Air Garnett sneakers than could be sold, and thousands fewer Air Jordans than were needed. This mistake ended up costing Nike more than $100 million in lost sales. The company’s stock price dropped by 20 percent, and they faced a flurry of class-action lawsuits.

Nike claimed that system problems including being too slow, failing to integrate well, and the presence of some bugs. It also conceded that Nike’s planners were inadequately trained in how to use the system before it went live.

However, a CIO article in 2014 described the situation this way: “If there was a strategic failure in Nike’s supply chain project, it was that Nike had bought in to software designed to crystal ball demand. Throwing a bunch of historical sales numbers into a program and waiting for a magic number to emerge from the algorithm—the basic concept behind demand-planning software—doesn’t work well anywhere, and in this case didn’t even support Nike’s business model.” By the spring of 2001, Nike had moved its short- and medium-range sneaker planning to its SAP ERP system, which is grounded more in orders and invoices and less in predictive algorithms.

The 2001 i2 glitch was one only a company like Nike could bounce back from.

The lesson

Software—and new frontiers like machine learning—aren’t silver bullets. You need to understand how your system makes decisions and/or recommendations to ensure that they will align with your business goals.

St. Mary’s Mercy Medical Center

In February 2003, a software error at St. Mary’s Mercy Medical Center in Grand Rapids, Michigan, showed 8,500 patients had died—when they hadn’t. All the patients had procedures between October 25 and December 11 of the previous year, some as simple as a blood test as part of a routine physical examination. And while they were still alive, the hospital’s patient management system incorrectly notified Social Security, patient insurance companies, and the patients themselves that they had deceased.

It turns out that the hospital had recently completed an upgrade of its patient management system. A "mapping error" in the conversion process assigned the disposition code of "20"—which meant the patient had died—instead of the intended "01" code, which meant the patient had been discharged.

Besides the obvious embarrassment, this error had serious implications in terms of patient trust. Would you go back to a hospital system that declared you dead? It also forced people to deal with the significant hassle of correcting an error in the Social Security death index.

The lesson

Software isn’t just a system. It impacts your brand and the customer experience you create. If you don’t get it right, the impacts will likely reach farther than the walls of your business.

Heathrow Terminal 5 Opening 

Terminal 5 of Heathrow Airport opened in March 2008, along with a new baggage handling system. Before the opening, staff tested the new baggage system with over 12,000 pieces of luggage, and it worked flawlessly. But on the Terminal’s opening day, the system fell apart. As a result, over the next 10 days approximately 42,000 pieces of luggage didn’t make it onto the right planes, and over 500 flights were cancelled. Analysts suggested that real-life scenarios such as removing a bag from the system manually had confused the entire system.

The lesson

This is a stark reminder that users are creative in what they will actually do. Your software needs to find ways to account for the uncertainty and complexity that users will undoubtedly create.

Knight Capital Group

Think you could lose $440 million in 30 minutes? Knight Capital Group did on August 1, 2012, thanks to a trading algorithm that bought high and sold low on 150 different stocks. As a result, the market-making firm’s stock price dropped by 62 percent in one day, and the total loss was four times its 2011 net income.

In its analysis after the event, Nanex elaborated on the problem with the glitch: “[A]lmost all these trades alternate between buying at the offer and selling at the bid, which means losing the difference in price. In the case of EXC (Exelon), that means losing about 15 cents on every pair of trades. Do that 40 times a second, 2,400 times a minute, and you now have a system that’s very efficient at burning money.”

The lesson

You need visibility into your software so you can spot errors quickly—because things can go very badly in the blink of an eye. Automated error reporting is a key feature that’s worth the effort.

South Carolina Department of Social Services

This one is personal to us, because it’s happening in our state and costing us as taxpayers a lot of money.

Due to long-running delays in implementing a child support enforcement computer system—which are still ongoing—the state of South Carolina expects to incur $13.5 million in federal fines in 2017. And that’s on top of the $134 million already incurred. The system, which has been a work in progress since the 1990s, is being rebuilt response to a 1988 federal law. And yet, it’s not expected to be operational until the fall of 2019—20 years later!—making South Carolina the only state not to have the required automated system up and running.

The state was pursuing a made-from-scratch system but abandoned those plans to enter into contract with Xerox and copy the system currently used in Delaware and some other states in an effort to get the system live. Jimmy Early, who has been overseeing the project for DSS since last year, expects the agency to pay another $63 million in fines before the project is complete and running statewide. That’s $200-plus million in costs without paying for a single line of code—because of poor planning and project management that lead to cost and timeline overruns.

The lesson

Process matters. Pick the right partner and the right development approach from the beginning, and follow a proven process that will actually get to the finish line.

In Conclusion

Software projects can go wrong. In fact, the industry chaos report says that 20-30% fail and that less than 10% of medium- and large-sized projects are a resounding success.

So take the lessons from these five failures into account as you prepare for your next software project. That way, you won’t be on the list for the next post like this.

About Worthwhile Storyteller

We'll never tell you a lie, but we might tell you a success story that protects the intellectual property of our clients and partners. Our Worthwhile Storyteller is an amalgamation of all of our thoughts, experience, and expertise brought together to give you the facts about our relentless improvement in the software development space.