Why software projects fail

14 March 2024 · 14,583 views

Some of you know that I work in the agency business — how that translates to my technical experience is that I used to work on many highly different projects over the years — in contrast to a product-based company where you work on a single product, your whole career. As one of my friends, an agency owner, said: “A year in the agency equals three years in the product company.” You can agree or disagree with that statement, but my life is filled with many different projects on a daily basis from various domains with different vendors and clients.

As you might guess, over time, this experience accumulates into a unique perspective on why projects fail — regardless of the industry, who’s leading the initiative, and who the stakeholders are. Sometimes, projects fail, and you can’t point to a specific thing that went wrong.

My experience, though highly subjective, suggests that even well-planned, well-funded projects tend to fail. I’ve been part of projects, even the ones that seem to have everything going for them, and there was always a chance they’d hit an iceberg. You've got deadlines, budget constraints, people falling out, new stakeholders getting onboarded, and all that jazz. Those are the apparent reasons the project can get derailed; they lay on the surface and are easy to spot.

But the silent killers? Those are the ones that creep up on you — the subtle issues that aren't immediately obvious but can seriously mess up your timing if you're not careful.

Making decisions in a good mood results in unexpected outcomes.
Source: commitstrip.com

Why software projects fail is not a new topic, nor is this article groundbreaking. Over the last week, I’ve read a dozen white papers about failed software projects and tried to combine them with my experience. This essay is the result of this combination — I will not focus on things already discussed by many. I will try to show you the not-so-obvious reasons why projects fail.

Overconfident developers

Confidence is cool; overconfidence, not so much. One of the things that I’ve noticed is that developers often underestimate not just the complexity of tasks, but there’s a general overconfidence in their abilities, not limited by programming:

  1. Overconfidence in their coding skills.
  2. Overconfidence in learning new technologies.
  3. Overconfidence in our abstractions.
  4. Overconfidence in external dependencies, e.g., third-party services or some open-source library.

If you ask a developer if they’ve ever written bad code, they will most likely say no. I’m guilty of this as well. I don’t remember ever writing bad code; sure, some of it wasn’t perfect and could be improved, but I don’t think it was terrible, per se. Nonetheless, if we asked everyone who reviewed my PRs if I ever wrote bad code, the answer would be yes; some of the code was probably garbage.

On average, I write decent code; on average, I can solve a task quite quickly; on average, I can build the module on time.

Sounds about right. Source: xkcd

If you ask me If I can build some feature in three weeks — after analyzing the requirements, I will tell you with confidence, based on my average past performance, that I’ve built similar things in the past and am pretty sure I can do it again. But there are always outliers, and we neglect them during estimations. We, as software developers, are guilty of overconfidence, leading to failing software projects.

💡 There’s also this concept called The Planning Fallacy, a term coined by psychologists Daniel Kahneman and Amos Tversky, which describes our tendency to be overly optimistic about our capacity to complete tasks quickly. 

Another great example of overconfidence are bad abstractions. Unless you live in a perfect world, you probably know that duplication is better than bad abstractions. The problem with that statement is that you always assume that your abstraction is better — you’re not a junior dev, you're confident in your skills (overconfident one might say) you know what you’re doing, you want to make everything neat and tidy and you've done this a dozen of times. The code looks perfect and will probably pass the code review.

After some time, new requirements pop up, almost fitting the existing abstraction, but not quite — someone tweaks the abstraction, adds a parameter, adds some condition, adds some switch statements. Repeat this several times, and you have a piece of code that’s so complex that nobody wants to touch it. Everyone continues using the overblown abstraction because they think this piece of code was written by someone smarter than them — and the cycle continues until someone brave enough confronts the complexity and starts adding duplication again to untangle the code.

The problem with bad abstractions is that their effect on the project timeline is like a snowball. At the start of the project, any feature that is related to that abstraction is done on time and doesn’t raise any red flags, but as the project goes on longer and longer, the time to modify the abstraction increases exponentially until it hits the point where the abstraction becomes unmaintainable and impossible to extend. Delays start piling up, until the project comes to a halt.

There’s an even better example of overconfidence — the learning curve of new technologies. If you’ve been coding in Golang and the project will be done in Rust, how much effort will you need to become proficient in the language? Developers often jump onto new technologies without fully appreciating the time and effort required to master them. This underestimation leads to delays as the steep learning curve is not sufficiently accounted for in project timelines or risk assessments. And if we dive even more deeply — new technologies always have a risk that the framework/language/driver/adapter has limitations that your team has not yet seen or knows about, which will result in figuring out workarounds to make things happen.

The need for workarounds is even more true when dealing with external dependencies and third-party vendors — anything you do not directly control is a risk of hidden complexity. The assumption that third-party services, libraries, packages, or APIs will work as expected without bugs is a common oversight. Unless you’re building simple CRUD applications, projects often get derailed when these external dependencies get deprecated, become incompatible with other dependencies, or plainly stop providing the service.

Third-party dependency reality — update that breaks everything. Source xkcd, I love them.

It’s hard to plan for that. We’re overconfident because, on average, it works out, except when it doesn’t. If we try to quantify this and multiply this probability by the number of developers on a project, even the best projects, if they get big enough, have a hundred percent chance to either go over the budget or deliver too late.

Inexperienced Managers

So, what’s the next thing that you can’t plan for? You guessed it: inexperience and/or incompetence and/or negligence. I’m using these interchangeably, as you never know where one ends and the other starts. There’s a massive mismatch between what makes a manager successful in the eyes of their superiors versus what truly benefits a project and its team.

Hitting those bonus targets.

On one side of the spectrum is a manager with their own ambitions in mind rather than the project’s success. A good example is promotion-based development, where managers leave projects before they are completed, leveraging their position for a career boost. Peter Principle says that people get promoted to their highest level of incompetence. It's not that these managers are inherently bad at their jobs; it's just that what got them the promotion isn't what's needed to lead the project to success — it’s a different skill set.

On the other side, we have managers without enough experience who a) assume someone else will do the job and b) assume that their job is to satisfy all the stakeholders and say yes to everything. They overlook the necessity of robust and firm planning, allowing process debt and scope creep to derail projects.

If you see these red flags, you should be ringing alarm bells:

  1. Information from upper management is not shared, leaving the team in the dark about broader project developments. I call it, “Trust me, we’re moving in the right direction.”
  2. Tasks are assigned without context, necessary resources, or contacts, only mentioning deadlines. I call these “figure-it-out-yourself” tasks.
  3. They will often turn the team's work in as if they’re responsible for the whole thing, getting praise from the upper management but not sharing it with the team.
  4. Always afraid of taking on responsibility, never makes decisions independently, always defers them to upper management… which only they are allowed to talk with.
  5. Blames the team for mistakes in front of the upper management.

These are just some of the characteristics of an inexperienced manager; let me know if you have more.

The biggest mistake an inexperienced manager can make is allowing frequent changes to the project’s scope, especially those imposed by higher-ups, without proper consideration of the implications. For example, “The CEO said his daughter thinks we should have Login with TikTok on our banking app, and I think this is a great idea!” This leads to many sighs in the team, resource strain, and shifting priorities, resulting in project failure.

Mismanaged Stakeholders

A project's true north should always be its value — real, tangible, and aligned with the business goals it seeks to fulfill. Nobody would want to build a project that’s just a drain on the resources, right? Of course not, but the problem is you never really know if your project has turned into a zombie until it’s too late.

Our minds play tricks on us - confirmation bias makes us blind to the signs that the project is going in the wrong direction, and the affect heuristic leads us to downplay the bad and inflate the good, especially when we're emotionally invested in the project's success.

These projects always start with a promise - a clear goal, a vision aligning with the company’s strategy, reasonable financial forecasts, and a decent development timeline. It feels like a great opportunity; it’s hard to notice anything wrong during the initial 30-40% of the project.

But then things start to change, and unexpected challenges arise.

Technology that seemed straightforward refuses to cooperate, external competitors launch similar ideas, key partners back out, and internal business stakeholders focus more on the projects that include AI in their name. Things slow down, and as months turn into years, enthusiasm wanes.

Then the snowball continues — key members leave, and new people join, each departure a slight shift in direction. New tech lead steps in, eager to leave their mark, steering the project further from its original course. At this point, nobody knows where the project is headed, and nobody wants to admit the project has failed. It's a tough spot, especially when everyone's playing it safe, avoiding the embarrassment or penalties of admitting failure.

💡 The silent killer in this scenario isn't just the project's failure to deliver value; it's the collective reluctance to acknowledge that failure. Fear of admitting defeat, of facing the repercussions of a project gone south, keeps the team from developing the product further.

Why? Well, it's awkward, isn't it? Admitting defeat means facing some uncomfortable truths. And nobody wants to be the bearer of bad news; nobody wants their name attached to “that project” that failed, so they keep the project alive, using creative budgeting, overpromising, and forever pushing success just out of reach, hoping for a miracle or reassignment to some other project.

Deep down, everyone knows the truth, but they will never admit it. This project isn't going to hit its targets; it’s not getting released, or maybe it will, but the financials won’t be that good. It's not going to revolutionize the market. It's just… going to continue consuming resources without really achieving anything.

When such software projects start failing, a negative mindset quickly spreads. Instead of fixing the issues, people say, "Well, that's not my job." or “I’m doing my part; everything else is not my concern.” Everyone's looking out for themselves, trying to avoid the consequences. This attitude stalls any attempt at genuine problem-solving and deepens the project's issues, making recovery even more challenging.

When a "not my problem" attitude prevails, it becomes the seed for a toxic culture. In such an environment, accountability is dodged, and collaboration dwindles. Team members focus solely on their tasks, ignoring the bigger picture and the collective goal. This lack of shared responsibility leads to blame-shifting when things go awry, creating an atmosphere of distrust. The energy that could have been used for problem-solving gets wasted on internal politics and protecting one's turf.

Conclusion

Projects fail. It sucks, yes, but it's the truth. It's not just about bad coding or missed deadlines. Inexperience, ego, overconfidence. All of these human traits have a huge impact. So let's try to avoid them and be better role models?

Let me know your stories about failing software projects in the comments.

Other Newsletter Issues:

  • Pietro

    Jumped into a project with a new stack thinking it’d speed things up. Reality hit hard with the learning curve slowing us way down. Everyone got bogged down with tutorials instead of coding. Ended up switching back to our old stack to meet the deadline.

  • Quentin Ash

    I once jumped too quickly into using a new programming language for a project, thinking it’d make us more efficient. Turned out, the steep learning curve actually slowed us down and stressed the team out. We learned the hard way that sticking to familiar tools can sometimes be the smarter move.

  • Izzy

    Jumping on new tech without considering the ramp-up time is a common pitfall. It’s essential to realistically assess the learning curve and challenges. Also, recognizing when a project is no longer viable and having the courage to make tough decisions early can save resources and prevent bigger failures.

  • Jack Summers

    Jumping on new tech? Cool, but comes with a learning curve that can mess up timelines. Gotta weigh if it’s worth the hassle. Also, knowing when to kill a failing project saves $$$ and stress. Fail fast, learn faster. Cut the crap and move on. This ain’t rocket science, just common sense in the dev world.

  • Louis Schilling

    I see a continued failure to do a “crawl, walk, run” on projects. Come up with the “crawl” and say “I can do that for you in 2w, and then we can look at where we are and figure out what the walk looks like”. There’s no way you can accurately estimate what the “walk” or the “run” looks like in terms of time and expense until you’ve validated your assumptions and re-verified the goals and requirements of the project. Obviously this doesn’t work for all types of projects, but the vast majority in my 30y experience.

    9 times out of 10, once you get to “crawl” you will have found 10 things that were wrong about your original assumptions or requirements, and a “crawl” or “MVP” will give you time to adjust before you continue on.

    1. Vadim Kravcenko

      Hey Louis, very interesting concept, thanks for sharing

  • Dave Bartlett

    In economic terms, a software project fails when the cost of implementing change exceeds the anticipated benefit of the change. That’s my insight from observing software projects over 30 years.

  • Gifford Nicholson

    I have been a software coder (engineer) for over 25 years. Recently retired. No problems. Mostly worked in FoxPro 2.6 and Visual FoxPro and traditioned to VS C# with some incursions into VB6.
    I have worked as a company employee and as a contract employee. As a contract employee I had much less to do with company politics. As a company employee I was constantly exposed to company politics and fired sometimes due to company internal stresses. At one place I think I was a threat to the VP. Not that I ever thought of myself as a threat to his job. After leaving that employer I found other employments as a contract employee and was very happy and productive in that position. My last twelve years I was a company employee.
    Projects fail because of three + main reasons. Programmers who are overconfident in their ability. Programmers who do not realize the scope/complexity of the project. Programmers and manager who do not know how the product works at all levels. Managers who do not understand what is needed (coder ability, skill sets, knowledge) or the scope of the project and the need to limit the scope creep that management always? will push for. Scope creep is a killer and needs a strong management team to oversee the project.

    Thank you,

    Gifford T Nicholson

  • Otto

    Exploring new tech is thrilling but can lead to delays if not carefully planned. It’s equally important to know when to end a failing project to save time and effort. Learning from such experiences can guide future decisions, making project management more effective.

  • Tom

    Good writing. These are all good insights indeed. In my experience the key fallacy continues to be thinking “we know it all” – and it concerns all stakeholders, not just developers. What’s worse non-tech management often thinks that software development is for the most part just coding, and coding in their eyes is as simple as getting an “obvious task” and “just get done with it”, which if course couldn’t be further from the truth. Managers in general also assume they know how to make developers go faster, mainly by meeting often and agreeing every detail upfront, which of course is mostly a bad idea.

    Developers on the other hand assume they have it all figured out technically, and also they know best what to do. This concerns in particular adding third party dependencies and in tendency going after shiny new things instead of using well known libraries.

  • Anonymous

    Dependency is a code smell for an over confidence in abstraction.

  • David

    Great article. I’d add another common failing , estimating development effort at the very beginning of a project when there are so many unknowns, setting these in stone as deadlines and never revising these deadlines as more is known as what is to be done

  • AH

    In over 30 years, I have never worked on a project that did not deliver. Original delivery date was moved back in a few cases, but deliver we did. In summary, this is down to having good project managers most of the time, hence planning was done properly most of the time, but the most important and crucial factor is having a clear objective at the beginning of the project, and deviation from this was never agreed to or kept to a bare minimum in all cases. As an engineer or a project manager, I have always ensured that I am clear on the objective before agreeing to work on any project.

  • Seb

    Jumping into projects thinking you’ll figure out the tech stack as you go can really mess things up. Always better to budget extra time at the start for learning. Ever noticed how overconfidence can trip up even the best of plans? Always overshoot how long learning something new will take—it saves stress later. And honestly, talking straight with stakeholders about what can go wrong and how long things will really take can save a lot of headaches. Anyone else had to learn this the hard way?

  • Jonah T

    Nice article