Enterprise IT Projects

Large-scale projects across industries, particularly in Enterprise IT, Artificial Intelligence (AI), Machine Learning (ML), and Software Engineering, are notoriously susceptible to failure, frequently exceeding budgets, timelines, and failing to deliver promised benefits. This white paper, drawing extensively from Bent Flyvbjerg and Dan Gardner's insights in How Big Things Get Done, explores the underlying causes of these widespread failures, including psychological biases, strategic misrepresentation, and a "bias against thinking". It then outlines a comprehensive framework for success, rooted in the core philosophy of "Think Slow, Act Fast". By integrating robust leadership, critical thinking methodologies like Reference Class Forecasting and iterative planning, and fostering a disciplined culture of psychological safety and modularity, organizations and countries can dramatically improve the delivery of ambitious technology projects.

White Paper: Navigating the Labyrinth of Large-Scale Projects – Leadership, Problem-Solving, and Culture in Enterprise IT, AI-Machine Learning , and Software Engineering

Abstract

Large-scale projects across industries, particularly in Enterprise IT, Artificial Intelligence (AI), Machine Learning (ML), and Software Engineering, are notoriously susceptible to failure, frequently exceeding budgets, timelines, and failing to deliver promised benefits. This white paper, drawing extensively from Bent Flyvbjerg and Dan Gardner's insights in How Big Things Get Done, explores the underlying causes of these widespread failures, including psychological biases, strategic misrepresentation, and a "bias against thinking". It then outlines a comprehensive framework for success, rooted in the core philosophy of "Think Slow, Act Fast". By integrating robust leadership, critical thinking methodologies like Reference Class Forecasting and iterative planning, and fostering a disciplined culture of psychological safety and modularity, organizations and countries can dramatically improve the delivery of ambitious technology projects.

1. Introduction: The Persistent Challenge of "Big Things" in the Digital Age

The modern world is increasingly defined by ambitious, large-scale projects, from smart city initiatives to complex AI deployments. However, the track record for these "big things" is alarmingly poor. Research indicates that only a minuscule 0.5% of major projects are completed on time, within budget, and deliver the expected benefits. This consistent pattern of failure is so pronounced that it is termed the "Iron Law of Megaprojects": most large-scale projects habitually exceed their time and budget and underdeliver on promised benefits. For instance, transportation projects, on average, experienced budget overruns of 28%, with rail projects being the worst at 45% over budget. Startlingly, information technology (IT) projects demonstrate an even more dismal record of disasters than traditional transportation projects.

This paper asserts that the insights from How Big Things Get Done offer critical lessons universally applicable to all project types, including the rapidly evolving fields of Enterprise IT, AI, ML, and Software Engineering. It will explain how effective leadership, rigorous problem-solving through critical thinking, and the cultivation of an enabling culture and discipline can transform project outcomes, even when addressing the "fat tails" of risk inherent in complex digital initiatives.

2. The "Iron Law" and the Complexities of Digital Project Failure

The pervasive failure of big projects, particularly in IT, is not merely due to technical challenges but stems from a confluence of human and systemic factors.

2.1 Psychological Biases and Strategic Misrepresentation

Project leaders and teams are often susceptible to cognitive biases. Optimism bias, identified by Daniel Kahneman as a significant cognitive bias, leads individuals to underestimate the time and cost required to complete tasks, even when historical data suggest otherwise. The "planning fallacy," a subcategory of optimism bias, specifically describes the tendency to underestimate task completion times. This "this time is different" mindset is further exacerbated by "uniqueness bias," where planners perceive their projects as unique, believing they have little to learn from past ventures. Even Daniel Kahneman himself fell prey to uniqueness bias when forecasting the completion time of a textbook, which took eight years instead of the estimated two, despite knowing that similar projects usually took at least seven years and often weren't finished at all.

Beyond innocent bias, "strategic misrepresentation" — the deliberate distortion of information for strategic purposes — is a significant driver of failure. Those seeking project approval or contracts may gloss over major challenges to keep estimated costs and timelines low. This leads to "premature lock-in" or the "commitment fallacy," where shallow analysis is followed by a quick decision, making it difficult to turn back even when problems arise. This is often reinforced by a "bias against thinking," a preference for "doing over talking" that discounts the value of thorough planning, mistakenly viewing it as wasted effort, especially under time pressure. Powerful individuals are particularly susceptible to availability bias, trusting their intuition over slow, deliberate planning.

2.2 The "Fat Tails" of Risk in Technology Projects

Most project types, including IT, exhibit "fat-tailed distributions," meaning they are not merely at risk of going seriously wrong, but disastrously wrong. This implies that extreme outcomes, such as massive cost overruns or catastrophic failures, are far more probable than in a normal distribution. For IT projects, 18% experience cost overruns above 50% in real terms, with the average overrun in that "tail" being 447%. Examples include the US government's HealthCare.gov website, Kmart's massive IT projects contributing to bankruptcy, and Levi Strauss's $200 million loss due to an IT blowout. Ignoring this "fat-tailedness" in project risk is a significant oversight in much of project management literature.

3. "Think Slow, Act Fast": A Paradigm Shift for Digital Success

The central tenet for successful project delivery is "Think Slow, Act Fast". This means dedicating ample time and resources to meticulous planning, testing, and refining solutions before rushing into implementation. Planning, when done rigorously, is relatively cheap and safe, while delivery is expensive and dangerous. The longer a project's delivery phase, the greater its "window of doom" – the opportunity for unforeseen problems or "black swans" to cause trouble. Projects that succeed tend to "zip along" and finish quickly, while failures "drag on". This is because "projects don't go wrong; they start wrong," typically due to rushed, superficial planning that leads to a "break-fix cycle" during delivery.

4. Leadership in Digital Megaprojects

Effective leadership is paramount for transforming visions into successful projects. This involves more than just oversight; it demands specific qualities and actions.

4.1 The "Master-Builder" and Experienced Teams

The most valuable asset a project can have is an experienced leader with an experienced team, often referred to as a "master-builder". Such leaders and their teams possess profound "tacit knowledge" – intuitive understanding gained through long experience that cannot be fully captured in words or manuals. This "phronesis," or practical wisdom, allows them to discern what is right to do and to get it done, making their intuitions highly reliable under the right conditions. The success of projects like the Hoover Dam, led by engineer Frank Crowe and his loyal, experienced team, exemplifies this. In the context of Enterprise IT, this means prioritizing leaders who have a proven track record not just in general management, but specifically in managing complex IT, AI, or software development initiatives.

4.2 Cultivating Psychological Safety

A crucial aspect of effective leadership is fostering "psychological safety" within the team. This is an environment where every team member feels they have the right and responsibility to speak up, share ideas, and voice concerns without fear of retribution. Psychological safety boosts morale, encourages innovation, and, critically, ensures that "bad news travels fast". In IT, where complex interdependencies can lead to unforeseen issues, the ability for any team member, from a junior developer to a senior architect, to flag problems early is invaluable in preventing minor issues from escalating into major disasters.

5. Problem-Solving and Critical Thinking Methodologies

To "Think Slow" effectively, project leaders must employ rigorous problem-solving and critical thinking methodologies.

5.1 Right-to-Left Thinking

Successful projects begin by clarifying the ultimate purpose and desired outcome, then working backward to plan the necessary steps. This "thinking from right to left" ensures that the project remains focused on its true goal rather than getting lost in the details of execution. For example, Amazon implements this by requiring project proposers to write a press release (PR) and a "frequently asked questions" (FAQ) document for a successfully completed product before development begins. This forces clear, customer-centric thinking and lays bare any fuzzy, illogical, or unsupported assumptions. This approach is particularly beneficial in AI and ML, where the "why" and "what" (e.g., specific business problem solved, ethical implications) must be meticulously defined before diving into complex model development.

5.2 Iterative Planning and Testing ("Pixar Planning" / Maximum Virtual Product)

Rigorous testing and iterative refinement of plans are essential to address potential issues early, minimizing costly failures during execution. This is encapsulated in "Pixar Planning," where animated movies undergo numerous iterations of rough video mock-ups and audience feedback before full production. This process corrects for the "illusion of explanatory depth," where people mistakenly believe they understand complex phenomena more deeply than they do.

While some in Silicon Valley advocate the "lean startup" model of rapidly releasing a "minimum viable product" (MVP) and iterating based on consumer feedback, Flyvbjerg argues that this is effectively a form of "planning" through testing. The key is the method of testing. For large-scale Enterprise IT, AI, or ML projects that are too expensive or dangerous to release a true MVP (e.g., critical infrastructure software, medical AI), a "maximum virtual product" model is preferable. This involves extensive simulation and virtual testing, creating a detailed, tested plan that significantly increases the odds of smooth and swift delivery. The failure of Theranos, which applied a software-centric lean startup model to medical testing with catastrophic results, serves as a stark warning.

5.3 Reference Class Forecasting (RCF)

To combat optimism bias and provide more accurate estimates, RCF uses data from similar past projects as a "reference class" to predict success for current projects. This method provides a more accurate timeline and budget by incorporating historical examples and accounting for "unknown unknowns" that cannot be predicted individually but are encoded in the data of the reference class. The UK government, for example, made RCF mandatory for all big transport infrastructure projects after preliminary research showed its effectiveness. For Enterprise IT, AI, and ML, this means building and utilising databases of past project performance (e.g., cost, time, benefit delivery for similar AI model developments, software platform migrations, or data engineering initiatives) to inform new project forecasts.

5.4 Black Swan Management

Understanding that projects are exposed to "black swan" events – low-probability, high-consequence occurrences – is crucial. Instead of viewing these as unpredictable accidents, they can be studied and mitigated. For example, an analysis of high-speed rail projects in the "fat tail" revealed that failures were not due to "catastrophic" risks like terrorism, but rather compound effects of standard risks like archaeology or procurement delays. For IT and AI projects, this entails identifying potential points of systemic vulnerability, such as complex dependencies, security breaches, or data integrity issues, and proactively developing bundles of measures to reduce these risks, effectively "cutting the tail" of disastrous outcomes.

5.5 Modularity ("Build with Lego")

Breaking down ambitious projects into smaller, scalable modules, like LEGO blocks, allows for easier planning, testing, refinement, adaptation, and scaling. This "many small things" approach is more reliable and efficient than building "one huge thing". For example, Heathrow's Terminal 5 was constructed by manufacturing components in factories and then assembling them on-site, akin to the hyper-efficient car industry. In software engineering, this aligns perfectly with microservices architectures, containerization, and component-based development, enabling teams to develop, test, and deploy smaller, independent units faster and with fewer interdependencies. This principle extends to AI/ML development, where modular data pipelines, model components, and reusable feature stores can significantly de-risk and accelerate deployment. The success of wind and solar power projects, the best-performing project types, is largely attributed to their modularity.

6. Cultivating a Culture of Success and Discipline

Beyond methodologies, the organizational culture and inherent discipline play a decisive role in project success.

6.1 Building Cohesive and Empowered Teams

A strong, cohesive team, united and mutually committed towards a common goal, is critical for swift and successful delivery. This involves not only "getting the right people on the bus and placing them in the right seats" but also ensuring their well-being and listening to their input. BAA's success with Heathrow's T5 demonstrated the power of fostering a shared identity and purpose, providing excellent on-site facilities, ensuring immediate access to necessary equipment (like PPE), and consulting workers in design and workflow improvements. This level of care and respect translates into ownership and effectiveness in implementation. For Enterprise IT, this means moving beyond siloed teams, investing in cross-functional collaboration, and empowering engineers and data scientists to voice concerns and contribute solutions.

6.2 Overcoming Behavioral Obstacles

Disciplined adherence to "Think Slow, Act Fast" requires overcoming entrenched behavioral biases. Leaders must consciously commit to not committing prematurely, embracing an "open mind" and allowing thorough investigation before locking into a solution. This commitment to open-mindedness and iterative planning helps mitigate the "uniqueness bias" and the "commitment fallacy". Organizations should integrate lessons learned into their processes, documenting what happened, why, its impact, and what can be improved. These insights should be stored accessibly and applied proactively during project kick-offs and ongoing check-ins to avoid repeating mistakes.

While the sources provided do not directly discuss the role of religion in solving big problems, they strongly emphasize discipline as a foundational element of successful project management. This discipline manifests in several ways:

• Intellectual Discipline: The rigorous application of critical thinking, challenging assumptions, and seeking accurate data, even when it is uncomfortable.
• Planning Discipline: Dedicating sufficient time and resources to meticulous planning and iterative testing before execution.
• Execution Discipline: Maintaining focus on the end goal, acting quickly once a solid plan is in place, and continuously monitoring progress against defined "inchstones" and milestones.
• Cultural Discipline: Fostering an environment where transparency, accountability, and psychological safety are consistently upheld, enabling effective communication and problem-solving.

7. Application to Enterprise IT, AI, and Software Engineering

The principles from How Big Things Get Done are directly applicable to the unique challenges of Enterprise IT, AI, ML, and Software Engineering:

• Addressing IT's "Fat Tail": Given that IT projects have some of the fattest tails in terms of cost overruns and are prone to disastrous outcomes, adopting Flyvbjerg's methodologies is not just beneficial but imperative. Implementing Reference Class Forecasting, using historical data from similar software development projects, cloud migrations, or AI model training efforts, can provide far more realistic budget and schedule estimates than traditional, often optimistic, internal forecasts.
• Strategic Planning for AI/ML Initiatives: For AI and ML, "thinking from right to left" is crucial. Before embarking on complex model development or data pipeline construction, organizations must clearly define the business problem the AI is solving, the desired customer experience, and the precise metrics of success. This ensures that AI projects remain aligned with strategic objectives and avoid becoming "vaporware" – projects that promise much but deliver little.
• Iterative Development and Testing: The "Pixar Planning" model directly translates to agile and DevOps methodologies in software and AI. Rigorous, continuous testing throughout the development lifecycle (unit, integration, system, user acceptance testing for software; validation, robustness, bias testing for AI models) acts as the "experiri" (experiment and experience) phase, catching flaws when they are cheap to fix, before costly deployment. This involves adopting a "maximum virtual product" mindset for critical systems, simulating real-world conditions extensively before release.
• Modularity in System Design: The "Build with Lego" heuristic is a cornerstone of modern software architecture, promoting microservices, APIs, and component-based design. This reduces complexity, enables parallel development, facilitates independent deployment, and allows for easier scaling and adaptation, directly countering the inherent complexity of large IT systems. For AI/ML, this means modularizing data ingestion, feature engineering, model training, and deployment pipelines.
• Leadership and Culture in Tech: Tech leaders must foster psychological safety to ensure that engineers and data scientists feel comfortable raising concerns about technical debt, architectural flaws, or potential biases in AI models. They must challenge the "bias for action" when it comes to irreversible decisions, prioritizing thorough planning and risk mitigation over rushed implementation. The ongoing learning from "lessons learned" should be embedded in post-mortems and knowledge-sharing, using structured templates and accessible repositories for future reference.
• Leveraging AI for Project Management: While the book highlights AI support for "inchstones" to monitor project timelines, external sources mentioned in the original query point to AI's growing role in generating schedules, risk logs, and assisting with cost estimation. This suggests a synergistic relationship: the principles of "Think Slow, Act Fast" provide the human-driven strategic framework, while AI can enhance the speed and accuracy of tactical project management functions, potentially eliminating 80% of traditional PM work by 2030.

8. Conclusion: A Path to Success for Digital "Big Things"

The pervasive failure of big projects, especially in the complex and rapidly evolving domains of Enterprise IT, AI, ML, and Software Engineering, is not an insurmountable fate. By confronting psychological biases and strategic misrepresentation head-on, organizations and countries can shift from a pattern of "Think fast, act slow" to the transformative "Think Slow, Act Fast" approach. This requires:

• Strategic Leadership: Cultivating experienced leaders with "phronesis" who build cohesive teams and foster psychological safety, empowering open communication and rapid problem identification.
• Rigorous Planning: Adopting "right-to-left" thinking to clarify ultimate goals, employing "Pixar Planning" for iterative testing and refinement, and utilising "Reference Class Forecasting" to generate accurate, data-driven estimates and manage inherent "fat-tailed" risks.
• Structural Discipline: Embracing "modularity" to break down complex digital projects into manageable, scalable components, simplifying development and deployment.
• Continuous Learning: Embedding "lessons learned" processes throughout the project lifecycle to drive ongoing improvement and avoid repeating past mistakes.

By embracing these evidence-based principles, Enterprise IT, AI, and Software Engineering initiatives can move beyond the "Iron Law of Megaprojects" and consistently deliver successful outcomes, turning ambitious visions into triumphant realities for organizations and countries worldwide.

Reference List

• ActiveCollab. (2025, May 12). Lessons learned in project management: Key steps and breakdown. ActiveCollab (Blog).
• BCG Henderson Institute. (n.d.). How Big Things Get Done with Bent Flyvbjerg [Interview]. BCG Henderson Institute.
• Cooper, D., Grey, S., Raymond, G., & Walker, P. (n.d.). Project Risk Management Guidelines: Managing Risk in Large Projects and Complex Procurements. Available from Amazon.ca.
• Effective Altruism Forum. (n.d.). Lessons on project management from “How Big Things Get Done”. Effective Altruism Forum.
• Elevate Constructionist. (2024, October 3). Reference Class Forecasting: Predicting Construction Success. Elevate Constructionist (Blog).
• First Friday Book Synopsis. (n.d.). How BIG Things Get Done: The Surprising Factors That Determine the Fate of Every Project, from Home Renovations to Space Exploration and Everything In Between – Here are my Seven Lessons and Takeaways. First Friday Book Synopsis (Blog).
• Flyvbjerg, B., & Gardner, D. (2023). How Big Things Get Done: The Surprising Factors That Determine the Fate of Every Project, from Home Renovations to Space Exploration and Everything In Between. Currency, an imprint of Random House, a division of Penguin Random House LLC.
• Hewitt, J. (2025). 4 ways to tackle optimism bias. APM (Blog).
• Paul, R., Elder, L., & Niewoehner, R. (n.d.). Engineering Reasoning: A Miniature Guide (Based on Critical Thinking Concepts & Principles). CriticalThinking.org. (Sourced from pdfcoffee.com_engineering-reasoning-critical-thinking-pdf-free.pdf).
• r/20minutebooks. (n.d.). How Big Things Get Done - Book Summary. Reddit.
• Readingraphics. (2025). Book Summary - How Big Things Get Done. Readingraphics.
• Taylor & Francis. (n.d.). Book Review: How Big Things Get Done by Bent Flyvbjerg and Dan Gardner. Taylor & Francis (Journals/Books platform).
• Wexler, N. (2019). The Knowledge Gap: The Hidden Cause of America’s Broken Education System—and How to Fix It. Avery.