What Governance Actually Means
Governance is not control or committee theater. It sets boundaries, assigns accountability, and brings evidence into ...
Software Will Fail. Recovery Is the Real Metric
9 min read
Stop promising that software will never fail
05.09.2026, By Stephan Schwab
Every post-incident meeting eventually reaches the same impossible request: guarantee it can never happen again. That promise does not produce reliability. It produces larger releases, slower feedback, and approval theater. Software operates in a changing world; failure cannot be abolished by committee. A CTO’s job is to match reliability to business risk, keep changes small, expose trouble quickly, and make recovery ordinary enough that nobody needs a hero.
The meeting after an outage usually begins with evidence and ends with theater. Someone shows the timeline. Someone asks why the test environment missed the problem. A supplier explains that production behaved differently. Then comes the sentence everybody knew was coming: “Can you guarantee this will never happen again?”
The room can tell the truth about risk and recovery. Or it can promise perfection, add three approval gates, and make the next failure larger.
The second choice is surprisingly popular. It produces visible action: more sign-offs, a longer checklist, perhaps a new committee with an impressive name. None of that has to improve detection, containment, or recovery. It only has to make failure look less likely on a slide.
Once zero failure becomes the target, resisting change looks prudent. Releases grow larger so they can happen less often. Rollback becomes harder because one deployment now contains three urgent fixes, a database migration, two regulatory changes, and the feature the CEO already announced. The organization pursued perfection and built a hostage situation.
Software still needs serious quality: automated tests, continuous integration, observability, careful design, security, recovery plans, and competent developers. But quality is not the absence of every failure. It is delivering the behavior users need at the reliability the business requires while preserving the ability to change it again.
A perfectly stable system nobody can safely change is not a triumph. It is a museum exhibit with a payroll.
MTBF rewards the wrong behavior
MTBF can describe how often a service fails. It cannot tell you whether the organization is good at changing that service.
Mean time between failures has an intuitive appeal. Observe a system, count the failures, and calculate the average interval between them. Longer sounds better. For physical equipment, where components age and eventually need replacement, that can be useful operational information.
Software is not a bearing.
Code does not become thinner by Friday or rust after another million requests. Its operating context simply refuses to stay still. Teams deploy. Users find new combinations. Data grows. Certificates expire. Traffic changes shape. Providers update services. Vendor APIs reinterpret fields they swore were stable. Attackers try inputs nobody designed for.
MTBF is not useless. For a stable service, it can describe incident frequency and support staffing or capacity decisions. It becomes dangerous when leadership treats it as the primary target for software delivery.
There is a wonderfully easy way to increase the time between deployment failures: stop deploying.
The metric improves immediately. Nobody’s business does.
Changes accumulate into larger batches. Feedback arrives later. Developers forget why decisions were made. Production becomes sacred ground that must be protected from development, which is a clever way to protect today’s defects from tomorrow’s fixes.
MTBF measures the distance between visible failures. It does not measure batch size, release fear, recovery capability, or the opportunities lost while everyone waits for the perfect release.
DORA measures the delivery system
DORA does not ask whether you move fast or stay stable. It asks whether you have built the capability to do both.
The familiar DORA model was known as the Four Keys: deployment frequency, lead time for changes, change fail rate, and mean time to recover.
The model evolved. DORA now describes five software delivery performance metrics, grouped into throughput and instability. Its history of the metrics explains why the old MTTR label became failed deployment recovery time and why deployment rework rate was added.
| View | Metric | What it reveals |
|---|---|---|
| Throughput | Change lead time | How long a committed change takes to reach production successfully |
| Throughput | Deployment frequency | How often the organization can put changes into production |
| Throughput | Failed deployment recovery time | How long it takes to recover when a deployment impairs the service and needs intervention |
| Instability | Change fail rate | What proportion of deployments needs a rollback, hotfix, patch, or other immediate remediation |
| Instability | Deployment rework rate | What proportion of deployments is unplanned work to correct a user-facing production problem |
The value lies in the tension. Deployment frequency without change fail rate rewards reckless output. Change fail rate without deployment frequency rewards inactivity. Lead time without rework can celebrate a pipeline that rapidly ships rubbish. Recovery time without user-facing reliability can make repeated incidents look efficient.
Together, the metrics describe whether a delivery system moves useful changes safely. DORA’s research rejects the lazy assumption that speed and stability are opposites. Strong performers tend to do well at both. Smaller changes provide evidence sooner and are easier to recover.
The metrics belong to an application or service in its own context. Blending every team into one corporate number produces an impressive average and very little truth. Comparing a mobile application with a regulated mainframe system is management theater with decimals.
Use DORA to see whether a delivery system improves over time, not to build a league table. The moment a metric becomes a quota, people learn to satisfy the quota. The software remains stubbornly uninterested in the bonus scheme.
Recovery is a delivery capability
The useful recovery question is not "Who caused the incident?" It is "Why did restoring service require heroics?"
MTTR challenged the fantasy that prevention was the only respectable reliability strategy. Mean time to recover, restore, repair, or resolve—choose your favorite expansion and begin the acronym argument—shifted attention from blame to capability.
DORA narrowed the term for good reason. Failed deployment recovery time covers impairments caused by production changes. It does not mix those failures with a data-center outage, a severed network cable, or every other incident that might affect a service. That makes it a cleaner measure of software delivery.
The broader recovery conversation still matters. Services fail without deployments, and customers do not care whose infrastructure caused the outage. A CTO should still keep the questions separate:
- Did our delivery process introduce the failure?
- How quickly did users receive acceptable service again?
- Did we detect the problem before users reported it?
- Did recovery use the normal path, or did it require privileged improvisation?
- How much follow-up work did the incident create?
A single average can hide a great deal. Ten two-minute rollbacks and one eight-hour disaster can still produce a respectable mean. Severity, blast radius, detection time, and the distribution of recovery times matter. Metrics are instruments, not absolution.
Fast recovery is not an operations trick performed after developers finish. It is a property of the whole delivery system, designed when the change is shaped.
A small change is easier to understand, test, observe, reverse, and replace. An automated deployment gives the emergency fix the same path as ordinary work. A feature flag can separate deployment from exposure. A canary release limits the blast radius. Useful monitoring makes user harm visible. Clear ownership removes the ten-minute conference call devoted to discovering who is allowed to act. A tested rollback prevents the database from becoming a one-way door.
None of those practices guarantees success. They make failure smaller and less dramatic.
That is why the technical practices that drive business results reinforce one another. Tests increase confidence. Continuous integration exposes incompatible changes while they are still small. Refactoring keeps recovery paths understandable. Frequent small releases reduce the amount of uncertainty in each deployment.
The result is control in the only form complex software allows: observe reality and respond without making it worse.
Long approval chains rarely improve recovery. They may prevent some errors, but they cannot replace production evidence. When something breaks, the same approval path becomes part of the outage. The emergency change waits because the only person allowed to approve it is on an airplane.
If the normal path is too slow for recovery, the normal path is too slow.
Reliability needs a business target
The right reliability target is not "as much as possible." It is the level users and the business actually require.
DORA measures software delivery performance. It does not replace service-level indicators and objectives for user-facing reliability.
That distinction prevents two equally bad conclusions. Product cannot use frequent deployment as proof that the service is healthy. Operations cannot use uptime as proof that the delivery system is healthy. A service can look reliable because nobody dares change it. A team can deploy frequently while users suffer repeated degradation. Both dashboards can be green.
Google’s SRE guidance on embracing risk makes the uncomfortable point directly: 100% reliability is usually the wrong target. Extreme reliability has a nonlinear cost. Users often cannot perceive the final increment, and the effort spent chasing it cannot improve features, reduce debt, or lower operating cost.
The useful alternative is an explicit service-level objective and an error budget. Decide what reliability users need and what the business can justify. Measure actual user experience against that target. When the service consumes too much of its error budget, invest in reliability and slow risky change. When the service operates comfortably within the budget, keep improving the product.
For safety-critical, privacy-critical, or financially critical behavior, the acceptable budget may be tiny. The principle still holds: make the risk explicit, choose controls that match the consequence, and verify them. “Never fail” is not a control. It is a wish wearing a tie.
MTBF may remain useful for hardware fleets, infrastructure components, and incident analysis. It cannot substitute for an SLO or become the north star for an organization whose job is to change software.
What a CTO can credibly promise
Do not promise that failure is impossible. Show that failure cannot quietly become catastrophe.
The board does not need a lecture about distributed systems. It needs a credible statement of risk and evidence that the organization can manage it:
We cannot guarantee that software will never fail. We can show that changes are small, independently recoverable, and observed in production. We know how often deployments require intervention, how quickly we restore service, how much unplanned rework failures create, and whether users receive the reliability we agreed to provide. When those signals worsen, we invest in the constraint instead of hiding it behind a release freeze.
That commitment is stronger than “never again.” It is measurable, gives management a decision rule, and makes reliability a shared business concern instead of a moral burden placed on developers after every incident.
It also gives the CTO better questions for the next review:
- Are we deploying less often because the system is stable, or because change is frightening?
- How large is the typical production change?
- Can a failed change be reversed without a special meeting?
- Does our recovery path use the same tested pipeline as an ordinary release?
- Which service-level objective represents what users actually experience?
- Is unplanned corrective work rising even while incident counts look stable?
Those answers reveal far more than a longer MTBF ever will.
Software lives in a changing world. The mature response is not resignation or carelessness. It is relentless work to prevent the failures you understand, expose the ones you do not, contain their impact, and recover while the incident is still small.
Perfection is not the standard. A delivery system that tells the truth and recovers without heroics is.
Talk It Through
Tell me what is happening. I listen, ask a few practical questions, and reflect back what I see: where the risk may sit, what may be blocking delivery, and what looks worth checking next. No pitch, no obligation. Confidential and direct.
Talk it through. Practical reflection, no pitch.
Start a Conversation
🎭 This Week's Episodes
Telenovelas show what we can't say in client meetings. The drama is heightened, but the patterns are real.
Three Ways to Understand a Country
Stefan drives Sophie to the Pacific for two days. The beach is big enough to make her feel appropriately small. Back at the finca the rain keeps th...
The Resignations
Nathan Cole resigns on a Monday morning with two sentences and no argument left in his body. By Tuesday his desk is empty and the company's institu...
Technical Consultancy
Embedded Delivery Partner
Embedded into your team as an active contributor, reducing delivery friction and helping important work move cleanly.
Technical Advisor
Peer-style technical assessments before far-reaching decisions; reduce architectural and product risk early.
Product & Delivery
Ship working software to real users earlier. Measure impact and adapt based on evidence rather than assumption.
Custom Software Development
High quality, maintainable software. Short-term augmentation that leaves lasting capability in your own team.
Recent Articles
- When Everything Is Finally Under Control (28.09.2026)
- AI Makes Technical Execution Cheap. Judgment Stays Scarce. (24.09.2026)
- The LLM Will Not Come Back Tomorrow (19.09.2026)
- The One-Person AI Company Is a Greed Fantasy (15.09.2026)
- Test Production Without Leaving a Mess (10.09.2026)