Software Will Fail. Recovery Is the Real Metric
MTBF can reward inactivity. DORA shows whether your organization changes software safely and recovers when reality di...
Test Production Without Leaving a Mess
10 min read
A Green Deployment Is Not a Working Product
10.09.2026, By Stephan Schwab
A green deployment proves that a server answers, not that a customer can complete the journey the business sells. Staging cannot close that gap; it only proves staging works. A controlled, disposable production journey can create temporary data, exercise a bounded path, verify the result, and remove the data through supported behavior. Safety depends on strict limits and cleanup that fails loudly when it cannot be confirmed.
The release pipeline is green.
The container started. The health endpoint returned 200. The database migration completed. The load balancer can see the service. Somewhere, a dashboard has acquired a reassuring shade of green.
None of that proves the product works.
It proves the infrastructure can answer the questions the infrastructure team thought to ask. The customer asks different questions. Can I create an account? Can I sign in? Do my permissions match what I bought? Does the browser load the same application the deployment claims to have shipped? Can I reach the workflow that justifies the invoice?
Those questions cross boundaries. DNS, certificates, routing, static assets, runtime configuration, authentication, authorization, database state, background work, and the browser itself all get a vote. A deployment check can validate each component and still miss the broken journey between them.
That is how teams arrive at the classic software paradox: everything is healthy except the thing customers came to use.
Staging Proves Staging
The closer a test gets to the real customer journey, the more embarrassing the differences between staging and production become.
Staging environments are useful. They catch integration defects, give people somewhere to explore, and reduce the number of surprises that reach customers.
They are also magnificent liars when asked to certify production.
Staging often has different data, traffic, credentials, network rules, feature switches, domains, certificates, identity-provider settings, third-party integrations, and deployment history. Sometimes it runs a smaller version of the production architecture. Sometimes it runs a hopeful approximation assembled three redesigns ago. Everyone knows this. The release ritual proceeds anyway because the organization needs a box marked validated.
The problem is not that staging differs from production. A perfect replica would be expensive, difficult to protect, and still wrong five minutes after production changes.
The problem is expecting an imitation to answer a question about the original.
Reliable software comes from controlled change and fast evidence. Production verification is one way to shorten the distance between what the organization believes and what customers can actually do.
The useful question is not, “Did this pass before deployment?”
It is, “Can we prove the essential journey works where customers use it now?”
Verification Is Not a Second Deployment
A production journey should verify the product independently of the event that changed it.
Many teams attach a smoke test to the end of deployment and call the problem solved. Better than nothing. Still incomplete.
When verification only runs as a deployment step, it inherits the deployment’s assumptions and urgency. The same event changes the system, checks the system, and declares success. If the check flakes, release pressure encourages someone to rerun it until the color improves. If production drifts later because of configuration, credentials, data, certificates, or a vendor dependency, the test sleeps until the next release.
A serious production journey can run independently.
Run it after a release. Run it before a major commercial event. Run it after an infrastructure change. Run it when support reports a vague problem nobody can reproduce. Run it manually when a CTO wants evidence rather than a chorus of “looks fine from here.”
Independence changes the meaning of the result. The test stops being a ceremonial appendix to deployment and becomes an operational capability.
That capability should still be used with restraint. Not every test belongs in production. A complete browser suite is usually too broad, too slow, and far too creative with state. The goal is not to turn live customers into involuntary test fixtures. The goal is to select one or two representative journeys whose success proves that the product’s critical seams still connect.
Small. Deliberate. Boring.
Boring is underrated in production.
Make the Journey Disposable
Temporary test data is only temporary when the system can prove it has gone.
The safe pattern begins by refusing to borrow customer state.
Do not log in as a friendly customer. Do not select a record that “should always be there.” Do not reserve a permanent test account that slowly accumulates permissions, preferences, corrupted state, and folklore. Shared test identities become tiny production pets. Nobody knows who changed them, which assumptions still hold, or whether deleting them would break an executive demo next Thursday.
Create what the journey needs, mark it unmistakably, and give it a unique identity. Use the public product path where that path is what you want to verify. Exercise the bounded behavior. Then remove the state through a supported administrative capability and confirm that it is gone.
This does more than keep the database tidy.
It tests lifecycle ownership.
Can the system create a customer cleanly? Can administrators remove one cleanly? Do organization boundaries hold? Does deletion actually remove the product records the company claims belong to that customer, or does it leave a sedimentary layer across forgotten tables and services?
Most privacy and data-retention discussions happen in policy documents. A disposable production journey asks the software.
The answer can be uncomfortable. Good. An uncomfortable answer from a controlled test is cheaper than the same discovery during a customer deletion request, an audit, or an incident.
Cleanup Is an Assertion
The usual cleanup strategy is a line near the bottom of the test that runs if everything above it behaved nicely.
Production has opinions about that optimism.
Browsers crash. Networks time out. Assertions fail halfway through. Authentication expires. An administrator screen changes. The test runner gets interrupted. If cleanup is merely the happy path’s final step, the live system eventually becomes a museum of abandoned synthetic customers.
Cleanup belongs in the failure path too. It must run when the journey succeeds, when the journey fails, and when the assertion halfway through discovers something ugly. More importantly, cleanup must be verified.
“We sent the delete request” is not verification.
“The interface confirmed success” is better, but still only half the claim.
The useful assertion is that the temporary identity and its product state can no longer be found through the supported administrative view. If that cannot be confirmed, the run fails loudly even if every customer-facing step passed.
That changes cleanup from housekeeping into product behavior.
It also exposes a class of defects conventional testing often ignores. Creation gets lavish attention because creation drives growth. Deletion gets a button, a confirmation dialog, and hope. Yet deletion is where tenant isolation, data ownership, retention rules, and administrative control reveal whether the architecture has a coherent model or merely a collection of foreign keys with good intentions.
Prevent Side Effects Before They Become Apologies
The safest production test is designed around what it must never trigger.
A live journey cannot be treated like an ordinary end-to-end suite pointed at a more exciting URL.
Production actions may charge money, send messages, create documents, trigger integrations, start fulfillment, notify salespeople, wake support systems, or feed analytics. A browser does not care whether the action is commercially awkward. It sees a button.
The journey needs an explicit side-effect budget: ideally zero external effects and a very small, reversible amount of internal state.
That means choosing read-only behavior after setup wherever possible. It means giving the temporary identity only the entitlement required for the verification. It means using a dedicated administrative identity with narrow authority, not an all-powerful credential passed around like a conference-room key. It means designing test-aware boundaries where irreversible actions cannot occur accidentally.
Serial execution matters too. Two production journeys racing to create and remove similar state can turn an elegant test into a small distributed-systems seminar. The CTO does not need another seminar. Run the critical path one at a time unless the purpose is explicitly to test concurrency.
These constraints are not signs that production testing is inherently foolish.
They are signs that production is real.
Any organization unable to name which actions have external consequences already has a larger governance problem than testing.
Do Not Promise Invisibility
No retained business data is a credible target. No trace anywhere is usually fiction.
The phrase “leave no trace” sounds responsible and usually collapses under five minutes of scrutiny.
Production systems emit access logs, security events, metrics, traces, audit records, and workflow metadata. The verification itself should produce a report. Those artifacts exist for good reasons. Erasing them to make a test metaphysically invisible would weaken the evidence and may violate the very controls the organization claims to value.
Use precise language instead.
The temporary customer and product state should be removed. Operational evidence may remain according to normal retention and security policy. The test report should remain long enough to explain what was checked, what failed, and whether cleanup completed.
This distinction matters to CTOs because vague promises become liabilities. “Trace-free” invites someone to discover a log line and declare the control dishonest. “No retained customer or product state, with normal operational records preserved” describes an architecture people can actually build and audit.
Honesty is not a downgrade.
It is a control.
The Report Is Part of the Product Evidence
A production test that prints “passed” in a terminal and disappears has limited organizational value.
Keep a report whether the journey passes or fails. Capture which steps ran, where the failure occurred, and whether cleanup was confirmed. Make the result accessible to the people who must decide whether a release is safe, an incident is resolved, or a customer report is credible.
This is where browser-level verification earns its keep. A health check can say the application responded. A browser report can show that a user reached the expected screen after navigating the same layers a real customer crosses.
Do not confuse this with surveillance theater. More screenshots and longer videos do not automatically create more truth. Collect enough evidence to diagnose the journey and prove the cleanup contract. Avoid turning temporary customer data into a permanent report attachment.
Evidence needs its own boundaries.
The best report answers three questions quickly:
- Did the essential production journey work?
- If not, where did observed behavior diverge?
- Was every temporary piece of customer and product state removed?
Anything else is supporting material.
What This Capability Reveals About the System
The interesting part is not the browser automation.
The interesting part is what the organization must understand before it can automate the journey safely.
Someone must know the real customer path. Someone must know which actions create external consequences. Someone must know how entitlements are granted and revoked. Someone must know what data belongs to a customer and how deletion propagates. Someone must know which evidence must remain. Someone must decide what a failed cleanup means operationally.
That knowledge usually sits across product, development, operations, security, and support. Building the verification forces those views into one executable statement.
This is software responsibility in unusually honest clothing.
The company may describe the work as customer assurance, release governance, operational readiness, or risk reduction. Fine. Underneath, it is still requirements, permissions, browser behavior, data lifecycle, observability, and automated verification.
The CTO’s opportunity is not to sponsor another testing project. It is to turn a critical business claim into something the live system can prove repeatedly:
A real user can complete the essential journey, and the organization can exercise that proof without harming customers or polluting their data.
That is a much stronger claim than “deployment succeeded.”
It is also harder to fake with a dashboard.
Start With the Journey That Would Embarrass You Most
Do not begin by automating everything.
Choose the smallest customer journey whose failure would make the green infrastructure dashboard look ridiculous. Define the exact production evidence it should produce. List the side effects it must never trigger. Give it disposable state and narrow permissions. Make cleanup unavoidable and verifiable. Preserve an honest report.
Then run it when it matters.
The first result may fail. That is not an argument against production verification. It is the reason to have it.
Staging told you the rehearsal worked.
Production just told you whether the audience can hear the play.
Talk It Through
Tell me what is happening. I listen, ask a few practical questions, and reflect back what I see: where the risk may sit, what may be blocking delivery, and what looks worth checking next. No pitch, no obligation. Confidential and direct.
Talk it through. Practical reflection, no pitch.
Start a Conversation
🎭 This Week's Episodes
Telenovelas show what we can't say in client meetings. The drama is heightened, but the patterns are real.
Three Ways to Understand a Country
Stefan drives Sophie to the Pacific for two days. The beach is big enough to make her feel appropriately small. Back at the finca the rain keeps th...
The Resignations
Nathan Cole resigns on a Monday morning with two sentences and no argument left in his body. By Tuesday his desk is empty and the company's institu...
Technical Consultancy
Embedded Delivery Partner
Embedded into your team as an active contributor, reducing delivery friction and helping important work move cleanly.
Technical Advisor
Peer-style technical assessments before far-reaching decisions; reduce architectural and product risk early.
Product & Delivery
Ship working software to real users earlier. Measure impact and adapt based on evidence rather than assumption.
Custom Software Development
High quality, maintainable software. Short-term augmentation that leaves lasting capability in your own team.
Recent Articles
- When Everything Is Finally Under Control (28.09.2026)
- AI Makes Technical Execution Cheap. Judgment Stays Scarce. (24.09.2026)
- The LLM Will Not Come Back Tomorrow (19.09.2026)
- The One-Person AI Company Is a Greed Fantasy (15.09.2026)
- Test Production Without Leaving a Mess (10.09.2026)