Make the product dependable before deciding to rewrite it
A practical sequence for reducing recurring failures and learning whether the architecture really needs replacement.
Use incidents to identify specific failure mechanisms, improve recovery and establish a fair basis for a larger architectural decision.
Understand what is actually failing
Repeated incidents make a rewrite feel attractive. A fresh system seems to promise relief from accumulated mistakes and a design the team can finally understand. Sometimes replacement is warranted. Before committing to it, distinguish failures caused by a fundamental limitation from failures caused by missing visibility, unsafe releases or an unhandled dependency.
Collect a small set of recent incidents and reconstruct them in customer terms. What could people no longer do? How long did the disruption last? How was it detected? What restored service? Include the steps that made the impact worse or prolonged recovery. The sequence often reveals a more specific problem than “the old system is unreliable.”
For example, a hypothetical collaboration tool might repeatedly fail after configuration changes. If the application is otherwise stable, replacing its core data model may do little to prevent the next incident. Validating configuration, checking the release and making recovery straightforward would address the observed mechanism more directly.
Choose the experience to protect
Reliability work needs priorities. Identify the customer journeys whose interruption is most damaging and define how to measure acceptable service. A product can have different expectations for signing in, saving work and producing an occasional export. Treating every feature as equally urgent makes it difficult to decide where effort belongs.
A service level objective makes one of those expectations measurable. Google’s SRE workbook explains how objectives and error budgets can inform the balance between reliability and feature work. The value comes from using the measure to make decisions, rather than placing a percentage on a dashboard and leaving priorities unchanged.
Discuss what happens when the objective is missed. The response could be investigating a recurring failure, reducing the size of releases or completing a recovery improvement before adding related functionality. Keep the action tied to the evidence. Reliability targets are useful when they help the team protect a real customer experience and explain its choices.
Contain failures at their boundaries
Look at the dependencies on the protected journey. An unavailable provider should lead to a defined outcome: a clear failure, a bounded wait, a recoverable pending state or a useful reduced service. The appropriate choice depends on what the operation means. Returning stale information might be acceptable for a recommendation and harmful for a decision that requires current data.
For background work, make the state visible and understandable. Accepted, processing, completed and failed should represent different conditions. Decide what happens when a worker stops midway through a task. Can it resume safely? Can the same task be attempted again without creating a second result? Who can identify work that has become stuck?
Review concurrency as well. Two individually correct actions may behave incorrectly when they overlap. A hypothetical reservation system must protect the rule that one place cannot be assigned twice. PostgreSQL’s constraints offer mechanisms for enforcing certain stored-data rules. The application must still interpret a rejected operation and give the customer a useful outcome. Reliability includes correct behavior, not simply a lack of crashes.
Make releases easier to evaluate
A large release changes many things at once and makes an unexpected result harder to explain. Smaller releases reduce the number of competing explanations. Pair them with checks that exercise the affected customer journey and a clear view of the deployed version. This creates a shorter path from “something changed” to an actionable diagnosis.
A canary release exposes a change to a limited portion of traffic before wider rollout. Google’s guidance describes comparing the canary with a control and evaluating meaningful signals. The technique needs enough representative activity to reveal problems. A quiet canary receiving no relevant requests has provided little evidence about the changed feature.
Decide what will stop a rollout before beginning it. That decision should include customer failures and harmful data behavior, not only infrastructure alarms. Also check whether recovery is truly available. Reverting application code may be straightforward while reversing a destructive data change is not. A release plan should account for both.
Let the evidence inform replacement
Once the team can observe failures and make safer changes, revisit the larger architectural question. Which important failure mechanisms remain? Why are targeted repairs insufficient? What behavior would a replacement make easier to guarantee? These questions create a testable argument for change rather than a general expression of frustration.
Include the transition itself in the comparison. Customers will continue to use the current product while a replacement is built. Existing data must remain interpretable, integrations must keep working and the team must operate the overlap. A proposed architecture should be judged together with the work required to reach it.
The immediate aim is a product that people can depend on and a team that can reason about its failures. That may lead to a careful replacement of one component. It may lead to a broader migration. It may show that the existing structure is adequate once its operational weaknesses are addressed. Each outcome is useful when supported by observed behavior and a clear account of the remaining limits.
Frequently asked questions
Do recurring outages mean we need a full rewrite?
Not by themselves. Investigate the failures, protect the customer journeys that matter and improve recovery. Consider replacement when a specific limitation cannot reasonably be resolved in the existing product.
What should we improve before considering a rewrite?
Identify the failure mechanism and its customer impact. Review timeouts, repeated requests, data rules, monitoring, rollback and release checks. Make targeted changes and compare the results before deciding on a larger replacement.
How can we release changes with less risk?
Define the customer behaviour to preserve, expose a change to a limited scope where appropriate, observe its effect and prepare a way to recover. Release checks should look at customer outcomes rather than server availability alone.