Timeout, Retry, and Circuit Breaker: correct order, configuration, and common mistakes

With three layers and four attempts per layer, one logical operation can generate up to 64 calls to the deepest component. If every attempt waits one second and backoff is added between attempts, the system may continue consuming resources long after the result has stopped being useful to the user.

The central idea is simple:

This article explains what each pattern must limit, how to choose a defensible composition order, which failures may be retried, how to prevent retry storms, and which signals prove that the policy works in production.

What problem each pattern solves

All three patterns act on remote calls, but they do not protect against the same risk.

A timeout prevents a call from occupying resources indefinitely. A retry attempts to recover from a transient failure. A Circuit Breaker stops insisting when recent evidence suggests that new calls have a low probability of succeeding.

The design error appears when each pattern is configured in isolation:

• The timeout is chosen without calculating how many attempts fit inside it.
• Retries are enabled at several layers.
• The Circuit Breaker observes a different signal from the one the team believes it is measuring.
• Every service resets the time budget instead of propagating the remaining deadline.
• The system repeats an operation without an idempotency guarantee.

None of these patterns eliminates the need for sufficient capacity, concurrency limits, and observability. A Circuit Breaker is not a bulkhead: it may reject calls while open, but it does not necessarily limit how many concurrent calls pass through while closed.

Timeout: limit waiting and preserve capacity

An unbounded wait is not only a user-experience problem. While a request is waiting, it may retain threads, connections, memory, pool slots, locks, buffers, or execution capacity. As volume increases, a slow dependency can turn waiting into resource exhaustion.

Per-attempt timeout

A per-attempt timeout limits one physical interaction with the dependency.

Examples include:

• An HTTP request.
• A gRPC call.
• A database query.
• An operation against a queue or external service.

Do not assume that one timeout configuration covers every phase of communication. Depending on the library, separate limits may exist for DNS resolution, connection establishment, the TLS handshake, connection-pool acquisition, writing, reading, and total duration. The only reliable way to know what is being limited is to inspect the actual client semantics and verify them through tests.

Per-operation timeout

A logical operation may contain more than one attempt and more than one remote call.

For example, registerOrder may perform:

1. Remote validation.
2. Inventory reservation.
3. Persistence.
4. Event publication.

A 400 ms timeout on each call does not mean the complete operation lasts 400 ms. Sequential calls, retries, backoff waits, and internal queues all contribute to end-to-end latency.

End-to-end deadline

A deadline is the point in time after which the result is no longer useful to the consumer. Unlike a local timeout, it should travel with the request across service boundaries.

Total deadline: 2,000 ms
Time consumed in the first layer: 700 ms
Remaining time: 1,300 ms

The next layer should not start a fresh 2,000 ms timer. It should operate within the remaining 1,300 ms and preserve enough margin to return the response.

gRPC distinguishes a timeout, expressed as a maximum duration, from a deadline, expressed as a point in time, and documents propagation of the remaining time to downstream calls. This prevents each service from resetting the budget and continuing work after the original caller has abandoned the request.

Operational decision

A timeout should be derived from four inputs:

1. The operation's SLO or functional limit.
2. The dependency's historical latency distribution.
3. The time required by the remaining layers.
4. The cost of cutting off too early versus the cost of waiting too long.

Using only the average is dangerous. Budget design requires dependency latency percentiles segmented by operation, region, payload size, and load condition.

A timeout that is too low creates false failures and unnecessary retries. One that is too high keeps resources occupied, increases queueing, and delays detection of a degradation.

Validation

A timeout configuration is not validated merely because it produced no errors in a functional test. It must be tested under:

• Normal load.
• Load close to the system limit.
• Induced latency.
• Connection loss or reset.
• Pool saturation.
• Partial or slow responses.
• A completely unreachable dependency.

The primary metric is not only timeout_count. The system must also measure how much downstream work continued after the consumer exhausted its deadline.

Retry: repeat only when another attempt has a reasonable chance of succeeding

A retry is useful only when the next attempt may observe a different condition from the previous one.

This may be true when:

• A connection was reset transiently.
• A packet was lost.
• The request reached a faulty instance and the next attempt may be routed elsewhere.
• A service reports temporary overload and communicates when to try again.
• A known transient conflict can be retried safely according to the operation's semantics.

A technical error alone is not enough. Three conditions must also hold:

1. Recovery probability: another attempt may produce a different result.
2. Semantic safety: repeating the operation will not create duplicate or inconsistent effects.
3. Available budget: enough time and capacity remain for another attempt.

Common candidates

429 Too Many Requests and 503 Service Unavailable do not mean "retry immediately." HTTP allows the server to communicate a delay through Retry-After. Ignoring that signal and retrying immediately may intensify the overload that caused the response.

Failures that normally should not be retried

These categories are not universal. A 404 may be transient in an eventually consistent system. A 409 may represent a recoverable conflict. A 500 may be permanent for one specific payload. The policy must be based on contract semantics, not only on the HTTP status-code family.

Retry ownership

There must be one explicit owner for retries.

In a chain such as client -> API -> service -> database, enabling automatic retries at every hop without a coordinated budget is not defensible. The layer closest to the dependency understands its technical failures best; the upstream layer understands whether the complete operation is still useful. The design must make that tension explicit.

A defensible policy usually assigns one primary layer to retry and requires the others to propagate distinguishable failures, deadlines, and overload signals.

Idempotency: the requirement that prevents duplicate effects

An operation is idempotent when repeating the same intent produces the same observable effect as executing it once.

The critical case is not when the server rejects the request. It is when the server executes it and the response is lost:

1. The client sends the operation.
2. The server processes it and commits the change.
3. The response is lost.
4. The client observes a timeout.
5. The client retries.

From the client's perspective, the outcome is uncertain. From the server's perspective, the first attempt may have completed successfully.

HTTP idempotency is not the same as functional idempotency

RFC 9110 defines methods such as PUT, DELETE, and the safe methods as idempotent. It also warns that a non-idempotent request should not be retried automatically unless the client knows the actual semantics are idempotent or can determine that the first attempt was not applied.

That does not mean every PUT is safe in a real implementation, nor that every POST is impossible to retry. Safety depends on the actual contract.

Idempotency key

A common strategy is to associate a unique key with the business intent:

POST /orders
Idempotency-Key: 0f34d7b8-5f6e-4c23-a0a5-54e5f2a8a921

The server should:

1. Record the key and operation state.
2. Detect repeated requests.
3. Return the previous result or continue an incomplete execution.
4. Prevent the effect from being applied again.
5. Retain the record longer than the maximum retry horizon.

The key must not be reused for different payloads. The server may store a normalized request hash and reject an incompatible repetition.

Deduplication and operation state

Deduplication can be implemented at different levels:

• A unique database constraint.
• A log of received commands.
• An idempotency table.
• A stable business identifier.
• An operation state machine.
• A transactional inbox for message consumers.

The correct choice depends on the effect being protected. A key stored only in memory does not prevent duplicates after a restart. A table without an expiration policy grows indefinitely. A retention window that is too short allows a late retry to execute the operation again.

Validation

The minimum test should inject response loss after commit but before the client receives confirmation. The expected outcome is:

• One persisted effect.
• Two or more observable attempts.
• A consistent response for the same key.
• No duplicate publication or, in an at-least-once system, consumers capable of deduplicating.

Backoff and jitter: spread out and desynchronize attempts

Retrying immediately assumes that the condition that caused the failure disappeared within a few microseconds. For a saturated dependency, that assumption is usually false.

Exponential backoff

Backoff progressively increases the delay between attempts.

A conceptual sequence:

100 ms -> 200 ms -> 400 ms -> 800 ms

A common expression is:

delay_n = min(max_delay, base_delay × 2^(n-1))

The maximum cap prevents unbounded waits. The deadline must take precedence: if the delay plus the minimum time required for another attempt no longer fits, the system should not retry.

Jitter

If thousands of clients use the exact same sequence, they may all retry at the same time. Jitter adds randomness to break that synchronization.

Example of full jitter:

cap_n   = min(max_delay, base_delay × 2^(n-1))
delay_n = random(0, cap_n)

Backoff reduces frequency. Jitter reduces temporal correlation.

Google SRE recommends randomized exponential backoff and warns that synchronized retries can form amplifying waves. AWS documents the same failure mode in its guidance on timeouts, retries, backoff, and jitter.

What can go wrong

• Backoff without jitter: lowers retry frequency but preserves synchronized spikes.
• Jitter without a bound: may consume the deadline before the next attempt starts.
• Backoff reset at every layer: multiplies the total duration again.
• Ignoring Retry-After: contradicts an explicit server signal.
• A deterministic sequence per process: instances started together may remain synchronized.
• Immediate retry on overload errors: converts a recovery mechanism into additional traffic.

Retry storm: when recovery becomes additional load

A retry storm is a positive feedback loop:

1. The dependency degrades.
2. Latency increases.
3. More clients exhaust their timeout.
4. Clients retry.
5. Effective traffic exceeds original demand.
6. The dependency loses more capacity.
7. Latency and errors increase again.

Observable signals

A retry storm commonly appears as:

• QPS to the dependency exceeding the volume of original requests.
• An increase in attempts_per_request.
• Simultaneous growth in latency, timeouts, and retries.
• A falling proportion of failures recovered by retry.
• Saturated pools, queues, or connections.
• High traffic even though user demand has not increased.
• Retry-budget or Circuit Breaker rejections.

A common incident-response mistake is to treat retries as a secondary symptom. They may be part of the cause that keeps the platform outside its capacity envelope.

Circuit Breaker: stop calling when persistence increases damage

A Circuit Breaker summarizes the recent behavior of a dependency and decides whether a new call should be executed.

Its basic states are:

• Closed: allows calls and records their outcomes.
• Open: rejects calls without reaching the dependency.
• Half-Open: permits a limited number of probe calls to evaluate recovery.

A breaker may consider failure rate, slow calls, a minimum observation count, and specific exception classes. Frameworks such as Resilience4j separate failure rate from slow-call rate and require a minimum number of calls before calculating either threshold.

A Circuit Breaker does not necessarily limit concurrency while closed. If the risk is thread, connection, or in-flight request exhaustion, a concurrency limit or bulkhead is also required.

The full treatment of states, windows, thresholds, and recovery belongs in the dedicated article on Circuit Breaker and cascading failures. The decision addressed here is different: what call the breaker observes when retries are also present.

What the Circuit Breaker should observe

Before choosing thresholds, define the unit being observed.

Option 1: observe the logical operation

An operation with three attempts may look like this:

Attempt 1: timeout
Attempt 2: 503
Attempt 3: success
Logical result: success

If the breaker sees only the logical outcome, it records one success. This prevents one transient failure from opening the circuit after the retry recovered it, but it hides two physical failures and the extra load they created.

Option 2: observe every physical attempt

The same flow produces:

Failure
Failure
Success

The breaker sees three calls. It detects degradation earlier and may open during the sequence, but its failure rate no longer represents user operations. It may also open too aggressively if the policy was designed around final outcomes.

Option 3: observe two levels

A system with sufficient operational maturity may maintain:

• A technical metric or breaker per attempt against the dependency.
• A logical-outcome metric after retries.
• A separate retry-budget limit.
• An SLO based on end-to-end experience.

Two breakers are not always necessary. Two signal sets are: original operations and physical attempts.

Failures and slow calls

A service may return no explicit errors and still destroy the latency budget. A breaker may therefore consider:

• Failure rate.
• Slow-call rate.
• The duration above which a call is classified as slow.
• The minimum number of calls required before making a decision.
• Ignored exceptions.
• Exceptions counted as failures.
• A time-based or count-based window.

The slow-call threshold should not be copied from a generic configuration. It must be derived from the operation budget and the dependency's actual latency distribution.

Composition order

There is no universal order because changing the order changes the unit observed by each policy.

The notation below shows the outer wrapper first.

Option A: Retry inside the Circuit Breaker

Circuit Breaker
  └── Retry
        └── Per-attempt timeout
              └── Dependency

Expressed as a function:

CircuitBreaker(
    Retry(
        Timeout(call)
    )
)

The breaker receives one result after the retry policy finishes.

Advantages

• Measures success or failure of the logical operation.
• A recovered transient failure does not open the circuit.
• Its semantics are close to what the consumer observes.

Risks

• Intermediate attempts remain invisible to the breaker.
• The dependency may receive several calls before one failure is recorded.
• A breaker configured only around final failures may react too late.
• The success rate may look healthy while cost per operation has multiplied.

When it may be defensible

• A small number of total attempts.
• A strict retry budget.
• Idempotent operations.
• Explicit per-attempt metrics.
• A dependency able to absorb brief transient failures.
• A breaker supplemented by slow-call rate or a saturation signal.

Option B: Circuit Breaker inside Retry

Retry
  └── Circuit Breaker
        └── Per-attempt timeout
              └── Dependency

Expressed as a function:

Retry(
    CircuitBreaker(
        Timeout(call)
    )
)

Every attempt consults and updates the breaker.

Advantages

• Physical failures immediately influence breaker state.
• If the breaker opens, later attempts may be cut off before reaching the dependency.
• The signal is closer to the actual backend impact.

Risks

• Failure rate represents attempts, not user operations.
• A small set of requests with multiple retries can fill the window quickly.
• The retry policy may repeat after a breaker rejection if that rejection is incorrectly classified as retryable.
• The breaker may open on transient failures the system would otherwise recover.

When it may be defensible

• The dependency is sensitive to additional load.
• Every physical failure matters for capacity.
• A breaker rejection is classified as non-retryable within the same logical operation.
• The window and minimum-call threshold are calibrated against real traffic.
• Separate telemetry exists for logical outcomes.

Decision criteria

Before choosing an order, answer:

1. Which event should increase the breaker's failure rate?
2. Should a success after retry hide the failed attempts?
3. How much additional load can the dependency tolerate?
4. Can a breaker rejection itself be retried?
5. Does the policy have a deadline and retry budget?
6. Do the metrics distinguish a logical operation from a physical attempt?
7. Does degradation primarily appear as errors, latency, or saturation?

The correct answer is not "Retry first" or "Circuit Breaker first." It is a composition whose semantics can be explained, measured, and tested.

How calls multiply across layers

Assume three layers:

Client -> Service A -> Service B -> Database

Each layer calls the next once and applies the same policy.

Case 1: three additional retries

"Three retries" means:

1 initial attempt + 3 retries = 4 total attempts

The theoretical maximum against the final dependency is:

4 × 4 × 4 = 64 calls

Case 2: three total attempts

If the setting means three attempts including the initial one:

3 × 3 × 3 = 27 calls

The difference is not merely linguistic. It changes the amplification factor from 27 to 64.

Google SRE uses the same example—three layers with four attempts per layer—to show how one action can generate 64 attempts against an overloaded database.

General formula

For n layers:

Maximum calls to the final dependency = ∏ total_attempts_per_layer

If the tool configures additional retries:

total_attempts = 1 + additional_retries

Operational decision

The architecture should declare:

• Total attempts, not only the number of retries.
• Which layer owns the retry policy.
• Which clients or proxies already retry automatically.
• Whether the SDK enables retries by default.
• Whether the service mesh or gateway adds another policy.
• How an upstream retry is prevented from repeating an operation that already exhausted its downstream budget.

During an architecture review, "we have three retries" is an incomplete answer. The correct question is: how many physical attempts can one end-to-end operation generate in the worst case?

How to calculate the time cost

For one layer with N attempts:

Approximate maximum time =
Σ per-attempt timeouts
+ Σ backoff delays
+ overhead

Overhead includes connection acquisition, scheduling, serialization, queueing, load balancing, and local work not covered by the timeout.

Example

Assumptions:

• Per-attempt timeout: 1,000 ms.
• Total attempts: 4.
• Backoffs: 100 ms, 200 ms, and 400 ms.
• Overhead omitted for simplicity.

Attempt time = 4 × 1,000 ms = 4,000 ms
Accumulated backoff = 100 + 200 + 400 = 700 ms
Potential time = 4,700 ms

If the user's deadline is 2,000 ms, the policy is internally contradictory. The library may allow those values, but the operation does not have enough time to execute them.

Budgeting against a deadline

Assumptions:

• End-to-end deadline: 2,000 ms.
• Time consumed before the dependency: 300 ms.
• Return and final-processing margin: 200 ms.
• Time available to the dependency: 1,500 ms.

A conceptual policy could allocate:

Attempt 1: maximum 500 ms
Backoff with jitter: up to 100 ms
Attempt 2: maximum 500 ms
Unallocated margin: 400 ms

The margin absorbs scheduling, network, and processing variation. It is not wasted capacity. It prevents a design that works only when every component consumes exactly its theoretical maximum.

Conceptual pseudocode

The following fragment expresses decisions. It is not production-ready code:

deadline = operation_deadline
attempt = 0

while attempt < max_total_attempts:
    remaining = deadline - now()

    if remaining <= minimum_completion_margin:
        return DEADLINE_EXHAUSTED

    if circuit_breaker_rejects():
        return DEPENDENCY_UNAVAILABLE

    per_attempt_timeout = min(
        configured_attempt_timeout,
        remaining - minimum_completion_margin
    )

    result = call_dependency(timeout=per_attempt_timeout)
    attempt += 1

    if result.success:
        return result

    if not is_retryable(result):
        return result

    if not operation_is_retry_safe:
        return INDETERMINATE_RESULT

    delay = jittered_backoff(attempt)

    if delay + minimum_attempt_time >= deadline - now():
        return result

    sleep(delay)

return LAST_RESULT

A real implementation must account for cancellation, context propagation, metrics, concurrency, idempotency, framework-specific status codes, and breaker behavior.

Dangerous configuration versus defensible configuration

Dangerous configuration

Per-attempt timeout: 5 s
Retries: 3 additional
Backoff: none
Jitter: none
Retries in client, gateway, and service
Non-idempotent operation
Circuit Breaker: final errors only
Slow-call threshold: none
End-to-end deadline: none

This policy may produce:

• Four consecutive five-second attempts per layer.
• Synchronized retries.
• Multiplication across client, gateway, and service.
• Duplicate effects.
• A breaker that remains closed while retries partially recover requests.
• Work that continues after the user has abandoned the operation.
• Queues and pools occupied by work with no remaining value.

Defensible configuration

Conceptual example:

End-to-end deadline: 2 s
Per-attempt timeout: up to 400 ms, limited by remaining time
Total attempts: 2
Backoff: exponential with jitter
Retry: classified transient failures only
Retry-After: honored when applicable
Idempotency: guaranteed
Retry owner: one defined layer
Circuit Breaker: failure rate + slow-call rate
Minimum sample size: calibrated against volume
Retry budget: traffic and concurrency limits
Telemetry: operations, attempts, and total elapsed time

These values are not a recipe. They only demonstrate internal coherence. The final configuration must be derived from:

• The operation's SLO and deadline.
• Historical latency and percentiles.
• Available dependency capacity.
• Failure semantics.
• The cost of duplicating the operation.
• Expected duration of the transient condition.
• SDK, proxy, gateway, and service-mesh behavior.
• The functional degradation strategy.

Retry budget

Retries should consume a controlled fraction of capacity, not become a second invisible workload.

A retry budget can impose complementary limits.

1. Per-request budget

Maximum total attempts: 2

This prevents unbounded loops and makes the worst case calculable.

2. Time budget

Do not start another attempt if it cannot fit inside the remaining deadline.

This prevents late results and useless work.

3. Traffic budget

Illustrative example:

Original requests per minute: 10,000
Retry budget: 8%
Allowed retries: 800 per minute

The percentage is not universal. It must remain below the dependency's actual headroom and account for the possibility that original traffic may increase during the incident.

4. Concurrency budget

Limit how many retries may be active simultaneously. Envoy, for example, can express a retry budget as a percentage of active and pending requests, with an additional minimum retry concurrency.

5. Per-client or per-tenant budget

Prevent one faulty consumer from exhausting shared retry capacity.

Derived metrics

retry_amplification_factor =
    physical_attempts / original_operations

retry_recovery_rate =
    operations_succeeded_after_retry / operations_with_retry

retry_budget_consumption =
    retries_executed / retries_allowed

Interpretation:

• A rising amplification factor means more cost per logical operation.
• A falling recovery rate means retries are becoming less useful.
• An exhausted budget means the policy is protecting capacity, but it also indicates that the dependency or error classification requires investigation.

When not to retry

Do not retry when:

1. The failure is permanent for the same request.
2. The request is invalid.
3. The operation is non-idempotent and no deduplication mechanism exists.
4. Insufficient time remains in the deadline.
5. The breaker is open and its rejection is not a retryable condition within the same operation.
6. The dependency is saturated and communicates no reasonable recovery window.
7. The attempt is expensive and increases potential damage.
8. The result is uncertain and repetition may duplicate effects.
9. The contract requires human intervention or compensation.
10. The system has already exhausted the retry budget.

Automatic retry should also be avoided when the correct response is to degrade functionality, enqueue the work, return a pending state, or ask the consumer to check again later.

Minimum instrumentation

The policy must answer two different questions:

1. What did the user observe?
2. How much physical work did the system perform?

Metrics

Do not mix "requests" and "attempts" in the same series without a clear label. If a dashboard presents an error rate without identifying its denominator, the team may interpret physical failures as user failures, or the reverse.

Structured logs

Recommended fields:

operation_id
idempotency_key_hash
attempt_number
max_total_attempts
retry_reason
retryable
backoff_ms
remaining_deadline_ms
attempt_timeout_ms
breaker_state
breaker_decision
dependency
outcome
elapsed_total_ms

Avoid logging complete idempotency keys if they may be sensitive. Use a hash or safe identifier and apply cardinality controls.

Distributed traces

For HTTP, OpenTelemetry defines http.request.resend_count to indicate how many times a request has been resent. Depending on the instrumentation model, every physical send may have its own span while retaining a clear relationship with the logical operation.

A useful structure is:

Span: logical operation
  ├── Span: attempt 1
  ├── Event: retry scheduled, reason=timeout, backoff_ms=87
  └── Span: attempt 2, resend_count=1

Additional attributes or events may include:

• Attempt number.
• Remaining deadline.
• Retry reason.
• Breaker state.
• Breaker rejection.
• Applied timeout.
• Physical outcome and logical outcome.

The dedicated article on observability with metrics, logs, and traces explains how to correlate these signals. For step-by-step diagnosis during a degradation, see the structured incident investigation.

How to validate the policy

A resilience policy must be validated with controlled failures, not only with successful traffic.

Minimum test matrix

Success indicators

The solution worked if:

• End-to-end duration does not systematically exceed the deadline.
• Cancelled operations stop consuming downstream work.
• The amplification factor remains within budget.
• The degraded dependency receives less traffic.
• Retries recover a measurable proportion of transient failures.
• No duplicate effects appear.
• The breaker opens and recovers using a sufficient sample.
• Breaker rejections do not create a new retry loop.
• Dashboards distinguish original requests, attempts, and final failures.
• Functional degradation is predictable.

Signals that the policy made the system worse

• Apparent availability rises, but end-to-end P99 also rises.
• The backend receives more QPS during an outage.
• Most retries fail.
• The breaker repeatedly opens and closes.
• The retry budget is exhausted during normal traffic.
• Clients report timeouts while the server continues processing.
• Metrics show final success, but cost per operation grows.
• Duplicate effects or compensating actions become frequent.

End-to-end latency must be evaluated together with success and availability. An operation that completes successfully after ten seconds may count as a technical success and still be a failure for the consumer.

Operational checklist

Semantics

• Is the operation idempotent?
• Is an idempotency key or deduplication mechanism available where required?
• Which failures are truly transient?
• Which responses are final?
• Is a breaker rejection retryable within the same operation?
• Is response loss after commit covered?

Time

• Is there an end-to-end deadline?
• Is remaining time propagated?
• Does the per-attempt timeout cover the correct phases?
• How many attempts actually fit?
• Is margin reserved for returning the response?
• Can backoff consume the deadline?

Capacity

• Which layer owns retries?
• Does the SDK already retry automatically?
• Does the gateway, proxy, or service mesh add retries?
• Is there a per-request budget?
• Is there a traffic or concurrency budget?
• Does the dependency have headroom for the worst case?

Circuit Breaker

• Does it observe attempts or logical outcomes?
• Does it record failures and slow calls?
• Is there a minimum sample size?
• Are ignored exceptions explicitly defined?
• Does Half-Open limit probe calls?
• Is a bulkhead also required?

Observability

• Are original operations distinguished from physical attempts?
• Is the amplification factor measured?
• Is success after retry measured?
• Is remaining deadline recorded?
• Do traces show every attempt?
• Are breaker transitions and rejections observable?
• Is there an alert for retry-budget consumption?

Validation

• Have errors, latency, resets, and overload been tested?
• Has response loss after the effect been tested?
• Has downstream cancellation been verified?
• Has partial recovery been tested?
• Has original load been compared with physical load?
• Is there a functional degradation strategy?

Frequently asked questions

Which comes first: Retry or Circuit Breaker?

It depends on which event the breaker must observe.

When the Circuit Breaker wraps Retry, it usually records the logical result after all attempts. When Retry wraps the Circuit Breaker, every attempt passes through and updates the breaker. The decision must consider dependency capacity, threshold semantics, the retry budget, and available telemetry.

How many retries should I configure?

There is no universal number. First define the deadline, per-attempt timeout, backoff, operation cost, and dependency headroom. Express the setting as total attempts, because "three retries" may mean four attempts.

For an interactive operation with a short deadline, one or two total attempts may consume the entire budget. An asynchronous process may retry over a longer interval, but it requires persistence, deduplication, and global limits.

What is the difference between a timeout and a deadline?

A timeout is a maximum duration applied to one call or phase. A deadline is the time limit for the complete operation. When a request crosses several services, every layer should respect the remaining deadline instead of starting a fresh full timeout.

What is jitter?

Jitter is random variation applied to backoff. It prevents many clients that failed at the same time from retrying at the same instant. Backoff separates attempts; jitter prevents them from remaining synchronized.

When is an operation idempotent?

An operation is idempotent when repeating the same intent produces the same observable effect as executing it once. Idempotency may come from the contract, an idempotency key, a unique constraint, or a deduplication strategy. The HTTP method alone does not guarantee a correct implementation.

Can a retry make an outage worse?

Yes. If the root cause is overload, every retry adds work to a component that already lacks capacity. Without limits, backoff, jitter, and a global budget, retries can turn a partial degradation into a cascading failure.

Conclusion

Timeout, Retry, and Circuit Breaker should be designed as one policy.

The timeout limits a wait. The retry consumes another opportunity. The Circuit Breaker decides whether that opportunity should reach the dependency. All three compete for the same deadline and the same capacity.

Before enabling retries, calculate how many physical attempts one complete operation can generate. Before configuring a breaker, define whether it observes attempts or logical outcomes. Before repeating an operation, guarantee idempotency. After deployment, measure original requests, attempts, end-to-end latency, recovery, and amplification.

The final test is not whether the system retries. It is whether, during degradation, the system performs less useless work, preserves capacity, and fails within known limits.

Technical sources

• Google SRE — Addressing Cascading Failures: sre.google/sre-book/addressing-cascading-failures
• Google SRE — Handling Overload: sre.google/sre-book/handling-overload
• AWS — Timeouts, retries, and backoff with jitter: builder.aws.com/content/timeouts-retries-and-backoff-with-jitter
• AWS — Making retries safe with idempotent APIs: aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs
• gRPC — Deadlines: grpc.io/docs/guides/deadlines
• IETF — RFC 9110: HTTP Semantics: datatracker.ietf.org/doc/html/rfc9110
• IETF — RFC 6585: Additional HTTP Status Codes: datatracker.ietf.org/doc/html/rfc6585
• Resilience4j — CircuitBreaker: resilience4j.readme.io/docs/circuitbreaker
• OpenTelemetry — Semantic conventions for HTTP spans: opentelemetry.io/docs/specs/semconv/http/http-spans
• Envoy — Circuit breakers (RetryBudget): envoyproxy.io/docs/envoy/latest/api-v3/config/cluster/v3/circuit_breaker.proto

Jorel del Portal

Systems engineer specialized in enterprise software architecture and high-availability platforms.

LinkedIn YouTube Tiktok Instagram X Threads My website