Latency in Distributed Systems: P50, P95, P99, and Latency Budgets

The main API spends 180 ms in application logic, the database call takes 220 ms, and an external dependency consumes another 250 ms. No component looks catastrophic in isolation. The user still waits 800 ms.

That gap is the operational problem: latency does not belong to one service. It is the accumulated result of every connection, queue, dependency, retry, lock, serialization step, and architectural decision on the request's critical path.

A useful latency analysis must therefore answer more than «which service is slow?». It must identify where elapsed time is spent, which part is variable, which waits are avoidable, which work can leave the synchronous path, and whether the proposed fix improves the user-visible percentile without moving the bottleneck elsewhere.

This article explains how to read P50, P95, P99, and tail latency; design an end-to-end latency budget; detect fan-out amplification; distinguish processing time from waiting time; and investigate a slow API using metrics, traces, logs, and controlled validation.

Latency is elapsed time, not speed

Latency is the elapsed time between the start of an operation and the point at which its result becomes available to the observer that matters.

That observer must be explicit. A backend may report 120 ms of server processing while the browser observes 430 ms from request initiation to the final byte. Both measurements can be correct because they cover different boundaries.

In a distributed request, elapsed time can be separated into several categories:

Calling latency «speed» hides the decision that matters. A system is not slow for one universal reason. It may be computing slowly, waiting for capacity, establishing connections repeatedly, executing unnecessary synchronous work, or amplifying a small tail through fan-out.

The diagnostic objective is to separate those mechanisms before changing configuration.

Latency and throughput answer different questions

Latency and throughput are related, but they are not interchangeable.

A system can exhibit any of the following combinations:

• Low latency and low throughput: a lightly used service completes each request quickly but processes little total work.
• Low latency and high throughput: the desired state, provided error rate and resource headroom remain acceptable.
• High latency and high throughput: batching or deep queues may keep completions high while individual requests wait too long.
• High latency and falling throughput: saturation, contention, retries, or failure recovery may have crossed a nonlinear degradation point.

The most dangerous interpretation error is to treat stable throughput as proof that latency is healthy. A worker pool may continue completing 1,000 requests per second while its queue grows by 100 requests per second. Throughput appears stable; response time is already deteriorating.

The inverse mistake also occurs. Reducing concurrency can lower queueing latency while lowering maximum throughput. Whether that is an improvement depends on the service-level objective, workload shape, and capacity requirement.

Why averages hide production impact

Latency distributions are usually asymmetric. Most requests cluster around a central range, while a smaller set stretches into a long tail because of cache misses, lock contention, garbage collection, retries, cold connections, noisy neighbors, slow storage, or overloaded dependencies.

Consider 100 requests:

99 requests:   100 ms each
1 request:  10,000 ms

The arithmetic mean is:

((99 × 100) + 10,000) / 100 = 199 ms

An average of 199 ms sounds acceptable if the target is 250 ms. It also conceals a request that took ten seconds.

The average is not false. It answers a weak operational question: «What was total elapsed time divided by request count?» It does not show how many users experienced the slow tail, how extreme that tail was, or whether the affected requests shared a region, version, operation, payload shape, or dependency.

Percentiles provide a more useful view of the distribution, but they still require a defined population and time window. A P99 calculated across every endpoint, every status code, and an entire day can hide a severe regression in one high-value operation.

For a deeper treatment of quantile calculation, histogram accuracy, sample windows, and alert design, use the dedicated article on how to read latency percentiles (P50, P95, and P99).

How to interpret P50, P95, P99, and P99.9

A percentile is a threshold within an observed distribution.

• P50: 50% of observations completed at or below this value. It represents the median, not the average.
• P95: 95% completed at or below this value. It exposes a meaningful part of the slow tail without focusing only on the most extreme cases.
• P99: 99% completed at or below this value. At scale, the remaining 1% may still represent a large number of requests.
• P99.9: 99.9% completed at or below this value. It becomes operationally relevant when request volume makes one request in a thousand frequent rather than exceptional.

A P99 of 1.8 seconds means that, for the defined population and window, 99% of observations were 1.8 seconds or faster. It does not mean that every request in the slowest 1% took exactly 1.8 seconds.

Three details must accompany any percentile:

1. Population: endpoint, operation, customer class, status, region, version, payload class.
2. Window: one minute, five minutes, one hour, deployment interval, peak period.
3. Estimator: exact samples, client-side summary, classic histogram, native histogram, or another approximation.

Histogram bucket boundaries affect quantile accuracy. A P95 estimated from coarse buckets may be suitable for an SLO threshold and still be too imprecise for comparing a 20 ms optimization. Prometheus documents the trade-offs between histograms and summaries and the estimation error introduced by bucket design.

Do not compare percentiles unless their scope and calculation are compatible. Comparing a client-side P95 over mobile users with a server-side P95 over successful requests can be useful, but the difference must be interpreted as a boundary difference, not a contradiction.

Why tail latency becomes system latency

A single slow request is a local event. A distributed operation that waits for many subrequests turns local tails into aggregate behavior.

Suppose a service calls 20 dependencies in parallel and cannot respond until all 20 complete. If each dependency independently has a 1% probability of being «slow,» the probability that at least one is slow is:

P(at least one slow dependency)
= 1 - P(all dependencies are fast)
= 1 - 0.99^20
≈ 18.2%

The 1% local tail has become an 18.2% chance that the aggregate operation encounters at least one slow branch.

The independence assumption is only a model. Production dependencies often share networks, runtimes, clusters, storage systems, deployment events, or traffic bursts. Correlated slowdown can make the aggregate behavior worse than an independence model predicts. In other cases, shared caching or co-location can move branches together and change the distribution differently.

The important architectural fact remains: when completion depends on the slowest branch, the maximum branch duration dominates the critical path. The Google paper The Tail at Scale describes this effect and the need to build tail-tolerant services rather than assuming predictable components will emerge automatically from variable ones.

End-to-end latency is a critical-path problem

Measuring only the main service produces an incomplete latency model. The user-visible path can include:

• client scheduling and local network delay;
• DNS resolution;
• TCP, QUIC, or another transport setup;
• TLS negotiation;
• CDN, load balancer, reverse proxy, WAF, or API gateway processing;
• authentication and authorization;
• application queueing and execution;
• database acquisition, query, and result transfer;
• downstream APIs, messaging systems, or storage;
• serialization, compression, and response transfer;
• client rendering or post-processing.

A useful first question is therefore:

That question prevents a team from optimizing application code when the dominant cost is connection establishment, or increasing database capacity when the dominant span is pool acquisition.

How to design a latency budget

A latency budget converts a vague performance target into an architectural constraint. It defines how much elapsed time the complete operation may consume and how that time is allocated across the critical path.

Start with an end-to-end objective

Example:

Checkout API objective: P95 ≤ 800 ms
Measurement boundary: client request start to complete response body
Population: successful checkout requests in the primary region
Window: rolling 5 minutes for alerting; 28 days for SLO reporting

The boundary and population are part of the objective. «P95 under 800 ms» is incomplete without them.

Google SRE treats latency as one of the four golden signals and explicitly recommends separating successful-request latency from failed-request latency, because failures can be fast and distort the combined distribution.

Allocate the budget by critical-path component

A first draft might allocate the full 800 ms:

The arithmetic is correct and the design is fragile. It assumes every component simultaneously remains at or below its allocation. There is no room for network jitter, runtime pauses, cache misses, deployment effects, scheduler delay, or measurement error.

An operational budget should reserve headroom:

The reserve is not unowned time. It is protection against variance and future change. If teams consume it continuously, it is no longer reserve.

Allocate using compatible percentiles

A P95 end-to-end objective cannot be guaranteed by simply summing every component's P95. Percentiles are not generally additive, especially when operations overlap or are correlated.

Use component percentiles as constraints and diagnostic indicators, then validate the aggregate objective with end-to-end measurements under representative traffic. When a formal model is needed, use the joint distribution or simulation rather than pretending that P95(A) + P95(B) equals P95(A + B).

Make deadlines propagate through the call graph

A downstream timeout must fit inside the time remaining for the caller. If an upstream request has 300 ms left before its deadline, giving a dependency a 500 ms timeout cannot protect the end-to-end objective.

A practical sequence is:

remaining budget
- local cleanup and response serialization
- retry allowance, if justified
- safety margin
= maximum downstream attempt duration

Timeouts should be derived from the operation's deadline, dependency distribution, and failure semantics — not copied from a framework default. The dedicated article on correctly configured timeouts should own the full timeout and retry policy.

Case study: decomposing an 800 ms request

Assume a trace shows the following serial critical path:

40 + 60 + 180 + 220 + 250 + 50 = 800 ms

The largest component is the external dependency at 250 ms, but «optimize the largest number» is not yet a decision. Ask:

1. Which component dominates the critical path? Here, the external call is largest, but the database is close.
2. Which component has the greatest variance? A stable 250 ms call may be less responsible for P99 than a database call that ranges from 40 ms to 1.5 seconds.
3. Which operations are actually serial? If database and external calls can run concurrently without violating semantics, the branch maximum could replace their sum.
4. Which work can leave the synchronous path? Auditing, enrichment, notifications, or secondary indexing may not need to block the user response.
5. Which call can be avoided? Cached, precomputed, denormalized, or request-carried data may remove a dependency, but freshness and correctness requirements constrain that choice.
6. Where should a timeout apply? It must preserve enough time for fallback, compensation, or a controlled error before the upstream deadline.
7. What is the validation metric? The change must improve end-to-end P95 or P99 for the target operation without unacceptable error, staleness, resource, or cost regressions.

A local optimization that removes 40 ms from service CPU but leaves a 900 ms P99 dependency tail unchanged may improve benchmarks without changing user experience.

Fan-out and latency amplification

Fan-out occurs when one operation issues multiple downstream requests. Parallel execution reduces the sum of branch durations, but the join still waits for the slowest required branch.

The aggregate P99 can be materially worse than each dependency's individual P99 because every user operation creates more opportunities to encounter a slow branch. Retries can add new branches while the original work is still running, increasing load and correlation.

Possible responses depend on semantics:

Hedging is not a generic retry. It intentionally issues a second attempt before the first has failed, usually after a delay derived from the normal latency distribution. Without capacity controls and idempotent semantics, it can make overload worse.

DNS, TCP, TLS, and connection setup

Latency can accumulate before application processing begins.

A cold request may require:

1. DNS resolution;
2. transport connection establishment;
3. TLS negotiation;
4. proxy or gateway negotiation;
5. application protocol setup;
6. only then, request transmission and server processing.

TCP connection establishment uses a three-way handshake as specified by RFC 9293. TLS adds cryptographic negotiation and authentication; the current TLS 1.3 specification is RFC 9846. The actual round trips depend on protocol version, resumption, transport, packet loss, middleboxes, and implementation.

The operational distinction is between cold and reused connections:

Do not infer the phase from one aggregate timer. Capture client-side phase timings where available, and correlate them with server traces. A fast server span cannot explain time spent before the server receives the request.

Connection reuse usually reduces setup cost, but unlimited lifetime is not automatically correct. DNS changes, load-balancer rotation, stale connections, certificate changes, and uneven connection distribution can require bounded lifetime or health validation.

Connection pools can remove or create waits

A connection pool amortizes setup cost and bounds concurrent use of a downstream resource. It can also become a queue whose wait time is larger than the operation itself.

Consider this trace:

Database pool acquisition: 380 ms
Database query execution:   42 ms
Result decoding:             8 ms
Total database span:        430 ms

Optimizing the SQL query cannot recover the 380 ms spent waiting for a connection.

The relevant pool signals are:

• active connections;
• idle connections;
• maximum configured connections;
• pending borrowers or waiters;
• acquisition duration distribution;
• acquisition timeout count;
• connection creation rate and failures;
• connection age and eviction reason;
• downstream server connection utilization;
• transaction or lease duration.

Pool sizing is a concurrency-control decision, not a rule of thumb.

A valid change improves end-to-end latency and pool wait without causing downstream saturation. Validate at representative concurrency, not only with single-user tests.

Cold starts are several different problems

«Cold start» is often used as a label for any first-request slowdown. The underlying mechanisms differ:

• process or container creation;
• runtime initialization;
• dependency injection or module loading;
• JIT compilation;
• lazy data loading;
• empty connection pools;
• cold filesystem or page cache;
• cold application cache;
• first database query compiling a plan or loading pages;
• autoscaling that adds capacity after demand arrives.

These mechanisms require different fixes. Provisioned instances may address process creation but not a cold cache. Preopening connections may reduce setup time but create a connection storm during rollout. Warming every code path may lengthen deployment and waste resources.

To diagnose a suspected cold start, segment latency by:

• instance age;
• request ordinal since instance start;
• deployment or scale event;
• runtime initialization duration;
• connection creation count;
• cache hit ratio;
• JIT or compilation activity;
• first-query versus steady-state execution.

A useful validation compares the first N requests on new instances with steady-state requests under the same input and load. If only the first request is slow, the trace should show which initialization segment consumed the time.

Queues, saturation, and backpressure

Latency often rises before errors become visible. When arrivals exceed sustainable processing capacity, work waits in a queue. The service may still be completing requests and reporting moderate average CPU while specific executors, partitions, connections, or downstream resources are saturated.

Google SRE identifies overload as a common cause of cascading failure and recommends designing explicit overload behavior rather than allowing queues and retries to grow without control.

Little's Law provides a useful steady-state relationship:

L = λ × W

Where:

• L is the average number of items in the system;
• λ is the average arrival or completion rate;
• W is average time in the system.

If a stable service processes 500 requests per second and average time in the system is 200 ms:

L = 500 × 0.2 = 100 requests in flight on average

If the observed in-flight population rises toward 600 while the arrival rate remains near 500 requests per second, either average time has moved toward 1.2 seconds or the system is not in steady state. The equation is a diagnostic relationship, not proof of where the delay occurs.

Backpressure is the mechanism by which a system communicates or enforces that it cannot accept unlimited work at the current rate. Depending on the protocol, it may take the form of bounded queues, flow control, concurrency limits, admission control, rate limits, load shedding, or explicit overload responses.

The decision is not «queue or no queue». A bounded queue can absorb short bursts. An unbounded queue converts overload into memory growth and stale work. A queue that is too small may reject harmless bursts. The correct bound depends on acceptable waiting time, service rate, burst characteristics, and failure semantics.

Retries can turn latency into overload

A retry consumes time and creates more work. It is justified when the failure is transient, the operation is safe to repeat, the remaining deadline is sufficient, and the additional load does not undermine recovery.

Layered retries are especially dangerous. Suppose three layers each allow three total attempts: the initial attempt plus two retries. In the worst case, one original operation can produce:

3 × 3 × 3 = 27 downstream attempts

If each layer instead allows three additional retries, there are four total attempts per layer:

4 × 4 × 4 = 64 downstream attempts

This distinction between total attempts and additional retries must be explicit in configuration and documentation.

AWS describes retries as «selfish» because they consume server resources to improve one caller's chance of success, and recommends timeouts, capped exponential backoff, and jitter rather than immediate synchronized retries. Google SRE similarly warns that retries can amplify overload and contribute to cascading failure.

A defensible retry policy specifies:

• which outcomes are retryable;
• whether the operation is idempotent or protected by an idempotency key;
• total attempts, not ambiguous «retry count» wording;
• per-attempt timeout;
• overall deadline;
• backoff function and maximum delay;
• jitter strategy;
• retry budget or concurrency limit;
• observability for original attempts and retries separately;
• cancellation when the upstream request has ended.

The full treatment belongs in the article on retry storms. This article's narrower point is that retries consume the same latency and capacity budget being investigated. When degradation becomes persistent, a circuit breaker can cut off attempts before they amplify the outage.

Synchronous and asynchronous paths

Moving work to an asynchronous path does not eliminate latency. It changes which latency the user waits for and where completion, retry, ordering, and failure handling occur.

The key question is:

An asynchronous design is appropriate when the product contract tolerates delayed completion. It is not appropriate when the user must know whether inventory was reserved, access was granted, or a transaction committed before proceeding.

Measure both:

• acknowledgement latency: time until the producer receives acceptance;
• completion latency: time until the requested state is actually reached.

Reporting only the first can make a system look fast while the backlog grows for minutes.

Minimum instrumentation for latency analysis

Latency diagnosis requires compatible evidence across metrics, traces, and logs.

Operation-level metrics

At minimum, capture:

• request count;
• success and error count;
• duration histogram;
• P50, P95, and P99 derived from an appropriate distribution instrument;
• queue or scheduler wait;
• connection acquisition wait;
• dependency duration by low-cardinality operation and destination class;
• timeout count;
• retry attempt count;
• in-flight requests or concurrency;
• saturation signals for the constrained resource.

OpenTelemetry defines http.server.request.duration and http.client.request.duration as HTTP duration histograms in its semantic conventions. Use the conventions supported by the deployed instrumentation version and avoid silently mixing legacy and stable names in dashboards.

Separate successful and failed latency. A fast rejected request and a slow successful request represent different user outcomes. Combining them can produce a lower latency graph during an outage.

Distributed traces

A trace should make the critical path visible through:

• server and client spans;
• parent-child relationships;
• start time and duration;
• dependency name or class;
• status and error type;
• retries and redirects;
• queue or acquisition events where instrumentation supports them;
• selected low-cardinality attributes such as operation, region, version, and route.

OpenTelemetry's HTTP and database semantic conventions define common span structures for these operations, and W3C Trace Context standardizes traceparent and tracestate propagation so a request can remain correlated across components and vendors. To expand trace architecture, context propagation, and sampling, see observability with metrics, logs, and traces.

Trace sampling must match the diagnostic objective. Low-rate random sampling can miss rare P99.9 events. Tail-based sampling can retain slow or failed traces, but it adds collector state, delay, cost, and failure modes. The selection is an observability architecture decision, not a universal default.

Structured logs

Useful latency log fields include:

trace_id
span_id
operation
route
region
version
dependency
duration_ms
queue_wait_ms
connection_wait_ms
attempt_number
outcome
error_type

Do not place unbounded identifiers, full queries, secrets, or sensitive payloads into metrics labels or unrestricted trace attributes. High cardinality can make the telemetry system expensive or unusable precisely during an incident.

Correlation requirements

Metrics identify the distribution and affected segment. Traces explain the path. Logs provide detailed state and error context. The instruments should share stable dimensions so an engineer can move from:

P99 regression on operation X in region Y and version Z

to:

slow traces for operation X, region Y, version Z

to:

logs for the dominant span and its dependency outcome

For a complete incident workflow, see how to investigate an incident with metrics, logs, and traces.

A production method for diagnosing a slow API

The method below is intentionally ordered. Skipping directly to code optimization often wastes time because the dominant delay may be queueing, connection setup, or a dependency.

Step 1: Confirm user impact and measurement boundaries

Establish:

• the affected operation;
• client-side and server-side P50, P95, and P99;
• success and error latency separately;
• traffic volume and request mix;
• SLO or explicit user-facing objective;
• start time and relationship to deployment, scale, or dependency events.

A P99 increase from 700 ms to 1.4 seconds under ten requests per minute has different significance and estimator stability than the same increase under 20,000 requests per second.

Step 2: Segment before averaging

Break the distribution down by low-cardinality dimensions:

• route or operation;
• region or zone;
• application version;
• client class;
• status or outcome;
• dependency;
• payload or workload class;
• cache hit or miss, when safely modeled.

The objective is to find a population where the slow behavior is concentrated. Do not create unbounded labels to do this.

Step 3: Separate processing from waiting

Inspect:

• CPU and run-queue pressure;
• thread or event-loop delay;
• executor queue time;
• connection pool wait;
• database lock and I/O waits;
• downstream request duration;
• garbage collection or runtime pauses;
• filesystem, network, or storage waits.

Low CPU does not prove spare capacity. A service waiting on a saturated connection pool can have low CPU and high latency. A single hot partition can be overloaded while fleet-average CPU remains moderate.

Step 4: Follow the distributed trace and identify the critical path

Find:

• the longest required branch;
• serial dependency chains;
• overlapping spans;
• repeated attempts;
• missing instrumentation gaps;
• time before the first server span;
• time after application completion;
• large parent duration with little child or self-time, which may indicate an uninstrumented wait.

Do not sum every span. Determine which spans lie on the wall-clock critical path.

Step 5: Compare healthy and slow traces

Use the same operation and similar inputs. Compare:

• branch count;
• cache state;
• pool acquisition;
• query shape;
• payload size;
• connection reuse;
• retry count;
• region and instance age;
• version and feature flags.

A single slow trace suggests hypotheses. A repeated difference across a representative sample is stronger evidence.

Step 6: Form a falsifiable hypothesis

Weak hypothesis:

The database is slow.

Falsifiable hypothesis:

P99 increased because requests on version 4.2 wait for a database connection.
Prediction: slow traces will show acquisition wait above 300 ms while query execution remains below 60 ms; the issue will correlate with pool exhaustion after the new concurrency setting.

The prediction defines what evidence would support or reject the claim.

Step 7: Mitigate at the correct layer

Possible mitigations include:

• reduce or bound concurrency;
• correct a pool leak;
• shorten transaction duration;
• optimize the dominant query;
• remove a serial dependency;
• reuse connections;
• precompute or cache within freshness constraints;
• move nonessential work off the synchronous path;
• apply a downstream deadline;
• disable unjustified retries;
• shed load or return a controlled degraded result;
• roll back the change that introduced the regression.

The lowest-risk mitigation during an incident may differ from the permanent architectural fix.

Step 8: Verify with the original user-facing metric

A mitigation is not validated because a local span became shorter. Confirm:

• end-to-end P95 and P99 improved for the affected population;
• error rate did not increase beyond the accepted trade-off;
• throughput remains sufficient;
• queue depth and retry volume stabilized;
• the bottleneck did not move to another dependency;
• the change holds under representative peak load;
• resource and cost impact remain acceptable.

Worked diagnosis: low CPU, high latency

Assume the API shows:

P50: 210 ms
P95: 920 ms
P99: 1,480 ms
Error rate: 0.2%
Average CPU: 42%

The initial claim is «CPU is low, so capacity is not the problem». Traces show otherwise:

Database pool acquisition P95: 510 ms
Database query execution P95:   58 ms
Pool pending requests peak:     140
Database CPU:                   37%
Database lock waits:            low

The dominant wait is before query execution. Increasing the pool might reduce acquisition wait, but only if the database and total connection budget can support more concurrent queries. The next checks are:

1. total pool capacity across all application instances;
2. connection lease duration and possible leaks;
3. transaction boundaries;
4. request concurrency change after the latest deployment;
5. database maximum connections and safe working range;
6. query throughput and I/O under a controlled pool increase.

Suppose the deployment doubled application concurrency but left the pool unchanged. A controlled increase reduces pool wait to 70 ms, database query P95 rises modestly from 58 ms to 65 ms, and end-to-end P95 falls from 920 ms to 470 ms without increasing errors. That supports the hypothesis.

If database query P95 instead rises to 600 ms and lock waits surge, the pool increase merely moved the queue into the database. The correct response would be to bound concurrency, reduce transaction time, or increase downstream capacity — not continue increasing connections.

Common implementation errors

1. Reporting only the average

The average hides the shape and tail. Report a distribution, traffic volume, and the relevant percentile by operation and outcome.

2. Treating P99 as a universal property of a service

P99 depends on population, window, load, region, status, and estimator. Attach those dimensions to the number.

3. Adding span durations to compute end-to-end time

Parallel child spans overlap. Use wall-clock duration and identify the critical path.

4. Assuming low CPU means no saturation

Queues, pools, locks, event loops, storage, downstream limits, and hot partitions can dominate while fleet CPU is low.

5. Increasing every pool and thread limit

Larger limits can transfer waiting to a less controlled downstream resource and trigger contention or collapse.

6. Using cache as a generic patch

A cache can reduce latency and load, but it introduces freshness, invalidation, consistency, memory, and failure behavior. Define what data may be stale and how misses behave.

7. Retrying at every layer

Layered retries multiply attempts and consume the deadline. Assign retry ownership and an explicit total-attempt budget.

8. Setting timeouts longer than the caller's remaining deadline

The downstream operation continues after the user-visible request is already doomed. Propagate deadlines and cancel abandoned work when possible.

9. Calling every first-request delay a cold start

Separate process initialization, JIT, cache warming, pool creation, and first-query effects. Each has different evidence and mitigation.

10. Claiming asynchronous processing removes latency

It reduces acknowledgement latency only when the contract permits delayed completion. Completion latency and backlog still require measurement.

11. Alerting on a volatile percentile without volume context

Very low sample counts can make high percentiles unstable. Pair percentile alerts with traffic thresholds, longer windows, or SLO-based methods appropriate to the workload.

12. Optimizing a local component without validating the user path

A faster query is useful only if it changes the critical path or creates needed headroom. Verify the original end-to-end objective.

Operational checklist

Define

• Specify the user-visible measurement boundary.
• Define operation, region, outcome, and traffic population.
• Set an end-to-end percentile objective and reporting window.
• Reserve latency headroom rather than allocating 100% continuously.
• Define which work is mandatory before responding.

Instrument

• Record duration as a distribution, not only an average.
• Separate successful and failed-request latency.
• Capture queue, pool, dependency, timeout, retry, and concurrency signals.
• Propagate trace context across every supported boundary.
• Control metric and trace attribute cardinality.
• Make client and server measurement boundaries distinguishable.

Diagnose

• Segment by operation, region, version, outcome, and dependency.
• Separate processing time from waiting time.
• Identify the trace critical path rather than summing all spans.
• Compare healthy and slow traces with similar inputs.
• State a falsifiable hypothesis and expected evidence.

Change

• Confirm downstream capacity before increasing concurrency or pool size.
• Derive timeouts from remaining deadlines and dependency behavior.
• Assign retry ownership and total attempts.
• Define degraded or partial behavior before applying load shedding.
• Document cache freshness and invalidation constraints.
• Measure acknowledgement and completion latency for asynchronous flows.

Validate

• Recheck end-to-end P95 and P99 for the original population.
• Verify error rate, throughput, queue depth, retry load, and saturation.
• Test under representative peak traffic and failure conditions.
• Confirm the bottleneck did not move.
• Retain a regression dashboard and deployment comparison.

Frequently asked questions

What is a good latency for an API?

There is no universal number. The acceptable target depends on the user interaction, protocol, geography, dependency graph, payload, consistency requirement, and business consequence of waiting. Define a user-visible SLO, then allocate and validate a budget for that operation.

What is the difference between P95 and P99?

P95 is the threshold at or below which 95% of observations completed; P99 is the equivalent threshold for 99%. P99 focuses further into the tail. At high traffic, the slowest 1% can still affect thousands or millions of requests.

Why can latency rise while CPU remains low?

The service may be waiting on a queue, connection pool, lock, database, network, storage system, rate limit, or external dependency. Average CPU may also hide a saturated core, shard, partition, or instance subset.

What is tail latency?

Tail latency is the slower end of the latency distribution, commonly examined through P95, P99, P99.9, or maximum values. It matters because distributed fan-out increases the probability that an operation encounters at least one slow component.

How is a latency budget built?

Define an end-to-end percentile objective and measurement boundary, map the critical path, allocate component constraints with headroom, propagate deadlines, instrument each wait, and validate the aggregate distribution under realistic traffic. Do not assume component percentiles add arithmetically.

Do asynchronous systems eliminate latency?

No. They can reduce the time until acknowledgement by moving work after the response, but completion latency, queueing, retries, and failure recovery still exist and must be measured.

Is P99 always more useful than P95?

No. P99 is more sensitive to the tail but may be noisier at low volume and more expensive to optimize. Select the percentile that represents the user and business risk, then use additional percentiles to understand the distribution.

Conclusion

Latency is not a property that can be assigned to one service and optimized in isolation. It is the elapsed-time consequence of the whole request path: connection setup, queues, resource acquisition, processing, dependency calls, fan-out, retries, and response delivery.

The actionable method is:

1. define an end-to-end percentile objective and measurement boundary;
2. preserve headroom through a latency budget;
3. measure distributions rather than averages;
4. identify the wall-clock critical path;
5. separate computation from waiting;
6. change the layer that owns the dominant delay;
7. validate the same user-visible percentile under representative load.

A platform with available CPU and few errors can still be operationally slow. The evidence that resolves that tension is not another aggregate dashboard. It is a coherent path from the affected percentile to the exact wait, dependency, queue, connection, or retry behavior that consumed the budget.

Technical sources

• Google SRE — «Monitoring Distributed Systems»: sre.google/sre-book/monitoring-distributed-systems
• Jeffrey Dean and Luiz André Barroso — «The Tail at Scale»: research.google/pubs/the-tail-at-scale
• Google SRE — «Addressing Cascading Failures»: sre.google/sre-book/addressing-cascading-failures
• Google SRE — «Handling Overload»: sre.google/sre-book/handling-overload
• Marc Brooker, AWS Builders' Library — «Timeouts, retries, and backoff with jitter»: builder.aws.com — timeouts, retries, and backoff with jitter
• OpenTelemetry — «Semantic conventions for HTTP metrics»: opentelemetry.io/docs/specs/semconv/http/http-metrics
• OpenTelemetry — «Semantic conventions for HTTP spans»: opentelemetry.io/docs/specs/semconv/http/http-spans
• OpenTelemetry — «Semantic conventions for database client spans»: opentelemetry.io/docs/specs/semconv/db/database-spans
• W3C — «Trace Context»: w3.org/TR/trace-context
• Prometheus — «Histograms and summaries»: prometheus.io/docs/practices/histograms
• IETF — RFC 9293, «Transmission Control Protocol (TCP)»: rfc-editor.org/info/rfc9293
• IETF — RFC 9846, «The Transport Layer Security (TLS) Protocol Version 1.3»: rfc-editor.org/info/rfc9846

Jorel del Portal

Jorel del Portal is a systems engineer specialized in architecture, integration, resilience and observability of critical platforms. He designs systems, builds products and documents real engineering decisions.

About LinkedIn YouTube X TikTok Instagram