P50, P95, and P99: how to interpret latency percentiles without being misled by the average

100,000 requests
P50 = 120 ms
P95 = 480 ms
P99 = 1.8 s

Is the system fast because half of its requests complete in 120 milliseconds, or is it slow because approximately 1,000 requests take longer than 1.8 seconds?

Both statements can be true.

That is the problem with interpreting latency through a single number. The average can hide a severe tail. P50 can accurately describe the center of the experience while ignoring thousands of slow operations. P99 can expose material impact, but it can also be unstable, incorrectly aggregated, or calculated over a population that mixes incomparable operations.

The central idea of this article is simple:

By the end, you will be able to read P50, P95, and P99 without confusing them with availability, convert a tail percentile into approximate impact, decide which percentile to observe, and identify measurements that look precise but are not comparable.

Latency is a distribution, not a number

Every completed request produces one observation:

84 ms
91 ms
105 ms
118 ms
...
2,400 ms

The set of observations for one operation, population, and time window forms a distribution. That distribution may be tightly concentrated around a narrow range, or it may contain a long tail of requests that are much slower than the majority.

In distributed systems, latency distributions are commonly asymmetric. Most operations may complete near a central value while a smaller fraction accumulates delay in queues, contention, runtime pauses, I/O, retries, locks, or slow dependencies. For that reason, the arithmetic mean rarely describes the full shape of the distribution on its own.

OpenTelemetry models histograms as a population of measurements compressed into counts, a sum, bucket boundaries, and a time window. Prometheus uses the same principle to calculate quantiles from aggregatable histograms.

A percentile summarizes a position. A histogram or heatmap shows the shape more clearly. Operating a system usually requires both: percentiles to track thresholds and distributions to understand what changed.

The complete treatment of queues, latency budgets, and the components that accumulate time belongs in the specialized article on latency in distributed systems. The scope here is narrower: how to interpret a distribution without assigning properties to it that it does not measure.

What a percentile actually means

The percentile Px is the value below or equal to which approximately x% of the observations in the measured population and time window fall.

If a dashboard shows:

P95 = 480 ms

the operational interpretation is:

• Approximately 95% of observations completed in 480 ms or less.
• Approximately 5% took longer than 480 ms.

It does not mean:

• The system had 95% availability.
• 95% of requests succeeded.
• 95% of requests took exactly 480 ms.
• The remaining 5% were only slightly slower.
• The same value applies to every endpoint, region, or time window.

Availability and latency are separate indicators. A request can succeed and still arrive too late. It can also fail quickly. Google SRE explicitly separates the availability SLI—the proportion of successful requests—from the latency SLI—the proportion of requests below a threshold.

The word “approximately” matters

For a finite population, the exact result depends on the estimator, ranking convention, interpolation method, and, in observability platforms, the bucket schema. Hyndman and Fan documented multiple definitions of sample quantiles used by statistical software.

This has two consequences in production:

1. Two tools may report slightly different values for the same sample.
2. A percentile calculated from buckets is an estimate whose precision depends on the bucket boundaries.

The correct decision is not to chase false decimal precision. It is to verify whether the difference is material to the operational threshold and whether both values were calculated with comparable methods.

How to interpret P50, P95, P99, and P99.9

P50: the central experience

P50 is the median. It divides observations so that approximately half fall below or equal to the value and half fall above it.

It answers this question:

It is less sensitive than the average to extreme values. If one request takes 30 seconds, the median may not move. That makes it useful for describing the typical experience, but insufficient for evaluating the tail.

Observable signal: P50 rises consistently across all relevant segments.

Possible interpretations:

• A general increase in processing time.
• A routing or dependency change shared by most requests.
• Saturation that now affects the majority.
• Larger payloads or a different traffic composition.

Risk of misinterpretation: declaring a service healthy because P50 remains stable while P95 and P99 deteriorate.

Validation: compare P50 with throughput, saturation, errors, and segmentation by operation. If only the tail changes, the cause probably does not affect all requests uniformly.

P95: the frequent operational tail

P95 leaves approximately 5% of observations above it. In services with moderate or high traffic, that 5% can represent a substantial volume.

1,000,000 requests per day
5% above P95
≈ 50,000 requests

P95 answers a different question:

It is often useful on dashboards for interactive operations because it is more sensitive to degradation than P50 and less volatile than more extreme percentiles. It is not universally “the correct percentile.” Its usefulness depends on volume, criticality, and the shape of the distribution.

P99: extremes that stop being rare at scale

P99 leaves approximately 1% of observations above it.

100,000 requests
1% above P99
≈ 1,000 requests

At low volume, P99 may depend on very few observations. At large scale, the same 1% can affect hundreds of thousands of operations.

P99 is especially useful when:

• The platform processes high volume.
• The operation is sensitive to delay.
• One request depends on multiple downstream calls.
• Tail latency can trigger timeouts or retries.
• Extreme delays hold resources for longer periods.

Its trade-off is clear: it gives more sensitivity to the tail, but requires a larger sample and usually shows more variation in short windows.

P99.9: extreme precision with cost and volatility

P99.9 leaves approximately 0.1% above it. In ten million operations, that segment represents about ten thousand observations. In a window containing one thousand requests, it nominally represents a single observation.

It can be relevant for extremely high-volume platforms or operations where the far tail has material consequences. It also requires:

• More observations for a stable signal.
• Higher histogram resolution.
• Better control of truncation and timeouts.
• More care with cardinality and storage cost.
• Consistent comparison methods across tools.

Moving from P99 to P99.9 does not automatically make the measurement more useful. It only shifts the question toward a more extreme region of the distribution.

The 100,000-request example

Return to the initial window:

Volume = 100,000 requests
P50 = 120 ms
P95 = 480 ms
P99 = 1.8 s

A valid reading is:

• Approximately 50,000 requests completed in 120 ms or less.
• Approximately 95,000 completed in 480 ms or less.
• Approximately 99,000 completed in 1.8 s or less.
• Approximately 1,000 exceeded 1.8 s.

But essential questions are still unanswered:

• Was the window one minute, one hour, or one day?
• Did all 100,000 requests belong to the same endpoint?
• Were errors and timeouts included?
• Was latency measured at the client or the server?
• Which regions, versions, and payload sizes were mixed?
• How slow was the worst 1%: 1.9 seconds or 30 seconds?
• Was the percentile calculated from raw events, a summary, or a histogram?

The three percentiles do not reconstruct the complete distribution. Two systems can share the same P50, P95, and P99 while having radically different maxima, intermediate densities, and root causes.

Why the average can mislead you

Consider this sample:

99 requests = 100 ms each
1 request = 10,000 ms

The average is:

((99 × 100 ms) + 10,000 ms) / 100
= 19,900 ms / 100
= 199 ms

A dashboard that only shows average = 199 ms loses two important facts:

• The central experience was close to 100 ms.
• One request took 10 seconds.

P50 would be 100 ms. The maximum would be 10,000 ms. P99 would depend on the calculation convention and the way the sample is represented.

The average does not “lie” mathematically. It answers another question:

The error is using that answer to infer the shape of an asymmetric distribution.

Average, median, maximum, and histogram serve different purposes

The operational recommendation is not to replace the average with one percentile. It is to observe a minimal set that preserves the center, tail, traffic volume, and distribution shape.

Traffic volume changes the meaning of the tail

The same “1%” represents radically different impact depending on the number of operations.

A P99 of 1.8 seconds without volume does not tell you how many operations were above that value. It also does not reveal how many unique users were affected: one client can generate multiple requests and appear repeatedly in the tail.

Do not confuse requests with users

Converting percentiles into user impact requires another dimension:

• Identity or session, when privacy and cardinality permit it.
• Functional flow.
• Number of technical calls per user action.
• Transparent retries.
• Asynchronous operations triggered by one interaction.

One user action may generate ten calls. A per-request P99 does not automatically mean that 1% of users were affected.

Fan-out: when a backend P99 enters the critical path

A frontend request may query multiple partitions or dependencies in parallel and wait for the slowest response. In that design, the probability of encountering at least one slow call increases with fan-out.

Assume, only to illustrate the effect, that:

• Each call has a 1% probability of falling into its slow tail.
• Calls are independent.
• One frontend operation waits for all 100 responses.

The probability that none are slow is:

0.99^100 ≈ 36.6%

The probability that at least one is slow is:

1 - 0.99^100 ≈ 63.4%

A tail condition affecting 1% of individual calls can appear in most aggregated operations when fan-out is high. Independence is rarely perfect in real systems, but the calculation shows why the tail stops being rare when the critical path queries many components. Dean and Barroso's The Tail at Scale explains how infrequent episodes can dominate the performance of large services.

This section does not replace the complete analysis of tail latency. Its purpose is to explain why an extreme percentile in a dependency can become a frequent frontend experience.

The measurement window determines which incident you can see

Every percentile belongs to a time window. A value without its window is incomplete.

The incident a long window can hide

Assume one hour of stable traffic:

• 55 minutes at P99 = 300 ms.
• 5 minutes at P99 = 8 s.

The P99 for the full hour does not necessarily become 8 seconds. It depends on traffic volume in each interval and the combined distribution. If the incident affected fewer than 1% of all requests in the hour, it may disappear from the hourly P99 even though it was obvious during those five minutes.

Operational decision

Use multiple windows for different questions:

• Detection: Is degradation happening now?
• Confirmation: Is it sustained or was it an isolated point?
• Trend: Has behavior changed from the baseline?
• SLO: How much budget has been consumed during the objective period?

Do not compare a five-minute P99 with a one-hour P99 and present them as though they describe the same population.

Population and segmentation determine what you are measuring

A global percentile may combine incompatible distributions:

• Read and write endpoints.
• Regions with different network distances.
• Versions before and after a deployment.
• Cache hits and cache misses.
• Small and large payloads.
• Internal and external clients.
• Synchronous and batch operations.
• Successful responses, errors, and timeouts.

Example: a fast endpoint hides a slow one

Assume:

GET /catalog
900,000 requests
P99 = 180 ms

POST /order
100,000 requests
P99 = 2.4 s

If both operations are mixed, the higher-volume endpoint may dominate the global distribution. The aggregate percentile does not answer either of these questions well:

• Is catalog retrieval healthy?
• Does order creation meet its objective?

Decision

Segment by dimensions that change the nature of the operation, but avoid uncontrolled cardinality. A reasonable initial taxonomy may include:

• Normalized route or operation type.
• Grouped result code.
• Region.
• Deployment version.
• Payload class.
• Cache state, when operationally relevant.

Do not use user IDs, request IDs, or unnormalized URLs as metric labels. Those identifiers belong in logs or traces, not in high-cardinality time series.

OpenTelemetry allows attributes to be attached to histogram data points. That capability improves filtering, but every attribute combination can produce another series and increase cost.

Clients and servers do not observe the same latency

Server-side latency commonly begins after the request reaches the server and ends when the response is emitted. Client-side latency may additionally include:

• DNS resolution.
• Connection establishment.
• TLS negotiation.
• Network transit.
• Waiting in proxies or gateways.
• Body transfer.
• Client retries.
• Local processing.

Google SRE notes that client-observed latency is usually more relevant to the user, although only server-side measurements may be available in some systems.

These values are therefore not interchangeable:

Server P99 = 400 ms
Client P99 = 1.2 s

The difference does not prove that the network is the cause. It may include queues in intermediate layers, automatic retries, handshakes, streaming, or instrumentation with different boundaries.

Measurement decision

Define explicitly:

• Start point.
• End point.
• Unit.
• Whether retries are included.
• Whether timeouts are included.
• Treatment of cancellations.
• Segmentation by protocol and route.

Only then compare client and server observations through metrics, logs, and traces.

How a metrics platform calculates percentiles

A dashboard may show P99 = 742 ms, but that value can originate from different mechanisms.

Raw events

The system retains each observation and sorts or estimates directly from the data.

Advantage: maximum analytical flexibility.

Cost: storage, transfer, and query volume.

This is rarely the primary strategy for long-retention, high-volume telemetry.

Summary with precomputed quantiles

The instrumented process calculates quantiles over a window and exports values such as P50, P95, or P99.

Advantage: the client can control the quantile algorithm and precision.

Limitation: precomputed quantiles are generally not aggregatable across instances. OpenTelemetry retains Summary as a legacy type and does not recommend it for new applications because its quantiles cannot always be combined meaningfully.

Classic histogram

Each observation increments cumulative buckets with predefined boundaries:

≤ 0.1 s
≤ 0.3 s
≤ 0.5 s
≤ 1.0 s
≤ 2.0 s
+Inf

The backend estimates the percentile by interpolating inside the bucket containing the requested rank.

Advantage: compatible buckets can be aggregated across instances and dimensions.

Trade-off: precision depends on boundary placement. A bucket that is too wide around an SLO threshold produces an estimate that may be operationally weak.

Exponential or native histogram

This representation uses buckets with relative resolution and can cover a broad dynamic range efficiently. OpenTelemetry defines ExponentialHistogram as stable, and Prometheus recommends native histograms when they are available and their resolution meets the precision requirement.

A PromQL example

For a classic histogram:

histogram_quantile(
  0.99,
  sum by (le, route) (
    rate(http_server_request_duration_seconds_bucket[5m])
  )
)

This query:

1. Calculates the rate of bucket increments over five minutes.
2. Aggregates instances while preserving le and route.
3. Estimates P99 per route.

This is a conceptual example. Metric names, labels, and filters must match the actual instrumentation. The implementation must also define whether errors, cancellations, and low-traffic routes are included.

Bucket error is part of the design

Prometheus documents that histogram_quantile() estimates within a bucket. If a percentile falls in a range from 200 to 300 ms, the actual value may be anywhere inside that interval depending on the internal distribution. Interpolation can produce a number that appears precise even though the real resolution is still bounded by the bucket.

Bucket boundaries should therefore be concentrated around important decisions:

• SLO threshold.
• Client timeout.
• Limit of acceptable user experience.
• Boundary where functional degradation begins.

There is no universal schema. An endpoint that normally completes in 5 ms needs different buckets from a batch operation that takes several minutes.

Why you should not average percentiles

Consider two instances:

Instance A
100,000 requests
P99 = 200 ms

Instance B
1,000 requests
P99 = 2,000 ms

The simple average of both P99 values would be:

(200 + 2,000) / 2 = 1,100 ms

That result treats both instances as though they had equal weight and preserves neither original distribution. It is not the global P99.

Prometheus describes averaging precomputed quantiles as statistically invalid and recommends aggregating histograms before calculating the quantile.

# Incorrect: average of precomputed P99 values
avg(http_request_duration_seconds{quantile="0.99"})

# Correct for compatible classic histograms
histogram_quantile(
  0.99,
  sum by (le) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

To obtain a global percentile, you need

• The underlying observations, or
• Aggregatable histograms with compatible semantics, or
• A mergeable sketch data structure, when the platform supports one.

The following must also match:

• Unit.
• Time window.
• Bucket boundaries or a compatible schema.
• Population definition.
• Treatment of errors and timeouts.

How to relate percentiles to an SLO

An SLO must specify the indicator, objective, population, and time window. The following statement appears complete, but still needs an operational definition:

99% of requests must complete in less than 800 ms over 30 days.

At minimum, it must clarify:

• Which operations are eligible.
• Where measurement starts.
• Which responses are excluded and why.
• How timeouts and cancellations are treated.
• The exact evaluation window.
• Which telemetry source is authoritative.

SLO based on good events

A directly aggregatable approach classifies each request:

Good event: duration ≤ 800 ms
Bad event: duration > 800 ms

The SLI is:

good_requests / total_requests

The objective can be:

≥ 99% over 30 days

For request-driven services, Google SRE recommends defining latency as the proportion of requests faster than a threshold and illustrates multiple objectives to capture both the central experience and the tail.

Example:

90% of requests < 300 ms
99% of requests < 800 ms

This prevents one tail objective from permitting a mediocre central experience, or one central objective from ignoring the tail.

SLO expressed as a percentile

The expression:

P99 ≤ 800 ms over the complete 30-day population

can be approximately equivalent to requiring at least 99% of those observations to remain below the threshold, provided that population, window, tie handling, and calculation method are the same.

It stops being equivalent when aggregation changes. For example:

P99 of every 5-minute window ≤ 800 ms

is not the same as:

99% of all requests during the month ≤ 800 ms

The first objective evaluates windows. The second evaluates events. A low-traffic minute can carry the same weight as a high-traffic minute when windows are counted, while an event-based SLO weights every request.

Recommended decision

For SLOs and error budgets, use a good-event ratio when you can measure it consistently. Keep P50, P95, P99, and histograms on dashboards to understand distribution shape and localize degradation.

A percentile is a view of a distribution. An SLO is a quantitative contract for acceptable experience.

How to alert without turning P99 into noise

Paging every time one P99 point crosses a threshold produces false positives in low-traffic services and alert fatigue in naturally variable systems.

Before paging, evaluate:

• Duration of the violation.
• Minimum traffic volume in the window.
• Magnitude of the excess.
• Error-budget consumption.
• Impact by operation or region.
• Correlation with errors, saturation, or timeouts.
• Comparison across short and long windows.

Google SRE recommends designing alerts around significant events and error-budget consumption, while evaluating precision, recall, detection time, and reset time.

A defensible percentile alert

Condition A:
P99 > 800 ms for 10 minutes

Condition B:
volume > 10,000 requests in the window

Condition C:
the degradation appears in at least two consecutive evaluations

This is not a universal configuration. Minimum volume, duration, and threshold must be derived from traffic, criticality, and the operational response time.

Better still: alert on SLO consumption

If the SLO is 99% ≤ 800 ms, every request above 800 ms consumes budget. A burn-rate alert can detect whether the budget is being depleted too quickly instead of reacting to an isolated P99 value.

This does not eliminate percentile dashboards. It separates two functions:

• Paging: protect the objective and user experience.
• Diagnosis: understand which part of the distribution changed.

What to do with low-traffic services

With 50 requests in five minutes, P99 is determined by the extreme end of a small sample. In that context:

• Widen the window.
• Use absolute counts of slow events.
• Display traffic volume next to the percentile.
• Avoid paging on one observation unless the operation is critical.
• Consider synthetic monitoring when a stable periodic signal is required.

There is no magic sample size. With N observations, the expected count above P99 is 0.01 × N:

Having one hundred observations does not make P99 stable; it merely places one nominal observation in the upper tail. Precision depends on the distribution, estimator, and required confidence level.

Coordinated omission: the load test that omits the damage

A load test can report excellent percentiles while the system was unable to accept the planned arrival rate.

This occurs when the generator operates in a closed loop:

1. Send a request.
2. Wait for the response.
3. Only then send the next request.

If the system stalls for one second, the generator also stops creating new work. Requests that should have arrived during the wait never enter the dataset. The measurement records one slow response but omits the queue that an independent external arrival stream would have created.

Timeline example

Target load:

100 requests per second
1 request every 10 ms

The system stalls for one second.

A closed-loop generator may behave like this:

t = 0 ms     send request A
t = 1000 ms  receive A
t = 1000 ms  send request B

During that second, approximately 99 additional arrivals from the target schedule were not recorded.

In an open model, operations continue to be scheduled every 10 ms. The result reflects queue buildup, waiting, rejection, or capacity loss.

HdrHistogram provides mechanisms to correct data for coordinated omission by adding estimated values based on the expected interval between samples. That correction can be useful, but it does not replace designing a test that preserves the intended arrival pattern.

Decision for load testing

• Define whether the test models closed users or open external arrivals.
• Preserve the planned arrival rate when the real use case requires it.
• Record offered throughput and completed throughput.
• Include rejections, timeouts, and cancellations.
• Verify whether the tool corrects coordinated omission and how it does so.
• Do not compare results from different load methodologies as though they were equivalent.

When a saturated test appears to “improve” P99 while throughput falls, inspect the load model before celebrating the result.

How to diagnose percentile degradation

A high percentile is a symptom. It does not identify the cause.

1. Confirm that the data is comparable

Verify:

• Same operation.
• Same time window.
• Same population.
• Same measurement point.
• Same histogram method.
• Sufficient traffic volume.
• No changes in units or bucket boundaries.

If instrumentation changed during a deployment, the jump may be telemetry rather than system behavior.

2. Determine which part of the distribution moved

These are hypotheses, not causes. They determine which signal to inspect next.

3. Segment before searching for a global cause

Slice the distribution by:

• Endpoint.
• Region.
• Version.
• Response type.
• Cache hit or miss.
• Dependency.
• Payload class.

A global degradation may be the result of one slow route receiving more traffic.

4. Correlate with the four signals

Review:

• Latency: which percentiles and buckets moved.
• Traffic: volume, composition, and fan-out.
• Errors: timeouts, rejections, cancellations, and failures.
• Saturation: CPU, memory, pools, queues, connections, and I/O.

Google SRE notes that a short-window increase in P99 can be an early saturation signal, but it must be analyzed with the rest of the system.

5. Use exemplars or traces to move from metric to request

A histogram shows that slow requests exist. A distributed trace can locate where time accumulated.

The operational flow is:

P99 rises
→ identify bucket and segment
→ select exemplar or trace ID
→ inspect spans
→ correlate logs and resources
→ formulate a testable cause

The complete methodology belongs in the specialized article on structured incident investigation.

6. Validate the fix across the full distribution

After a change, do not verify only that P99 fell. Validate:

• P50, P95, and P99.
• Comparable traffic volume.
• Error rate.
• Throughput.
• Saturation.
• Distribution by endpoint and region.
• Resource consumption.
• No regression in another operation.

An optimization can reduce P99 by dropping requests, shortening a timeout, or returning a degraded fallback. Latency improves, but quality or availability worsens.

Common mistakes

Interpreting P99 without traffic volume

Problem: the number of observations represented by the tail is unknown.

Consequence: impact is either minimized or exaggerated.

Correction: display total traffic volume and the approximate count above the percentile.

Confusing P95 with availability

Problem: assuming P95 implies 95% success.

Consequence: two SLIs are mixed, hiding fast failures or slow successes.

Correction: measure latency and success separately.

Mixing endpoints

Problem: one high-volume route dominates the global percentile.

Consequence: the slow critical operation disappears inside the aggregate.

Correction: segment by normalized route or functional class.

Comparing different windows

Problem: comparing a five-minute P99 with a daily P99.

Consequence: attributing an aggregation difference to system performance.

Correction: align the time window and population.

Averaging percentiles

Problem: averaging P99 values across instances or regions.

Consequence: producing a number with no valid statistical interpretation.

Correction: aggregate compatible histograms or underlying data before calculating the quantile.

Using insufficient samples

Problem: calculating P99 from very few observations.

Consequence: high volatility and alerts triggered by one isolated extreme.

Correction: display N, widen the window, or use counts of slow events.

Excluding errors and timeouts without declaring it

Problem: measuring only requests that completed successfully.

Consequence: the worst operations disappear from the distribution.

Correction: define eligibility and represent timeouts as bad events or in a complementary metric.

Truncating the metric at the timeout

Problem: recording every cancelled operation as exactly one second.

Consequence: creating an artificial spike and losing information about residual server work.

Correction: distinguish client-observed duration, cancellation time, and work that continued on the server.

Treating client and server percentiles as equivalent

Problem: comparing different measurement boundaries.

Consequence: diagnosing the network or application incorrectly.

Correction: document the scope of every timer.

Comparing histograms with different buckets

Problem: assuming two estimates have equal precision.

Consequence: interpreting instrumentation differences as performance changes.

Correction: inspect the schema, interpolation, and resolution.

Ignoring coordinated omission

Problem: the generator stops sending work while responses are slow.

Consequence: artificially optimistic percentiles under saturation.

Correction: use a load model consistent with the real arrival pattern and validate offered throughput.

Optimizing the percentile while making the system worse

Problem: lowering P99 with overly aggressive timeouts, rejection, or fallback.

Consequence: apparent latency improvement combined with lower success or quality.

Correction: validate latency, availability, quality, and load together.

Wait limits and retries must be designed as one system. The relationship with time budgets and load amplification belongs in the article on correctly configured timeouts and retry storms.

Circuit behavior under slow dependencies belongs in the specialized article on cascading failures.

Operational checklist

Before interpreting a percentile, answer:

• Which operation does it represent?
• Which population does it include?
• What is the time window?
• How many observations does it contain?
• Is the measurement taken at the client, gateway, or server?
• Are failed responses included?
• Are timeouts and cancellations included?
• Are routes normalized?
• Is the data segmented by region, version, or payload class?
• Does the value come from raw events, a summary, or a histogram?
• Do buckets have sufficient resolution near the threshold?
• Were distributions aggregated before the percentile was calculated?
• Does the comparison use the same method and unit?
• Is there enough data for the selected percentile?
• Is the SLO defined by events or by windows?
• Does the alert consider duration, traffic volume, and impact?
• Does the load test avoid coordinated omission?
• Does post-change validation include success, throughput, and saturation?

Conclusion

P50, P95, and P99 do not compete with one another. Each observes a different region of the distribution.

P50 describes the center. P95 shows a frequent tail. P99 exposes extremes that can become massive at scale. None of them, in isolation, determines whether the system is available, whether it meets an SLO, or whether the experience is acceptable.

A correct interpretation requires five pieces of information:

1. Percentile: which position is being observed.
2. Window: over what period.
3. Volume: how many observations support it.
4. Population: which operations, regions, and outcomes were mixed.
5. Method: how the distribution was captured, aggregated, and estimated.

The concrete action is to review every dashboard that currently shows an isolated P95 or P99. Add traffic volume, time window, segmentation, the SLO threshold, and access to the histogram or heatmap. Then verify that percentiles are calculated from aggregatable distributions and that errors, timeouts, and load-test methodology are not removing precisely the observations you need to see.

A percentile without context is a number. A percentile connected to an operational decision is a signal.

Frequently asked questions

Does P95 mean that 95% of requests took exactly that value?

No. It means that approximately 95% completed in that value or less within the measured population and time window. Those requests can be distributed across many lower values.

Which is better: P95 or P99?

Neither is universally better. P95 is usually more stable and exposes a frequent tail. P99 examines a more extreme region and requires more traffic. The choice depends on criticality, volume, fan-out, experience thresholds, and the cost of false positives.

Can percentiles be averaged?

Generally, no. Averaging percentiles calculated over separate groups does not produce the global percentile. You need the underlying observations or compatible aggregatable histograms or sketches.

What is the difference between the average and P50?

The average divides total duration by the number of observations. P50 is the median: the value that divides the sample approximately in half. Extreme values can move the average without changing P50.

What is coordinated omission?

It is an underestimation of latency that occurs when a load generator stops producing new operations while waiting for a slow response. Arrivals that should have occurred during the wait are never recorded.

How many requests do I need to calculate P99?

There is no universal number. With one hundred observations, only one is expected in the top 1%, which makes the estimate highly sensitive. The required sample depends on the distribution, desired precision, window, estimator, and operational use. Always display traffic volume next to the percentile.

Is P99 below 800 ms equivalent to an SLO of 99% below 800 ms?

It can be approximately equivalent when calculated over exactly the same population and time window with a compatible definition. It is not equivalent when percentiles are averaged, separate windows are evaluated, or events are excluded differently. For error budgets, counting good and bad events against the threshold is usually clearer.

Does a lower P99 always represent an improvement?

No. It may fall because work was reduced, requests were rejected, timeouts increased, or the population changed. Validate it together with success, quality, throughput, saturation, and traffic composition.

Technical sources

• Prometheus — Histograms and summaries: prometheus.io/docs/practices/histograms
• OpenTelemetry — Metrics Data Model: opentelemetry.io/docs/specs/otel/metrics/data-model
• Google SRE — Monitoring Distributed Systems: sre.google/sre-book/monitoring-distributed-systems
• Google SRE — Service Level Objectives: sre.google/sre-book/service-level-objectives
• Google SRE Workbook — Implementing SLOs: sre.google/workbook/implementing-slos
• Google SRE Workbook — Alerting on SLOs: sre.google/workbook/alerting-on-slos
• Jeffrey Dean and Luiz André Barroso — The Tail at Scale: research.google/pubs/the-tail-at-scale
• HdrHistogram — IntCountsHistogram JavaDoc (coordinated omission): hdrhistogram.github.io/HdrHistogram/JavaDoc/org/HdrHistogram/IntCountsHistogram.html
• Rob J. Hyndman and Yanan Fan — Sample Quantiles in Statistical Packages: robjhyndman.com/publications/quantiles

Jorel del Portal

Systems engineer specialized in enterprise software architecture and high-availability platforms.

LinkedIn YouTube Tiktok Instagram X Threads My website