Observability vs. Monitoring: Metrics, Logs, and Traces in Distributed Systems

An API begins responding slowly. The dashboard confirms that P95 latency has increased, but traffic is stable and the error rate has barely moved. The service is technically available, yet users are waiting almost a second for an operation that normally completes in 200 milliseconds.

The monitoring system has detected the symptom. It has not explained which dependency consumed the time, whether every request was affected, what changed before the degradation, or which mitigation is safe.

That distinction is the practical boundary between monitoring and observability. Monitoring checks expected conditions. Observability gives engineers enough correlated evidence to investigate behavior they did not anticipate.

Metrics quantify the symptom. Traces reconstruct the execution path. Logs provide event-level context. None of them is sufficient by itself, and collecting all three does not automatically make a system observable. The capability emerges only when the signals share consistent identity, semantics, ownership, and operational purpose.

This guide explains how to design that capability, how to avoid telemetry that is expensive but diagnostically weak, and how to validate that the resulting system actually improves production decisions.

Monitoring and observability are not synonyms

Monitoring and observability overlap, but they describe different capabilities.

Google's SRE definition of monitoring focuses on collecting, processing, aggregating, and displaying quantitative data about a system. Monitoring is therefore an operational activity: define signals, evaluate conditions, visualize behavior, and notify people or automation when a condition requires action.

Observability is a property of the system and its instrumentation. A system is observable when an engineer can use its externally emitted evidence to ask new questions about internal behavior without first deploying new diagnostic code. OpenTelemetry describes this in terms of being able to investigate novel problems and answer why a behavior is occurring.

The distinction is not that monitoring uses metrics while observability uses three tools. Modern monitoring may use metrics, logs, traces, synthetic checks, profiles, and events. The distinction is the kind of question the system lets you answer.

What monitoring answers

Monitoring is effective when the condition can be defined before it happens:

• Is the service accepting requests?
• Has the error ratio exceeded the SLO threshold?
• Is P95 latency above the operational budget?
• Is the connection pool close to exhaustion?
• Has a queue stopped draining?
• Is the current release behaving differently from the previous one?

These are known questions with explicit measurements and evaluation logic.

What observability supports

Observability becomes necessary when the investigation is not known in advance:

• Why are only requests from one region slow?
• Which dependency dominates the latency of a specific operation?
• Why do two instances running the same release behave differently?
• Did the deployment cause the degradation, or did traffic composition change at the same time?
• Which customer-visible operations were affected by a shared infrastructure event?
• Is the queue growing because consumers are slower, producers are faster, or retries are duplicating work?

Observability does not guarantee automatic root-cause identification. It provides the evidence needed to reduce the search space, form a hypothesis, and test it. A trace can show where time accumulated without proving why it accumulated. A log can show a timeout without proving whether the dependency, the network, or local resource starvation caused it.

Operational comparison

The monitoring path determines that a known condition requires attention. The observability path supplies the evidence used after detection to understand scope, construct a hypothesis, and validate the response.

The three core telemetry signals

Metrics, logs, and traces remain the most useful operational model for understanding distributed applications. Calling them the “three pillars” is convenient, but it is not a complete definition of observability. OpenTelemetry also models context, baggage, events, resources, and profiles. The three-signal model is valuable because each signal has a different cost and diagnostic shape.

Metrics: detect and quantify

A metric is a numeric measurement associated with time and a bounded set of dimensions. Metrics are designed for aggregation. They answer how much, how often, how long, and how behavior changes over time.

Common metric instruments include:

• Counter: a monotonic total, such as completed requests or timeout events.
• Up-down counter: a value that may increase or decrease, such as active jobs or open connections.
• Gauge: the most recent observed value, such as queue depth or memory pressure.
• Histogram: a distribution of measurements, such as request duration or payload size.

The exact instrument model depends on the telemetry standard and backend. The OpenTelemetry Metrics API, for example, defines counters, gauges, up-down counters, and histograms, while its data model describes how aggregated values and distributions are represented.

Useful service metrics include:

• Request rate by operation.
• Success and error ratios.
• P50, P95, and P99 latency.
• Queue depth and queue age.
• Active, idle, and waiting connections.
• Saturated worker or thread pools.
• Retry attempts and rejected requests.
• SLO compliance and error-budget burn rate.

A metric is efficient because many events become one time series. That efficiency is also its limitation. A P95 value can show that 5% of requests exceeded a duration threshold, but it normally cannot tell you which individual requests were slow or what happened inside them.

For a deeper treatment of end-to-end performance and latency budgets, keep the system-level design question separate from the statistical interpretation of percentiles. For the latter, use the dedicated article on latency percentiles P50, P95, and P99.

Decision rule for metrics

Use a metric when the question requires aggregation across many events and the dimensions can remain bounded.

Do not put request IDs, transaction IDs, timestamps, raw URLs, email addresses, or user IDs into metric labels. Those values belong in logs or traces because their near-unique nature creates a new time series for each distinct label set.

Logs: preserve event context

A log is a record of an event. Its diagnostic value depends less on the sentence it contains than on the structure and context attached to it.

A weak log entry says:

Something went wrong while processing the request.

It provides no stable operation name, service identity, result, error classification, duration, or correlation field. During an incident, engineers must infer context from surrounding lines, hostnames, and timestamps. That approach fails quickly when requests cross concurrent services.

A stronger structured record might be:

{
  "timestamp": "2026-07-12T22:14:08.417Z",
  "severity": "ERROR",
  "service.name": "authorization-api",
  "service.version": "2026.07.12.3",
  "deployment.environment": "production",
  "operation": "authorize",
  "result": "timeout",
  "trace_id": "4e8f2c7b6bb44df19cd9d6f0db8916a2",
  "span_id": "a1729f6b1c83e402",
  "correlation_id": "order-8129",
  "dependency": "risk-service",
  "duration_ms": 1842,
  "pool_wait_ms": 1614,
  "error_code": "DEPENDENCY_CONNECTION_ACQUIRE_TIMEOUT"
}

This record supports several investigative paths:

• Search all logs associated with one trace.
• Compare failures by service version.
• Separate dependency execution time from local pool waiting time.
• Group a larger business workflow by correlation ID.
• Measure whether the same error code increased after a deployment.

OpenTelemetry's logging specification explicitly supports correlation through time, trace context, and resource context. Including Trace ID and Span ID in log records lets an engineer move from a trace waterfall to the precise events emitted during that execution.

Fields that should be consistent

At minimum, application logs should use stable names for:

• Event timestamp and observed timestamp when both matter.
• Severity.
• Service name and version.
• Deployment environment.
• Operation or event name.
• Result and error classification.
• Trace ID and Span ID when a trace context exists.
• Correlation ID when the domain workflow requires one.
• Duration and unit.
• Dependency or resource involved.

The specific schema should follow a shared convention rather than each team inventing its own names. OpenTelemetry Semantic Conventions exist precisely to provide common naming across languages, libraries, and platforms.

What not to log

More detail is not automatically better. Logs can create security, privacy, and cost risks. Avoid recording credentials, access tokens, complete payment data, personal data without a defined purpose, or full request and response bodies by default. Redaction and filtering must be part of the telemetry pipeline, not an afterthought after sensitive data reaches a backend.

Distributed traces: reconstruct execution

A distributed trace represents the path of an operation across process and network boundaries. It is composed of spans. Each span describes one operation with a start time, end time, attributes, events, status, and a parent relationship or link.

A simplified request path might be:

Client
  -> API Gateway
    -> Authorization API
      -> Risk Service
        -> Database
        -> External Rules API

A trace turns that topology into a timed execution. It can show that the Authorization API completed 70 milliseconds of local work, waited 620 milliseconds for a connection to the Risk Service, spent 130 milliseconds in the dependency itself, and returned after 820 milliseconds overall.

That distinction matters. Without the waiting span or attribute, an engineer may blame the downstream service because it appears in the request path. With the complete timing, the evidence points instead to local connection acquisition or concurrency control.

Metrics establish impact and trend. Traces identify the degraded path. Logs explain the state and events associated with that path. The diagnosis depends on joining them, not reading them as isolated dashboards.

A trace localizes time; it does not automatically prove cause

A long span is evidence of where elapsed time was observed. It may represent:

• Work performed by the span owner.
• Waiting for a connection, lock, thread, or queue.
• Network delay.
• Retries hidden inside a client library.
• Time spent in a remote dependency.
• Missing child instrumentation that collapses several operations into one span.

The next decision is to compare traces, inspect resource metrics, and find state-specific logs. Observability supports causal analysis; it does not replace it.

Trace ID, Span ID, and Correlation ID

These identifiers are related but solve different problems.

Trace ID

A Trace ID identifies the complete distributed execution represented by one trace. Every span in that trace shares the same Trace ID.

The W3C Trace Context specification defines interoperable propagation through the traceparent and optional tracestate headers. The traceparent value carries the trace identifier, parent span identifier, version, and trace flags in a vendor-neutral format.

Example:

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

The Trace ID in this example is:

0af7651916cd43dd8448eb211c80319c

Span ID

A Span ID identifies one operation inside a trace. Parent and child Span IDs reconstruct the causal tree or graph.

A trace may contain spans for:

• The inbound HTTP request.
• An internal authorization operation.
• A database query.
• A message publication.
• A remote service call.

Each span has its own Span ID but retains the Trace ID of the distributed execution.

Correlation ID

A Correlation ID is an application-defined identifier used to associate events that belong to a logical workflow. It is not a replacement for trace context and is not standardized by W3C Trace Context.

A single business workflow may outlive one trace. For example, an order may be created synchronously, processed asynchronously, retried hours later, and reconciled the next day. Those executions may produce several traces while sharing one stable order or workflow correlation identifier.

Conversely, one trace may involve several business entities. Forcing a single domain identifier to serve as the Trace ID can break trace generation, sampling assumptions, and interoperability.

When to use each identifier

Propagation decision

Propagate W3C trace context across supported synchronous and asynchronous boundaries. Preserve application correlation fields only when they have a defined owner, lifecycle, and data classification.

Do not copy arbitrary user-supplied values into trusted correlation fields without validation. Do not overload baggage with large or sensitive values: propagated context travels through service boundaries and can increase request size or expose data to components that do not need it.

OpenTelemetry without backend lock-in

OpenTelemetry provides vendor-neutral APIs, SDKs, semantic conventions, context propagation, the OpenTelemetry Protocol, and the Collector. It standardizes how telemetry is produced and moved. It does not provide the complete observability outcome by itself.

What OpenTelemetry solves

OpenTelemetry helps standardize:

• Trace, metric, and log instrumentation.
• Context propagation across service boundaries.
• Automatic or zero-code instrumentation for supported libraries and runtimes.
• Code-based instrumentation for application-specific operations.
• Resource identity such as service name and version.
• Semantic naming across languages and frameworks.
• Export through OTLP and other supported protocols.
• Collection, batching, filtering, transformation, sampling, and routing through the Collector.

What OpenTelemetry does not decide

It does not decide:

• Which user journeys require an SLO.
• Which business operation deserves a manual span.
• Which labels are safe for metrics.
• Which data is sensitive.
• Which traces must be retained.
• Which alert should page an engineer.
• Which dashboard question matters during an incident.
• Which backend retention and query model fits the organization.

Those remain architecture and operating-model decisions.

Recommended signal path

The Collector decouples applications from backend-specific ingestion and centralizes cross-cutting controls such as batching, retries, filtering, redaction, and sampling. Small environments may export directly, but a Collector becomes valuable when telemetry policy must be changed without redeploying every service.

Automatic instrumentation and manual instrumentation

OpenTelemetry documentation distinguishes code-based and zero-code approaches. They should be combined rather than treated as competing choices.

Automatic or zero-code instrumentation is useful for:

• Inbound and outbound HTTP calls.
• Framework middleware.
• Database clients.
• Messaging clients.
• Runtime and process telemetry.
• Common network operations.

It quickly reveals technical topology and baseline timing.

Manual instrumentation is required for:

• Domain operations such as authorize_order or calculate_quote.
• Functional stages inside one request.
• Attributes that classify the operation in a bounded, meaningful way.
• Events such as fallback activation or policy rejection.
• Boundaries that generic library instrumentation cannot infer.

Automatic instrumentation may show that an HTTP call took 800 milliseconds. Manual instrumentation may reveal that 650 milliseconds were spent waiting for a limited permit before the HTTP client was invoked.

A defensible instrumentation sequence

1. Establish consistent resource identity: service name, version, environment, region, and instance or workload identity where appropriate.
2. Enable automatic instrumentation for standard protocols and libraries.
3. Define the critical user and business operations that require manual spans or metrics.
4. Apply semantic conventions before inventing custom attribute names.
5. Add trace and span context to structured logs.
6. Define attribute cardinality budgets and sensitive-data rules.
7. Load-test the telemetry path and measure application overhead, Collector saturation, dropped data, and backend ingestion cost.
8. Validate the signals through a controlled failure or game-day scenario.

The last step is essential. Instrumentation that looks complete in a demo can fail under concurrency, asynchronous execution, partial sampling, or incident-level volume.

How to choose between RED, USE, and the Golden Signals

RED, USE, and the Four Golden Signals are not competing standards. They organize different views of the same system.

RED for request-driven services

The RED method, created by Tom Wilkie for service monitoring, focuses on:

• Rate: how many requests or operations are processed.
• Errors: how many fail, including semantically failed responses where applicable.
• Duration: the distribution of execution time.

RED is a strong default for APIs, RPC services, consumers, and other request-driven components. It reveals whether demand, failure, or latency changed.

USE for resources

Brendan Gregg's USE method evaluates each resource through:

• Utilization: how busy the resource is.
• Saturation: how much work is waiting because the resource cannot serve it immediately.
• Errors: error events associated with the resource.

USE applies to CPUs, disks, network interfaces, connection pools, thread pools, queues, memory capacity, and other finite resources.

The saturation dimension is critical. Average utilization can look acceptable while brief intervals reach full capacity and create queues. A five-minute CPU average of 70% does not prove that no one-second interval reached 100%.

The Four Golden Signals for service health

Google SRE's Four Golden Signals are:

• Latency.
• Traffic.
• Errors.
• Saturation.

They provide a compact operational view of user-facing service behavior and resource pressure.

How to choose

RED and the Golden Signals show the user-visible symptom. USE examines whether a finite resource explains it. The methods become useful when they lead to the next decision, not when they exist as three separate dashboard folders.

Cardinality: when one label breaks the metrics system

Metric systems identify a time series by the metric name plus its complete label set. Every unique combination creates another series with storage, memory, indexing, query, and network cost.

This metric is dangerous:

http_requests_total{
  user_id,
  transaction_id,
  request_id,
  timestamp
}

Every request can create a new label combination. The metric stops behaving like an aggregate and becomes an expensive event store implemented in the wrong signal.

A more defensible design is:

http_requests_total{
  service,
  operation,
  status_code,
  region
}

These dimensions are bounded and support operational aggregation.

Prometheus documentation explicitly warns that high-cardinality or unbounded values such as user IDs and email addresses should not be used as labels. The same principle applies beyond Prometheus: almost-unique dimensions belong in logs or traces, not metric identity.

Why cardinality multiplies

Assume one request-duration metric uses these observed dimensions:

• 12 services.
• 40 operations.
• 5 status classes.
• 6 regions.

The theoretical upper bound is:

12 × 40 × 5 × 6 = 14,400 time series

Add 100,000 active user IDs:

14,400 × 100,000 = 1,440,000,000 possible combinations

The backend may not observe every combination, but each new unique label set still becomes a new time series. Even a small fraction of that upper bound can overwhelm ingestion, memory, retention, and query performance.

Cardinality is not only a storage problem

High cardinality also causes:

• Slow dashboards and alerts.
• Expensive wide queries.
• Increased Collector and backend memory pressure.
• Unpredictable cost during traffic spikes.
• Longer incident investigations because queries time out.
• Operational blind spots when the platform begins dropping telemetry.

Bounded operational dimensions already multiply. Adding a user-level identifier changes the metric from an aggregate signal into an unbounded series generator.

A label decision test

Before adding a metric label, ask:

1. Is the set of values bounded?
2. What is the current and projected number of distinct values?
3. Does the dimension support an operational decision or only curiosity?
4. Can the question be answered through logs or traces instead?
5. What happens if a malformed client creates arbitrary values?
6. Is the dimension required at full resolution or can it be normalized?

For HTTP metrics, use normalized routes such as /orders/{orderId} rather than raw paths such as /orders/8129.

Trace sampling as an engineering decision

Tracing every request may be acceptable at low volume, but it can become expensive in high-throughput systems. Sampling is the decision to retain and export only part of the trace population.

The objective is not to minimize data at any cost. It is to preserve enough evidence to answer operational questions while keeping application overhead, Collector capacity, backend ingestion, and retention within budget.

Head sampling

Head sampling decides near the start of the trace, before the final outcome is known.

Advantages:

• Simple.
• Low infrastructure overhead.
• Predictable volume.
• Available in SDKs and early pipeline stages.

Limitations:

• The sampler may discard a request that later becomes slow or fails.
• A fixed probability can underrepresent rare failure modes.
• Decisions must use information available at trace creation.

Tail sampling

Tail sampling decides after enough spans have arrived to evaluate the trace outcome.

Advantages:

• Retain traces with errors.
• Retain slow traces.
• Apply rules based on span attributes or complete duration.
• Keep a smaller representative sample of successful traffic.

Limitations:

• Requires buffering and additional memory.
• Introduces decision latency.
• Requires all spans for one trace to reach the same decision point.
• Becomes more complex to scale and operate.

The OpenTelemetry Collector tail-sampling processor groups spans by Trace ID and requires consistent routing so all spans for a trace reach the same Collector instance responsible for the decision.

A defensible sampling policy

A common starting policy is:

• Retain all traces classified as errors.
• Retain traces above operation-specific latency thresholds.
• Retain traces for rare or high-value operations.
• Retain a statistically useful sample of successful, normal-latency traffic.
• Apply stricter limits to known noisy endpoints.
• Increase sampling temporarily during a controlled investigation only if the pipeline has capacity.

Thresholds must be operation-specific. A 500-millisecond threshold may be severe for a cache lookup and normal for a long-running report.

What can go wrong

• Sampling only failures removes the healthy baseline needed for comparison.
• Sampling 1% uniformly can miss low-volume operations entirely.
• Tail sampling can overload the Collector during an incident, exactly when trace volume and value increase.
• Independent sampling decisions across services can produce broken traces.
• Increasing the sample rate without checking backend quotas can turn an application incident into a telemetry incident.

How to validate sampling

Track the sampling system itself:

• Received, accepted, sampled, and dropped spans.
• Decision latency.
• Collector memory and CPU.
• Queue and exporter failures.
• Traces retained by policy.
• Coverage by service and operation.
• Broken or incomplete traces.

Sampling is successful when investigators can still compare healthy and degraded executions, preserve rare failures, and query the trace backend during peak incidents.

Alert on impact, diagnose with causes

An alert should communicate that a condition requires action. The most reliable paging signals usually express user or service impact:

• Availability below an objective.
• Error ratio consuming the error budget too quickly.
• Latency SLI outside the agreed threshold.
• A critical workflow not completing.
• A durable queue violating its maximum acceptable age.

Google SRE recommends SLO-based alerting because it aligns the page with reliability as users experience it. Prometheus guidance similarly recommends alerting on symptoms and using consoles to identify causes.

Symptom signals

Examples:

• More than 2% of authorization requests fail over a multi-window burn-rate rule.
• P95 latency for a critical operation exceeds its objective and enough traffic exists to make the signal reliable.
• Successful checkout completion falls below the SLI target.
• Oldest unprocessed message age exceeds the recovery objective.

Cause signals

Examples:

• CPU utilization is high.
• Connection pool has no idle connections.
• Disk space is below a threshold.
• Thread pool queue is growing.
• A dependency returns throttling responses.

Cause signals are valuable for diagnosis and sometimes for prevention. They are weaker as primary paging rules because the same cause may have no user impact, while user impact may occur through a cause that was never encoded.

When a cause alert is justified

A cause-based page can be appropriate when:

• The resource is about to cross an irreversible or slow-to-recover boundary.
• The condition threatens data integrity.
• The condition requires action before user-visible impact appears.
• The signal has a clear owner and runbook.
• Historical evidence shows a reliable relationship between the condition and severe impact.

Disk exhaustion, certificate expiration, exhausted capacity reservations, and replication failure can justify preventive alerts. The decision depends on actionability and consequence, not a blanket rule that cause alerts are always wrong.

Alert quality criteria

A page should have:

• A named owner.
• A clear statement of impact.
• A link to the affected service, SLO, dashboard, and runbook.
• Enough labels to route correctly, but not enough to create duplicate pages for one incident.
• A condition that resolves automatically when the service recovers.
• A tested response path.
• A severity that reflects urgency, not team preference.

How to prevent alert fatigue

Alert fatigue is not simply “too many alerts.” It is the loss of trust caused by alerts that are noisy, duplicated, non-actionable, stale, or poorly prioritized.

Google SRE warns that low-priority pages disrupt work and can cause serious alerts to receive less attention. It also recommends controlling alert fan-out so one abnormal condition does not generate several independent pages.

Common causes

• Static thresholds that ignore traffic level or time window.
• Separate alerts for every downstream symptom of one incident.
• No distinction between page, ticket, and informational event.
• Alerts with no owner.
• Alerts that describe a metric but not the required action.
• Rules that stay open after recovery.
• Repeated flapping around a threshold.
• Every instance paging independently for a service-level event.
• Alerts created during a past incident and never reviewed afterward.

A practical review

For each alert, ask:

1. What decision should the recipient make?
2. Must a human act now?
3. What user or system objective is threatened?
4. Can the same condition be grouped at service or incident level?
5. Does the notification contain enough context to begin triage?
6. Is there a runbook, and has it been exercised?
7. What is the expected alert-to-incident ratio?
8. When was the rule last useful?

An alert with no immediate action should become a ticket, report, or dashboard annotation rather than a page.

What an operational dashboard must show

A dashboard is not a wall of available metrics. It is a decision interface.

Each panel should answer a production question. A service landing dashboard should let an on-call engineer determine impact, scope, recent change, and the next investigative path within minutes.

Recommended hierarchy

1. User and SLO impact

• Availability or successful completion SLI.
• Latency SLI and relevant percentiles.
• Error-budget status and burn rate.
• Critical journey completion.

2. Demand and scope

• Traffic or work arrival rate.
• Operation, region, tenant class, or channel where bounded and operationally relevant.
• Current release and deployment distribution.

3. Service behavior

• Error ratio by stable classification.
• Duration distribution by operation.
• Retry, timeout, cancellation, and rejection rates.

4. Saturation and dependencies

• Queue depth and oldest age.
• Connection and thread-pool pressure.
• Dependency latency and error ratio.
• CPU throttling, memory pressure, disk, and network indicators as relevant.

5. Recent changes

• Deployments.
• Configuration changes.
• Feature-flag changes.
• Capacity events.
• Dependency incidents.

What not to include

• Dozens of panels with equal visual weight.
• Metrics without units.
• Averages without distributions when tail latency matters.
• Raw instance-level data on the primary service page unless it supports immediate drill-down.
• Charts with no owner or known decision.
• Decorative gauges whose thresholds have no operational meaning.
• Business KPIs mixed with technical symptoms without a clear relationship.

Dashboard decision table

The primary dashboard should not attempt to replace the trace or log backend. It should direct the engineer to the next high-value query.

Production case: an API becomes slow without failing

Consider a generic authorization-api with the following normal behavior:

• 8,000 to 10,000 requests per second.
• P50 latency: 95 ms.
• P95 latency: 220 ms.
• P99 latency: 410 ms.
• Error ratio: 0.4%.
• Latency objective: 95% of requests below 350 ms.

At 22:10 UTC, P95 rises to 890 ms. P99 reaches 1.7 seconds. Traffic remains near 9,200 requests per second and the error ratio increases only from 0.4% to 0.6%.

The service is not “down,” but it is consuming its latency budget and user-visible performance has degraded.

Step 1 — Detect the symptom

The alert should be tied to the latency SLI or error-budget burn, not merely to CPU or a pool metric.

Observed:

P95: 220 ms -> 890 ms
P99: 410 ms -> 1,700 ms
Request rate: 9,100 -> 9,200 requests/s
Error ratio: 0.4% -> 0.6%

Initial conclusion:

• The problem is primarily latency, not availability.
• Increased demand is not an obvious explanation.
• The stable error ratio does not make the incident harmless.

Step 2 — Establish scope

Break down bounded dimensions:

• Operation.
• Region.
• Service version.
• Response classification.
• Dependency route.

The degradation appears only in:

operation = authorize
region = west
service.version = 2026.07.12.3

Other operations and regions remain within their normal distributions.

Decision:

• Avoid a global mitigation until evidence shows global risk.
• Compare the affected version and region with a healthy control.

Step 3 — Compare healthy and slow traces

A healthy trace:

Gateway                         12 ms
Authorization API local work    58 ms
Risk Service call              118 ms
Database lookup                 31 ms
Total                          219 ms

A slow trace:

Gateway                         13 ms
Authorization API local work    71 ms
Connection acquisition wait    623 ms
Risk Service call              131 ms
Database lookup                 34 ms
Total                          872 ms

The remote call itself is only 13 milliseconds slower than normal. Most additional latency occurs before the call begins.

The trace localizes the dominant delay to connection acquisition. It does not yet prove why the pool is saturated or whether the new version caused it.

Step 4 — Correlate logs by Trace ID

Search structured logs for slow traces:

{
  "service.name": "authorization-api",
  "service.version": "2026.07.12.3",
  "operation": "authorize",
  "trace_id": "4e8f2c7b6bb44df19cd9d6f0db8916a2",
  "event": "connection_pool.acquire",
  "pool.active": 40,
  "pool.idle": 0,
  "pool.waiters": 173,
  "pool_wait_ms": 623,
  "result": "acquired"
}

A second log shows that requests still complete, explaining the small error increase. They are waiting rather than immediately failing.

Step 5 — Test competing hypotheses

Possible hypotheses:

1. The downstream Risk Service is slow.
2. Network latency increased.
3. The local connection pool is too small for current concurrency.
4. Connections are leaked or held longer by version 2026.07.12.3.
5. A retry policy increased concurrent dependency calls.

Evidence so far weakens hypotheses 1 and 2 because dependency execution time is close to baseline. It supports a local wait hypothesis but does not distinguish insufficient capacity, longer hold time, a leak, or retry amplification.

Inspect:

• Pool active, idle, waiter count, acquire duration, and timeout count.
• Connection hold duration.
• Calls per incoming request.
• Retry attempts by outcome.
• Version comparison.
• Dependency concurrency and rate limits.

The version comparison reveals:

2026.07.12.2: 1.02 dependency calls per request
2026.07.12.3: 1.84 dependency calls per request

A new retry path is retrying a class of responses that should not be retried. The extra attempts consume connections, increase waiters, and amplify latency without yet causing a large error spike.

This is a potential retry storm, even though the traffic arriving at the public API has not changed.

Step 6 — Choose a mitigation

Possible mitigations include:

• Disable the incorrect retry path through a feature flag.
• Roll back the affected version in the region.
• Reduce concurrency for the affected operation.
• Apply controlled degradation for the optional dependency path.
• Increase pool size only after validating downstream capacity and local resource cost.

Increasing the pool immediately is tempting but risky. It may transfer saturation to the downstream service, increase connection overhead, or hide the retry defect. The mitigation should remove amplification before adding capacity.

Decision:

• Disable the retry behavior for the non-retriable response class.
• Keep the rollback ready if latency does not recover.

For the design relationship between timeouts, retries, and cascading failures, treat each resilience mechanism as part of one load-control system rather than as an isolated library option.

Step 7 — Validate recovery

Do not close the incident because a configuration change succeeded. Validate the system-level outcome:

P95: 890 ms -> 245 ms
P99: 1,700 ms -> 460 ms
Pool waiters: 173 -> 4
Dependency calls/request: 1.84 -> 1.03
Error ratio: 0.6% -> 0.4%
Latency SLO burn: returned below alert threshold

Also verify:

• No regional error increase.
• No dependency saturation.
• Queue and pool metrics remain stable for an appropriate observation window.
• Healthy and previously slow traces now have comparable shapes.

Step 8 — Prevent recurrence

Add or improve:

• A metric for retry attempts by operation and outcome.
• A bounded trace attribute for retry reason.
• Pool acquisition duration as a histogram.
• A dashboard comparison of incoming requests versus dependency attempts.
• A load test that includes the response class that triggered retries.
• A policy test that proves non-retriable responses remain non-retriable.
• A release comparison panel by service version.

The complete incident method belongs in a dedicated piece so this article does not duplicate the full investigation framework: how to investigate an incident with metrics, logs, and traces.

The process moves from impact to scope, execution evidence, hypothesis testing, reversible mitigation, and measurable validation. It avoids jumping directly from one suspicious metric to a permanent configuration change.

Common instrumentation failures

1. Collecting signals that cannot be correlated

Metrics use one service name, logs use another, and traces omit service version. Time zones differ and deployment markers are absent.

Consequence: Engineers spend the incident proving that records belong to the same component.

Correction: Standardize resource identity and timestamps. Inject Trace ID and Span ID into logs. Keep deployment metadata queryable.

2. Instrumenting only framework boundaries

Automatic instrumentation captures HTTP and database calls but not the domain stage that waits for a permit, evaluates a rule set, or activates a fallback.

Consequence: One large span hides the operation that matters.

Correction: Add manual spans or events at decision boundaries, not around every function.

3. Creating one span per trivial function

Excessive spans increase cost and make traces unreadable.

Consequence: Important timing is buried in implementation detail.

Correction: Create spans for remote calls, asynchronous boundaries, expensive stages, and diagnostically meaningful operations.

4. Using raw identifiers as metric labels

Request IDs, user IDs, and raw URLs create unbounded time series.

Consequence: The metrics backend becomes expensive or unavailable during high traffic.

Correction: Normalize dimensions and move unique context to logs or traces.

5. Logging the same exception at every layer

A dependency exception is logged by the client, service, controller, gateway, and global handler.

Consequence: One failure appears as five independent events and can trigger duplicate alerts.

Correction: Record the event where ownership and context are strongest; propagate structured status without duplicating noise.

6. Sampling without checking coverage

A global 1% head sample is enabled and considered complete.

Consequence: Low-volume operations and rare failures disappear.

Correction: Measure coverage by operation and outcome; preserve errors, slow traces, and a healthy baseline.

7. Alerting on every anomaly

Every resource threshold pages independently.

Consequence: One incident creates a notification storm and the on-call engineer cannot identify the primary impact.

Correction: Page on significant service symptoms, group related causes, and use cause metrics for diagnosis or preventive tickets.

8. Treating observability as a backend migration

The organization changes vendors but keeps inconsistent names, missing context, weak alerts, and decorative dashboards.

Consequence: Query syntax changes; diagnostic capability does not.

Correction: Define questions, schemas, ownership, and validation before selecting or migrating the backend.

Operational checklist

Instrumentation

• Every service has stable service.name, version, environment, and deployment identity.
• W3C trace context is propagated across supported synchronous and asynchronous boundaries.
• Structured logs include Trace ID and Span ID when context exists.
• Critical domain operations have manual instrumentation where automatic instrumentation is insufficient.
• Metric names include units and follow one naming convention.
• High-cardinality and sensitive attributes are explicitly reviewed.
• Instrumentation overhead is measured under realistic load.

Metrics

• RED or equivalent service metrics exist for request-driven components.
• USE signals exist for finite resources that can saturate.
• Latency is recorded as a distribution suitable for required percentiles.
• Queue age is measured where waiting time matters more than queue length.
• Retry, timeout, rejection, and fallback activity is measurable.
• SLI and error-budget signals are visible.

Logs

• Logs use structured fields rather than relying on message parsing.
• Error codes are stable and documented.
• Duplicate exception logging is controlled.
• Sensitive fields are redacted before export.
• Retention and indexing match investigative value.

Traces

• Healthy and degraded traces can be compared.
• Span names are bounded and do not contain unique IDs.
• Remote calls, queues, and meaningful waits are visible.
• Sampling preserves errors, slow requests, and a successful baseline.
• Collector routing does not break tail-sampling decisions.
• Broken and dropped traces are monitored.

Alerts and dashboards

• Paging alerts express impact or imminent high-consequence failure.
• Every page has an owner and runbook.
• One incident does not produce uncontrolled alert fan-out.
• The service dashboard starts with SLO, traffic, errors, latency, and saturation.
• Deployment and configuration changes are visible in the same time frame.
• Every panel answers a named operational question.

Validation

• A controlled failure has been used to test the full investigation path.
• The team can move from alert to affected traces and logs without adding instrumentation.
• Mitigation success is validated through user, service, and resource signals.
• Telemetry pipeline saturation and data loss are observable.
• Post-incident actions include signal quality and regression coverage, not only more alerts.

Conclusion

Monitoring is necessary because engineers cannot respond to a condition they do not detect. Observability is necessary because detection rarely explains a distributed failure.

The practical objective is not to collect the maximum amount of telemetry. It is to create a controlled evidence system:

• Metrics reveal impact and trend.
• Traces localize the execution path.
• Logs explain event and state context.
• Shared identity lets the signals be joined.
• Sampling and cardinality controls keep the platform operable.
• SLO-oriented alerts bring humans in for significant conditions.
• Dashboards direct the next decision.

The final validation is operational. When an unfamiliar incident occurs, the team should be able to determine scope, compare healthy and degraded behavior, test competing hypotheses, apply a reversible mitigation, and prove that the service recovered.

That sequence is also the limit of what is worth automating. Once evidence is obtained verifiably, the question stops being what to instrument and becomes who interprets the result: enterprise observability with AI covers where to place a model above these controls without letting it decide whether the system is healthy.

Frequently asked questions

Are observability and monitoring the same thing?

No. Monitoring is the activity of collecting and evaluating signals against expected conditions. Observability is the system's capability to support investigation through its emitted evidence. A mature monitoring system contributes to observability, but dashboards and alerts alone do not guarantee it.

What are the three pillars of observability?

Metrics, logs, and traces are a useful operational model because they provide aggregation, event context, and execution paths. “Three pillars” is a simplification, not a complete technical definition. Context propagation, resource identity, events, baggage, semantic conventions, and profiles may also be part of an observability system.

What is the difference between a Trace ID and a Correlation ID?

A Trace ID identifies one distributed execution and is propagated through tracing standards. A Correlation ID is application-defined and may represent a business workflow that spans several traces. They can coexist, but they should not be treated as interchangeable.

What is OpenTelemetry?

OpenTelemetry is a vendor-neutral observability framework that defines APIs, SDKs, semantic conventions, context propagation, protocols, and a Collector for telemetry such as traces, metrics, and logs. It standardizes instrumentation and transport; it does not choose the operational questions or backend strategy for you.

Should I use RED or USE?

Use RED for request-driven service behavior and USE for finite resources. RED helps detect user-visible changes in rate, errors, and duration. USE helps test whether utilization, saturation, or resource errors explain the symptom. The Four Golden Signals provide a compact service-level view that overlaps with both.

When should I use P95 or P99?

Use the percentile that corresponds to the user population, operation criticality, traffic volume, and SLO you need to protect. P99 exposes tail behavior but can be noisy for low-volume windows. Do not select a percentile because it is fashionable; select it because it represents an explicit reliability objective and has enough observations to be interpreted.

For the full statistical treatment, use the dedicated guide to latency percentiles P50, P95, and P99.

How do I prevent alert fatigue?

Reduce pages to significant, actionable conditions. Group related symptoms, align pages with SLO or high-consequence risk, attach ownership and runbooks, and review alerts that repeatedly fire without requiring action. A page that does not change a decision should become a ticket, report, or dashboard signal.

Technical references

• OpenTelemetry — Observability primer: opentelemetry.io/docs/concepts/observability-primer
• OpenTelemetry — Signals: opentelemetry.io/docs/concepts/signals
• OpenTelemetry — Instrumentation: opentelemetry.io/docs/concepts/instrumentation
• OpenTelemetry — Semantic Conventions: opentelemetry.io/docs/concepts/semantic-conventions
• OpenTelemetry Specification — Metrics API: opentelemetry.io/docs/specs/otel/metrics/api
• OpenTelemetry Specification — Metrics Data Model: opentelemetry.io/docs/specs/otel/metrics/data-model
• OpenTelemetry Specification — OpenTelemetry Logging: opentelemetry.io/docs/specs/otel/logs
• OpenTelemetry Specification — Tracing API: opentelemetry.io/docs/specs/otel/trace/api
• OpenTelemetry — Context propagation: opentelemetry.io/docs/concepts/context-propagation
• W3C — Trace Context: w3.org/TR/trace-context
• OpenTelemetry — Collector: opentelemetry.io/docs/collector
• OpenTelemetry — Sampling: opentelemetry.io/docs/concepts/sampling
• OpenTelemetry Collector Contrib — Tail Sampling Processor: github.com/open-telemetry/opentelemetry-collector-contrib
• Google — SRE: Monitoring Distributed Systems: sre.google/sre-book/monitoring-distributed-systems
• Google — The Site Reliability Workbook: Alerting on SLOs: sre.google/workbook/alerting-on-slos
• Google — SRE: Being On-Call: sre.google/sre-book/being-on-call
• Prometheus — Metric and label naming: prometheus.io/docs/practices/naming
• Prometheus — Alerting: prometheus.io/docs/practices/alerting
• Brendan Gregg — The USE Method: brendangregg.com/usemethod.html
• Grafana Labs and Tom Wilkie — The RED Method: grafana.com/blog/the-red-method

Jorel del Portal

Jorel del Portal is a systems engineer specialized in architecture, integration, resilience and observability of critical platforms. He designs systems, builds products and documents real engineering decisions.

About LinkedIn YouTube X TikTok Instagram