How to investigate an incident with metrics, logs, and distributed traces

An alert reports that latency has increased. The service is still accepting traffic. The team opens five dashboards, searches for ERROR across several log indexes, and proposes restarting two components. No one has yet defined which operation, version, region, or dependency is affected.

This pattern is common because activity is easy to confuse with progress during an incident. Each engineer finds a different signal, every chart appears relevant, and every change seems capable of restoring the service. The risk is that the team changes several variables at once, destroys evidence, and temporarily reduces the symptom without understanding what produced it.

Effective investigation follows a different logic: metrics show the impact and delimit the affected population; traces identify where time is consumed or an error appears; logs provide event-level context; and diagnosis emerges when those signals are organized into a hypothesis that can be tested or disproved. OpenTelemetry defines metrics, traces, and logs as distinct signals, while context propagation makes it possible to correlate them across processes and services.

This article develops an operational method in seven movements:

1. Confirm the impact.
2. Scope the incident.
3. Build a timeline.
4. Correlate metrics, traces, and logs.
5. Formulate a testable hypothesis.
6. Mitigate and verify recovery.
7. Convert the findings into concrete improvements.

This is not a guide to observability tools. It is a sequence for making decisions under pressure.

Before investigating: protect the operation

The immediate objective of incident response is not to produce a perfect explanation. It is to limit damage safely.

The order of priorities depends on the system, but it should usually consider:

1. Safety of people and assets.
2. Data integrity and confidentiality.
3. Actual impact on users and critical processes.
4. Containment of the failure.
5. Operational communication.
6. Preservation of enough evidence to continue diagnosis.

Google SRE distinguishes incident management from technical investigation: an explicit response structure coordinates actions, communication, and recovery without turning every participant into an investigator at the same time.

The operational decision can be summarized as follows:

The inverse, however, is not valid:

A restart can release memory, drain or recreate queues, rebuild connections, or remove a transient lock condition. The service may improve while the causal mechanism remains intact. If the team records “the restart solved the incident” as the root cause, it converts a temporal observation into an unproven explanation.

Initial decision: investigate, mitigate, or do both

Step 1: detect and confirm impact

An alert is not yet a confirmed incident. It is an automated claim about an observed condition. It may represent real impact, an internal degradation with no user-visible effect, delayed telemetry, a poorly calibrated threshold, or a population that is irrelevant to the service objective.

The first obligation is to convert the alert into an operational description:

• What changed.
• When it started.
• Which population is affected.
• The magnitude of the change.
• Which user experience or business process is affected.
• Whether the behavior is sustained, intermittent, or already over.

Start with Rate, Errors, and Duration

For request-oriented services, RED organizes initial triage around three signals:

• Rate: how many requests or spans are processed per unit of time.
• Errors: how many end in a condition considered unsuccessful.
• Duration: how long they take, ideally represented as a distribution rather than only as an average.

RED is an effective entry point, not a complete diagnosis. It must be complemented with saturation, queues, pools, and dependencies. Google SRE summarizes the minimum coverage for a user-facing system as latency, traffic, errors, and saturation.

Do not mix availability, success, and latency

During an incident, these dimensions must be recorded separately:

The exact meaning depends on the SLI. An operation can be technically available and functionally degraded. It can also maintain an acceptable success rate while a queue grows and prepares the next failure.

The distribution matters because an average can hide a slow tail. Google SRE recommends observing percentiles to distinguish which fraction of requests is degraded instead of relying only on the mean. Detailed calculation and interpretation of latency percentiles belong in the specialized article.

Confirm from more than one perspective

A single source can mislead because of instrumentation errors, delay, or scope. Confirmation should combine, when available:

• Internal service telemetry.
• External synthetic monitoring.
• Client-side or channel-side metrics.
• SLO consumption or burn rate.
• Functional evidence: rejected transactions, backlogs, or business processing times.

The question is not “Is the dashboard red?” It is:

Step 2: define the scope

“The platform is slow” is not an investigable scope.

A useful description looks like this:

Scoping reduces the number of possible components, queries, and changes. Segment by dimensions that can actually alter behavior:

• Operation or endpoint.
• Service and dependency.
• Region, zone, or cluster.
• Deployed version.
• Channel or client type.
• Tenant, product, or plan.
• Payload type and size.
• Authentication method.
• Synchronous or asynchronous path.
• Partition, shard, or node.

Cardinality must be designed before the incident

Not every dimension should become a metric label. User, order, or trace identifiers create extreme cardinality and generally belong in logs or traces, not in time-series dimensions. Metric dimensions must enable segmentation without making the telemetry backend operationally unviable.

OpenTelemetry uses resource attributes and conventions such as service.name, service.namespace, and deployment.environment.name to describe telemetry origin consistently. This consistency allows responders to compare signals without translating different naming systems during a crisis.

Scoping technique: split the population

For each scope hypothesis, compare two groups:

• Affected versus unaffected.
• New version versus previous version.
• Degraded region versus healthy region.
• Large payload versus small payload.
• Mobile client versus web client.
• Dependency A versus dependency B.

Comparison is more useful than observing an isolated group because it reveals which dimension explains the difference.

Step 3: build a timeline

The timeline is not an administrative document to complete afterward. It is a diagnostic tool during the incident.

It should record four types of information:

1. Observed events: alerts, metric changes, errors, and saturation.
2. System changes: deployments, configuration updates, credential rotations, and dependency failures.
3. Decisions: what was decided and based on which evidence.
4. Results: what changed after each action.

Which time should be recorded

Whenever possible, use a single time zone, preferably UTC, and preserve the original timestamp from the source. In distributed logs, the time at which an event occurred can differ from the time at which the collector observed it. The OpenTelemetry log data model separates Timestamp from ObservedTimestamp precisely to represent both.

Temporal order prioritizes; it does not prove

If retries increase before pool wait time, that sequence makes retries a candidate cause. It does not prove causality. Both changes may depend on a third factor, such as external latency or a concurrency change.

The timeline helps answer:

• What changed first?
• What occurred immediately before impact?
• Which signal reacted after a mitigation?
• Which events merely coincide because of aggregation windows?

Step 4: use metrics to locate the symptom

Metrics describe complete populations and changes over time. They are the appropriate signal for confirming magnitude, trend, distribution, and saturation.

Investigation should move from the user-visible symptom toward the resources that could explain it.

First layer: service experience

Review by operation and affected population:

• Logical request rate.
• Total technical attempts.
• Technical and functional error rates.
• P50, P95, and P99.
• Timeouts observed by client and server.
• SLO consumption.

Separating logical requests from technical attempts is critical. If 220 requests per second generate 88 additional retries, user traffic remains 220 requests/s, but the system executes 308 attempts/s. Mixing those numbers hides amplification.

Second layer: saturation and waiting

Review:

• Queue length and queue age.
• Active and waiting threads.
• Used, free, and maximum connections.
• Requests waiting to acquire a connection.
• Pool wait time.
• CPU, memory, and disk utilization.
• Concurrency limits.
• Backpressure or rejection signals.

Saturation does not require CPU or memory to reach 100 percent. A connection pool, semaphore, partition, lock, or external quota can become the scarce resource while the host appears healthy.

OpenTelemetry defines database-pool metrics such as connections by state, pending requests, timeouts, and wait time. As of July 12, 2026, several of these conventions are marked Development, so the names and support available must be validated in the SDK or library actually used.

Third layer: dependencies and resilience mechanisms

Review:

• Latency and errors by dependency.
• Connection and read timeouts.
• Additional retries by operation.
• Circuit Breaker state.
• Bulkhead or rate-limit rejections.
• Activated fallbacks.
• DNS, TLS, or connection-acquisition time when instrumented.

Resilience mechanisms also create load. A retry is not a free abstraction: it is another execution that consumes time, connections, threads, and dependency quota. The design of correctly configured timeouts and the prevention of retry storms belong in the specialized article.

Primary question: what changed first, and in which population?

A useful sequence may look like this:

1. P99 increases for only one operation.
2. Logical traffic remains stable.
3. Technical attempts increase.
4. Pending pool requests grow.
5. Query duration remains stable.

This combination lowers the probability that the database is executing slower queries. It moves the investigation toward pre-query waiting and the mechanism that multiplied attempts.

Do not use an average to close the incident

An average can return to baseline while a residual queue keeps a fraction of requests degraded. Histograms and percentiles expose the distribution, but their interpretation depends on buckets, window, and population. Prometheus documents that histograms and summaries record count and sum, from which an average can be derived; that average does not replace the distribution.

For a deeper treatment of end-to-end allocation and latency distributions, see the latency budget guide.

Step 5: use traces to locate the degraded path

Metrics show that an anomalous population exists. Traces allow the team to inspect individual executions from that population and see how time is distributed across services and operations.

A trace should not be selected at random. It should match the scope already established:

• Affected operation.
• Exact incident window.
• Region or zone.
• Version.
• Slow or failed outcome.
• Relevant payload attributes without sensitive data.

Compare three trace types

Select, when available:

1. Healthy trace: same operation and comparable conditions outside the incident.
2. Slow trace: completes, but exceeds the threshold.
3. Failed trace: ends in an error or timeout.

Compare:

• Critical path.
• Span that dominates duration.
• Uninstrumented gaps.
• Waiting before a dependency call.
• Retries represented as spans or events.
• Serialization and validation.
• Parallel versus sequential work.
• Queues or messaging.
• Status and error attributes.

The longest span is not always the cause

A span may be long because it is waiting for a resource that is already saturated. Duration locates where time manifests, but the cause may be earlier:

• Excessive concurrency.
• A retry that duplicated calls.
• Missing backpressure.
• Lock contention.
• Pool exhaustion.
• A dependency that does not honor cancellation.

The trace reduces the search space. It does not replace a hypothesis.

Propagate context without breaking the trace

W3C Trace Context standardizes the traceparent and tracestate headers. A compatible implementation must at least propagate them to preserve trace continuity across components.

This context allows a trace_id to represent the complete request path and each span_id to identify a specific operation within it. OpenTelemetry uses context propagation as the common mechanism for correlating signals across processes and networks.

Sampling can hide exactly the cases being investigated

With probabilistic head sampling, a slow or failed trace may never be retained. OpenTelemetry notes that head sampling makes a decision without seeing the complete trace and therefore cannot guarantee retention of every error. Tail sampling can decide based on latency, error, attributes, or version, at the cost of more state, complexity, and operational expense.

Before concluding that “there are no slow traces,” validate:

• Sampling rate and policy.
• Whether spans reached the collector.
• Loss caused by queues or export failures.
• Whether the query uses the correct time window and attributes.
• Whether the tail sampler retains errors and high-latency traces.

Step 6: correlate logs without searching blindly

Searching for ERROR across the entire platform usually produces a large amount of noise and little evidence. Logs become more useful after the investigation has an operation, time window, version, dependency, and trace ID.

Minimum fields for an operational log

A useful event should include, when applicable:

• Source timestamp.
• observed_timestamp or ingestion timestamp.
• service.name and version.
• Environment, region, zone, and instance.
• trace_id and span_id.
• Normalized operation or route.
• Dependency.
• Outcome and stable error code.
• Relevant duration.
• Attempt number.
• Non-sensitive functional attributes with controlled cardinality.

OpenTelemetry includes Timestamp, ObservedTimestamp, TraceId, SpanId, severity, resource, and attributes in its stable log data model. It also defines correlation by time, execution context, and source resource.

Example:

{
  "timestamp": "2026-07-12T22:14:08.417Z",
  "observed_timestamp": "2026-07-12T22:14:08.463Z",
  "severity": "WARN",
  "service": "orders-api",
  "service_version": "2026.07.12-3",
  "region": "south-1",
  "trace_id": "4e8f9b87d1354f4f9c9770c7cb8a66d2",
  "span_id": "63a3a6f34dcf1e80",
  "operation": "create-order",
  "dependency": "inventory-db",
  "attempt": 2,
  "pool_wait_ms": 1842,
  "outcome": "timeout",
  "error_code": "POOL_WAIT_TIMEOUT"
}

The example is illustrative. Field names must align with the platform telemetry model and with the conventions supported by the actual instrumentation.

Recommended search order

1. Find the trace_id from a slow or failed trace.
2. Filter by the span window and relevant service.
3. Review events for the specific span using span_id.
4. Expand to other instances only when evidence requires it.
5. Group by stable codes rather than free-form message text.

What to avoid

• Searching by full exception messages that change on each occurrence.
• Using severity as a substitute for impact.
• Assuming the most dramatic log entry is the cause.
• Logging secrets, tokens, complete payloads, or personal data.
• Emitting a log on every iteration of a massive failure and worsening saturation.
• Concluding that no error occurred because no log was found.

An error may appear in a metric or span without producing a log because of sampling, log level, ingestion loss, or missing instrumentation. Absence of evidence in one signal is not evidence of absence in the system.

Step 7: formulate a testable hypothesis

Investigation advances when the team stops listing possibilities and makes a statement that can be disproved.

Recommended structure:

Example:

A useful hypothesis contains a prediction

Do not use a label as an explanation

Statements such as “network issue,” “slow database,” “Kubernetes,” “GC,” or “saturation” describe categories. They do not explain the mechanism.

A hypothesis must specify:

• Which resource or interaction fails.
• Under which condition.
• How it produces the symptom.
• Which observation would support it.
• Which observation would refute it.

Step 8: separate evidence, hypothesis, decision, and result

Statements quickly become mixed in an incident room:

• “The pool is exhausted.”
• “Increase the number of connections.”
• “The database is healthy.”
• “The deployment caused it.”

Without classification, a repeated assumption can become accepted as fact.

Use an explicit table:

Step 9: apply a mitigation

A mitigation aims to reduce impact. Evaluate it by risk, reversibility, speed, integrity, and blast radius.

Options and trade-offs

Change one variable at a time when risk allows it

A severe incident may require combined actions. When several actions are applied, record them precisely and stagger them where possible. Otherwise, recovery cannot be attributed to a specific change.

Define abort criteria

Before the change, define:

• Which metric must improve.
• In which direction and population.
• Which metric must not worsen.
• When to revert the mitigation.
• Who decides whether to continue.

A mitigation without abort criteria is another uncontrolled production experiment.

Step 10: verify recovery

“It is responding again” is not verification.

Recovery must be checked in the same population and with the same signals that confirmed the incident:

• P95 and P99.
• Technical and functional error rates.
• Logical traffic and total attempts.
• Timeouts and retries.
• Queue length and age.
• Saturation of the candidate resource.
• SLO or burn rate.
• Data integrity.
• Backlog accumulated during degradation.

Compare against baseline, not against the worst minute

A P99 reduction from 2.4 s to 1.5 s is an improvement, not recovery, when the baseline was 600 ms.

Define:

• A comparable baseline by time of day, day, and load.
• A normal range, not only a single point.
• An observation window long enough to cover telemetry delay, retry cycles, backlog processing, and traffic variation.

There is no universal observation window. A high-volume service may demonstrate statistical stability in minutes; an hourly batch process must be observed through at least the next relevant cycle.

Verify from outside and inside

Robust recovery combines:

• External signal: user, synthetic check, or SLI.
• Internal signal: queues, pools, and dependencies.
• Individual evidence: healthy traces after mitigation.
• Functional consistency: no operations were lost or duplicated.

Google SRE notes that monitoring should make it possible to compare behavior before and after a change. It also emphasizes telemetry freshness: when data arrives too late, responders may attribute an effect to the wrong action.

Case study: latency caused by connection-pool wait time

The following scenario is constructed to demonstrate the method. Its values are not a recommended configuration or a universal limit.

Scenario

An order-creation API maintains 220 logical requests per second. Its normal P99 is 600 ms. At 22:14 UTC, P99 rises to 2.4 s and the error rate increases from 0.3 percent to 2.1 percent.

CPU remains between 48 and 55 percent. The duration of the main database queries remains close to 85 ms. The time required to acquire a connection, however, increases sharply.

Step A: confirm and scope

The degradation affects:

• Operation: create-order.
• Region: south-1.
• Version: 2026.07.12-3.
• Nodes using a new concurrency configuration.

It does not affect:

• Order reads.
• Region north-1.
• The previous version.

Step B: observe attempts, not only requests

Before the incident:

• Logical requests: 220/s.
• Additional retries: 4.4/s.
• Total attempts: 224.4/s.

During the incident:

• Logical requests: 220/s.
• Additional retries: 88/s.
• Total attempts: 308/s.

Calculation:

total_attempts = logical_requests + additional_retries
total_attempts = 220 + 88 = 308 attempts/s
amplification = 308 / 220 = 1.40

User traffic did not increase. Technical load increased by 40 percent.

Step C: review capacity and waiting

Observed assumptions:

• Maximum pool size: 100 connections.
• Mean connection hold time: 300 ms.
• Simplified theoretical capacity: 100 / 0.3 = 333 acquisitions/s.
• Attempts during the incident: 308/s.

This calculation does not represent safe capacity because it ignores variance, overhead, long transactions, and the distribution of hold times. It shows that the system is operating close to a theoretical limit. Near saturation, small variations create queues and wait time grows before CPU or query duration appears abnormal.

Observed metrics:

Step D: compare traces

Healthy trace:

create-order                  540 ms
├─ validate-request            35 ms
├─ acquire-db-connection       18 ms
├─ insert-order               102 ms
├─ reserve-inventory          210 ms
└─ publish-event               75 ms

Slow trace:

create-order                2,320 ms
├─ validate-request            37 ms
├─ acquire-db-connection    1,760 ms
├─ insert-order               108 ms
├─ reserve-inventory          225 ms
└─ publish-event               82 ms

The query does not dominate the difference. The wait occurs before query execution.

Step E: correlate logs

Logs associated with slow trace IDs show:

• attempt=2 more frequently.
• POOL_WAIT_TIMEOUT.
• The same query version.
• No equivalent increase in db_query_timeout.

Step F: formulate the hypothesis

Prediction:

Step G: mitigate

Staged actions:

1. Disable retries on acquisition timeout through a feature flag.
2. Limit internal concurrency for the affected version.
3. Keep the pool at 100 connections until database headroom is verified.

The pool is not increased automatically because the database could become the next saturated resource.

Step H: verify

After mitigation:

The sequence matters:

1. Retries fall.
2. Total attempts fall.
3. Waiters decrease.
4. API P99 falls.
5. Query duration remains stable.

This result supports the amplification-and-pool-wait hypothesis. The team still needs to prove which specific change increased concurrency and why the retry policy treated a saturation timeout as a recoverable transient failure.

Prevention derived from the case

• Explicit metrics for pending requests and pool wait time.
• Separate counters for logical requests, attempts, and additional retries.
• Retry policy by failure type and idempotency.
• Concurrency limit before the pool.
• Load tests that include variance and partial failures.
• Saturation alerting before P99 exceeds the SLO.
• Traces that represent connection acquisition and every attempt.
• A runbook that explains how to reduce load without increasing the pool blindly.

The same mechanism can propagate into cascading failures when timeouts and retries consume resources in upstream services.

What to do when signals contradict one another

A contradiction usually means that the signals observe different populations, clocks, or stages of the request path.

Case 1: metrics show errors, but logs do not

Possible explanations:

• The proxy records the error, but the application does not.
• The process terminates before emitting or exporting the log.
• The logging level filters the event.
• Ingestion lost data.
• The metric counts a client timeout that the server never observes.

Decision: validate the measurement point, look for a trace or independent external signal, and check for loss in the log pipeline.

Case 2: logs show errors, but there is no visible impact

Possible explanations:

• A retry or fallback recovered the error.
• It affects a path with insignificant traffic.
• The application uses ERROR for an expected condition.
• The success metric is aggregated and hides a small population.

Decision: measure frequency, population, and functional outcome before escalating the incident.

Case 3: traces do not show slow cases

Possible explanations:

• Probabilistic sampling.
• Misconfigured tail sampling.
• Querying the wrong operation or version.
• Incomplete spans or broken context propagation.
• Saturated exporter.

Decision: review sampling policy, span loss, and traceparent propagation.

Case 4: client and server report different latency

Possible explanations:

• DNS, connection establishment, TLS, or network time before the request reaches the server.
• Queuing in a gateway or proxy.
• The server sent the response, but the client received it later.
• Population difference: the server excludes aborted requests.

Decision: decompose the path and define exactly where each metric starts and ends.

Method for resolving contradictions

1. Validate the same time window.
2. Validate the same population and labels.
3. Confirm the unit and metric definition.
4. Compare source and ingestion timestamps.
5. Review sampling and telemetry loss.
6. Find an independent signal.
7. Prefer evidence closest to the outcome being explained.

Common mistakes during incidents

Changing several variables without recording them

This prevents attribution of recovery and may introduce a second failure.

Restarting before preserving evidence

A restart removes heaps, connections, queues, locks, and state that could demonstrate the mechanism. Restarting may be necessary, but it must be a conscious decision.

Investigating the entire platform

This increases noise. Scope first by operation, population, version, and dependency.

Using averages to declare recovery

This hides slow tails and extreme user experiences. Review the distribution.

Confusing correlation with causality

Two simultaneous signals can share a cause. A prediction and a test are required.

Automatically blaming the latest deployment

A deployment is a temporal candidate, not a demonstrated cause. It may activate a latent condition or merely coincide with an external change.

Searching logs before scoping

This produces broad, slow queries dominated by irrelevant events.

Treating every retry as recovery

A retry can recover a transient failure or amplify saturation. Measure it as additional load.

Declaring root cause from the mitigation

“Latency fell after increasing the pool” does not prove that the pool was undersized. The change may have hidden a leak, a concurrency increase, or a transaction holding connections too long.

Closing without checking integrity and backlog

Latency can return to baseline while accumulated messages, duplicated transactions, or incomplete processes remain.

For a broader framework for structured incident investigation, the troubleshooting article covers diagnosis beyond the specific correlation of observability signals.

Minimum roles in an incident room

Not everyone should run queries or propose changes at the same time.

A minimum structure includes:

• Incident lead: maintains priority, scope, decisions, and closure criteria.
• Technical investigator: coordinates hypotheses and tests.
• Communicator: updates stakeholders without interrupting investigation.
• Timeline owner: records events, decisions, and results.

In small teams, one person can hold more than one role. What matters is that the functions exist and leadership is not implicit.

The incident-response model described by Google relies on defined functions and standardized communication so the response can scale without losing coordination.

After the incident: document without duplicating the diagnosis

The postmortem should record:

• Impact and affected population.
• Detection.
• Timeline.
• Demonstrated technical cause.
• Contributing factors.
• Mitigation and recovery.
• What accelerated or delayed diagnosis.
• Preventive actions with an owner and due date.

Google SRE defines a postmortem as a record of the incident, impact, mitigation, causes, and follow-up actions. It also recommends a blameless culture that examines system conditions instead of turning the document into an individual sanction.

Actions without an owner tend not to be completed. The SRE Workbook recommends explicit ownership and a single accountable owner supported by collaborators.

This article does not develop the full postmortem discipline in order to avoid cannibalizing the dedicated piece. See how to write a postmortem that prevents recurrence.

Close observability gaps

The output of an incident should not be an indiscriminate instruction to “add more logs.”

The correct question is which decision the team could not make and which signal would have reduced that uncertainty.

Review questions

• Which impact metric was missing?
• Which dimension prevented scoping the population?
• Which span was absent from the critical path?
• Which trace attribute was missing?
• Which log lacked a trace_id, stable code, or version?
• Which clock or timestamp prevented event ordering?
• Which sampling policy discarded the relevant cases?
• Which alert arrived late or lacked context?
• Which dashboard mixed logical requests and attempts?
• Which runbook recommended an unvalidated action?

Convert the gap into a requirement

Weak action item:

Executable action item:

Useful telemetry is not the telemetry that accumulates the most data. It is the telemetry that reduces uncertainty in a decision.

Operational checklist

Detection

• The alert represents real impact or operational risk.
• The primary changed metric has been identified.
• The incident start time has been estimated.
• The affected volume and population have been measured.
• Availability, success, latency, and percentiles have been separated.
• Logical requests, total attempts, and additional retries have been separated.

Scope

• Operation or endpoint.
• Service and dependency.
• Region, zone, or cluster.
• Version.
• Channel, tenant, or payload type when relevant.
• Comparable healthy group.

Timeline

• First alert.
• First confirmed impact.
• Deployments and configuration changes.
• Traffic or dependency variations.
• Mitigations with exact timestamps.
• Result of each action.

Evidence

• RED metrics.
• Saturation, queues, and pools.
• Healthy trace.
• Slow trace.
• Failed trace, if available.
• Logs correlated by trace/span ID.
• Sampling policy validated.
• Clocks and windows aligned.

Hypothesis

• Candidate causal mechanism.
• Supporting evidence.
• Measurable prediction.
• Reversible test or mitigation.
• Condition that would refute the hypothesis.

Action

• Risk and blast radius evaluated.
• Data integrity protected.
• Abort criteria defined.
• One variable changed at a time when risk allows it.
• Decision and owner recorded.

Recovery

• Metrics compared against baseline.
• P95/P99 and error rate normalized.
• Queues and backlogs under control.
• Retries and timeouts normalized.
• SLO or burn rate recovered.
• Integrity and duplicates verified.
• Observation window sufficient for the load pattern.

Learning

• Technical cause and contributing factors separated.
• Mitigation distinguished from cause.
• Observability gaps identified.
• Actions have an owner, due date, and acceptance criterion.
• Postmortem required according to previously defined criteria.

Frequently asked questions

What should be checked first during an incident?

First confirm the impact and the affected population. For request-oriented services, Rate, Errors, and Duration provide an initial triage frame; then review saturation, queues, dependencies, and resilience mechanisms. Searching all platform logs before scoping usually increases noise.

What is the difference between mitigation and root cause?

A mitigation reduces impact. The cause explains the mechanism that produced the incident and the factors that allowed it to escalate. A restart, rollback, or capacity increase can restore service without proving the cause.

How is a Trace ID used?

A trace_id identifies the distributed path of a request. It is used to open the complete trace and filter logs emitted in that context. The span_id narrows the search to a specific operation. Continuity depends on correct context propagation between services.

What should a timeline include?

Alerts, metric changes, deployments, configuration changes, dependency failures, decisions, mitigations, and results. It should be built during the incident, use a common time zone, and separate event time from observation time when ingestion delays exist.

When should an incident be closed?

When impact is controlled, primary signals have returned to an acceptable range, backlog and integrity have been verified, and the observation period covers a sufficient cycle for the load pattern. Closure does not require the postmortem to be complete, but pending risks and actions must be transferred explicitly.

What should a postmortem include?

Impact, detection, timeline, technical cause, contributing factors, mitigation, recovery, lessons, and actions with owners and dates. It should focus on the system and decision conditions rather than blaming individuals.

Conclusion

Investigating an incident does not mean opening every available tool. It means reducing uncertainty in a controlled way.

Metrics answer what changed, by how much, and for whom. Traces show where time is consumed or an error appears along the request path. Logs provide the specific event and state. No individual signal automatically provides the cause.

The operational sequence is:

1. Protect the operation and data integrity.
2. Confirm real impact.
3. Scope the affected population.
4. Build the timeline while the incident is happening.
5. Use metrics to locate the symptom.
6. Use traces to narrow the degraded path.
7. Use logs to obtain precise context.
8. Formulate a hypothesis with a prediction and refutation criterion.
9. Mitigate reversibly.
10. Verify against baseline, SLO, queues, and integrity.

The test of a strong investigation is not the number of dashboards consulted. It is whether every decision can be connected to evidence, whether recovery is measurable, and whether the next incident encounters fewer blind spots than the previous one.

Technical sources

• OpenTelemetry — Signals: opentelemetry.io/docs/concepts/signals
• OpenTelemetry — Context propagation: opentelemetry.io/docs/concepts/context-propagation
• OpenTelemetry — Logging: opentelemetry.io/docs/specs/otel/logs
• OpenTelemetry — Logs Data Model: opentelemetry.io/docs/specs/otel/logs/data-model
• OpenTelemetry — Sampling: opentelemetry.io/docs/concepts/sampling
• OpenTelemetry — Database client metrics: opentelemetry.io/docs/specs/semconv/db/database-metrics
• W3C — Trace Context: w3.org/TR/trace-context
• Google SRE — Monitoring Distributed Systems: sre.google/sre-book/monitoring-distributed-systems
• Google SRE Workbook — Monitoring: sre.google/workbook/monitoring
• Google SRE Workbook — Incident Response: sre.google/workbook/incident-response
• Google SRE — Postmortem Culture: sre.google/sre-book/postmortem-culture
• Google SRE Workbook — Postmortem Culture: sre.google/workbook/postmortem-culture
• Grafana Labs — Choose a RED metric: grafana.com/docs/grafana/latest/explore/simplified-exploration/traces/investigate/choose-red-metric
• Prometheus — Histograms and summaries: prometheus.io/docs/practices/histograms

Jorel del Portal

Systems engineer specialized in enterprise software architecture and high-availability platforms.

LinkedIn YouTube Tiktok Instagram X Threads My website