Enterprise observability with AI

Automate diagnosis without delegating the truth.

Enterprise observability with AI

A platform in the green that was not delivering service

I operated an enterprise platform that processed transfers and moved information between different systems. It was not an isolated server: it was a chain of applications, scheduled jobs, queues, files and external dependencies.

One day the technical signals said everything was fine. Servers available, main processes running, dashboard clear of critical errors. Infrastructure in the green.

But the operations were not arriving.

One of the components responsible for consuming them had stopped working correctly. The process showed as started, but the queue kept growing and operations were not reaching their destination within the expected time. The operational window was missed. The system was technically available and functionally stopped.

It took us several hours to identify it. Not for lack of metrics, but because the ones we had were focused on infrastructure health rather than on business-flow compliance. The user area noticed before we did that operations were not arriving: we saw available components; the business saw a service that was not fulfilling its purpose.

What surfaced it was a question from the user area. Then came the manual checks on queues, pending files, processing times and last recorded movements. Someone who knew the platform's normal behavior immediately spotted that the number of pending items did not match the hour or the expected volume. The knowledge existed, but it lived in a few people's experience and not inside the monitoring system.

The technical details of this case have been simplified to preserve confidentiality. The operational pattern and the lesson are real.

What I changed afterwards was not adding another dashboard. It was changing the question.

The change of question
Before  Is the component running?

After   Is the service processing what it should,
        at the expected rate, and within the
        committed window?

Out of that change came controls that did not exist before: queue depth, age of the oldest pending item, ingress rate versus consumption rate, time since the last completed operation, processing-window compliance, comparison against normal behavior, alerts for running processes with no real movement, and end-to-end validation through synthetic operations.

A runbook also appeared, to tell apart three conditions that until then had looked identical:

Observability must not prove that components are switched on. It must prove that the business purpose is being met.

Short on time? Read the executive version (6 min).

From isolated metrics to enterprise observability

The four golden signals Google popularized for monitoring services —latency, traffic, errors and saturation— are a useful baseline, but on their own they do not describe whether a business process is fulfilling its purpose. A mature strategy connects several levels:

Level Question it must answer Examples
Infrastructure Are the base resources available? CPU, memory, disk, network, processes, operating system
Platform Can the internal components operate? Queues, pools, threads, sessions, certificates, storage
Service Does technical capacity meet the expected level? Availability, latency, errors, saturation, SLOs
Business flow Is the process moving end to end? Completed operations, files processed, reconciliations, pending items
Experience Does the business receive the expected result? End-to-end time, rejections, abandonment, retries
Automation and AI Is the diagnostic layer reliable and efficient? Tokens, latency, tools executed, evidence, cost per analysis

The common mistake is stopping at the first level. A server can look healthy while a queue is not being consumed. A database can answer queries while an application exhausts its connection pool. A service can stay up while hundreds of operations pile up outside their processing window.

Enterprise observability appears when these signals are correlated with the system's purpose. Knowing that a process is running is not enough: you need to know whether it is processing, at what rate, with what error rate, which dependencies it uses, and what impact its degradation would have.

The right place for AI

AI should not compute disk utilization, verify that a process exists, check certificate validity, or determine whether a transfer completed. Deterministic software solves that: scripts, SQL queries, APIs, rules, thresholds, synthetic tests and monitoring platforms.

AI adds value in three specific functions.

1. Assisted selection

In a predictable flow, the orchestrator runs an explicit sequence of controls defined by the catalog: AI does not freely decide what to do. In an open investigation it may propose an additional check — reviewing a queue's consumption rate, comparing latency before and after a deployment — but that proposal goes through the policy engine, which validates parameters, permissions, scope and risk before any tool runs.

2. Interpretation

It analyzes structured results and relates them to current documentation, prior incidents, dependencies and authorized procedures.

3. Communication

It turns evidence into a clear, prioritized, actionable explanation for different audiences: operator, specialist, service owner or user area.

Current models can use function calling to interact with external tools through defined contracts. The model requests execution, but it is the application that validates the parameters, enforces permissions and actually performs the action. The correct division of responsibility is this:

Division of responsibility
AI proposes a check when the flow allows it.
The policy engine decides whether it is authorized.
The orchestrator runs the flow.
Code obtains the evidence.
Rules compute verifiable states.
Retrieval supplies versioned context.
AI interprets and communicates.
The validator checks the response.
The specialist approves critical decisions.

Back to the opening incident. With AI wired to reliable controls, the system could have run the health triages, checked queue depth, compared ingress and consumption rates, queried recent errors and retrieved the applicable runbook, to conclude something like this:

Conclusion with evidence
Main processes are running, but the queue keeps growing and no
completed operations have been recorded in the last 40 minutes.
The evidence confirms functional degradation, though it does not
yet allow the root cause to be determined.

That would have significantly cut the time spent gathering information. But if the AI had received only the surface-level dashboard, it would have produced a convincing and wrong explanation, faster: "the platform is operational because servers and processes report a healthy state".

An AI wired to incomplete signals only interprets an incomplete view of the system faster.

From the isolated prompt to an enterprise capability

One of the most frequent mistakes when introducing AI into operations is handing users a text box and expecting them to know how to write the right prompt. That approach makes everyone describe the incident differently, leaves the model with incomplete information, makes permissions ambiguous and makes token consumption depend on how much text the user types. It also makes results hard to compare, quality hard to measure and auditing hard to keep consistent.

In an enterprise environment the user should not need to know the internal prompt, the model, the observability tool, the specific script or the format each integration expects. They should consume a previously designed operational capability: diagnose service degradation, analyze queue growth, verify a platform's functional health, compare behavior before and after a deployment, review certificate expirations, evaluate processing-window compliance or prepare the evidence for an incident.

The interface asks for controlled parameters:

Request
Capability  : Processing delay diagnosis
System      : Transfer platform
Environment : Production
Period      : Last hour
Flow        : Outbound transfers
Context     : Optional field

The user may add context, but that text does not replace the structured parameters. The interaction stops being "write a good prompt and hope the AI understands the problem" and becomes "select a capability that runs a governed, measurable, repeatable process".

The user should be an expert in their business or their platform. Prompt engineering belongs inside the solution.

Reference technical architecture

The solution can be organized into seven operational layers plus one transversal control for model access:

Reference architecture — 7 layers + transversal control

01 Consumption channel User interaction.Web portal, internal application, corporate messaging client, chatbot, alert or ITSM ticket.
02 API and security Identity, authorization, quotas and traceability.API gateway, OAuth 2.0, OpenID Connect, corporate identity provider, IAM, RBAC.
03 Orchestration Run the flow and control state.Java with Spring Boot, C# with .NET, Python with FastAPI, Node.js, workflow engines.
04 Deterministic tooling Obtain verifiable technical evidence.Shell or PowerShell scripts, Python, SQL, platform APIs, configuration automation.
05 Observability Collect metrics, logs, traces and events.OpenTelemetry plus whatever metrics, logging and APM platform the organization already runs.
06 Enterprise knowledge Retrieve current context.Relational store, search engine, vector database, versioned document repository.
07 Artificial intelligence Classify, correlate, interpret and communicate.Small models, LLMs with tool calling, embeddings and RAG.
Model Gateway Transversal control: govern model access.Routing, data redaction, budget, caching, auditing and fallback.
Product selection depends on each organization's ecosystem. The vendor is not the point. The separation of responsibilities is.

Two zones of uncertainty

A nuance that often gets missed: uncertainty does not live only inside the LLM. Semantic retrieval is probabilistic too. A vector search can return a runbook that looks similar but belongs to another environment, an earlier version, or an architecture that is no longer current. So it pays to separate the two zones explicitly:

Zones of the system

Verifiable zone Can be proven

Identity and permissions · tool execution · metrics and queries · rules and thresholds · auditing · schema validation.

Controlled probabilistic zone Must be bounded and exposed

Semantic classification · document retrieval · hypothesis prioritization · interpretation and drafting.

Both zones must be traceable. The second must additionally expose its sources, versions, scores and limits.

The Model Gateway: governing AI consumption

In an enterprise architecture, applications should not connect directly to each model provider. The Model Gateway — or AI control plane — establishes a central boundary between internal systems and the models, and concentrates the decisions each team would otherwise reimplement on its own:

Its deeper benefit is that it lets you swap models without redesigning the operational capability. The model becomes replaceable; the contracts and the evidence remain.

The capability catalog

The catalog holds previously designed operational functions. Each capability declares its scope, version, tools, permissions, budget and approval policy — and, where relevant, how many additional checks the AI may propose:

YAML
capability:
  id: diagnose_service_degradation
  name: Diagnose service degradation
  version: 2.2

  allowed_environments:
    - development
    - testing
    - production

  required_roles:
    - operator
    - platform_engineer

  deterministic_flow:
    - get_service_metrics
    - get_error_summary
    - get_dependency_status
    - get_recent_changes
    - run_synthetic_test

  optional_ai_proposals:
    enabled: true
    maximum_additional_checks: 2
    require_policy_validation: true

  ai_profile:
    model_tier: medium
    rag_enabled: true
    maximum_context_tokens: 12000

  approval:
    read_operations: automatic
    write_operations: human_required

This turns AI usage into a versioned, governed capability. The user does not freely choose which commands to run: they select an authorized function and the system determines the corresponding flow. The identity layer validates on every request the role, the authorized systems and environments, the permitted tools, and whether the case requires human approval. AI should never decide on its own what permissions a user has.

The deterministic tooling layer

Tools are narrow-scope functions that perform one concrete task:

Tool contracts
get_cpu_metrics()            get_queue_depth()
get_memory_metrics()         get_consumer_rate()
get_disk_growth()            get_database_connections()
get_process_status()         get_recent_deployments()
get_certificate_expiration() get_error_summary()
run_connectivity_test()      run_synthetic_transaction()

Each tool must define its allowed parameters, data types, timeout, authorized systems, risk level, the information it returns, the data it must mask, its retry policy, its audit evidence, its owner, its version and its maintenance criteria.

A tool should not accept an arbitrary command generated by the model. Instead of exposing execute_command("whatever the AI generated"), you expose closed contracts:

Read vs. write
get_disk_usage(server_id, filesystem)
restart_service(service_id, environment, change_ticket)

The first is read-only. The second needs additional permissions, operational-window validation and human approval or a previously authorized automatic policy.

The result should not be free text either:

JSON
{
  "check": "queue_health",
  "target": "service-a",
  "status": "degraded",
  "value": 1842,
  "threshold": 500,
  "timestamp": "2026-07-19T15:30:00-05:00",
  "evidence": "queue-main",
  "execution_status": "success",
  "tool_version": "2.3.1"
}

The execution_status field is the easiest to forget and the most important. Without it you cannot distinguish between four situations that demand different responses: the system is failing, the control could not run, the source did not answer, or the result was partial. A diagnosis that confuses "it is broken" with "I could not measure it" is worse than no diagnosis.

Normalizing the evidence

Tools return different formats. Before calling the model, a normalization layer must unify source, value, baseline, threshold, status and validity — and assign each piece a stable identifier that conclusions can cite:

JSON
{
  "diagnostic_id": "DGN-2026-0719-0042",
  "system": "transfer-platform",
  "environment": "production",
  "time_window": {
    "start": "2026-07-19T15:00:00-05:00",
    "end":   "2026-07-19T15:30:00-05:00"
  },
  "evidence": [
    {
      "evidence_id": "metric.queue_depth.01",
      "source": "metrics_backend",
      "type": "metric",
      "name": "queue_depth",
      "value": 1842,
      "baseline": 320,
      "threshold": 500,
      "status": "critical",
      "execution_status": "success"
    },
    {
      "evidence_id": "check.consumer_process.02",
      "source": "application_script",
      "type": "process_check",
      "name": "consumer_process",
      "value": "running",
      "status": "healthy",
      "execution_status": "success"
    },
    {
      "evidence_id": "metric.consumer_rate.03",
      "source": "metrics_backend",
      "type": "metric",
      "name": "consumer_rate_per_minute",
      "value": 68,
      "baseline": 120,
      "threshold": 90,
      "status": "critical",
      "execution_status": "success"
    },
    {
      "evidence_id": "change.deployment.04",
      "source": "deployment_api",
      "type": "change",
      "name": "last_deployment",
      "value": "release-2026.07.19.2",
      "timestamp": "2026-07-19T14:47:00-05:00",
      "execution_status": "success"
    }
  ]
}

This layer reduces volume and strips noise before spending tokens. The AI does not need thousands of log lines when code can first produce the error count by code, the first and last occurrence, the deviation from the previous period, the affected components, the correlation IDs, representative samples and identifiers to fetch detail.

The flow: consumption, analysis and result

From request to result

  1. User → Channel

    Selects capability, system, environment and period. Writes no prompt.

  2. Channel → API gateway

    Sends structured parameters, not free text.

  3. Gateway → Policy engine

    Validates identity, role and environment. Returns the authorized scope.

  4. Orchestrator → Tools

    Runs the deterministic flow declared in the catalog, with bounded timeouts and retries.

  5. Tools → Data

    Query metrics, logs, traces and states. Return normalized evidence with its execution_status.

  6. Orchestrator → Rules

    Computes states and confirmed conditions: thresholds and baseline comparison.

  7. Orchestrator → Retrieval

    Brings only relevant, current documents, with source, version and score.

  8. Model Gateway → AI

    Selects the authorized model, caps tokens, redacts sensitive data and records the call.

  9. Validator

    Checks evidence, schema and authorizations. Approves or rejects before anything is shown.

  10. Channel → User

    Delivers the structured result: diagnosis, cited evidence and next steps.

The main flow is explicit and every step writes to audit and telemetry. If the model proposes an additional check, the proposal returns to the policy engine, which decides whether it belongs to the catalog, whether the user has permission, and whether the risk is acceptable.

Example: a processing delay

A user notices transfers are taking longer than expected. They do not type "analyze why the system is slow": they select the capability, the system, the environment, the period and the flow.

1. Validation

The platform confirms the user has read access to that system and environment.

2. Deterministic execution

The orchestrator queries process status, queue depth, ingress rate, consumption rate, disk space, database latency, recent errors, external dependencies, deployed changes and a synthetic test.

3. Rules

JSON
{
  "overall_status": "degraded",
  "confirmed_conditions": [
    "queue_depth_above_threshold",
    "consumer_rate_below_baseline"
  ],
  "healthy_conditions": [
    "process_running",
    "database_available",
    "disk_capacity_available"
  ]
}

The degraded state is not invented by the model. It comes from configured rules.

4. Knowledge retrieval

The engine retrieves the current runbook for queue growth, similar incidents, the flow architecture, the consumer's dependencies and the changes applied during the period. Each fragment must carry version, owner, environment and validity date.

5. Interpretation

The AI receives only normalized evidence, rule results, retrieved context, the request scope and the expected schema. It receives no full logs and no isolated screenshots.

6. Validation

The system checks that every finding cites a valid evidence_id, that the metrics exist, that timestamps belong to the period, that recommendations are permitted, that hypotheses are not presented as facts, that retrieved documents are current and from the right environment, and that the response satisfies the schema.

7. Result

JSON
{
  "status": "degraded",
  "executive_summary": "The flow remains available, but its consumption capacity is below baseline.",
  "confirmed_findings": [
    {
      "finding": "The queue grew from 320 to 1842 items.",
      "evidence_id": "metric.queue_depth.01"
    },
    {
      "finding": "Consumer rate dropped 43% versus baseline.",
      "evidence_id": "metric.consumer_rate.03"
    },
    {
      "finding": "The consumer process remains running.",
      "evidence_id": "check.consumer_process.02"
    }
  ],
  "hypotheses": [
    {
      "hypothesis": "The consumer may be processing each operation with higher latency.",
      "confidence": "medium",
      "required_validation": "Compare internal per-stage duration."
    }
  ],
  "missing_evidence": [
    "Internal per-stage latency before and after the last deployment."
  ],
  "recommended_next_steps": [
    "Analyze per-operation duration over the last 60 minutes.",
    "Compare behavior before and after the deployment.",
    "Review calls to the external validation dependency."
  ],
  "automatic_actions_executed": [],
  "human_approval_required": false
}

The values are illustrative, not measured. What matters is that the AI did not arbitrarily determine the state: it interpreted conditions computed through verifiable controls, and explicitly declared what evidence it was missing.

The prompt as an internal component

The prompt still exists, but it stops being the user's responsibility. The system builds a versioned instruction:

System instruction
Analyze exclusively the evidence provided.

Rules:
1. Do not invent metrics, systems, dates or dependencies.
2. Distinguish confirmed facts, correlations and hypotheses.
3. Every finding must include a valid evidence_id.
4. Do not declare a root cause when the evidence only shows correlation.
5. Explicitly declare the missing evidence.
6. Propose only checks permitted by the catalog.
7. Do not propose write actions without approval.
8. Return the response using the defined schema.

This instruction can be versioned, tested, compared across models and audited. Quality stops depending on each operator's prompt-writing skill.

Model strategy by complexity

Not every request should consume the largest model. A cheap classifier picks the route before you spend advanced capacity:

Routing by complexity

Simple Rules or small model

Process status, available space, certificates nearing expiration, alert classification.

Intermediate LLM with a fixed flow

Metric correlation, period comparison, runbook selection, explaining a degradation with sufficient evidence.

Complex Advanced LLM with retrieval

Distributed incident, multiple services and dependencies, correlation of changes and traces, prioritized hypotheses.

Critical Analysis + human approval

Rollback, production restart, blocking an integration, capacity change, isolating a component.

At the critical tier AI may prepare the analysis, but the action stays under human control or under a previously approved automatic policy.

The enterprise rule should be: do not use autonomy where an explicit sequence of steps can solve the problem.

Which kind of AI should you use?

Not every problem requires an autonomous agent or the largest model available.

Small models or classifiers

Suitable for classifying alerts, recognizing incident categories, prioritizing requests, extracting fields and selecting a known procedure. Fast, cheap and sufficient when the decision space is well bounded.

Embeddings and semantic retrieval

Useful for locating runbooks, documentation, configurations, postmortems and similar incidents. Here the primary function is not writing but finding relevant evidence inside authorized knowledge — remembering that this layer lives in the probabilistic zone.

LLM with RAG

Appropriate when you need to interpret technical evidence and contrast it with enterprise documentation. RAG reduces reliance on the model's internal knowledge, but does not eliminate the risks: OWASP warns that RAG and fine-tuning can improve relevance and still not fully resolve vulnerabilities such as prompt injection.

LLM with tool calling

This is the right fit for the triage described above. The model can select a function, fill in parameters and process the response. The function remains conventional code subject to authentication, authorization, validation and auditing.

Agentic AI

Reserve it for open-ended investigations requiring multiple steps, several sources and intermediate decisions. When the flow is predictable, a non-agentic solution is usually simpler, more controllable and cheaper. Google Cloud's architectural guidance distinguishes agents —suited to open-ended problems and complex flows— from assistive, RAG or single-call solutions, which are enough for predictable tasks.

How to contain hallucination risk

There is no instruction that guarantees a model will never be wrong. OWASP defines LLM-generated misinformation as false or misleading information that can appear credible, and recommends verified sources, automated validation and human oversight in critical contexts.

The architectural goal is not to promise the absence of errors. It is to prevent an incorrect response from becoming an untraceable conclusion or an unauthorized operational action.

Mandatory evidence

Every conclusion must cite concrete metrics, tools, periods and documents. When the information is not enough, the correct answer is: this cannot be determined with the available evidence.

Structured output

The response should conform to a defined schema, with correlations as their own category — distinct from facts and from hypotheses:

JSON
{
  "observed_status": "degraded",
  "confirmed_findings": [],
  "correlations": [],
  "hypotheses": [],
  "missing_evidence": [],
  "recommended_next_checks": [],
  "confidence": "medium"
}

Structured output mechanisms let you require fields, types and enumerations. A schema does not guarantee the content is true, but it reduces ambiguity and makes automated validation easier.

Separate levels of assertion

A hypothesis cannot be presented as a root cause.

Restricted tooling

The model can only request functions included in the catalog, with read and write kept separate. Destructive actions require human approval, authorized windows and change control.

Versioned documentation

Runbooks used by semantic retrieval must declare version, owner, environment, validity and last-updated date. Retrieving an obsolete document with high similarity is still retrieving wrong information.

Post-hoc validation in code

Critical claims can be checked: that the cited metric exists, that the timestamp belongs to the period, that the system is within scope, that the document matches the environment, that the recommendation uses an authorized tool, and that every proposed cause has associated evidence.

Evaluation before production

The solution must be tested against historical incidents, synthetic scenarios, downed sources, partial results, cases with insufficient evidence, obsolete or contradictory documents, and attempts to request unauthorized actions. Measuring whether the answer sounds convincing is not enough. The NIST AI RMF proposes managing AI risk across the whole lifecycle, building trustworthiness criteria into design, development, use and evaluation.

Tokens are an architectural decision

Token consumption is not only a financial problem. Sending too much context increases latency, noise, contradictions, exposure of unnecessary data, the surface for malicious instructions, and the difficulty of reconstructing which evidence the model used.

I would not send gigabytes of logs to an LLM and ask what happened. Deterministic software must filter, aggregate and structure first:

As a provider example, OpenAI documents reductions of up to 80% in time to first token and up to 90% in input token cost for requests compatible with its prompt caching. Real results depend on the model, the provider and how effectively context repeats.

Efficiency should not be measured as cost per million tokens, but as AI cost per correctly diagnosed incident.

AI must be observable too

Adding AI to operations creates another system that needs metrics, logs and traces. At minimum I would measure total analysis latency and each tool's duration, the number of model calls, input, output and cached tokens, cost per diagnostic, the percentage of responses with sufficient evidence, tools proposed, approved and rejected, errors and partial results, human escalations, hypotheses later confirmed, retrieved documents and their version, actions blocked by policy, and cases resolved without further intervention.

Modern platforms are already extending logs, metrics and traces to agent behavior: OpenTelemetry's semantic conventions for generative AI normalize the agent invocation span, tool execution spans, and token and duration metrics.

This makes it possible to answer real questions: is AI reducing triage time? On which incidents does it add value? Which tools fail most? What does each useful case cost? Which hypotheses tend to be confirmed? When should it abstain? Is it reducing work, or only generating more text?

Without that information, an AI observability solution becomes, paradoxically, one more unobservable component.

Where to start (and when not to)

Everything above describes the destination, not the first step. If I had to build this tomorrow, I would not start with a chatbot, with RAG, or by picking a model. I would start at layer 04: the deterministic tooling.

Evidence first, AI second

That layer has an advantage no other one has: it pays off even if the AI never arrives. A triage script, a reliable query or a synthetic test already cut operational time, standardize checks, remove manual steps, formalize tribal knowledge and produce auditable evidence.

But I would not build isolated scripts that print different text depending on who wrote them. My order would be:

  1. Define the operational questions I need to answer.
  2. Define a common evidence contract.
  3. Build the deterministic tools.
  4. Apply rules, thresholds and baselines.
  5. Integrate the flow into an orchestrator.
  6. Add retrieval of current knowledge.
  7. Introduce AI for interpretation and communication.
  8. Enable assisted selection only where a real need exists.

AI should not be the foundation. It should be an acceleration layer over a foundation that can stand on its own.

What is realistic in a quarter, and what at a year

Three months is enough for a first production capability if the scope is controlled: one critical platform and one or two recurring incident types. In that window you can deliver five to ten deterministic controls with a common result contract, integration with existing metrics and APIs, a triage runnable from a simple interface, rules to classify states, retrieval of one or two current runbooks, interpretation by a model, response validation, and auditing of executions, tokens and results — all restricted to reads.

What I would not attempt yet: full autonomy, automatic restarts, rollbacks, coverage of the whole organization, unbounded investigation, integration with all corporate knowledge, or automatic root-cause declarations.

The first quarter should prove exactly one thing: that triage time can be reduced using reliable evidence and a repeatable flow.

At twelve months a more mature platform is within reach: more capabilities and systems, permissions per environment and action, versioned knowledge, baseline comparison, continuous evaluation, model routing and integration with the ticketing tool. Only then does it make sense to propose low-risk, reversible, previously authorized remediations — retry a synthetic test, escalate an incident, open a ticket with the evidence attached. The boundary does not change: autonomy is not granted because the technology can do it, but because the organization can govern it.

The trap layer

The trap layer is not the model. It is the combination of orchestration, normalization, tool governance and lifecycle management of the controls. On the diagram it looks simple: a function runs, returns a result and the AI explains it. In production the hard questions show up:

The model is relatively interchangeable. Reliable integration with real systems is not. A catalog of twenty scripts does not amount to an enterprise capability either: the capability exists when each control has an owner, a version, a contract, permissions, a timeout, evidence, success criteria, error handling, auditing and maintenance.

The hard part is not getting the AI to call a function. It is guaranteeing that the function is still reliable six months later.

When it is not worth it

Not every team needs this. It can be over-engineering when several conditions coincide: few systems, a simple architecture, infrequent and easily reproducible incidents, a small stable team, low dependence on specific specialists, short up-to-date runbooks, limited operational impact and an acceptable resolution time. In that scenario you do not need AI: you need better controls, scripts and documentation.

Nor should the decision rest on "the company needs to use AI". That is a bad architectural reason. I would use this relation as the criterion:

Decision criterion
Potential value = frequency × complexity × impact × repetitive manual work

Criticality can break any frequency threshold: a system can have few incidents a year and still justify advanced controls if each interruption compromises critical operations or regulatory compliance. I would also look at technical hours consumed, how many people get pulled in, how long it takes to identify the affected component, how often the same checks are repeated, and the risk of a wrong diagnosis. When only one person knows what to look at, there is already an opportunity to formalize — but that does not mean the first solution should be AI.

The one capability I would build

If I could build only one: automated end-to-end degradation triage. I would not start with postmortem generation, document lookup or alert summarization: those are useful, but they happen afterwards or work on information someone already gathered.

I would pick it not because it replaces analysis, but because it removes the repetitive part: opening different tools, running the same queries, lining up timestamps, comparing metrics and rebuilding context. My time should go to interpreting anomalous behavior and deciding, not to collecting by hand what a system can gather consistently.

The expected answer would not be "the system may be experiencing degradation", but a bounded conclusion: what is available, what degraded and by how much, what changed nearby in time, what is not yet proven, and what the next check is. The AI would not have resolved the incident for me. It would have removed a significant part of the collection work and handed me the problem properly bounded.

Conclusion

The most solid enterprise use of artificial intelligence is not replacing scripts, metrics, rules or procedures. It is connecting them without moving the operational core outside what is verifiable, and leaving approval of critical decisions where it always belonged: with the specialist.

AI delivers speed precisely because it does not need to replace what already works. Its value disappears the moment it is allowed to invent system state, execute actions without limits, or replace checks that conventional software performs more precisely.

The question, then, is not whether an organization already uses AI in operations. It is whether that AI is connected to verifiable evidence, or only to a more sophisticated version of the same incomplete dashboard.

Automate how evidence is obtained. Accelerate how it is interpreted. Never delegate the truth.

Frequently asked questions

Should AI determine whether a system is healthy?

No. State must be computed with deterministic tools, rules, thresholds and verifiable evidence. AI interprets and communicates those results.

Can AI decide which tool to run?

In known flows it does not need to: the orchestrator follows an explicit sequence. In open investigations it may propose additional checks, but the policy engine validates permissions, parameters and risk before they run.

Does RAG eliminate hallucination risk?

No. It reduces reliance on the model's internal knowledge, but it can also retrieve incorrect, outdated or wrong-environment documentation. Sources must be versioned and validated.

When is agentic AI appropriate?

In open-ended investigations requiring several steps and intermediate decisions. When the flow is known and repeatable, a deterministic sequence is simpler, more controllable and cheaper.

Why are tokens an architectural decision?

Because unnecessary context increases cost, latency, noise, information exposure and auditing difficulty. Evidence must be filtered and structured before calling the model.

References

Jorel del Portal

Jorel del Portal

Jorel del Portal is a systems engineer specialized in architecture, integration, resilience and observability of critical platforms. He designs systems, builds products and documents real engineering decisions.