AI API Monitoring and Observability: Track LLM Calls in Production

Published 2026-08-16 · 2,146 words · 8 min read

AI API monitoring is the practice of capturing, analyzing, and alerting on every LLM API call your application makes — requests, tokens, latency, errors, and cost. Classic API monitoring tracks status codes and response times; LLM observability goes further because every call carries hidden dimensions: token counts that translate directly into money, non-deterministic outputs that vary on retry, and latency that depends on model size and queue depth. Teams that skip LLM observability discover problems the expensive way: a $4,000 monthly bill from a runaway loop, a customer-facing chatbot that silently degrades, or a prompt regression nobody noticed for two weeks. This guide covers the full stack of AI API monitoring — what to measure, how to instrument it, and how to alert on it — with production-ready code you can adapt today.

Why LLM Observability Is Different From Classic API Monitoring

Your existing monitoring stack (Prometheus, Datadog, Grafana) was built for deterministic services. LLM APIs break four of its core assumptions:

• Cost is per-request, not per-minute. A single API call can cost $0.01 or $10.00 depending on model and token count. Traditional infrastructure metrics give you no visibility into spend.
• Outputs are non-deterministic. The same input can return different results — and retrying on error can produce a worse answer than the original failure. Monitoring must capture outputs, not just status codes.
• Latency is multi-phase. Time-to-first-token (TTFT), inter-token time, and total completion time each tell a different story about provider health, model size, and your prompt complexity.
• Quality is a first-class signal. A 200 OK response can still be a hallucination, a refusal, or a broken JSON payload. Error rate alone understates real failures by an order of magnitude.

LLM observability therefore means logging rich structured events per call, aggregating them into dashboards, and alerting on business-impacting signals — not just HTTP codes.

The 7 Metrics Every LLM Application Must Track

Track these per model, per feature, and per customer — a rate limit that's invisible in the aggregate can be crushing for a single tenant.

Structured Request Logging: The Foundation

Every LLM call should emit one structured log line with a shared request_id so you can trace a user's request through your app, the gateway, and the provider. Log JSON, never text — it must be queryable. A minimal Python instrumentation wrapper:

import json, time, uuid, logging
from openai import OpenAI

logger = logging.getLogger("llm.usage")
client = OpenAI(api_key="your-drai-key", base_url="https://api.dr-ai.top/v1")

def llm_call(messages, model="gpt-5-mini", feature="chat", user_id=None):
    request_id = uuid.uuid4().hex[:12]
    started = time.monotonic()
    try:
        resp = client.chat.completions.create(model=model, messages=messages)
        usage = resp.usage
        logger.info(json.dumps({
            "event": "llm_call", "request_id": request_id,
            "feature": feature, "user_id": user_id, "model": model,
            "status": "ok",
            "prompt_tokens": usage.prompt_tokens,
            "completion_tokens": usage.completion_tokens,
            "total_tokens": usage.total_tokens,
            "latency_ms": round((time.monotonic() - started) * 1000),
            "ttft_ms": None,  # capture with streaming
            "cached_tokens": getattr(usage, "prompt_tokens_details", None) or {},
        }))
        return resp
    except Exception as e:
        logger.info(json.dumps({
            "event": "llm_call", "request_id": request_id,
            "feature": feature, "model": model, "status": "error",
            "error_type": type(e).__name__, "error": str(e)[:300],
            "latency_ms": round((time.monotonic() - started) * 1000),
        }))
        raise

Ship this to a log sink (Elasticsearch, Loki, ClickHouse, or a SaaS LLM observability tool) and you have the raw material for every dashboard and alert in this guide.

Token Usage Tracking: Count Every Token, Attribute Every Cost

Token accounting is the heart of AI API monitoring. Providers return usage in the response — always capture all three fields plus cache details:

{
  "usage": {
    "prompt_tokens": 812,
    "completion_tokens": 340,
    "total_tokens": 1152,
    "prompt_tokens_details": {"cached_tokens": 600},
    "completion_tokens_details": {"reasoning_tokens": 120}
  }
}

Three disciplines make token data actionable:

• Attribution tags. Add feature, user_id, tenant, and model to every usage record. Without attribution, you can't answer "which customer is burning our budget?"
• Normalize to cost. Store token counts and compute cost at query time with a pricing table per model. Token counts drift from dollar impact as pricing changes; keep the price map in one place.
• Watch cached vs. fresh. Modern providers return large discounts for cached prompts. If your cached-token ratio drops, your prompts are changing too often — a cheap fix for a real cost leak.

DrAI's gateway does this bookkeeping server-side: every API key gets per-model usage breakdowns in the dashboard, so you get cost attribution without building your own accounting layer.

Latency Breakdown: TTFT, Inter-Token Time, and End-to-End

LLM latency is not one number. Split it into phases — each has a different root cause and fix:

• Time-to-first-token (TTFT): time from request to the first output token. Driven by queue depth, input length, and model size. Small models: 100-400ms; large reasoning models: 1-10s.
• Inter-token time (ITT): time between tokens — the perceived "typing speed." Drives the streaming experience.
• End-to-end: total wall time, including your app, network, and retries.

def stream_llm(messages, model="gpt-5-mini"):
    started = time.monotonic()
    first_token_at = None
    tokens = 0
    for chunk in client.chat.completions.create(
        model=model, messages=messages, stream=True
    ):
        if first_token_at is None:
            first_token_at = time.monotonic()
            logger.info(json.dumps({
                "event": "llm_ttft", "model": model,
                "ttft_ms": round((first_token_at - started) * 1000),
            }))
        delta = chunk.choices[0].delta.content
        if delta:
            tokens += 1
    total = time.monotonic() - started
    logger.info(json.dumps({
        "event": "llm_stream_stats", "model": model,
        "total_ms": round(total * 1000),
        "tokens": tokens,
        "inter_token_ms": round(total * 1000 / max(tokens, 1)),
    }))

Track p50, p95, and p99 of each phase separately. A p95 TTFT regression almost always means provider congestion or a model change — not your code.

AI Cost Monitoring: Dashboards and Budget Burn Alerts

LLM spend compounds silently. A production cost dashboard should show, per day:

• Total spend split by model (you'll usually find 80% of cost in one model)
• Spend by feature and by top-10 users
• Cost per successful request and cost per user
• Projected monthly spend vs. budget (burn rate)

Then set three alert tiers:

1. Budget burn: projected monthly spend crosses 80% of budget → warn; 100% → page. Compute projection as (spend / days_elapsed) × days_in_month.
2. Cost anomaly: daily spend deviates >50% from the 7-day rolling average. Catches runaway loops, retry storms, and prompt bloat before the bill arrives.
3. Per-tenant spikes: any single user's spend doubles day-over-day — usually an integration bug or abuse.

Common cost leaks: unbounded max_tokens, retries on 429s that re-bill full prompts, missing prompt caching, and using a reasoning model where a mini model suffices. See our LLM cost optimization strategies for the full playbook.

OpenTelemetry Integration: Standard Spans for LLM Calls

OpenTelemetry is the vendor-neutral standard, and its GenAI semantic conventions give your LLM calls first-class trace support. Instrument the call as a span with gen_ai.* attributes:

from opentelemetry import trace
from opentelemetry.semconv.gen_ai import SpanAttributes as GenAI

tracer = trace.get_tracer("llm-app")

def traced_llm_call(messages, model):
    with tracer.start_as_current_span("chat.completions") as span:
        span.set_attribute(GenAI.OPERATION_NAME, "chat")
        span.set_attribute(GenAI.REQUEST_MODEL, model)
        span.set_attribute(GenAI.USAGE_INPUT_TOKENS, 812)
        span.set_attribute(GenAI.USAGE_OUTPUT_TOKENS, 340)
        span.set_attribute("llm.request_id", request_id)
        span.set_attribute("llm.feature", feature)
        return resp

Export spans to any OTLP-compatible backend (Grafana Tempo, Jaeger, Datadog, Honeycomb). Trace context propagation lets you connect the LLM call to the user's full request journey — the database query that built the prompt, the retry, and the downstream action. For teams that want zero-code tracing, the DrAI gateway logs every request with request IDs and token usage, so you can start from gateway logs and add client spans as you grow.

Alerting Rules That Actually Fire

Most monitoring setups alert on everything and page on nothing. A sane LLM alerting baseline:

Wire alerts to PagerDuty/Opsgenie for pages, Slack for warns. Add a weekly LLM cost and quality digest so regressions surface even when thresholds don't trip.

Output Quality Monitoring: The Metric Most Teams Skip

A hallucination is a 200 OK. Quality monitoring closes the gap:

• Schema validation: every structured output should be JSON-validated and schema-checked at the edge. A rising parse-failure rate is your earliest prompt regression signal.
• Refusal tracking: log when the model declines to answer ("I can't help with that"). A spike usually means your system prompt or content policy changed, or the model was swapped.
• Sampled human review: store response samples (with PII scrubbed) for spot-checking and building eval sets.
• LLM-as-judge scoring: for high-volume features, score a sample of outputs nightly with a cheaper model against rubric criteria (relevance, format, safety). See AI model evaluation for rubric design.

Choosing Your Observability Stack: Build vs. Buy

Teams building LLM observability face a real decision: assemble from open-source parts or adopt a purpose-built LLM observability platform. The honest comparison:

Most teams converge on: gateway-native usage data (cost, tokens, per-key) plus OpenTelemetry spans for the application-level view, plus a log sink for raw payloads. That covers the three questions you'll actually ask — "what did we spend", "where was it slow", and "what did the model return" — without building a data platform.

Log Retention and Privacy: Store Prompts Responsibly

Request logs contain your users' prompts — which may contain PII, trade secrets, or health data. Before you store everything forever, decide deliberately:

• Redact before storing: strip emails, phone numbers, API keys, and obvious PII at ingestion with a redaction pipeline, not at query time.
• Tiered retention: full payloads for 7-30 days for debugging; token counts, latency, and error aggregates forever. Most debugging happens in the first week.
• Sample the tail: store 100% of errors and 5-10% of successes — you need the failures in full, successes mostly for quality review.
• Know your obligations: GDPR, HIPAA, and SOC 2 all have opinions about prompt logs. If your provider processes data outside your region, that's a data-processing decision, not an accident.

Getting this wrong is how AI startups end up in the news. A redaction function in your logging path is twenty lines and saves you a compliance investigation.

The LLM Outage Playbook: When the Provider Goes Down

Provider outages happen every quarter somewhere in the ecosystem. Your monitoring setup should make the response boring:

1. Detect in under a minute: the 5xx-rate alert from the rules above is your tripwire. Don't discover outages from user complaints.
2. Fail over immediately: if your gateway has fallback routing configured, this is automatic — the alert becomes informational. Without fallbacks, flip the routing config and redeploy.
3. Communicate: post a status-page notice and a banner in your app. Users forgive outages they're told about; they don't forgive silent degradation.
4. Protect your budget: outage traffic often retries in a loop — pause retries or cap them, or the recovery bill spikes. Rate-limit client retry storms at the gateway.
5. Post-mortem: record time-to-detect, time-to-failover, and time-to-recovery. Each outage should shorten the next one; if your time-to-failover is over five minutes, that's the metric to fix.

LLM observability exists to make incidents boring. If your monitoring generates alerts but your team still scrambles, close the loop: alerts must map to runbook actions.

The Complete LLM Observability Checklist

1. Log every call as JSON with request_id, model, feature, user, tokens, and latency phases
2. Track the 7 core metrics per model, per feature, per tenant
3. Attribute cost to users and features; alert on burn rate and anomalies
4. Instrument TTFT and inter-token time separately from end-to-end latency
5. Export OpenTelemetry spans with GenAI semantic conventions
6. Validate outputs and track refusal and parse-failure rates as quality signals
7. Alert on error spikes, latency regressions, and budget burn — not on noise
8. Review dashboards weekly and eval sets monthly

DrAI's gateway includes built-in per-key request logs, token usage, and cost breakdowns, so you get production observability from day one — pair it with your own tracing for full visibility. Start with a free key at sign in, and see pricing for usage-based plans.