AI API Observability Costs: What to Track and What to Skip

Observability has a dirty secret: it costs money, and for AI APIs it can cost a surprisingly large fraction of what the models themselves cost. Every request you log stores tokens you paid for twice — once to generate, once to persist. Every full-prompt trace you keep inflates storage bills that grow with your traffic. Every metric you ship to a vendor SaaS multiplies per-event fees. Teams that instrument their LLM stack enthusiastically on day one routinely discover on month three that their observability bill is 10-20% of their model spend — and that most of it is paying to store data nobody ever queries. This guide is about the economics of AI observability: what to track on every request, what to sample away, how to attribute cost to teams and features, and — just as importantly — what to skip entirely. The companion monitoring and observability guide covers how to build the tracking stack; this one covers how to keep it from eating your budget.

The Cost Stack of LLM Observability

LLM observability spend comes from four places, and teams usually notice only one of them:

• Storage — prompt and completion bodies, metadata, and traces. This is the big one: token-heavy payloads are megabytes per thousand requests, and retention multiplies it linearly.
• Compute for processing — parsing, tokenizing, embedding, and aggregating the data you collect. Cheap per event, real in aggregate at scale.
• Vendor fees — per-event ingestion, per-trace, per-seat, or per-GB pricing from observability SaaS (Langfuse, Helicone, Arize, LangSmith, Datadog LLM Observability, and similar).
• Engineering time — the least visible line item: dashboards nobody reads, alerts nobody acts on, and the instrumentation code that must be maintained across every model call site.

A useful mental model: the cost of observing a request should be a small, bounded fraction of the cost of the request itself. For a $0.01 model call, spending $0.002 on observability is sane; spending $0.01 is not. Before you add any instrumentation, estimate its per-request cost against that budget.

What to Track on Every Request

The default event schema for an LLM call is small — under 200 bytes if you exclude bodies. Track these on 100% of requests:

Notice what is deliberately absent: the prompt and completion bodies. Metadata-only logging keeps per-event cost near zero and still answers the questions that matter — who called what model, how much it cost, how fast it was, and whether it failed. Bodies are a separate, sampled, opt-in layer (next section).

The Logging Storage Math Nobody Does

Here is the arithmetic that decides whether your observability bill is trivial or terrifying. A typical chat completion is ~500 prompt tokens and ~300 completion tokens, or roughly 3 KB of text in JSON. At 1 million requests per day:

# Storage cost of logging full bodies vs metadata only
REQUESTS_PER_DAY = 1_000_000
BODY_BYTES = 3_000          # prompt + completion text as JSON
META_BYTES = 180            # metadata-only event

body_gb_per_day = REQUESTS_PER_DAY * BODY_BYTES / (1024**3)   # ~2.8 GB/day
meta_gb_per_day = REQUESTS_PER_DAY * META_BYTES / (1024**3)   # ~0.17 GB/day

# 90-day retention:
body_total = body_gb_per_day * 90      # ~252 GB
meta_total = meta_gb_per_day * 90      # ~15 GB
print(f"bodies: {body_total:.0f} GB | metadata-only: {meta_total:.0f} GB for 90 days")

At typical object-storage prices ($0.02/GB/month) and observability-vendor ingestion prices (often $0.50-$1 per GB or per million events), that 252 GB of bodies costs roughly $6-$250 per month just to store — before query cost, before the compute that tokenized it, before the vendor markup. Multiply by a year of retention and you are funding a data warehouse for data you almost never read. The fix is not "store nothing" — it is store metadata at 100%, bodies at a sample, and put the full-fidelity path behind an explicit flag for the requests that need it (errors, high-value transactions, on-demand debug captures).

Sampling Strategies: Get 95% of the Value for 5% of the Cost

Sampling is the single highest-leverage cost control in LLM observability. The goal is a representative slice of traffic plus complete coverage of the events that matter. Four strategies, in increasing sophistication:

• Random sampling — keep 5-10% of requests at full fidelity. Perfect for latency percentiles, token distributions, and cost aggregates, because those statistics survive unbiased sampling. Cost drops to the sample rate.
• Error-first sampling — keep 100% of failures (status != 200, validation errors, timeouts) at full fidelity, and sample successes. Failures are rare and priceless; successes are plentiful and mostly identical.
• Tail-based sampling — keep the slowest N% of requests (the "tail") plus all errors. This is what you actually want for latency work: the median is easy to reason about, the tail is where the problems live.
• Head-based + trace-level sampling — decide at the request root whether the whole trace is kept (e.g., keep if user is in the beta cohort, or if a header like x-debug: 1 is present, or 1-in-N). Trace-level decisions preserve causality — a sampled trace is complete, never half-logged.

# Head-based sampling at the gateway: cheap to decide, complete traces
import random, hashlib

def should_trace(request, sample_rate=0.05):
    if request.get("x-debug") == "1":
        return True                    # explicit full capture
    if request.get("error") or request.get("status") != 200:
        return True                    # all failures, always
    # deterministic hash-based sampling: stable per request_id
    h = int(hashlib.sha256(request["request_id"].encode()).hexdigest(), 16)
    return (h % 1000) < sample_rate * 1000

for req in requests:
    if should_trace(req):
        capture_full_trace(req)        # bodies, spans, everything
    else:
        log_metadata(req)              # 180-byte event

Whatever strategy you pick, enforce it in one place — the gateway or the SDK layer — so sampling cannot be bypassed by a future developer adding a log line. And record the sample decision in the metadata event itself; aggregate queries can then re-weight the sample back to true population estimates.

Cost Attribution: Tags Are the Cheapest Feature You Will Ever Build

Most teams can answer "how much did we spend on GPT-5 last month?" but cannot answer "how much did the checkout feature spend?" or "which customer is 30% of our token bill?" Cost attribution turns observability data into a budget weapon. The mechanism is trivial — tags on every event — and the payoff is structural:

# Tag everything at the call site; attribute later
client.chat.completions.create(
    model="gpt-5-mini",
    messages=msgs,
    extra_body={
        "metadata": {
            "team": "checkout",
            "feature": "order-summary",
            "customer_tier": "pro",
            "env": "production",
        }
    },
)

With tags in place, the same metadata stream answers: per-team spend this week, per-feature token efficiency (cost per completed checkout), per-customer usage for billing disputes, and per-environment waste (staging models running in production-shaped traffic). Three rules make attribution stick: default everything (untagged requests land in an "unknown" bucket that gets reviewed weekly), propagate from the edge (the gateway should inject tenant and project tags so SDKs cannot forget), and roll up daily (aggregate per tag-combination into a cost table; raw per-request events get sampled). Aggregates are where dashboards read from — nobody needs 30 days of raw events to know team X spent $4,000.

Dashboards vs Alerts: Skip What Nobody Reads

A large share of observability spend produces dashboards that are opened once at the demo and never again. The cost is not just the storage behind them — it is the alert fatigue that makes every real incident quieter. The discipline: every dashboard answers a question someone asked this month, and every alert has a runbook. The cheap, high-value core is small:

• Daily cost by model and team — the report that pays for the whole system.
• Error rate and p95 latency by model — the health signal.
• Cache hit rate — the untapped savings meter (see the caching guide for how to act on it).
• Budget burn rate — spend vs. the forecast line for the month.

Everything else — real-time token counters, per-prompt similarity heatmaps, weekly "insights" emails — can wait until someone actually asks for it. When they do, the data is still there in the aggregates.

Budget Alarms and Anomaly Detection

The most important alert in LLM observability is not technical — it is financial. A runaway prompt loop, a leaked key, or a misconfigured retry can multiply spend by 100x in an hour, and the models will happily comply. Budget alarms are the tripwire:

# Daily spend alarm against a forecast (runs in your scheduler)
import datetime

def check_budget(today_spend, month_spend, monthly_budget=10_000):
    today = datetime.date.today()
    days_in_month = 30
    day_of_month = today.day
    expected_so_far = monthly_budget * day_of_month / days_in_month
    burn_rate_ok = month_spend < expected_so_far * 1.3     # 30% over pace
    if not burn_rate_ok:
        alert("Budget pace exceeded: %s spent by day %s of %s budget"
              % (month_spend, day_of_month, monthly_budget))
    if today_spend > monthly_budget / days_in_month * 5:   # 5x daily pace
        alert("Anomalous daily spend: %s in one day" % today_spend, severity="high")

check_budget(today_spend, month_spend)

Beyond fixed budgets, three anomaly signals catch most runaway scenarios early: per-key spend spikes (a single API key burning 10x its baseline — often a leak or a bug), token-per-request inflation (average prompt size doubling usually means a code path is re-sending full conversation history), and retry amplification (error rate stable but spend up 3x — retries are multiplying failed calls). Each maps to a simple alert with a runbook that starts with "check the last deploy."

Self-Hosted vs Vendor Observability: The Real Comparison

The pattern that works at most companies: self-host the metadata store (it is small — the 15 GB/90-day example above fits on a single cheap disk), and keep the vendor only for the features that require product depth — evals, prompt versioning, and debugging UIs — pointed at sampled traffic. Splitting by fidelity (full metadata local, sampled bodies in the vendor) gives you the best of both and keeps the vendor bill proportional to what you actually use. For teams with strict data-residency requirements, the privacy and compliance guide covers the constraints that may decide this for you.

What to Skip: The Observability Anti-Patterns

For every metric worth keeping, there are three worth dropping. The highest-cost, lowest-value patterns observed across production LLM stacks:

• Full prompt bodies on every request — the storage sink described above. Sample them; keep them only for errors and debug-flagged traffic.
• Embedding vectors in logs — a 1536-dimension float array is 6 KB of near-useless log data. Store hashes or references, not vectors.
• Per-token breakdowns at the token level — you need totals, not the token-by-token ledger. Aggregate at ingestion.
• Real-time dashboards on wall screens — nobody reads them, and they force streaming pipelines you do not need. Daily aggregates suffice.
• Duplicate instrumentation layers — an SDK span, a gateway log, and a vendor trace of the same request. Pick one primary path and derive the rest.
• Retention beyond a quarter — for most debugging, 30-90 days of metadata and 7-14 days of sampled bodies is enough; archive anything older to cold storage at pennies per GB.

Summary

LLM observability has a budget problem, and it is solvable with four decisions: track metadata on 100% of requests and bodies on a sample; sample with an error-first or tail-based strategy decided in one place; tag everything so cost attribution becomes a daily report instead of a forensic project; and skip what nobody reads — full bodies, vector logs, real-time wallboards. Add budget alarms and anomaly detection on per-key spend, and choose self-hosted storage for metadata with vendor tooling only for the product-depth features. When the bill for observing your models stays under a few percent of the bill for running them, your observability is earning its keep — and the cost optimization playbook can take the next slice off the model side.