AI API Disaster Recovery: Survive Provider Outages

AI features are now business-critical, but most teams have no disaster recovery plan for them — a provider outage means silent feature degradation or a hard outage with no playbook. AI disaster recovery means defining your RTO/RPO, running multi-provider hot standby, practicing traffic switches, and having degradation modes (cached answers, rule-engine fallback) that keep the product usable even when every LLM is down. This guide builds the complete plan with concrete numbers and code.

RTO and RPO for AI Services

Start by defining what "recovery" means for your AI features:

# Your DR targets, written down and reviewed quarterly:
DR_TARGETS = {
    "rto_minutes": 10,           # hot standby switch
    "rpo_transcripts": "stream", # query logs streamed continuously
    "degradation_hours": 4,      # cached-mode endurance
    "drill_frequency": "quarterly",
}

If you can't meet a 10-minute RTO with your current architecture, the fix is architectural (multi-provider failover), not procedural (a faster runbook).

Multi-Provider Hot Standby: The Core Strategy

Hot standby means a second provider can take traffic within minutes — not hours. Because the API surface is OpenAI-compatible, standby is a config, not a redeploy:

# Provider registry — switch = change env, no code change
PROVIDERS = {
    "primary":   {"base_url": "https://api.dr-ai.top/v1", "key": os.environ["KEY_A"]},
    "standby":   {"base_url": "https://api.openai.com/v1", "key": os.environ["KEY_B"]},
    # or standby via the same gateway's fallback chains:
    # "fallback_chain": ["gpt-5", "claude-sonnet-4", "deepseek-chat"],
}

def get_client():
    active = os.environ.get("ACTIVE_PROVIDER", "primary")
    p = PROVIDERS[active]
    return OpenAI(api_key=p["key"], base_url=p["base_url"])

Two ways to operationalize standby:

1. Gateway-managed (recommended) — your gateway (DrAI or similar) runs automatic circuit breakers and fallback chains: GPT-5 fails 5x → requests route to Claude Sonnet 4 → then DeepSeek. You do nothing; the switch is per-request and invisible.
2. Self-managed — you run health checks against each provider and flip an env var / DNS record when one fails. Slower (minutes), but gives explicit control and lets you stage by traffic percentage.

Whichever you choose, the standby must be tested, not assumed — see drills below.

Degradation Modes: Keep the Product Alive

Even with failover, a total AI outage (all providers down — rare but real) needs a degradation ladder:

def serve(request):
    try:
        return llm_call(request)                 # level 0-1
    except AllProvidersDown:
        cached = cache_lookup(request)           # level 2
        if cached: return cached
        if intent := rule_engine.match(request): # level 3
            return rule_engine.answer(intent)
        return graceful_decline(request)         # level 4

The rule engine matters more than teams expect: for support bots, 30-50% of queries are deterministic (order status, hours, shipping policy). A tiny intent→answer table keeps those alive through any outage. This pairs naturally with the response caching layer — your cache IS your degradation mode.

Data Protection and RPO

• Stream query logs — every prompt/response to durable storage (S3/Postgres) as it happens; RPO ≈ seconds
• Back up fine-tuned artifacts — model configs, LoRA weights, eval datasets (small, but irreplaceable) — nightly to a second region/provider
• Exportable config — routing rules, fallback chains, prompt templates in version control (they're code)
• Encryption keys — provider keys in a secrets manager with cross-region replication; a lost key is a self-inflicted outage

Traffic Switch Drills: Practice the Outage

An untested DR plan is a wish. Run quarterly drills that simulate real failure modes:

# Drill scenarios (rotate quarterly):
#   1. Primary provider outage (block its domain at your firewall)
#   2. Gateway outage (simulate by pointing base_url to a dead port)
#   3. Total AI outage (all providers blocked) → degradation ladder
#   4. Key rotation failure (rotate keys without updating config)
# For each: measure time-to-switch, error rates during switch, recovery time.

1. Announced drill — schedule, run, document. Measures baseline competence.
2. Unannounced drill — someone blocks the provider without telling the team. Measures actual readiness.
3. Chaos drill — random provider latency injection (p95 +2s) to test circuit breakers, not just hard outages.

Post-drill, the output is always the same: a list of gaps (too-slow switch, missing cache coverage, alert fatigue) turned into concrete fixes with owners.

Monitoring and Alerting for Outages

• Error-rate anomaly — alert when provider error rate crosses 2% over 5 minutes (normal is <0.5%)
• Latency anomaly — alert when p95 TTFT doubles vs rolling baseline
• Circuit-breaker trips — every trip is an event; repeated trips = provider incident
• Cache hit-rate drop — a sudden drop signals degraded retrieval or cache failure
• Status-page integration — subscribe to provider status feeds; don't discover outages from user complaints

# Sample alert rule (Pseudocode — wire into your monitoring):
RULE provider_error_rate:
  when: rate(provider_errors[5m]) > 0.02
  for: 5m
  then: page on-call + open incident + announce degradation mode

Communication Plan

During an outage, internal and external communication is part of the recovery:

• Internal — a shared incident channel with a pinned runbook (who switches, who tests, who communicates)
• External — a public status page (your own, updated within 15 min of Sev-1) and a message template for customers: what's affected, what you're doing, ETA if known
• Post-incident — an RCA within 5 business days, covering root cause, detection time, response time, and the three fixes you'll ship

The DR Checklist

1. RTO/RPO defined and reviewed quarterly
2. Multi-provider standby configured AND tested (gateway-managed or self-managed)
3. Degradation ladder implemented (cache → rule engine → graceful decline)
4. Query logs streamed; fine-tune artifacts backed up
5. Circuit breakers + anomaly alerting live
6. Quarterly drills with documented gaps and owners
7. Status page + customer communication templates ready

AI disaster recovery is mostly boring engineering: redundancy, caching, and practice. The boring parts are what make an outage a page you resolve in 10 minutes instead of a headline. For the supporting disciplines, read the reliability and SLA guide, the error handling guide (retry/circuit-breaker code), and the caching guide. DrAI's gateway implements circuit breakers, fallback chains, and caching out of the box — evaluate it free at ai.dr-ai.top/signin, with plans at pricing.