AI Model Evaluation Guide: Build Your Own LLM Test Suite

Building your own LLM test suite catches quality regressions before users do. A production evaluation stack needs four components: golden datasets (input-expected output pairs), automatic metrics (exact, semantic, LLM-judge), regression CI (every prompt/model change re-scored), and online monitoring (drift detection on live traffic). Teams that add even 50 test cases catch 80% of prompt-regression incidents pre-deploy — this guide builds the whole loop in Python.

Why Vibe-Checking Isn't Evaluation

Trying 5 prompts manually after each change catches obvious breaks and nothing else. Model updates, prompt tweaks, and temperature changes shift behavior in ways casual testing misses — your classification prompt silently drops from 94% to 87% accuracy, and you find out from users. Systematic evaluation is the difference.

Component 1: The Golden Dataset

Start with 50-200 curated test cases covering your real distribution:

# evals/dataset.jsonl — one case per line
{"id": "t001", "input": "Refund my order #4521", 
 "expected": "REFUND", "category": "intent"}
{"id": "t002", "input": "How do I reset my password?",
 "expected": "HELP_ACCOUNT", "category": "intent"}
{"id": "t003", 
 "input": "Summarize: [meeting transcript...]",
 "expected_contains": ["action items", "deadline"],
 "category": "summarization"}
{"id": "t004", "input": "Write a haiku about testing",
 "semantic_ref": "A poem about quality checking, 5-7-5 form",
 "category": "generation"}

Three case types by check style: exact (classification, extraction), contains (summaries must mention key facts), semantic (open generation judged by similarity or LLM-judge). Source cases from real logs — your top 100 real queries beat 1,000 invented ones.

Component 2: Automatic Metrics

import json, re
from openai import OpenAI

client = OpenAI(base_url="https://api.dr-ai.top/v1")

def check_exact(case, output):
    return output.strip() == case["expected"]

def check_contains(case, output):
    return all(k.lower() in output.lower() 
               for k in case["expected_contains"])

def check_semantic(case, output):
    """LLM-as-judge with a strict rubric."""
    verdict = client.chat.completions.create(
        model="gpt-5", temperature=0,
        response_format={"type": "json_object"},
        messages=[{"role": "user", "content": f"""
You are a strict grader. Score the OUTPUT against the REFERENCE.
Return JSON: {{"pass": true/false, "reason": "..."}} 
Fail on: missing key facts, contradictions, format violations.

REFERENCE: {case['semantic_ref']}
OUTPUT: {output}
"""}])
    return json.loads(verdict.choices[0].message.content)["pass"]

def run_case(case, predict_fn):
    output = predict_fn(case["input"])
    if "expected" in case:
        return check_exact(case, output)
    if "expected_contains" in case:
        return check_contains(case, output)
    return check_semantic(case, output)

LLM-judge calibration tip: run the judge against 20 human-labeled cases first. If judge-human agreement is under 85%, tighten the rubric — an uncalibrated judge adds noise, not signal. Use GPT-5-class models for judging; mini-class judges are inconsistent graders.

Component 3: Regression CI

Wire the suite into CI so every prompt or model change re-scores automatically:

# evals/run.py — exits non-zero on regression
import sys, statistics

def run_suite(dataset_path, predict_fn, threshold=0.90):
    cases = [json.loads(l) for l in open(dataset_path)]
    results = [run_case(c, predict_fn) for c in cases]
    overall = statistics.mean(results)
    
    # Per-category breakdown
    by_cat = {}
    for c, r in zip(cases, results):
        by_cat.setdefault(c["category"], []).append(r)
    
    print(f"Overall: {overall:.1%}")
    for cat, rs in sorted(by_cat.items()):
        rate = statistics.mean(rs)
        print(f"  {cat}: {rate:.1%} ({sum(rs)}/{len(rs)})")
    
    sys.exit(0 if overall >= threshold else 1)

# .github/workflows/eval.yml
# - name: LLM eval suite
#   run: python evals/run.py --threshold 0.92

Set the threshold from your current baseline: if you're at 93.4%, gate at 93%. Any dip becomes a failed build, not a user complaint.

Component 4: Online Monitoring

Offline suites can't see distribution drift — live traffic brings inputs your golden set never imagined:

# Log every production call with sampling
{"ts": ..., "model": "gpt-5-mini", "prompt_hash": "a3f...",
 "input_len": 412, "output_len": 89, "latency_ms": 240,
 "user_feedback": null, "flags": []}

# Three drift alarms (daily job):
# 1. Distribution drift: new prompt patterns absent from 
#    the golden set (cluster inputs; flag new clusters >5%)
# 2. Length drift: mean output length ±30% vs 7-day baseline
# 3. Feedback drop: thumbs-down rate +2pp week-over-week
# Each alarm ⇒ mine 20 samples ⇒ add top patterns to golden set

The flywheel: online drift feeds the golden dataset, the dataset guards CI, CI protects quality. Evaluation compounds.

Testing Model Swaps and Routing

The same suite prices your routing decisions. Score each candidate model, then compare quality-per-dollar:

models = ["gpt-5", "gpt-5-mini", "claude-sonnet-4", "deepseek-chat"]
for m in models:
    accuracy = run_suite(dataset, lambda x: predict(x, model=m))
    cost = estimate_cost(dataset, model=m)
    print(f"{m}: {accuracy:.1%} at ${cost:.2f} "
          f"(${cost/accuracy:.2f}/point)")

Run suites like this before any model routing change — it turns routing from folklore into measured policy.

Common Pitfalls

• Fixed seeds expectation — temperature 0 isn't fully deterministic across providers; assert with contains/semantic, not exact, except for classification
• Testing only happy paths — 20% of cases should be adversarial: injection attempts, malformed input, out-of-scope queries
• Judge grade inflation — re-calibrate the LLM-judge monthly against human labels
• One mega-suite — split per use case; a 92% gate on support-intent says nothing about your summarizer
• No versioning — commit dataset + prompts + results together so every score is reproducible

Start today: extract your top 50 real queries, write the three check functions above, and gate your next prompt change. The full loop takes a day to build and pays back on the first caught regression. For adjacent infrastructure, see benchmark methodology for public-benchmark context and model evaluation frameworks — and run every candidate model behind one DrAI key at passthrough pricing.

Sources & Further Reading

• OpenAI evals documentation
• OpenAI Evals — open-source evaluation framework (GitHub)
• MMLU benchmark leaderboard (Papers with Code)
• DrAI: LLM Benchmark Methodology
• DrAI: AI Model Routing Strategy