AI API Pagination and Filtering: Handle Large Result Sets

Every AI API eventually hands you a list too long to return in one response: usage logs with a million token events, vector search results across a hundred-thousand-document corpus, chat history for a busy workspace, or the full audit trail of model calls behind a billing dashboard. How you page through those lists decides whether your integration feels instant or crawls, whether it survives rate limits, and whether it stays consistent while new items keep arriving. This guide covers the two dominant pagination models — offset and cursor — plus filtering composition, streaming-aware pagination, and the rate-limit-friendly patterns that keep large result sets manageable in production. By the end you will know which pagination style fits which API, how to combine filters with paging without losing results, and how to design the client side so that ten-thousand-item pulls complete in minutes instead of hours.

Why AI APIs Return Large Result Sets

Classic CRUD APIs page through rows of user records; AI APIs page through something stranger: tokens, vectors, and generated artifacts. Four workloads produce the biggest lists, and each stresses pagination differently:

• Usage and billing logs — every completion, embedding call, and image generation writes a row with token counts, model name, latency, and cost. A busy gateway accumulates hundreds of thousands of rows per day, and customers want to pull "all of last month" for their own dashboards.
• Vector search results — similarity search returns a ranked list of document chunks. Top-k truncates, but analytical queries often want the full ranked set or large slices of it, which means paging through potentially millions of embeddings.
• Model and catalog listings — model inventories, tool registries, and agent marketplaces return thousands of items. These lists change slowly, but they are fetched by every client on startup, so pagination must be cheap and cache-friendly.
• Batch job outputs — a single batch endpoint can return tens of thousands of generated items. Clients resume failed downloads, so the pagination must be resumable and idempotent.

Each workload has a different consistency requirement: usage logs tolerate slight drift, vector rankings should stay stable during a page sequence, and batch outputs must never skip or duplicate items. That requirement — more than any other factor — decides which pagination style you should use.

Offset Pagination: Simple but Fragile

Offset pagination is the LIMIT ?offset model familiar from SQL and early REST APIs. The client requests ?limit=50&offset=200 and the server skips the first 200 rows, returning rows 201-250. It is trivial to implement, easy to reason about, and wrong for most large, fast-changing datasets.

# Offset pagination — simple, but degrades at scale
def fetch_usage(offset=0, limit=50):
    r = requests.get(f"https://api.example.com/v1/usage?offset={offset}&limit={limit}")
    rows = r.json()["data"]
    return rows, offset + len(rows)   # next offset = current + fetched

offset = 0
all_rows = []
while True:
    rows, offset = fetch_usage(offset)
    if not rows:
        break
    all_rows.extend(rows)
    if len(rows) < 50:
        break

Three failure modes make offset pagination dangerous on AI APIs:

• Deep-offset cost — skipping 100,000 rows requires the server to scan, sort, and discard them on every request. Latency grows linearly with offset, and the API eventually times out or rate-limits you into a corner.
• Insertion and deletion drift — if a new row lands at the front of the list between page 1 and page 2, every subsequent page shifts by one and you silently miss or duplicate rows. On a busy usage log, rows are inserted constantly.
• No stable anchor — nothing in the response tells you where you are in the dataset, so resuming a failed multi-hour pull means re-scanning from zero.

Offset pagination is still the right tool for small, static, admin-style lists (a model catalog with 200 entries, a user table in a back office). The moment the list can grow or change while you page, switch to cursors.

Cursor Pagination: The Production Standard

Cursor pagination replaces the numeric offset with an opaque token that encodes the server-side position. The client sends ?cursor=abc123, the server resumes from exactly that position, and returns the next page plus a next_cursor value — or null when the list ends. Most modern AI APIs — OpenAI's usage endpoints, Anthropic's message batches, Stripe-style listing patterns everywhere — use cursors because they are immune to drift and cheap to resume.

# Cursor pagination — stable under concurrent writes
def fetch_page(cursor=None, limit=50):
    params = {"limit": limit}
    if cursor:
        params["cursor"] = cursor
    r = requests.get("https://api.example.com/v1/usage", params=params, timeout=30)
    data = r.json()
    return data["data"], data.get("next_cursor")  # None when exhausted

cursor = None
all_rows = []
while True:
    rows, cursor = fetch_page(cursor)
    all_rows.extend(rows)
    if not cursor:
        break
    if len(all_rows) >= 100_000:   # safety valve
        break

The cursor itself is opaque to the client — never decode it, never construct one yourself. Behind the scenes it is usually a keyset value: the timestamp, ID, or composite sort key of the last row of the previous page. A well-designed cursor API gives you four guarantees:

• Stability — new rows appended to the end of the list do not shift pages you already fetched.
• Resumability — you can pause for hours, save the cursor, and continue exactly where you stopped.
• Constant-time cost — the server seeks to the cursor with an index instead of scanning from zero, so page latency stays flat no matter how deep you go.
• Consistency — because the cursor encodes the sort key, rows deleted mid-pull simply vanish rather than silently shifting everything after them.

Keyset Pagination: Cursors You Can See

Some APIs expose keyset pagination directly instead of hiding it behind an opaque token. You pass the last sort key you saw: ?after=2026-08-01T00:00:00Z or ?created_before=...&limit=50. This is cursor pagination with the curtain pulled back — the cursor is a real column value, which makes it debuggable and lets you jump to arbitrary points in the dataset.

# Keyset pagination over created_at (index-backed seek)
def fetch_usage_after(ts, limit=50):
    r = requests.get("https://api.example.com/v1/usage",
                     params={"after": ts.isoformat(), "limit": limit})
    rows = r.json()["data"]
    return rows, rows[-1]["created_at"] if rows else None

ts = None
while True:
    rows, ts = fetch_usage_after(ts)
    if not rows:
        break
    process(rows)

Keyset pagination is ideal for time-ordered data — exactly what usage logs, chat transcripts, and event streams are. It also composes beautifully with filters: ?after= plus ?model=gpt-5-mini means "all rows for this model after this timestamp," which is the query behind most cost dashboards. The one rule to respect: the sort key you page on must be the same column you filter on, or you risk skipping rows that match the filter but sort before your cursor.

Filtering: Composition and Semantics

Pagination and filtering are two halves of one query language. A page size of 50 means nothing if you cannot first narrow "everything" to "the things I care about." Production AI APIs expose filters that combine in three layers:

• Field filters — model, status, user_id, date_from/date_to narrow the dataset before pagination applies.
• Query filters — q or search terms for semantic or lexical search over the list (common in vector and document endpoints).
• Pagination parameters — limit, cursor/offset, and often sort plus order.

# Composing filters + cursor pagination (order matters)
def fetch_filtered(model="gpt-5-mini", date_from=None, cursor=None, limit=100):
    params = {"model": model, "limit": limit}
    if date_from:
        params["date_from"] = date_from
    if cursor:
        params["cursor"] = cursor
    r = requests.get("https://api.dr-ai.top/v1/usage", params=params, timeout=30)
    r.raise_for_status()
    data = r.json()
    return data["data"], data.get("next_cursor")

# Pull all usage for one model over a week
cursor = None
total_tokens = 0
while True:
    rows, cursor = fetch_filtered(date_from="2026-08-01", cursor=cursor)
    total_tokens += sum(r["total_tokens"] for r in rows)
    if not cursor:
        break
print(f"week tokens: {total_tokens}")

Two filter-composition pitfalls dominate real integrations. First, server-side filter application — filter at the API, never client-side after paging. Filtering in your code after pagination means you can miss matches that sit beyond your page window, and you pay token cost for rows you discard. Second, cursor + filter coupling — the cursor is only valid for the filter set that produced it. If you change model or date_from mid-pull, discard the old cursor and restart; reusing a cursor across different filters is undefined behavior on most APIs and silently wrong on the rest.

Streaming Pagination for Very Large Result Sets

Some result sets are too big to page through comfortably even with cursors: a million-embedding similarity scan, a full-corpus export, a year of chat logs. Two patterns solve this: server-side streaming and cursor-resumable batch endpoints.

Server-side streaming — the API returns a single response whose body streams rows as they are produced (newline-delimited JSON or SSE). The client processes rows incrementally without ever holding the full list in memory. This is how vector index exports and large usage downloads typically work:

# NDJSON streaming download — process as rows arrive
def stream_export(url, headers):
    with requests.get(url, headers=headers, stream=True, timeout=600) as r:
        r.raise_for_status()
        for line in r.iter_lines():
            if line:
                yield json.loads(line)

for row in stream_export("https://api.example.com/v1/usage/export?date_from=2026-08-01",
                         {"Authorization": "Bearer " + KEY}):
    ingest(row)   # write to warehouse as it arrives

Cursor-resumable batch — the export endpoint returns a job ID plus a cursor; each chunk of the export advances the cursor, and a dropped connection resumes from the last cursor instead of restarting. This is the pattern to demand from any API you export from on a schedule. Combined with a checkpoint file on your side, it makes multi-hour exports effectively failure-proof:

# Resumable export with a local checkpoint
import os, json
def checkpoint_path(): return "/tmp/export_cursor.json"

def save_cursor(c):
    with open(checkpoint_path(), "w") as f:
        json.dump({"cursor": c}, f)

def load_cursor():
    if os.path.exists(checkpoint_path()):
        with open(checkpoint_path()) as f:
            return json.load(f).get("cursor")
    return None

cursor = load_cursor()
while True:
    rows, cursor = fetch_page(cursor, limit=200)
    save_cursor(cursor)          # checkpoint after every page
    for row in rows:
        ingest(row)
    if not cursor:
        break
os.remove(checkpoint_path())

Streaming pagination changes your failure mode from "restart everything" to "continue from the last good page" — the difference between a two-minute recovery and a two-hour re-run.

Rate-Limit-Friendly Pagination

Large result sets and rate limits are natural enemies: pulling 10,000 rows at 50 per page is 200 requests, and if your API budget is 60 requests per minute, that is a three-and-a-half-minute walk through a minefield. The pagination patterns that survive rate limits share five habits:

• Request the largest page the API allows — 100 rows per request instead of 20 cuts request count fivefold. Cost per page barely changes, but rate-limit headroom grows proportionally.
• Honor Retry-After — when a 429 arrives, read the Retry-After header and sleep exactly that long, then resume from the same cursor. Never restart the pull from scratch on a 429.
• Backoff on page failures — exponential backoff with jitter (1s, 2s, 4s, 8s, capped) applied only to the failed page keeps the rest of the sequence intact.
• Cache pages you already fetched — re-running a cost dashboard that pages through the same 50 pages daily should hit an HTTP cache or a local store, not the live API.
• Parallelize only within budget — parallel page fetching (one worker per cursor segment) speeds pulls up, but only if the API supports it and you reserve headroom for interactive traffic. When in doubt, serialize.

# Rate-limit-aware pagination with backoff
import time, random

def fetch_page_safe(cursor, limit=100, max_retries=5):
    for attempt in range(max_retries):
        r = requests.get("https://api.example.com/v1/usage",
                         params={"limit": limit, "cursor": cursor}, timeout=30)
        if r.status_code == 429:
            wait = float(r.headers.get("Retry-After", 1)) + random.uniform(0, 0.5)
            time.sleep(wait)
            continue
        r.raise_for_status()
        return r.json()["data"], r.json().get("next_cursor")
    raise RuntimeError("rate limited after retries")

cursor = None
while True:
    rows, cursor = fetch_page_safe(cursor)
    ingest(rows)
    if not cursor:
        break

Notice what makes this robust: the cursor is the only state that moves forward, retries never replay ingested rows, and the 429 path sleeps instead of burning requests. The full retry-and-fallback playbook for AI APIs is covered in our error handling guide.

Choosing a Pagination Style: Decision Guide

If the API you are integrating offers only offset pagination on a dataset you know is large and changing, ask the provider for a cursor — most gateway platforms, including the usage endpoints behind DrAI's API, support cursor-based listing precisely because customers hit these limits. And when you build your own AI API, ship cursors from day one; retrofitting them later is a breaking change for every client (the versioning strategy guide explains how to land such changes without breaking anyone).

Common Pitfalls and How to Debug Them

• Duplicate rows across pages — usually a missing or unstable sort: rows reorder between requests, so the cursor jumps. Fix: ensure the API sorts by a unique, immutable key (ID or created_at + ID).
• Missing rows at page boundaries — the classic offset-drift symptom. If the API insists on offset, re-query with a timestamp filter and dedupe client-side by ID.
• Cursor reuse after filter change — undefined results. Always restart the sequence when any filter parameter changes.
• Infinite loop when next_cursor never ends — guard every loop with a max-pages or max-rows safety valve, as in the examples above.
• Timezone bugs in keyset timestamps — always send ISO-8601 UTC (2026-08-01T00:00:00Z), never local time; a DST shift silently drops or duplicates an hour of rows.
• Ignoring has_more vs next_cursor — some APIs signal continuation with a boolean and return an empty cursor page at the end. Treat "no next cursor" and "has_more=false" identically as the stop condition.

Summary

Pagination is the quiet engineering layer between your application and the data an AI API produces. Offset pagination still has a home on small static lists, but production-scale result sets — usage logs, vector rankings, batch outputs — need cursors: stable under writes, resumable after failure, and constant-time at any depth. Combine cursors with server-side filters, prefer streaming or checkpointed exports for million-row pulls, and shape every loop around rate limits with page-size maximization, Retry-After handling, and backoff. Get those four decisions right and a 10,000-item pull becomes a background routine instead of a fire drill.