AI Embeddings Practical Guide: Semantic Search in Production

Published 2026-08-16 · 2,424 words · 10 min read

Embeddings are the quiet workhorse behind modern AI search: they convert text into vectors so that meaning, not just spelling, can be matched. But between the tutorial demo and a production semantic search service there is a long list of decisions that determine whether your system returns great results at low cost or mediocre results at high cost. This guide walks through the full production path: choosing an embedding model, picking a vector dimension, deciding how to chunk documents, selecting an index (HNSW, IVF, or flat), computing similarity correctly, combining embeddings with BM25 keyword search, and keeping the whole pipeline affordable as it scales.

What Embeddings Actually Do Under the Hood

An embedding model maps a piece of text to a dense vector of floating-point numbers, typically 256 to 3,072 dimensions. The training objective is simple to state: texts that mean similar things should land close together in vector space, while unrelated texts should land far apart. Semantic search then becomes a nearest-neighbor problem. You embed every document once at indexing time, embed the query at search time, and return the documents whose vectors are closest to the query vector.

The crucial property that makes embeddings useful is that they capture meaning beyond vocabulary. The query "how do I refund a subscription" will match a document that says "cancel your monthly plan and request a reimbursement" even though the two share almost no words. That is impossible with pure keyword search and is the reason embeddings became the backbone of RAG pipelines, recommendation systems, and deduplication jobs. It is also the source of their failure modes: embeddings can conflate topics, miss rare proper nouns, and degrade on out-of-distribution text, which is why production systems pair them with lexical search rather than replacing it.

Choosing an Embedding Model: What Actually Matters

The model you pick determines the ceiling of your search quality. The shortlist of serious contenders in 2026 includes OpenAI's text-embedding-3 family, Cohere's embed-v4, Google's Gemini embeddings, and open-source models such as BGE, E5, and Qwen-based embedders, often served through a model comparison that tracks MTEB scores and pricing side by side. For production selection, weigh four things, in this order.

First, retrieval quality on your own domain. The Massive Text Embedding Benchmark (MTEB) is a useful starting point, but a model that ranks first on general benchmarks can still lose on legal documents or support tickets. Build a small eval set of 200-500 real queries with known relevant documents and measure recall@k before committing. Second, latency and throughput: embedding 10 million documents at 100 documents per second takes over 27 hours; the same job at 1,000 per second takes under 3 hours. Third, pricing per million tokens, which varies by more than 10x between providers. Fourth, ecosystem friction: some providers only expose their embedding API through their own SDK, while others offer a plain OpenAI-compatible endpoint that works with every tool you already run.

One rule of thumb: if your corpus is under a few million chunks and you need fast iteration, start with a hosted API model in the 1,024-dimension range. If you need data residency, offline indexing, or very high volume, evaluate open-source models you can self-host. Re-embedding a corpus is expensive, so spend the time on model selection before you index, not after.

Dimension Trade-Offs: Bigger Is Not Always Better

Embedding dimension is the most misunderstood knob in the stack. High dimensions (3,072) capture more nuance and historically scored slightly better on benchmarks; low dimensions (256-512) cost less to store, less to transfer, and less to compute similarity against. Modern models can project down cheaply. OpenAI's text-embedding-3 models, for example, are trained at 3,072 dimensions but support truncation to 512 or 256 with minimal quality loss via Matryoshka-style training.

Dimension also drives index performance. ANN (approximate nearest neighbor) indexes like HNSW degrade gracefully as dimension grows, but recall and query latency both suffer: similarity in high-dimensional space becomes noisy, and distance computations cost more per vector. If you are uncertain, benchmark 512 versus 1,024 on your eval set. In most production RAG systems the quality delta is within noise, and the cost delta is not.

Chunking: The Decision That Determines Recall

Semantic search matches chunks, not documents. If you embed whole documents, the vector averages away the specific passage a user is looking for; if you embed single sentences, you lose context and generate thousands of tiny vectors that are expensive to store and noisy to match. Chunking strategy is therefore a first-class decision. The dominant patterns are fixed-size chunks with overlap, structure-aware chunking, and semantic chunking.

Fixed-size chunking is the baseline: split text every 300-800 tokens with a 10-20% overlap so that passages straddling a boundary still appear in at least one chunk. It is simple, deterministic, and works surprisingly well. Structure-aware chunking improves on it for documents with real structure: split on markdown headings, PDF sections, or HTML headings so each chunk is a coherent unit. Semantic chunking, where you split at points where embedding similarity between adjacent sentences drops, produces the most coherent chunks but costs an extra embedding pass over your corpus.

Chunk size interacts with the model's context window and your query type. Small chunks (150-300 tokens) win when queries target specific facts ("what is the refund policy?"); larger chunks (500-1,000 tokens) win when queries target whole sections ("summarize the pricing section"). Store the chunk's parent document reference and character offsets with every vector so you can return precise citations. Whatever you choose, make chunking reproducible: store the parameters in your index metadata, because re-chunking means re-embedding, and re-embedding is the most expensive mistake you can make.

Index Selection: HNSW, IVF, or Flat

With vectors in hand, you need an index that finds nearest neighbors fast. Brute-force flat search computes the exact distance to every vector and is perfectly fine up to a few hundred thousand vectors with caching. Beyond that, approximate indexes trade a small recall loss for orders of magnitude in speed. The two families that dominate are HNSW (hierarchical navigable small world graphs) and IVF (inverted file / product quantization).

HNSW is the default choice for most teams: it delivers single-digit millisecond queries at millions of vectors with parameters you can tune. The two knobs that matter are M (connections per node, default 16; higher improves recall at memory cost) and ef_search (how many candidate nodes to explore per query; higher improves recall at latency cost). Set ef_search adaptively — a cheap query path at 32, an expensive path at 256 for reranking — rather than paying maximum latency on every request. IVF with product quantization compresses vectors aggressively and shines past roughly 100 million vectors or when RAM is tight, at the cost of more tuning (nlist, nprobe) and lower recall ceilings. For most teams that are not yet at hundreds of millions of vectors, HNSW with scalar or product quantization on top is the sweet spot.

Similarity Computation: Cosine, Dot Product, and Normalization

Distance metrics are a classic source of subtle bugs. Cosine similarity measures the angle between vectors and ignores magnitude; dot product rewards both direction and magnitude; Euclidean distance measures straight-line distance. If you normalize every vector to unit length before storing (divide by its L2 norm), then cosine similarity, dot product, and negative Euclidean distance become ranking-equivalent — a trick that lets you use fast dot-product kernels in every index library and hardware accelerator.

def normalize(vec):
    norm = sum(x*x for x in vec) ** 0.5
    return [x / norm for x in vec]

# After normalization, these rank identically:
#   cosine_sim(v1, v2)  ==  dot(v1, v2)
#   l2_distance(v1, v2) ==  sqrt(2 - 2*dot(v1, v2))

Two practical consequences follow. First, normalize at embedding time on the ingestion path so queries and documents live in the same space — an unnormalized document corpus with a normalized query silently degrades ranking. Second, decide whether your score needs to be a calibrated similarity (0 to 1) or just a ranking. If you filter by threshold ("only return results above 0.75"), calibrate thresholds on your own corpus; embedding models are not calibrated across domains, and a threshold that works for one corpus can be useless on another.

Hybrid Search: BM25 Plus Vectors, Fused Properly

Pure semantic search has a blind spot for exact terms: product codes, version numbers, names, and typos. A query for "GPT-5" or "SKU-4472" can return semantically related but factually wrong documents because embeddings rarely preserve exact tokens. Hybrid search solves this by running a lexical index (BM25 or the newer BM25L/BM25+ variants) alongside the vector index and fusing the two result lists. This is the single highest-leverage quality improvement available to a production search system, and it is why virtually every serious RAG deployment now ships both.

# Reciprocal Rank Fusion: merge two ranked lists
def rrf(results_a, results_b, k=60):
    scores = {}
    for rank, doc in enumerate(results_a + results_b):
        scores[doc] = scores.get(doc, 0) + 1.0 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

Reciprocal Rank Fusion (RRF) is the pragmatic default: it requires no training, no score calibration between the two systems, and reliably beats either system alone on most corpora. If you have labeled data, a learned reranker (cross-encoder) on top of the fused top-50 is the strongest option, at the cost of one extra model call per query. Start with RRF, measure, and only add a reranker when the incremental latency budget allows it. Also remember the escape hatch: when a query matches an exact token pattern, let the lexical result win outright. A query containing "SKU-4472" should not be decided by semantic fuzz.

Keeping It Fresh: Upserts, Deletes, and Multi-Tenancy

Production corpora change. Documents get edited, removed, or reclassified, and your index must reflect that within a bounded time. Design your ingestion pipeline around immutable chunk IDs (for example doc_id:chunk_index), so upserts are idempotent and re-indexing a document is a delete-plus-insert rather than a blind append. Most vector databases support point deletes but with a caveat: HNSW graph structures need occasional optimization passes (some backends call it "cleanup" or "merge") to reclaim deleted nodes, otherwise memory grows and recall drifts. Schedule a nightly or weekly optimize job and monitor the deleted-vector ratio.

Multi-tenancy is the other scaling trap. Filtering by tenant ID after a global ANN search works at small scale but collapses once one tenant's vectors dominate the index. Prefer tenant-isolated indexes or metadata filtering pushed into the index itself (most modern engines support filtered ANN natively), and partition by tenant before approximate search rather than after. If a tenant has millions of vectors and others have thousands, consider separate index replicas per large tenant rather than one shared index where the small tenants drown.

Cost Optimization: The Embedding Bill Nobody Budgets For

Embedding costs hide in three places: initial corpus embedding, re-embedding after model or chunking changes, and query-time embedding. Attack all three. Batch initial ingestion with concurrency within the provider's rate limits — embedding APIs price per token, not per request, so throughput is free within limits. Cache query embeddings by normalized query text; real traffic has a long tail of repeats, and a cache hit is a zero-cost query. Monitor embedding token volume the same way you monitor completion tokens, and set the same cost guardrails you use for chat. Finally, match dimension to need: cutting 1,024 dimensions to 512 halves storage and speeds queries for a fraction of a percent of recall on most domains, which is often the cheapest optimization available in the whole stack.

Evaluating Semantic Search in Production

You cannot tune what you do not measure. Build an eval set from real user queries: take 200-500 logged queries, have a human (or a strong LLM, spot-checked) mark the relevant document IDs, then compute hit rate at k (did the right document appear in the top 5?) and mean reciprocal rank. Track these numbers when you change the model, the chunk size, the index parameters, or the fusion weights — every one of those knobs will move the metric, and the eval set is the only honest referee. A simple weekly regression job that runs the eval set and diffs against last week's numbers will catch embedding API model updates that silently shift behavior, which are far more common than vendors admit.

The Bottom Line

Production semantic search is a pipeline of deliberate trade-offs: a model chosen on your own eval set, a dimension that balances fidelity against storage, chunking that matches your query types, an index sized to your corpus, normalized vectors with a calibrated score, hybrid BM25 fusion for exact matches, and a budget with visibility into every token. Get those six decisions right and the system compounds; get one wrong and no amount of tuning elsewhere fully compensates. DrAI gives you the pragmatic side of the stack: OpenAI-compatible embedding and chat endpoints across 40+ models with per-key usage dashboards, so you can compare embedding models on price and quality before you index a single document. Start free at sign in, or review pricing.