How RAG Pipelines Scale to Millions of Documents

RAG works well in demos. It often fails silently in production. The failure isn't usually catastrophic — the system returns answers, just not always the right ones. Users stop trusting it. Usage drops. The project stalls. Understanding where naive RAG breaks down is the first step to building a retrieval pipeline that actually holds up at enterprise scale.

After deploying RAG systems across legal, financial services, and healthcare clients with document corpora ranging from thousands to millions of documents, we have developed a set of architectural principles that consistently produce reliable retrieval at scale.

Chunking Is an Underrated Engineering Problem

Most teams chunk documents by fixed token count with a small overlap. This is the right starting point and the wrong finishing point. Fixed-size chunking destroys semantic coherence — a crucial sentence about a contract clause ends up split across two chunks, and the retrieval system finds neither. For documents with structure — headings, sections, tables — semantic chunking that respects document hierarchy consistently outperforms fixed-size chunking on retrieval precision.

"The retrieval quality ceiling is set by your chunking strategy. Even the best embedding model cannot retrieve meaning that was cut in half during preprocessing."

The Re-ranking Step That Changes Everything

First-stage retrieval using approximate nearest-neighbour search over embeddings is fast but imprecise. Adding a cross-encoder re-ranker as a second stage — a model that scores the relevance of each retrieved chunk against the query with full bidirectional attention — dramatically improves precision without prohibitive latency cost. For enterprise search use cases, the addition of re-ranking consistently delivers 15–30% improvement in answer quality on our benchmark suite.

Infrastructure for Scale

For corpora below 5 million chunks, most managed vector databases (Pinecone, Weaviate, Qdrant) handle the load with appropriate indexing. Above that, the infrastructure decisions matter more: partition strategy, index type (HNSW vs. IVF), and embedding batch processing pipeline design all have meaningful impact on both query latency and cost. Plan for your scale, not your current state.