RETRIEVAL-AUGMENTED GENERATION (RAG)

CHUNKING, INDEXING, AND RETRIEVAL QUALITY Before documents can be searched, they must be prepared. First comes chunking. Chunking means splitting long documents into smaller pieces, called chunks, each of which gets its own embedding. Why split at all? A fifty-page manual embedded as a single vector would be a blurry average of everything in it, and retrieving it would flood the context window with irrelevant pages. Small chunks let you retrieve just the relevant paragraph. But chunk size is a balancing act. If chunks are too small, each one lacks context. A chunk that says only "It must be done within thirty days" does not reveal what "it" is. If chunks are too large, each one mixes several topics, making the embedding vague and wasting context space. Chunk boundaries also matter. Splitting in the middle of a sentence, or in the middle of a table, produces fragments that make little sense on their own. A better approach is to split along natural boundaries, such as sections, headings, paragraphs, or list items, so each chunk is self-contained and understandable alone. A common technique is overlap, in which neighboring chunks share a little text, such as a sentence or two at the edges, so an idea that straddles a boundary is not lost. As a rough starting point, many practitioners try chunks of a few hundred words with a small overlap, such as ten to twenty percent, and then adjust based on testing. There is no universally correct number, because the right choice depends on your documents and your questions. It is also wise to attach metadata to each chunk, meaning extra labeled information such as the document title, section heading, date, and source. Metadata supports filtering, such as "only search documents from this year," and it supports citations. Indexing is the step of storing all the chunk embeddings, with their text and metadata, in the vector database so they can be searched. It must be repeated or updated whenever documents change, or the search will return stale content, a subject we return to in Chapter 48. Retrieval quality is judged by two main ideas. Precision asks, of the chunks retrieved, how many were actually relevant? Recall asks, of all the relevant chunks that exist, how many did we retrieve? High precision with low recall means what you get is good but you miss things. Low precision with high recall means you get the needed pieces buried among junk. Both failures hurt, because the model can only use what it is given, and irrelevant text distracts it. You improve retrieval quality by trying several levers: better chunking, a better embedding model, hybrid search, reranking, rewriting the user's question into a clearer search query, and retrieving a sensible number of chunks, commonly a handful rather than dozens. The way to know whether your changes help is to measure. Build a small test set of realistic questions along with the chunks that should be retrieved for each, then compute precision and recall before and after each change. This is the same evaluation discipline you learned in Part Eight.