RETRIEVAL-AUGMENTED GENERATION (RAG)

WHAT RETRIEVAL-AUGMENTED GENERATION IS RAG is a technique in which, before the model answers, the harness first searches a collection of documents for the passages most relevant to the question, places those passages into the model's context, and instructs the model to answer based on them. The name has three parts. Retrieval means fetching relevant information. Augmented means adding to or boosting something. Generation means the model producing text. So the model's generation is augmented by retrieved information. Why is RAG needed? An LLM has several limits that RAG helps with. First, the model's knowledge comes from its training data, which has a cutoff date, so it does not know about recent events or documents created after training. Second, the model has never seen your private documents, such as your company handbook or your customer records. Third, the model can hallucinate, meaning it can produce fluent, confident statements that are false, especially when it lacks the true information. Fourth, you cannot fit an entire library into the context window. RAG solves these by fetching only the small number of relevant passages when needed. The basic RAG flow has five steps. Step one: the user asks a question. Step two: the harness turns the question into a search query and searches the document collection. Step three: the search returns the top few most relevant passages. Step four: the harness builds a prompt containing the question, the passages, and an instruction such as "answer using only the provided passages, and say you do not know if the answer is not there." Step five: the model writes the answer, ideally citing which passage supported each claim. This connects directly to the context construction ideas in Chapter 20: retrieval is one of the ways the context builder decides what to place in the model's view. EMBEDDINGS: NUMBERS THAT CAPTURE MEANING How does a computer search by meaning? The answer is embeddings. An embedding is a list of numbers that represents the meaning of a piece of text, produced by a special kind of model, called an embedding model, so that texts with similar meanings get similar lists of numbers. Here is a way to picture this. Imagine a giant map where every sentence in the world has a location. Sentences that mean similar things are placed close together, and sentences that mean different things are placed far apart. The embedding is simply the sentence's coordinates on this map. Real embeddings use many numbers, often hundreds or thousands, rather than two, so the map has many dimensions. You cannot picture it, but the principle is the same: closeness equals similarity of meaning. Consider two sentences: "The cat sat on the mat" and "A feline rested on the rug." They share almost no words, but their meanings are nearly the same. Their embeddings would be very close together. By contrast, "The stock market fell today" would be far away from both. To measure closeness, systems use mathematical measures of similarity between two lists of numbers. One widely used measure is called cosine similarity, which compares the direction the two lists of numbers point. You do not need to calculate it yourself, because software libraries do it. You only need to know that a higher similarity score means closer meaning. One caution: embedding models are not magic. They can be fooled by rare technical terms, names, or numbers, and different embedding models work better for different languages and subjects. You should test on your own data. VECTOR DATABASES: SEARCHING BY MEANING A vector is a list of numbers, so an embedding is a vector. A vector database is a storage system designed to hold many embeddings and to quickly find the ones closest in meaning to a query embedding. Contrast this with traditional keyword search. Keyword search looks for the exact words you typed. If you search for "how do I reset my password" and the help article is titled "Account access recovery steps," keyword search may miss it because the words do not match. A vector search converts your question into an embedding and finds stored items whose embeddings are nearby, so it can retrieve the "Account access recovery steps" article since its meaning is close. Storing and searching millions of vectors efficiently uses special techniques, often grouped under the idea of approximate nearest neighbor search, which finds very-close-but-not-guaranteed-closest matches quickly. Many databases and search services offer vector search, including dedicated vector databases and extensions of ordinary databases. Here is an important practical insight. Meaning search and keyword search have different strengths. Meaning search is good for paraphrases and vague questions. Keyword search is good for exact identifiers, such as an error code or a product part number. Many strong systems combine both, an approach called hybrid search, and then optionally apply a reranker, a second model that re-sorts the top results by how well they truly answer the question.