REPLAY AND DEBUGGING

CHECK YOUR UNDERSTANDING • What is replay? ◦ Answer: Re-running a recorded session offline to study it. • Why record every tool response? ◦ Answer: A live re-run might meet changed data and fail to reproduce the problem. • What does an HTTP 503 in a trace suggest? ◦ Answer: A tool or network problem, since the service was unavailable. • What follows a good root-cause analysis? ◦ Answer: A fix and a new regression test. • What is the difference between a harness problem and a reasoning problem? ◦ Answer: A harness problem comes from your software, such as a missing limit, and is fixed in code, while a reasoning problem comes from the model's judgment and is improved through prompts, context, or model choice. PART EIGHT: THE VERIFICATION LAYER (EVALUATION, TESTING, AND QUALITY) Welcome to Part Eight. In the earlier parts of this book, you learned how to build the parts of a harness that run an agent, give it tools, manage its memory, and let you watch what it does. A harness, remember, is the supporting software wrapped around an LLM so that the model can act as a reliable agent. Recall from Chapter 2 that an LLM predicts and produces language, and from Chapter 5 that an agent is a software system that takes actions on its own to reach a goal. Now we turn to a question that every builder must eventually face: how do you know the thing actually works? This part is called the Verification Layer because it is about verifying, which simply means checking that something is true. Verification is the sixth layer of the seven-layer framework called ETCLOVG, which stands for Execution, Tooling, Context, Lifecycle, Observability, Verification, and Governance. Think of a restaurant kitchen. A cook can be talented, the stove can be excellent, and the recipe can be perfect, but a good restaurant still has someone who tastes the food before it leaves the kitchen. Verification is the tasting step for AI systems. In this part you will learn how to build a system that measures quality (Chapter 31), how to design good tests (Chapter 32), how to score answers fairly (Chapter 33), how to use shared public tests and understand their limits (Chapter 34), and how to test the harness itself as ordinary software (Chapter 35).