THE EVALUATION HARNESS

WHY MEASURING QUALITY IS HARD FOR AI Suppose you build a calculator. You can press two plus two, see four, and know the calculator is right. The answer is exact, and anyone can check it. Now suppose you build an agent that reads a customer complaint, searches a company database, drafts a reply, and sends it. Is the reply good? Maybe it is polite but slightly wrong about the refund policy. Maybe it is correct but too long. Maybe it got the right answer by a strange and risky route, such as searching the database twelve times before finding the right record. An AI agent produces open-ended results, and the same input can produce different outputs on different days, because language models include a controlled amount of randomness. That is why we need a dedicated system for measuring quality, rather than just eyeballing a few answers and hoping. WHAT IS AN EVALUATION HARNESS? An evaluation harness is the system that measures agent quality. Let us unpack that sentence slowly. The word evaluation means measuring how good something is. The word harness, as you know, means the supporting software around something that makes it usable. So an evaluation harness is the supporting software that takes a collection of test situations, feeds them to your agent, collects what the agent does, and then scores the results automatically and repeatably. Many people shorten the word evaluation to the word eval, and you will see that short form throughout this part. An analogy helps here. Imagine a driving school that tests new drivers. The school has a fixed test route, a clipboard with a checklist, and an examiner who writes down a score. Every student drives the same route and is judged against the same checklist. That route, checklist, and examiner together are the evaluation harness. The student driver is your agent. Without the harness, you would rely on a vague feeling that a student seems like a decent driver. With the harness, you can say that this student passed 18 of 20 checklist items, and last week's student passed 15, so we are improving.