SCORING METHODS

EXAMPLES Example 1: Exact-match scoring A math eval asks, "What is 6 times 7?" and the golden answer is 42. The scorer first normalizes the agent's output by removing spaces and punctuation, then checks whether it equals 42. An agent that answers "42" passes in a fraction of a second. An agent that answers "forty-two" fails, even though a person would accept it. That is the brittleness described earlier. The team can fix this by adding a rule that also accepts the spelled-out number, but notice that every new variation needs a new rule, which is why exact match is best for simple, single-valued answers. Example 2: LLM as a judge A customer-support team wants to score thousands of free-text replies. They give a judge model the customer's question, the golden answer, and the agent's reply, plus a rubric: score from 1 to 5 for similarity and correctness, and write two sentences of justification. For one reply, the judge returns a 4 and explains that the reply gave the correct 90-day window but forgot to mention the store credit option. A human reading that justification can immediately see whether the judgment was fair. The team has scored an open-ended answer at scale, which neither exact match nor simple rules could do. Example 3: Human review catches what the judge missed An agent states that a certain medication is safe at a dose that is actually unsafe. The judge model, built on a model family that shares the same mistaken belief, gives the reply a 5 out of 5. A pharmacist reviewing a random sample spots the error immediately. The team learns that their judge shares a blind spot with the agent, and they add a human review step for all medical content and a rule-based check against a trusted dose table. The lesson is that automated judges are helpful but not infallible. Example 4: Reducing bias and variance A team compares two harness versions, A and B, using an LLM judge. At first the judge picks whichever answer appears first about 60 percent of the time, regardless of quality, which reveals position bias. The team fixes this by scoring each pair twice with the order swapped, and by running the judge three times and averaging. They also add a second judge from a different model family, and when the two judges disagree by more than one point, the case goes to a human. After these changes the scores become far more stable, and the team trusts the comparison between A and B.