SCORING METHODS

RULE-BASED SCORING Rule-based scoring applies a set of explicit checks written in ordinary code. Examples include checking that a reply contains no more than 150 words, that it includes a required disclaimer, that it is valid JSON, that it never mentions a forbidden word, or that a generated program passes a set of automated unit tests. A unit test is a small automatic check of one piece of code, and you will study these in Chapter 35. Rule-based scoring is also fast, cheap, and consistent, and it can check structure and policy very well. It cannot easily judge qualities like helpfulness, tone, or whether an explanation is actually correct and clear. Think of rule-based scoring as a ruler. A ruler is wonderful for measuring length and useless for measuring kindness. Whenever a rule can capture what you care about, use the rule, because it is the most trustworthy and least expensive scorer. MODEL-GRADED SCORING (LLM AS A JUDGE) Many qualities cannot be captured by a ruler. Is this summary faithful to the document? Is this explanation clear? Did this reply follow our tone guidelines? For such questions, teams often use another model as the grader. This technique is called model-graded scoring, and it is more commonly known as LLM as a judge.