SCORING METHODS

HUMAN EVALUATION AND HUMAN REVIEW Human evaluation means that real people read outputs and score them. It is slower and more expensive than automatic scoring, but it is the closest thing to ground truth for many qualities, such as whether advice is wise or a message sounds respectful. Humans serve several roles. First, they create the golden answers and the rubrics. Second, they review a sample of the judge's scores, checking whether the automatic judge agrees with careful human opinion. Third, they audit hard or high-stakes cases, such as medical, legal, or financial outputs. Fourth, they discover new kinds of problems that nobody thought to write a rule or rubric for. Humans also have limits. Different reviewers may disagree, get tired, or be influenced by an answer's confident tone. To manage this, teams give reviewers clear guidelines, have two reviewers score the same items, and measure how often they agree. If two humans agree only half the time, the rubric is probably unclear. A practical pattern is to calibrate the judge: have humans score a small set, run the judge on the same set, and compare. If the two agree closely, the judge can be trusted on similar cases at scale, with humans spot-checking a small sample forever after.