SCORING METHODS

LEARNING GOALS By the end of this chapter you will be able to describe exact-match and rule-based scoring, explain model-graded scoring (often called LLM as a judge), explain the role of human evaluation, and describe practical ways to reduce bias and variance in scoring. THE SCORING PROBLEM Once an agent produces an answer and you hold a golden reference answer, you still need to decide how good the answer is. That decision is called scoring. Scoring seems easy when the answer is a single number, and it gets surprisingly hard when the answer is a paragraph of free text. Consider two correct answers to the question "Why is the sky blue?" One might say that air scatters blue light more than red light, and the other might say that shorter wavelengths bounce off air molecules more strongly. The words differ, yet both are right. A scoring method must be able to recognize such equivalence, or it will unfairly punish good answers. This chapter covers four families of scoring approaches, from simple and rigid to flexible and human. EXACT-MATCH AND RULE-BASED SCORING