SCORING METHODS

CHECK YOUR UNDERSTANDING • When is exact-match scoring a good choice? ◦ Answer: When the answer has one short, precise correct form, such as a number or a multiple-choice letter. • Why should an LLM judge give a written justification? ◦ Answer: So humans can check whether the reasoning was sound, and it often improves consistency. • What is position bias and how can you reduce it? ◦ Answer: It is favoring an answer based on its position; you can reduce it by scoring both orders and combining the results. • Why can a judge and an agent share a blind spot? ◦ Answer: If they are built on similar models, they may hold the same mistaken beliefs, so the judge cannot see the agent's error. • What is the purpose of calibrating a judge? ◦ Answer: To confirm that its scores agree closely with careful human scores before trusting it at scale.