SCORING METHODS

REDUCING BIAS AND VARIANCE IN SCORING Bias is a consistent lean in one direction, such as always favoring long answers. Variance is random wobble, where the same input gets different scores on different tries. Think of archery. Bias is when all your arrows land to the left of the target. Variance is when your arrows are scattered all over the place. You want arrows clustered tightly on the bullseye. PRACTICAL TECHNIQUES First, run each judgment several times and average the scores, which smooths out random wobble. Second, use two or more different judge models and check whether they agree; disagreement is a signal to send that case to a human. Third, swap the order of candidate answers and score both orders to cancel position bias. Fourth, give the judge a precise rubric and, when possible, a golden answer to compare against rather than asking it to judge from scratch. Fifth, ask the judge to give its reasoning before its final score, since this often improves consistency. Sixth, keep the judge's settings fixed, such as using a low randomness setting, so that scoring conditions do not change between runs. Seventh, measure the judge itself by comparing its scores against human scores on a sample and tracking how well they agree over time. Eighth, hide irrelevant clues, such as which model or version produced an answer, so the judge cannot be swayed by a label.