SCORING METHODS

KNOWN WEAKNESSES OF LLM JUDGES A judge is not an oracle, which is a source of perfect truth. It has quirks, and you must know them. Position bias is a tendency to prefer an answer simply because of where it appears, such as favoring the first of two candidates. Verbosity bias is a tendency to prefer longer answers, even when a shorter one is better. Self-preference or shared blind spot means a judge may favor answers that resemble its own style or may share the same factual mistakes as the agent being judged, especially if both are built on similar models. Inconsistency means the same judge can give different scores to the same answer on different runs. Because of these weaknesses, judges must be checked, not blindly trusted.