The Judge is Part of the Experiment: Auditing Peer Context Sensitivity in LLM-Based Evaluation
Alicia Guerra, Jiaming Qu, Yibo Hu
原文摘要
Large language model (LLM)-based judges are increasingly used to evaluate AI systems. An underexplored area is applying these judges to multi-agent systems, where they may see the interactions behind a final answer. These interactions, which we call peer context, can affect an evaluation in two ways: (1) they can change the answer produced by the multi-agent system, because an agent might be swayed by other agents, and (2) they can change how a judge evaluates that same answer, depending on whether the judge sees them. In this paper, we separate these two effects through a controlled experiment. We first let an agent answer a question and then answer again after seeing peer responses. We then prompt judges to rate the same final answer with and without the peer context. Finally, we check whether the judges' ratings establish a valid measure of answer quality. Across three benchmarks, four open-weight LLMs as agents, and six judges, we find that peer context changes not only the answer but also how an identical answer is judged, and judges react to it in opposite directions. Moreover, providing peer context for a wrong answer misleads agents more than correct peers help them; without the correct answer, judges struggle to tell known-correct from known-wrong answers---a problem that a well-fitted statistical model does not reveal. Our results show that even when evaluating the same answer, how a judge is set up can change the conclusions differently. We therefore call for evaluations to report what each judge sees, document detailed results per judge, and validate rating scales. These practices are an important step toward the accountable use of LLM-based judges.
cs.CL