← 研踪 PaperTrailACL Anthology · ACL 2026:acl:2026.acl-long.922 · 记录 #1
League of LLMs: A Benchmark-Free Paradigm for Mutual Evaluation of Large Language Models
来源日期（仅到月份） 2026-07
来源记录日期 2026-07
雷达观测 2026-11-01 21:58
摘要
Although large language models (LLMs) have shown exceptional capabilities across a wide range of tasks, reliable evaluation remains a critical challenge due to data contamination, opaque operation, and subjective preferences. To address these issues, we propose League of LLMs (LOL), a novel benchmark-free evaluation paradigm that organizes multiple LLMs into a self-governed league for multi-round mutual evaluation. LOL integrates four core criteria (dynamic, transparent, objective, and professional) to mitigate key limitations of existing paradigms. Experiments on eight mainstream LLMs in mathematics and programming demonstrate that LOL can effectively distinguish LLM capabilities while maintaining high internal ranking stability (Top-k consistency = 70.7\%). Beyond ranking, LOL reveals empirical findings that are difficult for traditional paradigms to capture. For instance, “memorization-based answering” behaviors are observed in some models, and higher in-family scores are found in the OpenAI model family (\Delta = 9, p < 0.05). Finally, we make our framework and code publicly available as a valuable complement to the current LLM evaluation ecosystem.
论文发现出处
这些页面提供论文线索，不代表同行评审或质量认证。
观测到的版本
平台记录号代表本站观测到的元数据变化，不是作者修订号。来源日期仅到月份或年份时，按该月或该年首日存档。
记录 #1 · 2026-07 · 最新观测版本
League of LLMs: A Benchmark-Free Paradigm for Mutual Evaluation of Large Language Models
Although large language models (LLMs) have shown exceptional capabilities across a wide range of tasks, reliable evaluation remains a critical challenge due to data contamination, opaque operation, and subjective preferences. To address these issues, we propose League of LLMs (LOL), a novel benchmark-free evaluation paradigm that organizes multiple LLMs into a self-governed league for multi-round mutual evaluation. LOL integrates four core criteria (dynamic, transparent, objective, and professional) to mitigate key limitations of existing paradigms. Experiments on eight mainstream LLMs in mathematics and programming demonstrate that LOL can effectively distinguish LLM capabilities while maintaining high internal ranking stability (Top-k consistency = 70.7\%). Beyond ranking, LOL reveals empirical findings that are difficult for traditional paradigms to capture. For instance, “memorization-based answering” behaviors are observed in some models, and higher in-family scores are found in the OpenAI model family (\Delta = 9, p < 0.05). Finally, we make our framework and code publicly available as a valuable complement to the current LLM evaluation ecosystem.
查看此版本解读 →查看此版本原文 ↗