← 研踪 PaperTrailarXiv:2608.09128 · v2
SocialGym: Evaluating LLM Social Intelligence through Verifiable Multi-Agent Games
首次提交 2026-08-10 13:12
版本更新 2026-11-09 17:19
雷达观测 2026-11-10 15:22
摘要
Large Language Model (LLM) agents are increasingly deployed in multi-agent social settings where they must cooperate, negotiate, and adapt to other agents. Measuring these social skills is hard because most social interactions offer no objective ground truth. Evaluations often fall back on LLM judges, which are costly, subjective, and noisy. We introduce SocialGym, an environment of 22 multi-agent social games, from matrix games to hidden-role deduction, with verifiable outcomes determined by game rules. We benchmark ten LLMs across all games and roles using an Elo tournament. Our results show that no model is best in every game, and rankings shift across game families and sides. Secret keeping correlates more strongly with theory-of-mind and privacy benchmarks than with general knowledge benchmarks. Half of the models give away their side's secret in Chameleon and Spyfall, and the smallest models are not reliably better than chance at voting for hidden opponents in most hidden-role games. Finally, we examine whether models can improve their play by reflecting on self-play and writing strategy playbooks, finding mixed results across models and games. SocialGym provides verifiable measures for diagnosing LLM social capabilities across settings and roles.
作者备注：Code and data: https://github.com/Keyu-He/Social_Gym_Spartan
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-09 17:19 · 最新观测版本
SocialGym: Evaluating LLM Social Intelligence through Verifiable Multi-Agent Games
Large Language Model (LLM) agents are increasingly deployed in multi-agent social settings where they must cooperate, negotiate, and adapt to other agents. Measuring these social skills is hard because most social interactions offer no objective ground truth. Evaluations often fall back on LLM judges, which are costly, subjective, and noisy. We introduce SocialGym, an environment of 22 multi-agent social games, from matrix games to hidden-role deduction, with verifiable outcomes determined by game rules. We benchmark ten LLMs across all games and roles using an Elo tournament. Our results show that no model is best in every game, and rankings shift across game families and sides. Secret keeping correlates more strongly with theory-of-mind and privacy benchmarks than with general knowledge benchmarks. Half of the models give away their side's secret in Chameleon and Spyfall, and the smallest models are not reliably better than chance at voting for hidden opponents in most hidden-role games. Finally, we examine whether models can improve their play by reflecting on self-play and writing strategy playbooks, finding mixed results across models and games. SocialGym provides verifiable measures for diagnosing LLM social capabilities across settings and roles.
查看此版本解读 →查看此版本原文 ↗