← 研踪 PaperTrailarXiv:2608.29455 · v2
Item-Mean Surrogates: Why Richer Personas Fail to Deliver Individual Fidelity in LLMs as Human Surrogates
首次提交 2026-08-30 06:17
版本更新 2026-11-02 08:27
雷达观测 2026-11-04 23:42
摘要
LLMs are increasingly used to simulate and study human behavior. Such studies assume richer personas improve predictions of specific people's answers. We test this premise across four public datasets. First, we split each person's answer to an item into the item mean (the average human answer to that item) and the person's deviation from the item mean. Much of the accuracy that surrogate studies report comes from predicting item means. LLM predictions explain little of the deviations. We call this pattern item-mean surrogacy. Second, we split each deviation into stable signal and noise using a dataset in which the same participants answered the same questions twice. Stable signal is the part of a deviation that repeated answers share. Noise is the part that changes. In this dataset, more than half of the variance in deviations persists across occasions. Most of what the LLM misses is stable signal rather than noise. Third, we split stable signal into the person signal, a person's average deviation across questions, and the person-by-item signal, their remaining question-specific deviations. The person-by-item signal accounts for about 8.5 times as much variance as the person signal. The LLM recovers about a tenth of it. Richer personas, different models, and fine-tuning on human answers still leave most of the person-by-item signal unrecovered. Both fine-tuned models recover less of this signal than their untuned versions. These findings call for closer empirical scrutiny of LLMs as human surrogates.
作者备注：43 pages, 7 figures, 20 tables. Includes Supporting Information (main text pages 1-11; SI Appendix pages 1-32)
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-02 08:27 · 最新观测版本
Item-Mean Surrogates: Why Richer Personas Fail to Deliver Individual Fidelity in LLMs as Human Surrogates
LLMs are increasingly used to simulate and study human behavior. Such studies assume richer personas improve predictions of specific people's answers. We test this premise across four public datasets. First, we split each person's answer to an item into the item mean (the average human answer to that item) and the person's deviation from the item mean. Much of the accuracy that surrogate studies report comes from predicting item means. LLM predictions explain little of the deviations. We call this pattern item-mean surrogacy. Second, we split each deviation into stable signal and noise using a dataset in which the same participants answered the same questions twice. Stable signal is the part of a deviation that repeated answers share. Noise is the part that changes. In this dataset, more than half of the variance in deviations persists across occasions. Most of what the LLM misses is stable signal rather than noise. Third, we split stable signal into the person signal, a person's average deviation across questions, and the person-by-item signal, their remaining question-specific deviations. The person-by-item signal accounts for about 8.5 times as much variance as the person signal. The LLM recovers about a tenth of it. Richer personas, different models, and fine-tuning on human answers still leave most of the person-by-item signal unrecovered. Both fine-tuned models recover less of this signal than their untuned versions. These findings call for closer empirical scrutiny of LLMs as human surrogates.
查看此版本解读 →查看此版本原文 ↗