When Can You Trust Your Synthetic Users? Diagnostics and Corrections for LLM Consumer Panels
Robson Tigre, Hugo Gobato Souto
Large language models are increasingly used to simulate consumers, yet agreement with aggregate survey outcomes can conceal errors in the subgroup estimates that guide decisions. We develop a framework for deciding when to trust, correct, or abandon a synthetic consumer panel by distinguishing population mismatch from differences in conditional responses. The framework combines diagnostics for covariate overlap, conditional calibration, and stability across models with an augmented inverse probability weighting estimator that treats LLM predictions as auxiliary information. Correction requires a complete invitation frame, response indicators for everyone invited, and human outcomes for respondents, together with assumptions of missing at random and positivity. We evaluate the framework using simulations and controlled nonresponse experiments constructed from paired human and LLM responses in ANES and Twin-2K-500. In identified simulation regimes, point bias remains small when either the response model or the outcome model is correct, but nominal 95\% confidence intervals cover only 88--94\% in some regimes. When nonresponse depends on missing human outcomes, substantial bias persists and coverage approaches zero. Controlled masking of the paired datasets reproduces this contrast between successful point correction under imposed missing at random and failure under nonignorable nonresponse. These findings show why validation must address the intended population and task, and why the scope for correcting synthetic estimates depends on the available human data and the assumptions that identify the target.
尚无 AI 解读，展示原文摘要