Total Simulated Survey Error: Designing and Diagnosing Survey Responses from Large Language Models
Indira Sen, Georg Ahnert, Leah von der Heyde, Jana Lasser, Bernd Weiß, Markus Strohmaier
原文摘要
Large Language models (LLMs), having been trained on vast amounts of human-generated data, may encode the opinions, attitudes, and behaviors of these humans. As such, LLMs show promise in mimicking human-like patterns that facilitate their use in simulating people in a wide variety of contexts. One such context is using LLMs as 'silicon samples', i.e., proxies of people in answering survey questions to establish public opinion, design policies, or use as (social) scientific data. However, several critical questions of social biases, generalization, and technical limitations remain, further complicated by a vast design space open to simulation designers. In addition, LLM-simulated surveys often lean on existing survey infrastructure, e.g., survey questions and samples. These artifacts might not be suitable for LLMs or themselves might have errors that are then inherited by the LLM-powered survey simulations. As a result, we lack a systematic understanding of the design space of LLM-generated surveys as well as how these decisions interplay with inherent LLM limitations. In this article, we aim to systematically identify, trace, and document limitations in LLM-generated survey responses. Building on traditions in the quantitative social sciences, specifically survey methodology and measurement theory, we investigate threats to the validity of LLM-generated survey responses. To do so, we design a framework that enumerates conceptual errors and systematic biases that can occur at different stages of the survey simulation lifecycle. Our framework - called the Total Simulated Survey Error (TS2E) Framework - provides a unified and end-to-end perspective on errors in LLM-generated survey data. We illustrate the framework through a theoretical and empirical case study, which enables survey simulation designers to systematically understand, identify and reflect on errors in LLM-generated surveys.
cs.CY