GenPT: Beyond Self-Report for Reliable LLM Psychometrics via Generative Projective Testing
Ming Wang Singapore Management University and Northeastern University, Shuang Wu Northeastern University, Bixuan Wang, Lu Lin, Yuxin Chen Central University of Finance and Economics, Xiaocui Yang Northeastern University, Daling Wang, Shi Feng Northeastern University, China 等 10 位作者
Self-report questionnaires remain the default tool for probing the psychological characteristics of Large Language Model (LLM) agents, yet classical instruments (BFI, BDI, MBTI, BSS) inherit three well-known threats under LLMs: contamination from training corpora, directional bias under social-desirability framing, and limited responsiveness to context beyond the item text. We ask whether a projective paradigm can be adapted into a usable psychometric tool for LLM agents. We introduce GenPT (Generative Projective Testing), which reformulates TAT, Rorschach, and SCT with newly generated stimuli and organises assessment as a three-stage pipeline (Behavior Collection \to Interpretation \to Diagnosis) grounded in SCORS-G and a Simplified Rorschach Analysis System. On personality traits (Big Five, MBTI) and mental-health risks (depression, suicide ideation), questionnaires exhibit systematic directional shifts under social-desirability framing, most strongly on suicide ideation, whereas GenPT’s collected behavioral patterns stay near the symmetric baseline; under a longitudinal counselling context, GenPT-based depression assessment shifts by roughly an order of magnitude more than its questionnaire counterpart. Questionnaires remain competitive on clean-persona trait tasks where items align lexically with the persona description. Overall, GenPT complements rather than replaces self-report when contamination resistance, bias asymmetry, and context sensitivity matter. Code and stimuli: https://github.com/sci-m-wang/GenPT.
尚无 AI 解读，展示原文摘要
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · 2026 · 正式发表