Same Illness, Different Voice: Clinical Language Models Lose Accuracy When Patients Speak for Themselves
Prabhjot Singh, Pritam Deka, Vijay Chennareddy
Clinical language models are evaluated on vignettes written in a clinician's voice, but patients describe the same illness in their own. We pair 1,000 MedQA cases with three audited retellings (clinical, socioeconomic, cultural) and test 12 open models. Every model is less accurate when the patient speaks in a marked voice, by 9.6 points on average (95% CI 8.2 to 11.0), with no detectable shrinkage at larger scale. Matched controls trace the penalty. Generation temperature does not explain it, a plain first-person paraphrase that states every fact accounts for 2.4 points, and attaching the original vignette to the marked narrative recovers 7.0. We read this mainly as a failure to use facts expressed in patients' own words, not as a reaction to social cues, although our voices bundle both. Chain-of-thought and explicit debiasing instructions together close about a fifth of the gap. Having the same model first restate the account as a clinical vignette halves it and raises marked-voice accuracy by 3.9 points, whereas a structured-schema pipeline closes the same gap but is 7.1 points less accurate than the rewrite. Rewriting also invents plausible details, which we quantify. Code, data and analysis scripts: https://github.com/prabhjotschugh/narrativeshield-sdoh-medqa
尚无 AI 解读，展示原文摘要