Multi-Estimate Elicitation Does Not Uniformly Debias Language Model Judgments
An-Sheng Jhang
原文摘要
Does eliciting five weighted estimates within a single response consistently improve numerical judgments? We compare direct and multi-estimate elicitation across anchoring, hindsight, base-rate, conjunction, and status-quo/default tasks in six frozen configurations. Each configuration contributes 618 admitted development units and 3,702 admitted formal units; the six-configuration formal cohort contains 22,212 units, including 21 missing outcomes. Of 45 preregistered confirmatory tests across five configurations, 12 reject on complete cases and 11 retain directional confirmation after the prespecified missingness-protection rule. Only one adjusted bound exceeds its smallest effect size of interest: gpt-oss-120b shows a 10.68-percentage-point increase in post-outcome confirmation error under multi-estimate elicitation (worst-case estimate 10.04 points). The remaining ten retained effects have sub-threshold point estimates; non-rejection does not establish equivalence. Joint presentation lowers signed conjunction excess in four confirmatory configurations, but this comparison changes both constraint visibility and call structure. Flash-high combines two disjoint execution-window batches under an approved continuation protocol, preserving their identities and non-causal interpretation. Post-hoc candidate-spread associations across all six configurations remain in-sample and require independent predictive validation. The completed cohort provides no evidence for uniform debiasing by multi-estimate elicitation. Materials: https://github.com/asj252/llm-judgment-studies
cs.CL · cs.AI