Learning a risk-sensitive individualized treatment rule (ITR) with patient covariates is challenging when the treatment criterion depends on the full conditional outcome distribution rather than only its conditional mean. Existing methods are largely criterion-specific, making it difficult to compare multiple treatment criteria within a common pipeline, such as quantiles, conditional value-at-risk (CVaR), and complex preference-based objectives such as cumulative prospect theory. We propose a distribution-first framework for risk-sensitive ITR learning that first estimates treatment-specific conditional outcome distributions and then constructs the target ITR by plug-in criterion evaluation. The proposed framework unifies treatment rule learning across multiple criteria and accommodates hard-to-optimize objectives within the same estimation pipeline. On the theoretical side, it yields a general error propagation result for locally Hölder-stable criteria, linking first-stage distribution estimation error to downstream criterion estimation and treatment regret, together with sharper regret bounds under standard margin conditions. Empirically, we show that the proposed framework outperforms direct criterion-specific methods in multiple settings and further evaluate it on the ACTG175 HIV clinical trial, a randomized study comparing several antiretroviral regimens in HIV-infected adults.
Gabriel Sargent, Will Wei Sun, Zhengwu Zhang, Yufeng Liu
原文摘要
Ranking large language models (LLMs) through human evaluations is essential for downstream decision-making. Because human feedback is expensive, evaluation platforms often collect it adaptively, prioritizing informative model comparisons using previously observed feedback. To further improve precision with limited human feedback, evaluators can draw on abundant unrated responses and relatively inexpensive LLM judges; however, judge bias and adaptive dependence challenge standard i.i.d. inference frameworks. We propose Prediction-Powered Ranking (PP-Rank), a framework that combines LLM-judge predictions with human feedback to improve precision while providing valid inference under adaptive sampling. Data-driven prediction weights minimize the asymptotic covariance trace within the proposed class. In theory, we establish asymptotic normality for general evaluation targets and simultaneous ranking inference. A joint martingale analysis handles dependence between adaptive human feedback and auxiliary data. We further derive a rule for reducing exploration that prioritizes lower-cost comparisons while matching a conservative variance bound, and characterize allocation limits for regular $\sqrt{T}$-rate estimation of pairwise means. Semi-synthetic experiments on UltraFeedback-Extended show improvements over human-only baselines in score estimation, confidence interval width, and ranking certification. Synthetic experiments demonstrate robustness to judge quality and adaptive allocation, while showing that PP-Rank can substantially reduce evaluation costs without sacrificing precision.
Changcheng Song, Yi Yang, Xiangyu Chang, Xiaoling Lu, Yufeng Liu
原文摘要
Fairness-aware classification seeks to balance predictive accuracy and fairness across demographic groups. Although surrogate losses facilitate the optimization of fairness-aware objectives, it remains unclear when minimizing these losses yields optimal decisions under the original fairness-aware risk. This paper investigates the statistical validity of surrogate-based fairness-aware classification under demographic parity and equal opportunity, considering both attribute-blind and attribute-aware settings. We introduce fairness-aware Fisher consistency and characterize it through conditions on the derivatives of the surrogate losses. The characterization shows that consistency depends not only on the individual loss functions but also on their interaction within the fairness-regularized objective. We further establish a calibration inequality that relates the excess fairness-aware risk to the excess surrogate risk, providing a quantitative connection between surrogate optimization and the original classification objective. Numerical experiments illustrate how deviations from the consistency conditions affect fairness-aware excess risk and how the regularization parameter influences the well-posedness of surrogate optimization. An application to consumer credit default prediction demonstrates that appropriately calibrated fairness regularization can substantially reduce demographic disparities while maintaining predictive performance.