Marco Gutierrez, Xinyi Leng, Hannah Cyberey, Jonathan Richard Schwarz, Ahmed Alaa, Thomas Hartvigsen
原文摘要
Language model benchmarks are pervasive and computationally-efficient proxies for real-world downstream performance. However, many recent works find that benchmarks often fail to predict downstream utility. While some works have begun diagnosing sources of misalignment, there remain no ways to systematically update benchmarks to align their scores with downstream usage. Towards bridging this gap, we introduce and study \textit{benchmark alignment}, where we use information about downstream model performance to automatically update benchmarks, specifically aiming to update static benchmarks so they generalizably rank models according to new pairwise preferences. Our experiments involving 4576 language models and 6 benchmarks show that reweighting benchmark items can successfully rank unseen models, even generalizing across model scales in most cases. And while naive alignment unsurprisingly requires large numbers of models and benchmark questions, an oracle experiment suggests this could be reduced to as few as 20 well-chosen models. Overall, our work takes a step towards efficiently aligning benchmark development with downstream tasks. All of our code, models, and data are publicly-available: https://github.com/hartvigsen-group/benchalign
Jiaxing Qiu, Luke Hakso, Roxana Daneshjou, Ahmed Alaa, Thomas Hartvigsen
原文摘要
Model editing aims to correct errors in large, pretrained models by replacing model generations with user-provided responses. Existing editors assume a single target answer for each edit. However, many real tasks inherently entail uncertainty, resulting in multiple plausible answers with varying likelihoods in user feedback. Such feedback forms a target distribution to not only correct but also calibrate model generations. We therefore propose SoftEdit, the first model editor to leverage uncertainty in user responses and perform one-shot calibration during editing. To generalize calibration to unseen samples, SoftEdit uses co-plausibility relationships between answers while preserving real distributional shifts, balancing the new distributional generality-locality tradeoff. Across three language models, three vision-language models, and two large datasets, SoftEdit achieves state-of-the-art performance, ultimately showing that uncertainty in user responses enables generalizable calibration of pretrained models beyond single-answer corrections.
Deep models for radiological diagnosis are typically trained to predict disease labels directly from images, which limits them to diseases seen during training and makes their decisions hard to interpret in radiologists' terms. Radiology, however, proceeds from visual findings to disease through published guidelines that make this decision rule explicit, interpretable, and reproducible. We introduce diagnosis by composition (DbC), a representation learning framework that encodes this findings-first logic. DbC learns a vocabulary of radiological findings by aligning image features with the text embeddings of the findings, each finding represented as a dense map over the scan. A disease is defined by a set of required findings derived from its guidelines, and its diagnosis map is a fixed, parameter-free composition of the corresponding finding maps. Because the finding-to-disease mapping is derived from guidelines rather than learned, DbC is interpretable and auditable by design, can extend to new diseases by specifying their findings without retraining, and can diagnose unseen diseases by composing their findings. On multi-phase liver CT, DbC detects and localizes unseen diseases, rare diseases from a different institution, and an external cohort significantly better than all comparators, including models that align images with disease names. DbC also improves detection and localization in the fully supervised setting and remains superior across training-set sizes. Although demonstrated on liver CT, DbC applies wherever radiological guidelines define diseases through findings.