跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2610.16825 · v2

One Checkpoint, Many Runs: Recipe Cards for Fine-Tuned Models

首次提交 2026-10-12 22:08
版本更新 2026-11-18 13:09
雷达观测 2026-11-19 19:08
arXiv ↗PDF ↗

摘要

Fine-tuned language models are shared with model cards that describe the training setup in detail, often down to the default random seed, but report the score of a single released checkpoint. Such setups are retrained routinely, by fine-tuning services for each customer, by organizations at each update, and by users on their own data, so the model that is deployed comes from a new run of the setup. Some runs fail, and a failed run can go unnoticed and reach the people the model serves. We call such a setup a model recipe and argue that its documentation should state how often a run of the recipe misses a required performance bar. In a survey of the model cards of 300 fine-tuned language models on Hugging Face, none reports this failure probability, and only three report results from more than one training run. Across 2,124 fine-tuning runs, we find that reading the mean, the median, or a bootstrap interval as an approval often approves recipes whose failure rate exceeds a stated failure tolerance, because these summaries do not measure how often a run fails. Model cards do not report how often the runs of a recipe fail, although whoever runs the training is best placed to count them, while those who rely on a retrained model bear the cost of a failed run. To report this failure probability directly, we propose recipe cards, an addendum to model cards that documents the model recipe behind a checkpoint. A recipe card states the bar and the tolerance and who chose them, reports how many runs are made and how many of them miss the bar, gives an upper bound on the failure probability, and warns when this bound exceeds the tolerance. Recipe cards record who set and who accepted this risk, and let deployers and regulators state acceptance criteria for retrained models.

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

本站首次收录为 v2,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v2 · 2026-11-18 13:09 · 最新观测版本

One Checkpoint, Many Runs: Recipe Cards for Fine-Tuned Models

Fine-tuned language models are shared with model cards that describe the training setup in detail, often down to the default random seed, but report the score of a single released checkpoint. Such setups are retrained routinely, by fine-tuning services for each customer, by organizations at each update, and by users on their own data, so the model that is deployed comes from a new run of the setup. Some runs fail, and a failed run can go unnoticed and reach the people the model serves. We call such a setup a model recipe and argue that its documentation should state how often a run of the recipe misses a required performance bar. In a survey of the model cards of 300 fine-tuned language models on Hugging Face, none reports this failure probability, and only three report results from more than one training run. Across 2,124 fine-tuning runs, we find that reading the mean, the median, or a bootstrap interval as an approval often approves recipes whose failure rate exceeds a stated failure tolerance, because these summaries do not measure how often a run fails. Model cards do not report how often the runs of a recipe fail, although whoever runs the training is best placed to count them, while those who rely on a retrained model bear the cost of a failed run. To report this failure probability directly, we propose recipe cards, an addendum to model cards that documents the model recipe behind a checkpoint. A recipe card states the bar and the tolerance and who chose them, reports how many runs are made and how many of them miss the bar, gives an upper bound on the failure probability, and warns when this bound exceeds the tolerance. Recipe cards record who set and who accepted this risk, and let deployers and regulators state acceptance criteria for retrained models.

查看此版本解读 →查看此版本原文 ↗