跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2606.04326 · v2

Measuring What Matters: Synthetic Benchmarks for Concept Bottleneck Models

首次提交 2026-06-03 09:01
版本更新 2026-11-18 03:59
雷达观测 2026-11-19 19:08
arXiv ↗PDF ↗

摘要

Concept bottleneck models predict using high-level concepts detected in inputs. Although these models provide a simple way to reap benefits from interpretability, they are hard to train and evaluate because very few real-world datasets include concept labels. In this paper, we develop synthetic benchmarks for concept bottleneck models. With our benchmarks, researchers can generate labeled datasets and control properties that affect performance, such as data modality, the choice of concepts, or annotation quality. They can thus find problems that suit these models and isolate the factors behind their failures. We demonstrate how our benchmarks can be used to evaluate major use cases for concept bottleneck models: decision support, i.e., helping human experts make better decisions; and automation, i.e., handling routine tasks without supervision. Our demonstrations show how the benchmarks can identify new failure modes and attribute them to either the architecture or the data.

作者备注:Camera-ready version, NeurIPS 2026 (Evaluations and Datasets Track). Benchmarks available at https://github.com/ustunb/concept-benchmark

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

本站首次收录为 v2,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v2 · 2026-11-18 03:59 · 最新观测版本

Measuring What Matters: Synthetic Benchmarks for Concept Bottleneck Models

Concept bottleneck models predict using high-level concepts detected in inputs. Although these models provide a simple way to reap benefits from interpretability, they are hard to train and evaluate because very few real-world datasets include concept labels. In this paper, we develop synthetic benchmarks for concept bottleneck models. With our benchmarks, researchers can generate labeled datasets and control properties that affect performance, such as data modality, the choice of concepts, or annotation quality. They can thus find problems that suit these models and isolate the factors behind their failures. We demonstrate how our benchmarks can be used to evaluate major use cases for concept bottleneck models: decision support, i.e., helping human experts make better decisions; and automation, i.e., handling routine tasks without supervision. Our demonstrations show how the benchmarks can identify new failure modes and attribute them to either the architecture or the data.

查看此版本解读 →查看此版本原文 ↗