少数派报告：在真值数据标注中平衡成本与质量
Minority Reports: Balancing Cost and Quality in Ground Truth Data Annotation
Hsuan Wei Liao, Christopher Klugmann, Daniel Kondermann, Rafid Mahmood
AI 速读 基于摘要
该研究通过识别与多数投票标签不一致的“少数派报告”，探索在保证标注质量的前提下削减重复标注以降低成本。
cs.LG · cs.CV · cs.HC
查看原文摘要
Data annotation is an essential but costly aspect of developing machine learning-based technology, requiring multiple human workers to repeatedly label each data point in a data set. We explore the tradeoff between annotation accuracy and cost by identifying minority reports---worker responses disagreeing with the reference majority-vote label---that may indicate unnecessary redundancy, and can therefore be proactively pruned before they are executed. We develop a theoretical framework for choosing the number of repeated annotations required for a task and pruning policies at a target label quality. We estimate minority-report risk on two computer vision datasets re-annotated through a professional platform, then evaluate a rolling policy on the assignment stream to prune over 35% of repeats while preserving more than 99.9% of reference majority-vote labels.