Md Tanveer Hossain Munim, Bijoy Ahmed Saiem, Muhammad Abdullah Adnan
原文摘要
Open-vocabulary segmenters accept class names at inference. Conformal risk control can bound how often their prediction sets miss the true label, but we show that this bound breaks when names are added after calibration. On ADE20K, a threshold calibrated for 10% risk at 150 names reaches up to 15.4% risk at 845 names in released CAT-Seg models. The risk exceeds the target on every calibration split, yet the prediction sets grow far less than the target requires. In SED they instead grow larger than needed. The cause is that pruning, cross-label attention and vocabulary normalisation make each label's score depend on the other requested labels. Scores can move either way, so a threshold recalibrated at a larger vocabulary need not hold at smaller ones. We remove this dependence without training by scoring each label against a fixed reference bank of names. A label's score then stays the same when names are added, so one calibration keeps its guarantee as the vocabulary grows. The guarantee holds whatever names are added, even ones chosen per image after seeing it. On three benchmarks and five segmenters, the converted models meet the target at every vocabulary size we evaluate. For CAT-Seg and SED at alpha=10%, they use at most one more label per pixel than the released models recalibrated at the grown vocabulary. Treating names instead of images as calibration samples also bounds the miss rate of added names. If the annotated names are a random sample of the names in the calibration images, an added name's expected miss rate on those images is at most the target. For harder added names, the excess is at most their gap in mean miss rate. Applied to Grounding DINO, which splits large vocabularies into chunks, the conversion also makes inference about six times faster.
Many benchmarks for rare events score a prediction in three steps: pool the prediction and the target over spatial blocks, threshold each pooled field, and score the resulting contingency table. We show that at a fixed rare threshold the max-pooled Critical Success Index (CSI) cannot separate two different things: how well a system places events, and how well its amplitudes are calibrated. Sharp observations push many blocks above the threshold, while the attenuated predictions of squared-error regression leave those same blocks below it. A monotone calibration map is fitted on held-out data and applied separately to each system. The map needs no retraining and cannot change the order of any two pixels, so whatever part of a difference it reproduces cannot be evidence of better spatial ranking. We report how much of a reported difference the map removes. On SEVIR, two released checkpoints of one architecture differ by -29.5% in extreme-threshold CSI before the control and by +5.3% after it. Across 450 pairwise contrasts among 6 systems, how far apart two systems sit in pooled frequency bias predicts how far their CSI difference moves under the control (r = 0.896 when the map is fitted after pooling), and 64 contrasts reverse sign. At CasCast's published extreme-event operating point, calibration alone removes 88%-106% of the cascade's gain over its backbone across three controls. Calibration also reveals advantages hidden by a better-calibrated baseline. On geostationary infrared imagery, a larger share of the score can be explained by calibration as events get rarer. The problem therefore needs two things at the same time: a threshold that is fixed in advance, and a model whose output is poorly calibrated at that threshold. We recommend reporting pooled frequency bias and a symmetric held-out FreeKnob Audit alongside rare-event pool-and-threshold scores.