Seeing the Options: What Should a Visual Candidate See?
Fengxin Li
Agents often choose among candidates that are images: elements of a screenshot, product photos, charts. A single-pass decision model either encodes the request and all candidates as one sequence, making its answer depend on candidate order, or encodes each candidate alone, which keeps order invariance and reusable encodings but hides the request from the candidate. We ask where the joint encoder's advantage comes from: from candidates seeing the request, or from candidates seeing each other while they are encoded. With one backbone and the same fine-tuning steps on screenshot crops of Multimodal-Mind2Web, the answer is mostly the request. Continuing each candidate from a forked cache of the encoded request recovers 11 to 13 of the 12 to 17 points that independent encoding loses to joint encoding, at the price of cross-request reuse, and we prove that order invariance stays exact whenever candidates never see each other. A joint encoder retrained with its candidates isolated from one another during encoding loses 1.1 and 0.4 ranking points in one paired run (0.7 and 0.3 against the mean of five joint runs), and permutation-equivariant interactions among candidates, over pooled vectors or inside the encoder, gain nothing beyond continued training. At a matched abstention operating point the state-conditioned model is 0.6 and 1.0 points below the joint encoder (mean of four runs) with a flip rate of zero, against 8-10% for the joint encoder; on AndroidControl crops (three and five runs) it is 2.5 to 4 ranking points below. Finally, the abstention threshold drifts: the choice term of the loss ignores the score level, and the coverage term pins it only weakly once positives saturate; a quantile threshold aligns operating points.
尚无 AI 解读，展示原文摘要