Label-Wording Augmentation in a Typed-Decision Encoder: Gains on Unfamiliar Words, Not on Abbreviations or Codes
Vladimir Beskorovainyi
原文摘要
Typed-decision models score options that the caller supplies at request time, so their accuracy depends on label wording that the model's developer does not control. We compare two releases of a 307M-parameter multilingual cross-encoder, Ru-Jev, trained on thirteen Russian-language tasks; the later release adds an augmentation that presents labels in several surface forms. Holding questions fixed and varying only how eighteen routing labels are worded, we evaluate ten label vocabularies at five and eighteen candidates, checked for overlap with the training data. On held-out vocabularies made of ordinary or borrowed words (jargon, office slang, department names), the augmented release has higher point accuracy in all fourteen cells, by up to 0.24. Several of these gains rest partly on one label that the earlier release almost never chose; fitting one bias per option removes part of them, while the jargon gain largely survives. On abbreviations and opaque codes the augmented release gives no usable gain: its loss on 17 abbreviations unrelated to training (-0.060) is one label and disappears after the same correction. The comparison is between two trained releases, not a controlled ablation, and the earlier recipe varies across seeds by up to 0.098 on the strictest slice. On two datasets neither release saw, accuracy moves by -8.4 and +5.6 points, narrowed to -3.4 and +1.4 by the per-option biases; a second training run of each recipe, the augmented one under a rule recorded beforehand, puts the recipe gap on jargon far above seed noise and on a third unseen dataset, scientific rubrics, only just above it. Rankings of training checkpoints disagree across the three unseen datasets, so none of them should choose the checkpoint. We also report four evaluation hazards from comparing the two releases.
cs.CL · cs.AI · cs.LG