arXiv:2410.23066 · v5
Don't Pay Attention, PLANT It: Pretraining Attention via Learning-to-Rank
摘要
State-of-the-art Extreme Multi-Label Text Classification models rely on multi-label attention to focus on key tokens in input text, but learning good attention weights is challenging. We introduce PLANT (Pretrained and Leveraged AtteNTion), a plug-and-play strategy for initializing the label-attention module that operates on top of an LLM's output representations. PLANT works by planting label-specific attention using a Learning-to-Rank model trained on mutual information gain, before end-to-end fine-tuning. This architecture-agnostic approach integrates with LLM backbones (e.g., Mistral-7B, Llama-3.1-8B, DeepSeek-V3, and Phi-3-mini). PLANT outperforms SOTA methods across tasks like ICD coding, legal topic classification, and content recommendation, and improves fine-tuned LLMs in controlled comparisons outside the medical domain. Gains are largest on rare labels, although a training-free LLM extraction baseline remains competitive on rarely seen labels. Ablation studies confirm that attention initialization is a key driver of these gains. We make our code available at https://github.com/deb-official/plant
作者备注:Accepted at NeurIPS 2026. v5 is the camera-ready version, revised after the NeurIPS reviews: new comparisons with fine-tuned LLMs outside the medical domain and with zero- and few-shot LLM baselines. 41 pages, 5 figures, 20 tables. Code: https://github.com/deb-official/plant