用可学习图位置编码扩展语言模型中的图推理
Scaling Graph Reasoning in Language Models with Learnable Graph Positional Encodings
Javier Porras-Valenzuela, Arush Arora, Alejandro Ribeiro
AI 速览：作者提出用可学习的、置换等变的图位置编码调制语言模型注意力，并以GNN加Graph Transformer计算编码，从而避免序列化邻接表、把上下文负担降到O(N)。
Solving reasoning tasks supported on graphs typically requires providing a Language Reasoning Model (LRM) with a serialized description of the graph. This approach results in the input length growing with the number of edges, which limits the size of the problems that can be solved. Furthermore, even when graphs fit in context, performance sharply decays as the graphs grow. We propose to address this by augmenting LRMs with scalable and permutation equivariant graph positional encodings that modulate attention logits. An additional post-fusion-style module incorporates structural information into hidden states. We compute these positional encodings using a GNN followed by a Graph Transformer. This eliminates the need for serialized adjacencies, reducing the context burden to $O(N)$. We jointly fine-tune the positional encoder and the LRM LoRA adapters using optimal solutions generated by graph-theoretic solvers, resulting in a computationally efficient method. Trained on small graphs, the model exhibits improved size-generalization across graph-theoretic tasks on graphs with up to 1,000 nodes, and produces fewer tokens than a conventional LRM. Additionally, we observe that our model remains robust to node relabelings.