检索增强生成中的上下文优化:梯度下降视角
In-Context Optimization for Retrieval-Augmented Generation: A Gradient-Descent Perspective
Mingchen Li, Jiatan Huang, Chuxu Zhang, Liang Zhao, Hong Yu
AI 速读 基于摘要
作者从梯度下降视角将RAG的上下文利用解释为一种上下文内优化,并提出一种轻量级前向更新方法,在冻结的RAG LLM上改进证据使用接口。
cs.CL
查看原文摘要
In-context learning has recently been linked to implicit gradient descent in linear self-attention models, suggesting that context can induce a forward-pass update. Retrieval-augmented generation (RAG) also relies on context, but retrieved documents are usually treated as static evidence rather than signals for adaptation. We study RAG as an in-context optimization process. First, we show that one linear self-attention layer can implement one gradient-descent step on a unified linearized RAG objective covering both projection-based and dot-product retrieval interfaces. This gives an exact regime where retrieval-augmented prediction and in-context optimization coincide. We use this result not as a literal model of LLM computation, but as a guide for adapting the interaction between queries and retrieved evidence. We then test the boundary of this correspondence: it remains stable under controlled linear extensions, but becomes feature-distribution dependent under nonlinear architectures. Finally, we turn this view into a lightweight method for frozen RAG LLMs. The method keeps the retriever and backbone fixed, and predicts a context-conditioned update to a generator-side evidence-use interface. Across seven QA benchmarks, two retrievers, and two frozen LLM backbones, this forward-only update improves a shared-interface baseline, transfers to held-out tasks, and approaches test-time gradient adaptation at much lower per-query cost.