Learning Retrieval Refinement from Generation Gradients

Retrieval-Augmented Generation (RAG) systems are fundamentally constrained by retrieval quality, yet user queries often fail to capture the evidence required for accurate downstream generation. We observe that LLM generation gradients inherently encode missing semantic information in retrieved context, providing a useful signal for constructing retrieval refinement pseudo-labels from answer supervision. Based on this insight, we propose Gradient-Driven Query Embedding Refinement (GDER), a novel framework that reformulates query optimization as gradient-driven embedding refinement in dense retrieval space. Instead of costly autoregressive query rewriting, GDER learns lightweight retrieval-space correction offsets directly from generation gradients, substantially reducing both training and inference overhead. GDER requires no modification to either the underlying LLM or retriever and can be seamlessly integrated into existing RAG pipelines. Experiments across multiple open-domain question answering benchmarks demonstrate that GDER consistently improves both retrieval and answer quality while achieving significantly higher efficiency than existing query optimization methods. The model, data, and code are available at https://anonymous.4open.science/r/gder-60EB.