acceptodds
Under review as a conference paper at ICLR 2027

Learning Retrieval Refinement from Generation Gradients

Abstract

Retrieval-Augmented Generation (RAG) systems are fundamentally constrained by retrieval quality, yet user queries often fail to capture the evidence required for accurate downstream generation. We observe that LLM generation gradients inherently encode missing semantic information in retrieved context, providing a useful signal for constructing retrieval refinement pseudo-labels from answer supervision. Based on this insight, we propose Gradient-Driven Query Embedding Refinement (GDER), a novel framework that reformulates query optimization as gradient-driven embedding refinement in dense retrieval space. Instead of costly autoregressive query rewriting, GDER learns lightweight retrieval-space correction offsets directly from generation gradients, substantially reducing both training and inference overhead. GDER requires no modification to either the underlying LLM or retriever and can be seamlessly integrated into existing RAG pipelines. Experiments across multiple open-domain question answering benchmarks demonstrate that GDER consistently improves both retrieval and answer quality while achieving significantly higher efficiency than existing query optimization methods. The model, data, and code are available at https://anonymous.4open.science/r/gder-60EB.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Information Retrieval, LLM, RAG, Retrieval Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5UmSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GDER, a method that refines dense-retrieval query embeddings using signals derived from LLM generation gradients. In an offline stage, it computes the answer cross-entropy loss given top-5 retrieved documents and backpropagates it to the LLM input embeddings, averaging over answer positions. It then scores each query token by the inner product of its LLM input embedding with this aggregated gradient. The top-K tokens are re-encoded with BGE, softmax-weighted into a direction d_grad, and added to the query embedding. The resulting shift e*_q − e_q is kept as a pseudo-label only if re-retrieval reduces the generation loss by at least τ; otherwise the target is zero. A small MLP is trained to predict this offset from the LLM’s final-layer hidden state at the query position, using a cosine loss for positive samples and an L2-norm penalty for zero samples. At inference, the predicted offset’s norm gates whether to re-retrieve. Training data comes from 10k HotpotQA samples with Qwen2.5-7B. The method is evaluated on HotpotQA, 2WikiMultiHopQA, MuSiQue and StrategyQA with three generator LLMs, reporting gains of about 1–3 points over Naive RAG and over prompt-rewriting, LeReT, DeepRetrieval, Dense PRF and HyDE. It also reports retrieval metrics, ablations (random perturbation, no gradient, learned space alignment), and lower latency than autoregressive rewriting methods.

Critical strengths
  1. GDER modifies neither the retriever nor the generator, requires no autoregressive rewriting, and adds about 233 ms on average with a small MLP. For deployed RAG systems, an embedding-space correction that is learned once offline and gated per query is an attractive engineering point compared with RL-trained query rewriters.

  2. Turning answer-level loss into retrieval-space pseudo-labels, verified by an actual loss reduction after re-retrieval, is a reasonable and cheap way to obtain supervision. The offline cost is modest.

  3. The random-perturbation and no-gradient ablations, the gate-threshold sweep, and the triggered/skipped subset breakdown help show where the gains come from. The method also transfers from HotpotQA-only training to other datasets and generator backbones.

Critical weaknesses
  1. The paper claims gradients “point toward missing semantic information.” However, d_grad is a weighted combination of BGE embeddings of tokens already in the query, so the method can only re-emphasize existing query terms, not add missing content. In effect, it is gradient-saliency-based term reweighting, a much narrower contribution than the framing suggests, and it is not compared to simple term-weighting baselines.

  2. The paper does not state how the gold and document representations are computed, in which space the cosine is measured, or on how many samples. Random documents show higher alignment than top-2 retrieval, which would be expected trivially because random documents are farther from gold. The figure therefore does not establish that gradients encode useful retrieval directions.

  3. On HotpotQA, Recall@5 rises from 70.00 to 79.77 overall, yet only about 16% of queries trigger refinement and only about 14% of documents change in Table 8. This implies roughly a 60-point recall gain on the triggered subset. Meanwhile, the QA accuracy gain is only 2.1 points. This needs explanation and verification.

  4. Improvements are typically 1–3 points from a single run, with no variance or significance tests. The paper does not compare with retriever fine-tuning from LM feedback (e.g., REPLUG-LSR, Atlas-style training), rerankers, the closely related gradient-based GRO-RAG (Chen et al., 2026), or iterative multi-hop retrieval methods (e.g., IRCoT), even though three of the four benchmarks are multi-hop.

Questions for authors
  1. Can GDER ever introduce information absent from the query text? How does it compare with a simple baseline that reweights query tokens, for example by IDF, by LLM attention, or by the best K-token subset found through brute-force search with the same loss check?

  2. Please clarify the scoring in Eq. (3): why dot answer-position gradients with query-token input embeddings, and why select the maximum of ⟨E, g⟩ rather than of ⟨E, −g⟩? What happens if the sign is flipped?

  3. How was Figure 1(c) computed (spaces, representations, sample size)? What does the same plot look like for a random-direction or random-token baseline?

  4. Can you explain the Recall@5 gain of +9.77 on HotpotQA given a 16% trigger rate and 14% document change rate? Please report retrieval metrics separately for the triggered subset.

1 comment
Reviewer_5UmSOpenReview ↗

Response to Authors

Thank authors for the response. I still have a few concerns remaining:

  1. Framing. You confirm that GDER adds no information beyond the query, yet the paper still claims that gradients encode "missing semantic information". The IDF and attention baselines also seem to skip the gate and the loss check; please run them through the same pipeline.
  2. Fig. 1(c). For random Gaussian directions in 3584 dimensions, the cosine should have mean ≈ 0 and std ≈ 0.017, not the reported 0.11–0.21 / 0.37–0.41. Please explain how they were generated.
  3. Recall@5. On the changed queries, recall jumps from 17 to 86, yet accuracy on the triggered subset stays well below the unchanged subset (36.8 vs. 45.5). Please report accuracy on the changed subset and a few examples.
  4. Baselines. Were the new baselines rerun in your setup? GRO-RAG uses web retrieval, and IRCoT scoring below Naive RAG needs explanation.
Reviewer_PCBFOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

GDER uses generation feedback to learn corrections to query embeddings for dense retrieval. During offline target construction, it computes answer-loss gradients, ranks query tokens by their alignment with an aggregated gradient, and re-encodes the selected tokens to obtain a correction direction. A second retrieval checks whether the correction improves generation loss enough to retain it; other examples receive zero-offset targets. A lightweight network then learns to predict offsets from LLM hidden states. At inference, the predicted norm determines whether to retrieve again. The experiments assess QA quality, transfer, ablations and latency in several RAG settings.

Critical strengths
  1. The offline construction avoids using the target answer or a backward pass at inference. This makes the feedback practical to use through a small predictor. The reported QA and transfer results suggest that the complete pipeline can retrieve more useful evidence.

  2. The loss-gain filter and zero-offset targets give the model a way to abstain when a proposed correction is unhelpful. I also appreciate the latency measurements and the controls for non-gradient weighting and cross-space alignment. They help examine the refinement beyond the main quality comparison.

Critical weaknesses
  1. Accuracy needs an explicit definition. Table 1 gives Qwen2.5-14B on HotpotQA 49.80 Accuracy and 46.68 token F1; some other settings have larger gaps. To interpret this ordering, the authors need to clarify whether Accuracy is answer-level EM and whether both metrics use the same predictions, references, answer extraction, normalization and averaging. The numbers alone do not establish an evaluation error. Until the authors specify the metric, answer extraction and evaluation implementation, I cannot reliably interpret the quality gains.

  2. The paper has not shown what the gradient-derived targets add. https://aclanthology.org/2024.naacl-long.463/ and https://aclanthology.org/2024.acl-long.194/ already use generation feedback for retrieval.

Questions for authors

Do gradient-derived targets outperform non-gradient proposals when each alternative gets its own trained predictor, with the same data, proposal count, loss-gain filter, predictor capacity and inference budget? Please explain the advance over existing feedback-supervised retrieval and lightweight embedding adaptation. Why should the positive-alignment token ranking identify information missing from retrieval? Please distinguish the empirical support for this step from what follows from the Taylor argument.

Reviewer_HSbtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper utilizes the gradient generated by a large model when generating answers as a supervisory signal to train a lightweight network to optimize the retrieval vector, enabling the RAG system to find the required knowledge more accurately and efficiently. The author's experiments on four question-answering benchmark datasets and three generation models demonstrate that this method can improve retrieval quality and downstream question-answering accuracy. Furthermore, the author also proves the effectiveness and computational efficiency of the method through ablation experiments.

Critical strengths
  1. The method utilizes a large model to generate gradient guidance for retrieval optimization, combining the retrieval process with downstream generation tasks, providing a new approach.

  2. GDER outperforms the basic RAG method on multiple datasets and achieves better comparative results. At the same time, it has certain efficiency advantages.

Critical weaknesses
  1. The paper has not yet fully demonstrated that the generated gradient can identify the semantic information missing in retrieval, and further theoretical or experimental analysis is needed.

  2. The experiment only uses the BGE-base retrieval model, and its generalization ability remains unclear. Furthermore, some baselines do not utilize the same supervisory information, raising doubts about the fairness of the comparison.

Questions for authors
  1. How to distinguish whether the gradient reflects the lack of retrieval information or the difficulty of answer prediction?

  2. Is this method still effective on E5 or larger-scale BGE models?

  3. Can you verify the independent contribution of gradients by comparing it with methods that do not use gradients under the same supervised conditions?

  4. Can you provide a complete comparison of training costs to verify the overall efficiency advantage?

  5. How sensitive is the method to the threshold?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.