acceptodds
Withdrawn from ICLR 2027

When Does RAG Need Structure? Enhancing Evidence Retrieval via Selective Structural Calibration

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Retrieval-augmented generation (RAG) grounds large language model outputs in external knowledge. Graph structures capture relations among candidates, but directly adding structural signals can disrupt a reliable relevance-based ranking. To address this limitation, we propose TRACER (Task-Conditioned Readout and Anchored Calibration for Evidence Retrieval), a selective structural calibration framework. TRACER merges candidates from embedding and structural retrieval and represents them with simple score-, rank-, source-, and propagation-based features. A lightweight task-conditioned reranker then adjusts a frozen base score through bounded residual and within-query reordering branches while discouraging unnecessary changes. Under our controlled evaluation protocol, TRACER ranks first on every reported retrieval metric and yields the highest average question-answering performance; on 2WikiMultihopQA, its N@5 improves by about 12% relative to the strongest baseline. With candidate access fixed, TRACER exceeds a matched query-adaptive fusion control by 7.96 average N@5 points. Controlled decompositions identify structural evidence and within-query reordering as the primary sources of retrieval improvement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Retrieval-Augmented Generation, Graph Structure, Selective Structural Calibration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SocHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how structural information should be incorporated into retrieval-augmented generation (RAG) without unnecessarily disrupting reliable semantic rankings. The authors propose TRACER, a selective structural calibration framework that combines candidates from dense and structure-based retrieval and constructs compact candidate, structural, and task-level features.

TRACER employs a task-conditioned multi-head readout to extract candidate-specific structural signals. These signals are subsequently used to adjust a frozen base score through two complementary components: a bounded residual correction and a within-query reordering branch. The training objective combines ranking losses with regularization terms designed to discourage excessive deviations from the base ranking.

Experiments on LongMemEval, 2WikiMultihopQA, and MuSiQue demonstrate improvements in retrieval and downstream QA performance. The authors further provide matched candidate-pool comparisons, component ablations, structural-noise analysis, and runtime measurements.

Critical strengths

S1. A practically relevant formulation of structure-aware retrieval.

The paper addresses an important limitation of existing structure-enhanced RAG systems: structural connectivity does not necessarily imply relevance to the current query. Rather than treating structural scores as uniformly beneficial, TRACER formulates their integration as candidate-specific corrections to a semantic ranking. This perspective is meaningful, particularly for multi-hop retrieval and long-term memory applications where relational evidence may be necessary but structural noise can also be harmful.

S2. Strong empirical results with useful controlled analyses.

The reported retrieval and QA improvements are substantial, particularly on 2WikiMultihopQA. The authors make a valuable effort to separate improvements due to candidate coverage, adapted base scoring, and reranking. On the fixed candidate pool, TRACER improves average NDCG@5 from 71.07 for the matched QuDAR-style fusion control to 79.03. The component ablations and structural corruption experiments also provide useful evidence about the contributions and limitations of the proposed framework.

Critical weaknesses

W1. Insufficient validation of the proposed architecture's necessity.

The ablation results suggest that most improvements come from structural features and within-query reordering, while task-conditioned readout contributes relatively little. Although the matched fusion comparisons are useful, the paper lacks feature-matched supervised reranking baselines, such as MLP or LambdaMART. This makes it difficult to isolate the contribution of the proposed architecture beyond conventional learning-to-rank methods.

W2. Limited evidence for reliable selective calibration.

Although TRACER aims to preserve reliable base rankings, the within-query reordering branch is not explicitly bounded, and its strength is controlled by a global scalar rather than a query-specific gate. The performance degradation under severe structural corruption further raises concerns about robustness. More systematic evidence is needed to demonstrate that TRACER avoids unnecessary or harmful ranking changes.

Questions for authors

Q1. How much of TRACER's improvement is attributable to its proposed architecture rather than structural features and supervised reranking?

Table 5 suggests that structural features and within-query reordering account for most of the performance gains, while task conditioning and other calibration components contribute comparatively smaller improvements. Could the authors include feature-matched supervised learning-to-rank baselines, such as an MLP reranker or LambdaMART, using identical candidate pools, structural features, and training supervision? This would help clarify the incremental contribution of the proposed architecture beyond conventional supervised reranking.

Q2. How effectively does TRACER preserve reliable base rankings while correcting unreliable ones?

Could the authors provide a query-level analysis distinguishing cases where the frozen base ranking already contains all gold evidence within the top five from those where evidence is missing? In particular, how frequently does TRACER degrade an already correct ranking, and how effectively does it repair incorrect rankings with high base-score margins? An ablation of the confidence-weighted regularizer would also help establish its contribution to selective calibration.

Q3. How sensitive is TRACER to candidate-pool size and structural evidence coverage?

The main experiments use top-512 dense and structural candidate lists, with the structural pool achieving 100% gold-evidence coverage on 2WikiMultihopQA. How does TRACER perform with smaller candidate budgets, such as top-128 or top-256, or when structural evidence coverage is reduced? Do its advantages over matched fusion baselines and the relative contributions of its calibration components persist under these conditions?

Q4. To what extent are the dataset-specific feature representations transferable to new structural backends or domains?

Although TRACER uses a shared architecture, the feature-construction functions (F_\phi), (F_\psi), and (F_t) rely on dataset-specific structural statistics. Could the authors clarify which feature families are essential and which require backend-specific engineering? How much feature redesign or retraining would be required when applying TRACER to a new domain with a different structural retrieval pipeline

Reviewer_yD36OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TRACER, a selective structural calibration framework that uses task and candidate context to modulate how structural evidence revises the base ranking. Specifically, TRACER merges candidates from embedding and structural retrieval and represents them with simple score-, rank-, source-, and propagation-based features. Meanwhile, a lightweight task-conditioned reranker is used to adjust a base score by residual and within-query reordering branches while discouraging unnecessary changes. Extensive experiments show the effectiveness of the proposed method.

Critical strengths
  1. The research problem in this paper is well-motivated, which aims to address the limitation for structure-based retrieval methods.
  2. This paper conducts extensive experiments, which show the effectiveness of the proposed method.
  3. This paper introduces a lightweight reranker, which may be efficient.
Critical weaknesses
  1. The novelty of this paper seems limited, as TRACER primarily combines embedding-based and structure-based retrieval results through a learned reranking mechanism, without introducing substantial methodological innovations.
  2. As shown in Table 5, the proposed method involves too many designed components. It would be beneficial to clarify the necessity of each component.
  3. In experimental section, this paper claims to run the experiments 5 times. It would be beneficial to include the statistical significance tests for experimental results as the evaluation dataset size is small.
Questions for authors

Please refer to weaknesses section

Reviewer_FPPZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TRACER is a lightweight, structure-aware reranking framework for retrieval-augmented generation. Rather than constructing another graph encoder or replacing the base retriever, TRACER merges candidates from a dense retrieval path and a structural retrieval path, summarizes them with compact retrieval statistics, and learns a candidate-specific correction around a frozen base score. The correction has two branches: a bounded residual in base-score space and a within-query reordering term. A task-conditioned multi-head readout uses the task state as a query and candidate structural states as keys to produce candidate-specific attention weights, which are concatenated with local candidate representations before calibration.

The paper formulates evidence retrieval as selective structural calibration, trains TRACER with listwise and pairwise ranking losses plus anchor and confidence regularizers, and evaluates on LongMemEval, 2WikiMultihopQA, and MuSiQue. It reports first-place retrieval and QA results under a controlled protocol, decomposes gains into candidate access, base-score adaptation, and reranking, and provides component ablations, structural-noise robustness tests, and CPU query-time measurements.

Critical strengths

1.The candidate-specific calibration design is well motivated and cleanly decomposed. TRACER does not replace the frozen base score; it learns a bounded residual plus a within-query reordering term, with an explicit anchor regularizer. This is consequential because it addresses a real failure mode of structure-enhanced retrieval: structural association does not necessarily imply query relevance, and a highly activated candidate may be a necessary bridge or may introduce redundant evidence. The bounded correction and confidence-weighted anchoring are a sensible way to let structure revise the ranking without letting it override a confident base ranking.

2.The controlled decomposition and ablations substantially support the empirical claims. The paper separates candidate access, base-score adaptation, matched fusion, and component-level reranking, and reports that full TRACER gains 13.48 N@5 points over the frozen adapted base scorer and 7.96 over a matched QuDAR-style fusion control. The component ablations identify structural features and within-query reordering as the primary contributors. This is consequential because it makes the source of the gains more attributable than a simple “structure helps” comparison, and the robustness and CPU-cost analyses add practical context.

Critical weaknesses

1.The paper does not actually answer “when does RAG need structure.” The method applies structural calibration to candidates in essentially all cases, but there is no explicit query-level or candidate-level gate that decides whether structure should be used. The term “selective” refers to task- and candidate-conditioned modulation, not to a binary decision about structure use. This is consequential because the title and framing promise a conditional-use analysis, but the method is better described as a structure-aware reranker with bounded calibration. The paper needs either a clearer reframing or an explicit analysis of when structural correction helps versus when the adapted-base fallback is preferable.

2.Candidate-pool coverage and base-scorer supervision complicate the attribution of gains. On 2Wiki, the structural top-512 pool and the union cover 100% of gold evidence, versus 83.38% for dense; on MuSiQue, the union covers 95.92% versus 83.04% dense. Although the fixed-pool experiments are a good control, the base scorer is trained with training-split gold evidence and frozen, while several baselines use their native scoring rules. This is consequential because the headline “first on every metric” claim combines candidate access, base-score adaptation, and reranking; the paper should more explicitly state what a practitioner would obtain with the same supervision budget and the same candidate pool across methods.

Questions for authors

1.Explicit gating / conditional use: Can you provide an analysis or an ablated gate that decides when to apply structural calibration versus when to fall back to the adapted base ranking? Under what query or candidate conditions does structural correction help most, and when does it hurt? This would directly address the “when does RAG need structure” framing.

2.Candidate-pool versus reranking separation: Can you report results with all methods restricted to exactly the same candidate union and the same base scorer, including the structure-enhanced baselines? How much of TRACER’s advantage remains when candidate access and base-score supervision are matched exactly?

3.Base-scorer supervision fairness: The base scorer is trained with training-split gold evidence and then frozen. How do the baselines’ supervision budgets compare? Can you provide a matched-supervision control, for example a baseline reranker trained with the same labels and the same candidate pool?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.