acceptodds
Under review as a conference paper at ICLR 2027

Generating Leakage-Free Benchmarks for Robust RAG Evaluation

Abstract

Retrieval-augmented generation (RAG) is widely used to augment large language models (LLMs) with external knowledge. However, many benchmark datasets, designed to test RAG performance, comprise many questions that can already be answered from an LLM’s parametric memory. This leads to unreliable evaluation. We refer to this phenomenon as knowledge leakage—cases where RAG tasks are solvable without retrieval. This issue worsens over time due to benchmark aging. As benchmarks are reused for training, their contents are increasingly absorbed into model parameters, making them less effective for evaluating retrieval. We introduce SeedRG, a semi-synthetic benchmark generation pipeline that mitigates knowledge leakage and addresses the issue of benchmark aging. Starting from a seed benchmark dataset, SeedRG extracts a reasoning graph from question–context pairs to capture their underlying reasoning structure, and then generates new examples via type-constrained entity replacement. This process produces structurally similar but novel instances that are unlikely to exist in the model’s parametric knowledge, while preserving the original reasoning patterns. To ensure quality, we incorporate two verification steps: (1) a reasoning-graph consistency check to maintain task difficulty, and (2) a knowledge-leakage filter to exclude instances answerable without retrieval. We evaluate SeedRG on three seed benchmarks (HotpotQA, 2WikiMultihopQA, QASC) and three popular LLMs (GPT-5, Claude Sonnet 4.5, Gemini 2.5 Flash). SeedRG reduces knowledge leakage by at least 78% while preserving reasoning difficulty. By removing the confounding effect of parametric knowledge, SeedRG reveals meaningful variability across RAG systems that is otherwise obscured. Prior benchmarks show uniformly high performance across RAG methods (HippoRAG, GraphRAG, OGRAG, SemanticRAG), because performance is dominated by model knowledge. In contrast, SeedRG surfaces clear differences in retrieval and reasoning ability across the methods. Beyond benchmark construction, we provide a systematic analysis linking reasoning difficulty to graph structure, showing how structural variations induce predictable changes in model accuracy. Together, these results demonstrate that SeedRG enables more discriminative and robust evaluation of RAG systems.

Primary area:
datasets and benchmarks
Keywords:
RAG, dataset, knowledge leakage

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_iRajOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper investigates an important question in RAG evaluation: how to distinguish improvements attributable to retrieval quality from those arising from the underlying language model's parametric knowledge. The authors identify two major factors that undermine reliable evaluation: knowledge leakage and benchmark aging. To address these issues, the paper introduces SeedRG, a semi-structured pipeline that constructs a knowledge graph from each query-context pair, substitutes entities with others of the same semantic type to generate new queries, and verifies both the absence of knowledge leakage and the answerability of the generated queries. Experimental results demonstrate the effectiveness of the proposed approach.

Critical strengths
  1. The paper provides a systematic analysis of benchmark aging and knowledge leakage, helping the community better understand the sources of performance improvements in RAG systems.

  2. The paper proposes a semi-structured pipeline for automatically generating high-quality, answerable queries while minimizing knowledge leakage.

Critical weaknesses
  1. Insufficient comparison with simpler alternatives. Why not simply evaluate the model on the original questions without providing any retrieved context, and then filter out questions that the model can answer correctly? It remains unclear what advantages the proposed pipeline offers over this simpler approach.

  2. Unclear generalizability to complex benchmarks. Can the proposed method be extended to more challenging benchmarks, such as those requiring multi-hop reasoning or complex compositional reasoning? Entity substitution may be insufficient to preserve the reasoning structure of such questions.

Questions for authors

Beyond the critical weaknesses mentioned above, there are two extra questions.

  1. Could the authors provide a detailed breakdown of the computational cost of the proposed pipeline, including the number of LLM calls, token consumption, and generation time per query?

  2. Could the proposed pipeline be implemented using a smaller, open-weight model, such as Qwen3-32B? How would this affect the quality of the generated queries and the reliability of the verification process?

If the authors can address my concerns, I am glad to raise my score.

Update 11/18: The authors have addressed my concerns.

2 comments
AuthorsOpenReview ↗

Thank you for your feedback! (1/2)

Dear Reviewer iRaj,

Thank you for your valuable feedback and for recognizing our systematic analysis of benchmark aging and knowledge leakage.

1. Comparison with simple filtering

We discuss this alternative in Section 5.2 of our submission (page 7): as Table 1 on the same page shows, filtering alone would discard up to 82% of a benchmark, even before removing low-answerability items, and the retained set keeps shrinking as future models absorb more questions. SeedRG instead renews each seed instance while preserving its reasoning structure. Additional analysis on the paper's data supports this (Appendix B.2 of the revised paper):

a. The filtered pool shrinks with stronger models: filtering with GPT-5 keeps far fewer original questions than filtering with the weaker Qwen3-32B (e.g. 37% vs 83% of 2WikiMultihopQA).

b. The questions that filtering keeps have low gold-context accuracy. We split the original questions by no-context filtering and compare their gold-context accuracy (table below): even with the gold evidence, the questions it keeps are answered correctly less often than the questions it removes, so a filtered benchmark is not a reliable test.

No-context filtering of the original questions: we compare the gold-context accuracy of the questions it filters out (the model answers them without context) and of those it keeps (the model does not). Each model filters with its own answers. Each cell: GPT-5 / Claude / Gemini.

DatasetFiltered outKept
HotpotQA1.00 / 1.00 / 1.000.68 / 0.88 / 0.73
2WikiMultihopQA0.95 / 0.96 / 0.960.62 / 0.91 / 0.71
QASC1.00 / 1.00 / 1.000.72 / 0.94 / 0.91

c. Filtering cannot remove partial-hop shortcuts, where the model already knows one hop and only retrieves the other. On MuSiQue, most original questions have such a hop, but few SeedRG questions do.

2. Complex compositional reasoning

Entity substitution acts on each entity independently, so it does not depend on the length of the reasoning path. What matters is whether the reasoning structure is preserved, and SeedRG controls this explicitly: every accepted instance must keep the seed's reasoning graph (Section 4.2, page 6), and Section 5.3.2 (page 8) shows that changing this graph changes the difficulty. Empirically, we applied SeedRG to MuSiQue, from its single-hop sub-questions up to 4-hop questions: SeedRG transforms a similar share of the 2-, 3- and 4-hop seeds, and at every depth models almost never answer the SeedRG questions without evidence, while gold-context accuracy stays close to that of the original questions (table below; Appendix B.4).

MuSiQue, generated for this response. 1-, 2-, 3- and 4-hop: full MuSiQue questions. Each cell: GPT-5 / Claude / Gemini.

HopsQuestionsNo-context acc.Gold-context acc.
1-hopOriginal0.69 / 0.62 / 0.560.91 / 0.98 / 0.96
SeedRG0.00 / 0.00 / 0.000.87 / 0.99 / 0.94
2-hopOriginal0.56 / 0.35 / 0.290.90 / 0.96 / 0.94
SeedRG0.02 / 0.00 / 0.020.90 / 0.96 / 0.81
3-hopOriginal0.28 / 0.15 / 0.180.87 / 0.95 / 0.69
SeedRG0.00 / 0.00 / 0.030.69 / 0.92 / 0.59
4-hopOriginal0.25 / 0.14 / 0.140.77 / 0.91 / 0.84
SeedRG0.00 / 0.00 / 0.000.84 / 0.91 / 0.86

AuthorsOpenReview ↗

Thank you for your feedback! (2/2)

3. Computational cost

Generation is cheap. On MuSiQue (2- to 4-hop questions, GPT-4o-mini as generator and verifier), one accepted instance costs about 65 LLM calls, 19.8k input and 11.0k output tokens (about USD 0.01 at list prices), and 203 seconds of LLM time when run sequentially, including rejected attempts and rejected seeds. Seeds are independent, so generation can be parallelized trivially. We have added this breakdown to Appendix B.6.

4. A smaller open-weight model (Qwen3-32B)

Yes, Qwen3-32B can take over the pipeline's checking steps. We ran it as an additional evaluator on all four datasets, without evidence and with the gold evidence, and as a verifier (Appendix B.5).

a. Leakage check: without evidence, Qwen3-32B answers few SeedRG questions, and fewer than the original ones on every dataset (table below), so it clearly separates the two.

New for this response: Qwen3-32B (non-thinking) as an additional evaluator, answering without evidence (No-Ctx) and with the gold evidence (Gold). Each cell: SeedRG / Original. HotpotQA, 2WikiMultihopQA and QASC: the submitted data; MuSiQue: generated for this response.

Qwen3-32BHotpotQA2WikiMultihopQAQASCMuSiQue
No-Ctx0.04 / 0.140.10 / 0.170.08 / 0.740.02 / 0.05
Gold0.45 / 0.780.64 / 0.830.85 / 0.980.85 / 0.82

b. Verification: the instances it accepts are answered from the gold evidence more often than those it rejects, by a wide margin on HotpotQA and 2WikiMultihopQA (table below), so a 32B open model is a reliable verifier on HotpotQA and 2WikiMultihopQA; on QASC, most of its rejections come from the reasoning-graph check rather than from answerability.

Reliability of Qwen3-32B as verifier: gold-context accuracy of frontier models on instances Qwen3-32B accepts vs rejects. Each cell: GPT-5 / Claude / Gemini.

DatasetAccepted by Qwen3-32BRejected by Qwen3-32B
HotpotQA0.83 / 1.00 / 0.830.30 / 0.64 / 0.41
2WikiMultihopQA0.92 / 0.92 / 0.850.36 / 0.70 / 0.41
QASC0.89 / 0.86 / 0.950.82 / 0.86 / 0.89
Reviewer_pR1hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper first shows with systematic evidence that existing RAG benchmarks include many questions that can already be answered by LLMs' internalized knowledge. To address the issue, the paper then introduces SeedRG, a benchmark generation pipeline that mitigates knowledge leakage. The idea is to generate new examples based on a seed benchmark dataset through type-constrained entity replacement. The paper reports evaluation results on 3 seed benchmark datasets, showing that SeedRG can significantly reduce knowledge leakage.

Critical strengths
  • The paper presents systematic evidence to show the significance of the knowledge leakage issue (up to 82% of questions) in 3 benchmark datasets.
  • The paper introduces a new pipeline to generate low-leakage benchmarks and shows its effectiveness.
Critical weaknesses
  • My major concern is that I am not fully convinced of the necessity of the pipeline. There are two cheaper ways to get a low-leakage benchmark: 1) simply discarding the leaking instances. The paper does not show evidence that SeedRG separates methods better compared to such straightforward filtering. 2) employing benchmarks with questions whose answers are fast-changing or real-time, such as [1].

[1] CRAG -- Comprehensive RAG Benchmark

Questions for authors
  • Can you discuss the necessity of the work and compare with the two cheaper ways mentioned above?

  • It appears to me that the leakage issue is not specific to multi-hop. Could you 1) explain why the work focuses on multi-hop and 2) show that transformed examples still require combining multiple documents rather than reading one?

6 comments
AuthorsOpenReview ↗

Thank you for your feedback!

Dear Reviewer pR1h,

Thank you for your constructive feedback and for recognizing our systematic evidence of knowledge leakage.

1. Comparison with discarding leaking instances

a. We agree that filtering is a natural baseline; in fact, SeedRG itself ends with this filter. However, as shown in Table 1 of our submission (page 7), filtering alone would discard up to 82% of a benchmark, and the loss grows as models become stronger: filtering with GPT-5 keeps far fewer original questions than filtering with the weaker Qwen3-32B (e.g. 37% vs 83% of 2WikiMultihopQA). On the other hand, SeedRG can regenerate new instances, so the benchmark keeps its capacity as models improve.

b. Filtering also keeps the noisiest questions. We split the original questions by no-context filtering and compare their gold-context accuracy (table below): even with the gold evidence, the questions it keeps are answered correctly less often than the questions it removes (those the model answers without evidence). Many of the kept questions are ambiguous or mislabelled, so a filtered benchmark is not reliable.

No-context filtering of the original questions: we compare the gold-context accuracy of the questions it filters out (the model answers them without context) and of those it keeps (the model does not). Each model filters with its own answers. Each cell: GPT-5 / Claude / Gemini.

DatasetFiltered outKept
HotpotQA1.00 / 1.00 / 1.000.68 / 0.88 / 0.73
2WikiMultihopQA0.95 / 0.96 / 0.960.62 / 0.91 / 0.71
QASC1.00 / 1.00 / 1.000.72 / 0.94 / 0.91

c. We have added this comparison to the discussion of Table 1 (page 7) and to Appendix B.2 of the revised paper.

2. Fast-changing benchmarks such as CRAG

We see them as complementary. However, once released as a snapshot, their answers also enter later training data, so they age like any static benchmark, and refreshing them requires new human annotation; SeedRG regenerates automatically. In addition, they are restricted to volatile domains such as news, finance and sports, whereas SeedRG applies to any existing benchmark and keeps a paired original for every instance.

3. Why multi-hop

We agree that leakage is not specific to multi-hop, and SeedRG applies unchanged to single-hop questions: on the single-hop sub-questions of MuSiQue, no model answers any SeedRG question without evidence (1-hop rows of the table below). We focus on multi-hop for two reasons. First, multi-hop benchmarks are the standard testbed for the graph-based RAG methods we compare. Second, multi-hop questions have an additional form of leakage: the model may already know one hop and only need to retrieve the other. When we decompose MuSiQue questions and ask each hop without evidence, most original questions have at least one hop the model already knows, but few SeedRG questions do (Appendix B.4).

MuSiQue, generated for this response. 1-, 2-, 3- and 4-hop: full MuSiQue questions. Each cell: GPT-5 / Claude / Gemini.

HopsQuestionsNo-context acc.Gold-context acc.
1-hopOriginal0.69 / 0.62 / 0.560.91 / 0.98 / 0.96
SeedRG0.00 / 0.00 / 0.000.87 / 0.99 / 0.94
2-hopOriginal0.56 / 0.35 / 0.290.90 / 0.96 / 0.94
SeedRG0.02 / 0.00 / 0.020.90 / 0.96 / 0.81
3-hopOriginal0.28 / 0.15 / 0.180.87 / 0.95 / 0.69
SeedRG0.00 / 0.00 / 0.030.69 / 0.92 / 0.59
4-hopOriginal0.25 / 0.14 / 0.140.77 / 0.91 / 0.84
SeedRG0.00 / 0.00 / 0.000.84 / 0.91 / 0.86

4. Do transformed examples still require multiple documents?

Yes. We gave each model either one gold document at a time or all gold documents. The gap between one and all gold documents (Diff. column, in bold) is larger on SeedRG for every dataset and model, which shows that SeedRG questions require combining the documents. We have added this experiment to Appendix B.3.

Accuracy with one gold document (averaged over the documents) and with all gold documents, on the paper's data. Diff.: all minus one; bold: a larger gap than on the original questions. Each cell: GPT-5 / Claude / Gemini.

DatasetQuestionsOne gold documentAll gold documentsDiff. (All – One)
HotpotQAOriginal0.74 / 0.70 / 0.530.84 / 0.93 / 0.810.10 / 0.23 / 0.28
SeedRG0.21 / 0.44 / 0.210.43 / 0.73 / 0.510.22 / 0.29 / 0.30
2WikiMultihopQAOriginal0.78 / 0.69 / 0.480.83 / 0.93 / 0.810.05 / 0.24 / 0.33
SeedRG0.32 / 0.47 / 0.210.57 / 0.79 / 0.570.25 / 0.32 / 0.36
QASCOriginal0.84 / 0.92 / 0.900.93 / 0.99 / 0.980.09 / 0.07 / 0.08
SeedRG0.49 / 0.56 / 0.570.85 / 0.86 / 0.920.36 / 0.30 / 0.35

Thanks for your rebuttal. My major concern was specifically that the paper does not show SeedRG separating RAG methods better than filtering does. Rebuttal 1a and 1b argue that filtering is unsustainable and noisy, which is a reasonable case, but neither is that measurement. Could you report the 4 RAG systems on the non-leaking subset alongside the SeedRG numbers? If the spread is smaller there, my concern is resolved.

AuthorsOpenReview ↗

Thank you for you feedback, Reviewer pR1h!

Dear Reviewer pR1h,

Thank you for clarifying! We appreciate your feedback on the rebuttal!

Below we report the four RAG systems on the non-leaking subset comparison across original dataset vs SeedRG, on the four RAG algorithms.

Table: The four RAG algorithms performances and relative spread on the non-leaking questions across 3 LLM models

DatasetEvaluatorQuestionsSemanticHippoRAGGraphRAGOGRAGRel. spread (max-min)/min (%)
HotpotQAGPT-5Original0.6760.7300.7030.7308.0
SeedRG0.4860.4860.4050.27080.0
ClaudeOriginal0.9250.9000.8500.80015.6
SeedRG0.6750.6750.6750.52528.6
GeminiOriginal0.6670.7060.6270.64712.5
SeedRG0.5690.5690.4710.39245.0
2WikiMultihopQAGPT-5Original0.6920.7690.4620.61566.7
SeedRG0.3850.5000.2690.26985.7
ClaudeOriginal0.9070.9300.6510.76742.9
SeedRG0.6280.6280.4650.48835.0
GeminiOriginal0.7620.7620.5240.57145.5
SeedRG0.4290.4760.2620.33381.8
QASCGPT-5Original0.6400.7600.7200.80025.0
SeedRG0.6800.7600.5200.72046.2
ClaudeOriginal0.8890.8890.8890.9446.3
SeedRG0.6670.6110.6110.72218.2
GeminiOriginal0.9130.9130.8260.87010.5
SeedRG0.7830.7830.6960.73912.5

We quantify differentiation as the relative spread (max-min)/min across RAG methods, as in Section 5.4 of our paper.

Our conclusions are nearly same:

  • For non-leakage original questions (the filter ones), SeedRG questions give a larger relative spread than original questions in eight of the nine settings (except WikiHop with Claude).
  • For full original questions, the two exceptions are WikiHop with GPT-5 and Claude. (In Section 5.4)

We added this result to Appendix B.2, Table 9.

Reviewer_WKyqOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper addresses a critical flaw in current RAG evaluation, i.e., knowledge leakage and benchmark aging, where evaluation metrics are dominated by an LLM's internal parametric memory rather than retrieval performance. The authors present SeedRG, a semi-synthetic data generation framework that extracts a reasoning graph from existing multi-hop QA benchmarks and generates new, leakage-free benchmark instances via type-constrained entity replacement.

Critical strengths
  1. The motivation is clear and easy to follow. Currently, the performance of graphrag on multi-hop QA datasets could already achieve over 90 ACC. It is necessary to propose a new and challenging benchmark.

  2. Using reasoning graph extraction to constrain the entity replacement process ensures that the underlying logic chains (e.g., hop counts, dependency chains) remain intact, preventing the regenerated benchmark from suffering uncontrolled difficulty shifts.

  3. The empirical demonstration showing how parametric shortcuts collapse performance spreads across RAG baselines is compelling. SeedRG effectively removes the ceiling effect and makes variations in retrieval strategy more noticeable.

Critical weaknesses
  1. The idea is not new. The paper also misses a directly relevant prior work: AnonyRAG (introduced in Youtu-GraphRAG, ICLR 2026). AnonyRAG similarly tackled knowledge leakage in complex GraphRAG settings by replacing entities and relations with anonymized/novel placeholders to create strict zero-knowledge evaluation sets. The authors must cite and thoroughly compare SeedRG against AnonyRAG in both Related Work and Methodology.

  2. SeedRG relies on type-constrained entity replacement. However, as frontier models scale, newly sampled long-tail or synthetic entities often still trigger implicit parametric associations or logical conflicts if the surrounding context prose isn't perfectly disentangled. Unlike total symbol/token anonymization, like AnonyRAG, type-constrained entity replacement remains vulnerable to leaking through semantic priors embedded in LLM pre-training corpora.

  3. The seed benchmarks evaluated consist of basic 2-hop or 3-hop linear/tree reasoning paths. While SeedRG works well on these small multi-hop chains, it is unproven whether simple entity replacement scales to complex scenarios.

Questions for authors

Please see above.

2 comments
AuthorsOpenReview ↗

Thank you for your feedback!

Dear Reviewer WKyq,

Thank you for your valuable feedback and for recognizing that our motivation is clear and that the reasoning-graph constraint keeps the logic chains intact.

1. Novelty and comparison with AnonyRAG

Thank you for pointing us to AnonyRAG, which is indeed closely related. We have cited and discussed it in the related work (Section 2) and in Section 4.1 of the revised paper.

The two works pursue different goals. AnonyRAG masks people and locations in four novels with typed placeholders such as [PERSON#277], and its Anonymity Reversion task asks models to recover the masked entities from context. SeedRG instead transforms an existing benchmark's own question–evidence–answer triples: it keeps each seed's reasoning graph, so the difficulty is preserved, and it gives the replaced entities natural invented names, so the model has no background knowledge to fall back on.

We also compared the two directly on the paper's three benchmarks (table below; Appendix B.1).

No-context accuracy between Original, AnonyRAG, SeedRG. Each cell: GPT-5 / Claude / Gemini.

QuestionsHotpotQA2WikiMultihopQAQASC
Original0.50 / 0.46 / 0.310.63 / 0.39 / 0.400.75 / 0.82 / 0.77
AnonyRAG (people & location)0.31 / 0.23 / 0.180.20 / 0.14 / 0.170.74 / 0.80 / 0.73
SeedRG0.01 / 0.14 / 0.040.11 / 0.13 / 0.060.14 / 0.17 / 0.16

2. Residual semantic priors

We agree that entity replacement cannot rule out every residual association by construction, so we measured it directly: if residual priors still revealed the answer, models would answer SeedRG questions without any evidence. The table in point 1 shows that this rarely happens. AnonyRAG's masking leaks more than SeedRG, though less than the original questions, because AnonyRAG is designed to test whether models can recover the masked entities, not to remove leakage.

3. Beyond 2–3-hop chains

We applied SeedRG to MuSiQue, from its single-hop sub-questions up to 4-hop questions. The results do not degrade with the number of hops (table below): SeedRG transforms a similar share of the 2-, 3- and 4-hop seeds, and at every depth models almost never answer the SeedRG questions without evidence, while gold-context accuracy stays close to that of the original questions. We have added these results to Appendix B.4.

MuSiQue, generated for this response. 1-, 2-, 3- and 4-hop: full MuSiQue questions. Each cell: GPT-5 / Claude / Gemini.

HopsQuestionsNo-context acc.Gold-context acc.
1-hopOriginal0.69 / 0.62 / 0.560.91 / 0.98 / 0.96
SeedRG0.00 / 0.00 / 0.000.87 / 0.99 / 0.94
2-hopOriginal0.56 / 0.35 / 0.290.90 / 0.96 / 0.94
SeedRG0.02 / 0.00 / 0.020.90 / 0.96 / 0.81
3-hopOriginal0.28 / 0.15 / 0.180.87 / 0.95 / 0.69
SeedRG0.00 / 0.00 / 0.030.69 / 0.92 / 0.59
4-hopOriginal0.25 / 0.14 / 0.140.77 / 0.91 / 0.84
SeedRG0.00 / 0.00 / 0.000.84 / 0.91 / 0.86

AuthorsOpenReview ↗

Follow-up and Summary of Revisions for Reviewer WKyq

Dear Reviewer WKyq,

Thank you again for your time and insightful feedback on our submission. As the discussion period is drawing to a close, we wanted to kindly follow up for the rebuttal.

To save your time, here is a brief summary of how we addressed your main concerns in the revised manuscript

  1. The mentioned AnonyRAG (original paper is Youtu-GraphRAG), has been cited and compared in our revised version. SeedRG shows a smaller leakage comparing to AnnonyRAG.
  2. Type-constrained replacement helps with maintaining the graph structure & reasoning difficulty. For concerns on the leakage, we proved it in point 1, comparing to AnonyRAG.
  3. We added MuSiQue and comparing the diff between simple-hops to multi-hops. We proved that SeedRG applies to complex scenarios. Besides that, following the suggestion of Reviewer iRaj, we added BrowseComp, to prove the SeedRG ability in complex scenarios.

We would greatly appreciate it if you could let us know whether our revisions address your concerns, or if any further clarification is needed before the discussion deadline.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the rebuttal

Dear Area Chairs and Reviewers,

We thank all reviewers for their constructive feedback. SeedRG renews existing RAG benchmarks by replacing the entities of each seed question and context while keeping its reasoning graph to maintain reasoning difficulty.

Below is a summary of our discussion. All related results have been added to Appendix B of the revised paper.

I. Reviewer WKyq

1. Comparison with AnonyRAG. The reviewer asked us to cite and compare with AnonyRAG (introduced in Youtu-GraphRAG).

  • Response: AnonyRAG is now cited and compared (Sections 2 and 4.1, Appendix B.1): SeedRG leaks less than AnonyRAG-style anonymization.

2. Residual semantic priors. The reviewer asked whether the replaced entities could still leak through semantic priors comparing to AnonyRAG.

  • Response: We measured this directly: models rarely answer SeedRG questions without evidence (Appendix B.1).

3. More complex scenarios. The reviewer asked whether SeedRG scales beyond 2–3-hop chains.

  • Response: On MuSiQue with 1 to 4 hops, models almost never answer the SeedRG questions without evidence at any depth, while gold-context accuracy stays close to that of the original questions (Appendix B.4). We also applied SeedRG to BrowseComp-Plus (Appendix B.7).

II. Reviewer pR1h

1. Why not filter the leaking questions, or use fast-changing / real-time benchmarks such as CRAG? This was the reviewer's main concern.

  • Filtering: as discussed in our paper, a filtered benchmark ages: as LLMs become stronger, previously non-leaking questions start to leak, so the filtered set keeps shrinking. Filtering also keeps the noisier questions. The reviewer found these arguments "a reasonable case".
  • Real-time benchmarks: they age like any static benchmark once a snapshot is released, and refreshing them requires new human annotation, whereas SeedRG regenerates automatically. They are also restricted to volatile domains such as news.
  • Separating RAG systems: we reported the four RAG systems on the non-leaking subset, which the reviewer confirmed was the requested experiment, with both absolute and relative spreads and on three new random samples (Appendix B.2). The comparison shows why SeedRG works: on the leaking part, the RAG systems cluster because the LLMs already know the answers, whereas SeedRG separates them; on the non-leaking part, SeedRG has the larger relative spread in most settings, while the absolute spread depends on the dataset; and SeedRG keeps all of its questions.

2. Other questions. The need for multiple documents and single-hop questions are covered in Appendices B.3 and B.4.

III. Reviewer iRaj (raised the score to 4, Clear Accept)

1. Why not filter the leaking questions? The same concern as Reviewer pR1h's.

  • Response: as shown in our paper (Section 5.2, Table 1), a filtered benchmark ages: the retained set keeps shrinking as models become stronger. In addition, the questions that filtering keeps are noisier: even with the gold evidence, models answer them correctly less often than the questions it removes.

2. More complex benchmarks. The reviewer asked whether SeedRG applies to benchmarks that require complex compositional reasoning.

  • Response: yes. We applied SeedRG to BrowseComp-Plus. The original questions already leak to strong reasoning models (GPT-5, Gemini 4 Hydrogen); SeedRG removes this leakage for all evaluated models (Appendix B.7).

3. Computational cost.

  • Response: we reported LLM calls, tokens and time per instance; generation costs about USD 0.01 per accepted instance (Appendix B.6).

4. A smaller open-weight model. The reviewer asked whether Qwen3-32B could run the pipeline, and how reliable its verification would be.

  • Response: yes. Qwen3-32B can take over the pipeline's leakage check and verification (Appendix B.5).

The reviewer was very satisfied with our clarifications and offered to advocate for acceptance.

We thank the reviewers and the AC again for their time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.