- Rating
- 4(+2)
- Confidence
- 4
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper investigates an important question in RAG evaluation: how to distinguish improvements attributable to retrieval quality from those arising from the underlying language model's parametric knowledge. The authors identify two major factors that undermine reliable evaluation: knowledge leakage and benchmark aging. To address these issues, the paper introduces SeedRG, a semi-structured pipeline that constructs a knowledge graph from each query-context pair, substitutes entities with others of the same semantic type to generate new queries, and verifies both the absence of knowledge leakage and the answerability of the generated queries. Experimental results demonstrate the effectiveness of the proposed approach.
Critical strengths
-
The paper provides a systematic analysis of benchmark aging and knowledge leakage, helping the community better understand the sources of performance improvements in RAG systems.
-
The paper proposes a semi-structured pipeline for automatically generating high-quality, answerable queries while minimizing knowledge leakage.
Critical weaknesses
-
Insufficient comparison with simpler alternatives. Why not simply evaluate the model on the original questions without providing any retrieved context, and then filter out questions that the model can answer correctly? It remains unclear what advantages the proposed pipeline offers over this simpler approach.
-
Unclear generalizability to complex benchmarks. Can the proposed method be extended to more challenging benchmarks, such as those requiring multi-hop reasoning or complex compositional reasoning? Entity substitution may be insufficient to preserve the reasoning structure of such questions.
Questions for authors
Beyond the critical weaknesses mentioned above, there are two extra questions.
-
Could the authors provide a detailed breakdown of the computational cost of the proposed pipeline, including the number of LLM calls, token consumption, and generation time per query?
-
Could the proposed pipeline be implemented using a smaller, open-weight model, such as Qwen3-32B? How would this affect the quality of the generated queries and the reliability of the verification process?
If the authors can address my concerns, I am glad to raise my score.
Update 11/18: The authors have addressed my concerns.
2 comments
Thank you for your feedback! (2/2)
3. Computational cost
Generation is cheap. On MuSiQue (2- to 4-hop questions, GPT-4o-mini as generator and verifier), one accepted instance costs about 65 LLM calls, 19.8k input and 11.0k output tokens (about USD 0.01 at list prices), and 203 seconds of LLM time when run sequentially, including rejected attempts and rejected seeds. Seeds are independent, so generation can be parallelized trivially. We have added this breakdown to Appendix B.6.
4. A smaller open-weight model (Qwen3-32B)
Yes, Qwen3-32B can take over the pipeline's checking steps. We ran it as an additional evaluator on all four datasets, without evidence and with the gold evidence, and as a verifier (Appendix B.5).
a. Leakage check: without evidence, Qwen3-32B answers few SeedRG questions, and fewer than the original ones on every dataset (table below), so it clearly separates the two.
New for this response: Qwen3-32B (non-thinking) as an additional evaluator, answering without evidence (No-Ctx) and with the gold evidence (Gold). Each cell: SeedRG / Original. HotpotQA, 2WikiMultihopQA and QASC: the submitted data; MuSiQue: generated for this response.
| Qwen3-32B | HotpotQA | 2WikiMultihopQA | QASC | MuSiQue |
|---|---|---|---|---|
| No-Ctx | 0.04 / 0.14 | 0.10 / 0.17 | 0.08 / 0.74 | 0.02 / 0.05 |
| Gold | 0.45 / 0.78 | 0.64 / 0.83 | 0.85 / 0.98 | 0.85 / 0.82 |
b. Verification: the instances it accepts are answered from the gold evidence more often than those it rejects, by a wide margin on HotpotQA and 2WikiMultihopQA (table below), so a 32B open model is a reliable verifier on HotpotQA and 2WikiMultihopQA; on QASC, most of its rejections come from the reasoning-graph check rather than from answerability.
Reliability of Qwen3-32B as verifier: gold-context accuracy of frontier models on instances Qwen3-32B accepts vs rejects. Each cell: GPT-5 / Claude / Gemini.
| Dataset | Accepted by Qwen3-32B | Rejected by Qwen3-32B |
|---|---|---|
| HotpotQA | 0.83 / 1.00 / 0.83 | 0.30 / 0.64 / 0.41 |
| 2WikiMultihopQA | 0.92 / 0.92 / 0.85 | 0.36 / 0.70 / 0.41 |
| QASC | 0.89 / 0.86 / 0.95 | 0.82 / 0.86 / 0.89 |
Thank you for your feedback! (1/2)
Dear Reviewer iRaj,
Thank you for your valuable feedback and for recognizing our systematic analysis of benchmark aging and knowledge leakage.
1. Comparison with simple filtering
We discuss this alternative in Section 5.2 of our submission (page 7): as Table 1 on the same page shows, filtering alone would discard up to 82% of a benchmark, even before removing low-answerability items, and the retained set keeps shrinking as future models absorb more questions. SeedRG instead renews each seed instance while preserving its reasoning structure. Additional analysis on the paper's data supports this (Appendix B.2 of the revised paper):
a. The filtered pool shrinks with stronger models: filtering with GPT-5 keeps far fewer original questions than filtering with the weaker Qwen3-32B (e.g. 37% vs 83% of 2WikiMultihopQA).
b. The questions that filtering keeps have low gold-context accuracy. We split the original questions by no-context filtering and compare their gold-context accuracy (table below): even with the gold evidence, the questions it keeps are answered correctly less often than the questions it removes, so a filtered benchmark is not a reliable test.
No-context filtering of the original questions: we compare the gold-context accuracy of the questions it filters out (the model answers them without context) and of those it keeps (the model does not). Each model filters with its own answers. Each cell: GPT-5 / Claude / Gemini.
c. Filtering cannot remove partial-hop shortcuts, where the model already knows one hop and only retrieves the other. On MuSiQue, most original questions have such a hop, but few SeedRG questions do.
2. Complex compositional reasoning
Entity substitution acts on each entity independently, so it does not depend on the length of the reasoning path. What matters is whether the reasoning structure is preserved, and SeedRG controls this explicitly: every accepted instance must keep the seed's reasoning graph (Section 4.2, page 6), and Section 5.3.2 (page 8) shows that changing this graph changes the difficulty. Empirically, we applied SeedRG to MuSiQue, from its single-hop sub-questions up to 4-hop questions: SeedRG transforms a similar share of the 2-, 3- and 4-hop seeds, and at every depth models almost never answer the SeedRG questions without evidence, while gold-context accuracy stays close to that of the original questions (table below; Appendix B.4).
MuSiQue, generated for this response. 1-, 2-, 3- and 4-hop: full MuSiQue questions. Each cell: GPT-5 / Claude / Gemini.