acceptodds
Under review as a conference paper at ICLR 2027

AG-RAG: Robust Agentic Graph RAG through Stress-Aware Self-Improvement

Abstract

Graph retrieval-augmented generation (Graph RAG) organizes external knowledge into entities, relations, and supporting documents to improve LLM reasoning. However, existing Graph RAG systems are typically optimized and evaluated on clean queries, where their advantages over text retrieval can be modest. We argue that graph structure can be especially valuable when retrieval fails. We introduce AG-RAG, an agentic Graph RAG framework designed for such failures. AG-RAG exposes a corpus-derived knowledge graph through MCP tools for fuzzy entity resolution, graph traversal, and provenance-bound document retrieval, together with stress-aware self-improvement that optimizes tool policies, retrieval budgets, ranking, and instructions while rejecting changes that sacrifice robustness. We evaluate three retrieval failures: misspelled entities, unavailable source text, and adversarially injected passages. With the same answering model, AG-RAG remains competitive with HippoRAG 2 and LinearRAG on clean HotpotQA, 2WikiMultiHopQA, and MuSiQue, while substantially outperforming both under stress. On 2Wiki-Extra, a set of 2WikiMultiHopQA questions never used during development, AG-RAG achieves only 2.2% poisoning attack success, compared with 57.2% for HippoRAG 2 and 48.0% for LinearRAG. These results show that graph structure provides complementary robustness against retrieval failures and suggest that self-improving retrieval agents should optimize capability subject to explicit stress-aware constraints.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Retrieval-augmented generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5AkGOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AG-RAG, an agentic Graph RAG framework that exposes a corpus-derived graph through MCP tools and uses stress-aware self-improvement to optimize the agent's retrieval harness. The paper studies three retrieval failure conditions: misspelled entities, unavailable source text, and adversarially injected passages. The central idea is to accept harness modifications only when they improve or preserve performance under both clean and stressed validation conditions.

The paper reports compelling robustness improvements on the proposed 2Wiki-Extra benchmark. The self-improved AG-RAG achieves 66.5 clean F1, 63.0 typo F1, 63.2 F1 when 50% of source text is unavailable, and 2.2% poisoning ASR. In contrast, the BM25 text agent obtains higher clean F1 (69.1) but substantially worse performance under stress, including 68.4% poisoning ASR. The comparison between clean-only and stress-aware self-improvement is also interesting: clean-only optimization tends to favor free-text search, whereas stress-aware promotion rejects such edits.

Critical strengths

The paper addresses an important question: whether agentic retrieval systems should optimize for robustness under retrieval failures rather than only clean benchmark accuracy. The observation that clean-only optimization can select a tool that improves nominal accuracy while substantially weakening poisoning robustness is useful and practically relevant.

The proposed promotion rule provides a simple mechanism for incorporating robustness constraints into automated harness optimization. The comparison with clean-only optimization is particularly informative.

Critical weaknesses
  1. The paper frames the three stress conditions as evidence that graph structure provides complementary robustness. However, the mechanisms underlying these results should be separated more carefully. For typos, AG-RAG uses a character-trigram index followed by RapidFuzz reranking. Thus, the experiment does not establish that graph edges are necessary for typo robustness. More importantly, the paper seems does not compare against a directly comparable non-graph fuzzy retrieval mechanism such as BM25 + character n-gram matching or a fuzzy text/entity index.For poisoning, the protection primarily comes from the snapshot/provenance boundary. For missing text, in contrast, the large degradation after removing graph edges provides substantially stronger evidence that graph structure itself contributes robustness.

  2. The text agent is based on BM25, while AG-RAG includes a dedicated fuzzy entity-resolution mechanism. The paper does evaluate an LLM spelling-fix intervention, but this is not the most direct comparison to the mechanism used by AG-RAG. The paper itself notes that a fuzzy text index remains untested.

Questions for authors

Clarifications

  1. The paper attributes robustness to three mechanisms: fuzzy entity matching, graph edges, and snapshot/provenance-bound retrieval. Could the authors clarify which parts of the reported gains they consider specifically attributable to the graph structure itself, rather than to the entity index or provenance restrictions?
  2. The entity-index-only ablation retains essentially the same typo robustness as the full system. Does this mean that the authors view typo robustness primarily as a property of fuzzy entity resolution rather than graph traversal? If so, could the paper make this attribution more explicit?
  3. The main poisoning experiments concern malicious documents added after graph construction. Should the security claim therefore be interpreted specifically as robustness to post-snapshot text-index poisoning, rather than poisoning in general? In particular, how should the reported ASR numbers be interpreted when the malicious documents are ingested into the graph?

Suggestions

  1. A directly comparable fuzzy text-retrieval baseline would help isolate the contribution of the graph from fuzzy matching. This seems particularly important because the current typo experiment does not establish that graph edges are necessary for typo robustness.
  2. Since the paper itself notes that provenance-bound access does not require graph edges, comparing against a text-only agent restricted to the same snapshot would more cleanly quantify the benefit of graph structure for poisoning robustness.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 5AkG

Thank you for the expert and constructive review, and for the two suggested baselines. We ran both, together with several further controls. They make the role of each of AG-RAG's three mechanisms explicit: fuzzy entity resolution, snapshot-bound reading and relation edges.

All results below are on the 2Wiki-Extra test set (n = 500) with the paper's answering model, unless noted. The three mechanism columns show which of AG-RAG's mechanisms each system has. ↑ means higher is better and ↓ means lower is better. ASR is the attack success rate under poisoning, the share of poisoned questions answered with the attacker's target answer, so lower is better. "Stress avg" is the mean F1 over the three stress conditions.

SystemFuzzy namesSnapshot-boundEdgesTypos ↑50% text ↑Poisoned ↑Stress avg ↑ASR ↓
Text agent (BM25)---53.344.324.240.668.4%
Word + char-trigram BM25 (new)partial--58.244.526.243.065.2%
Fuzzy title lookup (new)✓--67.742.344.551.541.6%
Snapshot text agent (new)-✓-53.344.369.555.72.4%
Entity index only (ablation)✓✓-63.129.967.653.62.8%
AG-RAG static✓✓✓59.660.461.860.62.4%
AG-RAG self-improved✓✓✓63.063.267.264.52.2%

The snapshot text agent matches the plain text agent under typos and 50% text, because no passages are added in those conditions, so its index is identical.

W1, W2 and Suggestion 1: a directly comparable fuzzy retrieval baseline. Thank you for this suggestion. We added two non-graph fuzzy baselines.

  • Word + character-trigram BM25. BM25 over word and character-trigram tokens of every passage. Under typos it scores 58.2, which is 4.9 F1 above plain BM25.
  • Fuzzy title lookup. The same character-trigram index and RapidFuzz reranking that AG-RAG uses, applied to document titles, with no extraction and no edges. Under typos it scores 67.7. That is 8.0 F1 above AG-RAG static and 4.6 above the entity-index-only ablation.

These results confirm the design of AG-RAG: typo robustness comes from its fuzzy name resolution mechanism, not from graph edges. On its own, however, fuzzy resolution does not handle the other two stresses:

  • Fuzzy title lookup falls to 42.3 F1 when half the text is missing.
  • It reaches 41.6% ASR under poisoning, because it reads from the current, poisoned collection.

This is the benefit of combining the mechanisms in one framework. AG-RAG keeps 59.6-67.2 F1 and 2.2-2.4% ASR across all three stresses, because it pairs fuzzy resolution with snapshot-bound reading and relation edges. Its stress average is 60.6 (static) and 64.5 (self-improved), against 51.5 for fuzzy title lookup. We will make this attribution explicit, and add both baselines to the paper.

Suggestion 2: a text-only agent restricted to the same snapshot. We ran a BM25 text agent whose index contains only the documents in the graph snapshot.

  • Its ASR falls from 68.4% to 2.4%, and its poisoned F1 rises from 24.2 to 69.5, above AG-RAG static (61.8).
  • It read no poison passages.

This confirms that snapshot-bound reading, AG-RAG's second mechanism, is a simple and effective poisoning defence that does not need graph edges. Alone, however, it does not handle the other two stresses: this agent scores 53.3 under typos and 44.3 under missing text, for a stress average of 55.7. AG-RAG builds snapshot-bound reading in as a default and combines it with the other two mechanisms, which raises the stress average to 60.6-64.5. We will report this baseline and attribute the poisoning result to snapshot-bound reading.

Q1. What is attributable to graph structure itself? Keeping facts reachable when their source text is missing. This is a benefit only the graph provides. On 2Wiki-Extra, no edge-free system we tested comes close.

Removing the edges from AG-RAG, which leaves the entity index only, drops F1 with half the text removed from 60.4 to 29.9, a loss of 30.5 F1. The table below compares AG-RAG with the best edge-free system on 2Wiki-Extra and in the new settings. In each row, the edge-free system is the strongest one we ran in that setting. All values are F1 with 50% of the text removed.

Setting (50% text)Best edge-free system ↑AG-RAG static ↑AG-RAG lead
2Wiki-Extra, paper's answering model44.5 (char-trigram BM25)60.4+15.9
2Wiki-Extra, DeepSeek V4.1 Flash (open weights)42.5 (text agent)73.8+31.3
2Wiki-Extra, Qwen3-8B (open weights)35.6 (text agent)41.9+6.3
MultiHop-RAG news corpus, paper's answering model59.7 (text agent)55.4−4.4
  • On entity-centric data, the edge advantage holds with all three answering models. The answers there are carried by relations between entities, which the extracted triples keep even when the source passage is gone.
  • On document-centric data, such as the news corpus, it does not appear. Most of its questions (305 of 500) are comparison or temporal questions about what two specific articles report. Extracted triples capture such article-level claims poorly, so the graph has little to fall back on when those articles are removed.

We will state that the graph's contribution is preserving facts that are carried by extracted relations, and report the news result as the boundary of this claim.

Q2. Is typo robustness a property of fuzzy entity resolution rather than graph traversal? The fuzzy baselines under W1 answer this: fuzzy title lookup scores 67.7 under typos with no graph, and the entity-index-only ablation scores 63.1 with no edges. AG-RAG includes fuzzy entity resolution as a dedicated mechanism for this reason, alongside snapshot-bound reading and relation edges. We will state this attribution explicitly in the paper.

Q3. Scope of the security claim, and ingestion into the graph. Thank you for this question. The paper's threat model is post-snapshot poisoning: the graph is built from a vetted corpus, and malicious documents enter the text collection afterwards, as in deployments where the knowledge graph is curated offline while the document store keeps growing. The reported ASR numbers should be read under this threat model, and we will state it explicitly in the abstract and the threat-model section.

Poisoning at ingestion time, where malicious documents are admitted into the trusted source itself, is a different threat model. The attacker then controls what the system treats as trusted knowledge, so any defence has to decide which sources to trust. In our poisoning condition, HippoRAG 2 and LinearRAG index the poison passages through their normal document path: HippoRAG 2 extracts triples from them with its own OpenIE prompts and adds them to its graph, and LinearRAG adds them to its entity graph and chunk index. Neither system has a trusted snapshot, so their reported ASR already reflects poison entering the graph without vetting, which corresponds to the "No checks" row below, although each system uses its own extraction pipeline. For completeness, we also measured AG-RAG under this threat model. We extracted the test questions' 1,000 poison passages into triples and committed them to the graph under four write policies, with document links, so the poison documents also become readable through their entities. The genuine documents and facts stay in the graph, so the system faces conflicting evidence. "Exposure" is the share of questions where the agent read a poison passage (lower is better).

GraphTriples committedStatic F1 ↑Static ASR ↓Static exposure ↓Self-impr. F1 ↑Self-impr. ASR ↓Self-impr. exposure ↓
Frozen snapshot (paper)061.82.4%0%67.22.2%0%
No checks3,09737.451.6%11%23.967.1%79%
Grounding + verifier2,48841.046.0%13%27.062.9%78%
+ contradiction check2,33543.544.0%13%29.560.0%78%
+ trusted corroboration72459.711.4%12%51.226.8%50%
  • In its threat model, AG-RAG's snapshot-bound reading reduces ASR to near chance. With a frozen snapshot, AG-RAG has 2.2-2.4% ASR, against 54.8-64.2% for the matched baselines.
  • Under ingestion, static AG-RAG is no worse than the baselines. It reaches 51.6% ASR, below LinearRAG (54.8%) and HippoRAG 2 (64.2%) with matched readers, on the same poison passages ingested through each system's own pipeline. No system in our comparison is robust when the trusted source itself is poisoned.
  • Content-level checks cannot detect a consistent lie. Checking each triple against its source document helps little (46.0% and 44.0%), because the poison document does state the false fact.
  • AG-RAG's provenance tracking enables the defence that works. Because AG-RAG records which document each fact comes from, it can require corroboration from a trusted snapshot document before a new fact is used. This lowers static ASR to 11.4%. The cost is coverage: in a control with 300 genuine new documents, it also quarantines 95.8% of genuine new facts.
  • The stress-aware rule protects against the threats it checks. The self-improved harness reaches 67.1% ASR here. Its tuned policy reads documents about 2.5× as often, which is safe under the frozen snapshot it was checked on, but carries the poison once poison documents are linked to entities. Adding ingestion poisoning as a gate condition is a direct extension of the same rule.

We will add ingestion-time poisoning to the paper as a separate threat model with these results, move this table into the main text, and discuss source-trust mechanisms as future work.

Reviewer_jTjoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies agent‑style Graph‑RAG by introducing stress‑aware harness self‑improvement instead of optimizing agent configuration only using clean validation data. Candidate harness configurations (tool budgets, ranking strategies, tool‑use policies, short strategy notes) must pass both clean‑set performance checks and multiple stress‑condition checks (typos, 50% unavailable text, poisoning attacks). Configurations that degrade robustness are rejected even if they achieve higher clean‑set F1. Experiments show that AG‑RAG achieves competitive performance on standard clean multi‑hop QA benchmarks, while obtaining drastically better robustness under stress compared with HippoRAG 2 and LinearRAG. They also construct the 2Wiki‑Extra test set for robustness evaluation.

Critical strengths
  1. The paper demonstrates that graph‑derived robustness benefits can be erased by clean‑only agent optimization, since the optimizer will favor enabling unsafe free‑text retrieval to boost clean‑set scores.
  2. Three representative failure modes (entity typos, source‑document unavailability, post‑construction poisoning) are defined, together with the released 2Wiki‑Extra benchmark.
  3. Component‑wise ablations clearly assign robustness gains to each graph primitive.
  4. It includes comparison experiments against strong published Graph‑RAG baselines.
Critical weaknesses
  1. The high‑level idea of filtering candidate configurations under multi‑constraints (main‑task performance plus out‑of‑distribution / robustness constraints) is a well‑established paradigm. The paper applies this known paradigm to Graph‑RAG harness search without proposing a new optimization algorithm.
  2. The framework can only select among pre‑specified options for budget, ranking and tool‑policy. It cannot invent new agent primitives or logic. If no satisfactory candidate exists within this predefined space, the method has no fallback mechanism to produce a good solution.
  3. As acknowledged in the appendix, the robustness advantage largely disappears when poison facts are ingested during the graph‑extraction phase.
  4. It remains unvalidated whether the stress‑aware‑selected harness and the core empirical observations transfer to other LLMs, e.g., open‑weight LLMs.
  5. Improvements come purely from picking better agent runtime configurations. The framework cannot fix fundamental reasoning weaknesses of the underlying LLM.
Questions for authors

Please address the concerns described in the above weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer jTjo

Thank you for the thoughtful review, and for recognizing that clean-only optimization can erase graph-derived robustness. In all tables, ↑ means higher is better and ↓ means lower is better. ASR is the attack success rate under poisoning, the share of poisoned questions answered with the attacker's target answer, so lower is better.

W1. Relation to established paradigms. Thank you for pointing us to this line of work. Our promotion rule is related to constrained and safe policy improvement [1-3], and we will cite and discuss these. Our contribution is the AG-RAG framework and the empirical finding that clean-only tuning of an agent harness can silently adopt a provenance bypass that erases poisoning robustness. It recurs on a news corpus (+50.8 ASR points) and with DeepSeek V4.1 Flash (+10.5). The table below shows the results of a new control experiment. It uses the same streams, gates and test split as the paper, with 2 proposer seeds per arm:

GrammarRuleStress avg ↑Poisoned ↑ASR ↓
(static harness)-60.661.82.4%
Fullclean-only62.762.114.2%
Fullstress-aware (paper)64.567.22.2%

We can see that:

  • AG-RAG's self-improvement makes the agent stronger without giving up robustness. With the stress-aware rule, harness search raises the stress average from 60.6 to 64.5 and cuts turns by about 16%, while ASR stays at 2.2%.
  • The stress-aware rule blocks the provenance bypass automatically. Given the full grammar, clean-only selection adopted text search in one of its two seeds. The table's 14.2% ASR is the average of the two: the seed that adopted text search reached 26.0%, while the other stayed at 2.4%. Under the stress-aware rule, none of the four runs in the paper promoted text search, because its poisoning check rejects edits that raise ASR on the gate questions.

W2. Fixed grammar. Thank you for this point. The fixed grammar is what makes AG-RAG's self-improvement safe to deploy. Every edit is typed and auditable, so each promoted harness can be inspected and explained. A candidate is promoted only if it passes the clean and stress checks on held-out gate questions; when no candidate passes, AG-RAG keeps the incumbent, preferring no change over an unverified one. The gate is used in practice: in the paper's stress-aware runs, 4 of 6 generations kept the incumbent, and in the new runs 11 of 18 did. We will discuss extending the grammar with new primitives, which the same gate would check before promotion.

W3. Ingestion poisoning. Thank you for raising this. The paper's threat model is post-snapshot poisoning: the graph is built from a vetted corpus, and malicious documents enter the text collection afterwards, as in deployments where the knowledge graph is curated offline while the document store keeps growing. The reported ASR numbers should be read under this threat model, and we will state it explicitly in the abstract and the threat-model section.

Poisoning at ingestion time, where malicious documents are admitted into the trusted source itself, is a different threat model. The attacker then controls what the system treats as trusted knowledge, so any defence has to decide which sources to trust. In our poisoning condition, HippoRAG 2 and LinearRAG index the poison passages through their normal document path: HippoRAG 2 extracts triples from them with its own OpenIE prompts and adds them to its graph, and LinearRAG adds them to its entity graph and chunk index. Neither system has a trusted snapshot, so their reported ASR already reflects poison entering the graph without vetting, which corresponds to the "No checks" row below, although each system uses its own extraction pipeline. For completeness, we also measured AG-RAG under this threat model. We extracted the test questions' 1,000 poison passages into triples and committed them to the graph under four write policies, with document links, so the poison documents also become readable through their entities. The genuine documents and facts stay in the graph, so the system faces conflicting evidence. "Exposure" is the share of questions where the agent read a poison passage (lower is better).

GraphTriples committedStatic ASR ↓Static exposure ↓Self-impr. ASR ↓Self-impr. exposure ↓
Frozen snapshot (paper)02.4%0%2.2%0%
No checks3,09751.6%11%67.1%79%
Grounding + verifier2,48846.0%13%62.9%78%
+ contradiction check2,33544.0%13%60.0%78%
+ trusted corroboration72411.4%12%26.8%50%
  • In its threat model, AG-RAG's snapshot-bound reading reduces ASR to near chance. With a frozen snapshot, AG-RAG has 2.2-2.4% ASR, against 54.8-64.2% for the matched baselines.
  • Under ingestion, static AG-RAG is no worse than the baselines. It reaches 51.6% ASR, below LinearRAG (54.8%) and HippoRAG 2 (64.2%) with matched readers, on the same poison passages ingested through each system's own pipeline. No system in our comparison is robust when the trusted source itself is poisoned.
  • Content-level checks cannot detect a consistent lie. Checking each triple against its source document helps little (46.0% and 44.0%), because the poison document does state the false fact.
  • AG-RAG's provenance tracking enables the defence that works. Because AG-RAG records which document each fact comes from, it can require corroboration from a trusted snapshot document before a new fact is used. This lowers static ASR to 11.4%. The cost is coverage: in a control with 300 genuine new documents, it also quarantines 95.8% of genuine new facts.
  • The stress-aware rule protects against the threats it checks. The self-improved harness reaches 67.1% ASR here. Its tuned policy reads documents about 2.5× as often, which is safe under the frozen snapshot it was checked on, but carries the poison once poison documents are linked to entities. Adding ingestion poisoning as a gate condition is a direct extension of the same rule.

We will add ingestion-time poisoning to the paper as a separate threat model with these results, move this table into the main text, and discuss source-trust mechanisms as future work.

W4. Open-weight models. We tested two open-weight answering models, DeepSeek V4.1 Flash and Qwen3-8B, with no prompt changes and at most 1.8% failed answers.

DeepSeek V4.1 Flash:

SystemTypos ↑50% text ↑Poisoned ↑Stress avg ↑ASR ↓
Text agent63.742.535.447.259.2%
HippoRAG 2 (matched)61.750.224.845.572.0%
LinearRAG (matched)43.445.816.135.175.4%
AG-RAG static68.773.874.672.41.8%
AG-RAG self-improved70.274.976.773.91.9%

Qwen3-8B:

SystemTypos ↑50% text ↑Poisoned ↑Stress avg ↑ASR ↓
Text agent22.935.614.124.284.6%
HippoRAG 2 (matched)47.833.823.435.074.4%
LinearRAG (matched)33.125.817.725.572.8%
AG-RAG static35.141.946.941.310.6%
AG-RAG self-improved34.345.046.541.910.7%
  • AG-RAG's advantage does not depend on a proprietary model. With both open-weight models, AG-RAG has the highest stress average (73.9 with DeepSeek and 41.9 with Qwen3-8B, against at most 47.2 and 35.0 for the baselines in the tables).
  • Its poisoning protection is structural, so it holds for every model. Graph agents read no poison, and their ASR stays within 0.6 points of the model's own chance level, measured on the clean corpus (DeepSeek: 1.8-1.9% against 1.6%; Qwen3-8B: 10.6-10.7% against 10.1-10.6%). The text agent and graph-RAG baselines reach 59.2-84.6%.
  • Relation edges help with every model. AG-RAG leads every baseline under missing text, by up to 31 F1.
  • With a capable open model, AG-RAG leads every baseline under every stress. With DeepSeek, it also leads under typos (by 7 and 25 F1 over HippoRAG 2 and LinearRAG), and the stress-aware rule remains necessary, as W1 shows.
  • With a small 8B model, AG-RAG still leads the text agent under typos (35.1 against 22.9) but trails HippoRAG 2 (47.8); see W5.

We will add these results.

W5. Model reasoning. Thank you for this point. AG-RAG's robustness does not rely on the model's reasoning: as W4 shows, its structural protections hold with all three answering models. Harness search adds to this by improving how a given model uses its tools, and harnesses tuned with the paper's model transfer to DeepSeek without retuning, adding up to about 2 F1. The gain is smaller with the 8B model, which stops earlier (2.8 tool calls per question under typos, against 4.3 for DeepSeek and 5.0 for the paper's model). We will add this discussion and state the scope clearly.

References

[1] Achiam, Held, Tamar, Abbeel. Constrained Policy Optimization. ICML 2017.

[2] Laroche, Trichelair, Tachet des Combes. Safe Policy Improvement with Baseline Bootstrapping. ICML 2019.

[3] Thomas, Castro da Silva, Barto, Giguere, Brun, Brunskill. Preventing Undesirable Behavior of Intelligent Machines. Science, 2019.

Dear Reviewer jTjo,

We sincerely appreciate your careful review and constructive suggestions. We have provided additional analyses and experiments to address your concerns and would welcome any further feedback or discussion!

Reviewer_7s5eOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that Graph RAG systems are usually built and evaluated on clean queries, where their advantage over text retrieval is modest, and that graph structure should instead be valued for its behavior when retrieval fails. It introduces AG-RAG, an agentic Graph RAG framework with two parts. First, an MCP server exposes a corpus-derived knowledge graph, built with the GFM-RAG extraction pipeline, through tools for fuzzy entity resolution, single and batched neighbour expansion, and document retrieval ranked by how many selected entities each document mentions. Fuzzy resolution returns character-trigram-prefiltered and RapidFuzz-reranked suggestions instead of an empty result. Retrieval is provenance-bound: the agent reads documents only through entity links, so a passage that enters the text collection after the graph snapshot has no link and cannot be returned. An optional free-text BM25 tool over the current collection can be enabled by the harness. Second, a stress-aware self-improvement loop searches a typed configuration grammar covering budgets, ranking, tool policy, optional text search or path finding, and short strategy notes. In each generation the incumbent answers 110 stream questions and an LLM proposer returns three type-checked edits; each edit must pass a clean check on 200 held-out validation questions and then a stress check on 60 questions for each of three stresses, and the best passing edit becomes the next incumbent. The clean-only rule is the control. The three stresses are character-level typos in entity names, removal of 50% of source text while keeping graph edges, and two answer-targeted poisoned passages injected after the graph is built. Evaluation uses the standard GFM-RAG test sets for HotpotQA, 2Wiki, and MuSiQue on clean questions, and a new 2Wiki-Extra test set of 500 questions from the 2WikiMultiHopQA development set, disjoint from the standard test set, for all stress, ablation, and self-improvement results. With the same answering model, AG-RAG (self-improved) reaches 72.7/70.5/52.0 F1 on the three standard clean sets, roughly competitive with HippoRAG 2 and LinearRAG; on 2Wiki-Extra it reaches 66.5 clean, 63.0 under typos, 63.2 under removed text, and 67.2 under poisoning, with 2.2% attack success rate against 57.2% for HippoRAG 2 and 48.0% for LinearRAG. Ablations attribute typo robustness to fuzzy suggestions, missing-text robustness to retained relation edges, and poisoning resistance to provenance-bound reading. Under the clean-only promotion rule, both the LLM proposer and matched-budget random search adopt free-text search and attack success rises to 13.8-16.6%. The paper concludes that the objective used for self-improvement is part of the trust boundary.

Critical strengths
  1. The paper identifies a real failure of clean-only optimization and demonstrates it with a controlled experiment. The stress-aware and clean-only promotion rules are interesting,
  2. The stress suite is defined precisely and the ablations are tied to individual mechanisms.
  3. The main comparison uses a genuinely held-out question set and reports paired intervals.
  4. The system design is simple and auditable. Costs and scope are reported.
Critical weaknesses
  1. The clean comparison against HippoRAG 2 is reader-confounded, and the headline clean gap on 2Wiki-Extra is partly an artifact of the baseline's prompt. The paper states that, with HippoRAG 2's prompt, the shared answering model replies "not specified" on 22.4% of 2Wiki questions even though 91.9% of gold documents are in its top five (Page 7, §5.3; lines 350–355). This is a reader failure, not a retrieval failure, yet Table 2 reports HippoRAG 2 at 50.1 clean F1 on 2Wiki-Extra against 66.5 for AG-RAG, and the abstract and Section 5.3 use that gap to support the "competitive on clean, far more robust under stress" framing (Abstract, p. 1; Table 2, pp. 6–7). A reader-matched baseline — adapting the answer prompt so that every system is required to produce an answer when gold evidence is retrieved — is needed before the clean comparison can be read as a system-level result. Without it, the clean advantage is overstated and the stress comparison, which is the paper's real contribution, is unnecessarily entangled with a baseline implementation artifact.

  2. The poisoning result establishes an access-control property, not a property of graph structure, and the abstract's attribution to graph structure is not supported. Section 3.3 states that the protection "comes from which documents a tool may reach, not from edges", and Section 5.4 states that "a text tool restricted to the same snapshot might provide it with higher clean accuracy" (pp. 3–4, §3.3; lines 182–185; Page 7, §5.4; lines 370–373). The ablation confirms this: the entity-index-only variant, which has no relation edges, still holds ASR at 2.8% because post-snapshot passages have no entity links (Table 3, p. 7). The abstract nevertheless concludes that "graph structure provides complementary robustness against retrieval failures" (Abstract, p. 1; lines 27–29). The evidence supports this conclusion for fuzzy resolution under typos and for edge retention under missing text, but for poisoning the mechanism is snapshot-bounded reading, which a timestamped text index would reproduce. The paper acknowledges the mechanism in the body but not in its headline claim, and the distinction matters for practitioners deciding whether they need a graph or only an immutable index.

  3. The self-improvement contribution may reduce to "do not enable free-text search", which the paper itself leaves untested. The accepted edits adjust budgets, ranking, and tool policy without changing the provenance boundary, and every text-search edit that passed the clean check failed the stress check; the paper states that "removing the text-search tool from the grammar might recover most of the gain (untested)" (Page 9, §5.6; lines 454–457; Page 8, §5.5; lines 406–413). If a grammar without the text-search option, plus the same clean validation, reproduces most of the reported +4.0 clean F1 and the turn reduction, then the stress-aware promotion rule is not doing the work — the static design decision to keep retrieval behind the snapshot is. This is a decisive question for the paper's second contribution, and it can be answered with an experiment the authors already have the infrastructure to run. The random-search control shows the rule separates from the proposer, but not from the simpler alternative of removing the harmful tool.

Questions for authors
  1. If the answer prompt is adapted so that a system must answer when gold evidence is in the retrieved set, what are HippoRAG 2's clean and poisoned F1 on 2Wiki-Extra?
  2. What does a snapshot-restricted BM25 tool do? You state that a text tool restricted to the graph snapshot would share the poisoning protection. Running that condition would let readers see whether the graph edges add anything beyond an immutable, timestamped text index on clean and stressed questions.
  3. How sensitive is the result to the answering model and corpus? Appendix A acknowledges one closed model, one proposer, two seeds, and Wikipedia-style QA. Would the ASR gap and the self-improvement gain survive an open-weight answering model and a non-Wikipedia corpus with less templated relations?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 7s5e

Thank you for the detailed and constructive review. We ran an experiment for each of the three concerns. In all tables, ↑ means higher is better and ↓ means lower is better. ASR is the attack success rate under poisoning, the share of poisoned questions answered with the attacker's target answer, so lower is better.

W1 / Q1. Reader-matched baselines. Thank you for this question. We re-read each baseline's saved retrieved passages with its native prompt plus one shared answer rule, which tells the reader that the answer can be found in the given passages and that it should always give its best short answer, never replying that the information is not provided, not available or not specified. Retrieval is unchanged; only the final answering step differs.

SystemTypos ↑50% text ↑Poisoned ↑ASR ↓2Wiki-Extra clean ↑HotpotQA ↑2Wiki ↑MuSiQue ↑
HippoRAG 2, native41.727.031.957.2%50.175.864.148.0
HippoRAG 2, matched50.745.228.364.2%60.377.973.053.1
LinearRAG, native35.128.035.048.0%56.773.970.946.4
LinearRAG, matched41.549.630.554.8%62.776.976.452.1
AG-RAG self-improved63.063.267.22.2%66.572.770.552.0
  • HippoRAG 2 with the matched reader scores 60.3 clean F1 and 28.3 poisoned F1 on 2Wiki-Extra (native reader: 50.1 and 31.9).
  • The matched reader helps the baselines answer, but makes them more vulnerable to poisoning. HippoRAG 2's abstentions on clean 2Wiki-Extra questions fall from 37.2% to 6.4%, and LinearRAG's from 27.4% to 5.0%. Here an abstention is any answer that declines to answer, such as "cannot be determined from the provided information". Appendix G reports 9.8% for HippoRAG 2 because it counted only answers containing the literal phrase "not specified", unlike the 22.4% it reports for standard 2Wiki. Both baselines gain 2-9 F1 on the standard benchmarks and improve under typos and missing text. However, once the reader commits to an answer, it also adopts the attacker's answer more often: ASR rises from 57.2% to 64.2% for HippoRAG 2 and from 48.0% to 54.8% for LinearRAG. Answering readily and resisting poisoning pull in opposite directions for these systems.
  • AG-RAG does not face this trade-off. It abstains about as rarely as the matched readers (5.8-7.8% of clean questions, against 5.0-6.4%) and keeps ASR at 2.2%, because snapshot-bound reading keeps injected passages out of what it reads.
  • AG-RAG still leads on 2Wiki-Extra. It scores 66.5 clean and 67.2 poisoned F1, leads both matched baselines by 12-39 F1 under every stress condition, and its stress average is 64.5, against 41.4 for HippoRAG 2 and 40.5 for LinearRAG. On the standard benchmarks, the matched baselines are ahead on HotpotQA and 2Wiki, and AG-RAG is within 1.1 F1 on MuSiQue.

We will report matched baselines in the main table and correct the "not specified" rates in Appendix G.

W2 / Q2. Access control vs graph structure. Thank you for this suggestion. The snapshot-restricted BM25 agent reaches 2.4% ASR and 69.5 poisoned F1: the poisoning protection comes from snapshot-bound reading, one of AG-RAG's three mechanisms, while relation edges add +16.1 F1 under missing text. AG-RAG combines both and is the only system robust to all three stresses, with a stress average of 60.6-64.5 against 55.7 for the snapshot text agent (see General Response). We will attribute each result to its mechanism in the abstract.

W3. Self-improvement vs removing text search. Thank you for proposing this control. We ran an experiment with the same streams, gates and test split, with 2 proposer seeds per arm:

GrammarRuleClean ↑Turns per question (clean) ↓Stress avg ↑Poisoned ↑ASR ↓
(static harness)-62.54.6660.661.82.4%
Fullclean-only68.23.8962.762.114.2%
No text searchclean-only67.53.8665.067.71.8%
Fullstress-aware (paper)66.53.9464.567.22.2%
  • AG-RAG's self-improvement makes the agent stronger without giving up robustness. With the stress-aware rule, harness search raises clean F1 from 62.5 to 66.5, cuts turns per question from 4.66 to 3.94 (about 16% fewer), and raises the stress average from 60.6 to 64.5, while ASR stays at 2.2%.
  • The stress-aware rule blocks the provenance bypass automatically. Given the full grammar, clean-only selection adopted text search in one of its two seeds. The table's 14.2% ASR is the average of the two: the seed that adopted text search reached 26.0%, while the other stayed at 2.4%. Text search bought no clean gain in return: that seed scored 68.2 clean F1, against 68.1 for the seed that did not adopt it. Under the stress-aware rule, none of the four runs in the paper promoted text search, because its poisoning check rejects edits that raise ASR on the gate questions.

On the no-text-search control. Removing text search by hand also recovers the clean gain and the turn reduction: clean F1 rises to 67.5 and turns fall to 3.86 per question, with 1.8% ASR. This control scores slightly higher than our stress-aware harness (65.0 against 64.5 stress-average F1). Importantly, its search space is a subset of our full grammar, which already contains every configuration without text search. The control excludes the text-search edit by hand, whereas our rule rejects it automatically. The gap is within run-to-run variation: the clean and poisoned differences are not statistically significant, and running our stress-aware rule on the same restricted grammar gives 64.1, slightly below both.

Our current search explores only a limited number of candidates and does not guarantee finding the best configuration within the full grammar. A more effective search algorithm could find similarly strong configurations automatically, without manually restricting the search space. Improving search efficiency and exploration is an interesting direction for future work.

Nevertheless, these results show that stress-aware promotion achieves comparable performance while preventing harmful configurations, such as text search here, from being adopted during optimization, without anyone having to identify them in advance. We did not test bypasses outside the listed grammar.

We will add this control and revise the section 5.6 sentence accordingly.

Q3. Other models and corpora.

Open-weight models. We ran additional experiments with two open-weight answering models, DeepSeek V4.1 Flash and Qwen3-8B, with the same prompts, tools, frozen harnesses and baseline contexts.

DeepSeek V4.1 Flash:

SystemTypos ↑50% text ↑Poisoned ↑Stress avg ↑ASR ↓
Text agent63.742.535.447.259.2%
HippoRAG 2 (matched)61.750.224.845.572.0%
LinearRAG (matched)43.445.816.135.175.4%
AG-RAG static68.773.874.672.41.8%
AG-RAG self-improved70.274.976.773.91.9%

Qwen3-8B:

SystemTypos ↑50% text ↑Poisoned ↑Stress avg ↑ASR ↓
Text agent22.935.614.124.284.6%
HippoRAG 2 (matched)47.833.823.435.074.4%
LinearRAG (matched)33.125.817.725.572.8%
AG-RAG static35.141.946.941.310.6%
AG-RAG self-improved34.345.046.541.910.7%
  • Both models: AG-RAG has the highest stress average (73.9 with DeepSeek and 41.9 with Qwen3-8B, against at most 47.2 and 35.0 for the baselines in the tables). Graph agents read no poison, and their ASR stays within 0.6 points of the model's own chance level, measured on the clean corpus (DeepSeek: 1.8-1.9% against 1.6%; Qwen3-8B: 10.6-10.7% against 10.1-10.6%). AG-RAG leads every baseline under missing text, by up to 31 F1.
  • DeepSeek: AG-RAG also leads both baselines under typos (by 7 and 25 F1). The clean-only bypass also recurs: the clean-only harness that adopted text search has 10.5 more ASR points than the stress-aware harness.
  • Qwen3-8B: under typos, AG-RAG still leads the text agent (35.1 against 22.9) but trails HippoRAG 2 (47.8). The 8B model stops after fewer tool calls (2.8 per question against 4.3 for DeepSeek).

News corpus. We ran additional experiments on MultiHop-RAG (609 news articles, 500 questions) with the paper's answering model. Every harness is frozen from 2Wiki-Extra, so nothing was tuned on this corpus. Chance ASR is the rate at which the attacker's target answer is given on the clean corpus, without any poison.

SystemTypos ↑50% text ↑Poisoned ↑ASR ↓Chance ASR
Text agent65.559.77.964.6%3.4%
Snapshot text agent65.559.766.34.6%3.4%
AG-RAG clean-only harness (adopted text search)--14.156.2%4.2%
AG-RAG static58.755.459.44.6%4.4%
AG-RAG self-improved60.959.962.75.4%5.1%
  • AG-RAG's poisoning protection transfers to a new domain without retuning. Its graph agents read no poison and stay at chance ASR, while the text agent reaches 64.6%. The harnesses tuned on 2Wiki-Extra still add 2.2-4.5 F1 under the three stresses, and the clean-only bypass recurs (+50.8 ASR), so the stress-aware rule remains necessary on this corpus.
  • AG-RAG is most beneficial on entity- and relation-centric data. On 2Wiki-Extra, answers are chains of relations between named entities, so all three mechanisms contribute: fuzzy resolution recovers misspelled names, and relation edges keep facts reachable when their text is missing. MultiHop-RAG is document-centric: most questions (305 of 500) compare what two news articles report, extracted triples capture such article-level claims poorly, and the text agent is barely affected by typos. On such data, snapshot-bound reading provides AG-RAG's robustness, and a snapshot-restricted text agent performs as well or better. We see AG-RAG as best suited to knowledge organised around entities and their relations, with snapshot-bound reading as the component that carries over to document-centric corpora.

We will add these results and a discussion of which kinds of data benefit from each mechanism.

Dear Reviewer 7s5e,

Thank you again for your thoughtful review. We have conducted additional experiments to address your concerns and look forward to hearing your thoughts!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all three reviewers for careful and constructive reviews. Each main concern could be tested, so we ran new experiments. Results are on the 2Wiki-Extra test set (n = 500), unless noted. In all tables, ↑ means higher is better and ↓ means lower is better.

AG-RAG's contributions:

  1. One agentic framework that combines three mechanisms, each targeting a different retrieval failure:
    • fuzzy entity resolution for misspelled names;
    • snapshot-bound (provenance-bound) reading for passages injected after graph construction;
    • relation-edge traversal for facts whose source text is unavailable.
  2. Robust across diverse stresses. Each mechanism alone handles one stress. Together they let one system handle all three: on 2Wiki-Extra, AG-RAG outperforms HippoRAG 2 and LinearRAG under every stress condition.
  3. A finding about agent self-improvement. Clean-only tuning can silently adopt a provenance bypass that erases poisoning robustness. Stress-aware promotion prevents it for the threats it checks. The finding recurs on a news corpus and with an open-weight model (see the replies to 7s5e and jTjo).

Main comparison: AG-RAG outperforms every other method under stress. HippoRAG 2 and LinearRAG use reader-matched prompts (see the reply to 7s5e). "Stress avg" is the mean F1 over the three stress conditions: typos, 50% of the text removed, and poisoning.

SystemMechanismsStress avg ↑
Text agent (BM25)-40.6
Word + char-trigram BM25 (new)-43.0
Fuzzy title lookup (new)fuzzy51.5
Snapshot text agent (new)snapshot55.7
Entity index only (ablation)fuzzy + snapshot53.6
HippoRAG 2 (matched)-41.4
LinearRAG (matched)-40.5
AG-RAG staticall three60.6
AG-RAG self-improvedall three64.5
  • AG-RAG outperforms every other method. The self-improved harness reaches 64.5 and the static one 60.6. The published graph-RAG baselines reach 41.4 (HippoRAG 2) and 40.5 (LinearRAG), so AG-RAG leads them by 19-24 F1. The strongest other system, the snapshot text agent, reaches 55.7, and AG-RAG leads it by 4.9 (static) and 8.8 (self-improved).
  • Each single-mechanism system is held back by the stresses it does not cover: fuzzy title lookup by poisoning, the snapshot text agent by typos and missing text, and the entity index without edges by missing text.

The per-condition breakdown is in the reply to 5AkG. We will update the abstract and contributions to present AG-RAG this way, with each robustness result attributed to its mechanism.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.