- Rating
- 3(+1)
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes AG-RAG, an agentic Graph RAG framework that exposes a corpus-derived graph through MCP tools and uses stress-aware self-improvement to optimize the agent's retrieval harness. The paper studies three retrieval failure conditions: misspelled entities, unavailable source text, and adversarially injected passages. The central idea is to accept harness modifications only when they improve or preserve performance under both clean and stressed validation conditions.
The paper reports compelling robustness improvements on the proposed 2Wiki-Extra benchmark. The self-improved AG-RAG achieves 66.5 clean F1, 63.0 typo F1, 63.2 F1 when 50% of source text is unavailable, and 2.2% poisoning ASR. In contrast, the BM25 text agent obtains higher clean F1 (69.1) but substantially worse performance under stress, including 68.4% poisoning ASR. The comparison between clean-only and stress-aware self-improvement is also interesting: clean-only optimization tends to favor free-text search, whereas stress-aware promotion rejects such edits.
Critical strengths
The paper addresses an important question: whether agentic retrieval systems should optimize for robustness under retrieval failures rather than only clean benchmark accuracy. The observation that clean-only optimization can select a tool that improves nominal accuracy while substantially weakening poisoning robustness is useful and practically relevant.
The proposed promotion rule provides a simple mechanism for incorporating robustness constraints into automated harness optimization. The comparison with clean-only optimization is particularly informative.
Critical weaknesses
-
The paper frames the three stress conditions as evidence that graph structure provides complementary robustness. However, the mechanisms underlying these results should be separated more carefully. For typos, AG-RAG uses a character-trigram index followed by RapidFuzz reranking. Thus, the experiment does not establish that graph edges are necessary for typo robustness. More importantly, the paper seems does not compare against a directly comparable non-graph fuzzy retrieval mechanism such as BM25 + character n-gram matching or a fuzzy text/entity index.For poisoning, the protection primarily comes from the snapshot/provenance boundary. For missing text, in contrast, the large degradation after removing graph edges provides substantially stronger evidence that graph structure itself contributes robustness.
-
The text agent is based on BM25, while AG-RAG includes a dedicated fuzzy entity-resolution mechanism. The paper does evaluate an LLM spelling-fix intervention, but this is not the most direct comparison to the mechanism used by AG-RAG. The paper itself notes that a fuzzy text index remains untested.
Questions for authors
Clarifications
- The paper attributes robustness to three mechanisms: fuzzy entity matching, graph edges, and snapshot/provenance-bound retrieval. Could the authors clarify which parts of the reported gains they consider specifically attributable to the graph structure itself, rather than to the entity index or provenance restrictions?
- The entity-index-only ablation retains essentially the same typo robustness as the full system. Does this mean that the authors view typo robustness primarily as a property of fuzzy entity resolution rather than graph traversal? If so, could the paper make this attribution more explicit?
- The main poisoning experiments concern malicious documents added after graph construction. Should the security claim therefore be interpreted specifically as robustness to post-snapshot text-index poisoning, rather than poisoning in general? In particular, how should the reported ASR numbers be interpreted when the malicious documents are ingested into the graph?
Suggestions
- A directly comparable fuzzy text-retrieval baseline would help isolate the contribution of the graph from fuzzy matching. This seems particularly important because the current typo experiment does not establish that graph edges are necessary for typo robustness.
- Since the paper itself notes that provenance-bound access does not require graph edges, comparing against a text-only agent restricted to the same snapshot would more cleanly quantify the benefit of graph structure for poisoning robustness.
5 comments
Q1. What is attributable to graph structure itself? Keeping facts reachable when their source text is missing. This is a benefit only the graph provides. On 2Wiki-Extra, no edge-free system we tested comes close.
Removing the edges from AG-RAG, which leaves the entity index only, drops F1 with half the text removed from 60.4 to 29.9, a loss of 30.5 F1. The table below compares AG-RAG with the best edge-free system on 2Wiki-Extra and in the new settings. In each row, the edge-free system is the strongest one we ran in that setting. All values are F1 with 50% of the text removed.
| Setting (50% text) | Best edge-free system ↑ | AG-RAG static ↑ | AG-RAG lead |
|---|---|---|---|
| 2Wiki-Extra, paper's answering model | 44.5 (char-trigram BM25) | 60.4 | +15.9 |
| 2Wiki-Extra, DeepSeek V4.1 Flash (open weights) | 42.5 (text agent) | 73.8 | +31.3 |
| 2Wiki-Extra, Qwen3-8B (open weights) | 35.6 (text agent) | 41.9 | +6.3 |
| MultiHop-RAG news corpus, paper's answering model | 59.7 (text agent) | 55.4 | −4.4 |
- On entity-centric data, the edge advantage holds with all three answering models. The answers there are carried by relations between entities, which the extracted triples keep even when the source passage is gone.
- On document-centric data, such as the news corpus, it does not appear. Most of its questions (305 of 500) are comparison or temporal questions about what two specific articles report. Extracted triples capture such article-level claims poorly, so the graph has little to fall back on when those articles are removed.
We will state that the graph's contribution is preserving facts that are carried by extracted relations, and report the news result as the boundary of this claim.
Q2. Is typo robustness a property of fuzzy entity resolution rather than graph traversal? The fuzzy baselines under W1 answer this: fuzzy title lookup scores 67.7 under typos with no graph, and the entity-index-only ablation scores 63.1 with no edges. AG-RAG includes fuzzy entity resolution as a dedicated mechanism for this reason, alongside snapshot-bound reading and relation edges. We will state this attribution explicitly in the paper.
Q3. Scope of the security claim, and ingestion into the graph. Thank you for this question. The paper's threat model is post-snapshot poisoning: the graph is built from a vetted corpus, and malicious documents enter the text collection afterwards, as in deployments where the knowledge graph is curated offline while the document store keeps growing. The reported ASR numbers should be read under this threat model, and we will state it explicitly in the abstract and the threat-model section.
Poisoning at ingestion time, where malicious documents are admitted into the trusted source itself, is a different threat model. The attacker then controls what the system treats as trusted knowledge, so any defence has to decide which sources to trust. In our poisoning condition, HippoRAG 2 and LinearRAG index the poison passages through their normal document path: HippoRAG 2 extracts triples from them with its own OpenIE prompts and adds them to its graph, and LinearRAG adds them to its entity graph and chunk index. Neither system has a trusted snapshot, so their reported ASR already reflects poison entering the graph without vetting, which corresponds to the "No checks" row below, although each system uses its own extraction pipeline. For completeness, we also measured AG-RAG under this threat model. We extracted the test questions' 1,000 poison passages into triples and committed them to the graph under four write policies, with document links, so the poison documents also become readable through their entities. The genuine documents and facts stay in the graph, so the system faces conflicting evidence. "Exposure" is the share of questions where the agent read a poison passage (lower is better).
| Graph | Triples committed | Static F1 ↑ | Static ASR ↓ | Static exposure ↓ | Self-impr. F1 ↑ | Self-impr. ASR ↓ | Self-impr. exposure ↓ |
|---|---|---|---|---|---|---|---|
| Frozen snapshot (paper) | 0 | 61.8 | 2.4% | 0% | 67.2 | 2.2% | 0% |
| No checks | 3,097 | 37.4 | 51.6% | 11% | 23.9 | 67.1% | 79% |
| Grounding + verifier | 2,488 | 41.0 | 46.0% | 13% | 27.0 | 62.9% | 78% |
| + contradiction check | 2,335 | 43.5 | 44.0% | 13% | 29.5 | 60.0% | 78% |
| + trusted corroboration | 724 | 59.7 | 11.4% | 12% | 51.2 | 26.8% | 50% |
- In its threat model, AG-RAG's snapshot-bound reading reduces ASR to near chance. With a frozen snapshot, AG-RAG has 2.2-2.4% ASR, against 54.8-64.2% for the matched baselines.
- Under ingestion, static AG-RAG is no worse than the baselines. It reaches 51.6% ASR, below LinearRAG (54.8%) and HippoRAG 2 (64.2%) with matched readers, on the same poison passages ingested through each system's own pipeline. No system in our comparison is robust when the trusted source itself is poisoned.
- Content-level checks cannot detect a consistent lie. Checking each triple against its source document helps little (46.0% and 44.0%), because the poison document does state the false fact.
- AG-RAG's provenance tracking enables the defence that works. Because AG-RAG records which document each fact comes from, it can require corroboration from a trusted snapshot document before a new fact is used. This lowers static ASR to 11.4%. The cost is coverage: in a control with 300 genuine new documents, it also quarantines 95.8% of genuine new facts.
- The stress-aware rule protects against the threats it checks. The self-improved harness reaches 67.1% ASR here. Its tuned policy reads documents about 2.5× as often, which is safe under the frozen snapshot it was checked on, but carries the poison once poison documents are linked to entities. Adding ingestion poisoning as a gate condition is a direct extension of the same rule.
We will add ingestion-time poisoning to the paper as a separate threat model with these results, move this table into the main text, and discuss source-trust mechanisms as future work.
Response to Reviewer 5AkG
Thank you for the expert and constructive review, and for the two suggested baselines. We ran both, together with several further controls. They make the role of each of AG-RAG's three mechanisms explicit: fuzzy entity resolution, snapshot-bound reading and relation edges.
All results below are on the 2Wiki-Extra test set (n = 500) with the paper's answering model, unless noted. The three mechanism columns show which of AG-RAG's mechanisms each system has. ↑ means higher is better and ↓ means lower is better. ASR is the attack success rate under poisoning, the share of poisoned questions answered with the attacker's target answer, so lower is better. "Stress avg" is the mean F1 over the three stress conditions.
The snapshot text agent matches the plain text agent under typos and 50% text, because no passages are added in those conditions, so its index is identical.
W1, W2 and Suggestion 1: a directly comparable fuzzy retrieval baseline. Thank you for this suggestion. We added two non-graph fuzzy baselines.
These results confirm the design of AG-RAG: typo robustness comes from its fuzzy name resolution mechanism, not from graph edges. On its own, however, fuzzy resolution does not handle the other two stresses:
This is the benefit of combining the mechanisms in one framework. AG-RAG keeps 59.6-67.2 F1 and 2.2-2.4% ASR across all three stresses, because it pairs fuzzy resolution with snapshot-bound reading and relation edges. Its stress average is 60.6 (static) and 64.5 (self-improved), against 51.5 for fuzzy title lookup. We will make this attribution explicit, and add both baselines to the paper.
Suggestion 2: a text-only agent restricted to the same snapshot. We ran a BM25 text agent whose index contains only the documents in the graph snapshot.
This confirms that snapshot-bound reading, AG-RAG's second mechanism, is a simple and effective poisoning defence that does not need graph edges. Alone, however, it does not handle the other two stresses: this agent scores 53.3 under typos and 44.3 under missing text, for a stress average of 55.7. AG-RAG builds snapshot-bound reading in as a default and combines it with the other two mechanisms, which raises the stress average to 60.6-64.5. We will report this baseline and attribute the poisoning result to snapshot-bound reading.