acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Retrieval and Verification: A Shared Agentic Controller for Cybersecurity RAG and GraphRAG

Abstract

Cybersecurity question answering often requires connecting evidence across heterogeneous sources and reasoning over multiple entities and relationships. Conventional Retrieval-Augmented Generation (RAG) retrieves semantically similar evidence as a flat context, limiting its ability to support cross-domain and multi-hop questions. We propose an agentic, multi-layer cybersecurity GraphRAG framework that integrates heterogeneous repositories through explicit bridge relationships and combines semantic retrieval with intent-aware graph traversal and re-ranking. A shared, bounded controller is instantiated over both flat RAG and GraphRAG to plan evidence requirements, assess retrieval sufficiency, perform targeted retries, and verify generated claims before returning a complete answer, partial answer, or abstention. We evaluate Basic RAG, Agentic RAG, GraphRAG, and Agentic GraphRAG on a 40-question cybersecurity benchmark. Agentic GraphRAG achieves the highest overall answer-quality score of 4.04/5, compared with 3.82 for GraphRAG, 3.05 for Agentic RAG, and 3.00 for Basic RAG. Relative to GraphRAG, it provides numerical improvements of 5.8% in overall answer quality, 20.5% in pooled identifier recall, and 30.9% on multi-hop questions. These gains increase mean latency from 4.23 to 30.96 seconds. The results indicate that agentic control is most effective when supported by structured, relational evidence, while revealing a substantial quality-efficiency trade-off.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic RAG, GraphRAG, Cybersecurity, Knowledge Graphs, Adaptive Retrieval, Evidence Verification, Multi-Hop Reasoning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_CWNdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to improve cybersecurity question answering when the required information is spread across several sources. The authors build a knowledge graph from MITRE ATT&CK, NVD, Atomic Red Team, and threat-intelligence reports. They then add a controller that checks whether enough evidence has been retrieved, searches again when information is missing, and checks the answer before returning it. The same controller is used with both ordinary vector retrieval and graph retrieval. The paper compares four systems on 40 questions. Agentic GraphRAG achieves the highest average answer-quality score, 4.04 compared with 3.82 for GraphRAG. It also retrieves more of the required identifiers and includes more correct identifiers in its answers. However, its average response time increases from 4.23 to 30.96 seconds. The main contribution is the combined system and the comparison of how the controller works with the two retrieval methods.

Critical strengths

1.The paper addresses a useful problem and proposes a sensible system.

Cybersecurity questions often require connecting information about vulnerabilities, attack techniques, software, and defensive measures. Organizing this information as a graph and allowing the system to search again when evidence is missing is a reasonable approach. Using the same controller with both retrieval methods is also a useful way to compare their behavior.

2.Several results support an improvement over one-shot GraphRAG.

In Table 1, accuracy increases from 3.12 to 3.83 and completeness from 2.48 to 2.92. In Table 4, both the precision and recall of identifiers in the answers improve. These results suggest that the system is recovering useful information, rather than merely producing answers that receive a higher overall score. The paper also reports cases where performance declines and clearly states the additional time cost.

Critical weaknesses

1.The comparisons do not show whether the gains come from graph retrieval or other differences between the systems.

All 40 questions contain explicit identifiers, such as CVE or ATT&CK IDs. The graph retriever can match these identifiers directly and retain their attributes, while the vector baseline only searches by similarity. The graph version can also change its traversal depth. It therefore has several advantages besides the use of graph relationships. The improvement from GraphRAG to Agentic GraphRAG is still a useful result. However, comparing the two retrieval methods does not establish why the controller helps the graph version more. A vector baseline with exact identifier matching would help answer this question. The paper also changes the generation prompts and temperature when adding the controller, as acknowledged in Sections 4.3–4.5. This is acceptable for comparing the complete systems, but it does not show how much of the improvement comes from repeated retrieval, answer checking, or the changed generation settings. Since the paper’s main contribution is to study how these parts work together, this missing analysis is important.

2.The paper does not sufficiently check the inferred relationships or show whether answer verification actually helps.

Section 3.3 explains that some CVE–technique links are inferred through software matching and do not confirm that a technique was used to exploit a vulnerability. Yet Section 4.2 includes questions asking about exactly such exploitation relationships. The authors state that the reference answers were manually checked, but do not show how this distinction was handled. Retrieving the correct identifiers is not enough if the answer states the wrong relationship between them. The effect of answer verification is also unclear. In Table 1, Agentic GraphRAG answers more of the question correctly, but receives lower grounding and hallucination-control scores than GraphRAG. A comparison with verification switched off would help determine whether this step reduces unsupported claims. The results also vary across related tasks: the Multi-Hop category improves in Table 6, while depth-3 traversal, set intersection, and evidence overlap decline in Table 7. These outcomes deserve closer analysis. With only 40 questions, one automated judge, and no paired uncertainty estimates, the current results do not give enough confidence about when the method is reliable.

Questions for authors

1.Does the difference between the two retrieval methods remain when the vector baseline can also match identifiers exactly? When the generation settings are kept the same, what happens if answer verification is removed?

2.For the CVE exploitation example in Section 4.2, could you show the original source, the graph links, and the reference answer? How do you prevent an inferred association from being treated as confirmed evidence of exploitation? How many questions rely on inferred or manually added links?

3.Could you provide the scores for each question, a paired confidence interval for the main comparison, and the proportions of complete answers, partial answers, and abstentions? Were any of the judge’s assessments checked by humans? The failures in Table 7 would also be easier to interpret with a few examples showing where the process went wrong.

4.Could Figures 1 and 3 be supported by one complete example showing the missing evidence, the next retrieval request, and the resulting answer? Is the “future: cmd similarity” link in Figure 2 included in the evaluated system? Appendix A.1 would also benefit from the actual trace fields mentioned in Section 3.7 and the settings used in Equations (2)–(3).

5.Are anonymized code and evaluation materials available, including the graph-building settings, source versions, all 40 questions and reference evidence, prompts, and the 160 answers with their scores? Please also explain how the five question categories relate to the ten “remaining” stress-test questions in Appendix A.3.3, and give the number of questions in each group.

Reviewer_rPfdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces agentic RAG/GraphRAG, where they introduce an agentic controller that plan evidence requirements, assess retrieval sufficiency, perform targeted retries, and verify generated claims before returning a complete answer, partial answer, or abstention. Results show that it is effective in improving the performance on cybersecurity benchmark, but introduces more latency, a tradeoff between efficiency and effectiveness

Critical strengths
  • the paper is motivated well, and gives its limits for tradeoff btw efficiency and effectiveness
  • the paper is written clearly, and communicates effectively
  • the topic is relevant and suitable for this conference
Critical weaknesses
  • the method introduced is somewhat outdated, it seems its a subset / fixed workflow of coding agents, where it is not clear if coding agents can do better with the flexibility and never compared
  • the dataset used to evaluate is cybersecurity questions, and not tested in other domains, making the method limited in applications
Questions for authors

See weakness

Reviewer_G4f6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies cybersecurity question answering by connecting information about vulnerabilities, software, attack techniques, threat groups, and mitigations. The authors construct a knowledge graph with 38,950 nodes and 88,215 edges and propose a shared controller for RAG and GraphRAG. Basic RAG uses vector similarity search, whereas GraphRAG combines exact identifier matching, graph traversal, and re-ranking to retrieve relevant entities and the paths connecting them.

The controller uses Gemini 2.5 Flash to plan retrieval, assess evidence, and verify answers. When evidence is insufficient, it retrieves again, with a limit of three retrieval calls. If the evidence remains insufficient, it provides a partial answer or abstains. The graph-based version can also adjust traversal depth. All systems use Gemini 2.5 Flash to generate answers and all-MiniLM-L6-v2 for embeddings.

The authors compare Basic RAG, GraphRAG, and their controller-augmented versions on 40 manually verified questions. Each system generates one answer per question. Retrieval is evaluated against reference entity identifiers, while Nemotron scores the answers on six dimensions, including accuracy, grounding, and completeness. Adding the controller increases GraphRAG's average score from 3.82 to 4.04, while latency rises from 4.23 to 30.96 seconds.

Critical strengths

The knowledge base integrates real-world security resources, including NVD, MITRE ATT&CK, and public threat reports.

Critical weaknesses

First, competitive baselines are missing. The paper mainly compares four configurations of its own system, without comparisons against existing methods such as Self-RAG or CRAG. In addition, the evaluation includes only 40 questions, which is too small.

Second, the correctness of the AI-generated scores is not validated. The paper uses Nemotron to score answers but does not compare its scores with human ratings or report the accuracy of its judgments. Manually verified reference answers do not establish that the AI judge can score responses correctly. The main result improves by only 0.22 points. The authors need a comparison with human ratings to show that this difference reflects an actual improvement in answer quality.

Third, the choice of controller model is not justified. Planning, evidence assessment, and answer verification all use Gemini 2.5 Flash, a model released in June 2025. The paper does not explain why it uses this model rather than a newer one and provides no ablation using alternative models. The authors should test other models to determine whether the controller's gains hold and whether the results depend on the controller design or the model used.

Questions for authors

How does the proposed method compare with established RAG-based approaches? How well do Nemotron’s scores agree with human ratings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.