- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to improve cybersecurity question answering when the required information is spread across several sources. The authors build a knowledge graph from MITRE ATT&CK, NVD, Atomic Red Team, and threat-intelligence reports. They then add a controller that checks whether enough evidence has been retrieved, searches again when information is missing, and checks the answer before returning it. The same controller is used with both ordinary vector retrieval and graph retrieval. The paper compares four systems on 40 questions. Agentic GraphRAG achieves the highest average answer-quality score, 4.04 compared with 3.82 for GraphRAG. It also retrieves more of the required identifiers and includes more correct identifiers in its answers. However, its average response time increases from 4.23 to 30.96 seconds. The main contribution is the combined system and the comparison of how the controller works with the two retrieval methods.
Critical strengths
1.The paper addresses a useful problem and proposes a sensible system.
Cybersecurity questions often require connecting information about vulnerabilities, attack techniques, software, and defensive measures. Organizing this information as a graph and allowing the system to search again when evidence is missing is a reasonable approach. Using the same controller with both retrieval methods is also a useful way to compare their behavior.
2.Several results support an improvement over one-shot GraphRAG.
In Table 1, accuracy increases from 3.12 to 3.83 and completeness from 2.48 to 2.92. In Table 4, both the precision and recall of identifiers in the answers improve. These results suggest that the system is recovering useful information, rather than merely producing answers that receive a higher overall score. The paper also reports cases where performance declines and clearly states the additional time cost.
Critical weaknesses
1.The comparisons do not show whether the gains come from graph retrieval or other differences between the systems.
All 40 questions contain explicit identifiers, such as CVE or ATT&CK IDs. The graph retriever can match these identifiers directly and retain their attributes, while the vector baseline only searches by similarity. The graph version can also change its traversal depth. It therefore has several advantages besides the use of graph relationships. The improvement from GraphRAG to Agentic GraphRAG is still a useful result. However, comparing the two retrieval methods does not establish why the controller helps the graph version more. A vector baseline with exact identifier matching would help answer this question. The paper also changes the generation prompts and temperature when adding the controller, as acknowledged in Sections 4.3–4.5. This is acceptable for comparing the complete systems, but it does not show how much of the improvement comes from repeated retrieval, answer checking, or the changed generation settings. Since the paper’s main contribution is to study how these parts work together, this missing analysis is important.
2.The paper does not sufficiently check the inferred relationships or show whether answer verification actually helps.
Section 3.3 explains that some CVE–technique links are inferred through software matching and do not confirm that a technique was used to exploit a vulnerability. Yet Section 4.2 includes questions asking about exactly such exploitation relationships. The authors state that the reference answers were manually checked, but do not show how this distinction was handled. Retrieving the correct identifiers is not enough if the answer states the wrong relationship between them. The effect of answer verification is also unclear. In Table 1, Agentic GraphRAG answers more of the question correctly, but receives lower grounding and hallucination-control scores than GraphRAG. A comparison with verification switched off would help determine whether this step reduces unsupported claims. The results also vary across related tasks: the Multi-Hop category improves in Table 6, while depth-3 traversal, set intersection, and evidence overlap decline in Table 7. These outcomes deserve closer analysis. With only 40 questions, one automated judge, and no paired uncertainty estimates, the current results do not give enough confidence about when the method is reliable.
Questions for authors
1.Does the difference between the two retrieval methods remain when the vector baseline can also match identifiers exactly? When the generation settings are kept the same, what happens if answer verification is removed?
2.For the CVE exploitation example in Section 4.2, could you show the original source, the graph links, and the reference answer? How do you prevent an inferred association from being treated as confirmed evidence of exploitation? How many questions rely on inferred or manually added links?
3.Could you provide the scores for each question, a paired confidence interval for the main comparison, and the proportions of complete answers, partial answers, and abstentions? Were any of the judge’s assessments checked by humans? The failures in Table 7 would also be easier to interpret with a few examples showing where the process went wrong.
4.Could Figures 1 and 3 be supported by one complete example showing the missing evidence, the next retrieval request, and the resulting answer? Is the “future: cmd similarity” link in Figure 2 included in the evaluated system? Appendix A.1 would also benefit from the actual trace fields mentioned in Section 3.7 and the settings used in Equations (2)–(3).
5.Are anonymized code and evaluation materials available, including the graph-building settings, source versions, all 40 questions and reference evidence, prompts, and the 160 answers with their scores? Please also explain how the five question categories relate to the ten “remaining” stress-test questions in Appendix A.3.3, and give the number of questions in each group.