- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a multi-agent framework for vulnerbility exploitation with open weight LLMs to gain a comparable ability with frontier close source LLMs. It splits exploitation into three stages, uses a route planner with parallel executors, and keeps a shared memory of verified and falsified facts. Knowledge is disclosed progressively by stage and role. Experiments on Cybench, CVE-Bench, and ExploitGym show gains over ReAct and OpenCode baselines on the same backbones.
Critical strengths
- The controlled comparison (same backbone, same budget) against ReAct and OpenCode is clean and shows a real gain from the architectural design.
- The ablation isolates each component and reports solved count, tokens, cost, and time together, which is methodologically useful.
- The writing is clear and the architecture figure matches the text.
Critical weaknesses
- Limited methodological novelty. Hierarchical planner/executor, parallel search, staged knowledge disclosure, and verified/falsified memory are standard agent-design patterns, not specific to exploitation.
- The related work section names purpose-built exploitation multi-agent systems (CoRedTeam, Automation-Exploit) but none are used as empirical baselines. ReAct and OpenCode are general-purpose coding agents, not exploitation-specific. Beating them does not show that this specific design is better than other purpose-built competitors.
- The ExploitGym subset is chosen by the authors’ own team with no rules stated to reviewers. This is the benchmark the paper emphasizes the most for its “long-horizon” and “beats closed-source” claims, so an undisclosed selection process is a serious weakness.
- No variance is reported anywhere. All Pass@1 numbers, including 97.5% and 100%, are single runs, and small ablation differences (30 vs 28 out of 40) have no statistical support.
- Comparisons to closed-source frontier models use different task subsets and third-party budgets.
Questions for authors
- Why were purpose-built exploitation agents (e.g., CoRedTeam) not included as baselines?
- What is the selection rule for the ExploitGym subsets?
- Can results be reported over multiple runs, even on a small subset?
- Without code or prompts, what part of this paper can another group actually reuse or verify?