acceptodds
Under review as a conference paper at ICLR 2027

DoGNAVY-Exploit: Harnessing Parallel Open-Weight Agents for Stage-Aware Vulnerability Exploitation

Abstract

Frontier LLM agents can turn known vulnerabilities into working exploits, but are costly and often access-restricted due to their potential for malicious use. Open-weight models offer a more accessible alternative, but struggle to maintain progress across long exploitation chains. They also often fail to apply relevant security knowledge at the appropriate stage and in the right context to identify and evaluate promising attack directions. To address these limitations, we propose DoGNAVY-Exploit, a multi-agent harness built around two complementary designs: stage-aware decomposition with progressive expert knowledge disclosure, and hierarchical orchestration with parallel exploration. Specifically, DoGNAVY-Exploit uses configurable milestones to recast long-horizon exploitation as short, verifiable capability transitions and provides only the knowledge relevant to each stage. Within each milestone, a hierarchical architecture separates strategic planning from parallel execution: a planner proposes diverse attack routes, executors test them concurrently, and a controller accepts progress only when supported by runtime evidence. A shared structured memory consolidates verified facts, artifacts, replay steps, and failed routes across branches, allowing agents to build on collective progress and carry validated knowledge into subsequent milestones. In controlled benchmark environments, DoGNAVY-Exploit with GLM-5.2 achieves 97.5% Pass@1 on Cybench, and 97.5% zero-day and 100% one-day Pass@1 on CVE-Bench. On ExploitGym, it solves 79 tasks with GLM-5.2 and 157 with GLM-5.3, while the corresponding public Claude Code baselines solve 39 and 130 tasks, respectively. Ablation results further demonstrate that the stage-aware orchestration, expert knowledge, parallel exploration, and memory designs can substantially improve the exploitation capabilities of open-weight agents.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agent, Cybersecurity, Agent Harness, Vulnerability Exploitation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_B1b8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a multi-agent framework for vulnerbility exploitation with open weight LLMs to gain a comparable ability with frontier close source LLMs. It splits exploitation into three stages, uses a route planner with parallel executors, and keeps a shared memory of verified and falsified facts. Knowledge is disclosed progressively by stage and role. Experiments on Cybench, CVE-Bench, and ExploitGym show gains over ReAct and OpenCode baselines on the same backbones.

Critical strengths
  1. The controlled comparison (same backbone, same budget) against ReAct and OpenCode is clean and shows a real gain from the architectural design.
  2. The ablation isolates each component and reports solved count, tokens, cost, and time together, which is methodologically useful.
  3. The writing is clear and the architecture figure matches the text.
Critical weaknesses
  1. Limited methodological novelty. Hierarchical planner/executor, parallel search, staged knowledge disclosure, and verified/falsified memory are standard agent-design patterns, not specific to exploitation.
  2. The related work section names purpose-built exploitation multi-agent systems (CoRedTeam, Automation-Exploit) but none are used as empirical baselines. ReAct and OpenCode are general-purpose coding agents, not exploitation-specific. Beating them does not show that this specific design is better than other purpose-built competitors.
  3. The ExploitGym subset is chosen by the authors’ own team with no rules stated to reviewers. This is the benchmark the paper emphasizes the most for its “long-horizon” and “beats closed-source” claims, so an undisclosed selection process is a serious weakness.
  4. No variance is reported anywhere. All Pass@1 numbers, including 97.5% and 100%, are single runs, and small ablation differences (30 vs 28 out of 40) have no statistical support.
  5. Comparisons to closed-source frontier models use different task subsets and third-party budgets.
Questions for authors
  1. Why were purpose-built exploitation agents (e.g., CoRedTeam) not included as baselines?
  2. What is the selection rule for the ExploitGym subsets?
  3. Can results be reported over multiple runs, even on a small subset?
  4. Without code or prompts, what part of this paper can another group actually reuse or verify?
Reviewer_3paKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DoGNAVY-Exploit, a multi-agent framework for vulnerability exploitation that aims to bring low-cost open-weight models (GLM-5.2/5.3) close to frontier closed-source models in exploitation capability. The method has two components: (1) the exploitation process is divided into three stages, with expert knowledge provided according to both stage and agent role; (2) within each stage, a planner proposes candidate routes, executors test them in parallel, and a controller accepts or rejects progress based on runtime evidence, supported by a shared structured memory.

Critical strengths
  1. The controlled comparisons isolate the contribution of the orchestration design. On Cybench and CVE-Bench, DoGNAVY, ReAct, and OpenCode are compared using the same backbone. Since each DoGNAVY agent is itself built on OpenCode, these results indicate that the gains come from the orchestration rather than the underlying model.
  2. The motivation is practical. Inference cost and safety refusals of frontier models are reasonable grounds for focusing on open-weight models, and the cost-normalized analysis is a useful perspective.
Critical weaknesses
  1. The primary benchmark lacks a comparable controlled baseline. On ExploitGym, DoGNAVY is evaluated only on expert-selected subsets (225 and 485 tasks), whereas the baseline results cover all 869 tasks. The paper does not state whether the two use the same success criterion, and no same-backbone controlled baseline is provided. Meanwhile, Cybench and CVE-Bench, where controlled baselines do exist, are close to saturation and offer limited discrimination.

  2. Key mechanisms are underspecified, and the source of the gains cannot be attributed. The method section does not specify the form, source, and injection method of the expert knowledge; the criteria the controller uses to judge evidence; how stage completion is determined; or how routes are generated and memory is managed. Removing expert knowledge reduces solved tasks from 30 to 24 and increases cost by 84.6%, but the current ablation cannot distinguish whether this gain comes from how the knowledge is organized or from the knowledge content itself. The authors' decision not to release code and prompts is understandable given misuse concerns, but architecture-level descriptions of these mechanisms would not provide directly exploitable material and should be included.

Questions for authors
  1. Do the Claude Code results in Table 3 use the on-target or flag-captured criterion?
  2. What is the form, source, and size of the expert knowledge? How are stages and milestones defined for non-binary tasks such as those in Cybench and CVE-Bench?
  3. Why does the framework fail to keep the agent focused on the target vulnerability?
Reviewer_fc74OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents DoGNAVY-Exploit, a multi-agent harness for vulnerability exploitation with open-weight models. Built on OpenCode, it organizes exploitation around verifiable milestones and combines hierarchical planning, parallel exploration, stage-specific expert knowledge, and shared memory. These components aim to preserve progress and reduce unproductive exploration during long tasks. Evaluation uses GLM-5.2 on Cybench, CVE-Bench, and selected ExploitGym tasks, with additional GLM-5.3 results on ExploitGym. The authors report improved success over general-purpose agent baselines and favorable API costs, alongside component ablations.

Critical strengths
  1. The gains over the same-model baselines are substantial, especially on CVE-Bench. With GLM-5.2, zero-day success improves from 26/40 with OpenCode to 39/40 with DoGNAVY.

  2. The workflow makes sense for long exploitation tasks. Verifiable milestones and shared records of successful and failed attempts give agents a concrete way to preserve progress.

  3. The evaluation covers CTF tasks, web vulnerabilities, and binary exploitation. Reporting on-target exploits separately from flag captures is also useful, since completing a task does not always mean exploiting the intended vulnerability.

Critical weaknesses
  1. The contribution over existing security agents is unclear. Hierarchical planning, expert knowledge, execution feedback, and shared memory already appear in prior systems. The paper mainly compares against general-purpose agents. It needs a closer comparison with security-specific frameworks to establish what its stage-aware design adds.

  2. The comparisons do not separate better orchestration from more search. Matching wall-clock time does not match total compute when DoGNAVY runs three executors concurrently. A simple parallel OpenCode baseline would help. It is also unclear whether the baselines receive the same expert knowledge.

  3. The ExploitGym results do not establish the claimed cost advantage. DoGNAVY uses expert-selected subsets and an eight-hour limit, while public baselines use the full benchmark and six hours. The cost metric combines subset-specific average costs with success counts normalized over all 869 tasks. Counting untested tasks as failures is conservative, but does not make this a controlled comparison at equal cost.

  4. The ablations do not directly test some central design choices. Removing expert knowledge shows that knowledge helps, not that providing it progressively is better than providing it upfront. Similarly, removing stages does not establish that reduced context pressure explains the improvement. The proposed mechanisms remain plausible but insufficiently tested.

  5. Important implementation details are missing. The paper does not sufficiently explain how milestones are checked, expert knowledge is constructed and selected, or shared memory is updated. Since the implementation and full prompts are withheld, these omissions make the system difficult to assess and reproduce.

  6. Generality and stability remain unclear. Evaluation uses only two versions of GLM, and the paper does not report repeated-run results or confidence intervals. This limits broader claims about open-weight models and makes smaller gains, such as the four additional Cybench successes, harder to assess.

Questions for authors
  1. Do you have existing comparisons with a security-specific agent or parallel OpenCode? Were total compute and access to expert knowledge matched?

  2. How were the ExploitGym and ablation subsets selected? Were they fixed before inspecting outcomes? Can the public baselines be scored on the same task IDs?

  3. Is the expert knowledge generic or task-specific? Do any existing results compare progressive disclosure with providing the same knowledge upfront?

  4. How are milestones checked: by executable tests, model judgment, or human review? Is any human guidance used during a run?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.