- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
DIVE encodes how an agent should make the decisions it faces during a long task (continue, check its work, change direction, stop) as explicit, reusable "Cognitive Judgment Skills". Each Skill states when it applies, what evidence the decision needs, and when the decision should be revisited. A Detect-Invoke-Verify loop calls these Skills at decision points, while the host agent still generates the actions. The method is evaluated on software repair (SWE-bench Verified and Pro, several backbones) and on program search in OpenEvolve, and individual decisions are measured by comparing continuations from the same state. The headline results are 27 vs 24 resolved tasks on 30 SWE-bench Pro tasks (outer-loop DIVE with Codex SDK and GPT-5.6-Luna) and a 72.52% token saving from a guarded stopping rule in program search. The authors note that the decision-level results are less consistent and that the task-level benefits "do not establish a general advantage for its selection policy".
Critical strengths
-
The paper is candid about what it does not show. It separates the value of an action from the value of choosing it (Sec. 5.1) and reports results that go against its own method. The held-out v3 direction selector is worse than Native on average (Sec. 7.3), and the reuse selector loses to always-reuse in two of four cohorts (Sec. 7.4, Table 3). This shows that a useful action and a useful judgment can come apart, which is a fair warning for work on metacognitive agents.
-
The same-state evaluation design (fixed parent state, with Native, selected and opposite continuations, Sec. 5.1, Sec. 7.3 and App. E.6) is a reasonable template for evaluating single agent decisions.
Both points concern how the evaluation is done. Neither shows that the method itself helps.
Critical weaknesses
-
The repair results do not isolate what judgment contributes, and the evidence is small.
- Each DIVE arm changes Skill content, extra deliberation and deployment together. The paper itself says a matched generic-review control is needed and not provided (Sec. 8, App. B). Without it, the gains cannot be credited to Cognitive Judgment Skills.
- The gains amount to a few tasks and come with no statistical test. For 27 vs 24 on 30 Pro tasks, an exact McNemar test gives p = 0.25 even in the most favourable case. The Verified numbers are "projected to 100 tasks". The Native and DIVE-Target resolved and action counts, and the DIVE-Jev token counts, equal the 33-task archive in App. B (Table 5; lines 833-834) scaled by 100/33, so the 82 vs 76 gap may be two tasks.
- App. A.1 states that the task-ID lists, prompts, Skill hashes and per-request usage ledgers for the added Pro cohorts have not been assembled into a reconciled reproduction package. The Reproducibility Statement also says the archive does not yet provide a complete, reconciled task-level reproduction package for these summaries.
- DIVE-Target roughly doubles backbone tokens (Table 1). In the MiniMax and DeepSeek cohorts, tokens go down only when judgment is routed to "Jev". I could not find a definition of Jev (Sec. 4.3 only names its role). The Codex token saving is measured in a different setup with an outer loop.
What would change my view: paired per-task outcomes with exact tests, the actual Verified sample size, a matched-budget generic-review arm, and a definition of Jev.
-
The program-search results do not show what judgment adds.
- The 72.52% saving comes from stopping once a preset target is met, while Native runs to a fixed budget. In the longer-budget runs Native reached the target within the first three iterations (App. D), and the saving is larger under these longer budgets (94.59% and 90.11% vs 70.79% and 73.67%; Table 2). The paper states that the stopping rule is itself a guarded threshold and does not compare it with an equivalent plain rule. The study covers two tasks.
- The selection studies are negative or mixed. The v3 direction selector beats Native on 1 of 6 forks (Sec. 7.3). Fixed reuse helps locally in all four cohorts, but the gain from letting the Skill choose reuse changes sign across these small cohorts, with no intervals (Sec. 7.4, Table 3).
For a measurement framework at ICLR, I would expect either a positive controlled result or a broader evaluation with intervals. The related work also misses EET (Guo et al., arXiv 2601.05777, 2026), which reports early-termination savings on SWE-bench Verified with negligible loss and should be discussed next to the repair cost claims.
Questions for authors
- What is "Jev" (model, provider, prompting)? Why is its cost reported in USD while the backbone cost is in CNY (Table 1)?
- How many SWE-bench Verified tasks were actually run per arm? Is the "projection to 100" a 100/33 rescaling of the 33-task archive, and did all arms use the same tasks?
- Please give paired per-task outcomes with exact McNemar p-values for all repair cohorts.
- Does any model call take part in the stop decision, or is it fully determined by the threshold and guards? How do the savings compare with a Native run whose budget is set to the typical time-to-target?
- Why does DIVE-Target double backbone tokens, and how much of the Codex saving comes from the outer loop and not from judgment?
- Have the Table 3 token changes marked "remains to be verified" been checked?