acceptodds
Under review as a conference paper at ICLR 2027

DIVE: Making Judgment a First-Class Capability in Long-Horizon Agents

Abstract

Agents working on long tasks must decide when to continue, check their work, change direction, or stop. A final task score gives little information about the value of these decisions. We introduce DIVE, which encodes judgment procedures as reusable Cognitive Judgment Skills. Each Skill specifies when prior experience applies, what evidence a decision requires, and when to reconsider it. A Detect–Invoke–Verify loop connects these procedures to the host agent without replacing its action generator. We evaluate DIVE in software repair and scientific program search, using task outcomes and continuations from the same state. Guarded stopping reduces aggregate Target tokens by 72.52% across 16 matched search pairs. Every pair passes its task-specific final-quality check, though five have lower anytime quality. In the MiniMax and DeepSeek repair summaries, Jev uses fewer reported tokens than Native or DIVE with Target-based judgment. On 30 SWE-bench Pro tasks, outer-loop DIVE with Codex SDK and GPT-5.6-Luna resolves 27 tasks versus 24 for Native, using 19.41% fewer input-plus-output tokens. The decision-level results are less consistent. Reuse has positive mean local effects in four cohorts, but selection gains and five-step outcomes vary. DIVE provides a way to measure these decisions and their costs; the observed task-level benefits do not establish a general advantage for its selection policy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model agents, metacognitive control, agent decision-making, cognitive judgment skills, agent evaluation, software engineering agents, evolutionary program search

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YYadOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

DIVE encodes how an agent should make the decisions it faces during a long task (continue, check its work, change direction, stop) as explicit, reusable "Cognitive Judgment Skills". Each Skill states when it applies, what evidence the decision needs, and when the decision should be revisited. A Detect-Invoke-Verify loop calls these Skills at decision points, while the host agent still generates the actions. The method is evaluated on software repair (SWE-bench Verified and Pro, several backbones) and on program search in OpenEvolve, and individual decisions are measured by comparing continuations from the same state. The headline results are 27 vs 24 resolved tasks on 30 SWE-bench Pro tasks (outer-loop DIVE with Codex SDK and GPT-5.6-Luna) and a 72.52% token saving from a guarded stopping rule in program search. The authors note that the decision-level results are less consistent and that the task-level benefits "do not establish a general advantage for its selection policy".

Critical strengths
  1. The paper is candid about what it does not show. It separates the value of an action from the value of choosing it (Sec. 5.1) and reports results that go against its own method. The held-out v3 direction selector is worse than Native on average (Sec. 7.3), and the reuse selector loses to always-reuse in two of four cohorts (Sec. 7.4, Table 3). This shows that a useful action and a useful judgment can come apart, which is a fair warning for work on metacognitive agents.

  2. The same-state evaluation design (fixed parent state, with Native, selected and opposite continuations, Sec. 5.1, Sec. 7.3 and App. E.6) is a reasonable template for evaluating single agent decisions.

Both points concern how the evaluation is done. Neither shows that the method itself helps.

Critical weaknesses
  1. The repair results do not isolate what judgment contributes, and the evidence is small.

    • Each DIVE arm changes Skill content, extra deliberation and deployment together. The paper itself says a matched generic-review control is needed and not provided (Sec. 8, App. B). Without it, the gains cannot be credited to Cognitive Judgment Skills.
    • The gains amount to a few tasks and come with no statistical test. For 27 vs 24 on 30 Pro tasks, an exact McNemar test gives p = 0.25 even in the most favourable case. The Verified numbers are "projected to 100 tasks". The Native and DIVE-Target resolved and action counts, and the DIVE-Jev token counts, equal the 33-task archive in App. B (Table 5; lines 833-834) scaled by 100/33, so the 82 vs 76 gap may be two tasks.
    • App. A.1 states that the task-ID lists, prompts, Skill hashes and per-request usage ledgers for the added Pro cohorts have not been assembled into a reconciled reproduction package. The Reproducibility Statement also says the archive does not yet provide a complete, reconciled task-level reproduction package for these summaries.
    • DIVE-Target roughly doubles backbone tokens (Table 1). In the MiniMax and DeepSeek cohorts, tokens go down only when judgment is routed to "Jev". I could not find a definition of Jev (Sec. 4.3 only names its role). The Codex token saving is measured in a different setup with an outer loop.

    What would change my view: paired per-task outcomes with exact tests, the actual Verified sample size, a matched-budget generic-review arm, and a definition of Jev.

  2. The program-search results do not show what judgment adds.

    • The 72.52% saving comes from stopping once a preset target is met, while Native runs to a fixed budget. In the longer-budget runs Native reached the target within the first three iterations (App. D), and the saving is larger under these longer budgets (94.59% and 90.11% vs 70.79% and 73.67%; Table 2). The paper states that the stopping rule is itself a guarded threshold and does not compare it with an equivalent plain rule. The study covers two tasks.
    • The selection studies are negative or mixed. The v3 direction selector beats Native on 1 of 6 forks (Sec. 7.3). Fixed reuse helps locally in all four cohorts, but the gain from letting the Skill choose reuse changes sign across these small cohorts, with no intervals (Sec. 7.4, Table 3).

    For a measurement framework at ICLR, I would expect either a positive controlled result or a broader evaluation with intervals. The related work also misses EET (Guo et al., arXiv 2601.05777, 2026), which reports early-termination savings on SWE-bench Verified with negligible loss and should be discussed next to the repair cost claims.

Questions for authors
  1. What is "Jev" (model, provider, prompting)? Why is its cost reported in USD while the backbone cost is in CNY (Table 1)?
  2. How many SWE-bench Verified tasks were actually run per arm? Is the "projection to 100" a 100/33 rescaling of the 33-task archive, and did all arms use the same tasks?
  3. Please give paired per-task outcomes with exact McNemar p-values for all repair cohorts.
  4. Does any model call take part in the stop decision, or is it fully determined by the threshold and guards? How do the savings compare with a Native run whose budget is set to the typical time-to-target?
  5. Why does DIVE-Target double backbone tokens, and how much of the Codex saving comes from the outer loop and not from judgment?
  6. Have the Table 3 token changes marked "remains to be verified" been checked?
Reviewer_UXSzOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces DIVE, a framework that treats judgment as a separate capability in long-horizon agents. It represents decisions about when to continue, verify, change direction, or stop as reusable Cognitive Judgment Skills, specifying their applicability, required evidence, and revision conditions. A detect-invoke-verify loop applies these skills while leaving action generation to the original agent. The authors evaluate DIVE in software repair and scientific program search, examining both task-level outcomes and alternative continuations from the same decision state. The decision-level experiments give mixed results. Direction selection often fails to outperform alternatives, and locally beneficial experience reuse does not consistently improve longer continuations. The work provides a framework for making agent judgments explicit and evaluating their consequences, although the experiments do not establish a general advantage for its judgment-selection policy.

Critical strengths
  1. The paper addresses an important problem in long-horizon agents which is deciding when to continue, verify, revise, or stop. Separating these judgments from action generation makes it possible to examine decisions that final task success alone cannot explain.
  2. The Cognitive Judgment Skills provide a structured way to represent decision criteria, including when a skill applies, what evidence it requires, and when a decision should be reconsidered. The framework also allows different judgment implementations without replacing the host agent's action generator.
  3. The experiments cover both software repair and scientific program search. The Codex repair results show improvements in resolution with lower reported token usage, while guarded stopping substantially reduces search computation without violating the specified final-quality checks.
Critical weaknesses
  1. It is unclear how much of the improvement comes from the proposed Judgment Skills. In Section 5.1, the authors state that the repair experiments evaluate "skill and consultation bundles without a matched-compute ablation." This means the improvement could come from giving the agent more opportunities to think and review its actions, rather than from the specific decision criteria in the Skills. The authors acknowledge this limitation, but it remains important for understanding the framework's contribution. I suggest adding a baseline in Section 6.1, Table 1, where the agent receives the same evidence and review budget but no Cognitive Judgment Skills. This would help show what the Skills themselves add.
  2. The paper does not sufficiently check whether the agent correctly understands when a Skill should be used. Section 4.3 states that the parser checks the format and consistency of the model's response but "cannot verify the model's assessment." Appendix E.6 also states that the reuse experiments lack independent labels of whether a Skill actually applies to a state. This matters because an agent may produce a valid decision without correctly understanding the available evidence. I suggest adding a small manually examined set of decision states in Section 7.4, where the authors check whether the evidence supports using or rejecting a Skill.
  3. The software repair results need clearer task-level evidence. Section 6.1 acknowledges that aggregate resolution counts do not establish a statistically reliable repair advantage. For example, the Codex experiment reports more resolved tasks with DIVE, but the totals do not show which tasks were newly solved and which previously solved tasks may have failed. This makes it difficult to judge how consistent the improvement is. I suggest expanding Table 1 in Section 6.1 or adding a table with paired outcomes, indicating improvements and regressions on individual tasks, along with uncertainty estimates.
  4. The decision-level experiments are too small to support broader conclusions about selection behavior. Section 7.3 evaluates direction selection using six forks from only three source runs. The authors correctly acknowledge that this provides limited evidence about expected branch rankings. I see the negative selection results as useful. My concern is that the small number of independent starting states makes it difficult to know whether these findings are specific to the selected cases. I suggest extending Section 7.3 and Appendix E.6 with more independent source states and reporting uncertainty at the source-run level. If additional experiments are not feasible, the findings should remain framed as a focused diagnostic study.
  5. Some important experimental details are still missing for reproduction. Appendix A.1 states that the authors have not assembled a complete "reconciled reproduction package" for the added repair cohorts. These are important because the paper's positive results depend on both task resolution and computational savings. I suggest completing the records listed in Appendix A.1 and making them available. This would allow readers to check the reported comparisons and better understand how the different judgment configurations were evaluated.
Questions for authors

See the weakness

Reviewer_F49LOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces DIVE, which allows an agent's choices to continue, verify, redirect, or cease to be an explicit, reusable element. Cognitive Judgment Skills indicate when to utilize experience and what proof a decision requires, while a Detect–Invoke–Verify loop connects them to a host agent that continues to formulate its own actions. The judge may represent the agent itself, a rule, or Jev, an independent judge responsible for managing valid self-judgment and Skill-Ask requests. It is evaluated on SWE-bench repair, in guarded stopping within OpenEvolve program search, and in same-state forked comparisons of both direction guidance and experience reuse. The task-level outcomes are favorable, but the decision-level outcomes are varied, and the authors claim they do not demonstrate a universal benefit for their selection strategy.

Critical strengths
  1. Same-state evaluation of individual decisions. Comparing the chosen action, a fixed policy, and a hindsight bound separates “the action helped” from “selecting it helped”. It is a transferable idea, and it shows that the selector does not beat fixed reuse.
  2. Candid reporting. Negative results are shown, fallbacks are kept apart from abstentions, projections are labeled, and the limitations name the missing control. This makes the claims checkable.
Critical weaknesses
  1. The thesis is not isolated, and the repair gains are tiny and unpaired. There is no matched generic-review or equal-budget control, and no ablation across models or components. Each model appears in one setting only (MiniMax on Verified, DeepSeek on Pro, GPT-5.6-Luna via Codex), so the model is confounded with the benchmark and host. The task gaps are 19 to 21 of 50 and 24 to 27 of 30, with no paired outcomes or intervals. The Verified rows appear to be a 33-task archive rescaled to 100.
  2. The “long-horizon” claim is not tested, and the decision-level evidence is thin. Main stopping runs allow only 5 or 20 iterations, and the forks look two or five steps ahead. The clean stopping saving is a threshold at a known target. The direction selector beats Native on 1 of 6 forks, and reuse selection gains change sign across cohorts of 8 to 16 states, with no intervals. Table 3’s Historical and v2 rows appear to repeat archived-subset values from App. E.5, despite being labeled n = 16 and n = 8.
Questions for authors
  1. What model is Jev (size, prompt, price), and does its lower token total come from fewer steps rather than cheaper judgment?
  2. What are the true state counts behind the Historical and v2 rows of Table 3? Please provide per-state data and intervals.
  3. Add a matched generic-review, equal-budget control, with an ablation of the same tasks across at least two backbones.
  4. Add tasks with substantially longer horizons, plus fixed-iteration and random-stopping baselines across several Native budgets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.