- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies when software agents should call a tool rather than continue reasoning. It proposes Counterfactual Necessity Optimization (CNO), which compares a concrete tool call with a reasoning step from the same saved state. Both branches then finish under a frozen continuation policy, and their return difference is used to train a necessity head and guide tool routing. An obligation mask enforces required interactions. On SWE-bench Verified, the paper reports 45.8% resolution compared with 38.8% for outcome-only RL, and 39.1% fewer optional calls per resolved issue.
Critical strengths
(1) The problem is interesting and well motivated. Final task rewards do not directly show whether individual tool calls are useful, and the paired comparison provides a concrete way to study this.
(2) The reported results show higher issue resolution and fewer optional calls per resolved issue on Verified. The mask-only baseline and component ablations are useful for understanding the method.
Critical weaknesses
(1) Some experimental settings are missing.
(2) It is not clear whether the training comparison is fair. CNO uses additional complete rollouts, but only the number of training instances is matched.
(3) The SWE-Bench Pro setup also needs clarification: which version and subsets were evaluated? The reported 1,865 issues include non-public subsets in the original benchmark; how were these evaluated, and what are the per-subset results?
(4) Could you provide a breakdown of call counts and costs by tool type (e.g., search, file reading, editing, testing)? If most of the reduction comes from cheap calls, the cost savings might be limited.
Questions for authors
Please refer to the weakness section.
We thank the reviewer for the valuable feedback.
CNO’s intended benefit is more informative credit assignment per unit of rollout. At a selected state, both branches share the same context and budget. Their return difference directly supervises whether the proposed tool call improves the continuation relative to reasoning.
To test whether this mechanism provides benefits beyond additional sampling, we further conduct comparison that match total generated rollout tokens (original budget 1x together with paired suffix rollouts 2x).
From the results, additional rollouts improve Outcome RL by 3.8%, while CNO retains a 3.2% advantage at the same total rollout-token budget. CNO also exceed TRM by 2.2% and the unpaired-target ablation by 2.8%, without using more environment interactions. This pattern supports the interpretation that CNO benefits from how it allocates rollouts and constructs state-specific supervision, beyond the benefit of additional sampling alone.
The 1865 refers to the complete original benchmark, whereas our evaluation uses only the original public test set (731 samples), and we will correct the dataset description and label the corresponding results as SWE-Bench Pro Public V1. We will include our recipe for other settings in the revised manuscript.
CNO aims to improve when an agent calls a tool by estimating how the call affects downstream task outcomes. This can reduce redundant calls and recover useful calls that the agent would otherwise skip. On SWE-bench Verified, resolution rises from 38.8% to 45.8%, while optional calls per resolved issue fall by 39.1%. Here an training example from SWE-Gym
dask__dask-10422shows how CNO can identify and avoid expensive, low-value execution calls rather than merely eliminating cheap inspection calls.Case
dask__dask-10422. The agent had written a script to reproduce a bug in Dask’s distributedto_zarrpath. The script startedClient()at module scope, so spawned workers would try to start new processes during their own initialization.python3 reproduce_error.pyimmediately.Client()needs to run under anif __name__ == "__main__":guard.to_zarrand reproduces the target error: Dask rejects a disk-backed Zarr store as an in-memory store.