- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies a coding-agent harness rule that makes an extra model call to review a patch whenever an edit applies. In the paper's three-call harness on 60 SWE-bench Verified tasks, this review takes 20.8 and 23.7 percent of all tokens for two Qwen coder models, over 83 percent of it input. Forking the same saved patch on 200 tasks, review raises resolution from 38 to 43 percent under one prompt, while a second prompt is not significant. On SWE-bench-Live, a learned controller choosing among stop, edit, review, inspect and test beats a state-blind schedule by 6.5 points at higher cost, and showing an already-paid test result to later decisions adds 15.5 points.
Critical strengths
- The question is practical. Post-edit compute is a real cost in coding agents, and measuring it per stage rather than per run is useful.
- The individual controls are careful. Forks start from byte-identical patches with a sham branch, controller decisions replay exactly, and the reveal/mask test charges for the test in both arms.
Critical weaknesses
- The target problem is the paper's own setting, not common practice. Per-edit review is a rule of the paper's harness, and the main body gives no evidence that deployed agents use it. Widely used agents review on request or at the pull request, for example Codex CLI's
/review(https://learn.chatgpt.com/docs/codex/cli) and Copilot (https://docs.github.com/en/copilot/concepts/agents/coding-agent/about-coding-agent). - The main findings are expected. Input-heavy review, review repairing weak patches, and test output helping later decisions all match the self-repair and execution-feedback work the paper cites. The forked comparison is a standard paired design, which the paper describes as applying an existing replay principle.
- The evidence does not answer the paper's question. Cost and benefit come from different tasks, models and context limits, and the paper states they "cannot be combined into one cost-effectiveness estimate". The learned policy resolves 143 and then 123 on the same 308 entries, and in the repeat a cheaper correctness gate matches it.
Questions for authors
All three are clarification requests.
- On the forked states, what is the review cost per additional resolved task?
- What explains 143 versus 123 for the same policy on the same entries, and which run is representative?
- The main body does not name the four equal-budget alternatives that tie original review. Does a fresh attempt with the same tokens match review?