acceptodds
Withdrawn from ICLR 2027

The Review Tax: From Automatic Review to Post-Edit Control in Coding Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

After a coding agent produces an edit that can be applied, should the harness automatically spend another model call reviewing it? We study this _**review tax**_: computation committed to review simply because a patch exists, before there is evidence that review is worthwhile. In a controlled three-call harness on 60 SWE-bench Verified tasks under the same 32k context limit, review consumes 20.8% and 23.7% of all model tokens for two Qwen coder configurations. More than 83% of those review tokens are spent reading the input. Measuring the return requires care, since separately rerun review and no-review arms can produce different patches before review begins. When we instead fork the same saved patch on 200 tasks at 64k, review gains 5.0 percentage points, with a 95% interval of , while a prompt variant gives a smaller estimate whose interval includes zero. These findings motivate treating the stage after an edit as a decision. In a separate sequential campaign with 308 task entries from 305 SWE-bench-Live issues, a learned policy resolves 6.5 points more than a state-blind schedule, at higher reported cost. A further repetition again favors learned control over the schedule but leaves its advantage over a correctness gate unresolved. When a public-test result has already been obtained and charged for, showing it to later decisions yields 15.5 points higher resolution than masking it. An applicable edit is therefore an opportunity to review. Whether to do so should depend on the candidate patch, the evidence already available and the cost of continuing.

Primary area:
foundation or frontier models, including LLMs
Keywords:
coding agents, LLM agents, self-review, test-time compute, adaptive computation, agent harness design, counterfactual evaluation, value of information, SWE-bench

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DAziOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a coding-agent harness rule that makes an extra model call to review a patch whenever an edit applies. In the paper's three-call harness on 60 SWE-bench Verified tasks, this review takes 20.8 and 23.7 percent of all tokens for two Qwen coder models, over 83 percent of it input. Forking the same saved patch on 200 tasks, review raises resolution from 38 to 43 percent under one prompt, while a second prompt is not significant. On SWE-bench-Live, a learned controller choosing among stop, edit, review, inspect and test beats a state-blind schedule by 6.5 points at higher cost, and showing an already-paid test result to later decisions adds 15.5 points.

Critical strengths
  1. The question is practical. Post-edit compute is a real cost in coding agents, and measuring it per stage rather than per run is useful.
  2. The individual controls are careful. Forks start from byte-identical patches with a sham branch, controller decisions replay exactly, and the reveal/mask test charges for the test in both arms.
Critical weaknesses
  1. The target problem is the paper's own setting, not common practice. Per-edit review is a rule of the paper's harness, and the main body gives no evidence that deployed agents use it. Widely used agents review on request or at the pull request, for example Codex CLI's /review (https://learn.chatgpt.com/docs/codex/cli) and Copilot (https://docs.github.com/en/copilot/concepts/agents/coding-agent/about-coding-agent).
  2. The main findings are expected. Input-heavy review, review repairing weak patches, and test output helping later decisions all match the self-repair and execution-feedback work the paper cites. The forked comparison is a standard paired design, which the paper describes as applying an existing replay principle.
  3. The evidence does not answer the paper's question. Cost and benefit come from different tasks, models and context limits, and the paper states they "cannot be combined into one cost-effectiveness estimate". The learned policy resolves 143 and then 123 on the same 308 entries, and in the repeat a cheaper correctness gate matches it.
Questions for authors

All three are clarification requests.

  1. On the forked states, what is the review cost per additional resolved task?
  2. What explains 143 versus 123 for the same policy on the same entries, and which run is representative?
  3. The main body does not name the four equal-budget alternatives that tie original review. Does a fresh attempt with the same tokens match review?
Reviewer_ZMV5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper explores an interesting research question: should harness automatically spend compute on reviewing its own work? The authors call this review tax. The paper makes three contributions:

  1. Quantify the review tax: Automatic review increases token consumption up to 23.7%. Input tokens are responsible for 83% of the review tax.
  2. Quantifies benefit of review: Authors run review and no-review configuration and find that review gains 5 percentage points improvement over no-review configuration.
  3. The authors train a controller which decides whether or not to perform the review step after a model makes an edit. This learned controller outperformed state-blind schedule by 6.5 points.
Critical strengths
  1. The three-call harness is a strong design choice to quantify the review tax and the benefits of review. This methodology has potential to be extended to other agentic tasks such as deepsearch to quantify self-review tax.
  2. The empirical finding regarding review tax is informative. More than 80% of the review tax is due to input tokens is an interesting finding.
Critical weaknesses
  1. Few analysis is missing for the learned controllers. One of the paper’s primary contributions is quantifying review tax, and subsequent quantifying benefits of review. However, these findings are not connected to the proposed learned controller. It is unclear whether learned controllers reduce review tax while preversing the benefits of review.

  2. Limited generalizability to natural coding-agent workflow: The current study relies on oracle file localization, and does not permit interactive testing or repository exploration during implementation. This helps in the initial analysis as it removes confounding but does not guarantee generalizability of the work. I believe the controlled harness is a useful starting point, but extending it to more natural agentic workflows will substantially strengthen the work.

  3. Lack of coherence in the paper: I found following the paper difficult in some sections. My major concern is several research questions and their respective findings are loosely connected. The experiments change the underlying model and task population. As a result, it is difficult to connect the findings across experiments.

Questions for authors
  1. Can the authors perform the experiments to address W2?
  2. Can authors clarify what constitutes a "public test" in the execution-feedback experiments?
Reviewer_PLJ1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines the cost and benefit of automatic patch review in coding agents. The authors measure how much computation review consumes, evaluate its effect by comparing continuations from the same saved patch, and train an agent to decide whether to review, edit, test, inspect, or stop.

Review accounts for roughly 20–24% of tokens in the authors' harness, largely from rereading the input. On 200 SWE-bench Verified tasks, reviewing a fixed patch improves resolution by 5 percentage points. The learned controller beats a fixed schedule, but not a simple rule that predicts whether the patch is correct.

The paper provides a useful way to measure the value of review, but does not establish when review is worth the additional computation.

Critical strengths

Aims as useful practical insight: The results try to challenge the assumption that every patch should trigger review and motivate more selective use of computation.

Critical weaknesses

Inconsistent results: Review gains vary across experiments and do not clearly generalize across benchmarks. Unexplained experimental variation: The same review condition scores 43% in E1 and 39% in E3, while all E3 alternatives achieve identical aggregate scores.

Questions for authors

What explains this difference across benchmarks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.