- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how to get work done with several LLMs sequentially. The task is, for example, fixing a bug in one Python function, making one planned edit in a repository (e.g. "add a separator parameter to these functions"), or reaching one subgoal in a simulated science game (e.g. "focus on the thermometer"). In repository edits and the science game, each step is scored alone, starting from a correct state supplied by the benchmark or by the reference model. The models are tried one after another on that step, cheapest first. After each attempt, an automatic check (the paper calls it a "gate") decides whether to keep the result. The check is executable code, but its origin differs per study: written by the authors for bug fixing (does the code parse, do the problem's examples pass), written by the 27B planner model for repository edits (does a file contain given strings), and built into the game for the science subgoals. If the result is kept, the process stops. Otherwise the task's files or environment are reset to their initial state, and the next model tries. The last model in the list is the most expensive one, called the "reference". The models are Qwen3-8B and Qwen3-14B, each with thinking on or off; the reference is Qwen3-14B with thinking on.
The paper reports three studies:
- Bug fixing on 134 HumanEvalPack Python problems, with two models: Qwen3-14B without thinking first, then with thinking. Each model gets the same prompt: the function's docstring, the buggy code, and the problem's public example asserts, e.g. for Python/52 the docstring of
below_thresholdandassert below_threshold([1, 2, 4, 10], 100). The checks are applied after the attempt and are never shown to the model. Each model's output is generated once, and only the check changes. Check A keeps the first output if the code was changed, still parses, and still defines the function. Check B additionally requires the public example asserts to pass. For example, on Python/52 the first model only flipsreturn Truetoreturn False: check A keeps this wrong fix, while check B rejects it, so the second model's correct fix is returned (Python/57 is the same case). Measured on the benchmark's hidden tests, check B raises the share of correct results from 78.4% to 90.3%, at 29.1% more tokens. - Code edits on 357 CooperBench steps, taken from plans written by a 27B model. Trying three models in sequence instead of the reference alone raises the share of steps passing the check from 81.5% to 92.7%, at 64% of the reference's token cost. The middle model costs more than the reference work it saves.
- ScienceWorld subgoals, each started by replaying the benchmark's reference actions up to that subgoal. Trying three models in sequence passes more subgoals than the reference alone, but costs more.
In studies 2 and 3, the gain comes only from trying more models: no information passes from one attempt to the next.
Critical strengths
Good transparency and reproducibility. The supplement contains the code, the registered protocols, every recorded attempt. The authors' reproduce.py runs without a GPU and passes: it checks the file hashes and recomputes the CooperBench counts (291, 324 and 331 accepted steps) from the 1,428 recorded cells.
Critical weaknesses
W1. The technical contribution is very thin. The method tries a list of models one after another, checks each result, resets the files on failure, and stops at the first result that passes. This is the standard model cascade studied by FrugalGPT, AutoMix and cascade routing (Dekoninck et al., 2025), all cited by the paper. It is also the "filter by public tests" step of AlphaCode and CodeT. The models share nothing: no feedback from a failed attempt reaches the next model. The added formulas (Eqs. 1–4) are bookkeeping, e.g. "extra successes = cases where the cheap model passes and the reference fails".
W2. The main results are expected by construction. Running a stronger check and moving to the next model when it fails will of course return more correct results. In bug fixing, check B runs the problem's public example asserts. These asserts are already in the model's prompt. They also largely copy the hidden tests used for scoring: on the 134 evaluation problems, all public asserts appear word for word in the hidden tests for 85 problems, and some do for 26 more. The buggy code fails these examples on 124 of 134 problems (the authors' own eligibility_audit.json). So check B accepts only 2 wrong outputs, and correct outputs reach the ceiling (121 of 121 available). Check A only verifies that the code changed and parses; on Python/52 it accepts an edit that makes the function always return False. The +11.9 points therefore compare a check that runs part of the scoring tests with a check that runs nothing. In CooperBench and ScienceWorld, the gain over the reference is guaranteed, because the reference always runs last.
W3. The three benchmarks are hardly comparable, and the coding one is simplistic. Only bug fixing has an evaluation separate from the check, so it is the only study where "a correct result existed" and "the check returned it" can differ. In CooperBench, the check is written by the 27B planner and only searches files for strings. For example, it requires normalize_image_bytes and image_to_base64( in a file; the second string is already there before any edit, and adding a comment would satisfy the first. CooperBench's own tests are never run, and the paper's AI-assisted review found defects in all 20 sampled accepted states. In ScienceWorld, the check is the game's subgoal flag, and many subgoals take one command (e.g. "focus on thermometer"). Failures are dominated by a broken command interface: 61–81% of commands are unparseable, e.g. 84–88 of 90 on grow-plant@v3. Scores from these three studies measure different things and cannot support one common conclusion. The coding benchmark itself is HumanEvalPack: single short functions, many fixed by swapping two return values (Python/52, Python/57). It is from 2023, likely in the pretraining data, as the paper admits. Results on repository-level repair would be needed to support claims about coding agents.
Questions for authors
see main weaknesses.