acceptodds
Under review as a conference paper at ICLR 2027

AgentClutch: Verifier-Gated Computation Allocation for Stateful Agents

Abstract

An agent cascade can contain a test-passing fallback yet return an earlier failing output. We study this failure with AgentClutch, a fixed-order cascade that restores a registered execution state before fallback. Our analysis separates the success available among retained outputs, the success delivered by the stopping rule, and the executor cost of acquiring it. On 134 Python repair problems, replacing syntax stopping with public-example checks, while holding candidate and reference outputs fixed, raises delivered test success from 78.4% to 90.3%. The stronger gate recovers 16 blocked test-passing fallbacks at 29.1% higher executor-token use. In CooperBench and ScienceWorld, a middle executor adds unique operational acceptances but costs more than the downstream reference work its successful stops avoid. The full cascade nevertheless remains cheaper than standalone reference execution in the CooperBench census under a fixed token tariff. These findings show why complementary outputs, reliable selection, and economical acquisition must be evaluated separately. The studies concern restorable execution units rather than uninterrupted task completion.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Model Routing, Plan Execution, Cost-Aware Delegation, Agentic Workflows, Adaptive Computation, Multi-Agent Systems

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_joFzOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how to get work done with several LLMs sequentially. The task is, for example, fixing a bug in one Python function, making one planned edit in a repository (e.g. "add a separator parameter to these functions"), or reaching one subgoal in a simulated science game (e.g. "focus on the thermometer"). In repository edits and the science game, each step is scored alone, starting from a correct state supplied by the benchmark or by the reference model. The models are tried one after another on that step, cheapest first. After each attempt, an automatic check (the paper calls it a "gate") decides whether to keep the result. The check is executable code, but its origin differs per study: written by the authors for bug fixing (does the code parse, do the problem's examples pass), written by the 27B planner model for repository edits (does a file contain given strings), and built into the game for the science subgoals. If the result is kept, the process stops. Otherwise the task's files or environment are reset to their initial state, and the next model tries. The last model in the list is the most expensive one, called the "reference". The models are Qwen3-8B and Qwen3-14B, each with thinking on or off; the reference is Qwen3-14B with thinking on.

The paper reports three studies:

  1. Bug fixing on 134 HumanEvalPack Python problems, with two models: Qwen3-14B without thinking first, then with thinking. Each model gets the same prompt: the function's docstring, the buggy code, and the problem's public example asserts, e.g. for Python/52 the docstring of below_threshold and assert below_threshold([1, 2, 4, 10], 100). The checks are applied after the attempt and are never shown to the model. Each model's output is generated once, and only the check changes. Check A keeps the first output if the code was changed, still parses, and still defines the function. Check B additionally requires the public example asserts to pass. For example, on Python/52 the first model only flips return True to return False: check A keeps this wrong fix, while check B rejects it, so the second model's correct fix is returned (Python/57 is the same case). Measured on the benchmark's hidden tests, check B raises the share of correct results from 78.4% to 90.3%, at 29.1% more tokens.
  2. Code edits on 357 CooperBench steps, taken from plans written by a 27B model. Trying three models in sequence instead of the reference alone raises the share of steps passing the check from 81.5% to 92.7%, at 64% of the reference's token cost. The middle model costs more than the reference work it saves.
  3. ScienceWorld subgoals, each started by replaying the benchmark's reference actions up to that subgoal. Trying three models in sequence passes more subgoals than the reference alone, but costs more.

In studies 2 and 3, the gain comes only from trying more models: no information passes from one attempt to the next.

Critical strengths

Good transparency and reproducibility. The supplement contains the code, the registered protocols, every recorded attempt. The authors' reproduce.py runs without a GPU and passes: it checks the file hashes and recomputes the CooperBench counts (291, 324 and 331 accepted steps) from the 1,428 recorded cells.

Critical weaknesses

W1. The technical contribution is very thin. The method tries a list of models one after another, checks each result, resets the files on failure, and stops at the first result that passes. This is the standard model cascade studied by FrugalGPT, AutoMix and cascade routing (Dekoninck et al., 2025), all cited by the paper. It is also the "filter by public tests" step of AlphaCode and CodeT. The models share nothing: no feedback from a failed attempt reaches the next model. The added formulas (Eqs. 1–4) are bookkeeping, e.g. "extra successes = cases where the cheap model passes and the reference fails".

W2. The main results are expected by construction. Running a stronger check and moving to the next model when it fails will of course return more correct results. In bug fixing, check B runs the problem's public example asserts. These asserts are already in the model's prompt. They also largely copy the hidden tests used for scoring: on the 134 evaluation problems, all public asserts appear word for word in the hidden tests for 85 problems, and some do for 26 more. The buggy code fails these examples on 124 of 134 problems (the authors' own eligibility_audit.json). So check B accepts only 2 wrong outputs, and correct outputs reach the ceiling (121 of 121 available). Check A only verifies that the code changed and parses; on Python/52 it accepts an edit that makes the function always return False. The +11.9 points therefore compare a check that runs part of the scoring tests with a check that runs nothing. In CooperBench and ScienceWorld, the gain over the reference is guaranteed, because the reference always runs last.

W3. The three benchmarks are hardly comparable, and the coding one is simplistic. Only bug fixing has an evaluation separate from the check, so it is the only study where "a correct result existed" and "the check returned it" can differ. In CooperBench, the check is written by the 27B planner and only searches files for strings. For example, it requires normalize_image_bytes and image_to_base64( in a file; the second string is already there before any edit, and adding a comment would satisfy the first. CooperBench's own tests are never run, and the paper's AI-assisted review found defects in all 20 sampled accepted states. In ScienceWorld, the check is the game's subgoal flag, and many subgoals take one command (e.g. "focus on thermometer"). Failures are dominated by a broken command interface: 61–81% of commands are unparseable, e.g. 84–88 of 90 on grow-plant@v3. Scores from these three studies measure different things and cannot support one common conclusion. The coding benchmark itself is HumanEvalPack: single short functions, many fixed by swapping two return values (Python/52, Python/57). It is from 2023, likely in the pretraining data, as the paper admits. Results on repository-level repair would be needed to support claims about coding agents.

Questions for authors

see main weaknesses.

Reviewer_zB5LOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors discuss a very simple retry mechanism. An agent tries to solve a problem (for example writing python code). The candidate solution is checked with a simple verified (for example a syntax check). On failure, retry. The authors then compare this with an alternative setup that includes additional checks (for example running the candidate solution against a public example test-set). On failure, retry.

Retries can use different models/agents/executors. The additional tests catch additional mistakes ...

This behaviour is discussed on a variety of benchmarks and presented as an insightful failure analysis.

Critical strengths

Sorry, none.

Critical weaknesses

I find the central repair result entirely unsurprising: adding functional checks detects errors that simple syntax checks miss. The paper does not establish why the failure mode (early stopping, even though more discriminative tests could prevent wrong solution attempts) is representative of existing agentic systems. The experiments offer additional checks but compare against a constellation that doesn't use them. Why would these checks be omitted in practice? I don't believe that this syntax-only stopping rule is representative of existing agentic systems. Either additional tests are available, and then I believe that a reasonable agentic runtime would use them. Or they're not available, but then not running them is not a failure, it's simply not possible.

My understanding of the paper is: The authors construct a tailor-made workflow themselves, show that it fails, and then discuss this failure. As far as I can see, the paper provides no evidence that this would happen with any real agent harness.

There are additional experiments that investigate acceptance and cost of fallback configurations. That seems like an orthogonal question and does not resolve the missing motivation for the central experiment.

The experimental methodology in itself is OK, but the setup that is studied is not interesting.

Questions for authors

Do you have any evidence that the stopping failures presented here occur in existing systems?

Reviewer_4c6xOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies verifier-gated fallback for stateful agents: if a check fails, the agent restores its initial state and hands the task to the next executor. The key contribution is separating three quantities: the successes available among candidate outputs, the successes the gate actually delivers, and the execution cost of getting them. On 134 repair tasks with fixed candidate outputs, adding public-example checks to syntax checks raises success from 78.4% to 90.3% at the cost of 29.1% more execution tokens. The paper also uses CooperBench and ScienceWorld to examine the benefits and costs of adding more executors.

Critical strengths
  1. Holding outputs fixed and varying only the gate is a clean design. It cleanly separates selection effects from generation effects on those outputs.
  2. The cost analysis is insightful. It shows that an intermediate executor can produce extra passes without saving enough downstream cost to pay for itself.
Critical weaknesses
  1. The local analysis does not yet show practical value. The paper shows how gating shapes which outputs are selected and what they cost, but the evidence is mostly limited to local execution units. It remains unclear whether these gains carry over to end-to-end task accuracy and total cost.
  2. The benefit of switching executors is unclear. The current baselines do not show how much switching executors adds over well-designed same-executor retries at the same compute budget. Without this comparison, it is hard to judge whether this way of spending compute is worthwhile.
Questions for authors
  1. Is there evidence that the gains from local gating improve end-to-end task accuracy and total cost?
  2. At the same compute budget, how much does switching executors gain over same-executor retries that produce diverse candidates?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.