acceptodds
Withdrawn from ICLR 2027

Funnel: Public-Check-Guided Budget Allocation for Native Agent Harnesses

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Batch-oriented agent systems often improve task success by repeating complete trajectories, but uniform candidate budgets spend additional computation on both tasks that already pass a check and tasks that still require rescue. We introduce FUNNEL, a training-free, task-level budget scheduler that operates outside a native agent harness. FUNNEL runs one complete native trajectory for every task, then uses only protocol-permitted public-check outcomes to launch seven additional state-isolated trajectories for first-check failures. Candidates share neither mutable execution state nor feedback, and a fixed selector freezes the final candidate before independent verification. Across four configurations spanning two models, two harnesses, and two task families, Funnel-8 differs from fixed eight-candidate search by only -0.52 to +1.04 percentage points in final pass rate while reducing the number of generated candidates by 28.3–58.6% and reported GPU-hours by 18.2–48.2%. In matched Qwen3-32B/Hermes comparisons on Reasoning Gym and LiveCodeBench-v6, Funnel remains within -0.26 to +1.30 points of ASC and DSC while using fewer candidates and reported GPU-hours. A fixed-average-four UAB adaptation uses a slightly lower budget but trails Funnel by 3.12–7.03 points. These results show that concentrating attempts on first-check failures provides a competitive accuracy–cost operating point without changing native trajectories, while candidate savings do not imply proportional savings in GPU-hours.

Primary area:
generative models
Keywords:
Language model agents, Test-time compute, Adaptive budget allocation, Agent harnesses, Public-check-guided search

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_dZYNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces FUNNEL, a training-free, per-problem budget scheduler that operates externally to existing agent harnesses. For each problem, it first executes a full run and checks for success using the task’s publicly available verification criteria. If the solution passes, execution stops; if it fails, the problem is run seven additional, independent times. A fixed set of rules is then applied to select one candidate from the results, which is finalized and submitted for the hidden, final evaluation.

Experiments: conducted using Qwen3-32B and gpt-oss-20b models, integrated with the HERMES and NANOBOT harnesses, across three benchmarks: Reasoning Gym (384 problems), LiveCodeBench-v6 (96 problems), and Harness-Bench (32 problems). Each configuration was executed exactly once.

Critical strengths
  • Adapting existing per-task allocation methods to the agent setting and comparing them. The paper adapts three existing per-question budget allocation methods to an agent-based setting—evaluating them with one model and one harness (Qwen3-32B/HERMES) on three benchmarks—and provides a comparison.
  • The idea is straightforward and easy to implement.
Critical weaknesses

1. The motivation may not hold: the assumed usage scenario is not established

Both the first sentence of the abstract and the introduction take repeating complete agent trajectories on the same task to raise success as the starting point, without citing any support. In reviewer's daily using of Claude Code, OpenClaw and other LLM Agent tools, I haven't seen such batch-style usage. As far as I know, this practice appears mainly in evaluation and data sampling, whereas the paper presents its contribution as saving compute for agent harnesses. My main concern is if this scenario exists and common. This issue is related to the fundamental motivation of the paper.

2. Insufficient originality and contribution

The method is essentially "run once, check, stop if it passes, otherwise run seven more", which has limited originality. The allocation rule is a fixed rule without derivation and is hard to call a method; and it overlaps with the cited Zuo & Zhu (2026) work without adequate disclosure.

3. Key design choices are unjustified, and the most direct baseline and cost analysis are missing

The fixed rescue count of seven and the absence of early stopping after each check during rescue are key design choices, yet the paper gives no justification or ablation; the most direct alternative, sequential retry, is not tested; what the public check actually is on each benchmark is not specified, and its cost is not reported, even though whether FUNNEL makes sense depends on exactly these costs; and the choice not to use failure information is not compared against methods that do use feedback.


These weaknesses, especially the second one, raise my major concerns, so my current inclination is to reject. I look forward to hearing back from the authors; as long as these issues can be resolved, I would be very glad to raise my score.

Questions for authors
  • Why use 8 times?
  • What is the checking mechanism?
  • Why not stop at the first passing attempt (sequential retry)? Is it too expensive? What's the cost for each checking?
Reviewer_fJG3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces Funnel, a framework for reducing the cost of multi-trajectory sampling by first sampling one native trajectory for each task and then only sampling an additional 7 trajectories when the first was incorrect. The paper evaluates Reasoning Gym and LiveCodeBench with different harnesses and base models. The paper also conducts some analysis on the outcomes vs. GPU-hours cost, and observe some improvement compared to depth-8, the main baseline.

Critical strengths
  • The paper studies an important problem on how to improve the cost efficiency of sampling multiple trajectories in agentic settings, and compares a few different harnesses and base models.
Critical weaknesses
  • Although the overall framework reduces redundant computation, it can also significantly increase the latency as the trajectories become sequential (1 then 7) as opposed to in parallel, and the paper fails to discuss this trade off in a meaningful way.
  • The paper also assumes access to “public pass”, which is essentially a ground truth to the task. This doesn’t reflect many real-world applications where access to the ground truth is not available. This brings the question of the practicality of this framework.
  • In terms of presentation, the proposition in sec 3.4 adds very little to the informativeness of the paper. The results in Table 1 are also difficult to parse.
Questions for authors
  • how would Funnel be used in settings where public pass is not accessible?
Reviewer_Y4H3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Current agentic systems relies on repeating complete trajectories to improve task success, which can be wasteful since not all tasks need the same amount of test time scaling. The paper proposes Funnel to tackle this problem by running the agent once at first and scale to seven agent runs for the failed tasks on certain public checks. The experiment consists of two smaller open source models across different harnesses, evaluated on reasoning, coding, and agentic benchmarks. The results indicate that Funnel is able to closely match baseline methods with decent savings in terms of number of generated candidates and GPU hours.

Critical strengths
  1. The paper targets a specifically timely problem: as test time scaling becoming commonly used agentic techniques, how can we efficiently allocate budget for most of the rollouts assuming not all tasks need the same amount of scaling?
  2. The experiments and results are organized in a clear way that is easy to read and understand. The tables and plots are intuitive and demostrative.
Critical weaknesses
  1. The quality of the public checks seem very important in the paper's setting, since public success at the first try is the primary gate for skipping test time scaling. However the public checks used in the tasks are not disclosed and the overlap between public and private/evaluation checks are not discussed.
  2. The method does not have comparable baselines. To compare the method fairly, other adaptive baselines should be reported, such as sequential runs. Other baselines should be controlling the following factors: number of trajectories, per task total runtime, adaptiveness. For example, per task total runtime of FUNNEL is 2x of a depth-8 run; is there other methods that is 2x runtime? What would happen if we change from 1-7 split of the same budget to 2-6?
  3. The interplay between task difficulty, public checker quality, and model capabiliy is not discussed. For example, difficult tasks and weaker public checks may require a change of spit strategy to more front loaded, while public checks may compensate for model capabilities to save cost, etc.
Questions for authors
  1. Corresponding to W1, can you discuss public/private checks more?
  2. Corresponding to W2, can you add more baselines?
  3. Corresponding to W3, can you discuss the aforementioned interplay?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.