- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces FUNNEL, a training-free, per-problem budget scheduler that operates externally to existing agent harnesses. For each problem, it first executes a full run and checks for success using the task’s publicly available verification criteria. If the solution passes, execution stops; if it fails, the problem is run seven additional, independent times. A fixed set of rules is then applied to select one candidate from the results, which is finalized and submitted for the hidden, final evaluation.
Experiments: conducted using Qwen3-32B and gpt-oss-20b models, integrated with the HERMES and NANOBOT harnesses, across three benchmarks: Reasoning Gym (384 problems), LiveCodeBench-v6 (96 problems), and Harness-Bench (32 problems). Each configuration was executed exactly once.
Critical strengths
- Adapting existing per-task allocation methods to the agent setting and comparing them. The paper adapts three existing per-question budget allocation methods to an agent-based setting—evaluating them with one model and one harness (Qwen3-32B/HERMES) on three benchmarks—and provides a comparison.
- The idea is straightforward and easy to implement.
Critical weaknesses
1. The motivation may not hold: the assumed usage scenario is not established
Both the first sentence of the abstract and the introduction take repeating complete agent trajectories on the same task to raise success as the starting point, without citing any support. In reviewer's daily using of Claude Code, OpenClaw and other LLM Agent tools, I haven't seen such batch-style usage. As far as I know, this practice appears mainly in evaluation and data sampling, whereas the paper presents its contribution as saving compute for agent harnesses. My main concern is if this scenario exists and common. This issue is related to the fundamental motivation of the paper.
2. Insufficient originality and contribution
The method is essentially "run once, check, stop if it passes, otherwise run seven more", which has limited originality. The allocation rule is a fixed rule without derivation and is hard to call a method; and it overlaps with the cited Zuo & Zhu (2026) work without adequate disclosure.
3. Key design choices are unjustified, and the most direct baseline and cost analysis are missing
The fixed rescue count of seven and the absence of early stopping after each check during rescue are key design choices, yet the paper gives no justification or ablation; the most direct alternative, sequential retry, is not tested; what the public check actually is on each benchmark is not specified, and its cost is not reported, even though whether FUNNEL makes sense depends on exactly these costs; and the choice not to use failure information is not compared against methods that do use feedback.
These weaknesses, especially the second one, raise my major concerns, so my current inclination is to reject. I look forward to hearing back from the authors; as long as these issues can be resolved, I would be very glad to raise my score.
Questions for authors
- Why use 8 times?
- What is the checking mechanism?
- Why not stop at the first passing attempt (sequential retry)? Is it too expensive? What's the cost for each checking?