- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ForkPilot, a self-evolving search controller for long-horizon LLM agents. It learns to dynamically allocate retrospective search based on predicted benefits, computational costs, and uncertainty. Experiments across six benchmarks and multiple LLM backbones demonstrate substantial token savings while maintaining comparable task performance.
Critical strengths
The paper addresses an important practical problem of reducing inference costs in long-horizon agents. It provides extensive evaluations across diverse benchmarks and model scales, including frontier proprietary models. The ablation studies also provide evidence supporting the effectiveness of its adaptive search allocation and uncertainty-aware decisions.
Critical weaknesses
-
Uncertain baseline reliability. Several baselines lack official implementations or validated results on the evaluated benchmarks, requiring adaptations by the authors. It remains unclear whether these adaptations faithfully reproduce the original methods or adequately optimize them for the new environments, potentially weakening the comparisons.
-
Insufficient justification of necessity. While the paper demonstrates the effectiveness of a separately trained search controller, it does not establish whether such a mechanism is necessary. Stronger backbone models or direct LLM-based search decisions might achieve similar efficiency without expensive paired-replay training, whose costs are not adequately accounted for in the main comparisons.
Questions for authors
-
How did the authors validate the correctness and competitiveness of baseline implementations on benchmarks not originally evaluated by those methods?
-
Can the authors compare ForkPilot against direct LLM-based search allocation and evaluate whether its efficiency gains justify the additional training and data collection costs?
2 comments
Rebuttal Part II
W2 / Q2. Is a trained controller necessary, and does it justify its cost?
In the new experiment in which GPT-5.6 Sol controls its own search on Automation (part (b) below), ForkPilot reaches nearly the same score (69.00 vs. 69.33) with 16% fewer raw and 28% fewer uncached tokens, and its controller needs no additional LLM call per decision. With Sol deciding for the Qwen actor, price-weighted cost is 22% higher than ForkPilot's on Automation (17% on τ³; draft Table 25). The advantage over the adaptive allocators holds on both Qwen3.5-27B and Qwen3.5-122B-A10B.
Table 2. Task performance and token usage of ForkPilot, RTS baselines, and LLM-as-controller variants. (a) Qwen3.5-27B on AutomationBench and τ³. (b) GPT-5.6 Sol on AutomationBench with 600 tasks; the LLM-as-controller rows, where Sol controls its own search, are a new experiment (draft Table 25), and the RTS and ForkPilot rows are from draft Tables 3 and 12. S denotes task score; R and U denote raw and uncached token usage, respectively, normalized to the corresponding RTS baseline within each backbone–benchmark setting. Higher scores and lower token ratios are better. Boldface highlights ForkPilot. † Sequential host schedule; all other rows use the parallel host.
(a) Qwen3.5-27B
| Method (Qwen3.5-27B) | Automation: Score (S ↑) | Automation: Raw / Uncached ratio (R / U ↓) | τ³: Score (S ↑) | τ³: Raw / Uncached ratio (R / U ↓) |
|---|---|---|---|---|
| RTS | 28.7 | 1.00 / 1.00 | 70.6 | 1.00 / 1.00 |
| Efficient RTS / RTS(Sequential), ρ = 0.8† | 27.3 | 0.58 / 0.63 | 69.2 | 0.61 / 0.64 |
| Backbone as controller (zero-shot) | 27.6 | 0.93 / 0.95 | 69.7 | 0.91 / 0.94 |
| Backbone as controller (few-shot) | 28.3 | 0.85 / 0.88 | 70.4 | 0.83 / 0.87 |
| Backbone as controller (cost-aware) | 27.2 | 0.68 / 0.73 | 69.0 | 0.68 / 0.71 |
| ForkPilot | 29.2 | 0.61 / 0.66 | 71.0 | 0.63 / 0.67 |
(b) GPT-5.6 Sol
| Method | Score (S ↑) | R / U ↓ |
|---|---|---|
| RTS | 71.83 | 1.000 / 1.000 |
| Efficient RTS / RTS(Sequential), ρ = 0.8† | 60.17 | 0.555 / 0.530 |
| LLM-as-controller (zero-shot) | 69.33 | 0.912 / 0.768 |
| LLM-as-controller (few-shot) | 69.33 | 1.123 / 0.895 |
| ForkPilot (parallel) | 69.00 | 0.764 / 0.556 |
| ForkPilot (sequential)† | 75.00 | 0.679 / 0.402 |
ForkPilot learns fork-level gain and cost estimates from paired outcomes, whereas prompted LLM controllers either over-search (zero-shot, few-shot) or sacrifice task quality (cost-aware). Unlike these alternatives, ForkPilot requires no additional LLM calls per allocation decision. The remaining question is whether its upfront learning cost is justified.
Our source-size experiments show that effective learned allocation does not require full-scale source collection. Across the four frontier settings, full-pool raw-token amortization takes 332–4,687 deployed tasks. On GPT-5.6 Sol, using just 25% of the source pool preserves near-full-pool task quality while reducing raw-token break-even to 60–1,030 tasks. We additionally evaluate 50% and 75% training fractions to characterize source-size sensitivity:
Table 3. Source-size sensitivity and learning-inclusive token amortization on frontier backbones (draft Table 11). ΔS is relative to RTS; break-even extrapolates observed per-task token savings to repeated deployment.
| Setting | Training split | ΔS vs. RTS | Source tokens, raw / uncached (M) | Break-even, raw / uncached (tasks) |
|---|---|---|---|---|
| Sol · Automation | Full | −2.83 | 881 / 564 | 4,136 / 2,035 |
| 75% | −2.71 | 649 / 417 | 3,078 / 1,361 | |
| 50% | −4.83 | 442 / 282 | 1,997 / 993 | |
| 25% | −3.16 | 219 / 140 | 1,030 / 508 | |
| Sol · τ³ | Full | −3.10 | 1,067 / 359 | 354 / 231 |
| 50% | −3.10 | 466 / 156 | 143 / 96 | |
| 25% | −2.58 | 203 / 69 | 60 / 42 |
On Sol / τ³, 25% training achieves 64.95 versus 64.43 with full training (both solve 63/97 tasks), using 9.3% fewer inference raw tokens and 81% fewer source raw tokens. Raw-token break-even consequently falls from 354 to 60 tasks (231 to 42 for uncached tokens). On Automation, 25% training achieves 68.67 versus 69.00 with full training, reducing raw-token break-even from 4,136 to 1,030 tasks. At 75%, Automation reaches 69.12 with 26% fewer source raw tokens than full training.
Together, these results demonstrate that ForkPilot's learned allocation can provide substantial deployment-token savings with considerably less source supervision. The complete sweep also shows that performance is not monotonic in training size. Break-even figures quantify token amortization rather than quality-adjusted utility recovery; full accounting appears in draft Table 11 (App. C.3).
Rebuttal Part I
Thank you for the review and the focused questions. We ran a new experiment with GPT-5.6 Sol controlling its own search on Automation: ForkPilot scores 69.00 versus 69.33, with 16% fewer raw and 28% fewer uncached tokens (W2 below).
W1 / Q1. Validity of the baseline implementations.
Following the reviewer's suggestion, we reproduced each memory and context baseline on a benchmark from its original paper and re-tuned all five with the same source-validation budget as ForkPilot. The implementations run from the authors' official repositories and prompts, with only the environment interface adapted; draft Table 16 (App. C.7) lists the commits; we restate the implementation setup here as well. Koh et al.'s official tree search has also been added. The reproductions (new, draft App. C.7):
Table 1. Reproduction of the five memory and context baselines on a benchmark from each original paper.
To check sensitivity to tuning in the new environments, we re-tuned each of these five baselines on Qwen3.5-27B / Automation with the same number of source-validation configurations as ForkPilot (5). No baseline moved by more than 0.3 points, and none improved on ForkPilot's score–cost operating point: the strongest, Trace2Skill, went from 29.2 to 29.3, which reaches ForkPilot's score of 29.2 but at 0.66 / 0.74 raw / uncached tokens against ForkPilot's 0.61 / 0.66. All methods share the same host infrastructure, tools, scorer, cache policy, source split and token accounting, and nothing is tuned on test tasks. These memory and context methods also complement ForkPilot: combining ReasoningBank with ForkPilot improves on both (draft Table 8).