- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper splits AutoResearch into three roles (Orchestrator, Worker, Verifier) and trains a small model on one role at a time, keeping a strong model in the others. Training data comes from trajectories on about 2K synthesized research tasks. Experiments on MLS-Bench-Lite and AutoLab show that training the Verifier gives the largest gain, keeping about 93% of the all-strong-model score while using fewer strong-model tokens.
Critical strengths
(1) The paper asks a practical question, which role is worth training, and shows that the role easiest to replace is not the one that gains most from training. The Worker starts strongest but barely improves, while the Verifier gains the most. This is useful for people who build agent systems with mixed model sizes.
(2) A lot of careful work went into the data pipeline. Turning about 2K research sources into executable tasks with protected evaluators is not easy, and the role-structured harness alone helps all three strong models on four benchmarks.
Critical weaknesses
(1) The trained small-Verifier system is still worse than just running GLM-5.2 without any role structure. On MLS-Lite, Qwen Verifier-SFT gets 37.1 and Gemma gets 38.4, while the plain GLM-5.2 baseline gets 40.4. On AutoLab, Qwen gets 64.3 against 68.5. Inserting an untrained small Verifier drops the score by more than 15 points, so much of the 10.6-point “training gain” just repairs damage the small model caused. A run with GLM Orchestrator and Worker and no Verifier at all is missing, and without it I cannot tell whether the small Verifier helps.
(2) I am not convinced the token savings come from training or from moving review to the small model. The untrained Verifier already cuts GLM tokens from 366.1M to 204.4M (about 44%). After training, the small model handles only 16.8M tokens, yet GLM usage drops by 172M, which means the GLM Orchestrator and Worker are simply doing less work. Appendix E.3 also shows the trained Verifier returns usable output in only 27 of 90 calls, against 90 of 90 for GLM, and is slower (187.7 vs. 70.5 seconds median).
(3) The claim that the Verifier is the best role to train is confounded by data size. The Verifier has 7,300 training examples, the Worker 4,700, and the Orchestrator only 1,900. At a similar number of examples the gap mostly disappears, since the Verifier at 25% (about 1,800 examples) gains 4.8 points and the Orchestrator at 100% (1,900 examples) gains 5.2. The Worker comparison is not clean either. After training, the small Worker uses only 1.6M tokens versus 53.3M before, so it is rarely called at all.
(4) Several claims go beyond the evidence. The abstract says the 51.1% token reduction holds “across various autoresearch benchmarks,” but tokens are only measured on MLS-Lite. The contribution list says Verifier-SFT gives larger gains per improvement than Joint-SFT, yet every confidence interval in Figure 10 crosses zero. Table 2, the main result, also has no standard deviations.
(5) Training a small critic to guide a stronger agent has been studied before [1, 2, 3]. Steer, Don’t Solve [1] does almost the same thing for code agents, a small SFT critic giving mid-trajectory feedback to a frozen strong agent at lower cost. None of these are cited, so the novelty of the main finding is unclear.
References: [1] Steer, Don’t Solve: Training Small Critic Models for Large Code Agents. arXiv 2606.21811, 2026. [2] Teaching Language Models to Critique via Reinforcement Learning (CTRL). arXiv 2502.03492, 2025. [3] The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement. NeurIPS 2025.
Questions for authors
(1) What does the system score with GLM-5.2 as Orchestrator and Worker and no Verifier at all? If it is close to 40, the small Verifier may not be adding anything.
(2) Could you compare the three roles with the same number of training examples, or at least the same number of tokens? Right now the Verifier has about four times more data than the Orchestrator, and at similar sizes their gains look almost the same.
(3) I would like to see how often the trained Verifier is actually called per task, and how much of its feedback is usable in the real runs, not just in the E.3 probe. This would show whether the token savings come from offloading review or just from the strong model doing less research.
2 comments
Reply to Reviewer AXVc, Part 2/2
W3 / Q2 — Corpus-size confound; Worker rarely called after training
Thank you for pressing on both points — each is addressed by the data. At equal compute the Verifier leads on both benchmarks (MLS-Lite +10.6 vs +5.2 Orchestrator; AutoLab +10.5 vs +5.1). And the trained Worker is barely called yet the score is unchanged, which indicates execution is not the AutoResearch bottleneck.
| Worker | Small-model tokens | MLS-Lite |
|---|---|---|
| Untrained | 53.3M | 32.3 |
| Trained | 1.6M | 32.9 |
This mirrors the core finding of the paper: the Worker is the easiest role to drop in untrained and the least rewarding to train.
W4 — Overclaims (token scope; per-improvement CIs; missing std)
Thank you for these rigorous checks — all three are now addressed. The scope covers both benchmarks, the CI is significant, and std is added; no conclusion changes.
-
Scope. Token reduction is measured on both benchmarks; 51.1% is their mean (abstract, introduction, and the inference-allocation section updated):
GLM-token reduction Qwen Gemma MLS-Lite 47.0% 55.1% AutoLab 47.5% 54.7% -
Per-improvement gain, N=54 paired trajectories:
Metric Joint-SFT Verifier-SFT Gain per improvement 9.6 13.0 New-best rate 58.7% 58.9% Bootstrap 95% CI on the difference [0.8, 6.1], excluding zero.
-
Std. Following the benchmark papers' convention we had reported 3-run averages only; variance is indeed necessary, so we computed it and updated the paper (Table 2): Qwen Verifier-SFT 37.1 ± 0.7 / 64.3 ± 0.6; Gemma 38.4 ± 1.1 / 69.3 ± 0.9; no-review (O+W) 39.8 ± 1.3 / 68.3 ± 1.0. The variance does not change the picture: the Verifier gain (+11.0) far exceeds it, while the Worker's +0.6 sits within it, consistent with our claim that the Worker is not the role to train.
W5 — Prior small-critic work (Steer-Don't-Solve, CTRL, Lighthouse)
All three will be cited. That a small critic can help is accepted; what it looks like on AutoResearch (which roles, why, how supervision scales) is what we add.
- Steer-Don't-Solve is closest: a small SFT critic steering a frozen strong code agent, added externally. We place the reviewer inside the AutoResearch execution system and study which capability repays training.
We thank the reviewers again and are glad to run any further experiment.
Reply to Reviewer AXVc, Part 1/2
Dear Reviewer AXVc,
Thank you for the detailed questions; we ran an experiment for each.
The no-Verifier control now exists. The small Verifier's recovered performance comes from training, not downsizing, and we show this causally. Every statistic you flagged is tightened. Beyond efficiency, the trained small Verifier adds capability on fine-grained domains (e.g., RL, vision, LLM post-training).
Without the Verifier the system falls back to baseline level, so the review role supplies the role-structure gain. The small Verifier's job is efficiency plus domain capability, not absolute parity.
To us, replacing a role is about more than savings: the decomposition serves role-specific training — the one adaptation lever a domain expert can afford — and on RL, vision, and LLM post-training the trained small Verifier exceeds the all-strong system (61.1 vs 60.1; 28.8 vs 20.2; 84.4 vs 82.2).
An expert who cannot retrain a backbone can train a small Verifier on self-built domain knowledge and obtain review ability the strong general model lacks.
With the strong-model budget essentially fixed, training alone buys +11 points — the saving comes from a smarter Verifier making GLM waste less work, not GLM doing less. Live usability is also far better than the offline probe suggests.