acceptodds
Under review as a conference paper at ICLR 2027

Small Critics Make Strong Models Count: Learning to Verify Stronger Models for Efficient AutoResearch

Abstract

AutoResearch agents improve solutions through repeated experiments, but the intertwined demands of decision-making, execution, and analysis make them difficult to build and train. Using strong models throughout is also costly. We study which small-model capability yields the largest system-level gains from training. We introduce ResearchWeaver, which decomposes research into an Orchestrator, Worker, and Verifier. This makes research experience usable as role-specific supervision, so a small model can learn one responsibility rather than the entire process. We develop an automated task synthesis pipeline and collect role-structured trajectories on the resulting executable tasks. By training and replacing one role at a time while retaining strong models in the others, we identify where small-model learning helps most: a trained Verifier can influence subsequent research through its diagnoses and suggested next checks. Experiments on diverse benchmarks show that ResearchWeaver consistently improves performance across all three evaluated frontier models by explicitly structuring research into complementary roles. Training small models on role-structured experience further improves performance at the same source-task fraction, and role-specific experiments identify Verifier training as the highest-gain target. With supervision derived from only ∼2K source tasks, training small Verifiers delivers a 10.6-point gain on average, retaining 92.9% of the all-strong-model reference performance while reducing strong-model token use by 51.1% on average across various autoresearch benchmarks. These findings identify Verifier training as a high-leverage path to efficient AutoResearch.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AutoResearch, LLM agents, Verifier training, Agent efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AXVcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper splits AutoResearch into three roles (Orchestrator, Worker, Verifier) and trains a small model on one role at a time, keeping a strong model in the others. Training data comes from trajectories on about 2K synthesized research tasks. Experiments on MLS-Bench-Lite and AutoLab show that training the Verifier gives the largest gain, keeping about 93% of the all-strong-model score while using fewer strong-model tokens.

Critical strengths

(1) The paper asks a practical question, which role is worth training, and shows that the role easiest to replace is not the one that gains most from training. The Worker starts strongest but barely improves, while the Verifier gains the most. This is useful for people who build agent systems with mixed model sizes.

(2) A lot of careful work went into the data pipeline. Turning about 2K research sources into executable tasks with protected evaluators is not easy, and the role-structured harness alone helps all three strong models on four benchmarks.

Critical weaknesses

(1) The trained small-Verifier system is still worse than just running GLM-5.2 without any role structure. On MLS-Lite, Qwen Verifier-SFT gets 37.1 and Gemma gets 38.4, while the plain GLM-5.2 baseline gets 40.4. On AutoLab, Qwen gets 64.3 against 68.5. Inserting an untrained small Verifier drops the score by more than 15 points, so much of the 10.6-point “training gain” just repairs damage the small model caused. A run with GLM Orchestrator and Worker and no Verifier at all is missing, and without it I cannot tell whether the small Verifier helps.

(2) I am not convinced the token savings come from training or from moving review to the small model. The untrained Verifier already cuts GLM tokens from 366.1M to 204.4M (about 44%). After training, the small model handles only 16.8M tokens, yet GLM usage drops by 172M, which means the GLM Orchestrator and Worker are simply doing less work. Appendix E.3 also shows the trained Verifier returns usable output in only 27 of 90 calls, against 90 of 90 for GLM, and is slower (187.7 vs. 70.5 seconds median).

(3) The claim that the Verifier is the best role to train is confounded by data size. The Verifier has 7,300 training examples, the Worker 4,700, and the Orchestrator only 1,900. At a similar number of examples the gap mostly disappears, since the Verifier at 25% (about 1,800 examples) gains 4.8 points and the Orchestrator at 100% (1,900 examples) gains 5.2. The Worker comparison is not clean either. After training, the small Worker uses only 1.6M tokens versus 53.3M before, so it is rarely called at all.

(4) Several claims go beyond the evidence. The abstract says the 51.1% token reduction holds “across various autoresearch benchmarks,” but tokens are only measured on MLS-Lite. The contribution list says Verifier-SFT gives larger gains per improvement than Joint-SFT, yet every confidence interval in Figure 10 crosses zero. Table 2, the main result, also has no standard deviations.

(5) Training a small critic to guide a stronger agent has been studied before [1, 2, 3]. Steer, Don’t Solve [1] does almost the same thing for code agents, a small SFT critic giving mid-trajectory feedback to a frozen strong agent at lower cost. None of these are cited, so the novelty of the main finding is unclear.

References: [1] Steer, Don’t Solve: Training Small Critic Models for Large Code Agents. arXiv 2606.21811, 2026. [2] Teaching Language Models to Critique via Reinforcement Learning (CTRL). arXiv 2502.03492, 2025. [3] The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement. NeurIPS 2025.

Questions for authors

(1) What does the system score with GLM-5.2 as Orchestrator and Worker and no Verifier at all? If it is close to 40, the small Verifier may not be adding anything.

(2) Could you compare the three roles with the same number of training examples, or at least the same number of tokens? Right now the Verifier has about four times more data than the Orchestrator, and at similar sizes their gains look almost the same.

(3) I would like to see how often the trained Verifier is actually called per task, and how much of its feedback is usable in the real runs, not just in the E.3 probe. This would show whether the token savings come from offloading review or just from the strong model doing less research.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer AXVc, Part 1/2

Dear Reviewer AXVc,

Thank you for the detailed questions; we ran an experiment for each.

The no-Verifier control now exists. The small Verifier's recovered performance comes from training, not downsizing, and we show this causally. Every statistic you flagged is tightened. Beyond efficiency, the trained small Verifier adds capability on fine-grained domains (e.g., RL, vision, LLM post-training).

W1 / Q1 — No-Verifier baseline: does the small Verifier help?

Without the Verifier the system falls back to baseline level, so the review role supplies the role-structure gain. The small Verifier's job is efficiency plus domain capability, not absolute parity.

SystemMLS-LiteAutoLab
Single-agent baseline40.468.5
Orchestrator + Worker, no Verifier39.868.3
Full (O + W + Verifier)41.570.5
Trained small Verifier (Qwen / Gemma)37.1 / 38.464.3 / 69.3
  • A direct answer, including the unfavorable cells. Against the no-Verifier control, the trained small Verifier sits below it on MLS-Lite (37.1/38.4 vs 39.8) and above it on AutoLab with Gemma (69.3 vs 68.3). So on MLS-Lite the small Verifier is a cost lever at slightly lower pooled quality; on AutoLab a trained small Verifier adds net value even over omitting the role. Note also that the no-Verifier control is a scientific baseline, not a deployment option: removing the Verifier saves few strong-model tokens (the Orchestrator still re-checks the work itself) while giving up the structure gain.
  • What the review role itself contributes. Without it the system stays at baseline level (39.8 vs 40.4; 68.3 vs 68.5) and far below the full system (41.5 / 70.5); the full structure beats the baseline in all 12 cells of Table 1.
  • What the small Verifier buys, and what it does not. It buys roughly half the strong-model token cost, and per-domain capability beyond the all-strong system (RL 61.1 vs 60.1; CV 28.8 vs 20.2; grpo post-training 84.4 vs 82.2 — full table in the Global Response); it does not buy pooled-score parity with all-strong. That its recovery comes from training rather than downsizing is shown two ways: the untrained-to-trained jump in a fixed slot (26.1→37.1), and the causal matched-continuation test, where on identical frozen states a trained review doubles per-step progress and an untrained one does worse than no review.

On "what is replacing a role for"

To us, replacing a role is about more than savings: the decomposition serves role-specific training — the one adaptation lever a domain expert can afford — and on RL, vision, and LLM post-training the trained small Verifier exceeds the all-strong system (61.1 vs 60.1; 28.8 vs 20.2; 84.4 vs 82.2).

  • MLS-Lite: the trained small Verifier exceeds all-strong on RL (61.1 vs 60.1) and CV (28.8 vs 20.2); the largest gain sits where the strong model is weakest (GLM-5.2 baseline 12.7 on RL; training +58.5).
  • AutoLab: the untrained Verifier scores 0.0 on whole task families; training restores or exceeds all-strong (0 → 84.4 > 82.2; 0 → 41.3 ≈ 41.5).
  • Worker training barely helps in any domain we tested.

An expert who cannot retrain a backbone can train a small Verifier on self-built domain knowledge and obtain review ability the strong general model lacks.

W2 / Q3 — Training or just a smaller model? Usability (27/90) and latency

With the strong-model budget essentially fixed, training alone buys +11 points — the saving comes from a smarter Verifier making GLM waste less work, not GLM doing less. Live usability is also far better than the offline probe suggests.

Verifier in slotGLM-5.2 tokensMLS-Lite score
Untrained204.4M26.1
Trained (SFT)194.0M37.1
  • Budget unchanged, +11 points; the matched-continuation test confirms this causally.
  • 27/90 and the 187.7s median latency both come from the same offline stress probe (Appendix E.3, 64K cap, concurrency 15). Live runs behave differently: per task 4.1 review requests, 71% usable outputs, 48% adopted, and review is issued while experiments execute, so it sits off the critical path; wall-clock per task barely moves (≈2.1 → 2.2 h).
AuthorsOpenReview ↗

Reply to Reviewer AXVc, Part 2/2

W3 / Q2 — Corpus-size confound; Worker rarely called after training

Thank you for pressing on both points — each is addressed by the data. At equal compute the Verifier leads on both benchmarks (MLS-Lite +10.6 vs +5.2 Orchestrator; AutoLab +10.5 vs +5.1). And the trained Worker is barely called yet the score is unchanged, which indicates execution is not the AutoResearch bottleneck.

WorkerSmall-model tokensMLS-Lite
Untrained53.3M32.3
Trained1.6M32.9

This mirrors the core finding of the paper: the Worker is the easiest role to drop in untrained and the least rewarding to train.

W4 — Overclaims (token scope; per-improvement CIs; missing std)

Thank you for these rigorous checks — all three are now addressed. The scope covers both benchmarks, the CI is significant, and std is added; no conclusion changes.

  • Scope. Token reduction is measured on both benchmarks; 51.1% is their mean (abstract, introduction, and the inference-allocation section updated):

    GLM-token reductionQwenGemma
    MLS-Lite47.0%55.1%
    AutoLab47.5%54.7%
  • Per-improvement gain, N=54 paired trajectories:

    MetricJoint-SFTVerifier-SFT
    Gain per improvement9.613.0
    New-best rate58.7%58.9%

    Bootstrap 95% CI on the difference [0.8, 6.1], excluding zero.

  • Std. Following the benchmark papers' convention we had reported 3-run averages only; variance is indeed necessary, so we computed it and updated the paper (Table 2): Qwen Verifier-SFT 37.1 ± 0.7 / 64.3 ± 0.6; Gemma 38.4 ± 1.1 / 69.3 ± 0.9; no-review (O+W) 39.8 ± 1.3 / 68.3 ± 1.0. The variance does not change the picture: the Verifier gain (+11.0) far exceeds it, while the Worker's +0.6 sits within it, consistent with our claim that the Worker is not the role to train.

W5 — Prior small-critic work (Steer-Don't-Solve, CTRL, Lighthouse)

All three will be cited. That a small critic can help is accepted; what it looks like on AutoResearch (which roles, why, how supervision scales) is what we add.

  • Steer-Don't-Solve is closest: a small SFT critic steering a frozen strong code agent, added externally. We place the reviewer inside the AutoResearch execution system and study which capability repays training.

We thank the reviewers again and are glad to run any further experiment.

Reviewer_iLwpOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors show that a role-based agentic framework (Orchestrator, worker, & Verifier) improves LLM performance across automated research tasks; on Opus, the average score across benchmarks improves from 38.9 to 41.7 (GLM’s 42.0 -> 43.6). They replicate this across multiple large LLMs (Opus 4.6, GLM-5.2, DeepSeek-V4.1-Flash) and across benchmarks.

They also run several ablations, showing which roles a distilled, smaller LLM can replace with the least hit to accuracy. On MLS-Lite, GLM’s score drops from 41.5 to 37.1 when they substitute the verifier role with a distilled Qwen3.5-9B model; replacing the worker role with Qwen drops it to 32.9, and replacing the orchestrator drops the score to just 19.7.

Critical strengths
  1. Ablations that show which roles, when substituted with a smaller model, benefit most from training. Using a 9B model for all roles drops accuracy on MLS-Lite from 41.5 to 9.1. Substituting only the orchestrator, worker, and verifier gets you 14.5, 32.3, and 26.1. While this suggests the worker is the most easily replaceable, it also shows that SFT benefits the verifier most. After distilling, the readings are 19.7, 32.9, and 37.1. While SFT didn’t help the replacing worker (32.3 to 32.9), the verifier benefited massively, going from 26.1 to 37.1.
  2. While the paper doesn’t highlight it, the data above shows something practical. When cutting costs, replace the verifier role so the overall agent takes the smallest accuracy hit after SFT.
  3. An ablation shows that even if a model is distilled using vanilla trajectories from a larger model, plugging it into a 3-role-based harness improves performance. Training it on role-based trajectories improves it further. On MLS-Lite, adding a role-based harness on a 9B model distilled from vanilla trajectories improves performance from 12.1 to 13.6, and distilling it from role-based trajectories pushes it up to 15.6
  4. Ablations show that training a small model only on verifier traces is superior to training it on traces from all roles.
Critical weaknesses
  1. Role-based agents are a known technique in the academic and AI engineering community. For example, one of their references studies the accuracy-cost trade-offs of role-specific setups: “Specialize Roles, Mix Deployments: Pushing the Cost–Accuracy Frontier of LLM Agent Teams”

  2. The Literature Survey is anemic, missing many important related prior works. Examples:

Multi-Agent Role Playing:

MetaGPT, (ICLR 2024), which also has roles for software engineering. https://proceedings.iclr.cc/paper_files/paper/2024/hash/6507b115562bb0a305f1958ccc87355a-Abstract-Conference.html CAMEL, NeurIPS 2023 https://proceedings.neurips.cc/paper/2023/hash/a3621ee907def47c1b952ade25c67698-Abstract-Conference.html

ML Engineering Benchmarks:

MLE-Bench: https://proceedings.iclr.cc/paper_files/paper/2025/hash/7e3767db483c942b883eb4f8cfb74e31-Abstract-Conference.html RE Bench https://icml.cc/virtual/2025/poster/46519

Trajectory Distillation papers Agent-FLAN https://aclanthology.org/2024.findings-acl.557/

Usefulness of Verifiers: CTRL - Notes that critics are useful for even critiquing stronger generation models. https://proceedings.mlr.press/v267/xie25a.html

  1. The orchestrator decides when to invoke a role. So it’s unclear how often the strong orchestrator skipped calling the verifier and how often it reverified or corrected the verifier.

  2. The final training datasets contain 1,900 Orchestrator, 4,700 Worker, and 7,300 Verifier lifecycles. The difference in data volume confounds the claim that verifier benefits most from training. Figure 4 also shows that performance improvements have not saturated for the variants. While there may naturally be more verifier lifecycles, this can be controlled by a training-data-matched comparison, for example, verifier at X% of training data roughly matching the orchestrator’s 100%. Figure 4 already shows that the verifier at 25% of training data (1,825 examples) is close to the orchestrator at 100% (1,900 examples).

  3. The paper could be clearer. It was hard to understand. An example paragraph: “Both system-prompt variants in the audited historical Verifier corpus permit reading, analyzing and fixing issues, explicitly naming Edit. The later deployment prompt instead forbids editing and running tests. The native training traces contain no named Edit or Write call, but retain broad Bash/REPL activity and one BenchSubmit call; this is not evidence that every source invocation was read-only. Call/response counts are balanced, which certifies neither permission compliance nor diagnostic correctness. Thus reinsertion preserves the role position, not literal identity of historical prompts or tool schemas.”

  4. The ablations don’t run on all benchmarks they use in the main table. It would be useful to see whether the effects replicate across datasets.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Reply to Reviewer iLwp, Part 1/2

Dear Reviewer iLwp,

Thank you for the close reading and the related-work pointers.

Several of your observations in fact support our thesis. Most directly, the Verifier matching the Orchestrator on a quarter of the data is the data-flywheel advantage we claim.

W1/W2 — "Role-based agents are known; the survey is thin"

Thank you for these pointers — agreed on coverage, and we will add the citations. The story is also bigger than the decomposition: to the best of our knowledge, decomposition on AutoResearch remains under-explored, and what it unlocked is a series of findings prior setups were not designed to produce.

1. The decomposition is a pioneer on AutoResearch — prior decompositions do not transfer.

Existing structures were built for other settings: MetaGPT and CAMEL for software-style collaboration, AgentCARD for cost-aware deployment of existing models, ECHO for co-training a policy and critic with RL, MLE-Bench for ML engineering. For AutoResearch — long-horizon, environment-feedback-dependent, interleaving decision, execution, and analysis — decomposition remains under-explored. Its capability boundaries still had to be revealed, and that required a structure fitted to the loop.

2. The decomposition is the entry point, not the ceiling.

Building AutoResearch on role interfaces let us collect role-specific supervision at scale (2,212 tasks → 7,300 Verifier lifecycles) and produce findings with practical value:

  • the easiest role to replace is not the one to train (Worker +0.6 to +1.6 vs Verifier +7.5 to +13.2);
  • review-only supervision beats joint supervision (+8.8 / +5.9);
  • a trained small Verifier exceeds the all-strong system on fine-grained domains (RL 61.1 vs 60.1, CV 28.8 vs 20.2).

3. On citations.

We do cite AgentCARD ("Specialize Roles, Mix Deployments") and ECHO. For the rest, the omissions had reasons, though none are reasons to leave them out — we will add all of them, each with a one-line distinction like the above:

  • MetaGPT, CAMEL — role-based multi-agent collaboration for software-style tasks. Our setting is a single AutoResearch loop decomposed into capabilities, so we did not view them as direct comparisons; but they are the right context for role-based design, and we will cite them as such.
  • MLE-Bench, RE-Bench — ML-engineering benchmarks. Our ML evaluation runs on our own MLS suite, built precisely because we need a long-horizon, environment-feedback loop across research stages rather than competition-style ML engineering; since MLS already filled that slot in our experiments, these two slipped — an oversight, as they are the closest benchmark relatives, and we will cite them in the benchmark discussion.
  • Agent-FLAN — trajectory distillation for general agent ability. Orthogonal to our question (which role repays training), so it sat outside our related-work scope; we will cite it in the training-data discussion.
  • CTRL — the closest in spirit, showing critics can usefully critique stronger models. We anchored that discussion on ECHO and "Steer, Don't Solve" for the in-loop setting; CTRL belongs there too and will be added.

W3 — "How often does the Orchestrator skip/override the Verifier?"

Thank you for the question. The Verifier is called often and used selectively — and that frequency is itself why it is worth training and its data is cheap to collect.

On MLS-Lite, per task on average:

Orchestrator–Verifier interactionValue
Review requests4.1
Suggestions adopted as the next action48%
Overridden or re-checked with its own judgment52%

This is consistent with two results elsewhere in the paper: in the matched-continuation test, the Orchestrator follows a trained Verifier's review to double per-step progress but does worse with an untrained one's; and these frequent calls are exactly what make Verifier supervision abundant and cheap to collect (the same 2,212 tasks yield 7,300 Verifier lifecycles vs 1,900 Orchestrator).

AuthorsOpenReview ↗

Reply to Reviewer iLwp, Part 2/2

W4 — "Data volume confounds 'Verifier benefits most'"

Thank you for this careful reading — it in fact points at the heart of our claim. As you note, the Verifier at 25% of its data already matches the Orchestrator at 100%; that is exactly the data advantage we describe, and at equal compute the Verifier clearly leads.

ControlOrchestratorWorkerVerifier
Equal examples (~1,900), MLS-Lite+5.2−0.3+4.8
Equal training compute (~37.6M tok), MLS-Lite+5.2+0.5+10.6
Equal training compute, AutoLab+5.1+1.5+10.5
  • Equal examples: the roles look similar, but the Verifier gets there on under a third of the tokens.
  • Equal compute: the Verifier leads clearly on both benchmarks, curves unsaturated (Fig 4). Abundant review data is a property of the design, not a confound.

W5 — "The appendix permissions paragraph is hard to read"

Thank you — agreed; it will be rewritten as a two-column table.

  • Training data: the Verifier could edit and run. Deployed Verifier: read-only (inspect, diagnose, suggest); no Edit/Write, test execution, or submission. Reinsertion preserves the role position, not the historical permissions.

W6 — "Ablations not run on all four benchmarks"

Thank you for the suggestion — we have now run them. Verifier training gains +8 to +13 points on all four benchmarks, and the ablation benchmarks are themselves cross-domain.

BenchmarkUntrained VTrained VGain
MLS-Lite26.137.1+11.0
AutoLab51.164.3+13.2
PostTrainBench22.530.8+8.3
EdgeBench18.026.8+8.8
  • Every benchmark in the paper is a mixture of research domains, not a single-domain suite. MLS-Lite spans RL, optimization, time series, and vision; AutoLab spans LLM post-training, kernel optimization, and systems. The ablations therefore already cover many domains.
  • The main table adds PostTrainBench and EdgeBench precisely to make the setting more solid; they are recent and widely used, and their conclusions are consistent with the two role-training benchmarks.
  • Running every ablation on all four benchmarks with strong-model APIs throughout would be prohibitively expensive; we prioritized the two benchmarks where role training is the core question, and now extend the key ablation to all four.
  • Across both industry reports and the academic literature, the conclusions on these leaderboards are consistent, which is further evidence the finding is not benchmark-specific.
Reviewer_qKDbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose ResearchWeaver which breaks down the auto research task into 3 sub tasks - orchestrator (an agent that plans experiments), worker (an agent that conducts them) and verifier (an agent that checks the results and suggests improvements).

The authors compare a holistic agent that does all 3 sub tasks with a multi-agent system where each agent does one of the 3 sub-tasks on MLS-Bench-Lite, AutoLab, PostTrainBench, and EdgeBench. The goal is to show that it is possible to use a small model for the verifier while keeping the two other models bigger in order to save a considerable amount of compute and token use while retaining the performance of having each of them as a big model.

Critical strengths
  • the paper is well written and easy to follow.
  • I found the idea of breaking down an agent into several roles and identifying roles that would make the system more efficient if chosen as a small model to be very interesting.
  • the authors did an exhausive experimentation on multiple benchmarks MLS-Bench-Lite, AutoLab, PostTrainBench, and EdgeBench
  • The authors show that training a small model verifier can retain 92% of the performance of an all-strong model while cutting off 51% of the strong model tokens, making things a lot cheaper.
  • the ablation studies are exhaustive in that the authors isolate the impact of each role in the system.
Critical weaknesses
  • the idea of decomposing the auto research workflow into multiple roles such planner-worker-verifier (as shown as contribution 1) is not new, this structure was shown in existing works like AgentCARD and verifier-like feedback loop was shown in ECHO

  • The idea of mixing strong and weak models to retain strong model performance and have cheap costs is not uncommon as well, this was shown in existing methods in RouteLLM (ICLR 2025 - https://arxiv.org/abs/2406.18665), and Hybrid LLM (ICLR 2024 - https://arxiv.org/abs/2404.14618) where routing queries to the most efficient model that can resolve the query is the strategy. The finding in this work is not that interesting in that we know that some weaker models are sufficient for some roles like the verifier in this case.

  • just reporting strong model token reduction is not sufficient in identifying how efficient the resultant system is, there should be a report on wall-clock time and monetary savings.

Questions for authors

Please see Weaknesses.

1 comment
AuthorsOpenReview ↗

Reply to Reviewer qKDb

Dear Reviewer qKDb,

Thank you for recognizing the breadth of the experiments and the value of identifying small-model roles.

Strong/weak mixing to cut cost is indeed known, but ours is the AutoResearch-specific mapping of which capability repays training, which remains unexplored in prior work. The cost accounting is added below.

W1 — "The three-role decomposition is not new"

The decomposition is part of the contribution, not all of it. Our claim is the AutoResearch-specific question it opens — which capability repays training — a question AgentCARD and ECHO do not ask, and one that remains unexplored.

  • AgentCARD studies the deployment cost of heterogeneous roles; ECHO co-trains a policy and a critic with RL. We cite both.
  • Neither asks which single capability, trained offline, gives the largest system gain in AutoResearch.
  • The decomposition is also the entry point for efficient, scalable role-specific data collection — the same 2,212 tasks yield 7,300 Verifier lifecycles at no extra task cost. The value is there, not in the role names.

W2 — "Strong/weak mixing is not new; the finding is unsurprising"

For AutoResearch, "obvious" remains an assumption rather than an established result, and we think routing is the wrong comparison: routing assigns different tasks to different models, whereas we decompose a single task into capabilities and ask which of them repays training.

  • Decomposition on AutoResearch remains under-explored. The loop is strongly environment-feedback-dependent and long-horizon, with decision, execution, and analysis interleaved; measurements of each capability's boundary, bottleneck, and training return are still limited. Indeed, the reviews themselves show that whether a weak Verifier suffices is contested — exactly the question a systematic study settles.
  • Routing ≠ capability decomposition. RouteLLM and Hybrid LLM choose among existing models per query and cannot create missing capability. Our capability map (which role to serve small, where to spend expert supervision, at what cost) is what a vertical-domain AutoResearch team actually deploys on.

Direct evidence that the answer is not obvious — on 40 identical frozen states, we inject one review and take one step:

Injected reviewMean next-step gain
No review+3.1
Untrained small Verifier+1.4
Trained small Verifier+6.5
  • A trained Verifier doubles per-step progress; an untrained one is worse than no review, so "a weak verifier suffices" is an untested extrapolation — pooled across the benchmark, dropping the trained Verifier for an untrained one loses ~15 points. The Orchestrator's own behavior is consistent: it uses reviews selectively, adopting 48% of suggestions and re-checking 52%.

W3 — "Report wall-clock and money, not only tokens"

Strong-model cost drops by about half with quality retained, and wall-clock is unchanged because review sits off the critical path. Full accounting on MLS-Lite (30 tasks):

DeploymentGLM-5.2 tokensSmall-modelAPI costWall-clock / task
All GLM-5.2366.1M—$214≈2.1 h
Qwen Verifier-SFT194.0M16.8M$118≈2.2 h
Gemma Verifier-SFT164.4M35.4M$101≈2.0 h

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.