- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether keeping more research trajectories active helps an LLM research agent improve a model. Claude Opus 4.6 serves as the planner for autonomous post-training of a VLM2Vec-Qwen2-VL-7B embedding model, scored on a 12-task MMEB-V2 subset. A greedy single-trajectory policy (K=1) is compared with a top-K multi-trajectory policy (K=3); everything else is fixed: agent, memory, training setup, and evaluator. With three runs per policy over 22 rounds, both policies raise the score from 0.576 to about 0.70. K=3 plateaus sooner but not higher; its runs are more consistent with each other, and they explore more intervention families without finding more productive ones. A case study shows the same intervention (false-negative filtering) changing sign across parent states. The authors conclude that the bottleneck for wider search is exploiting the preserved states, not generating them.
Critical strengths
Clean controlled comparison: Only trajectory width varies; the agent, persistent memory, environment, and evaluator are held fixed. This isolates the search-control question better than most autonomous-research comparisons.
Useful process analysis: Distinguishing attempted breadth (coverage, entropy) from productive breadth is a helpful lens. It shows that wider search really does explore more, rather than just repeating the greedy policy.
Transparent reporting: Per-run numbers, discovered recipes, artifact construction, and explicit caveats about causal interpretation make the study auditable.
Critical weaknesses
The null result is underpowered: With three runs per policy, the mean difference (0.7054 vs 0.7016) is far smaller than the K=1 standard deviation (0.018). The study cannot distinguish “wider search doesn’t help” from a modest effect in either direction, and its headline “best K=3 below median K=1” rests on a single run.
Selection and reporting use the same benchmark: Every candidate, soup, and ensemble is selected on the same 12 tasks that produce the reported scores, and no held-out tasks are used. The 22% gain may therefore partly reflect selection on the benchmark, and K=3 performs three times more selections, which confounds the comparison.
Artifacts and single models are mixed in the comparison: The top K=1 run (0.722) is a 4-member soup and one K=3 run is a 5-model embedding ensemble. On single models alone the policies are 0.7015 vs 0.6965, which changes the framing, and the ensembles have different inference costs.
Weak support for the state-dependence claim: The evidence is one intervention with deltas of 0.002–0.006 from single runs. The “<0.001 seed noise” baseline comes from an unspecified number of seeds of one configuration, so it may not hold across states.
Narrow scope and limited novelty: The study covers one model, one benchmark, one planner, K∈{1,3}, and 22 rounds, with no compute-matched comparison (e.g., K=1 for 66 rounds). Toledo et al. (2025) already compared greedy and wider search strategies, so the generality and novelty of the conclusion are limited.
Questions for authors
Can you report how the final models from both policies perform on held-out MMEB-V2 tasks that were never used for selection?
How does the comparison look when restricted to single trained models, and with more runs per policy, or at least bootstrap confidence intervals?
How was the “<0.001 seed noise” estimated (how many seeds, which configurations), and does it hold for parent states other than the starting configuration?
Who assigned interventions to taxonomy families (authors or an LLM), and was this done blind to policy and outcome?