- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents SPOT, a framework in which an LLM-based researcher repeatedly proposes a joint post-training recipe. Each proposal is validated, executed, and evaluated through a shared pipeline, and the outcomes and new checkpoint are returned to the researcher's history for the next round. The authors instantiate SPOT on tool-use post-training of Qwen3-8B with LoRA, using four LLM researchers (GLM-5.3, Qwen3.8-27B, Kimi-K2.6, and Nemotron-3-Ultra) plus TPE and random search as classical references. The best search-time composite score rises from 0.4092 to 0.5152. They further run interface ablations (one-component proposals and no auxiliary annotations) and use the recorded training trajectories to propose a three-question diagnostic hierarchy.
Critical strengths
- The paper studies a timely and important question, namely, automated LLM post-training with LLM-based researchers (a form of AutoML/auto-research).
Critical weaknesses
-
Limited novelty and unclear positioning. The core loop (i.e., LLMs proposes a configuration, observes the outcome, and revises the next proposal) already exists in LLM-guided hyperparameter optimization and autonomous post-training systems [1-5]. SPOT mainly enlarges the action space and adds detailed logging, which is an engineering improvement rather than methodological innovation. The diagnostic hierarchy, which the paper presents as
more important, consists of retrospective case analyses of known phenomena (zero within-group reward variance and reward over-optimization [6,7]) and is not validated prospectively. -
The central
statefulclaim is not supported by the evidence. 24 of the 34 parent-relative updates regress, three of the four best LLM-found checkpoints (including the overall best, 0.5152) were trained from the base model, and no controlled ablation isolates the effect of checkpoint inheritance. In addition, each campaign is run only once on one student model, and the selection utility is computed on the same benchmarks used to report results, without a held-out test. The headline improvement can therefore neither be attributed to stateful search nor shown to be statistically reliable.
[1] Large Language Models to Enhance Bayesian Optimization. ICLR, 2024.
[2] AgentHPO: Large Language Model Agent for Hyper-Parameter Optimization. CPAL, 2025.
[3] Lamdagent: An autonomous framework for post-training pipeline optimization via llm agents. EMNLP, 2025.
[4] PostTrainBench: Can LLM Agents Automate LLM Post-Training? ICML, 2026.
[5] TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Exploration. arXiv, 2026.
[6] DAPO: An Open-Source LLM Reinforcement Learning System at Scale. NeurIPS, 2026.
[7] Scaling Laws for Reward Model Overoptimization. ICML, 2023.
Questions for authors
- What is methodologically new in SPOT compared with LLAMBO, AgentHPO, LaMDAgent, PostTrainBench, and TREX, beyond a larger action space? Have the authors compared SPOT with any of them under the same budget?
- Under the same budget, how does the Full interface perform when checkpoint inheritance is disabled? Among the top-k candidates, how many come from inherited runs and how many from restarted runs?
- How do the selected checkpoints perform on a held-out benchmark that was never used for selection, and how large is the run-to-run variance of a campaign?
- Have the code, proposer prompts, ledgers, and evaluation scripts referred to in
data/appendix_20260925/be released through an anonymous repository? If not, why are these paths placed in the paper? - Please explain or make the candidate counts consistent in the main text. The caption of Figure 4 mentions 97 LLM candidates, but the data in Figure 4(c) add up to 99. Additionally, Section 3.1 reports a total of 141 fully evaluated candidates, while Appendix F lists 150.
- Suggestion: Extend the study to other student model families and sizes, full-parameter training, and other post-training domains (e.g., math reasoning).