- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces HeuriTrain, a framework for autonomous exploration of LLM post-training configurations. An AI agent proposes changes to training data and recipes, while a harness validates the changes, executes training and evaluation, and records results under a fixed task contract and budget. The paper compares Global Greedy, Global Archive, and Factorized Experimental State (FES), which separates data and recipe exploration. Experiments on IFEval, Countdown, and a deterministic Splay task examine search performance, interactions between edits, and the reliability of results under re-evaluation and re-execution.
Critical strengths
- The framework records proposed edits, executed changes, evaluation results, and computational costs, providing useful traceability for inspecting and comparing agent-driven training experiments.
Critical weaknesses
- The framework largely follows the standard process of proposing data or hyperparameter changes, running training experiments, and using evaluation feedback to guide further trials. This is a common workflow for human researchers, and this paper only validates that modern agents can do the same thing. The paper claims they find that data and recipes should not be seperated. This is a common sense for doing research, not a finding or contribution of this paper.
- The central FES policy receives only a brief definition in the main text, and its contribution is not clearly separated from that of the HeuriTrain harness. FES changes parent eligibility, visible history, edit permissions, and opportunity allocation together, making it unclear which design choice drives the reported gains.
Questions for authors
- What specific limitation of AI auto-research does HeuriTrain address beyond the standard experiment–feedback loop, and what evidence demonstrates that its proposed components are necessary to address this limitation?
- Can you provide ablations that independently vary parent selection, visible history, edit scope, and opportunity allocation, so that the contribution of factorized experimental state can be distinguished from the effects of restricting the search space?
2 comments
Thanks for the response from the authors. I'm still concern about the position and contribution of this paper. The data exploration, recipe exploration, config inheriting, are all common practice for human researcher. The core of this paper seems to just define some contracts to ensure the agents follow a common research workflow. The model improvement brought by such workflow is expected. In my view, this paper seems to not identify a specific critical problem about AI auto-research and solve it, or bring some new insights to this field. It is more like an engineering practice. Besides, there's a lof of room for improving the writing of this paper. For example, the paper names their framework as HeuriTrain, while in the experiment, FES is refered as their method, which is only described in a short paragraph (line 187-192) and confusing.
Response to Reviewer yQX7
Thank you for the review, for recognizing the value of the experimental records, and for the specific request to separate the effects of FES's design choices. We address the contribution, the interpretation of composition, and the additional ablations below.
W1 and Q1 — Research question and contribution. We agree that the experiment–feedback loop is a standard research workflow. The paper studies choices within that loop: which measured configuration the agent inherits, which earlier results it can consult, what it may change, and how its remaining opportunities are allocated. HeuriTrain makes these choices explicit so that different search organizations can be compared under a common task contract.
The experimental records connect those choices to actual interventions and outcomes. This enables analyses of inherited versus consulted evidence, permitted versus executed edits, interactions between separately discovered changes, and the persistence of discovery results under verification. The additional ablation examines which parts of the tested policy account for its observed behavior; it does not establish that every component is necessary.
W1 — Separate exploration and composition. We would like to clarify a distinction central to the paper. FES separates Data and Recipe exploration by construction (Sec. 2.3). The composition experiments ask whether changes that were beneficial separately remain beneficial when combined (Secs. 3.3–3.4).
The executed IFEval crossing has a −26-prompt interaction contrast. In the deterministic Splay composition study, 10 of 12 combinations score below the stronger constituent, and 4 score below the root. These measurements show that individually favorable changes need not compose favorably in the tested configurations. They do not imply that Data and Recipe exploration should never be separated.
W2 and Q2 — Attribution to individual design choices. Thank you for this suggestion. We evaluated three intermediate policies alongside Greedy and FES on the same 18 Splay blocks with six proposals per search. Greedy + scope introduces alternating single-surface edits while retaining Greedy's parent rule and history. Adding the FES parent rule or history separately, and then jointly in full FES, gives a 2×2 comparison among the four constrained variants (global response, item 2).
All four constrained variants have higher observed means than Greedy; Greedy + scope has the highest mean (675.5 rotations of gain, versus 467.1 for Greedy and 569.0 for FES). The individual confidence intervals against Greedy include zero, and the results do not establish an additional benefit from separate parent pools or histories in this six-proposal setting. Opportunity allocation remains coupled to the alternation schedule, so the available ablation does not independently identify all four factors.
For clarity, FES maintains two surface-specific states with their own eligible parents and visible histories. The fixed alternation schedule determines the active edit surface, while the inactive surface remains at its root configuration. The harness enforces the task contract; FES specifies how the search context and edit permissions are organized within that contract.
We hope these clarifications and additional comparisons help make the empirical contribution and its scope easier to assess. We appreciate your consideration of this evidence.