- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the use of LLM-based agents to automatically optimize asynchronous RL post-training configurations. Using the official reference configuration as the baseline, the authors set the objective as improving throughput and reducing per-run training GPU-hours without degrading model quality, while requiring the search to satisfy GPU-hour and wall-clock budgets. The agent reads the VeRL code, reference configurations, and experiment history, proposes hypotheses for configuration changes, chooses short-horizon PROXY or full FULL experiments, and continues searching based on feedback. After the search, it summarizes the experience into a natural-language optimization prior (lessons.md) to assist configuration optimization for other tasks.
The paper uses two agent systems, Claude Code + Claude Opus 5 and OpenCode + Qwen3.8-27B, and conducts experiments on three tasks: mathematical reasoning, multimodal geometric reasoning, and mathematical reasoning with code tool calls. All three tasks use 7B-scale Qwen2.5 series models, VeRL, and GRPO. Compared with the official reference configuration, the authors report that some settings reduce per-run training cost by 4.5%–49.5%, increase throughput by up to about 2x, and satisfy the non-degradation constraint on the reported quality point estimates. The cross-task prior shortens the time to first find a qualified configuration in the tested Claude settings, but its effect is inconsistent on Qwen.
Critical strengths
- The problem has practical value. Asynchronous RL post-training involves coupled factors such as rollout, policy training, resource allocation, and policy staleness. The paper takes configuration optimization in a real training pipeline as its research object and targets a problem with clear engineering needs.
- The evaluation objective is more complete than optimizing throughput alone. The authors simultaneously consider model quality, throughput, and training GPU-hours, avoiding equating throughput improvement directly with total training cost reduction. Placing the optimization process within explicit resource budgets is also more consistent with practical usage conditions.
- The experiments are based on real infrastructure and training processes. The work is not purely offline simulation; it includes actual configuration changes, training execution, failed explorations, and budget consumption. The paper provides the reference configuration, differences in final configurations, search trajectories, partial experience summaries, and token usage statistics for the closed-source agent, offering some engineering reference value.
- The paper provides preliminary observations on experience transfer and failure cases. The authors report not only successful configurations but also cases where Qwen failed to find qualified configurations under some settings, as well as inconsistent benefits from the prior. These results help understand the capability boundaries of off-the-shelf agents in high-cost optimization tasks.
Critical weaknesses
- The core workflow is highly similar to OPRO and AgentHPO, and the newly added configuration-item discovery, PROXY/FULL scheduling, and experience transfer lack independent validation. Being better than the official default configuration alone cannot prove that agentic search is superior to existing tuning methods; as empirical research, it also lacks controlled experiments supporting generalizable configuration principles.
- Each configuration was run only once, with no multi-seed results or confidence intervals; the difference between 18.0% and 18.1% in reasoning is insufficient to reliably demonstrate no quality degradation. Whether search evaluation and final testing are independent is also not clearly explained, raising the risk that the test set was repeatedly used for configuration selection.
- System parameters vary simultaneously with training recipe factors such as learning rate, training length, and KL. For example, Claude reasoning cost decreased by about 49.5%, while training sample size decreased by about 40%. There are no grouped ablations, so it is impossible to distinguish the contributions of asynchronous system optimization from regular training recipe adjustments.
- Although some search costs are disclosed, there is no complete accounting per setting of search, agent inference, and prior construction overhead, nor any explanation of how many reuses are needed to offset the optimization cost. Cheaper single training does not equal net end-to-end resource savings.
- Finding a qualified configuration earlier does not mean the final configuration is better; after Claude reasoning used the prior, the final training cost instead increased from 42.5 to 60.8 GPU-hours. There are also no consistent gains on Qwen, and there is a lack of controls such as human experience or direct YAML transfer, so the independent value of the natural-language experience summary cannot be determined.
- All three tasks are mathematical reasoning, use VeRL, GRPO, and 7B models, and two of them also share training and evaluation data. Execution mechanisms and configuration semantics may differ significantly across RL frameworks; even within the same framework, rollout length, tool interaction, and computational load of different tasks can change bottlenecks and effective configurations. Current evidence mainly supports optimization effects on the tested tasks and limited transfer between similar tasks, and has not proven that these experiences remain valid when framework or task conditions change. Therefore, the output is closer to tuning experience dependent on specific infrastructure and tasks, rather than validated general optimization principles. In addition, the two agent systems simultaneously change the model, agent framework, and context length, so the effects of these factors cannot be separated.
- Full instructions, context-management rules, and search logs are not provided, and the PROXY/FULL scheduling lacks clear details. The relationship between proxy metrics and full training results is not validated, so the reliability of the filtering design and its actual benefit cannot be judged.
- It is mainly a concrete application of the existing agent-based training optimization paradigm. OPRO has already established an LLM optimization paradigm based on historical evaluation feedback, and AgentHPO more directly studies agents proposing training configurations, executing experiments, and iteratively optimizing. This paper changes the optimization target to asynchronous RL but does not present substantive new search strategies, experiment scheduling mechanisms, or knowledge transfer methods addressing its unique difficulties. Quality and resource constraints, PROXY/FULL experiments, and natural-language experience summaries mainly reflect task adaptation of existing workflows, and their independent contributions are not validated. Therefore, the paper has not gone beyond the application level of “using off-the-shelf agents to optimize another class of training configurations”; gains over the default configuration are insufficient to prove methodological innovation, and the post-hoc summarized configuration experience lacks controlled validation, making it difficult to constitute a sufficient empirical research contribution.
Questions for authors
- How does agent-based tuning compare with conventional hyperparameter optimization or expert tuning under the same compute budget? Meanwhile, I want to investigate more on the 4.5% improvements, which is not very impressive.
- Are the quality and efficiency improvements consistent across multiple random seeds?
- How many training runs are needed to recover the computational cost of the optimization search?
- Which configuration changes contribute most to the gains, and how much comes from simply shortening training?
- Do the findings generalize to larger models, other RL algorithms, or different training infrastructures?