acceptodds
Withdrawn from ICLR 2027

Agentic Optimization for Asynchronous RL Post-Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning (RL) post-training is computationally expensive, motivating asynchronous training for improved efficiency. Asynchronous RL, however, exposes a large, complex, and coupled configuration space across training, rollout, and resource orchestration. If configured improperly, asynchronous training will lose the efficiency advantages while introducing policy staleness. In this paper, we formulate a joint objective over model quality and training efficiency for asynchronous RL training and task LLM-based agents with iteratively optimizing it through hypothesis formulation, experimentation and evaluation. We evaluate both open- and closed-source agents across reasoning, multimodal and agentic post-training tasks using VeRL, a widely adopted open-source RL infrastructure. Our agents improve model quality and training throughput over human reference configurations while reducing per-task training GPU-hours by up to 50%, and distill transferable insights that accelerate optimization on a new task. Analysis of optimization trajectories provides insights into configuring asynchronous RL training and suggests directions for efficient RL post-training.

Primary area:
reinforcement learning
Keywords:
Asynchronous Reinforcement Learning, RL Post-Training, Agentic Optimization, Hyperparameter Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4szpOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the use of LLM-based agents to automatically optimize asynchronous RL post-training configurations. Using the official reference configuration as the baseline, the authors set the objective as improving throughput and reducing per-run training GPU-hours without degrading model quality, while requiring the search to satisfy GPU-hour and wall-clock budgets. The agent reads the VeRL code, reference configurations, and experiment history, proposes hypotheses for configuration changes, chooses short-horizon PROXY or full FULL experiments, and continues searching based on feedback. After the search, it summarizes the experience into a natural-language optimization prior (lessons.md) to assist configuration optimization for other tasks.

The paper uses two agent systems, Claude Code + Claude Opus 5 and OpenCode + Qwen3.8-27B, and conducts experiments on three tasks: mathematical reasoning, multimodal geometric reasoning, and mathematical reasoning with code tool calls. All three tasks use 7B-scale Qwen2.5 series models, VeRL, and GRPO. Compared with the official reference configuration, the authors report that some settings reduce per-run training cost by 4.5%–49.5%, increase throughput by up to about 2x, and satisfy the non-degradation constraint on the reported quality point estimates. The cross-task prior shortens the time to first find a qualified configuration in the tested Claude settings, but its effect is inconsistent on Qwen.

Critical strengths
  1. The problem has practical value. Asynchronous RL post-training involves coupled factors such as rollout, policy training, resource allocation, and policy staleness. The paper takes configuration optimization in a real training pipeline as its research object and targets a problem with clear engineering needs.
  2. The evaluation objective is more complete than optimizing throughput alone. The authors simultaneously consider model quality, throughput, and training GPU-hours, avoiding equating throughput improvement directly with total training cost reduction. Placing the optimization process within explicit resource budgets is also more consistent with practical usage conditions.
  3. The experiments are based on real infrastructure and training processes. The work is not purely offline simulation; it includes actual configuration changes, training execution, failed explorations, and budget consumption. The paper provides the reference configuration, differences in final configurations, search trajectories, partial experience summaries, and token usage statistics for the closed-source agent, offering some engineering reference value.
  4. The paper provides preliminary observations on experience transfer and failure cases. The authors report not only successful configurations but also cases where Qwen failed to find qualified configurations under some settings, as well as inconsistent benefits from the prior. These results help understand the capability boundaries of off-the-shelf agents in high-cost optimization tasks.
Critical weaknesses
  1. The core workflow is highly similar to OPRO and AgentHPO, and the newly added configuration-item discovery, PROXY/FULL scheduling, and experience transfer lack independent validation. Being better than the official default configuration alone cannot prove that agentic search is superior to existing tuning methods; as empirical research, it also lacks controlled experiments supporting generalizable configuration principles.
  2. Each configuration was run only once, with no multi-seed results or confidence intervals; the difference between 18.0% and 18.1% in reasoning is insufficient to reliably demonstrate no quality degradation. Whether search evaluation and final testing are independent is also not clearly explained, raising the risk that the test set was repeatedly used for configuration selection.
  3. System parameters vary simultaneously with training recipe factors such as learning rate, training length, and KL. For example, Claude reasoning cost decreased by about 49.5%, while training sample size decreased by about 40%. There are no grouped ablations, so it is impossible to distinguish the contributions of asynchronous system optimization from regular training recipe adjustments.
  4. Although some search costs are disclosed, there is no complete accounting per setting of search, agent inference, and prior construction overhead, nor any explanation of how many reuses are needed to offset the optimization cost. Cheaper single training does not equal net end-to-end resource savings.
  5. Finding a qualified configuration earlier does not mean the final configuration is better; after Claude reasoning used the prior, the final training cost instead increased from 42.5 to 60.8 GPU-hours. There are also no consistent gains on Qwen, and there is a lack of controls such as human experience or direct YAML transfer, so the independent value of the natural-language experience summary cannot be determined.
  6. All three tasks are mathematical reasoning, use VeRL, GRPO, and 7B models, and two of them also share training and evaluation data. Execution mechanisms and configuration semantics may differ significantly across RL frameworks; even within the same framework, rollout length, tool interaction, and computational load of different tasks can change bottlenecks and effective configurations. Current evidence mainly supports optimization effects on the tested tasks and limited transfer between similar tasks, and has not proven that these experiences remain valid when framework or task conditions change. Therefore, the output is closer to tuning experience dependent on specific infrastructure and tasks, rather than validated general optimization principles. In addition, the two agent systems simultaneously change the model, agent framework, and context length, so the effects of these factors cannot be separated.
  7. Full instructions, context-management rules, and search logs are not provided, and the PROXY/FULL scheduling lacks clear details. The relationship between proxy metrics and full training results is not validated, so the reliability of the filtering design and its actual benefit cannot be judged.
  8. It is mainly a concrete application of the existing agent-based training optimization paradigm. OPRO has already established an LLM optimization paradigm based on historical evaluation feedback, and AgentHPO more directly studies agents proposing training configurations, executing experiments, and iteratively optimizing. This paper changes the optimization target to asynchronous RL but does not present substantive new search strategies, experiment scheduling mechanisms, or knowledge transfer methods addressing its unique difficulties. Quality and resource constraints, PROXY/FULL experiments, and natural-language experience summaries mainly reflect task adaptation of existing workflows, and their independent contributions are not validated. Therefore, the paper has not gone beyond the application level of “using off-the-shelf agents to optimize another class of training configurations”; gains over the default configuration are insufficient to prove methodological innovation, and the post-hoc summarized configuration experience lacks controlled validation, making it difficult to constitute a sufficient empirical research contribution.
Questions for authors
  1. How does agent-based tuning compare with conventional hyperparameter optimization or expert tuning under the same compute budget? Meanwhile, I want to investigate more on the 4.5% improvements, which is not very impressive.
  2. Are the quality and efficiency improvements consistent across multiple random seeds?
  3. How many training runs are needed to recover the computational cost of the optimization search?
  4. Which configuration changes contribute most to the gains, and how much comes from simply shortening training?
  5. Do the findings generalize to larger models, other RL algorithms, or different training infrastructures?
Reviewer_8GEjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether LLM agents can make asynchronous RL post-training cheaper while maintaining model quality. The agents inspect VeRL, suggest configuration changes, and run short efficiency tests and full training experiments. They also summarize lessons for use on another task. Tests with Claude and Qwen agents cover text reasoning, multimodal reasoning, and reasoning with code tools. The resulting configurations reduce training GPU-hours by 4.5–49.5% compared with the released recipes.

Critical strengths
  1. Claude cuts reasoning cost from 84.1 to 42.5 GPU-hours, with reported quality going from 18.0% to 18.1%. Qwen lowers multimodal cost from 45.2 to 31.6 GPU-hours. Its throughput rises from 1.13K to 2.27K tokens/GPU-s. These savings are useful in practice.

  2. With lessons from another task, Claude finds its first qualifying reasoning configuration after 127.8 GPU-hours. Without them, it needs 394.8 GPU-hours. This is promising.

Critical weaknesses
  1. Each configuration has one training run, but some quality margins are small: 18.1% versus 18.0% and 27.3% versus 27.2%. Independent reruns would help check whether quality is preserved.

  2. Table 2 changes learning rate, training length, and system settings at the same time. The largest reasoning cost reduction also cuts training samples from 819.2K to 491.5K. Appendix 12 gives useful observations on individual changes. But it is still unclear how much each change contributes to the total savings in the final configurations.

Questions for authors
  1. Which quality evaluations can the agent see during optimization? Which are held back for the final report? Are independent reruns available for the reference and selected configurations, especially those with very small quality margins?

  2. Can the recorded experiments separate savings from learning rate and training length changes from those due to system and asynchronous-execution settings? A comparison on one representative task would strongly affect how I judge the paper’s insights.

Reviewer_PtUtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether LLM-based agents can automatically optimize the configurations of asynchronous RL. The authors formulate a constrained optimization objective involving model quality, training throughput, and GPU-hour consumption. They employ LLM agents to inspect the VeRL infrastructure, propose configuration changes, conduct proxy or full training experiments, and iteratively refine configurations based on observed results. The paper also investigates cross-task knowledge transfer by summarizing optimization experiences into reusable textual priors.

Critical strengths
  1. This is a practical and relevant optimization problem.

  2. Empirical improvements on real training workloads.

  3. Useful engineering observations.

Critical weaknesses
  1. Methodological novelty compared with existing agent-based optimization methods is limited. The core procedure in Algorithm 1 follows a standard iterative optimization loop: an LLM proposes configuration changes, executes experiments, observes performance, and uses previous results to guide subsequent trials. Similar ideas have already been explored in AgentHPO (2025), which performs iterative LLM-driven hyper-parameter optimization, and in more recent systems such as ASAP (2025) and OptiCo (2026), which use LLM agents to optimize distributed training configurations. Moreover, JF-HPO (ACL 2026) specifically investigates efficient hyper-parameter optimization for LLM RL, while PROMPTS (2026) considers agent-based performance optimization across training, serving, and post-training workloads. The paper's main differentiation appears to be the specific asynchronous RL setting and the joint consideration of training quality and efficiency. However, the proposed objective in Eq. (1) is essentially a constrained feasibility criterion followed by training-cost minimization, rather than a new optimization algorithm. The optimization-prior mechanism is also a simple textual summary of previous experiments.

  2. Missing comparisons with established optimization baselines. Despite positioning the approach as a solution to a complex configuration optimization problem, the experiments compare agent-discovered configurations almost exclusively against the released reference configurations. This is a significant limitation. Demonstrating that an LLM agent can improve a default configuration does not establish that agent-based optimization is necessary or particularly effective. The reported improvements might also be obtained through conventional HPO, random search, or straightforward engineering heuristics.

  3. The reported training-cost reductions are confounded by changes to the training recipe. The headline efficiency improvements require more careful interpretation. For example, Table 2 shows that, on the reasoning task, Claude reduces training cost from 84.1 to 42.5 GPU-hours. However, the total number of training samples also decreases from 819.2K to 491.5K, a reduction of approximately 40%. Therefore, the reported 49.5% cost reduction cannot be attributed solely to improved system efficiency or asynchronous execution.

  4. The experiments do not adequately isolate asynchronous RL-specific optimization. Although the paper emphasizes asynchronous RL, many of the successful changes are generic optimizations applicable to synchronous training as well. The paper does not provide an ablation separating asynchronous execution parameters from general training hyper-parameters, nor does it evaluate whether similar improvements would arise when optimizing synchronous RL.

  5. Insufficient statistical validation and potential evaluation-selection bias. The paper explicitly states that each reported configuration is evaluated in a single training run. This is concerning because RL post-training outcomes can exhibit substantial variability across random seeds and training trajectories. Some reported quality differences are very small. For example, the reasoning baseline achieves 18.0% accuracy, while Claude's optimized configuration achieves 18.1%. Without repeated training runs or confidence intervals, it is unclear whether this difference is statistically meaningful.

Questions for authors

See weakness above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.