acceptodds
Under review as a conference paper at ICLR 2027

MART: Multi-Agent Hyperparameter Tuning for LLM Reinforcement Learning

Abstract

LLM reinforcement learning (RL) is computationally intensive, and well-chosen hyperparameters can improve both training efficiency and model performance. However, parameter tuning is highly dependent on expert knowledge and repeated trial-and-error experiments, incurring considerable human and computational overhead. Recent agent-based approaches have demonstrated the potential of automating system configuration optimization for LLM training and serving. However, extending such approaches to RL requires accounting for its distinct execution and learning dynamics: rollout and policy-update phases impose different memory and compute demands, while response lengths evolve during training, changing the workload and complicating configuration evaluation. Moreover, effective tuning must consider both execution efficiency and the effects of learning hyperparameters on training stability and task performance. To address these challenges, we propose MART, a multi-agent framework with human-in-the-loop review for hyperparameter tuning in RL post-training. Guided by phase-wise resource measurements and training feedback, specialized agents propose and assess candidate configurations. A human reviewer approves each selected candidate before execution or rejects it with feedback for subsequent revision. The search proceeds in two stages: short-horizon trials identify system configurations that improve end-to-end throughput while retaining a prescribed GPU memory margin; longer-horizon trials then optimize learning hyperparameters with the system configuration fixed. During learning tuning, online training-health assessment supports early termination of trials exhibiting persistent anomalies. Experiments on GRPO-based mathematical reasoning show that, relative to baseline configurations, MART improves throughput averaged over updates 3–5 by 13.5% and 7.7% for Qwen3-8B, and by 69.5% and 91.3% for Qwen3-0.6B-Base, under single-/dual-node deployments, respectively. Across AMC23, OlympiadBench, AIME24, and AIME25, MART improves mean pass@1 by 1.98–2.51 percentage points for Qwen3-8B and 0.74–0.81 percentage points for Qwen3-0.6B-Base.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Reinforcement Learning, Hyperparameter Tuning, Multi-Agent Systems, Large Language Models, ML Systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_pgVxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MART, a multi-agent framework designed to automate the complex hyperparameter tuning process for LLM reinforcement learning (RL) by addressing the distinct memory and compute demands of evolving rollout and policy-update phases. Utilizing a human-in-the-loop review system, specialized agents optimize system configurations for throughput in short-horizon trials, followed by longer-horizon trials that fine-tune learning hyperparameters.

Critical strengths
  • The paper addresses the uniquely challenging and novel problem of automating hyperparameter tuning specifically for LLM reinforcement learning, effectively managing the dynamic workloads caused by alternating rollout and policy-update phases.

  • The MART framework demonstrates substantial empirical superiority.

Critical weaknesses
  • Whether the framework can be extended to long horizon agentic tasks like webshop and alfworld is unknown, despite these being a current research focus for RL post training.

  • A human reviewer is needed, and results fluctuate significantly once the human review is removed (e.g., a significant drop in performance for AIME 25 but less of a drop for AIME 24), which may signify that the current experiments are insufficient or heavily reliant on the human reviewer.

  • The MAS framework of proposing and validating/refining is a very common collaboration topology and is not scientifically novel, which significantly hurts the overall novelty of the work.

Questions for authors
  • Why are all the experiments conducted on math problems when the framework does not appear specific to math settings, and can the work be extended to other tasks, especially long horizon agentic tasks like webshop and alfworld?

  • How are the human reviewers selected, what are their credentials and experience with these math problems, how many reviewers were involved, and what exactly do they see?

  • Can the authors include more RL algorithms, as well as different model architectures and sizes?

  • Can the authors include more advanced hyperparameter baselines?

  • How does the proposed framework compare with human experts?

3 comments

Dear Reviewer, we are grateful for your thorough review and the excellent questions posed. Your queries touch upon crucial deployment and scalability concerns of our framework. We have carefully considered each point and provide our detailed responses below.

Q1: Applicability beyond mathematical reasoning.

We focus on mathematical reasoning because rule-based rewards allow us to evaluate MART without introducing an additional reward model. However, MART is not restricted to this setting and supports broader model and algorithm configurations within verl. To demonstrate this, we additionally evaluate MART/no-HITL on Llama-3.1-8B-Instruct using PPO, which introduces a critic model compared with GRPO. The experimental results are presented below. Extending the evaluation to long-horizon agentic tasks such as WebShop and ALFWorld remains future work.

Stage 1 — hardware tuning

Parameter changethroughput (token/s/gpu)Reason
1Baseline559.4Reference point for the throughput search
2rollout.max_num_seqs 256→1024580.3The cap was binding: running requests pinned at 256 while KV usage was only 47.6% with zero preemptions, so the admission limit is raised to the measured queue-demand p95 (1024)
3actor.megatron.pipeline_model_parallel_size2→1 actor.ppo_max_token_len_per_gpu 16384→8192590.5— rejected: rollout phase reached 61691 MiBPP=2 leaves only DP=2 and adds stage bubbles, so PP=1 raises actor DP to 4. token cap: PP=1 doubles per-GPU weights/optimizer state, so the training token budget must shrink to stay in memory
4rollout.gpu_memory_utilization 0.6→0.55582.3The rollout phase exceeded the throughput memory reserve; 0.55 is the smallest value whose estimate (55137 MiB) clears the limit
5rollout.log_prob_max_token_len_per_gpu 16384→32768 ref.log_prob_max_token_len_per_gpu 16384→65536587.0actor log-prob: halve the number of micro-batches in a phase that must score the full batch each update ref log-prob: same overhead removal in the phase with the largest memory slack
6critic.megatron.tensor_model_parallel_size 8→4 critic.ppo_max_token_len_per_gpu 16384→8192597.6Decrease the TP of the critic such that its architecture is consistent with the actor.

Stage 2 — stability tuning

Parameter changeReasonResult
7Baseline (hardware frozen)Reference trajectory with the default stability knobsstopped at 28/100: entropy 7.09→0.91, ppo_kl→0.85, pg_clipfrac→0.37; reward 0.569, MATH 0.4922
8actor.entropy_coeff 0→0.005Entropy had no restoring term in the loss and collapsed monotonically toward a deterministic policystopped at 41/100: entropy 7.65→1.77, ppo_kl→1.27, pg_clipfrac→0.43; reward 0.569, MATH 0.4872
9actor.optim.lr 1e-6→5e-7The ppo_kl>1 excursions and pg_clipfrac saturation scale with the per-update policy move, which is set by the learning ratecompleted 100/100: reward 0.574, MATH 0.4866 (peak 0.5016 at step 60); entropy still decayed 7.3→0.40
10actor.kl_loss_coef 0.001→0.01The KL penalty contributed only ~0.007 against a pg_loss of ~0.14, i.e. the reference anchor was numerically inertinterrupted at 34/100: entropy 6.89→1.28, pg_clipfrac→0.38; reward 0.570, MATH 0.4918

Q2:Human reviewer background and review process.

One human reviewer with substantial experience in model training and experimental evaluation participated in our experiments. The review concerns training configurations rather than mathematical solutions. The reviewer sees the Proposal Agent’s suggested changes, the Feasibility Agent’s assessment, and relevant experimental records and metrics. MART’s HITL process can therefore be viewed as an interaction between a user and an LLM, in which the user reviews proposed adjustments and provides approval or feedback.

Q3 Evaluation across RL algorithms and models.

The additional results presented above demonstrate MART’s tuning of Llama-3.1-8B-Instruct using PPO, extending our evaluation to another model family and an additional RL algorithm.

Q4&Q5 Comparison with advanced tuning baselines and human experts.

MART can be viewed as human–LLM collaborative tuning. One human reviewer with substantial experience in model training and experimental evaluation participated in our experiments, reviewing the Proposal Agent’s suggested changes, the Feasibility Agent’s assessment, and relevant experimental records and metrics. The reviewer provides approval or feedback, while MART/no-HITL performs tuning without human intervention.

Reviewer_22cuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

MART is a multi-agent framework for tuning hyperparameters in LLM reinforcement learning. It first uses short trials to optimize system efficiency, then uses longer trials to tune learning parameters, combining runtime measurements, historical experience, training-health monitoring, and optional human review.

Critical strengths
  1. Separating the system and learning tuning matches their different evaluation needs and avoids using long training runs to evaluate system configurations.
  2. Structured experiment records and experience summaries help guide future decisions, with measurable benefits in the ablation study.
  3. MART improves average test accuracy and early-step throughput across all four main settings, showing practical potential.
Critical weaknesses
  1. The experiments mainly compare against existing configurations, so it remains unclear whether MART outperforms established tuning methods under the same search budget.
  2. The experiments do not directly show whether multiple agents outperform a single agent or whether two-stage tuning outperforms joint search.
  3. The method looks more like a heuristic hyperparameter tuning with a wrapping of LLM without too many intuitions and explanations.
Questions for authors
  1. The method fixes system parameters before tuning learning parameters, but learning changes can alter response lengths and memory demands. How does MART handle cases where the selected system configuration becomes unsuitable later?
  2. Does the method need specific tuning when using for different base models?
2 comments

We thank the reviewer for the constructive comments and for recognizing the practical potential of MART. We appreciate the concerns raised and will address each point carefully in our response.

Q1: System configuration stability during learning tuning.

Thank you for raising this concern. We agree that learning-parameter changes can affect response lengths and memory requirements. MART addresses this risk proactively through fixed execution constraints and explicit GPU-memory headroom during system tuning.

In the Qwen3-0.6B-Base experiment, the increase in total tokens mainly comes from increasing rollout.n from 5 to 8, which increases the workload and training time without causing OOM in an individual training micro-batch. Peak training memory is primarily determined by micro-batch size and response length. The former is fixed after system tuning, while the latter is bounded by the fixed max_response_length, beyond which responses are truncated. Our logs also show that mean response lengths generally decrease: from approximately 2,000 to 1,500 tokens for the 8B model and from 850 to 600 for the 0.6B model.

Moreover, MART records the maximum memory usage of each execution phase, which generally captures memory demand at the maximum response length, and reserves additional memory headroom for potential growth. These safeguards keep the selected system configuration suitable during subsequent learning tuning.

Q2: Does MART require model-specific tuning

MART primarily tunes configurations within the verl framework and therefore supports the models and algorithms supported by verl, without requiring model-specific adaptation of the agent. To illustrate this, we provide the complete tuning process of a MART/no-HITL run on Llama-3.1-8B-Instruct using GRPO.

Stage 1 — System tuning

Parameter changethroughput (token/s/gpu)Reason
1Baseline791.9Locate the throughput bottleneck (rollout concurrency vs. actor topology)
2actor.megatron.tensor_model_parallel_size 4→2; ppo/ref/rollout max_token_len_per_gpu 16384→8192840.18B model was over-sharded across 4 GPUs; smaller token caps offset the doubled per-GPU weights
3rollout.max_num_seqs 256→1024; rollout.gpu_memory_utilization 0.6→0.75878.6Rollout was admission-limited (running pinned at the cap, requests waiting); larger KV pool to serve the larger in-flight set
4rollout/ref log_prob_max_token_len_per_gpu 8192→16384902.2Log-prob phases ran only ~2 sequences per micro-batch, dominated by launch/collective overhead
5rollout/ref log_prob_max_token_len_per_gpu 16384→32768896.0Test whether the same lever still yields gains (it saturated)
6virtual_pipeline_model_parallel_size None→2895.8Reduce the PP=2 pipeline bubble; reverted the log-prob caps to stay within memory (failed with memory-headroom exceeded)

Stage 2 — Learning tuning

Parameter changeReasonResult
7Baseline (hardware frozen)Establish the stability reference on the best hardware configurationcompleted 100/100; reward 0.625, MATH 0.5006 — but entropy 6.52→0.003,
8actor.entropy_coeff 0→0.005Counter the unopposed entropy collapse toward a near-deterministic policystopped at 62/100 (entropy 1.95→0.126 in 6 updates, MATH 0.4918); reward 0.493
9actor.kl_loss_coef 0.001→0.005KL penalty (≈0.01) was negligible against pg_loss (≈0.036) and could not restrain drift from the referencestopped at 87/100 final-window reward 0.546, after sliding from 0.593 to 0.539 over steps 60–86
10actor.optim.lr 1e-6→5e-7; actor.kl_loss_coef back to 0.001Alternative hypothesis: the pathology is excessive per-step policy displacement rather than a missing regularizercompleted 100/100; reward 0.589, MATH 0.4952,

Comments on the paper

2 comments

Dear Reviewer, thank you very much for your valuable time and the insightful questions. We sincerely appreciate your feedback, which has helped us clarify several important practical aspects of our work. Below, we provide detailed, point-by-point responses to your questions.

Q1: Human effort and performance trade-offs of HITL.

Generating a complete candidate takes approximately 5–10 minutes. With an average of two human rejections before approval, the overall process takes around 30 minutes, including agent execution rather than continuous human attention. MART automatically monitors training metrics and presents candidate adjustments for review, so human involvement resembles a brief conversational exchange with an LLM rather than manually inspecting training logs. Given the modest performance differences, we view HITL as an optional way to incorporate user judgment, while MART/no-HITL supports fully unattended tuning.

Q2:LLM api and ROI

Based on the token consumption recorded by the API, a complete tuning run costs approximately US$0.43–0.85 at current DeepSeek Flash rates.

The two tuning stages serve different purposes. System tuning uses short trials to identify efficient execution configurations before committing to longer training runs. For Qwen3-8B, MART increases throughput from 731 to 830 tokens/s/GPU on a single node (13.5%) and from 724 to 780 on two nodes (7.7%), providing efficiency benefits during subsequent training. Learning tuning searches for better learning configurations; manual hyperparameter tuning likewise requires training trials and time to inspect their outcomes. MART automates this monitoring and adjustment process, reducing the need for repeated manual analysis.

Q3: Reliability of experience summaries.

Experience summaries can introduce incorrect or outdated priors across tuning runs. In our experiments, we carefully inspected the summaries against the underlying trial records to verify their consistency and applicability. However, this manual verification does not guarantee their validity for future runs, and MART currently does not include an automatic mechanism for detecting and retiring outdated summaries. We acknowledge this as a limitation.

The answer did not fully address my question. I keep my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.