- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces MART, a multi-agent framework designed to automate the complex hyperparameter tuning process for LLM reinforcement learning (RL) by addressing the distinct memory and compute demands of evolving rollout and policy-update phases. Utilizing a human-in-the-loop review system, specialized agents optimize system configurations for throughput in short-horizon trials, followed by longer-horizon trials that fine-tune learning hyperparameters.
Critical strengths
-
The paper addresses the uniquely challenging and novel problem of automating hyperparameter tuning specifically for LLM reinforcement learning, effectively managing the dynamic workloads caused by alternating rollout and policy-update phases.
-
The MART framework demonstrates substantial empirical superiority.
Critical weaknesses
-
Whether the framework can be extended to long horizon agentic tasks like webshop and alfworld is unknown, despite these being a current research focus for RL post training.
-
A human reviewer is needed, and results fluctuate significantly once the human review is removed (e.g., a significant drop in performance for AIME 25 but less of a drop for AIME 24), which may signify that the current experiments are insufficient or heavily reliant on the human reviewer.
-
The MAS framework of proposing and validating/refining is a very common collaboration topology and is not scientifically novel, which significantly hurts the overall novelty of the work.
Questions for authors
-
Why are all the experiments conducted on math problems when the framework does not appear specific to math settings, and can the work be extended to other tasks, especially long horizon agentic tasks like webshop and alfworld?
-
How are the human reviewers selected, what are their credentials and experience with these math problems, how many reviewers were involved, and what exactly do they see?
-
Can the authors include more RL algorithms, as well as different model architectures and sizes?
-
Can the authors include more advanced hyperparameter baselines?
-
How does the proposed framework compare with human experts?
3 comments
Stage 1 — hardware tuning
| Parameter change | throughput (token/s/gpu) | Reason | |
|---|---|---|---|
| 1 | Baseline | 559.4 | Reference point for the throughput search |
| 2 | rollout.max_num_seqs 256→1024 | 580.3 | The cap was binding: running requests pinned at 256 while KV usage was only 47.6% with zero preemptions, so the admission limit is raised to the measured queue-demand p95 (1024) |
| 3 | actor.megatron.pipeline_model_parallel_size2→1 actor.ppo_max_token_len_per_gpu 16384→8192 | 590.5— rejected: rollout phase reached 61691 MiB | PP=2 leaves only DP=2 and adds stage bubbles, so PP=1 raises actor DP to 4. token cap: PP=1 doubles per-GPU weights/optimizer state, so the training token budget must shrink to stay in memory |
| 4 | rollout.gpu_memory_utilization 0.6→0.55 | 582.3 | The rollout phase exceeded the throughput memory reserve; 0.55 is the smallest value whose estimate (55137 MiB) clears the limit |
| 5 | rollout.log_prob_max_token_len_per_gpu 16384→32768 ref.log_prob_max_token_len_per_gpu 16384→65536 | 587.0 | actor log-prob: halve the number of micro-batches in a phase that must score the full batch each update ref log-prob: same overhead removal in the phase with the largest memory slack |
| 6 | critic.megatron.tensor_model_parallel_size 8→4 critic.ppo_max_token_len_per_gpu 16384→8192 | 597.6 | Decrease the TP of the critic such that its architecture is consistent with the actor. |
Stage 2 — stability tuning
| Parameter change | Reason | Result | |
|---|---|---|---|
| 7 | Baseline (hardware frozen) | Reference trajectory with the default stability knobs | stopped at 28/100: entropy 7.09→0.91, ppo_kl→0.85, pg_clipfrac→0.37; reward 0.569, MATH 0.4922 |
| 8 | actor.entropy_coeff 0→0.005 | Entropy had no restoring term in the loss and collapsed monotonically toward a deterministic policy | stopped at 41/100: entropy 7.65→1.77, ppo_kl→1.27, pg_clipfrac→0.43; reward 0.569, MATH 0.4872 |
| 9 | actor.optim.lr 1e-6→5e-7 | The ppo_kl>1 excursions and pg_clipfrac saturation scale with the per-update policy move, which is set by the learning rate | completed 100/100: reward 0.574, MATH 0.4866 (peak 0.5016 at step 60); entropy still decayed 7.3→0.40 |
| 10 | actor.kl_loss_coef 0.001→0.01 | The KL penalty contributed only ~0.007 against a pg_loss of ~0.14, i.e. the reference anchor was numerically inert | interrupted at 34/100: entropy 6.89→1.28, pg_clipfrac→0.38; reward 0.570, MATH 0.4918 |
Q2:Human reviewer background and review process.
One human reviewer with substantial experience in model training and experimental evaluation participated in our experiments. The review concerns training configurations rather than mathematical solutions. The reviewer sees the Proposal Agent’s suggested changes, the Feasibility Agent’s assessment, and relevant experimental records and metrics. MART’s HITL process can therefore be viewed as an interaction between a user and an LLM, in which the user reviews proposed adjustments and provides approval or feedback.
Q3 Evaluation across RL algorithms and models.
The additional results presented above demonstrate MART’s tuning of Llama-3.1-8B-Instruct using PPO, extending our evaluation to another model family and an additional RL algorithm.
Q4&Q5 Comparison with advanced tuning baselines and human experts.
MART can be viewed as human–LLM collaborative tuning. One human reviewer with substantial experience in model training and experimental evaluation participated in our experiments, reviewing the Proposal Agent’s suggested changes, the Feasibility Agent’s assessment, and relevant experimental records and metrics. The reviewer provides approval or feedback, while MART/no-HITL performs tuning without human intervention.
Dear Reviewer, we are grateful for your thorough review and the excellent questions posed. Your queries touch upon crucial deployment and scalability concerns of our framework. We have carefully considered each point and provide our detailed responses below.
Q1: Applicability beyond mathematical reasoning.
We focus on mathematical reasoning because rule-based rewards allow us to evaluate MART without introducing an additional reward model. However, MART is not restricted to this setting and supports broader model and algorithm configurations within verl. To demonstrate this, we additionally evaluate MART/no-HITL on Llama-3.1-8B-Instruct using PPO, which introduces a critic model compared with GRPO. The experimental results are presented below. Extending the evaluation to long-horizon agentic tasks such as WebShop and ALFWorld remains future work.