acceptodds
Under review as a conference paper at ICLR 2027

GrowMTP: Can RL Grow Its Own Draft Head?

Abstract

Reinforcement learning (RL) post-training drives the frontier capabilities of large language models, with its wall-clock dominated by autoregressive rollout generation. Speculative decoding is an established remedy for this bottleneck, but existing draft heads must be pretrained or warmed up before RL, introducing substantial training cost outside the RL run to be accelerated. We observe that RL training itself provides both conditions required for online draft-head training: its rollout distribution is far narrower than that of pretraining, and its verification step continuously produces supervision signals aligned with this distribution. Building on these observations, we propose GrowMTP, which uses this supervision to train a draft head from scratch entirely within the RL loop, with all head updates detached from the policy backbone. On Qwen3-4B (no draft head), MiMo-7B-SFT (weak head), and Qwen3.5-4B-Base (strong head), GrowMTP achieves rollout speedups of 2.13×, 1.93×, and 1.36×, and end-to-end speedups of 1.60×, 1.41×, and 1.20×, respectively. GrowMTP therefore serves existing RL training frameworks as a modular component, particularly offering a from-scratch acceleration path for models without pretrained draft heads.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, Large Language Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vFRTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GrowMTP learns a speculative-decoding draft head during RL, using the target distributions and draft trajectories produced by rollout verification. The method reconstructs each draft cycle, applies a logarithmic transformation to an acceptance-chain objective, and retains supervision through the first rejected position. The head loss is detached from the policy backbone. On Qwen3-4B, a randomly initialized recurrent head learns quickly enough to accelerate the same 500-step RL run by 1.60× on math and 1.56× on code, including the cost of training the head. Experiments on two models with pretrained heads examine online adaptation, drafting depth, and the individual components. Held-out evaluations assess both acceptance length and policy quality.

Critical strengths
  1. The from-scratch result is convincing and practically useful. On Qwen3-4B, acceptance length grows from 1.00 to 2.91 on math and 2.64 on code. Appendix B.4 makes clear that timings cover the entire run, including the initial learning period and head updates, whereas these acceptance figures describe the final ten steps. Thus, the reported savings account for the cost of acquiring the drafting capability. The held-out results show no evident loss of policy quality in either domain. This is a useful option for RL runs on models without a pretrained draft head.

  2. The paper treats the training and serving details carefully. Reconstruction preserves the cycle boundaries and the distinction between backbone-derived prefix features and the head's recurrent draft features; Appendix G explains why simply unrolling the head over the committed response does not reproduce that computation. The experiments also track the quantities that matter in practice: joint CE can increase acceptance while destroying policy quality, and increasing MiMo's depth from five to seven improves acceptance while slowing the full step. The top-64 approximation check and the clear separation between the acceptance surrogate and actual rollout acceptance make the method easier to assess.

Critical weaknesses
  1. The ablations leave the main methodological question partly open. Detached CE is compared on Qwen3.5, and the more detailed reconstruction, objective, and masking ablations use MiMo; both begin with pretrained heads. Qwen3-4B is evaluated only with the complete method. I would like to know whether its successful start from random initialization depends on DCA and masking, or whether detached CE or the TV-chain objective would learn a similarly useful head. This distinction matters because online drafter adaptation is already present in ReSpec, and Bebop introduces the multi-step TV objective, as the paper acknowledges. The complete system works, but the evidence for what enables its central result is less conclusive.

  2. The finer efficiency comparisons need more careful attribution. DCA reduces step time from 382.69 seconds with TV to 371.02 seconds in Table 10, about 3%, without uncertainty across independent runs or generated-token counts. These omissions matter more for this close comparison than for the large gain over autoregressive decoding. In Table 5, reconstruction reaches nearly the same aggregate acceptance as teacher forcing while supervising fewer positions, so the result establishes a useful cost saving but provides weaker evidence that correcting the state mismatch is responsible for the gain. Separately, Table 4 compares a frozen MiMo head at K=1 with trained heads at larger K, combining the effects of online training and draft depth. None of these issues invalidates the main result, but they limit the conclusions about which ingredients account for the savings.

Questions for authors
  1. Main comparison: Can you compare the randomly initialized Qwen3-4B head under detached CE or TV with the complete method, keeping reconstruction and the other settings fixed? Acceptance during the initial learning phase and cumulative GPU-hours would directly address the first weakness.
  2. Attribution of speedups: What acceptance and step time does the frozen MiMo head achieve at K=5? For the close DCA–TV comparison, can you provide generated-token counts and any available evidence of run-to-run variability?
  3. Cost clarification: How was the 161.6 GPU-hour offline budget in Table 2 selected? Also, at which step does GrowMTP first use less cumulative compute than autoregressive RL? Figures 5–6 identify when individual steps become faster, which is a different point.
Reviewer_x4NsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GrowMTP, a from-scratch online draft-head training method for accelerating the rollout stage of RL training. GrowMTP trains the draft head online from rollout verification signals, with all head updates detached from the policy backbone. Experiments show rollout and end-to-end speedups with comparable held-out benchmark scores.

Critical strengths
  1. The paper studies an important and practical problem in accelerating RL rollouts.
  2. The experiments cover multiple models and include useful ablation studies.
Critical weaknesses
  1. The paper lacks experimental comparisons with prior online drafter training methods for RL, including TLT [1], FastGRPO [2], and ReSpec [3]. TLT trains a lightweight EAGLE-style drafter online on idle GPUs. FastGRPO already evaluates online training without drafter pretraining. Although ReSpec's offline warm-up is briefly discussed, no experimental comparison is provided.

  2. The evaluation also lacks experimental comparisons with methods that accelerate RL rollouts without drafter training, such as DAS [4] and EfficientRollout [5]. It remains unclear whether GrowMTP offers a practical advantage over these alternatives once its online training overhead is included. I believe end-to-end comparisons under matched settings are necessary for a fair evaluation.

[1] Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter (ASPLOS 2026)
[2] FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learning (ICLR 2026)
[3] ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems (MLSys 2026)
[4] Beat the long tail: Distribution-Aware Speculative Decoding for RL Training (MLSys 2026)
[5] EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts (NeurIPS 2026)

Questions for authors

Please refer to the Weaknesses section above.

Reviewer_U89UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GrowMTP trains a speculative draft head within the RL loop without a separate offline warm-up. It combines draft-path reconstruction, a logarithmic transformation of an acceptance-chain objective, and masking after the first rejection, with head gradients detached from the policy. Experiments on three models and math/code RL report up to 1.60x end-to-end speedup, including head updates, with comparable benchmark performance.

Critical strengths
  1. Growing a useful draft head from random initialization during the RL run addresses a practical deployment barrier. The Qwen3-4B experiments report 1.60x and 1.56x full-run step speedups on math and code, including online training costs.

  2. The evaluation covers absent, weak, and strong initial heads and separates acceptance length from policy quality. The depth study usefully shows that higher acceptance does not necessarily improve end-to-end efficiency.

Critical weaknesses
  1. The incremental novelty is limited. DCA adds cycle-level reweighting to Bebop's TV-chain objective [1]; VGM's principle of retaining supervision through the first failure and masking later positions already appears in Spec-AUF [2]; draft-path replay has precedent in Draft-OPD [3]; and online MTP training with gradient isolation is already supported by RL frameworks. Overall, the work appears primarily to combine existing methods with engineering improvements, without establishing a sufficiently substantial methodological advance.

  2. The distinctive cold-start contribution is not sufficiently isolated. Core component ablations use a pretrained MiMo head, rather than the randomly initialized Qwen3-4B head. A matched cold-start baseline using existing objectives is needed to establish the necessity of the proposed modifications; the reported DCA gain over TV-chain training is modest (Table 10).

Questions for authors
  1. Can the authors compare DCA against tuned TV-chain and detached CE objectives in the random-initialization setting, keeping reconstruction, masking, data, and compute comparable? Which components are necessary for a net reduction in total training cost?

  2. Table 10 compares the TV-chain objective, while Table 2 uses an offline head trained with the paper's own objective. How does GrowMTP compare with a Bebop-style pre-RL TV-chain adaptation followed by frozen-head RL, accounting for all preparation costs and varying the offline budget and RL duration?

  3. What specifically distinguishes the training method from Bebop [1], and what gains are attributable to these differences? What new technical insight or capability does GrowMTP establish beyond integrating existing components?

References:

[1] Bebop: Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Sampling, https://arxiv.org/abs/2606.12370

[2] Spec-AUF: Accept-Until-Fail Training under Train-Inference Misalignment for Masked Block Drafters, https://arxiv.org/html/2607.01893v1

[3] Draft-OPD: On-Policy Distillation for Speculative Draft Models, https://arxiv.org/html/2605.29343v1

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.