acceptodds
Under review as a conference paper at ICLR 2027

HIGAP: Bridging the Overhead Gap in History-Based Speculative Rollout for MoE RL

Abstract

Rollout generation is a major bottleneck in reinforcement learning (RL) for large language models. History-based speculative rollout can reduce this cost by reusing trajectories from previous training steps as drafts, but its rollout-stage savings do not necessarily translate into end-to-end speedup, especially for mixture-of-experts (MoE) models. Expensive actor verification, redundant old-log-probability computation, and actor drift can substantially offset the benefits of history reuse. We present , a history-based speculative rollout framework that bridges this overhead gap through full-pipeline optimization. HiGAP adaptively bounds historical draft lengths to regulate history reuse and verification overhead, leverages verification probabilities to partially bypass old-log-probability recomputation, and mitigates history staleness under actor drift through post-update probability caching and advantage-guided history updates. Across three MoE and one dense models, HiGAP consistently reduces end-to-end step time and improves training throughput while maintaining competitive training effectiveness. On Qwen3.5-35B-A3B, HiGAP reduces step time by 11.6% and improves throughput by 12.7% over vanilla GRPO, while improving the average score by 12.3 points across nine mathematical reasoning benchmarks.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, History-Based Speculative Rollout, Mixture-of-Experts, LLM Training, Training Efficiency

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_26EuOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HiGAP, a framework for improving the end-to-end efficiency of history-based speculative rollouts, particularly for MoE reinforcement learning. The method combines adaptive history pruning, reuse of verification and generation log-probabilities, probability-cache refresh, and advantage-guided history updates. Experiments on three MoE models and one dense model compare HiGAP with vanilla GRPO and SPEC-RL. The results show that reducing autoregressive generation alone does not necessarily reduce total training time, and that optimizing verification and training-side overheads can yield end-to-end improvements. On the evaluated MoE models, HiGAP reduces step time by 11.6–14.9% relative to vanilla GRPO.

Critical strengths
  1. A practically important focus on end-to-end MoE RL efficiency. The paper examines an important limitation of history-based speculation: savings in autoregressive generation can be offset by verification and training-side computation. The timing breakdowns make this trade-off concrete, and the proposed optimizations deliver consistent step-time reductions across the three evaluated MoE actors. This provides useful guidance for deploying speculative rollouts in practical training systems.

  2. Concrete optimizations supported by informative ablations. The proposed components address identifiable sources of overhead and history-reuse inefficiency. The paper also reports meaningful trade-offs rather than presenting every component as an unconditional improvement. In particular, removing history pruning improves speed but reduces downstream accuracy, while advantage-guided history updates are not uniformly beneficial across models. I appreciate this transparent reporting and find the overall systems contribution valuable.

Critical weaknesses
  1. The SPEC-RL baseline configuration is insufficiently documented. Appendix A reports shared training and rollout settings, but does not specify the lenience parameter used for SPEC-RL, whether it was tuned for each model, or whether HiGAP uses the same setting. This is consequential because lenience directly affects acceptance, fresh generation, and the efficiency–quality trade-off. In particular, interpreting the reported 3.2% accepted-token ratio on Qwen3-30B-A3B requires knowing this configuration. Reporting the relevant settings and a focused sensitivity analysis would strengthen the baseline comparison and the motivation.

  2. Some implementation details connecting history maintenance to verification need clarification. The paper explains how cached probabilities are refreshed and how positive-advantage trajectories replace negative ones, but does not explicitly specify the complete acceptance rule or how the refreshed probabilities enter it. Evaluating an existing trajectory under a newer policy is distinct from sampling that trajectory from the newer policy. Clarifying the intended sampling semantics, any approximation involved, and the handling of rejection would make the method easier to assess and reproduce. I view this primarily as a need for more precise methodological documentation rather than evidence that the reported practical gains are invalid.

Questions for authors
  1. What lenience values were used for SPEC-RL and HiGAP on each model? Were these values tuned, and do the efficiency and quality conclusions remain consistent across a reasonable range of lenience settings?

  2. Could you provide the explicit verification acceptance rule and explain how the refreshed cached probabilities are used? How are advantage-based cache replacement and rejection handled within this procedure?

  3. What portion of the improvement remains when SPEC-RL is equipped with log-probability reuse alone? This comparison would help distinguish the benefits of eliminating redundant computation from those of the proposed history-maintenance mechanisms.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 26Eu

Thank you for recognizing the practical value of the end-to-end analysis and for identifying the details needed to assess the method. Revisions and additions in the updated manuscript are highlighted in blue.

1. Baseline configuration and lenience (W1, Q1).

We added Section 3.2 and Appendix A.1 to specify the acceptance-and-continuation procedure: ai=min⁡{1,exp⁡[s(log⁡pi−log⁡ci+b)]},a_i=\min\{1,\exp[s(\log p_i-\log c_i+b)]\}, where pip_i is the current verification score, cic_i the cached reference score, ss the log-ratio scale, and bb the acceptance-lenience bias. All reported SPEC-RL and HiGAP runs use the settings adopted from SPEC-RL, s=1,b=0.5s=1,b=0.5, without additional tuning for HiGAP or individual models. This corresponds to multiplicative lenience ℓ=e0.5≈1.65\ell=e^{0.5}\approx1.65 and is the setting underlying the reported 3.2% accepted-token ratio on Qwen3-30B-A3B.

Appendix A.3, Table 6 documents shared training sampling settings—temperature 1.0, top-p=1p=1, disabled top-kk—and method-specific pruning, cache, replacement, and bypass settings.

The requested lenience sensitivity experiments are underway. The fixed-setting comparisons alone do not establish robustness across a parameter range; we will report the additional results once available.

2. Verification and cache maintenance (W2, Q2).

Section 3.2 and Appendix A.1 specify that only the consecutive accepted prefix is retained. Rejection discards the rejected token and remaining historical suffix, followed by ordinary generation from that prefix. If the verification budget is exhausted without rejection, generation continues after the accepted prefix; a fully accepted complete response is reused.

Actor-update caching stores scores computed during training, before each minibatch's optimizer step, for future use as cic_i. No additional final-actor forward pass is required. Advantage-guided replacement copies the highest-advantage eligible response and its scores into negative-advantage cache entries for the same prompt. Both operations affect only the future cache, leaving current training responses and PPO anchors unchanged.

Rescoring does not change how a response was generated, and paired copying does not correct changes in draft selection. Verification does not correct for the complete selected-draft distribution; the inherited SPEC-RL procedure also uses ordinary continuation rather than residual-distribution correction. We therefore describe HiGAP as approximate history reuse and identify exact preservation of the current-policy sampling distribution as a direction for future work.

3. Improvement beyond probability reuse (Q3).

We added SPEC-RL+bypass using HiGAP's verification-prefix/generation-suffix score reuse, while retaining SPEC-RL's history configuration and disabling HiGAP's pruning, actor-update score refresh, and advantage-guided replacement:

MethodVerification (s)Step time (s)Five-seed accuracy (%)
SPEC-RL311.82838.564.81 ± 1.00
SPEC-RL+bypass319.72727.366.03 ± 3.34
HiGAP140.02487.576.43 ± 0.30

These Qwen3.5-35B-A3B timings average common steps 7–23. Accuracy is the nine-benchmark macro mean ± sample SD across decoding seeds 0–4. HiGAP further reduces step time by 8.8%, with a matched-seed accuracy advantage of 10.40 ± 3.61 percentage points, positive at every seed. These results show that HiGAP provides additional efficiency gains beyond probability reuse alone in the evaluated Qwen3.5-35B-A3B setting.

Locations: Section 4.4; Appendix A.8, Tables 9–10.

Reviewer_26EuOpenReview ↗

Thanks for you rebuttal

Thanks for your rebuttal, I'll keep my score.

AuthorsOpenReview ↗

Follow-up on lenience sensitivity (W1, Q1)

Thank you for reviewing our rebuttal and for your helpful feedback.

As a follow-up, we completed additional HiGAP experiments on Qwen3.5-35B-A3B and Qwen3-8B. The main experiments used s=1s=1 and b=0.50b=0.50 for both SPEC-RL and HiGAP on all models, following SPEC-RL without further tuning. In this sweep, we kept s=1s=1 and compared three bias values, with the corresponding multiplicative lenience ℓ=eb\ell=e^b shown below.

ModelBias bbLenience ℓ\ellAccepted prefix (tokens)Rollout (s)Step time (s)Accuracy (%)
Qwen3.5-35B-A3B0.451.57606.6917.62522.276.84 ± 0.57
0.50 (default)1.65639.6884.32487.576.43 ± 0.30
0.551.73703.5855.32462.673.19 ± 1.19
Qwen3-8B0.451.57197.0143.2516.558.46 ± 2.46
0.50 (default)1.65214.5145.0525.057.89 ± 2.66
0.551.73249.1138.0510.257.31 ± 3.22

Prefix lengths and timings are averaged over training steps 7–23. Accuracy is the nine-benchmark macro mean ± sample SD across decoding seeds 0–4 for each fixed checkpoint.

Efficiency. Higher lenience was associated with longer accepted prefixes on both models, and ℓ=1.73\ell=1.73 gave the shortest rollout and step times. Across the tested settings, HiGAP took 2462.6–2522.2 s/step on Qwen3.5 (GRPO: 2813.5 s/step) and 510.2–525.0 s/step on Qwen3-8B (GRPO: 764.4 s/step). All step times stayed within 2.9% of the respective defaults.

Accuracy. The efficiency gains at ℓ=1.73\ell=1.73 came with lower mean accuracy than at the default setting. The trade-off was more pronounced on Qwen3.5, where accuracy fell by 3.2 percentage points, though HiGAP stayed above GRPO (67.04 ± 2.36%) across the sweep. On Qwen3-8B, both alternative settings were within 0.6 percentage points of the default, and all three means were below GRPO (59.78 ± 1.59%).

We plan to extend this sensitivity analysis to more models, a wider lenience range, and SPEC-RL.

Reviewer_3GA9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The authors proposed HiGAP, a framework for accelerating history-based speculative rollout for MoE RL. It combines adaptive history pruning, reuse of verification probabilities to bypass old-log-probability computation, probability caching during actor updates, and advantage-guided history replacement. Experiments on three MoE models and one dense model report lower end-to-end step time and mixed downstream accuracy gains.

Critical strengths
  • The paper identifies a practical limitation of history reuse: rollout savings can be offset by verification and training overhead. The end-to-end timing breakdown is useful.
  • Evaluation across three MoE actors and a dense actor provides evidence that the implementation reduces step time beyond a single model setting.
Critical weaknesses
  1. The claim that the original GRPO objective is preserved is not established. Eq. (5) uses verification probabilities for accepted historical tokens, but the paper does not specify the acceptance rule or how the first rejected token is sampled. Eq. (6) refreshes probabilities without resampling the cached trajectory, while Eqs. (7)-(8) select and duplicate high-advantage histories. These operations change how the proposals are obtained. Evaluating an old response under a newer policy does not make it a sample from that policy. The actual proposal distribution and the corresponding correction need to be specified before the on-policy claim can be assessed.

  2. The experiments do not isolate the contribution beyond log-probability bypass. On Qwen3.5-35B-A3B, removing bypass gives 2747.5 s per step and 66.2 average accuracy, compared with 2813.5 s and 65.2 for vanilla GRPO. Full HiGAP reaches 2487.5 s and 77.5. GRPO+bypass and SPEC-RL+bypass controls are missing, so it is unclear how much benefit comes from the proposed history mechanisms. A single seed and one sampled response per evaluation question also leave the large accuracy gain insufficiently supported.

Questions for authors
  1. What is the complete acceptance/rejection algorithm, including lenience and replacement-token sampling? After probability refresh and advantage-based replacement, which distribution generated each draft, and how is it accounted for in verification?

  2. How do GRPO+bypass and SPEC-RL+bypass perform under the same settings? Does the reported accuracy gain persist across training seeds and repeated evaluation samples?

  3. What is the specific advance over RhymeRL's reward-prioritized drafts and adaptive windows, and SEER's treatment of MoE verification costs? A comparison with these closer baselines would help establish the contribution.

  4. What causes the periodic, synchronized spikes and dips in Figure 5?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3GA9 (Part 1)

Thank you for your thoughtful review and constructive feedback. We address your comments and questions below.

1. Draft origin, verification, and the GRPO objective (W1, Q1).

Acceptance and rejection. HiGAP inherits SPEC-RL's verification rule (Section 3.2 and Appendix A.1). A valid draft token ii is accepted when its uniform pseudorandom threshold uiu_i satisfies ui≤aiu_i\le a_i, where

ai=min⁡{1,exp⁡[s(log⁡pi−log⁡ci+b)]}.a_i=\min\{1,\exp[s(\log p_i-\log c_i+b)]\}.

Here, pip_i is the current actor's verification score and cic_i is the draft token's cached reference score. Both methods use s=1s=1 and b=0.50b=0.50 without further tuning, giving multiplicative lenience ℓ=e0.50≈1.65\ell=e^{0.50}\approx1.65. At the first rejection, we discard the rejected token and the remaining draft suffix. The ordinary rollout sampler draws the next token and continues from the accepted prefix without residual-distribution correction. If all tokens within the verification budget pass, generation resumes after the accepted prefix, or the response is reused in full if it is already complete.

Draft origin and sampling distribution. In the first epoch, the rollout engine samples each full response from its rollout distribution. The same applies to cache misses in later epochs. From the second epoch onward, on a cache hit, the rollout engine samples only the new suffix, conditional on the accepted historical prefix. The full draft distribution therefore also depends on historical draft selection and prefix acceptance. For both the reused prefix and the new suffix, HiGAP caches token log-probabilities from actor forward, which serve as references for future verification.

Cached scores and draft selection. For drafts saved in any epoch, HiGAP caches token log-probabilities from the training actor's forward pass before the corresponding minibatch optimizer step. This refresh rescores the existing tokens without resampling them. Advantage-guided replacement copies a higher-advantage response and its scores into other cache entries for the same prompt, changing which drafts are proposed for reuse. Verification uses the cached scores as cic_i in the acceptance rule. These scores do not encode the draft's actual sampling probabilities, and the verifier applies no correction for the distribution induced by rollout generation and cache selection.

Implication for GRPO. HiGAP retains the clipped GRPO loss and applies it to rollouts produced by approximate history reuse. The inherited SPEC-RL procedure saves generation work by reusing accepted prefixes, without a guarantee of exact current-policy sampling. Achieving exact on-policy sampling while retaining these efficiency gains is a direction for future work.

AuthorsOpenReview ↗

Response to Reviewer 3GA9 (Part 2)

2. Contribution beyond bypass and repeated evaluation (W2, Q2)

We added both requested controls on Qwen3.5-35B-A3B (Section 4.4 and Appendix A.8, Tables 9–10).

  • GRPO+bypass uses rollout-engine log-probabilities as old-policy loss anchors, bypassing the separate old-log-probability forward pass.
  • SPEC-RL+bypass reuses verification scores for accepted historical prefixes and rollout-engine scores for newly generated suffixes. It retains SPEC-RL's history configuration, with HiGAP's pruning, actor-update score refresh, and advantage-guided replacement disabled.
MethodAccepted prefix (tokens)Rollout (s)Verification (s)Old-log-prob (s)Step time (s)Five-seed accuracy (%)
GRPO—1043.7—317.42813.567.04 ± 2.36
GRPO+bypass—1068.2——2620.666.43 ± 2.57
SPEC-RL995.6710.5311.8313.92838.564.81 ± 1.00
SPEC-RL+bypass617.6857.5319.71.32727.366.03 ± 3.34
HiGAP639.6884.3140.01.22487.576.43 ± 0.30

Prefix lengths and timings average steps 7–23. Accuracy is the nine-benchmark macro mean ± sample SD across decoding seeds 0–4.

SPEC-RL+bypass reduces old-log-probability time from 313.9 to 1.3 s. In these runs, this saving was accompanied by a shorter accepted prefix (995.6 → 617.6 tokens) and longer rollout time (710.5 → 857.5 s), partly offsetting the computational saving. Compared with SPEC-RL+bypass, HiGAP achieves a similar old-log-probability cost and reduces verification time from 319.7 to 140.0 s, yielding a further 8.8% reduction in step time despite slightly longer rollout generation. Relative to GRPO+bypass, the step-time reduction is 5.1% and improves five-seed mean accuracy from 66.43% to 76.43%, a gain of around 10.00 percentage points.

We also reevaluated all four models using decoding seeds 0–4 to assess whether the accuracy findings persist across evaluation sampling. The table compares original single-evaluation → five-seed mean ± sample SD macro accuracy over nine benchmarks (Section 4.3 and Appendix A.6, Table 11).

ModelBaseGRPOSPEC-RLHiGAP
Qwen3.5-35B-A3B66.7 → 67.64 ± 1.1565.2 → 67.04 ± 2.3660.3 → 64.81 ± 1.0077.5 → 76.43 ± 0.30
Qwen3-30B-A3B48.0 → 48.01 ± 0.9064.3 → 64.08 ± 0.6162.6 → 62.90 ± 0.6065.9 → 65.21 ± 1.04
Moonlight-16B-A3B14.7 → 14.97 ± 3.3537.1 → 36.12 ± 3.2633.8 → 33.15 ± 2.2234.3 → 33.01 ± 1.65
Qwen3-8B46.3 → 44.91 ± 6.4659.1 → 59.78 ± 1.5958.1 → 60.71 ± 1.8257.4 → 57.89 ± 2.66

The repeated evaluations broadly preserve the original model-dependent trends. HiGAP's accuracy gains over GRPO persist on both Qwen MoE models, supporting their consistency across evaluation seeds. Moonlight and Qwen3-8B show modest decreases relative to GRPO. We also reevaluated HiGAP without advantage-guided update on Qwen3-8B across the same five decoding seeds. Its macro accuracy is 59.00 ± 2.54%, removing advantage-guided update increases mean accuracy by 1.11 percentage points, bringing it closer to GRPO.

3. RhymeRL and SEER (Q3).

HiGAP differs in draft management and computation reuse. RhymeRL builds reward-aware suffix trees; HiGAP directly caches historical responses and scores. RhymeRL adjusts decoding-time windows, while HiGAP sets verification budgets from previous-epoch acceptance statistics. RhymeRL's main evaluation uses dense actors; ours additionally evaluates MoE actors. SEER uses within-group response context for drafting, whereas HiGAP reuses responses across epochs. HiGAP's verification and score caching do not require within-step group sampling, potentially extending beyond GRPO, which could be tested in future work.

Both RhymeRL and SEER optimize system-level scheduling, so their reported gains extend beyond speculative generation. RhymeRL comprises HistoSpec (generation) and HistoPipe (scheduling). The public verl integration includes HistoSpec but not HistoPipe, so our component-level attempt does not reproduce the complete system. And we could not identify an official public SEER implementation in the checked sources. We will clarify these distinctions and pursue validated full-system comparisons.

4. Periodic fluctuations in Figure 5 (Q4).

Figure 5 reports mean reward for each training batch. Without shuffling, the same final batch recurs each epoch; its higher or lower reward relative to neighboring batches produces periodic peaks or dips. Training logs show six batches per epoch, placing these recurring fluctuations at steps 6, 12, and 18. Rewards at these steps tend to be higher for the Qwen models and lower for Moonlight. Qwen3-8B GRPO used shuffling, so the fixed-batch explanation does not apply to that run.

Reviewer_NgxaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary
  • This paper proposes HiGAP to improve end-to-end efficiency of history-based speculative rollouts for MoE RL.

  • It combines adaptive draft truncation, log-probability reuse, probability-cache refresh, and advantage-guided history selection to reduce verification and recomputation overheads.

  • Experiments with GRPO on three MoE models and one dense model show step-time reductions of 11.6–14.9% and 31.3%, respectively.

  • Evaluation on nine reasoning benchmarks and component ablations demonstrate efficiency–quality trade-offs, with accuracy improvements on some models and declines on others.

Critical strengths
  1. Consistent end-to-end speedups. HiGAP reduces RL step time by 11.6–14.9% across three MoE models, while SPEC-RL increases it, demonstrating practical benefits beyond rollout-only acceleration.

  2. Effective computation reuse. Reusing verification and generation probabilities nearly eliminates separate old-log-probability computation, with ablations confirming its contribution to overall speedup.

  3. Low-overhead history maintenance. The method refreshes cached probabilities using existing actor-update computations and selects histories using available advantages, avoiding an additional draft or quality model.

  4. Informative efficiency–quality analysis. Timing breakdowns and ablations show that maximizing history reuse can improve speed while hurting accuracy, motivating controlled reuse.

Critical weaknesses
  1. On-policy correctness is insufficiently established. The paper does not fully specify the acceptance/rejection procedure or explain how refreshed cached probabilities and advantage-based history selection preserve the target sampling distribution. This leaves unclear whether the reported gains come from an equivalent sampling procedure or a biased training process.

  2. The accuracy gains lack sufficient controls. The headline 12.3-point improvement occurs on Qwen3.5, where vanilla GRPO underperforms the initial checkpoint, while HiGAP reduces accuracy on two other models. Without repeated runs or a vanilla-GRPO baseline using the same log-probability bypass, it is difficult to separate the benefits of history reuse from probability-handling effects and experimental variability.

Questions for authors
  1. Clarification on sampling correctness: What are the exact acceptance and rejection-resampling rules? After refreshing cached probabilities and applying advantage-guided history replacement, does the procedure still preserve the current policy’s sampling distribution? Please justify this claim.

  2. Clarification on the accuracy gains: How does vanilla GRPO perform with the same log-probability bypass but without history reuse? Are the reported accuracy differences reproducible across seeds? These controls would help determine whether the gains arise from speculative history reuse or changes in probability handling.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Ngxa

Thank you for recognizing the end-to-end efficiency benefits and for requesting clearer sampling semantics and accuracy controls. Revisions and additions in the updated manuscript are highlighted in blue.

1. Verification and history maintenance (W1, Q1).

Section 3.2 and Appendix A.1 specify the acceptance gate inherited from SPEC-RL: ai=min⁡{1,exp⁡[s(log⁡pi−log⁡ci+b)]},a_i=\min\{1,\exp[s(\log p_i-\log c_i+b)]\}, where pip_i is the current actor's verification score and cic_i its cached reference score. Here, ss scales the log-ratio and bb controls lenience. Both methods inherit s=1,b=0.5s=1,b=0.5 without additional tuning, equivalent to multiplicative lenience ℓ=e0.5≈1.65\ell=e^{0.5}\approx1.65.

At the first rejection, the rejected token and remaining historical suffix are discarded, and ordinary generation resumes from the accepted prefix. If the verification budget is exhausted without rejection, generation continues after that prefix; a fully accepted complete response is reused. Without residual-distribution correction, this inherited procedure does not generally guarantee exact current-policy sampling.

HiGAP modifies two inputs to this procedure. Probability caching saves scores computed during actor training, before each minibatch's optimizer step, as references for future verification. These can reflect more recent actor parameters without an extra forward pass. Advantage-guided replacement retains the highest-advantage eligible response in place of negative-advantage cache entries, copying the response and its scores together. These operations leave current training responses and PPO anchors unchanged, but can affect future rollout distributions. We therefore characterize HiGAP as approximate history reuse, without guaranteeing the same trajectory distribution as fresh current-policy generation. Exact distribution preservation remains a direction for future work.

2. Accuracy controls and repeated evaluation (W2, Q2).

We added the requested GRPO+bypass control on Qwen3.5-35B-A3B (Section 4.4 and Appendix A.8, Tables 9–10). It uses rollout-engine log-probabilities as old-policy loss anchors, eliminating the separate old-log-probability forward pass while generating all responses without history reuse.

MethodRollout (s)Old-log-prob (s)Step time (s)Five-seed accuracy (%)
Base checkpoint———67.64 ± 1.15
GRPO1043.7317.42813.567.04 ± 2.36
GRPO+bypass1068.2—2620.666.43 ± 2.57
HiGAP884.31.22487.576.43 ± 0.30

Timings average common steps 7–23. GRPO+bypass skips the old-log-probability block entirely. Accuracy is the nine-benchmark macro mean ± sample SD across decoding seeds 0–4, with one response per question per seed.

Contribution beyond bypass. Relative to GRPO+bypass, HiGAP reduces rollout time from 1068.2 to 884.3 s and overall step time by 5.1%, while increasing mean accuracy from 66.43% to 76.43%. Its matched-seed accuracy advantage is 10.00 ± 2.85 percentage points, positive at all five seeds. Thus, the observed Qwen3.5 gain persists against the requested control. We also added SPEC-RL+bypass, which achieves 2727.3 s/step and 66.03 ± 3.34% accuracy, as a complementary control in Appendix A.8.

Repeated evaluation. We reevaluated all four models using the same five decoding seeds (Section 4.3 and Appendix A.6, Table 11).

ModelGRPOSPEC-RLHiGAP
Qwen3.5-35B-A3B67.04 ± 2.3664.81 ± 1.0076.43 ± 0.30
Qwen3-30B-A3B64.08 ± 0.6162.90 ± 0.6065.21 ± 1.04
Moonlight-16B-A3B36.12 ± 3.2633.15 ± 2.2233.01 ± 1.65
Qwen3-8B59.78 ± 1.5960.71 ± 1.8257.89 ± 2.66

The results broadly preserve the original model-dependent trends. HiGAP's gains over GRPO persist on both Qwen MoE models, while Moonlight and Qwen3-8B show modest decreases. On Qwen3.5, the advantage over GRPO is 9.39 percentage points, compared with the original 12.3-point difference.

We also reevaluated Qwen3-8B without advantage-guided update, obtaining 59.00 ± 2.54%, compared with 57.89 ± 2.66% for full HiGAP. This further illustrates the model-dependent quality trade-off.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the AC and reviewers for their feedback. HiGAP’s main contribution is to translate historical response reuse into end-to-end RL training savings, reducing step time by 11.6–14.9% relative to GRPO across three MoE models. In our five-seed evaluation, both Qwen MoE models also achieved higher mean accuracy across nine benchmarks than GRPO, with a 9.39-point gain on Qwen3.5-35B-A3B. In response to the reviews, we clarified the sampling procedure, added controls to measure gains beyond log-probability bypass, repeated accuracy evaluations across decoding seeds, and compared HiGAP’s design with related systems. We summarize these findings below. Revisions to the manuscript are highlighted in blue.

1. Sampling and lenience

We clarified the sampling procedure and added a lenience sensitivity study. Sections 3.2–3.4 and Appendix A.1 explain that, after a rejection, the inherited SPEC-RL rule resumes ordinary generation from the accepted prefix without residual-distribution correction. HiGAP also refreshes verification scores and changes which drafts enter the future cache. We therefore describe HiGAP as approximate history reuse, with no guarantee of exact current-policy sampling.

Both methods used SPEC-RL's s=1s=1 and b=0.50b=0.50 without tuning in the main experiments. We tested two additional HiGAP settings on one MoE model and one dense model, keeping s=1s=1 so that lenience ℓ=eb\ell=e^b.

ModelBias bbLenience ℓ\ellStep time (s)Accuracy (%)
Qwen3.5-35B-A3B0.451.572522.276.84 ± 0.57
0.50 (default)1.652487.576.43 ± 0.30
0.551.732462.673.19 ± 1.19
Qwen3-8B0.451.57516.558.46 ± 2.46
0.50 (default)1.65525.057.89 ± 2.66
0.551.73510.257.31 ± 3.22

HiGAP remained faster than GRPO at all three settings. The highest lenience gave the shortest steps on both models, with mean accuracy 3.2 percentage points below the default on Qwen3.5 and 0.6 points below on Qwen3-8B, revealed an efficiency–accuracy trade-off.

2. Gains beyond bypass

We added GRPO+bypass and SPEC-RL+bypass controls. GRPO+bypass reuses rollout log-probabilities for freshly generated responses. SPEC-RL+bypass retains SPEC-RL's history configuration and reuses verification scores for accepted prefixes and rollout scores for generated suffixes. Both controls were evaluated on Qwen3.5-35B-A3B (Section 4.4 and Appendix A.8, Tables 9–10).

MethodStep time (s)Accuracy (%)
GRPO2813.567.04 ± 2.36
GRPO+bypass2620.666.43 ± 2.57
SPEC-RL2838.564.81 ± 1.00
SPEC-RL+bypass2727.366.03 ± 3.34
HiGAP2487.576.43 ± 0.30

HiGAP reduced step time by 5.1% vs. GRPO+bypass and 8.8% vs. SPEC-RL+bypass. Its matched-seed accuracy gains were 10.00 ± 2.85 and 10.40 ± 3.61 percentage points, respectively, positive at all five decoding seeds. These comparisons measure the combined benefit beyond bypass. Both timing tables average steps 7–23.

3. Accuracy across seeds

We reevaluated fixed checkpoints for all four models using decoding seeds 0–4. The table shows original single-evaluation accuracy → five-seed mean ± sample SD over nine benchmarks (Section 4.3 and Appendix A.6, Table 11).

ModelBaseGRPOSPEC-RLHiGAP
Qwen3.5-35B-A3B66.7 → 67.64 ± 1.1565.2 → 67.04 ± 2.3660.3 → 64.81 ± 1.0077.5 → 76.43 ± 0.30
Qwen3-30B-A3B48.0 → 48.01 ± 0.9064.3 → 64.08 ± 0.6162.6 → 62.90 ± 0.6065.9 → 65.21 ± 1.04
Moonlight-16B-A3B14.7 → 14.97 ± 3.3537.1 → 36.12 ± 3.2633.8 → 33.15 ± 2.2234.3 → 33.01 ± 1.65
Qwen3-8B46.3 → 44.91 ± 6.4659.1 → 59.78 ± 1.5958.1 → 60.71 ± 1.8257.4 → 57.89 ± 2.66

Across five decoding seeds, the mean accuracies reproduced the trend seen in the original single evaluation: compared with GRPO, HiGAP achieved higher accuracy on both Qwen MoE models, with modest declines on Moonlight and Qwen3-8B.

4. Related systems

We clarified how HiGAP differs from RhymeRL and SEER. RhymeRL builds reward-aware suffix trees and primarily evaluates dense actors. SEER drafts from responses within the same prompt group. HiGAP directly caches responses across epochs and reuses scores from verification and actor training, including for MoE actors.

Both RhymeRL and SEER combine speculative generation with scheduling optimizations, so their reported end-to-end speedups reflect the combined effects of both. The public RhymeRL integration we tested covers speculative generation (HistoSpec) but omits scheduling (HistoPipe). And we could not identify an official public SEER implementation in the sources we checked. We will incorporate these distinctions and pursue matched full-system comparisons.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.