- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents HiGAP, a framework for improving the end-to-end efficiency of history-based speculative rollouts, particularly for MoE reinforcement learning. The method combines adaptive history pruning, reuse of verification and generation log-probabilities, probability-cache refresh, and advantage-guided history updates. Experiments on three MoE models and one dense model compare HiGAP with vanilla GRPO and SPEC-RL. The results show that reducing autoregressive generation alone does not necessarily reduce total training time, and that optimizing verification and training-side overheads can yield end-to-end improvements. On the evaluated MoE models, HiGAP reduces step time by 11.6–14.9% relative to vanilla GRPO.
Critical strengths
-
A practically important focus on end-to-end MoE RL efficiency. The paper examines an important limitation of history-based speculation: savings in autoregressive generation can be offset by verification and training-side computation. The timing breakdowns make this trade-off concrete, and the proposed optimizations deliver consistent step-time reductions across the three evaluated MoE actors. This provides useful guidance for deploying speculative rollouts in practical training systems.
-
Concrete optimizations supported by informative ablations. The proposed components address identifiable sources of overhead and history-reuse inefficiency. The paper also reports meaningful trade-offs rather than presenting every component as an unconditional improvement. In particular, removing history pruning improves speed but reduces downstream accuracy, while advantage-guided history updates are not uniformly beneficial across models. I appreciate this transparent reporting and find the overall systems contribution valuable.
Critical weaknesses
-
The SPEC-RL baseline configuration is insufficiently documented. Appendix A reports shared training and rollout settings, but does not specify the lenience parameter used for SPEC-RL, whether it was tuned for each model, or whether HiGAP uses the same setting. This is consequential because lenience directly affects acceptance, fresh generation, and the efficiency–quality trade-off. In particular, interpreting the reported 3.2% accepted-token ratio on Qwen3-30B-A3B requires knowing this configuration. Reporting the relevant settings and a focused sensitivity analysis would strengthen the baseline comparison and the motivation.
-
Some implementation details connecting history maintenance to verification need clarification. The paper explains how cached probabilities are refreshed and how positive-advantage trajectories replace negative ones, but does not explicitly specify the complete acceptance rule or how the refreshed probabilities enter it. Evaluating an existing trajectory under a newer policy is distinct from sampling that trajectory from the newer policy. Clarifying the intended sampling semantics, any approximation involved, and the handling of rejection would make the method easier to assess and reproduce. I view this primarily as a need for more precise methodological documentation rather than evidence that the reported practical gains are invalid.
Questions for authors
-
What lenience values were used for SPEC-RL and HiGAP on each model? Were these values tuned, and do the efficiency and quality conclusions remain consistent across a reasonable range of lenience settings?
-
Could you provide the explicit verification acceptance rule and explain how the refreshed cached probabilities are used? How are advantage-based cache replacement and rejection handled within this procedure?
-
What portion of the improvement remains when SPEC-RL is equipped with log-probability reuse alone? This comparison would help distinguish the benefits of eliminating redundant computation from those of the proposed history-maintenance mechanisms.
3 comments
Thanks for you rebuttal
Thanks for your rebuttal, I'll keep my score.
Follow-up on lenience sensitivity (W1, Q1)
Thank you for reviewing our rebuttal and for your helpful feedback.
As a follow-up, we completed additional HiGAP experiments on Qwen3.5-35B-A3B and Qwen3-8B. The main experiments used and for both SPEC-RL and HiGAP on all models, following SPEC-RL without further tuning. In this sweep, we kept and compared three bias values, with the corresponding multiplicative lenience shown below.
| Model | Bias | Lenience | Accepted prefix (tokens) | Rollout (s) | Step time (s) | Accuracy (%) |
|---|---|---|---|---|---|---|
| Qwen3.5-35B-A3B | 0.45 | 1.57 | 606.6 | 917.6 | 2522.2 | 76.84 ± 0.57 |
| 0.50 (default) | 1.65 | 639.6 | 884.3 | 2487.5 | 76.43 ± 0.30 | |
| 0.55 | 1.73 | 703.5 | 855.3 | 2462.6 | 73.19 ± 1.19 | |
| Qwen3-8B | 0.45 | 1.57 | 197.0 | 143.2 | 516.5 | 58.46 ± 2.46 |
| 0.50 (default) | 1.65 | 214.5 | 145.0 | 525.0 | 57.89 ± 2.66 | |
| 0.55 | 1.73 | 249.1 | 138.0 | 510.2 | 57.31 ± 3.22 |
Prefix lengths and timings are averaged over training steps 7–23. Accuracy is the nine-benchmark macro mean ± sample SD across decoding seeds 0–4 for each fixed checkpoint.
Efficiency. Higher lenience was associated with longer accepted prefixes on both models, and gave the shortest rollout and step times. Across the tested settings, HiGAP took 2462.6–2522.2 s/step on Qwen3.5 (GRPO: 2813.5 s/step) and 510.2–525.0 s/step on Qwen3-8B (GRPO: 764.4 s/step). All step times stayed within 2.9% of the respective defaults.
Accuracy. The efficiency gains at came with lower mean accuracy than at the default setting. The trade-off was more pronounced on Qwen3.5, where accuracy fell by 3.2 percentage points, though HiGAP stayed above GRPO (67.04 ± 2.36%) across the sweep. On Qwen3-8B, both alternative settings were within 0.6 percentage points of the default, and all three means were below GRPO (59.78 ± 1.59%).
We plan to extend this sensitivity analysis to more models, a wider lenience range, and SPEC-RL.
Response to Reviewer 26Eu
Thank you for recognizing the practical value of the end-to-end analysis and for identifying the details needed to assess the method. Revisions and additions in the updated manuscript are highlighted in blue.
1. Baseline configuration and lenience (W1, Q1).
We added Section 3.2 and Appendix A.1 to specify the acceptance-and-continuation procedure: ai=min{1,exp[s(logpi−logci+b)]}, where pi is the current verification score, ci the cached reference score, s the log-ratio scale, and b the acceptance-lenience bias. All reported SPEC-RL and HiGAP runs use the settings adopted from SPEC-RL, s=1,b=0.5, without additional tuning for HiGAP or individual models. This corresponds to multiplicative lenience ℓ=e0.5≈1.65 and is the setting underlying the reported 3.2% accepted-token ratio on Qwen3-30B-A3B.
Appendix A.3, Table 6 documents shared training sampling settings—temperature 1.0, top-p=1, disabled top-k—and method-specific pruning, cache, replacement, and bypass settings.
The requested lenience sensitivity experiments are underway. The fixed-setting comparisons alone do not establish robustness across a parameter range; we will report the additional results once available.
2. Verification and cache maintenance (W2, Q2).
Section 3.2 and Appendix A.1 specify that only the consecutive accepted prefix is retained. Rejection discards the rejected token and remaining historical suffix, followed by ordinary generation from that prefix. If the verification budget is exhausted without rejection, generation continues after the accepted prefix; a fully accepted complete response is reused.
Actor-update caching stores scores computed during training, before each minibatch's optimizer step, for future use as ci. No additional final-actor forward pass is required. Advantage-guided replacement copies the highest-advantage eligible response and its scores into negative-advantage cache entries for the same prompt. Both operations affect only the future cache, leaving current training responses and PPO anchors unchanged.
Rescoring does not change how a response was generated, and paired copying does not correct changes in draft selection. Verification does not correct for the complete selected-draft distribution; the inherited SPEC-RL procedure also uses ordinary continuation rather than residual-distribution correction. We therefore describe HiGAP as approximate history reuse and identify exact preservation of the current-policy sampling distribution as a direction for future work.
3. Improvement beyond probability reuse (Q3).
We added SPEC-RL+bypass using HiGAP's verification-prefix/generation-suffix score reuse, while retaining SPEC-RL's history configuration and disabling HiGAP's pruning, actor-update score refresh, and advantage-guided replacement:
These Qwen3.5-35B-A3B timings average common steps 7–23. Accuracy is the nine-benchmark macro mean ± sample SD across decoding seeds 0–4. HiGAP further reduces step time by 8.8%, with a matched-seed accuracy advantage of 10.40 ± 3.61 percentage points, positive at every seed. These results show that HiGAP provides additional efficiency gains beyond probability reuse alone in the evaluated Qwen3.5-35B-A3B setting.
Locations: Section 4.4; Appendix A.8, Tables 9–10.