acceptodds
Under review as a conference paper at ICLR 2027

RANA: Accelerating Multi-Turn Agentic Reinforcement Learning with Speculative Decoding

Abstract

The evolution of Large Language Models (LLMs) toward reasoning-intensive agents relies heavily on Reinforcement Learning (RL) for alignment, e.g., RL human feedback (RLHF). However, the RLHF rollout phase suffers from severe computational inefficiencies, particularly in multi-turn interactions where sequential dependencies exacerbate the long-tail effect and lead to GPU underutilization. While recent work has explored speculative decoding (SD) to accelerate RLHF rollouts, they overlook the structural properties of agentic multi-turn interactions, i.e., contextual continuity, entropy decay, and hardware sensitivity, leaving substantial optimization potential unexploited. In this paper, we propose RANA (Rollout Adaptive N-Gram Accelerator), a system that accelerates multi-turn agentic RLHF rollouts by systematically exploiting these structural properties. RANA consists of three innovative components: (1) a Tree-based N-gram Speculative Decoding mechanism that leverages historical context for model-free drafting; (2) a Load-Adaptive Controller that dynamically toggles speculation based on Roofline analysis to avoid compute-bound contention; and (3) an Entropy-aware Scheduler that leverages entropy decay to align low-entropy rollouts with the execution tail, where speculative decoding yields the greatest benefit. Experiments show that RANA achieves up to 3.74x speedup, significantly outperforming prior approaches.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Reinforcement Learning, Speculative Decoding, Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_e2EhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RANA accelerates multi-turn agentic RL rollouts by combining a history-based n-gram speculative decoder, a load-adaptive controller, and an entropy-aware scheduler. The drafter reuses previously generated tokens without an auxiliary model; the controller disables speculation at high concurrency; and the scheduler prioritizes high-entropy requests to reserve more predictable requests for the low-load tail. Experiments on mathematical self-correction, maze navigation, and code generation report an average rollout speedup of 2.09× and a maximum of 3.74× over vanilla decoding. The paper also compares with EAGLE-3 and provides a component ablation.

Critical strengths

S1. The design connects speculative decoding to the changing execution regime of RL rollouts. RANA recognizes that verification overhead can be detrimental during compute-saturated execution but beneficial during the memory-bound tail. Dynamically enabling speculation according to load is a practical way to exploit this distinction. The substantial makespan reductions across the evaluated workloads make this a consequential systems contribution, although the controller's isolated gain in the reported ablation is modest.

S2. The system offers a lightweight approach to exploiting repeated content in agentic interactions. Reusing historical outputs avoids maintaining or training an auxiliary draft model and is particularly suitable for iterative refinement and debugging. The end-to-end rollout measurements, together with the additional time reduction from adding the scheduler on top of SD and the controller in Table 3, provide evidence that the integrated design has practical value beyond an isolated decoding microbenchmark.

Critical weaknesses

W1. The multi-turn-specific contribution is not sufficiently isolated. RANA is motivated by prior accelerators’ failure to exploit multi-turn structure, yet comparisons primarily involve vanilla decoding and EAGLE-3. Its advance over historical speculation remains unclear, particularly since SuffixDecoding already exploits repetition in agentic and self-refinement requests. Load-adaptive speculation is largely workload-agnostic, and the experiments do not separate cross-turn reuse from cross-rollout sharing and cross-epoch persistence. The previous-turn entropy scheduler is more distinctly multi-turn-aware, but its advantage over simpler turn- or length-based scheduling is not established. The results therefore demonstrate overall effectiveness without sufficiently identifying the claimed multi-turn-specific advance.

W2. The justification for entropy-aware scheduling is incomplete. Low entropy does not necessarily imply high acceptance of historical drafts, and the paper does not directly establish that previous-turn entropy predicts next-turn acceptance. Appendix B.3 also does not justify optimality or strict improvement over FCFS under variable request lengths and multi-turn dependencies. The ablation supports the heuristic’s usefulness, but not the stronger claims about its mechanism and guarantees.

Questions for authors

Q1. Could the authors clarify which technical advances specifically depend on multi-turn structure, addressing Weakness 1? In particular, what evidence distinguishes cross-turn reuse from other historical reuse mechanisms, and demonstrates that previous-turn entropy provides scheduling benefits beyond simpler turn- or length-based priorities?

Q2. The main results compare RANA against vanilla decoding, while the additional comparison covers EAGLE-3. Could the authors include representative speculative decoding approaches discussed in the related work, evaluated or adapted for the same multi-turn RL setting? This would help establish RANA’s contribution beyond existing rollout acceleration methods.

Q3. Could baseline + n-gram SD and baseline + EAGLE-3 be compared under matched hardware and serving conditions, with the controller and scheduler disabled for both? If feasible, please also compare baseline + full RANA against baseline + EAGLE-3 + the proposed scheduler and controller, to distinguish the drafting mechanism’s contribution from that of scheduling and load control.

Q4. Does previous-turn entropy reliably predict next-turn n-gram acceptance? Please clarify how Figure 2 aggregates entropy and handles terminated sessions, and specify the assumptions under which the optimality and strict-improvement claims in Appendix B.3 hold.

Reviewer_b95SOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RANA to accelerate multi-turn agentic RL rollouts. It observes three key characteristics in agentic RL rollouts: (1) Contextual Continuity (2) Hardware Regime Sensitivity (3) Entropy Decay. To leverage these attributes, RANA designs three corresponding components: N-gram Based Drafter, Load-Adaptive Controller and Entropy-Aware Scheduler to improve the efficiency.

Critical strengths
  1. This paper targets the long-tail effects in RL rollout phase, which is an important and practical problem. The three observed characteristics are well demonstrated and the paper designs corresponding components to address them.
  2. RANA uses model-free drafting approach and introduce entropy-aware scheduling strategy. On different benchmarks and models, RANA shows acceleration compared to vanilla baselines.
Critical weaknesses
  1. Missing baselines: The N-gram based drafter in RANA is very similar to the one in SRT[1]. Both use historical rollouts to construct tree-based caches, retrieve matching token suffixes and update the cache online. For now, RANA only compares with vanilla baselines and integrated EAGLE, which is not enough. Closely related retrieval-based speculative rollout methods should be added to baselines. Such comparison would be very necessary to establish the motivation and contribution of RANA.
  2. RANA proposes entropy-aware scheduler. It finds that average per-round token-level entropy decreases with more interaction round. This observation is interesting, but this paper does not directly validate the relationship between entropy and speculative acceptance rates, especially whether previous-turn entropy can reliably predict next-turn acceptance rates. Such experiments are critical to establish the motivation of entropy-aware scheduler. I would consider entropy-aware scheduler as the main novel module in RANA, since other components share much similarity with some prior works. Therefore, more ablations and evaluations on scheduler would be valuable to validate the effectiveness of entropy-aware scheduler, e.g. adding other simple heuristic baselines or adding other ablation settings. [1] Chi-Chih Chang et al. SRT: Accelerating Reinforcement Learning via Speculative Rollout with Tree-Structured Cache. NeurIPS 2025 Workshop on Efficient Reasoning,
Questions for authors

See weakness.

Reviewer_b5reOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RANA accelerates multi-turn agentic RL rollouts through history-based N-gram drafting, a load-adaptive speculation controller, and entropy-based scheduling. Experiments cover GSM8K, Maze, and BigCodeBench with PPO and GRPO rollout configurations. The paper reports an average rollout speedup of 2.09× and a maximum of 3.74×.

Critical strengths

The approach exploits a practical property of multi-turn workloads. Reusing previous actions provides lightweight drafts without maintaining an auxiliary model. Reported improvements span two main backbones and three tasks; for example, Qwen2.5-7B PPO Maze rollout time decreases from 415.97s to 111.24s.

The ablations show benefits from combining drafting and scheduling. On Qwen3-8B with GRPO on Maze, speculative decoding reduces rollout time from 1695.02s to 915.91s; adding the controller and scheduler further reduces it to 785.14s. Scheduler-only execution reaches 1583.89s, suggesting that its contribution interacts with speculation.

Critical weaknesses

Sampling correctness and theoretical guarantees require stronger justification. Evaluation uses temperature 0.7, but the verification description does not adequately specify how the target sampling distribution is preserved. Separately, Appendix B.1 bounds trie transitions using final output length without accounting for rejected drafts. The scheduling optimality argument also assumes an entropy–acceptance relationship without addressing variable execution times or dynamically arriving turns. These gaps undermine the stated correctness and complexity guarantees.

The evaluation does not sufficiently isolate the proposed advantages or establish preserved training quality. Close model-free baselines under identical scheduling are missing. EAGLE-3 uses different CUDA-graph and resource settings, complicating attribution of its performance gap to draft-policy mismatch. Moreover, the main results report rollout duration without corresponding policy-learning curves or final task quality, leaving the end-to-end RL benefit uncertain.

Questions for authors

What exact verification and sampling-correction procedure preserves the target distribution at temperature 0.7?

How does the complexity analysis account for rejected draft tokens, child selection, and online trie updates?

What gains remain over a simple N-gram baseline with the same controller, scheduler, and resource settings?

Are actual policy-update experiments available showing preserved reward or success rates? Are calibration and cache-maintenance costs included in the reported timings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.