acceptodds
Under review as a conference paper at ICLR 2027

Tail-W4: Predictive Precision Scheduling for the Long Tail of RL Rollouts

Abstract

Reinforcement learning (RL) post-training can be dominated by rollout, where synchronous execution waits for a few long responses to finish. Weight-only 4-bit quantization (W4) is promising for addressing those small-batch tails because it reduces the cost of loading model weights. However, naïvely applying W4 in RL post-training does not guarantee faster RL steps: quantized sampling can generate more tokens, increasing both decoding and downstream RL work. To address this challenge, we propose TAIL-W4, a rollout precision scheduler that chooses when to switch from BF16 to W4 by jointly predicting response lengths and execution costs. Reserving W4 for the tail can mitigate token inflation, but switching too early risks extra work and switching too late misses acceleration. Fixed schedules require repeated trials to tune and cannot fully adapt to changing training workloads. TAIL-W4 calibrates estimates of token inflation, updates them from training rollouts, and periodically predicts the benefit of switching to W4. On Qwen3.5-4B, TAIL-W4 achieves 1.38× rollout and 1.26× measured RL-step speedup over BF16, versus 1.12× and 1.03× for uniform W4 with the same quantized weights.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning Post Training, LLM Reasoning, Rollout Efficiency, Weight-Only Quantization LoRA, Long-tail Generation, Systems for Machien Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8k4iOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a method for accelerating RL rollouts using weight-only 4-bit quantization. The authors observe that trajectories generated entirely with 4-bit weights tend to contain more tokens, which can offset the gains from faster decoding. To address this issue, they introduce a scheduler that dynamically switches from BF16 to 4-bit weights during rollout, guided by runtime estimates of response lengths and execution costs. Their experiments demonstrate reductions in rollout time.

Critical strengths

None

Critical weaknesses
  1. The impact on training quality is not sufficiently evaluated. The authors observe that W4 quantization substantially changes rollout behavior, as reflected in longer generated trajectories. This raises an essential concern whether the resulting distribution shift affects final model quality. Although the paper reports limited rollout rewards and held-out accuracy results, these evaluations do not establish that delaying the switch to W4 adequately mitigates this issue.
  2. The choice of W4A16 is not clear. Alternatives such as W8A16 or W8A8 might offer comparable speedups with less distortion of the rollout distribution. Without comparisons in both efficiency and learning quality, It is unclear why the authors focus exclusively on W4A16.
  3. The end-to-end speedups are modest, but keeping both BF16 and W4 weights on GPU incurs additional memory overhead.
Questions for authors
  1. Why did you choose weight-only 4-bit quantization over other quantization configurations?
1 comment

Dear Reviewer 8k4i, Thank you for the review and the valuable comments. We address the concerns as follows.

W1. Impact on training quality. In RL, quantized rollouts do not always make training faster, because low precision lengthens responses. Tail-W4 recovers the speedup while keeping quality comparable to BF16.

  • Prior work. QeRL, QaRL, QuRL and Jet-RL show that RL with low-precision rollouts can train well when the rollout–training mismatch is handled. We build on this.
  • Smaller mismatch. When every token is rescored by BF16, Tail-W4 has 1/3.8 of uniform W4's TIS-truncated tokens.
  • Comparable rollout accuracy. On Eurus-2-RL, Tail-W4 is not significantly different from BF16 in two runs, while uniform W4 loses 8.9 points.
  • Comparable accuracy after training. On a held-out test (Phi-4-mini, 32 steps), Tail-W4 + TIS is comparable to BF16 (63.1% vs. 61.7%).

W2 / Q1. Why W4A16 rather than W8A16 or W8A8? We chose W4A16 because it decodes fastest in the small-batch tail. W8 is also a good option, and it fits naturally into our framework.

  • Why W4. With only a few requests left, decoding is limited by reading weights, so fewer weight bytes means faster steps. At batch 1–8, W4A16 is 1.33–1.54× faster than BF16 on 4B and 1.57–1.97× on 9B. W8A16 is 1.24–1.40× and 1.30–1.49× (vLLM with CUDA graphs, RTX PRO 6000).
  • W8 as an extension. W8 could be used for the tail, the prefix, or both, with the same cost model choosing when to switch. We will add it as future work.

W3. Modest speedups and extra memory. In our RL settings, the extra memory does not raise peak memory, so the speedup comes with little memory cost. The speedup is also what uniform W4 fails to deliver.

  • Memory. Peak memory is set by training (activations, gradients, optimizer states). The INT4 copy only uses KV-cache space inside the rollout's fixed budget and is offloaded during training (see our response to F2w1, W1-b).
  • Speedup. Uniform W4: 1.03× per step on 4B, 0.88× on 9B. Tail-W4: 1.26× and 1.09× (Table 1).
  • Uniform W4 also costs memory. It generates 20% more tokens on 4B, and longer responses keep more KV cache.
Reviewer_F2w1OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This work proposes scheduling a single BF16-to-W4 switch during synchronous PEFT rollouts. It models both decoding latency and quantization-induced response-length changes, including the downstream cost of processing extra tokens. It updates calibrated survival curves online. The implementation also optimizes the shared LoRA execution path.

For experiments, it evaluates three models, obtaining RL-Step speedup of 1.09-1.26 over BF16. A short Phi experiment reports comparable held-out accuracy at lower accumulated step cost. However, a separate longer Phi configuration reports lower held-out accuracy, and complete preparation and memory costs are not fully measured.

Critical strengths
  • The work is well-motivated, with good analysis of the issue.

  • The proposal can improve training efficiency with comparable performance. (239 vs. 302 seconds)

  • The paper provides an informative analysis of different parts of the model and designs LoRA packing and stream overlap. It also checks CPU/GPU overlap, data-parallel extensions, and fixed schedules.

Critical weaknesses
  • The full cost advantage over simple schedules remains incompletely measured. The 3.02-step preparation estimate combines earlier 9B profiling with current calibration and assumes BF16 training-rollout reuse. Those costs are outside the cumulative step timers. Both weight copies and graph sets remain resident, yet peak-memory and KV-capacity tradeoffs are not quantified. The expensive fixed-rule sweep is informative, but it is not a matched-budget comparison with a cheaply selected heuristic. These gaps affect the practical advantage in short or memory-constrained runs.

  • Comparison with some related works is missing: PMPD and FlexQuant. Not clear whether the proposed method is better than these existing works.

Questions for authors
  • Under identical data, training budgets, correction, and independent seeds, how do BF16, uniform W4, a fixed switch, and TAIL-W4 compare on common-precision held-out quality versus total cost? What explains or bounds the negative 48-step result in Appendix A.11?
2 comments

Dear Reviewer F2w1, Thank you for the review and the valuable comments. We address the concerns as follows.

W1-a. Preparation cost. Preparation takes about 30 minutes for 4B (measured on A100) and pays for itself after about 29 RL steps.

  • What it includes. Generating tail-W4 continuations (22 min) and timing decode latency (8 min). The BF16 calibration reuses the first few training rollouts.
  • Payback. Tail-W4 saves 63 s per step (302 s vs. 239 s; Table 1). Including preparation, the total speedup is about 1.23× after 256 steps and 1.25× after one epoch (934 steps).
  • One-time and shared. It is paid once per model and dataset, and reused across runs.

W1-b. Memory with both weight copies. In our RL settings, the second copy does not increase peak GPU memory, because peak memory is set by training, not rollout.

  • Training is generally the memory bottleneck. Training must store activations, gradients and optimizer states. Rollout and training take turns on the same GPU. During training, the rollout engine sleeps: its weights are offloaded and its KV cache is freed.
  • Rollout has a fixed memory budget. In our setup on A100 80GB, with both weight copies loaded, a rollout with 32 prompts uses at most 17% of the available KV cache, and no request was preempted.
  • If memory is tight (e.g., much larger models), we can load the INT4 copy only at the switch and offload the BF16 copy, since only one is needed at a time; this is a straightforward extension. The transfer can run in parallel with computation, and should be compatible with CUDA graphs.

W1-c. Matched budget. Tail-W4 is faster than fixed switch points chosen cheaply, and equal to the best one found after the fact, without trying any candidates.

  • Chosen from calibration only: 10.0% slower than Tail-W4.
  • Chosen by an 8-step pilot: 6.3% slower (4B, seed 42).
  • Best fixed switch found after the fact: equal to Tail-W4 (seeds 42 and 43).

W2. PMPD and FlexQuant. PMPD and FlexQuant are good inference methods, but they solve a different problem. We will add both to related work.

  • They only make decoding faster. In RL, low precision also makes responses longer, and every extra token must also be trained on. Neither paper measures output length.
  • This matters in RL. Uniform W4 makes the 4B rollout 1.12× faster, but it generates 20% more tokens, so the full RL step is only 1.03× faster. On 9B it is slower (0.88×; Table 1).
  • They can work with Tail-W4. Their precision choices could be used in the tail. Tail-W4 decides when to switch. Closest comparison. PMPD switches at a fixed position. This is our fixed-switch baseline: Tail-W4 is 6–10% faster than a fixed switch chosen cheaply, and equal to the best fixed switch found after the fact (W1-c).

Q1. Training quality. In RL, quantized rollouts do not always make training faster, because low precision lengthens responses. Tail-W4 recovers the speedup while keeping quality comparable to BF16.

  • Prior work. QeRL, QaRL, QuRL and Jet-RL show that RL with low-precision rollouts can train well when the rollout–training mismatch is handled. We build on this.
  • Smaller mismatch. When every token is rescored by BF16, Tail-W4 has 1/3.8 of uniform W4's TIS-truncated tokens.
  • Comparable rollout accuracy. On Eurus-2-RL, Tail-W4 is not significantly different from BF16 in two runs, while uniform W4 loses 8.9 points.
  • Comparable accuracy after training. On a held-out test (Phi-4-mini, 32 steps), Tail-W4 + TIS is comparable to BF16 (63.1% vs. 61.7%).

Q2. The 48-step result in Appendix A.11. Preserving training quality under quantized rollouts is addressed by prior work on quantized RL training (e.g., QeRL, QaRL). These techniques are orthogonal to our efficiency-focused method and can be combined with it, including in the longer setting of Appendix A.11.

Dear Reviewer F2w1,

We have uploaded a revised version of the paper; changes are marked in blue. PMPD and FlexQuant (W2). Related work (Appendix C) now discusses both, as promised in our response, and positions quantized RL training methods (QeRL, QaRL) as orthogonal to Tail-W4. Appendix A.11 (Q2). Appendix A.11 now notes that quantized RL training methods such as QeRL and QaRL can be combined with Tail-W4.

Reviewer_FnYHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper proposes TAIL-W4, an adaptive precision scheduling method for accelerating LLM RL training.

  • It dynamically switches from BF16 to INT4 decoding to balance inference speed, response length, and downstream training costs.

  • Experiments on Qwen3.5 and Phi-4-mini show improved end-to-end efficiency, including a 20.9% reduction in RL step time on Qwen3.5-4B.

Critical strengths
  1. Practical problem formulation: The paper identifies that INT4 quantization can increase response lengths and downstream costs, and optimizes end-to-end RL efficiency rather than decoding speed alone.

  2. Strong empirical validation: Experiments across multiple models demonstrate meaningful efficiency gains, including a 20.9% reduction in RL step time on Qwen3.5-4B.

Critical weaknesses
  1. Reliance on prediction accuracy: The switching policy depends on calibrated response-length and latency estimates. Prediction errors may lead to suboptimal switching decisions, and the robustness of these estimates under substantial policy or workload shifts remains unclear.

  2. Limited evaluation scope: Experiments focus primarily on a small set of models and GSM8K, which is a pretty outdated eval. Broader evaluation is needed to establish whether the efficiency gains and accuracy preservation generalize to other tasks and model scales.

Questions for authors
  1. How sensitive is TAIL-W4’s switching policy to errors in response-length predictions, especially when the RL policy evolves significantly during training? Does the paper evaluate how such prediction errors affect end-to-end speedup?

  2. Since the evaluation focuses primarily on GSM8K and a few small models, how well do the authors expect the method to generalize to other tasks and larger models, where quantization-induced token inflation and decoding bottlenecks may differ?

2 comments

Dear Reviewer FnYH, Thank you for the review and the valuable comments. We address the concerns as follows.

W1 / Q1. Robustness to prediction errors and policy drift. Tail-W4 is robust in our experiments: even without the online update, it is still faster than BF16 and uniform W4.

  • Without the online update. On 4B GSM8K, the speedup is 1.14× (1.26× with the update). Uniform W4 gives 1.03×.
  • The online update adapts during training. The switch point moves from 2.75K to 7.25K tokens, and all three seeds adapt (Section 5.4, Appendix A.8).
  • Long training. For very long runs, the scheduler can periodically fall back to a full-BF16 rollout to recalibrate. This costs little: with one BF16 step in ten, the 4B speedup would be about 1.23× instead of 1.26× (estimated from Table 1).

W2 / Q2. Generalization to other tasks. The token inflation is not specific to GSM8K. We additionally tested MATH-500 and Eurus-2-RL. On both datasets (4B), and on Eurus-2-RL with 9B, using W4 from the start makes responses about 1.15× longer than BF16, similar to GSM8K.

Dear Reviewer FnYH,

We have uploaded a revised version of the paper; changes are marked in blue. Other datasets (W2). The new Appendix A.13 reports response length under W4 on GSM8K and MATH-500 (Qwen3.5-4B) and on Eurus-2-RL (Qwen3.5-4B and 9B). Uniform W4 lengthens the mean response by 1.16–1.31× in every setting in the table, which motivates confining W4 to the tail of the response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.