- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a method for accelerating RL rollouts using weight-only 4-bit quantization. The authors observe that trajectories generated entirely with 4-bit weights tend to contain more tokens, which can offset the gains from faster decoding. To address this issue, they introduce a scheduler that dynamically switches from BF16 to 4-bit weights during rollout, guided by runtime estimates of response lengths and execution costs. Their experiments demonstrate reductions in rollout time.
Critical strengths
None
Critical weaknesses
- The impact on training quality is not sufficiently evaluated. The authors observe that W4 quantization substantially changes rollout behavior, as reflected in longer generated trajectories. This raises an essential concern whether the resulting distribution shift affects final model quality. Although the paper reports limited rollout rewards and held-out accuracy results, these evaluations do not establish that delaying the switch to W4 adequately mitigates this issue.
- The choice of W4A16 is not clear. Alternatives such as W8A16 or W8A8 might offer comparable speedups with less distortion of the rollout distribution. Without comparisons in both efficiency and learning quality, It is unclear why the authors focus exclusively on W4A16.
- The end-to-end speedups are modest, but keeping both BF16 and W4 weights on GPU incurs additional memory overhead.
Questions for authors
- Why did you choose weight-only 4-bit quantization over other quantization configurations?
Dear Reviewer 8k4i, Thank you for the review and the valuable comments. We address the concerns as follows.
W1. Impact on training quality. In RL, quantized rollouts do not always make training faster, because low precision lengthens responses. Tail-W4 recovers the speedup while keeping quality comparable to BF16.
W2 / Q1. Why W4A16 rather than W8A16 or W8A8? We chose W4A16 because it decodes fastest in the small-batch tail. W8 is also a good option, and it fits naturally into our framework.
W3. Modest speedups and extra memory. In our RL settings, the extra memory does not raise peak memory, so the speedup comes with little memory cost. The speedup is also what uniform W4 fails to deliver.