- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work investigates the impact of strategy distribution mismatch between quantized actors and learners on multi-round agent reinforcement learning training, and proposes InterQ-RL to alleviate this issue through quantization alignment, policy prefix correction, and correction strength allocation based on effective sample size. Experiments are conducted in three environments using different models for validation, and compared with BF16, QAT, and various quantization reinforcement learning baselines. The results show that InterQ-RL achieves the best quantization method endpoint performance in all four environment and model combinations, and performs close to or better than the BF16 baseline on multiple metrics.
Critical strengths
-
The research question is clear and has practical value. The goal of quantizing the rollout is to reduce the computational and memory costs of generation and KV Cache, but the low-precision executor simultaneously determines the behavior policy of the training data.
-
The method structure is clear, and the theoretical derivation has a certain completeness. Has a clear algorithmic definition, computational process, and verifiable optimization properties.
Critical weaknesses
-
The model and environment coverage of the experiments are still limited. Although the experiments include models of different scales from 4B to 30B and both dense and MoE architectures, the backbone models all belong to the Qwen3 series, the quantization settings focus on W8A8-KV8.
-
The paper takes reducing rollout computation cost as its research motivation and emphasizes that InterQ-RL does not require additional model forward computations. However, current experiments mainly compare task performance, training curves, divergence, and interaction rounds. The paper does not systematically quantify the tokens generated per second and rollout throughput, and the extra overhead from prefix weight statistics and the optimizer is not compared with QAT.
Questions for authors
see weakness