- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work tries to address the mismatch between NVFP4 rollout and BF16 actor training. It adjusts two-sided truncated importance weights in log space, adapting the correction strength using historical mismatch and advantage-coefficient disagreement. A second controller uses persistent mismatch and degradation proxies to activate a sampled-token drift penalty and reduce the policy-gradient contribution, with a per-microbatch auxiliary-gradient budget. The method reuses quantities already computed during GRPO.
For experiments, it uses Qwen2.5-Omni-3B and Qwen3-8B, and reports slightly higher mean offline accuracies of 45.26% and 60.82%, versus BF16’s 44.72% and 58.47%, and a 1.17-times Qwen3 training speedup.
Critical strengths
- The proposed method demonstrates better performance than BF16 baselines, although not significantly.
Critical weaknesses
-
The evaluation or RL training is not robust. The 0.54-point Omni gain over BF16 is small, and the gain over TIS is concentrated on WeMath; on Qwen3, AIS exceeds ACU-RL by 1.39 points. The failure of AIS in one configuration does not establish a general stability advantage, especially when transferring its controller from FP8 to NVFP4.
-
The necessity of the coupled controller is not isolated. Figure 4 enables PG scaling and drift regularization together on Omni. It does not compare those components separately, apply the same control to ordinary TIS/AIS, or test a simpler reduced-learning-rate or fixed-regularization alternative.
Questions for authors
-
Do you repeat evaluation with different training seeds?
-
Does the proposed method lead to slower RL training?