acceptodds
Under review as a conference paper at ICLR 2027

ACU-RL: Adaptive Correction and Update Control for Low-Precision LLM Reinforcement Learning

Abstract

Low-precision rollout can reduce the dominant generation cost of reinforcement learning for large language models, but it also creates a policy mismatch between the quantized behavior policy and the BF16 learner. Existing truncated importance sampling (TIS) mitigates this mismatch by reweighting sampled tokens, yet it can strongly attenuate tokens favored by the quantized rollout policy. We find that relaxing this lower-side attenuation improves early learning but can amplify late-stage policy drift, revealing that sample correction and actor stabilization must be controlled jointly. To address this trade-off, we propose Adaptive Correction and Update Control for RL (ACU-RL), which couples Adaptive Two-Sided Truncated Importance Sampling (AT-TIS) with risk-gated actor-update control. AT-TIS adapts the correction strength while retaining more of the gradient contributions from tokens favored by the quantized rollout policy. Complementing this sample-side correction, risk-gated actor-update control increases drift regularization when persistent excess mismatch coincides with degradation signals and reduces the policy-gradient multiplier when the mismatch becomes more severe. ACU-RL requires no additional sampling or model forward passes. We evaluate ACU-RL on Qwen2.5-Omni-3B and Qwen3-8B-Instruct across six multimodal and four mathematical reasoning benchmarks, respectively. It achieves average accuracies of 45.26% and 60.82%, compared with 44.72% and 58.47% for BF16 RL. NVFP4 rollout further provides a end-to-end training speedup.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Low-Precision Rollouts, Rollout–Training Mismatch, Adaptive Importance Sampling, Training Stability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zCmqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work tries to address the mismatch between NVFP4 rollout and BF16 actor training. It adjusts two-sided truncated importance weights in log space, adapting the correction strength using historical mismatch and advantage-coefficient disagreement. A second controller uses persistent mismatch and degradation proxies to activate a sampled-token drift penalty and reduce the policy-gradient contribution, with a per-microbatch auxiliary-gradient budget. The method reuses quantities already computed during GRPO.

For experiments, it uses Qwen2.5-Omni-3B and Qwen3-8B, and reports slightly higher mean offline accuracies of 45.26% and 60.82%, versus BF16’s 44.72% and 58.47%, and a 1.17-times Qwen3 training speedup.

Critical strengths
  • The proposed method demonstrates better performance than BF16 baselines, although not significantly.
Critical weaknesses
  • The evaluation or RL training is not robust. The 0.54-point Omni gain over BF16 is small, and the gain over TIS is concentrated on WeMath; on Qwen3, AIS exceeds ACU-RL by 1.39 points. The failure of AIS in one configuration does not establish a general stability advantage, especially when transferring its controller from FP8 to NVFP4.

  • The necessity of the coupled controller is not isolated. Figure 4 enables PG scaling and drift regularization together on Omni. It does not compare those components separately, apply the same control to ordinary TIS/AIS, or test a simpler reduced-learning-rate or fixed-regularization alternative.

Questions for authors
  • Do you repeat evaluation with different training seeds?

  • Does the proposed method lead to slower RL training?

Reviewer_iRFLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ACU-RL combines two responses to the mismatch between NVFP4 rollouts and BF16 actor training. AT-TIS bounds and tempers the importance weights, adjusting correction strength from mismatch and advantage statistics. A separate controller scales the policy gradient and penalizes actor-to-rollout log-probability drift when mismatch persists alongside degradation signals. The paper evaluates Omni-3B on multimodal tasks and Qwen3-8B on math tasks, and reports a 1.17x training speedup for the Qwen3 setup. ACU-RL has the highest multimodal average, while AIS has the highest math average. The main claim is therefore more consistent training across the two settings.

Critical strengths

The lower-weight experiments give a useful reason to study correction and update control together. Raising the floor helps early learning, but can be followed by substantial degradation. Figure 4 then shows that adding update control to AT-TIS reduces the observed drops. This is a worthwhile practical observation about low-precision RL, where an importance-weight rule alone may not be enough.

Testing both multimodal and math training is helpful, as is using BF16 for the final offline evaluation. The paper reports the stronger AIS result on Qwen3 openly, so the reader can see the trade-off. The controller is described in enough detail to examine its behavior, including the important qualification that the gradient budget is in output space.

Critical weaknesses

There are several moving parts in the controller, and the ablation does not yet explain which ones matter. Figure 4 turns PG scaling and drift regularization on together, on Omni. It does not separate their effects or show whether a lower learning rate, fixed regularization, or the same update control added to TIS/AIS would work similarly. I therefore find the case for update damping stronger than the case for this particular set of risk gates. The sensitivity of the calibration and gating settings in Appendix B is also unclear.

The robustness claim needs repeated training runs. Table 1 gives point estimates, and the curves do not show seed variation. The Omni gain over BF16 is only 0.54 points, while AIS is 1.39 points ahead on Qwen3. Those numbers are compatible with a useful method, but they do not establish a reliable cross-setting advantage on their own. The AIS configuration in Appendix C is another concern: it is chosen to favor stronger correction, leaving the mismatch gate saturated and the variance penalty inactive on Omni. A comparison with validation-tuned AIS would help determine how specific that failure is to the chosen setting.

Questions for authors
  1. What happens with PG scaling alone, drift regularization alone, and the full controller? How do these compare with a reduced learning rate or fixed regularization, and with adding update control to TIS/AIS?

  2. How many independent training runs support the main results and Figure 4? Please report variation in final accuracy and in the timing or occurrence of reward drops.

  3. How were the ACU-RL thresholds and AIS settings chosen? Were they fixed using a comparable validation budget? In particular, does AIS still fail on Omni with standard or validation-tuned settings, and do the ACU-RL settings transfer between the two tasks without retuning?

Reviewer_eKfYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ACU-RL addresses instability from NVFP4 rollouts with a BF16 learner. It combines adaptive, two-sided importance weighting with risk-gated gradient scaling and policy-drift regularization. Experiments on multimodal and mathematical reasoning show improved stability relative to fixed correction, with a reported 1.17× end-to-end speedup.

Critical strengths

The paper identifies a useful trade-off: preserving low-ratio token contributions can help early learning while worsening later instability. This motivates controlling both correction and actor updates.

Evaluation covers two different task settings and includes AIS, which performs differently across them. The results support the practical value of seeking stability across settings.

Critical weaknesses

The controller combines numerous thresholds and diagnostics, but the ablation enables gradient scaling and drift regularization together. Their individual necessity—and the advantage over simpler update control—remains unclear.

Independent training-seed uncertainty is not reported. The Omni gain over BF16 is only 0.54 points, while AIS outperforms ACU-RL on Qwen3, making the robustness claim more compelling than a general accuracy advantage.

Questions for authors

What happens when gradient scaling and drift regularization are enabled separately, and when the same controls are added to TIS or AIS?

How sensitive are results to the calibration window and intervention thresholds, and were these settings fixed before evaluating both models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.