acceptodds
Under review as a conference paper at ICLR 2027

Auditing Dynamic Sensitivity-Selective Quantization For Low-Precision LLM Reinforcement Learning

Abstract

Reinforcement learning has become a critical post-training phase for large language models, where the rollout stage dominates the computational and memory overhead. Low-precision rollout reduces the cost of reinforcement learning for large language models but introduces a rollout-training mismatch that can destabilize optimization and even leads to collapse. Selective high-precision protection mitigates this mismatch, but the modules that benefit most from such protection can change as the policy evolves. Consequently, a protection set determined through one-time calibration may become suboptimal during training. We formulate selective protection as a time-varying allocation problem under a fixed high-precision budget, periodically assessing the sensitivity of each module to low-precision quantization and reconfiguring the protection strategy based on their current benefit-to-cost ratios. This allocation mechanism complements Quantization-Aware Training (QAT): while QAT adapts weights to low-precision computation, our method adapts precision assignments to the evolving module sensitivities. Experimental results demonstrate that combining dynamic allocation with QAT effectively controls the rollout-training mismatch throughout the evaluation window, whereas using QAT alone or QAT with static protection exhibits earlier signs of instability.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Low Precision, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2fAzOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes dynamic sensitivity-selective quantization (DYNSENS). It periodically changes which modules remain in BF16 during low-precision LLM RL training. The main contribution is adaptive precision allocation. This paper claims that the quantization sensitivity o modules changes during RL training, making a protection set is not enough. Experimental results show that it improves the training stability, compared with the baseline. The authors test the method with BF16 training with fake-QAT. Experiments uses Qwen3-8B, DAPO-17K on 8 H100 GPUs.

Critical strengths
  1. The problem of sensitivity changing across modules is a clear motivation.

  2. The method is concrete and can complement weight adaptation through QAT. Ranking benefit per protected parameter accounts for different module sizes.

  3. The paper reports results of several seeds, initialization situations, late instability, and implementation details. These are very helpful to understand the method and performance in details.

Critical weaknesses
  1. There is no final performance on benchmarks comparisons. The authors should at least finish training and testing a 7-8B size LLM and report the results with comparison to full-precision results and other quantized RL methods, like QeRL.

  2. There is no end-to-end efficiency benefit reported. Both rollout time, and GPU memory should be benchmarked in the paper.

  3. The sensitivity proxy is indirectly related to RL training. There is no evidence show that better sensitivity proxy leads to good final performance on benchmarks or less exploration during RL training.

Questions for authors

It would be much helpful if the authors could conducted more extensive experiments and comparisons with other quantized RL methods.

Reviewer_msDWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to allocate a fixed high-precision protection budget during low-precision LLM reinforcement learning. The proposed method periodically measures module sensitivity and updates the protected set using a benefit-to-cost ranking. The experiments use Qwen3-8B Base on DAPO-17K, with BF16 actor training and a separate W4A16 fake-QAT configuration. Dynamic allocation improves the reported inference-only stability and validation results. An active-set variant reduces probing work at selected reselection boundaries.

Critical strengths

1.The sensitivity measurements provide a concrete motivation for revisiting precision assignments during RL. Module rankings change between checkpoints, and allocating protection by benefit per parameter accounts for differences in module size. The resulting method is straightforward and addresses a practical allocation problem.

2.The inference-only results show substantial improvements across three seeds. At step 120, Dynamic reaches a mean signed mismatch of 0.217, compared with 2.871 for None and 3.114 for Static. Validation macro accuracy increases to 0.413, compared with 0.306 and 0.227. The per-seed trajectories and endpoint accounting also make the variability and later failures visible.

Critical weaknesses

1.Dynamic restarts every 20 steps, while Static runs continuously (Section 4.1 and Supplement C.2). The supplement does not state which training states are restored after each restart. A Static run with the same restart schedule would separate the effect of reselection from changes introduced by restarting. The segmented Infer-None baseline does not provide this same-budget comparison.

2.The paper does not compare sensitivity-guided selection with random reselection under the same budget and schedule. This comparison would test whether the sensitivity ranking chooses better modules, rather than whether periodically changing the protected set is sufficient to obtain the gains.

Questions for authors

1.How are optimizer, scheduler, RNG, and data-loader states handled at each restart? Does Dynamic retain its advantage over Static when both use the same segmented execution protocol?

2.With the protection budget and reselection schedule held fixed, how does sensitivity-guided selection compare with random reselection or a shuffled sensitivity ranking?

Reviewer_j2dbOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies dynamic allocation of high-precision protection during low-precision LLM reinforcement learning. The proposed method measures module-level sensitivity using teacher-forced probes and reallocates a fixed protection budget according to estimated benefit per parameter. The inference-only experiments show encouraging improvements. However, the current experimental results are not strong enough to support the proposed contribution.

Critical strengths
  1. Relevant motivation: Since policy weights and rollout distributions evolve during RL, reconsidering a precision assignment determined at initialization is a reasonable research direction.

  2. Encouraging empirical observations: In the inference-only setting, Dynamic achieves substantially better mean validation accuracy and lower mismatch at step 120 than Static or None.

  3. Transparent reporting: The paper reports individual seeds, late failures, missing runs, and implementation limitations.

Critical weaknesses
  1. The experiment results are not strong.

Section 4.1 states that Dynamic uses checkpoint/restart segments every 20 steps, whereas Static and QAT-None run continuously; Infer-None is also segmented. Consequently, the central Dynamic–Static comparison changes both the allocation strategy and the execution schedule. Restarting may affect random states, data ordering, quantization state, or other runtime behavior. The paper does not proof that these effects are negligible. A matched control in which Static follows the same checkpoint/restart procedure while retaining its original protection set is needed to attribute the observed gains to reallocation. Furthermore, the QAT branch jointly changes fake-quantization, fused-slice, and MBridge settings. These bundled changes limit conclusions about the specific interaction between QAT and dynamic allocation.

  1. The novelty is limited. The authors should compare with related work about the idea of dynamic precision.

Sensitivity-based mixed precision and dynamic precision assignment are established ideas. The potentially distinctive contribution here is their application to checkpoint-dependent allocation during RL under a fixed protection budget. The discussion should include QeRL, which studies NVFP4-based LLM RL, and DP-LLM, which studies changing quantization sensitivity and dynamic precision assignment during decoding. Their settings differ from this paper, but explaining those differences would clarify the methodological contribution.

  1. The presentation makes the central argument unnecessarily difficult to follow. The manuscript frequently switches between precision branches, selectors, evaluation horizons, and implementation qualifications. The roles of steps 120, 170, and 172 are eventually explained, but the reader must repeatedly reconstruct which comparison supports which claim. The addition of SAIR further disperses attention from the central hypothesis. I recommend organizing the experiments around explicit claims, with each claim linked to a controlled comparison and a clearly defined outcome. The main text should also identify the validation benchmarks and explain how macro accuracy is computed. Several essential details are deferred to a supplementary appendix that was not included in the material I reviewed, so my reproducibility assessment is provisional.
Questions for authors

The paper states that the experiments use H100 GPUs with a Marlin W4A16 execution path for NVFP4. Could the authors clarify the following points?

  1. Numerical implementation: How closely does the training-side fake quantization match the rollout-side NVFP4 weight quantization, including the quantization codebook, scaling factors, and rounding rules? Please specify the activation and accumulation precisions.

  2. Scope of the conclusions: Have the sensitivity rankings and stability improvements been validated under native W4A4 NVFP4 execution on Blackwell, where activation quantization introduces additional errors? If such experiments are unavailable, please explicitly delimit which findings are established for the evaluated W4A16 setting and which remain unverified for native W4A4 execution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.