acceptodds
Under review as a conference paper at ICLR 2027

TRIAGE: Direction-Aware Mismatch Stabilization for Native NVFP4 Reinforcement Learning

Abstract

Low-precision execution can substantially accelerate reinforcement learning (RL) for large language models, but discrepancies between learner and sampler execution can destabilize policy optimization. In this paper, we characterize the interaction between mismatch and the policy-gradient direction, distinguishing locally amplifying from contracting update contributions that mismatch magnitude alone cannot identify. In native NVFP4 runs, we observe an early imbalance between the two amplifying regions, favoring negative-advantage, negative-gap updates. Their tail tokens become concentrated in a small fraction of response segments before mismatch spreads globally. Motivated by these findings, we introduce TRIAGE, a direction-aware stabilization method that uses segment-level diagnosis to selectively rebalance policy-gradient updates and applies bounded repair to residual severe mismatch. TRIAGE modifies the optimization objective while retaining native NVFP4 weight-and-activation 4-bit (W4A4) forward execution on both the sampler and learner. Experiments on Qwen3-4B and Qwen3-30B-A3B show stable optimization throughout the evaluated training horizon and achieve full precision level performance across five mathematical reasoning benchmarks, while native NVFP4 with TRIAGE provides up to 2.3 higher rollout throughput than BF16. Our code is available at https://anonymous.4open.science/r/TRIAGE.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Quantization, Reinforcement Learning, LLMs

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZX3YOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates training instability caused by learner–sampler log-probability mismatch δi,t=log⁡πl(yi,t∣x,yi,<t)−log⁡πs(yi,t∣x,yi,<t)\delta _ {i,t} = \log \pi _ l(y _ {i,t} \mid x, y _ {i,<t}) - \log \pi _ s(y _ {i,t} \mid x, y _ {i,<t}) (Eq. 2) during native NVFP4 (W4A4, Eq. 4) reinforcement learning for large language models. In Section 3.1 (Eqs. 5–9, Appendix B Eqs. 21–34), a first-order Taylor expansion of the sampled-token mismatch energy D=12∑i,tδi,t2D = \frac{1}{2}\sum _ {i,t}\delta _ {i,t}^2 under a fixed sampler decomposes the single-step change into ΔD=η∑i,tκi,tδi,tgi,t+ηRcross+O(η2)\Delta D = \eta\sum _ {i,t}\kappa _ {i,t}\delta _ {i,t} g _ {i,t} + \eta R _ {\mathrm{cross}} + \mathcal{O}(\eta^2), motivating two locally amplifying sign quadrants: A−={Ai<0,δi,t<0}\mathcal{A}^- = \lbrace A _ i < 0, \delta _ {i,t} < 0 \rbrace and A+={Ai>0,δi,t>0}\mathcal{A}^+ = \lbrace A _ i > 0, \delta _ {i,t} > 0 \rbrace. Based on trajectory diagnostics showing pre-collapse growth in raw advantage-weighted A−/A+\mathcal{A}^- / \mathcal{A}^+ occupancy (Section 3.2, Eq. 10, Figure 2) and concentration of severe negative-gap tokens within 64-token segments (Section 3.3, Eq. 11, Figure 3), Section 4 proposes TRIAGE (Figure 1, Eqs. 12–20): (1) a 64-token Segment Gate (Eqs. 13–17) that downweights A−\mathcal{A}^- tokens in tail-heavy segments (wS∈[0.1,1]w _ S \in [0.1, 1]) and renormalizes weights within each response (αi,t=wi,t/wˉi\alpha _ {i,t} = w _ {i,t}/\bar{w} _ i), and (2) a 128-token Positive-Advantage Repair penalty (Eqs. 18–20) using a one-sided pseudo-Huber loss ϕβ([τrep−δˉR]+)\phi _ \beta([\tau _ {\mathrm{rep}} - \bar{\delta} _ R] _ +) when Ai>0A _ i > 0 and δˉR<−0.03\bar{\delta} _ R < -0.03.

Evaluated on GRPO training of Qwen3-4B-Base (600 steps, MLP projections in NVFP4) and Qwen3-30B-A3B-Base (1,700 steps, routed expert MLPs in NVFP4) on DAPO-Math-17K across five math benchmarks (AIME24, AIME25, MATH500, AMC23, GSM8K, Table 1), TRIAGE prevents the collapse seen in naive NVFP4 (at step 304 on 4B and step 724 on 30B), achieving average accuracy of 58.49% vs. 58.26% for BF16 on 4B and 70.96% vs. 72.41% for BF16 on 30B, alongside up to 2.30x rollout throughput and 1.30x end-to-end iteration speedup on 30B (Table 2). While the directional formulation and native NVFP4 systems effort are valuable, I lean toward weak rejection due to an explicit mismatch between the paper's response-wise objective (Eqs. 16–17) and the global token normalization implemented in the supplementary code, negative mechanistic audit results on the primary 30B model in the appendix (Tables 5 and 8b), checkpoint-inherited ablations that start at step 300 (Figure 5), and backend confounds in the headline throughput numbers.

Critical strengths

1. Insightful directional and segment-localized diagnosis of low-precision RL mismatch.
By decomposing ΔD\Delta D into diagonal current-gradient terms κi,tδi,tgi,t\kappa _ {i,t}\delta _ {i,t} g _ {i,t} and off-diagonal couplings (Eqs. 6–8) and tracking 64-token segment statistics across training stages (Figures 2–3), the paper moves beyond simple scalar KL or importance-ratio thresholds and shows how localized negative-gap bursts can accumulate inside otherwise normal-looking long rollouts.

2. Large-scale native NVFP4 GRPO evaluation on both dense (4B) and MoE (30B-A3B) models with transparent appendix auditing.
Maintaining native NVFP4 execution on both the sampler and learner MLP/expert forward paths over 600 steps (4B, up to 16k tokens) and 1,700 steps (30B-A3B, up to 20k tokens) on B300 hardware is a nontrivial systems achievement, and the authors deserve credit for including thorough negative audits and null-model permutations in Appendices C and F.

Critical weaknesses

1. Unresolved discrepancy between the paper's response-wise objective (Eqs. 16–17) and the released global token-normalization implementation, alongside negative mechanistic evidence on 30B.
As detailed under Soundness, the supplementary codebase (docs/PAPER_AND_CODE.md lines 32–40, policy_extension/TRIAGE.py lines 817–878) optimizes a global weighted-token average across all DP ranks and micro-batches rather than the per-response weighted mean in Eqs. (16)–(17). Under global token normalization, gating a segment in a long response changes the relative weight of other responses in the batch rather than strictly reallocating weight αi,t=wi,t/wˉi\alpha _ {i,t} = w _ {i,t}/\bar{w} _ i within response ii, and also alters length normalization (1/ni1/n _ i). Compounding this, on the flagship Qwen3-30B-A3B model, the diagonal directional predictor has only 25.27% weighted sign agreement with actual token motion (Table 5), and the effective length-normalized pre-terminal A−/A+\mathcal{A}^-/\mathcal{A}^+ ratio is just 1.049 (Table 8b). Consequently, it is unclear both which mathematical objective produced Table 1 / Figures 4–5 and why the method stabilizes 30B when the local directional assumptions fail on 30B.

2. Confounded step-300 branched ablations (Figure 5), single-seed trajectories, large benchmark trade-offs on 30B (-10.32 pp on AMC23), and backend-confounded throughput claims.

  • Step-300 checkpoint inheritance in Figure 5 (Section 5.4, p. 9): The only component ablation separating Gate and Repair (TRIAGE, Gate+TIS, TIS, NVFP4) is run solely on 4B and branches all four variants from a step-300 checkpoint that was already trained with full TRIAGE (and omits a Repair+TIS branch as well as final benchmark accuracy numbers). This does not test whether Gate or Repair can stabilize training from scratch (step 0).
  • Missing quantized-RL baselines, and 30B benchmark trade-offs: Closely related methods discussed in Section 6 (QaRL, QUADS, ACRL, FP8-RL) are not compared (and TIS is only tested at C=2.0C=2.0); and on 30B (Table 1), NVFP4+TRIAGE drops by -10.32 percentage points relative to BF16 on AMC23 (77.72% vs. 88.04%), offsetting that drop in the unweighted average only via a +4.66 pp gain on GSM8K (95.47% vs. 90.81%).
  • Backend and output-length confounds in the 2.30x throughput claim (Table 2 vs. Appendix Table 10 & examples/common.sh): In examples/common.sh (lines 71–83), nvfp4 runs with SGLANG_MOE_RUNNER_BACKEND=flashinfer_trtllm whereas bf16 defaults to triton, and Table 2 measures natural-EOS generation where output lengths vary across runs. When both NVFP4 and BF16 use the same FlashInfer TRT-LLM MoE backend at a fixed 512×4096512 \times 4096 output length in Appendix F.3.1 (Table 10, p. 28), the standalone 30B rollout speedup is only 1.175x (18,175.67 vs. 15,470.01 tokens/s)—and on dense 4B (benchmarks/README.md), only 1.014x–1.026x.
Questions for authors
  1. Assessment-critical (Reconciling the paper's response-wise objective in Eqs. 16–17 with the code's global token normalization):
    Which exact loss reduction produced the results in Table 1 and Figures 4–5: the per-response weighted mean in Eqs. (16)–(17) or the global weighted-token normalization across all DP ranks and micro-batches in policy_extension/TRIAGE.py (docs/PAPER_AND_CODE.md lines 32–40)? Also, were the main runs executed with GATE_SCOPE="q3" and REPAIR_SCOPE="adv_positive" (RECENTER_ENABLE=0) as written in Section 4, or with the code defaults in triage_config.py ("whole" and RECENTER_ENABLE=True)?

  2. Assessment-critical (From-scratch ablations and mechanism on Qwen3-30B-A3B):

    • Can you provide from-scratch (step 0) ablations comparing NVFP4+TIS, Gate+TIS, Repair+TIS, and Full TRIAGE (with final five-benchmark evaluation scores and ≥2\ge 2 seeds on 4B), rather than branching from the step-300 TRIAGE checkpoint as in Figure 5?
    • Given that on 30B the diagonal sign agreement is only 25.27% (Table 5) and the length-normalized pre-terminal A−/A+\mathcal{A}^-/\mathcal{A}^+ ratio is 1.049 (Table 8b), what accounts for TRIAGE's stabilizing effect on 30B, and why does 30B NVFP4+TRIAGE lose 10.32 percentage points on AMC23 relative to BF16 in Table 1?
  3. Clarification (MATH-500 92.65% denominator and matched-backend iteration speedup):

    • How was 92.65% obtained for 30B BF16 on MATH-500 in Table 1 under 500 problems ×\times 6 binary-scored samples (3,000 samples, where 2779/3000=2779/3000 = 92.63% and 2780/3000=2780/3000 = 92.67%)?
    • In Table 2, what are the 30B rollout throughput and end-to-end iteration speedups when BF16 and NVFP4+TRIAGE both use the exact same FlashInfer MoE runner backend (rather than triton for BF16 vs. flashinfer_trtllm for NVFP4 in examples/common.sh)?
2 comments
AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer ZX3Y,

Thank you again for your time and effort in reviewing our work. We have posted our point-by-point responses and additional experimental results under your review. As the discussion period closes within the next two days, we would appreciate any feedback on whether our responses address your concerns, or any remaining questions we can clarify.

Best regards,

The TRIAGE Authors

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer ZX3Y,

We wanted to send a brief follow-up as the author-reviewer discussion period closes in less than 24 hours. Whenever you have a moment, we would greatly appreciate it if you could look over our point-by-point responses and additional experimental results, and let us know if our revisions address your concerns or any remaining questions that we can clarify.

Thank you very much for your time and consideration.

Best regards,

The TRIAGE Authors

Reviewer_NNNXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces a new method called TRIAGE to stablize low-precision RL training. This method is inspired by analysis on the interaction of the policy-gradient direction and the train-inference mismatch. The authors distinguished two cases where the gradient amplifies this mismatch, and separately designed methods to rebalance the gradient update and control the mismatch, and hence maintain the training stability. TRIAGE used segment-level reweighing and weight gating on negative samples, and solely weight gating on positive samples to reduce potential mismatch. In the following experiments they further validated the stability and efficiency of RL with this correction.

Critical strengths
  • Stablizing low-precision RL training is an important topic in modern LLMs for its practical value. The topic studied in this paper is close to real application.

  • The work demonstrates solid engineering capabilities in the relevant experiments, and the presentation of the paper also reflects a sufficient level of care and rigor.

Critical weaknesses
  • The final algorithm design appears to rely heavily on heuristic choices and manual intervention. Its effectiveness lacks theoretical guarantees, and the overall design is not particularly elegant. As a result, the performance of the proposed algorithm relies largely on empirical validation. However, the amount of experimental evidence provided, both in the main text and in the appendix, remains insufficient. The current results do not convincingly establish the effectiveness or generalizability of the proposed method, nor do they rule out the possibility that certain model or algorithmic choices were made post hoc based on observed experimental performance, potentially leading to a form of experimental overfitting or "hacking".

  • The insight behind TRIAGE is to control the self-amplification of train-inference mismatch, but with this insight, a more direct method would be introducing a train-inference KL penalty with a proper strength. This work lacks comparison with this direct baseline and other baselines as well.

Questions for authors
  • Compared with the train–inference KL-constraint approach discussed in point 2 in weaknesses, where do you see the main advantages of TRIAGE? Can these advantages be empirically validated through additional experiments?
2 comments
AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer NNNX,

Thank you again for your time and effort in reviewing our work. We have posted our point-by-point responses and additional experimental results under your review. As the discussion period closes within the next two days, we would appreciate any feedback on whether our responses address your concerns, or any remaining questions we can clarify.

Best regards,

The TRIAGE Authors

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer NNNX,

We wanted to send a brief follow-up as the author-reviewer discussion period closes in less than 24 hours. Whenever you have a moment, we would greatly appreciate it if you could look over our point-by-point responses and additional experimental results, and let us know if our revisions address your concerns or any remaining questions that we can clarify.

Thank you very much for your time and consideration.

Best regards,

The TRIAGE Authors

Reviewer_rqKEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates learner–sampler policy mismatch and training instability in large language model reinforcement learning with native NVFP4 execution. Motivated by the interaction between mismatch and policy-update direction, the authors propose TRIAGE, which combines segment-level risk diagnosis, direction-selective gating, within-response gradient reweighting, and bounded repair. The method retains native W4A4 forward execution on both the learner and sampler for the quantized layers. Experiments on Qwen3-4B and Qwen3-30B-A3B demonstrate stable optimization over the evaluated horizons and average performance close to BF16 across five mathematical reasoning benchmarks. The paper reports up to 2.30× higher rollout throughput and 1.30× end-to-end iteration speedup. The work offers a practically useful approach, although its generalization across settings, hyperparameter robustness, and training efficiency at matched performance require further validation.

Critical strengths
  1. Low-precision execution can reduce the computational cost of LLM reinforcement learning, but learner–sampler mismatch can compromise training stability. TRIAGE directly addresses this challenge through an objective-level intervention that preserves native low-precision execution.

  2. Considering mismatch magnitude together with the update direction determined by advantage provides an insightful perspective on low-precision RL instability. Segment-level diagnosis further allows the method to identify localized risk and motivates selective intervention.

  3. TRIAGE improves the stability of native NVFP4 training on both a dense model and an MoE model, while achieving better average benchmark performance than the tested TIS configuration. Reporting both rollout throughput and end-to-end iteration time also helps distinguish inference acceleration from improvements to the complete RL iteration.

Critical weaknesses
  1. The experimental coverage is limited, leaving generalization across settings insufficiently established.
    All training experiments use mathematical reasoning data, both models belong to the Qwen3 family, and policy optimization is restricted to GRPO. Although the dense and MoE models provide some architectural diversity, the current evidence does not establish whether the observed mismatch patterns and TRIAGE’s benefits extend to other tasks, model families, or optimization objectives.

  2. Sensitivity analysis for key hyperparameters is missing.
    Table 9 specifies fixed values for several gating thresholds, attenuation weights, segment lengths, and repair parameters, but their effects on training stability and final accuracy are not sufficiently evaluated. The appendix’s segment-length analysis primarily concerns mismatch statistics and does not directly establish the training method’s robustness to these choices. Consequently, it remains unclear whether TRIAGE works reliably over a reasonable parameter range or depends on a particular configuration.

  3. The efficiency evaluation does not compare training time to matched performance.
    Table 2 measures average iteration time over ten post-warm-up iterations starting from the base checkpoint, whereas Table 1 primarily reports accuracy at specified training steps. Lower per-step cost does not necessarily imply less time to reach the same accuracy: if BF16 reaches the accuracy attained by NVFP4+TRIAGE in fewer updates, faster convergence could offset its higher iteration cost. Although BF16 achieves higher final average accuracy on 30B, final accuracy alone does not establish which method reaches a shared target sooner. The current evidence therefore supports up to 1.30× iteration speedup, but does not establish training acceleration at matched performance. The authors should compare cumulative end-to-end time to predefined accuracy targets achievable by both methods under the same evaluation protocol, and report accuracy against cumulative training time.

Questions for authors

see above

3 comments
AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer rqKE,

Thank you again for your time and effort in reviewing our work. We have posted our point-by-point responses and additional experimental results under your review. As the discussion period closes within the next two days, we would appreciate any feedback on whether our responses address your concerns, or any remaining questions we can clarify.

Best regards,

The TRIAGE Authors

Thank you for your responses.

However, we still have some doubts regarding the experiment of the first question. Our original intention is to observe the results after training and evaluation on different tasks separately, in order to demonstrate that this method is effective for various data tasks. This is because different tasks in reinforcement learning will affect the output length, training difficulty, hyperparameters, etc., which in turn will impact the training stability.

You only trained on mathematical tasks, but evaluated on OOD datasets such as Code, GPQA, and MMLU. We believe this is an unreasonable evaluation method, and it can be seen that many of these evaluation scores are much lower than those of the original Qwen model in the technical report. We hope to see new experiments and solutions in your updated version.

We have already understood the latter two issues and have decided to raise the score. Good Luck!

Dear Reviewer rqKE,

Thank you for revisiting our responses and clarifying the remaining concern. We also appreciate your consideration of our explanations regarding hyperparameters and training efficiency.

  • Regarding independent RL training on different tasks, we agree with your suggestion. Response length and task difficulty can indeed affect training stability. Our current experiments focus on demanding mathematical reasoning tasks, with evaluations on challenging benchmarks such as AIME and a maximum response length of 20k tokens for training from Base checkpoints. TRIAGE maintains stable training through 600 updates on 4B and 1,700 updates on 30B (Figure 4). Your suggested experiments would extend this validation to additional RL training domains. These experiments require additional data preparation, verification infrastructure, and complete training runs, which we could not finish within the available rebuttal time and GPU budget. We will prioritize dedicated coding-RL training and evaluation, together with extensions to other model families, in a subsequent version.

  • Regarding the existing OOD evaluations, we would like to clarify their purpose. These experiments assess whether, under a common mathematical RL training setup, models trained with TRIAGE achieve downstream performance comparable to their BF16-trained counterparts beyond the training task. Both methods start from the same Base checkpoint and are evaluated under the same protocol, providing a direct comparison of the resulting model quality. Across the four additional benchmarks, TRIAGE achieves an average accuracy of 61.97% versus 58.70% for BF16 on 30B, and 46.34% versus 46.38% on 4B. These results demonstrate competitive performance relative to BF16 on broader downstream tasks, complementing the independent task-specific training experiments you suggest.

  • Regarding the post-trained model results in the Qwen technical report, a key distinction is that our experiments apply mathematical RL starting from Qwen3-4B-Base and Qwen3-30B-A3B-Base, whereas the official post-trained 4B and 30B-A3B models undergo off-policy and on-policy distillation from stronger teacher models, as described in Section 4.5 [1]. The officially reported scores therefore reflect a different post-training pipeline, while our BF16-trained counterparts provide the direct reference for assessing TRIAGE’s training outcomes.

Thank you again for identifying this valuable direction and for your thoughtful engagement throughout the discussion. We will carefully incorporate your suggestions into our continued experimental work.

With gratitude and best regards,

The TRIAGE Authors

[1]. Qwen Team, Qwen3 Technical Report, Section 4.5.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions

Dear Reviewers and Area Chair,

Thank you for your thorough reviews and constructive suggestions. We have updated the main text and appendix and provided additional supplementary materials to further strengthen the empirical evidence for TRIAGE regarding training stability, downstream task performance, and practical training

For your convenience, we summarize the key additions and their locations below.

  1. More Comprehensive End-to-End Efficiency Evaluation and Weight-Update Optimization. We added a stage-wise breakdown of a full RL iteration, comparisons implementation details for batched quantization and weight-update pipelining. On Qwen3-30B-A3B-Base, TRIAGE achieves a 1.46–1.51× end-to-end speedup and 2.33–2.34× rollout throughput, while incurring a median learner-update overhead of only 0.84%. Furthermore, since the Base model does not initially possess long-CoT reasoning capabilities, its response length increases steadily as RL training progresses. For long-horizon reasoning or agentic workloads where generation dominates runtime, while rollout acceleration holds, the per-iteration weight-update cost is amortized over a larger number of generated tokens, further amplifying the end-to-end gains. (Section 5.3, Table 2, Appendix F.1–F.2, Table 13, and Figure 11)

  2. Component Ablations Starting from the Base Model and Final Benchmark Evaluations. We added comparisons of Gate+TIS, Repair+TIS, and full TRIAGE trained from scratch (step 0), along with the final benchmark results for the original step-300 branched ablations. (Section 5.4, Figures 5–6, and Table 3)

  3. Broader Out-of-Domain Evaluations. We expanded our evaluation suite to include LiveCodeBench, OlympiadBench, GPQA-Diamond, and MMLU, covering code generation, competition-level reasoning, scientific QA, and general knowledge.

  4. Comparison with Direct Learner–Sampler KL Regularization. We added experiments across three KL coefficient settings, providing the complete objective formulation, training curves, and downstream evaluations. Although that KL configuration constrains the mean absolute log-probability gap to a level comparable to TRIAGE, it still yields lower late-stage rewards and final task performance, which further validating our design of selective intervention conditioned on both reward direction and local risk. (Appendix E.4, Table 12, and Figure 10)

  5. QaRL and ACRL Baselines and Reproducibility Code. Within a unified native NVFP4 training stack, we added algorithmic component comparisons against QaRL and ACRL, complete with training curves, checkpoint evaluations, concrete implementations, and launch scripts. (See training/code and training/training_curve in the supplementary material)

  6. Clearer Appendix Navigation. We added a table of contents for the appendix to help readers quickly locate method definitions, experimental setups, additional evaluations, and efficiency analyses.

  7. Others. For additional non-overlapping questions and points requiring individual clarification raised by the three reviewers, we have conducted supplementary experiments or provided detailed explanations accordingly. Please refer to our individual responses to each reviewer below for details.

Thank you again for your valuable feedback, I think these suggestions has helped us substantially improve our experimental coverage and presentation of results. Please let me know if your have furrher questions, we would be happy to provide additional clarification and engage in further discussion.

Best regards,

The TRIAGE Authors

AuthorsOpenReview ↗

TRIAGE: Author Summary of Key Results and Revisions

Dear Reviewers, Area Chair, and Senior Area Chair,

Good day! Thank you for your time and effort in reviewing our submission (TRIAGE: Direction-Aware Mismatch Stabilization for Native NVFP4 Reinforcement Learning).

The reviewers recognized the practical importance of stabilizing low-precision RL (rqKE, NNNX), the value of analyzing learner–sampler mismatch through update direction and segment-level diagnosis (rqKE, ZX3Y), and the engineering work behind the native NVFP4 RL experiments (NNNX, ZX3Y). The main questions concerned how TRIAGE compares with alternative methods, how sensitive it is to hyperparameter choices, how well it generalizes and how much TRIAGE improves end-to-end training efficiency. As the discussion concludes, we summarize the evidence from the revised manuscript and our responses below.


  1. TRIAGE sustains stable training on both dense and MoE models while recovering near-BF16 quality. Naive NVFP4 becomes unstable after approximately 300 updates on 4B and 700 on 30B; TRIAGE completes 600 and 1,700 updates, respectively, with stable training. This stability translates into substantial improvements in model quality: on 4B, average accuracy across the five benchmarks reaches 58.49%, compared with 58.26% for BF16; on 30B, it reaches 71.88%, only 0.37 percentage points below BF16 and 8.08 points above NVFP4+TIS. TRIAGE turns native NVFP4 execution into sustained, effective RL training. (Figure 4, Table 1.)

  1. Direction-aware diagnosis delivers learning benefits beyond average mismatch control. Following NNNX’s suggestion, we compared direct learner–sampler KL regularization at three strengths using the same native NVFP4 execution path. TRIAGE achieves 58.49% average accuracy across the five benchmarks, outperforming the best tested KL configuration’s 55.78%, with higher scores on all five benchmarks. That KL configuration maintains a mean absolute log-probability gap comparable to TRIAGE’s, yet yields lower late-training reward and final accuracy. This comparison supports TRIAGE’s central insight: learning performance also depends on how mismatch interacts with the update direction and local risk, which its average magnitude alone cannot capture. We also added comparisons with QaRL and ACRL algorithmic components within the same NVFP4 training stack, together with implementation details for reproducibility. (Appendix E.4, Table 12, Figure 10; supplementary code.)

  1. Ablations from step 0 and reuse of a fixed configuration validate component contributions and method reusability. Starting from the same base checkpoint, full TRIAGE achieves 58.49% average accuracy, compared with 50.10% for Gate+TIS and 47.51% for Repair+TIS, supporting the complementary roles of the two components throughout training. All TRIAGE-specific hyperparameters were selected on 4B and frozen before the 30B experiments; 30B directly reused these configurations without retuning. We also reported training-level sensitivity comparisons. Additional downstream evaluations on LiveCodeBench, OlympiadBench, GPQA-Diamond, and MMLU yield averages of 61.97% versus 58.70% for BF16 on 30B, and 46.34% versus 46.38% on 4B. (Section 5.4, Figure 6, Table 3; Appendix E.3, Table 11; response to rqKE W2.)

  1. TRIAGE achieves stable native NVFP4 training with minimal incremental overhead while preserving end-to-end gains. TRIAGE’s incremental learner-update overhead is just 0.84%. With optimized weight updates design, the complete NVFP4 training pipeline achieves 2.33–2.34× rollout throughput and 1.46–1.51× end-to-end iteration speedup over the original BF16 triton baseline. These end-to-end gains account for learner updates, weight conversion and synchronization, and orchestration, demonstrating the practical training value of stable low-precision RL. (Section 5.3, Table 2, Appendix F.)

We have also verified the main training objective and runtime configurations, corrected inconsistencies in the released implementation and evaluation protocol, and updated the corresponding code and manuscript results. (Responses to ZX3Y W1 and Q3.1.)

Validated through extensive experiments, TRIAGE delivers both scientific and practical contributions: new insight into low-precision RL instability, an effective method for stable native NVFP4 training, and tangible end-to-end efficiency gains. The additional baselines, ablations, transfer of a fixed configuration across dense and MoE architectures, and full-iteration measurements further substantiate these three contributions. The revised manuscript and expanded results firmly establish TRIAGE’s scientific and real-world impact.

We respectfully ask that the revised manuscript, additional results, and clarifications be considered together in the final assessment. Thank you for your time and consideration.

With appreciation,

The TRIAGE Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.