- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates training instability caused by learner–sampler log-probability mismatch (Eq. 2) during native NVFP4 (W4A4, Eq. 4) reinforcement learning for large language models. In Section 3.1 (Eqs. 5–9, Appendix B Eqs. 21–34), a first-order Taylor expansion of the sampled-token mismatch energy under a fixed sampler decomposes the single-step change into , motivating two locally amplifying sign quadrants: and . Based on trajectory diagnostics showing pre-collapse growth in raw advantage-weighted occupancy (Section 3.2, Eq. 10, Figure 2) and concentration of severe negative-gap tokens within 64-token segments (Section 3.3, Eq. 11, Figure 3), Section 4 proposes TRIAGE (Figure 1, Eqs. 12–20): (1) a 64-token Segment Gate (Eqs. 13–17) that downweights tokens in tail-heavy segments () and renormalizes weights within each response (), and (2) a 128-token Positive-Advantage Repair penalty (Eqs. 18–20) using a one-sided pseudo-Huber loss when and .
Evaluated on GRPO training of Qwen3-4B-Base (600 steps, MLP projections in NVFP4) and Qwen3-30B-A3B-Base (1,700 steps, routed expert MLPs in NVFP4) on DAPO-Math-17K across five math benchmarks (AIME24, AIME25, MATH500, AMC23, GSM8K, Table 1), TRIAGE prevents the collapse seen in naive NVFP4 (at step 304 on 4B and step 724 on 30B), achieving average accuracy of 58.49% vs. 58.26% for BF16 on 4B and 70.96% vs. 72.41% for BF16 on 30B, alongside up to 2.30x rollout throughput and 1.30x end-to-end iteration speedup on 30B (Table 2). While the directional formulation and native NVFP4 systems effort are valuable, I lean toward weak rejection due to an explicit mismatch between the paper's response-wise objective (Eqs. 16–17) and the global token normalization implemented in the supplementary code, negative mechanistic audit results on the primary 30B model in the appendix (Tables 5 and 8b), checkpoint-inherited ablations that start at step 300 (Figure 5), and backend confounds in the headline throughput numbers.
Critical strengths
1. Insightful directional and segment-localized diagnosis of low-precision RL mismatch.
By decomposing into diagonal current-gradient terms and off-diagonal couplings (Eqs. 6–8) and tracking 64-token segment statistics across training stages (Figures 2–3), the paper moves beyond simple scalar KL or importance-ratio thresholds and shows how localized negative-gap bursts can accumulate inside otherwise normal-looking long rollouts.
2. Large-scale native NVFP4 GRPO evaluation on both dense (4B) and MoE (30B-A3B) models with transparent appendix auditing.
Maintaining native NVFP4 execution on both the sampler and learner MLP/expert forward paths over 600 steps (4B, up to 16k tokens) and 1,700 steps (30B-A3B, up to 20k tokens) on B300 hardware is a nontrivial systems achievement, and the authors deserve credit for including thorough negative audits and null-model permutations in Appendices C and F.
Critical weaknesses
1. Unresolved discrepancy between the paper's response-wise objective (Eqs. 16–17) and the released global token-normalization implementation, alongside negative mechanistic evidence on 30B.
As detailed under Soundness, the supplementary codebase (docs/PAPER_AND_CODE.md lines 32–40, policy_extension/TRIAGE.py lines 817–878) optimizes a global weighted-token average across all DP ranks and micro-batches rather than the per-response weighted mean in Eqs. (16)–(17). Under global token normalization, gating a segment in a long response changes the relative weight of other responses in the batch rather than strictly reallocating weight within response , and also alters length normalization (). Compounding this, on the flagship Qwen3-30B-A3B model, the diagonal directional predictor has only 25.27% weighted sign agreement with actual token motion (Table 5), and the effective length-normalized pre-terminal ratio is just 1.049 (Table 8b). Consequently, it is unclear both which mathematical objective produced Table 1 / Figures 4–5 and why the method stabilizes 30B when the local directional assumptions fail on 30B.
2. Confounded step-300 branched ablations (Figure 5), single-seed trajectories, large benchmark trade-offs on 30B (-10.32 pp on AMC23), and backend-confounded throughput claims.
- Step-300 checkpoint inheritance in Figure 5 (Section 5.4, p. 9): The only component ablation separating
GateandRepair(TRIAGE,Gate+TIS,TIS,NVFP4) is run solely on4Band branches all four variants from a step-300 checkpoint that was already trained with full TRIAGE (and omits aRepair+TISbranch as well as final benchmark accuracy numbers). This does not test whetherGateorRepaircan stabilize training from scratch (step 0). - Missing quantized-RL baselines, and
30Bbenchmark trade-offs: Closely related methods discussed in Section 6 (QaRL,QUADS,ACRL,FP8-RL) are not compared (andTISis only tested at ); and on30B(Table 1),NVFP4+TRIAGEdrops by-10.32percentage points relative toBF16onAMC23(77.72%vs.88.04%), offsetting that drop in the unweighted average only via a+4.66pp gain onGSM8K(95.47%vs.90.81%). - Backend and output-length confounds in the
2.30xthroughput claim (Table 2 vs. Appendix Table 10 &examples/common.sh): Inexamples/common.sh(lines 71–83),nvfp4runs withSGLANG_MOE_RUNNER_BACKEND=flashinfer_trtllmwhereasbf16defaults totriton, and Table 2 measures natural-EOS generation where output lengths vary across runs. When bothNVFP4andBF16use the sameFlashInfer TRT-LLMMoE backend at a fixed output length in Appendix F.3.1 (Table 10, p. 28), the standalone30Brollout speedup is only1.175x(18,175.67vs.15,470.01tokens/s)—and on dense4B(benchmarks/README.md), only1.014x–1.026x.
Questions for authors
-
Assessment-critical (Reconciling the paper's response-wise objective in Eqs. 16–17 with the code's global token normalization):
Which exact loss reduction produced the results in Table 1 and Figures 4–5: the per-response weighted mean in Eqs. (16)–(17) or the global weighted-token normalization across all DP ranks and micro-batches inpolicy_extension/TRIAGE.py(docs/PAPER_AND_CODE.mdlines 32–40)? Also, were the main runs executed withGATE_SCOPE="q3"andREPAIR_SCOPE="adv_positive"(RECENTER_ENABLE=0) as written in Section 4, or with the code defaults intriage_config.py("whole"andRECENTER_ENABLE=True)? -
Assessment-critical (From-scratch ablations and mechanism on
Qwen3-30B-A3B):- Can you provide from-scratch (
step 0) ablations comparingNVFP4+TIS,Gate+TIS,Repair+TIS, andFull TRIAGE(with final five-benchmark evaluation scores and seeds on4B), rather than branching from the step-300 TRIAGE checkpoint as in Figure 5? - Given that on
30Bthe diagonal sign agreement is only25.27%(Table 5) and the length-normalized pre-terminal ratio is1.049(Table 8b), what accounts for TRIAGE's stabilizing effect on30B, and why does30BNVFP4+TRIAGElose10.32percentage points onAMC23relative toBF16in Table 1?
- Can you provide from-scratch (
-
Clarification (
MATH-50092.65%denominator and matched-backend iteration speedup):- How was
92.65%obtained for30B BF16onMATH-500in Table 1 under 500 problems 6 binary-scored samples (3,000samples, where92.63%and92.67%)? - In Table 2, what are the
30Brollout throughput and end-to-end iteration speedups whenBF16andNVFP4+TRIAGEboth use the exact sameFlashInferMoE runner backend (rather thantritonforBF16vs.flashinfer_trtllmforNVFP4inexamples/common.sh)?
- How was
2 comments
Follow-up on our rebuttal
Dear Reviewer ZX3Y,
We wanted to send a brief follow-up as the author-reviewer discussion period closes in less than 24 hours. Whenever you have a moment, we would greatly appreciate it if you could look over our point-by-point responses and additional experimental results, and let us know if our revisions address your concerns or any remaining questions that we can clarify.
Thank you very much for your time and consideration.
Best regards,
The TRIAGE Authors
Follow-up on our rebuttal
Dear Reviewer ZX3Y,
Thank you again for your time and effort in reviewing our work. We have posted our point-by-point responses and additional experimental results under your review. As the discussion period closes within the next two days, we would appreciate any feedback on whether our responses address your concerns, or any remaining questions we can clarify.
Best regards,
The TRIAGE Authors