acceptodds
Under review as a conference paper at ICLR 2027

miniTE: Diagnosing and Bridging the Execution Gap in Native NVFP4 Reinforcement Learning

Abstract

Native NVFP4 promises to accelerate reinforcement learning (RL) for large language models (LLM), but achieving stable learning across separate training and sampling engines remains challenging. We show that matching model weights and numerical formats is insufficient to ensure consistent policy evaluation, highlighting the importance of cross-engine numerical execution. We introduce miniTE, a lightweight framework that combines execution diagnostics, cross-engine numerical alignment, and efficient native W4A4 (4-bit weights and activations) GEMM computation for LLM RL. It improves numerical execution consistency without modifying the RL objective, while reducing implementation overhead to realize the benefits of native low-bit computation. Long-horizon RL experiments on Qwen3-30B-A3B demonstrate stable training and benchmark performance near to BF16, outperforming naive Transformer Engine (TE) NVFP4 in evaluation quality. On single NVIDIA B300 GPU node, the optimized learner reaches 33,540 tokens/s, yielding 1.39 and 1.47 the throughput of BF16 and TE NVFP4, respectively, for complete learner updates. We released the framework, kernels, diagnostics, and training configurations to make native low-bit RL easier. Our framework, kernels, diagnostics, and training environments are available at https://anonymous.4open.science/r/miniTE.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Quantization, Reinforcement Learning, LLMs, High-performance Training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RwJPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents miniTE, a framework for native NVFP4 reinforcement learning that combines cross-engine numerical diagnostics with optimized MoE expert computation. It shows that matching weights and numerical formats does not ensure consistent learner–sampler policy evaluation. Using W4A4 expert forward computation, BF16 backward computation, and optimized quantization and grouped GEMM, miniTE improves mathematical benchmark performance over standard NVFP4 and achieves 1.39× BF16 learner throughput on eight B300 GPUs. However, increased weight conversion and synchronization costs leave end-to-end RL efficiency unresolved.

Critical strengths
  • The paper provides useful diagnostics for examining numerical discrepancies between the learner and sampler.
  • The authors report learner-side speedups and provide reasonably detailed implementation and measurement protocols.
Critical weaknesses
  • Limited end-to-end benefit. In many RL workloads, learner computation is not the dominant bottleneck. Quantizing only the expert forward pass further limits the potential overall speedup.

  • Insufficient long-horizon evaluation. The experiments provide limited evidence on genuinely long-horizon reasoning or agentic tasks, making it difficult to assess how learner–sampler discrepancies accumulate with trajectory length or affect kl.

  • Substantial synchronization overhead. Weight conversion and synchronization time increases from 9.94 to 55.26 seconds in the reported colocated experiment, potentially outweighing learner-side savings. This raises scalability concerns for larger, more sparsely activated MoE models.

  • Limited methodological novelty. The contribution primarily consists of numerical diagnostics and engineering optimizations, with limited theoretical or algorithmic innovation.

  • Suggestions. The authors could profile large-scale agentic RL workloads to identify and address actual end-to-end bottlenecks. Broader investigation of rollout–learner scheduling, load balancing, synchronization, and fault tolerance could offer greater practical impact and stronger systems contributions.

Questions for authors

same as weakness

6 comments
AuthorsOpenReview ↗

Author response (1/3)

Thank you for recognizing the value of the cross-engine numerical diagnostics and for your suggestions regarding complete RL efficiency, long reasoning trajectories, and systems contributions. We have added complete-cycle measurements and weight-update optimizations, and clarify below how the experimental evidence supports the different contributions of the work.

W1 and W3: End-to-end benefit and synchronization overhead

A1: Thank you for highlighting practical RL training costs and synchronization overhead in MoE models. The additional complete-cycle measurements show that miniTE achieves 1.50× and 1.46× speedups over the BF16/Triton baseline used in our training experiments at 256 and 512 responses per cycle, respectively. It also achieves 1.16× and 1.05× speedups over TE NVFP4. These measurements include rollout, learner updates, model offload and reload, and the actual costs of weight conversion and synchronization.

Please see our Common Response: End-to-End Performance for the complete-cycle measurements, major-stage timings, and results for the optimized system. The recovered complete-cycle timings for the historical configurations are reported in Appendix F.3, alongside Table 15. For MoE weight synchronization, compact transfer reduces the measured logical tensor payload by 68.2% relative to BF16, while grouped quantization and pipelined transfer further reduce weight-preparation and synchronization costs. The TE NVFP4 comparison also uses the optimized weight-update pipeline; miniTE therefore retains additional complete-cycle gains with this infrastructure shared.

For larger and more sparsely activated MoE models, the distinction between sparse computation and weight synchronization is important. In the full-weight refresh used here, the synchronization payload depends on the total expert weights being refreshed, even though each token activates only a subset of experts. Compact representation, grouped quantization, and pipelining directly address the associated transfer volume, repeated preparation, and serial execution costs.

W2: Long reasoning trajectories and learner–sampler discrepancies

A2: Thank you for raising the question of long reasoning trajectories and multi-turn agentic tasks. We recovered and analyzed response-length records from an independent historical TE NVFP4 diagnostic run of Qwen3-30B-A3B. Across 185,600 generated responses, the 99th-percentile length is approximately 7,818 tokens, and the maximum is 20,480 tokens, establishing that the actual native NVFP4 RL workload includes long generated responses.

The records cover 725 rollouts and approximately 232 million response tokens. All lengths come from individual response records:

Recorded responsesP95 lengthP99 lengthMaximum lengthResponses marked as truncated
185,6002,969 tokensApproximately 7,818 tokens20,480 tokens538, or 0.29%

These statistics establish the actual response-length coverage of the diagnostic workload; the main miniTE experiments provide the evidence for sustained learning and task quality. Figure 3 reports training behavior over the first 1,000 updates, during which miniTE maintains sustained reward improvement. Table 1 evaluates the resulting checkpoints, with miniTE achieving higher mean scores than standard TE NVFP4 on all four mathematics benchmarks: AIME24, AIME25, MATH-500, and GSM8K. Training and evaluation both use the 20,480-token maximum generation length specified in Table 10.

Regarding accumulation along a trajectory, the sequence log-probability ratio for a fixed generated response is the sum of its token-level log-probability differences. The signs and distribution of these local differences therefore matter alongside trajectory length; an average absolute gap alone does not determine sequence-level KL.

The diagnostics in Section 3 examine these local execution differences through frozen weights, scoring of identical sequences, tail statistics, and routing-conditioned measurements. They identify numerical discrepancies that remain after matching the nominal quantization recipe and target-layer expert assignments, including differences that average metrics can obscure. These diagnostics provide a reusable basis for studying mismatch across token positions and interaction turns.

AuthorsOpenReview ↗

Author response (2/3)

W4: Methodological and systems contributions

A3: Thank you for asking us to clarify the research contribution. miniTE combines reproducible diagnosis of cross-engine execution differences, a native NVFP4 training path that sustains learning, and a lightweight, efficient framework validated through numerical checks. Together, these contributions make low-precision RL execution measurable, inspectable, and reproducible.

First, the diagnostics reveal execution differences that remain after nominal quantization recipes are matched. Frozen-weight measurements show that identical parameters and numerical formats are insufficient to ensure identical policy computation. The routing-conditioned comparison further shows residual FC1/FC2 output differences even when expert assignments at the target layer are matched. These findings motivate an explicit execution specification covering scales, rounding, layout, rescaling, casts, and MoE composition, with diagnostics at three boundaries: quantized operands, expert outputs, and token log-probabilities (Section 3 – Section 4.1).

Second, the numerical design is examined through controlled experiments and complete training runs. The forward context-swap experiment verifies that the tested per-token path is independent of changes in the numerical ranges of other tokens in the batch. The gradient comparison establishes the numerical effect of independently selecting backward precision. The separately constructed strict reference in Figure 4 demonstrates that tight cross-engine alignment and sustained learning can coexist with native NVFP4 computation. Complete RL curves and benchmark evaluations assess the learning behavior of the practical training configuration. This is consistent with our explanation of learning gains: an average forward log-probability gap cannot fully characterize forward execution, backward gradients, and the policy updates they jointly produce.

Third, miniTE organizes these designs into an efficient framework with configurable recipes, replaceable operators, and reusable execution components. Quantization directly produces the contiguous FP4 and scale buffers consumed by grouped GEMM, while device-side expert metadata, workspaces, and activation amax values are reused across the execution path. This joint organization reduces repeated scans, representation conversions, and preparation overhead.

Under the same forward recipe, miniTE achieves a 2.46× geometric-mean expert-forward speedup over the TE per-token reference, and the tested API and layout transformations pass the numerical-preservation checks reported in the paper. Configurable quantization recipes, independent backward precision, replaceable expert operators, and reproducible execution environments make these diagnostics and implementations usable within existing RL pipelines. The complete learner and complete RL-cycle measurements in the common response provide the corresponding evidence for practical efficiency.

AuthorsOpenReview ↗

Author response (3/3)

W5: Profiling real workloads and broader systems optimization

A4: Thank you for suggesting that systems optimization be guided by bottlenecks in actual RL workloads. We have implemented this direction through complete RL-cycle profiling and concrete improvements to the weight-update bottleneck identified in the experiment.

The weight conversion, packing, and synchronization costs discussed in §5.3 and Appendix A are quantified in the historical colocated profile, now Table 15. The new implementation reduces these costs through grouped weight quantization, overlap of weight preparation with asynchronous transfer, and compact representations with improved bucket and buffer organization.

At 512 responses per cycle, the historical configuration recorded 55.26 s for weight updates and 209.44 s for the complete RL cycle. The optimized configuration records 22.61 s and 174.20 s, respectively—reductions of 59.1% and 16.8%.

The controlled weight-update measurements in Appendix F.5, Table 17, further identify the effects of bucket size and pipelining. Using identical frozen weights and grouped quantization, increasing the bucket size reduces serial update time, and enabling pipelining at the same bucket size reduces it further:

ExecutionBucket sizeWeight-update time
Serial512 MiB48.07 ± 1.17 s
Serial2 GiB28.74 ± 0.20 s
Pipelined2 GiB20.36 ± 1.55 s

These separate controlled measurements report mean ± sample standard deviation over three measured updates after one warm-up update.

We believe this improvement extends miniTE’s systems design from the quantization and grouped-GEMM dataflow within the learner to weight updates across the learner–sampler boundary. Rollout–learner scheduling, load balancing, and fault tolerance are important directions for extending the system to larger-scale and multi-turn agentic RL. miniTE’s modular interfaces, execution specification, and cross-engine checks provide reusable numerical validation tools for investigating those changes in execution organization.

Reviewer_e1QCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

the paper studies native nvfp4 (w4a4) computation for llm RL where the learner and sampler are separate engines. a frozen-policy diagnosis shows that even at a shared weight version, nvfp4 roughly doubles the p99 learner-sampler log-prob gap over bf16, and that matching routing or switching the sampler to per-token scaling does not remove the fc1/fc2 output discrepancy. it then introduces miniTE: a per-token nvfp4 expert-forward path with bf16 backward, a fused quantization plus grouped-gemm dataflow, and cross-engine diagnostic hooks. on qwen3-30b-a3b with TIS-corrected RL, miniTE trains stably for 1,000 steps and beats transformer engine’s per-tensor nvfp4 on four math benchmarks while staying below bf16.

Critical strengths
  • the diagnosis is careful. it separates quantization error, cross-engine execution difference and implementation difference, uses teacher-forced scoring at a frozen checkpoint with bootstrap CIS, and the routing-matched control (fig. 1 right) is a real result: expert-id agreement is not sufficient.

  • the systems work is concrete and verified: bitwise checks against the upstream reference, bitwise preservation of target rows under context swaps, and a released artifact with dockerfiles.

  • the paper is honest about its limits. it reports the colocated RL measurement where weight conversion and sync rises from 9.9 s to 55.3 s per iteration (table 13), and it states in appendix A that all learning evidence is one model and one run per configuration.

Critical weaknesses
  • the learning results are a single RL run per configuration. the headline +10 points on aime24 (table 1) is 30 problems from one run, and the six evaluation rounds average only sampling noise. IFeval reverses the ranking (naive nvfp4 43.99 vs minite 38.40). with n=1 the ordering between the two nvfp4 paths is not established.

  • the main comparison conflates recipe and implementation. “naive nvfp4” is TE’s per-tensor recipe with hadamard transforms; miniTE is per-token without them plus a different kernel path. the paper’s own evidence says both nvfp4 paths have similar mean log-prob discrepancy (sec. 5.2) and that per-token scaling alone does not close the gap (sec. 3.2), so what produces the reward difference is not isolated. the strict reference that does close the gap (1e-8, fig. 4) is a third configuration, run without tis and with a different step count (871/1199/1189 endpoints, appendix e).

  • the throughput claim is learner-only on fixed text. in the one colocated RL measurement the learner saves 4.5 s per iteration while weight conversion costs 45s more, and the paper says it cannot determine whether a full iteration is faster (table 13). the abstract leads with 1.39x without this qualifier.

Questions for authors
  • can you run TE’s upstream per-token path (u3045) as the learner with the same TIS pipeline? that would separate the recipe change from the implementation change in table 1.
  • why is strict execution not the default training path, and what is its learner throughput relative to table 2?
  • what is the full RL iteration time for bf16 vs miniTE, including rollout? can the bf16-to-nvfp4 weight conversion be moved off the critical path?
  • can you add at least two more training seeds for the three arms in table 1, or report per-round variance so readers can judge the math gains?
6 comments
AuthorsOpenReview ↗

Author response (1/3)

Thank you for the careful assessment and for recognizing the cross-engine diagnostics, numerical validation, and reproducible artifact. Your questions have helped us clarify the learning evidence and quantify the practical costs of the different execution paths. We apologize for the delayed response: limited GPU availability meant that the long-horizon PR #3045 experiment took longer than expected to complete. We hope the additional experiments and results below address your concerns.

W1 and Q4: Evaluation variability and paired comparisons

A1: Thank you for this suggestion and for your patience. We have supplemented the original evaluation results with round-to-round standard deviations and problem-level paired analyses using the per-item evaluation records of the original step-999 checkpoints. The additional statistics provide positive evidence for the AIME24 improvement: miniTE exceeds naive TE NVFP4 by 10.00 percentage points, with a nominal 95% paired t confidence interval of [0.67, 19.33] percentage points.

The table below reports mean accuracy and the standard deviation across evaluation rounds. AIME24 and AIME25 each have six rounds; the other benchmarks each have four.

BenchmarkBF16Naive TE NVFP4miniTE
AIME2441.11 ± 5.4426.67 ± 2.1136.67 ± 5.16
AIME2531.11 ± 5.4420.56 ± 3.2824.44 ± 4.55
MATH-50090.05 ± 0.5386.90 ± 0.6287.45 ± 0.66
GSM8K93.57 ± 0.5594.71 ± 0.1094.79 ± 0.19
IFEval38.36 ± 1.0643.99 ± 1.2238.40 ± 1.24

Entries report mean accuracy (%) ± the sample standard deviation across evaluation rounds (percentage points).

On AIME24, miniTE achieves a mean accuracy of 36.67% across six evaluation rounds, compared with 26.67% for naive TE NVFP4. Its round-to-round standard deviation is 5.16 percentage points, close to BF16’s 5.44. Across the 30 problems, miniTE has a higher mean correctness rate on 12 problems, a lower rate on 2, and the same rate on 16, showing how the improvement is distributed at the problem level.

For the paired analysis, we first average repeated answers’ correctness within each problem and then compare the two methods on the same problems. The six answers to each AIME problem therefore remain within a single analysis unit; the 180 answers are not treated as 180 independent problems.

The full results also show task-dependent differences. The point estimates for AIME25, MATH-500, and GSM8K are positive, while the IFEval difference is negative. The paired analysis provides the clearest positive support for the AIME24 improvement and quantifies evaluation variability and differences on the other tasks. These statistics characterize evaluation uncertainty for the original checkpoints, with one RL training run per configuration. The supplementary results allow mean performance, repeated-evaluation variability, and problem-level paired differences to be examined together.

AuthorsOpenReview ↗

Author response (2/3)

W2 and Q1: Upstream PR#3045 per-token comparison and attribution of learning gains

A2: Thank you for suggesting the TE upstream per-token learner as an additional comparison under the same TIS pipeline, and for asking us to distinguish the numerical recipe from implementation optimizations. We have completed a long-horizon training comparison with PR #3045 under the same GRPO+TIS pipeline and aligned training hyperparameters. miniTE maintains an overall upward reward trend, whereas PR #3045 exhibits recurrent instability and substantial reward degradation later in training.

Additional upstream comparison. As shown in the training curves in the supplementary miniTE_vs_PR3045.pdf, the two paths have similar reward trends for approximately the first 600 steps. Subsequently, the reward of PR #3045 drops sharply and remains near 0.2 for an extended period. Although it partially recovers at times, substantial fluctuations recur later in training. Meanwhile, its learner–sampler log-probability mismatch increases markedly, and its mean response length decreases from approximately 2,500 tokens to a few hundred tokens. miniTE continues to improve in reward while maintaining a comparatively low log-probability mismatch.

The numerical recipe is also a core part of miniTE’s design. miniTE specifies forward execution, backward precision, and the scope of quantization together: expert forward computation uses per-token NVFP4; backward computation retains the original, pre-quantization BF16 activations and weights for BF16 dgrad and wgrad; and non-expert modules retain their configured higher precision. This precision allocation combines native W4A4 expert forward computation with high-precision gradient computation.

The controlled experiments in the paper provide direct numerical evidence for these choices. The context-swap experiment in Appendix D verifies that, for the tested operator path, the target token’s quantization results and GEMM outputs remain unchanged when the numerical range of other tokens in the batch changes. Gradient comparisons against an FP32 STE reference establish the numerical effect of using the original BF16 operands for backward computation. Thus, miniTE’s numerical design affects both forward dependencies and the gradients actually computed, neither of which is fully characterized by an average forward log-probability gap.

Regarding the similar average gaps in the original experiments, identical TIS settings and similar mean gaps do not imply identical policy updates. TIS uses learner–sampler probability ratios to apply truncated importance weights to policy-gradient contributions. It does not directly remove the raw log-probability differences recorded in the curves, nor does it make the underlying quantization rules, backward operand sources, or computation precision identical. Even when the mean absolute gaps are similar, the signs of token-level differences, advantages, importance weights, clipping states, and actual gradients can produce different updates. These updates affect subsequent parameters, sampled trajectories, and numerical distributions, allowing long-term learning behavior to diverge.

We view the co-occurrence of increased mismatch and reward degradation in the additional comparison as consistent with this online update feedback mechanism. TIS and miniTE’s numerical recipe therefore play complementary roles: TIS adjusts gradient weights to account for sampling discrepancies, while the numerical recipe specifies the forward and backward computations actually executed.

We evaluate implementation optimizations through separate comparisons. The 28 common-input checks in Appendix D verify bitwise preservation of quantized operands, scales, and GEMM outputs for the tested forward API paths. Under the same forward recipe, miniTE achieves a 2.46× over the TE reference. The complete RL comparisons evaluate the learning behavior of the numerical configurations, while the numerical-preservation checks and same-recipe performance comparisons support the implementation-efficiency contribution.

AuthorsOpenReview ↗

Author response (3/3)

W2 and Q2: Why is strict execution not the default training path, and what does it cost?

A3: Thank you for asking us to clarify the practical cost of strict execution and the rationale for choosing the default training path. Strict execution provides a rigorous reference for cross-engine numerical alignment. Additional measurements during real RL updates show a mean absolute learner–sampler log-probability difference below 10−710^{-7}; its higher runtime cost explains our choice of a more efficient default training configuration.

Numerical alignment. Strict execution jointly aligns learner and sampler operators, formulas, and reduction orders, including the surrounding normalization and communication paths, while retaining native NVFP4 forward computation. During actual RL updates, the mean absolute learner–sampler log-probability differences are:

Batch sizeStrict executionBF16 diagnostic reference
2569.166572×10−89.166572 \times 10^{-8}2.176722×10−22.176722 \times 10^{-2}
5129.121627×10−89.121627 \times 10^{-8}2.173935×10−22.173935 \times 10^{-2}

Values are averaged over training steps 2–6 of each configuration’s own RL trajectory using the same metric reduction.

The cross-engine difference under strict execution is more than five orders of magnitude smaller than that of the measured BF16 reference, showing that tight execution alignment is maintained during these actual policy updates.

Runtime configuration. As specified in Appendix E.3, strict execution uses a single eight-GPU DP8/EP8 sampler with CUTLASS/FA4 and no TIS.

Learner throughput. We additionally report learner throughput from the actual RL runs. Strict execution achieves 71.7% and 58.6% of default miniTE’s learner throughput at batch sizes 256 and 512, respectively.

Batch sizeDefault miniTEStrict executionStrict / miniTE
2568,825 tokens/s6,326 tokens/s0.717×
51216,141 tokens/s9,451 tokens/s0.586×

Rollout and complete-cycle costs. Rollout accounts for 82.3% and 83.3% of strict execution’s total runtime at batch sizes 256 and 512, respectively, making it the dominant cost of the current strict configuration.

At these two batch sizes, strict execution takes 586.40 s and 715.00 s per complete RL cycle, compared with 138.90 s and 174.20 s for default miniTE. Its complete-cycle times are therefore 4.22× and 4.10× those of the default path.

These results motivate using strict execution as an executable reference for numerical alignment and diagnosis, while selecting the more efficient miniTE path with TIS as the default for routine RL training.

W3 and Q3: End-to-end performance

Please see our Common Response: End-to-End Performance, which reports complete RL cycle times and stage measurements for BF16, TE NVFP4, and miniTE, together with the weight-update optimizations and their measured costs.

Reviewer_4V5GOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces miniTE to make 4bit RL training faster and more reliable. It investigates why the trainer and sampler produce inconsistent log probabilities even with the same weights. It introduces a framework that locates the differences, enforces higher-precision 4-bit gradient computation, and reduces GPU overhead. On a Qwen MoE model trained on math questions, it improves results over standard NVFP4 training and makes learner updates up to 1.39× faster than bf16.

Critical strengths

There are two clear strengths. First, it measures the difference between the learner and sampler under NVFP4 and reduces the compute overhead between quantization and the GPU. Second, its BF16 backward pass is faster than the MXFP8 backward pass since it skips the dequantization preparation cost.

Critical weaknesses
  1. The idea of per-token NVFP4 forward + BF16 backward has been explored in MIles, so it is not counted as a novel contribution. The self-implemented CUTLASS NVFP4 with BF16 backward achieves learner throughput similar to miniTE with BF16 backward; however, it doesn't compare these two settings on related benchmarks such as MATH and AIME.
  2. Also, it is unclear to me how the observed improvements from miniTE are generated. For example, the average execution gap between miniTE and naive TE NVFP4 still seems very small, it would be great to figure out which part improves the training performance.
Questions for authors
  1. Could you add the benchmark scores (AIME24, AIME25, MATH-500, etc.) for your CUTLASS NVFP4 + BF16 backward implementation under the same training and evaluation settings?
  2. Which changes in miniTE really contribute to the higher reward in Figure 3?
  3. Could you also report rollout time and total time per RL epoch for the BF16 and miniTE runs in Table 13?
7 comments
AuthorsOpenReview ↗

Author response (1/2)

Thank you for your constructive comments and questions. We apologize for responding later in the discussion period; limited GPU availability meant that completing the additional long-run CUTLASS comparison took longer than expected. We hope the results and explanations below address your concerns.

W1 and Q1: Benchmark comparison between CUTLASS NVFP4 + BF16 backward and miniTE + BF16 backward

Thank you for suggesting this comparison. Comparing complete RL training with the same high-precision backward pass helps clarify miniTE’s learning behavior relative to another native NVFP4 implementation.

We have added training curves and five-benchmark results for CUTLASS NVFP4 with BF16 backward under the same training and evaluation settings as miniTE. Both configurations sustain learning over 1,000 updates, with miniTE achieving a higher mean reward in the later training stage and a higher five-task average score.

The additional curves show broadly similar learning trajectories. Over the final 200 updates (steps 800–999), miniTE achieves a mean raw reward of 0.6086, compared with 0.5989 for CUTLASS.

At the final evaluation checkpoint, miniTE achieves a five-task average of 56.35%, compared with 55.52% for CUTLASS, an improvement of 0.83 percentage points. Its scores on AIME24, AIME25, and MATH-500 are 36.67%, 24.44%, and 87.45%, respectively.

ConfigurationAIME24AIME25MATH-500GSM8KIFEvalAvg.
miniTE + BF16 backward36.6724.4487.4594.7938.4056.35
CUTLASS NVFP4 + BF16 backward33.3323.3387.3095.0338.6355.52

Together with the learner-throughput comparisons in the paper, these training and evaluation results show that miniTE maintains competitive overall quality against this native NVFP4 configuration with BF16 backward, while achieving a higher average score across the mathematics tasks.

Regarding Miles, we discuss its per-token NVFP4 and high-precision backward recipe in Section 6. Our response A2 below explains miniTE’s additional execution diagnostics, numerical specification, and efficient native implementation.

We have added the CUTLASS comparison to the reward and learner–sampler log-probability difference curves in Figure 3 and to the evaluation results in Table 1. We also provide the raw training data for all four configurations in the supplementary material.

AuthorsOpenReview ↗

Author response (2/2)

W2 and Q2: Which changes in miniTE contribute to the higher reward, and which components improve training performance?

A2: Thank you for asking us to clarify the sources of the learning improvements. miniTE’s numerical design changes both the dependence of forward quantization on other tokens and the gradient computation used for policy updates. The average learner–sampler log-probability gap summarizes forward probability differences and therefore does not capture the full effect of these choices on learning.

The effect of an execution difference depends on which tokens it affects and how it interacts with their advantages, importance weights, clipping status, and backward gradients. TIS operates through importance weighting of policy-gradient contributions; the numerical forward and backward paths still determine the resulting update. Consequently, similar average gaps can coexist with different policy updates and subsequent training trajectories.

Our numerical checks and complete training comparisons provide complementary evidence:

  1. Forward computation: miniTE’s per-token quantization removes the tested operator’s dependence on the numerical ranges of other tokens in the batch. Each activation row’s outer scale is determined by that row alone. This addresses a concrete source of variability when the learner and sampler process tokens in different groups. In the context-swap experiment in Appendix D, we hold the target tokens, expert weights, input shape, and padding fixed while replacing only the companion tokens. miniTE preserves the target FP4 payload and native GEMM outputs bitwise across all six swaps, whereas the RHT-enabled per-tensor reference changes in five of them. This directly verifies the change in numerical behavior of the forward path.
  2. Backward computation: miniTE retains the original BF16 activations and weights for gradient computation while maintaining native W4A4 forward execution. Against the FP32 gradient reference under straight-through-estimator semantics, the maximum relative-L2 error is 0.001663 for BF16 backward, compared with 0.037653 for the tested MXFP8-backward alternative (Table 9). These measurements demonstrate the numerical effect of choosing backward precision independently—a property that the forward log-probability gap does not measure.

The additional CUTLASS comparison in A1 connects these numerical considerations to complete RL training. miniTE achieves the higher late-training mean reward and five-task evaluation mean. With the backward choice shared, this comparison further evaluates the learning behavior of the two native NVFP4 forward paths.

Beyond the numerical recipe, miniTE provides reproducible cross-engine diagnostics, an explicit execution specification, and an efficient native implementation. As discussed in Section 6, Miles also combines per-token NVFP4 with high-precision backward. miniTE further investigates execution differences that remain after nominal recipe agreement: frozen-weight and routing-conditioned measurements reveal residual discrepancies even when target-layer expert assignments agree. It provides diagnostics at three boundaries—quantized operands, expert outputs, and policy log-probabilities—and explicitly specifies and checks scale domains, rounding, layouts, rescaling, casts, and MoE composition.

At the implementation level, miniTE jointly designs quantization and grouped-GEMM dataflow, using contiguous buffers, device-side metadata, amax reuse, and native epilogues to reduce execution overhead. All 28 reported forward-API comparisons pass bitwise checks, and the fused-versus-unfused layout tests preserve the corresponding operands and outputs. These checks establish numerical preservation for the tested implementation transformations. The common response reports the resulting efficiency gains at the operator, complete learner, and complete RL cycle levels.

W3: End-to-end efficiency

A3: Please see our Common Response: End-to-End Performance, which reports complete RL cycle times and stage measurements for BF16, TE NVFP4, and miniTE, together with the weight-update optimizations and their measured costs.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Revised Manuscript

Dear Reviewer 4V5G,

Thank you once again for your constructive review and valuable time. We have posted our detailed responses along with new experimental results under your review. With about 36 hours remaining in the discussion period, we would be grateful to know if our rebuttal resolves your concerns, or if there are any further points we should address.

Best regards,

The miniTE Authors

Comments on the paper

4 comments
AuthorsOpenReview ↗

Common Response: End-to-End Performance (1/2)

We thank all three reviewers for highlighting complete RL training efficiency and the cost of weight synchronization. These costs are central to the practical value of native low-precision RL, and we provide a consolidated response below.

Our additional end-to-end measurements show that, at 256 and 512 responses per cycle, miniTE achieves complete-cycle speedups of 1.50× and 1.46× over the BF16 baseline used in our training experiments, and further speedups of 1.16× and 1.05× over TE NVFP4, respectively. All measurements include rollout, learner updates, model offload and reload, and weight conversion and synchronization. Our system combines efficient low-precision MoE execution with optimized weight updates across the learner–sampler boundary: the quantization, layout, and grouped-GEMM designs presented in the paper improve learner execution efficiency.

To address the weight-update bottleneck identified in the original submission, we redesigned the update path with grouped quantization and pipelined transfers. At 512 responses per cycle, weight-update time decreases from the historical 55.26 s to 22.61 s, a 59.1% reduction. The logical tensor payload is 19.39 GB, 68.2% smaller than BF16’s 61.06 GB.

1. Gains over end-to-end complete RL cycles.

We benchmark Qwen3-30B-A3B-Base on eight B300 GPUs. All configurations share same generation settings, and perform actual RL updates on DAPO-Math-17k. Each prompt generates 16 responses, with generation terminating naturally at EOS or at the 20,480-token limit. Each configuration runs for 12 cycles; the first two are warm-up, and we report the mean over the remaining ten cycles.

Table R1: Complete RL cycle times and selected stage times, in seconds. miniTE achieves complete-cycle gains at both response counts.

Responses per cycleConfigurationRolloutWeight updateFull RL cycleCycle speedup over BF16
256BF16115.7614.16208.201.00×
256TE NVFP458.8318.94161.501.29×
256miniTE41.8822.69138.901.50×
512BF16152.1313.99254.401.00×
512TE NVFP469.1519.36182.301.40×
512miniTE69.6822.61174.201.46×

BF16 uses the Triton MoE backend. Both NVFP4 configurations share the optimized weight-update pipeline.Full-cycle wall time is measured independently and includes all regular training and generation stages; rollout and weight update are selected components. Under natural EOS termination, each configuration executes its actual generated trajectories.

2. Turning the weight-update bottleneck into a concrete systems improvement.

The original submission identified weight conversion, packing, and synchronization at the learner–sampler boundary as a bottleneck and prioritized reducing these repeated operations (§5.3 and Appendix A). We redesigned this execution path through three complementary changes:

  • Grouped weight quantization. Expert matrices with compatible shapes are quantized in batches, reducing repeated per-matrix dispatch while preserving the original scaling rules and native NVFP4 block-scaling structure.
  • Pipelined weight preparation and transfer. Preparation of the next weight bucket overlaps with asynchronous transfer and loading of the current bucket. Generation resumes after all updated weights have been installed.
  • Compact representation and bucket organization. We transfer packed NVFP4 weights and their scales, retain the prescribed higher precision for non-expert parameters, and optimize transfer granularity and buffer organization.

In the additional RL measurements, weight updates take 22.61 s, a 59.1% reduction from the historical 55.26 s. The optimized configuration completes a full RL cycle in 174.20 s, compared with 254.40 s for BF16 and 182.30 s for TE NVFP4 in the same additional evaluation.

Controlled synchronization measurements on identical frozen weights further clarify the effects of bucket organization and pipelining. Update time decreases from 48.07 s with 512 MiB serial buckets to 28.74 s with 2 GiB serial buckets, and then to 20.36 s with 2 GiB pipelined updates (Table 17).

The compact representation carries 19.39 GB of logical tensor payload, compared with 61.06 GB for BF16, a 68.2% reduction. Grouped quantization and pipelined transfers further reduce the preparation and synchronization costs of using this representation in actual RL, and Table R1 includes these operations in full.

AuthorsOpenReview ↗

Common Response: End-to-End Performance (2/2)

3. Further performance potential as RL trajectories grow longer.

The end-to-end measurements above start from Qwen3-30B-A3B-Base and use naturally generated responses on DAPO mathematics tasks. At 256 and 512 responses per cycle, miniTE’s mean response lengths are approximately 923 and 1,029 tokens, respectively. In this setting, miniTE already achieves complete-cycle speedups of 1.50× and 1.46× over BF16/Triton, while its rollout output-token throughput reaches 2.21× and 2.06× that of BF16. These results demonstrate a substantial throughput advantage for the measured Blackwell NVFP4 generation path, while complete-cycle timing also accounts for weight conversion and synchronization, learner state transitions, and training computation.

When RL training produces longer reasoning trajectories, or workloads shift toward long-horizon agent tasks in which model generation accounts for a larger share of execution, each weight synchronization can serve more generated tokens. This amortizes the per-cycle costs of conversion, transfer, and execution-state transitions over more generation work, allowing the rollout-throughput advantage to contribute more strongly to end-to-end performance. Combined with the measured generation-throughput gains, we expect further scope for larger end-to-end benefits relative to BF16 in such workloads.

4. miniTE’s systems contributions.

Within the learner, miniTE jointly designs quantization, data layout, and grouped GEMM. Quantization directly produces contiguous FP4 and scale buffers consumed by GEMM, while device-side expert offsets, metadata, and workspaces are reused. FC1 collects and passes activation maxima during permutation; FC2 obtains the corresponding statistics from the producer of its router-weighted activations. Together with fused scale-layout preparation and GEMM configurations adapted to FC1 and FC2, these designs reduce repeated scans, interface conversions, and argument-preparation overhead.

Under the same forward recipe, miniTE achieves a 2.46× geometric-mean speedup in expert-forward execution over the TE per-token reference implementation. The fixed-text complete-learner measurements in Table 2 further report 1.39× the throughput of BF16 and 1.47× that of TE NVFP4. The new weight-update optimizations extend this systems design to the dataflow between learner and sampler, and the complete-cycle measurements establish its practical efficiency benefits in actual RL.

miniTE’s systems contributions span native low-precision MoE operator execution, data organization across operators, and weight updates across engines, supported by measurements at the operator, complete learner, and complete RL cycle levels.

The weight-update path is reusable. For a fair comparison, we apply the same optimizations to the current TE NVFP4 baseline. miniTE achieves further complete-cycle gains even when both configurations share this infrastructure. We have incorporated the new weight-update design, complete-cycle results, and corresponding experimental configurations into the revised manuscript, and will publicly release the new weight-update implementation so that the community can adopt these optimizations.

AuthorsOpenReview ↗

Summary of Revisions and Invitation for Further Discussion

Dear Reviewers and Area Chair,

We sincerely thank you for your constructive feedback. We have updated the manuscript and supplementary materials and posted detailed point-by-point responses to all three reviewers. We would like to briefly summarize the major additions:

  1. Expanded training comparisons: We added long-run RL comparisons with CUTLASS NVFP4 + BF16 backward and upstream TE PR #3045, together with further analyses of miniTE's numerical design and training behavior.

  2. Strengthened numerical and evaluation evidence: We added per-round evaluation variability, problem-level paired statistical analyses, strict-execution accuracy and runtime measurements, and additional long-trajectory diagnostics.

  3. Demonstrated end-to-end efficiency: We optimized weight synchronization, reducing weight-update time by 59.1%, and conducted complete RL-cycle measurements. miniTE achieves 1.50×/1.46× speedups over BF16 and 1.16×/1.05× over TE NVFP4 at 256/512 responses per cycle, respectively. These results complement the 2.46× same-recipe expert-forward speedup and complete-learner measurements.

  4. Others: Regarding the remaining non-overlapping inquiries and specific points raised individually by the three reviewers, we have performed supplementary experiments and furnished comprehensive explanations. Detailed answers are provided in our individual responses to each reviewer below.

We hope these additions directly address the major concerns regarding learning quality, numerical consistency, methodological contributions, and practical end-to-end efficiency.

We warmly welcome further discussion and would be happy to address any remaining questions or concerns during the rebuttal period. Thank you again for your time and consideration.

Best regards,

The miniTE Authors

AuthorsOpenReview ↗

Rebuttal Summary: Key Evidence and Contributions of miniTE

Dear Reviewers, Area Chair, and Senior Area Chair,

We sincerely thank you for your time and effort in reviewing our submission. The reviews recognized the scientific value of miniTE's cross-engine numerical diagnosis (4V5G, e1QC, RwJP), its numerically validated system implementation and reproducibility (e1QC), and its practical efficiency improvements and engineering contributions (4V5G, RwJP). After reviewing our clarification, 4V5G explicitly acknowledged our response and maintained their Weak Accept rating, while e1QC and RwJP have not yet provided follow-up comments. We would like to summarize the most important evidence and contributions below.

  1. miniTE not only accelerates the learner but also delivers substantial end-to-end RL training improvements, directly addressing all three reviewers' shared concerns about practical efficiency and synchronization bottlenecks. We supplemented our evaluation with full RL iteration and systematically optimized the weight conversion and synchronization bottlenecks identified in actual execution, reducing weight update time by 59.1%. The optimized miniTE achieves 1.46–1.50× end-to-end speedup over the BF16 implementation and a further 1.05–1.16× speedup over TE NVFP4. All measurements include rollout, learner updates, weight conversion, and synchronization costs. These results demonstrate that miniTE's computational advantages translate into practical efficiency gains, outperforming the existing TE NVFP4. (End-to-End Performance Common Response.)

  1. miniTE demonstrates stable and effective native NVFP4 learning, directly addressing concerns about the reliability of its training improvements. We added long-running comparisons against CUTLASS NVFP4 and the TE per-token path, together with per-run evaluation variance and per-question paired statistical analyses. miniTE completes 1000 RL updates, improves upon standard TE NVFP4 across all four mathematical benchmarks, and achieves an average accuracy of 56.35%, approaching BF16's 58.84%. miniTE also achieves higher average performance than the CUTLASS and maintains continued learning when the TE per-token path becomes unstable. These additional results strengthen the evidence supporting miniTE's training quality, learning stability, and numerical design. (Fig 3, Table 1, App E.2; Responses to 4V5G and e1QC.)

  1. Controlled numerical experiments reveal execution differences beyond nominally identical NVFP4 recipes, clarifying miniTE's scientific contribution and the rationale behind its training design. We systematically investigated cross-engine numerical discrepancies and their effects through experiments. Our experiments demonstrate that identical weights, quantization formats, and even expert routing do not necessarily imply identical numerical execution behavior; moreover, the average log-probability gap alone is insufficient to fully explain actual policy updates and training performance. Strict execution reduces cross-engine log-probability discrepancies during actual training to below 10−710^{-7}, further addressing concerns related to long reasoning trajectories. (Sec 3–4, App D and E.3; Responses to 4V5G W2, e1QC W2, and RwJP W2/W4.)

  1. Beyond providing an effective numerical recipe, miniTE establishes an efficient, accessible, and reusable open-source framework for native low-precision RL through joint operator optimization and rigorous numerical validation. We further validated miniTE's joint design of quantization, data layout, and grouped GEMM, achieving an average 2.46× speedup over the TE. Together with the newly added weight update optimization, miniTE's systems contributions span operator execution, learner updates, and the complete RL pipeline. miniTE integrates all these core configurations into a lightweight framework. Providing a more efficient NVFP4 implementation and offers reliable and reusable infrastructure for the community to investigate and extend native low-precision RL. (Sec 4.3 and 5.3–5.4; App D and F; Repository.)

In summary, extensive validation on large-scale, real-world long-reasoning workloads and a production-grade RL stack supports miniTE's three core contributions: Execution-gap diagnosis provides new insights and systematic diagnostics of cross-engine numerical discrepancies; Practical native NVFP4 RL achieves stable training with near-BF16 quality and end-to-end acceleration; and A lightweight framework with numerically faithful acceleration delivers high-performance, numerically verifiable, plug-and-play open-source infrastructure.

We believe our additional evidence and clarifications have substantively addressed the reviewers' main concerns. We respectfully ask that you consider the revised manuscript, new evidence, and point-by-point responses in your final evaluation.

Thank you again for your time and thoughtful consideration.

Best regards,
The miniTE Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.