- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents miniTE, a framework for native NVFP4 reinforcement learning that combines cross-engine numerical diagnostics with optimized MoE expert computation. It shows that matching weights and numerical formats does not ensure consistent learner–sampler policy evaluation. Using W4A4 expert forward computation, BF16 backward computation, and optimized quantization and grouped GEMM, miniTE improves mathematical benchmark performance over standard NVFP4 and achieves 1.39× BF16 learner throughput on eight B300 GPUs. However, increased weight conversion and synchronization costs leave end-to-end RL efficiency unresolved.
Critical strengths
- The paper provides useful diagnostics for examining numerical discrepancies between the learner and sampler.
- The authors report learner-side speedups and provide reasonably detailed implementation and measurement protocols.
Critical weaknesses
-
Limited end-to-end benefit. In many RL workloads, learner computation is not the dominant bottleneck. Quantizing only the expert forward pass further limits the potential overall speedup.
-
Insufficient long-horizon evaluation. The experiments provide limited evidence on genuinely long-horizon reasoning or agentic tasks, making it difficult to assess how learner–sampler discrepancies accumulate with trajectory length or affect kl.
-
Substantial synchronization overhead. Weight conversion and synchronization time increases from 9.94 to 55.26 seconds in the reported colocated experiment, potentially outweighing learner-side savings. This raises scalability concerns for larger, more sparsely activated MoE models.
-
Limited methodological novelty. The contribution primarily consists of numerical diagnostics and engineering optimizations, with limited theoretical or algorithmic innovation.
-
Suggestions. The authors could profile large-scale agentic RL workloads to identify and address actual end-to-end bottlenecks. Broader investigation of rollout–learner scheduling, load balancing, synchronization, and fault tolerance could offer greater practical impact and stronger systems contributions.
Questions for authors
same as weakness
6 comments
Author response (2/3)
W4: Methodological and systems contributions
A3: Thank you for asking us to clarify the research contribution. miniTE combines reproducible diagnosis of cross-engine execution differences, a native NVFP4 training path that sustains learning, and a lightweight, efficient framework validated through numerical checks. Together, these contributions make low-precision RL execution measurable, inspectable, and reproducible.
First, the diagnostics reveal execution differences that remain after nominal quantization recipes are matched. Frozen-weight measurements show that identical parameters and numerical formats are insufficient to ensure identical policy computation. The routing-conditioned comparison further shows residual FC1/FC2 output differences even when expert assignments at the target layer are matched. These findings motivate an explicit execution specification covering scales, rounding, layout, rescaling, casts, and MoE composition, with diagnostics at three boundaries: quantized operands, expert outputs, and token log-probabilities (Section 3 – Section 4.1).
Second, the numerical design is examined through controlled experiments and complete training runs. The forward context-swap experiment verifies that the tested per-token path is independent of changes in the numerical ranges of other tokens in the batch. The gradient comparison establishes the numerical effect of independently selecting backward precision. The separately constructed strict reference in Figure 4 demonstrates that tight cross-engine alignment and sustained learning can coexist with native NVFP4 computation. Complete RL curves and benchmark evaluations assess the learning behavior of the practical training configuration. This is consistent with our explanation of learning gains: an average forward log-probability gap cannot fully characterize forward execution, backward gradients, and the policy updates they jointly produce.
Third, miniTE organizes these designs into an efficient framework with configurable recipes, replaceable operators, and reusable execution components. Quantization directly produces the contiguous FP4 and scale buffers consumed by grouped GEMM, while device-side expert metadata, workspaces, and activation amax values are reused across the execution path. This joint organization reduces repeated scans, representation conversions, and preparation overhead.
Under the same forward recipe, miniTE achieves a 2.46× geometric-mean expert-forward speedup over the TE per-token reference, and the tested API and layout transformations pass the numerical-preservation checks reported in the paper. Configurable quantization recipes, independent backward precision, replaceable expert operators, and reproducible execution environments make these diagnostics and implementations usable within existing RL pipelines. The complete learner and complete RL-cycle measurements in the common response provide the corresponding evidence for practical efficiency.
Author response (3/3)
W5: Profiling real workloads and broader systems optimization
A4: Thank you for suggesting that systems optimization be guided by bottlenecks in actual RL workloads. We have implemented this direction through complete RL-cycle profiling and concrete improvements to the weight-update bottleneck identified in the experiment.
The weight conversion, packing, and synchronization costs discussed in §5.3 and Appendix A are quantified in the historical colocated profile, now Table 15. The new implementation reduces these costs through grouped weight quantization, overlap of weight preparation with asynchronous transfer, and compact representations with improved bucket and buffer organization.
At 512 responses per cycle, the historical configuration recorded 55.26 s for weight updates and 209.44 s for the complete RL cycle. The optimized configuration records 22.61 s and 174.20 s, respectively—reductions of 59.1% and 16.8%.
The controlled weight-update measurements in Appendix F.5, Table 17, further identify the effects of bucket size and pipelining. Using identical frozen weights and grouped quantization, increasing the bucket size reduces serial update time, and enabling pipelining at the same bucket size reduces it further:
| Execution | Bucket size | Weight-update time |
|---|---|---|
| Serial | 512 MiB | 48.07 ± 1.17 s |
| Serial | 2 GiB | 28.74 ± 0.20 s |
| Pipelined | 2 GiB | 20.36 ± 1.55 s |
These separate controlled measurements report mean ± sample standard deviation over three measured updates after one warm-up update.
We believe this improvement extends miniTE’s systems design from the quantization and grouped-GEMM dataflow within the learner to weight updates across the learner–sampler boundary. Rollout–learner scheduling, load balancing, and fault tolerance are important directions for extending the system to larger-scale and multi-turn agentic RL. miniTE’s modular interfaces, execution specification, and cross-engine checks provide reusable numerical validation tools for investigating those changes in execution organization.
Author response (1/3)
Thank you for recognizing the value of the cross-engine numerical diagnostics and for your suggestions regarding complete RL efficiency, long reasoning trajectories, and systems contributions. We have added complete-cycle measurements and weight-update optimizations, and clarify below how the experimental evidence supports the different contributions of the work.
A1: Thank you for highlighting practical RL training costs and synchronization overhead in MoE models. The additional complete-cycle measurements show that miniTE achieves 1.50× and 1.46× speedups over the BF16/Triton baseline used in our training experiments at 256 and 512 responses per cycle, respectively. It also achieves 1.16× and 1.05× speedups over TE NVFP4. These measurements include rollout, learner updates, model offload and reload, and the actual costs of weight conversion and synchronization.
Please see our Common Response: End-to-End Performance for the complete-cycle measurements, major-stage timings, and results for the optimized system. The recovered complete-cycle timings for the historical configurations are reported in Appendix F.3, alongside Table 15. For MoE weight synchronization, compact transfer reduces the measured logical tensor payload by 68.2% relative to BF16, while grouped quantization and pipelined transfer further reduce weight-preparation and synchronization costs. The TE NVFP4 comparison also uses the optimized weight-update pipeline; miniTE therefore retains additional complete-cycle gains with this infrastructure shared.
For larger and more sparsely activated MoE models, the distinction between sparse computation and weight synchronization is important. In the full-weight refresh used here, the synchronization payload depends on the total expert weights being refreshed, even though each token activates only a subset of experts. Compact representation, grouped quantization, and pipelining directly address the associated transfer volume, repeated preparation, and serial execution costs.
A2: Thank you for raising the question of long reasoning trajectories and multi-turn agentic tasks. We recovered and analyzed response-length records from an independent historical TE NVFP4 diagnostic run of Qwen3-30B-A3B. Across 185,600 generated responses, the 99th-percentile length is approximately 7,818 tokens, and the maximum is 20,480 tokens, establishing that the actual native NVFP4 RL workload includes long generated responses.
The records cover 725 rollouts and approximately 232 million response tokens. All lengths come from individual response records:
These statistics establish the actual response-length coverage of the diagnostic workload; the main miniTE experiments provide the evidence for sustained learning and task quality. Figure 3 reports training behavior over the first 1,000 updates, during which miniTE maintains sustained reward improvement. Table 1 evaluates the resulting checkpoints, with miniTE achieving higher mean scores than standard TE NVFP4 on all four mathematics benchmarks: AIME24, AIME25, MATH-500, and GSM8K. Training and evaluation both use the 20,480-token maximum generation length specified in Table 10.
Regarding accumulation along a trajectory, the sequence log-probability ratio for a fixed generated response is the sum of its token-level log-probability differences. The signs and distribution of these local differences therefore matter alongside trajectory length; an average absolute gap alone does not determine sequence-level KL.
The diagnostics in Section 3 examine these local execution differences through frozen weights, scoring of identical sequences, tail statistics, and routing-conditioned measurements. They identify numerical discrepancies that remain after matching the nominal quantization recipe and target-layer expert assignments, including differences that average metrics can obscure. These diagnostics provide a reusable basis for studying mismatch across token positions and interaction turns.