- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents NoisyGEMM, which enables efficient rollout generation using rank-R perturbed noise added to the weight of the model during its forward pass, added by using randomized basis coefficients per candidate in a shared rank-R Hadamard transformed space with a CMA-ES optimizer, which effectively skips the materialization of perturbed weights on memory. This enables efficient zeroth-order RL training using inference systems such as vLLM, without the need for costly backpropagation.
Critical strengths
The paper provides a novel and effective way of inference-time weight perturbation and batched computation on top of an existing production serving system. Combining such a method with the important problem of efficient RL rollout generation and enabling zeroth-order training of RL problems is interesting and highly valuable.
Critical weaknesses
-
Zeroth-order methods are often compute and time inefficient due to their random nature. There are no analyses of GPU-hour or per-sample training efficiency results provided by the authors. This raises questions about whether the suggested method is a practical alternative to the traditional backpropagation-based training methods.
-
It is unclear which training scenario this method is targeted at. The introduction says it is for inference systems without the resources required to support costly backpropagation, such as on local devices fine-tuning on private data. However, the method requires generating thousands (lambda=512 or 2048, 8192 rollouts) of traces and scoring them to provide feedback to the weight update. It is unclear if such a scale of rollout generation and scoring infrastructure would be feasible on local devices and private data. For idling inference infrastructures on the cloud, traditional RL frameworks such as verl can still utilize the infrastructure for their compute-intensive rollouts. The paper's method has real benefits in that it has smaller resource requirements, but it is hard to know if that would lead to real end-to-end benefits in time or compute use without the missing compute and sample-efficiency analysis.
Questions for authors
-
Can you please provide the total number of candidate traces and the number of generated tokens used to achieve the results in Table 2 for both GRPO (verl) and NoisyGEMM?
-
Can you please provide the total FLOPs and GPU time spent to achieve the results in Table 2, for both GRPO (verl) and NoisyGEMM?
-
In Figure 3, what were GRPO (verl)’s group size, prompts per step, step count, and learning rate, and how were they tuned? Are the number of generated tokens for GRPO (verl) and NoisyGEMM per step equivalent to each other?
-
What kind of training and validation dataset did you use for the experiments? Does the training set contain content from the evaluation benchmarks used in Table 2 in any way?
-
What kind of use case is this method targeting? Is it for replacing the general RL training step during post-training? Is it for user-specific model updates in batched user serving in large-scale inference providers? Or is it for personalized finetuning of on-device models using private data?
2 comments
Thank you for your feedback
Dear Reviewer,
Thank you again for your appreciation of our work and your valuable feedback. We have provided a comprehensive response to your questions, especially with a detailed GPU-hr comparison between GRPO and our NoisyGEMM. We'd very appreciate it if you could acknowledge our response if it has fully addressed your concerns. If you have further questions, feel free to follow up and we will do our best to respond/address them. Thank you!
Response to Reviewer pbiU
Thank you for the incisive questions on sample/FLOP accounting, baseline setup, overlap checks, and deployment scope.
1. Traces, tokens, FLOPs, and GPU time (W1, Q1, Q2; §5.4, Table 3). Node-hours and generated tokens are in the General Response table. To reach GRPO's converged accuracy (2,048 rollouts/step for GRPO vs. 8,192 for NoisyGEMM), it takes GRPO and NoisyGEMM 160 vs. 100 steps (0.33M vs. 0.82M traces) on 1.7B, 180 vs. 80 steps (0.37M vs. 0.66M traces) on 4B, and 300 vs. 120 steps (0.61M vs. 0.98M traces) on 30B-A3B, while NoisyGEMM reaches its higher Table 2 peak at 140 / 140 / 160 steps (1.15M–1.31M traces). Therefore, although GRPO only generates 1/4 rollouts per step (2,048 vs 8,192), it requires more steps to reach the same accuracy as CMA-ES. We also tuned GRPO's batch size and rollouts per question independently to have the total rollouts per step to match NoisyGEMM, but in each configuration, CMA-ES (NoisyGEMM) requires less GPU hours to reach the same performance as GRPO on Qwen3-4B/Qwen3-30B-A3B.
At ≈2P_active forward FLOPs/token (3.4, 8.0, 6.1 GFLOPs/tok on 1.7B, 4B, 30B-A3B), one 16k math step costs 76M tokens / 2.6×10¹⁷ FLOPs on 1.7B and 71M tokens / 5.7×10¹⁷ FLOPs on 4B. Because NoisyGEMM runs forward passes only (1×) while GRPO adds policy log-prob and backward passes (≈5× FLOPs/token; §5.4), multiplying by Table 3's token totals shows NoisyGEMM (@ GRPO) uses fewer total FLOPs at every scale (2.4×10¹⁹ vs. 5.6×10¹⁹ on 1.7B; 4.2×10¹⁹ vs. 1.2×10²⁰ on 4B; 4.6×10¹⁹ vs. 1.8×10²⁰ on 30B-A3B).
2. Fig. 3a GRPO (verl) configuration (Q3). In Fig. 3a (Qwen3-4B, 8k cap, 100 steps), NoisyGEMM uses 1024 × 8 = 8,192 rollouts/step (56.6M tokens/step). GRPO uses 128 prompts × 16 rollouts = 2,048 rollouts/step (14.8M tokens/step), learning rate 3e-5, and no KL penalty, tuned over batch sizes {128, 256} × group sizes {8, 16, 32}. Per-step tokens in Fig. 3a are therefore 4× smaller for GRPO, whereas we give GRPO up to 4× the step budget to match the total tokens used in CMA-ES (NoisyGEMM). Table 2 and 3 reports the accuracy and node-h when GRPO and CMA-ES reach their best validation performance.
3. Datasets and overlap check (Q4; Appendix H). We train on OpenR1-Math-220k, DAPO-Math-17k, and NuminaMath-CoT for math, and Nemotron and Eurus-2-RL for coding, holding out 10k math and 2k coding questions for validation and checkpoint selection. An exact-match and 8-gram check across the 93 math test problems did find one overlapping problem (AIME26 Problem 1; none on AIME25 or HMMT26). Since all runs, including GRPO, share the same training pool, this single overlap should not materially affect our conclusions.
4. Practical deployment regimes vs. verl (W2, Q5). Indeed, 8,192 × 16k math/coding RL is a datacenter workload; however, as a proof of concept, the efficiency advantage of NoisyGEMM on training large models using only decodings opens up the potential of small-population on-device personalization, and we leave this to future work (§6). In datacenters, NoisyGEMM addresses three regimes where verl struggles:
To summarize, idle inference servers can generate rollouts, but not complete a training step (W2). We agree with the reviewer that rollout generation is just inference and can run on idle inference GPUs, for any RL method including verl. The distinction is the update. verl's rollouts are only useful once shipped to a separate trainer that runs the backward pass and optimizer, after which the new policy must be broadcast back to every rollout worker every step (the policy changes each step). So "idle inference infrastructure" buys verl only the rollout phase; it still needs dedicated training hardware and a per-step weight-sync channel, and on one 80 GB A100 that update does not fit beside an active engine.