acceptodds
Under review as a conference paper at ICLR 2027

Measuring Gradient Geometry in Language-Model RLVR with Sketched Gradients

Abstract

Reinforcement learning with verifiable rewards (RLVR) has emerged as a dominant paradigm in training large language models to solve a wide range of tasks, including mathematical reasoning. Most RLVR pipelines build on a variant of the standard REINFORCE policy-gradient algorithm. Many methods make claims about the noise of policy-gradient estimators, and about the relative geometry between policy gradients on different prompts: for example, baselines as control variates, and pass@ objectives to shift gradient weight from easier to harder problems. However, to our knowledge, no prior work has directly measured these quantities in language-model RLVR, at scale and across training. We compress per-sequence gradients of online rollouts with CountSketch, a linear operation which allows offline analysis to measure the effect of RLVR objectives, rollout allocations and baselines on policy-gradient estimation. After validating that sketches preserve gradient geometry better than common proxies, such as the gradient through the language-model head, we use them to study within-prompt noise and cross-prompt structure. Per-prompt policy-gradient estimates are noisy: at 16 rollouts per prompt, the median cosine similarity between an estimate and a held-out, high-budget estimate for the same prompt is between 0.02 and 0.20, depending on model and checkpoint. We find that, at the beginning of training, prompts share a common policy-gradient direction; at the reward plateau, per-prompt gradients are easier to estimate, but full-batch gradients become more difficult to estimate as the early shared cross-prompt direction disappears. Further, we show that group-relative baselines typically produce more reliable policy-gradient estimates than fixed baselines, that allocating rollouts by estimated pass rate improves reliability over uniform allocation, and that pass@ gradients that shift weight toward harder prompts are harder to estimate as grows. We propose this sketched gradient analysis as a tool to measure the effect of various algorithmic choices in RLVR.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language models, reinforcement learning, rlvr, gradients, sketching, interpretability, training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kX2vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies gradient noise and cross-prompt geometry in language-model RLVR. Its main methodological contribution is to store CountSketch projections of full-model, per-sequence score gradients and exact norms, enabling offline comparisons of baselines, objectives, and rollout allocations. The authors validate sketch fidelity against exact gradients and partial-model proxies, then analyze four models spanning from 0.6B to 8B parameters at three training stages. The main findings are that individual-prompt gradient estimates remain noisy at practical rollout budgets; the shared gradient direction across prompts largely disappears after training; group-relative baselines and pass-rate-based allocation improve estimation reliability; and objectives emphasizing harder prompts become less reliable. The paper also establishes the relationship between GRPO’s successful and failed trajectory contributions and pass@2 and all@2 gradients, and introduces trajectory tracing with a momentum-filtered noise reference.

Critical strengths
  1. The measurement methodology is useful and receives meaningful validation. Full-model sketches consistently outperform exact LM-head and final-block proxies in the reported geometry comparisons, while substantially reducing storage. The sketch-width sweeps, exact-gradient comparisons, and displacement reconstruction provide complementary checks rather than relying solely on theoretical sketch guarantees. This is consequential because the paper makes direct measurement of full-model geometry practical and demonstrates that commonly used proxies can give misleading results.

  2. The distinction between within-prompt noise and cross-prompt coherence provides a substantive empirical insight. The observation that individual-prompt estimation can improve while aggregate estimation becomes harder is more informative than simply reporting noisy gradients. The authors connect this behavior to measured cross-prompt structure and test the resulting batch-noise predictions. Appendix E’s reliable-prompt subsets and Appendix C.4’s Adam-preconditioned analyses strengthen the evidence that the disappearance of broad alignment is substantive. This supports acceptance as an empirical analysis contribution.

Critical weaknesses

1.The headline rollout requirements are less directly established than the main text suggests. Reliability is measured against a finite, disjoint reference, which itself remains noisy. Appendix A.4 correctly distinguishes equal-budget comparisons, an effectively exact reference, and comparison against the remaining pool: reaching cosine 0.5 requires different budgets under these conventions. However, the main text’s quoted ranges match the median noise scales in Table 8, while the operational reliability definition uses the remaining rollouts. Appendix B.3 also acknowledges that some crossings cannot occur within the pool and that extrapolated values are only indicative; Table 14 nevertheless reports extrapolations using a sigmoid with ceiling 1. Thus, the qualitative conclusion of substantial noise is convincing, but the numerical interpretation of “rollouts required” is ambiguous. Uncertainty intervals are particularly important when the estimated signal norm is small or unresolved, as illustrated in Figure 8. This affects a central quantitative claim, rather than merely presentation.

  1. The trajectory analysis does not sufficiently justify interpreting near-null displacement as noise-dominated learning. The null assumes independent noise filtered through stationary momentum, whereas the actual runs use AdamW, changing policies, warmup, clipping, and repeated optimizer steps within a rollout batch. Appendix A.7 motivates extending the momentum argument to slowly changing preconditioning, but the paper does not demonstrate that the null reproduces the relevant temporal correlations and step magnitudes of these runs. Moreover, a small excess in displacement can arise from changing signal directions or cancellation, and does not by itself identify the fraction of updates attributable to estimator noise. Table 4 also validates reconstruction against Adam-only displacement, making the treatment of weight decay relevant. The tracing tool is credible, but the interpretation that the directed component is small, which is proposed as a standard diagnostic, needs stronger support.
Questions for authors
  1. Which reference convention defines the headline n0.5n_{0.5} ranges? Please reconcile those ranges with Table 8 and the finite-reference formula in Appendix A.4, and distinguish directly observed crossings from model-derived estimates.

  2. Can you provide confidence intervals for the median noise scales and budget estimates, together with their sensitivity to pool size and the treatment of unresolved signal norms? A larger independent reference for a representative subset would help establish whether the numerical estimates are stable.

  3. Table 3 describes the snapshot stream as a “token-level importance-weighted score gradient.” What is its exact formula, and under what conditions does it equal the sequence score gradient defined in Section 2? Please explain whether sampling corrections and token normalization preserve the score identity used in the baseline and objective analyses.

  4. How are the null simulations calibrated to the observed update magnitudes, effective dimension, learning-rate schedule, and temporal dependence? Does the 2-8% excess persist under a reference that accounts for those quantities, and how is AdamW weight decay handled in both displacement and the null comparison?

5 comments

Thank you for your review!

To respond to your questions and feedback:

W1, Q1 - Conventions and Table 8.

The ranges in Section 4.1 are the median noise scales from Table 8. In other words, if two estimates each have BqB_q rollouts, then the estimates have expected cosine of 0.5; against the exact gradient, Bq/3B_q/3 rollouts suffice. These values are computed from the measured noise scales; they are not observed crossings. Measured curves use the prompt's remaining rollouts as the reference, against which a curve can reach 0.5 only if BqB_q is below half the pool.

Our updated revision clarifies which convention is used for which tables (Section 2, the Section 4.1 takeaway and Appendix B.3). All measured curves have one reference, and we report the extrapolated crossings of Table 14, now shaded and labeled as estimates, alongside the median reliability values measured at 512 rollouts.

Observed crossings and the model agree where both exist: on the 16 depth prompts, the 46 of 142 curves that cross 0.5 within the pool are exactly those the model predicts, with crossings within 5% of the predicted ones at the median. Out of sample, BqB_q from 512 rollouts per prompt predicts each prompt's reliability on the other 1,536 with a median absolute error of 0.01–0.04 at 10 of 12 checkpoints and up to 0.06 at the other two (Appendix B.3).

Q2 - Intervals, pool size and unresolved prompts.

A prompt whose unbiased estimate of ∣gˉq∣2\\|\bar g_q\\|^2 is not positive has Bq=∞B_q = \infty (the triangles in Figure 8). The submitted medians left these prompts out; they now count as Bq=∞B_q = \infty, which raises the medians by 7–35%, and from 13,898 to 43,022 at Qwen3-0.6B base, where 53 of 170 prompts are unresolved. In six of the eight checkpoints of Table 8, a lower bound on BqB_q places every unresolved prompt above the median.

Table 8 now gives 95% intervals that combine a bootstrap over prompts with the estimation error of each BqB_q. At Qwen3-0.6B base this interval is unbounded, and the prompt bootstrap alone gives [25,122, 107,565]. The paper now says that the base and final intervals do not overlap for three of the four models (Qwen3-0.6B on the prompt bootstrap alone; not Qwen3-4B).

From the first half of each prompt's rollouts, the medians change by at most 14% in six of eight checkpoints, and rise by 29% and 92% at Qwen3-0.6B step 200 and base, where more prompts become unresolved (Appendix D.1). We did not collect a larger reference, and the extrapolated budgets of Table 14 have no intervals; the held-out test above is our out-of-sample check.

Q3 - Sketched stream.

We sketch ∇θ∑twtlog⁡πθ(ot∣o<t)\nabla_\theta \sum_t w_t \log \pi_\theta(o_t \mid o_{ < t}), with wt=πθ(ot∣o<t)/πsamp(ot∣o<t)w_t = \pi_\theta(o_t \mid o_{ < t}) / \pi_{\mathrm{samp}}(o_t \mid o_{ < t}), the ratio of trainer to sampler (vLLM) probabilities, held constant and not truncated, and a plain sum over tokens, without length or loss normalization. It equals the score gradient of Section 2 when wt=1w_t = 1. Under the sampler, E[wt∇θlog⁡πθ(ot∣o<t)∣o<t]=0\mathbb{E}[w_t \nabla_\theta \log \pi_\theta(o_t \mid o_{ < t}) \mid o_{ < t}] = 0 for every token, so the expected estimate does not depend on the baseline; this needs the plain sum, since dividing by the length would make each token's weight depend on later tokens. The weights correct each token's own probability but not the distribution of the tokens before and after it, so the stream's reward-weighted mean equals ∇V\nabla V, the identity Proposition 1 relies on, only when sampler and policy agree.

At a probed checkpoint they hold the same weights, and the log-probabilities differ by a median of 0.31–0.70 nats summed over a response. On fresh Qwen3-4B rollouts, per-prompt gradients from the weighted and unweighted streams have median cosine 0.997, and the median BqB_q is 630 against 622. Appendix B.2 now states all of this.

W2, Q4 - Trajectory null and weight decay.

We agree that an RR close to the null is consistent with updates that are mostly noise, or with a signal that changes direction, and does not measure the noise share; Section 5 now says so. We also tested the null against the runs (Appendix H):

  • A null built from each run's recorded preconditioned gradients (g/v^g/\sqrt{\hat v}), given random signs and passed through the same momentum, bias correction and learning-rate schedule, keeps the step magnitudes, the warmup, the effective dimension of the noise and the changing policy's gradient statistics. It is centered at 4.35 for each run, like the simulated null. Over the full runs, Qwen3-1.7B (4.68) and Qwen3-4B (4.52) exceed its range, and Qwen3-0.6B (4.44) lies inside it ([4.25, 4.45]).
  • Gradient clipping is never active, and each rollout enters one optimizer step. Steps from different rollout batches are nearly uncorrelated (∣cos⁡∣≤0.008|\cos| \le 0.008 at lags of 16 or more); within a batch the cosine is +0.011 to +0.018 at lag 1 and −0.015 to −0.020 at lag 8. A null that flips the sign of whole batches keeps the within-batch correlations; it is lower (4.29–4.31), and all three runs exceed its range.
  • After the warmup (from step 160), RR is 4.34 for Qwen3-0.6B and 4.41 for Qwen3-4B, inside the range of the data-driven null (Table 19); Qwen3-1.7B stays above it (4.56 against [4.23, 4.43]). A constant direction of 1% of each step's norm raises RR from 4.35 to 4.65, while 0.5% gives 4.43, inside the range.

Weight decay is not part of the recorded updates, so RR, the null and Table 4 refer to the Adam update. The decay moves the weights by 33%, 38% and 10% of the Adam displacement at the last checkpoint compared (Qwen3-0.6B, 1.7B, 4B), nearly orthogonally. Adding it to every step raises RR by 0.14, 0.15 and 0.02 (at 1,584, 1,600 and 1,200 steps), and would raise the null by about as much (0.15, 0.16 and 0.02, taking the decay orthogonal to the null's displacement), so we report it separately (Appendix B.4).

Once again, thank you for your detailed review. We would appreciate it if you would consider raising your score if these additions have sufficiently addressed your concerns; if not, please let us know and we are happy to answer any remaining questions.

Thank you again for your review!

Given that the discussion period closes at the end of November 18 (AoE), we wanted to follow up to see if our response has addressed your concerns. The revised PDF has changes in blue. To highlight those relevant to your review:

  • We count unresolved prompts and provide 95% confidence intervals in Table 8
  • We define the sketched stream specifically in Appendix B.2, and describe the null behavior in more detail in Section 5
  • We extend our analysis in Appendix B.4 / H with a null from recorded gradients and include the effects of weight decay

If these changes help to address your concerns, we would be grateful if you would consider updating your score. We are also happy to answer any remaining questions before the deadline.

Reviewer_YfxFOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper develops a CountSketch based framework for measuring full model gradient geometry in language-model RLVR. By storing compressed sequence gradients and their exact norms, the framework enables offline analysis of different baselines, optimization objectives, and rollout allocation strategies without repeatedly computing full-model gradients.

This is primarily a measurement and analysis study rather than a proposal for a new RLVR training algorithm.

Critical strengths
  1. Many RLVR methods are designed based on assumptions regarding gradient variance, cross prompt interference, and sampling efficiency. However, these assumptions are typically studied only through theoretical approximations or local gradient proxies. This paper provides a practical approach to measure the relevant full-model gradient quantities and validates their measurement accuracy against exact gradients.

  2. Per prompt gradients gradually become easier to estimate, the aggregated batch gradients conversely become more difficult to estimate. Meanwhile, the decrease in gradient alignment across prompts provides a geometric perspective for explaining why improvements in single question gradient estimation do not necessarily translate to enhanced batch level gradient reliability.

Critical weaknesses
  1. Section 4.1 and Appendix A.4 discuss different reference protocols, but it is not always clear which one applies to the reported n₀.₅ values. Table 14 also extrapolates beyond the 2,048 rollout pool. In addition, some cross objective comparisons involve gradients with low split half reliability.

  2. For a group of T rollouts, the centered estimator appears to correspond to truncation order T−1 rather than T. The T=1 case makes this clear, since the centered update is zero. It is worth checking whether the reported comparisons are affected.

Questions for authors
  1. Which reference protocol is used for the n₀.₅ values in Section 4.1? How should the extrapolated results in Table 14 be interpreted?
  2. Do the MaxRL comparisons use the population objective or the finite-group centered estimator? Would correcting T to T−1 change the results?
  3. Can offline analysis faithfully represent the long term behavior of online RL? Many experiments collect a large number of rollouts at a fixed checkpoint and then recombine them offline.
3 comments

Thank you for your review! We have made several changes to the paper in response to your feedback.

W1, Q1 - Reference protocol.

We agree with the critique, and have modified our submission to be more clear about which reliability measure is used where.

Specifically, in the updated submission, unless mentioned otherwise, every per-prompt reliability measure is measured against all held-out rollouts, as opposed to equal-budget matching. In other words, for NN total rollouts and an estimator budget of nn rollouts, we match the nn-rollout estimator against an (N−n)(N-n)-rollout target.

The n0.5n_{0.5} values in Section 4.1 were the median per-prompt noise scales BqB_q from Table 8, using the noise model of Appendix A.4. These n0.5n_{0.5} values follow the definition of the gradient noise scale from McCandlish et al.; under our noise model, they are the budgets required for two independent estimates of n0.5n_{0.5} rollouts to have an expected cosine similarity of 0.5. We now report these values directly as BqB_q for clarity. The medians now also count unresolved prompts (Bq=∞B_q = \infty), which raises them by 7–35%, and from 13,898 to 43,022 at Qwen3-0.6B base (Table 8).

Importantly, comparisons at the same budget nn and pool NN come out in the same order under either reference target, since under the noise model reliability against either reference decreases with BqB_q. The equal-budget target simply has additional noise due to its smaller size (nn rollouts instead of N−nN - n).

W1, Q1 - Extrapolation.

You are correct that marked entries in Table 14 are extrapolated beyond 512 rollouts. The way to interpret these numbers is as the budget, measured when possible and otherwise estimated, at which the median prompt's curve would reach 0.5 with a larger pool, extrapolated with a sigmoid.

For many model checkpoints, our rollout budgets are not large enough to achieve this level of reliability: with a pool of 2,048 rollouts, a prompt's curve can reach 0.5 only if its noise scale is below 1,024. We now more clearly demarcate which values are extrapolated versus directly measured, and additionally include the measured reliability values at the highest rollout budget (512 rollouts) and the median noise scale of each baseline in Table 14.

W1 - Comparisons with low split-half reliability.

As pointed out, the noise within a single estimator / objective can lower the measured cosine similarity between it and another estimator. Accordingly, we now report the cross-objective cosines after correcting for per-objective noise in Table 17. On Figure 4 and Figure 14, we also mark values for which the underlying targets have low reliability: for Qwen3, only log⁡V\log V (the T→∞T \to \infty limit of MaxRL) has this much noise.

Under the noise-corrected cosines, our conclusions remain unchanged. GRPO vs. pass@8 ranges from 0.24 to 0.41 at the trained Qwen3 checkpoints, and pass@8 vs. all@2 from −0.22 to 0.07, against 0.94–1.00 for both at the base models. The noise-corrected cosine between positive/negative GRPO signals and pass@2 / all@2 objectives is >0.97 at every trained checkpoint. The exception is log⁡V\log V, whose low cosines with the other objectives are mostly noise, as marked in the updated Figures 4 and 14.

W2, Q2 - MaxRL.

We use the population objective for our comparisons, in which each prompt's gradient is weighted by (1−(1−V)T)/V(1 - (1 - V)^T) / V, where VV is the prompt's pass rate over the full pool rather than a finite-group estimator.

You are correct about the order of the centered estimator, and we have corrected Appendix G.2 accordingly. The estimator has order T−1T - 1, not TT as was originally reported. After updating the description, the results are unchanged. The order-TT and order-(T−1)(T-1) targets have cosine similarity of at least 0.9998 at each checkpoint, and all reported cosine and reliability values stay within 0.012 of their original value with order T−1T - 1.

Q3 - Offline vs. online analysis.

It is certainly challenging to tie offline gradient reliability directly to online performance. Nonetheless, the statistical question we ask is how to best approximate high-group-size policy gradients with smaller group sizes. Under the standard view that higher group sizes train more stably with all else held equal, we consider our proxy of gradient reliability as a reasonable target of study while avoiding the confounds of measuring compounding effects over the course of optimization.

For the estimators we analyze, each snapshot is exact for the first optimizer step on a fresh batch, but it does not show how a choice compounds over training; we now state this scope in the limitations (Appendix J), and hope to study this further.

Once again, thank you for your review! If our changes have sufficiently addressed your concerns, we would appreciate it if you would consider raising your score; if not, please let us know and we are happy to answer any remaining questions.

Thank you again for your review!

Given that the discussion period closes at the end of November 18 (AoE), we wanted to follow up to see if our response has addressed your concerns. The revised PDF has changes in blue. To highlight those relevant to your review:

  • We now use all held-out rollouts as the reference for every per-prompt reliability value. We still report the noise scale BqB_q of McCandlish et al. (Sections 2 and 4.1, Table 8), now under its own name instead of as n0.5n_{0.5}.
  • We mark extrapolated values in Table 14, and report the values measured with 512 rollouts.
  • We report noise-corrected cosines between the gradients of different objectives (Table 17), and mark particularly noisy objectives (log⁡V\log V in Figures 4 and 14).
  • Appendix G.2 now uses order T−1T - 1; the results are unchanged.

If these changes help to address your concerns, we would be grateful if you would consider updating your score. We are also happy to answer any remaining questions before the deadline.

Thank you for your thorough response and for addressing my specific technical questions in the revised PDF. The clarifications regarding the noise scale notation, extrapolated values, and estimator order improve the paper's rigor to some extent. I believe my initial assessment remains appropriate and will maintain my score.

Reviewer_t7WgOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

The authors adopt the CountSketch algorithm as lightweight yet accurate proxy for studying the gradient signals in LLM RLVR training. The authors show the validness of using CountSketch through empirical experiments. Using the tool, they observe that (1) accurate estimation of per-prompt gradient requires a large amount of rollouts (2) careful allocation between prompt/rollout number improves gradient estimation (3) the correlation between the positive/negative gradients of GRPO and pass@2/all@2 objectives (4) learning pace of RL. Besides, some findings are further discussed from theoretical perspectives.

Critical strengths

The adoption of sketch algorithms in LLM RL is novel and interesting. And I do learn new insights into the group-relative objectives such as GRPO. The experiments are generally comprehensive across differnet model sizes and of proper scalibility.

Critical weaknesses

Personally speaking, the paper would benefit from more organized and clear way of writing. The first paragraph of several sections serves as setup and results simultaneously based on my reading. I find the right part of Figure 3 slightly redundant since all of the numbers convey the same message. Also, "relative to uniform-allocation baseline" is confusing at first glance.

Questions for authors

(1) Will you open-source your work as a toolkit for public use?

(2) Have you experimented the method on objectives without group-relative baselines (PPO), and clipping-free variants like CISPO.

3 comments

Thank you for your review! We have made several changes to the paper in response to your feedback.

Presentation.

Per your feedback, we have reorganized the writing in some sections of the paper. Specifically, we modified the following results sections for clarity:

  • Section 4.1: We merge the repeated results paragraphs into one paragraph, and we more clearly and uniformly separate setup from results.
  • Section 4.2: We now introduce the multi-prompt noise model before the results that use it, and label the setup and results.
  • Section 4.3: We include setup and results labels, and do not repeat Appendix F.
  • Section 4.4: We separate out descriptions of objectives and baselines, and explicitly state the baseline results in the text.
  • Figure 3: Panel (b) is now a plot of the median ratio across models against the total budget, in place of the table of per-model values. The caption defines the ratio as the cosine under Neyman allocation divided by the cosine under uniform allocation at the same budget, and no longer calls uniform allocation a "baseline".

Q1 - Open-source.

Yes, we plan to open-source our work in two ways: as scripts for direct reproducibility of the results reported in the paper; and second, as a tool in verl so that the broader community can experiment with gradient sketching as a diagnostic tool in their own runs. The anonymized code for training, sketching and analysis is already in the supplementary material, with the data behind each figure and table.

We expect that adoption should be relatively smooth, given that the cost of our method is quite low: tracing a Qwen3-1.7B run only requires about 2MB per optimizer step, and computing the sketch requires only a linear scan over the gradient that can be built directly into the backward pass.

Q2 - Non-group-relative baselines.

We compare group-mean baselines against fixed baselines in 0,0.25,0.5,0.75,1\\{0, 0.25, 0.5, 0.75, 1\\} and the batch's pooled pass rate. With 16 rollouts per prompt over 256 prompts, we find that the group mean is more reliable than every fixed baseline and the pooled pass rate at every trained checkpoint. For weaker models before training (Qwen3-0.6B, Qwen3-1.7B), we find that a fixed baseline b=0b = 0 can be as or more reliable than the group-relative baseline.

We now include these results in Section 4.4.

Q2 - PPO.

We did not train a learned critic. For the scope of this paper, we study prompt-level baselines. We list token-level PPO in future work (Appendix J): the design space around token-level advantage estimators is larger than that of Monte-Carlo sequence-level baselines, and GAE introduces bias. We agree that this is an interesting exploration and include it as a potential future research direction.

Q2 - CISPO.

CISPO and PPO both clip on the ratio πnew/πold\pi_{\mathrm{new}} / \pi_{\mathrm{old}}: PPO clips its surrogate objective, and CISPO clips a stop-gradient weight on each token while keeping every token's gradient.

In our offline-rollouts setup, we sample every rollout from the checkpoint being measured. Therefore, πnew/πold=1\pi_{\mathrm{new}} / \pi_{\mathrm{old}} = 1, and CISPO-style and PPO-style clipping yield the same gradient as no clipping.

They only differ when the gradient is calculated over stale rollouts. We measure this setup (new Appendix I) by sampling rollouts from Qwen3-4B at training step 20 and computing gradients from training step 21. We compare the stale-rollout policy gradient against a policy gradient computed from fresh rollouts generated from step 21.

Without token-level importance sampling, the stale estimate's expected direction has cosine 0.91 with the on-policy gradient (95% interval [0.69, 1.00]). With token-level importance sampling, it is 0.99 (95% interval [0.97, 1.00]), at about the same reliability as fresh rollouts (0.213 against 0.225 at 16 rollouts). Per-token gradients were not stored, so we could not apply a token-level PPO or CISPO clip to these data; after this step of staleness, only 0.88% of tokens have a ratio outside [0.8, 1.2]. Comparing GRPO and CISPO over full runs comprehensively is outside the scope of this paper, and we have included it in notes for future work.

Once again, thank you for your review. Please let us know if you have any remaining concerns, and we will do our best to address them.

Reviewer_t7WgOpenReview ↗

Thank you for the rebuttal

My major concerns are generally addressed. I have one further question to be discussed. The author said "We expect that adoption should be relatively smooth". Have you tried integrating your method into existing RL libraries like VeRL and TRL. How would you handle situations like expert/tensor-parallel, and different distributed computing backends like FSDP/Magatron? I believe it would highly expensive to acquire full-model gradient in large-scale training, in which gradient is scattered across thousands of GPUs.

Overall the technique is built and verified solidly. I maintain my positive score and look forward to the open-source of your project.

Thank you for your response, and for maintaining your positive score!

To answer your questions: we have begun development to integrate this into verl, but not yet into other frameworks like TRL or slime. Our experiments already run on a modified version of SkyRL and use FSDP across 8 GPUs. Still, integration into a large open-source framework will require more careful implementation and optimization.

Highly sharded training is a good fit for gradient sketching. Because CountSketch is linear, the full gradient does not need to be collected on one device: each shard can compute its own sketches and corresponding inner products, and the sums of these inner products (scalars, cheap to communicate across devices) give the estimate of the overall inner product. We already do this with per-group sketches (Appendix C.1); a tensor-, pipeline- or expert-parallel shard can be treated as a group.

Data parallelism does require communication across ranks, but communication scales with the sketch width, which is orders of magnitude less than the model / gradient size (~6,000x to 34,000x reduction in our Qwen3 experiments).

Once again, thank you, and we’re also looking forward to bringing our work to the broader community.

Comments on the paper

1 comment

Thank you for your questions and feedback. We have uploaded a revised PDF with our changes highlighted in blue (removed text is not shown; the OpenReview diff against the original submission shows every change). Below is a summary of the changes, grouped by topic, with the reviewers who raised each point.

Correction.

  • Probe prompts: Section 2 and Table 3 incorrectly stated that the probe prompts were from DAPO-Math-17k. The prompts used for offline gradient collection were from DeepMath-103K. Section 2, Table 3 and Appendix B.2 now say this.

Reliability and noise scales (YfxF, kX2v).

  • Reference protocol: All per-prompt reliability values now compare an estimate from nn of the NN rollouts with the estimate from the other N−nN-n rollouts. This measures the same noise scale as an equal-size reference (nn rollouts against nn), with less noise for small nn. The gradient noise scale BqB_q (Section 4.1, Table 8, and per baseline in Table 14) is now reported under its own name instead of as n0.5n_{0.5}.
  • Table 8: We now include unresolved prompts (prompts whose unbiased estimate of the squared signal norm is not positive, so Bq=∞B_q = \infty), which the original medians left out. We also add 95% confidence intervals and recompute the medians with half of each pool (Appendix D.1).
  • Table 14: We now report measured values at 512 rollouts next to the extrapolated crossings, and mark which values are extrapolated.

Objectives (YfxF).

  • Comparisons across objectives: We report noise-corrected cosines for the pairs of objectives we discuss (Table 17), and mark objectives whose gradients are too noisy to compare (Figures 4 and 14). The disagreements we discuss remain after correction; the low cosines of log⁡V\log V are mostly noise.
  • MaxRL: We correct the truncation order of the centered estimator in Appendix G.2 to T−1T - 1. The results do not change, since our comparisons use the population objective.

Method (kX2v, t7Wg).

  • Sketched stream: Appendix B.2 now gives the formula and properties of the sketched stream.
  • Trajectory tracing: We add a second null that gives each run's recorded gradients random signs; like the Gaussian null, it is centered at 4.35. Part of the excess over the null comes from the learning-rate warmup: after the warmup, only Qwen3-1.7B stays above this null. Section 5 now notes that an RR near the null is also consistent with a signal that changes direction. Leaving decoupled weight decay out of RR does not change the comparison, since adding it raises RR and the null by about the same amount (Section 5, Appendices B.4 and H).
  • Staleness and clipping: A new Appendix I measures how one step of staleness changes the gradient, with and without importance weights, and compares clipping rules.

Presentation and claims (t7Wg, kX2v).

  • Section 4: We remove repeated results and give each subsection the same structure (setup, results, takeaway). Figure 3(b) is now a plot.
  • Updated claims (added conditions): the group mean is more reliable than every fixed baseline at trained checkpoints; the median per-prompt noise scale is lower at the plateau than at the start of training, with non-overlapping 95% intervals for three of the four models; the excess in tracing stems partly from the learning-rate warmup.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.