- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies gradient noise and cross-prompt geometry in language-model RLVR. Its main methodological contribution is to store CountSketch projections of full-model, per-sequence score gradients and exact norms, enabling offline comparisons of baselines, objectives, and rollout allocations. The authors validate sketch fidelity against exact gradients and partial-model proxies, then analyze four models spanning from 0.6B to 8B parameters at three training stages. The main findings are that individual-prompt gradient estimates remain noisy at practical rollout budgets; the shared gradient direction across prompts largely disappears after training; group-relative baselines and pass-rate-based allocation improve estimation reliability; and objectives emphasizing harder prompts become less reliable. The paper also establishes the relationship between GRPO’s successful and failed trajectory contributions and pass@2 and all@2 gradients, and introduces trajectory tracing with a momentum-filtered noise reference.
Critical strengths
-
The measurement methodology is useful and receives meaningful validation. Full-model sketches consistently outperform exact LM-head and final-block proxies in the reported geometry comparisons, while substantially reducing storage. The sketch-width sweeps, exact-gradient comparisons, and displacement reconstruction provide complementary checks rather than relying solely on theoretical sketch guarantees. This is consequential because the paper makes direct measurement of full-model geometry practical and demonstrates that commonly used proxies can give misleading results.
-
The distinction between within-prompt noise and cross-prompt coherence provides a substantive empirical insight. The observation that individual-prompt estimation can improve while aggregate estimation becomes harder is more informative than simply reporting noisy gradients. The authors connect this behavior to measured cross-prompt structure and test the resulting batch-noise predictions. Appendix E’s reliable-prompt subsets and Appendix C.4’s Adam-preconditioned analyses strengthen the evidence that the disappearance of broad alignment is substantive. This supports acceptance as an empirical analysis contribution.
Critical weaknesses
1.The headline rollout requirements are less directly established than the main text suggests. Reliability is measured against a finite, disjoint reference, which itself remains noisy. Appendix A.4 correctly distinguishes equal-budget comparisons, an effectively exact reference, and comparison against the remaining pool: reaching cosine 0.5 requires different budgets under these conventions. However, the main text’s quoted ranges match the median noise scales in Table 8, while the operational reliability definition uses the remaining rollouts. Appendix B.3 also acknowledges that some crossings cannot occur within the pool and that extrapolated values are only indicative; Table 14 nevertheless reports extrapolations using a sigmoid with ceiling 1. Thus, the qualitative conclusion of substantial noise is convincing, but the numerical interpretation of “rollouts required” is ambiguous. Uncertainty intervals are particularly important when the estimated signal norm is small or unresolved, as illustrated in Figure 8. This affects a central quantitative claim, rather than merely presentation.
- The trajectory analysis does not sufficiently justify interpreting near-null displacement as noise-dominated learning. The null assumes independent noise filtered through stationary momentum, whereas the actual runs use AdamW, changing policies, warmup, clipping, and repeated optimizer steps within a rollout batch. Appendix A.7 motivates extending the momentum argument to slowly changing preconditioning, but the paper does not demonstrate that the null reproduces the relevant temporal correlations and step magnitudes of these runs. Moreover, a small excess in displacement can arise from changing signal directions or cancellation, and does not by itself identify the fraction of updates attributable to estimator noise. Table 4 also validates reconstruction against Adam-only displacement, making the treatment of weight decay relevant. The tracing tool is credible, but the interpretation that the directed component is small, which is proposed as a standard diagnostic, needs stronger support.
Questions for authors
-
Which reference convention defines the headline ranges? Please reconcile those ranges with Table 8 and the finite-reference formula in Appendix A.4, and distinguish directly observed crossings from model-derived estimates.
-
Can you provide confidence intervals for the median noise scales and budget estimates, together with their sensitivity to pool size and the treatment of unresolved signal norms? A larger independent reference for a representative subset would help establish whether the numerical estimates are stable.
-
Table 3 describes the snapshot stream as a “token-level importance-weighted score gradient.” What is its exact formula, and under what conditions does it equal the sequence score gradient defined in Section 2? Please explain whether sampling corrections and token normalization preserve the score identity used in the baseline and objective analyses.
-
How are the null simulations calibrated to the observed update magnitudes, effective dimension, learning-rate schedule, and temporal dependence? Does the 2-8% excess persist under a reference that accounts for those quantities, and how is AdamW weight decay handled in both displacement and the null comparison?
5 comments
W2, Q4 - Trajectory null and weight decay.
We agree that an close to the null is consistent with updates that are mostly noise, or with a signal that changes direction, and does not measure the noise share; Section 5 now says so. We also tested the null against the runs (Appendix H):
- A null built from each run's recorded preconditioned gradients (), given random signs and passed through the same momentum, bias correction and learning-rate schedule, keeps the step magnitudes, the warmup, the effective dimension of the noise and the changing policy's gradient statistics. It is centered at 4.35 for each run, like the simulated null. Over the full runs, Qwen3-1.7B (4.68) and Qwen3-4B (4.52) exceed its range, and Qwen3-0.6B (4.44) lies inside it ([4.25, 4.45]).
- Gradient clipping is never active, and each rollout enters one optimizer step. Steps from different rollout batches are nearly uncorrelated ( at lags of 16 or more); within a batch the cosine is +0.011 to +0.018 at lag 1 and −0.015 to −0.020 at lag 8. A null that flips the sign of whole batches keeps the within-batch correlations; it is lower (4.29–4.31), and all three runs exceed its range.
- After the warmup (from step 160), is 4.34 for Qwen3-0.6B and 4.41 for Qwen3-4B, inside the range of the data-driven null (Table 19); Qwen3-1.7B stays above it (4.56 against [4.23, 4.43]). A constant direction of 1% of each step's norm raises from 4.35 to 4.65, while 0.5% gives 4.43, inside the range.
Weight decay is not part of the recorded updates, so , the null and Table 4 refer to the Adam update. The decay moves the weights by 33%, 38% and 10% of the Adam displacement at the last checkpoint compared (Qwen3-0.6B, 1.7B, 4B), nearly orthogonally. Adding it to every step raises by 0.14, 0.15 and 0.02 (at 1,584, 1,600 and 1,200 steps), and would raise the null by about as much (0.15, 0.16 and 0.02, taking the decay orthogonal to the null's displacement), so we report it separately (Appendix B.4).
Once again, thank you for your detailed review. We would appreciate it if you would consider raising your score if these additions have sufficiently addressed your concerns; if not, please let us know and we are happy to answer any remaining questions.
Thank you again for your review!
Given that the discussion period closes at the end of November 18 (AoE), we wanted to follow up to see if our response has addressed your concerns. The revised PDF has changes in blue. To highlight those relevant to your review:
- We count unresolved prompts and provide 95% confidence intervals in Table 8
- We define the sketched stream specifically in Appendix B.2, and describe the null behavior in more detail in Section 5
- We extend our analysis in Appendix B.4 / H with a null from recorded gradients and include the effects of weight decay
If these changes help to address your concerns, we would be grateful if you would consider updating your score. We are also happy to answer any remaining questions before the deadline.
Thank you for your review!
To respond to your questions and feedback:
W1, Q1 - Conventions and Table 8.
The ranges in Section 4.1 are the median noise scales from Table 8. In other words, if two estimates each have Bq rollouts, then the estimates have expected cosine of 0.5; against the exact gradient, Bq/3 rollouts suffice. These values are computed from the measured noise scales; they are not observed crossings. Measured curves use the prompt's remaining rollouts as the reference, against which a curve can reach 0.5 only if Bq is below half the pool.
Our updated revision clarifies which convention is used for which tables (Section 2, the Section 4.1 takeaway and Appendix B.3). All measured curves have one reference, and we report the extrapolated crossings of Table 14, now shaded and labeled as estimates, alongside the median reliability values measured at 512 rollouts.
Observed crossings and the model agree where both exist: on the 16 depth prompts, the 46 of 142 curves that cross 0.5 within the pool are exactly those the model predicts, with crossings within 5% of the predicted ones at the median. Out of sample, Bq from 512 rollouts per prompt predicts each prompt's reliability on the other 1,536 with a median absolute error of 0.01–0.04 at 10 of 12 checkpoints and up to 0.06 at the other two (Appendix B.3).
Q2 - Intervals, pool size and unresolved prompts.
A prompt whose unbiased estimate of ∣gˉq∣2 is not positive has Bq=∞ (the triangles in Figure 8). The submitted medians left these prompts out; they now count as Bq=∞, which raises the medians by 7–35%, and from 13,898 to 43,022 at Qwen3-0.6B base, where 53 of 170 prompts are unresolved. In six of the eight checkpoints of Table 8, a lower bound on Bq places every unresolved prompt above the median.
Table 8 now gives 95% intervals that combine a bootstrap over prompts with the estimation error of each Bq. At Qwen3-0.6B base this interval is unbounded, and the prompt bootstrap alone gives [25,122, 107,565]. The paper now says that the base and final intervals do not overlap for three of the four models (Qwen3-0.6B on the prompt bootstrap alone; not Qwen3-4B).
From the first half of each prompt's rollouts, the medians change by at most 14% in six of eight checkpoints, and rise by 29% and 92% at Qwen3-0.6B step 200 and base, where more prompts become unresolved (Appendix D.1). We did not collect a larger reference, and the extrapolated budgets of Table 14 have no intervals; the held-out test above is our out-of-sample check.
Q3 - Sketched stream.
We sketch ∇θ∑twtlogπθ(ot∣o<t), with wt=πθ(ot∣o<t)/πsamp(ot∣o<t), the ratio of trainer to sampler (vLLM) probabilities, held constant and not truncated, and a plain sum over tokens, without length or loss normalization. It equals the score gradient of Section 2 when wt=1. Under the sampler, E[wt∇θlogπθ(ot∣o<t)∣o<t]=0 for every token, so the expected estimate does not depend on the baseline; this needs the plain sum, since dividing by the length would make each token's weight depend on later tokens. The weights correct each token's own probability but not the distribution of the tokens before and after it, so the stream's reward-weighted mean equals ∇V, the identity Proposition 1 relies on, only when sampler and policy agree.
At a probed checkpoint they hold the same weights, and the log-probabilities differ by a median of 0.31–0.70 nats summed over a response. On fresh Qwen3-4B rollouts, per-prompt gradients from the weighted and unweighted streams have median cosine 0.997, and the median Bq is 630 against 622. Appendix B.2 now states all of this.