- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies reinforcement learning with verifiable rewards (RLVR) through prompt-conditioned policy-gradient flows. Let
be the objective for prompt , and let
be the population objective. The ideal greedy oracle selects the prompt whose gradient has the largest inner product with the population gradient,
\left\langle \nabla J(\theta),\nabla J_x(\theta)\right\rangle.$$ At a common parameter value, this control has at least as large an instantaneous population-objective derivative as the population-gradient flow. The paper observes, however, that this comparison can be explained partly by the winner having a larger component parallel to $\nabla J$. It therefore introduces a normalized greedy flow that divides the selected update by its projection ratio, so that, in the exact-flow setting, its instantaneous projection onto $\nabla J$ matches that of vanilla gradient flow. The practical algorithm samples a batch of prompts, constructs prompt-level policy-gradient candidates, scores each candidate against a leave-one-out mean gradient, and commits the candidate with the largest estimated score. A blockwise Johnson–Lindenstrauss (JL) projection is used to approximate the required inner products at LLM scale. The paper evaluates raw greedy, normalized greedy, uniform prompt selection, and batch-mean SGD using Qwen3-8B-Base with LoRA and baseline-free REINFORCE on a 2,048-prompt subset of DAPO-Math-17K. It reports monitor-objective curves, final evaluations on a held-out in-distribution bank and six mathematical-reasoning benchmarks, and several gradient-geometry diagnostics. The theoretical part gives sufficient conditions under which exact normalized greedy globally dominates exact population-gradient flow under a shared continuous-time clock. The central condition couples a candidate’s projection coefficient to the effect of its orthogonal component on future population-gradient power. An analytical two-dimensional construction shows that the assumptions are non-vacuous. I see two possible ways to understand the paper's intended contribution. The first is as a new optimization method for RLVR. Under this interpretation, the current evaluation is incomplete: it compares variants of baseline-free REINFORCE with SGD under one shared learning-rate schedule, but does not establish an advantage over tuned, mainstream alternatives such as GRPO with AdamW. The second, and seemingly more central, contribution is the discovery and explanation of a previously undocumented phenomenon: maximum-projection prompt selection remains effective even after its estimated parallel advantage is normalized away. This empirical observation is genuinely surprising, but the paper does not yet explain it. The experiments show that a large orthogonal component remains, yet do not establish that the greedy winner's orthogonal component is specially beneficial rather than interchangeable with a random or uniformly sampled orthogonal perturbation; the projection–steering assumption that would provide such an explanation is not empirically tested. The paper therefore currently sits between an optimization-method paper and a mechanistic paper: it presents an interesting algorithmic result and a potentially important phenomenon, but does not yet provide the comparisons needed for the former or the causal evidence needed for the latter.Critical strengths
-
The normalized-greedy phenomenon is genuinely surprising and potentially important.
Under the paper's fixed Qwen3-8B mathematical RLVR setup, selecting the prompt with the largest estimated population-gradient projection substantially outperforms uniform prompt selection and batch-mean SGD. More importantly, much of this advantage remains after the winner's estimated excess parallel projection is normalized away. This is not explained by the elementary maximum-versus-average argument and reveals a nontrivial optimization phenomenon worthy of further study. The fact that the effect appears consistently over training and transfers across several held-out mathematical-reasoning benchmarks makes the observation especially compelling.
-
The experimental design used to expose the phenomenon is thoughtful.
The paper does more than show that raw greedy selection outperforms a baseline. Uniform prompt selection and batch-mean SGD provide useful controls under a common training recipe, and normalized greedy directly addresses the most immediate alternative explanation: that the winner merely takes a larger effective step along the population-gradient direction. Separate training, monitoring, and final-evaluation prompt banks further reduce the risk that the reported effect is an artifact of evaluating on the selected training prompts. These experiments make the normalized-greedy phenomenon credible within the tested setting, even though they do not yet explain its mechanism or establish its generality.
Critical weaknesses
1. The practical normalization is not shown to remove the true parallel advantage
The paper's interpretation of normalized greedy relies on the exact identity
\rho(\theta)^{-1}\nabla J_{x^*}\right\rangle = \|\nabla J\|_2^2$$ where $\nabla J$ is the true population gradient. The implementation cannot compute this quantity. It selects the winner and estimates the normalization ratio from finite, noisy batch gradients. Consequently, the implemented update is normalized relative to a batch-dependent surrogate direction, not necessarily relative to the true $\nabla J$. Conditioning on the maximum-scoring candidate can further bias this relationship even if each candidate's score is unbiased before selection. The paper's batch-size sweep does not test this issue: it changes the candidate pool used by the algorithm. What is needed instead is an independent *evaluation* of the resulting update. At several checkpoints, the authors could use a much larger held-out prompt/rollout bank to obtain a high-accuracy gradient estimate $\widehat{\nabla J}_{\mathrm{ind}}$ and report $$r_{\parallel}=\frac{\langle g_{\mathrm{ind}},u_{\mathrm{NG}}\rangle}{\lVert g_{\mathrm{ind}}\rVert_2^2}.$$ which should be close to one if the practical normalization has actually removed the population-parallel advantage. The same measurement for raw greedy should be larger than one. Until this is verified, the experiments show that a particular batch-normalized greedy algorithm performs well, but do not establish the paper's stronger causal claim that excess projection onto the population gradient has been removed. ### 2. The paper observes but does not explain the normalized-greedy phenomenon The central empirical observation is genuinely surprising. After the maximum-projection prompt has been selected and its estimated excess parallel projection has been normalized away, the resulting update still outperforms the batch-gradient baseline. Because the retained orthogonal component has zero instantaneous contribution to the first-order change in the population objective at the current parameter, this result points to a trajectory effect beyond the elementary maximum-versus-average argument. The experiments, however, do not establish that the greedy winner's orthogonal component is specially beneficial. They do not distinguish the paper's intended explanation, > selecting the maximum-projection prompt and then normalizing its parallel component retains a specially useful orthogonal component, from the weaker possibility, > adding an arbitrary orthogonal component produces a similar trajectory benefit under this training setup. The reported orthogonal-energy and coherence diagnostics show that the winner retains a large, non-degenerate residual, but magnitude and directional diversity do not imply favorable steering. A direct test only requires one clean control. Write the normalized-greedy update as $$u_{\mathrm{NG}}=g_{\mathrm{pop}}+v_{\mathrm{greedy},\perp}.$$ The control should replace its orthogonal component by an isotropically random direction in the trainable parameter space, $$u^{\mathrm{rand}}=g+v^{\mathrm{rand}},\qquad \langle v^{\mathrm{rand}},g\rangle=0,\qquad \lVert v^{\mathrm{rand}}\rVert=\lVert v^{\mathrm{greedy}}\rVert.$$ Thus, the two updates have the same population-gradient component and the same orthogonal magnitude; only the orthogonal direction differs. Comparing their learning curves over multiple random directions would directly test whether the greedy residual is specially useful or whether an arbitrary orthogonal perturbation of the same size produces the same benefit. ### 3. The breadth and prompt-specific nature of the phenomenon are not established The phenomenon is demonstrated in one narrow setting: Qwen3-8B-Base, one LoRA parameterization, a 2,048-prompt subset of DAPO-Math-17K, baseline-free REINFORCE, and SGD. The six reported out-of-distribution evaluations are useful, but they are all mathematical-reasoning benchmarks and evaluate policies trained by the same single procedure. Consequently, the paper does not establish whether normalized-greedy improvement is a broad property of RLVR or a feature of this particular model, domain, estimator, and parameterization. This matters especially because the paper frames the control variable as the **prompt**. The population objective is an expectation over both prompts and sampled responses, and the practical prompt gradient is itself estimated from multiple responses. It is therefore unclear why the relevant candidate should be a prompt-level averaged gradient rather than an individual prompt–response gradient. A tournament over prompt–response pairs could select the sampled trajectory gradient with the largest independently estimated population-gradient projection. If that method exhibits the same effect, the result is more naturally understood as a general per-sample gradient-selection phenomenon, not a prompt-specific one. If averaging responses within a prompt is essential, that would reveal important structure that the present paper does not identify.Questions for authors
I would change my assessment if my major concerns are addressed. I don't have additional questions.
We thank the reviewer for the careful reading, substantive technical comments, and concrete experimental suggestions. We particularly appreciate your distinction between population-level identities, finite-sample implementation, and evidence for the proposed mechanism.
(a) From population-level normalization to finite-sample implementation.
Thank you for the suggestion. Your question concerns the same population-to-finite-sample gap that motivated the analysis outlined below.
(Gap between in-batch and population-level estimation) We indeed have developed a finite-sample analysis of the batch tournament algorithm during preparation of the submission, though we eventually left it out due to it quickly becoming very lengthy and technically involved. We are happy to briefly sketch the analysis below, and will consider including it in the appendix in the future. The finite-sample analysis will, by standard concentration inequalities (e.g., Hoeffding's and Bernstein's inequalities), track how the rollout count G controls each prompt's conditional-gradient estimate, and how the batch size B and G jointly affect the population-gradient mean estimate. Approximating the global exact winner over the prompt space X from the finite batch of prompts also requires the batch to contain a sufficiently good candidate, and this can be guaranteed by assumptions on the tail probability of the prompt distribution. Extending these statements across K updates requires probability control along the adaptive trajectory. With uniform control over candidates and a normalization denominator bounded away from zero, these bounds can quantify the normalization error for the selected candidate.
(Independent evaluation of the resulting update) Your proposed measurement would help us check this gap empirically. However, even with a larger independent bank, the ratio still need not be unbiased since E[1/X] generally differs from 1/E[X]. We appreciate the suggestion and will consider adding this check in future experiments to better connect the theoretical normalization with its practical implementation.
(b) A matched random-orthogonal control.
Thank you for proposing this comparison. By holding the parallel component and orthogonal magnitude fixed while changing the orthogonal direction, it directly tests whether the selected residual is more useful than a matched random orthogonal perturbation, and we will consider incorporating this control in follow-up experiments and discuss its role in the revised manuscript.
(c) Breadth of the phenomenon and the role of prompt-level averaging.
We agree and plan to examine fixed-checkpoint geometry across models and LoRA ranks, including base and post-trained checkpoints.
We also appreciate the suggestion as an alternative candidate-set design (your proposed “tournament over prompt-response pairs”), which is an interesting proposal. Our formulation is motivated by few-shot RLVR, with each prompt defining a conditional objective Jx and its gradient ∇Jx as an admissible control (Section 3, Eqs. (3)-(4), lines 119-139). We will consider the alternative you suggest in future work.
(d) Positioning and extension to other algorithms and optimizers.
Thank you for pointing this out. Our work connects optimization with few-shot RLVR, and our main goal is a theory-driven study of the mechanisms behind prompt selection. REINFORCE with SGD provides a tractable starting point that keeps the experiments aligned with the analysis. As discussed in our limitations (Section 8, lines 480–485), other algorithms and optimizers introduce different update dynamics, so extending the guarantees requires further analysis specific to those choices. We leave these extensions, together with broader empirical comparisons such as tuned GRPO with AdamW, for future work.
Thank you again for the careful engagement. These suggestions provide concrete ways to strengthen the connection between our theoretical formulation, practical implementation, and empirical interpretation.