acceptodds
Under review as a conference paper at ICLR 2027

Greedy-Prompt Policy Gradient Flow

Abstract

Reinforcement learning with verifiable rewards (RLVR) optimizes a policy over a prompt distribution. Inspired by recent findings that RLVR remains effective when only one or a few prompts are used for gradient updates, we study how each prompt gradient shapes policy-optimization dynamics. This perspective leads to a greedy flow that selects the prompt maximizing instantaneous improvement. Although effective, one can hypothesize that the advantage has a trivial explanation: the selected update may simply have a larger component along the population gradient. We therefore remove this parallel advantage through normalization and find that the normalized flow remains effective, showing that parallel rescaling alone does not explain the observed gap in LLM RLVR. Our diagnostics further document substantial orthogonal motion and nontrivial temporal geometry. Motivated by these observations, we study projection-steering regularity as one possible sufficient condition for global dominance over standard gradient flow. Empirically, we design algorithms approximating these flows and evaluate them on Qwen3-8B-Base with LoRA over DAPO-Math-17K. The results show consistent dominance throughout the observed training horizon, with gains transferring across multiple out-of-distribution benchmarks, including MATH-500, AMC 2023, AIME 2024, AIME 2025, Minerva Math, and OlympiadBench.

Primary area:
reinforcement learning
Keywords:
reinforcement learning with verifiable rewards, policy optimization, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_r6PMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies reinforcement learning with verifiable rewards (RLVR) through prompt-conditioned policy-gradient flows. Let

Jx(θ)=Ey∼πθ(⋅∣x)[R(x,y)]J_x(\theta)=\mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)}[R(x,y)]

be the objective for prompt xx, and let

J(θ)=Ex∼d0[Jx(θ)]J(\theta)=\mathbb{E}_{x\sim d_0}[J_x(\theta)]

be the population objective. The ideal greedy oracle selects the prompt whose gradient has the largest inner product with the population gradient,

\left\langle \nabla J(\theta),\nabla J_x(\theta)\right\rangle.$$ At a common parameter value, this control has at least as large an instantaneous population-objective derivative as the population-gradient flow. The paper observes, however, that this comparison can be explained partly by the winner having a larger component parallel to $\nabla J$. It therefore introduces a normalized greedy flow that divides the selected update by its projection ratio, so that, in the exact-flow setting, its instantaneous projection onto $\nabla J$ matches that of vanilla gradient flow. The practical algorithm samples a batch of prompts, constructs prompt-level policy-gradient candidates, scores each candidate against a leave-one-out mean gradient, and commits the candidate with the largest estimated score. A blockwise Johnson–Lindenstrauss (JL) projection is used to approximate the required inner products at LLM scale. The paper evaluates raw greedy, normalized greedy, uniform prompt selection, and batch-mean SGD using Qwen3-8B-Base with LoRA and baseline-free REINFORCE on a 2,048-prompt subset of DAPO-Math-17K. It reports monitor-objective curves, final evaluations on a held-out in-distribution bank and six mathematical-reasoning benchmarks, and several gradient-geometry diagnostics. The theoretical part gives sufficient conditions under which exact normalized greedy globally dominates exact population-gradient flow under a shared continuous-time clock. The central condition couples a candidate’s projection coefficient to the effect of its orthogonal component on future population-gradient power. An analytical two-dimensional construction shows that the assumptions are non-vacuous. I see two possible ways to understand the paper's intended contribution. The first is as a new optimization method for RLVR. Under this interpretation, the current evaluation is incomplete: it compares variants of baseline-free REINFORCE with SGD under one shared learning-rate schedule, but does not establish an advantage over tuned, mainstream alternatives such as GRPO with AdamW. The second, and seemingly more central, contribution is the discovery and explanation of a previously undocumented phenomenon: maximum-projection prompt selection remains effective even after its estimated parallel advantage is normalized away. This empirical observation is genuinely surprising, but the paper does not yet explain it. The experiments show that a large orthogonal component remains, yet do not establish that the greedy winner's orthogonal component is specially beneficial rather than interchangeable with a random or uniformly sampled orthogonal perturbation; the projection–steering assumption that would provide such an explanation is not empirically tested. The paper therefore currently sits between an optimization-method paper and a mechanistic paper: it presents an interesting algorithmic result and a potentially important phenomenon, but does not yet provide the comparisons needed for the former or the causal evidence needed for the latter.
Critical strengths
  1. The normalized-greedy phenomenon is genuinely surprising and potentially important.

    Under the paper's fixed Qwen3-8B mathematical RLVR setup, selecting the prompt with the largest estimated population-gradient projection substantially outperforms uniform prompt selection and batch-mean SGD. More importantly, much of this advantage remains after the winner's estimated excess parallel projection is normalized away. This is not explained by the elementary maximum-versus-average argument and reveals a nontrivial optimization phenomenon worthy of further study. The fact that the effect appears consistently over training and transfers across several held-out mathematical-reasoning benchmarks makes the observation especially compelling.

  2. The experimental design used to expose the phenomenon is thoughtful.

    The paper does more than show that raw greedy selection outperforms a baseline. Uniform prompt selection and batch-mean SGD provide useful controls under a common training recipe, and normalized greedy directly addresses the most immediate alternative explanation: that the winner merely takes a larger effective step along the population-gradient direction. Separate training, monitoring, and final-evaluation prompt banks further reduce the risk that the reported effect is an artifact of evaluating on the selected training prompts. These experiments make the normalized-greedy phenomenon credible within the tested setting, even though they do not yet explain its mechanism or establish its generality.

Critical weaknesses

1. The practical normalization is not shown to remove the true parallel advantage

The paper's interpretation of normalized greedy relies on the exact identity

\rho(\theta)^{-1}\nabla J_{x^*}\right\rangle = \|\nabla J\|_2^2$$ where $\nabla J$ is the true population gradient. The implementation cannot compute this quantity. It selects the winner and estimates the normalization ratio from finite, noisy batch gradients. Consequently, the implemented update is normalized relative to a batch-dependent surrogate direction, not necessarily relative to the true $\nabla J$. Conditioning on the maximum-scoring candidate can further bias this relationship even if each candidate's score is unbiased before selection. The paper's batch-size sweep does not test this issue: it changes the candidate pool used by the algorithm. What is needed instead is an independent *evaluation* of the resulting update. At several checkpoints, the authors could use a much larger held-out prompt/rollout bank to obtain a high-accuracy gradient estimate $\widehat{\nabla J}_{\mathrm{ind}}$ and report $$r_{\parallel}=\frac{\langle g_{\mathrm{ind}},u_{\mathrm{NG}}\rangle}{\lVert g_{\mathrm{ind}}\rVert_2^2}.$$ which should be close to one if the practical normalization has actually removed the population-parallel advantage. The same measurement for raw greedy should be larger than one. Until this is verified, the experiments show that a particular batch-normalized greedy algorithm performs well, but do not establish the paper's stronger causal claim that excess projection onto the population gradient has been removed. ### 2. The paper observes but does not explain the normalized-greedy phenomenon The central empirical observation is genuinely surprising. After the maximum-projection prompt has been selected and its estimated excess parallel projection has been normalized away, the resulting update still outperforms the batch-gradient baseline. Because the retained orthogonal component has zero instantaneous contribution to the first-order change in the population objective at the current parameter, this result points to a trajectory effect beyond the elementary maximum-versus-average argument. The experiments, however, do not establish that the greedy winner's orthogonal component is specially beneficial. They do not distinguish the paper's intended explanation, > selecting the maximum-projection prompt and then normalizing its parallel component retains a specially useful orthogonal component, from the weaker possibility, > adding an arbitrary orthogonal component produces a similar trajectory benefit under this training setup. The reported orthogonal-energy and coherence diagnostics show that the winner retains a large, non-degenerate residual, but magnitude and directional diversity do not imply favorable steering. A direct test only requires one clean control. Write the normalized-greedy update as $$u_{\mathrm{NG}}=g_{\mathrm{pop}}+v_{\mathrm{greedy},\perp}.$$ The control should replace its orthogonal component by an isotropically random direction in the trainable parameter space, $$u^{\mathrm{rand}}=g+v^{\mathrm{rand}},\qquad \langle v^{\mathrm{rand}},g\rangle=0,\qquad \lVert v^{\mathrm{rand}}\rVert=\lVert v^{\mathrm{greedy}}\rVert.$$ Thus, the two updates have the same population-gradient component and the same orthogonal magnitude; only the orthogonal direction differs. Comparing their learning curves over multiple random directions would directly test whether the greedy residual is specially useful or whether an arbitrary orthogonal perturbation of the same size produces the same benefit. ### 3. The breadth and prompt-specific nature of the phenomenon are not established The phenomenon is demonstrated in one narrow setting: Qwen3-8B-Base, one LoRA parameterization, a 2,048-prompt subset of DAPO-Math-17K, baseline-free REINFORCE, and SGD. The six reported out-of-distribution evaluations are useful, but they are all mathematical-reasoning benchmarks and evaluate policies trained by the same single procedure. Consequently, the paper does not establish whether normalized-greedy improvement is a broad property of RLVR or a feature of this particular model, domain, estimator, and parameterization. This matters especially because the paper frames the control variable as the **prompt**. The population objective is an expectation over both prompts and sampled responses, and the practical prompt gradient is itself estimated from multiple responses. It is therefore unclear why the relevant candidate should be a prompt-level averaged gradient rather than an individual prompt–response gradient. A tournament over prompt–response pairs could select the sampled trajectory gradient with the largest independently estimated population-gradient projection. If that method exhibits the same effect, the result is more naturally understood as a general per-sample gradient-selection phenomenon, not a prompt-specific one. If averaging responses within a prompt is essential, that would reveal important structure that the present paper does not identify.
Questions for authors

I would change my assessment if my major concerns are addressed. I don't have additional questions.

1 comment

We thank the reviewer for the careful reading, substantive technical comments, and concrete experimental suggestions. We particularly appreciate your distinction between population-level identities, finite-sample implementation, and evidence for the proposed mechanism.


(a) From population-level normalization to finite-sample implementation.

Thank you for the suggestion. Your question concerns the same population-to-finite-sample gap that motivated the analysis outlined below.

  • (Gap between in-batch and population-level estimation) We indeed have developed a finite-sample analysis of the batch tournament algorithm during preparation of the submission, though we eventually left it out due to it quickly becoming very lengthy and technically involved. We are happy to briefly sketch the analysis below, and will consider including it in the appendix in the future. The finite-sample analysis will, by standard concentration inequalities (e.g., Hoeffding's and Bernstein's inequalities), track how the rollout count GG controls each prompt's conditional-gradient estimate, and how the batch size BB and GG jointly affect the population-gradient mean estimate. Approximating the global exact winner over the prompt space X\mathcal{X} from the finite batch of prompts also requires the batch to contain a sufficiently good candidate, and this can be guaranteed by assumptions on the tail probability of the prompt distribution. Extending these statements across KK updates requires probability control along the adaptive trajectory. With uniform control over candidates and a normalization denominator bounded away from zero, these bounds can quantify the normalization error for the selected candidate.

  • (Independent evaluation of the resulting update) Your proposed measurement would help us check this gap empirically. However, even with a larger independent bank, the ratio still need not be unbiased since E[1/X]\mathbb{E}[1/X] generally differs from 1/E[X]1/\mathbb{E}[X]. We appreciate the suggestion and will consider adding this check in future experiments to better connect the theoretical normalization with its practical implementation.


(b) A matched random-orthogonal control.

Thank you for proposing this comparison. By holding the parallel component and orthogonal magnitude fixed while changing the orthogonal direction, it directly tests whether the selected residual is more useful than a matched random orthogonal perturbation, and we will consider incorporating this control in follow-up experiments and discuss its role in the revised manuscript.


(c) Breadth of the phenomenon and the role of prompt-level averaging.

We agree and plan to examine fixed-checkpoint geometry across models and LoRA ranks, including base and post-trained checkpoints.

We also appreciate the suggestion as an alternative candidate-set design (your proposed “tournament over prompt-response pairs”), which is an interesting proposal. Our formulation is motivated by few-shot RLVR, with each prompt defining a conditional objective JxJ_x and its gradient ∇Jx\nabla J_x as an admissible control (Section 3, Eqs. (3)-(4), lines 119-139). We will consider the alternative you suggest in future work.

(d) Positioning and extension to other algorithms and optimizers.

Thank you for pointing this out. Our work connects optimization with few-shot RLVR, and our main goal is a theory-driven study of the mechanisms behind prompt selection. REINFORCE with SGD provides a tractable starting point that keeps the experiments aligned with the analysis. As discussed in our limitations (Section 8, lines 480–485), other algorithms and optimizers introduce different update dynamics, so extending the guarantees requires further analysis specific to those choices. We leave these extensions, together with broader empirical comparisons such as tuned GRPO with AdamW, for future work.


Thank you again for the careful engagement. These suggestions provide concrete ways to strengthen the connection between our theoretical formulation, practical implementation, and empirical interpretation.

Reviewer_aMcFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes the use of gradient evaluated at a particular prompt input instead of population gradient for updating the LLM policy in the reinforcement learning from verifiable reward setting. The author studies two gradient direction: one obtained from a prompt maximizing the inner product of gradient evaluated at that prompt and the population prompt, the other obtained from the normalizing the previous update direction. Through ablation study it was proven that the proposed gradient direction is effective in improving the performance of the RLVR algorithm because of the contribution from the component orthogonal to the population gradient.

Critical strengths
  1. The gradient direction obtained using the greedy selection rule obtained better Pass@1 score, boxed answer rate compared to Batch SGD.
  2. The proposed method is able to obtain shorter mean response length compared to Batch SGD.
  3. The ablation studies in Figure 1 shows that the improvement in the performance is due to component orthogonal to the population gradient. Further, normalized greedy direction is also proposed to eliminate the possibility of amplification of the parallel component.
Critical weaknesses
  1. Several terms are introduced in the introduction without proper qualification. It is difficult to understand the introduction without completely going through the paper first. For example, terms like orthogonal energy, leave-one-out tournament, and winner's excess projection are difficult to understand just from the introduction section.
  2. The Related Work section is not informative enough.
  3. There is no description about i^∗\hat{i}^* at the end of section 3.
  4. Figure 2 is not present in the main text. If the appendix is the intended location, please mention that.
  5. In section 4.2, numerical values of the performance metric are mentioned without a description of the performance metric.
  6. The order of sections 4, 5, and 6 seems unnatural. Section 4 starts talking about experiments, and then sections 5 and 6 complete the discussion of the proposed method.
  7. Figure 1.c requires a more elaborate discussion.

If the authors can explain how they will improve the readability of the manuscript considering the above-mentioned weaknesses, I will increase my score.

Questions for authors
  1. What does cumulative-learning-rate clock mean?
  2. Why is Assumption 5 of the PL condition justified in the context of RLVR using an LLM?
  3. It is established that the orthogonal component is useful. Is there any explanation for why the orthogonal component is useful?
2 comments

We thank the reviewer for the constructive feedback and technical questions. Below, we clarify our argument and planned revisions.


(a) Terminology and context. We will clarify the terms by explaining orthogonal energy as the squared magnitude of the component perpendicular to the population gradient, the leave-one-out tournament as scoring against the other prompts' mean gradient, and the winner's excess projection as its parallel advantage over the population update (lines 159-170, 258-284). We will also strengthen Related Work by bringing relevant comparisons from Appendix E.3 into the main text and relating them to our formulation. We are also sorry for not clarifying the notation i^∗\hat i^* at its first use in Section 3, which denotes the index of the candidate with the highest leave-one-out score (Algorithm 2, lines 817–819).

(b) Logical progression. We will refine the presentation and transitions across Sections 4-6 to make the argument clearer. For details, please see our response to Reviewer XHxG, where we take care to illustrate the connections between motivation, theory, and experiments. Detailed preliminaries remain in the appendix for space, and we will explicitly locate Figure 2 there.

(c) Metrics and Figure 1(c). We will clarify Section 4.2 by defining the reported quantity as the held-out estimate of the expected reward objective, with higher values better, and distinguishing it from accuracy (lines 200-208, 250-251, Table 3). For Figure 1(c), larger values of the reciprocal mean squared off-diagonal pairwise cosine indicate lower angular coherence among orthogonal components. We will explain how this spread motivates studying orthogonal steering (lines 238-245, 283-296).


(d) Questions.

1. What is the cumulative-learning-rate clock?

In our paper, this clock is the cumulative learning rate, tk=∑j<kμjt_k=\sum_{j<k}\mu_j. All methods share the same prescribed learning-rate family and initial parameters, and we take their continuous-time limits as the step sizes tend to zero. This common clock isolates prompt control in the first-order limit, abstracting away higher-order discretization effects (Appendix E.2, lines 859-923).

2. Why assume the PL condition for LLM RLVR?

Assumption 5 uses the standard PL condition ∣∇J∣2≥2ν(J∗−J)\\|\nabla J\\|^2\geq 2\nu(J^*-J) (lines 348–349) for convergence guarantees [1]. PL arguments also appear in Transformer theory, including linear self-attention trained on regression tasks [2, Lemma 5.4]. In policy-gradient theory, adequate state coverage yields a PL-type gradient-domination bound under direct policy parameterization, with coverage mismatch controlling its strength [3, Lemma 4.1 and Theorem 4.1]. Recent LLM post-training theory makes a similar connection and shows that coverage collapse can cause exponentially slow convergence [4, Eq. (3) and Theorem 2.1]. These connections motivate our use of PL to relate gradient strength to the remaining objective gap.

Our analysis needs a starting point, and PL is a simple, widely used choice in nonconvex optimization.

3. Why can the orthogonal component help?

Normalized greedy retains much of the empirical gain (lines 258–279), suggesting that its orthogonal component may benefit the optimization on the trajectory level. Since normalization matches vanilla's instantaneous objective growth at the same state in the idealized flow, a possible explanation is that orthogonal movement helps the model make faster progress later.

This brings us to a second-order effect, as orthogonal movement can change the gradient strength that drives subsequent learning. Our analysis examines this possibility by comparing the methods at equal objective levels (Appendix J.1, lines 1661–1707, 1782–1805), and Theorem 1 gives sufficient conditions for this mechanism to yield global dominance (lines 329–375).

We currently view this mechanism as a hypothesis for interpreting the observed gains. Our next revision will aim to provide more direct geometric evidence to assess whether selection by parallel projection also favors beneficial orthogonal steering.


References

[1] Karimi, H., Nutini, J., & Schmidt, M. (2020). Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition. arXiv [Cs.LG]. https://arxiv.org/abs/1608.04636

[2] Zhang, R., Frei, S., & Bartlett, P. L. (2023). Trained Transformers Learn Linear Models In-Context. arXiv [Stat.ML]. https://arxiv.org/abs/2306.09927

[3] Agarwal, A., Kakade, S. M., Lee, J. D., & Mahajan, G. (2020). On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift. arXiv [Cs.LG]. http://arxiv.org/abs/1908.00261

[4] Huang, A., Jiang, N., Kakade, S. M., Malladi, S., Krishnamurthy, A., & Foster, D. J. (2026). A Theory of Policy Optimization in Post-Training: Coverage Collapse and Reference Resets. arXiv [Cs.LG]. http://arxiv.org/abs/2611.14467

The authors have mentioned that they will address the issues raised. However, I don't see the updates in the manuscript. Hence, at the moment it is not possible for me to evaluate whether my concerns are addressed. Given the original version of the manuscript, I will retain my scores.

Reviewer_XHxGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is motivated by the fact that RL works even with small number of prompts. It provides a gradient flow analysis of this idea, and performs some practical experiments.

Critical strengths

None

Critical weaknesses

The main issue is that the paper is not really readable. I understood almost nothing despite being familiar with LLM RLVR. The introduction starts with an unmotivated gradient flow equality and inequality. Some of the explanations are moved to appendix E and are referenced back in main text. Section 4.1 provides excessive engineering explanations that could have been moved to appendix. I tried to skip the theory part and understand what practical implications the paper has, but I could not understand what the paper is exactly implementing and what it is comparing against. Algorithms 1 and 2 should have been included in main text with a better explanation. I believe the paper need significant rewriting and a better flow. Currently each section seems entirely disconnected from its previous and next section without connection or motivation.

Questions for authors

See weaknesses

1 comment

We thank the reviewer for the presentation feedback. We will go through the insights of our paper below, though we believe some of the main points are not that hard to understand given appropriate background in optimization. The detailed preliminaries had to be deferred to the appendix given the space constraints. We will revisit the presentation in the next version to make the progression clearer.


(a) From few-shot RLVR to prompt selection as control.

If a few prompts can improve performance broadly, which prompts should we select, and why? We formulate prompt selection as a control problem through prompt-conditioned gradient flow, a continuous-time idealization that isolates the first-order effect of prompt choice (lines 030-048, 128-146).

Write JxJ_x for the expected reward on prompt xx, J=Ex[Jx]J=\mathbb{E}_x[J_x], and g=∇Jg=\nabla J. By the chain rule, updating along ∇Jx\nabla J_x changes JJ at rate ⟨g,∇Jx⟩\langle g,\nabla J_x\rangle. Thus,

\geq \mathbb{E}_x[\langle g,\nabla J_x\rangle] =\\|g\\|^2.$$ This is the elementary “maximum is at least the average” inequality. Its role is to motivate the greedy control by selecting the prompt giving the greatest instantaneous improvement, and the substantive question is what this local choice implies for the subsequent trajectory (Eqs. (1)-(2), lines 044-053, Eq. (5), lines 147-155, lines 307-319). --- **(b) From instantaneous improvement to global dominance.** For the raw greedy flow, it is easy to show that it will initially dominate the vanilla gradient flow under standard assumptions (Appendix I, lines 1568-1627). Global dominance asks a stronger question: does the greedy flow remain ahead throughout the trajectory? This distinction motivates the subsequent analysis (lines 307-319). --- **(c) Normalized greedy tests whether rescaling the mean gradient explains the gain.** The gain of raw greedy flow could, however, have a largely trivial explanation if the selected gradient were approximately a scaled-up version of the population gradient, or of the batch mean in practice. In that case, much of the gain could come from effectively using a larger learning rate. To examine this possibility, we consider normalized greedy flow, which removes the excess parallel component at the population level (lines 258–269). Its practical counterpart still retains much of the empirical advantage over uniform selection and batch-mean SGD (Figure 1, lines 275–279). This is the central observation motivating our investigation of orthogonal steering. --- **(d) How orthogonal movement can affect later progress.** One possible explanation is that, even when instantaneous improvement is matched in the idealized flow, the orthogonal component may influence the trajectory in a way that helps later progress. To explore this possibility, our analysis compares the methods at equal objective levels and examines how their geometry could affect the time needed for further improvement (Appendix J.1, lines 1661–1707, 1782–1805). Theorem 1 gives sufficient conditions under which this mechanism yields global dominance (lines 329–375). We currently view the proposed orthogonal-steering mechanism as a hypothesis for interpreting the observed gains of normalized greedy, and our next revision will aim to provide more direct geometric evidence to assess whether selection by parallel projection also favors beneficial orthogonal steering. --- **(e) What the experiments implement and compare.** The practical procedure samples a prompt batch, constructs candidate updates and independent gradient estimates, scores each candidate by its instantaneous growth estimate w.r.t. $J$, and commits the highest-scoring update. Normalized greedy rescales that update by following the aforementioned protocol in (c). At a fixed batch size, baselines include uniform prompt selection and batch-mean SGD with matched rollout and optimizer-step budgets (Algorithms 1-2, lines 791-826, and lines 177-184, 200-215, 475-476). --- We will revisit the presentation of Algorithms 1-2 and Sections 4.1-4.2 to make the procedure and baselines easier to follow. We have taken care to respond substantively and professionally, and would appreciate the same standard of engagement in return. In particular, please identify the specific claim, assumption, derivation, or experimental comparison underlying the soundness assessment, so that we can address it precisely.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.