acceptodds
Under review as a conference paper at ICLR 2027

PROSE: Enhancing Sample Efficiency in Code Generation via Plan-Aware Rollout Reorganization

Abstract

Group Relative Policy Optimization (GRPO) and related execution-based RL methods for code generation subtract a flat group-relative baseline from every rollout in a group; this baseline is variance-suboptimal as soon as rewards carry latent-plan structure. Programs instantiating the same latent plan tend to succeed or fail together, so execution rewards follow a one-way random-effects mixture whose within-plan intraclass correlation (ICC) is ANOVA-measurable ( on APPS+, on GSM8K). GRPO's i.i.d. rollouts remain marginally exchangeable, so the deficiency lies in the estimator and not in the sampling, but under this mixture the efficient subtractive baseline is plan-conditioned rather than flat. We propose PROSE, which couples explicit plan clustering, posterior-UCB budget reallocation across plans, and an allocation-aware training objective through a single shared plan posterior. Its hierarchical advantage is the uniformly minimum-variance unbiased (UMVU) estimator within the symmetric subtractive class, with a variance-reduction fraction set by . We test this prediction directly: at the training operating point (, ) the theory predicts an advantage-variance ratio of against the flat estimator recomputed on the same batch; the measured value on stored rollouts is . On LiveCodeBench pass@1, PROSE at () exceeds GRPO+CoT at () — strict superiority at half the execution budget — and PROSE at () exceeds GRPO+CoT at (); against vanilla GRPO, and in execution budget only, PROSE at matches GRPO at under a pre-specified equivalence margin of pass@1. On mathematical reasoning, where the ICC is lower, the gain is correspondingly smaller, as the diagnosis predicts.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
reinforcement learning for code generation, GRPO, variance reduction, intraclass correlation, sample efficiency, latent plans, execution-based RL, RLVR

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_idRuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PROSE reorganizes execution-based code-generation RL around explicit plans. A per-plan Beta–Bernoulli posterior guides allocation of the remaining execution budget after a uniform probe, while hierarchical plan- and code-level advantages and allocation-dependent weights determine the policy update. The paper analyzes the estimator under a random-effects formulation and evaluates the method against GRPO and GRPO+CoT, with component attribution, budget comparisons, and additional model and domain evaluations.

Critical strengths
  1. Plan-level budget allocation and hierarchical credit assignment address a meaningful training-efficiency problem without changing the execution reward. The shared plan statistics provide a coherent way to connect allocation with the update rule.

  2. The empirical attribution is relatively thorough. At the matched execution budget, LiveCodeBench pass@1 increases from 17.1 for GRPO to 18.4 with CoT, 20.3 with PVE, and 22.4 with the complete method. Comparisons with GRPO+CoT and post-hoc credit assignment help establish a contribution beyond adding a planning prompt. These results support my weak-accept recommendation.

Critical weaknesses
  1. The method combines familiar ideas from hierarchical credit assignment and bandit-style budget allocation. The paper could be clearer about what is new in coupling them through shared plan statistics.

  2. The optimality result covers a restricted estimator class under balanced allocation, while the full method uses adaptive allocation and biased weighting. The theory therefore supports a narrower claim than the complete algorithm’s framing might suggest.

Questions for authors
  1. Relative to existing hierarchical policy-gradient methods and group-conditioned baselines, what is the precise new contribution of the plan/code advantage decomposition? Please separate what follows from introducing an explicit plan variable from what is specific to PROSE.

  2. What is the conceptual necessity of using the same posterior for allocation and backward weighting? In particular, why is multiplying by the realized allocation ratio the distinctive contribution, rather than a heuristic preference for promising plans layered on a standard bandit-allocation rule?

  3. The optimality result is scoped to balanced allocation and a symmetric subtractive class, whereas the complete method uses adaptive allocation and biased weighting. What novel theoretical conclusion applies to the deployed method itself, and how should the contribution be stated so that optimality within the restricted estimator class is not mistaken for optimality of the full algorithm?

1 comment

We understand your questions to be what PROSE adds beyond hierarchical policy gradients, why the same allocation statistics enter the backward update, and what the theory establishes for the deployed method. We clarify the role of repeated implementations per plan and within-problem allocation, explain how realized counts determine the optional backward weight, and compare UCB, Thompson sampling, and a split-sample probability correction under the same execution cap. This lets us state the contribution and its theoretical scope directly. We address these points below.

Q1–Q3 — Contribution and limits

Introducing P into π(p,y|x)=π(p|x)π(y|x,p) and conditioning a baseline on P are standard operations. Our contribution is to make a textual plan the unit of replication, execution allocation, and credit assignment in code RL. Independent implementations of the same plan let us estimate implementation uncertainty separately from differences across plans, and let allocation change within one problem rather than only across prompts. The posterior directly determines the forward allocation score. The backward weight uses the realized count ωᵢ=kMᵢ/N, after rounding and the exploration floor. The backward weight is therefore coupled to the posterior through allocation. This design is optional and serves to align plan updates with execution commitment. It is not an importance correction or a theoretical necessity. We further tested a split-sample, propensity-corrected APRA variant. At N=16, it uses k=4 plans with two fixed probe executions each and eight further draws. The plan-score term uses the probe means and a leave-one-plan-out baseline. For each further draw, we sample plan I from the UCB allocation distribution q conditional on the complete probe batch, with qᵢ≥ε/k. We draw a fresh implementation, subtract its plan's probe mean bᵢ, and multiply the code-score term H(R−bᵢ) by 1/(kqᵢ). Conditional on the plans and probes, its expectation equals the uniform average of the k conditional code gradients, because E[H|pᵢ]=0. With independent on-policy plan draws, the probe-only plan term estimates the plan gradient in expectation. This argument concerns the sequence-level update at the sampling policy, before PPO clipping and regularization. Small qᵢ can increase variance. The correction is specific to the randomized allocator and does not justify an inverse realized-count weight for the original allocation rule.

From the same PVE checkpoint, N=16LiveCodeBench pass@1 (%, three seeds)
Continue balanced PVE with leave-one-out20.4
Thompson sampling, with ωᵢ=121.5
UCB with ωᵢ=121.8
Split-sample propensity-corrected APRA22.0
UCB with realized-count weighting22.3

The two unweighted allocator arms use the same probes, allocation floor, remaining execution cap, and pooled credit rule. The paired gain of realized-count weighting over unweighted UCB is +0.5 points (95% CI [−0.10, 1.10]). The split-sample variant differs from the original weighted method by −0.3 points (95% CI [−0.82, 0.22]). These comparisons leave the gain from count weighting and the performance cost of targeting the uniform objective uncertain. The propensity-corrected variant uses only fresh post-probe implementations for the code term and targets the uniform average over plans. For one exploitation draw and fixed probes, its second moment is (1/k²)Σᵢ E[||H(R−bᵢ)||² | pᵢ, probes]/qᵢ, so rare allocations can contribute substantial variance even though the mean is corrected. The original weighting instead gives more plan-level influence to plans receiving more executions. These differences could affect learning through both gradient variance and the effective optimization target. The performance interval alone cannot distinguish their effects. Per-plan allocation probabilities, the squared norm of the weighted code update, and matched runs varying only the weighting rule would help separate them.

Thank you again for your positive assessment of our paper and for your questions about the method’s contributions and the scope of our theoretical analysis.

Reviewer_m9QoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PROSE reorganizes code-generation rollouts around explicitly sampled plans. It first estimates plan values using uniform sampling, then reallocates the remaining execution budget through a Beta–Bernoulli posterior and a UCB-style score. Training uses separate plan-level and code-level advantages, with additional weighting under adaptive allocation. The paper motivates this design through within-plan reward correlation and derives a variance identity for plan-centered rewards. Experiments report improvements over GRPO and GRPO+CoT, including higher LiveCodeBench pass@1 with half the execution budget. Ablations examine planning, hierarchical credit assignment, allocation, warm-up, and transfer to another model family and mathematical reasoning.

Critical strengths

The main empirical comparison is reasonably convincing. At (N=16), PROSE improves LiveCodeBench pass@1 from 18.4 for GRPO+CoT to 22.4, and also exceeds GRPO+CoT at (N=32). The comparison includes three training seeds, matched training settings, and a planning baseline. The authors distinguish external checkpoints from training-matched methods and appropriately restrict the fourfold execution-budget claim to vanilla GRPO. The ablations go beyond showing that planning helps. PVE improves over GRPO+CoT, and the post-hoc grouping experiment attempts to hold generation fixed while changing credit assignment. The warm-up and allocation studies provide further evidence that the full procedure benefits from more than a different prompt. The explicit reporting of unfavorable or unresolved findings, including the incomplete factorial design and uncertain interaction, is also useful.

Critical weaknesses
  1. The claimed unbiasedness does not follow from the baseline property used in the proof.

Appendix I states that plan-conditional unbiasedness of the baseline is the condition under which the REINFORCE update is unbiased. This is insufficient when the baseline includes the reward of the same rollout.

\left(1-\frac1M\right) \mathbb E[h_jr_j\mid x,p]. ] Thus, although (\mathbb E[\bar r\mid x,p]=v_p), the resulting code-gradient estimator is attenuated. The plan-level grand mean produces an analogous finite-group effect.

This does not invalidate the reward-residual variance identity or the observed performance gains. It does undermine the transition to Appendix H, which starts with an unbiased estimator of the same gradient. At the default (M=4), the code-level factor is 0.75 and cannot simply be ignored in a quantitative comparison. A correction or leave-one-out construction may address this, but its variance and the combined plan/code update must then be rederived. The minimum second moment of a reward residual is not, by itself, minimum variance for an unbiased gradient estimator.

  1. The gradient-level guarantee relies on approximations that are not controlled by the stated structural assumptions.

Appendix H introduces score–advantage decoupling as Convention C2 and explicitly calls it an approximation. This factorization is used in the gradient-variance calculation, but no bound is given for its error. Applying it to both methods does not establish that the errors cancel: the methods use different score components and different advantages.

Similarly, unweighted plan/code score orthogonality does not alone imply that the reward-weighted gradient channels are uncorrelated. That step requires additional justification. Equation (60) also comes from optimizing a smoothness-based improvement lower bound; ordering such bounds should be distinguished from ordering actual expected improvements.

The paper is careful about several limitations, but these issues affect the derivation itself, beyond checking the score-norm condition A3. I would regard the reward-level calculation as established separately from the claimed policy-improvement guarantee. The reported empirical gains remain interesting, but the current theory does not yet justify the stronger connection.

Questions for authors

Does the implementation correct the finite-group attenuation factors at the plan and code levels? Please give the exact loss normalization and derive the expected combined update before clipping and regularization. How does this change under unequal, reward-dependent (M_i)?

Can the gradient-variance analysis retain the score–reward dependence rather than assume C2? If an approximation is necessary, please quantify its error and show why it preserves the comparison. Please also distinguish a guarantee on improvement lower bounds from a guarantee on actual improvement.

Can the authors measure the full gradient estimator’s mean, variance, and signal-to-noise ratio in a tractable setting where the true gradient is available? The agreement between 0.262 and 0.26 concerns reward advantages, not the full gradient. The proposed split-half validation would be useful, but would not resolve this distinction by itself.

Can the post-hoc control be repeated across training seeds, and how is its plan-level gradient computed when plans are extracted after generation? Also, a flat-baseline control using exactly the same forward shared-plan rollouts as PVE would help separate generation changes from credit assignment.

What connects Proposition F.1’s bound on pooled reward-estimation bias to the bias of the (\omega_i)-weighted policy gradient? Please make this step explicit, including how the warm-up establishes the required posterior-calibration condition when each new plan still receives only one or two probe executions.

3 comments

We understand your main concern to be whether the reward-residual analysis supports the claims about the actual policy update, given finite-group attenuation, reward-weighted covariance, and adaptive sampling. We derive the balanced finite-group factors and the leave-one-out correction, state the conditions for the gradient-variance comparison, and compare the full estimators by exact enumeration in a finite policy. We also compare credit assignment under the same generation protocol and explain which APRA conclusions follow from Proposition F.1 and which are addressed by the split-sample analysis and calibration checks. We address each point below.

Q1 — Finite-group attenuation and the published objective

Let Sᵢ be the score of the sampled plan and Hᵢⱼ the score of code conditional on that plan, using sums of token log-probabilities for sequence scores. Write g_p=E[Sv(P)] and g_c=E[HR]. In balanced sampling with k independent plans and M conditional draws, define P=(1/k)Σᵢ Sᵢ(Vᵢ−V̄) and C=(1/N)Σᵢⱼ Hᵢⱼ(rᵢⱼ−Vᵢ). Only the self term in each mean survives multiplication by its own score. Therefore E[P]=(1−1/k)g_p and E[C]=(1−1/M)g_c. At k=M=4 both factors are 0.75. A mean-zero residual does not establish an unbiased policy gradient. Equation (4) prints a sum over plans plus a sum over programs, with ωᵢ only on the plan term. The printed ratios are πθ(pᵢ|x)/πold(pᵢ|x) and πθ(yᵢⱼ|x,pᵢ)/πold(yᵢⱼ|x,pᵢ). The printed balanced objective includes no leave-one-out factors. Under the sequence-score formulation of Eq. (4), its unclipped, unregularized expected gradient at the sampling policy is k(1−1/k)g_p + N(1−1/M)g_c, up to any common outer loss multiplier. If the trainer uses separate channel means it yields the normalized expectation above. Token-length averaging and plan duplication in rollout buffers can change the effective coefficients. For the reported implementation, we will document the reducer, token masks, and plan deduplication from the run configuration so the effective coefficients can be identified. For an unbiased balanced reference estimator of ∇E[R] at the sampling policy, use P_LOO=kP/(k−1) and C_LOO=MC/(M−1), with one sequence score per plan and one per code sample. These are equivalent to subtracting the corresponding leave-one-out means. At the same operating point, scaling the code residual changes its variance ratio from 0.262 to 0.465 relative to the original uncorrected flat residual on the same batch. A gradient comparison must correct both comparators for their own sampling dependencies and recompute the full covariance. Clipping, KL and later PPO-style updates are outside this on-policy derivation. For reward-dependent counts, let aᵢ be the number of probe successes, mᵢ the probe count and nᵢ=Mᵢ−mᵢ. Conditional on the plans and all probes, E[Vᵢ]=(aᵢ+nᵢvᵢ)/Mᵢ. For the summed stage-two code term, the conditional expectation is nᵢ(1−1/Mᵢ)E[Hr|pᵢ]. The stage-one term contributes its observed scores times rᵢⱼ−(aᵢ+nᵢvᵢ)/Mᵢ. Averaging these probe-dependent quantities and the ωᵢ-weighted plan term does not yield a constant attenuation factor. Applying the balanced LOO correction to all adaptively pooled samples therefore lacks a general unbiasedness guarantee.

Q2 — Gradient covariance and improvement claims

The plan–code score orthogonality in the appendix follows from the conditional score identity E[H|x,p]=0 and the fact that the plan score is fixed given (x,p). That identity continues to hold. For the reward-weighted update G=P+C, the exact covariance is Cov(P)+Cov(C)+Cov(P,C)+Cov(C,P), including the trace contribution 2E[⟨P−EP,C−EC⟩]. The unweighted score identity leaves this reward-weighted cross term to be evaluated separately. To make the C2 approximation explicit, let 𝒫 be the sampled plan set for a fixed input, X=⟨h_a,h_b⟩, Y=A_aA_b, and μ_s=E[A_s | 𝒫]. Relative to the conditional version of the C2 covariance-product term in Appendix H, E[X | 𝒫]Cov(A_a,A_b | 𝒫), the exact remainder is Cov(X,Y | 𝒫)+E[X | 𝒫]μ_aμ_b. The second term matters when the advantages have nonzero conditional means, as can happen with flat and plan-level advantages for a realized set of plans. Both remainder terms can differ across estimators. To express when the variance comparison holds, let V̂_F and V̂_H be the gradient variance traces in the factored model for the flat and hierarchical estimators on the same balanced shared-plan design, after calibrating each estimator to the same mean gradient. Define Δ=V̂_F−V̂_H and δ_j=V_j−V̂_j, where V_j is the exact variance trace. Then V_F−V_H=Δ+δ_F−δ_H. Thus a positive model gap yields the same ordering for the exact estimators whenever |δ_F|+|δ_H|<Δ. The discrepancies account for the omitted cross-channel and factorization terms. In the finite-policy example below we compute the exact covariance directly and compare both estimators at the same mean. The balanced reward-residual identity and the restricted baseline optimality result hold at the reward level. With a common mean gradient, the smoothness analysis relates the variance comparison to a comparison of optimized sufficient one-step improvement bounds. The PL result concerns sufficient iteration bounds under the additional PL assumptions. Actual improvement and actual iteration counts can also depend on how the two trajectories evolve.

Q3 — Exact full-gradient measurement

We used a finite two-stage policy with one shared scalar logit shift θ. At θ=0, P(plan=1)=0.5 and conditional Bernoulli rewards have probabilities 0.2 and 0.7. The shift is added to the plan logit and to both conditional code logits. The exact gradient is 0.3100, consisting of 0.1250 through the plan and 0.1850 through code. At 16 executions, we enumerate the joint distribution, using k=M=4 for shared plans. The shared-plan flat reference weights the plan and code score channels by 4/3 and 16/15, respectively. The i.i.d. flat reference uses standard leave-one-out. SNR is |mean|/SD.

EstimatorMeanVarianceSNR
Shared-plan flat, raw0.267190.0033044.648
Shared-plan flat, corrected0.310000.0055764.152
Hierarchical, raw0.232500.0029774.262
Hierarchical, corrected0.310000.0052924.262
Independent-rollout flat, corrected0.310000.0018047.298

On the same shared-plan design, the sum of the plan and code variances is 0.008118 for the corrected flat estimator and 0.010917 for the corrected hierarchical estimator. The respective plan–code covariances are −0.001271 and −0.002813. Including both cross terms using unrounded values gives total variances of 0.005576 and 0.005292, respectively. The ordering therefore reverses when the cross terms are included: the corrected hierarchy has about 5% lower variance than the corrected flat estimator. The independent-rollout flat comparator has lower variance still in this example. This check isolates the effect of the cross-channel covariance and does not measure the full C2 approximation error. This is a setting where the shared parameter makes the reward-weighted covariance observable. It demonstrates why comparing raw variances or extrapolating from reward residuals is insufficient. The result is local to this finite policy.

Q4 — Same-generation credit assignment controls

An extracted plan is a label derived from a completed program. It is not an on-policy sampled action and has no plan log-probability in the generating policy. In the control below the extracted text receives no gradient. Both retrospective arms retain a common CoT scratchpad objective on tokens actually sampled by the policy. Only the code-token baseline changes. Because extracted membership depends on the sampled completion, we make no unbiasedness claim for this retrospective baseline. For forward plans, both arms score the same sampled plan and code spans, with the same masks and loss reduction. The flat arm uses the grand mean baseline for code; its plan term is held fixed. Both arms use sequence-level corrections for their respective code baseline self terms. We will show frozen-batch diagnostics alongside end-to-end repeats, since policies trained with different objectives can diverge even with identical generation protocols. Five paired seeds gave 18.5 ± 0.39 for CoT/flat and 19.6 ± 0.46 for CoT/retrospective code credit (paired Δ +1.1, 95% CI [0.48, 1.72]). Under the same forward shared-plan protocol, forward/flat reached 19.2 ± 0.42 and forward/hierarchical reached 20.4 ± 0.38 (paired Δ +1.2, 95% CI [0.65, 1.75]). All four arms used uniform allocation and N=16. The forward comparison holds the generator and the plan-token objective fixed by design. The retrospective comparison tests code credit on structure recovered after generation.

Q5 — APRA target, gradient bias and warm-up

Proposition F.1 concerns a pooled scalar reward statistic under the two-stage sampling model. The ω-weighted update includes scores and a plan-relative baseline. Since ωᵢ depends on the probes, terms such as E[ωᵢSᵢ(Vᵢ−V̄)] generally contain score–allocation and reward–allocation dependencies. A bound on the scalar statistic does not bound this vector update. Even treating counts as fixed within an optimization step yields a stop-gradient update on the observed buffer. It does not establish the gradient of an expected policy-dependent allocation objective. We state Proposition F.1 as a bound for the pooled reward statistic, and assess the APRA policy update through the allocation ablation and the split-sample analysis. The Beta posterior for each new plan is initialized anew. PVE warm-up changes the policy that generates the plans and programs, so it may change probe informativeness. It does not guarantee posterior calibration from one or two executions. We will call it a training schedule and evaluate calibration against fresh outcomes. For plans sampled from the step-0 and step-500 checkpoints, we scored the two-probe posterior mean against 32 independently generated audit implementations per plan. Across 400 plans per checkpoint, the Brier score changed from 0.213 to 0.197, and ECE from 0.094 to 0.067, using fixed calibration bins. The audit executions were accounted for separately and were excluded from training. These numbers characterize prediction in the measured population. They give no uniform per-plan calibration or unbiased-gradient guarantee. Our response to idRu details the split-sample update and the conditions under which it estimates the uniform objective.

Reviewer_MVu1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that execution rewards in code RL follow a latent-plan mixture with high within-plan correlation (ICC ≈ 0.72 on APPS+), under which GRPO's flat group baseline is variance-suboptimal. It proposes PROSE, which samples k explicit textual plans with several implementations each, reallocates the execution budget across plans with a Beta–Bernoulli posterior-UCB rule, and applies a hierarchical (plan-level and code-level) advantage with allocation-aware weighting. With Qwen2.5-Coder-7B-Instruct trained on APPS+, PROSE improves LiveCodeBench pass@1 from 17.1 (GRPO) to 22.4 at a matched budget of 16 executions per problem, with smaller gains on HumanEval, MBPP, and GSM8K.

Critical strengths

(S1) Well-motivated direction. The paper identifies a real structure in grouped-sampling RL: rollouts that share a latent strategy tend to succeed or fail together, so treating every rollout in a group as an independent sample wastes budget and mixes plan-level with implementation-level credit. Organizing rollouts and baselines around latent plans, rather than around individual samples, is a sensible response.

(S2) Plan-level credit assignment. Splitting the advantage into a plan-level term on plan tokens and a code-level term on code tokens is a simple way to separate strategy quality from implementation quality. It is also cheap to adopt: roughly 100–200 lines over a GRPO trainer, with about 5% wall-clock overhead.

(S3) Careful attribution and scoping. The gain is decomposed into plan prompting, hierarchical advantage, and adaptive allocation, and a post-hoc control holds generation fixed to isolate the estimator. Execution budget, token usage, and wall-clock are reported separately, and several stronger efficiency claims are explicitly re-scoped.

Critical weaknesses

(W1) Unvalidated plan grouping. The method assumes that rollouts sharing a plan text share a strategy, but neither failure mode of this assumption is measured. Semantically identical plans may be split into different groups; this is only proxied by an unvalidated cosine > 0.85 rule (9% → 13% of pairs). Implementations may also not follow their plan, which is not checked at all. The high ICC (0.72) may partly reflect the shared text prefix rather than semantic plans, and no control with a shared but non-semantic prefix is reported. For unconditioned GRPO rollouts, plans are recovered by LLM extraction followed by forced 4-way cosine clustering without ground-truth labels, so the precision and recall of the grouping are unknown, as the authors acknowledge (App. N).

(W2) Outdated models and benchmarks. All policies are 2023–2024 models (Qwen2.5-Coder 3B/7B, DeepSeek-Coder), and all mechanism analyses (ICC, variance test, ablations) use a single model. Evaluation relies on HumanEval/MBPP, which are near ceiling (88–91%), and a single LiveCodeBench window (Sep–Dec 2024). This matters beyond recency: current reasoning models may explore several strategies within one long rollout, which would weaken the one-plan-per-rollout premise, as the authors themselves note in the limitations.

(W3) Weak baseline and missing comparisons. Vanilla GRPO barely improves over the base model (LCB 16.9 → 17.1 at N=16 and 15.8 at N=8, below base; HumanEval +0.3, MBPP +0.2; Tables 2 and 25), which raises the question of whether the baseline is adequately tuned; the external AceCoder checkpoint in Table 2 reaches 20.4, although it is not training-matched. Adaptive-rollout methods are compared only by design checkmarks (Table 8); in particular, XRPO [2], published at ICML 2026, is not compared empirically, and the allocation ablation (Table 15) covers only variants of the proposed rule. Comparison with the concurrent arXiv work Reinforce-Ada [1] would also be informative. In addition, the Base rows match the officially reported numbers exactly (88.4/83.5 for 7B and 84.1/73.6 for 3B [3]), even though the protocol decodes at T=0.7 and averages over 3 seeds (Table 31). The "DeepSeekCoder-7B-Instruct" checkpoint is also not precisely identified; the DeepSeek-Coder paper reports 78.6/65.4 for its 6.7B-Instruct model [4], versus 83.1/80.4 here. It is unclear whether these baseline numbers were measured under the paper's protocol.

(W4) The theoretical test is close to an identity. With balanced plans, the flat advantage decomposes exactly as A_flat = A_plan + A_code (Secs. 3.2 and 4.1). The reported 73.8% variance reduction of A_code therefore reflects moving between-plan variance to the plan-token term of Eq. (4), not removing it from the gradient. The predicted 0.262 follows directly from the one-way ANOVA decomposition, so its agreement with the measured 0.26 mainly shows that the ICC estimated on one batch carries over to another batch of the same 500 problems. The paper also describes the provenance of ρ_r inconsistently ("the same rollouts" in Secs. 4.2 and 5.3, but a separate M=8 batch in App. K.4), and the split-half check is specified but not run. The component the theory actually covers is modest: +1.9 pass@1, and +1.3 in the post-hoc control, which is reported without seed dispersion (Table 13).

[1] W. Xiong et al. Reinforce-Ada: An Adaptive Sampling Framework for Reinforce-Style LLM Training. arXiv:2510.04996, 2025.

[2] U. Bamba et al. XRPO: Pushing the Limits of GRPO with Targeted Exploration and Exploitation. ICML 2026.

[3] B. Hui et al. Qwen2.5-Coder Technical Report. arXiv:2409.12186, 2024.

[4] D. Guo et al. DeepSeek-Coder: When the Large Language Model Meets Programming – The Rise of Code Intelligence. arXiv:2401.14196, 2024.

Questions for authors
  • Q1. (W1) Can the authors validate plan grouping against an external reference, e.g., human strategy labels on a sample of problems, and report the plan-adherence rate? What is the ICC when rollouts share a non-semantic prefix of similar length?
  • Q2. (W2) Does the method, and the latent-plan correlation it relies on, hold for recent models, including reasoning models that may explore multiple strategies within a single rollout, and on later LiveCodeBench windows?
  • Q3. (W3) Why does vanilla GRPO barely improve over the base model, and fall below it at N=8? Can the authors provide empirical comparisons with Reinforce-Ada and XRPO, confirm whether the Base rows were measured under the paper's protocol, and specify which DeepSeek-Coder checkpoint was used?
  • Q4. (W4) What is the result of the pre-specified split-half check? Can the authors report the variance of the full gradient estimator, including both the plan-level and code-level terms, rather than of the code-level advantage alone?
  • Q5. (Suggestion) Have the authors considered defining latent plans from execution behavior instead of shared plan text, e.g., sampling diverse plans with one implementation each and grouping them by per-test pass/fail vectors? This would give an objective, verifiable notion of plan membership, avoid spending budget on redundant samples of the same plan, and distinguish failure modes among zero-reward rollouts, which the binary-reward ICC cannot.
3 comments

We understand your main concern to be whether PROSE’s gains come from grouping programs that implement the same strategy, or can be explained by shared prefixes, an older evaluation setting, and weaker baselines. We address this with independent expert-LLM strategy verification and neutral-prefix controls, an evaluation on a newer reasoning model and a later LiveCodeBench window, and execution-budget comparisons against tuned and planning-matched baselines. We also clarify which batches support the ICC result and test your proposal to group programs by their execution behavior. We address each point below.

Q1 — Grouping validity and shared-prefix control

A shared plan ID records the sampling intervention. It is an imperfect proxy for the algorithm implemented by the program. We evaluate adherence, cross-ID semantic collisions, and the retrospective grouping pipeline as separate outcomes. We used expert LLMs to review each of 480 programs and 360 program pairs from 120 APPS+ problems. Two distinct verifier models, separate from the strategy extractor used by our grouping pipeline, judged each item independently. For plan adherence, they compared the proposed plan with the program. For strategy equivalence, they inspected only the two programs. The verifiers cited code evidence for their decisions, with execution outcomes and cluster assignments hidden. A third expert LLM resolved disagreements. Before adjudication, Cohen’s κ between the two verifiers was 0.79 for adherence and 0.81 for pairwise equivalence. The adjudicated adherence rate was 84% (problem-bootstrap 95% CI 80–88%). Among sampled pairs with distinct forward plan IDs, the verifiers judged 14% to implement the same algorithmic strategy. Against the adjudicated LLM reference labels, the cosine >0.85 near-duplicate rule had precision 0.80 and recall 0.59. On separately sampled GRPO pairs, the retrospective four-cluster pipeline had pairwise precision 0.81, recall 0.73, and F1 0.77 against the same type of reference. With k=M=4 on a common set of prompts and a fixed checkpoint, ICC was 0.70 [0.65, 0.74] for shared algorithmic plans, 0.12 [0.07, 0.17] for shared neutral prefixes, and 0.07 [0.03, 0.11] for independent neutral prefixes. The neutral texts used the same delimiters and length distribution and contained no solution steps. We resampled prompts, keeping each prompt’s conditions paired. Shared text can account for part of the correlation. The contrast under these controls supports the additional role of algorithmic content. We will report the strategy-label errors alongside the ICC and treat retrospective clusters as noisy labels.

Q2 — Reasoning models and a later window

We trained Qwen3-8B in thinking mode on the same APPS+ split and evaluated on the January–May 2025 LiveCodeBench window. We froze the window, removed overlaps with the RL training set, matched the maximum completion length and execution count, and paired five training seeds. GRPO+CoT and PROSE scored 33.8 ± 0.6 and 35.6 ± 0.5 at N=16, a paired gain of +1.8 points (95% CI [1.13, 2.47]). Within-forward-plan reward ICC was 0.49 [0.42, 0.55]. In a blinded audit of 200 thinking traces, 24% explicitly considered multiple algorithms, while 77% of final programs implemented the assigned strategy. We distinguish strategies explored in the trace from the algorithm ultimately implemented. PROSE improves over GRPO+CoT in this newer model and evaluation window. The gain is smaller than in the original Qwen2.5 setting. Because the model and the task window changed together, this comparison cannot attribute the smaller gain to either factor separately. Transfer to other recent model families and longer agentic tasks remains open.

Q3 — Tuned baselines and checkpoint provenance

We will use the base-checkpoint reruns under a common evaluator for the revised tables, archive the decoded outputs, and record each checkpoint repository and resolved revision. For DeepSeek, we report the rerun on the official checkpoint as a standalone reference. We will report the matched comparison in Table 28 once the run manifests confirm a common revision across methods, or after rerunning all methods from one verified revision. We reran the base evaluator at the declared decoding settings using five decoding seeds. Qwen2.5-Coder-7B-Instruct gave HumanEval/MBPP/LiveCodeBench 88.1/83.0/16.8; the 3B checkpoint gave 83.9/73.2/12.8. A fresh DeepSeek-Coder run using the official deepseek-ai/deepseek-coder-6.7b-instruct checkpoint gave 79.0/66.1/14.9. The planning-matched baselines use the same plan and code prompt templates as PROSE, drawing a fresh plan for each implementation. CoT+RLOO uses a whole-rollout leave-one-out baseline under independent sampling. The CoT versions of Reinforce-Ada and XRPO retain their native prompt-level allocation rules. Every comparison uses the same starting checkpoint, training prompts and held-out LiveCodeBench evaluator. Each method is tuned on the same development split under the same search allowance. The cap averages 16 candidate-program executions per training prompt. Probe and discarded program executions count toward the cap. We record generated tokens and elapsed time separately.

MethodLiveCodeBench pass@1 (%, three seeds, mean ± SD)
Tuned GRPO18.1 ± 0.45
Tuned GRPO+CoT19.2 ± 0.46
CoT+RLOO19.7 ± 0.44
Reinforce-Ada20.0 ± 0.48
XRPO20.8 ± 0.49
CoT+Reinforce-Ada20.9 ± 0.46
CoT+XRPO21.4 ± 0.43
PROSE22.3 ± 0.37

The paired gain over CoT+XRPO, the strongest alternative in this comparison, is +0.9 percentage points (95% seed-level CI [0.39, 1.41]). The gain over XRPO without CoT is +1.5 points (95% CI [0.78, 2.22]). At N=8, tuned vanilla GRPO reaches 16.6 ± 0.48 against the evaluator base of 16.8. The tuned and planning-matched rows narrow the gap relative to the original GRPO baseline. The table supports an execution-budget comparison. Prompt-level allocation and plan-level allocation act at different stages. CoT+XRPO can move executions to training problems where further exploration is useful, and the common planning templates give it access to explicit plans. PROSE can additionally reuse a promising plan and sample another implementation within the same problem. The remaining gap is consistent with that mechanism, but the end-to-end comparison also changes the advantage rule and plan reuse. The same-generator credit comparison in our reply to m9Qo isolates the baseline intervention more directly. A factorial comparison that independently toggles cross-prompt allocation and within-prompt plan allocation would establish whether these mechanisms are complementary.

Q4 — Variance identity and split-half check

For balanced sampling, A_flat = A_plan + A_code exactly. The 73.8% figure concerns code reward residuals. The between-plan component still contributes through the plan-score term. The manuscript’s protocol appendix attributes ρ=0.72 to the M=8 ICC batch and recomputes both residual estimators on a shared M=4 batch using the same problem set. We will report the batch identifiers and aggregation rule alongside this comparison. This is a check of the random-effects model across batch designs on the same prompts. In the specified 250/250 problem split, the unweighted mean of the M=8 fit-side problem ICCs was 0.70, giving an approximate M=4 same-batch ratio of 0.279 when plugged into the formula. The M=4 held-out ratio was 0.254 (95% problem-bootstrap CI [0.232, 0.280]), or −0.025 relative to that plug-in value. The held-out statistic divides two separately averaged variances. In the population formula for that statistic, the matching ICC is weighted by each problem’s total reward variance, since reward variances can vary across problems. We report 0.279 as a plug-in check using the unweighted mean. The displayed interval quantifies uncertainty in the held-out ratio. Our reply to m9Qo Q3 gives the joint gradient mean, variance and signal-to-noise in a setting with an exact reference gradient.

Q5 — Behavioral grouping

We implemented a one-code-per-plan control. For each training prompt, we sample 16 independent plans and one implementation per draw. We compare flat code credit with three post-generation grouping rules on that same candidate-generation protocol. The semantic arm uses algorithm labels extracted without test outcomes. The behavioral arm clusters test pass/fail vectors by normalized Hamming distance. The hybrid uses behavior to refine semantic clusters. Each arm uses the same grouping-test executions, separate reward tests, a common sampled-plan objective, and the same flat baseline fallback for singleton groups. Evaluation uses held-out LiveCodeBench tasks. Grouping and reward tests are disjoint, and both test suites are charged to every arm. The cluster rules and cutoffs are fixed on the development split.

One-code-per-plan training armLiveCodeBench pass@1 (%, three seeds, mean ± SD)
Flat code credit19.3 ± 0.42
Post-generation semantic grouping19.8 ± 0.43
Behavioral grouping20.1 ± 0.44
Semantic + behavioral grouping20.4 ± 0.42

The hybrid improves over flat code credit by +1.1 points (paired 95% CI [0.58, 1.62]). Its advantage over behavioral grouping alone is +0.3 points ([−0.17, 0.77]). Test signatures can distinguish some programs that receive the same scalar reward. They can also merge correct programs that implement different algorithms. This experiment tests grouping for post-generation credit assignment. It provides no evidence that test signatures can replace the forward allocation of implementations to plans, and the resulting score update may be biased because the groups depend on the sampled programs. Two programs with identical scalar rewards can fail different test subsets. Behavioral clusters can therefore give a more specific reference group for the code baseline. Semantic labels may help when the test signatures are ambiguous, while over-refinement can create small clusters and trigger more singleton fallbacks. The current intervals support a hybrid-versus-flat gain in this comparison and leave its incremental value over behavioral grouping unresolved. Agreement with expert-LLM strategy judgments and the cluster-size and fallback distributions would test these explanations.

Comments on the paper

1 comment

We thank the reviewers for their questions about the policy update and the variance analysis. First, self-inclusive baselines attenuate the raw balanced REINFORCE update. We give the two finite-group factors and the corrected estimator, and state the conditions under which the gradient-variance and PL iteration-bound comparisons apply. Second, the original ICC comparison describes reward residuals. In the original LiveCodeBench setting PROSE@16 scored 22.4 against GRPO+CoT@16 at 18.4 and GRPO+CoT@32 at 21.1. These comparisons measure pass@1 at the stated execution budgets. We distinguish that identity from measurements of the complete gradient. The revised empirical evaluation tests strategy labels and plan adherence, separates shared wording from algorithmic content, and compares against tuned and adaptive baselines under a common evaluator. The contribution is a code-RL training design with an explicit plan-level allocation unit, a scoped analysis of reward residuals, and empirical evaluation of allocation and credit assignment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.