- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PROSE reorganizes execution-based code-generation RL around explicit plans. A per-plan Beta–Bernoulli posterior guides allocation of the remaining execution budget after a uniform probe, while hierarchical plan- and code-level advantages and allocation-dependent weights determine the policy update. The paper analyzes the estimator under a random-effects formulation and evaluates the method against GRPO and GRPO+CoT, with component attribution, budget comparisons, and additional model and domain evaluations.
Critical strengths
-
Plan-level budget allocation and hierarchical credit assignment address a meaningful training-efficiency problem without changing the execution reward. The shared plan statistics provide a coherent way to connect allocation with the update rule.
-
The empirical attribution is relatively thorough. At the matched execution budget, LiveCodeBench pass@1 increases from 17.1 for GRPO to 18.4 with CoT, 20.3 with PVE, and 22.4 with the complete method. Comparisons with GRPO+CoT and post-hoc credit assignment help establish a contribution beyond adding a planning prompt. These results support my weak-accept recommendation.
Critical weaknesses
-
The method combines familiar ideas from hierarchical credit assignment and bandit-style budget allocation. The paper could be clearer about what is new in coupling them through shared plan statistics.
-
The optimality result covers a restricted estimator class under balanced allocation, while the full method uses adaptive allocation and biased weighting. The theory therefore supports a narrower claim than the complete algorithm’s framing might suggest.
Questions for authors
-
Relative to existing hierarchical policy-gradient methods and group-conditioned baselines, what is the precise new contribution of the plan/code advantage decomposition? Please separate what follows from introducing an explicit plan variable from what is specific to PROSE.
-
What is the conceptual necessity of using the same posterior for allocation and backward weighting? In particular, why is multiplying by the realized allocation ratio the distinctive contribution, rather than a heuristic preference for promising plans layered on a standard bandit-allocation rule?
-
The optimality result is scoped to balanced allocation and a symmetric subtractive class, whereas the complete method uses adaptive allocation and biased weighting. What novel theoretical conclusion applies to the deployed method itself, and how should the contribution be stated so that optimality within the restricted estimator class is not mistaken for optimality of the full algorithm?
We understand your questions to be what PROSE adds beyond hierarchical policy gradients, why the same allocation statistics enter the backward update, and what the theory establishes for the deployed method. We clarify the role of repeated implementations per plan and within-problem allocation, explain how realized counts determine the optional backward weight, and compare UCB, Thompson sampling, and a split-sample probability correction under the same execution cap. This lets us state the contribution and its theoretical scope directly. We address these points below.
Q1–Q3 — Contribution and limits
Introducing P into π(p,y|x)=π(p|x)π(y|x,p) and conditioning a baseline on P are standard operations. Our contribution is to make a textual plan the unit of replication, execution allocation, and credit assignment in code RL. Independent implementations of the same plan let us estimate implementation uncertainty separately from differences across plans, and let allocation change within one problem rather than only across prompts. The posterior directly determines the forward allocation score. The backward weight uses the realized count ωᵢ=kMᵢ/N, after rounding and the exploration floor. The backward weight is therefore coupled to the posterior through allocation. This design is optional and serves to align plan updates with execution commitment. It is not an importance correction or a theoretical necessity. We further tested a split-sample, propensity-corrected APRA variant. At N=16, it uses k=4 plans with two fixed probe executions each and eight further draws. The plan-score term uses the probe means and a leave-one-plan-out baseline. For each further draw, we sample plan I from the UCB allocation distribution q conditional on the complete probe batch, with qᵢ≥ε/k. We draw a fresh implementation, subtract its plan's probe mean bᵢ, and multiply the code-score term H(R−bᵢ) by 1/(kqᵢ). Conditional on the plans and probes, its expectation equals the uniform average of the k conditional code gradients, because E[H|pᵢ]=0. With independent on-policy plan draws, the probe-only plan term estimates the plan gradient in expectation. This argument concerns the sequence-level update at the sampling policy, before PPO clipping and regularization. Small qᵢ can increase variance. The correction is specific to the randomized allocator and does not justify an inverse realized-count weight for the original allocation rule.
The two unweighted allocator arms use the same probes, allocation floor, remaining execution cap, and pooled credit rule. The paired gain of realized-count weighting over unweighted UCB is +0.5 points (95% CI [−0.10, 1.10]). The split-sample variant differs from the original weighted method by −0.3 points (95% CI [−0.82, 0.22]). These comparisons leave the gain from count weighting and the performance cost of targeting the uniform objective uncertain. The propensity-corrected variant uses only fresh post-probe implementations for the code term and targets the uniform average over plans. For one exploitation draw and fixed probes, its second moment is (1/k²)Σᵢ E[||H(R−bᵢ)||² | pᵢ, probes]/qᵢ, so rare allocations can contribute substantial variance even though the mean is corrected. The original weighting instead gives more plan-level influence to plans receiving more executions. These differences could affect learning through both gradient variance and the effective optimization target. The performance interval alone cannot distinguish their effects. Per-plan allocation probabilities, the squared norm of the weighted code update, and matched runs varying only the weighting rule would help separate them.
Thank you again for your positive assessment of our paper and for your questions about the method’s contributions and the scope of our theoretical analysis.