- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Confidence-Guided Policy Refinement (CGPR), a training-time framework for recovering learning signals from low-dispersion rollout groups in GRPO-based code generation. When an initial group provides little reward contrast, the method constructs a bounded tree of feedback-conditioned code refinements and allocates the search budget using either a UCB-style rule (CVPR) or Thompson sampling (CGPR-TS). A refined candidate replaces a non-anchor trajectory only when it increases group reward variance, while one highest-reward trajectory from the initial group is preserved. The resulting fixed-size group is then used for policy optimization without changing the inference procedure. Experiments with Qwen3-4B-Instruct show improvements over GRPO on APPS and Codeforces, including a 5.0-percentage-point gain in both pass@1 and pass@10 on Codeforces. However, results on the remaining benchmarks are mixed, and several methodological and experimental issues limit the strength of the conclusions.
Critical strengths
-
The paper addresses a concrete limitation of GRPO. When all trajectories receive the same reward, group-centered advantages vanish. Reusing execution feedback from these groups is a sensible way to seek additional useful training examples, particularly when independent generation repeatedly fails.
-
The framework is practical and insightful. Refinement is confined to training, so the method does not require additional search or verifier calls at inference time. Keeping the backbone and reward fixed across the principal GRPO-based comparisons also helps control some sources of variation.
-
The improvements on harder benchmarks are promising. Both variants improve APPS and Codeforces pass@1 and pass@10 relative to GRPO. The paper also reports unfavorable results and acknowledges that the benefits are not uniform, together with the additional training cost and distributional mismatch.
Critical weaknesses
-
Refinements are sampled conditional on the parent program and execution feedback in Eq. (4), whereas their likelihoods in the GRPO update are evaluated under the original prompt, as explicitly acknowledged in Section 6. Adaptive tree selection and variance-based acceptance further change the distribution of the training groups. Please specify the objective under the actual group-construction distribution and explain which aspects of the update are heuristic.
-
Table 1 describes the fine-tuned methods as compute-matched, but Section 5.1 establishes matching optimizer updates rather than total training resources. Figure 2 reports additional rollout expenditure, and Section 6 reports a 37.5% wall-clock increase relative to GRPO. Equal update counts therefore do not separate the benefit of better group construction from the benefit of additional generation and verification. This distinction is particularly relevant because the proposed method explicitly spends extra interactions on selected training examples. Please clarify or correct the comparison label and include at least one downstream comparison at a matched total rollout-token or GPU-hour budget.
-
The comparison between UCB and Thompson sampling does not establish that confidence-guided allocation outperforms simpler feedback-conditioned refinement. It is better to provide more expreiment on Uniform or greedy expansion versus UCB/Thompson sampling. and Variance-based acceptance versus reward-improving acceptance.
-
Table 1 claims five-seed averages and 95% bootstrap confidence intervals, but only point estimates are shown. Appendix C lists seeds under inference settings without clarifying whether they correspond to independent training runs, decoding runs, or both.
Questions for authors
-
How are newly created nodes and expansion counts initialized in Eq. (5)? If a new node has zero prior expansions, the reported makes its exploration bonus overwhelmingly larger than that of a previously expanded node. Under the 16-candidate budget, does the implementation therefore behave mainly as greedy selection among unexpanded nodes? Please provide pseudocode or a short expansion trace.
-
What precisely counts as an unsuccessful refinement in Eqs. (6)–(7)? Does it mean failing all tests, failing to improve reward, or failing the variance-acceptance criterion? How is the Beta distribution updated after a successful refinement, and are successful parents retired? These details are necessary to interpret the stated posterior over refinement success.
-
Which variant does the single CGPR curve in Figure 2 represent? Are the reported fractions measured before or after refinement? Does “uninformative” mean exactly equal rewards or variance below the configured threshold? Reporting both pre-refinement and post-refinement fractions for the same groups would make the diagnostic easier to interpret.
Thank you for the detailed review. Several of the changes in the revision come directly from your questions, especially the one about UCB initialization. Note that we renamed the UCB variant from "CVPR" to CGPR-UCB, which matches CGPR-TS and avoids the clash with the conference name.
W1 (objective under the actual distribution). This is now Sec. 4.3. We write CGPR's objective as the clipped GRPO surrogate taken over the group distribution induced by our construction (initial sampling, feedback-conditioned proposals, adaptive selection, variance-gated replacement), with all trajectories scored under x. Groups that are informative from the start, the anchor, and un-replaced trajectories are on-policy. The importance ratio of replaced trajectories, the tree selection and the variance gating are heuristic and uncorrected. We relate the resulting update to STaR rationalization and make no convergence claim.
W2 (compute-matched label). Corrected to "update-matched" in the Table 1 caption and in Sec. 5.1. We have not yet run a GRPO baseline at matched total rollout budget. In the meantime, the paper reports per-informative-group token cost (Fig. 2b: CGPR 1.17x to 1.65x vs DAPO 1.4x to 2.1x of GRPO) and the 37.5% wall-clock overhead. Sec. 6 states that part of the gain over GRPO may come from the extra interaction. We plan to add the budget-matched run for the final version.
W3 (simpler baselines, acceptance rule). Added in Sec. 5.3 and Table 2 (pass@1, APPS / Codeforces): Random 23.5 / 18.2, Greedy 23.1 / 18.8, UCB 25.2 / 20.6, TS 24.5 / 20.7, against GRPO at 22.5 / 16.0. At the same refinement budget, UCB and TS beat the better of the two simple rules by 1.0 to 1.9 pp, and Greedy is not better than Random. On variance-based vs reward-improving acceptance, we did not run the comparison in this revision. Sec. 4.2 now states when a lower-reward candidate can be accepted: only in uninformative groups that share a non-zero partial reward, never in all-failed groups, since R = 0 is the minimum. Sec. 6 lists the reward-improving variant as open.
W4 (CIs and seeds). The Table 1 caption no longer claims intervals it does not show. Results are means over five seeds (42, 123, 456, 789, 999). CIs are percentile bootstrap over test problems. For GRPO and CGPR-UCB on HumanEval(+) and MBPP(+), the half-widths are at most 0.9 pp, so we now treat differences on those two benchmarks as noise (Sec. 5.1, 5.2). The full per-entry CI table will go into the final version.
Q1 (UCB initialization). Your reading is correct. New nodes start with N_v = 0, so with eps = 1e-6 any unexpanded node outscores every expanded one, and all unexpanded nodes get the same bonus. Appendix F shows that with G = 4, depth 5 and a budget of 16, an unexpanded expandable node always exists. In our configuration, CGPR-UCB therefore never expands a node twice: it is best-first expansion of unrefined attempts by reward, and C has no effect. Sec. 4.1 and Sec. 6 now say this directly. It is not quite greedy, though. Our Greedy baseline keeps re-expanding the current best node, and UCB beats it by 2.1 pp on APPS and 1.8 pp on Codeforces. Pseudocode is in Algorithm 1 (App. E), and App. F walks through the expansion order.
Q2 (TS update). An attempt from node rho is unsuccessful if the child does not improve on the parent, R(rho') <= R(rho). Each failure increments N_rho and thus beta_rho. A new node starts from the reward-based prior Beta(1 + C R_norm, 1 + C (1 - R_norm)) with N = 0. Parents are not retired and stay expandable until they reach the branching factor. Since successes never increment alpha, this is a pseudo-posterior (reward prior plus a failure penalty, similar to the arm penalty in REx) rather than an exact Bayesian posterior; Sec. 4.1 now uses that wording.
Q3 (Figure 2). "Uninformative" means V < tau_group = 1e-4, the same test that triggers refinement, so it includes exact ties and slightly more. For CGPR, the plotted fraction is measured after refinement, on the groups actually passed to the update. Both points are now stated in Sec. 5.4 and in the caption. We agree that pre-refinement fractions for the same groups would make the plot easier to read and will add them.