acceptodds
Withdrawn from ICLR 2027

Structured Reward Reconstruction for Group-Normalized Policy Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Group-normalized policy updates couple the rewards of all sampled completions. This coupling makes selective verification difficult: an unknown reward changes the update weights even for verified answers. For tasks with one canonical correct answer, we show that the complete update has a finite representation with one state per distinct answer class and one none-correct state. The representation makes classical prediction-correction estimators applicable to this nonlinear target and supports an optimal projection on its observation channel. Public coefficient vectors guide query selection, and detached reconstruction weights implement the estimate through an ordinary language-model loss. Online calibration adapts the reference distribution under a statewise covariance constraint. Experiments on GSM8K use three models, five seeds and a common training recipe. At 400 updates, generation-matched comparisons show fewer new checks than full feedback under shared stopping and answer-cache rules. Half-batch full feedback offers a competitive lower-generation alternative. Fixed-history analyses distinguish query-design effects from prediction correction and statewise covariance control; cost models quantify when fewer checks can offset additional computation.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ZLN9OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies selective verification for GRPO-style group-normalized policy updates when completions can be partitioned into mutually exclusive canonical answer classes. It shows that, under a single-correct-answer assumption, the nonlinear full-feedback update can be represented as a finite state target, then develops a statewise-unbiased channel-projection estimator, query-law optimization, online reference prediction, and covariance-constrained calibration.

Critical strengths

None

Critical weaknesses
  1. The writing is chaotic, difficult to understand, and there are formatting issues, such as some tables and formulas not being numbered, which appear to be generated purely by LLM
  2. The article only conducted experiments on GSM8K
  3. The LLM used in the experiment is outdated
Questions for authors
  1. Can you provide more experimental results besides GSM8K?
  2. Can you use some updated LLMs for experiments? For example, Qwen-3.5, Gemma-4.
Reviewer_rPkKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a structured reward reconstruction method for group-normalized policy learning. For tasks with one canonical correct answer, it represents the complete update using distinct answer classes and a none-correct state, enabling classical prediction-correction estimators and optimal projection on the observation channel. The method uses public coefficient vectors for query selection, detached reconstruction weights for policy updates, and online calibration under a statewise covariance constraint. Experiments on GSM8K with three models and five seeds show fewer new checks than full feedback under generation-matched comparisons, while half-batch full feedback offers a competitive lower-generation alternative. Fixed-history analyses distinguish query-design effects from prediction correction and covariance control, and cost models quantify when verification savings can offset additional computation.

Critical strengths

1.This paper identifies reward coupling in group-normalized policy updates as a key challenge for selective verification and proposes a structured reward reconstruction method to address it.

2.Experiments on GSM8K across three models and five seeds evaluate the trade-off between verification cost and accuracy, while fixed-history analyses examine the contributions of query design, prediction correction, and covariance control.

Critical weaknesses

1.Insufficient comparison with competitive baselines. The evaluation relies primarily on full feedback, Uniform HT, and variants developed within the proposed framework. These comparisons are useful for component analysis but do not sufficiently establish the method's advantage over existing approaches. The authors also acknowledge that a trained Dr. GRPO comparison remains unmeasured.

2.Structured does not establish a clear advantage over Independent. In Table 1, Independent requires fewer verification calls than Structured on all three models, reducing the mean counts from 136.6 to 122.2 for Qwen2.5-3B, from 64.4 to 59.2 for Qwen2.5-7B, and from 482.0 to 455.0 for SmolLM2-1.7B. Meanwhile, the observed accuracy differences are small and statistically inconclusive under the authors' family-adjusted comparisons. Although Structured achieves higher mean accuracy on two models, the paper does not establish that these gains are reliable enough to justify the additional verification cost. This weakens the empirical case for the proposed structured query design over the Independent alternative.

3.The claimed efficiency gains do not translate into measured training speedups. Section 5.3 reports higher training times for Structured than for full feedback on all three models: 90.2 vs. 82.5 minutes for Qwen2.5-3B, 64.2 vs. 63.4 minutes for Qwen2.5-7B, and 83.5 vs. 65.6 minutes for SmolLM2-1.7B. The reported verification savings therefore do not establish an end-to-end efficiency improvement.

4.The evaluation protocol raises concerns about selection bias. The authors acknowledge in Section 7 that the GSM8K test set was used during method development and that the emphasis on update 400, the 21-comparison accuracy family, and stopping/cache analyses was chosen after the initial analysis. These choices weaken the independence of the evaluation and make it difficult to determine whether the reported findings generalize to an untouched test set.

5.The figures do not sufficiently clarify the method or support direct comparisons. Figure 1 presents a basic verification pipeline but does not clearly communicate the relationships among finite-state reconstruction, query design, online adaptation, and covariance-constrained calibration. Figure 2 uses intermediate checkpoints triggered by cumulative label-request thresholds, resulting in non-aligned checkpoints across methods. This makes direct comparisons under matched optimizer steps or verification budgets difficult.

6.The paper lacks sufficient literature support. The paper's reference list is relatively limited given the breadth of the work, which spans partial-feedback estimation, RLVR efficiency, and policy optimization. The limited coverage of relevant literature weakens the support for the problem formulation, methodological choices, and novelty claims. A more comprehensive review of existing work is needed to better contextualize the proposed approach and establish its contributions relative to prior research.

Questions for authors

See above.

Reviewer_2kqMOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies selective verification for group-normalized policy updates when each question has one canonical correct answer. Grouping duplicate answers reduces the possible reward vectors to a finite set, allowing an unbiased estimate of the complete update from selectively observed class labels. The method combines a statewise-unbiased channel projection, query design using public coefficients, and online reference calibration with a covariance constraint. It is implemented through detached weights in a standard language-model loss. Experiments on GSM8K compare verification calls and learning outcomes across three models and five seeds, alongside full-feedback and simpler partial-feedback controls.

Critical strengths
  1. The finite-state reduction is the main technical insight. An unknown reward normally changes the normalization of every completion, so simply estimating each missing reward is insufficient. Exploiting mutually exclusive answer classes makes the complete update estimable without that substitution. The public coefficient matrix and detached weighted loss also make the construction reasonably straightforward to implement (Sections 2 and 3.4).
  2. The statistical statements distinguish reference optimality from statewise unbiasedness. In particular, the reference need not be correct for unbiasedness, and the calibration constraint controls covariance separately in each possible reward state. The paper also states where these guarantees stop: before clipping and AdamW. These qualifications make the theory more useful than an undifferentiated claim of lower-variance training (Section 4).
Critical weaknesses
  1. The practical advantage over the simpler controls is not yet clear. On Qwen 3B, half-batch full feedback reaches 80.03% versus 80.23% with half the generation and slightly fewer checks; Independent uses fewer checks on all three models (Tables 1-2). Structured also takes longer in the recorded original-service runs. The cost scenarios identify possible operating regimes, but measured performance with an expensive verifier is still needed to establish an overall resource benefit.
  2. The additional components have clearer estimator-level justification than learning-level evidence. In Table 3, channel projection and generalized difference have almost identical coefficient errors on the same histories. Guarded calibration reduces that error by 2.34-4.71%, but the accuracy differences against the calibration ablation are small, and all family-adjusted intervals against trained controls include zero. This does not establish equivalence; it leaves uncertain which components improve policy learning beyond the shared finite-state prediction-correction construction.
  3. Frequent gradient clipping makes the connection between the guarantees and the training results harder to interpret. Appendix I.2 reports clipping on 99.28-100% of steps with nonzero coefficients, while the unbiasedness and covariance results apply before clipping and AdamW. The paper acknowledges this limitation, but does not measure how the mean reconstructed batch update changes after clipping. Coefficient-error reductions alone therefore do not show whether the actual training signal is improved in this regime.
  4. The learning evidence is confined to GSM8K after test-set use during development, as disclosed in Section 7. Repeated sampling from the same 256 training questions also gives the answer cache substantial value. The stop-only accounting helps separate the caching effect, but does not test adaptation to a new task or answer distribution. An untouched evaluation with frozen settings would help determine whether the observed tradeoffs extend beyond this development setting.
Questions for authors
  1. Would replacing channel projection with cyclic generalized difference change the learning results when the other components are kept fixed?
  2. Could you include a small comparison with measured verifier costs, alongside Independent and half-batch full feedback?
  3. How much does batch-level gradient clipping affect the agreement between reconstructed and full-feedback updates?
  4. Which settings were informed by test performance, and could you report a frozen-configuration result on an untouched dataset?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.