acceptodds
Under review as a conference paper at ICLR 2027

Selective Partition Repair for Group-Relative Policy Optimization

Abstract

Executable verifiers provide inexpensive rewards for visual-symbolic reasoning, but false negatives can eliminate useful training signals. In group-relative policy optimization (GRPO), an all-rejected group receives no ranking signal even when it contains correct responses. Correcting individual verdicts can still give a missed correct response a negative advantage, because each advantage depends on the rewards of the entire group. We formulate this problem as hidden-partition recovery and introduce Structured Group-Relative Correction (SGRC). SGRC learns a joint distribution over binary correctness masks from judge-annotated rollout groups. Its energy combines evidence from each response with interactions between the predicted correct and incorrect subsets. On subsequent groups, a confidence gate evaluates each complete predicted mask before applying it, leaving uncertain groups unchanged. The repaired rewards enter the original GRPO normalizer without changing the functional form of the policy objective. On the geometry sym subset of MathV360K, the complete SGRC system, using Qwen2.5-VL-3B-Instruct and 6,912 annotation attempts per run, achieves mean strict answer accuracy 2.96 percentage points above fixed-verifier GRPO across five paired training seeds. On held-out mixed groups, its joint predictor achieves exact-mask recovery 16.7 points above a parameter-matched factorized predictor. These results motivate evaluating group-relative reward repair by the complete partition it recovers, rather than only by individual verdict accuracy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Reinforcement Learning with Verifiable Rewards, GRPO, Structured Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LMXDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper propose SGRC, a new rewarding approach for LLM training. The standard GRPO training rewards sparsely, and thus, fails to recognize and reward equivalent expressions or formatting variants (they get zero rewards). The proposed method addresses this problem by having a repair system to let the system realize other possible correct representations, resulting in denser rewarding. The experiment is demonstrated using a subset of MathV360k.

Critical strengths
  • The paper presents a well-motivated method to make a better reward to train LLM models on structured reasoning tasks. It points out a suboptimal mechanism of the popular GRPO method when it has incorrect outcomes from verifiers. The paper reframes reward repair as hidden-partition recovery problem, which is insightful. The method is presented consistently and empirical results are delivered.
  • The quality of presentation is good overall, offering a fair clarity of explanations of essential concepts and information.
Critical weaknesses
  • The experiments are solely based on a single dataset, even it is a subset of MathV360K dataset. Although it is clarified that it contains image-question pairs from different sources, the result on a single dataset would not be enough to support empirically the claim introduced in the paper. As GRPO is successfully applied and demonstrated in various tasks and domains, it would not be fair to claim that the proposed method outperformed GRPO with a single dataset. It is not clear if the proposed method enhances the performance universally in different scenarios, or it worked well specifically on this dataset.
Questions for authors

How the method will scale further tasks and datasets? Would it enhance the performance overall, beating the GRPO baseline?

Reviewer_sTCWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies verifier false negatives in GRPO for visual-symbolic geometry tasks. When every response in a rollout group is rejected, standard GRPO provides no learning signal. Repairing only some missed correct responses can also assign negative normalized advantages to the remaining correct responses. To address this issue, SGRC predicts a joint correctness mask for the entire group using frozen policy hidden states and parser features. A calibrated confidence gate accepts a repair only when its estimated exact-mask error satisfies a prescribed risk bound; otherwise, the method abstains. The predictor and gate are periodically refreshed using external judge annotations. Evaluated on the geometry_sym subset with Qwen2.5-VL-3B-Instruct and groups of eight responses, SGRC improves held-out accuracy from 20.0% to 23.0% across five paired seeds, compared with 21.6% for a parameter-matched factorized predictor.

Critical strengths
  • The paper clearly explains how partial reward repair can assign negative advantages to missed correct responses.
  • Experiments include parameter- and annotation-matched baselines, five paired seeds, random-partition controls, and human audits.
Critical weaknesses
  • SGRC intervenes only when v=0v=\mathbf{0}, leaving false negatives in partially accepted groups untreated. Figure 3 also reports 0% exact-mask recovery for k≥3k\ge3, which accounts for 22.2% of mixed test groups. The predictor lacks explicit problem or reference inputs, although its contextual hidden states may encode some problem information.
  • In Eq. 5, it stated confidence bound requires at least 57 accepted groups even with zero errors. At 5% coverage, this would require approximately 1140 gate-selection groups. Table 9 reports only 2150 groups per checkpoint across all three splits, while Table7’s split counts are truncated. This raises a feasibility question.
  • Exact inference enumerates 2G2^G masks, with no scaling results beyond G=8G=8. Each run uses 13362 judge calls, yet repairs only about 5% of eligible groups. Evaluation is limited to one model and dataset, and the gain over the factorized baseline is 1.4%.
Questions for authors

Does gate-selection coverage differ from runtime coverage? How common are false negatives in partially accepted groups, and why does recovery fail for k≥3k\ge3? What are the runtime and memory costs at larger group sizes, and how does SGRC compare with simpler uses of the same annotation budget?

Reviewer_LMNwOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The author proposes SGRC, which repairs false-negative verifier rewards in GRPO by jointly predicting the correct/incorrect split of an all-rejected rollout group and only applying the repair when confidence is high. On MathV360K with Qwen2.5-VL-3B, it improves accuracy by 2.96 points over standard GRPO and substantially improves exact-mask recovery over a factorized predictor.

Critical strengths

Here the author identifies GRPO failure mode where partial reward repair can wrongly penalize other correct rollouts, and it gives a clean group-level formulation for fixing it.

Critical weaknesses
  • Here main claim is about joint partition recovery, but at the chosen confidence threshold the method appears to repair only easy single-positive groups, so the key multi-positive case is never actually used during training.

  • cost-effectiveness is unclear: SGRC uses substantial judge supervision, while the dense-judge baseline appears cheaper and performs better, and there is no compute- or annotation-matched comparison.

  • The evaluation is narrow, with one 3B model, one geometry subset, and limited baselines, so it is unclear whether the method generalizes. Or how effective the method is.

Questions for authors
  • Does SGRC still help across other domains, model sizes, and verifier-error patterns, and does the gate actually repair multi-positive groups where joint partition recovery matters?

  • How does SGRC compare with simpler alternatives under matched annotation and compute budgets, especially the dense-judge baseline, and can the authors clarify the unusual logging/count patterns in Tables 9 and 12?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.