acceptodds
Under review as a conference paper at ICLR 2027

TRUER: Test-Time Reward Correction Using an Error-Aware Rectifier for Reinforcement Learning

Abstract

Test-time reinforcement learning enables language models to adapt using rewards derived from their own answers. However, majority voting can reinforce incorrect solutions even when correct alternatives are available. Our training audit shows that false positives—incorrect responses receiving positive rewards—dominate reward errors. We focus on a practical regime where ground-truth rewards are difficult to obtain at scale, while a limited labeled source set is available. We introduce TRUER, a framework that uses limited supervision to train a reward rectifier that distinguishes correctness from consensus. To achieve this goal, we train the rectifier with an emphasis on incorrect majorities and correct minorities, targeting cases where consensus and correctness disagree. After that, we freeze the rectifier and use it to correct rewards during subsequent test-time adaptation. By improving reward accuracy and thereby model performance, TRUER consistently outperforms naive TTRL in both metrics across all evaluated configurations.

Primary area:
reinforcement learning
Keywords:
test-time reinforcement learning, reward estimation, LLM verification, self-consistency, mathematical reasoning, reinforcement learning from verifiable rewards

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pRj9OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2(−1)
Soundness
2
Read review
Paper summary

The proposed TRUER addresses a key flaw in Test-Time Reinforcement Learning (TTRL): majority-voting rewards are unreliable because consensus does not equal correctness. The authors train an error-aware rectifier on a small labeled source dataset to distinguish "widely agreed" answers from "actually correct" ones, then use it to correct reward signals at test time without any target-domain labels. Across several configurations (various models and math tasks), TRUER consistently outperforms naive TTRL, narrowing the gap to an Oracle baseline that uses ground-truth rewards.

Critical strengths
  1. The motivation is solid, majority voting rewards are unreliable because consensus does not equal correctness, with false positives dominating reward errors.
  2. The method is described in a well-structured and easy-to-follow manner, with clear figures that make the framework intuitive to grasp.
  3. The paper presents extensive experiments across diverse models and tasks, with consistent improvements and thoughtful ablation studies that strengthen the overall claims.
Critical weaknesses

1.The observation that majority voting is unreliable in TTRL has been widely discussed in recent literature, and TRUER appears more as an engineering patch than a fundamentally new paradigm. 2.Although the method avoids target-domain labels, it still relies on a labeled source dataset, which may become a distribution-shift bottleneck if the source and target domains differ significantly. 3.Since the rectifier is trained using GRPO on pseudo-reward signals derived from majority voting, it risks learning to reinforce systematic errors rather than truly distinguishing correctness. 4.While the reported overhead is modest, training an additional rectifier with its own sampling, scoring, and GRPO updates could scale poorly with problem difficulty and sample size in practice. 5.The rectifier's generated rationales are self-produced text that may not faithfully reflect its actual decision process, offering readability rather than genuine explainability. 6.Since stronger models already produce more reliable majority votes, the marginal benefit of the rectifier may shrink significantly as base model capability increases. 7.Key design choices such as the acceptance/rejection threshold, rank-2 checking strategy, and category weights lack systematic sensitivity analysis, raising concerns about reproducibility and tuning effort. 8.The paper primarily compares against naive TTRL and an Oracle baseline, without adequately benchmarking against other recent TTRL improvements such as confidence-weighted rewards, negative penalties, or self-consistency verification. 9.Experiments are largely confined to mathematical reasoning benchmarks, leaving the method's applicability to code generation, open-ended QA, and multi-turn tool use largely unverified. 10.The paper does not examine whether the rectifier could be misled by adversarial inputs, reward hijacking, or prompt injection, leaving potential failure modes unaddressed.

Questions for authors

1.Given that the unreliability of majority voting is well-documented, how does TRUER fundamentally differ from existing confidence-weighted or consistency-based reward correction methods beyond the specific implementation of the rectifier? 2.To what extent does the rectifier's performance degrade when there is a significant distribution shift between the labeled source dataset and the unlabeled target domain? 3.Since the rectifier is trained via GRPO on pseudo-rewards, what theoretical or empirical evidence guarantees it learns true correctness rather than simply memorizing and reinforcing systematic consensus biases? 4.How does the abstention mechanism—assigning zero rewards when both top candidates are rejected—impact training stability and convergence speed on difficult problem sets? 5.Why does the evaluation lack comparisons with other recent TTRL improvements? 6.Have you validated TRUER on non-mathematical tasks such as code generation or open-ended reasoning, where "correctness" is less binary and harder to verify via majority voting? 7.Considering the rectifier requires its own sampling and GRPO updates, how does the total computational overhead scale when applied to larger models or more complex reasoning chains? 8.Have you analyzed whether the rectifier is robust against adversarial inputs or reward hijacking, or if it merely shifts the vulnerability from the policy model to the rectifier itself?

Reviewer_GC6COpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new method, TRUER, to address the limitations of majority voting in test-time reinforcement learning. By training an Error-Aware Rectifier to verify the pseudo-labels used in TTRL, TRUER aims to provide higher-quality reward signals for GRPO. Across different experimental settings, TRUER achieves better performance than TTRL.

Critical strengths
  1. The proposed method is effective in selecting higher-quality pseudo-labels for TTRL to improve reward quality.
  2. The authors provide valuable insights into the issue of false positives, where incorrect responses receive positive rewards in TTRL.
Critical weaknesses
  1. In all experimental settings, the verifier appears to have stronger reasoning capabilities than the policy model, as suggested by the base column in Table 2. This raises a concern that the improvements may primarily come from transferring the verifier's stronger capabilities to the policy through evaluation. Would the proposed method still be effective if the verifier were weaker than the policy?
  2. The verifier requires GRPO training, while the policy is also trained using GRPO. As mentioned in Weakness 1, the verifier appears to be stronger than the policy in the experiments. Under this setting, how would alternative methods such as OPD perform?
  3. The paper does not report the training cost of the verifier, making it difficult to assess the overall computational cost of the proposed method.
Questions for authors

Would the proposed method still be effective when the policy model is stronger than the verifier? For example, how would TRUER perform with Qwen3-4B as the policy model and Qwen3-1.7B as the verifier?

Reviewer_15v8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TRUER to address false-positive rewards in majority-vote-based test-time reinforcement learning, where incorrect responses receive positive rewards. TRUER trains a reward rectifier using limited labeled source data, with an emphasis on incorrect majorities and correct minorities.

Critical strengths

1 Experiments show improvements in reward accuracy and final policy Pass@1 over naive TTRL across the evaluated configurations. TRUER uses a small labeled source set to train a reward rectifier, which then supports target adaptation without access to target ground-truth answers.

2 By comparing reward assignments on the same responses, the paper shows that TRUER reduces false-positive rewards while preserving most correct positive rewards.

Critical weaknesses

1 The benefit on difficult questions remains unclear. Abstention avoids reinforcing incorrect answers. However, it also removes the group’s direct outcome-based learning signal.

2 The added benefit of rectifier training needs stronger evidence. Table 14 shows that there is only a little increase in final policy Pass@1 using trained rectifier compared with using the original pretrained rectifier. This suggests that much of the gain in this setting may come from introducing a rectifier itself, regardless of whether it is trained or not.

Questions for authors

1 Table 3 shows that the main results in Table 2 use seed 42 for some configurations and seed 43 for others. Why did you choose different seed?

2 How does TRUER handle responses that contain irrelevant or invalid reasoning but share the selected final answer? Since rewards are assigned by answer matching after checking a representative response, can such responses also receive positive rewards?

3 Could this study still work for other RL loss in GRPO family? Such as GSPO,DAPO, DrGRPO...

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.