acceptodds
Under review as a conference paper at ICLR 2027

From Coverage to Reliable Success: Resampled-Evidence Policy Optimization for Test-Time Reinforcement Learning

Abstract

We study how label-free reinforcement learning can improve sampled accuracy () while retaining broad solution coverage () and repeated correctness. The first two metrics are our primary outcomes; , the probability that four independent responses to the same question are all correct, provides a supporting reliability assessment. A central challenge is that finite verification evidence can support different responses unequally, even after a supervision target has been selected. We propose Resampled-Evidence Policy Optimization (REPO), which retains coverage-oriented training while adapting each response's contribution to its evidence support. The student and teacher maintain separate parameter states; the teacher supplies judgments and is updated only by an exponential moving average (EMA) of the student parameters. By resampling cached verification judgments and reconstructing supervision on fixed student responses, REPO attenuates less reproducible decisions without additional verification decoding. For each response, agreement and conflict between replayed and original advantage signs determine a bounded update weight. This weight preserves the original direction while reducing the influence of decisions sensitive to the sampled evidence. Across five external benchmarks, REPO improves macro-averaged over Co-rewarding by 2.52 and 1.60 percentage points at 4B and 8B, with gains of 2.12 and 2.18 points. Macro-averaged remains close to Base (+0.18/+1.06 points), despite larger gaps to Co-rewarding (+11.50/+12.00 points). These results indicate aggregate coverage retention, with losses on some individual benchmarks.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Test-Time Training

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_LynKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes REPO, a label-free reinforcement learning framework combining coverage-oriented policy optimization with response-specific evidence weighting. It resamples cached EMA-teacher judgments, reconstructs targets and advantages, and attenuates updates whose directions are less reproducible. Experiments on mathematical and scientific benchmarks evaluate sampled accuracy, solution coverage, and repeated correctness, with component ablations examining the allocation of update weights.

Critical strengths
  1. The paper jointly examines accuracy, solution coverage, and repeated correctness, making a relevant tradeoff in label-free RL explicit.
  2. Cached-evidence replay provides a practical way to assess update stability without additional verification decoding.
  3. The matched ablations support the benefit of response-level weighting over unweighted and question-level controls for repeated success.
Critical weaknesses
  1. The baseline coverage results need reconciliation with the stated evaluation protocol. The TTRL entries in Table 1 exactly match those in TTRL-CoCoV [1], which estimates pass@16 through resampling with replacement, whereas Appendix E.2 specifies an unbiased subset estimator. In particular, the 4B AIME24 entry of 10.8/14.9 is incompatible with the stated 30-question, 32-response subset protocol. The provenance and calculation of these results require clarification.

  2. A closely related coverage-oriented baseline is missing. TTRL-CoCoV [1] combines confidence-conditioned verification with pass@k training and evaluates the same Qwen3-4B/8B backbones on overlapping benchmarks. Although the paper cites it as a design precedent, it is not compared experimentally, leaving REPO’s advantage over this closely related approach unestablished.

  3. The ablations do not isolate the value of replay-based sign support. Tables 3 and 5 compare REPO with unweighted and question-level weighting, but not with a simpler response-level weight derived from cached teacher verdict fractions. Such a control would clarify whether replaying advantage signs adds value beyond ordinary confidence-based weighting.

  4. The computational cost of the main comparisons is unclear. Table 4 specifies 64 student responses and up to 40 teacher judgments per low-consensus question, but corresponding baseline budgets and comparative training costs are not reported. The absence of additional decoding during replay establishes its incremental cost advantage, but does not establish the efficiency of the complete training framework.

Questions for authors

See the Critical Weaknesses above.

Reviewer_JkR2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies label-free test-time RL and argues that a shaky pseudo-label, applied uniformly to every response, can raise pass@1 while shrinking pass@16 and making answers less consistently correct (pass4). REPO keeps an EMA teacher for judgments, then resamples those cached verdicts to shrink (never flip) unstable response-level updates; on Qwen3-4B/8B versus Co-rewarding they report +2.52/+1.60 pass@1 and +2.12/+2.18 pass4, with macro pass@16 near the unadapted base.

Critical strengths

Besides pass@1 and pass@16, they also report pass^4, so the evaluation covers how reliable the model is after Test-time reinforcement learning.

Critical weaknesses

The manuscript would benefit from substantial revisions to improve clarity. The exposition often comes across as defensive, the proposed method lacks a straightforward, self-contained description, and the presentation of results is difficult to follow.

Questions for authors
  1. Please add the exact setting behind Figure 1(b). At step 0, every method except yours has pass@16 well below the Base line. If no parameters have been updated yet, those runs are still the base model, so pass@16 should match. Is the plot offset?
  2. The method section is hard to follow, and Figure 3 does not make the pipeline easier. Please describe REPO in simpler terms, and add a short comparison with CoVerRL. As I read it, this looks like a change on top of CoVerRL rather than a separate setup, and I cannot tell what you kept and what you replaced.
  3. Appendix C.2 sets max generation length to 2048, while the baselines you compare against typically allow at least 3072. Please report truncation rates under your setting for both training and evaluation. If those rates are high, please rerun with a longer budget and report the new numbers. I need this to judge whether the comparison is fair.
Reviewer_sLoWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes REPO, a reinforcement learning method for improving the reliability of label-free test-time adaptation in LLM reasoning. The key idea is to resample cached teacher judgments and evaluate the consistency of the resulting advantage directions. The method then adjusts the contribution of each response to the policy update based on this consistency. Experiments on several mathematical reasoning and knowledge benchmarks demonstrate improvements in pass@1 and repeated correctness while largely preserving pass@16.

Critical strengths
  1. The paper addresses a meaningful problem in label-free reinforcement learning, particularly the trade-off between improving prediction accuracy and maintaining solution coverage.

  2. The proposed method is relatively simple and intuitive. Using the consistency of advantage directions to adjust policy updates is an interesting idea.

  3. The experimental results are encouraging, with improvements across multiple benchmarks and two model sizes. The ablation studies also provide evidence supporting the proposed design.

Critical weaknesses
  1. The proposed method assumes that consistent advantage directions under resampled teacher judgments indicate more reliable supervision. However, consistency does not necessarily imply correctness, particularly when the teacher has systematic biases. The authors should further clarify this assumption.

  2. The core method essentially estimates the stability of advantage signs through resampling and uses this information to reweight policy updates. Although effective, the technical novelty appears relatively modest.

  3. It would be useful to compare the proposed sign-consistency weighting against simpler alternatives, such as weighting based on reward variance, teacher confidence, or advantage magnitude. Such comparisons would help establish whether the proposed replay procedure is necessary.

Questions for authors
  1. What causes the teacher to produce different judgments for the same candidate answer, and to what extent does this variation reflect meaningful uncertainty rather than stochastic decoding?

  2. Why does the method preserve the original advantage direction even when the resampled evidence consistently disagrees with it? Would stronger attenuation or discarding such updates improve performance?

Reviewer_FRq7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates pseudo-label uncertainty in label-free test-time reinforcement learning and proposes Resampled-Evidence Policy Optimization (REPO). The method resamples cached teacher verification judgments to estimate the stability of each response’s update direction and accordingly adjusts response-level policy-update weights. The authors conduct experiments on multiple reasoning benchmarks and two model scales, showing that REPO improves sampled accuracy and repeated correctness while largely retaining solution coverage. However, the core motivation and the practical consequences of low-consensus questions are not clearly established, and the distinction between REPO and existing bootstrap-based uncertainty estimation or sample-level reweighting methods is insufficiently discussed. Moreover, the current ablations do not fully explain the sources of the performance gains, the construction of pseudo-labels and reward signals requires further clarification, and key analyses of pseudo-label selection quality and computational efficiency are missing. Therefore, the paper still requires substantial improvement in its motivation, methodological novelty, and experimental validation.

Critical strengths
  1. The authors propose a new TTRL method.
  2. The authors validate the effectiveness of the proposed method on multiple datasets.
Critical weaknesses
  1. The motivation of the paper is relatively weak and is not clearly presented. In Section 2, the authors discuss how the proportion of “low-consensus questions” changes during training, but they do not clearly explain the eventual impact of this phenomenon on the model. For example, does it reduce the model’s reasoning accuracy, disrupt the TTRL training process, or cause the model to converge prematurely toward low-consensus questions? Moreover, the paper does not provide a detailed explanation of what constitutes a low-consensus or high-consensus question, nor does it clearly specify the criterion used to distinguish between them. Finally, an empirical study would generally examine existing methods to determine whether they fail to alleviate the identified problem. However, the authors conduct this analysis only on their proposed method, which is rather confusing.
  2. The methodological novelty is relatively limited, and the distinction from bootstrap-based uncertainty estimation and sample-level reweighting is insufficiently discussed. The core operation of REPO is to resample cached teacher judgments with replacement, estimate update stability according to the consistency of advantage signs, and then rescale the original advantages. In essence, the method can be viewed as a combination of “bootstrap resampling + sign-consistency estimation + sample-level advantage weighting.” The paper does not adequately explain the substantive algorithmic novelty of REPO compared with existing uncertainty-weighting, confidence-weighting, or robust learning methods for noisy labels.
  3. The paper lacks comprehensive component ablations, making it difficult to accurately identify the sources of the performance improvements. REPO consists of several designs, including the EMA teacher, low-consensus question filtering, cached verification-evidence resampling, response-level sign support, and bounded advantage weighting. However, the paper only ablates evidence weighting (w/o EW) and response-level allocation (w/o RA), without separately evaluating the contributions of the EMA teacher, the target-selection mechanism, or the pass@K advantage function. The current ablation results mainly show that evidence weighting and response-level allocation substantially improve pass4pass^4, while having only minor effects on pass@1 and pass@16. Therefore, these results cannot fully explain REPO’s overall improvements in accuracy and coverage over TTRL, Co-rewarding, and CoVerRL.
  4. The construction of the reward signal is not clearly described. In general, TTRL methods select more plausible answers as pseudo-labels by designing various rules or introducing external models, which is usually the primary source of their performance improvements. However, the description in Section 3.4 on constructing the training signals is rather vague. The authors do not clearly explain how the pseudo-labels are constructed, why there are teacher-derived student rewards, or how these rewards are related to the student model’s self-consistency voting.
  5. The paper also lacks several key analyses. For example, after applying the proposed method, can REPO select pseudo-labels more accurately than TTRL and reduce the proportion of incorrectly selected labels? This could be evaluated by plotting label accuracy against training steps. In addition, compared with TTRL, the authors introduce a more complex filtering procedure, but the paper does not analyze how this additional complexity affects computational and time efficiency.
Questions for authors

Please see above weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.