- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes REPO, a label-free reinforcement learning framework combining coverage-oriented policy optimization with response-specific evidence weighting. It resamples cached EMA-teacher judgments, reconstructs targets and advantages, and attenuates updates whose directions are less reproducible. Experiments on mathematical and scientific benchmarks evaluate sampled accuracy, solution coverage, and repeated correctness, with component ablations examining the allocation of update weights.
Critical strengths
- The paper jointly examines accuracy, solution coverage, and repeated correctness, making a relevant tradeoff in label-free RL explicit.
- Cached-evidence replay provides a practical way to assess update stability without additional verification decoding.
- The matched ablations support the benefit of response-level weighting over unweighted and question-level controls for repeated success.
Critical weaknesses
-
The baseline coverage results need reconciliation with the stated evaluation protocol. The TTRL entries in Table 1 exactly match those in TTRL-CoCoV [1], which estimates pass@16 through resampling with replacement, whereas Appendix E.2 specifies an unbiased subset estimator. In particular, the 4B AIME24 entry of 10.8/14.9 is incompatible with the stated 30-question, 32-response subset protocol. The provenance and calculation of these results require clarification.
-
A closely related coverage-oriented baseline is missing. TTRL-CoCoV [1] combines confidence-conditioned verification with pass@k training and evaluates the same Qwen3-4B/8B backbones on overlapping benchmarks. Although the paper cites it as a design precedent, it is not compared experimentally, leaving REPO’s advantage over this closely related approach unestablished.
-
The ablations do not isolate the value of replay-based sign support. Tables 3 and 5 compare REPO with unweighted and question-level weighting, but not with a simpler response-level weight derived from cached teacher verdict fractions. Such a control would clarify whether replaying advantage signs adds value beyond ordinary confidence-based weighting.
-
The computational cost of the main comparisons is unclear. Table 4 specifies 64 student responses and up to 40 teacher judgments per low-consensus question, but corresponding baseline budgets and comparative training costs are not reported. The absence of additional decoding during replay establishes its incremental cost advantage, but does not establish the efficiency of the complete training framework.
Questions for authors
See the Critical Weaknesses above.