- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RLCER, a framework that trains a single model to both generate solutions as a reasoner and develop evaluation criteria as a rubricator. To connect these criteria to solution quality, the authors use a separate frozen verifier to evaluate rubric satisfaction and retain rubrics whose satisfaction correlates positively with answer correctness across sampled solutions. The selected rubrics then guide training: the reasoner receives both outcome and rubric rewards, while the rubricator is rewarded for producing valid criteria. Through this interaction, the framework aims to provide adaptive, interpretable reasoning supervision that evolves with the policy. Experiments from matched starting checkpoints show modest improvements over outcome-only RLVR on mathematical and knowledge-reasoning benchmarks. Additional ablations examine the effects of frozen rubrics, externally generated rubrics, and the rubric-selection threshold.
Critical strengths
- Useful problem setting. The paper explores how evaluation criteria can evolve alongside a reasoning model, addressing a useful question about adapting supervision during training. The experimental results largely support the proposed method.
- The paper is relatively well written and easy to follow
Critical weaknesses
- Inconsistent filtering of penalty rubrics. The method allows negative-point criteria for flaws, with verifier output True when a flaw is present, but retains every rubric only when corr(vₖ, z) > α. A useful penalty should correlate negatively with correctness, so this rule rejects useful penalties and can retain penalties for behavior associated with correct answers. I think a sign-aware rule such as corr(sₖvₖ, z) > α would resolve the mismatch. The authors are also encouraged to clarify the implemented rule and handling of constant labels or empty valid sets.
- Unclear source of supervision gains. The method selects rubrics using correctness correlations across only eight solutions, then rewards those same solutions. As a result, the gains may reflect incidental batch correlations rather than meaningful reasoning criteria, and the random-reward control does not isolate these effects. Moreover, the “outcome-free” experiment still uses correctness labels for rubric selection, so it remains indirectly outcome-supervised. In this case, I think selecting rubrics on one batch and evaluating them on another would help clarify the source of the gains.
- Missing comparison at equal training cost. The paper reports average gains across seven benchmarks of approximately 0.86 points for 4B and 1.33 for 8B, with regressions on some tasks. However, achieving these gains requires additional rubric generation, verification across solutions, and optimization of the rubricator role. Consequently, training both methods for 1,500 steps does not ensure equal computational cost, which makes it difficult to determine whether the gains justify the additional resources. The authors are encouraged to quantify this overhead and compare RLCER with RLVR at the same total training budget to support the claimed benefit and the “free-lunch” characterization.
- Missing independent training runs. The appendix repeats evaluation sampling rather than independent RL training. AIME2024 and AIME2025 evaluation standard deviations are 4.5 and 3.6 points, while the corresponding 8B gains are 2.71 and 0.83 points. Given AIME is a 15-question exam, the aggregate significance statement does not clearly isolate the gains. Normally I’m not inclined to ask for more seeds but in this case I encourage the authors to provide more independent runs to justify the incremental gains.
Questions for authors
- Does the implementation use sign-aware validity filtering for negative-point rubrics? How are constant correctness labels and empty valid-rubric sets handled?
- Do rubric gains persist when criteria are selected on one rollout batch and evaluated on an independent batch, compared with randomized satisfaction indicators passed through the same filter?
- Can the authors clarify the teacher, 40,000-example cold-start data, verifier-training set, filtering logic, and inference-time rubric procedures?
Official Rebuttal
Thank you for the careful review. Revisions in the updated PDF are marked in blue.
W1 & Q1: Penalty rubrics and edge cases
Our implementation already uses the sign-aware rule you suggest: for penalty rubrics (v_k = 1 means the flaw is present), the sign of the correlation is flipped before checking corr > α, i.e., corr(s_k·v_k, z) > α. All reported results use this rule; the manuscript omitted it. We revised the definition of v_k in Sec. 3.3 (Eq. 7) accordingly, while Eq. 6 still aggregates the raw verifier outputs with signed scores. Questions with constant correctness labels (all 8 rollouts correct or incorrect) or with no valid rubric are discarded during training; this rule is now stated in Sec. 3.3.
W2 & Q2: Source of the gains
W3: Equal training cost
We report the computation comparison as follows: Normalizing a full 1,500-step RLVR run to 1.00, RLCER costs 1.65 (+0.60 for rubric generation, verification and rubricator updates; +0.05 for verifier distillation), so the RLVR budget equals ≈900 RLCER steps. Fig. 1(a) shows that RLCER at ≈900 steps already exceeds the best RLVR performance within 1,500 steps (dashed line). The breakdown is added to Appendix C (Table 7) and the Limitations. We also replaced the "free-lunch" wording: the paper now states that the gains need no human annotation or extra inference cost, not that training is free.
W4: Independent runs and AIME size
AIME24 and AIME25 each have 30 questions (AIME I + II), and each question is sampled 16 times, i.e., 480 responses per benchmark (now stated in Sec. 4.1 and Appendix B). A full run takes >100 GPUs for over a week, so we could afford one extra run, which we used to retrain RLCER at 4B with an independent seed (seed 123; Appendix C, Table 8). SuperGPQA is omitted because its 100-question subsets were randomly sampled and cannot be re-aligned.
The new run again beats RLVR on all four benchmarks. We also clarified that the earlier seed analysis re-evaluates one checkpoint and thus reflects evaluation variance only.
Q3: Implementation details (now in Appendix B)