acceptodds
Under review as a conference paper at ICLR 2027

Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubrics

Abstract

Despite chain-of-thought (CoT) playing crucial roles in LLM reasoning, directly rewarding it is difficult: training a reward model demands heavy human labeling efforts, and static RMs struggle with evolving CoT distributions and reward hacking. These challenges motivate us to seek an autonomous CoT rewarding approach that requires no human annotation efforts and can evolve gradually. Inspired by recent self-evolving training methods, we propose RLCER (Reinforcement Learning with CoT Supervision via Self-Evolving Rubrics), which enhances the outcome-centric RLVR by rewarding CoTs with self-proposed and self-evolving rubrics. We show that self-proposed and self-evolving rubrics provide reliable CoT supervision signals even without outcome rewards, enabling RLCER to outperform outcome-centric RLVR. Moreover, when used as in-prompt hints, these self-proposed rubrics further improve inference-time performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, reinforcement learning, rubrics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MNSyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RLCER, a framework that trains a single model to both generate solutions as a reasoner and develop evaluation criteria as a rubricator. To connect these criteria to solution quality, the authors use a separate frozen verifier to evaluate rubric satisfaction and retain rubrics whose satisfaction correlates positively with answer correctness across sampled solutions. The selected rubrics then guide training: the reasoner receives both outcome and rubric rewards, while the rubricator is rewarded for producing valid criteria. Through this interaction, the framework aims to provide adaptive, interpretable reasoning supervision that evolves with the policy. Experiments from matched starting checkpoints show modest improvements over outcome-only RLVR on mathematical and knowledge-reasoning benchmarks. Additional ablations examine the effects of frozen rubrics, externally generated rubrics, and the rubric-selection threshold.

Critical strengths
  1. Useful problem setting. The paper explores how evaluation criteria can evolve alongside a reasoning model, addressing a useful question about adapting supervision during training. The experimental results largely support the proposed method.
  2. The paper is relatively well written and easy to follow
Critical weaknesses
  1. Inconsistent filtering of penalty rubrics. The method allows negative-point criteria for flaws, with verifier output True when a flaw is present, but retains every rubric only when corr(vₖ, z) > α. A useful penalty should correlate negatively with correctness, so this rule rejects useful penalties and can retain penalties for behavior associated with correct answers. I think a sign-aware rule such as corr(sₖvₖ, z) > α would resolve the mismatch. The authors are also encouraged to clarify the implemented rule and handling of constant labels or empty valid sets.
  2. Unclear source of supervision gains. The method selects rubrics using correctness correlations across only eight solutions, then rewards those same solutions. As a result, the gains may reflect incidental batch correlations rather than meaningful reasoning criteria, and the random-reward control does not isolate these effects. Moreover, the “outcome-free” experiment still uses correctness labels for rubric selection, so it remains indirectly outcome-supervised. In this case, I think selecting rubrics on one batch and evaluating them on another would help clarify the source of the gains.
  3. Missing comparison at equal training cost. The paper reports average gains across seven benchmarks of approximately 0.86 points for 4B and 1.33 for 8B, with regressions on some tasks. However, achieving these gains requires additional rubric generation, verification across solutions, and optimization of the rubricator role. Consequently, training both methods for 1,500 steps does not ensure equal computational cost, which makes it difficult to determine whether the gains justify the additional resources. The authors are encouraged to quantify this overhead and compare RLCER with RLVR at the same total training budget to support the claimed benefit and the “free-lunch” characterization.
  4. Missing independent training runs. The appendix repeats evaluation sampling rather than independent RL training. AIME2024 and AIME2025 evaluation standard deviations are 4.5 and 3.6 points, while the corresponding 8B gains are 2.71 and 0.83 points. Given AIME is a 15-question exam, the aggregate significance statement does not clearly isolate the gains. Normally I’m not inclined to ask for more seeds but in this case I encourage the authors to provide more independent runs to justify the incremental gains.
Questions for authors
  1. Does the implementation use sign-aware validity filtering for negative-point rubrics? How are constant correctness labels and empty valid-rubric sets handled?
  2. Do rubric gains persist when criteria are selected on one rollout batch and evaluated on an independent batch, compared with randomized satisfaction indicators passed through the same filter?
  3. Can the authors clarify the teacher, 40,000-example cold-start data, verifier-training set, filtering logic, and inference-time rubric procedures?
1 comment
AuthorsOpenReview ↗

Official Rebuttal

Thank you for the careful review. Revisions in the updated PDF are marked in blue.

W1 & Q1: Penalty rubrics and edge cases

Our implementation already uses the sign-aware rule you suggest: for penalty rubrics (v_k = 1 means the flaw is present), the sign of the correlation is flipped before checking corr > α, i.e., corr(s_k·v_k, z) > α. All reported results use this rule; the manuscript omitted it. We revised the definition of v_k in Sec. 3.3 (Eq. 7) accordingly, while Eq. 6 still aggregates the raw verifier outputs with signed scores. Questions with constant correctness labels (all 8 rollouts correct or incorrect) or with no valid rubric are discarded during training; this rule is now stated in Sec. 3.3.

W2 & Q2: Source of the gains

  • Randomized control through the same filter (Appendix C, Fig. 8). We ran Rubric Only with rubrics shuffled across questions as suggested by the reviewer 31UN, keeping the same frozen verifier and the identical correlation filter. Under this setting, the accuracy then only fluctuates around its initial level, without the steady gain of the real rubrics. This indicates that that the gains come from rubric content rather than incidental correlations induced by filtering.
  • Independent samples. Rubrics are instance-specific, so rubrics selected on one batch of questions cannot be applied to another batch during training. Instead, Sec. 4.5 uses rubrics generated beforehand as in-prompt hints on freshly sampled solutions, without any in-batch selection: the average accuracy on AIME24/AIME25/AMC23 rises from 71.51 to 73.16 (avg@16) and from 85.00 to 88.21 (BoN). We are happy to discuss further if we have misread your suggestion.

W3: Equal training cost

We report the computation comparison as follows: Normalizing a full 1,500-step RLVR run to 1.00, RLCER costs 1.65 (+0.60 for rubric generation, verification and rubricator updates; +0.05 for verifier distillation), so the RLVR budget equals ≈900 RLCER steps. Fig. 1(a) shows that RLCER at ≈900 steps already exceeds the best RLVR performance within 1,500 steps (dashed line). The breakdown is added to Appendix C (Table 7) and the Limitations. We also replaced the "free-lunch" wording: the paper now states that the gains need no human annotation or extra inference cost, not that training is free.

W4: Independent runs and AIME size

AIME24 and AIME25 each have 30 questions (AIME I + II), and each question is sampled 16 times, i.e., 480 responses per benchmark (now stated in Sec. 4.1 and Appendix B). A full run takes >100 GPUs for over a week, so we could afford one extra run, which we used to retrain RLCER at 4B with an independent seed (seed 123; Appendix C, Table 8). SuperGPQA is omitted because its 100-question subsets were randomly sampled and cannot be re-aligned.

4BAIME24AIME25AMC23GPQA-D
RLVR (Table 1)29.3830.2179.5344.16
RLCER (Table 1)29.3830.6381.8844.91
RLCER (seed 123)30.0030.4282.8144.70

The new run again beats RLVR on all four benchmarks. We also clarified that the earlier seed analysis re-evaluates one checkpoint and thus reflects evaluation variance only.

Q3: Implementation details (now in Appendix B)

  • Teacher and cold-start data: a proprietary model comparable to Seed 1.6 Pro generates trajectories on the RL training questions; after rejection sampling, we keep 20k reasoning and 20k rubricator trajectories for SFT. RLVR and RLCER start from the same SFT checkpoint.
  • Verifier data: the teacher's verification judgments on the rejection-sampled reasoning trajectories paired with their rubrics.
  • Filtering logic: Sec. 3.3 (the sign-aware validity filter and the discarding rule above).
  • Inference: Table 1 uses the policy only as the reasoner, with no rubric generation or verifier calls; rubric hints are used only in the optional experiment of Sec. 4.5.
Reviewer_31UNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RLCER, a way to reward the chain of thought (CoT) in reinforcement learning with verifiable rewards (RLVR) using rubrics written by the model itself, without human annotation. The motivation is that outcome-only RLVR gives every trajectory with the same final answer the same reward. Process reward models, by contrast, need costly labels and can fall out of step with the policy as training proceeds.

A single policy plays two roles, set by the prompt. As a reasoner, it solves each problem (8 rollouts per question). As a rubricator, it reads one of its own CoTs and writes 5–15 weighted natural-language rubrics. A separate, frozen Qwen3-4B verifier, distilled from a proprietary teacher, judges whether each CoT satisfies each rubric. A rubric is treated as valid if, across the question's rollouts, its satisfaction correlates with answer correctness (corr > 0.2) and is not constant. Both roles are trained jointly with PPO, using role-specific advantages: a) the reasoner is rewarded with ±1 for correctness plus a normalized score over the valid rubrics it satisfies; b) the rubricator is rewarded with the fraction of its rubrics that are valid ("self-evolving"), plus a format reward.

Experiments use Qwen3-4B/8B-Base, cold-started with teacher-generated SFT data and trained on DAPO-Math-17k. Evaluation covers AIME24/25, AMC23, GPQA-Diamond and three SuperGPQA subsets, reported as avg@16.

Critical strengths
  1. A novel and practical idea for CoT supervision without step labels. The paper targets a real gap. Outcome-only RLVR does not look at how the model reasons, and process reward models need annotation and can drift out of step with the policy. RLCER's answer is simple. The policy writes its own rubrics. A rubric is kept only if its satisfaction across a question's rollouts correlates with answer correctness. The rubric writer is then trained by rewarding the fraction of valid rubrics among its proposals.

To my knowledge, this specific combination is new for natural-language rubrics: scoring each rubric by how it lines up with correctness across the group, and RL-training the policy's own rubric writer on that signal. It is also easy to adopt, needing only one extra prompt role and a small verifier on top of a standard RLVR setup. And the rewards are readable criteria rather than opaque scores.

  1. A sensible reward design and a controlled starting point for evaluating it.

The reward keeps the outcome primary. With r_outcome = ±1 and r_cot normalized to [0, 1], every correct rollout scores at least 1 and every incorrect one at most 0. A well-written wrong answer can never outscore a correct one, which is a common failure mode of additive process rewards. The std(vₖ) > 0 condition also discards rubrics that every rollout satisfies, which offers partial protection against trivially gameable criteria.

On the experimental side, we see consistent improvement in performance: RLCER is ahead in 11 of 13 non-tied cells, including on out-of-domain GPQA/SuperGPQA. The gains are small, and I discuss in Critical Weakness 2 why they are not yet conclusive, but the direction is encouraging.

Critical weaknesses
  1. The "outcome-free" result is not actually outcome-free, so it is unclear whether the rubrics themselves provide any training signal. The paper's central claim is that self-proposed rubrics give useful CoT supervision even without outcome rewards. However, the way rubrics are filtered makes this hard to test. A rubric only counts as valid if its satisfaction across the 8 rollouts of a question correlates with whether those rollouts got the right answer. That correctness vector is computed against the ground-truth answer, and the surviving rubrics are then used to reward those same 8 rollouts. So in the rubric only run of RQ1, the labels still enter the reward through the selection step. This matters because selection alone is enough to produce a reward that tracks correctness. Further, the random-reward control idoes not address this, because it replaces the reward entirely and skips the filter. Two other results point the same way such as a fixed external rubricator does as well as RLCER in Table 6.

  2. The improvement over RLVR is small, comes likely from single runs, and is measured against a baseline that may not be well tuned. The gains in Table 1 average about 0.9 points at 4B and 1.3 points at 8B, with one tie and two cells going the other way. On AIME24 at 8B, the largest math gain (+2.71) amounts to ~13 more correct samples out of 480. As far as I can tell, each arm was trained once. The only variance analysis re-evaluates a single RLCER checkpoint with different sampling seeds. Even those reported spreads such as AIME24 and SuperGPQA-Eng are larger than most of the gaps, and training-seed variance would add to them.

Questions for authors
  1. A leakage-controlled null for RQ1. Could you run rubric Only with the same validity filter, but with verifier judgments shuffled across rollouts (or rubric texts shuffled across questions)? Please plot it next to the real rubric only curve and an RLVR curve. If the shuffled version doesn't learn and the real rubrics do, I would raise my score substantially. A cross-fit RLCER run, where rubrics are selected on half the rollouts and the other half is rewarded, would also address this. Both are cheap compared with the main runs.

  2. Training-seed variance. Could you report at least three training seeds per arm, at 8B or at 4B if that's more feasible, with mean ± SD and a paired, question-level comparison? If RLCER's advantage over RLVR holds across seeds, I would also consider raising my score.

  3. A stronger and compute-matched baseline. How does RLCER compare with RLVR using the DAPO recipe (clip-higher, dynamic sampling, token-level loss) or an entropy-controlled variant, with entropy curves for both arms? What about RLVR given RLCER's extra compute, e.g. more rollouts per question? I know this is a bigger ask for the discussion period, but even one 8B run would be informative.

  4. Rubric content and verifier robustness. What fraction of valid rubrics at steps 0, 500 and 1500 refer to the final answer or its form, such as a specific value or "the answer is an integer"? How accurate is the verifier against the teacher on CoTs from late training? A clean result here would address my concern that the rubricator and the frozen verifier are being gamed.

1 comment
AuthorsOpenReview ↗

Official Rebuttal

Thank you for the detailed and constructive review. Revisions in the updated PDF are marked in blue.

W1 & Q1: Leakage-controlled null, cross-fit, and Table 6

  • Shuffled-rubric control (Appendix C, Fig. 8). As suggested, we ran Rubric Only with rubric texts shuffled across questions, keeping the same frozen verifier and the identical validity filter (corr > 0.2, non-constant). Since we have left the institution where this work was done and no longer have the raw logs of the original curves, we re-ran this control with our saved training code and plot it next to the original Rubric Only curve on AIME25. With shuffled rubrics, accuracy only fluctuates around its initial level (≈23–25%) over 600 steps, without the steady gain of the real rubrics. This indicates that the gain comes from the rubric content rather than from correctness-based selection alone. A pointer is added in Sec. 4.2.
  • Cross-fit. Splitting the 8 rollouts into 4+4 leaves only 4 rollouts to estimate each rubric's correlation, which is too unstable (only a few discrete values, and often undefined when all 4 are correct or incorrect). Given the limited compute, we prioritized the shuffled control.
  • Transfer to fresh samples (Sec. 4.5). Rubrics generated beforehand and used as in-prompt hints, without any in-batch selection, raise the average accuracy on AIME24/AIME25/AMC23 from 71.51 to 73.16 (avg@16) and from 85.00 to 88.21 (BoN).
  • Results of Table 6. The external rubricator is Seed 1.6 Pro, far larger than our 8B policy. Table 6 tests whether a self-evolving 8B rubricator can match it (it does), not whether rubric content matters; the latter is tested by the shuffled control. We clarified this in Appendix C.

W2 & Q2: Training-seed variance

All results are avg@16 on the full 30-question AIME sets (now stated in Sec. 4.1). A full 8B run takes >100 GPUs for over a week, so three seeds per arm is infeasible within the discussion period, especially after the shuffled control. We used our remaining budget to retrain RLCER at 4B with an independent seed (seed 123), from the same SFT checkpoint and configuration (Appendix C, Table 8). SuperGPQA is omitted because its 100-question subsets were randomly sampled and cannot be re-aligned.

4BAIME24AIME25AMC23GPQA-D
RLVR (Table 1)29.3830.2179.5344.16
RLCER (Table 1)29.3830.6381.8844.91
RLCER (seed 123)30.0030.4282.8144.70

The new run again beats RLVR on all four benchmarks. We also clarified that the earlier seed analysis re-evaluates one checkpoint and thus reflects evaluation variance only.

Q3: Baseline strength and equal compute

  • Our RLVR baseline already follows the DAPO recipe via VAPO (its PPO-based variant with clip-higher and dynamic sampling), was carefully tuned, and shares the same RL configuration as RLCER (Appendix B). We no longer have the training logs needed to plot entropy curves.
  • Normalizing a full RLVR run to 1.00, RLCER costs 1.65 (+0.60 for rubric generation, verification and rubricator updates; +0.05 for verifier distillation). The budget of 1,500 RLVR steps thus equals ≈900 RLCER steps, and Fig. 1(a) shows that RLCER at ≈900 steps already exceeds the best RLVR performance within 1,500 steps (dashed line). The breakdown is added to Appendix C (Table 7) and the Limitations.

Q4: Rubric content and verifier robustness

Thanks for the suggestion. However, we no longer have access to the intermediate checkpoints (steps 0/500/1500) as I have left the place where I was finishing this work, so we cannot recompute these statistics. However, the audit at the end of Appendix C checked whether rubrics leak final-answer values or encourage shortcuts: 200 sampled rubrics score 8.65/10 for reflecting genuine reasoning, and the verifier agrees with the teacher on 91.6% of 1,000 CoTs. Together with the shuffled control, this suggests that the rubrics are not merely gaming the verifier.

Reviewer_yTRvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes RLCER, in which a single policy model plays two roles: a reasoner that generates solutions and a rubricator that proposes grading criteria based on the question and the generated solutions. A separate, frozen verifier checks whether each criterion is met. Criteria whose satisfaction positively correlates with answer correctness are retained to construct the CoT reward, while the rubricator is trained using the proportion of valid criteria it generates. Experiments use Qwen3-4B and Qwen3-8B on math and knowledge reasoning benchmarks. Giving equal weight to the seven evaluation columns in Table 1, the average gains over RLVR are approximately 0.86 and 1.33 percentage points, respectively. The appendix includes experiments on frozen rubrics, external rubricators, correlation thresholds, and verifier agreement. The main unresolved questions are whether the reward definition is consistent and whether the evidence sufficiently separates useful process supervision from correctness-based reward shaping and the benefits of additional computation.

Critical strengths
  1. The paper addresses a worthwhile problem.

Outcome-only RL struggles to distinguish between different reasoning processes. Exploring natural-language grading criteria that evolve during training is therefore well motivated.

  1. The paper includes relevant control experiments.

The authors test removing the evolving reward, freezing the rubrics, using an external large model to generate rubrics, and varying the correlation threshold. It would therefore be inaccurate to say that the paper has no mechanism ablations.

  1. The method makes the feedback easier to inspect.

Rubrics explicitly describe the behaviors being rewarded or penalized, making it easier to understand the supervision than with a single scalar score. The design also remains relatively simple by sharing one policy across roles while using role-specific rewards.

Critical weaknesses
  1. Negative rubric scores conflict with the validity filter.

Eq. (6) uses signed rubric scores, but §3.3 applies the same requirement, corr(v, z) > α, to all criteria. The appendix explicitly states that negative-score criteria describe flaws and that v = 1 means the flaw is present. A useful flaw-detection criterion that negatively correlates with correctness would therefore be filtered out. The authors should clarify whether the implementation applies a sign correction and revise the equations accordingly.

  1. Using the same small batch to select criteria and assign rewards raises reliability concerns.

For each question, the method samples only N = 8 solutions, selects criteria based on correlations across those solutions, and then uses the selected criteria to reward the same solutions. This makes the procedure vulnerable to chance correlations and patterns specific to the current batch. The existing sensitivity analysis for α does not establish whether the selected criteria remain useful on newly sampled CoTs.

  1. The Rubric Only experiment does not isolate the value of semantic CoT supervision.

Even after removing the direct outcome reward, rubric selection still depends on ground-truth correctness. The control, however, uses random rewards without applying the same filtering procedure. The experiment therefore cannot adequately separate the semantic value of the rubrics from the effect of selecting rewards based on answer correctness. A random-reward control that preserves the filtering mechanism, or validation on independent samples, is needed.

Questions for authors

please refer to the weaknesses

1 comment
AuthorsOpenReview ↗

Official Rebuttal

Thank you for the positive and constructive review. Revisions in the updated PDF are marked in blue.

W1: Negative scores vs. the validity filter

Thank you for catching this. Our code already applies the sign correction you describe: for penalty rubrics (v_k = 1 means the flaw is present), the correlation sign is flipped, i.e., corr(s_k·v_k, z) > α, so useful flaw-detection rubrics are kept. All reported results use this rule. We revised Sec. 3.3 accordingly: Eq. 7 now uses the sign-corrected verifier output, while Eq. 6 keeps the raw output with the signed score.

W2: Selecting and rewarding on the same 8 rollouts

Rubrics are instance-specific, so they cannot be selected on one batch of questions and reused on another during training. To test whether they remain useful on newly sampled CoTs, Sec. 4.5 feeds rubrics generated beforehand as in-prompt hints, without any in-batch selection: the average accuracy on AIME24/AIME25/AMC23 rises from 71.51 to 73.16 (avg@16) and from 85.00 to 88.21 (BoN). In addition, the shuffled-rubric control below shows that rubrics passing the same filter but mismatched to the question do not help, which argues against chance correlations within the batch. We are happy to discuss further if we have missed your point.

W3: Isolating semantic supervision in Rubric Only

As suggested, we added a control that preserves the correctness-based filter: rubric texts are shuffled across questions, while the same frozen verifier and the identical filter (corr > α) are used to build the reward (Appendix C, Fig. 8). Having left the institution where this work was done, we re-ran it with our saved training code. With shuffled rubrics, accuracy only fluctuates around its initial level, without the steady gain of the real rubrics. This indicates that the gain of Rubric Only stems from the rubric content rather than from correctness-based selection alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.