acceptodds
Under review as a conference paper at ICLR 2027

RecurSE: Recursive Self-Evaluation for LLM Rubric Judges

Abstract

LLM-as-judge is essential for evaluating open-ended text and steering post-training, yet improving the judge itself typically relies on expensive human annotations, auxiliary reward models, or distillation from stronger teachers. In this work, we consider that judging another model's response and judging one's own evaluative reasoning are essentially the same capability. We therefore improve the judge model's evaluative capability without expensive external gold supervision: the model's own evaluations generate the learning signal in a closed-loop regime of recursive self-improvement (RSI) that we term Recursive Self-Evaluation (). Specifically, employs two-pass reinforcement learning: a trainable judge model evaluates candidate responses under per-rule rubrics (Pass 1), while a synchronized copy of the current policy (the checker) audits the judge's reasoning against meta-rubrics to supply a scalar process reward (Pass 2). We then investigate two governing questions of this loop: when can self-improvement occur, and when must it stop? For the first question, we make learning viable via interface decoupling, which structurally isolates the checker's scalar score from the judge's verdict tokens and closes a degenerative token-copying shortcut that inflates self-assigned rewards. For the second, because unanchored recursive learning is inherently bounded, we introduce Pairwise Advantage Validity (PAV), a validation monitor that jointly tracks judge accuracy and checker fidelity to reliably identify a useful early-stopping window. Across Qwen3.5-9B, Gemma-4-E4B-it, and Qwen3.6-27B, achieves consistent generalization gains across held-out medical, pairwise, summarization, and professional benchmarks. Extensive ablations show that synchronized judge–checker co-evolution outperforms frozen checkers, external meta-judges, self-consistency, and scaled teacher distillation. Preference pairs curated by our judge model further improve downstream policy alignment. RSI for LLM-as-judge is thus feasible when self-produced reward validity is explicitly decoupled and monitored.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Recursive Self-Improvement, Rubric, Judge Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4Yc1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a recursive self-improvement method for LLM-as-a-judge. Specifically, the model first acts as a judge, evaluating whether a response satisfies each rubric criterion and producing YES/NO verdicts. A synchronized copy of the model then acts as a checker, evaluating the judge's reasoning and assigning a score from 0 to 4 as the RL reward. The authors also propose an early-stopping metric that combines judge accuracy and checker fidelity to identify an appropriate stopping point before unreliable self-generated rewards lead to performance degradation. Experiments on Qwen3.5-9B, Qwen3.6-27B, and Gemma-4-E4B-it show improvements across medical, preference evaluation, summarization, and professional benchmarks.

Critical strengths
  1. The proposed method is relatively simple and achieves consistent improvements across multiple models and benchmarks, demonstrating its potential effectiveness and generalizability.

  2. The paper studies an important problem, as recursive self-improvement (RSI) offers a promising direction for improving models when external supervision or labeled data is limited.

Critical weaknesses
  1. Interface decoupling appears to address a specific form of reward hacking caused by shared output tokens, but it remains unclear whether it can prevent more general reward exploitation. Since the judge and checker evolve together, stronger models may discover alternative shortcuts, especially during longer training. Moreover, the two-pass evaluation framework resembles existing meta-judging approaches, such as Meta-Rewarding [1]. Direct comparisons under matched computational budgets would help establish the methodological contribution.

  2. Although the RL rewards do not use gold labels, PAV relies on a human-verified validation set for checkpoint selection. Since the reported improvements are often substantially larger at PAV-selected checkpoints than at the final training step, it is important to distinguish the contributions of recursive self-improvement and human-supervised checkpoint selection. While the paper compares different stopping criteria, further analysis of the dependence on human validation data would strengthen its claims.

  3. The improvements on several out-of-distribution benchmarks are relatively modest, often around 1–3 percentage points. Although the results are encouraging and cover multiple model families, additional experiments with independent training seeds would help establish their robustness. Further analysis of reward hacking across model scales and training durations would also clarify the conditions under which the proposed approach remains effective.

References:

[1] Wu, Tianhao, et al. "Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Questions for authors
  1. Could the authors provide compute-matched comparisons with existing meta-rewarding or meta-judging methods to better establish the benefits of the proposed approach?

  2. Could the authors clarify how much of the reported improvement comes from recursive self-training versus PAV-based early stopping?

2 comments

W1

Regarding "prevent more general reward exploitation" and "alternative shortcuts": thank you for this suggestion. Interface decoupling targets the direct shortcut induced by shared labels but does not guarantee protection against other forms of reward exploitation. The late-stage performance degradation reported in the paper underscores the need to monitor reward reliability. PAV supports checkpoint selection but does not guarantee the absence of other shortcuts. We will discuss this limitation in the revised manuscript. (And we will incorporate this and all other updates into the revised PDF once all reviewer feedback has been collected)

Regarding "existing meta-judging approaches": Please see the Meta-Rewarding adaptation and results in our response to tvju's second critical weakness.

W2

Regarding "distinguish the contributions of recursive self-improvement and human-supervised checkpoint selection": thank you for this suggestion. We compare Base, the final checkpoint, and the PAV-selected checkpoint on the same training trajectory. The final checkpoint improves SV-HARD and two OOD metrics, but declines on CheckEval and ProfBench. The PAV-selected checkpoint improves all four OOD metrics. This shows that self-rewarded training produces useful intermediate checkpoints and that validation-based selection helps retain their transfer gains.

MetricFinal − BasePAV-selected − FinalPAV-selected − Base
SV-Hard+15.1−2.2+12.9
SV-Full−2.4+5.6+3.2
HealthBench+0.4+3.4+3.8
RubricBench+1.8+1.3+3.1
CheckEval (Pearson ρ)−0.077+0.096+0.019
ProfBench−1.6+3.2+1.6

Regarding "further analysis of the dependence on human validation data": please see our response to Reviewer tvju’s first question. Replacing human annotations with labels from frontier model selects the same best checkpoint.

W3

Regarding "additional experiments": Thank you for this suggestion. We conducted experiments with two additional seeds. The results below show consistent improvements over Base on the same-style SV-Hard benchmark and both evaluated out-of-domain benchmarks across all three seeds.

ModelSV-Hard-RHealthBench-RProfBench-R
Base60.882.573.1
Paper Best (seed 0)73.786.374.7
Seed 174.885.975.2
Seed 273.584.474.5

Regarding "further analysis of reward hacking across model scales and training durations": Our preliminary observations suggest that larger models have stronger initial capabilities and reach the PAV-identified window earlier in training.

Q1

Please see the Meta-Rewarding adaptation in our response to tvju's second critical weakness.

Q2

Thank you for this suggestion. Please see our response to the second critical weakness.

Dear Reviewer 4Yc1,

Since the discussion phase is ending soon, we would like to kindly ask you to take a look at our rebuttal. We have carefully addressed all the concerns raised in your review and provided point-by-point responses in the rebuttal. If our response has addressed your concerns, we would be grateful if you could consider updating your assessment during the discussion phase.

Thank you very much for your time and thoughtful feedback. We would be happy to provide any further clarifications.

Best regards, The Authors

Reviewer_tvjuOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

RecurSE trains rubric-based LLM judges via closed-loop RL: a weight-synchronized copy of the policy (checker) audits the judge's reasoning against meta-rubrics and emits a 5-tier scalar score as the sole reward — no gold labels or teachers in the RL signal. Interface decoupling closes a token-copying reward shortcut, and Pairwise Advantage Validity (PAV), a holdout monitor, localizes early stopping. Experiments on three model families up to 27B show gains on four held-out benchmarks and downstream DPO transfer.

Critical strengths

The surface-coupling shortcut (Sec 3.2, Fig 3, Fig 5) is a crisp, non-obvious finding: coupled YES/NO readouts inflate rewards with flat accuracy while the scalar readout stays calibrated — a transferable lesson for self-rewarding RL.

PAV (Sec 3.3) is unusually rigorous for a stopping monitor: a margin-weighted ordering-loss bound fixes the weight-2 checker term a priori; bootstrap CIs, fold stability (Table 6), and oracle-regret comparison (Table 7) support reliability.

Critical weaknesses

No gold-reward RL reference (Sec 4, App T). An upper-bound arm driven by gold rule labels on even a subset of training data would substantially strengthen the "RSI is viable" claim; as it stands, the gap to gold rewards — and thus the headroom left by self-supervision — is unquantified.

No empirical comparison to the closest prior work. Meta-Rewarding, Self-Taught Evaluators, and Grad2Reward are distinguished only conceptually (Table 9); running even one under a matched budget would show that removing external anchors is better, not just different.

Modest effect sizes. CheckEval ρ +0.019 and ProfBench +1.6pp (Table 2) are significant but small, SV-Full sits near ceiling, and DPO gains (+0.59 to +2.37pp, Table 4) are modest; temper the "robust generalization" phrasing.

Speculative SFT-plateau explanation. The dual-variable coupling conjecture (Sec 5.4, App C) is untested; the proposed diagnostic (zero-variance rollout fraction) is cheap to measure and would convert conjecture into evidence.

Questions for authors

How sensitive is the PAV stopping window to gold-label noise in SV-Hard (e.g., LLM-panel labels instead of human verification)?

Does decoupling still work if only the judge's verdict format changes while the checker keeps a coupled readout — is interface asymmetry necessary, or just non-identity?

Why does the external 27B checker (Table 3) underperform the synchronized 9B checker — co-evolution, or style mismatch with judge rollouts?

Does the RSI window shrink or shift with scale, and does PAV still localize it under the shorter 75-step 27B budget?

Can you report the zero-variance rollout-group fraction over training (App C) to test the dual-variable coupling conjecture?

3 comments

W1

Thank you for this valuable suggestion. We agree that a gold-reward RL reference, including one on a training subset, would provide a useful comparison and help quantify the remaining headroom of self-generated rewards. Such an experiment requires human verification of individual rubric criteria, together with annotation quality control. Given the annotation resources available during the rebuttal period, we have not yet been able to complete this additional reference. We will clarify this scope in the revised manuscript and regard a carefully verified gold-reward comparison as an important direction for future work.

W2

Thank you for this suggestion. We adapt Meta-Rewarding's mechanism of comparing the model's own judgments and improving judgment ability through preference training to our rubric-based evaluation task. We train only the Judge, not the response-generating Actor in the original method. We use the same initial model and training input pool, following these steps:

  1. Generate eight judgment trajectories for each training input.
  2. Use the current model to make all 56 ordered pairwise comparisons among the eight judgments.
  3. Correct for position bias and obtain a Bradley–Terry ranking.
  4. Select the highest-ranked judgment as the chosen response and the lowest-ranked judgment as the rejected response.
  5. Filter unsuitable preference pairs and train with DPO (at least 2k samples).
  6. Save and evaluate the updated model before deciding whether to proceed to the next round.

The first adaptation round exceeds the paper's RECURSE Best result on SV-Hard but performs worse on all four OOD benchmarks. The second round declines further on all five metrics. (And we will incorporate this and all other updates into the revised PDF once all reviewer feedback has been collected)

ModelSV-HardHealthBenchRubricBenchCheckEvalProfBench
Base60.8082.5076.000.422073.10
RECURSE Best73.7086.3079.100.441074.70
Meta-Rewarding (Iter. 1)76.3260.8270.880.325759.33
Meta-Rewarding (Iter. 2)61.5053.9962.430.278857.02

One possible explanation is that the DPO epochs in this adaptation allow training to continue beyond a suitable stopping point. In contrast, RECURSE combines recursive self-evolution with PAV-based checkpoint selection to identify an appropriate stopping point.

W3

We agree that several transfer and downstream gains are modest and will temper the corresponding claims. Two additional results show positive gains on both evaluated OOD benchmarks in all three runs. HealthBench reaches 85.53 ± 1.00 versus 82.50 for Base, and ProfBench reaches 74.80 ± 0.36 versus 73.10. Full per-seed results appear in our response to 4Yc1, weakness 3.

W4

Thank you for this suggestion. On the same 512 inputs with eight trajectories per input, the zero-reward-variance fraction increases from 37/512 (7.23%) at initialization to 125/512 (24.41%) at the retained checkpoint. This shows reduced within-group reward differentiation. It is consistent with some examples becoming less informative, but could also reflect changes in checker scoring. It does not directly establish the cause of the SFT plateau, and we will retain the dual-variable explanation as a hypothesis.

Q1

Thank you for this insightful suggestion. To evaluate this, we experimented with using gpt-5.6-sol to generate labels (5% inconsistency). The results show that the optimal stopping steps selected for both Qwen and Gemma remain identical to those derived from the original ground truth. This consistency demonstrates that our PAV stopping window is relatively robust against label noise.

Q2

Please see the label-control experiment in our response to WM4y’s first critical weakness. We examined different degrees of interface asymmetry. Replacing the Checker’s YES/NO labels with CORRECT/INCORRECT introduces token-level non-identity while retaining the same per-rubric binary readout structure. This weaker form of asymmetry partially mitigates degradation but performs worse than the scalar readout. These results suggest that token-level non-identity alone is insufficient to recover the full benefit, and support introducing asymmetry in the readout structure itself.

Q3

Thank you for pointing out these two possible explanations. The external 27B Checker performs well on the other benchmarks, with weaker performance concentrated on HealthBench. Our tentative explanation concerns rubric structure: most criteria in the other benchmarks resemble single-sentence rules, consistent with our training data, although their difficulty, domains, and styles vary. HealthBench criteria often combine multiple requirements within a single rule, which may contribute to the weaker performance. The current comparison does not isolate the contributions of co-evolution and style mismatch with judge rollouts.

Q4

Our preliminary observations suggest that larger models have stronger initial capabilities and reach the PAV-identified window earlier in training. The best 27B checkpoint reported in the paper was also selected by PAV within the 75-step training budget. The final choice was step 60. We greatly appreciate your suggestion and will incorporate it into the revised version.

Q5

Thank you for this suggestion. Please see the experimental results in our response to the fourth weakness. A more direct investigation of dual-variable difficulty coupling would require varying one factor while holding the other fixed. We plan to pursue this investigation in future work.

Dear Reviewer tvju,

Since the discussion phase is ending soon, we would like to kindly ask you to take a look at our rebuttal. We have carefully addressed all the concerns raised in your review and provided point-by-point responses in the rebuttal. If our response has addressed your concerns, we would be grateful if you could consider updating your assessment during the discussion phase.

Thank you very much for your time and thoughtful feedback. We would be happy to provide any further clarifications.

Best regards, The Authors

Reviewer_WM4yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RecurSE, a framework for improving LLM-based rubric judges through recursive self-evaluation, without requiring external gold labels or stronger teachers for RL rewards. The method uses a synchronized copy of the judge as a checker to evaluate its reasoning and provide rewards for reinforcement learning. It identifies two main challenges: reward hacking caused by shared output interfaces between the judge and checker, and the degradation of self-generated rewards over training. They address these with interface decoupling and a new early-stopping metric, Pairwise Advantage Validity. Experiments across three models and several held-out benchmarks show consistent improvements in judging performance.

Critical strengths
  • The paper deals with the problem of improving LLM judges without relying heavily on external supervision. The observation about surface coupling, where self-generated rewards can increase without actual accuracy improvements, is meaningful. The proposed interface decoupling is simple and effective, and the corresponding experiments provide useful evidence of this failure mode.
Critical weaknesses
  • The paper doesn’t compare process-based rewards against rewards based only on final verdict correctness. It is therefore unclear how much of the improvement actually comes from reasoning-level feedback. In addition, the interface-decoupling experiment changes both the output format and the reward representation (binary judgments v.s. a five-level scalar score). Therefore, it's difficult to attribute the improvements specifically to removing shared token bias.
  • The process reward framing is not supported. Meta-rubrics ask whether each verdict is correct, the checker tiers are defined by the fraction of correct verdicts, and the fidelity target U is purely outcome-based. The claim that flawed reasoning is penalized even when verdicts are correct has no supporting experiment.
Questions for authors
  • How much of the improvement comes from evaluating reasoning rather than final verdicts? Would a checker that only evaluates verdict correctness achieve similar results?
  • Could the authors isolate the effect of changing the output format from the effect of changing the reward representation?
  • Could the authors clarify how the pairwise ordering bound relates to the normalized advantages used in RL? Also, how does PAV compare against using checker fidelity alone for checkpoint selection?
4 comments

W1

Regarding "process-based rewards against rewards based only on final verdict correctness": Thank you for this insightful suggestion. Our checker design evolved through three variants: (1) per-rubric YES/NO outputs sharing the Judge’s readout; (2) a standalone 0–4 score based only on verdict correctness (As indicated below, it is referred to as "Verdict-only"); and (3) the same 0–4 readout with reasoning quality additionally included in the scoring criteria. We conducted an additional ablation study where the Checker provides rewards based solely on final verdict correctness (a standalone 0–4 score based only on verdict correctness): 0 = all rule judgments are wrong 1 = most rule judgments are wrong (only a small fraction correct) 2 = about half of the rule judgments are correct 3 = most rule judgments are correct (few wrong) 4 = all rule judgments are correct As shown in the table below, removing the evaluation of process quality slightly improves same-style performance (SV-HARD-R) but leads to a degradation in out-of-domain generalization (HealthBench-R and ProfBench-R). One possible explanation is the omission of reasoning patterns within the process.

SV-HARD-RHealthBench-RProfBench-R
Ours73.786.374.7
Verdict-only74.384.572.8

Regarding "changes both the output format and the reward representation": Thank you for this suggestion. We introduce a label control in which the first-pass Judge retains YES/NO verdicts and the second-pass Checker replaces YES/NO with CORRECT/INCORRECT. The per-rubric binary decisions, meta-rubrics, input content, analysis requirements, and reward aggregation remain unchanged. The results are available at: https://anonymous.4open.science/api/repo/5b762aa5924a9053-0631/file/images/interface-label-control.png. (And we will incorporate this and all other updates into the revised PDF once all reviewer feedback has been collected) The measured control shows that changing the labels partially mitigates degradation but does not match the scalar method (reward representation).

W2

Thank you for pointing this out. Regarding "process reward": Our original intent was to convey that our checker assigns a single scalar reward to the Judge’s complete response, with scoring criteria that consider both verdict correctness and reasoning quality, as specified in Appendix I. Our terminology was inaccurate, and we will correct all related descriptions in the revised manuscript.

Regarding "flawed reasoning is penalized even when verdicts are correct has no supporting experiment": Thank you for the reminder. We mentioned this in Subsection 3.1 and the appendix. Our intention was to express that the motivation behind our checker input template design includes supervising the reasoning process. We will remove the relevant statements in the revised version.

Q1

Please see our response to the first weakness. Including explanation quality yields +1.8 percentage points on HealthBench and +1.9 on ProfBench relative to verdict-only, while SV-HARD is 0.6 points lower.

Q2

Please refer to our response to the first weakness. Simply changing the output format (e.g., from YES/NO to CORRECT/INCORRECT) has limited effect. When we move from binary classification to 0–4 classification (with the corresponding need to specify the meaning of each number), performance improves markedly. Relabeling the binary outputs partially mitigates degradation, while the scalar scoring design (reward representation) performs better.

Q3

Regarding "pairwise ordering bound relates to the normalized advantages": Within each rollout group, advantage normalization subtracts a shared mean and divides by a shared positive denominator. Thus, Aᵢ-Aⱼ=(Sᵢ-Sⱼ)/(σ_{S}+ϵ), which preserves the reward ordering. Consequently, pairwise ordering errors are unchanged by normalization. The bound controls margin-weighted ordering loss, rather than errors in normalized advantage magnitudes.

Regarding "PAV compare against using checker fidelity alone for checkpoint selection": Using the available statistics, we compared PAV and fidelity-only selection, which maximizes 1−NMAE, over the same 20 candidate checkpoints (steps 10–200, at intervals of 10). Both select step 130 for Qwen3.5-9B. For Gemma-4-E4B-it, PAV selects step 160 and fidelity-only selects step 120, with HealthBench accuracies of 86.45% and 86.19%, respectively. Selection is therefore identical for Qwen, while PAV selects a checkpoint with 0.26 percentage points higher HealthBench accuracy for Gemma. Our primary motivation in defining PAV was to provide a broadly applicable checkpoint-selection criterion. PAV combines Judge accuracy and Checker fidelity, whereas fidelity-only selection considers only the latter. In our experiments, because Judge accuracy increases prior to reaching peak Checker fidelity, the two criteria converge to the same checkpoint for Qwen, while PAV yields a modest performance gain on HealthBench for Gemma.

Dear Reviewer WM4y,

Since the discussion phase is ending soon, we would like to kindly ask you to take a look at our rebuttal. We have carefully addressed all the concerns raised in your review and provided point-by-point responses in the rebuttal. If our response has addressed your concerns, we would be grateful if you could consider updating your assessment during the discussion phase.

Thank you very much for your time and thoughtful feedback. We would be happy to provide any further clarifications.

Best regards, The Authors

Dear Reviewer WM4y,

Since the discussion phase is ending soon, we would like to kindly ask you to take a look at our rebuttal. We have carefully addressed all the concerns raised in your review and provided point-by-point responses in the rebuttal. If our response has addressed your concerns, we would be grateful if you could consider updating your assessment during the discussion phase.

Thank you very much for your time and thoughtful feedback. We would be happy to provide any further clarifications.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.