- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a recursive self-improvement method for LLM-as-a-judge. Specifically, the model first acts as a judge, evaluating whether a response satisfies each rubric criterion and producing YES/NO verdicts. A synchronized copy of the model then acts as a checker, evaluating the judge's reasoning and assigning a score from 0 to 4 as the RL reward. The authors also propose an early-stopping metric that combines judge accuracy and checker fidelity to identify an appropriate stopping point before unreliable self-generated rewards lead to performance degradation. Experiments on Qwen3.5-9B, Qwen3.6-27B, and Gemma-4-E4B-it show improvements across medical, preference evaluation, summarization, and professional benchmarks.
Critical strengths
-
The proposed method is relatively simple and achieves consistent improvements across multiple models and benchmarks, demonstrating its potential effectiveness and generalizability.
-
The paper studies an important problem, as recursive self-improvement (RSI) offers a promising direction for improving models when external supervision or labeled data is limited.
Critical weaknesses
-
Interface decoupling appears to address a specific form of reward hacking caused by shared output tokens, but it remains unclear whether it can prevent more general reward exploitation. Since the judge and checker evolve together, stronger models may discover alternative shortcuts, especially during longer training. Moreover, the two-pass evaluation framework resembles existing meta-judging approaches, such as Meta-Rewarding [1]. Direct comparisons under matched computational budgets would help establish the methodological contribution.
-
Although the RL rewards do not use gold labels, PAV relies on a human-verified validation set for checkpoint selection. Since the reported improvements are often substantially larger at PAV-selected checkpoints than at the final training step, it is important to distinguish the contributions of recursive self-improvement and human-supervised checkpoint selection. While the paper compares different stopping criteria, further analysis of the dependence on human validation data would strengthen its claims.
-
The improvements on several out-of-distribution benchmarks are relatively modest, often around 1–3 percentage points. Although the results are encouraging and cover multiple model families, additional experiments with independent training seeds would help establish their robustness. Further analysis of reward hacking across model scales and training durations would also clarify the conditions under which the proposed approach remains effective.
References:
[1] Wu, Tianhao, et al. "Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.
Questions for authors
-
Could the authors provide compute-matched comparisons with existing meta-rewarding or meta-judging methods to better establish the benefits of the proposed approach?
-
Could the authors clarify how much of the reported improvement comes from recursive self-training versus PAV-based early stopping?
2 comments
Dear Reviewer 4Yc1,
Since the discussion phase is ending soon, we would like to kindly ask you to take a look at our rebuttal. We have carefully addressed all the concerns raised in your review and provided point-by-point responses in the rebuttal. If our response has addressed your concerns, we would be grateful if you could consider updating your assessment during the discussion phase.
Thank you very much for your time and thoughtful feedback. We would be happy to provide any further clarifications.
Best regards, The Authors
W1
Regarding "prevent more general reward exploitation" and "alternative shortcuts": thank you for this suggestion. Interface decoupling targets the direct shortcut induced by shared labels but does not guarantee protection against other forms of reward exploitation. The late-stage performance degradation reported in the paper underscores the need to monitor reward reliability. PAV supports checkpoint selection but does not guarantee the absence of other shortcuts. We will discuss this limitation in the revised manuscript. (And we will incorporate this and all other updates into the revised PDF once all reviewer feedback has been collected)
Regarding "existing meta-judging approaches": Please see the Meta-Rewarding adaptation and results in our response to tvju's second critical weakness.
W2
Regarding "distinguish the contributions of recursive self-improvement and human-supervised checkpoint selection": thank you for this suggestion. We compare Base, the final checkpoint, and the PAV-selected checkpoint on the same training trajectory. The final checkpoint improves SV-HARD and two OOD metrics, but declines on CheckEval and ProfBench. The PAV-selected checkpoint improves all four OOD metrics. This shows that self-rewarded training produces useful intermediate checkpoints and that validation-based selection helps retain their transfer gains.
Regarding "further analysis of the dependence on human validation data": please see our response to Reviewer tvju’s first question. Replacing human annotations with labels from frontier model selects the same best checkpoint.
W3
Regarding "additional experiments": Thank you for this suggestion. We conducted experiments with two additional seeds. The results below show consistent improvements over Base on the same-style SV-Hard benchmark and both evaluated out-of-domain benchmarks across all three seeds.
Regarding "further analysis of reward hacking across model scales and training durations": Our preliminary observations suggest that larger models have stronger initial capabilities and reach the PAV-identified window earlier in training.
Q1
Please see the Meta-Rewarding adaptation in our response to tvju's second critical weakness.
Q2
Thank you for this suggestion. Please see our response to the second critical weakness.