acceptodds
Under review as a conference paper at ICLR 2027

When Self-Reinforcement Ends Self-Improvement

Abstract

Self-improvement systems use a judge to select revisions and reinforce those that pass. The quality of the judge is typically assessed through evaluation accuracy, but this metric does not capture whether errors recur on the same inputs. We show that whether the judge's errors are deterministic determines whether reinforcement ends improvement. In a bounded utility model with a deterministic binary judge and expected losses exceeding expected gains, we prove that every fixed acceptance rule that initially improves utility eventually produces negative drift. The preference for positive outputs that enables improvement also reinforces the judge's errors. For adaptive rules and judges with finitely many outputs, we derive a tradeoff between the number of revisions and cumulative negative drift, and we characterize when a suitable acceptance rule can sustain improvement. A comparison with independently refreshed errors isolates the role of deterministic errors: two judges with the same evaluation accuracy produce opposite signs of eventual drift. A sequential test measures expected utility change with sample bounds that distinguish revision frequency, magnitude, and evaluation noise. Experiments with language model judges corroborate the theoretical predictions. We identify whether errors are deterministic as a property that determines the outcome of self-reinforcement and is not captured by evaluation accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
self-improving systems, llm-as-a-judge

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FTDWOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The issue this paper tries to model is that when a AI is trained against a judge with consistent blind spots, the AI learns to exploit them which drives further training into those blind spots. The main theoretical result is that, under their assumptions, any deterministic rule for which updates to judge as positive eventually hurts performance. This is validated in experiments where on common benchmarks.

Critical strengths

This paper introduces valuable formalization of the issue that self-reinforcement can lead to propagating issues with the judge, along with a stylized model of how that could evolve. The technical results appear sound, as do the experiments, conditional on the assumptions made.

Critical weaknesses

The main insight of this paper, that systemic errors can lead to degenerate feedback loops, is not novel. Given that, the main results are not actionable since it's not clear how to create a judge without deterministic biases. Adding a clear proposal for how to do so would significantly improve this paper.

Relatedly, the experiments feel very contrived, since they impose whether a judge is deterministic or not through scaffolding that would not exist in practical scenarios.

Questions for authors

What kind of approaches appear promising to you for generating judges that do not make deterministic errors?

How would you position your work within the existing literature? The literature review section of your paper is extremely brief?

Reviewer_b34LOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper makes a self-improvement model to show judge errors will reach to negative improvement eventually under some assumptions. Some theoretical derivations are provided to justify the phenomenon. Both synthetic and practical benchmarks are given to support the theory.

Critical strengths

The paper provides a simple yet interesting theoretical model to answer when self-improvement stops improving. The model shows that the expected improvement eventually decay to negative due to judge errors. The model is supported by synthetic and practical tasks.

Critical weaknesses

The paper does not establish whether its conclusions extend to stochastic judges with input-dependent error probabilities. Section 2.1 assumes that beneficial proposal probability pp remains fixed and only the multiplicities of harmful templates change. Under this assumption, accepted harmful templates accumulate, while beneficial proposals cannot become more likely through reinforcement. Real self-improvement systems would be able to reinforce successful strategies, prompts, or solutions alongside unsuccessful ones. Section 4.2 replays previously released responses without generating new candidates. So, the experiments do not capture important features of genuine self-improvement: changes in proposal quality, discovery of new strategies, generalization from successful revisions, or evolving correlations between the generator and judge. Theorem 2 assumes a fixed finite judge-output space, fixed output distributions, and acceptance rules that cannot observe the current proposal's identity or true utility. The proposed sequential test can be substantially less sample-efficient than existing methods.

Questions for authors

Proposition 2 contrasts deterministic errors with independently refreshed, uniformly distributed errors. However, a stochastic judge with persistent input-dependent probabilities could also preferentially reinforce error-prone templates. Would the eventual negative-drift result extend to this setting? If so, is persistent error heterogeneity, rather than determinism itself, the fundamental mechanism? Section 2.1 assumes that only harmful template multiplicities change, while the beneficial proposal probability remains fixed. In practical self-improvement systems, successful proposals may also be reinforced. Would Theorem 1 or Corollary 1 continue to hold if both beneficial and harmful templates were copied according to acceptance? Section 4.2 replays a fixed collection of responses without generating new proposals. How do the conclusions generalize to systems where the generator itself evolves, potentially improving proposal quality or discovering new beneficial strategies through reinforcement? The JETTS experiment introduces a suffix claiming ground-truth agreement and verification by independent graders. Can the authors demonstrate similar degradation under naturally occurring judge errors, without deliberately injecting misleading verification claims? Section 4.2 reports that the suffix increases the judge's YES rate on incorrect responses from 23% to 76%, whereas Appendix J.2 reports an increase from 13% to 63%. Do these percentages correspond to different subsets or evaluation protocols?

Reviewer_ATSEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how deterministic errors in LLM judges can be amplified through self-reinforcing revision processes, potentially reversing initially positive utility improvements. The authors develop a theoretical framework based on reinforced sampling, deriving conditions under which fixed acceptance policies eventually exhibit negative expected utility drift. They further extend the analysis to adaptive acceptance policies, establish a trade-off between sustained revisions and cumulative negative drift, and propose a sequential statistical test for detecting drift. Experiments on synthetic processes and LLM-based evaluation benchmarks (RM-Bench and JETTS) are presented to support the theoretical findings.

Critical strengths
  1. Rigorous theoretical analysis. The paper provides a mathematically well-developed characterization of error amplification under its proposed reinforcement model. In particular, the analysis of adaptive acceptance policies and the associated cumulative negative-drift bound (Theorem 2) offers a nontrivial theoretical result.

  2. Clear analytical characterization. The explicit threshold for drift reversal and the comparison between deterministic and independently refreshed errors provide an interpretable illustration of how temporal error dependence can interact with feedback mechanisms.

  3. Multiple experimental settings. The authors investigate the proposed mechanism using both synthetic simulations and LLM-based evaluation benchmarks, including several controlled comparisons intended to isolate the effects of reinforcement.

Critical weaknesses
  1. Limited novelty of the central observation. The main phenomenon appears largely predetermined by the model construction. Under the proposed copying mechanism, incorrectly accepted harmful templates are repeatedly reinforced, inevitably increasing their representation in the proposal distribution. Given the assumption that expected losses exceed expected gains, the eventual reversal of utility drift follows rather directly. The resulting phenomenon closely resembles established selection-bias amplification and reward-hacking mechanisms. Although the mathematical analysis is relatively complete, the central conceptual insight seems limited.

  2. Questionable attribution to deterministic errors. The paper emphasizes error determinism as the distinguishing factor behind the failure of self-improvement. However, determinism does not appear necessary for the proposed mechanism. A stochastic judge with persistent identity-dependent error probabilities can produce similar reinforcement dynamics: templates with higher false-positive probabilities receive greater reinforcement and can eventually dominate the proposal distribution. This suggests that the essential mechanism is the correlation between error propensity and reinforcement, rather than determinism itself. The current formulation may therefore overstate the conceptual significance of deterministic errors.

  3. Substantial gap between theoretical assumptions and empirical settings. The theoretical model assumes fixed probabilities of beneficial and harmful proposals, a fixed gain-to-loss ratio, and a particular harmful-template copying mechanism. In contrast, the LLM experiments use binary correctness labels and reinforce response styles or model-source weights. These settings do not directly satisfy several key theoretical assumptions. Consequently, the experiments demonstrate related feedback effects rather than directly validating the theoretical predictions in realistic self-improvement systems.

  4. Insufficient causal isolation in the LLM experiments. The synthetic experiments largely reproduce behavior already guaranteed by the theoretical construction. More importantly, the real LLM experiments do not directly compare deterministic and stochastic judges with matched marginal accuracy while varying temporal error dependence. The RM-Bench controls also modify the relationship between proposal style and correctness, while the JETTS experiment deliberately increases the acceptance rate of a weak model through persuasive suffixes. These experiments support the amplification of biased selection but do not convincingly establish that error determinism is the critical explanatory factor.

  5. Limited practical relevance of the sequential testing component. The proposed sequential test addresses the estimation of expected utility drift using an independent utility evaluator, rather than directly diagnosing deterministic judge errors. Its connection to the central reinforcement mechanism is therefore somewhat indirect. Furthermore, the experimental comparisons in Appendix J.1 do not demonstrate a consistent efficiency advantage over existing confidence-bound methods, leaving its practical contribution unclear.

Questions for authors
  1. Is determinism genuinely necessary? Consider a stochastic judge whose errors are independent across repeated evaluations conditional on template identity, but whose false-positive probabilities differ across identities. Under acceptance-driven copying, templates with larger false-positive probabilities would be preferentially reinforced. Would this not produce the same eventual drift reversal? If so, what distinguishes the proposed mechanism from general amplification of structured evaluation bias?

  2. How do the theoretical results transfer to the LLM experiments? In particular, how should the fixed gain-to-loss ratio and the assumptions about harmful-template copying be interpreted in RM-Bench and JETTS, where utility is binary and the reinforced objects are styles or model sources? Which theoretical predictions are being tested quantitatively, rather than illustrated qualitatively?

  3. Can the authors provide a more direct causal comparison? It would be informative to compare judges with matched marginal accuracy and false-positive rates but different temporal error correlations in an actual LLM-based reinforcement loop. This would more directly substantiate the paper's central claim about determinism.

  4. How does the proposed explanation differ from distribution shift and reward hacking? In Proposition 2, the judges have identical accuracy on a fixed evaluation panel, while their effective error rates under the evolving proposal distribution differ. To what extent can the observed failure be explained by conventional on-policy distribution shift, without explicitly invoking error determinism?

  5. How robust are the results to alternative reinforcement mechanisms? For example, what happens when accepted beneficial proposals are also reinforced, when copying is regularized or capped, or when the acceptance policy can use source identity or additional proposal features? These cases seem important for understanding the relevance of the conclusions to practical self-improvement systems.

  6. Could the authors clarify the reported JETTS statistics? Section 4.2 reports that the persuasive suffix increases the YES rate on incorrect responses from 23% to 76%, whereas Appendix J.2 reports 13% to 63%. Which figures are correct?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.