acceptodds
Under review as a conference paper at ICLR 2027

What Do Self-Improvement Methods Optimize For? An Empirical-Mechanistic Study

Abstract

Self-improvement methods such as debate, simple bootstrap, Gibbs sampling, and internal coherence maximization (ICM) can improve language model accuracy without external supervision, yet the signal they optimize remains unclear. What drives the gains, and when? We test the hypothesis that these methods improve accuracy through *coherence optimization* (Qiu et al., 2026), finding behaviors that are jointly predictable under a model's prior. For debate, bootstrap, and Gibbs sampling, we find that, overall, coherence and training-pool accuracy increase in sync, and rejecting update steps that increase coherence also lowers accuracy, suggesting a causal relation. For the remaining method, ICM, where ablation is inapplicable, Qiu et al. (2026) proved its analytical equivalence to coherence optimization under an information saturation condition, which we experimentally verified. Our results suggest that coherence optimization provides a common objective for understanding and designing feedback-free self-improvement methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language models, self-improvement, coherence optimization, Gibbs sampling, multi-agent debate, unsupervised learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_hAsWOpenReview ↗
Rating
1(−2)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper empirically investigates whether several feedback-free self-improvement methods share a common objective: coherence, understood as the predictability of a set of responses under a model’s prior. Building on an earlier theoretical account, it measures per-token leave-one-out coherence during Gibbs sampling, bootstrap, and cooperative debate, and introduces rejection filtering that discourages coherence-increasing updates. It also examines the informational-saturation condition connecting ICM’s mutual-predictability objective to autoregressive coherence.

Critical strengths
  1. The paper goes beyond reporting that iterative self-improvement works and asks what measurable signal accompanies its gains across different procedures. Applying a common analysis to Gibbs sampling, bootstrap, and debate provides a useful basis for comparing their behavior.
  2. The main Gibbs and bootstrap trajectories each use 100 data seeds, and the bootstrap and debate rejection experiments use paired comparisons. Reporting within-seed associations, final-round correlations, and intervention outcomes provides more evidence than a single aggregate trajectory.
Critical weaknesses
  1. The rejection intervention does not isolate coherence as the causal mechanism. The paired experiments establish that adding coherence-based rejection changes accuracy, but they do not uniquely identify coherence as the mediating factor.
  2. The gap between the theoretical objective and the measured metric is central. The theory concerns unnormalized autoregressive coherence, whereas the main experiments measure length-normalized leave-one-out log-probability. These changes affect both the conditioning structure and the objective’s dependence on response length; they are not merely implementation details. The paper explicitly notes that its normalized metric is not an approximation of the unnormalized theoretical objective.
Questions for authors
  1. Can the rejection experiment isolate the effect of coherence-based selection?
  2. Can autoregressive coherence, raw leave-one-out coherence, and normalized leave-one-out coherence be measured on the same manageable subsets?
Reviewer_LhzROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper empirically examines the theory that coherence optimization offers a unified explanation for language model self-improvement, studying Gibbs sampling, bootstrapping, cooperative debate, and ICM. The authors track per-token coherence and accuracy over successive rounds and reject coherence-increasing updates to investigate their association and possible causal relationship. The main experiments show positive correlations between the two measures. Rejection filtering significantly reduces accuracy in bootstrapping and debate, while Gibbs sampling improves held-out accuracy at a fixed final round. For ICM, the gap between its mutual-predictability objective and autoregressive coherence narrows as more context is provided, offering evidence of partial informational saturation.

Critical strengths
  1. This paper addresses an important question beyond improving the performance of an individual algorithm. It asks whether different self-improvement methods share a common mechanism, rather than simply proposing another prompting or training technique. Studying these methods through a common framework of measurement and intervention helps clarify when self-generated data can improve performance and may inform the design of future methods.
  2. The main experiments are well replicated and transparently reported. The authors use multiple random seeds, report uncertainty estimates and paired comparisons, and demonstrate held-out gains from Gibbs sampling. They clearly distinguish training from test results and acknowledge the limits of the evidence.
Critical weaknesses
  1. The citation practices and overall presentation could be more consistent. References to Qiu et al. (2026) are repeated frequently within individual paragraphs and could be consolidated where attribution remains clear. GSM8K is cited only at a later mention rather than when first introduced. Citation formatting also needs attention. Some pages contain large gaps between text and figures, and the wording, terminology, and notation would benefit from further editing.
  2. The connection between the measured metric and the theoretical objective needs further support. The theory concerns unnormalized autoregressive coherence, whereas the main experiments measure token-normalized leave-one-out coherence. The authors explicitly acknowledge this distinction, but it remains unclear how strongly correlations between the latter and accuracy support predictions about the former. The ICM experiments show that the two objectives become closer, yet a gap remains under gold labels, and the setting differs from the main generation experiments. The evidence therefore supports an association between measured predictability and performance more directly than it validates the proposed unifying objective.
  3. The research question is interesting, but it appears to have already been addressed by Qiu et al. (2026). The present paper mainly builds on that work through further empirical analysis, which raises concerns about the scope and novelty of its contribution.
Questions for authors
  1. What evidence connects the measured coherence metric to the theoretical objective? In the main Gibbs, bootstrap, and debate settings, have you compared changes in autoregressive coherence, unnormalized leave-one-out coherence, and per-token coherence?
  2. Which sampler and evaluation procedure are used for the main Gibbs results? Why is round-0 test accuracy higher than the zero-shot baseline, and how much of the final improvement can be attributed to mixing in fresh zero-shot responses?
  3. Could the authors clarify what substantive new work and insights this paper offers beyond Qiu et al. (2026)?
Reviewer_sYcKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper tries to find empirical evidence that the gains for self-improvement methods such as debate, bootstrapping, Gibbs Sampling, internal coherence maximization are driven by 'coherence optimization' introduced by Qiu et al. For this they design counterfactual training where if they remove samples where coherence decreases the performance also decreases which goes on to establish a possible causal relationship.

Critical strengths

The paper is an interesting study on trying to find the root cause which leads to improvement in self training methods.

Critical weaknesses
  1. The paper is difficult to understand, with too many numerical claims in the experimental section without any space for explanation or interpretation. For instance, it's hard to parse what is the takeaway for Section 5.1. Also, there is heavy jargon such as 'partial informational saturation' which is not clearly explained or defined.

  2. The paper doesn't clearly state the claims it is trying to make. Is the paper an experimental extension to Qiu et al? Is it trying to go deeper than what they claim? Its hard to say what the key idea in the entire paper is.

Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.