- Rating
- 1(−2)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper empirically investigates whether several feedback-free self-improvement methods share a common objective: coherence, understood as the predictability of a set of responses under a model’s prior. Building on an earlier theoretical account, it measures per-token leave-one-out coherence during Gibbs sampling, bootstrap, and cooperative debate, and introduces rejection filtering that discourages coherence-increasing updates. It also examines the informational-saturation condition connecting ICM’s mutual-predictability objective to autoregressive coherence.
Critical strengths
- The paper goes beyond reporting that iterative self-improvement works and asks what measurable signal accompanies its gains across different procedures. Applying a common analysis to Gibbs sampling, bootstrap, and debate provides a useful basis for comparing their behavior.
- The main Gibbs and bootstrap trajectories each use 100 data seeds, and the bootstrap and debate rejection experiments use paired comparisons. Reporting within-seed associations, final-round correlations, and intervention outcomes provides more evidence than a single aggregate trajectory.
Critical weaknesses
- The rejection intervention does not isolate coherence as the causal mechanism. The paired experiments establish that adding coherence-based rejection changes accuracy, but they do not uniquely identify coherence as the mediating factor.
- The gap between the theoretical objective and the measured metric is central. The theory concerns unnormalized autoregressive coherence, whereas the main experiments measure length-normalized leave-one-out log-probability. These changes affect both the conditioning structure and the objective’s dependence on response length; they are not merely implementation details. The paper explicitly notes that its normalized metric is not an approximation of the unnormalized theoretical objective.
Questions for authors
- Can the rejection experiment isolate the effect of coherence-based selection?
- Can autoregressive coherence, raw leave-one-out coherence, and normalized leave-one-out coherence be measured on the same manageable subsets?