acceptodds
Withdrawn from ICLR 2027

Independence over Accuracy: Generator–Verifier Error Correlation in Recursive Self-Improvement in Small Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Verifier accuracy does not specify whether erroneous feedback repeats or corrects a policy’s mistaken judgments. We study this distinction using a binary verdict channel that fixes expected verifier accuracy while varying error correlation with a specified self-judge. Under an explicit conditional-independence assumption, the expected reward difference between correct and incorrect responses with the same self-judgment decreases linearly with correlation when the self-judge error rate is below one half. The resulting correlation tax defines an equivalent independent accuracy by matching conditional reward gaps. At a self-judge error rate of 0.30, a maximally correlated 90% verifier matches the gap of an independent 83.3% verifier, while a maximally correlated 97% verifier retains a 95% equivalent accuracy. Exact analysis identifies the feasible correlation range, the region where independence offsets lower accuracy, and the bounded tax as the self-judge error rate decreases. Reported Monte Carlo channel checks agree with these population calculations. A proposed five-round GRPO–LoRA protocol tests whether the conditional reward gap predicts learning, using controlled and natural verifiers and a frozen control. No language-model training outcome is claimed.

Primary area:
reinforcement learning
Keywords:
language model self-improvement, generator-verifier error correlation, verifier reliability, reinforcement learning, reward signals, controlled experiments

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oh74OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that verification error should be considered in combination with how correlated it is with the generator’s error pattern. It shows that a verifier whose error pattern has higher correlation with the generator can match another verifier that has less correlation, even if it has lower accuracy. The paper introduces a mathematical framework to discuss this aspect.

Critical strengths
  • The scope of the work, verification error, and its influence on GRPO-style training is practical and important.
Critical weaknesses
  • The argument of the paper reads like its topic is “the influence of verification error on LLM training (i.e., GRPO).” However, it does not have any experimental results on LLM training, making its practical takeaway unclear and limited.
  • There is no discussion of the relation to prior works on imperfect verifiers that draw various conclusions, such as [1-3]. This paper's stance, that the verifier’s quality should be considered beyond its error rate, sounds particularly relevant to [3].

[1] https://arxiv.org/abs/2601.04411 [2] https://arxiv.org/abs/2604.07666 [3] https://arxiv.org/abs/2605.02909

Questions for authors

N/A

Reviewer_QYnSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a verifier's accuracy suffices to describe the reward it gives in LLM self-improvement, or whether it also matters where its errors fall relative to the policy's own binary self-judgment s^\hat{s}. It defines a controlled verdict channel v=c⊕Ev=c\oplus E, where cc is correctness, s^=c⊕e\hat s=c\oplus e, the verifier error depends on the self-judge error only (E⊥c∣eE\perp c\mid e), expected accuracy aa is fixed, and the error correlation ρ=corr(E,e)\rho=\mathrm{corr}(E,e) varies within a feasible range (Eqs. 1–3). The conditional reward gap Δ=E[v∣c=1,s^]−E[v∣c=0,s^]\Delta=E[v\mid c=1,\hat s]-E[v\mid c=0,\hat s] is the same in both s^\hat s strata and equals (2a−1)−κρ(2a-1)-\kappa\rho (Eq. 5, Prop. 1), which gives an "equivalent independent accuracy" aeq=(1+Δ)/2a_{eq}=(1+\Delta)/2 (Eq. 6). The paper derives the region where a weaker independent verifier beats a stronger correlated one (Eq. 8, Fig. 5), a bounded maximal "tax" (Eq. 9, Table 2), and the non-monotone behavior of I(v;c∣s^)I(v;c\mid\hat s) in ρ\rho (Sec. 4.5). At ps=0.30p_s=0.30, a maximally correlated 90% verifier matches an independent 83.3% one (Table 1). Validation is limited to exact calculations and Monte Carlo checks of the channel on 4×1054\times10^5 synthetic responses (Table 3). A five-round GRPO–LoRA protocol (Qwen2.5-1.5B-Instruct, GSM8K, 15 arms, 3 seeds, hypotheses P1–P7) is specified but not run, and the paper states that no training outcome is claimed.

Critical strengths
  1. The analytical results are correct and carefully scoped. I recomputed ρmax⁡\rho_{\max}, Δ\Delta, aeqa_{eq} and the maximal tax from Eqs. 2, 3, 5 and 9. They reproduce Tables 1 and 2, the exact mutual-information values in Table 3 and the Sec. 4.3 example. The paper separates the copying endpoint (ρ=1\rho=1, v=s^v=\hat s) from same-model or majority-vote judges (Sec. 3.4), and says that Δ\Delta does not determine normalized GRPO advantages. My concerns are about significance and evidence, not correctness.

  2. It corrects a folk intuition. "An independent weaker verifier beats a correlated stronger one" holds only in a band that depends on psp_s (Eq. 8, Fig. 5): at ps=0.30p_s=0.30, a maximally correlated 97% verifier keeps a larger gap than an independent 90% one (0.900 vs 0.800, Tables 1 and 4, P4b). At ps=0.30p_s=0.30 the bounded tax is small (0.020 for a=0.97a=0.97, Table 2), although it grows to 0.07–0.27 for ps≤0.15p_s\le0.15.

Critical weaknesses
  1. The central question, whether error correlation with the self-judge affects self-improvement, is never tested, and the analytical part alone is too thin.

    • The abstract states that no training outcome is claimed, and the Fig. 1 caption and the Reproducibility Statement call the protocol "unexecuted". The protocol and hypotheses P1–P7 (Sec. 5, Table 4, App. C–D) are not run. The only empirical content is a Monte Carlo check that a sampler reproduces its own marginals (Table 3), which validates the code, not the claim.
    • The paper's own results show that the more accurate correlated verifier has the larger Δ\Delta at ps=0.30p_s=0.30, ties at ps=0.15p_s=0.15 and loses only at smaller psp_s such as 0.10 (Table 1, Sec. 4.2, P4b, App. D), so the title "Independence over Accuracy" is an overclaim.
    • The derivations are elementary: Δ=1−π1−π0\Delta=1-\pi_1-\pi_0 follows in two lines from E⊥c∣eE\perp c\mid e (App. A.2), and the bounded tax follows from evaluating Eq. 5 at the two endpoint cases of ρmax⁡\rho_{\max} (App. A.3).

    This is decisive because the missing study is the core of the paper's motivation, not a small add-on, and the full protocol is too large to expect in a rebuttal (an upper bound of about 6.3×1096.3\times10^9 generated tokens, Table 5, App. F), although a reduced version is feasible. I do not penalise the paper for the absence of training results as such, since it says so openly. My concern is that, without them, the remaining contribution (correct but elementary algebra) is of limited novelty and significance for ICLR.

  2. Δ\Delta is not connected to what GRPO optimizes, and under the paper's own population the unconditional reward signal does not depend on ρ\rho.

    • Training uses group-relative rewards over the KK samples of a prompt (App. C.1), not strata of s^\hat s, which the learner never observes.
    • Under the worked assumptions, P(E=1∣c)=1−aP(E=1\mid c)=1-a for both classes, so the unconditional gap 2a−12a-1 and I(v;c)I(v;c) are invariant to ρ\rho (App. B.1, Fig. 6). The "correlation tax" appears only after conditioning on s^\hat s.
    • The paper concedes that at Δ=0\Delta=0 "learning can still occur through differences across strata" (Sec. 3.4). It offers no gradient calculation, toy simulation or other argument for why within-stratum separation should rank verifiers, so the predictive hypotheses P1, P4 and P7 have no grounding beyond the definition of Δ\Delta.

    This is the bridge between the algebra and the motivation, and its absence weighs heavily with Weakness 1.

Questions for authors
  1. [Clarification] Under e⊥ce\perp c and E⊥c∣eE\perp c\mid e, the unconditional gap is 2a−12a-1 and I(v;c)I(v;c) is constant in ρ\rho (App. B.1). What makes the s^\hat s-conditioned gap Δ\Delta the quantity that should predict GRPO gains? Can you give even a toy calculation (e.g., the expected policy-gradient direction for a softmax policy whose sampling correlates with s^\hat s) in which a lower Δ\Delta at fixed aa gives a smaller improvement? This could change my view of Weakness 2.
  2. [Clarification] Were any rounds of the GRPO protocol run, even on a reduced grid? App. D says predictions were recorded "before training outcomes were inspected", which reads as if outcomes exist, whereas the abstract, Fig. 1 and the Reproducibility Statement call the protocol unexecuted. Please reword App. D. If some reduced-scale runs do exist, please report them (e.g. one seed, primary policy, the oracle and frozen controls, and the (0.90,0)(0.90,0) vs (0.90,ρmax⁡)(0.90,\rho_{\max}) and (0.97,ρmax⁡)(0.97,\rho_{\max}) vs (0.90,0)(0.90,0) arms), since they would already bear on P4a/P4b.
  3. [Clarification] Natural judges can err depending on difficulty and correctness (Secs. 3.1, 6), and the proposed self-judges (a response-judging variant and a commit-first variant, Fig. 1) change as the policy updates. How does Δ\Delta split into Δ0≠Δ1\Delta_0\ne\Delta_1 when E⊥c∣eE\perp c\mid e is relaxed, and does the ordering in Eq. 8 survive plausible dependence?
Reviewer_1FznOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates verifier error correlation against a fixed self‑judgment for LLM recursive self‑improvement. The core argument is that raw verifier accuracy alone cannot predict the quality of training signals. The authors build a controlled binary channel model and derive closed‑form results for conditional reward gap, equivalent independent accuracy, and feasible correlation bounds. They introduce the concept of correlation tax, which describes how high overlap between verifier errors and self‑judge errors degrades reward separation even when verifier accuracy stays fixed. Monte‑Carlo simulations validate the mathematical derivations. However, the full GRPO‑LoRA training study designed to test these theoretical predictions is only outlined in the appendix and has not been executed. The paper provides pre‑registered hypotheses and a complete experimental blueprint, but lacks real language model training outcomes to validate whether correlation tax and equivalent accuracy translate to practical learning behaviour.

Critical strengths
  1. The paper raises an under‑appreciated practical point. Most existing work evaluates verifiers purely by overall accuracy. This work highlights that error overlap between verifier outputs and the model’s own judgements can meaningfully degrade reward quality, even for high‑accuracy verifiers. This observation is relevant for many self‑reward and self‑play training setups.
  2. The analytical derivation is carefully handled. The authors clearly state model assumptions, derive feasibility constraints for the controlled channel, and separate mathematical results from real‑world training claims. They repeatedly warn that conditional reward gap does not directly equal final policy improvement. Synthetic Monte‑Carlo checks confirm the algebra.
  3. The proposed empirical protocol is well‑thought‑out. It includes frozen policy baselines, multiple verifier types, statistical auditing procedures and pre‑written hypotheses. This sets a good example for rigorous empirical work on self‑improvement, where null effects and measurement noise are common.
Critical weaknesses
  1. The key empirical evaluation is missing. The whole point of this analysis is to understand real‑world self‑improvement. All GRPO training experiments are only planned but not run. We have no evidence showing that correlation tax or equivalent independent accuracy actually predicts model performance changes. Right now this remains a theoretical exercise with a proposed experiment blueprint.
  2. The core channel model builds on the conditional independence assumption E perpendicular to c given e. Real LLM verifiers can violate this assumption in complicated ways. The paper provides almost no real‑world measurement of correlation values from existing self‑reward pipelines. It is hard to judge how applicable these formulas are to practical judge implementations.
  3. The analysis is restricted strictly to binary yes‑no verdicts. A large number of practical verifiers produce continuous scores or multi‑level feedback. The paper only gives very limited discussion on extending the framework beyond binary cases.
  4. There is limited grounding against existing empirical results. The authors do not take existing public self‑reward datasets and measure the practical rho values for common judges such as majority vote or self‑reflection judges. Without such numbers, readers cannot tell what magnitude of correlation tax to expect in real experiments.
Questions for authors
  1. What kinds of real‑world verifier behaviours would violate your conditional independence assumption, and how would that alter your conditional reward gap results?
  2. Could you provide measured rho values from real self‑reward datasets for commonly used judge approaches, even if you cannot run full training sweeps?
  3. If verifiers output continuous soft scores instead of binary verdicts, which parts of your theoretical framework carry over, and which need to be reworked?
  4. Are there simple proxy metrics that practitioners could compute during training to estimate correlation tax without heavy additional computation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.