acceptodds
Under review as a conference paper at ICLR 2027

On the Edge of Understanding: Churn in LLM Post-Training

Abstract

Beneath steady accuracy improvements in LLM post-training, prior work finds that many queries flip between solved and unsolved across training epochs. However, the underlying mechanisms driving these flips remain poorly understood. Through large scale analysis, we find a single gradient update flips of learnable queries between solved and unsolved throughout training, across both in-distribution and out-of-distribution datasets — a phenomenon we term . Accuracy gain is the surplus of positive over negative churn, both persisting even after accuracy plateaus. Churning queries reside on LLM's edge of understanding, razor-thin confidence margin between correct and incorrect answers makes correct answer more vulnerable to perturbation. Persistent churn incurs substantial cost: solved queries flips to unsolved lead to pp forgetting gap, and of final accuracy comes from unreliable answer on queries that flips repeatedly across training checkpoints. Finally, we find that churn is not benign: applying sharpness-aware minimization after accuracy plateaus reduces churn while lifting final accuracy by percentage points for models up to parameters, with no benefits on larger models. Our work highlights the importance of analyzing the evolution of query-level performance during post-training and points to a promising path for improving learning efficiency via reduced churn.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Churning, edge of understanding, LLM post-training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YSKhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies changes in individual questions' correctness during LLM post-training, distinguishing positive flips from negative flips that aggregate accuracy can conceal. It reports results across eight models, four post-training methods, and multiple reasoning benchmarks, relating repeated flips to a response-confidence margin and sensitivity to parameter perturbations

Critical strengths
  1. A useful question with broad empirical coverage.

  2. The research is interesting

  3. A promising intervention result with an explicit limitation.

Critical weaknesses
  1. The headline single-update churn result is not clearly supported by the measurement description. The paper repeatedly claims that one gradient update flips approximately 21% of learnable queries, and a footnote describes one-step probes, but Tables 5–8 and Figure 9 describe measurements between checkpoints separated by 25 updates. These are different quantities, so the manuscript needs to establish which measurement supports its central claim. Please report one-step and 25-step results separately, explain the probe procedure, and show rates over all evaluated queries as well as the retrospectively defined learnable set.

  2. The confidence analysis does not establish the claimed decision-boundary mechanism. The proposed margin compares length-normalized probabilities of complete responses, whereas greedy decoding chooses tokens sequentially and need not select the highest-scoring complete response. Its estimate also relies on eight sampled alternatives and averages signed margins, so a near-zero average does not establish that individual predictions are close to a correctness boundary or correct only by chance. The analysis needs a decoding-relevant margin, coverage of valid correct/incorrect candidate pairs, and absolute-margin distributions before drawing the stronger causal conclusions.

  3. The forgetting gap should not be interpreted as directly recoverable lost knowledge. The union of questions solved across twenty checkpoints is a retrospective oracle: different questions may be solved by different checkpoints, and occasional successes can accumulate even when average competence does not improve. Consequently, the 25.8-point union-to-final gap does not demonstrate that a single model previously possessed all those capabilities simultaneously or that reducing churn can recover the entire gap. Sensitivity to checkpoint count and spacing, together with measures of sustained correctness, would support a more precise interpretation of forgetting and final-answer reliability.

  4. Several reported statistics require clarification or correction Table 19 reports 0.0 ± 2.1% accuracy for one cell, which is incompatible with the stated mean and standard deviation over three nonnegative accuracy measurements, even allowing for rounding. Also, the per-dataset best-checkpoint rows in Table 3 average approximately 9.63 and 10.42 points above the final checkpoint for SFT and SDFT, respectively, whereas the text says early stopping reduces the gap by only 1–5 points. The authors should reconcile the selection and aggregation procedures, audit the per-seed summaries, and account for dependence between checkpoints when reporting uncertainty.

Questions for authors
  1. Which figures and tables report genuine one-update probes, and which use 25-update checkpoint differences? How are probe examples, optimizer state, and the learnable-set denominator defined?

  2. Does the margin analysis remain convincing when using absolute per-checkpoint margins, more candidate responses, and a measure aligned with greedy decoding? How are missing opposite-correctness candidates handled?

  3. How does the forgetting gap change with checkpoint density, and does historical churn predict future failure among currently correct answers under a matched evaluation?

  4. Can you provide the exact SAM/baseline configurations, compute-matched controls, and corrected run-level statistics supporting the reported effects?

If the author can solve these questions, I will increase my score.

Reviewer_ssQXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies query-level instability in LLM post-training. It tracks the greedy correctness of every evaluation query across 20 checkpoints (500 updates, three seeds) of four methods: RLVR (Dr.GRPO), SFT, self-distillation fine-tuning (SDFT) and on-policy distillation (OPD). The models are eight open models up to 9B (Qwen3, Qwen3.5, Gemma4), and evaluation covers one in-distribution set and nine OOD math, science and code benchmarks, using deterministic, batch-invariant greedy decoding. Churn is the fraction of learnable queries (solved by at least one checkpoint) that flip between consecutive checkpoints, split into positive (c+c^+) and negative (c−c^-) parts. Claimed contributions: Phenomenon. Churn averages 20.9% (18.0–24.1% across methods) on both ID and OOD data, with c+≈c−c^+ \approx c^-. In a closed-book QA SFT setting where accuracy rises and then plateaus, net churn tracks the accuracy change, and churn stays at about 13% per epoch after the plateau. Cause. Queries are split into unforgettable, unsolvable and churning groups. Churning queries have a near-zero margin g2g_2 between the best correct and best incorrect response. Under Gaussian weight noise (ρ=0.001\rho = 0.001) they flip 12.5× more often than unforgettable queries. Consequence. The forgetting gap (learnable fraction minus final accuracy) averages 25.8 pp, and early stopping or majority voting over checkpoints recovers little of it. 28.5% of final-correct queries went from correct to incorrect at least three times during training. Intervention. On closed-book QA SFT, weight decay, replay, SDFT, PCGrad, and learning-rate and batch-size changes leave churn and accuracy essentially unchanged. SAM lowers churn by about 30% for models up to 4B. Enabled after the plateau, it raises final accuracy by about 5 pp; enabled from the start, it costs 1–2 pp. There is no gain at 9B.

Critical strengths

S1. Broad, practically relevant query-level measurement with an important determinism control. The study tracks correctness trajectories across multiple post-training methods, model families and reasoning benchmarks rather than relying only on aggregate accuracy. Batch-invariant greedy evaluation is an important control against batch-related inference nondeterminism. Substantial turnover persists across the studied settings, including after the initial transient reported for SFT/SDFT. These observations are relevant to regression testing and checkpoint evaluation, even if their mechanistic interpretation requires further evidence.

S2. Churn is predictable, not just descriptive. Groups defined on the first half of training predict churn in the second half (28–31% vs. 2–7%, App. B.3), which is a non-circular check. Low-margin, perturbation-sensitive queries are the ones that churn. This points to concrete uses, such as flagging unstable items in evaluation sets and stability-aware checkpoint selection. The schedule-dependent SAM result (Sec. 3.5) is a first hint that late-stage instability can be controlled.

Critical weaknesses

W1. Churn is not separated from readout noise, so the "forgetting" framing is overclaimed. Correctness comes from a single greedy decode of a long chain of thought, which is a discontinuous function of the weights. For a query with intermediate success probability, a small perturbation can flip the outcome. The union over 20 checkpoints then acts like pass@20, so Union − Final would be large even if no query's success probability changed. The paper's own results fit this reading: Random noise with ρ=0.001\rho = 0.001–0.0030.003 flips about 12–37% of churning queries (Fig. 7), the same order as their 28–31% per-interval churn (App. B.3). A 10× change in learning rate moves churn by less than 1% relative (Tables 11, 13–16). No checkpoint comes close to the union. Early stopping recovers only 1–5 of the ~26 pp, and majority voting does not help (Table 3), so it is unclear how the gap could be "harvested" (Sec. 4). Without a null baseline (Q5), claims such as "the governing dynamic behind post-training" are unsupported, since Δacc=∣L∣∣Q∣(c+−c−)\Delta\text{acc} = \frac{|L|}{|Q|}(c^+ - c^-) holds by definition. The margin result is also partly circular. g2g_2 is averaged with its sign across checkpoints, and the sign largely follows the greedy outcome. It is therefore near zero by construction for the group defined by alternating outcomes (Table 2).

Questions for authors
  1. Is the headline churn measured over one gradient step, 25-update intervals, or both? Please report the two separately if both were measured and clarify the construction of the learnable-set denominator.
  2. Given the different OOD subsets in Appendix B.5, can you first reconcile the base scores on matched subsets?
Reviewer_BVLPOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies changes in problem-level correctness during LLM post-training, showing that individual queries can repeatedly switch between correct and incorrect even when accuracy remains relatively stable or increasing. The authors characterize this phenomenon as churn and analyze it across multiple model families, model sizes, datasets, and post-training methods. To characterize these dynamics, the authors propose or adopt complementary metrics, examine how they relate to confidence margins and parameter perturbations, and show that applying sharpness-aware minimization (SAM) once accuracy has plateaued can lower churn and boost accuracy in smaller models.

Critical strengths
  1. Broad empirical coverage. The strongest aspect of the paper is its extensive evaluation across multiple model families and sizes, diverse datasets, and several post-training methods. This breadth helps establish that the observed behavior is not confined to one particular model or task and provides a useful empirical characterization of churn in LLM post-training.

  2. Complementary metrics for query-level behavior. The use of several complementary metrics, including positive and negative churn, the forgetting gap, and retention, makes it easier to examine changes that aggregate accuracy can obscure. These metrics provide a useful vocabulary for describing how individual predictions evolve throughout training, even where the underlying concepts build on prior work.

Critical weaknesses
  1. The central contribution beyond documenting a known phenomenon is not sufficiently developed.

As the authors acknowledge, correctness flips during training have already been observed in prior work. The broader experimental coverage is valuable, but I found the paper’s main advance beyond establishing the prevalence of this behavior less clear. A descriptive study does not necessarily need to propose a new solution; however, given the existing evidence for the phenomenon, a stronger mechanistic insight or a more developed account of how the findings guide training would make the contribution more compelling.

The SAM intervention is particularly underdeveloped in this respect. Although the paper reports ablation results, the discussion provides little follow-up analysis of why the SAM works. For example, why does applying SAM from the beginning hurt accuracy, whereas enabling it after the accuracy plateau helps? The contrast between these schedules is interesting, but reporting the contrast does not explain it. A clearer connection between the preceding analyses and this timing-dependent effect would substantially strengthen the paper.

  1. The diagnostic experiments do not sufficiently separate changes induced by learning from the underlying variability of model responses.

The confidence-margin and parameter-perturbation experiments offer plausible evidence of sensitivity, but they are conducted as separate analyses, making it difficult to determine how the different sources of variability relate to the observed churn. Deterministic greedy decoding rules out sampling randomness as the direct cause of the measured flips. Nevertheless, it leaves an important question unanswered: how large are the changes induced by training relative to the variability obtained by sampling repeatedly from a fixed checkpoint?

Experiments at nonzero decoding temperatures, with matched sampling budgets and metrics such as pass@k or avg@k, would help distinguish changes in the probability of producing a correct answer from changes in which single response greedy decoding selects.

The confidence-margin argument also needs more granular evidence. The reported group-level summaries do not reveal whether individual churning queries consistently have small absolute margins. A mean signed margin near zero could instead result from positive and negative margins canceling across queries or checkpoints. Per-query distributions are therefore important for evaluating the proposed explanation.

Questions for authors
  1. How large is training-induced churn relative to sampling variability? How does the variation obtained by repeatedly sampling from a fixed checkpoint compare with the changes observed across training checkpoints, particularly for queries with small confidence margins? A comparison using matched temperatures and sampling budgets, including pass@k and avg@k, would help clarify whether the observed greedy churn corresponds to substantial changes in the probability of generating a correct answer.

  2. Could the authors provide the distribution of individual g2 values? While the confidence-margin explanation is plausible, positive and negative values can cancel when averaged. Distributions of per-query signed and absolute margins, separated by query group and evaluated at individual checkpoints, would help establish whether churning queries are actually concentrated near zero.

  3. Where do the benefits of SAM come from, and how do they depend on training duration? Given that training lasts 12 epochs while accuracy appears to plateau earlier, could the authors compare the checkpoint-wise performance of plain SFT and SAM, including their best intermediate checkpoints? This would help determine whether SAM improves upon the best SFT performance or primarily mitigates deterioration during prolonged training. Using the evaluation protocol in Question 1, how large are SAM’s effects on accuracy and churn relative to fixed-checkpoint sampling variability, and do its benefits persist under stochastic decoding?

  4. Why was SAM evaluated only on closed-book QA tasks? Is there a specific methodological or practical reason for excluding it from the main reasoning experiments? Clarifying this choice would help assess how broadly the conclusions drawn from the QA intervention apply to the churn observed in the main experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.