- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies changes in individual questions' correctness during LLM post-training, distinguishing positive flips from negative flips that aggregate accuracy can conceal. It reports results across eight models, four post-training methods, and multiple reasoning benchmarks, relating repeated flips to a response-confidence margin and sensitivity to parameter perturbations
Critical strengths
-
A useful question with broad empirical coverage.
-
The research is interesting
-
A promising intervention result with an explicit limitation.
Critical weaknesses
-
The headline single-update churn result is not clearly supported by the measurement description. The paper repeatedly claims that one gradient update flips approximately 21% of learnable queries, and a footnote describes one-step probes, but Tables 5–8 and Figure 9 describe measurements between checkpoints separated by 25 updates. These are different quantities, so the manuscript needs to establish which measurement supports its central claim. Please report one-step and 25-step results separately, explain the probe procedure, and show rates over all evaluated queries as well as the retrospectively defined learnable set.
-
The confidence analysis does not establish the claimed decision-boundary mechanism. The proposed margin compares length-normalized probabilities of complete responses, whereas greedy decoding chooses tokens sequentially and need not select the highest-scoring complete response. Its estimate also relies on eight sampled alternatives and averages signed margins, so a near-zero average does not establish that individual predictions are close to a correctness boundary or correct only by chance. The analysis needs a decoding-relevant margin, coverage of valid correct/incorrect candidate pairs, and absolute-margin distributions before drawing the stronger causal conclusions.
-
The forgetting gap should not be interpreted as directly recoverable lost knowledge. The union of questions solved across twenty checkpoints is a retrospective oracle: different questions may be solved by different checkpoints, and occasional successes can accumulate even when average competence does not improve. Consequently, the 25.8-point union-to-final gap does not demonstrate that a single model previously possessed all those capabilities simultaneously or that reducing churn can recover the entire gap. Sensitivity to checkpoint count and spacing, together with measures of sustained correctness, would support a more precise interpretation of forgetting and final-answer reliability.
-
Several reported statistics require clarification or correction Table 19 reports 0.0 ± 2.1% accuracy for one cell, which is incompatible with the stated mean and standard deviation over three nonnegative accuracy measurements, even allowing for rounding. Also, the per-dataset best-checkpoint rows in Table 3 average approximately 9.63 and 10.42 points above the final checkpoint for SFT and SDFT, respectively, whereas the text says early stopping reduces the gap by only 1–5 points. The authors should reconcile the selection and aggregation procedures, audit the per-seed summaries, and account for dependence between checkpoints when reporting uncertainty.
Questions for authors
-
Which figures and tables report genuine one-update probes, and which use 25-update checkpoint differences? How are probe examples, optimizer state, and the learnable-set denominator defined?
-
Does the margin analysis remain convincing when using absolute per-checkpoint margins, more candidate responses, and a measure aligned with greedy decoding? How are missing opposite-correctness candidates handled?
-
How does the forgetting gap change with checkpoint density, and does historical churn predict future failure among currently correct answers under a matched evaluation?
-
Can you provide the exact SAM/baseline configurations, compute-matched controls, and corrected run-level statistics supporting the reported effects?
If the author can solve these questions, I will increase my score.