acceptodds
Under review as a conference paper at ICLR 2027

Learning Dynamics of LLM Calibration under Supervised Fine-Tuning

Abstract

Calibrated confidence helps assess the reliability of an LLM’s response. In the literature, LLM fine-tuning is often associated with poor calibration, but previous analyses only compared the calibration performance of pre-trained and post-trained models, leaving the training dynamics largely unexplored. In this work, we study the calibration dynamics of LLMs during standard supervised fine-tuning (SFT), and find that the calibration of an overconfident LLM is improved first and then worsens during training. The underlying reason is that hard training questions that LLMs generate incorrect responses can improve calibration, but their influence weakens as their proportion decreases during training. Based on this insight, we propose State-Gated Corrective Replay (SGCR), a novel method that amplifies the impact of those hard training questions through oversampling. In effect, SGCR can reduce calibration error and improve calibration stability during training, thereby extending the iteration window with excellent calibration. Extensive experiments on MMLU and MedMCQA validate the effectiveness of our method, reducing time-averaged ECE by 36.0–47.6% relative to standard SFT across the evaluated models.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Confidence Calibration, Large Language Models, Supervised Fine-Tuning, Learning Dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DbgzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies confidence calibration during supervised fine-tuning on multiple-choice questions. For Qwen Instruct, which starts overconfident, ECE, Brier score, and NLL first improve, then worsen. The authors link this pattern to gradients from training questions answered correctly or incorrectly, and to logit scale. They run one-step group updates from the same model and optimizer state. Based on these tests, they propose SGCR, which dynamically resamples hard questions the model currently gets wrong. SGCR lowers average ECE along the training path and extends the period of low ECE.

Critical strengths
  1. The authors track the full training run and test one-step updates from the same model parameters and AdamW state, changing only the next batch. This shows more directly than training curves that currently wrong and correct examples affect next-step calibration differently.
  2. Logit-scale controls separate calibration changes from accuracy changes. SGCR lowers ECE despite lower accuracy on Qwen-MMLU and nearly equal accuracy on Qwen-MedMCQA.
Critical weaknesses
  1. The model starts overconfident, so early fine-tuning may improve ECE before larger logits make it worse; NLL also gives stronger corrections to wrong answers. The one-step tests support this pattern, but they do not show that SGCR does more than change confidence or train more often on hard questions.
  2. On Qwen-MMLU, the SGCR checkpoint with the lowest validation ECE cuts test ECE from 3.35% to 1.48%, but test accuracy falls from 70.33% to 68.50%, and Brier score and NLL worsen. At the same final training step, ECE improves only from 11.36% to 10.72% while the other metrics worsen again, so the paper should explain when this tradeoff is useful.
  3. SGCR repeatedly checks which training questions the model gets wrong, so it uses extra forward passes. The paper compares methods after the same number of training steps, not the same computing cost, and does not show whether SGCR beats standard SFT, early stopping, temperature scaling, or simple hard-question resampling at equal cost.
Questions for authors
  1. Could you compare SGCR, standard SFT, early stopping, and post-training temperature scaling across several seeds and equal compute budgets? Please report ECE, accuracy, Brier score, and NLL together.
  2. Does SGCR still beat these baselines at checkpoints with matched accuracy? Could you report ECE changes on examples whose predictions stay fixed, to separate confidence changes from accuracy changes?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Dbgz

Thank you for pointing out the missing comparisons. We added paired multi-seed runs, proper scoring rules (Brier/NLL), equal-budget evaluations, and fixed-prediction analyses.

1. Q1: Shared selection rules and all four metrics (Raw and Post-TS) SGCR's update rule uses no held-out data, whereas checkpoint selection and post-hoc Temperature Scaling (TS) use validation data equally. Across 3 paired seeds over [0, 429], selecting each run's highest-validation-accuracy checkpoint gives:

MethodAcc (%) ↑ECE (%) ↓Brier ↓NLL ↓
Vanilla69.65 ± 0.446.74 ± 3.310.4088 ± 0.01280.7840 ± 0.0309
SGCR69.58 ± 0.762.84 ± 1.680.4020 ± 0.01450.7683 ± 0.0404
Vanilla + TS69.65 ± 0.442.12 ± 1.140.3994 ± 0.00490.7569 ± 0.0103
SGCR + TS69.58 ± 0.761.55 ± 0.410.4004 ± 0.01310.7587 ± 0.0262

With the same accuracy-based selector, SGCR improves raw ECE, Brier, and NLL simultaneously without losing accuracy. After TS, SGCR + TS reaches the lowest ECE (1.55%) with nearly identical Brier/NLL.

2. Q1 (Continued): Equal-budget comparisons and screening cost Across 1,170 updates, both methods run 299,520 gradient examples, while SGCR adds 1,198,104 refresh and 1,284,236 gate-screening forwards (48.22 vs. 12.38 GPU-hrs). Enforcing a strict cap of 299,520 total algorithmic forward passes (counting all training, refresh, and gate-check passes against SGCR; step 195 vs. 1,170) gives:

Method (≤ 299,520 Forward Passes)Acc (%) ↑ECE (%) ↓Brier ↓NLL ↓
Vanilla69.65 ± 0.446.74 ± 3.310.4088 ± 0.01280.7840 ± 0.0309
SGCR (Capped)69.81 ± 0.512.96 ± 1.850.3987 ± 0.00890.7556 ± 0.0190
Vanilla + TS69.65 ± 0.442.12 ± 1.140.3994 ± 0.00490.7569 ± 0.0103
SGCR + TS (Capped)69.81 ± 0.511.46 ± 0.260.3970 ± 0.00750.7524 ± 0.0160

Even when penalized for every auxiliary forward pass (and under 3:1 FLOP-equivalent and time budgets), SGCR outperforms Vanilla across all four metrics within the shared budget.

3. Q2: Confidence changes with predictions held fixed On the test subset where the selected Vanilla and SGCR checkpoints predict the exact same option (84.93 ± 5.40% of test items; identical accuracy of 75.56 ± 1.85% by construction):

Metric on Common-Prediction SubsetVanillaSGCR
ECE (%) ↓5.13 ± 2.592.58 ± 0.56
Brier ↓0.3531 ± 0.02080.3477 ± 0.0138
NLL ↓0.6919 ± 0.03660.6787 ± 0.0225
Mean Confidence on Shared Wrong Answers (%) ↓64.95 ± 3.9556.03 ± 5.00

On the exact same answers and mistakes, SGCR cuts overconfidence on shared errors by 8.92 pp and improves ECE, Brier, and NLL.

4. Dynamic verification vs. 6 controls & 1,170-step error consumption

  • Why current state matters: On Seed 1 ([0, 429]), average test ECE is 6.05% for SGCR, beating High-loss (7.97%), No-recheck (8.02%), Static-Wrong (9.82%), and Random-pool (11.02%) because 0% of SGCR replay slots have flipped to Correct before the gradient step (vs. 43.34%–91.61% in controls). Still, Static-Wrong, High-loss, Label Smoothing, and Wrong-loss reweighting offer useful low-cost checkpoints.
  • 1,170-step error consumption: Learning consumes the SGCR Wrong pool (14.87% -> 2.17% -> 0.083%, where 20/20 top-replayed items are conflicting-label duplicates). Once real errors are exhausted, forcing replay trains on label noise and raises final ECE (26.60% vs. 15.08%; full average 17.71% vs. 12.47%).

Thanks the authors for the clarifications. However, I remain concerned about the effectiveness and validity of SGCR if sacrificing the accuracy and achieving similar calibration with logit-based post-hoc methods such as temperature scaling. Therefore, I'm leaning towards weak rejection.

Reviewer_MvVyOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies confidence calibration during supervised fine-tuning and finds that calibration can initially improve and subsequently deteriorate for overconfident models. Through controlled interventions, the authors investigate the contributions of currently correct and incorrect training examples and propose State-Gated Corrective Replay (SGCR) to sustain the influence of corrective updates. The reported reductions in calibration error are substantial. However, the main experiments define confidence through probabilities over multiple-choice answer tokens. Consequently, the findings primarily concern classification-style confidence, and their relevance to confidence in multi-step reasoning and open-ended generation remains unestablished. This limitation substantially narrows the paper’s broader implications for LLM reliability.

Critical strengths

Informative analysis of training dynamics. Tracking calibration throughout training reveals behavior that comparisons between initial and final checkpoints would miss. The distinction between early improvement and subsequent deterioration provides a useful empirical observation.

Controlled interventions support the analysis. Restoring the same model parameters and optimizer state while varying batch composition provides stronger evidence than observational trajectories alone. The logit-scale experiments further help distinguish changes in prediction confidence from changes in answer accuracy.

A simple intervention with transparent reporting. SGCR preserves the standard training objective and modifies sampling. The paper also reports complementary metrics and acknowledges accuracy trade-offs, incomplete trajectories, and limitations of the experimental evidence.

Critical weaknesses

The central notion of confidence is restricted to answer classification. The main experiments measure confidence as the maximum probability among predefined answer tokens, typically A–D, under an answer-only interface. This does not directly characterize uncertainty in generated reasoning, intermediate inference steps, or complete open-ended responses. Although selecting an answer may involve internal reasoning, the experiments do not isolate or evaluate that reasoning. The most important unanswered question is whether the reported dynamics and SGCR’s benefits persist when reasoning is explicitly elicited and confidence concerns the correctness of the resulting solution.

The contribution is insufficiently connected to contemporary questions about reasoning-model reliability. The paper provides a careful analysis within its chosen setting, but its broader positioning as a study of LLM calibration is not fully supported by the experimental scope. It remains unclear what the findings establish beyond calibration dynamics in classification-style fine-tuning. The full-response training experiments do not resolve this concern because evaluation still centers on multiple-choice answer probabilities. A stronger contribution would demonstrate that the proposed explanation yields actionable insights for uncertainty estimation in reasoning-oriented models.

Lower calibration error does not consistently translate into a better-performing model. In the Qwen–MMLU comparison, checkpoints selected by minimum validation ECE yield test accuracy of 70.33% for standard SFT and 68.50% for SGCR, with SGCR also having worse Brier score and NLL despite lower ECE. Thus, extending a low-ECE window is not sufficient to establish improved overall reliability. Evaluation under an explicit accuracy constraint or a downstream confidence-based decision rule would better demonstrate practical value.

The distinctive benefit and robustness of SGCR require stronger validation. The end-to-end SGCR comparisons use a single seed per configuration, and the main MMLU comparison covers only part of the planned training horizon. Comparisons against fixed-error replay, high-loss sampling, and replay without correctness rechecking would help establish the value of state gating. Additional screening passes also introduce computational overhead that is not captured by matching optimizer steps.

Questions for authors

see weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer MvVy

Thank you for recognizing our controlled interventions and transparent reporting.

1. Why the fixed-candidate (MCQ) setup is standard, connected to open-ended QA, and practically valuable Following established LLM calibration studies (Luo et al., NeurIPS 2025; Kadavath et al., 2022; Zhu et al., 2023), we focus on fixed-candidate answer confidence for three reasons:

  • Standard paradigm & direct bridge to Open-Ended QA (P(True)): In open-ended generation, multi-token likelihoods are confounded by length bias, paraphrase splitting, and decoding variance. Thus, prior calibration studies build their core theory and main experiments on fixed-choice option logits, and handle open-ended QA by reducing answer verification to a fixed-candidate binary readout (P(True), scoring True vs. False option tokens). Understanding how SFT shapes fixed-candidate confidence is therefore foundational to both MCQA and P(True) open-ended QA.
  • Beyond single-token supervision (Appendix M): While fixed-candidate readouts provide an exact evaluation distribution, our mechanism is not an artifact of single-letter loss masks: Appendix M shows that supervising broader target responses (non-only-option SFT) preserves the same Wrong vs. Correct calibration dynamics.
  • Direct deployment value: Fixed-option probabilities are a primary operational interface in fast single-pass decision systems—such as Jev (TypeSafe AI, 2026) and open Jev-style decision models (Geuter et al., 2026) that output option probabilities directly and require both high accuracy and calibration.

2. Calibration gains do not require sacrificing accuracy Across 3 paired seeds over [0, 429], we applied the exact same highest-validation-accuracy rule to both methods, and also tested the Same-Answer Subset (84.93 ± 5.40% of test items) where both models predict the exact same option:

Setting (3 Seeds, Mean ± SD)MethodAcc (%) ↑Raw ECE (%) ↓Raw Brier ↓Raw NLL ↓
Full Test (Max Val Acc)Vanilla69.65 ± 0.446.74 ± 3.310.4088 ± 0.01280.7840 ± 0.0309
Full Test (Max Val Acc)SGCR69.58 ± 0.762.84 ± 1.680.4020 ± 0.01450.7683 ± 0.0404
Same-Answer (84.93% shared)Vanilla75.56 ± 1.855.13 ± 2.590.3531 ± 0.02080.6919 ± 0.0366
Same-Answer (Identical acc)SGCR75.56 ± 1.852.58 ± 0.560.3477 ± 0.01380.6787 ± 0.0225
  • No accuracy loss: SGCR cuts raw ECE (6.74% -> 2.84%) and improves Brier/NLL at matched accuracy (−0.07 pp). On the 84.93% same-answer subset (100% identical accuracy), SGCR still cuts ECE (5.13% -> 2.58%).
  • Downstream & post-TS metrics: Raw Risk@80% is matched (22.12% Vanilla vs. 22.16% SGCR). With validation-fitted TS, SGCR+TS gets the lowest ECE (1.55% vs. 2.12%), while post-TS Brier (0.3994 vs. 0.4004), NLL (0.7569 vs. 0.7587), and AURC slightly favor Vanilla+TS.

3. Multi-seed robustness, 6 controls, and 1,170-step error consumption

  • Early regime & 6 controls: Across 3 seeds ([0, 429]), SGCR lowers average test ECE (10.01 ± 0.89% -> 5.73 ± 0.45%) and extends the sub-5% ECE window (6.7 ± 11.5 -> 195.0 ± 140.6 steps). Across 6 controls on Seed 1, SGCR gets the lowest average ECE (6.05% vs. 7.97% High-loss, 8.02% No-recheck, 9.82% Static-Wrong, 11.02% Random-pool), though simpler reweighting and static/high-loss replay also yield good individual checkpoints without screening cost.
  • 1,170-step error consumption: Extending runs to 1,170 steps shrinks the Wrong pool by 99.4% (14.87% -> 0.083%, where 20/20 top-replayed items are conflicting-label duplicates). Forcing replay once real errors are gone overfits to label noise and raises step-1,170 ECE (26.60% vs. 15.08%; [0, 1170] max-val-acc raw ECE 10.00% vs. 6.74%), confirming that calibration stability relies on real current errors.

4. Computational overhead and equal-budget comparisons Full 1,170-step cost is 48.22 vs. 12.38 GPU-hrs. Under a shared cap of 299,520 forward passes (charging all refresh and screening passes against SGCR), highest-validation-accuracy selection gives SGCR 69.81% acc / 2.96% raw ECE (1.46% post-TS) vs. Vanilla's 69.65% acc / 6.74% raw ECE (2.12% post-TS).

i understand the paper as a idea of balance the mixing of data pattern to imporve the ability of calibration. I agree with the result and the idea. but i wander if the novelty and the importance match with iclr. i remain my negative attitude. but i will give score of 2 to appriecate the solid work and detailed rebuttal.

Reviewer_pj8mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how calibration changes during answer-token SFT on multiple-choice QA, mainly with Qwen3-4B on MMLU. When the initial model is overconfident (Instruct, ECE 28.7%), ECE drops to 4.4% by step 39 and rises to about 22% by the end of training. A calibrated Base model gets steadily worse (3.5% to 20.3%). Using one-step updates from restored model and optimizer states, the authors show that updates from currently wrong examples reduce held-out ECE and updates from correct examples increase it. The early gain fades as wrong examples drop from about 12% to below 1% of the training set, and logit-norm growth drives the late confidence increase. Based on this, they propose SGCR, which replays currently wrong examples in each batch. In single-seed runs over 429 updates, SGCR reduces time-averaged ECE by 36.0–47.6% and lengthens the period of low ECE.

Critical strengths
  1. The analysis is carefully controlled and honestly scoped. The restored-state one-step interventions isolate the effect of each training group. The 2×2 experiment separating output scale from loss-effective scale is clean, and the norm/direction ECE decomposition is exact. Limitations are stated openly throughout.

  2. The empirical message is useful. Comparing only the starting and final checkpoints can miss a large early calibration improvement, and whether SFT helps or hurts depends on the initial logit scale. This helps explain why prior work disagrees on whether fine-tuning damages calibration.

Critical weaknesses
  1. The mechanism adds little beyond known effects of NLL training. The early gain is mostly a logit-scale correction: temperature scaling alone lowers the untrained model’s ECE to 6.4%, and scale accounts for 90.4% of the ECE change by step 390 (Table 10). The late degradation follows training accuracy above 99% against about 69% test accuracy, and it is driven by norm growth. This is the NLL-overfitting story of Guo et al. (2017) and Mukhoti et al. (2020). The account also fails to predict the trajectory: it places the ECE minimum at about step 37, while the observed minima are at steps 98–293 (Appendix N).

  2. The evidence for SGCR is weak, and key baselines are missing. The main SGCR run stops at 449 of 1,170 updates. At step 429, SGCR’s ECE is nearly unchanged (10.72% vs. 11.36%) and its accuracy is lower. SGCR is not compared with early stopping on validation ECE, although standard SFT already reaches 3.35% ECE at 70.33% accuracy at step 78. Temperature scaling, label smoothing, calibration-aware fine-tuning and random replay are also absent, and compute is not matched. The analysis is the stronger contribution, and the paper would be better without SGCR or with SGCR presented only as a probe.

Questions for authors
  1. Under the same checkpoint-selection rule (e.g., highest validation accuracy subject to validation ECE ≤ ε), how does SGCR compare with Vanilla plus early stopping and temperature scaling? Please report ECE after temperature scaling for both.

  2. Does the gain require the correctness recheck? Results for same-quota random replay, static oversampling of the initial Wrong set, or loss reweighting of Wrong examples would clarify this.

  3. Training examples that remain wrong late in training may be enriched for mislabeled or ambiguous items. Did you check the label quality of replayed examples, and could replay be acting as label-noise regularization?

  4. Can you complete the SGCR runs to 1,170 steps and report total compute?

  5. Do the findings hold beyond fixed-candidate multiple choice? In open-ended QA, training on unknown facts is known to increase hallucination (Gekhman et al., 2024). Would Wrong-only updates still improve calibration there?

2 comments
AuthorsOpenReview ↗

Response to Reviewer pj8m

Thank you for recognizing our state-restoration interventions, scale controls, and decomposition.

1. Mechanism beyond NLL overconfidence and logit scale A training example's NLL residual does not determine the sign of the confidence shift on unseen held-out questions after a shared-parameter update, which depends on Jacobian coupling and AdamW state (Prop. 1):

  • Held-out transfer from identical states (Table 1): Resetting weights and AdamW states before a single step across 4 model–task pairs shows Wrong-only updates cut held-out ECE by 1.01–3.79 pp, while Correct-only updates raise it by 2.77–7.40 pp. Gradient diagnostics confirm wrong-example gradients stay favorably aligned even as their natural frequency drops.
  • Learning vs. evaluation scale (Table 2): Fixing evaluation scale while varying only learning loss scale yields 4.9% vs. 26.8% held-out ECE at step 390 (~70% acc), showing learning scale shapes the trajectory itself. Also, our gradient-balance ratio is a local directional diagnostic, not a global stopping predictor.

2. Q1: Shared selection, early stopping, and Temperature Scaling (TS) SGCR's update rule is validation-free, while checkpoint selection and post-hoc TS use validation data equally. Across 3 paired seeds ([0, 429]), selecting each run's highest-validation-accuracy checkpoint gives:

MethodAcc (%) ↑ECE (%) ↓Brier ↓NLL ↓
Vanilla69.65 ± 0.446.74 ± 3.310.4088 ± 0.01280.7840 ± 0.0309
SGCR69.58 ± 0.762.84 ± 1.680.4020 ± 0.01450.7683 ± 0.0404
Vanilla + TS69.65 ± 0.442.12 ± 1.140.3994 ± 0.00490.7569 ± 0.0103
SGCR + TS69.58 ± 0.761.55 ± 0.410.4004 ± 0.01310.7587 ± 0.0262

SGCR cuts raw ECE (6.74% -> 2.84%) and improves Brier/NLL at matched accuracy; SGCR+TS gets the lowest ECE (1.55%) with similar Brier/NLL. For max val accuracy with raw val ECE ≤ 5%, SGCR succeeds in 3/3 seeds vs. 2/3 for Vanilla.

3. Q2: Dynamic rechecking vs. 6 simpler controls On Seed 1 ([0, 429]), average test ECE is 6.05% for SGCR, beating High-loss (7.97%), No-recheck (8.02%), Static-Wrong (9.82%), and Random-pool (11.02%). At the gradient pass, replay slots already flipped to Correct are 0% for SGCR, vs. 43.34% (High-loss), 43.89% (No-recheck), 64.23% (Static-Wrong), and 91.61% (Random-pool). Still, Static-Wrong, High-loss, No-recheck, and Wrong-loss reweighting yield strong checkpoints without screening cost.

4. Q3 & Q4: 1,170-step error consumption, label audit, and compute budget

  • Error consumption: Over 1,170 steps, learning consumes the Wrong pool from 14.87% (step 0) -> 2.17% (step 390) -> 0.083% (step 780). Early average ECE drops (10.01 ± 0.89% -> 5.73 ± 0.45%), but forcing replay at 0.083% raises step-1,170 ECE (26.60 ± 0.56% vs. 15.08 ± 0.78%; full average 17.71% vs. 12.47%).
  • 110-item audit: Across 40 initial wrong, 50 random late, and 20 top-replayed late items, 31/40 and 34/50 random items are real errors, whereas 20/20 top-replayed late items are near-duplicates with conflicting labels.
  • Equal-budget win: Full cost is 48.22 ± 0.39 vs. 12.38 ± 1.93 GPU-hrs. Under a 299,520 forward-pass cap (counting all training, refresh, and screening passes; step 195 vs. 1,170), SGCR gets 69.81% acc / 2.96% raw ECE (1.46% post-TS) vs. Vanilla's 69.65% acc / 6.74% raw ECE (2.12% post-TS).

5. Q5: Why we use the MCQ setup and how it connects to open-ended QA Following standard LLM calibration studies (Luo et al., NeurIPS 2025; Kadavath et al., 2022; Zhu et al., 2023), we use the fixed-candidate MCQ setup for three reasons:

  • Avoiding generative confounders & bridging to Open-Ended QA (P(True)): Free-form sequence likelihoods are confounded by length bias and paraphrase splitting. Thus, prior works build their core theory and MMLU experiments on MCQ option logits, and evaluate open-ended QA by converting generated answers into a fixed-candidate binary readout (P(True), scoring True vs. False option logits). Fixed-candidate calibration is thus the direct foundation for both MCQA and P(True) open-ended calibration.
  • Beyond single-token training (Appendix M): Appendix M shows that supervising full target responses beyond the option letter preserves the same Wrong vs. Correct dynamics.
  • Application value: Single-pass option probabilities directly power fast decision systems like Jev (TypeSafe AI, 2026) and open Jev-style models (Geuter et al., 2026).

Thank you for the additional experiments and for reporting the unfavourable results candidly. My two main concerns remain. On the mechanism, the response does not address the 90.4% scale share in Table 10, and Tables 1–2 are both what a pure logit-scale account would predict. Scale-matched versions (per-arm optimal temperature) would be needed to show a residual effect. On SGCR, the new results weaken the case. After temperature scaling, the gap to Vanilla is within seed noise (1.55 ± 0.41 vs. 2.12 ± 1.14, n=3) with Brier and NLL tied. The ECE-constrained selection I asked for, label smoothing, and any open-ended evaluation are still missing. I continue to think the analysis is the contribution and that SGCR is best presented as a probe, so I am keeping my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of 12 New Runs and Clarified Scope

We thank all reviewers for their constructive feedback. To address all questions on checkpoint selection, proper scoring rules, simpler baselines, compute cost, and long training horizons directly with data, we ran 12 additional Qwen3-4B–MMLU experiments to 1,170 steps (3 paired Vanilla/SGCR seeds and 6 sampling/loss controls), alongside matched-prediction checks, equal-budget comparisons, and a 110-example label audit:

  1. Mechanism Beyond NLL Overconfidence: Standard NLL is known to push confidence up, but a training example's own residual does not determine how a shared-parameter update shifts confidence on unseen held-out questions. Our restored-state interventions (Table 1) and scale controls (Table 2) show that held-out calibration depends on the learning-time logit scale and the balance between currently Wrong and currently Correct training examples.
  2. No Accuracy Trade-off (3 Paired Seeds): Under a unified highest-validation-accuracy selection rule ([0, 429]), SGCR cuts raw test ECE from 6.74% to 2.84% and improves Brier (0.4088 to 0.4020) and NLL (0.7840 to 0.7683) at matched accuracy (69.58% vs. 69.65%). On the 84.93% test subset where both models predict the exact same option (identical 75.56% accuracy), SGCR still cuts ECE from 5.13% to 2.58% and lowers confidence on shared errors from 64.95% to 56.03%.
  3. Superiority Under a Strict Compute Cap: Capping total algorithmic forward passes at 299,520 (counting every training, pool-refresh, and gate-check pass against SGCR), SGCR achieves 69.81% accuracy and 2.96% raw ECE (1.46% with Temperature Scaling), outperforming Vanilla's full 1,170-step run (69.65% accuracy, 6.74% raw ECE, 2.12% with TS).
  4. The Error-Consumption Mechanism (1,170 Steps): A fixed dataset has a finite supply of real errors. As the model learns, it consumes the Wrong pool-shrinking it from 14.87% (step 0) to 2.17% (step 390) to 0.083% (step 780), where a manual audit shows 20/20 of the most-replayed items are conflicting-label duplicates. Once real errors are used up, forcing replay on this noisy tail raises late-stage ECE (26.60% vs. 15.08%). This confirms that real current errors are what stabilize SFT confidence, and the Wrong-pool fraction directly shows when that supply runs out.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.