- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies confidence calibration during supervised fine-tuning on multiple-choice questions. For Qwen Instruct, which starts overconfident, ECE, Brier score, and NLL first improve, then worsen. The authors link this pattern to gradients from training questions answered correctly or incorrectly, and to logit scale. They run one-step group updates from the same model and optimizer state. Based on these tests, they propose SGCR, which dynamically resamples hard questions the model currently gets wrong. SGCR lowers average ECE along the training path and extends the period of low ECE.
Critical strengths
- The authors track the full training run and test one-step updates from the same model parameters and AdamW state, changing only the next batch. This shows more directly than training curves that currently wrong and correct examples affect next-step calibration differently.
- Logit-scale controls separate calibration changes from accuracy changes. SGCR lowers ECE despite lower accuracy on Qwen-MMLU and nearly equal accuracy on Qwen-MedMCQA.
Critical weaknesses
- The model starts overconfident, so early fine-tuning may improve ECE before larger logits make it worse; NLL also gives stronger corrections to wrong answers. The one-step tests support this pattern, but they do not show that SGCR does more than change confidence or train more often on hard questions.
- On Qwen-MMLU, the SGCR checkpoint with the lowest validation ECE cuts test ECE from 3.35% to 1.48%, but test accuracy falls from 70.33% to 68.50%, and Brier score and NLL worsen. At the same final training step, ECE improves only from 11.36% to 10.72% while the other metrics worsen again, so the paper should explain when this tradeoff is useful.
- SGCR repeatedly checks which training questions the model gets wrong, so it uses extra forward passes. The paper compares methods after the same number of training steps, not the same computing cost, and does not show whether SGCR beats standard SFT, early stopping, temperature scaling, or simple hard-question resampling at equal cost.
Questions for authors
- Could you compare SGCR, standard SFT, early stopping, and post-training temperature scaling across several seeds and equal compute budgets? Please report ECE, accuracy, Brier score, and NLL together.
- Does SGCR still beat these baselines at checkpoints with matched accuracy? Could you report ECE changes on examples whose predictions stay fixed, to separate confidence changes from accuracy changes?
2 comments
Thanks the authors for the clarifications. However, I remain concerned about the effectiveness and validity of SGCR if sacrificing the accuracy and achieving similar calibration with logit-based post-hoc methods such as temperature scaling. Therefore, I'm leaning towards weak rejection.
Response to Reviewer Dbgz
Thank you for pointing out the missing comparisons. We added paired multi-seed runs, proper scoring rules (Brier/NLL), equal-budget evaluations, and fixed-prediction analyses.
1. Q1: Shared selection rules and all four metrics (Raw and Post-TS) SGCR's update rule uses no held-out data, whereas checkpoint selection and post-hoc Temperature Scaling (TS) use validation data equally. Across 3 paired seeds over [0, 429], selecting each run's highest-validation-accuracy checkpoint gives:
With the same accuracy-based selector, SGCR improves raw ECE, Brier, and NLL simultaneously without losing accuracy. After TS, SGCR + TS reaches the lowest ECE (1.55%) with nearly identical Brier/NLL.
2. Q1 (Continued): Equal-budget comparisons and screening cost Across 1,170 updates, both methods run 299,520 gradient examples, while SGCR adds 1,198,104 refresh and 1,284,236 gate-screening forwards (48.22 vs. 12.38 GPU-hrs). Enforcing a strict cap of 299,520 total algorithmic forward passes (counting all training, refresh, and gate-check passes against SGCR; step 195 vs. 1,170) gives:
Even when penalized for every auxiliary forward pass (and under 3:1 FLOP-equivalent and time budgets), SGCR outperforms Vanilla across all four metrics within the shared budget.
3. Q2: Confidence changes with predictions held fixed On the test subset where the selected Vanilla and SGCR checkpoints predict the exact same option (84.93 ± 5.40% of test items; identical accuracy of 75.56 ± 1.85% by construction):
On the exact same answers and mistakes, SGCR cuts overconfidence on shared errors by 8.92 pp and improves ECE, Brier, and NLL.
4. Dynamic verification vs. 6 controls & 1,170-step error consumption