- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
With a framework called PAAF, the paper tests whether lower training loss also brings better probabilities and answers under scheduled intervention in a frozen SmolVLM. Gain scheduling lowers held-out loss but worsens Q-Bench2 Brier score and NLL at similar accuracy, and an allocation shuffle changes 3 of 500 answers.
Critical strengths
-
The main comparison is carefully controlled, so I believe the narrow result. The 2 × 2 design varies the gain and temperature schedules separately, with matched seeds and time averages checked in logs.
-
The paper is careful with its claims. It reports Brier and NLL next to accuracy, bases its main statistics on seeds rather than questions, and states its limits openly.
Critical weaknesses
-
The main lesson seems to be known already, and this is the main reason for my rating. Guo et al. [1], cited by the paper, showed that loss and calibration can move apart, as the paper's own LoReFT sweep also shows (Table 12). PAAF itself mostly puts existing parts (VEFT [2], KnG [3], KSL [4]) into a new frame, so the framework adds little on its own.
-
The evidence seems too narrow for the paper's general advice. The main reliability results come from one backbone, one layer, one schedule shape, one benchmark family (Q-Bench), and three seeds. Brier and NLL are also shown only for Paired-coordinate KnG, which the paper says reacts differently from the other two Actions (Section 4.5).
-
A bigger gain at test time may explain the main result, which would weaken the lesson that matched averages are not enough. The gain ramps up during training, so the Gain-only model may be tested with a bigger gain than the Fixed model. The "fixed terminal" variant in Table 1 would check this, but its results are not shown.
-
The loss and the proper scores are measured on different data, and the training data is not named. The held-out objective is token cross-entropy plus allocation regularizers on held-out training data, and the cross-entropy part is itself a form of NLL. So the result may show a gap between in-domain fit and transfer to Q-Bench2, not a split between fit and probability quality.
-
The allocation test seems too weak to show that decision sensitivity is a separate property. The token shuffle barely moves the internal readouts (linear CKA 0.99996, routing JSD 0.00212), so few answer changes would be expected. The paper itself notes the test does not separate weak coupling from a perturbation that is too small.
-
The audited setups bring little benefit over the frozen model, which limits the practical value of the audit. On Q-Bench (Q1), every reported VEFT-family setup is less accurate than zero-shot (Tables 4 and 14). On Q-Bench2, they beat zero-shot accuracy by under two points but have worse ECE.
References
[1] Chuan Guo et al. On calibration of modern neural networks. ICML, 2017.
[2] Noriaki Kawamae. One-shot multi-view visual conversational recommendation. KDD, 2025.
[3] Noriaki Kawamae. Multi level vision language adapter. ICDM, 2025.
[4] Noriaki Kawamae. Knowledge-aligned domain shift tuning for efficient adaptation in large language models. KDD, 2025.
Questions for authors
Questions:
- Which gain and temperature values do you use at test time? If the proper-score gap remains against the fixed-terminal variant, the schedule claim would be more convincing and my score could rise.
- What is the training data? If the gap remains when loss and proper scores use the same data, the central claim would be much stronger.
- Does post-hoc temperature scaling of the four answer logits, as in Guo et al. [1], remove the Brier and NLL gap? If not, the reliability cost would matter more in practice.
Suggestions:
- Try stronger allocation changes, such as uniform weights or removing vector fields, to see when answers start to move.
- Report Brier, NLL, and accuracy for all three Actions, and repeat the 2 × 2 test on a second backbone.
- Use fewer new terms, and fix the table numbers in Appendix B.3, which point to Tables 1 to 3 instead of 13 to 15.