- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper identifies the calibration degradation problem of backpropagation-free test-time adaption of vision-language models, which stems to some extent from class-dependent norm growth due to introduction of anchor text and prototypical visual tokens without normalization. The paper introduces CRAFT, a calibration-aware routing method that (1) combines spherical interpolation and a rank-preserving Fisher Rao retraction through zero-shot predictions as the concentration center, and (2) employs a scale-invariant routing rule. The method reduces average ECE on 15 image recognition benchmarks with accuracy preservation of underlying adapters.
Critical strengths
-
The improvements in calibration seem significant for some adapters. As shown in table 1, the mean expected calibration error (ECE) of PTA adapter drops from 17.17% to 8.09% and the accuracy also slightly improves from 69.33% to 69.46%. This is a meaningful gain (little improvement in recognition accuracy for much improved calibration). And the empirical contribution might be stronger since results are reported across multiple adapters.
-
This experiment is not specific to ECE and explores whether there is value in confidence for selective prediction; by contrast, as observed in Table 9 on Image Net-A, the AURC decreases from 257.80 to 210.66 while the accuracy at 10% coverage increases from 80.32% to 92.86%. This provides support for real world value of being able to know what predictions to trust.
Critical weaknesses
-
It has not been possible to properly isolate the unique contribution of SLERP over normalized addition in this analysis; Table 14 simply compares normalized Euclidean fusion to the entire CRAFT pipeline, rather than the intended sub-procedure of fixing the routing and retraction parameters. At , Equation 7 exactly reduces to the normalized sum of the two unit vectors, and GPR similarly tends to converge to this weight when both top-logit Z-scores are large (as noted in the study). Therefore, a fixed-weight baseline, which inherently depends on well-matched controls, is especially significant in this case, as it remains unclear how much the proposed fusion geometry can do over the combination of normalization and confidence correction.
-
The zero-shot confidence reference deserves more attention. Despite the good overall calibration the fact it remains close to the perfect calibration point does not make the zero-shot concentration a sensible upper bound for any adapted prediction, especially when adaptation fixes a zero-shot mistake. Additionally, regressions seem to be quite significant: on EuroSAT, CRAFT makes TCA's ECE (Expected Calibration Error) jump from 3.46% to 14.06% and TDA's from 6.71% to 11.26%. The paper does not specify whether these regressions stem from the retraction or from the modified fusion. This distinction is necessary to establish when the technique could be used as a safe default.
Questions for authors
-
How does CRAFT compare with normalized fusion using the same GPR and IGFR settings, with a fixed ?
-
On EuroSAT, are the impacts of HCIF+GPR and IGFR on ECE and on the confidence-accuracy gap individually?
-
Can you also clarify the PTA accuracy of the Separate stream? It's shown as 67.39% in Table 3, but the average of eleven entries in Table 6 makes 65.57%. This significantly alters the apparent accuracy cost of CRAFT.
Response to the authors
Thank you for the detailed response and additional experiments. The clarification of prompt protocols and view aggregation substantially addresses my concern about discrepancies with published baselines. The paired confidence intervals are also helpful, and the distinction between fusion-related accuracy changes and IGFR’s rank-preserving behavior clarifies the accuracy–calibration trade-off. I no longer consider these issues sufficient grounds for clear rejection.
However, the additional ablations leave me unconvinced of the advantages of the full CRAFT design over simpler alternatives. SLERP and normalized interpolation perform essentially equivalently, while GPR provides little improvement over fixed equal weighting. More significantly, probability averaging achieves better average ECE, NLL, and AURC than CRAFT on PTA, TDA, and TCA, with similar accuracy. The EuroSAT decomposition also demonstrates that IGFR can substantially worsen an already well-calibrated fused prediction, without a reliable label-free criterion for identifying when it should be applied.
I appreciate the controlled magnitude intervention, which strengthens the evidence that concentration inflation contributes to miscalibration. Nevertheless, I remain uncertain whether this diagnostic contribution, together with the proposed correction, sufficiently establishes the value of CRAFT relative to the simpler alternatives evaluated during rebuttal.
I am therefore raising my score from 1 (clear rejection) to 2 (weak rejection), reflecting the substantial improvements in experimental support and clarification, while retaining reservations about the contribution and comparative effectiveness of the full framework.