acceptodds
Under review as a conference paper at ICLR 2027

Concentration-Invariant Geodesic Fusion for Calibrated Test-Time Adaptation

Abstract

Backpropagation-free test-time adaptation (TTA) provides an efficient alternative for adapting pre-trained vision-language models under distribution shift; however, the underlying causes of miscalibration in these methods remain poorly understood. In this paper, we propose CRAFT, a calibration-aware adaptation framework that retains the efficiency of the backpropagation-free paradigm. We attribute the accuracy-calibration trade-off to a phenomenon we term Geometric Collapse: the unnormalized Euclidean fusion of textual anchors and visual prototypes departs from the unit hypersphere upon which the model was pre-trained, inflating the concentration unevenly across classes and resulting in overconfident predictions. To address this, CRAFT performs logit fusion along the geodesic (SLERP) on the unit hypersphere, which preserves the pre-trained concentration geometry, with a tuning-free instance-level weighting. Subsequently, a Fisher-Rao retraction on the probability simplex caps the Fisher-Rao concentration radius of the fused prediction at the VLM's inherent zero-shot level, mitigating overconfidence without requiring labeled data or tuned thresholds. Across 15 image recognition benchmarks encompassing cross-dataset benchmarks and natural distribution shifts, CRAFT consistently reduces the average Expected Calibration Error (ECE) while maintaining competitive accuracy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Test-Time Adaptation, Vision-Language Models, Calibration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4TKaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies the calibration degradation problem of backpropagation-free test-time adaption of vision-language models, which stems to some extent from class-dependent norm growth due to introduction of anchor text and prototypical visual tokens without normalization. The paper introduces CRAFT, a calibration-aware routing method that (1) combines spherical interpolation and a rank-preserving Fisher Rao retraction through zero-shot predictions as the concentration center, and (2) employs a scale-invariant routing rule. The method reduces average ECE on 15 image recognition benchmarks with accuracy preservation of underlying adapters.

Critical strengths
  1. The improvements in calibration seem significant for some adapters. As shown in table 1, the mean expected calibration error (ECE) of PTA adapter drops from 17.17% to 8.09% and the accuracy also slightly improves from 69.33% to 69.46%. This is a meaningful gain (little improvement in recognition accuracy for much improved calibration). And the empirical contribution might be stronger since results are reported across multiple adapters.

  2. This experiment is not specific to ECE and explores whether there is value in confidence for selective prediction; by contrast, as observed in Table 9 on Image Net-A, the AURC decreases from 257.80 to 210.66 while the accuracy at 10% coverage increases from 80.32% to 92.86%. This provides support for real world value of being able to know what predictions to trust.

Critical weaknesses
  1. It has not been possible to properly isolate the unique contribution of SLERP over normalized addition in this analysis; Table 14 simply compares normalized Euclidean fusion to the entire CRAFT pipeline, rather than the intended sub-procedure of fixing the routing and retraction parameters. At β=0.5\beta=0.5, Equation 7 exactly reduces to the normalized sum of the two unit vectors, and GPR similarly tends to converge to this weight when both top-logit Z-scores are large (as noted in the study). Therefore, a fixed-weight baseline, which inherently depends on well-matched controls, is especially significant in this case, as it remains unclear how much the proposed fusion geometry can do over the combination of normalization and confidence correction.

  2. The zero-shot confidence reference deserves more attention. Despite the good overall calibration the fact it remains close to the perfect calibration point does not make the zero-shot concentration a sensible upper bound for any adapted prediction, especially when adaptation fixes a zero-shot mistake. Additionally, regressions seem to be quite significant: on EuroSAT, CRAFT makes TCA's ECE (Expected Calibration Error) jump from 3.46% to 14.06% and TDA's from 6.71% to 11.26%. The paper does not specify whether these regressions stem from the retraction or from the modified fusion. This distinction is necessary to establish when the technique could be used as a safe default.

Questions for authors
  1. How does CRAFT compare with normalized fusion using the same GPR and IGFR settings, with a fixed β=0.5β=0.5?

  2. On EuroSAT, are the impacts of HCIF+GPR and IGFR on ECE and on the confidence-accuracy gap individually?

  3. Can you also clarify the PTA accuracy of the Separate stream? It's shown as 67.39% in Table 3, but the average of eleven entries in Table 6 makes 65.57%. This significantly alters the apparent accuracy cost of CRAFT.

Reviewer_GdeSOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

The paper proposes an explanation to the miscalibration of VLMs when adapted with backpropagation-free TTA methods. Their main argument is that it is caused by unnormalized fusion of zero-shot and prototype logits, which causes predictions to leave the unit-hypersphere.

They propose a new method called CRAFT, which addresses this problem and delivers calibrated TTA for VLMs.

Critical strengths
  1. The paper tackles a significant problem in TTA for VLMs, where calibration seems to be signficantly worse compared to that of TTA methods using backpropgation.

  2. The paper gives some theoretical insights into why this miscalibration is happening and proposes a solution. CRAFT seems to reduce ECE significantly in many cases.

Critical weaknesses
  1. The results of the baselines are inconsistent with what is already published. For example in Table 1, all backpropagation based methods deliver worse accuracy on average. This is in conflict with what is reported by C-TPT. Confidence intervals are missing, making it hard to interpret why this differs.

  2. CRAFT seems to worsen accuracy on average. In Table 1 and 2, accuracy is decreased in the majority of cases when using CRAFT. Why is this?

  3. The paper is hard to follow at many places and it takes significant effort to understand what the authors mean.

Questions for authors
  1. Why do your results not match with published literature?

  2. Can you add confidence intervals to your results?

  3. Why does CRAFT worsen accuracy?

1 comment
Reviewer_GdeSOpenReview ↗

Response to the authors

Thank you for the detailed response and additional experiments. The clarification of prompt protocols and view aggregation substantially addresses my concern about discrepancies with published baselines. The paired confidence intervals are also helpful, and the distinction between fusion-related accuracy changes and IGFR’s rank-preserving behavior clarifies the accuracy–calibration trade-off. I no longer consider these issues sufficient grounds for clear rejection.

However, the additional ablations leave me unconvinced of the advantages of the full CRAFT design over simpler alternatives. SLERP and normalized interpolation perform essentially equivalently, while GPR provides little improvement over fixed equal weighting. More significantly, probability averaging achieves better average ECE, NLL, and AURC than CRAFT on PTA, TDA, and TCA, with similar accuracy. The EuroSAT decomposition also demonstrates that IGFR can substantially worsen an already well-calibrated fused prediction, without a reliable label-free criterion for identifying when it should be applied.

I appreciate the controlled magnitude intervention, which strengthens the evidence that concentration inflation contributes to miscalibration. Nevertheless, I remain uncertain whether this diagnostic contribution, together with the proposed correction, sufficiently establishes the value of CRAFT relative to the simpler alternatives evaluated during rebuttal.

I am therefore raising my score from 1 (clear rejection) to 2 (weak rejection), reflecting the substantial improvements in experimental support and clarification, while retaining reservations about the contribution and comparative effectiveness of the full framework.

Reviewer_WAhDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the deterioration in predictive calibration that accompanies improvements in classification accuracy in backpropagation-free test-time adaptation (TTA), and proposes the CRAFT framework. The authors argue that directly fusing textual anchors and visual prototypes causes class-dependent feature norm inflation, leading to imbalanced logit scales and overconfident predictions. To address this issue, CRAFT uses spherical linear interpolation (SLERP) in HCIF to maintain unit-norm fused vectors, adaptively determines fusion weights through GPR, and constrains the Fisher–Rao concentration radius of the predictive distribution using IGFR. Experiments show that the method significantly reduces average ECE across multiple TTA frameworks and distribution-shift scenarios while maintaining comparable classification accuracy in most cases.

Critical strengths

1. The paper addresses a practically meaningful problem.

This paper focuses on the decline in the reliability of predictive confidence despite improvements in accuracy in backpropagation-free TTA, addressing the limited attention that existing methods pay to calibration. The selective prediction experiments further show that the calibration improvements are reflected not only in ECE but also in the reliability of confidence-based prediction selection.

2. The method has a clear design, a reasonable theoretical interpretation, and good deployment efficiency.

CRAFT controls overconfidence at both the feature-fusion and probability-output levels, and provides mathematical derivations for the unit-norm property of the fused vectors and the preservation of class rankings after calibration. The method requires no additional labeled data, backpropagation, or model parameter updates, introduces no additional model forward passes, and incurs little additional GPU memory usage, making it promising for deployment.

3. The experiments provide broad coverage and relatively consistent calibration improvements.

The experiments cover multiple classification datasets, TTA methods, backbones, natural distribution shifts, and correlated data streams, and include component ablations, comparisons with normalized fusion, and selective prediction analyses. CRAFT significantly reduces average ECE, particularly for adaptation methods with severe calibration degradation, demonstrating a degree of generality and effectiveness.

Critical weaknesses

1. Insufficient Support for the Geometric Mechanism

The authors show that direct addition causes class-dependent changes in feature norms and affects the effective scale of the logits. However, norm inflation does not necessarily increase calibration error, as its effect also depends on prediction correctness and visual prototype quality. The paper lacks direct empirical analysis linking the degree of norm inflation to the actual deterioration in ECE, leaving insufficient evidence to support its interpretation as the primary cause of calibration degradation.

2. Unclear Independent Contributions of the Fusion Modules

The fusion weights produced by GPR are restricted to approximately the range from 1/3 to 2/3 and approach 0.5 when both branches have large Z-scores, which may limit its adaptive capability. Meanwhile, when the weight is 0.5, SLERP is equivalent to addition followed by L2 normalization, provided that the two unit vectors are not antipodal. Although the paper compares against normalized fusion, the full CRAFT framework also includes GPR and IGFR, making it difficult to distinguish the independent gains from each module. In addition, the paper lacks a baseline that directly performs weighted fusion of the two branches' Softmax probabilities, leaving the advantages of the more complex fusion design over simpler alternatives insufficiently established.

3. IGFR Depends on the Reliability of the Zero-Shot Reference

IGFR uses the concentration of the original zero-shot model's probability distribution as an upper bound, but the original model may itself be overconfident or underconfident under distribution shifts. When the adapter acquires useful information, higher predictive confidence may be justified, yet it is still suppressed by this constraint. Limiting the Fisher–Rao concentration radius only ensures that the distribution does not become excessively concentrated; it does not directly guarantee improved calibration. The paper does not sufficiently validate the conditions under which this reference assumption is appropriate.

4. Accuracy Degradation and Inconsistent Experimental Numbers

The paper generally emphasizes that CRAFT improves calibration while largely preserving classification accuracy, but some correlated-stream experiments show a noticeable accuracy cost. For example, under the γ=0.001\gamma=0.001 setting, PTA's average accuracy drops from 65.97% to 64.78%, while its accuracy on EuroSAT drops from 50.16% to 43.17%, a decrease of 6.99 percentage points. Furthermore, calculating the average accuracy of PTA under the Separate setting from the results for the 11 datasets reported in Table 6 gives approximately 65.57%, whereas both Tables 3 and 6 report 67.39%. This numerical inconsistency affects the assessment of the trade-off between accuracy and calibration under correlated data streams and needs to be verified by the authors.

Questions for authors

Please kindly refer to the weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.