acceptodds
Under review as a conference paper at ICLR 2027

Prompting with Connections: Graph-Guided Attribute Selection for Test-Time Calibration

Abstract

Test-time prompt tuning adapts vision-language models (VLMs) to unseen target domains without labels by minimizing prediction entropy over augmented views. While boosting zero-shot accuracy, this process often induces overconfidence and degrades calibration. Existing approaches initialize prompts with LLM-generated visual attributes under contrastive regularization, but treat attributes independently, ignoring their relational structure. We introduce ARGTCA, a graph-guided framework for calibrated test-time prompt tuning that explicitly models relationships among class-specific visual attributes. By representing class–attribute pairs as nodes and linking within-class siblings and identical attributes across classes in a Symbolic Attribute Graph, a GAT trained with a supervised contrastive objective produces relational embeddings and makes attribute genericity—how many classes share an attribute—legible as distance from the class centroid (Spearman on Caltech101). The embeddings drive two prompt-initialization strategies: ARGTCA-Div selects diverse, complementary attributes, while ARGTCA-Disc selects highly discriminative ones; the selected attributes then initialize the test-time prompts. Across ten visual-classification benchmarks on CLIP ViT-B/16, ARGTCA-Div reduces average ECE to , a reduction of up to relative to existing test-time prompt-tuning methods, while ARGTCA-DISC attains the highest average accuracy () at a competitive ECE of . On ResNet-50, ARGTCA-DIV again attains the lowest average ECE () and ARGTCA-DISC the second lowest (), confirming generalization across CNN and Transformer encoders. Attribute selection runs once, offline, adding no test-time overhead. The codebase will be released upon acceptance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision language models, calibration, trustworthy VLMs, graph neural networks, graph learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LcMvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a graph-guided test-time calibration method for VLM models. ArgTCA constructs a LLM-based symbolic asymmetric graph (SAG) for class-attribute pairs. The intra-class connects attribute from the same class, and the inter-class connects identical attributes across classes. The graph attention training then uses supervised contrastive learning to produce relation-aware attribute embeddings. ArgTCA-DIV and or ArgTCA-DISC use the embedding to select attributes (either diversity within each class, or discrimination from other classes). The selected attribute is then used in the TCA soft-prompt adaptation process.

The main contribution of the paper is that it addresses the intra-class redundancy and inter-class ambiguity issues in LLM-generated attributes and provides an intuitive graph-based learning for extracting relation-aware attribute embeddings.

The experimental results are convincing: ArgTCA consistently improves Expected Calibration Error over TCA and several prompt-tuning baselines across multiple datasets and two CLIP backbones.

Critical strengths
  1. Clear motivation and easy-to-follow: The paper provides two main limitations of current attribute-based calibration method: intra-class attributes redundancy, and inter-class attribute ambiguity.

  2. Graph-based design is intuitive and explainable: Representing each class–attribute pair as a node and modeling intra-class and inter-class relations provides a natural way to incorporate relational structure that independent attribute ranking ignores.

  3. Useful ablation studies: The paper provides several analyses, including edge removal, no-GAT variants, different graph encoders, and intra-/inter-edge contributions, and random selection, which help isolate the benefit of the graph-based design.

  4. Efficient during test-time: All the graph construction, GAT training, and attribute selection are performed offline, while inference reuses the original TCA pipeline, making the approach practical at test time.

Critical weaknesses
  1. Attribute grounding in CLIP is assumed rather than verified. The paper assumes that CLIP can recognize the selected attributes. However, some attributes may be linguistically meaningful but weakly grounded or not observable in a given image. How can you verify that all attributes are valid for a certain model?

  2. Why are DIV and DISC separated rather than jointly optimized? It is unclear why these two objectives are treated as separate variants rather than combined into a unified selection criterion.

  3. Calibration-accuracy tradeoff is not fully discussed. In Table 1, with ResNet-50, ArgTCA improves ECE while reducing top-1 accuracy, TCA: Acc=58.58, ECE=4.11; ArgTCA-DIV: Acc=57.70, ECE=3.66. It seems that ArgTCA improves the model's confidence calibration, but in some cases, it also makes the model less accurate at classification. The author should discuss this more.

Questions for authors

see weakness

5 comments

(Part 1 of 3)

We thank the reviewer for the positive assessment of our motivation, the explainable graph design, the ablation studies, and the test-time efficiency of the method, and for the thoughtful questions.

Is attribute grounding in CLIP verified?

We agree that grounding should be measured rather than assumed. For each candidate attribute we compute the AUROC of CLIP's score for "a photo of a {attribute}." at separating the class's test images from all others (0.5: not visible; 1: perfectly visible). Labels are used only for this offline diagnostic, never by the method. Mean over nine datasets (Table 31):

Selected atttributesViT-B/16 AUROCViT-B/16 picks < 0.6ResNet-50 AUROCResNet-50 picks < 0.6
TCA's two0.75724%0.72730%
ARGTCA-DISC0.74626%0.71532%
ARGTCA-DIV0.66239%0.64842%
Random 2 of 100.69434%0.66839%

ARGTCA-DISC's selections are about as visible as TCA's on both backbones. We also planted unrelated atttributes in 10% or 30% of each class's candidate pool, rebuilt the graph and reran the full method (ViT-B/16, three seeds). Percentage of selected attributes that are planted atttributes (Table 32):

PollutionDatasetTCAARGTCA-DIVARGTCA-DISCARGTCA-DIV + filter
10%Caltech10193802
DTD9310.71
OxfordPets83003
30%Caltech10131701610
DTD29500.45
OxfordPets28500.98

At 30% pollution ARGTCA-DISC selects planted atttributes only 5.8% of the time on average (chance: 30%), and its DTD results stay close to the clean ones (Acc./ECE 46.73/6.62 → 46.61/7.06). ARGTCA-DIV is vulnerable because maximizing the distance between a class's two atttributes favors atttributes unlike its other attributes, and unrelated atttributes fit that description. A label-free text-relevance filter applied before selection reduces ARGTCA-DIV's planted-atttribute rate to 7.7% at 30% pollution; it uses no images or labels, so it can be applied to any new model or attribute list. Changes in visibility are not significantly correlated with changes in ECE (Spearman, n=9n=9), so grounding acts as a robustness condition rather than a predictor of improvement (new Appendix P).

(Part 2 of 3)

Why are DIV and DISC separate variants?

They target different failure modes (intra-class redundancy vs. inter-class ambiguity) and pull the selection in opposite directions. After graph training, ARGTCA-DISC's picks move toward class-specific attributes, while ARGTCA-DIV's move toward generic ones: on Caltech101 the mean genericity of the selected attributes falls from 2.15 to 1.33 for ARGTCA-DISC and rises from 2.80 to 5.25 for ARGTCA-DIV (Appendix I). Their margins also scale differently with the number of classes: the discriminability margin concentrates as K−1/2K^{-1/2} (Corollary 1) and the diversity margin does not, so a fixed weighting of the two scores would shift with the label-space size.

During development we also tried three ways of combining the two criteria in a single rule: an unscaled sum of the pair scores, greedy maximal marginal relevance (MMR), and a product of the two scores. ECE (%, ViT-B/16; combined rules single seed, single criteria three-seed means of Table 1; new Appendix L.10, Table 27):

RuleCaltech101DTD
Sum3.174.56
MMR1.844.33
Product1.745.20
ARGTCA-DIV2.704.22
ARGTCA-DISC1.946.62

None consistently beats the better single criterion. On DTD all three have higher ECE than ARGTCA-DIV; on Caltech101 the MMR and product rules stay within the seed spread of ARGTCA-DISC (1.94±0.501.94 \pm 0.50). MMR, which we ran on nine datasets, averages 5.66 ECE, against 4.73 for ARGTCA-DIV and 5.47 for ARGTCA-DISC on the same datasets. We therefore report the two criteria as separate variants, recommend ARGTCA-DIV as the calibration-oriented default, and list a principled joint objective as future work.

(Part 3 of 3)

Accuracy–calibration trade-off

We agree that this deserves explicit discussion. ARGTCA selects attributes to improve confidence reliability, and its accuracy cost is small and depends on the backbone and variant.

The ResNet-50 averages quoted in the review (TCA 58.58 / 4.11) included a transcription error in TCA's ImageNet entry, which is now corrected (see our reply to Reviewer 7kSv). With the corrected entry, TCA's ResNet-50 average is 57.64 / 4.10, and ARGTCA-DIV (57.70 / 3.66) is no longer below TCA in accuracy. Average change relative to TCA (Appendix Q):

ViT-B/16: ΔAcc / ΔECEResNet-50: ΔAcc / ΔECE
ARGTCA-DIV, all 10 datasets−0.57 / −0.41+0.06 / −0.44
ARGTCA-DISC, all 10 datasets+0.09 / +0.28−0.78 / −0.11
ARGTCA-DIV, excl. EuroSAT+0.79 / −1.29+0.30 / −0.32
ARGTCA-DISC, excl. EuroSAT+0.79 / −0.96−0.10 / −0.44

Small trade-offs remain for ARGTCA-DIV on ViT-B/16 and for ARGTCA-DISC on ResNet-50, and EuroSAT drives most of them. Excluding EuroSAT, both variants gain accuracy on ViT-B/16 and ECE improves in all four cases. We added this discussion, with a per-dataset plot (Figure 27), as the new Appendix Q.

We sincerely thank the reviewer again for the constructive feedback. We hope that the additional analyses address the concerns raised, and we would be happy to provide any further clarification. If the concerns have been satisfactorily addressed, we would be grateful if the reviewer could kindly reconsider the evaluation score.

Reviewer_7kSvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

ArgTCA extends TCA by selecting attributes through a graph of class–attribute pairs. A graph attention network models within-class links and shared attributes across classes. DIV favors intra-class diversity, while DISC favors separation from competing classes. Graph processing runs offline, and selected attributes enter TCA’s unchanged test-time prompt-tuning pipeline. Evaluation covers ten datasets, four natural shifts, and two CLIP backbones.

Critical strengths
  • Practical integration: The method addresses attribute redundancy and ambiguity using class names and text features alone. Offline graph learning adds no inference step beyond TCA and requires no target-domain data for attribute selection.

  • Useful empirical evidence: DIV reduces average ECE from 4.87 to 4.46 on ViT-B/16 and from 4.11 to 3.66 on RN50. Ablations examine graph edges, raw CLIP features, random selection, and alternative networks, helping assess the contribution of relational attribute selection.

Critical weaknesses
  • Unclear baseline comparability: Table 1’s RN50 ImageNet baseline accuracies differ substantially from the published TCA hard-prompt results. Although the authors rerun baselines, matched configurations and logs are needed to explain these differences. Missing uncertainty estimates also make small improvements over TCA difficult to assess.

  • Uneven calibration benefits: On ViT-B/16 EuroSAT, DIV reduces accuracy from 45.80% to 32.91% and increases ECE from 8.37 to 15.80. Both variants worsen average Brier score relative to TCA and trail O-TPT’s average ECE across natural shifts, including substantial underperformance on ImageNet-R. DIV also nearly matches random selection on Caltech101 and exactly matches it on OxfordPets. These results weaken the evidence for a consistent graph-specific advantage.

Questions for authors
  • Can you reconcile the RN50 ImageNet baselines with published results and provide matched configurations, logs, and paired uncertainty estimates?

  • Do gains over random and edge-free selection persist across matched seeds, beyond the reported single-seed ablations?

  • Are the EuroSAT regressions reproducible under identical TCA settings? What explains the failures on EuroSAT and ImageNet-R?

  • What independence or dependence-control assumptions justify Corollary 1? Is the divisor applied to logits in Eqs. (2) and (18) a temperature or CLIP’s logit scale?

6 comments

(Part 1 of 4)

We thank the reviewer for the detailed and constructive review, and in particular for the careful check of our baselines. Changes in the revised PDF are marked in blue. We address each point below.

ResNet-50 ImageNet baselines

Thank you for catching this. The ResNet-50 ImageNet entries of TPT and TCA in Table 1 were transcription errors. They are now corrected, in line with the published results (TPT 60.7 / 11.4; TCA 58.72 / 1.76). The TPT value is the mean of our own three runs (60.73±0.0660.73 \pm 0.06 / 11.38±0.0611.38 \pm 0.06). Corrected ResNet-50 entries (Acc. / ECE):

MethodImageNetAverage over 10 datasets
TPT60.73 / 11.3857.39 / 11.92
TCA58.72 / 1.7657.64 / 4.10
ARGTCA-DIV58.30 / 1.5557.70 / 3.66
ARGTCA-DISC58.96 / 1.7456.86 / 3.99

The conclusions for ResNet-50 do not change: ARGTCA-DIV keeps the lowest average ECE and ARGTCA-DISC the second lowest, and ARGTCA-DIV's average accuracy is now also above TCA's. We also checked our TCA runs on ResNet-50 against the published numbers: they agree to within about 0.2 ECE on each of the other nine datasets.

Matched configurations

All methods are evaluated under one test-time protocol: the same CLIP weights and test splits, 64 views per test image with the 10% lowest-entropy views retained, a single AdamW step (learning rate 5×10−35\times10^{-3}) on a 4-token context initialized to "a photo of a", a prompt reset after every test image, and ECE with 20 equal-width bins (new Appendix A.3). Each method uses its own regularization weights, fixed across all datasets and identical for TCA and ARGTCA on both backbones (Table 8):

MethodTest-time objectiveWeights
TPTLTPT\mathcal{L}_{\text{TPT}}–
C-TPT$\mathcal{L}_{\text{TPT}} + $text-feature dispersionλ=50\lambda=50
O-TPT$\mathcal{L}_{\text{TPT}} + $orthogonalityλ=18\lambda=18
A-TPT$\mathcal{L}_{\text{TPT}} + $angular diversityλ=−10\lambda=-10, τ=0.99999\tau=0.99999
TCALTPT−αLinter+βLintra\mathcal{L}_{\text{TPT}} - \alpha\mathcal{L}_{\text{inter}} + \beta\mathcal{L}_{\text{intra}}α=10\alpha=10, β=35\beta=35
ARGTCAas TCA, graph-selected attributesα=10\alpha=10, β=35\beta=35

Logs

We will release the code and configuration files for all experiments with the paper.

Uncertainty estimates

Table 1 reports means over three random seeds for ARGTCA on ViT-B/16 (ImageNet: one seed, marked †). In the submission, per-dataset standard deviations were reported for three datasets (Appendix H; Tables 17–18 of the submission, now Tables 20–21). The revision reports them for nine datasets (new Appendix A.4, Table 9), with each seed rebuilding the attribute graph and repeating test-time adaptation:

DatasetDIV Acc.DIV ECEDISC Acc.DISC ECE
Caltech10192.95 ± 0.242.70 ± 0.4393.10 ± 0.281.94 ± 0.50
OxfordPets89.47 ± 0.103.55 ± 0.0988.91 ± 0.263.16 ± 0.33
StanfordCars65.02 ± 0.222.83 ± 0.0566.44 ± 0.074.04 ± 0.18
Flowers10270.56 ± 0.443.63 ± 0.5570.56 ± 0.613.33 ± 0.18
Food10183.88 ± 0.171.85 ± 0.1184.06 ± 0.152.03 ± 0.29
FGVCAircraft24.64 ± 0.425.10 ± 0.3924.87 ± 0.214.45 ± 0.22
DTD47.79 ± 0.304.22 ± 0.3946.73 ± 0.276.62 ± 0.56
UCF10169.50 ± 0.332.85 ± 0.1368.61 ± 0.273.85 ± 0.05
EuroSAT†32.91 ± 0.0915.80 ± 0.1139.53 ± 0.0519.84 ± 0.18

† For EuroSAT, Table 9 keeps the graph fixed and varies only test-time adaptation; Appendix O (Table 29) reports EuroSAT with the graph rebuilt for every seed.

The standard deviation of ECE is at most 0.56 on every dataset, well below the differences to TCA on the datasets where ARGTCA gains most (e.g., StanfordCars 7.85 → 2.83, Food101 5.28 → 1.85, OxfordPets 6.30 → 3.55). Seed-matched comparisons for the ablations follow in the next comment.

(Part 2 of 4)

Do the gains over random and edge-free selection persist across matched seeds?

We repeated the ablations of Table 2 on three seeds, with the graph rebuilt for every seed and each ablation paired with the full model on the same seed. Random selection uses five independent draws of two attributes per class from the same ten candidates (new Appendix L.9, Table 26; Appendix L.8 gives the same comparisons as per-component ECE differences). ECE (%, ViT-B/16, mean ± std):

DatasetVariantARGTCANo edgesNo GAT trainingRandom (5 draws)
Caltech101DIV2.70 ± 0.432.27 ± 0.412.69 ± 0.072.44 ± 0.51
DISC1.94 ± 0.501.73 ± 0.302.68 ± 0.062.44 ± 0.51
OxfordPetsDIV3.55 ± 0.094.23 ± 0.173.93 ± 0.063.46 ± 0.38
DISC3.16 ± 0.334.27 ± 0.233.73 ± 0.063.46 ± 0.38
DTDDIV4.22 ± 0.395.84 ± 1.845.95 ± 0.125.65 ± 0.89
DISC6.62 ± 0.565.87 ± 2.008.90 ± 0.165.65 ± 0.89

The answer depends on the dataset and the variant, and we state it per dataset:

  • Over random selection, the gain persists for the variant matched to the dataset. ARGTCA-DIV beats all five random draws on DTD (4.22 vs. 5.65 ± 0.89 ECE, with 47.79 vs. 45.15 accuracy), and ARGTCA-DISC beats four of the five draws on Caltech101 and on OxfordPets. As the reviewer observed, ARGTCA-DIV is within seed variation of random selection on Caltech101 and OxfordPets. This matches the motivation of ARGTCA-DIV: its benefit is largest where a class's candidate attributes are near-synonyms, as for the texture descriptors of DTD (rough, bumpy, grainy), and smaller where the candidates are already varied.
  • Over edge-free training, message passing lowers ECE on OxfordPets for both variants on all three seeds and for ARGTCA-DIV on DTD (with +3.2 accuracy). On Caltech101, the edge-free variant is slightly better for both variants.
  • Graph training itself (versus applying the same selection rule to the frozen CLIP embeddings) lowers ECE in five of the six dataset–variant pairs, by up to 2.3 points (DTD, DISC: 6.62 vs. 8.90).

Section 6 and Appendix L now cite these three-seed results alongside the single-seed table.

Brier score

The Brier score couples calibration with accuracy, and its average is dominated by EuroSAT, where both variants lose accuracy. Per-dataset Brier scores (ViT-B/16, Table 15; lower is better):

DatasetTCAARGTCA-DIVARGTCA-DISC
Caltech1010.10130.10750.1008
OxfordPets0.14430.15100.1599
StanfordCars0.44790.44820.4402
Flowers1020.41130.40750.3977
Aircraft0.84580.84410.8471
DTD0.70730.68260.6808
EuroSAT0.70620.82870.8150
UCF1010.43330.40750.4060
Food1010.23410.24170.2348
Average0.44790.45760.4536
Average w/o EuroSAT0.41560.41120.4084

On the eight datasets other than EuroSAT, both variants have a lower average Brier score than TCA, and ARGTCA-DISC has a lower Brier score than TCA on five of the nine datasets.

Is the EuroSAT regression reproducible under identical settings?

Yes. We re-ran TCA, ARGTCA-DIV and ARGTCA-DISC on EuroSAT over three seeds under the same protocol, on both backbones (new Appendix O, Table 29; mean ± std, graph rebuilt for every seed):

BackboneMethodAccuracyECE
ViT-B/16TCA45.82 ± 0.108.39 ± 0.05
ARGTCA-DIV32.15 ± 1.4215.92 ± 1.04
ARGTCA-DISC38.02 ± 1.2319.88 ± 1.63
ResNet-50TCA31.38 ± 0.065.63 ± 0.17
ARGTCA-DIV28.52 ± 2.654.83 ± 0.94
ARGTCA-DISC24.12 ± 1.6010.55 ± 2.13

On ViT-B/16 the regression appears on every seed. On ResNet-50 it does not: ARGTCA-DIV is on par with TCA, and only ARGTCA-DISC degrades.

(Part 3 of 4)

What explains the EuroSAT failure?

EuroSAT is the one benchmark whose candidate attributes are mostly generic remote-sensing atttributes rather than class-specific ones: imagery is a candidate for 7 of the 10 classes, rural and landscape for 5 each. Because ARGTCA-DIV maximizes the distance between the two attributes it picks, on this pool it selects such generic atttributes: only 3% of its selected attributes contain a atttribute of the class name, against 70% for TCA. Generic descriptions fit satellite images of any class, so predictions collapse onto a few classes (ARGTCA-DIV assigns 54% of test images to Annual Crop Land, whose true share is 11%), and the entropy-minimizing tuning step then makes these predictions confident. Removing all graph edges lowers ARGTCA-DIV's ECE on EuroSAT from 15.92 to 10.83 (Appendix O), so the failure comes from relational selection meeting a generic candidate pool. We now state this as a limitation: ARGTCA assumes that the candidate pool contains class-specific attributes, and EuroSAT's does not.

ImageNet-R and the natural shifts

The natural-shift experiments use TCA's hyperparameters for distribution shift (α=45\alpha=45, β=15\beta=15), while the ViT-B/16 results of Table 1 use TCA's fine-grained setting (α=10\alpha=10, β=35\beta=35); the revision now states this in Appendix D. The ImageNet-R result is sensitive to this choice, not to the graph. With the same graph and selected attributes (Table 13, Acc. / ECE, ViT-B/16, single seed):

Variantα/β\alpha/\betaImageNet-AImageNet-RImageNet-V2
ARGTCA-DIV10/3546.80 / 12.9672.05 / 0.6760.38 / 4.58
45/1546.05 / 10.0671.78 / 2.6959.19 / 3.94
ARGTCA-DISC10/3547.73 / 12.6772.24 / 0.9460.99 / 4.33
45/1547.96 / 3.3572.58 / 7.7660.35 / 3.96

With α=10\alpha=10, β=35\beta=35, the ECE on ImageNet-R drops to 0.94 (DISC) and 0.67 (DIV), below the 1.46 reported for O-TPT, but ImageNet-A degrades (DISC: 12.67 instead of 3.35), so neither setting is best on all shifts. The attributes themselves remain visible in the renditions of ImageNet-R: the image–attribute AUROC of ARGTCA-DISC's selections (0.838) is close to that of TCA's (0.847).

ECE over the four shifts with the shift setting (Table 14):

MethodImageNet-AImageNet-V2ImageNet-RImageNet-SAverage
O-TPT7.223.971.466.874.88
ARGTCA-DIV10.063.942.696.985.92
ARGTCA-DISC3.353.967.766.615.42

We agree with the reviewer that O-TPT has the lower average ECE over the four shifts, and we have revised Appendix D to say so without qualification and to report both settings.

Reviewer_ojxwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ARGTCA, a graph-guided attribute selection framework for improving the calibration of vision-language models during test-time prompt tuning. Unlike existing approaches that independently rank LLM-generated attributes, ARGTCA models intra-class and inter-class attribute relationships using a Symbolic Attribute Graph and a contrastively trained Graph Attention Network. Two strategies are introduced: ARGTCA-DIV for attribute diversity and ARGTCA-DISC for discriminability. Experiments across ten classification benchmarks and two CLIP backbones demonstrate improved calibration, supported by component ablations, distribution-shift evaluations, and embedding analyses.

Critical strengths
  1. Effective calibration improvements: The proposed method achieves competitive calibration performance across ten classification benchmarks and two CLIP backbones. ARGTCA-DIV reduces average ECE to 4.46% on ViT-B/16 and 3.66% on ResNet-50, demonstrating the practical value of relational attribute selection.

  2. Well-motivated relational modeling: The Symbolic Attribute Graph captures intra-class redundancy and inter-class ambiguity overlooked by independent attribute ranking. The two selection strategies offer complementary objectives, and ablation studies provide evidence supporting the effectiveness of graph-based attribute representations.

Critical weaknesses
  1. Limited methodological novelty: The framework primarily extends TCA by incorporating a GAT-based attribute selection stage while retaining the original test-time optimization objective. The graph construction, supervised contrastive learning, and selection criteria largely build on established techniques, limiting the methodological novelty.

  2. Inconsistent improvements and incomplete statistical validation: Although average ECE improves, performance varies considerably across datasets. For example, ARGTCA-DIV obtains substantially worse ECE than TCA on EuroSAT (15.80 vs. 8.37), while ARGTCA-DISC reaches 19.84. Additionally, the graph component does not consistently outperform simpler alternatives across all ablations, and the main results lack standard deviations or confidence intervals, making the robustness of the improvements difficult to assess.

Questions for authors
  1. What explains the substantial calibration degradation on EuroSAT, particularly for both proposed attribute selection strategies? Can the authors quantify how cross-class attribute overlap affects performance?

  2. Since graph-based training does not consistently outperform simpler alternatives such as MLP-based representations or edge-free variants, can the authors clarify which graph components are essential and under what conditions they provide consistent benefits?

  3. Could the authors report standard deviations or confidence intervals for the main experimental results to establish whether the observed improvements over TCA are statistically significant?

6 comments

(Part 1 of 4)

We agree that ARGTCA builds on established components: Graph Attention Networks, supervised contrastive learning, and TCA's test-time objective. Its novelty lies in two parts that prior work does not address: how attributes are connected, and how those connections guide attribute selection.

Attribute graph construction. Prior graph-based VLM methods build class hierarchies, visual concept graphs (GraphAdapter, HGCLIP, VCGPrompt), or graphs within a single class description (HPT), all for supervised or few-shot adaptation. To our knowledge, ARGTCA is the first to build a graph over LLM-generated attributes across classes and use it for label-free test-time calibration. This matters because TCA scores each (class, attribute) pair independently, while the two failure modes we target are relational: intra-class redundancy depends on how a class's attributes relate to one another, and inter-class ambiguity arises when classes share attributes. Our Symbolic Attribute Graph encodes both. Each (class, attribute) pair is a node, connected by intra-class edges to the class's other attributes and by inter-class edges to identical attributes of other classes.

Graph-guided attribute selection. The graph yields relation-aware attribute embeddings, which we use to select attributes that are diverse within a class (DIV) or discriminative across classes (DISC). Proposition 1 and Corollary 1 characterize the geometry of both criteria.

Moreover, ARGTCA keeps TCA's test-time objective unchanged, so that any change in calibration therefore comes from graph construction and selection.

To further isolate the contribution of attribute selection from test-time adaptation, we evaluated the selected attributes directly using zero-shot CLIP with ResNet-50, without test-time tuning (Table 28):

Attributes used (zero-shot CLIP, ResNet-50)Acc.ECE
Class name only (no attribute)55.336.11
TCA's two Attribute55.786.15
Random 2 of 10 (5 draws)55.785.92
ARGTCA-DIV's Attributes56.025.51
ARGTCA-DISC's Attributes56.295.19

The atttributes selected by ARGTCA give lower ECE and slightly higher accuracy than TCA's atttributes or random atttributes. This provides additional evidence that relational attribute selection itself improves the calibration of the resulting class representations, independently of the subsequent TCA optimization.

Finally, graph construction, GAT training, and attribute selection are performed once, offline, and therefore introduce no additional test-time cost.

(Part 2 of 4)

EuroSAT and cross-class attribute overlap

We thank the reviewer for highlighting the degradation on EuroSAT. We investigated this failure mode in detail and have added the analysis to Appendix O.

EuroSAT differs from the other benchmarks in that several candidate attributes are highly generic. In particular, imagery, rural, and landscape are each shared by 5–7 of the 10 classes and together account for 41 of the graph's 63 inter-class edges. Under ARGTCA-DIV, the selected attributes consequently become much less class-specific: only 3% of them contain an atttribute from the class name, compared with 70% for TCA. This is accompanied by a pronounced prediction collapse: 54% of EuroSAT test images are predicted as Annual Crop Land, although that class makes up only 11% of the test set (Fig. 25).

The effect is also backbone-dependent (Table 29, mean ± std over three seeds, each rebuilding the graph):

BackboneMethodAccuracyECE
ViT-B/16TCA45.82 ± 0.108.39 ± 0.05
ARGTCA-DIV32.15 ± 1.4215.92 ± 1.04
ARGTCA-DISC38.02 ± 1.2319.88 ± 1.63
ResNet-50TCA31.38 ± 0.065.63 ± 0.17
ARGTCA-DIV28.52 ± 2.654.83 ± 0.94
ARGTCA-DISC24.12 ± 1.6010.55 ± 2.13

On ResNet-50, ARGTCA-DIV matches TCA within seed variation, with a lower mean ECE, whereas ARGTCA-DISC still degrades. The EuroSAT failure is therefore not an unavoidable consequence of graph-based selection.

To quantify the role of cross-class overlap, we varied the fraction of retained inter-class edges from 0% to 100%, keeping the intra-class edges unchanged (Table 30; ECE at 0% → 100% retention, slope per 100% with 95% CI, three seeds):

DatasetVariantECE (0% → 100%)Slope [95% CI]
Caltech101DIV2.25 → 2.70+0.25 [−0.37, +0.87]
DISC2.06 → 1.94−0.12 [−0.71, +0.47]
OxfordPetsDIV4.05 → 3.55−0.42 [−0.87, +0.03]
DISC3.18 → 3.16+0.03 [−0.66, +0.72]

Varying the retention changes the selected attribute pair for 54–98% of classes, so these edges strongly affect which attributes are selected, but all 95% confidence intervals of the ECE slope include zero. On EuroSAT the trend is likewise not significant (slopes −2.10 for DIV and −5.20 for DISC, both CIs including zero). Removing all graph edges, both intra- and inter-class, reduces ARGTCA-DIV's EuroSAT ECE from 15.92 to 10.83. Across these configurations, EuroSAT ECE tracks the degree of prediction collapse: the share of samples assigned to the dominant class correlates with ECE at r=0.83r=0.83.

Overall, the main cause is EuroSAT's generic candidate vocabulary, not the graph itself. ARGTCA-DIV's diversity objective favors attributes unlike a class's other candidates, which on EuroSAT are the generic ones. ARGTCA-DISC is limited by EuroSAT's crowded class names in CLIP's text space (Appendix L.8). Generic attributes match images of many classes, so predictions collapse onto one class, test-time tuning makes them confident, and ECE rises with the collapse. The graph amplifies the failure for ARGTCA-DIV: removing all edges lowers its ECE, but not to TCA's level. It is not the cause, since varying the inter-class edges has no significant effect. We state this as a limitation in Appendix O: ARGTCA assumes a candidate pool that contains class-specific attributes, and EuroSAT's does not.

(Part 3 of 4)

Standard deviations and statistical significance

Table 1 reports means over three random seeds (ImageNet: one seed, marked †). We kept standard deviations out of Table 1 for space and report them per dataset in the new Table 9 (Appendix A.4). The table below lists ECE (%, ViT-B/16): ARGTCA as mean ± std over three seeds, TCA as reported in Table 1. ↓/↑ marks a significantly lower/higher ECE than TCA (one-sample tt-test of our three seeds against TCA's value, α=0.05\alpha=0.05).

DatasetTCAARGTCA-DIVARGTCA-DISC
Caltech1013.012.70 ± 0.431.94 ± 0.50
OxfordPets6.303.55 ± 0.09 ↓3.16 ± 0.33 ↓
StanfordCars7.852.83 ± 0.05 ↓4.04 ± 0.18 ↓
Flowers1023.763.63 ± 0.553.33 ± 0.18
Food1015.281.85 ± 0.11 ↓2.03 ± 0.29 ↓
Aircraft3.605.10 ± 0.39 ↑4.45 ± 0.22 ↑
DTD5.484.22 ± 0.39 ↓6.62 ± 0.56
EuroSAT8.3715.80 ± 0.11 ↑19.84 ± 0.18 ↑
UCF1012.822.85 ± 0.133.85 ± 0.05 ↑

ARGTCA-DIV significantly lowers ECE on 4 of the 9 datasets and ARGTCA-DISC on 3. Both are significantly worse on Aircraft and EuroSAT, and ARGTCA-DISC also on UCF101. The significant gains occur where TCA is most miscalibrated (e.g., StanfordCars 7.85 → 2.83, Food101 5.28 → 1.85), so the improvement is dataset-dependent rather than uniform. The standard deviation of ECE is at most 0.56 on every dataset.

Comments on the paper

1 comment

Dear Reviewers,

The authors have submitted their responses to your comments. Please review their rebuttals and engage in further discussion, particularly regarding any remaining concerns, unresolved issues, or points requiring additional clarification.

Thank you for your time and continued contributions to the review process.

Regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.