- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a graph-guided test-time calibration method for VLM models. ArgTCA constructs a LLM-based symbolic asymmetric graph (SAG) for class-attribute pairs. The intra-class connects attribute from the same class, and the inter-class connects identical attributes across classes. The graph attention training then uses supervised contrastive learning to produce relation-aware attribute embeddings. ArgTCA-DIV and or ArgTCA-DISC use the embedding to select attributes (either diversity within each class, or discrimination from other classes). The selected attribute is then used in the TCA soft-prompt adaptation process.
The main contribution of the paper is that it addresses the intra-class redundancy and inter-class ambiguity issues in LLM-generated attributes and provides an intuitive graph-based learning for extracting relation-aware attribute embeddings.
The experimental results are convincing: ArgTCA consistently improves Expected Calibration Error over TCA and several prompt-tuning baselines across multiple datasets and two CLIP backbones.
Critical strengths
-
Clear motivation and easy-to-follow: The paper provides two main limitations of current attribute-based calibration method: intra-class attributes redundancy, and inter-class attribute ambiguity.
-
Graph-based design is intuitive and explainable: Representing each class–attribute pair as a node and modeling intra-class and inter-class relations provides a natural way to incorporate relational structure that independent attribute ranking ignores.
-
Useful ablation studies: The paper provides several analyses, including edge removal, no-GAT variants, different graph encoders, and intra-/inter-edge contributions, and random selection, which help isolate the benefit of the graph-based design.
-
Efficient during test-time: All the graph construction, GAT training, and attribute selection are performed offline, while inference reuses the original TCA pipeline, making the approach practical at test time.
Critical weaknesses
-
Attribute grounding in CLIP is assumed rather than verified. The paper assumes that CLIP can recognize the selected attributes. However, some attributes may be linguistically meaningful but weakly grounded or not observable in a given image. How can you verify that all attributes are valid for a certain model?
-
Why are DIV and DISC separated rather than jointly optimized? It is unclear why these two objectives are treated as separate variants rather than combined into a unified selection criterion.
-
Calibration-accuracy tradeoff is not fully discussed. In Table 1, with ResNet-50, ArgTCA improves ECE while reducing top-1 accuracy, TCA: Acc=58.58, ECE=4.11; ArgTCA-DIV: Acc=57.70, ECE=3.66. It seems that ArgTCA improves the model's confidence calibration, but in some cases, it also makes the model less accurate at classification. The author should discuss this more.
Questions for authors
see weakness
5 comments
(Part 2 of 3)
Why are DIV and DISC separate variants?
They target different failure modes (intra-class redundancy vs. inter-class ambiguity) and pull the selection in opposite directions. After graph training, ARGTCA-DISC's picks move toward class-specific attributes, while ARGTCA-DIV's move toward generic ones: on Caltech101 the mean genericity of the selected attributes falls from 2.15 to 1.33 for ARGTCA-DISC and rises from 2.80 to 5.25 for ARGTCA-DIV (Appendix I). Their margins also scale differently with the number of classes: the discriminability margin concentrates as (Corollary 1) and the diversity margin does not, so a fixed weighting of the two scores would shift with the label-space size.
During development we also tried three ways of combining the two criteria in a single rule: an unscaled sum of the pair scores, greedy maximal marginal relevance (MMR), and a product of the two scores. ECE (%, ViT-B/16; combined rules single seed, single criteria three-seed means of Table 1; new Appendix L.10, Table 27):
| Rule | Caltech101 | DTD |
|---|---|---|
| Sum | 3.17 | 4.56 |
| MMR | 1.84 | 4.33 |
| Product | 1.74 | 5.20 |
| ARGTCA-DIV | 2.70 | 4.22 |
| ARGTCA-DISC | 1.94 | 6.62 |
None consistently beats the better single criterion. On DTD all three have higher ECE than ARGTCA-DIV; on Caltech101 the MMR and product rules stay within the seed spread of ARGTCA-DISC (). MMR, which we ran on nine datasets, averages 5.66 ECE, against 4.73 for ARGTCA-DIV and 5.47 for ARGTCA-DISC on the same datasets. We therefore report the two criteria as separate variants, recommend ARGTCA-DIV as the calibration-oriented default, and list a principled joint objective as future work.
(Part 3 of 3)
Accuracy–calibration trade-off
We agree that this deserves explicit discussion. ARGTCA selects attributes to improve confidence reliability, and its accuracy cost is small and depends on the backbone and variant.
The ResNet-50 averages quoted in the review (TCA 58.58 / 4.11) included a transcription error in TCA's ImageNet entry, which is now corrected (see our reply to Reviewer 7kSv). With the corrected entry, TCA's ResNet-50 average is 57.64 / 4.10, and ARGTCA-DIV (57.70 / 3.66) is no longer below TCA in accuracy. Average change relative to TCA (Appendix Q):
| ViT-B/16: ΔAcc / ΔECE | ResNet-50: ΔAcc / ΔECE | |
|---|---|---|
| ARGTCA-DIV, all 10 datasets | −0.57 / −0.41 | +0.06 / −0.44 |
| ARGTCA-DISC, all 10 datasets | +0.09 / +0.28 | −0.78 / −0.11 |
| ARGTCA-DIV, excl. EuroSAT | +0.79 / −1.29 | +0.30 / −0.32 |
| ARGTCA-DISC, excl. EuroSAT | +0.79 / −0.96 | −0.10 / −0.44 |
Small trade-offs remain for ARGTCA-DIV on ViT-B/16 and for ARGTCA-DISC on ResNet-50, and EuroSAT drives most of them. Excluding EuroSAT, both variants gain accuracy on ViT-B/16 and ECE improves in all four cases. We added this discussion, with a per-dataset plot (Figure 27), as the new Appendix Q.
We sincerely thank the reviewer again for the constructive feedback. We hope that the additional analyses address the concerns raised, and we would be happy to provide any further clarification. If the concerns have been satisfactorily addressed, we would be grateful if the reviewer could kindly reconsider the evaluation score.
(Part 1 of 3)
We thank the reviewer for the positive assessment of our motivation, the explainable graph design, the ablation studies, and the test-time efficiency of the method, and for the thoughtful questions.
Is attribute grounding in CLIP verified?
We agree that grounding should be measured rather than assumed. For each candidate attribute we compute the AUROC of CLIP's score for "a photo of a {attribute}." at separating the class's test images from all others (0.5: not visible; 1: perfectly visible). Labels are used only for this offline diagnostic, never by the method. Mean over nine datasets (Table 31):
ARGTCA-DISC's selections are about as visible as TCA's on both backbones. We also planted unrelated atttributes in 10% or 30% of each class's candidate pool, rebuilt the graph and reran the full method (ViT-B/16, three seeds). Percentage of selected attributes that are planted atttributes (Table 32):
At 30% pollution ARGTCA-DISC selects planted atttributes only 5.8% of the time on average (chance: 30%), and its DTD results stay close to the clean ones (Acc./ECE 46.73/6.62 → 46.61/7.06). ARGTCA-DIV is vulnerable because maximizing the distance between a class's two atttributes favors atttributes unlike its other attributes, and unrelated atttributes fit that description. A label-free text-relevance filter applied before selection reduces ARGTCA-DIV's planted-atttribute rate to 7.7% at 30% pollution; it uses no images or labels, so it can be applied to any new model or attribute list. Changes in visibility are not significantly correlated with changes in ECE (Spearman, n=9), so grounding acts as a robustness condition rather than a predictor of improvement (new Appendix P).