acceptodds
Withdrawn from ICLR 2027

Not All Attributes Compose Equally: Family-Aware Structured Calibration for Compositional Zero-Shot Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Compositional zero-shot learning (CZSL) aims to recognize unseen attribute-object compositions by recombining known primitives. Existing methods mainly improve representations before pair scores are produced, but the final ranking usually does not explicitly use relations among attributes. However, attributes do not compose equally: semantically related attributes share family-level visual evidence, while their appearance and compatibility remain object dependent. We study the complementary problem of post-hoc pair calibration and introduce FASC (Family-Aware Structured Calibration), which preserves the backbone candidate space while using semantic attribute families as intermediate structure. FASC derives an image-specific family posterior by marginalizing the softmax distribution over averaged candidate-pair logits, then combines the original logits with training-frequency priors, a ranked family bias, a centered family-prior correction, and a pseudo-unseen ranking objective. On C-GQA, MIT-States, and UT-Zappos, FASC improves closed-world AUC over the same-source uncalibrated scores and exhibits low variation across five calibrator seeds. Fixed-source ensemble, component, and taxonomy ablations isolate the contribution of semantic family structure, while evaluations on two additional CZSL backbones demonstrate transfer beyond the main backbone.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Compositional Zero-Shot Learning, Multimodality, Image-Text Matching

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_Y66SOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FASC and investigates whether, after a VLM backbone has already produced composition scores, semantic relationships among attributes can further improve compositional zero-shot recognition. The method organizes attributes into fixed semantic families, aggregates the probability mass of candidate compositions to obtain image-dependent family evidence, and then combines this evidence with frequency priors to rerank the original candidates. A pseudo-unseen ranking objective is used to encourage a more favorable seen/unseen trade-off. Therefore, the contribution of this paper lies in structurally interpreting and adjusting existing scores, rather than proposing a new compositional representation. The closed-world and open-world results and the controlled experiments provide fairly credible evidence that this calibration procedure has practical value.

Critical strengths
  1. The use of intermediate semantic structure is well motivated. The paper recognizes that attribute prediction is heterogeneous. The reliability of color, physical state, and material evidence may differ, and uncertainty at the composition level may coexist with more consistent evidence at a coarser semantic level. Treating families as an intermediate structure while retaining predictions at the composition level is a reasonable design. It provides a path to exploit shared evidence without replacing the original task with family classification.

  2. The calibration method is practical and complementary. The interface based on logits makes FASC easy to combine with existing CZSL scoring models. The method does not require retraining representations and does not change the candidate vocabulary set, so it has practical value as an additional calibration stage. Experiments on other scoring models support this modular design and show that the method is not limited to the specific implementation of the main backbone.

  3. Using baselines from the same source and matched calibrators that do not use family information controls for two key confounders, checkpoint averaging and generic calibration. Taxonomy controls and repeated fitting further demonstrate attention to attribution. For a pipeline with multiple interacting components, such an experimental design is informative.

Critical weaknesses
  1. At the evidence level, the core question is whether semantic grouping can further improve recognition after the benefits of ensembling and generic score calibration have already been taken into account. When the source logits and non-family calibration components are held consistent, the remaining gain from family structure is limited. For example, on C-GQA the AUC difference relative to Generic is only 0.04. Therefore, uncertainty in paired comparisons is critical. Repeatedly fitting the calibrator after fixing the source subset and taxonomy can only establish fitting stability. The core claim also requires evidence that the gains relative to matched alternatives are similarly stable. We recommend that the authors add paired repeated experiments for family and non-family variants and use multiple random taxonomies to directly evaluate the semantic effect itself. This would help determine whether family structure provides reproducible useful information or whether the current evidence mainly supports the practical gains from combining existing scores with calibration terms.

  2. Family marginalization has a clear mathematical definition, but the resulting quantity mixes visual evidence with the influence of candidate space structure. Even when candidate logits are exactly the same, larger families receive more probability mass. Therefore, the posterior may favor a family because it contains more candidates or inherits object score biases, not because the image provides stronger evidence for that family's semantic cues. Keeping family size in the random control is useful, but it is not sufficient to establish that this aggregation statistic actually reflects the intended visual signal. We suggest that the authors compare with statistics adjusted for candidate count and add family prediction diagnostics as well as an analysis split by whether object recognition is correct. This would make the mechanism more convincing.

  3. We suggest that the authors conduct matched open-world comparisons on the same source logits, with particular attention to differences in performance when object recognition is correct versus incorrect. They should also explain how feasibility filtering interacts with family marginalization, since it changes the evidence that the calibrator receives. The additional backbone experiments establish interface compatibility, but they are limited to the closed-world setting on a single dataset and do not yet fully support broader generalization claims. These analyses would identify when semantic aggregation can improve compositional recognition and when it may amplify ambiguity.

  4. The writing of the paper needs a comprehensive revision so that readers can better understand the method. For example, the typesetting of Eq. 8 requires the authors to conduct a complete check of the paper.

Questions for authors

Please address the above issues.

Reviewer_JDEbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FASC, a post-hoc calibration framework for compositional zero-shot learning (CZSL) that leverages semantic attribute families to improve recognition. It combines image-dependent family posteriors, frequency priors, and family-level score corrections to recalibrate composition predictions, alongside a pseudo-unseen ranking objective to enhance generalization. Experiments on three CZSL benchmarks demonstrate improved closed-world performance, supported by ablation and transferability studies.

Critical strengths
  1. The paper addresses a limitation of existing CZSL methods by incorporating semantic attribute relationships into composition ranking, offering a reasonable calibration strategy without modifying visual representations.
  2. FASC operates directly on candidate-pair logits without requiring additional backbone training or modifications to the original representation learning pipeline.
  3. The paper provides comprehensive controlled experiments, including calibration baselines, random-taxonomy controls, and multiple training seeds, helping validate the contribution of semantic families and the method's stability.
Critical weaknesses
  1. The centered family-prior correction in Eq. 10 appears mathematically redundant for ranking. Since the subtracted mean is identical for all candidate compositions of the same image, it cancels out in pairwise score differences and softmax normalization. Consequently, the centering operation does not affect the final ranking or cross-entropy optimization.
  2. The methodological novelty of FASC is limited, as it mainly combines established calibration, hierarchical aggregation, and ensemble techniques. Although semantic family-based calibration is reasonable, the paper does not sufficiently differentiate its contributions from prior methods such as ProLT[1] and LOGICZSL[2].
  3. Although FASC achieves competitive results, its gains over generic calibration are marginal, with closed-world AUC improvements of only 0.04, 0.23, and 0.23 on C-GQA, MIT-States, and UT-Zappos, respectively. These limited gains raise concerns about whether the added semantic modeling complexity is justified.
  4. Despite consistent closed-world improvements, FASC shows limited or negative gains in open-world settings, with HM decreasing on C-GQA and both HM and AUC declining on UT-Zappos. These results raise concerns about its robustness against semantically plausible incorrect compositions, a limitation insufficiently addressed in the paper.
  5. The family calibration pipeline is inconsistent across datasets, as UT-Zappos uses a 16-channel attribute-level cache rather than the six semantic families. This raises concerns about whether the improvements stem from semantic family aggregation or attribute-specific calibration. Moreover, the contribution of auxiliary family evidence is not sufficiently isolated through ablation studies.

[1] Revealing the Proximate Long-Tail Distribution in Compositional Zero-Shot Learning, AAAI 2024. [2] LOGICZSL: Exploring Logic-Induced Representation for Compositional Zero-Shot Learning, CVPR 2025.

Questions for authors

Please see the Weaknesses.

Reviewer_uARSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FASC, a post-hoc calibration method for compositional zero-shot learning (CZSL). The method groups attributes into semantic families, derives image-dependent family evidence by aggregating candidate composition probabilities, and combines this evidence with frequency priors and a pseudo-unseen ranking objective to recalibrate existing composition scores. Experiments on three CZSL benchmarks show improvements in closed-world evaluation, while open-world results are less consistent. Overall, the method provides a practical way to adjust existing predictions, but its contribution mainly lies in score calibration rather than learning new compositional representations.

Critical strengths

The motivation for introducing semantic attribute families is reasonable. Different attributes exhibit heterogeneous visual characteristics, and exploiting their semantic relationships may help resolve ambiguities in composition predictions. Using families as intermediate calibration structures while retaining pair-level predictions is a sensible design.

Critical weaknesses
  1. The independent contribution of semantic family information is relatively limited. Although the overall pipeline improves closed-world performance, Table 3 shows that the improvement over the matched Generic calibrator is only 0.04 AUC on C-GQA and 0.23 on MIT-States and UT-Zappos. These results raise questions about how much of the improvement actually comes from semantic family information rather than generic score calibration. Moreover, the five-seed experiments only demonstrate the stability of calibrator fitting, not the stability of the incremental gains over non-family alternatives. I suggest paired repeated experiments comparing family and non-family variants, together with multiple random taxonomies, to better establish the contribution of semantic structure.

  2. The family posterior may be strongly influenced by the candidate-space distribution rather than genuine visual evidence. In Eq. (5), family evidence is obtained by summing the probabilities of all candidate compositions belonging to a family. However, larger families naturally accumulate more probability mass, even when candidate logits are identical. Object-level scoring biases may further affect this aggregation. Consequently, the resulting posterior does not necessarily reflect stronger image evidence for a particular semantic family. The authors should compare against candidate-count-normalized aggregation and provide quantitative diagnostics of family prediction quality. An analysis conditioned on object recognition correctness would also help clarify the underlying mechanism.

  3. The effectiveness of FASC in open-world settings remains insufficiently established. In Table 1, FASC slightly decreases HM on C-GQA and reduces both HM and AUC on UT-Zappos. Since the family posterior depends on the candidate distribution, expanding the candidate space may substantially change the aggregated evidence. I suggest conducting matched open-world comparisons using identical source logits and analyzing how feasibility filtering affects family marginalization. In addition, the transfer experiments are restricted to closed-world C-GQA, which provides limited evidence for broader generalization.

  4. The semantic family construction and presentation require further clarification. The appendix indicates that UT-Zappos uses a legacy 16-channel cache in the reported experiments, whereas a separate six-family semantic taxonomy is also provided. This raises questions about whether the same semantic grouping mechanism is consistently evaluated across datasets. Furthermore, the mathematical presentation needs improvement. For example, Eq. (8) has noticeable typesetting issues, the auxiliary family posterior is insufficiently explained in the main text, and the handling of attributes without family assignments is not explicitly defined in the relevant equations. These issues make the method unnecessarily difficult to follow.

Questions for authors

Please address the above concerns, particularly the independent contribution of semantic family information, the interpretation of family-level evidence, and the consistency of the experimental setup.

Reviewer_AwfUOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FASC ( family aware structured calibration ) for compositional zero shot learning. FASC uses an offline attribute taxonomy to compute an image specific family posterior from the softmax distribution over averaged candidate pair logits. Then the calibrator combines the original logits with training frequency priors, a ranked family bias, and a centered family prior correction. It can also use an auxiliary family cache. The authors train the calibrator with top-K cross entropy and a pseudo unseen ranking objective.

The paper evaluates FASC on C-GQA, MIT-states, and UT-Zappos. On all three datasets, the paper reports higher closed world AUC after calibrating the same source logits. The open world results are mixed.

Critical strengths
  1. FASC serves as a potentially useful addition to existing CZSL scorers, although the timing excludes the multi checkpoint backbone inference. It uses the family posterior as an intermediate statistic to adjust the candidate pair scores without modifying the backbone. The paper also reports the additional scoring cost once backbone logits are available.

  2. Based on the experiments, it seems that the calibration improves performance beyond checkpoint averaging (closed world AUC gains of 0.24, 0.41, and 0.31 points over identical source logits on C-GQA, MIT-States, and UT-Zappos, along with an improvement from 12.41 to 13.10 on a single Troika checkpoint without ensembling).

These results show that the calibration method can also improve scores from another backbone.

Critical weaknesses

So, in table 3, the full family dependent block improves over generic by 0.04, 0.23, and 0.23 AUC points, which seems fine. But table 5 reports variability only for full FASC and not for the paired improvement over generic.

Also, the paper introduces an independently computed auxiliary family cache without specifying its construction or which reported experiments use it. This matters because the paper’s distinctive claim concerns semantic family evidence derived from existing candidate pair logits, whereas the full calibrator can also use auxiliary predictions and family dependent priors.

The UT-Zappos experiment does not validate aggregation across semantically related attributes. In appendix I, the authors state that the experiment uses 16 channels, one per material attribute, rather than the six family semantic taxonomy. Under this mapping, eq. 5 reduces to an individual attribute marginal. Also, the paper describes the UT Zappos control as testing cache to family alignment, while the authors have presented table 4 more broadly as evidence for semantic family grouping. So, this limits the evidence for the central grouping mechanism across all three benchmarks. So the authors should explain what changes in the random taxonomy control. Evaluating the six-family taxonomy would directly test the grouping claim; alternatively, the authors should also distinguish the attribute level calibration on UT Zappos from semantic family aggregation on C-GQA and MIT-States.

Questions for authors
  1. How do the authors compute the auxiliary family posterior qaux(f∣x)q^{\mathrm{aux}}(f\mid x)? What inputs and supervision does it use? Which experiments use it? Could the authors report the results for the four modes in Table 11 with the same source logits as well?

  2. What does the UT-Zappos random-taxonomy control in Table 4 permute, and does the control remap the auxiliary channels and family-indexed statistics to match the permutation? Do any of the reported UT-Zappos experiments use the six-family taxonomy?

  3. Also, table 3 reports a UT-Zappos AUC of 48.45 without the pseudo-unseen objective. Table 12 reports 47.47 at zero objective weight. Which settings differ between these runs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.