acceptodds
Withdrawn from ICLR 2027

FACT: Factorized Attribute Conditioning for Compositional Zero-Shot Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Compositional Zero-Shot Learning (CZSL) recognizes unseen attribute-object compositions, given training that covers all primitives but only some of their pairings. Its central difficulty is the context-dependence of attributes, which manifest differently across objects and make attribute recognition the bottleneck. In CLIP-based methods, causal attention leaves the attribute token object-invariant, so every composition receives the same context-free attribute signal, and prior methods compensate visually by compressing an attribute's diverse appearances into one composition-invariant feature. We instead adapt the text side with FACT (Factorized Attribute Conditioning for prompt Tuning), which casts the per-composition offsets as a correction field over the composition space and solves it by a low-rank Canonical Polyadic (CP) decomposition into per-primitive factors, giving every composition, seen or unseen, its own offset at negligible cost. As these factors are fitted only by the seen-composition loss, we add Centripetal Regularization, a global-interaction regularizer against overfitting those pairings, and Compositional Smoothing, which exempts the ground truth's primitive-sharing neighbors from one-hot suppression. As a text-side plug-in, FACT improves two bases and achieves the best AUC on three benchmarks under both closed- and open-world settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
compositional zero-shot learning, prompt tuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gmjqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FACT for CLIP-based compositional zero-shot learning. FACT uses a low-rank CP parameterization to combine attribute and object factors into a composition-specific offset, which is added to the attribute token before text encoding. Offsets for unseen compositions are generated by recombining factors learned for their constituent primitives.

The method additionally introduces Centripetal Regularization, which couples primitive factors through averages of the opposing factors in the primitive branches, and Compositional Smoothing, which allocates part of the target probability mass to seen compositions sharing an attribute or object with the ground truth. Experiments evaluate FACT on top of Troika and CLEAR across four datasets under closed-world and open-world settings.

Critical strengths
  1. A simple design with practical applicability. FACT generates composition-specific offsets from shared attribute factors, object factors, and a projection, without storing independent parameters for every composition. The same construction can generate offsets for unseen pairs. Its evaluation on two base models supports its potential applicability as a text-prompt adaptation module.

  2. Broad evaluation with useful initial ablations. The paper evaluates four datasets, two base models, and both closed-world and open-world settings. Adding FACT improves AUC in all eight reported closed-world base-model–dataset combinations. Ablations on interaction forms, factor rank, and aggregation strategies, together with random-factor and non-interaction bias controls, provide initial empirical support for the design choices.

Critical weaknesses
  1. The central bottleneck diagnosis and the proposed explanation of the gains are insufficiently validated.

The motivation relies heavily on the observation that CLIP's causal attention prevents the attribute token from attending to the subsequent object token. However, this limitation of an intermediate representation does not establish a corresponding deficiency in the final composition representation used for image–text matching. The object token can attend to the attribute, and the final EOT representation can aggregate information from both. The paper does not directly demonstrate that the attribute-token limitation produces the claimed attribute-recognition bottleneck in the final representation.

Because FACT's offset depends jointly on the attribute and object, the improvements could also arise from the additional capacity for composition-conditioned adaptation. The existing interaction-form ablations do not isolate the benefit of injecting the offset specifically into the attribute token. A comparison that injects an analogous offset into the object token or the final text representation, with comparable parameter counts and identical training settings, would directly inform the proposed explanation.

The unseen-composition results also vary across base models. In Table 2, adding FACT to CLEAR on closed-world C-GQA changes unseen accuracy from 41.9 to 41.8 while increasing AUC from 16.6 to 17.1. On the same dataset, Troika's unseen accuracy increases from 35.7 to 41.0. These endpoint accuracies cannot identify the source of the AUC improvements. The full seen–unseen curves and targeted ablations would help explain how the proposed mechanism relates to these different outcomes.

The contribution should also be situated relative to Attributes as Operators (Nagarajan and Grauman, ECCV 2018). That work models attributes as operators acting on object representations, whereas Appendix B of FACT constructs object-indexed operators acting on attribute representations. Discussing their relationship and differences in operator direction, parameter sharing, and generalization assumptions would clarify the contribution.

  1. Incomplete configuration and statistical reporting limit the attribution and reliability of the experimental gains.

The implementation uses Text LoRA, but it is unclear whether each base model and its FACT counterpart use identical LoRA configurations, training procedures, and model-selection criteria. The paper also does not clearly identify which baseline results were reproduced and which were taken from prior publications. These details are necessary to determine how much of the improvement can be attributed to FACT.

Tables 3, 5, and 6 report an AUC of 41.5 for the full configuration, while the Hadamard interaction with CS and CR in Table 4 reports 41.1 and appears to represent the same configuration. The source of this difference is not explained. Several ablation differences are small, yet the number of repeated training runs and their variability are not reported, making the stability of the resulting comparisons difficult to assess.

Reproduction is further limited by missing details, including the regularization coefficient, gate initialization, training duration, batch size, model-selection rules, and open-world feasibility filtering and calibration. I could not find a code repository link in the submission materials available to me, which further limits independent verification.

Questions for authors
  1. What evidence connects the attribute token's inability to attend to the object with a deficiency in the final composition representation? Could you provide a controlled comparison that injects the offset into either the object token or the final text representation, using comparable parameter counts and identical training settings?

  2. Do Troika, CLEAR, and their FACT counterparts use identical Text LoRA, training, and model-selection configurations? Please identify which results are reproduced and which are taken from prior publications, and provide the missing training and open-world evaluation details. If configurations differ, please clarify their implications for attributing the improvements to FACT.

  3. Numerical differences and stability. Which configurations or runs produced the 41.1 AUC in Table 4 and the 41.5 AUC in Tables 3, 5, and 6? Please report any existing repeated-run statistics, or provide repeated runs for a small set of comparisons supporting the main conclusions, to clarify the stability of the reported differences.

  4. The role of CR. CR establishes global coupling through averages of the opposing factors, but its projections W_a and W_o are independent of the composition projection W_c. How does this coupling at the level of the mean improve interactions for specific unseen compositions? A control replacing CR with independently learned biases while retaining the primitive-branch supervision would help distinguish the contribution of factor coupling from additional adaptation capacity in those branches.

Reviewer_Key2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, this authors propose FACT, a text-side framework for compositional zero-shot learning (CZSL) that models object-dependent attribute semantics through low-rank CP decomposition. It further introduces Centripetal Regularization and Compositional Smoothing to improve generalization. FACT can be integrated into existing CLIP-based methods without modifying their visual branches. Experiments on four benchmarks demonstrate consistent improvements in both closed-world and open-world settings.

Critical strengths
  1. The paper addresses a key limitation of CLIP-based CZSL methods by modeling object-dependent attribute semantics on the text side, offering a reasonable alternative to visual feature disentanglement for improving compositional generalization.
  2. FACT employs factorized conditioning to generate composition-specific offsets efficiently and can be integrated into existing CZSL frameworks without modifying their visual encoders, which is intuitive and reasonable.
  3. The experiments cover four CZSL benchmarks under both closed-world and open-world settings. FACT consistently improves the AUC of two underlying architectures, while the component, interaction-form, and factor-rank ablations provide useful empirical evidence for the proposed design.
Critical weaknesses
  1. The methodological novelty of FACT appears relatively limited. The proposed framework mainly combines established techniques, including low-rank CP decomposition, multiplicative factor interactions, prompt adaptation, and structured label smoothing. Although applying these techniques to composition-specific attribute conditioning is reasonable, the paper does not sufficiently distinguish its core algorithmic contribution from existing factorized interaction models and conditional prompt-learning approaches.
  2. Compositional Smoothing assumes that compositions sharing an attribute or object should receive positive supervision during training. However, primitive sharing does not necessarily imply semantic similarity or visual compatibility. For example, compositions involving the same object but mutually exclusive attributes may represent substantially different visual concepts. Assigning positive probability mass to such compositions could introduce ambiguous or misleading supervision.
  3. The ablation results appear inconsistent across Tables 3 and 4. Table 3 reports an AUC of 41.5 for the complete FACT framework on MIT-States* under the closed-world setting, whereas Table 4 reports an AUC of 41.1 for the Hadamard interaction, which is described as the proposed design with both Centripetal Regularization and Compositional Smoothing enabled.
  4. The paper's central motivation is questionable, as causal attention does not necessarily prevent CLIP from capturing object-conditioned attribute semantics. Later tokens, including EOS, can attend to both attribute and object tokens and model their interactions across Transformer layers.
Questions for authors

Please see the Weaknesses.

Reviewer_jm4SOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

FACT conditions a CLIP attribute prompt on its paired object before text encoding. It generates a composition-specific offset through a CP factorization, allowing unseen pairs to reuse learned primitive factors. The module is evaluated on Troika and CLEAR across four benchmarks under closed- and open-world protocols, with component, interaction, rank, and aggregation ablations.

Critical strengths
  1. The conditioning mechanism is compact and directly compatible with unseen combinations. Equations (2)-(3) generate different offsets for different attribute-object pairs without storing a separate parameter vector per pair.
  2. The reported improvements span two bases and several distinct benchmarks. Tables 1-2 report positive CW AUC gains for both bases on all four datasets. Examples include CLEAR improving from 38.6 to 41.5 on MIT-States⋆ and Troika improving from 12.4 to 16.8 on C-GQA. The component, rank, and aggregation ablations explore meaningful design choices.
  3. Tables and figures are in good illustration.
Critical weaknesses
  1. The evaluation tables and training controls do not yet provide a reliable basis for the main empirical claims. The interpretation of the reported S/U metrics requires clarification.

  2. FACT operates only on the prompt side and leaves the visual pipeline untouched. Thus, some factors may mislead the text representation without considering the visual information.

  3. In the table 2, the state-of-the-art claim on C-GQA should be qualified in light of EVA (Mixture-of-Experts Semantic Variant Alignment for Compositional Zero-Shot Learning), which reports CW AUC/HM of 18.8/36.9, compared with FACT's best reported 17.1/35.3. The difference should be explained.

Questions for authors
  1. Contribution attribution: Do the baseline, FC-only, FC+CS, and full-FACT rows all use identical Text LoRA and other trainable modules? Please give those configurations and compare CS with ordinary label smoothing at the same strength under the same training setup. This comparison is consequential because structured smoothing supplies a substantial part of the cumulative gain, while earlier CZSL work such as TRIDENT, IJCAI 2025 already studies smoothing-based supervision. Equation (6) also requires clarification when either primitive-sharing neighbor set is empty.

  2. Computation cost should be reported about introducing a text-side plug-in.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.