- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes FACT for CLIP-based compositional zero-shot learning. FACT uses a low-rank CP parameterization to combine attribute and object factors into a composition-specific offset, which is added to the attribute token before text encoding. Offsets for unseen compositions are generated by recombining factors learned for their constituent primitives.
The method additionally introduces Centripetal Regularization, which couples primitive factors through averages of the opposing factors in the primitive branches, and Compositional Smoothing, which allocates part of the target probability mass to seen compositions sharing an attribute or object with the ground truth. Experiments evaluate FACT on top of Troika and CLEAR across four datasets under closed-world and open-world settings.
Critical strengths
-
A simple design with practical applicability. FACT generates composition-specific offsets from shared attribute factors, object factors, and a projection, without storing independent parameters for every composition. The same construction can generate offsets for unseen pairs. Its evaluation on two base models supports its potential applicability as a text-prompt adaptation module.
-
Broad evaluation with useful initial ablations. The paper evaluates four datasets, two base models, and both closed-world and open-world settings. Adding FACT improves AUC in all eight reported closed-world base-model–dataset combinations. Ablations on interaction forms, factor rank, and aggregation strategies, together with random-factor and non-interaction bias controls, provide initial empirical support for the design choices.
Critical weaknesses
- The central bottleneck diagnosis and the proposed explanation of the gains are insufficiently validated.
The motivation relies heavily on the observation that CLIP's causal attention prevents the attribute token from attending to the subsequent object token. However, this limitation of an intermediate representation does not establish a corresponding deficiency in the final composition representation used for image–text matching. The object token can attend to the attribute, and the final EOT representation can aggregate information from both. The paper does not directly demonstrate that the attribute-token limitation produces the claimed attribute-recognition bottleneck in the final representation.
Because FACT's offset depends jointly on the attribute and object, the improvements could also arise from the additional capacity for composition-conditioned adaptation. The existing interaction-form ablations do not isolate the benefit of injecting the offset specifically into the attribute token. A comparison that injects an analogous offset into the object token or the final text representation, with comparable parameter counts and identical training settings, would directly inform the proposed explanation.
The unseen-composition results also vary across base models. In Table 2, adding FACT to CLEAR on closed-world C-GQA changes unseen accuracy from 41.9 to 41.8 while increasing AUC from 16.6 to 17.1. On the same dataset, Troika's unseen accuracy increases from 35.7 to 41.0. These endpoint accuracies cannot identify the source of the AUC improvements. The full seen–unseen curves and targeted ablations would help explain how the proposed mechanism relates to these different outcomes.
The contribution should also be situated relative to Attributes as Operators (Nagarajan and Grauman, ECCV 2018). That work models attributes as operators acting on object representations, whereas Appendix B of FACT constructs object-indexed operators acting on attribute representations. Discussing their relationship and differences in operator direction, parameter sharing, and generalization assumptions would clarify the contribution.
- Incomplete configuration and statistical reporting limit the attribution and reliability of the experimental gains.
The implementation uses Text LoRA, but it is unclear whether each base model and its FACT counterpart use identical LoRA configurations, training procedures, and model-selection criteria. The paper also does not clearly identify which baseline results were reproduced and which were taken from prior publications. These details are necessary to determine how much of the improvement can be attributed to FACT.
Tables 3, 5, and 6 report an AUC of 41.5 for the full configuration, while the Hadamard interaction with CS and CR in Table 4 reports 41.1 and appears to represent the same configuration. The source of this difference is not explained. Several ablation differences are small, yet the number of repeated training runs and their variability are not reported, making the stability of the resulting comparisons difficult to assess.
Reproduction is further limited by missing details, including the regularization coefficient, gate initialization, training duration, batch size, model-selection rules, and open-world feasibility filtering and calibration. I could not find a code repository link in the submission materials available to me, which further limits independent verification.
Questions for authors
-
What evidence connects the attribute token's inability to attend to the object with a deficiency in the final composition representation? Could you provide a controlled comparison that injects the offset into either the object token or the final text representation, using comparable parameter counts and identical training settings?
-
Do Troika, CLEAR, and their FACT counterparts use identical Text LoRA, training, and model-selection configurations? Please identify which results are reproduced and which are taken from prior publications, and provide the missing training and open-world evaluation details. If configurations differ, please clarify their implications for attributing the improvements to FACT.
-
Numerical differences and stability. Which configurations or runs produced the 41.1 AUC in Table 4 and the 41.5 AUC in Tables 3, 5, and 6? Please report any existing repeated-run statistics, or provide repeated runs for a small set of comparisons supporting the main conclusions, to clarify the stability of the reported differences.
-
The role of CR. CR establishes global coupling through averages of the opposing factors, but its projections W_a and W_o are independent of the composition projection W_c. How does this coupling at the level of the mean improve interactions for specific unseen compositions? A control replacing CR with independently learned biases while retaining the primitive-branch supervision would help distinguish the contribution of factor coupling from additional adaptation capacity in those branches.