FACT: Factorized Attribute Conditioning for Compositional Zero-Shot Learning

Compositional Zero-Shot Learning (CZSL) recognizes unseen attribute-object compositions, given training that covers all primitives but only some of their pairings. Its central difficulty is the context-dependence of attributes, which manifest differently across objects and make attribute recognition the bottleneck. In CLIP-based methods, causal attention leaves the attribute token object-invariant, so every composition receives the same context-free attribute signal, and prior methods compensate visually by compressing an attribute's diverse appearances into one composition-invariant feature. We instead adapt the text side with FACT (Factorized Attribute Conditioning for prompt Tuning), which casts the per-composition offsets as a correction field over the composition space and solves it by a low-rank Canonical Polyadic (CP) decomposition into per-primitive factors, giving every composition, seen or unseen, its own offset at negligible cost. As these factors are fitted only by the seen-composition loss, we add Centripetal Regularization, a global-interaction regularizer against overfitting those pairings, and Compositional Smoothing, which exempts the ground truth's primitive-sharing neighbors from one-hot suppression. As a text-side plug-in, FACT improves two bases and achieves the best AUC on three benchmarks under both closed- and open-world settings.