acceptodds
Under review as a conference paper at ICLR 2027

POSS: Learning from Partially Observed Object States for Compositional Zero-Shot Learning

Abstract

Compositional zero-shot learning (CZSL) requires visual–semantic knowledge that can be reused in novel state–object combinations. Yet distinguishing seen composition labels need not produce such knowledge: competition between labels can discourage state evidence that remains visually valid. We study this mismatch through the semantic scope of supervision. A composition label specifies some properties of an object, while its classification objective compares it with candidates describing other properties. We introduce Partially Observed State Supervision (POSS), which organizes existing labels into sparse semantic slot–value facts and learns object-conditioned visual evidence only from the facts each annotation reveals. Shared state predictors pool supervision across training compositions, and their evidence contributes to the ranking of both seen and unseen candidates. POSS retains standard single-composition prediction and requires no additional image-level training annotations. Experiments on three benchmarks and multiple CZSL architectures show improvements in closed- and open-world recognition. State-retention diagnostics show improved preservation of visually supported states, while transfer analyses link reusable state evidence to gains on unseen compositions.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Compositional Zero-Shot Learning, Partial State Supervision, Compositional Generalization, Incomplete Supervision, State-Object Composition

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_tYXjOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the issue of coexisting attributes in CZSL. The authors organize state attributes into different semantic slots and introduce slot-specific supervision to improve the reusability of attribute representations. Experiments on three benchmarks demonstrate the effectiveness of the proposed method. However, the main paper lacks more fine-grained evaluation metrics. In addition, the tables contain considerable unused space, making the overall presentation less compact.

Critical strengths

1. Strong motivation and a novel perspective on attribute learning. The paper is well motivated and provides a fresh perspective on handling multiple coexisting attributes in CZSL. By reconsidering the supervision provided by a single composition label, the proposed method offers an interesting approach to learning reusable state representations.

2. Backbone-agnostic design. POSS has a modular and backbone-agnostic design, allowing it to be incorporated into different CZSL architectures.

Critical weaknesses

1. Similar supervision issues also exist at the object level. Similar issues also exist at the object level, as an image may contain multiple objects, and object categories may have hierarchical relationships. However, POSS only addresses state-level supervision without considering these cases. I wonder whether the same approach could also be applied to object-level supervision. If so, I would like to see some experimental results demonstrating its effectiveness.

2. Incomplete evaluation of state and object accuracy. Since the paper primarily focuses on state learning, state accuracy is an important metric. However, the paper lacks a comprehensive of state and object accuracy. In addition, the main results only report HM and AUC, while Seen and Unseen accuracy are presented in the appendix. This makes the results scattered across different tables. I suggest including Seen, Unseen, HM, AUC, as well as state and object accuracy in the main paper for a more comprehensive evaluation.

3. Insufficient discussion of existing multi-attribute CZSL methods. MAC: A Benchmark for Multiple Attributes Compositional Zero-Shot Learning (2024) has already explored multi-attribute learning in CZSL. The paper lacks a sufficient discussion of how POSS differs from MAC.

4. Lack of experiments on recent CZSL backbones. POSS is only integrated into CoOp, DFSP, and Troika, without experiments on more recent backbones such as ClusPro, Logic-Troika, or EVA. Although POSS already outperforms these methods, evaluating it on stronger backbones would better demonstrate its potential.

5. Insufficient discussion of prior object-conditioned state learning methods. Object-conditioned state learning has already been explored in prior works, such as Learning Conditional Attributes for Compositional Zero-Shot Learning (CVPR 2023) and Not Just Object, But State: Compositional Incremental Learning without Forgetting (NeurIPS 2024). However, these works are not cited in Section 4.2, where the authors introduce their object-conditioned state predictor. I would like to know how the proposed approach differs from these existing methods. The authors should also include the corresponding citations in the method section.

Questions for authors

Would independently running the same LLM with the same prompts produce different semantic slot structures? If so, how would these variations affect the downstream performance?

The four-point rating scale makes it difficult to reflect my current assessment accurately. My initial evaluation of this submission is around 2.5. If the authors can adequately address my concerns, I would be willing to raise my rating to 3 or higher.

Reviewer_3nhTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies compositional zero-shot learning (CZSL) from the angle of supervision design. Its claim: distinguishing seen composition labels does not necessarily yield reusable state knowledge — the classification competition (e.g., ranking "wet dog" over "brown dog") can suppress visually valid state evidence (the image may still be brown), tying state responses to the training set's annotation preferences. The authors introduce POSS, which organizes existing single-label annotations into sparse semantic slot-value facts Ω(a) (recovered offline: wet → {(moisture, wet)}), learns object-conditione slot predictors from only the facts each label reveals (partial-state NLL; unmentioned slots marginalized), and contributes reference-centered evidence G = (1/m)Σ log[q(v|x,o)/q̄(v)] to a base composition score via F + γG, with stop-gradient on G in the composition loss so composition gradients do not directly update state parameters. T

Critical strengths
  1. Identifying the mismatch between label-competition and reusable state evidence, formalizing it through slot-value facts, and designing supervision that only uses what each annotation actually reveals is a genuine, non-obvious contribution. The diagnostics (annotated vs. unobserved state accuracy; retention under cross-slot updates) provide direct evidence that standard objectives suppress visually valid evidence, and the reference-centered evidence scoring is an elegant way to convert state support into candidate margins.

  2. Consistent gains across three benchmarks and three architectures (including strong bases DFSP and Troika), with careful ablations isolating each design choice (slot grouping, partial supervision, object conditioning, stop-gradient, fusion vs. auxiliary representation learning), plus human-audited structure recovery, coverage and reference ablations, and mechanism-linked transfer analyses. The plug-in design (no new annotations, standard single-label protocol) gives the framework practical value.

Critical weaknesses

POSS (Troika) is best in all six settings, but the margins over EVA are small in several (UT-Zappos OW AUC 40.8 vs 40.2; MIT-States OW 10.3 vs 9.4), and the base POSS trails EVA on UT-Zappos OW (37.9 vs 40.2). With only 3-seed means and no significance testing in the main tables (bootstrap CIs appear only in Figure 3), it is difficult to tell whether the headline improvements are meaningful beyond noise — especially under the validation-AUC checkpoint selection and seen-unseen calibration sweep.

The observation structure Ω, the heart of POSS , is produced by GPT-5.5 + Gemini 2.5 Pro under a strict prompt/schema contract with manual verification, which limits reproducibility (API availability and version drift), generality (each new dataset requires fresh recovery plus human audit), and the paper's place in the CZSL literature. Relatedly, the paper argues that classification competition suppresses state evidence, but the key comparison (Dense-neg. vs POSS) confounds the supervision scheme with the negative-label regime, and no experiment directly measures the counterfactual state accuracy achievable without composition competition; the stop-gradient and reference-centering design choices are motivated partly by these ablations rather than by a direct mechanism test.

Questions for authors

In several settings the advantage over EVA/ClusPro is 1–3 AUC points (UT-Zappos OW 40.8 vs 40.2; C-GQA OW 6.1 vs 5.6; MIT-States OW 10.3 vs 9.4). Do these differences survive significance testing over seeds or composition bootstrap as in Figure 3? Please report per-seed values and confidence intervals for the Table 1 head-to-head comparisons.

Is there a more direct demonstration that classification competition suppresses state evidence, e.g., training the same state architecture with a flat/dense-negative objective but no composition head (removing competition), or ablating stop-gradient and partial supervision separately from the negative-label regime? The current Dense-neg. vs POSS comparison changes multiple factors at once.

How sensitive are the results to the specific LLM versions and the prompt contract? Since the released consensus mappings are fixed, could the pipeline be reproduced from the mappings alone, and would the method work on a new dataset without human verification of the recovered structure? Table 3 suggests human structure ≈ GPT-5.5 — is the LLM pipeline essential or just convenient?

1 comment
Reviewer_3nhTOpenReview ↗

Thanks for your response.

Thanks for the detailed responses, I have reviewed the responses and I think the responses have addressed my concerns.

Reviewer_QEfVOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a supervision mismatch in compositional zero-shot learning: a single state–object label identifies the classification target but does not describe every visually valid property of the object. The proposed Partially Observed State Supervision (POSS) maps state labels into semantic slot–value facts and trains object-conditioned state predictors only on the facts revealed by each annotation. Their predictions are combined with a base composition score using frequency-referenced evidence, while stop-gradient prevents the composition loss from directly updating the state branch. Experiments on MIT-States, UT-Zappos, and C-GQA show improvements across multiple architectures under closed- and open-world evaluation. Supporting analyses include component ablations, human audits of semantic mappings, a multi-state diagnostic set, and state-retention experiments.

Critical strengths
  1. The paper identifies a meaningful supervision issue and turns it into a coherent method. The distinction between an unobserved property and an absent property is consequential for compositional transfer. POSS incorporates this distinction into both state learning and candidate scoring while retaining the standard single-composition prediction task. The approach also improves multiple composition architectures, suggesting that its value extends beyond one particular scorer.
  2. The proposed explanation is supported by substantial diagnostic evidence. Beyond three benchmarks and three training seeds, the paper provides human verification of recovered mappings and additional visible-state labels on 612 held-out images. POSS improves recognition of unannotated but visible states over the flat auxiliary and dense-negative alternatives (Table 4). The ablations on supervision, stop-gradient, object conditioning, and evidence fusion further support the framework’s design.
Critical weaknesses
  1. The contribution of visual evidence is not fully separated from frequency-based score adjustment. The final score combines learned state probabilities with a candidate-dependent term derived from training frequencies. Table 5 compares several reference distributions, but does not include a control that adds only the frequency-derived term to the base composition score. Such a control would help establish how much of the recognition improvement comes from reusable visual state evidence rather than candidate-prior correction. Similarly, the gradient audit establishes the intended blocking of direct update paths by construction; it does not measure interference through the shared visual adapter. These limitations temper the mechanistic interpretation, although the additional-state diagnostic remains useful supporting evidence.
  2. The strongest benchmark comparisons need clearer experimental alignment and uncertainty reporting. POSS introduces a trainable visual adapter, structured semantic knowledge, and an additional prediction branch. The paper contains useful capacity controls and a matched composition-only comparison on MIT-States, but does not provide an equally explicit matched baseline table for each architecture and dataset. It is also unclear which competing results were reproduced under the same implementation settings. This matters for the strongest performance claims: for example, the closed-world UT-Zappos AUC advantage over EVA is only 0.2 points, while POSS(Troika)’s reported standard deviation is 0.44. Without corresponding baseline variability, this particular advantage should be interpreted cautiously.
Questions for authors
  1. How much improvement remains when the base scorer receives only the frequency-derived candidate correction, without image-dependent state evidence? This would help isolate the contribution of reusable visual evidence.

  2. Which Table 1 baselines were reproduced, and which numbers were taken from prior work? Please provide matched composition-only results for DFSP and Troika with the same visual adapter and training configuration, ideally with seed variability.

  3. Do the state-retention findings persist when the shared visual adapter is updated jointly, as in normal training? The current gradient audit excludes this path, and the controlled retention experiment isolates state-branch updates.

  4. How much do recognition gains differ between covered and uncovered primitives, particularly on C-GQA, where 130 of 413 state primitives have empty mappings? This would clarify the framework’s benefits and limitations under incomplete semantic coverage.

2 comments
Reviewer_QEfVOpenReview ↗

Response to authors' rebuttal

Thanks for your detailed response. My main concerns have been addressed and I would like to maintain my score at 3.

We sincerely thank the reviewer for their positive and constructive feedback. We greatly appreciate their careful consideration of our work and remain happy to provide further clarifications or address any additional concerns they may have.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.