acceptodds
Withdrawn from ICLR 2027

Misplaced, Not Forgotten: Compressed Codes Map Unseen Attribute Combinations to Seen Ones

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Models often have to handle combinations of familiar attributes that never appeared together in training, and they often fail on them. Existing work tests whether composition succeeds and which representation geometry supports it. We study where a failing model's predictions go, in models whose heads for several attributes all read one compressed representation, a shared read-out. Such a model maps an unseen combination onto one specific seen combination, a failure we call capture. The captured combination is predictable before training from the nearest seen mean image (0.76 against 0.07 chance). The relation is causal. Deleting that combination and retraining moves the prediction far more often than retraining alone, usually onto the next-nearest seen combination, and repeated deletion walks down that ordering. This holds on rendered benchmarks and on real camera photographs. The attributes remain decodable, and a composition operator on the same code recovers held-out combinations at 0.86 where the model scores 0.02, so capture is a failure of placement. Reserving part of the compression budget for each attribute does not reduce capture at matched rate. This holds on a controlled task and on frozen DINOv2, SigLIP and CLIP encoders. Specialising the read-out yields clear paired gains in held-out composition in six of seven settings and a positive but inconclusive one on MPI3D. Decoding every attribute on seen combinations does not guarantee composition. We show where these models send unseen combinations when they fail, and how changing the training support changes that destination.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
compositional generalization, representation learning, information bottleneck, disentanglement, causal intervention, shared read-out

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8ZYzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the compositional generalization task, where a model should recognize a new combination of seen attributes. It focuses on the problem that an unseen combination is often mapped onto one specific seen combination. The authors propose a method to predict such failure before training using the mean image, and then moves the prediction by deleting the captured targets. The method is tested on several datasets to show its effectivenss.

Critical strengths
  1. This paper makes the classificaiton failure more predictable with less costs. On dSprites, the nearest seen mean image predicts the captured combination with much higher accuracy than uniform chance. And it only uses the input and training split, without examining the trained model.

  2. The experiments provide evidence that deleting the predicted target and retraining moves the prediction more often than re-seeding or a distance matched deletion.

Critical weaknesses
  1. The paper is hard to follow. The authors use lots of unnecessarily long sentences separated by commas to express a simple meaning, making it difficult to understand.

  2. Missing citation. The method in Atzmon et al., learns separates attribute and object representations by encouraging each representation to be invariant to others. This directly relates to the proposed specialization mechanism.

  3. The explanation of the failure remains incomplete. Proposition 1 explains why the prediction heads select a nearby seen combination. However, it does not explain why the encoder places the unseen input near that combination. This gap matters because the deletion experiment retrains both the encoder and the heads, while the theory takes the encoded representation as given. The experiments show that changing the training support changes the wrong prediction, but the encoder’s role in this process remains unclear.

[1] Atzmon, Y., Kreuk, F., Shalit, U., & Chechik, G. (2020). A causal view of compositional zero-shot recognition. Advances in Neural Information Processing Systems, 33, 1462-1473.

Questions for authors

Can the authors explain the connection and differences between their specialisation mechanism and [1]?

Reviewer_16kdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper seems to study how a model treats unseen attributes. The paper claims that an unseen attribute combination is mapped to one of seen combinations for an encoder-decoder-type model.

Critical strengths

None

Critical weaknesses

Perhaps, I’m not an expert in this area, so my comment should be discounted.

It’s really hard to understand what the paper does. There are a lot of concepts that are not well-defined in the main text. For example, I don’t see what a head means. I also don’t see how a model is trained. It seems to use an encoder-decoder architecture with reparameterization, but the paper does not say how it is trained explicitly, so it’s hard to see the relationship between labels and factors with concepts like heads and codes. Also, the experiments are done with special datasets like dSprites (and a real-image dataset) but I’m not sure how the conclusion generalizes (the authors mention this in the paper, though).

Because of this, I cannot properly evaluate the paper and have to rate it negatively (I rate the paper weak reject because it can still give important insights that I don't understand).

Questions for authors

I would like to know the preconditons in the main text.

Reviewer_ut1FOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The work studies how learned decoders predict unseen combinations of factors whenever all, none, or some of them are known to be present in a latent representation of the support data.

Critical strengths

It is a very interesting problem that deserves more understanding. The experiments seem to be carefully designed and principled in nature, with a solid grounding in information theory.

Critical weaknesses

The paper is poorly written. It exceptionally lacks clarity. It was a very painful read. It is terse, repetitive, confusing, and hard to parse. A lot of jargon is introduced with no proper definition or citation. My research is related to this work, I am familiar with some of the related work, and yet I am not familiar with the jargon used here. For the most part, sentences are grammatically correct and almost parseable but they require substantial effort to get their gist.

Because of my background, I am able to understand what the goal of this work is and how its claims could be demonstrated. Hence, I think I am able to follow more than the general ML researcher, and yet I am very uncertain about what is being presented in this paper.

This paper should have been desk-rejected.

Questions for authors

I do not think any sort of rebuttal will change my scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.