- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1(−1)
- Soundness
- 1
Read review
Paper summary
This paper studies compositional generalization in contrastively trained vision-language models. The authors argue that analyzing each modality separately is not sufficient: both modalities might exhibit additive and orthogonal representations, but the corresponding concepts might still be misaligned across modalities. They study the conditions under which this correspondence can be recovered and introduce ROCA, which fits an orthogonal transformation to align image and text factor directions. They evaluate the geometry of pretrained models and the effect of alignment on retrieval of unseen combinations and attribute-object binding.
Critical strengths
- The topic of compositionality and the aim of understanding the impact of modalities on the representations is interesting.
- The scope of the study is good: it covers both compositionality within objects as well as across objects through binding.
My takeway from this work is that, potentially (since no comparison point exists), the method is akin to LAB-CLIP, a work the authors cite, but by taking a constarinted linear map (orthogonal transformation) as opposed to a general linear map as LAB-CLIP does.
Critical weaknesses
The main weakness of the paper is that it makes big claims but doesn't substantiate them; it provides general propositions that read like counter-examples in the cases where the result is intuitive (eg prop 1). Proposition 1 uses only diagonal training combinations, so the factors are confounded and their correspondence is unidentifiable by construction. Proposition 3 similarly gives standard rank and span conditions for identifiability, while Proposition 4 provides a generic perturbation bound whose connection to the proposed method is not established.
- The authors claim "Object-internal interactions or an equivalent binding mechanism must break this symmetry." But this is clear and has been established in the work the authors cite (arXiv:2605.31503).
- Proposition 2 is a well-known result on, closely related to the orthogonal Procrustes problem. This does not establish why such alignment is necessary for compositional generalization.
- There's no comparison with linear-alignment methods such as LABCLIP. It's an obvious choice since the authors here propose an orthogonal version of it. Even their existing comparisons are confounded: ROCA uses first-order differences, while the linear baselines use orders 1-3, despite the authors showing that higher orders hurt performance.
- The zero-shot results are surprisingly poor, even below random chance on all three datasets (Table 2). The authors do not sufficiently explain these results or provide per-model retrieval results. Even after alignment, the absolute accuracies remain quite low.
- It is also unclear to what extent the improvement comes from the particular alignment of concept directions, as opposed to a general supervised alignment of the two modalities. A baseline aligning centered img-text embeddings would be beneficial (as opposed to their differences).
- The presentation is quite poor. Important definitions and experimental details are scattered throughout the paper and appendix. Even the evaluated models are not identified in the main text. The motivation and relationship between the theoretical results and experiments are often unclear. Prop 1 is described in words only.
I think the work could technically be improved but it would require significant revisions.
Questions for authors
See above
13 comments
Response to Reviewer voKK (Part 2/4)
W4. LABCLIP comparison and matched factor-difference orders
Geometry-preserving adaptation improves matching on all four ARO settings and on What’sUp. We compare ROCA with LABCLIP [1], a general linear adaptation baseline. On What’sUp, PUG-SPAR, UT-Zappos, SPEC, and 3D Shapes, both methods freeze the same OpenAI CLIP ViT-B/32 encoders and share the splits, training pairs, hard negatives, and 50-epoch training budget. Each method uses its own preprocessing and optimization settings. ROCA initializes from matched factor directions via orthogonal Procrustes and then uses contrastive training under . On ARO, we compare the complete adaptation methods using the same source-image splits and test candidates. ROCA uses a margin-based Procrustes initialization constructed from training positives and negatives and optimizes the correct–incorrect score gap under orthogonality. Attribution also uses a training direction-pair constraint, and Relation uses training-set category weights. For held-out binding, we exclude candidates containing the reserved bindings from training and validation and test on positives that contain them, with constituent concepts seen in training. We reserve attribute–object pairs for Attribution and unordered object pairs for Relation. The table reports means ± sample SD over three splits or optimization seeds.
| Benchmark / setting | Metric (%) | LABCLIP | ROCA |
|---|---|---|---|
| ARO Attribution, image holdout | Category macro-accuracy | 78.38 ± 1.24 | 83.82 ± 2.09 |
| ARO Relation, image holdout | Category macro-accuracy | 81.52 ± 1.15 | 85.29 ± 2.08 |
| ARO Attribution, held-out bindings | Category macro-accuracy | 71.64 ± 2.37 | 75.98 ± 1.30 |
| ARO Relation, held-out bindings | Category macro-accuracy | 76.78 ± 7.35 | 81.66 ± 6.18 |
| What’sUp | Individual accuracy, equal weight for A/B | 52.78 ± 2.77 | 68.16 ± 4.54 |
| PUG-SPAR | Strict two-candidate accuracy, four tasks | 96.01 ± 1.20 | 94.71 ± 0.61 |
| UT-Zappos | Generalized CZSL best harmonic mean | 38.50 ± 3.97 | 38.55 ± 2.96 |
| SPEC | Accuracy, equal weight over six tasks and two directions | 52.42 ± 1.95 | 50.97 ± 0.82 |
| 3D Shapes | Unseen-composition Top-1, 40 candidates | 80.34 ± 5.97 | 80.10 ± 3.26 |
ROCA gains 5.44, 3.77, 4.34, and 4.88 percentage points across the four ARO settings and 15.38 points on What’sUp. The mean scores differ by at most 1.45 points on each of the other four benchmarks. For UT-Zappos, the table gives the best HM along the calibration curve, and the test HM at the validation-selected bias is 38.43 ± 2.88%. An orthogonal map in 512 dimensions has 130,816 intrinsic degrees of freedom, approximately half the 262,144 of a general linear map. It preserves the inner products, distances, and angles of the text embeddings used for matching.
Response to Reviewer voKK (Part 2.5/4)
Matched-order comparison. At each order, orthogonal Procrustes, ordinary least squares (OLS), and both Ridge variants fit exactly the same factor-difference rows and share the training support, centering, queries, and candidates. These are closed-form fits evaluated separately from the trained benchmark variants above. For Local I2T Top-1 (%), we average over three splits per encoder and report the mean ± sample SD across eight encoders. The nine orthogonal means are within 0.05 percentage points of their original Table 2 counterparts, and the 1+2+3 OLS means are within 0.01 points. First-order input gives the highest orthogonal mean on all three datasets. Orthogonal Procrustes has the highest mean at each matched order on dSprites and PUG-Animal. On MPI3D, Ridge performs best at all three orders, with 8.36% for training-tuned Ridge at order 1+2 and 4.22% for the corresponding orthogonal fit.
| Dataset | Difference orders | Orthogonal Procrustes | OLS | Ridge, fixed relative coefficient | Ridge, tuned within training |
|---|---|---|---|---|---|
| dSprites | 1 | 23.62 ± 4.16 | 0.01 ± 0.02 | 15.47 ± 9.17 | 17.62 ± 6.75 |
| dSprites | 1+2 | 23.15 ± 4.24 | 0.16 ± 0.14 | 14.51 ± 9.46 | 16.48 ± 6.71 |
| dSprites | 1+2+3 | 23.04 ± 4.47 | 0.23 ± 0.28 | 14.08 ± 8.98 | 15.99 ± 6.46 |
| PUG-Animal | 1 | 3.98 ± 0.92 | 1.02 ± 0.94 | 1.76 ± 1.09 | 2.11 ± 1.41 |
| PUG-Animal | 1+2 | 3.75 ± 0.97 | 1.39 ± 0.96 | 1.78 ± 1.08 | 2.30 ± 1.19 |
| PUG-Animal | 1+2+3 | 3.62 ± 0.98 | 1.38 ± 0.92 | 1.70 ± 1.11 | 2.16 ± 1.18 |
| MPI3D | 1 | 5.98 ± 0.64 | 2.48 ± 2.45 | 8.16 ± 3.68 | 7.96 ± 3.89 |
| MPI3D | 1+2 | 4.22 ± 0.44 | 5.90 ± 2.12 | 8.11 ± 1.45 | 8.36 ± 1.73 |
| MPI3D | 1+2+3 | 3.42 ± 0.45 | 3.47 ± 0.45 | 3.38 ± 0.54 | 3.48 ± 0.40 |
Why Ridge differs from the original Table 2. Reviewer c5KS’s question about separating the effects of regularization and model selection prompted this comparison. The original Ridge used 1+2+3 differences and a fixed absolute coefficient, . The new fits use , where is the training text-difference matrix and is the embedding dimension. We fix or select it using a split within the outer training set. On dSprites, the fixed relative coefficient gives an effective 294–646 times the original value. This stronger regularization suppresses amplification along weakly supported directions and changes the cosine rankings. With the 1+2+3 inputs and evaluation held fixed, switching the regularization protocol raises accuracy from 0.61% to 14.08%. Selecting within training under the same relative-coefficient protocol then raises it to 15.99%.
Response to Reviewer voKK (Part 1/4)
Thank you for recognizing the importance of compositionality and the value of studying it both within objects and across objects through binding. Our central question is how much cross-modal compositional matching can be recovered by adjusting the relative orientation of frozen image and text embeddings while preserving within-modality geometry. We test this directly through unseen-composition retrieval and attribute swaps, and extend the evaluation to held-out bindings in natural images.
W1. Positioning the theory and connecting it to the experiments
We agree with your assessment and apologize that the original presentation did not clearly distinguish established theory from our contribution or explain how it informs the experiments. The revision will organize the theory around within-modality factor structure, cross-modal compatibility, and attribute ownership. We will state the source and assumptions of each result and explain how the corresponding experiments test its implications. We will present Proposition 1 as an illustrative counterexample under confounded training support, Proposition 3 as an application of standard rank and span conditions, and Proposition 4 as a conditional analysis of retrieval margins.
W2. Binding symmetry and its relation to prior work
We agree that the role of concept interactions in breaking attribute-swap symmetry builds on Uselis et al. [2]. We will make this connection explicit in the revision. If a correct caption and its attribute-swapped counterpart have the same frozen text embedding, applying a shared map leaves their embeddings identical. The ability to distinguish the two captions through alignment therefore depends on the encoder having retained a difference relevant to attribute ownership.
The original CLEVR-2D experiment tests whether orthogonal alignment can make that existing difference useful for cross-modal matching. The two candidates contain the same colors and shapes and differ in which object owns each attribute. With the encoders frozen, ROCA-1 raises swap accuracy from 54.9% for the centered identity baseline to 76.0% and widens the score margin between the correct and swapped captions. W4 extends the matching evaluation to external benchmarks, including ARO with held-out bindings, and compares geometry-preserving adaptation with LABCLIP’s general linear adaptation.
W3. Orthogonal compatibility and compositional generalization
We agree that Proposition 2 is a classical compatibility criterion in the orthogonal Procrustes framework [3]. Given matrices DT and DI of semantically matched factor directions, an orthogonal Q satisfying QDT=DI exists exactly when DT⊤DT=DI⊤DI. This is the condition for a zero-residual orthogonal fit to the given directions.
In the experiments, the orthogonal constraint lets us vary the relative orientation of image and text representations while preserving the inner products, distances, and angles of the centered text representations. We compare ROCA with the identity map under the same centering and measure retrieval on held-out compositions. The proposition characterizes exact alignment of the factor directions, and the retrieval experiment measures how much that intervention changes performance on unseen combinations.