acceptodds
Under review as a conference paper at ICLR 2027

Compatible Concept Geometry for Cross-Modal Compositional Generalization

Abstract

Vision-language dual encoders must transfer familiar concepts to combinations that are absent from training. Existing work relates compositional generalization to additive and orthogonal representations within each modality; whether independently structured image and text factors can be correctly matched on unseen combinations remains unclear. We study this cross-modal gap. We show that even when the two encoders separately exhibit additive and cross-factor orthogonal representations, incorrect factor correspondence can still cause compositional matching to fail, and we characterize the compatibility and identifiability conditions of cross-modal factors. To test this property, we develop order-wise geometric diagnostics and use Relative Orthogonal Concept Alignment (ROCA) as a structure-preserving intervention. Across three compositional datasets and eight pretrained models, text representations are generally closer to first-order additive structure, while corresponding image and text factors retain transferable, structured misalignment; first-order orthogonal alignment consistently improves unseen-combination performance, and compatibility weakens at higher response orders. Controlled experiments further show that first-order factor correspondence alone does not produce attribute-object binding, which requires interaction structure capable of distinguishing factor swaps. These results show that cross-modal compositional generalization is jointly constrained by within-modality factor structure, cross-modal factor compatibility, and concept binding.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
vision-language models, compositional generalization, cross-modal alignment, representation geometry

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_voKKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1(−1)
Soundness
1
Read review
Paper summary

This paper studies compositional generalization in contrastively trained vision-language models. The authors argue that analyzing each modality separately is not sufficient: both modalities might exhibit additive and orthogonal representations, but the corresponding concepts might still be misaligned across modalities. They study the conditions under which this correspondence can be recovered and introduce ROCA, which fits an orthogonal transformation to align image and text factor directions. They evaluate the geometry of pretrained models and the effect of alignment on retrieval of unseen combinations and attribute-object binding.

Critical strengths
  1. The topic of compositionality and the aim of understanding the impact of modalities on the representations is interesting.
  2. The scope of the study is good: it covers both compositionality within objects as well as across objects through binding.

My takeway from this work is that, potentially (since no comparison point exists), the method is akin to LAB-CLIP, a work the authors cite, but by taking a constarinted linear map (orthogonal transformation) as opposed to a general linear map as LAB-CLIP does.

Critical weaknesses

The main weakness of the paper is that it makes big claims but doesn't substantiate them; it provides general propositions that read like counter-examples in the cases where the result is intuitive (eg prop 1). Proposition 1 uses only diagonal training combinations, so the factors are confounded and their correspondence is unidentifiable by construction. Proposition 3 similarly gives standard rank and span conditions for identifiability, while Proposition 4 provides a generic perturbation bound whose connection to the proposed method is not established.

  • The authors claim "Object-internal interactions or an equivalent binding mechanism must break this symmetry." But this is clear and has been established in the work the authors cite (arXiv:2605.31503).
  • Proposition 2 is a well-known result on, closely related to the orthogonal Procrustes problem. This does not establish why such alignment is necessary for compositional generalization.
  • There's no comparison with linear-alignment methods such as LABCLIP. It's an obvious choice since the authors here propose an orthogonal version of it. Even their existing comparisons are confounded: ROCA uses first-order differences, while the linear baselines use orders 1-3, despite the authors showing that higher orders hurt performance.
  • The zero-shot results are surprisingly poor, even below random chance on all three datasets (Table 2). The authors do not sufficiently explain these results or provide per-model retrieval results. Even after alignment, the absolute accuracies remain quite low.
  • It is also unclear to what extent the improvement comes from the particular alignment of concept directions, as opposed to a general supervised alignment of the two modalities. A baseline aligning centered img-text embeddings would be beneficial (as opposed to their differences).
  • The presentation is quite poor. Important definitions and experimental details are scattered throughout the paper and appendix. Even the evaluated models are not identified in the main text. The motivation and relationship between the theoretical results and experiments are often unclear. Prop 1 is described in words only.

I think the work could technically be improved but it would require significant revisions.

Questions for authors

See above

13 comments
AuthorsOpenReview ↗

Response to Reviewer voKK (Part 1/4)

Thank you for recognizing the importance of compositionality and the value of studying it both within objects and across objects through binding. Our central question is how much cross-modal compositional matching can be recovered by adjusting the relative orientation of frozen image and text embeddings while preserving within-modality geometry. We test this directly through unseen-composition retrieval and attribute swaps, and extend the evaluation to held-out bindings in natural images.

W1. Positioning the theory and connecting it to the experiments

We agree with your assessment and apologize that the original presentation did not clearly distinguish established theory from our contribution or explain how it informs the experiments. The revision will organize the theory around within-modality factor structure, cross-modal compatibility, and attribute ownership. We will state the source and assumptions of each result and explain how the corresponding experiments test its implications. We will present Proposition 1 as an illustrative counterexample under confounded training support, Proposition 3 as an application of standard rank and span conditions, and Proposition 4 as a conditional analysis of retrieval margins.

W2. Binding symmetry and its relation to prior work

We agree that the role of concept interactions in breaking attribute-swap symmetry builds on Uselis et al. [2]. We will make this connection explicit in the revision. If a correct caption and its attribute-swapped counterpart have the same frozen text embedding, applying a shared map leaves their embeddings identical. The ability to distinguish the two captions through alignment therefore depends on the encoder having retained a difference relevant to attribute ownership.

The original CLEVR-2D experiment tests whether orthogonal alignment can make that existing difference useful for cross-modal matching. The two candidates contain the same colors and shapes and differ in which object owns each attribute. With the encoders frozen, ROCA-1 raises swap accuracy from 54.9% for the centered identity baseline to 76.0% and widens the score margin between the correct and swapped captions. W4 extends the matching evaluation to external benchmarks, including ARO with held-out bindings, and compares geometry-preserving adaptation with LABCLIP’s general linear adaptation.

W3. Orthogonal compatibility and compositional generalization

We agree that Proposition 2 is a classical compatibility criterion in the orthogonal Procrustes framework [3]. Given matrices DTD_T and DID_I of semantically matched factor directions, an orthogonal QQ satisfying QDT=DIQD_T=D_I exists exactly when DT⊤DT=DI⊤DID_T^\top D_T=D_I^\top D_I. This is the condition for a zero-residual orthogonal fit to the given directions.

In the experiments, the orthogonal constraint lets us vary the relative orientation of image and text representations while preserving the inner products, distances, and angles of the centered text representations. We compare ROCA with the identity map under the same centering and measure retrieval on held-out compositions. The proposition characterizes exact alignment of the factor directions, and the retrieval experiment measures how much that intervention changes performance on unseen combinations.

AuthorsOpenReview ↗

Response to Reviewer voKK (Part 2/4)

W4. LABCLIP comparison and matched factor-difference orders

Geometry-preserving adaptation improves matching on all four ARO settings and on What’sUp. We compare ROCA with LABCLIP [1], a general linear adaptation baseline. On What’sUp, PUG-SPAR, UT-Zappos, SPEC, and 3D Shapes, both methods freeze the same OpenAI CLIP ViT-B/32 encoders and share the splits, training pairs, hard negatives, and 50-epoch training budget. Each method uses its own preprocessing and optimization settings. ROCA initializes QQ from matched factor directions via orthogonal Procrustes and then uses contrastive training under Q⊤Q=IQ^\top Q=I. On ARO, we compare the complete adaptation methods using the same source-image splits and test candidates. ROCA uses a margin-based Procrustes initialization constructed from training positives and negatives and optimizes the correct–incorrect score gap under orthogonality. Attribution also uses a training direction-pair constraint, and Relation uses training-set category weights. For held-out binding, we exclude candidates containing the reserved bindings from training and validation and test on positives that contain them, with constituent concepts seen in training. We reserve attribute–object pairs for Attribution and unordered object pairs for Relation. The table reports means ± sample SD over three splits or optimization seeds.

Benchmark / settingMetric (%)LABCLIPROCA
ARO Attribution, image holdoutCategory macro-accuracy78.38 ± 1.2483.82 ± 2.09
ARO Relation, image holdoutCategory macro-accuracy81.52 ± 1.1585.29 ± 2.08
ARO Attribution, held-out bindingsCategory macro-accuracy71.64 ± 2.3775.98 ± 1.30
ARO Relation, held-out bindingsCategory macro-accuracy76.78 ± 7.3581.66 ± 6.18
What’sUpIndividual accuracy, equal weight for A/B52.78 ± 2.7768.16 ± 4.54
PUG-SPARStrict two-candidate accuracy, four tasks96.01 ± 1.2094.71 ± 0.61
UT-ZapposGeneralized CZSL best harmonic mean38.50 ± 3.9738.55 ± 2.96
SPECAccuracy, equal weight over six tasks and two directions52.42 ± 1.9550.97 ± 0.82
3D ShapesUnseen-composition Top-1, 40 candidates80.34 ± 5.9780.10 ± 3.26

ROCA gains 5.44, 3.77, 4.34, and 4.88 percentage points across the four ARO settings and 15.38 points on What’sUp. The mean scores differ by at most 1.45 points on each of the other four benchmarks. For UT-Zappos, the table gives the best HM along the calibration curve, and the test HM at the validation-selected bias is 38.43 ± 2.88%. An orthogonal map in 512 dimensions has 130,816 intrinsic degrees of freedom, approximately half the 262,144 of a general linear map. It preserves the inner products, distances, and angles of the text embeddings used for matching.

AuthorsOpenReview ↗

Response to Reviewer voKK (Part 2.5/4)

Matched-order comparison. At each order, orthogonal Procrustes, ordinary least squares (OLS), and both Ridge variants fit exactly the same factor-difference rows and share the training support, centering, queries, and candidates. These are closed-form fits evaluated separately from the trained benchmark variants above. For Local I2T Top-1 (%), we average over three splits per encoder and report the mean ± sample SD across eight encoders. The nine orthogonal means are within 0.05 percentage points of their original Table 2 counterparts, and the 1+2+3 OLS means are within 0.01 points. First-order input gives the highest orthogonal mean on all three datasets. Orthogonal Procrustes has the highest mean at each matched order on dSprites and PUG-Animal. On MPI3D, Ridge performs best at all three orders, with 8.36% for training-tuned Ridge at order 1+2 and 4.22% for the corresponding orthogonal fit.

DatasetDifference ordersOrthogonal ProcrustesOLSRidge, fixed relative coefficientRidge, tuned within training
dSprites123.62 ± 4.160.01 ± 0.0215.47 ± 9.1717.62 ± 6.75
dSprites1+223.15 ± 4.240.16 ± 0.1414.51 ± 9.4616.48 ± 6.71
dSprites1+2+323.04 ± 4.470.23 ± 0.2814.08 ± 8.9815.99 ± 6.46
PUG-Animal13.98 ± 0.921.02 ± 0.941.76 ± 1.092.11 ± 1.41
PUG-Animal1+23.75 ± 0.971.39 ± 0.961.78 ± 1.082.30 ± 1.19
PUG-Animal1+2+33.62 ± 0.981.38 ± 0.921.70 ± 1.112.16 ± 1.18
MPI3D15.98 ± 0.642.48 ± 2.458.16 ± 3.687.96 ± 3.89
MPI3D1+24.22 ± 0.445.90 ± 2.128.11 ± 1.458.36 ± 1.73
MPI3D1+2+33.42 ± 0.453.47 ± 0.453.38 ± 0.543.48 ± 0.40

Why Ridge differs from the original Table 2. Reviewer c5KS’s question about separating the effects of regularization and model selection prompted this comparison. The original Ridge used 1+2+3 differences and a fixed absolute coefficient, λ=10−4\lambda=10^{-4}. The new fits use λ=α,tr(T⊤T)/d\lambda=\alpha,\mathrm{tr}(T^\top T)/d, where TT is the training text-difference matrix and dd is the embedding dimension. We fix α=0.01\alpha=0.01 or select it using a split within the outer training set. On dSprites, the fixed relative coefficient gives an effective λ\lambda 294–646 times the original value. This stronger regularization suppresses amplification along weakly supported directions and changes the cosine rankings. With the 1+2+3 inputs and evaluation held fixed, switching the regularization protocol raises accuracy from 0.61% to 14.08%. Selecting α\alpha within training under the same relative-coefficient protocol then raises it to 15.99%.

Reviewer_DHaVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In this paper, the authors analyzed when compositional structure in separately learned image and text representations is sufficient for cross-modal retrieval and proposes ROCA, which fits a rotation using matched finite differences of frozen embeddings. The author's experiments report substantial improvements over unaligned representations on controlled factor datasets and a two-object binding task.

Critical strengths

I think the conceptual question is well motivated. A representation can encode factors within each modality without arranging them in a mutually compatible geometry and by treating these as separate requirements is more informative than attributing every retrieval failure to a lack of compositional information. I like that this paper also makes clear that alignment cannot create binding information absent from the encoders.

Critical weaknesses

The CLEVR binding result is encouraging but I think it remains a restricted two-object task. A matched evaluation on an established binding benchmark, such as an appropriate attribute/relation subset of ARO would help distinguish a generally useful alignment procedure from one that is tailored.

I think this evaluation need not require new encoder training. At minimum, the paper should define the required supervision precisely and report split- and model-level variation for Table 2 instead of only aggregated means.

Questions for authors

Can the authors provide per model/per split retrieval results and one evaluation beyond the current controlled binding setup or explicitly restrict the claim to settings with known factor correspondences?

4 comments
AuthorsOpenReview ↗

Response to Reviewer DHaV (Part 1/3)

Thank you for your careful reading and constructive suggestions on clarifying the supervision, reporting results by encoder and split, and broadening the binding evaluation.

W2. Supervision in the original and new variants, and results by encoder and split

ROCA-1 in Table 2 of the original submission constructs semantically matched first-order factor differences across the image and text modalities using annotated training compositions. It computes an orthogonal map QQ in closed form via Procrustes and applies that map directly at retrieval time. Following another reviewer’s suggestion to compare with LABCLIP, we added two supervised adaptation variants for the new benchmark experiments. The orthogonal contrastive variant initializes QQ with the original closed-form solution and optimizes the map under Q⊤Q=IQ^\top Q=I using the same training positives and hard negatives as LABCLIP. The orthogonal pairwise-ranking variant for ARO uses correct and incorrect captions from the training set for adaptation. The original method and both variants keep the image and text encoders frozen.

StageSupervision and use
Closed-form ROCA-1 and initialization of the contrastive variantFactor or composition annotations identify corresponding first-order changes across the image and text modalities. We fit QQ to the resulting matched differences via orthogonal Procrustes. ROCA-1 uses this solution directly for retrieval, and the contrastive variant uses it as its initial map.
Orthogonal contrastive trainingROCA and LABCLIP share the positive training image–text pairs and hard-negative captions in each corresponding comparison. On controlled datasets, we construct negatives by substituting factors. For external tasks, we use the training negative pools defined by each protocol.
Orthogonal pairwise ranking on AROWe use correct and incorrect training captions for initialization and to compute the ranking loss. Attribution also imposes a constraint on matched directions from the training set, and Relation uses category weights computed from the training set.
Validation-based selectionWhere applicable, we use the validation set to select the learning rate, configuration, and checkpoint. Gradient updates use only training data.

For the original Table 2 experiments, the revision will report retrieval results on all three controlled datasets for eight encoders and three splits per encoder. For each split, we will report Local I2T Top-1, global recall on seen and unseen compositions, and their harmonic mean (H-mean). This breakdown will allow readers to compare ROCA with the corresponding baselines on the same encoder and split and inspect variation both across encoders and across splits within an encoder. For the newly trained variants, we will report per-split results separately for OpenAI CLIP ViT-B/32, with the map variant and supervision specified.

AuthorsOpenReview ↗

Response to Reviewer DHaV (Part 2/3)

W1. Additional matching and binding evaluations on established benchmarks

We use the ARO Attribution and Relation tasks to test whether ROCA can match unseen bindings in natural images. We retain the official image crops and the correct and incorrect candidate captions, and partition the data into training, validation, and test sets by source image. In the held-out binding setting, we additionally remove candidates containing reserved bindings from the training and validation sets. The correct captions in the test subset contain these bindings, and all their constituent concepts appeared in training. On this subset, ROCA achieves category macro-accuracies of 75.98% on Attribution and 81.66% on Relation, exceeding LABCLIP by 4.34 and 4.88 percentage points, respectively.

The full evaluation includes nine settings across six benchmarks spanning attribute ownership in natural images, spatial relations, attribute binding, unseen composition recognition, and fine-grained image–text matching. ROCA and LABCLIP [1] use the same OpenAI CLIP ViT-B/32 image and text encoders, data splits, and test candidates in each comparison. ARO uses the orthogonal pairwise-ranking variant described in W2, and the other five benchmarks use the orthogonal contrastive variant. On those five benchmarks, both methods use a 50-epoch training budget. The table reports means ± sample standard deviations over three splits or optimization seeds.

Benchmark / settingMetric (%)LABCLIPROCA
ARO Attribution, image holdout [7]Category macro-accuracy78.38 ± 1.2483.82 ± 2.09
ARO Relation, image holdout [7]Category macro-accuracy81.52 ± 1.1585.29 ± 2.08
ARO Attribution, held-out bindings [7]Category macro-accuracy71.64 ± 2.3775.98 ± 1.30
ARO Relation, held-out bindings [7]Category macro-accuracy76.78 ± 7.3581.66 ± 6.18
What’sUp [3]Individual accuracy, with equal weight for subsets A and B52.78 ± 2.7768.16 ± 4.54
PUG-SPAR [6]Strict two-candidate accuracy across four tasks96.01 ± 1.2094.71 ± 0.61
UT-Zappos [2]Best harmonic mean in generalized compositional zero-shot learning38.50 ± 3.9738.55 ± 2.96
SPEC [4]Accuracy averaged equally over six tasks and two directions52.42 ± 1.9550.97 ± 0.82
3D Shapes [5]Unseen-composition Top-1 accuracy with 40 candidates80.34 ± 5.9780.10 ± 3.26

On ARO with image holdout, ROCA exceeds LABCLIP by 5.44 and 3.77 percentage points on Attribution and Relation, respectively. On What’sUp, it scores 15.38 percentage points higher. The mean scores differ by at most 1.45 percentage points on each of the remaining four benchmarks. For 512-dimensional embeddings, the orthogonal map has d(d−1)/2=130,816d(d-1)/2=130{,}816 intrinsic degrees of freedom, approximately half the d2=262,144d^2=262{,}144 degrees of freedom of a general linear map. The orthogonality constraint lets ROCA adjust the relative orientation of the two modalities while preserving the inner products, angles, and distances among the centered, unit-normalized text embeddings used for matching. These results extend the binding evaluation to natural images and show that, under the stated supervision, a map can preserve within-text geometry while matching unseen bindings.

AuthorsOpenReview ↗

Response to Reviewer DHaV (Part 3/3)

Q1. Per-split results and scope of the conclusions

As described in W2, the revision will provide per-encoder and per-split retrieval results for Table 2 of the original submission. The ARO experiments in W1 extend the binding evaluation to natural images and test matching when the constituent concepts were seen during training and the full binding was held out. We will also clarify the scope of the theory and the experiments. The theoretical analysis of factor-direction alignment assumes known correspondences between factors in the image and text modalities. In natural-image experiments that use factor directions, we estimate these directions from observed differences between annotated compositions. The external experiments assess whether the adapted map can match unseen compositions given the training pairs, semantic annotations, and negative examples specified by each task.

References

  • [1] Koishigarina, D., Uselis, A., and Oh, S. J. CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modally. ICLR, 2026.

  • [2] Yu, A., and Grauman, K. Fine-Grained Visual Comparisons with Local Learning. CVPR, 2014.

  • [3] Kamath, A., Hessel, J., and Chang, K.-W. What’s “up” with vision-language models? Investigating their struggle with spatial reasoning. EMNLP, 2023.

  • [4] Peng, W., et al. Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding. CVPR, 2024.

  • [5] Burgess, C., and Kim, H. 3D Shapes Dataset. 2018.

  • [6] Bordes, F., et al. PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learning. arXiv:2308.03977, 2023.

  • [7] Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., and Zou, J. When and Why Vision-Language Models Behave like Bags-of-Words, and What to Do About It? ICLR, 2023.

AuthorsOpenReview ↗

Revised Manuscript Uploaded with Additional Experiments and Clarifications

Thank you again for your suggestions on broader binding evaluation, supervision, and reporting variation across encoders and splits. We have uploaded a revised manuscript incorporating these additions.

Section 5.3 and Table 3 extend the evaluation to six benchmarks, including ARO Attribution and Relation under both image holdout and binding holdout. The binding-holdout protocol excludes reserved bindings from training and validation while retaining their constituent concepts in training. ROCA achieves 75.98% and 81.66% category macro-accuracy on held-out attribute and relation bindings, respectively, exceeding LABCLIP by 4.34 and 4.88 percentage points.

Section 5.1 and Appendix B.12 explicitly distinguish the original closed-form intervention, orthogonal contrastive adaptation, and ARO pairwise-ranking adaptation, including their supervision and preprocessing. Appendix C, Tables 16–17, provides retrieval breakdowns by encoder and split, and Appendix E reports individual runs of the supervised experiments. Section 4.3 and Appendix B.12 also clarify the factor-correspondence assumptions and the supervision used to construct directions. These additions connect the controlled analysis to natural-image binding under precisely specified evaluation conditions.

Reviewer_c5KSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper aims to question vision and laguange dual encoders and why image and text factors may not match on unseen combinations. They have tried to prove that two encoders while having additive and cross factor orthoganal representations, may still fail due to their factor correspondence being wrong. They have tried to go into finding the conditions o make this correspondence recoverable using geometry and ROCA.They show that text representations are closer to first order additivity and while this is true, image and text representations remain misaligned. They have tested on dSprited, PUG Animal and MPI3D and ROCA gets top 1 accuracy on all 3. It also gets good accuracy on a 2 candidate attribute swap task.

Critical strengths

The paper has a nice flow and presents results in an easy to follow manner, within modality structure, cross modality compatibility and binding are explained clearly which is important as they form the base of the paper.

The paper contains results and experiments in depth and has evaluations on many models and splits, the tables of the appendix are also very detailed. Evaluation rigor is shown, and the choice of experiments is good to support the claims.

Limitations of the paper are correctly recognized where needed.

Critical weaknesses

The propositions and theorems seem somewhat forced and seem like a rewording of existing established methods. Please try to soften some of them in the final paper and link them to existing theory where appropriate.

I suspect some entries of the tables, which make me question the other experiments in favor of ROCA, for example, random and shuffled being nowhere close to chance on the local chance. Please check if this is correct and not a coding or typing error.

Questions for authors

Could I please see ROCA's performance on some established benchmarks?

Could you please do an experiment to distinguish the effect of using ridge-trained val split. Could there be a confirmation/ablation specifically showing that aligning factor directions is what is useful here?

It would be great if there were a way to know the effect of the template on text versus images.

7 comments
AuthorsOpenReview ↗

Response to Reviewer c5KS (Part 1/4)

Thank you for the careful review. We appreciate your recognition of the distinction between within-modality structure, cross-modal compatibility, and attribute binding, as well as your assessment of the experimental coverage and rigor.

W1. Theory and prior work

We will identify the established results that underpin our analysis, state their assumptions, and narrow claims that were phrased too broadly. The revision will organize these results around three questions: how factors are structured within each modality, when those structures are compatible across modalities, and how attributes are bound to their owners. We will connect each result to the corresponding experiment and clarify how the geometry-preserving orthogonal intervention tests which cross-modal discrepancies can be corrected by adjusting the modalities’ relative orientation.

W2. Why the mapping controls fall below local chance

The original hit counts matched an independent recomputation for Random Q and Shuffled Q on all three original dSprites splits with OpenAI CLIP ViT-B/32. This rules out candidate-indexing and similarity-scoring errors in this setting. Uniform guessing gives each of the NN candidates probability 1/N1/N of being selected. The two mapping controls rank candidates by similarity under a shared random orthogonal map or a map fitted to mismatched factors. The structure of the candidate pool can give different candidates different probabilities of ranking first. We repeated all four controls with 18 candidates per query.

ControlRepeats per splitLocal Top-1 (%)
Uniform candidate guessing1285.5573 ± 0.0619
IID random scores1285.5552 ± 0.0693
Shared random orthogonal map1280.0380 ± 0.0068
Map fitted to shuffled factor correspondences320.0488 ± 0.0211

Values are means ± sample SD across the three split means, each averaged over its repeats. Uniform guessing and IID scores agree with 1/18≈5.56%1/18 \approx 5.56\%. Both mapping controls reproduce the much lower accuracies in the original table. For example, if the correct candidate is (1,1)(1,1) and the negatives are (−1,1)(-1,1) and (1,−1)(1,-1), scoring along a uniformly random direction selects the correct candidate with probability 1/41/4 because it ranks first only in the first quadrant. Uniform guessing succeeds with probability 1/31/3. The table reports the single-encoder repeat experiment separately from the original aggregate. We will label the reference Uniform candidate guessing and document both mapping controls in the revision.

AuthorsOpenReview ↗

Response to Reviewer c5KS (Part 2/4)

Q1. Results on established benchmarks

ROCA leads LABCLIP on all four ARO settings and gains 15.38 percentage points on What’sUp. The mean scores differ by at most 1.45 percentage points on each of the other four benchmarks. The nine settings span six benchmarks and cover attribute ownership in natural images, spatial relations, attribute binding, unseen composition recognition, and fine-grained image–text matching.

Following another reviewer’s suggestion, we include LABCLIP [1] as a general linear adaptation baseline. Both methods freeze the same OpenAI CLIP ViT-B/32 encoder pair and use identical splits and test candidates. For ARO, ROCA uses orthogonally constrained pairwise ranking. For the other five benchmarks, ROCA starts from the orthogonal Procrustes solution fitted to matched factor directions and uses contrastive training based on the LABCLIP setup. Both methods have a 50-epoch budget on those five benchmarks. ROCA maintains Q⊤Q=IQ^\top Q=I throughout adaptation. For 512-dimensional embeddings, the general linear map has d2=262,144d^2=262{,}144 degrees of freedom and the orthogonal map has d(d−1)/2=130,816d(d-1)/2=130{,}816 intrinsic degrees of freedom. ROCA uses approximately half as many degrees of freedom to adjust the modalities’ relative orientation while preserving inner products, angles, and distances among the centered, unit-normalized text embeddings used for matching. Results are means ± sample SD over three splits or optimization seeds.

Benchmark / settingMetric (%)LABCLIPROCA
ARO Attribution, image holdout [7]Category macro-accuracy78.38 ± 1.2483.82 ± 2.09
ARO Relation, image holdout [7]Category macro-accuracy81.52 ± 1.1585.29 ± 2.08
ARO Attribution, held-out bindings [7]Category macro-accuracy71.64 ± 2.3775.98 ± 1.30
ARO Relation, held-out bindings [7]Category macro-accuracy76.78 ± 7.3581.66 ± 6.18
What’sUp [3]Individual accuracy, equal weight for A/B52.78 ± 2.7768.16 ± 4.54
PUG-SPAR [6]Strict two-candidate accuracy, four tasks96.01 ± 1.2094.71 ± 0.61
UT-Zappos [2]Generalized CZSL best harmonic mean38.50 ± 3.9738.55 ± 2.96
SPEC [4]Accuracy, equal weight over six tasks and two directions52.42 ± 1.9550.97 ± 0.82
3D Shapes [5]Unseen-composition Top-1, 40 candidates80.34 ± 5.9780.10 ± 3.26

ARO protocols. We retain the official crops and correct and incorrect candidate captions and split the data by source image into training, validation, and test sets. The held-out binding setting additionally removes candidates containing reserved bindings from training and validation. Its test subset contains those bindings in the positive captions and restricts their constituent concepts to ones seen in training. On image holdout, the gains over LABCLIP are 5.44 points for attribution and 3.77 for relation. On held-out bindings, they are 4.34 and 4.88 points, respectively. These results extend the evidence for geometry-preserving compositional matching to natural images and unseen bindings. We will include the full evaluation protocols in the revision.

AuthorsOpenReview ↗

Response to Reviewer c5KS (Part 3/4)

Q2. Factor initialization, validation, and ridge

Factor-direction initialization. All Q2 experiments freeze OpenAI CLIP ViT-B/32 and report Local I2T Top-1 over three splits (mean ± sample SD, %). The main ROCA setting, Matched factors, initializes QQ by fitting an orthogonal Procrustes map to corresponding image and text factor directions. The other conditions use an unaligned orthogonal start, a Procrustes fit to shuffled factor correspondences, or Sample Procrustes, which fits paired image–text embeddings. Each initialization is followed by the same contrastive training with Q⊤Q=IQ^\top Q=I, including the same preprocessing, positive pairs, factor-based hard negatives, optimizer, training budget, and validation rule. This objective uses no ridge penalty. Matched factors gives the highest mean on dSprites, PUG-Animal, and MPI3D. Its gains over Sample Procrustes are 0.78, 2.12, and 0.18 percentage points, respectively, and its gains over the unaligned start are 3.60, 1.10, and 1.56 points. The gap between matched and mismatched factors is largest on dSprites (17.23 points), followed by PUG-Animal (1.59) and MPI3D (0.32).

DatasetMatched factorsUnalignedMismatched factorsSample Procrustes
dSprites24.61 ± 5.5021.01 ± 6.527.38 ± 3.8123.83 ± 1.50
PUG-Animal17.64 ± 1.0816.54 ± 0.3916.05 ± 1.9015.52 ± 1.44
MPI3D30.01 ± 0.3928.45 ± 1.4929.69 ± 1.1029.83 ± 2.74

Validation-based selection. Both rows use the Matched factors initialization. Jointly selecting the learning rate and checkpoint on validation data yields 24.61%, 17.64%, and 30.01% on dSprites, PUG-Animal, and MPI3D, respectively. These means are 2.00, 5.21, and 9.70 percentage points higher than those from training for 50 epochs at a fixed learning rate of 0.001.

SettingdSpritesPUG-AnimalMPI3D
Validation-selected learning rate and checkpoint24.61 ± 5.5017.64 ± 1.0830.01 ± 0.39
Fixed learning rate of 0.001, 50 epochs22.61 ± 3.7812.43 ± 1.1820.31 ± 3.57

Ridge coefficient selection. We use the same first- through third-order factor differences and data splits for both ridge settings. We either fix the relative coefficient at 0.01 or select it using a split within the outer training set. Inner selection has the largest effect on dSprites, where mean accuracy goes from 3.91% to 8.20%. The changes on PUG-Animal and MPI3D are −0.07 and +0.30 percentage points, respectively. This experiment uses scale-normalized regularization and reports ViT-B/32 results. The original ridge baseline uses an absolute coefficient and averages across eight encoders. We will label both protocols explicitly. The updated main table reports Sample ridge separately because it fits centered, paired image–text embeddings and uses a different regularization protocol and aggregation scope.

CoefficientdSpritesPUG-AnimalMPI3D
Fixed relative coefficient of 0.013.91 ± 0.722.38 ± 0.342.47 ± 1.67
Selected within the outer training set8.20 ± 1.032.31 ± 0.482.77 ± 1.41

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.