acceptodds
Under review as a conference paper at ICLR 2027

Coordinates or Equivalence? Alignment Reallocates Sensitivity Across Input Variations

Abstract

Contrastive image-text training changes how an encoder responds to variations that its captions may leave unspecified. We measure this change using distributions induced by a fixed panel of input transformations. Each family's within-input covariance is compared with the covariance of the complete mixture, yielding response shares and cross-family covariance similarities that are invariant to invertible affine reparameterisations. An exact covariance budget separates within-component variation from variation of component means. A conditional linear-model analysis further expresses response changes through the encoder's input-space readout operator. On Stanford Cars, disrupting image-caption correspondence changes the mean per-seed correlation between a frozen description-based predictor and response-share changes from to . The five-dose series changes sign between corruption fractions and . Pairing contrasts also appear on CUB and Flowers and under a second public initialisation, with a gate-limited result on ConvNeXt. A separate analysis relates geometric reallocation to perturbation-specific accuracy changes, whose associations vary across datasets. Attributing the direction to description content needs separable caption sets; a pre-training screen passes one of six caption pairs across five datasets, and the content comparison on that pair is not significant. Together, these results provide an affine-invariant diagnostic of pairing-dependent response allocation, leaving open whether description content determines its direction.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multimodal alignment, contrastive learning, representation geometry, statistical tokens, vision-language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_c2rcOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper seems to attempt to measure how model's response changes by the perturbations of the input-caption in the contrastive image-text training. The authors seem to propose an approach to measure the change by fine-tuning the open-CLIP model on some downstream image classification datasets. They conduct experiments on several datasets and models and present some response analysis on the model's response change w.r.t the perturbation.

Critical strengths
  1. They seem to attempt to measure how the CLIP-like model's output changes by input variations. This kind of topic might be interesting to community.
Critical weaknesses
  1. I could not understand the scope of this paper due to the poor writing. Their contribution and problem setting are not clear at all from the abstract and introduction. The positioning of their paper is also not clear since they cite no papers in the introduction. They need major refinement w.r.t. the paper writing. For example, line 52, "We test this measurement by changing image-caption correspondence while holding the predictor and evaluation panel fixed.", what does the sentence mean?
  2. They do not mention the contribution considering the existing work. It is hard to measure the value of this paper.
  3. Their theory section is also hard to understand. What does "STATISTICAL TOKENS" mean? in the title of the Sec.3. The term is never mentioned after this title.
  4. Experiments sections are also poorly organized. They do not mention the goal of the experiments, detailed setting, and obtained insights.
  5. Their empirical results look very specific to the datasets. They do not mention any general observations from experiments.
Questions for authors

The authors need major refinement on this submission for better readability.

1 comment
AuthorsOpenReview ↗

Response on Problem Setup, Contribution, and Empirical Interpretation

We appreciate the reviewer's emphasis on readability and on making the contribution distinguishable from existing work. We agree that readers should be able to identify the scientific question, the statistical object, and the purpose of each experiment before reaching the technical details. The revised manuscript reorganizes those entry points as follows.

1. Problem setting and Introduction (Critical Weakness 1)

The Abstract (p. 1) now states two questions: how to measure the relative response to specified input variations without dependence on embedding coordinates, and whether that response allocation depends on image–caption pairing. The Introduction (pp. 1–2) begins with a concrete car example, cites related work, and lists three contributions. The previously compressed sentence about “changing correspondence” is now explained operationally: correct-caption and image-reassigned-caption runs share the images, caption multiset, evaluation panel, and frozen predictor. The manipulated factor is their training correspondence.

2. Contribution relative to prior work (Critical Weakness 2)

Section 2 (pp. 2–3) and Appendix C (Table 7) compare the approach with global CKA/Procrustes analyses and per-cell adaptations of similarity measures. The contribution is a transformation-resolved, affine-invariant covariance reading with normalized shares, cross-family overlap, and an exact covariance budget (Theorem 1). We do not claim that no existing measure detects a pairing reversal: the stochastic shape metric in Table 7 does so as well. The new information provided here is the interpretable allocation across declared variations.

3. Meaning of statistical tokens and theory (Critical Weakness 3)

Section 3.1 (p. 3) now introduces the term before Definition 1. Repeatedly applying a fixed transformation to one image creates a distribution of embeddings, called its statistical token. Its within-anchor covariance is aggregated across images for a cell; the share is computed only relative to the whole mixture. Section 3.2 explains the readout before the formulas; Table 4 (p. 14) provides a notation guide. The conditional directional theory is shortened in Section 4, with supporting statements in Appendix A.

4. Experimental organization and takeaways (Critical Weakness 4)

The new experiment map in Section 6 (p. 6) states the purpose of each part: validating the readout (§6.1), measuring allocation (§6.2), testing pairing (§6.3), grading corruption (§6.4), testing caption-content attribution (§6.5), and relating responses to accuracy (§6.6). Appendix E records the training arms, seeds, transform pipeline, and evaluation protocol. The main controlled Cars comparison gives +0.833+0.833 versus −0.443-0.443 (Table 1), with the family-block inference in Table 2.

5. Cross-setting observations and scope (Critical Weakness 5)

Section 6.3 and Figure 2(b) report the pairing-sign contrast on Cars, CUB, Flowers, and a second public ViT initialization. ConvNeXt has the same observed signs but does not pass the declared CCA gate, so it is explicitly a sensitivity check. Section 7 (p. 10) synthesizes the recurring pattern and its limits: effect magnitudes are not directly comparable across separately calibrated settings and unequal optimization budgets; behavioral correlations are also not uniform across datasets.

Summary (Question for Authors: overall readability)

The revisions directly address the requested presentation refinement: a concrete opening, an earlier intuitive definition of the main object, explicit positioning against nearby methods, a shorter conditional theory, and an experiment-by-experiment rationale. The scientific scope remains unchanged. We hope this makes the method, empirical finding, and interpretation limits straightforward to assess.

Reviewer_973oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how contrastive image-text finetuning reallocates the image encoder’s sensitivity across 15 input-transformation cells. The authors correlate the change in each cell’s affine-invariant covariance share with a frozen predictor from an image–caption CCA subspace, comparing correct vs permuted captions. They experiment on Stanford Cars + OpenCLIP (ViT-B/32), replicate on CUB, Flowers, and a different init, and add a gate-limited ConvNeXt check.

Critical strengths

(1) The readouts are exactly affine-invariant and the covariance budget closes to d, and many choices are fixed.

(2) A clean pairing intervention: class vs permuted captions on the same images and recipe give +.833 vs −.443, with consistent signs across seeds.

Critical weaknesses

The main concerns is to check whether the effect is due to the pairings or not, is not reported. The dose series is within the noise of different seeds (15 cells, 5 families), and the bootstrap of the cells in the arm with permuted pairings goes through 0 (tab.1). Is the story any clearer if one computes the p-values of the family blocks for both arms, and for their difference, with all 4 seeds of the permuted pairings?

It is not clear to what extent the “description” predictor works. Except for the change in opposite directions when the pairings are intact or permuted (which is to a large extent expected), does the direction depend on the descriptions, which the abstract and sec.7 leave open? The “class name” predictor is the largest for cropping and occlusion, the augmentations for which the rise in share is the largest, even though “none of their captions describes …” (p.9). Would experiments along these lines with the cross-entropy head, with true and shuffled labels, show reversal, too?

The theory is not related to the experiments. Since it takes a lot of space in the main text (sec.4), it would be good to know how it is related to \rhopart. Otherwise, it would be good to move it to the appendix. (The results are reported separately from the theory, p.5.)

The covariance estimate is extremely ill-conditioned; one can only read the directions and the ordering from it. Is it possible that the directions/ordering is due to the augmentations? Could the authors describe the augmentations and show the value of \rhopart for smaller values of NN?

Questions for authors

(1) What are the family-block p-values with 4 permuted seeds?

(2) What statistic and seeds give the family-block p = 8 / 120 for the two-text comparison on p.22?

1 comment
AuthorsOpenReview ↗

Response on Pairing Inference, Predictor Scope, Theory, and Conditioning

We appreciate the recognition of the affine-invariant readouts and pairing intervention. We added family-block inference and clarified the predictor and conditional theory.

1. Pairing contrast and family-block inference (Weakness 1; Question 1)

We now report all three class and four permuted seeds (§6.3, Table 2, p. 8; Appendix G). Each of the 5!=1205!=120 relabelings permutes the five three-strength Δπ\Delta\pi blocks as units, applying the same permutation across all seeds. The predictor and log⁡tr⁡Σ0g\log\operatorname{tr}\Sigma^g_0 stay fixed; each seed recomputes OLS residuals before Spearman. The primary statistic is the difference of arm-wise means of per-seed ρpart\rho_{\mathrm{part}}, not a correlation of seed-mean vectors.

TestOne-sided ppTwo-sided pp
FB-diff (primary)1/120=0.0081/120=0.0087/120=0.0587/120=0.058
FB-class1/120=0.0081/120=0.0082/120=0.0172/120=0.017
FB-permuted8/120=0.0678/120=0.06728/120=0.23328/120=0.233

The directional tails follow the observed cell-level signs and were not preregistered, so both sides are reported. The result supports a directional arm contrast under the fixed-family null; the two-sided result exceeds 0.050.05, and the permuted arm alone is not significant. Cell bootstrap and family-block permutation address different resampling questions; neither invalidates the other. The doses do not establish a monotone law (§6.4).

2. Description predictor and CE control (Weakness 2)

The frozen predictor measures reference response along caption-expressible directions. Cropping or occlusion can alter class evidence without being named in captions; this remains an interpretation. Appendix J provides control-subspace checks. On CUB, attribute captions never name species, yet paired/permuted correlations are +0.71/−0.34+0.71/-0.34 (§6.3), with the CCA gate narrowly passed at 0.1160.116. The contrast therefore extends beyond Cars class-name captions. Actual content attribution remains unestablished: the only separable Cars class/colour pair has p=10/56=0.179p=10/56=0.179 (arm labels) and 8/120=0.0678/120=0.067 (family blocks; §6.5).

We did not run a cross-entropy true/shuffled-label control. The covariance readouts can also be applied to a CE-trained encoder with the same reference-side predictor; such a test would address objective specificity. Our claims concern contrastive image–text fine-tuning, without claiming InfoNCE specificity.

3. Relation of the theory to ρpart\rho_{\mathrm{part}} (Weakness 3)

We agree the main-text theory was too long for its empirical role. Section 4 (p. 5) now states the boundary first; Theorem 2 and the two-source analysis move to Appendix A. Theorem 1's exact invariance and covariance budget remain in §3. The conditional model concerns Δs(g)/τg\Delta s(g)/\tau_g and input-space qgq_g, whereas Eq. (6) uses strength-residualized ranks of Δπ(g)\Delta\pi(g) and the embedding-side pgshp_g^{\mathrm{sh}}. Equation (4) connects ss and π\pi, but the model does not predict the observed reversal or correlation sign.

4. Conditioning, augmentation, and smaller values (Weakness 4)

Appendix E (p. 20) specifies the shared Cars training pipeline: RandomResizedCrop (area [0.9,1.0][0.9,1.0], aspect ratio [3/4,4/3][3/4,4/3], bicubic, 224×224224\times224), with no flip, color jitter, or erasing. Evaluation cropping overlaps in crop-area range but is a different operator. Equal training augmentation cannot exclude augmentation–supervision interactions (Table 12).

Removing cropping and occlusion together changes the class/permuted readings from +0.833/−0.443+0.833/-0.443 to +0.744/−0.096+0.744/-0.096 (Table 13); this shows family sensitivity without identifying cropping as its cause. On the two weakest levels, the readings are +0.919/−0.212+0.919/-0.212 (Table 13). Disjoint-half checks give +0.86/+0.87+0.86/+0.87 and −0.42/−0.40-0.42/-0.40; ridge and finite-view corrections retain the signs, as does frozen-reference truncation to k=64k=64 (Table 15; Appendix K). Truncation is diagnostic and not exactly affine-invariant. Please clarify if the requested smaller values refer to another parameter.

5. Definition of the two-text 8/1208/120 (Question 2)

This is Spearman over 15 cells between pclass−pcolourp_{\rm class}-p_{\rm colour} and Δπ‾class−Δπ‾colour\overline{\Delta\pi}_{\rm class}-\overline{\Delta\pi}_{\rm colour}. We average response vectors over three class and five colour seeds before correlation and do not residualize strength. Permuting the five intact family blocks gives 8/120=0.0678/120=0.067; the separate (83)=56\binom{8}{3}=56 arm-label assignments give 10/56=0.17910/56=0.179 (§6.5, p. 9; Appendix G). The two-text 8/1208/120 is distinct from the numerically equal FB-permuted 8/1208/120 in Table 2.

Summary

The revised evidence supports a controlled, fixed-panel pairing contrast while explicitly reporting its two-sided and per-arm limits. Content attribution, objective specificity, and a unique mechanism remain open.

Reviewer_oeq3OpenReview ↗
Rating
4
Confidence
1
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The authors investigate the effect of visual variations omitted from caption text when training contrastive image-text models. They define a grid of 15 variations (five transformation families across three strength levels), referring to each combination as a "cell." They further introduce the "statistical token," defined as the distribution of pre-normalization embeddings obtained by passing transformed images through the image encoder, paired with an "anchor" representation from the original source image. In Theorem 1 (pencil invariants), the authors prove that the readouts st(g)s_t(g), the share πt(g)\pi_t(g), and the eigenvalue spectrum of C−1ΣgC^{-1}\Sigma_g remain invariant under invertible affine transformations.Beyond their theoretical analysis, the authors conduct multi-arm experiments by fine-tuning an OpenCLIP ViT-B/32 model on the Stanford Cars dataset. Across arms, they vary the training captions while holding all other hyperparameters constant. The tested caption conditions include ground-truth class names, color-only descriptions, class names under a fixed image-label permutation, and a graded series that progressively replaces valid pairings with permuted class names. These experiments are also extended to the CUB and Oxford Flowers datasets.Finally, the authors evaluate the robustness of their findings across different pre-training initializations and architectures, comparing LAION-2B against OpenAI CLIP weights and substituting the ViT-B/32 backbone with a convnext_base_w architecture

Critical strengths

The paper combines both a rigorous theoretical analysis with extensive empirical evaluation. The author have experimented with different datasets, different weight initializations, and different model architectures to provide a comprehenisve evaluation.

Critical weaknesses

The paper is hard to read - especially for someone who doesn't have deep expertise in the geometric analysis.

Questions for authors

The theoretical analysis looks sound (although I lack the expertise to verify it meticulously) , the empirical experiments are convincing but the paper is hard to read - especially for someone who doesn't have deep expertise in the geometric analysis.

1 comment
AuthorsOpenReview ↗

Response on Presentation, Definitions, and Experimental Roadmap

We appreciate the reviewer's positive assessment of the theoretical construction and empirical evaluation. We agree that the geometric presentation placed too much burden on readers who are not already familiar with the underlying tools. The revision prioritizes a concrete explanation of the measurement and a clearer separation of exact mathematical properties from conditional interpretation.

1. Making the measurement accessible (Critical Weakness)

The revised Introduction (pp. 1–2) begins with a car image subjected to changes in colour, cropping, and occlusion, then identifies the two research questions and three contributions. Section 3.1 (p. 3) introduces a statistical token as the distribution of embeddings generated by repeatedly applying one declared transformation to an image, before giving Definition 1.

Section 3.2 (p. 4) explains the readout before the equations. In plain terms, a cell's within-anchor covariance is compared with the covariance of the complete embedding mixture; its normalized share measures how much of the total within-component reading is allocated to that cell. The construction is unchanged by any common invertible affine transformation of embedding coordinates. Table 4 (p. 14) collects the notation.

2. Connecting the mathematics to the experiments (Question for Authors)

Section 6 now opens with an experiment map: numerical validity, response allocation, pairing intervention, corruption doses, content attribution, and behavioral association. Section 4 (p. 5) is substantially shorter and immediately states its scope: the conditional input-space analysis does not predict the sign of the empirical residual Spearman correlation. Formal supporting statements are in Appendix A.

Summary

The central question can now be read directly: which declared input variations gain or lose relative encoder response after image–text fine-tuning, and how does this allocation depend on image–caption pairing? We hope the revised route from intuition to definitions and experiments addresses the presentation concern while preserving the necessary mathematical detail.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.