acceptodds
Withdrawn from ICLR 2027

STR: Auditing Shared-Direction Geometry in Multimodal Alignment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal contrastive learning involves inter-modal semantic agreement, inter-modal separation, and intra-modal diversity, yet most alignment objectives fold these concerns into a single aggregate similarity score. This paper decomposes them into separately weighted geometric terms, collectively Surface-Tension Regularization (STR), an auxiliary regularizer with no additional learnable parameters, and uses the decomposition as a measurement instrument. Across four VAST benchmarks and three seeds per benchmark, canonical STR increases common-mode energy and differs from the baseline in its response to shared-direction removal. Separating deletion from residual-norm rescaling in these two models shows that deletion drives the baseline loss, whereas under STR rescaling alone is disruptive and the combined operation approximately preserves aggregate top-1 recall. Replacing Gram-volume attraction with pairwise cosine also yields energy inflation and a similar combined-removal response on MSR-VTT and DiDeMo across three seeds per dataset. Seed-0 factorial and checkpoint-conditional gradient measurements provide complementary evidence about coupling and mean-margin changes. A single fixed STR configuration without per-dataset tuning produces small changes in final top-1 recall on the four benchmarks. The same metric remains similar across configurations with substantially different embedding-space behavior, motivating separate evaluation of the embeddings deployed directly.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multimodal representation learning, video-text retrieval, contrastive alignment objectives, embedding geometry, representation diagnostics

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_NnfDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces Surface-Tension Regularization (STR), an auxiliary objective that separates multimodal embedding pressures into agreement, separation, and diversity terms. Its main purpose is to audit embedding geometry rather than to substantially improve retrieval scores. Using video–text retrieval models, the authors compare a baseline with STR and measure how retrieval changes when a proposed shared direction is removed, the remaining vectors are rescaled, or both operations are applied. They report different responses to these interventions despite similar final retrieval scores. The main experiments cover four benchmarks with three seeds each; the paper also tests an alternative attraction term and reports factorial and gradient analyses.

Critical strengths
  1. The paper provides a careful audit of multimodal embedding geometry. It separates shared-direction deletion from residual-norm rescaling and reports how each operation affects retrieval. Random-direction, SVD1, and per-query controls make the central observation easier to assess.
  2. The main diagnostic results are supported by repeated experiments. The baseline–STR comparison covers four video-text retrieval datasets with three seeds each. The paper also tests alternative regularization terms and additional backbones, and clearly acknowledges that the final retrieval gains are small and the full geometric pattern does not transfer to every backbone.
Critical weaknesses
  1. The paper’s main removal result appears sensitive to how the direction is defined. In the baseline, the text and video centroids are nearly orthogonal, so their sum is not clearly a shared direction. At MSR-VTT seed 0, removing SVD1 changes R1 from 41.0 to 45.4 for the baseline and from 39.5 to 40.4 for STR (Table 15), reversing the sign of the between-model response difference. The paper should test whether the result holds under other direction definitions across datasets and seeds.

  2. The scope of the paper’s conclusion is unclear. The full pattern is observed with VAST under ITC+ITM finetuning, but none of the additional backbones reproduces it in full (Table 7). Although the paper compares final retrieval scores with GRAM, Symile, and TRIANGLE, it does not apply the geometric audit to those models. The evidence therefore does not establish whether the finding extends beyond this particular training setup.

  3. The paper does not demonstrate a consistent retrieval gain from STR. Its final text-to-video R1 differs from the baseline by only −0.1 to +0.4 points across the four datasets (Table 1), while its unmodified text–video embedding R1 is lower on all four (Table 14). Video-to-text R1 also falls on three datasets (Table 25). These results leave the practical value of the changed geometry unclear.

  4. The evidence for the paper’s three-term interpretation is limited. At the default setting, LsepL_{\mathrm{sep}} has very little effective pressure, and Lvol+LdivL_{\mathrm{vol}} + L_{\mathrm{div}} already produces much of the reported geometry. The factorial and gradient-cosine analyses use only seed 0, and the additional-backbone results are single runs. The smaller component effects are difficult to assess without estimates of their variability.

Questions for authors
  1. How is the sum of the text and video centroids justified as a shared direction when the two centroids are nearly orthogonal in the baseline? Does the main removal result hold under other direction definitions, including SVD1, across datasets and seeds?

  2. Does the full geometric pattern appear in any of the additional backbones under comparable training conditions? Can the same geometric audit be applied to GRAM, Symile, and TRIANGLE to clarify how broadly the conclusion applies?

  3. Given the small and inconsistent changes in final retrieval scores, what practical benefit does STR provide? How should the lower unmodified embedding R1 on all four datasets be interpreted?

  4. What is the effective contribution of LsepL_{\mathrm{sep}} at the default setting beyond Lvol+LdivL_{\mathrm{vol}} + L_{\mathrm{div}}? Can variability across seeds be reported for the factorial, gradient-cosine, and additional-backbone analyses?

Reviewer_aLP2OpenReview ↗
Rating
1
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper is a technical report on a collection of losses for trimodal learning.

Critical strengths

Literature and comparisons are relevant

Critical weaknesses

Writing should consistently reviewed. It is not acceptable to submit a scientific paper to such an important venue with seed indications on the abstract, it is meaningless from the scientific point of view, it is not a technical report. The introduction is ery harduous to understand, the motivation is not clear, the description is technical without being formal. Again in the contribution list the authors list the seeds: we do not care, these are experimental details. The notation is unclear: vectors and matrices should be bold, otherwise they represent scalars.

The loss combinatin described in sec. 3.2 should be thourughly explained. why involving both l_itc and l_vol if both cares about the alignment?

Page 3 and we already discuss failure? …

Loss functions introduced here are already well-known loss functions in the literature, so there is no novelty or no novel insights that this paper provides.

Experimental gains are non significant, the proposed method obtains comparable performance to existing methods.

The paper seems completely written and experimented by an agent, resulting in very poor scientific novelty, significance, and rigor. This is useless for the scientific community.

Questions for authors

No

Reviewer_zdGaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes new method for analyze multimodal embedding geometry by finetuning existing pretrained model (VAST) one more with additional loss, called Surface-Tension Regularization (STR). The STR is added on top of the standard contrastive (ITC) and matching (ITM) objectives, and consist of 3 terms with each separate weight. inter-modal agreement (Gram-volume term), inter-modal separation (quantile-based hinge), intra-modal diversity (short-range repulsion). authors compare model finetuned with and without the STR, also with term subset and gradient-matched substitute term. The authors report alignment pressure make embedding energy concentrate into a single shared direction, and remove this direction affect the baseline and the STR model differently. On the 4 video-text retrieval benchmarks (MSR-VTT, DiDeMo, VATEX, ActivityNet), the STR change final retrieval accuracy only little. authors interpret this as an evidence that a final pipeline metric can hide big difference in embedding space behavior.

Critical strengths

The paper proposes a new analysis approach that decomposes the multimodal alignment objective into separately weighted geometric terms and uses them to probe embedding-space behavior beyond what aggregate retrieval scores reveal.

Critical weaknesses
  • The paper appears to be written for readers already highly familiar with this specific line of work.: Core components are used without any explanation. For example, ITC and ITM are only expanded as acronyms in the introduction (ITM's role as the cross-encoder reranker behind the main metric is left to Sec. 4.1 and App. F), and VAST is used throughout as if already known to the reader, without describing what the model is or how it is structured. This makes the paper hard to follow for the broader ICLR audience.
  • Inaccurate naming and missing references.: VAST is referred to as a benchmark although it is a model and is not cited until Sec. 4.1, the ITC and ITM losses are never attributed to their sources (e.g., ALBEF; Li et al., 2021), and "common-mode energy" is introduced without reference although it is essentially an anisotropy measure (Ethayarajh, 2019).
  • The practical significance of the measured geometric differences is not demonstrated.: The differences between the baseline and STR (e.g., common-mode energy, removal sensitivity) are evaluated only through text-video embedding-cosine retrieval, whereas VAST actually retrieves with ITM reranking over fused candidates, where both models perform almost identically (Table 1). On the embedding-cosine path itself, STR is below the baseline on all four datasets (Table 14), and the paper provides neither a downstream setting that uses these embeddings directly nor a mitigation experiment, so it remains unclear what practical conclusion the measured differences support.
  • The relation to existing analysis methods could be made clearer. It is not discussed what the STR-based audit reveals beyond established diagnostics such as alignment and uniformity (Wang & Isola, 2020) or modality-gap analysis (Liang et al., 2022). Since the paper already reports related measures (e.g., MMD, Sym. NN, centroid cosine), a brief discussion of what the audit adds on top of them would help clarify its contribution.
  • Several findings may be expected consequences rather than new insights.: The shared-direction inflation is not an existing property uncovered by the analysis but is produced by STR's own Lvol, which alone collapses the video embeddings onto essentially one direction (∥µv∥² ≈ 1.0, Table 2), and the factorial attributions (e.g., Lvol dominating matched similarity) largely restate how each term is defined. Likewise, the insensitivity of the final ITM metric is expected for a retrieve-then-rerank pipeline, since both models already place the ground truth among the top-50 candidates for 93–96% of queries (Table 21) before the cross-encoder reranks them, and the collapse under rescaling alone (N) may follow mechanically from Eq. 7, whose candidate-dependent term grows with common-mode energy.
Questions for authors

Although the paper supports its claims with extensive and diverse experiments, these do not resolve the concerns I raised in the Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.