acceptodds
Withdrawn from ICLR 2027

NC-CLIP: Geometric Feature Decoupling for Robust Vision-Language Alignment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Contrastive pre-training dictates modern vision-language alignment, yet its conventional "align-everything" objective is fundamentally misaligned with the nature of multimodal data. By forcing the holistic alignment of images and texts, existing frameworks conflate shared semantics with modality-specific noise. This leads to the memorization of spurious correlations, the destruction of complementary information, and fragile generalization under distribution shifts. We propose that robust representation requires explicit decoupling: we must align the shared semantic core while retaining modality-specific traits. To this end, we present , a framework that mathematically enforces this separation via the geometry of Neural Collapse. leverages bidirectional feature masking alongside a fixed Equiangular Tight Frame (ETF) that acts as an absolute geometric anchor. This design strictly guides the alignment of pure semantic features while isolating modal noise. Extensive benchmarking demonstrates that drastically improves robustness to distribution shifts and spurious correlations, all while maintaining competitive zero-shot performance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
vision-language models, CLIP, contrastive learning, multimodal representation learning, neural collapse

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_72RvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes NC-CLIP, a vision-language alignment method built on SmartCLIP. It introduces symmetric shared routing to select a common feature support for image and text representations, fixed ETF anchors with Sinkhorn-balanced assignments to regularize the representation geometry, and shared-only cross-modal reconstruction to preserve useful information. The method is evaluated on image-text retrieval, zero-shot classification, and group-robustness benchmarks. The authors report improvements over SmartCLIP on retrieval and worst-group accuracy, along with several component ablations.

Critical strengths

S1. The paper addresses a relevant problem in vision-language alignment, and the reported experimental results are encouraging. In particular, NC-CLIP improves retrieval on several benchmarks and achieves noticeable worst-group accuracy gains on Waterbirds and CelebA. The authors also provide several ablations to examine the proposed components.

Critical weaknesses

W1. Poor presentation and inconsistencies in the method description. I found the paper unnecessarily difficult to follow, particularly the figures. Figure 1 introduces common and unique masks, but the unique mask is not explained in the subsequent method. Figure 2 instead uses fine and coarse masks without clarifying their relationship to Figure 1. The key averaging operation for shared routing is also not clearly illustrated. More importantly, Figure 2 shows reconstruction toward full image/text representations, which appears inconsistent with the shared-only reconstruction described in Eq. (11). There are also several apparent typos in the figures. These issues go beyond minor presentation problems and make it difficult to understand the actual method.

W2. The contribution of the individual components to robustness remains unclear. The paper reports noticeable worst-group accuracy improvements, but the targeted ablations mainly examine the proposed components using retrieval or geometric metrics. Shared routing and independent routing achieve nearly identical retrieval performance, while exact ETF and random fixed anchors also perform similarly on retrieval. Although the authors acknowledge these results, it remains unclear how much each component contributes to the robustness improvements. More direct worst-group accuracy ablations would help support the proposed design.

Questions for authors

Q1. Could the authors clarify the relationship between the common/unique masks in Figure 1 and the fine/coarse masks in Figure 2? Also, why does Figure 2 show full-representation reconstruction when Eq. (11) describes shared-only reconstruction?

Q2. Could the authors clarify how shared masks are computed during retrieval? In particular, are the masks generated separately for each query-candidate pair, as in pair-dependent scoring? A brief description of the inference procedure would be helpful.

Q3. Could the authors provide worst-group accuracy ablations for the main components, especially shared versus independent routing and exact ETF versus random fixed anchors? This would help clarify which design choices contribute most to the robustness improvements.

Reviewer_6Q3JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NC-CLIP, a vision-language alignment method that combines symmetric shared routing in a learned basis with fixed Neural-Collapse-inspired geometric anchoring. Shared routing uses modality-conditioned observers to produce one shared fine/coarse support per matched image-text pair without assuming semantic equivalence, while frozen ETF anchors with Sinkhorn-balanced assignment organize the routed space and are regularized by shared-only cross-modal reconstruction. Experiments report retrieval gains, improved group robustness, and controlled ablations that separate the roles of routing, anchoring, ETF geometry, and reconstruction.

Critical strengths

The paper proposes a unified framework that combines symmetric shared routing with fixed Neural-Collapse geometric anchoring. Specifically, it introduces modality-conditioned observers that produce a single shared fine/coarse support, fixed ETF anchors regularized by Sinkhorn-balanced assignment, and shared-only cross-modal reconstruction. These components separate feature selection from geometric organization, forming a reusable design principle whose contribution is verified through controlled ablations.

Critical weaknesses
  1. The paper argues that shared routing is justified because it "enforces shared support in a learned coordinate system," which SmartCLIP's asymmetric masks do not provide. But this explains how the mechanism works, not why shared support matters, and the underlying motivation remains unclear. Moreover, the experiments do not show that shared routing yields retrieval improvements (Table 3b). While it improves geometric diagnostics such as matched-group cosine and modality gap, how these improvements translate into any downstream benefit remains unclear.
  2. Qualitative results are missing. The paper should at least provide Grad-CAM visualizations and text-to-image generation examples, as SmartCLIP [1] does, to qualitatively show whether the learned representation is better or not.

[1] Shaoan Xie, Lingjing Lingjing, Yujia Zheng, Yu Yao, Zeyu Tang, Eric P. Xing, Guangyi Chen, Kun Zhang. SmartCLIP: Modular Vision-language Alignment with Identification Guarantees. CVPR, 2025.

Questions for authors

See weaknesses.

Reviewer_fNtyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes NC-CLIP, a fine-tuning framework with: (1) symmetric shared routing for image and text embeddings, (2) geometric anchoring via an Equiangular Tight Frame (ETF), and (3) shared-only cross-modal reconstruction.

Critical strengths

The integration of controlled ETF alongside fixed geometric anchoring effectively enhances both retrieval, which is an impressive strength.

Critical weaknesses
  • Lack of an equal-budget baseline confounds the main claims. NC-CLIP fine-tunes SmartCLIP checkpoints for 3 additional epochs on CC3M, yet no baseline for SmartCLIP trained with these extra 3 epochs is provided. Because the benchmark results in Table 1 match SmartCLIP, the reported performance gains could stem entirely from extended CC3M training rather than the proposed routing or anchoring mechanisms.
  • Under-explored motivation for the Fine vs. Coarse routers. The distinction between these two router designs is insufficiently analyzed, and the visualization in the appendix offer limited insight.
Questions for authors
  • Could the authors evaluate the performance of the proposed method when initialized directly from the standard CLIP baseline?
  • Could the authors elaborate on the results in Tables 3b and 3c? What accounts for the drop in performance on I2T and Rsum when using the proposed components?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.