acceptodds
Under review as a conference paper at ICLR 2027

Toward Unified Multimodal Representation Learning: Contrastive Alignment of Text, Image, and Point Cloud

Abstract

Contrastive Language-Image Pre-training (CLIP) has shown impressive performance in aligning visual and textual representations. Recent works have extended this paradigm to 3D vision, where a common strategy is to align point-cloud representations with text and image features through pairwise contrastive objectives. However, such pairwise formulations optimize modality pairs separately rather than multimodal tuples jointly. In this paper, we propose Contrastive Tensor Pre-training (CTP), a joint contrastive learning framework that extends the pairwise similarity matrix to a multimodal similarity tensor for text–image–point-cloud alignment. We further study two key aspects of joint multimodal learning. First, we show that a simple geometric scorer can effectively align the three modality representations. Second, we identify partial-match negatives introduced by joint tuple construction and show that masking them improves joint alignment. Experiments on real-world automotive LiDAR from nuScenes and dense point clouds from ULIP-ShapeNet show that CTP outperforms a matched pairwise baseline in both zero-shot classification and instance-level retrieval. The learned representations also transfer to KITTI and Waymo without target-domain training, further supporting the cross-dataset generalization of CTP.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multimodal representation learning, contrastive learning, joint multimodal alignment, 3D representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HTcvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Contrastive Tensor Pre-training (CTP) for jointly aligning text, image, and point-cloud embeddings. Instead of optimizing three separate pairwise contrastive objectives, CTP constructs a batch-wise similarity tensor over all text-image-point tuples. Tuple similarity is the sum of the three pairwise cosine similarities. For each anchored modality, the loss normalizes over the Cartesian product of candidates from the other two modalities. The authors identify “partial-match negatives” that preserve one matched pair involving the anchor and remove the corresponding row and column from the denominator. Experiments train CLIP text/image encoders jointly with a LitePT-S point encoder on object triplets from nuScenes and ULIP-ShapeNet, with cross-dataset evaluation on KITTI and Waymo.

Critical strengths
  1. The paper studies a clear and relevant question: whether text-image-point-cloud tuples should be contrasted jointly rather than only through separate modality pairs. The tensor formulation and partial-match-negative issue are clearly explained.
  2. The pairwise and joint methods use matched encoders, initialization, input processing, and optimization settings. Experiments cover both sparse real-world LiDAR and dense ShapeNet point clouds, with substantial retrieval gains and cross-dataset evaluation.
Critical weaknesses
  1. The experimental validation is insufficient. The current comparisons do not isolate the effects of the larger candidate set, tensor normalization, cosine-sum scorer, and partial-match masking. Candidate- and compute-matched pairwise baselines, together with factorial ablations of these components, are needed to establish where the gains come from.
  2. The methodological novelty is incremental. Joint contrastive learning over complete multimodal tuples has already been explored by Symile, GRAM, and TRIANGLE. CTP mainly replaces the tuple scorer with a simple sum of pairwise cosine similarities and masks partial-match negatives; moreover, its inference-time ranking is identical to the pairwise scorer. Thus, the substantive novelty is narrower than the paper’s framing suggests.
  3. Important 3D multimodal baselines are missing. The experiments compare mainly with a ULIP-style objective and general multimodal scorers, but not with representative text-image-3D pretraining methods such as Uni3D.
Questions for authors
  1. It would be helpful to compare CTP with a pairwise baseline using roughly the same number of negatives and a comparable compute budget. Do the gains remain in this setting?
  2. Could the authors disentangle the contributions of the tensor candidate space, cosine-sum scorer, and masking, ideally on both classification and retrieval?
  3. The related-work discussion could be broadened to cover multimodal learning with different 3D representations, including Uni3D (Zhou et al., 2024) for point clouds, UniGS (Li et al., 2025) and TIGaussian (Liu et al., 2026) for 3D Gaussian Splatting, MeshCLIP (Song et al., 2023) for meshes, and TriCoLo (Ruan et al., 2024) for voxels.
  4. Since ground-truth categories are provided when generating the pseudo-captions, please clarify in what sense the evaluation is “zero-shot.” Results with category-agnostic captions would help address this concern.
  5. I am also curious how the proposed anchor-sharing mask compares with no masking and with masking every tuple that contains a matched pair, especially for retrieval.

I will certainly consider raising my rating if the concerns above can be resolved.

2 comments

Thanks for the time and helpful comments. We address each point below, with revisions highlighted in blue.

W1: We have expanded the experiments to examine candidate count, scorer choice, and masking. Please refer to our responses to Q1, Q2, and Q5 for the corresponding comparisons and ablations.

W2: To our knowledge, prior work has not explored contrastive learning over complete multimodal tuples specifically for text–image–point-cloud alignment. Our work investigates the feasibility and effectiveness of this formulation. Our scorer design prioritizes alignment behavior rather than mathematical complexity. We explored SVD-based alternatives, but found that certain formulations can attain an optimal score of zero even when vectors point in opposite directions on the unit hypersphere, which does not reflect the desired alignment. We therefore adopt a cosine-sum scorer and introduce anchor-sharing masking to address partial-match negatives arising from multimodal tuple construction.

W3: We evaluated several additional 3D multimodal methods. PointCLIP and PointCLIP V2, evaluated in their training-free zero-shot settings, performed poorly on our sparse nuScenes point clouds. Our Uni3D adaptation achieved stronger joint classification performance, reaching 80.46% mAcc on nuScenes. However, it uses a different point-cloud backbone and initialization, making it a complementary method-level comparison rather than a controlled comparison of training objectives. We therefore use ULIP-style as the primary pairwise baseline, with the same LitePT-S backbone, data, and training settings as CTP.

Q1: We compare both methods with matched negative counts: 127 negatives per anchor for each pairwise contrastive term and 127 sampled negative triplets per anchor for CTP. Both use the same encoders, batch size, and 100-epoch training schedule. As shown in Appendix Figure 14, Pairwise achieves better retrieval early in training, while CTP improves more gradually and eventually surpasses it. At epoch 100, CTP achieves 86.54% mAcc and 4.21/14.64/23.22% R@1/R@5/R@10, compared with 71.42% and 3.84/12.57/19.57% for Pairwise.

Q2: To examine the joint candidate-space formulation, we compare ULIP-style with CTP without masking. On nuScenes, mAcc improves from 71.07% to 73.15%, and R@1/R@5/R@10 improves from 3.82/13.52/20.72% to 7.04/21.82/31.53%. Since tuple candidates require a scoring function, we compare cosine-sum with TRIANGLE and GRAM scorers under the same candidate-space framework. Updated Figure 5 reports results with and without masking for both classification and retrieval. Masking improves all three retrieval metrics for these scorers on ShapeNet, while on nuScenes, consistent retrieval improvements are observed for CTP. These comparisons examine scorer choice and masking separately within the joint framework. ULIP-style is excluded from the masking ablation because the proposed triplet mask does not directly apply to its pairwise objective.

Q3: Thank you for these suggestions. We have expanded the related-work discussion to include Uni3D for point clouds, UniGS and TIGaussian for 3D Gaussian Splatting, MeshCLIP for meshes, and TriCoLo for voxels, situating our work within multimodal learning across different 3D representations.

Q4: For zero-shot, we refer to classification using text-derived class prototypes, without training an additional classifier or performing further fine-tuning at evaluation time. But the categories in the test set are not no-seen, the similarity objects may exist in the training set. To address this concern, we additionally evaluated all methods using captions generated without ground-truth category prompts. The results are shown in Appendix Figure 15. On nuScenes, the pairwise baseline’s 8-class mAcc decreases from 71.07% to 51.59%, whereas CTP decreases from 74.26% to 60.63%. For joint image–text-to-point-cloud retrieval, the pairwise baseline’s R@1/R@5/R@10 changes from 3.82/13.52/20.72% to 3.88/13.18/20.51%, while CTP changes from 7.44/22.42/32.18% to 7.09/21.12/30.77%. Thus, CTP retains its advantage with category-agnostic captions. We suspect that removing category hints sometimes causes captions to describe surrounding objects rather than the intended target, which may contribute to the drop in classification performance.

Q5: We compared masking all negative triplets containing a matched pair with the proposed anchor-sharing mask. Both achieve similar classification accuracy, but additionally removing non-anchor diagonal triplets reduces retrieval recall on both datasets. We therefore adopt the anchor-sharing mask as our final strategy. Results are shown in Appendix Table 5.

Given that the discussion period ends soon, we'd appreciate any reviewer feedback on our analysis and experiments provided in the rebuttal, and know if there are any outstanding concerns. Thanks!

Reviewer_3VmqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Contrastive Tensor Pre-training (CTP) for the unified text, Image, and point cloud feature alignment. Be different with the existing methods, CTP aims to create a virtual anchor representation and force all the domain to be aligned at the same time. Experiments shows it's superious compared with previous methods.

Critical strengths
  1. The structure and design are easy to understand and follow.

  2. The new candidate feature space provides more combinations for three modality units, which constructed a more stable optimization signal.

Critical weaknesses
  1. Claims. CTP solves the pairwise contrastive objectives by introducing three-modality alignment. I'm concerned about the difficulty of construct suitable training batch in bigger candidate three modality units . Besides that, in experiments, CTP takes advantage of a more morden point cloud encoder. So the comparison is not fair.

  2. One of the main reason that previous methods don't take three dimension to align is the limiting of enough high quality paired data. The paper uses Qwen3-VL-8B-Instruct to create pseudo labels for training. The quality of the pseudo labels is not evaluated and reported.

Questions for authors
  1. How about to replace the ULIP-style method with the same point cloud encoder LitePT-S? I wonder a fairer comparison to support your claims.

  2. As reminded in weakness, is the setting of point cloud same across all the compared methods? As reported in ULIP, the choose of point cloud encoder would create huge impact for the final results.

2 comments

Thanks for the time and helpful comments. We address each point below, with revisions highlighted in blue.

W1: Given well-aligned text–image–point-cloud triplets, CTP can use the same batch construction strategy for both small or large batch sizes. We acknowledge the computational concern: the full candidate space grows cubically with batch size, compared with quadratic growth for pairwise methods. To reduce this overhead, we randomly subsample candidate triplets. Appendix Figure 14 shows that CTP continues to improve with additional training under reduced candidate budgets, illustrating the trade-off between candidate count, training cost, and performance. Regarding encoder fairness, all compared methods use the same text, image, and point-cloud encoders, including LitePT-S as the point-cloud backbone. We retrain all methods on the same datasets under matched training settings.

W2: We agree that obtaining high-quality paired data is a challenge for multimodal alignment, particularly for sparse real-world point clouds. We therefore construct aligned text–image–point-cloud triplets for training. To assess caption quality, we compared several prompt variants, ranging from a simple one-sentence instruction to more constrained prompts. We manually inspected captions generated for the same randomly sampled 10% subset across these variants. This inspection informed our choice of the final prompt, which produced more consistent, object-focused descriptions. We provided the category label to help direct the caption toward the corresponding object.

Q1: The ULIP-style baseline was trained using the same LitePT-S point-cloud encoder and datasets as CTP, under matched training settings. We also compared PointNet++ and LitePT-S and observed broadly comparable performance. PointNet++ performed slightly better on dense ShapeNet point clouds, while LitePT-S performed better on sparse nuScenes point clouds. We adopted LitePT-S for its performance on sparse data and its transformer-based architecture, which we plan to explore for integration with VLM decoders in future work. The backbone comparison is provided in Appendix Table 6.

Q2: Yes, we use the same setting of point cloud across all the compared methods.

Given that the discussion period ends soon, we'd appreciate any reviewer feedback on our analysis and experiments provided in the rebuttal, and know if there are any outstanding concerns. Thanks!

Reviewer_JwxLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the joint representation alignment of text, image, and point cloud modalities by identifying a key limitation in existing CLIP-extended 3D representation learning methods, which primarily rely on pairwise contrastive objectives that optimize separate modality pairs rather than multimodal tuples jointly. To overcome this issue, the authors introduce Contrastive Tensor Pre-training (CTP), a framework that expands the conventional 2D pairwise similarity matrix into a 3D similarity tensor S∈Rb×b×bS \in \mathbb{R}^{b \times b \times b}, enabling the direct contrasting of complete text-image-point-cloud triplets within a unified candidate space. Empirical evaluations on real-world automotive LiDAR from nuScenes and dense synthetic shapes from ULIP-ShapeNet, along with zero-shot cross-dataset transfer to KITTI and Waymo, show that CTP consistently outperforms both the matched pairwise baseline (ULIP-style) and alternative joint methods including Symile, TRIANGLE, and GRAM in zero-shot joint classification and instance-level cross-modal retrieval.

Critical strengths

The paper offers an intuitive and insightful geometric analysis of the inductive biases inherent in multimodal tuple scorers by modeling the relative configuration of unit-normalized representations across the three modalities as a tetrahedral feasible domain within a three-dimensional angle space (Figure 3 and Figure 12). Through this geometric perspective, the authors clearly elucidate the fundamental differences in optimization landscapes between the proposed additive cosine scorer and alternative area- or volume-based formulations such as TRIANGLE and GRAM.

Critical weaknesses

W1: The formulation and masking of partial-match negatives exhibit a noticeable theoretical omission by failing to account for the non-anchor positive pairs lying on the main diagonal of the candidate plane (i=j≠ki = j \neq k). In Section 3.3 and Figure 4, when fixing a modality anchor, the authors strictly define partial-match negatives as the anchor-associated row (i=k≠ji = k \neq j) and column (j=k≠ij = k \neq i). However, unmasked entries along the diagonal where i=j≠ki = j \neq k under the additive tuple scorer SiikCTPS_{iik}^{\text{CTP}} also intrinsically contain the ground-truth image-text pair (ti,vi)(t_i, v_i) of the ii-th instance and its similarity term ti⊤vit_i^\top v_i. Given that all three modality encoders are jointly updated (Section 4.1.2), retaining these diagonal candidates in the denominator during the optimization of the anchor loss lkPl_k^P inevitably exerts a suppressive gradient on ti⊤vit_i^\top v_i, thereby penalizing the alignment of the remaining b−1b-1 valid image-text positive pairs within the same batch.

W2: The additive tuple scorer mathematically degenerates into pairwise baselines during inference, while its theoretical advantage over pairwise contrastive objectives during training remains insufficiently unpacked. As shown in Equations 15 and 16, for both zero-shot classification and joint retrieval, the intra-query cross-modal inner product is strictly constant across candidates and drops out of the ranking function entirely. Consequently, CTP's inference scoring functions, $s_c $ and $r_{qk} $, are algebraically identical to those of the ULIP-style pairwise baseline, capturing zero high-order tri-modal interactions.

W3: The evaluation protocol fundamentally departs from standard practices in 3D cross-modal representation learning by completely omitting single-modality zero-shot assessments. All primary benchmarks in this paper (Tables 1, 2, 3, and Figure 5) enforce bi-modal joint queries during testing. Specifically, classification requires both the cropped image vqv_q and point cloud pqp_q, while retrieval strictly requires joint text-and-image queries (tq,vq)(t_q, v_q). Crucially, because the image tower is initialized with the strong pretrained CLIP ViT-B/32 backbone (Section 4.1.2), the visual representation vq⊤tcv_q^\top t_c alone could trivially dominate and achieve high zero-shot performance on cropped RGB objects in nuScenes. Without providing essential baselines, it is impossible for readers to discern whether CTP genuinely learns superior 3D geometric representations or merely relies on the dominant, pretrained 2D CLIP visual priors.

W4: The pseudo-caption generation pipeline suffers from severe ground-truth label leakage. As detailed in Appendix A.1 (lines 791–804), the descriptive pseudo-captions for nuScenes, KITTI, and Waymo were generated using Qwen3-VL-8B-Instruct by directly prompting the model with the ground-truth category label, resulting in generated captions that explicitly contain target class keywords.

W5: Jointly fine-tuning the pretrained CLIP encoders poses severe risks of catastrophic forgetting and semantic collapse, while the paper completely lacks comparisons with frozen backbones. Standard 3D-vision-language pre-training frameworks (e.g., ULIP, ULIP-2, and OpenShape) typically freeze the pretrained CLIP image and text towers and only optimize the point-cloud encoder to preserve CLIP's robust open-world alignment space. In contrast, this work jointly updates the pretrained CLIP ViT-B/32 image and text encoders on a relatively small driving cohort of only 29,917 low-resolution cropped objects and synthetic pseudo-captions (Table 4). This practice is highly susceptible to representational distortion and overfitting to localized driving crops.

Questions for authors

Please refer to the "Critical Weaknesses" section.

2 comments

Thanks for the time and helpful comments. We address each point below, with revisions highlighted in blue.

W1: Theoretically, when the point cloud serves as the anchor, the diagonal image–text pairs are indeed matched pairs. However, our experiments show that anchor-based masking achieves better joint retrieval than additionally masking these diagonal pairs. We interpret each matched image–text pair as a combined representation of another instance, which remains a valid negative relative to the point-cloud anchor. Retaining these candidates therefore helps distinguish the anchor from other instances, although it also introduces a suppressive gradient on their internal image–text alignment. The results in Appendix Table 5 support retaining this instance-discrimination signal, motivating the masking strategy adopted in our paper.

W2: Our primary focus is the training objective, where the tuple scorer measures multimodal compatibility and is jointly normalized over candidate tuples. This joint normalization produces different negative weights and gradients from separate pairwise contrastive losses. Inference evaluates the resulting cross-modal alignment. Thus, equivalent inference ranking functions do not imply equivalent training objectives or learned representations.

W3: This work investigates multimodal alignment, and we therefore adopt joint evaluation to assess the effectiveness of our method when multiple modalities are involved. To further examine alignment between point clouds and individual modalities, we provide additional results in Appendix Table 7 and 8 of the appendix. CTP achieves the best zero-shot classification performance using only point clouds and class text prompts, without RGB images at inference. For retrieval, the pairwise baseline performs better on text-to-point-cloud retrieval, with CTP trailing by approximately two percentage points, while CTP achieves the best image-to-point-cloud retrieval performance.

W4: In our setup, training-set category labels guide the VLM to generate richer, instance-specific captions while reducing category misidentification and hallucination. Only training-split captions are used for optimization, so no test-set labels are leaked into training. Classification uses predefined class prompts rather than instance captions, while label-guided retrieval queries contain category information. We additionally evaluate category-agnostic captions generated without label guidance. As shown in Appendix Figure 15, CTP remains the best-performing method across joint classification and all retrieval metrics. Its mAcc decreases from 74.26% to 60.63%, while R@5 decreases from 22.42% to 21.12%, showing that its relative advantage persists without category guidance.

W5: Our goal is to study multimodal pretraining, so we jointly optimize all three encoders to evaluate the effectiveness of the proposed training objective. We initialize the image and text encoders with pretrained CLIP weights to build on their existing alignment and reduce the training required to incorporate the point-cloud modality. We evaluate this approach on both nuScenes and ShapeNet, covering driving scenes and synthetic objects.

Given that the discussion period ends soon, we'd appreciate any reviewer feedback on our analysis and experiments provided in the rebuttal, and know if there are any outstanding concerns. Thanks!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.