- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Contrastive Tensor Pre-training (CTP) for jointly aligning text, image, and point-cloud embeddings. Instead of optimizing three separate pairwise contrastive objectives, CTP constructs a batch-wise similarity tensor over all text-image-point tuples. Tuple similarity is the sum of the three pairwise cosine similarities. For each anchored modality, the loss normalizes over the Cartesian product of candidates from the other two modalities. The authors identify “partial-match negatives” that preserve one matched pair involving the anchor and remove the corresponding row and column from the denominator. Experiments train CLIP text/image encoders jointly with a LitePT-S point encoder on object triplets from nuScenes and ULIP-ShapeNet, with cross-dataset evaluation on KITTI and Waymo.
Critical strengths
- The paper studies a clear and relevant question: whether text-image-point-cloud tuples should be contrasted jointly rather than only through separate modality pairs. The tensor formulation and partial-match-negative issue are clearly explained.
- The pairwise and joint methods use matched encoders, initialization, input processing, and optimization settings. Experiments cover both sparse real-world LiDAR and dense ShapeNet point clouds, with substantial retrieval gains and cross-dataset evaluation.
Critical weaknesses
- The experimental validation is insufficient. The current comparisons do not isolate the effects of the larger candidate set, tensor normalization, cosine-sum scorer, and partial-match masking. Candidate- and compute-matched pairwise baselines, together with factorial ablations of these components, are needed to establish where the gains come from.
- The methodological novelty is incremental. Joint contrastive learning over complete multimodal tuples has already been explored by Symile, GRAM, and TRIANGLE. CTP mainly replaces the tuple scorer with a simple sum of pairwise cosine similarities and masks partial-match negatives; moreover, its inference-time ranking is identical to the pairwise scorer. Thus, the substantive novelty is narrower than the paper’s framing suggests.
- Important 3D multimodal baselines are missing. The experiments compare mainly with a ULIP-style objective and general multimodal scorers, but not with representative text-image-3D pretraining methods such as Uni3D.
Questions for authors
- It would be helpful to compare CTP with a pairwise baseline using roughly the same number of negatives and a comparable compute budget. Do the gains remain in this setting?
- Could the authors disentangle the contributions of the tensor candidate space, cosine-sum scorer, and masking, ideally on both classification and retrieval?
- The related-work discussion could be broadened to cover multimodal learning with different 3D representations, including Uni3D (Zhou et al., 2024) for point clouds, UniGS (Li et al., 2025) and TIGaussian (Liu et al., 2026) for 3D Gaussian Splatting, MeshCLIP (Song et al., 2023) for meshes, and TriCoLo (Ruan et al., 2024) for voxels.
- Since ground-truth categories are provided when generating the pseudo-captions, please clarify in what sense the evaluation is “zero-shot.” Results with category-agnostic captions would help address this concern.
- I am also curious how the proposed anchor-sharing mask compares with no masking and with masking every tuple that contains a matched pair, especially for retrieval.
I will certainly consider raising my rating if the concerns above can be resolved.
2 comments
Given that the discussion period ends soon, we'd appreciate any reviewer feedback on our analysis and experiments provided in the rebuttal, and know if there are any outstanding concerns. Thanks!
Thanks for the time and helpful comments. We address each point below, with revisions highlighted in blue.
W1: We have expanded the experiments to examine candidate count, scorer choice, and masking. Please refer to our responses to Q1, Q2, and Q5 for the corresponding comparisons and ablations.
W2: To our knowledge, prior work has not explored contrastive learning over complete multimodal tuples specifically for text–image–point-cloud alignment. Our work investigates the feasibility and effectiveness of this formulation. Our scorer design prioritizes alignment behavior rather than mathematical complexity. We explored SVD-based alternatives, but found that certain formulations can attain an optimal score of zero even when vectors point in opposite directions on the unit hypersphere, which does not reflect the desired alignment. We therefore adopt a cosine-sum scorer and introduce anchor-sharing masking to address partial-match negatives arising from multimodal tuple construction.
W3: We evaluated several additional 3D multimodal methods. PointCLIP and PointCLIP V2, evaluated in their training-free zero-shot settings, performed poorly on our sparse nuScenes point clouds. Our Uni3D adaptation achieved stronger joint classification performance, reaching 80.46% mAcc on nuScenes. However, it uses a different point-cloud backbone and initialization, making it a complementary method-level comparison rather than a controlled comparison of training objectives. We therefore use ULIP-style as the primary pairwise baseline, with the same LitePT-S backbone, data, and training settings as CTP.
Q1: We compare both methods with matched negative counts: 127 negatives per anchor for each pairwise contrastive term and 127 sampled negative triplets per anchor for CTP. Both use the same encoders, batch size, and 100-epoch training schedule. As shown in Appendix Figure 14, Pairwise achieves better retrieval early in training, while CTP improves more gradually and eventually surpasses it. At epoch 100, CTP achieves 86.54% mAcc and 4.21/14.64/23.22% R@1/R@5/R@10, compared with 71.42% and 3.84/12.57/19.57% for Pairwise.
Q2: To examine the joint candidate-space formulation, we compare ULIP-style with CTP without masking. On nuScenes, mAcc improves from 71.07% to 73.15%, and R@1/R@5/R@10 improves from 3.82/13.52/20.72% to 7.04/21.82/31.53%. Since tuple candidates require a scoring function, we compare cosine-sum with TRIANGLE and GRAM scorers under the same candidate-space framework. Updated Figure 5 reports results with and without masking for both classification and retrieval. Masking improves all three retrieval metrics for these scorers on ShapeNet, while on nuScenes, consistent retrieval improvements are observed for CTP. These comparisons examine scorer choice and masking separately within the joint framework. ULIP-style is excluded from the masking ablation because the proposed triplet mask does not directly apply to its pairwise objective.
Q3: Thank you for these suggestions. We have expanded the related-work discussion to include Uni3D for point clouds, UniGS and TIGaussian for 3D Gaussian Splatting, MeshCLIP for meshes, and TriCoLo for voxels, situating our work within multimodal learning across different 3D representations.
Q4: For zero-shot, we refer to classification using text-derived class prototypes, without training an additional classifier or performing further fine-tuning at evaluation time. But the categories in the test set are not no-seen, the similarity objects may exist in the training set. To address this concern, we additionally evaluated all methods using captions generated without ground-truth category prompts. The results are shown in Appendix Figure 15. On nuScenes, the pairwise baseline’s 8-class mAcc decreases from 71.07% to 51.59%, whereas CTP decreases from 74.26% to 60.63%. For joint image–text-to-point-cloud retrieval, the pairwise baseline’s R@1/R@5/R@10 changes from 3.82/13.52/20.72% to 3.88/13.18/20.51%, while CTP changes from 7.44/22.42/32.18% to 7.09/21.12/30.77%. Thus, CTP retains its advantage with category-agnostic captions. We suspect that removing category hints sometimes causes captions to describe surrounding objects rather than the intended target, which may contribute to the drop in classification performance.
Q5: We compared masking all negative triplets containing a matched pair with the proposed anchor-sharing mask. Both achieve similar classification accuracy, but additionally removing non-anchor diagonal triplets reduces retrieval recall on both datasets. We therefore adopt the anchor-sharing mask as our final strategy. Results are shown in Appendix Table 5.