acceptodds
Withdrawn from ICLR 2027

Reweighted Contrastive Learning for Robust Point Cloud Representations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Contrastive self-supervised learning has achieved promising performance in 3D point cloud representation learning, typically using the standard NT-Xent loss. However, NT-Xent assigns the strongest repulsive gradients to high-similarity negatives, including false negatives that share similar geometric semantics, which can disrupt intra-class compactness and increase sensitivity to sampling noise. To address this issue, we propose Reweighted Contrastive Loss for Robust Representations (ReCoR²), derived from a maximum likelihood estimation perspective. ReCoR² treats anchor–candidate similarity as evidence of shared latent geometric semantics, applies weaker repulsion to higher-similarity candidates, and retains strong repulsion for low-similarity ones with larger geometric differences. Since similarity estimates can be unreliable early in training, we adopt a two-stage optimization strategy: NT-Xent is first used to establish a semantically structured representation space, followed by optimization with ReCoR². Theoretically, we establish an upper bound on the gradient norm of ReCoR² with respect to sampling noise, which is strictly lower than the corresponding lower bound for standard NT-Xent, demonstrating improved robustness to sampling perturbations. Integrated into a 3D–2D cross-modal pretraining framework, ReCoR² consistently improves performance over representative contrastive baselines on downstream classification and part segmentation tasks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
point cloud representation learning, contrastive learning, self-supervised learning, false negatives

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BHemOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper looks at false negatives in contrastive pretraining for point clouds. In the standard NT-Xent loss, the more similar a negative is to the anchor, the harder it is pushed away. If that negative is really from the same class, this hurts. The authors propose ReCoR2, which multiplies each candidate in the NT-Xent denominator by a weight that decreases with similarity (a softmax over negative similarities). The result is that similar candidates are pushed away weakly and dissimilar ones strongly. Because similarities are not reliable early in training, the method first trains with NT-Xent and then switches to ReCoR2.

The paper gives a gradient analysis. Under three assumptions about how false and true negatives are separated in feature space, the gradient that a noisy false negative receives under ReCoR2 is bounded below the corresponding NT-Xent gradient. The loss is used inside a CrossPoint-style 3D-2D framework, extended with attention-based fusion of two point cloud views and a Chamfer-based geometry regressor. It is evaluated with linear and few-shot classification on ModelNet40 and ScanObjectNN and part segmentation on ShapeNetPart, with PointNet and DGCNN backbones. Small ablations cover the loss terms, the two-stage schedule, the choice of weighting function, and an injected false-negative setting.

Critical strengths

S1. A simple, clearly motivated change. The method is a small change to a standard loss and is easy to add to existing pipelines. The gradient analysis in Section 3.2 makes the motivation concrete: NT-Xent pushes similar negatives harder, and ReCoR2 reverses this. The paper is honest that this only makes sense once the feature space is informative, and builds the two-stage schedule around that.

S2. The key assumption is checked. Before using similarity as a weight, the paper tests on the Stage I features whether higher similarity really means a higher chance of the same class (Figure 3). It also adds a controlled experiment with injected false negatives (Appendix B.2). The method gives small but consistent gains over the reported baselines on both backbones and three datasets.

Critical weaknesses

W1. What the loss actually does is different from how it is presented. This is my biggest concern. The weights are chosen so that they exactly cancel the exponential terms in the denominator. After simplification (Eq. 8), the loss is the positive similarity plus a log-sum-exp over negative similarities. This is a soft minimum: the gradient goes almost entirely to the most dissimilar candidates, which are already far from the anchor. With the small temperature used, nearby negatives, both false and true ones, get almost no gradient. So the loss is not a mild reweighting of NT-Xent. It removes hard-negative pressure completely and focuses on easy negatives. This explains why the loss fails when used from the start (Table 4 shows it is then worse than plain CrossPoint), and why it only works as a short fine-tuning stage after NT-Xent.

The theory has the same issue. The theorem shows that a high-similarity sample gets a smaller gradient under ReCoR2. This is true by construction of the weights. It also assumes that every true negative lies below a similarity threshold (Assumption 2), which means it assumes there are no hard negatives. Under the same analysis, a true negative that is close to the anchor would also be ignored. That is the case where contrastive learning needs the gradient most, and the paper does not discuss it. The "maximum likelihood" framing is also not backed by a likelihood. No model or likelihood is written down, and the decreasing weight is a design choice, not a derivation.

W2. The gains are small and not clearly due to the loss. The reported improvements over the strongest baseline are small on most benchmarks. Part segmentation is a tie. The linear and segmentation results have no standard deviations, and the few-shot gaps are smaller than the reported standard deviations. More importantly, the comparison is not controlled:

  • The framework is not the same as the baselines. It adds attention-based fusion of the two point cloud views and a geometry regressor with a Chamfer loss. The method also trains longer in total (NT-Xent stage plus ReCoR2 stage). The key control is missing: the same framework, trained for the same total epochs with NT-Xent only. The injected false-negative table suggests that, inside this framework, the gap between NT-Xent and ReCoR2 is small.
  • There is no comparison with existing false-negative methods (debiased contrastive learning, hard-negative sampling, incremental false-negative detection), even though they are cited in Section 2.2. These are the most relevant baselines for this paper.
  • The title and theory are about robustness to sampling noise, but there is no robustness experiment (for example, varying point density, jitter, or a corrupted benchmark such as ModelNet-C).
  • Baseline numbers are copied from earlier papers, and masked-modeling methods discussed in related work are left out of the tables.
Questions for authors

Q1. Given Eq. 8, the gradient on each negative is proportional to exp(-s/tau), so it concentrates on the least similar candidates. Can you show how the gradient is spread across negatives during Stage II? How does ReCoR2 treat a true negative that is very similar to the anchor (a hard negative of a different class)?

Q2. Can you report the same framework (with view fusion and the geometry regressor) trained with NT-Xent only for the same total number of epochs as Stage I plus Stage II? Can you also report CrossPoint trained for the same total epochs? If ReCoR2 still gives a clear gain over these, my main concern in W2 would be resolved.

Q3. How does ReCoR2 compare with debiased contrastive loss, hard-negative sampling, and incremental false-negative detection, applied in the same framework and with the same two-stage schedule?

Q4. The paper claims robustness to sampling noise. Can you add an experiment that tests this directly, such as accuracy under varying point density, jitter, or a corrupted point cloud benchmark?

Q5. With batches of 16 objects drawn from 55 categories, how often does a batch contain same-class pairs? Also, in Section 3.1 the false negatives are described as noisy observations of the same surface. In the standard setup, other views of the same object are positives, not negatives. Which samples are the false negatives in practice?

Q6. Injected false negatives (Appendix B.2). How are false negatives injected? Why does the reweighted accuracy in this table differ from the main result under the same backbone and dataset?

Q7. Which likelihood is being maximized? If there is a probabilistic model behind the weights, please state it. Otherwise, I suggest describing the weights as a design choice.

Minor:

Q8. Please report standard deviations over several pretraining seeds for the linear classification, segmentation, and ablation results (Tables 1, 3, 4, 5). Many of the differences there are very small.

Q9. Table 1 has no PointNet result for CCPoint on ScanObjectNN, so the best result in that column is compared against an incomplete set. Please note this in the table.

Q10. Figure 4 says accuracy "exceeds the baseline" after the switch. Which baseline is meant? Is it NT-Xent continued for the same number of epochs?

Reviewer_VnLBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem that similar negatives in point-cloud contrastive learning can be false negatives, and that NT-Xent provides a large backlash gradient to them. The proposed method reduces the backlash of highly similar negatives by pretraining with NT-Xent in Stage I and then fine-tuning with the proposed loss in Stage II. Evaluate classification, few-shot, and part-segmentation performance on ModelNet40, ScanObjectNN, and ShapeNetPart.

Critical strengths
  • The issue that false negatives can affect the representation quality of point-cloud SSL is valid.
  • A two-stage strategy that applies similarity-based weighting after Stage I is a reasonable design that takes into account the instability of early training.
  • I tried to expand the evaluation scope to include two backbones and several downstream tasks.
  • Results of improvements compared with CrossPoint and CCPoint are presented in Table 1 and Table 2.
Critical weaknesses
  • Key related research and comparisons are missing. Self-Supervised Image-to-Point Distillation via Semantically Tolerant Contrastive Loss at CVPR 2023 addresses the problem of large gradients for similar negatives in 3D–2D contrastive learning and proposes a loss that reduces the contribution of similar negatives. Although the exact form of the proposed loss differs, the problem setup and the direction of solution are very close. This manuscript does not cite or compare this paper. Therefore, the novelty and experimental superiority were not sufficiently demonstrated.
  • There is a lack of statistical evidence for the small magnitude of the improvement. For example, a difference of 0.2 percentage points in CCPoint and 0.1 percentage points in ShapeNetPart mIoU is reported for the DGCNN ModelNet40, but the table does not include the mean or standard deviation for the repeated experiments. Therefore, it is difficult to determine whether this difference is a reproducible improvement.
  • The protocol for the false-negative injection experiment is insufficient. Table 8 presents performance by false-negative ratio, but it is unclear how false negatives were defined and injected, how labels were used, and what the number of iterations and variance are.
Questions for authors
  • What is the key difference between Semantically Tolerant Contrastive Loss and the proposed method, and can they be directly compared under the same backbone and training budget?
  • Can the means, standard deviations, and number of repetitions for multiple seeds be provided for the results in Table 1, 3, and 8?
  • What is the exact procedure used to define and inject a false negative in Table 8?
Reviewer_QwZBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReCoR² (Reweighted Contrastive Loss for Robust Representations) to mitigate the false-negative problem in point cloud contrastive learning. The method observes that standard NT-Xent assigns stronger repulsive gradients to highly similar negatives, which may excessively separate semantically or geometrically related samples. ReCoR² therefore reweights negative gradients according to feature similarity, reducing repulsion for high-similarity candidates while preserving stronger repulsion for dissimilar ones. Since similarity estimates are unreliable early in training, the authors adopt a two-stage strategy that first learns a structured representation space with NT-Xent and then applies ReCoR² for reweighted optimization. Integrated into a 3D–2D cross-modal pre-training framework, ReCoR² achieves consistent improvements on ModelNet40, ScanObjectNN, and ShapeNetPart across classification, few-shot learning, and part segmentation, with additional analyses supporting the proposed mechanism.

Critical strengths

1. The paper revisits the false-negative problem from the perspective of gradient allocation, with a clear motivation

Rather than discussing false negatives only at a conceptual level, the paper explicitly analyzes the gradient response of NT-Xent to negative samples: the higher the similarity between an anchor and a negative sample, the stronger the repulsive gradient assigned to that negative. For point clouds, however, different instances or different samplings may exhibit highly similar geometric semantics, making such gradient allocation potentially harmful to false negatives. This analysis provides a clear problem formulation and establishes a direct motivation for the proposed reweighting strategy.

2. The two-stage training strategy is reasonable and explicitly addresses the dependency of the method on similarity reliability

ReCoR² relies on the assumption that similarity reflects potential semantic consistency. However, the representation space at the early stage of training is usually not semantically reliable. Instead of applying similarity-based reweighting from random initialization, the authors first train the model with NT-Xent for 300 epochs and then perform 100 epochs of ReCoR² fine-tuning. In the ablation study, applying reweighting from the beginning reduces performance from 92.6% to 91.1%, indicating that the two-stage design plays an important role in the effectiveness of the method.

3. The theoretical analysis and mechanism-oriented experiments form a relatively complete validation chain

The paper not only compares the negative-sample gradients of ReCoR² and NT-Xent, but also derives a sampling-noise gradient bound under several geometric assumptions and provides a sufficient condition under which the upper bound of ReCoR² is lower than the lower bound of NT-Xent. Meanwhile, the authors validate the relationship between similarity and same-class probability using the Stage-I representation space. In the controlled false-negative-ratio experiment, as the false-negative ratio increases from 0.1 to 0.5, the advantage of ReCoR² over NT-Xent increases from 0.28 to 0.44 percentage points. These analyses are well aligned with the central motivation of the paper.

Critical weaknesses

1. The central claim of improved robustness to sampling perturbations lacks direct validation through corruption or perturbation experiments

A major theoretical motivation of the paper is that ReCoR² reduces sensitivity to sampling noise. However, the main downstream experiments are still conducted on standard ModelNet40, ScanObjectNN, and ShapeNetPart. Although ScanObjectNN contains realistic variations such as background clutter, occlusion, and geometric variation, it does not directly verify robustness to controlled sampling noise, jitter, dropout, density variation, or other point-cloud corruptions. The controlled false-negative-ratio experiment validates sensitivity to false-negative prevalence rather than robustness to input sampling perturbations themselves.

Therefore, the claimed “robust point cloud representations” and the theoretical sampling-noise robustness are not yet fully supported by direct experimental evidence. It would be helpful to include evaluations on ModelNet-C or ScanObjectNN-C, or at least systematically vary perturbation strength for jitter, point dropout, random subsampling, and density variation, and report the resulting performance degradation.

2. The theoretical modeling of sampling noise remains distant from realistic point-cloud sampling perturbations

The theoretical analysis models the corrupted feature as a clean manifold feature with an additive feature-space perturbation and directly studies the loss sensitivity to this perturbation. It also assumes that false negatives remain in the high-similarity region, true negatives lie in a relatively lower-similarity region, and a positive manifold margin exists.

This analysis is useful for characterizing the difference between ReCoR² and NT-Xent under these assumptions. However, it does not explicitly model how perturbations introduced in the input point-cloud space propagate through a nonlinear encoder and eventually affect the learned feature representation. Therefore, the current theory is more accurately interpreted as a conditional feature-space robustness result rather than a complete robustness guarantee against realistic point-sampling noise. The paper should more carefully distinguish these two levels of interpretation.

3. The paper lacks direct comparisons with representative methods specifically designed for false-negative mitigation

The Related Work section discusses Debiased Contrastive Learning, hard-negative debiasing, clustering-based false-negative detection, and related approaches. However, the main experimental comparisons focus on point-cloud self-supervised learning methods such as CrossPoint, CrossNet, GSPCon, and CCPoint.

Since the core contribution of this work is specifically false-negative handling, comparison only against general point-cloud SSL methods is insufficient to demonstrate the advantage of the proposed reweighting strategy over existing false-negative mitigation approaches. It would be useful to compare against debiased contrastive loss, false-negative elimination or detection strategies, or other similarity-aware baselines under exactly the same 3D–2D backbone, augmentation, batch size, and training budget.

Questions for authors

1. Can the authors provide direct robustness experiments under sampling perturbations or corruptions?

The theoretical motivation of the paper is to reduce sensitivity to sampling noise, but the current experiments do not directly evaluate this claim under different perturbation strengths. It would be helpful to compare NT-Xent, ReCoR², and other baselines on ModelNet-C, ScanObjectNN-C, or controlled perturbation settings involving jitter, point dropout, random subsampling, and density variation. Accuracy-versus-corruption-severity curves would provide more direct evidence for the claimed robustness.

2. Can the authors include a strictly matched-compute NT-Xent baseline?

The full ReCoR² training procedure consists of 300 epochs in Stage I followed by 100 epochs in Stage II. Please report at least one baseline trained with NT-Xent for the full 400 epochs to verify that the Stage-II improvement is not mainly due to additional training time. If possible, it would also be useful to report the training time and GPU cost of the two-stage method and the matched-compute baseline.

3. How does ReCoR² handle highly similar samples that are actually hard true negatives?

The core assumption is that high similarity provides stronger evidence of shared semantics, but hard true negatives may also exhibit high similarity. Could the authors report the proportions of false negatives and true negatives across different similarity intervals, and separately analyze the gradient behavior and decision-boundary effects of ReCoR² on high-similarity true negatives? Experiments on a more fine-grained dataset would also be informative.

4. Can the authors compare ReCoR² with existing false-negative mitigation methods under a fully unified training framework?

For example, Debiased Contrastive Learning, false-negative detection or removal, hard-negative debiasing, or other similarity-aware weighting strategies could be directly substituted for the current NT-Xent or ReCoR² objective while keeping the backbone, batch size, augmentation, and training epochs unchanged. Such a comparison would more directly demonstrate the methodological contribution of ReCoR².

5. Can the relationship between feature-space perturbation and realistic point-sampling noise be established more directly?

The current analysis directly models perturbations in feature space, while real perturbations first occur in the input point-cloud space and then propagate through a nonlinear encoder. Could the authors provide an encoder stability analysis, a local robustness analysis, or an empirical measurement of how input perturbations affect feature representations? This would make the theoretical claim of sampling-noise robustness more directly connected to realistic point-cloud perturbations.

6. How sensitive is the method to the switching point between the two training stages?

The current implementation switches from NT-Xent to ReCoR² after 300 epochs. It would be useful to evaluate different switching points, such as 100, 200, 300, and 400 epochs, or to explore an adaptive switching criterion based on similarity reliability, class-agreement proxies, or feature convergence. This would further clarify the necessity of the two-stage strategy and the sensitivity of ReCoR² to the training schedule.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.