- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors use the teacher’s predicted class probabilities to estimate the probability that two images belong to the same class. These estimates provide soft pairwise supervision for the student to learn relationships between samples in feature space, rather than directly using the teacher’s pseudo-labels for classification training.
Critical strengths
The paper is clearly presented and easy to follow. The method is simple, and its main idea is easy to understand. Using soft pairwise targets to preserve uncertainty in the teacher’s predictions is a reasonable design choice. The method also requires neither source data nor additional auxiliary losses.
Critical weaknesses
Proposition 1 compares the expected squared errors of the unnormalized pairwise weights and , whereas Equation 3 uses the normalized weights . The proof therefore does not fully establish the connection to the loss used in practice.
Figure 2 uses binned calibration statistics to verify the conditions. However, the binning procedure, the estimation of the true same-class probabilities, and how these statistics correspond to the conditions in the proposition are not sufficiently explained. This analysis provides some intuition, but I find the explanation of why the method is more robust to noise insufficient.
Table 2 provides an ablation study, but lacks a controlled comparison that replaces PSCL with CE while keeping all other settings unchanged. Such a comparison would help isolate the contribution of the loss function from that of other design choices.
Another concrete concern is the experimental comparison. The authors directly use the baseline results reported in GOLD, which uses a batch size of 64 on ImageNet-C, whereas Table 3 of this paper specifies a batch size of 100. The experimental conditions are therefore not fully matched.
Furthermore, the authors have not provided source code for reproducing the results. This makes it difficult to independently verify the implementation and experimental settings, particularly whether the augmentation, buffering, and evaluation procedures are consistent with those used for the cited baseline results.
Questions for authors
- How does the error bound for in Proposition 1 translate to the normalized weights used in Equation 3? Could the authors provide an error analysis after normalization, or clarify what additional conditions are needed to connect the existing result to the noise robustness of the actual loss?
- As discussed in the weaknesses, could the authors provide an online ablation that replaces Soft PSCL with CE while keeping all other settings unchanged? This would help clarify how much of the improvement comes from the loss function itself.
- This paper uses a batch size of 100 on ImageNet-C, whereas the cited GOLD results use a batch size of 64. Could the authors explain this difference in experimental settings?
3 comments
Thank you for the detailed response and additional experiments. The new controlled comparisons with CE directly address my concern about the contribution of the loss function, and the ImageNet-C rerun with a matched batch size corrects the previous discrepancy. I appreciate these additions, which strengthen the empirical evidence for the method.
One detail in the theoretical analysis still needs clarification. In Eq.3, the denominator of is , which includes the current partner . However, Appendix A.4 explicitly excludes from all row sums. The normalized targets analyzed in the appendix therefore still differ from those used in the actual loss. If this difference becomes negligible as the number of partners grows, could the authors clarify the required conditions or provide an error analysis connecting the result to Eq.3?
Overall, the additional experiments have substantially alleviated my main experimental concerns, and I maintain my recommendation of weak acceptance.
Thanks for maintaining the recommendation.
Yes, partner is in the denominator of , as in the original SupCon definition (Khosla et al., 2020, Eq. 2). In the proof for the normalized targets, the true row sum is treated as given, although it contains , which we define as the random quantity. The row sums with and without differ only by the single term , so they become almost equal as grows:
We will highlight in Appendix A.4 that the proof neglects this difference for large enough m, and that the result holds for the targets of Eq. 3 as grows. Moreover, our empirical check in Appendix A.4 computes all row sums (true, soft, and hard) as in Eq. 3, including . We will mention this as well.
We thank the reviewer for the positive assessment and the precise questions.
[E1] Proposition 1 and the normalized weights (W1, Q1)
We extended Proposition 1 to the row-normalized targets of Eq. 3 (Section 3.1, Appendix A.4).
[E2] Binned calibration statistics (W2)
Appendix A.3 now describes the procedure: how pairs are binned, how the same-class probability is estimated per bin, and how each bin is checked against Eq. 4. The submission's calibration figure is replaced by Figure 4. Figure 4 adds ImageNet-C, separates true-positive and true-negative pairs, and varies the number of bins. Details are in [J5]. On why the loss is more robust to noise, please see [J3].
[E3] Loss-only comparison with cross-entropy (W3, Q2)
We added soft and hard cross-entropy under the same pipeline, changing only the loss. Soft PSCL is more accurate in every case:
Details are in [J2].
[E4] Batch size on ImageNet-C (W4, Q3)
Thank you for spotting this. The mismatch was a mistake on our side. We reran ImageNet-C with batch size 64 and the same 5,000 images per corruption as GOLD. Soft PSCL reaches 59.17% mean error with a standard deviation of 0.10 pp, compared with GOLD's 59.3% (Table 1, Section 5). Table 6 is corrected accordingly. The Soft-versus-Hard PSCL ablation in Table 3 still uses batch size 100 for both variants, as its caption now states.
[E5] Source code (W5)
We are preparing the code and will share an anonymous repository link in this discussion. Until then, the settings you mention are documented in the revision:
We thank the reviewer again for their efforts and the positive assessment.