- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes out-of-batch (OOB) regularization for joint-embedding SSL, which moves the anti-collapse term from the active minibatch to a dataset-level state. Every half epoch, the current encoder embeds the full training set, and per-instance targets are built globally. SIGReg-OOB assigns Gaussian quantiles along random 1D projections; Contrastive-OOB uses spherical anchors spread by a repulsion objective. These targets are stored on CPU, and during training each sample is only attracted to its own target, with no cross-sample interaction online. An oracle-gradient analysis shows that minibatch regularization gradients can point away from the full-dataset gradient at small batch sizes, while cached targets trade this error for staleness. Experiments on natural images and 2D/3D biomedical data cover classification, controlled imbalance, instance retrieval and DINOv2 continued pretraining. Gains are largest at small batch sizes, and GPU memory per batch is unchanged.
Critical strengths
- The core idea is simple and well motivated: decouple the statistical support of the regularizer from the optimization batch size. This is relevant for memory-limited settings such as 3D medical imaging and adapting large pretrained encoders.
- The target construction is simple and sound. For SIGReg-OOB, at each refresh the authors draw 256 random directions and project the embeddings of the whole training set onto each of them. On every direction, they sort the N projected values and assign the k-th ranked sample to the k-th quantile of a standard Gaussian, which in 1D is the optimal matching between the data and the Gaussian grid. During training, each sample’s projections are then regressed toward these fixed per-direction targets, and the directions are redrawn at the next refresh. The regularizer is thus computed against the whole dataset rather than the batch, and the online step reduces to a cheap per-sample regression.
- The oracle-gradient analysis is, to my knowledge, an original contribution. It shows that minibatch gradients can have negative cosine with the full-dataset gradient at small batch sizes, which is a relevant observation.
- The evaluation is broad (more than 19 datasets, though small), and the overhead is small.
Critical weaknesses
- The SIGReg baseline is unusually weak, it is apparently not the published Epps–Pulley objective but an in-batch sorted-quantile variant (Sec. 3.3), and it performs poorly (34.6 5-NN on natural images vs 50.2 for SimCLR). The headline SIGReg-OOB gains (+14pp) may therefore partly reflect a batch-fragile baseline rather than a general property of OOB.
- Contrastive-OOB gains on natural images are marginal, the average is +0.14pp linear probe (Table 2), and Contrastive-OOB is below SimCLR on CIFAR-10 (74.9/79.4 vs 76.2/81.1) and CIFAR-100 (33.4/49.4 vs 39.1/53.1) in Table 8.
- The MoCoV2 baseline, the closest analogue to OOB, is absent from table 2 and seems poorly configured for small datasets: its 4096-key queue is larger than many training sets, and its fixed 0.999 momentum barely updates the key encoder in short runs. It’s also missing from Table 2. A tuned version would be needed before concluding OOB beats queue-based methods.
- The most direct competitor, SogCLR (Yuan et al., 2022), is missing, is cited but not compared, although it targets exactly the minibatch bias of the global contrastive objective.
- The 2D experiments train ViT-S/16 from scratch on small datasets at 224×224 resolution. ViTs are known to underperform convolutional backbones in this regime, and the absolute numbers are low (e.g., CIFAR-10 5-NN of 49.2 for SIGReg and 76.2 for SimCLR). This makes it hard to tell whether OOB’s gains are general or compensate for a weak setup.
- In the oracle experiment, the gradients disagree (opposite sign) only when the batch size is extremely small (B= 1 or 2), which is rarely required, even in low-budget regime.
- (additional, not affecting the score) The mutual-information framing around Eq. 1 could cite more InfoMax-based SSL work, e.g., Hjelm et al. (2019), Poole et al. (2019), and Louiset et al. (ICLR 2024), which decompose InfoMax objectives into alignment and uniformity terms in the same spirit as the paper’s alignment/anti-collapse split.
Questions for authors
-Could you compare against the original Epps–Pulley SIGReg? Your in-batch quantile variant may make the +14pp gain look larger than it is. -Why does Contrastive-OOB barely help on natural images (+0.14pp linear) and even underperform SimCLR on CIFAR-10/100? -Could you rerun MoCoV2 with a queue smaller than N and a momentum schedule (0.99 → 0.999), and add it to Table 2? At N = 500–2k, the 4096-key queue contains stale copies of each image. -How does Contrastive-OOB compare with SogCLR, which targets the same minibatch bias at almost no cost? -Do the gains hold with a ResNet-18 at native resolution (maybe using GroupNorm instead of BatchNorm when batch sizes are particularly small), where baselines are stronger and comparable to published numbers? -Gradients only disagree in sign at B = 1–2, which is rarely needed in practice. Which realistic settings actually benefit, and do downstream gains track gradient agreement?
- At refresh time, the gradient of Eq. 7 seems to coincide with the gradient of the sliced Wasserstein-2 distance to an isotropic Gaussian, since sorting gives the optimal 1D coupling. If there is a connection, could you make it explicit?
5 comments
Reply to reviewer bKBA (2/3)
(2/3)
- Comparison with SogCLR. SogCLR is indeed a method designed to remove the minibatch bias of the global contrastive objective by keeping a per-sample moving average of the contrastive denominator, but each update still uses only the negatives in the current batch. Originally, we excluded it as the smallest batch size their paper considers is 128, but now we ran the comparison experiments using a 2D and a 3D dataset in the small-batch regime:
| Dataset | B | SogCLR | SIGReg-OOB | Contrastive-OOB |
|---|---|---|---|---|
| PathMNIST-5k | 2 | 83.5 / 88.6 | 90.3 / 91.9 | 91.7 / 92.8 |
| 4 | 88.6 / 91.7 | 90.3 / 92.8 | 91.5 / 93.0 | |
| 8 | 89.3 / 91.2 | 90.2 / 92.1 | 91.3 / 93.1 | |
| NoduleMNIST3D | 2 | 55.6 / 57.0 | 72.4 / 76.0 | 72.1 / 76.3 |
| 4 | 57.8 / 57.0 | 73.0 / 77.7 | 72.9 / 77.3 | |
| 8 | 69.0 / 71.3 | 75.0 / 79.0 | 73.5 / 74.9 |
Additionally, we run (only for batch size of 2) experiments on other datasets:
| Dataset | SogCLR | SIGReg-OOB | Contrastive-OOB |
|---|---|---|---|
| BloodMNIST-5k | 95.1 / 96.4 | 97.0 / 97.7 | 96.9 / 98.0 |
| STL-10 | 56.8 / 65.0 | 62.1 / 68.3 | 64.4 / 70.1 |
| ImageNet-100-5k | 49.5 / 64.8 | 57.8 / 67.5 | 62.8 / 70.4 |
| FractureMNIST3D | 35.6 / 39.6 | 36.1 / 42.4 | 36.2 / 39.7 |
To summarize, SogCLR does alleviate the small batch performance problems of SimCLR, but at the lowest batch sizes the negative effect is still pronounced, and both SIGReg-OOB and Contrastive-OOB outperform it.
- Backbone architecture and absolute performance. Originally, we wanted to use both ViT and a CNN to make sure our results are not associated with the model architecture. Therefore, our 3D experiments already use a 3D ResNet-18, where both OOB methods improve over their in-batch counterparts (Table 2). To test the 2D case as well, we now trained a ResNet-18 from scratch on CIFAR-10 and CIFAR-100, for 100 epochs and batch sizes 2, 16 and 64. SSL uses only the official training split, and frozen evaluation uses the official test split. Here, SIGReg uses the published Epps–Pulley objective. Results are 5-NN / linear-probe accuracy (%), averaged over three seeds:
| Dataset | B | EP-SIGReg | SIGReg-OOB |
|---|---|---|---|
| CIFAR-10 | 2 | 9.7 / 29.9 | 86.1 / 87.0 |
| CIFAR-10 | 16 | 78.2 / 82.3 | 80.1 / 82.7 |
| CIFAR-10 | 64 | 68.5 / 74.7 | 75.7 / 79.7 |
| CIFAR-100 | 2 | 1.0 / 9.7 | 48.5 / 52.0 |
| CIFAR-100 | 16 | 33.6 / 45.2 | 48.7 / 49.6 |
| CIFAR-100 | 64 | 25.9 / 38.7 | 47.7 / 49.2 |
With a convolutional backbone, the absolute numbers are much higher (e.g. 86.1 / 87.0 for SIGReg-OOB on CIFAR-10 at B=2), and the small-batch advantage remains. SIGReg-OOB outperforms SIGReg at every batch size. In-batch Epps–Pulley SIGReg collapses to near-chance 5-NN accuracy at B=2, which is exactly the failure mode that OOB is designed to avoid. More generally, our intention was to compare each OOB method to its in-batch counterpart under exactly the same backbone, augmentations and training recipe, which isolates the effect of where the regularizer is computed. We used a single, generic augmentation pipeline for all datasets, both natural and biomedical, rather than tuning it per dataset, so our absolute numbers are not directly comparable to published results where performance on individual datasets was the main goal.
Reply to reviewer bKBA (3/3)
(3/3)
-
Oracle gradient analysis. We agree that the gradient sign flips only at B=1, and we don't claim B=1 is typical. But the sign flip is just the extreme case. The cosine with the full-dataset gradient rises gradually and reaches around 0.90 only at B=64, so the anti-collapse term is already partially wrong for the batch sizes in between. Batch sizes of 2 to 8 are realistic when models or inputs are large: in Table 3 we continue pretraining ViT-G (1.1B) at B=2 and 8 on a single A100. The oracle analysis diagnoses the regularizer's estimation error, and we don't claim the oracle gradient is optimal for downstream performance.
-
InfoMax references. We thank the reviewer for providing new references for the MI framing. We will add Hjelm et al. (2019), Poole et al. (2019) and Louiset et al. (2024) to the discussion of Eq. 1 in the revised PDF.
-
Connection to sliced Wasserstein. Yes, we believe there is a connection and we are happy to put it in more explicitly. We will add the following to Sec. 3.3: "Along each direction , sorting gives the optimal 1D coupling between the projected embeddings and the Gaussian quantiles. So at refresh, Eq. 7 equals between the projected embeddings and a quantized . Averaged over directions, it is equivalent to the sliced distance to an isotropic Gaussian, and its gradient coincides with the sliced-Wasserstein gradient." This framing also clarifies two properties. First, between the refreshes the coupling between anchors and encoder outputs is kept fixed, thus our fixed coupling can only cost more than the optimal one (the cached loss upper-bounds , with equality at refresh). Staleness therefore loosens this bound but does not change the objective. Second, it explains the batch size dependency. In the in-batch variant, each sample's target depends on its rank among the other samples in the batch, so it changes with the batch. This makes it a minibatch OT loss, which is a different objective from the full-data distance. According to Fatras et al., 2020, it upper-bounds that distance, and the gap grows as B decreases. In OOB, each sample's loss depends only on its own fixed target, so the minibatch loss is an unbiased estimate of the dataset-level cached loss. The remaining error is the staleness of the targets, which we measure in Sec. 4.2 (Fig. 2C–D).
Reply to reviewer bKBA (1/3)
(1/3)
We thank Reviewer bKBA for the constructive review, and for recognizing the simplicity and motivation of the core idea, the oracle-gradient analysis as an original contribution, and the broad evaluation. We address each point below, including new experiments. In the coming days, we will upload a revised PDF that includes these changes and the new results.
EP-SIGReg loses 13 points in 5-NN from B=16 to B=2, while SIGReg-OOB stays stable. SIGReg-OOB outperforms EP-SIGReg in 31 of 32 experiments, with average gains of +9.5 (5-NN) and +7.8 (linear). Since the vulnerability to batch size comes from in-batch anti-collapse estimation, not from the quantile variant, OOB can address it for the EP variant as well. We think this is an important ablation and will include it in the revised pdf.
Contrastive-OOB on natural images. We agree that Contrastive-OOB does not uniformly outperform SimCLR on natural images. However, note that due to the lack of space, results in Table 8 show average performance across batch sizes. For CIFAR in particular, performance recovers much faster than for other datasets with the increase of the batch size. At B=2, Contrastive-OOB improves over SimCLR on CIFAR-10 by 7.16 pp in 5-NN and 4.94 pp in linear probing, and on CIFAR-100 by 4.26 pp and 2.83 pp, respectively. As batch size increases, in these two datasets, SimCLR improves substantially and overtakes Contrastive-OOB, explaining its higher batch-averaged CIFAR scores. The small average linear-probe gain on natural images (+0.14 pp in Table 2) is all due to CIFAR-10 and CIFAR-100. On the other five natural datasets (DTD, ImageNet-100-5k, Imagenette, Imagewoof, STL-10), Contrastive-OOB outperforms SimCLR on every dataset in both probes, with average gains of +7.3 pp in 5-NN and +2.6 pp in linear probing (Table 8). Beyond classification, the advantage of Contrastive-OOB on natural images is clearer in tasks that depend on the structure of the embedding space: in instance retrieval on COIL-100, UKBench and INRIA Holidays, it improves mAP over SimCLR by 13.7 points on average (Sec. 4.5), reaching 71.0 vs 57.3 in the transductive and 79.7 vs 66.7 in the held-out protocol (Table 9).
MoCoV2 configuration. We agree and now evaluate MoCoV2 with a 128-key queue, smaller than the training set, and a cosine momentum schedule from 0.99 to 0.999. We evaluated on BloodMNIST-500 and PathMNIST-500 at B ∈ {2, 16, 64}, with three seeds and examined their 5-NN performance:
These results show that the newly configured MoCov2 is not necessarily better and Contrastive-OOB outperforms both. Unfortunately, due to time limitations, we do not have the capability to run these tests in all our 19 datasets, so we can't include them in Table 2, but we'll make it part of the Supplementary.