acceptodds
Under review as a conference paper at ICLR 2027

Dataset-Level Regularization for Joint-Embedding Self-Supervised Learning

Abstract

Joint-embedding self-supervised learning aligns representations of augmented views while regularizing their distribution to prevent collapse. Most joint-embedding methods estimate latent regularization from the active minibatch, even though its purpose is to shape the broader representation distribution. Batch size therefore affects not only optimization and memory use, but also the computation of the self-supervised objective itself. We propose out-of-batch regularization, which maintains a dataset-level state in CPU memory and uses globally constructed per-instance anchors during minibatch training. An analysis on natural-image and biomedical datasets shows that these anchors approximate the full-population regularization gradient more faithfully than conventional minibatch estimation at small batch sizes. We apply this principle to contrastive and non-contrastive learning across 2D and 3D data. The resulting methods consistently improve over their corresponding in-batch objectives in image classification, minority-class performance under controlled imbalance, instance retrieval, and continued self-supervised pretraining, with the largest gains at small batch sizes. Our results show that the minibatch does not need to define the statistical scope of anti-collapse regularization.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, self-supervised learning, joint-embedding architectures

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bKBAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes out-of-batch (OOB) regularization for joint-embedding SSL, which moves the anti-collapse term from the active minibatch to a dataset-level state. Every half epoch, the current encoder embeds the full training set, and per-instance targets are built globally. SIGReg-OOB assigns Gaussian quantiles along random 1D projections; Contrastive-OOB uses spherical anchors spread by a repulsion objective. These targets are stored on CPU, and during training each sample is only attracted to its own target, with no cross-sample interaction online. An oracle-gradient analysis shows that minibatch regularization gradients can point away from the full-dataset gradient at small batch sizes, while cached targets trade this error for staleness. Experiments on natural images and 2D/3D biomedical data cover classification, controlled imbalance, instance retrieval and DINOv2 continued pretraining. Gains are largest at small batch sizes, and GPU memory per batch is unchanged.

Critical strengths
  • The core idea is simple and well motivated: decouple the statistical support of the regularizer from the optimization batch size. This is relevant for memory-limited settings such as 3D medical imaging and adapting large pretrained encoders.
  • The target construction is simple and sound. For SIGReg-OOB, at each refresh the authors draw 256 random directions and project the embeddings of the whole training set onto each of them. On every direction, they sort the N projected values and assign the k-th ranked sample to the k-th quantile of a standard Gaussian, which in 1D is the optimal matching between the data and the Gaussian grid. During training, each sample’s projections are then regressed toward these fixed per-direction targets, and the directions are redrawn at the next refresh. The regularizer is thus computed against the whole dataset rather than the batch, and the online step reduces to a cheap per-sample regression.
  • The oracle-gradient analysis is, to my knowledge, an original contribution. It shows that minibatch gradients can have negative cosine with the full-dataset gradient at small batch sizes, which is a relevant observation.
  • The evaluation is broad (more than 19 datasets, though small), and the overhead is small.
Critical weaknesses
  • The SIGReg baseline is unusually weak, it is apparently not the published Epps–Pulley objective but an in-batch sorted-quantile variant (Sec. 3.3), and it performs poorly (34.6 5-NN on natural images vs 50.2 for SimCLR). The headline SIGReg-OOB gains (+14pp) may therefore partly reflect a batch-fragile baseline rather than a general property of OOB.
  • Contrastive-OOB gains on natural images are marginal, the average is +0.14pp linear probe (Table 2), and Contrastive-OOB is below SimCLR on CIFAR-10 (74.9/79.4 vs 76.2/81.1) and CIFAR-100 (33.4/49.4 vs 39.1/53.1) in Table 8.
  • The MoCoV2 baseline, the closest analogue to OOB, is absent from table 2 and seems poorly configured for small datasets: its 4096-key queue is larger than many training sets, and its fixed 0.999 momentum barely updates the key encoder in short runs. It’s also missing from Table 2. A tuned version would be needed before concluding OOB beats queue-based methods.
  • The most direct competitor, SogCLR (Yuan et al., 2022), is missing, is cited but not compared, although it targets exactly the minibatch bias of the global contrastive objective.
  • The 2D experiments train ViT-S/16 from scratch on small datasets at 224×224 resolution. ViTs are known to underperform convolutional backbones in this regime, and the absolute numbers are low (e.g., CIFAR-10 5-NN of 49.2 for SIGReg and 76.2 for SimCLR). This makes it hard to tell whether OOB’s gains are general or compensate for a weak setup.
  • In the oracle experiment, the gradients disagree (opposite sign) only when the batch size is extremely small (B= 1 or 2), which is rarely required, even in low-budget regime.
  • (additional, not affecting the score) The mutual-information framing around Eq. 1 could cite more InfoMax-based SSL work, e.g., Hjelm et al. (2019), Poole et al. (2019), and Louiset et al. (ICLR 2024), which decompose InfoMax objectives into alignment and uniformity terms in the same spirit as the paper’s alignment/anti-collapse split.
Questions for authors

-Could you compare against the original Epps–Pulley SIGReg? Your in-batch quantile variant may make the +14pp gain look larger than it is. -Why does Contrastive-OOB barely help on natural images (+0.14pp linear) and even underperform SimCLR on CIFAR-10/100? -Could you rerun MoCoV2 with a queue smaller than N and a momentum schedule (0.99 → 0.999), and add it to Table 2? At N = 500–2k, the 4096-key queue contains stale copies of each image. -How does Contrastive-OOB compare with SogCLR, which targets the same minibatch bias at almost no cost? -Do the gains hold with a ResNet-18 at native resolution (maybe using GroupNorm instead of BatchNorm when batch sizes are particularly small), where baselines are stronger and comparable to published numbers? -Gradients only disagree in sign at B = 1–2, which is rarely needed in practice. Which realistic settings actually benefit, and do downstream gains track gradient agreement?

  • At refresh time, the gradient of Eq. 7 seems to coincide with the gradient of the sliced Wasserstein-2 distance to an isotropic Gaussian, since sorting gives the optimal 1D coupling. If there is a connection, could you make it explicit?
5 comments
AuthorsOpenReview ↗

Reply to reviewer bKBA (1/3)

(1/3)

We thank Reviewer bKBA for the constructive review, and for recognizing the simplicity and motivation of the core idea, the oracle-gradient analysis as an original contribution, and the broad evaluation. We address each point below, including new experiments. In the coming days, we will upload a revised PDF that includes these changes and the new results.

  • SIGReg baseline and the Epps–Pulley objective. Our in-batch SIGReg baseline used the same sorted-quantile target as SIGReg-OOB, so that the two methods differ only in whether the regularizer is computed on the batch or with our OOB framework. This is a valid instance of SIGReg since the original LeJEPA paper (Balestriero & LeCun, 2025) defines SIGReg for any consistent univariate test applied to random projections. It also lists sorting-based (CDF/order-statistic) tests among the valid options (Sec. 4.2.2). They recommend the Epps–Pulley test for practical reasons because sorting is hard to parallelize across GPUs and is not differentiable. In SIGReg-OOB neither issue arises, because sorting happens once per refresh on CPU and outside the gradient path, while the online loss is a simple differentiable MSE to fixed targets. In any case, we ran experiments with the Epps-Pulley implementation for in-batch SIGReg (EP-SIGReg) on BloodMNIST, PathMNIST, DTD and a 5k subset of ImageNet-100 at B ∈ {2, 8, 16, 64}, with 3 seeds. Averaged over datasets, the results show that Epps–Pulley SIGReg also suffers from small-batch training:
B5-NN: EP-SIGReg5-NN: SIGReg-OOBLinear: EP-SIGRegLinear: SIGReg-OOB
241.361.850.367.8
851.460.263.667.4
1654.558.463.867.8
6449.253.860.666.7

EP-SIGReg loses 13 points in 5-NN from B=16 to B=2, while SIGReg-OOB stays stable. SIGReg-OOB outperforms EP-SIGReg in 31 of 32 experiments, with average gains of +9.5 (5-NN) and +7.8 (linear). Since the vulnerability to batch size comes from in-batch anti-collapse estimation, not from the quantile variant, OOB can address it for the EP variant as well. We think this is an important ablation and will include it in the revised pdf.

  • Contrastive-OOB on natural images. We agree that Contrastive-OOB does not uniformly outperform SimCLR on natural images. However, note that due to the lack of space, results in Table 8 show average performance across batch sizes. For CIFAR in particular, performance recovers much faster than for other datasets with the increase of the batch size. At B=2, Contrastive-OOB improves over SimCLR on CIFAR-10 by 7.16 pp in 5-NN and 4.94 pp in linear probing, and on CIFAR-100 by 4.26 pp and 2.83 pp, respectively. As batch size increases, in these two datasets, SimCLR improves substantially and overtakes Contrastive-OOB, explaining its higher batch-averaged CIFAR scores. The small average linear-probe gain on natural images (+0.14 pp in Table 2) is all due to CIFAR-10 and CIFAR-100. On the other five natural datasets (DTD, ImageNet-100-5k, Imagenette, Imagewoof, STL-10), Contrastive-OOB outperforms SimCLR on every dataset in both probes, with average gains of +7.3 pp in 5-NN and +2.6 pp in linear probing (Table 8). Beyond classification, the advantage of Contrastive-OOB on natural images is clearer in tasks that depend on the structure of the embedding space: in instance retrieval on COIL-100, UKBench and INRIA Holidays, it improves mAP over SimCLR by 13.7 points on average (Sec. 4.5), reaching 71.0 vs 57.3 in the transductive and 79.7 vs 66.7 in the held-out protocol (Table 9).

  • MoCoV2 configuration. We agree and now evaluate MoCoV2 with a 128-key queue, smaller than the training set, and a cosine momentum schedule from 0.99 to 0.999. We evaluated on BloodMNIST-500 and PathMNIST-500 at B ∈ {2, 16, 64}, with three seeds and examined their 5-NN performance:

DatasetBMoCoV2 (original)MoCoV2 (reconfigured)Contrastive-OOB
BloodMNIST-500264.4 ± 2.860.4 ± 5.265.9 ± 2.8
1661.9 ± 3.761.5 ± 2.571.2 ± 0.9
6459.0 ± 2.857.9 ± 2.665.4 ± 4.3
PathMNIST-500257.7 ± 3.460.6 ± 4.464.5 ± 5.5
1658.7 ± 1.763.4 ± 1.970.0 ± 1.4
6450.3 ± 3.258.9 ± 1.570.1 ± 0.7

These results show that the newly configured MoCov2 is not necessarily better and Contrastive-OOB outperforms both. Unfortunately, due to time limitations, we do not have the capability to run these tests in all our 19 datasets, so we can't include them in Table 2, but we'll make it part of the Supplementary.

AuthorsOpenReview ↗

Reply to reviewer bKBA (2/3)

(2/3)

  • Comparison with SogCLR. SogCLR is indeed a method designed to remove the minibatch bias of the global contrastive objective by keeping a per-sample moving average of the contrastive denominator, but each update still uses only the negatives in the current batch. Originally, we excluded it as the smallest batch size their paper considers is 128, but now we ran the comparison experiments using a 2D and a 3D dataset in the small-batch regime:
DatasetBSogCLRSIGReg-OOBContrastive-OOB
PathMNIST-5k283.5 / 88.690.3 / 91.991.7 / 92.8
488.6 / 91.790.3 / 92.891.5 / 93.0
889.3 / 91.290.2 / 92.191.3 / 93.1
NoduleMNIST3D255.6 / 57.072.4 / 76.072.1 / 76.3
457.8 / 57.073.0 / 77.772.9 / 77.3
869.0 / 71.375.0 / 79.073.5 / 74.9

Additionally, we run (only for batch size of 2) experiments on other datasets:

DatasetSogCLRSIGReg-OOBContrastive-OOB
BloodMNIST-5k95.1 / 96.497.0 / 97.796.9 / 98.0
STL-1056.8 / 65.062.1 / 68.364.4 / 70.1
ImageNet-100-5k49.5 / 64.857.8 / 67.562.8 / 70.4
FractureMNIST3D35.6 / 39.636.1 / 42.436.2 / 39.7

To summarize, SogCLR does alleviate the small batch performance problems of SimCLR, but at the lowest batch sizes the negative effect is still pronounced, and both SIGReg-OOB and Contrastive-OOB outperform it.

  • Backbone architecture and absolute performance. Originally, we wanted to use both ViT and a CNN to make sure our results are not associated with the model architecture. Therefore, our 3D experiments already use a 3D ResNet-18, where both OOB methods improve over their in-batch counterparts (Table 2). To test the 2D case as well, we now trained a ResNet-18 from scratch on CIFAR-10 and CIFAR-100, for 100 epochs and batch sizes 2, 16 and 64. SSL uses only the official training split, and frozen evaluation uses the official test split. Here, SIGReg uses the published Epps–Pulley objective. Results are 5-NN / linear-probe accuracy (%), averaged over three seeds:
DatasetBEP-SIGRegSIGReg-OOB
CIFAR-1029.7 / 29.986.1 / 87.0
CIFAR-101678.2 / 82.380.1 / 82.7
CIFAR-106468.5 / 74.775.7 / 79.7
CIFAR-10021.0 / 9.748.5 / 52.0
CIFAR-1001633.6 / 45.248.7 / 49.6
CIFAR-1006425.9 / 38.747.7 / 49.2

With a convolutional backbone, the absolute numbers are much higher (e.g. 86.1 / 87.0 for SIGReg-OOB on CIFAR-10 at B=2), and the small-batch advantage remains. SIGReg-OOB outperforms SIGReg at every batch size. In-batch Epps–Pulley SIGReg collapses to near-chance 5-NN accuracy at B=2, which is exactly the failure mode that OOB is designed to avoid. More generally, our intention was to compare each OOB method to its in-batch counterpart under exactly the same backbone, augmentations and training recipe, which isolates the effect of where the regularizer is computed. We used a single, generic augmentation pipeline for all datasets, both natural and biomedical, rather than tuning it per dataset, so our absolute numbers are not directly comparable to published results where performance on individual datasets was the main goal.

AuthorsOpenReview ↗

Reply to reviewer bKBA (3/3)

(3/3)

  • Oracle gradient analysis. We agree that the gradient sign flips only at B=1, and we don't claim B=1 is typical. But the sign flip is just the extreme case. The cosine with the full-dataset gradient rises gradually and reaches around 0.90 only at B=64, so the anti-collapse term is already partially wrong for the batch sizes in between. Batch sizes of 2 to 8 are realistic when models or inputs are large: in Table 3 we continue pretraining ViT-G (1.1B) at B=2 and 8 on a single A100. The oracle analysis diagnoses the regularizer's estimation error, and we don't claim the oracle gradient is optimal for downstream performance.

  • InfoMax references. We thank the reviewer for providing new references for the MI framing. We will add Hjelm et al. (2019), Poole et al. (2019) and Louiset et al. (2024) to the discussion of Eq. 1 in the revised PDF.

  • Connection to sliced Wasserstein. Yes, we believe there is a connection and we are happy to put it in more explicitly. We will add the following to Sec. 3.3: "Along each direction vmv_m, sorting gives the optimal 1D coupling between the projected embeddings and the Gaussian quantiles. So at refresh, Eq. 7 equals W22W_2^2 between the projected embeddings and a quantized N(0,1)\mathcal{N}(0,1). Averaged over directions, it is equivalent to the sliced W22W_2^2 distance to an isotropic Gaussian, and its gradient coincides with the sliced-Wasserstein gradient." This framing also clarifies two properties. First, between the refreshes the coupling between anchors and encoder outputs is kept fixed, thus our fixed coupling can only cost more than the optimal one (the cached loss upper-bounds SW22SW_2^2, with equality at refresh). Staleness therefore loosens this bound but does not change the objective. Second, it explains the batch size dependency. In the in-batch variant, each sample's target depends on its rank among the other samples in the batch, so it changes with the batch. This makes it a minibatch OT loss, which is a different objective from the full-data distance. According to Fatras et al., 2020, it upper-bounds that distance, and the gap grows as B decreases. In OOB, each sample's loss depends only on its own fixed target, so the minibatch loss is an unbiased estimate of the dataset-level cached loss. The remaining error is the staleness of the targets, which we measure in Sec. 4.2 (Fig. 2C–D).

Reviewer_E6g7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work proposes a multi-view alignment approach for self-supervised learning which targets the low-memory setting. The core idea is to develop anchor representations for each sample in an off-line fashion and then use these during view alignment to mitigate the risk of representations collapsing. This is essentially trading complexity for The resulting representation is evaluated on a range of small-scale downstream tasks using kNN and linear probing.

Critical strengths
  • I appreciate the goal of making it easier to train embedding methods with reduced computational costs.
  • The presentation is fairly clear, although I did struggle a bit with S3.3 and 3.4.
  • Performing some out-of-GPU work and utilizing CPU memory is a natural strategy to reduce GPU memory.
Critical weaknesses
  • A major element that's missing for me is clarity around the time it takes to run these methods relative to the baselines. I appreciate the desire for lower batch sizes, but, in the end, there's always a tradeoff between run-time and batch sizes. While there's value in supporting running models on small hardware, it's often easy to just rent larger hardware for short training runs and so understanding the tradeoff helps practitioners translate the findings into cost tradeoffs.
  • There has been prior work that explored how hyperparameter settings have a significant impact on representation quality for SSL methods (https://arxiv.org/abs/2303.01986). These settings interact with batch size. So, a fair baseline would perform hyperparameter tuning across the baselines to ensure that they are using proper settings for different minibatch sizes.
  • An alternative strategy would be to use accumulated gradients across a collection of mini-batches to approximate a larger batch size. It's possible I missed that, but I would think that would be a very natural and strong baseline. In addition, it would be much simpler to implement with fewer hyperparameters to tune.
  • I would have liked to see a little more emphasis around issues related to staleness, which likely interacts with learning rate.
  • Often the differences between representations if the downstream task is allowed some small amount of fine-tuning. I would like to see some evaluation of that case to see if the differences we are seeing here hold up to that change.
  • I couldn't find what the batch size was for the experiments in 4.4.
  • I found the method descriptions in 3.3 and 3.4 difficult to follow. I'm sure I'll get used to it eventually, but many parts read like LLM generated text (especially L222-227), which I find difficult to follow because of sentences like "SimCLR obtains spherical spreading from minibatch negatives, while MoCoV2 extends its negative support with a momentum queue.". This is problematic because it's the key technical element in the paper.
  • The evaluation is on mostly toy datasets. I'd love to see some real examples where there are datasets that benefit from this approach and aren't outperformed by reasonable tuning of existing methods or using pre-trained network as initialization.
  • It seems a little odd to switch Dinov2 fine tuning to SIGReg/SimCLR as opposed to continuing to use the Dinov2 objectives.
Questions for authors
  • Why does SIGReg baseline use the "same sorted-quantile targets compute in the batch, rather than the Epps-Pulley statistic"? Is that fair?
  • What steps / hyper parameter tuning were performed to ensure that the baseline methods were strong. It would be unfair to use settings that were optimized for large batch sizes and then make a significant change like reducing the batch sizes significantly and expect performance to be preserved. There needs to be a clear win given the added complexity of this method.
  • The results in 4.4 seems a little strange. It seems like performance is trending down for larger batch sizes (except for SigREG). Why? Similar question for Figure 3. Is it because it's being held to a fixed number of steps and larger batch sizes get fewer steps? If that's the case, then I'd think a more fair comparison would be to use wall clock time or some other way of balancing cost as opposed to something that's arbitrary like steps. This is especially important because of the interaction between learning rate and batch sizes. Were learning rates increased for the larger batch sizes?
  • How would the resulting representations compare to just using strong pre-trained features (e.g., dino v2)? If there's no improvement, then an even simpler method than the proposed method would be to just use those features. In that scenario, it would be ideal if the paper included results on data types where there weren't existing strong features. Otherwise, it seems like there's not much practical value to the proposed work.
4 comments
AuthorsOpenReview ↗

Reply to Reviewer E6g7 (1/4)

(1/4)

We thank Reviewer E6g7 for the thoughtful review. We address each point below, including new experiments run. In the coming days, we will upload a revised PDF that includes these changes and the new results.

  • Runtime and cost trade-off. Training time is reported in Appendix G (Fig. 16), which we refer to from the main text in Sec. 4.7. All runs use 100 epochs on a single NVIDIA L40S. At the same batch size, OOB adds little training overhead. Across over 30 matched dataset–batch conditions, SIGReg-OOB takes 9.9% longer than SIGReg on average and Contrastive-OOB 2.3% longer than SimCLR. Example training times (minutes):
DatasetBSIGRegSIGReg-OOBSimCLRContrastive-OOB
BloodMNIST-5k2197.0239.8212.1214.2
BloodMNIST-5k64125.4128.1126.5127.5
STL-102537.8580.6543.9547.1
STL-1064312.3324.5314.9325.9

Small batches themselves are, of course, slower: on STL-10, B=2 takes about 1.7× longer than B=64, and we do not claim that OOB reduces runtime in general. However, renting larger hardware is not a solution in many of the settings we target. In scientific imaging, a single sample can be a multi-channel 3D volume or a 3D time series of several gigabytes, so that a batch of one or two samples already fills a large GPU. Reaching the batch sizes that in-batch regularizers need would then require tens to hundreds of GPUs with distributed training, which is prohibitively expensive for many research groups, adds engineering complexity, and is wasteful of resources. While additional computational resources can sometimes overcome batch-size limitations, we argue that making SSL effective regardless of batch size is an important goal in its own right, for both practical and methodological reasons.

  • Hyperparameter tuning of the baselines. We thank the reviewer for pointing to Bordes et al., 2023. To test if the advantages of OOB methods came from untuned baselines, we now tuned the baselines separately for each batch size. We tried 12 configurations (learning rate in {1e-4, 3e-4, 1e-3}, combined with the SimCLR temperature in {0.10, 0.15, 0.25, 0.50} or the Epps–Pulley SIGReg weight in {0.01, 0.05, 0.10, 0.20}), selected the best on a validation split, and report test accuracy over three seeds. Note that this favors the baselines because the OOB methods use a single default configuration across all datasets and batch sizes. Results on ImageNet-100-5k (5-NN / linear):
BSIGReg (paper)SIGReg (tuned, EP)SIGReg-OOBSimCLR (paper)SimCLR (tuned)Contrastive-OOB
229.6 / 40.331.4 / 41.357.8 / 67.544.8 / 60.253.8 / 65.662.8 / 70.4
427.3 / 46.040.3 / 56.958.0 / 68.351.3 / 64.457.8 / 67.663.5 / 68.4
840.5 / 58.845.6 / 61.756.4 / 67.656.3 / 66.562.7 / 68.564.1 / 69.4
1649.2 / 63.549.4 / 63.855.3 / 68.358.6 / 68.260.9 / 67.861.7 / 69.6

Tuning can substantially improve the baselines (naturally since we evaluate each configuration using downstream validation), especially SimCLR (e.g. +9.0 points in 5-NN at B=2). Even so, both OOB methods outperform their tuned counterparts at every batch size: SIGReg-OOB improves over tuned SIGReg by 5.9–26.4 points in 5-NN and 4.5–26.2 points in linear probing, and Contrastive-OOB improves over tuned SimCLR by 0.8–9.0 and 0.8–4.8 points, respectively. The gains are largest at B=2 and shrink as the batch grows, which is consistent with our claim that OOB matters most when batches are small. We will add this ablation and cite Bordes et al. in the revised PDF.

  • Gradient accumulation. For batch-dependent objectives, gradient accumulation does not reproduce a larger batch. This is because each minibatch still computes its own contrastive denominator, so negatives from different minibatches never interact, 1A∑a=1A∇θL(Ba)≠∇θL(⋃a=1ABa).\frac{1}{A}\sum_{a=1}^{A}\nabla_\theta L(\mathcal B_a) \neq \nabla_\theta L\Big(\bigcup_{a=1}^{A}\mathcal B_a\Big). Accumulation enlarges the optimizer batch, but not the population over which the regularizer is computed (which is the quantity OOB changes). We trained SimCLR on STL-10 for 100 epochs with three seeds, accumulating minibatches of 2 or 16 up to an optimizer batch of 64, with all other settings fixed:
MethodMinibatchAccum. stepsOptimizer batch5-NNLinear
SimCLR21253.3 ± 0.661.5 ± 0.8
SimCLR2326455.0 ± 0.464.5 ± 0.7
Contrastive-OOB21264.4 ± 0.770.1 ± 0.8

Accumulating from B=2 to 64 helps SimCLR only slightly (+1.7 / +3.0) and stays far below true B=64 training (−12.9 / −7.8). Contrastive-OOB at B=2, without accumulation, outperforms accumulated SimCLR by 9.4 / 5.6 points. We will add this experiment to the revised PDF.

AuthorsOpenReview ↗

Reply to Reviewer E6g7 (2/4)

(2/4)

  • Staleness and learning rate. We belive that our oracle analysis (Appendix B) measures exactly this interaction. We tracked how well a cached target agrees with the current full-dataset gradient as the encoder trains. After 128 optimizer steps, the cosine agreement at our default learning rate is 0.45, 0.72 and 0.91 at epochs 0, 50 and 100, while with a much smaller learning rate (10⁻⁶) it is 0.69, 0.91 and 0.91. Staleness is therefore driven by how fast the encoder changes, i.e. by the learning rate, rather than by the number of steps. Our cosine schedule lowers the learning rate over time so the cache stays accurate for longer later in training (Fig. 2C–D). We will move this discussion into the main text; we see the design of an adaptive policy based on measured representation drift as a natural next step for this research.

  • Fine-tuning evaluation. We ran a fine-tuning study on BloodMNIST-5k and STL-10. After 100 epochs of SSL pretraining (ViT-S/16, SSL batch sizes 2–16), we fine-tune the whole encoder for 20 epochs, using the same small labeled subsets (280 BloodMNIST and 728 STL-10 images). The table shows frozen head → full fine-tuning balanced accuracy (%), averaged over three seeds:

BloodMNIST-5k

SSL batchSIGRegSIGReg-OOBSimCLRContrastive-OOB
215.9 → 61.295.8 → 96.290.9 → 93.493.9 → 95.8
417.1 → 70.095.8 → 96.192.5 → 94.294.6 → 95.7
892.6 → 94.495.4 → 96.194.6 → 95.195.0 → 96.1
1693.7 → 94.595.3 → 95.995.3 → 95.895.1 → 95.7

STL-10

SSL batchSIGRegSIGReg-OOB
213.1 → 29.058.9 → 62.1
420.6 → 32.559.6 → 60.8
853.8 → 56.259.9 → 61.2
1658.3 → 59.060.3 → 61.0

Fine-tuning does not erase the small-batch advantage. For encoders pretrained at B=2, SIGReg-OOB exceeds SIGReg by 35.0 and 33.0 points and remains better than SIGReg in every comparison. These results cover a low-label, fine-tuning regime. We will add this study to the revised PDF.

  • Batch size in Sec. 4.4. The oracle diagnostic (Fig. 4) and the minority-recall results in Fig. 12 use B=64, as stated in Appendix D, while the aggregate gains reported in the text of Sec. 4.4 are averaged across batch sizes. We agree this is confusing, and in the revised version we will report the B=64 results in the text as well and state the batch size explicitly. We chose the specific batch size on purpose, to separate the effect of class imbalance from small-batch collapse. At B=64 the in-batch baselines train stably, so the remaining difference in minority-class performance reflects how well rare samples are represented in the regularizer, rather than the general small-batch failure studied elsewhere in the paper.

  • Real-world datasets. We would like to clarify that MedMNIST it is not a synthetic or toy benchmark. Each of its datasets is derived from an independent, published biomedical dataset (e.g. blood-cell microscopy, colorectal histology, chest CT, dermatoscopy), collected for real diagnostic or research tasks and only standardized in size and format. Our evaluation also includes BBBC010, a real high-throughput microscopy assay, and a private plankton dataset acquired with widefield microscopy.

  • Continued pretraining objective. Regarding the question about continued pretraining with SIGreg/SimCLR: our comparison in Sec. 4.6 is controlled, comparing OOB methods with their in-batch versions, so the gains reflect the performance advantage of computing the anti-collapse loss with our OOB framework, not the choice of a specific regularizer. We did not continue with the DINOv2 objective for three reasons. 1) Public checkpoints provide the backbone but not the training state of the original objective (projection heads, prototypes, teacher), so a continuation would restart those components from scratch, 2) continued pretraining does not require the original objective: DIET-CP (https://arxiv.org/abs/2509.06990) continues DINOv2/DINOv3 with a much simpler loss, without a teacher or projector, and still obtains significant gains from about 1000 images, and 3) DINOv2 also computes regularizers that need a big batch size so we expect a similar degradation in small batch sizes.

AuthorsOpenReview ↗

Reply to Reviewer E6g7 (3/4)

(3/4)

  • SIGReg baseline and the Epps–Pulley objective. Our in-batch SIGReg baseline used the same sorted-quantile target as SIGReg-OOB, so that the two methods differ only in whether the regularizer is computed on the batch or with our OOB framework. This is a valid instance of SIGReg since the original LeJEPA paper (Balestriero & LeCun, 2025) defines SIGReg for any consistent univariate test applied to random projections. It also lists sorting-based (CDF/order-statistic) tests among the valid options (Sec. 4.2.2). They recommend the Epps–Pulley test for practical reasons because sorting is hard to parallelize across GPUs and is not differentiable. In SIGReg-OOB neither issue arises, because sorting happens once per refresh on CPU and outside the gradient path, while the online loss is a simple differentiable MSE to fixed targets. In any case, we run experiments with the Epps-Pulley implementation for in-batch SIGReg (EP-SIGReg) on BloodMNIST, PathMNIST, DTD and a 5k subset of ImageNet-100 at B ∈ {2, 8, 16, 64}, with 3 seeds. Averaged over datasets, the results show that Epps–Pulley SIGReg also suffers from small-batch training:
B5-NN: EP-SIGReg5-NN: SIGReg-OOBLinear: EP-SIGRegLinear: SIGReg-OOB
241.361.850.367.8
851.460.263.667.4
1654.558.463.867.8
6449.253.860.666.7

EP-SIGReg loses 13 points in 5-NN from B=16 to B=2, while SIGReg-OOB stays stable. SIGReg-OOB outperforms EP-SIGReg in 31 of 32 experiments, with macro-average gains of +9.5 (5-NN) and +7.8 (linear). The vulnerability to batch size comes from in-batch anti-collapse estimation, not from the quantile variant, and OOB addresses it for the published objective as well. We think this is an important ablation and will be included in the revised pdf.

  • Batch size, number of steps and learning rate. All runs train for a fixed 100 epochs with a fixed learning rate of 10⁻⁴, so larger batch sizes get fewer optimizer steps. However, we believe our comparisons remain fair since each OOB method and its in-batch counterpart always use the same batch size, number of epochs and steps, learning rate and data, so they receive exactly the same number of updates. We chose a fixed number of epochs so that every method sees the same data, which is the standard SSL protocol. Matching wall-clock time instead would barely change these comparisons, since runtime is nearly constant from B≈8 to 64 (Fig. 16), and only the smallest batches take longer. Regarding the learning rate, our new tuning study (see our reply on hyperparameter tuning) selects the learning rate separately for each batch size, and the OOB advantage remains. Appendix H.1 also shows that the gains persist when training is extended to 200 epochs. Beyond the number of steps, the drop at larger batch sizes also reflects a potential issue we discuss in Sec. 1: on small datasets, a large batch covers a large fraction of the training set and leaves few parameter updates per epoch, which can hurt stochastic optimization. In-batch regularizers need large batches to estimate the dataset-level objective, while optimization on small datasets favors smaller batches. We will state the fixed-epoch protocol and its consequence for the number of steps explicitly in the revised version.
AuthorsOpenReview ↗

Reply to Reviewer E6g7 (4/4)

(4/4)

  • Comparison with pretrained features. We evaluated frozen DINOv2-base directly on our datasets, using the same 5-NN / linear-probe protocol. Even when trained from scratch for only 100 epochs at batch size 2, OOB outperforms the frozen DINOv2 features on several datasets, for example:
DatasetFrozen DINOv2-baseSIGReg-OOB (B=2)Contrastive-OOB (B=2)
BloodMNIST89.6 / 96.497.0 / 97.796.9 / 98.0
TissueMNIST36.3 / 39.937.6 / 49.639.2 / 50.6
NoduleMNIST3D64.7 / 73.072.4 / 76.072.1 / 76.3

Notably, for the 2D datasets our models use a ViT-S (22M parameters), whereas DINOv2-base has 86M parameters and was pretrained on 142M images, so a smaller model trained only on the target data outperforms a larger general-purpose one. Moreover, we do have the continued pretraining experiments that we present in subsection 4.6, that show that DINOv2 can be a good initialization but continuing self-supervised learning can improve the model representations further. Other literature has also recognized weaknesses of foundation models, e.g. many scientific modalities have many channels (multispectral data), which RGB-pretrained ViTs do not handle well (Lian et al., https://arxiv.org/pdf/2503.09826), or a native 3D/4D structure, such as volumetric time-lapse data, which 2D encoders are not designed for. Foundation models can also degrade where deep specialization is needed, for example in EM, PET and whole-slide pathology (Liu et al., https://arxiv.org/abs/2509.06467v2). Starting from a foundation model therefore does not remove the need for SSL on the target domain. Continued SSL improves pretrained models (Reed et al., 2022, https://arxiv.org/pdf/2103.12718), including DINOv3 with as few as 1000 images (Rodas et al., 2025, https://arxiv.org/pdf/2509.06990). This is consistent with our Table 3, where OOB improves over the original DINOv2 in 48/48 configurations (SIGReg-OOB) and 44/48 (Contrastive-OOB). Taken together, these arguments show that self-supervised training on the target data, either from scratch or as continued pretraining, remains a practical way to improve representations for a specific dataset, also because it lets practitioners design the augmentations (and architecture) to match their modality and downstream task, which is not possible with a fixed pretrained encoder.

Reviewer_V8rbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes out-of-batch (OOB) regularization for joint-embedding self-supervised learning (SSL), which decouples anti-collapse regularization from the active minibatch by periodically constructing per-instance, dataset-level anchors stored in CPU memory and using them as targets during online updates. First, The authors instantiate the idea for non-contrastive SSL (SIGREG-OOB via projected Gaussian quantile anchors) and contrastive SSL (CONTRASTIVE-OOB via spherical anchors optimized for uniformity with sample correspondence). Second, the authors analyze how OOB approximates a full-dataset “oracle” gradient compared to standard minibatch estimates. This paper includes comprehensive experiments across natural and biomedical 2D/3D datasets showing consistent gains over in-batch counterparts, especially at small batch sizes, along with improved minority-class recall under imbalance, better instance retrieval, and advantages for continued pretraining under tight memory budgets.

Critical strengths

1.Technical novelty and innovation

This paper introduces a clear and general framework to decouple population-level regularization from minibatch optimization via periodically refreshed, dataset-indexed, detached anchors. Besides, this paper provides two concrete, complementary instantiations (Gaussian projected anchors and spherical anchors) that map naturally onto non-contrastive and contrastive SSL families. Moreover, this paper frames collapse prevention as finite-population gradient estimation and carefully defines an oracle reference, enabling principled analysis of minibatch vs cache errors and staleness.

2.Experimental rigor and validation

This paper includes broad empirical coverage: 19+ datasets spanning natural images and 2D/3D biomedical domains, with evaluations on kNN/linear-probe classification, minority recall, instance retrieval, and continued SSL pretraining of DINOv2 variants. Second, this paper displays ablations (for SIGREG-OOB) on refresh period and projection resampling, and a well-designed oracle-gradient analysis to dissect estimation error vs population reweighting under imbalance. Third, consistent trends across batch-size sweeps shows large benefits in low-batch regimes.

3.Clarity of presentation

The method is well-motivated with the MI identity decomposition and a clean separation of alignment vs anti-collapse roles. Training interface and implementation steps are precise (refresh vs online phases) and equations are easy to follow; limitations are acknowledged.

Critical weaknesses

1.The mechanism underlying the contrastive improvements is not sufficiently isolated.

Although the paper motivates OOB primarily as a solution to minibatch estimation error, Contrastive-OOB differs from standard contrastive learning in more than the population used for regularization. Specifically, Eqs. (8)–(10) replace online negative-sample interactions with periodically optimized spherical anchors and a sample-to-anchor attraction loss. Therefore, the observed improvements over SimCLR and MoCoV2 may reflect both the expanded population context and the substantially different optimization objective.

2.The scalability and approximation quality of dataset-level anchor construction require stronger validation.

The proposed framework introduces a periodic full-dataset inference pass, persistent instance-indexed storage, and additional anchor construction. For Contrastive-OOB, Eq. (9) is optimized using only 20 stochastic steps with subsets of at most 256 anchors (Section 3.4). Thus, each refresh involves at most 5,120 anchor selections, potentially with repeated instances. It is unclear how closely this procedure approximates the intended population-level uniformity objective as the dataset grows, or how many anchors receive direct optimization updates.

Questions for authors
  1. How sensitive is CONTRASTIVE-OOB to its hyperparameters? Could you provide ablations and diagnostics (e.g., anchor pairwise similarities, effective rank) to assess whether anchors approach the intended spherical uniformity?

  2. How does OOB behave with other contrastive baselines (e.g., MoCo v3) or other non-contrastive methods (e.g., BYOL/DINO variants for ViTs) under matched training recipes? The appendix mentions additional baselines—could you highlight key aggregated numbers in the main text?

  3. For large-scale settings (e.g., ImageNet-1k), what refresh schedules and memory footprints do you anticipate for SIGREG-OOB (M, N, sorting costs) and CONTRASTIVE-OOB (anchor table size, optimization steps)? Are there streaming or sharded-refresh strategies that preserve benefits while curbing cost?

2 comments
AuthorsOpenReview ↗

Reply to reviewer V8rb (1/2)

(1/2)

We thank reviewer V8rb for their comments, careful reading and recognizing the technical novelty, the experimental breadth and the clarity of our work. We ran all the requested experiments and in the following days we will revise the pdf to include the new numbers.

  • Mechanism of Contrastive-OOB. The anti-collapse (repulsive) term of Contrastive-OOB is the standard uniformity loss of Wang & Isola (2020), which appears also in SimCLR's denominator and is also applied in MoCov2's momentum queue. On the unit sphere, Eq. 8 is a log-sum-exp over pairwise cosine similarities between different instances, and therefore, OOB does not change the regularizer. The additional online attraction term transfers this geometry to the encoder outputs, driving them toward the same target geometry. Since each anchor is obtained by a proximal step of the uniformity loss from its sample's current embedding (Eq. 9), attracting a sample to its anchor moves it, to first order, along the gradient that lowers the uniformity loss of the whole dataset, which is the same direction SimCLR obtains from in-batch negatives. The attraction is therefore a way of applying the same regularizer rather than an additional objective. We will aim to make this more explicit in the revised text.

  • Anchor coverage and number of anchors. We thank the reviewer for prompting a discussion about the training dynamics and hyperparameters of the Contrastive-OOB variant. Each refresh makes at most 20 × 256 anchor selections. With uniform sampling, the expected fraction of anchors updated in one refresh is 1−(1−256/N)201-(1-256/N)^{20}. That is about 100% for N=500, 93% for N=2k, 65% for N=5k and 33% for N=13k. Anchors are kept between refreshes, so updates accumulate: with a refresh every half epoch, every anchor is updated many times over training even at our largest N. We agree that for much larger datasets the anchor budget should grow with N. This costs O(NK) pair evaluations for subsets of size K, which is still far below the O(N²) of the exact objective and small next to the full-dataset inference pass. Regarding the number of anchors used for optimization, we varied the optimized anchor subset size K on six datasets (evaluation is 5-NN / linear, averaged over B ∈ {2, 16} and two seeds):

DatasetK = 128K = 256 (default)K = 1024
BloodMNIST96.8 / 97.497.1 / 97.792.3 / 95.0
PathMNIST90.8 / 92.791.4 / 92.989.7 / 92.3
OrganAMNIST82.5 / 88.782.4 / 88.582.5 / 89.2
DTD30.1 / 36.332.7 / 36.935.5 / 39.9
STL-1061.2 / 67.865.0 / 70.366.2 / 70.8
ImageNet-100-5k61.5 / 68.864.2 / 69.464.1 / 68.8

As this table shows, the number of anchors does not significantly affect the downstream performance, so we set K = 256 as it worked well across our dataset sizes. For substantially larger datasets, we expect K and the number of anchor steps should grow with N. We will add this experiment to the revised pdf to provide the reader with a practical guideline of how to tune this hyperparameter.

  • Approximation of the uniformity objective. To evaluate the approximation of the uniformity objective, we computed the uniformity loss of Eq. 8 for the encoder embeddings before anchor construction, for the stored anchors after construction (optimization), and for uniformly distributed random vectors of the same size, to be used as a reference. Furthermore, we report effective rank (also present in Appendix C.1 / Figure 11) and pairwise cosine similarity of the final anchors (last refresh averaged over experiments with B ∈ {2, 16}).
DatasetU beforeU afterUniform ref.Eff. rank before → afterMean cos.
BloodMNIST-5k−3.48−3.89−3.9489 → 1230.0007
DTD−3.57−3.89−3.9498 → 1220.0008
STL-10−2.89−3.51−3.9459 → 1010.044

The effective rank of the representations is increased during anchor construction and is also higher for Contrastive-OOB than for SimCLR and MoCoV2 in the small-batch regime (Figure 11). Mean cosine similarity is close to zero, showing a uniform distribution on the hypersphere.

AuthorsOpenReview ↗

Reply to reviewer V8rb (2/2)

(2/2)

  • Other baselines. MoCoV3 removes the momentum queue and takes its negatives from the current batch, so, just like SimCLR, it depends heavily on large batches. MoCoV2's queue extends the negatives beyond the batch, which makes it a stronger and more relevant baseline for the small-batch setting. We therefore compared against MoCoV2, including a version reconfigured for small datasets (we will include this in the revised manuscript based on a request from another reviewer). The other baselines (VICReg, BYOL, SwAV, DINO, IConE) are in Table 7 (Appendix C), and we will now summarize them in the main text. Averaged over the ten 2D MedMNIST datasets (linear probe), SIGReg-OOB and Contrastive-OOB are the two best methods at N=500 (66.2 / 63.1 vs at most 61.9 for the other baselines) and at N=5k (79.7 / 80.2 vs at most 72.4).

  • Large-scale settings. Even though we target small, high-dimensional datasets, the memory requirements of our approach do scale well with dataset size. For ImageNet-1k (N ≈ 1.28M), the SIGReg-OOB cache (N × M fp32, M = 256) takes about 1.3 GB and the Contrastive-OOB anchor table (N × 128 fp32) about 0.66 GB. Both are stored in CPU memory, and only the rows of the current batch are moved to the GPU, so GPU memory does not change. Sorting for SIGReg-OOB costs O(MN log N) per refresh, which is small compared to the inference pass over the dataset. For Contrastive-OOB, keeping the same per-refresh coverage as in our experiments means scaling the number of anchor steps with N/K, which costs O(NK) pair evaluations instead of the O(N²) of the exact objective. The dominant cost at this scale is the inference pass, so the refresh schedule matters most. Our oracle analysis (Sec. 4.2) shows that how long a cache stays accurate depends on how fast the encoder changes rather than on the number of steps. We agree with the reviewer that an adaptive refresh strategy is a promising direction for future work. It could be based on representation drift, follow the learning-rate schedule, take the dataset size into account, or combine these and could reduce costs further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.