acceptodds
Withdrawn from ICLR 2027

Balancing Representations by Identifying and Generating Under-represented Data

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Self-supervised learning inherits the coverage of its source data. When an unlabeled corpus is long-tailed, the representation becomes dense around frequent modes and thin where coverage is weak. Objectives for long-tailed SSL reweight the images already in the corpus and add no new ones. We introduce BRIDGE (Balancing Representations by Identifying and Generating Under-represented Data), a label-free feedback loop around the source set. The current encoder scores every source image by its mean kNN distance, and a budgeted acquisition rule selects images from a band of that score distribution where support is thin and the generator still produces faithful variants. A frozen image-to-image diffusion model then adds local variants of the selected anchors, and pretraining continues on the enlarged source set. We formulate acquisition as recoverability-weighted allocation of local support and derive when a finite budget should favor a sparse band that excludes the extreme tail and when reallocation after each repair is optimal under a fixed objective. Across five long-tailed, web-scraped and synthetic source regimes, BRIDGE has the highest seven-task linear-probe and kNN average of the compared methods, and paired experiments extend the gain to ViT-S under contrastive and self-distillation objectives, to low-label fine-tuning and to semantic segmentation. The linear-probe gain is smaller under masked reconstruction. Generated repair matches an oracle that restores the withheld real images, and under a frozen encoder the inserted images shrink the neighborhoods of the selected anchors about twice as much as those of matched unselected images. BRIDGE needs no labels and leaves the pretraining objective unchanged, so it can be added to pretraining on uncurated web data, where the encoder decides which images to generate.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-supervised Learning, Representation Learning, Computer Vision, Imbalanced Data

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xkyZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the authors propose BRIDGE, a label-free iterative framework that augments self-supervised learning (SSL) with synthetic data. In each cycle, mean squared kk-NN distances are computed using normalized embeddings from the current encoder. These distances are used to select anchors from relatively sparse regions of the embedding space. Then, a frozen Stable Diffusion (SD) 3 model generates variants of these anchors, which are added to the training set for continued SSL pre-training.

Critical strengths

[S1] Compatibility with Different SSL Objectives

BRIDGE augments the training data while leaving the underlying SSL loss unchanged. The paper applies the same framework across different SSL objectives and reports improvements in transfer performance for several of them.

Critical weaknesses

[W1] Limited Theoretical Justification and Unclear Novelty

As I understand it, the propositions in Section 3 are largely disconnected from the proposed method. Their assumptions are not established for BRIDGE, key quantities in the theoretical model are neither instantiated nor estimated, and the implemented selection rule is not derived from the analysis.

Proposition 1: It holds only in a fixed embedding space. Furthermore, the proposition uses rkr_k, while BRIDGE uses the mean squared kk-NN distance. Although fixed-space monotonicity also holds for this score, this property provides no guarantee after encoder updates.

Proposition 2: As stated in the paper (lines 161–163), the regions and parameters in Eq. (1) are neither instantiated nor estimated when using BRIDGE. Moreover, the paper does not establish that the image-level kk-NN score reflects the region-level effective support njn_j defined in Eq. (1).

Proposition 3: The conclusion follows directly from the assumptions. Since u′(d)u'(d) has the sign of g(d)g(d), the conditions "gg strictly decreasing" and "g(dmin⁡)>0>g(dmax⁡)g(d_{\min})>0>g(d_{\max})" already impose unimodality on uu with an interior maximum. Neither bb nor ρ\rho is measured, and the lack of fidelity measurements is explicitly acknowledged in line 472. The optimum d⋆=β/μd^\star=\beta/\mu, obtained for λ=0\lambda=0 in Proposition 6, is never estimated. Moreover, the proposition yields a single point d⋆d^\star, and it gives no reason why the fixed q75–q99 band, or a candidate pool centered at the histogram mode of the score distribution, should identify it. The upper cutoff q99 is never ablated and appears non-binding for the reported candidate pools, since the mode always lies at the lower edge of the band and the candidates are effectively the 4$B$ samples just above q75.

Proposition 4: As stated in the paper (lines 192–193), the guarantee does not apply to the changing representation in BRIDGE, since the image encoder is updated in each cycle. Even within its fixed-objective setting, the proposition does not establish an advantage over an optimal allocation computed in advance. If UU is fixed and known, the optimal fixed allocation can be computed once and attains the same objective value as the greedy allocation. The counterexample in Appendix C compares greedy against a naive allocation, (3,0)(3,0) yielding 5, rather than against the optimal fixed allocation, (1,2)(1,2), which matches the greedy gain of 7.

Unlike the greedy marginal-gain allocation analyzed in Proposition 4, BRIDGE selects anchors using kk-NN scoring, a percentile band, a mode-based candidate pool, and FPS. The combined selection rule is not shown to select the region with the largest marginal gain, and 2,500 images are added per cycle without re-estimation between individual repairs. kk-NN neighborhoods also overlap, so repairing one location can change the support of nearby points. Therefore, the assumption that UU is separable across regions is not established for this process.

Proposition 5: This proposition bounds the difference in BB between any two equal-budget allocations by Rγmax⁡jpjaj(nj+κ)−γ−1R\gamma\max_j p_j a_j(n_j+\kappa)^{-\gamma-1}. In particular, within the same fixed-representation setting, any absolute advantage of greedy over another equal-budget allocation is bounded by the same quantity. Therefore, this advantage vanishes in the stated R/nmin⁡→0R/n_{\min} \to 0 regime. The subsequent claim that the effect accumulates over multiple cycles is not established by this bound (lines 203–205). Proposition 4 does not establish this accumulation either.

The related-work discussion also leaves the novelty insufficiently established. MAK [1] already combines tailness, outlier avoidance and diversity (KK-center) for label-free acquisition in imbalanced SSL. These acquisition principles overlap substantially with the sparsity, outlier avoidance, and diversity criteria used in BRIDGE. D$^2$ Pruning [2] jointly considers difficulty and diversity, while Vo et al. [3] address balanced SSL data curation through hierarchical clustering and sampling. LoRA-based synthesis methods, such as DataDream [4], LoFT [5] and ChimeraLoRA [6], address faithful generation from scarce real examples.

These methods need not be directly equivalent to BRIDGE, but the paper should clarify what BRIDGE contributes beyond existing acquisition and synthesis techniques. Tables 5 and 6 include some alternative selection rules (e.g., TADA with the SSL loss, AIDE inverse occupancy, uniform, top-tail) and alternative generators (FLUX, caption conditioning), but closely related prior methods are missing. The paper should compare alternative anchor-selection rules, such as MAK- or D$^2$ Pruning-style selection, under the same frozen generator and image budget. For the synthesis component, BRIDGE keeps the generator frozen, whereas the LoRA-based methods discussed above adapt it to the real examples. Given that repair fidelity ρ\rho is not measured (line 472), the paper should provide evidence that the frozen generator produces faithful variants of the selected anchors.

[W2] Limited Scope and Inconsistent Reporting

The practical relevance of the experimental setting is insufficiently justified. (i) Substantial external knowledge is already available through the pre-trained SD 3 generator, and a pre-trained CLIP image encoder is also used for the AIDE baseline (line 374). However, the paper does not evaluate the more direct alternative of adapting a pre-trained image encoder, either with or without BRIDGE. (ii) The experiments are restricted to small source datasets, including standard long-tailed CIFAR configurations, a constructed ImageNet-100-LT source, and 10,000-image subsets of PASS and DiffusionDB. These experiments provide limited evidence for applicability to web-scale SSL pre-training. Larger, uncurated image collections, such as DataComp [7], would provide a more convincing setting.

Moreover, there are unexplained discrepancies across tables, some of which affect the highlighted result. (i) The imbalanced source in Table 1 is built with “the power-law profile of Section 5” (lines 313–315), which also defines ImageNet-100-LT (lines 268–269 and 298–299 in Section 5; Source Data paragraph in Appendix D). The paper does not describe a difference in source construction or training configuration that explains why the SimCLR linear-probe average is 31.2 in Table 1 but 39.8 in Table 2. This baseline discrepancy is consequential because the paper highlights a gain of 11.1 points over SimCLR in Section 1 (lines 078–079) and the Table 1 caption, whereas Table 2 reports a gain of only 2.7 points. (ii) The BRIDGE row of Table 1 (42.3) is identical to the default rows in the Selection, Cycle and Architecture groups of Table 6, whereas Table 2 reports 42.5 with different per-task values (e.g., Cars 15.2 vs. 16.4). (iii) In Table 6, the default BRIDGE result is 41.5 in the Generator group but 42.3 in the Selection, Cycle and Architecture groups.

The paper does not explain the discrepancies listed above or clearly identify which results should be directly comparable. The authors should clarify the configurations behind each baseline and BRIDGE row, and report the main gain against a consistent baseline. These reporting ambiguities reduce confidence in the reported gains.

References:

[1] Improving Contrastive Learning on Imbalanced Seed Data via Open-World Sampling (NeurIPS 2021; Jiang et al.)

[2] D$^2$ Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruning (ICLR 2024; Maharana et al.)

[3] Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approach (TMLR 2024; Vo et al.)

[4] DataDream: Few-shot Guided Dataset Generation (ECCV 2024; Kim et al.)

[5] LoFT: LoRA-fused Training Dataset Generation with Few-shot Guidance (BMVC 2025; Kim et al.)

[6] ChimeraLoRA: Multi-Head LoRA-Guided Synthetic Datasets (CVPR 2026; Kim et al.)

[7] DataComp: In search of the next generation of multimodal datasets (NeurIPS 2023; Gadre et al.)

Questions for authors

I have several questions, and I list them below in descending order of importance to my assessment.

[Q1] Consistency Across Tables

As mentioned in [W2], what differences in source data, initialization, training budget, or evaluation explain the SimCLR averages of 31.2 and 39.8 in Tables 1 and 2, the BRIDGE row in Table 1 that matches the default Selection, Cycle and Architecture rows in Table 6 but only partially matches Table 2, and the default BRIDGE averages of 41.5 and 42.3 across the ablation groups in Table 6? Please identify which rows should be directly comparable and correct any reporting errors, since these discrepancies materially affect the reported gain.

[Q2] Connection Between Theory and Method

As mentioned in [W1], which conclusions in Section 3 justify the implemented selection rule? In particular, what establishes that the kk-NN score reflects effective support or marginal repair utility, and that mode-window selection with FPS approximates the allocation studied in Proposition 4?

[Q3] Evaluation on Pre-trained Encoders

As mentioned in [W2], could the authors evaluate BRIDGE starting from a pre-trained image encoder and compare it with adaptation of the same encoder without repair?

[Q4] Anchor Selection and Generation Fidelity

As mentioned in [W1], could the authors compare alternative anchor-selection rules, such as MAK- or D$^2$ Pruning-style selection, under the same frozen generator and image budget? Could the authors also provide evidence that the frozen generator produces faithful variants of the selected anchors?

[Q5] Alternative Explanation via Conventional Augmentation

Table 6 reports similar performance for generated variants and the real-image oracle (41.5 vs. 40.7), while caption conditioning does not improve performance. These comparisons do not establish whether the benefit requires a pre-trained diffusion generator or could also be obtained through conventional augmentation of the selected anchors. Could the authors compare BRIDGE with a baseline that adds the same number of strongly augmented copies of the same anchors, for example using RandAugment [8], while matching sampling exposure and optimizer updates?

[Q6] Generated Images Selected as Anchors

Generated images account for 34–45% of anchors at cycle 4 (line 474). Could the authors provide qualitative examples that trace anchors and their generated descendants across cycles? Please explain whether later selections revisit already repaired regions, cover newly represented regions, or follow generator artifacts, and how the selection mechanism responds as a region receives additional images.

[Q7] Training Budget Control

BRIDGE uses approximately 1.5 times as many optimizer updates as the corresponding baselines in Tables 2–3 (lines 261–263) because the training set grows while the epoch budget remains fixed. Although the paired ViT-S experiments match optimizer updates, could the authors provide update-matched results for the main comparisons in Tables 2–3?

[Q8] Evaluation on DataComp

As mentioned in [W2], could the authors evaluate BRIDGE on a larger subset of DataComp images, using an update-matched SSL baseline, to assess whether the gains extend beyond the small source datasets used in the paper?

References:

[8] RandAugment: Practical automated data augmentation with a reduced search space (NeurIPS 2020; Cubuk et al.)

Reviewer_tdyvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BRIDGE (Balancing Representations by Identifying and Generaing Under-represented Data), which is a framework for augmenting self-supervised learning training data with synthetic data, in order to cover where there are gaps in the data distribution. Leveraging average kNN distance among the existing training set images, the pipeline selects images where its neighborhood is thin yet the generator is still capable of producing sufficient quality data, then creates those samples to augment the dataset. The paper also presents a theoretical formulation, motivating this. The paper demonstrates that across five long-tailed dataset regimes, BRIDGE results in SSL representations with highest downstream task performance average compared to other methods.

Critical strengths
  • The paper is clearly presented and the experimental protocol is extensive, covering multiple source datasets, SSL objectives, and ablations. The gains hold across SimCLR, MoCo v3, and DINO, which supports usefulness of the method beyond one training objective.
  • The paper shows gains over similar generative augmentation methods. Also, the comparisons against uniform and top-tail selection use the same generated-image budget and show substantial improvements, providing evidence that the proposed anchor selection is useful for deciding where to spend a limited generation budget.
Critical weaknesses
  • The results do not sufficiently support that long-tail recovery is what causes the gains. In Table 1, class imbalance barely affects performance: balanced and purposefully imbalanced SimCLR achieve 31.1 and 31.2, respectively. BRIDGE then adds 10,000 images, so its improvement could reflect additional data and training rather than correction of imbalance. Similarly, lowering the percentile cutoff for data selection from 75% to 1% gives essentially the same performance. These results significantly weaken the central explanation of why the method works (including the theoretical framework), even though the augmentation itself appears useful.

  • The diffusion generator brings substantially more external pretrained knowledge than the SSL baselines. Stable Diffusion 3 Medium was pretrained on approximately 1 billion images, compared with 142 million for DINOv2. This makes me question whether the gains primarily reflect transfer from a much stronger pretrained model. Comparisons with pretrained DINOv2/CLIP representations would help establish the value of accessing this knowledge through image generation, as with access to that level of foundation model would imply that you would also have access to representations at DINOv2/CLIP level, which you can finetune instead of training SSL models from scratch.

Questions for authors

Clarifications:

  • What are the exact image counts and optimizer update counts for the balanced, imbalanced, and BRIDGE settings in Table 1?
  • How were the 75th and 99th percentile thresholds selected, when the ablation shows no statistically significant difference? Was the upper cutoff varied, and how should the similar results with a 1st-percentile lower cutoff be interpreted?
  • What exactly is DiffAug + BRIDGE? Does it retain DiffAug’s learned generator while adding images from frozen SD3, and how are the two generation processes scheduled and used during training?

Additional experiments:

  • Can you compare BRIDGE against a balanced real-image baseline with the same final image count and matched optimizer updates? This would help separate the effect of imbalance repair from simply increasing the amount of training data.
  • Does the advantage over TADA/AIDE/DiffAug hold on pretraining sources other than ImageNet-100-LT, such as PASS or CIFAR-100-LT?
  • How does BRIDGE compare with cheaper augmentation or oversampling strategies applied to the same selected anchors, such as MixUp, CutMix, or copy-paste, with matched training budgets?
Reviewer_Q4m6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BRIDGE addresses imbalanced SSL through an iterative data-expansion/generation loop. It identifies sparse regions in the current representation, selects diverse anchors from a trimmed kNN-distance band, generates variants using a frozen image-to-image diffusion model, and continues SSL training on the expanded dataset.

Critical strengths
  1. The paper addresses a relevant problem with a straightforward, label-free mechanism that can be combined with different SSL objectives without changing their losses.

  2. Three-seed results, matched-update ViT-S comparisons across 4 SSL objectives, and selection, generator, oracle experiment, and cycle-count ablations provide useful evidence that the approach can improve representations. The fixed defaults showing improvment across several sources is also encouraging.

Critical weaknesses
  1. The incremental benefit and baseline adequacy are insufficiently established.
  • The contribution needs clearer positioning against tail-aware SSL acquisition methods such as MAK[1] and COLT[2], and pretrained diffusion augmentation methods such as GenView[3] and MixDiff[4].
  • The experiments also lack cheaper controls applied to the same selected anchors, such as targeted oversampling with standard augmentations (e.g., cropping and color jitter) or SSL-compatible Mixup/CutMix (e.g., i-Mix[5] or Un-Mix[6]).
  • Tables 2–3 give BRIDGE approximately 1.5× more optimizer updates.
  • Missing numerical SSL batch sizes and incomplete method-specific recipe adaptions on the the specified hardware used (2 A100 80G) further limit assessment of baseline adequacy.
  1. The framework shows substantial configuration sensitivity, without sufficiently explaining its robustness In the matched-budget cycle ablation (Tab 6.), the avg. falls from 42.3 with 5 cycles to 35.0 and 34.1 with 2 and 10 cycles. Sensitivity to initial training duration and generation strength remains unresolved. Moreover, repair fidelity is explicitly unmeasured, and no representative anchor–variant gallery is provided. Neighborhood-radius reduction alone cannot establish useful, semantically faithful diversity.

  2. Scalability and broader generalization remains unclear. Pretraining uses CIFAR, ImageNet-100-LT, and 10k-image PASS/DiffusionDB subsets. The multiple-objective evaluation and VOC segmentation experiment are useful, but larger-scale pretraining and transfer to naturally long-tailed tasks remain untested. It is unclear whether the effective repair configuration transfers reliably beyond these settings without substantial retuning.

[1] Jiang, Z., Chen, T., Chen, T. and Wang, Z., 2021. Improving contrastive learning on imbalanced data via open-world sampling. Advances in Neural Information Processing Systems, 34, pp.5997–6009. [2] Bai, J., Liu, Z., Wang, H., Hao, J., Feng, Y., Chu, H. and Hu, H., 2023. On the effectiveness of out-of-distribution data in self-supervised long-tail learning. International Conference on Learning Representations (ICLR). [3] Li, X., Yang, Y., Li, X., Wu, J., Yu, Y., Ghanem, B. and Zhang, M., 2025. GenView: Enhancing view quality with pretrained generative model for self-supervised learning. Computer Vision – ECCV 2024, Lecture Notes in Computer Science, 15126, pp.306–325. [4] Bafghi, R.A., Harilal, N., Monteleoni, C. and Raissi, M., 2025. MixDiff: Mixing natural and synthetic images for robust self-supervised representations. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp.7500–7511. [5] Lee, K., Zhu, Y., Sohn, K., Li, C.-L., Shin, J. and Lee, H., 2021. i-Mix: A domain-agnostic strategy for contrastive representation learning. International Conference on Learning Representations (ICLR). [6] Shen, Z., Liu, Z., Liu, Z., Savvides, M., Darrell, T. and Xing, E., 2022. Un-mix: Rethinking image mixtures for unsupervised visual representation learning. Proceedings of the AAAI Conference on Artificial Intelligence, 36(2), pp.2216–2224.

Questions for authors
  1. Were the SSL baselines adequately adapted to the experimental setting? Please provide the following info for transparency and reproducibility: Per-device/global batch sizes, contrastive negative-set sizes, accumulation, crop resolutions, initialization, and optimization schedules. Clarify gap from reference recipes, particularly DINO’s batch size, heads dimension, num of crops, global-crop resolution and SDCLR’s pruning schedule, and provide evidence of adequate baseline convergence.

  2. Which choices make repair reliably beneficial? Please disentangle initial training duration from subsequent repair frequency, and assess sensitivity to generation settings such as noise strength and variants per anchor. Do effective configurations transfer across sources and SSL objectives without substantial retuning?

  3. What evidence supports faithful, nonredundant generation? Please provide representative anchor–variant grids across rarity bands and cycles, including failures, and quantify fidelity and diversity. It would be meaningful to see purity diagnostic of generated-sample labels. Progressive embedding visualizations could complement quantitative neighborhood analyses of a fixed evaluation set.

  4. What does the diffusion generator used in this paper add beyond simpler interventions on the selected anchors? A meaningful comparison would be comparing against targeted oversampling with ordinary SSL augmentations and an SSL-compatible mixing baseline (e.g. Mixup/CutMix through i-mix or un-mix)—under comparable selection policies and encoder-update budgets.

  5. Do the gains persist at larger scale or on naturally long-tailed tasks? A representative experiment—for example, ImageNet-1K-scale pretraining, iNaturalist, or long-tailed detection/segmentation (LVIS, ADE20K)—would strengthen the generalization claim.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.