acceptodds
Under review as a conference paper at ICLR 2027

Learning a Flow to Self-Supervised Representations

Abstract

Explicit geometric references offer a direct way to structure self-supervised representations. Existing adversarial distribution-matching formulations, however, require costly encoder–critic optimization. We introduce Flow-Based Distribution Matching (FBDM), a non-adversarial framework that shapes the encoder toward an explicit geometric reference via spherical conditional velocity regression during pretraining, then discards the learned velocity field for downstream evaluation. An ETF-inspired reference allows its number of components K′ to exceed the auxiliary flow dimension \(d^\star\) while retaining structured geometric separation. We assign both augmented views of each image to the same target, while limiting how many images each reference center can receive. An explicit alignment loss further pulls the two views’ representations closer together. Experiments across benchmarks ranging from CIFAR to ImageNet show that FBDM achieves performance nearly on par with DM and remains competitive with existing SSL methods. Matched training-cost comparisons show a 1.48–1.83× speedup over DM with a negligible increase in GPU memory usage. We also provide a theoretical explanation for the usefulness of the learned representations: under stated conditions, we bound the downstream misclassification rate in terms of the FBDM pretraining loss.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-Supervised Learning, Representation Learning, FLow Matching

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZSnaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces FBDM, a non-adversarial self-supervised learning framework that aligns learned representations with a geometric reference through spherical conditional velocity regression. A structured reference design and an alignment loss are further proposed to enhance representation learning. Experimental results demonstrate that FBDM attains performance comparable to existing methods while substantially reducing training time, and offers theoretical guarantees for downstream classification.

Critical strengths

(1) This work presents a non-adversarial, flow-based framework for distribution matching that circumvents the expensive optimization associated with adversarial training.

(2) FBDM attains performance on par with competitors while delivering a 1.48–1.83× acceleration in training relative to DM.

(3) Theoretical assurances are established in the paper, linking the pretraining objective with performance on downstream classification tasks.

Critical weaknesses

(1) The ImageNet evaluation is confined to 100 epochs, and only a modest set of competing methods is included for comparison.

(2) The theoretical assurances rest on rather stringent assumptions, notably that of transport identifiability.

(3) The empirical study centers primarily on image classification, so how well the approach transfers to other downstream tasks remains uncertain.

Questions for authors
  1. Would it be possible for the authors to include broader ImageNet comparisons against stronger SSL baselines?

  2. To what extent does the transport identifiability assumption hold under realistic conditions?

  3. How effectively does FBDM transfer to downstream tasks beyond classification, e.g., object detection or segmentation?

4 comments
AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank you for the time and technical care you brought to this review and greatly appreciate your recognition of the framework, its efficiency, and its connection to downstream classification. Your suggestions to broaden the ImageNet comparison and test transfer beyond classification, and your question about when transport identifiability holds, helped us strengthen the evaluation and clarify the scope of the theory.

Summary of our response.

  • ImageNet (Table 2): FBDM reaches 61.55% linear top-1, versus 61.29% for VICReg and 59.94% for BYOL under the same 100-epoch budget and frozen-backbone evaluation.
  • Transport identifiability (Appendix C.4): Terminal Wasserstein error controls class–component leakage if a positive fraction of mismatched mass must travel a fixed minimum distance.
  • Transfer (Table 16): FBDM’s frozen features achieve 63.37% [email protected] on VOC 2007 detection and 95.54% mIoU on Oxford-IIIT Pet segmentation.

Weakness 1: The ImageNet evaluation is confined to 100 epochs, and only a modest set of competing methods is included for comparison. Question 1: Would it be possible for the authors to include broader ImageNet comparisons against stronger SSL baselines?

Response. Thank you for suggesting stronger ImageNet baselines. At 100 pretraining epochs and global batch size 512, FBDM reaches 61.55% linear top-1, compared with 61.29% for VICReg and 59.94% for BYOL, which we reproduced using the same frozen-backbone evaluation (Table 2). The SimCLR entry uses the same epoch budget and batch size and remains an external reference.

MethodLinear evaluation accuracy (%)
SimCLR (external reference)60.14
VICReg (reproduced, 100 epochs)61.29
BYOL (reproduced, 100 epochs)59.94
FBDM61.55

The revised manuscript reports these results in Section 4.1 (Table 2), with the evaluation settings and baseline protocol differences in Appendices D.2 and E.7.

Weakness 2: The theoretical assurances rest on rather stringent assumptions, notably that of transport identifiability. Question 2: To what extent does the transport identifiability assumption hold under realistic conditions?

Response. Thank you for asking when this assumption is plausible in practice. One situation in which the condition can hold is when a positive fraction of the mismatched mass still lies near the center matched to its own class. If the reference components are well separated, pairing that mass with a wrong component incurs a nontrivial distance cost, even if some other mismatches are cheap.

For one component per class, let Γ⋆\Gamma^\star be a Wasserstein-1 optimal coupling between the learned terminal distribution p^1\widehat p_1 and the reference distribution νε\nu_\varepsilon. Write qkj⋆q^\star_{kj} for the mass of class kk paired with component jj, and qkj⋆,<Δsepq_{kj}^{\star,<\Delta_{\mathrm{sep}}} for the part paired at a distance below Δsep\Delta_{\mathrm{sep}}. Let τ⋆\tau^\star be the one-to-one class–component matching that retains the most mass. The sufficient condition in Proposition 9 is

\le \alpha_{\mathrm{sep}} \sum_k\sum_{j\ne\tau^\star(k)}q^\star_{kj}, \qquad 0\le\alpha_{\mathrm{sep}}<1,\quad \Delta_{\mathrm{sep}}>0.$$ The right-hand sum is the total mismatched mass; the left counts only the low-cost part. Thus at least a fraction $1-\alpha_{\mathrm{sep}}$ of the mismatched mass incurs a distance of at least $\Delta_{\mathrm{sep}}$. This cost is included in $W_1$, giving $$\operatorname{Leak}(\Gamma^\star) \le \frac{2W_1(\widehat p_1,\nu_\varepsilon)} {(1-\alpha_{\mathrm{sep}})\Delta_{\mathrm{sep}}},$$ where leakage is the largest sum of outgoing and incoming mismatched mass for any matched class–component pair (Appendix C.4). The class labels are used only in the analysis. This is a data-dependent condition on the class-wise distribution of the learned terminal representations. For example, if two equally weighted classes are mixed equally within each of two reference components, the terminal distribution can match the reference exactly even though half of the mass is mismatched.
AuthorsOpenReview ↗

Official Comment by Authors 2

Weakness 3: The empirical study centers primarily on image classification, so how well the approach transfers to other downstream tasks remains uncertain. Question 3: How effectively does FBDM transfer to downstream tasks beyond classification, e.g., object detection or segmentation?

Response. Thank you for suggesting detection and segmentation as transfer tests. With a frozen ResNet-50 pretrained on ImageNet for 100 epochs with β=0\beta=0, FBDM reaches 63.37% VOC 2007 [email protected] and 95.54% Oxford-IIIT Pet mIoU (Table 16).

Object detection. We use Faster R-CNN with FPN on the complete VOC 2007 trainval/test splits. The backbone is frozen, and the detection components are trained for 15 epochs. We report VOC 2007 11-point mAP at IoU 0.5 from one run per configuration.

Backbone initializationDetection [email protected] (%)
Random initialization7.23
FBDM63.37

Foreground segmentation. We freeze the backbone and train a linear segmentation head on 224×224224\times224 inputs. We report binary mIoU and foreground IoU, excluding the unclassified trimap region, from one run per configuration.

InitializationmIoU (%)Foreground IoU (%)
Random initialization60.2244.81
FBDM95.5494.20

The revised manuscript reports the transfer protocols and results in Appendix E.6 (Table 16).

If you have any further questions, please feel free to let us know. We appreciate your recognition of our work and value your constructive advice.

Thank you for your detailed response and the considerable effort you have put into providing additional clarifications and addressing my some concerns. I sincerely appreciate the authors' dedication throughout the rebuttal process.

However, I still have substantial concerns regarding the paper's contributions, presentation quality, and experimental evidence. In my view, the current manuscript does not yet meet the acceptance standards of ICLR. Therefore, I have decided to maintain my original score.

I have also carefully reviewed the updated version of the manuscript and would like to offer a few constructive suggestions:

  1. Clarity and Presentation: I strongly encourage the authors to improve the clarity and organization of the manuscript. More informative tables and figures would help readers understand the key ideas and findings more effectively, rather than relying heavily on lengthy mathematical formulations.

  2. Contributions and Experimental Evidence: The authors should more clearly articulate and highlight the main contributions and advantages of the proposed method. For instance, based on the performance comparisons in Table 1, I still find it difficult to identify the clear advantages of FBDM over existing approaches.

  3. Computational Efficiency: The reported speedup appears relatively modest, and the efficiency results also seem inconsistent with those presented in the original submission. I encourage the authors to clarify these discrepancies and provide more convincing evidence of the claimed efficiency improvements.

  4. Overall Research Quality: A strong research paper requires a combination of clear writing, rigorous experiments, meaningful methodological contributions, and careful presentation. These aspects are equally important and should be developed together. I sincerely encourage the authors to study strong papers from previous ICLR conferences and continue refining their research, step by step, across these dimensions.

I hope these comments will be helpful in further improving the work. I appreciate the authors' efforts and wish them all the best in their future research.

AuthorsOpenReview ↗

Responds for Reviewer ZSna

Thank you for reading the revision. In response to your original questions, we added 100-epoch ImageNet comparisons with VICReg and BYOL (Table 2), evaluated transfer to detection and segmentation (Table 16), and clarified the geometric condition for transport identifiability (Appendix C.4).

Table 1 shows that FBDM achieves competitive linear-evaluation accuracy. Its practical advantage is that it lets us train representations against a chosen reference distribution without an adversarial critic, at a lower per-epoch cost than critic-based DM.

The original implementation repeatedly computed the same fixed reference centers across training iterations. We now compute them once and reuse them. We also matched the parameter counts of DM’s critic and FBDM’s velocity network and remeasured both methods. The measured per-epoch speedup over DM increased from 1.48–1.83× to 2.17–2.36×. The full timings are in revised Table 3 and in our response to Reviewer rZf5, Question 2.

And thank you for your suggestions. We would appreciate it if you could keep your feedback focused on the specific claims, experiments, and presentation of this manuscript. We also encourage reviewers to respond promptly to rebuttals and explain which concerns still weigh against acceptance in light of the new evidence. This gives authors time to address those concerns before the discussion closes.

Reviewer_NCUUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper introduces a self-supervised learning method in the “distribution matching” family. It builds on recent previous work on Distribution Matching, which pushes the encoded data distribution toward a fixed reference distribution using a Wasserstein critic. Paper goes on to proof the efficacy of the proposed method through on CIFAR and IMageNet dataset and also shows improvement on faster training iterations.

Critical strengths

Strength:

  1. Quality. The writing is careful, the appendices are detailed, and the main proofs appear correct.
  2. A simple, non-adversarial replacement for a critic-based method, with a measured speedup.
  3. Provides related experimental results: a. ResNet-18 on CIFAR-10/100, STL-10 and Tiny ImageNet: accuracy close to DM. b. ResNet-50 on ImageNet (100 epochs): 61.41% versus a SimCLR reproduction at 60.14%. c. Training is 1.48–1.83× faster per epoch than DM on one V100.
Critical weaknesses

WEAKNESS:

  1. The experiments do not support the central claim that the flow field shapes the representation: A. the ImageNet objective with β = 20 is very close to direct regression; B. there are no seeds, baselines are copied with unclear protocol matching, and the ImageNet comparison has a single baseline.
  2. Limited Novelty A. a simplex-spectral reference that allows K′ > d*. Any set of K′ points on the sphere already allows this, and the paper’s own ablation finds no stable gain from the choice of centers.

3.The method seems very close to NAT, which is not cited. NAT (Bojanowski & Joulin, 2017) assigns features to fixed points on the sphere and regresses onto them; FBDM without the flow is essentially this.

Typos: 1.Notation clash. z₁ and z₂ mean the two views (Fig. 1, Eq. 17), but Z₁ means the path endpoint (= R) in Eq. 10 and Table 4. Appendix E.3’s “the transported z₁” is ambiguous because of this. 2.Wrong method name. Table 1 lists “Zero-ICL”; the cited method is Zero-CL (Zhang et al., 2022). 3. Symbol rendering. The superscript renders as “d?” in many places and “d⋆” in others. 4. Spelling. “FlowBased” is missing a hyphen. “Centre” (C.3, Lemma 8) and “center” (elsewhere) are mixed. 5. Notation drift: R_CFM(φ; q_T) in Eq. 13 versus R_FM in Eq. 20.

Questions for authors

Clarifying Questions:

  1. Is the flow needed? On CIFAR-10 (or STL-10), how do the following compare under matched tuning? (a) alignment loss only; (b) v ≡ 0, i.e. loss E[ṡ(T)²]ω² + λR_align; (c) direct geodesic regression (Eq. 117) plus alignment. If (b) or (c) is close to FBDM, the flow is not what drives the results.

  2. ImageNet controls. With β = 20, what is the ratio E‖v‖² / E‖U‖² during training? Were the direct-regression controls in Table 8 tuned at all, for example with the same 1.3× weights and time weighting?

3.Seeds and model selection. Can you report mean ± std over at least 3 pretraining seeds? How were the per-dataset settings (clock family and a, capacity schedule, blur, warmup) chosen? Was there a validation split, or were choices made on test accuracy?

4 comments
AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank you for the careful assessment and constructive feedback. We appreciate the opportunity to clarify the role of the velocity field, the choice of reference centers, and our experimental protocols. Your questions have helped us strengthen the experimental evaluation and improve the clarity of our presentation.

Summary of our response. FBDM exceeds the better of the zero-field and direct geodesic regression controls by 2.06 percentage points on CIFAR-10 and 3.23 on STL-10 (Table 11). In the energy-regularized ImageNet run (β=20\beta=20), the angle between the learned-field and zero-field flow-matching gradients with respect to the starting representations was 3.25°–4.28° at the recorded iterations, showing that the learned field still changes the gradient direction (Table 17). With a 20-layer velocity network, FBDM reaches 61.55% ImageNet linear top-1 without the energy penalty (Table 2). The revision also reports three-seed results (Table 15), shows gains over random spherical centers (Table 14), and clarifies how FBDM differs from NAT.

Weakness 1A. The experiments do not support the central claim that the flow field shapes the representation: the objective with β = 20 is very close to direct regression. Question 2A. ImageNet controls. With β = 20, what is the ratio E‖v‖² / E‖U‖² during training?

Response. Thank you for asking whether low field energy makes the learned field behave like direct regression. With β=20\beta=20, the field-to-target energy ratio is about 0.10–0.11 at the measured checkpoints (Table 17). At those checkpoints, the angle between the learned-field and zero-field FM gradients ranges from 3.25° to 4.28°; these directional differences may accumulate over training updates. With the velocity network expanded to 20 layers, FBDM trains successfully at β=0\beta=0, reaching 61.55% ImageNet linear top-1 (Table 2).

Energy ratio and gradient direction. At each checkpoint, we hold the batch, assignments, and sampled times fixed and differentiate the FM loss with respect to each view's starting representation (Eq. (3)):

=\frac{h_\psi(f_\theta(\mathbf{x}_j^{(n)}))} {\\|h_\psi(f_\theta(\mathbf{x}_j^{(n)}))\\|_2}.$$ We concatenate these gradients across all images $n$ and both views $j$ to obtain $g_{\mathrm{flow}}$ and $g_0$ for the learned-field and zero-field branches. The zero-field branch still differentiates the analytic target velocity. We compute both metrics on these batch-level vectors: $$\delta_z = \frac{\\|g_{\mathrm{flow}}-g_0\\|_2}{\\|g_0\\|_2}, \qquad \vartheta_z = \arccos\left( \frac{\langle g_{\mathrm{flow}},g_0\rangle} {\\|g_{\mathrm{flow}}\\|_2\\|g_0\\|_2} \right).$$ We report $\delta_z$ as a percentage and $\vartheta_z$ in degrees. The energy ratio is $\mathbb E\\|v_\phi(Z_T,T)\\|_2^2/\mathbb E\\|U_T\\|_2^2$, with the two averages taken before division. | Epoch | Field/target energy | $\delta_z$ | $\vartheta_z$ | |---|---:|---:|---:| | 5 | 0.1124 | 8.71% | 4.28° | | 10 | 0.1119 | 8.21% | 3.92° | | 20 | 0.1083 | 8.01% | 3.43° | | 50 | 0.1056 | 7.71% | 3.25° | | 75 | 0.0984 | 7.83% | 3.46° | **Why the gradients can differ.** For one view, fix the assigned endpoint, sampled time $T$, and field parameters. Let $z$ denote its starting representation, $V=v_\phi(Z_T,T)$, and $U=U_T$. Omitting the common loss scale, the chain rule gives the learned-field minus zero-field FM gradient: $$\nabla_z\left(\\|V-U\\|_2^2-\\|U\\|_2^2\right) =2(D_zV)^\top(V-U)-2(D_zU)^\top V.$$ The energy penalty acts on the magnitude of $V$, but the gradient also depends on its derivative $D_zV$ through the path state $Z_T$ (Appendix G.1.2). The three-seed CIFAR-10 and STL-10 results in Question 1 separately show how the learned field affects downstream accuracy relative to the tested controls. **ImageNet without the energy penalty.** FBDM reaches **61.55%** linear top-1 with a 20-layer velocity network and $\beta=0$ (Table 2). All four runs below use a global batch size of 512 and 100 pretraining epochs. | Method | ImageNet Top-1 (%) | |---|---:| | Direct regression | 58.02 | | Geodesic regression | 58.75 | | FBDM (4-layer velocity network, β = 20) | 61.41 | | FBDM (20-layer velocity network, β = 0) | 61.55 | With the earlier 4-layer velocity network at batch size 512, changing only $\beta$ from 0 to 10 or 20 raised 50-epoch linear top-1 from 2.044% to 52.765% and 54.763% (Appendix G.1.1). A practical takeaway from our ImageNet experiments is to scale the capacity of the auxiliary velocity network alongside the encoder. Appendices E.1.2 and D.2 give the direct-regression and reported FBDM protocols.
AuthorsOpenReview ↗

Official Comment by Authors 2

Question 1. Is the flow needed? On CIFAR-10 (or STL-10), how do the following compare under matched tuning? (a) alignment loss only; (b) v ≡ 0, i.e. loss E[ṡ(T)²]ω² + λR_align; (c) direct geodesic regression (Eq. 117) plus alignment. If (b) or (c) is close to FBDM, the flow is not what drives the results.

Response. Thank you for pointing out that the original comparison left open whether learning a velocity field adds value beyond simpler objectives. Both controls retain FBDM's shared-target assignment and explicit view-alignment term. Across three seeds, FBDM exceeds the better control by 2.06 percentage points on CIFAR-10 and 3.23 on STL-10 (Table 11).

MethodCIFAR-10STL-10
Alignment onlyCollapse†Collapse†
Zero velocity field + alignment89.88±0.2489.88 \pm 0.2486.03±0.3586.03 \pm 0.35
Direct geodesic regression + alignment89.36±0.4189.36 \pm 0.4185.65±0.2785.65 \pm 0.27
FBDM91.94±0.38\mathbf{91.94 \pm 0.38}89.26±0.51\mathbf{89.26 \pm 0.51}

† The alignment-only objective admits representational collapse; this control was not evaluated (see Chen and He, CVPR 2021, §4.1 and Bardes et al., ICLR 2022, Table 7).

Results are reported as mean ±\pm standard deviation over three independent runs with different random seeds. For both the zero-velocity and geodesic-regression controls, we varied the learning rate, weight decay, and alignment weight for them. Then selected the best configuration and ran it with three different random seeds.

Weakness 2. The simplex-spectral construction is not the only way to obtain K′ > d*, and the manuscript's ablation does not show a stable gain from the choice of centers.

Response. Thank you for noting that random spherical centers also allow K′>d⋆K' > d^\star (Section 3.1). Simplex-spectral centers outperform them by 2.78 and 2.73 percentage points on CIFAR-10 and CIFAR-100, respectively, with other settings fixed and three runs per configuration (Appendix E.4, Table 14).

DatasetRandom normalizedSimplex-spectralImprovement
CIFAR-1089.16±0.4889.16 \pm 0.4891.94±0.3891.94 \pm 0.38+2.78 pp
CIFAR-10063.87±0.4263.87 \pm 0.4266.60±0.3266.60 \pm 0.32+2.73 pp

The benefit of separation can be understood through a simplified, idealized example. Suppose representations of class yy concentrate around a unit center cyc_y, and let δ=min⁡i≠j∣ci−cj∣\delta = \min_{i\ne j}\\|c_i-c_j\\|. The linear prototype classifier h(z)=arg⁡max⁡kck⊤zh(z)=\arg\max_k c_k^\top z has a geometric margin of at least δ/2−∣z−cy∣\delta/2-\\|z-c_y\\|. This mirrors the maximum-margin intuition of SVMs: greater center separation leaves more room for within-class variation before a representation crosses a decision boundary.

Weakness 3. The method seems close to Noise as Targets (NAT), which assigns features to fixed points on the sphere and regresses onto them.

Response. Thank you for drawing our attention to this relevant work. Both methods assign representations to reference targets. NAT [1] assigns each image to a distinct slot in a fixed set of spherical targets and directly regresses its representation toward the assigned target. Section 2 now discusses the relationship to NAT; Sections 3.2–3.4 and Algorithm 1 describe the FBDM assignment and training objective.

FBDM performs distribution matching through a capacity-constrained assignment to reference components. Multiple images can share a component up to its capacity. The two augmented views of each image share a target sampled from their assigned component, and an explicit alignment loss acts on their representations before transport. FBDM jointly learns the representations and an auxiliary velocity field by applying flow matching at intermediate states along the paths to the shared targets. The field is discarded after pretraining. Setting the field to zero yields a form of target regression that retains the capacity-constrained assignment and explicit two-view alignment.

[1] Unsupervised Learning by Predicting Noise. ICML, 2017.

Question 2B. Were the direct-regression controls in Table 8 tuned at all, for example with the same 1.3× weights and time weighting?

Response. Thank you for checking the loss scaling and time weighting of the controls. Both direct-regression controls use the same 1.3 multiplier on the regression loss that FBDM uses on the flow-matching loss. We tuned the alignment weight and learning rate and applied no time weighting (Appendix E.1.2, Eqs. (109)–(110) and the text below).

AuthorsOpenReview ↗

Official Comment by Authors 3

Weakness 1B. Baselines are copied with unclear protocol matching, and the ImageNet comparison has a single baseline.

Response. Thank you for identifying the need to make the evaluation protocol and the sources of the baseline results explicit. For CIFAR-10, CIFAR-100, STL-10, and Tiny ImageNet, FBDM follows the same frozen-feature linear-evaluation procedure as the W-MSE benchmark [2], subsequently adopted by CW-RGP [3]. We have checked our implementation against Section 4 and the official eval/sgd.py implementation of [2]. The relevant settings agree as follows:

ComponentShared linear-evaluation procedure
Feature extractionFreeze the backbone and use its outputs before the projector.
PreprocessingDeterministic evaluation preprocessing, with the same dataset-specific transforms.
ClassifierA linear classifier trained with cross-entropy loss and Adam.
Training budget500 classifier-training epochs; batch size 1,000.
Learning rateInitial learning rate 0.01, decaying to 10−610^{-6}.
Weight decay5×10−65 \times 10^{-6}.

The non-DM baseline values in Table 1 are taken from the common benchmark reported in [3]. In particular, its SimCLR, BYOL, and W-MSE results originate from [2], whose evaluation protocol is inherited by [3]. Thus, these values are published benchmark results evaluated under a common linear-evaluation protocol; their being cited from previous work does not imply that different classifier-training procedures were used. Table 1 and Appendix E.7 document the baseline sources; Appendix D.2 describes the FBDM evaluation protocol.

We also distinguish the scope of this matching:

  • Four datasets: The frozen-feature extraction, preprocessing, classifier optimization, and classifier-training budget listed above follow the common benchmark protocol. The baseline accuracies retain their published provenance. For pretraining, all methods use a ResNet-18 backbone. FBDM is trained for 1,000 epochs on all four datasets. The W-MSE/CW-RGP benchmark uses 1,000 epochs on CIFAR-10, CIFAR-100, and Tiny ImageNet, and 2,000 epochs on STL-10.

    In the order of CIFAR-10, CIFAR-100, STL-10, and Tiny ImageNet, the projection dimensions are 64, 72, 48, and 96 for FBDM; 64, 64, 128, and 128 for W-MSE; and 512, 512, 512, and 1,024 for CW-RGP. For optimization, FBDM uses Adam with a learning rate of 3×10−53 \times 10^{-5} and coupled L2L_2 regularization of 10−410^{-4}. The standard W-MSE/CW-RGP configuration uses Adam with a learning rate of 0.003 on CIFAR-10 and CIFAR-100, 0.002 on STL-10 and Tiny ImageNet, and weight decay of 10−610^{-6}. In the CW-RGP benchmark, SimSiam uses SGD with a cosine schedule.

  • ImageNet: The SimCLR value is an externally reported reference. FBDM and the external SimCLR reproduction both use a ResNet-50 backbone and are pretrained for 100 epochs with a global batch size of 512. Both use LARS with momentum 0.9, weight decay 10−610^{-6}, learning-rate warmup, and cosine decay. Their pretraining settings differ in the peak learning rate (0.9 for FBDM and 0.6 for SimCLR) and Gaussian blur probability (0.1 and 0.5, respectively).

    For linear evaluation, both methods freeze the backbone and use features before the projector. The classifier-training procedures differ: FBDM uses Adam for 500 epochs with a batch size of 1,000 and deterministic preprocessing, while the published SimCLR implementation uses SGD for 28,080 steps with a batch size of 4,096 and random cropping during classifier training. We also note that FBDM was not extensively tuned on ImageNet; its configuration was largely based on the settings used for the smaller datasets, with adaptations for ImageNet.

    Overall, the main differences lie in the pretraining hyperparameters and the linear-classifier training procedure.

    We reproduced VICReg and BYOL using the same 100-epoch budget and frozen-backbone linear evaluation protocol as FBDM. All four methods, including the external SimCLR reference, use global batch size 512. FBDM takes 29.58 minutes per epoch on four V100 GPUs (Table 2 and Appendix D.3).

    MethodLinear evaluation accuracy (%)
    SimCLR60.14
    VICReg61.29
    BYOL59.94
    FBDM61.55

    FBDM achieves competitive performance among these methods under the same pretraining epoch budget. The current comparison is in Section 4.1 (Table 2), with protocol differences in Appendix E.7 and the earlier FBDM result in Appendix G.1.1.

[2] Whitening for Self-Supervised Representation Learning. ICML, 2021.

[3] An Investigation into Whitening Loss for Self-supervised Learning. NeurIPS, 2022.

AuthorsOpenReview ↗

Official Comment by Authors 4

Weakness 1B. There are no seeds. Question 3. Seeds and model selection. Can you report mean ± std over at least 3 pretraining seeds? How were the per-dataset settings (clock family and a, capacity schedule, blur, warmup) chosen? Was there a validation split, or were choices made on test accuracy?

Response. Thank you for asking about variability and model selection. We selected settings using test-set accuracy and report mean and sample standard deviation over three pretraining seeds (Table 15):

DatasetLinear evaluation accuracy (%, mean ±\pm std)
CIFAR-1091.94±0.3891.94 \pm 0.38
CIFAR-10066.60±0.3266.60 \pm 0.32
STL-1089.26±0.5189.26 \pm 0.51
Tiny ImageNet48.38±0.3648.38 \pm 0.36

For the four smaller datasets, the per-dataset settings were selected based on test-set accuracy. This selection procedure and the three-run results are reported in Appendix E.5 (Table 15); Appendix D.2 (Tables 5–7) lists the configurations.

Notation and typographical issues.

Response. Thank you very much for your careful and detailed review and for identifying these issues. We appreciate the time and attention you devoted to examining the manuscript. Your constructive comments are very helpful for improving its clarity and presentation. We have corrected these issues in the revised manuscript.

View and time indices. In Fig. 1 and Eq. (17) of the original submission, z1\mathbf z_1 and z2\mathbf z_2 are the normalized projector outputs of the two augmented views before transport (revised Fig. 1 and Eq. (3)). In original Eq. (10), tt in Zt\mathbf Z_t indexes flow time, and Z1=R\mathbf Z_1=\mathbf R is the prescribed endpoint of the analytic path (revised Eq. (12)). Appendix E.3 clarifies that the ablation aligns the two view representations after evolving each to t=1t=1 with the learned velocity field.

Baseline name. The results in our Table 1 are for Zero-ICL, the instance-wise variant of Zero-CL [4]. As stated in Appendix E.7, the values come from the Zero-ICL reproduction in Table 1 of [3].

Flow-space dimension. The intended symbol is d⋆d^\star, which denotes the dimension of the auxiliary flow space (Section 3, Notation, and Eq. (3)). We checked its rendering in the revised PDF.

Spelling. We use "Flow-Based" and "center" consistently in the revised manuscript.

Flow-matching risk. In the original submission, RCFM(ϕ;qT)\mathcal R_{\mathrm{CFM}}(\phi;q_T) in Eq. (13) and RFM\mathcal R_{\mathrm{FM}} in Eq. (20) denote the same population flow-matching risk. In the revision, Eq. (15) defines it as RFM(f,ϕ;qT)\mathcal R_{\mathrm{FM}}(f,\phi;q_T), and Eq. (22) uses RFM\mathcal R_{\mathrm{FM}} as shorthand in the analysis with qTq_T fixed.

[4] Zero-CL: Instance and Feature Decorrelation for Negative-Free Symmetric Contrastive Learning. ICLR, 2022.

If you have any further questions, please feel free to let us know. We appreciate your recognition of our work and value your constructive advice.

Reviewer_rZf5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Flow-Based Distribution Matching (FBDM), a non-adversarial framework for self-supervised representation learning. It assigns two augmented views of an image to a shared geometric reference target and trains an auxiliary velocity field through spherical conditional flow matching, together with explicit view alignment. The auxiliary field is discarded after pretraining. Experiments report representation quality on several image benchmarks and higher training throughput than critic-based Distribution Matching (DM).

Critical strengths
  1. A relevant attempt to simplify explicit distribution matching. Replacing adversarial critic optimization with velocity regression is an interesting direction. The matched measurements in Table 3 demonstrate higher training throughput than the evaluated DM implementation, providing practical evidence for this motivation.

  2. Useful initial controls and implementation details. The appendix includes direct-target regression, spherical-versus-linear paths, and alignment diagnostics. These experiments provide initial support for the complete framework, while the detailed configurations help readers inspect its implementation.

Critical weaknesses
  1. The comparisons do not sufficiently establish the effectiveness of the proposed method.

    The paper should clarify whether the reported methods use comparable pretraining and classifier-training protocols. Table 1 largely adopts results from previous publications, while the ImageNet comparison uses an externally reported SimCLR result. Appendix D.3 specifies FBDM’s classifier-training procedure, but does not establish whether the corresponding procedures for the baselines are equivalent. Differences in feature extraction, preprocessing, classifier optimization, and evaluation budgets could affect the reported accuracy.

    Moreover, FBDM’s results are generally comparable to existing methods rather than consistently stronger. Apart from CIFAR-10 linear evaluation, Table 1 does not demonstrate a clear overall advantage. Comparable accuracy could still be valuable if accompanied by convincing efficiency improvements, but the cost comparison is limited to DM, whereas several established SSL alternatives already avoid adversarial training. Consequently, the current evidence does not sufficiently establish when FBDM should be preferred over these alternatives.

  2. The component analysis does not adequately explain the source of the benefits, and the presentation makes this difficult to assess.

    Although the appendix contains several useful ablations, important design choices remain insufficiently analyzed. In particular, the ImageNet implementation adds velocity-energy regularization, but its contribution is not isolated from that of flow matching. The proposed simplex-spectral reference is also not directly compared with a simpler random spherical reference.

    The efficiency analysis reports aggregate training throughput against DM, without sufficiently separating the costs of assignment, velocity prediction, and optimization. These measurements establish faster epochs for the tested implementation, but do not establish faster convergence or lower time to a comparable representation quality.

    The presentation further obscures this analysis. Sections 3–5 introduce numerous mathematical definitions and derivations before sufficiently developing an intuitive explanation of why the components are necessary. Figure 1 provides a useful overview, but the paper would benefit from a concrete running example and a clearer separation between the core algorithm, dataset-specific training modifications, and theoretical assumptions. This would help readers understand what the method contributes and which evidence supports each design choice.

Questions for authors
  1. Protocol clarification: Do the baseline results use the same frozen-feature extraction and classifier-training procedures as FBDM? Please identify which comparisons are strictly matched and which are cross-paper references.

  2. Effectiveness and efficiency: Under a matched training and evaluation protocol, how does FBDM compare with representative non-adversarial SSL methods in both accuracy and training cost? Can the authors distinguish per-epoch throughput from time to reach a comparable accuracy?

  3. Component attribution: How does ImageNet performance change without velocity-energy regularization? How does the proposed reference geometry compare with random normalized reference centers when the remaining components are fixed?

5 comments
AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank you for the time, care, and technical insight you brought to this review. Your questions about when FBDM is useful and how to assess its training cost helped us sharpen our claims. Your suggestions to test random spherical centers and add a running example strengthened the evaluation and made the method easier to follow.

Summary of our response.

  • Efficiency and use case. On a V100, FBDM is fastest per epoch on Tiny ImageNet and within two seconds of the fastest evaluated method on the other three ResNet-18 benchmarks (Table 8). It lets practitioners choose the reference distribution and match it without an adversarial critic.
  • ImageNet without energy regularization. FBDM reaches 61.55% linear top-1 (Table 2).
  • Reference geometry. Simplex-spectral centers improve over random spherical centers by 2.78 and 2.73 percentage points on CIFAR-10 and CIFAR-100, respectively (Table 14).

We also break down training costs, clarify baseline protocols, and add a running example.

Weakness 1: The cost comparison is limited to DM, whereas several established SSL alternatives already avoid adversarial training. Consequently, the current evidence does not sufficiently establish when FBDM should be preferred over these alternatives. Question 2: Effectiveness and efficiency: Under a matched training and evaluation protocol, how does FBDM compare with representative non-adversarial SSL methods in both accuracy and training cost? Can the authors distinguish per-epoch throughput from time to reach a comparable accuracy?

Response. Thank you for asking when FBDM is worth choosing. FBDM is useful when we want to specify the distribution that the learned representations should follow. Distribution matching offers a distinct route to self-supervised learning: a replaceable reference encodes the desired geometry and specifies what to match. FBDM addresses how to match it efficiently, using flow matching without an adversarial critic.

In additional CIFAR-10 experiments after 1,000 pretraining epochs, FBDM achieved 90.12% linear accuracy with a uniform spherical reference and 89.79% with an isotropic Gaussian reference in Euclidean space. The Gaussian targets were not normalized to the sphere.

On a V100, FBDM has the shortest reported epoch time on Tiny ImageNet and is within two seconds of the fastest evaluated method on each of the other three ResNet-18 benchmarks (Figure 3 and Table 8). In three separate matched comparisons with critic-based DM, FBDM was 2.17–2.36× faster per epoch (Table 3).

Accuracy and training cost. On CIFAR-10, CIFAR-100, STL-10, and Tiny ImageNet, FBDM's best linear top-1 scores are 92.37%, 66.92%, 89.79%, and 48.62%, versus 90.32%, 66.45%, 90.78%, and 48.71% for VICReg in the published benchmark (Table 1). The tables below give the per-epoch timings (Figure 3 and Table 8) and the ImageNet-1K accuracy comparison (Table 2).

Method (s/epoch)CIFAR-10CIFAR-100STL-10Tiny ImageNet
FBDM26.7226.9766.4261.67
SimCLR27.5127.1766.8963.31
BYOL30.9430.5073.4069.76
SimSiam25.8025.5664.8762.22
Barlow Twins26.8727.5465.8262.10
VICReg27.0928.8166.0463.05
Zero-ICL29.4128.9867.7665.16
W-MSE25.9226.4064.8562.88
CW-RGP80.4937.5285.7280.16
Shuffled-DBN25.8226.5264.5162.74
DM63.0562.62144.03--
MethodImageNet linear top-1 (%)Time (min/epoch)
SimCLR60.1430.02
BYOL59.9429.50
VICReg61.2929.54
FBDM61.5529.58

All ImageNet accuracy and timing comparisons use global batch size 512. Timings use four V100 GPUs with mixed precision; the FBDM measurement uses the reported 20-layer velocity network and per-batch assignment (Table 2 and Appendix D.3).

The earlier timing comparison included redundant regeneration of fixed reference centers. We optimized both implementations to reuse these centers and matched the parameter counts of DM’s critic and FBDM’s velocity network (Table 10). We then remeasured wall-clock time per epoch on one V100 in FP32 at batch size 512, averaging complete epochs 2 and 3 after one warm-up epoch and including assignment, network computation, and optimization. Under this protocol, FBDM is 2.172.17–2.36×2.36\times faster per epoch than DM on the three shared datasets.

DatasetDMFBDM
CIFAR-105.07 / 5.445.28 / 5.79
CIFAR-1005.07 / 5.435.28 / 5.79
STL-107.35 / 8.527.56 / 8.71

Peak allocated / reserved CUDA memory in GiB under the DM comparison settings above. See Section 4.2 (Table 3) for the paired timing and memory results and Appendix D.3 (Table 10) for the matched auxiliary parameter counts.

AuthorsOpenReview ↗

Official Comment by Authors 2

Weakness 2: The ImageNet implementation adds velocity-energy regularization, but its contribution is not isolated from that of flow matching. Question 3: How does ImageNet performance change without velocity-energy regularization?

Response. Thank you for asking whether the ImageNet result depends on energy regularization. With a 20-layer velocity network and global batch size 512, FBDM reaches 61.55% linear top-1 after 100 epochs with β=0\beta=0 (Table 2). At the same batch size, the earlier run with a 4-layer velocity network and β=0\beta=0 reached 2.044% after 50 epochs (Table 11 of the original submission). The new result suggests that a larger velocity network is needed for effective ImageNet training without the energy penalty.

Appendix D.2 details the reported accuracy configuration, and Appendix G.1.1 gives the energy ablation with the 4-layer velocity network.

Weakness 2: The proposed simplex-spectral reference is also not directly compared with a simpler random spherical reference. Question 3: How does the proposed reference geometry compare with random normalized reference centers when the remaining components are fixed?

Response. Thank you for suggesting random spherical centers as a control. Over three runs per configuration, simplex-spectral centers improve on this control by 2.78 and 2.73 percentage points on CIFAR-10 and CIFAR-100, respectively, with all other settings fixed (Appendix E.4, Table 14):

DatasetRandom normalizedSimplex-spectralImprovement
CIFAR-1089.16±0.4889.16 \pm 0.4891.94±0.3891.94 \pm 0.38+2.78 pp
CIFAR-10063.87±0.4263.87 \pm 0.4266.60±0.3266.60 \pm 0.32+2.73 pp

The remark in Section 3.1 gives the maximum-margin intuition for center separation. A simple illustration is to suppose that representations of each class yy concentrate around a corresponding unit-norm center cyc_y, and let δ=min⁡i≠j∣ci−cj∣\delta=\min_{i\ne j}\\|c_i-c_j\\| be the minimum distance between centers. The linear prototype classifier h(z)=arg⁡max⁡kck⊤zh(z)=\arg\max_k c_k^\top z has a geometric margin of at least δ/2−∣z−cy∣\delta/2-\\|z-c_y\\|. This mirrors the maximum-margin intuition of SVMs: greater center separation leaves more room for within-class variation before a representation crosses a decision boundary.

Weakness 2: The efficiency analysis reports aggregate training throughput against DM, without sufficiently separating the costs of assignment, velocity prediction, and optimization. These measurements establish faster epochs for the tested implementation, but do not establish faster convergence or lower time to a comparable representation quality.

Response. Thank you for asking which stages account for the per-epoch cost. Table 9 (Appendix D.3) breaks down training time over three complete epochs per dataset.

DatasetAssignmentVelocity forwardEncoder/projector forwardBackward/optimization/other
CIFAR-105.01%1.55%25.48%67.97%
CIFAR-1005.65%1.63%25.52%67.20%
STL-104.55%1.50%25.51%68.45%
Tiny ImageNet9.38%1.49%25.35%63.77%
ImageNet-1K5.78%0.13%52.86%41.23%
AuthorsOpenReview ↗

Official Comment by Authors 3

Weakness 2: The presentation further obscures this analysis. Sections 3–5 introduce numerous mathematical definitions and derivations before sufficiently developing an intuitive explanation of why the components are necessary. Figure 1 provides a useful overview, but the paper would benefit from a concrete running example and a clearer separation between the core algorithm, dataset-specific training modifications, and theoretical assumptions. This would help readers understand what the method contributes and which evidence supports each design choice.

Response. Thank you for this suggestion. At the start of Section 3, Figure 1 follows two overlapping crops of the same airplane. Blue and orange trace the crops through a shared backbone and projector to their respective starting points z1\mathbf z_1 and z2\mathbf z_2 on the sphere. The two views use their consensus to select one reference endpoint r\mathbf r, so the assignment does not send them toward different targets while the alignment loss brings their starting points closer. The figure traces the two paths to r\mathbf r and marks alignment separately between the starting points.

With the same shared target and view-alignment term, direct regression penalizes the distance from each starting representation to the target. FBDM uses intermediate states along both paths to train a velocity field shared across images. The field can model part of the transport, and the velocity prediction errors along the paths also guide the encoder.

Sections 3.1 and 3.2 then explain the reference construction and why capacities cap occupancy without enforcing equal counts. Section 4.3 and Appendix E.1.1 (Table 11) compare velocity regression with direct regression. Section 3 and Algorithm 1 present the core method, Appendices D.1 and D.2 give the training refinements and dataset-specific settings, and Section 5.2 states the assumptions for the theoretical analysis. Appendix E opens with a short guide to the ablations and additional evaluations.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Follow-Up on Our Responses

Dear Reviewers,

Thank you very much for the time and effort you have put into reviewing our paper and for your thoughtful feedback. We know this is a particularly busy period for you, and we apologize for the extra message.

With the discussion deadline approaching, we wanted to follow up on our responses. If you have a moment, could you share any remaining questions, concerns, or suggestions? We would appreciate the chance to clarify any points about our work while the discussion is still open.

Best regards,
The authors of Submission 12316

AuthorsOpenReview ↗

Rebuttal Summary for the Area Chair

Dear Area Chair,

Thank you for overseeing the review. With the discussion deadline approaching, we have received no response to our rebuttal or to our follow-up asking whether the reviewers had any remaining questions or concerns. We have put the key results and clarifications in one place to help with your assessment and meta-review.

FBDM achieves competitive linear-evaluation accuracy compared with established self-supervised methods. It matches a chosen reference distribution through flow matching, delivers a 2.17–2.36× per-epoch speedup over critic-based Distribution Matching (DM) in matched comparisons, and is among the fastest evaluated SSL methods by training time per epoch across the four ResNet-18 benchmarks.

The rebuttal adds the following evidence.

  • Learned velocity field (Table 11). The zero-field and direct geodesic regression controls both retain shared target assignment and explicit view alignment. Over three seeds per configuration, FBDM improves mean linear accuracy over the stronger control by 2.06 percentage points on CIFAR-10 and 3.23 points on STL-10.
  • Reference geometry (Table 14). With the other settings fixed, simplex-spectral centers improve over random spherical centers by 2.78 points on CIFAR-10 and 2.73 points on CIFAR-100, over three runs per configuration.
  • ImageNet comparison (Table 2). After 100 epochs at global batch size 512, FBDM reaches 61.55% linear top-1 without velocity-energy regularization. VICReg and BYOL reach 61.29% and 59.94% under the same frozen-backbone evaluation. All three take about 29.5 minutes per epoch on four V100s.
  • Training cost across the four ResNet-18 benchmarks (Figure 3 and Table 8). FBDM has the shortest measured epoch time on Tiny ImageNet and is within two seconds per epoch of the fastest evaluated method on each of the other three benchmarks.
  • Transfer beyond classification (Table 16). The frozen ImageNet-pretrained backbone reaches 63.37% [email protected] on VOC 2007 detection and 95.54% mIoU on Oxford-IIIT Pet segmentation, improving on the random-initialization controls.

We also clarified the geometric condition under which terminal Wasserstein error bounds class–component leakage in the analysis (Appendix C.4).

Best regards,
The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.