acceptodds
Under review as a conference paper at ICLR 2027

Paired Flow Matching: Representation Learning without a Reconstruction Bottleneck

Abstract

Variational autoencoders (VAEs) are a standard tool for representation learning, but their samples often have lower fidelity than diffusion and flow models that generate directly in observation space. Latent diffusion and latent flow matching reduce computation by generating in a VAE latent space, yet they too lose sample quality. The shared limitation is the encode-and-decode design: a compact representation should discard presentation details, but the decoder needs those details to reconstruct the observation. The representation must therefore either retain unwanted detail or incur reconstruction error. To resolve this conflict, we propose Paired Flow Matching (PairedFM), which uses representations to guide transport between full observations. From ordered pairs drawn from a single data distribution, PairedFM jointly learns a representation and a family of flows in observation space, indexed by the pair's representations. At generation, a source observation supplies the details and a target representation determines where to move. PairedFM also comes with theoretical guarantees: We characterize when its representation is identifiably and the resulting transport achieves exact recovery of the truth. Across empirical studies, we find that PairedFM learns informative representations while maintaining sample quality where VAE and latent flow matching baselines degrade.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Flow Matching, Representation Learning, Probabilistic Models, Identifiability, Latent variable models, Conditional generation.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YTQyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Paired Flow Matching (PairedFM), a framework that jointly learns representations and observation-space flows without an explicit reconstruction objective. By conditioning the flow on learned source and target representations, the method aims to overcome the reconstruction bottleneck of conventional autoencoders.

Critical strengths

1.The paper presents an interesting perspective on jointly learning representations and generative models without relying on an explicit reconstruction objective.

2.The theoretical analysis provides useful insights into representation identifiability and generation guarantees under certain assumptions.

Critical weaknesses

1.Lack of validation on modern generative models. The experiments are primarily conducted on synthetic or relatively simple datasets. Despite the paper's motivation concerning the limitations of VAEs and latent generative models, there are no evaluations on modern image generation architectures such as DiT or SiT. Without experiments on realistic image generation benchmarks, it is difficult to assess the practical effectiveness and scalability of the proposed framework. This is a major concern.

2.Insufficient comparisons with representative VAEs. The paper does not compare against widely adopted visual tokenizers, such as SD-VAE or the FLUX series of VAEs. These comparisons are essential to establish whether the proposed method offers meaningful advantages over modern autoencoding approaches, particularly in terms of representation quality, reconstruction fidelity, and downstream generation performance.

3.Limited evaluation of generalizability. The paper lacks experiments across different image resolutions and more challenging generation tasks, such as text-to-image (T2I) generation. Evaluations on realistic, high-resolution datasets would substantially strengthen the empirical evidence and clarify the applicability of the proposed method.

Questions for authors

1.Lack of validation on modern generative models.

2.Insufficient comparisons with representative VAEs.

3.Limited evaluation of generalizability.

3 comments
AuthorsOpenReview ↗

Response to Reviewer YTQy

We appreciate your recognition of the reconstruction-free learning perspective and the insights provided by the theoretical analysis. We have substantially expanded the empirical evaluation in the revised manuscript, adding a spatial DiT velocity network, pretrained SD-VAE comparisons, natural-image experiments across three resolutions, and an exploratory text-conditioned generation study. These additions also address R1's scalability questions and R2's request for more realistic representation evaluation. The added material is marked in dark red in the revised manuscript.

We respond to your concerns and questions point by point below.

W1. Validation on modern generative architectures

We have added natural-image experiments on CIFAR-10, Tiny ImageNet, and FFHQ (Appendix A.4), together with a spatial DiT implementation of the PairedFM velocity network (Appendix A.5, Table~14). The DiT and U-Net velocity networks are each trained across three seeds using the same frozen encoder and training budget. Both support code-directed image recovery, with the U-Net achieving better fidelity at the evaluated budget. These additions demonstrate PairedFM on natural images and its compatibility with a transformer-based velocity architecture.

W2. Comparisons with representative VAEs

We now compare against pretrained SD-VAE, evaluating image recovery and frozen-code classification (Appendix A.4, Tables 12 and 13). In the one-step comparison, PairedFM achieves lower pixel MSE on all three image benchmarks. SD-VAE achieves lower LPIPS using the resized reference on CIFAR-10 and native resolution on Tiny ImageNet and FFHQ. PairedFM also attains comparable classification accuracies under the common linear-probe procedure.

These results support useful representations and competitive recovery, with a tradeoff between pixel and perceptual fidelity. The manuscript specifies the differences in code size, source access, and training history.

W3. Generalizability across resolutions and text-conditioned generation

The expanded experiments cover image resolutions of 32×3232\times32, 64×6464\times64, and 128×128128\times128. We also added text-conditioned generation on FFHQ (Appendix A.6, Table15 and Figures15--16). A learned prior generates target codes from noise and captions, which condition the PairedFM flow from a source image. The target code is therefore sampled rather than computed from an observed target image.

At guidance 2, PairedFM with one image step achieves higher CLIP alignment than the decoder of the same sampled codes (0.26070.2607 versus 0.24800.2480), together with greater caption-swap responsiveness. The figures illustrate responses to face and clothing descriptions. These results demonstrate compatibility with text conditioning in the evaluated FFHQ setting. Within the revision period, we expanded the evaluation to natural images at three resolutions and text-conditioned generation on FFHQ. These additions demonstrate PairedFM's applicability beyond the original controlled settings, including caption-conditioned generation in the face domain.

Q1. Validation on modern generative models

Please see our response to W1 above.

Q2. Comparisons with representative VAEs

Please see our response to W2 above.

Q3. Generalizability

Please see our response to W3 above.

We hope these responses address your concerns and welcome any follow-up questions.

Can you please respond to the authors even in the last minute?

Thank you

Thank you very much for your efforts; however, the authors have not provided any additional, compelling experiments to demonstrate effectiveness. I have decided to maintain my score.

Reviewer_jedwOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PairedFlow Matching learns jointly an encoder and a flow from pairs of observations. Give (Y0,Y1)(Y_0,Y_1) the model predicts the endpoint along the linear interpolation conditioned on c0andc1c_0 and c_1 from ci=h(Yi)c_i=h(Y_i). The motivation is to do representation learning without requiring a decoder to reconstruct the full observation from the representation only. They have some nice theoretical results that connects the population flow matching loss to recovery of Y, representation geometry and endpoint error. They also have grouped PairedFM which considers pairs sharing nuisance variables GG while varying UU and provides some conditions under the learned representations across groups. Experiments are done on branching fractals, PMBC simulation prediction and shapes 3D.

Critical strengths
  • I like the core formulation. Using the same velocity-prediction objective to learn both the transport and the representation is quite clean, and I think the idea of avoiding an explicit reconstruction objective is interesting. The theory also gives a useful connection between low FM loss, recoverability from the code, and generation error.

  • I think the grouped-pairing idea is the strongest part of the paper. The source keeps the shared information G, while the target code is supposed to specify the changing factor U. This is a nice way to think about representation learning through transport. I also appreciated that the appendix explicitly discusses failure cases for coordinate alignment rather than pretending independence alone is enough.

Critical weaknesses
  1. I am not fully convinced by the branching-fractal generation comparison. PairedFM gets a held-out source together with h(Y1)h(Y_1) computed from an actual target observation, while FM, β\beta-VAE, and LatentFM sample from noise or from their priors. At the same time, Y1Y_1 is almost perfectly recoverable from the PairedFM code (8-NN R2≈0.9998R^2 \approx 0.9998). To me, this looks closer to conditional target recovery, or implicitly decoding the target code, than to the same generative problem faced by the other methods. I would like to see an ablation where h(Y1)h(Y_1) is zeroed, shuffled, or progressively corrupted, ideally also as a function of tt, to understand how much the model actually relies on the representation rather than information already present in YtY_t. A simple g(h(Y1))→Y1g(h(Y_1))\rightarrow Y_1 decoding baseline would also help establish whether the flow is doing substantially more than decoding a nearly invertible target representation.

  2. I also think the representation experiments are still a bit limited relative to the motivation of the paper. On Shapes3D the main experiment focuses on object hue, other scene information is still decodable from the code, and the harder hue/shape/scale intervention is clearly weaker. Since the dataset gives all the ground-truth factors, I would expect something like DCI, MIG/SAP, or at least a factor-by-latent importance matrix. I would also like to see compositional generalization to held-out factor combinations rather than mostly recombination inside a very structured factor grid. More broadly, the representation experiments remain quite controlled. I do not think a full ImageNet-scale study is strictly necessary, but some evidence on a more realistic representation-learning or structured-transfer problem would make the central claim much stronger. For example, large-scale frozen-code probing or object/region transfer using segmentation information could test whether the method preserves generation quality while learning useful semantic representations. On PBMC, the comparison is also not fully controlled because PairedFM uses a 16-dimensional code while the autoencoder baselines use 32 dimensions.

Questions for authors
  1. For the 4,096 fresh fractal source-target pairs, are Y0Y_0 and Y1Y_1 newly sampled from the underlying continuous distribution, or are these new pairings from a fixed held-out set? Also, what are the training-set size and the exact fractal sampling procedure?

  2. For Shapes3D, what exactly is excluded by the 5% scene holdout? Are the evaluated (U,G)(U,G) combinations genuinely unseen during training, or are they combinations of factor values that the model has already seen elsewhere in the grid?

  3. For Theorem 4, how should I interpret the assumptions in relation to the actual Shapes3D model? In particular, are h(Y0)⊥h(Y1)h(Y_0)\perp h(Y_1) and the class-A\mathcal A condition in Eq. (12) enforced or checked in the neural encoder, or is the theorem meant only as a sufficient-condition result that does not directly apply to the implementation?

5 comments
AuthorsOpenReview ↗

Response to Reviewer jedw [1]

Thank you for highlighting the clean joint objective and the grouped-pairing perspective. Your detailed suggestions helped us strengthen the revised manuscript through code interventions, decoder comparisons, expanded Shapes3D factor diagnostics, and natural-image representation probes. We also clarify the distinction between target recovery and unconditional generation, the sampling and holdout protocols, and how the theoretical assumptions relate to the implementation. The added material is marked in dark red in the revised manuscript.

We respond to your concerns and questions point by point below.

W1. Fractal comparison, target-code dependence, and decoder baseline

We distinguish the original target-conditioned recovery experiment from unconditional sampling by the baselines. We evaluated the frozen model with zeroed, shuffled, progressively noisy, and time-switched target codes. Shuffling redirects the output toward the coded donor: donor-recovery MSE is 1.12×10−51.12\times10^{-5}, close to the intact-code MSE of 1.11×10−51.11\times10^{-5}. Recovery error increases with zeroed or progressively noisy codes, and switching codes at t=0.75t=0.75 changes the endpoint. These results show that the target representation actively guides recovery, including late in the trajectory.

We also added a code-only MLP decoder and a restricted-code ablation (Appendix A.2, Figure 8). With full codes, both readouts reproduce the branching pattern, with lower coordinate MSE for the MLP. Under mean pooling, the decoder concentrates outputs along a narrow curve, while PairedFM produces a broader two-dimensional spread. With noisy pooling, the decoder curve becomes smoother, while the flow continues to produce dispersed outputs.

Across three seeds, the fractions below the fifth percentile of real-target log density average 48.2%48.2\% for PairedFM and 78.1%78.1\% for the decoder under pooling, and 51.5%51.5\% and 69.8%69.8\% under noisy pooling. The corresponding real-target reference is approximately 5%5\%. These results distinguish the source-conditioned flow from the code-only decoder under restricted conditioning: PairedFM produces a broader spread and a lower average fraction of low-density outputs.

W2. Representation evaluation and baseline comparisons

We have expanded the representation evaluation with DCI, MIG, SAP, linear and nonlinear factor probes, and a factor-by-code information matrix covering all six Shapes3D factors (Appendix A.1, Table 7 and Figure 3). The additional comparison uses ten-dimensional codes and a common training-scene partition for all methods. PairedFM achieves 99.98%99.98\% object-hue accuracy, while floor- and wall-hue accuracies are 34.60%34.60\% and 36.38%36.38\%, compared with approximately 99%99\% for the comparators. These results support strong recovery of the selected factor with lower background-factor predictability. Figure 3 shows hue information distributed across several code coordinates; the DCI, MIG, and SAP results provide a complementary assessment of coordinate-wise disentanglement.

The scene holdout excludes complete combinations of the five scene factors, including all ten object hues for each excluded scene. Evaluation therefore includes observations and scene configurations absent from training, using factor values represented elsewhere in the grid. The additional factor-diagnostic comparison applies this split to all three encoders. We report the joint hue/shape/scale intervention separately (Figure 5), which shows the additional challenge of transferring several factors together.

We have also added frozen-code probing on natural images alongside image-recovery experiments (Appendix A.4, Tables 12 and 13). PairedFM achieves linear classification accuracies of 43.3%43.3\% on CIFAR-10 and 5.0%5.0\% on 200-class Tiny ImageNet, compared with 42.8%42.8\% and 5.2%5.2\% for the pretrained SD-VAE reference under the same probe procedure. These results extend the evaluation beyond controlled factor grids: representations learned through velocity prediction retain class information while supporting observation recovery.

For PBMC, PairedFM uses a 16-dimensional code, whereas the autoencoder baselines use 32 dimensions. We report this configuration difference explicitly. The additional pairing ablation fixes the code dimension at 16 and uses the same architecture and training budget across samplers (Appendix A.3, Table 11), isolating the effect of pairing.

AuthorsOpenReview ↗

Response to Reviewer jedw [2]

Q1. Fractal sampling procedure and training-set size

The 4,096 evaluation pairs are newly sampled from the underlying continuous distribution, not new pairings of a fixed held-out observation bank. Training uses minibatches of 256 pairs drawn with replacement from a fixed bank containing 128 minibatch-equivalents. Optimization runs for 6,000 updates, with checkpoint selection on 1,024 separately sampled validation pairs. Each observation is a two-dimensional point.

The fractal distribution is a fixed mixture of 1,016 anisotropic Gaussians, with construction seed 2. Each observation is generated by

\epsilon\sim\mathcal N(0,I_2),\qquad Y=\mu_J+\Sigma_J^{1/2}\epsilon.$$ Source and target use independently drawn component indices and Gaussian noise. **Q2. Shapes3D holdout and unseen factor combinations** A scene is a combination of floor hue, wall hue, scale, shape, and orientation. The 5\% holdout excludes complete scene configurations, including all ten object hues for each excluded scene, from Grouped PairedFM training. Thus, the evaluated $(U,G)$ combinations are unseen, although their individual factor values appear elsewhere in the training grid. The original baseline encoders used the complete grid; the additional representation comparison in Table 7 uses a common training-scene partition for all three methods. **Q3. Theorem 4 assumptions and their relation to the implementation** Theorem 4 provides sufficient conditions for consistent factor recovery and transfer across groups. Paired-code independence and the class-$\mathcal A$ condition in Equation (12) serve as assumptions for this theoretical guarantee. The Shapes3D implementation learns the encoder through the paired velocity-prediction objective, and we complement the analysis with empirical diagnostics of paired-code dependence and cross-scene consistency. Specifically, Table 6 reports paired-code distance correlation of $0.165$, comparable to the target-code permutation reference of $0.183$. The cross-scene same-hue distance is only $0.249$ times the within-scene one-hue-step distance. Thus, on average, representations of the same hue across different scenes are substantially closer than representations of adjacent hues within a scene. These measurements provide empirical support for cross-scene consistency of the learned hue representation, complementing the theoretical analysis. We hope these responses address your concerns and welcome any follow-up questions.

first of all thanks for the authors for the detailed rebuttal and revised manuscript. Several of my concerns are clarified/solved.

  • W1. the decoder ablation is helpful and please correct me if i am misinterpreting (fig. 8). I was mainly concerned about the role of the source observation. I think the comment clarifies but with full representation a simple code-only decoder seems to achieve lower endpoint MSE than pairedFM. To me , this suggest that the target code contains almost all information needed for recovery. PairedFM seems interesting when the code is restricted but some generation quality is sacrificed.
  • W2. disentanglement metrics are lower (table 7) than β\beta-vae or latentFM. hue specifically is distributed across several coords. I think it is doing some representation learning but not on the coordinate wise for disentanglement.
Reviewer_65JjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the fundamental "reconstruction bottleneck" in generative representation learning (e.g., VAEs, Latent Diffusion, and Latent Flow Matching). In traditional encode-and-decode architectures, a compact representation should ideally discard low-level presentation details, yet a decoder requires the exact details to faithfully reconstruct the high-dimensional observation. To resolve this conflict, the authors propose Paired Flow Matching (PairedFM). Instead of relying on an explicit decoder to reconstruct complete observations from representations alone, PairedFM jointly learns an encoder and a conditional velocity field in observation space using a single velocity-prediction objective over ordered pairs drawn from the data distribution. The paper provides rigorous theoretical guarantees under standard regularity assumptions, and empirical evaluations across three benchmarks demonstrate that PairedFM achieves superior fidelity compared to VAE and Flow Matching baselines.

Critical strengths
  1. Bypassing explicit decoders by conditioning observation-space transport on paired representations offers an elegant and effective solution to the reconstruction bottleneck.

  2. Directly optimize latent variable recovery and continuous-time ODE trajectory errors through the population velocity prediction loss provides solid mathematical justification.

  3. Grouped PairedFM extends the methodology to disentangled factor transfer without altering the core architecture or loss function. This provides strong practical value, particularly in domains where domain knowledge allows for meaningful data grouping.

  4. Experimental results across synthetic fractals, high-dimensional single-cell genomics, and multi-factor image manipulation consistently support the effectiveness of the proposed algorithm.

Critical weaknesses
  1. Because generation relies on a real source observation as the initial state, unconditional generation from scratch requires first sampling a source point from the data distribution rather than sampling directly from a standard Gaussian prior. Furthermore, sampling requires solving an ODE in observation space, making inference inherently slower than a single forward pass of a standard VAE decoder.

  2. In practical applications, constructing high-quality training pairs with the desired group granularity can be challenging. Providing a generic pipeline, practical heuristics, or robustness analysis for pair construction would significantly enhance the method's real-world usability.

Questions for authors
  1. While 64×64 RGB images demonstrate strong performance in the experiment, how well does PairedFM scale to ultra-high-dimensional observation spaces (e.g., 512×512 natural images or video streams) where evaluating velocity fields across full resolution becomes computationally intensive?

  2. How should one approach unconditional sampling when a suitable initial source observation is unavailable? Is there a principled workaround or adaptation within the PairedFM framework?

  3. How sensitive is the model to the quality and structure of paired data? For instance, how would randomly sampled paired data compare against group-sampled paired data on the PBMC task?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 65Jj

We thank you for recognizing the theoretical grounding of PairedFM and the practical value of grouped factor transfer. Following your suggestions, we revised the manuscript to clarify sampling requirements and added a controlled PBMC comparison of independent and grouped pairing, alongside few-step recovery experiments on natural images. We also discuss an adaptation for generation when no stored source observation is available. The added material is marked in dark red in the revised manuscript.

We respond to your concerns and questions point by point below

W1. Source requirements and inference cost

PairedFM samples source and target observations independently during training (Algorithm 1). At generation time, the source can be selected independently from stored observations. A learned target-code prior enables sampling without an observed target, while the observation-space flow uses a source image as its initial state.

Regarding inference cost, our added CIFAR-10, Tiny ImageNet, and 128×128128\times128 FFHQ experiments demonstrate target recovery with few velocity evaluations, including one-step and four-step settings (Appendix A.4, Table 12). One-step recovery uses a single velocity-network evaluation. The preferred step count depends on the dataset and the balance between pixel and perceptual error.

W2. Practical construction of training pairs

Our added natural-image experiments in Appendix A.4 use independently sampled pairs. On PBMC, we also compare independent pairing with pairing within cell-type/patient groups across seven target cell types and three seeds, using the same architecture and training budget. Independent pairing achieves a higher mean response RΔ2R^2_\Delta across cell types (0.7980.798 versus 0.7880.788). Grouped pairing gives higher scores for CD4 T cells (0.7380.738 versus 0.6900.690) and dendritic cells (0.8270.827 versus 0.8050.805). These results support independent pairing as an effective option, with the benefit of grouping depending on the target population.

For selective factor transfer, grouping specifies which attributes should be preserved. In Shapes3D, pairs share scene factors while object hue varies (Appendix A.1). A practical approach is to define groups using available metadata, sample pairs independently within each group, and evaluate both the intended change and preservation of source attributes on held-out data. Including independent pairing as a baseline helps assess the benefit of the chosen grouping.

Q1. Scaling to higher resolutions

Our added natural-image experiments cover resolutions of 32×3232\times32, 64×6464\times64, and 128×128128\times128, with effective target recovery using few velocity evaluations, including one-step and four-step settings (Appendix A.4, Table 12). Appendix A.5 also evaluates U-Net and spatial DiT velocity networks within the same PairedFM objective.

Few-step inference reduces the number of full-resolution velocity evaluations. At higher resolutions, the cost and memory required for each evaluation also increase, making the velocity-network architecture important for scaling. The current experiments demonstrate recovery up to 128×128128\times128. A systematic study at 512×512512\times512 and on video would require additional computational resources and is left for future work.

Q2. Sampling without an observed source

A possible adaptation is to generate the initial source. Fit a prior over the learned codes and a decoder from codes to observations. At generation time, independently sample two codes c0,c1c_0,c_1 and decode the first into an initial observation y~0=D(c0)\widetilde y_0=D(c_0). PairedFM then evolves from y~0\widetilde y_0, conditioned on h(y~0)h(\widetilde y_0) and c1c_1. Both the source and target conditioning are therefore obtained from sampled codes.

To adapt the flow to these generated sources, the velocity network can be fine-tuned using decoded sources paired with real training targets. This retains the paired velocity-prediction objective while aligning training with the source distribution used at generation.

Q3. Independent versus grouped pairing on PBMC

The additional PBMC ablation directly compares independent pairing with pairing within cell-type/patient groups (Appendix A.3, Table~11). Both settings use the same architecture and training budget across seven target cell types and three seeds.

Independent pairing gives a higher average response RΔ2R^2_\Delta (0.7980.798 versus 0.7880.788). Grouped pairing gives higher scores for CD4 T cells and dendritic cells, with a smaller numerical gain for FCGR3A$^+$ monocytes. Independent pairing also achieves lower response MAE for every target cell type. These results show that the benefit of grouping depends on the target population and evaluation metric. The comparison isolates the effect of pairing structure under matched training conditions.

We hope these responses address your concerns and welcome any follow-up questions

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.