acceptodds
Under review as a conference paper at ICLR 2027

Semantic Latent Alignment for Flow Matching with Invertible Bi-Lipschitz Adapters

Abstract

Pretrained latent spaces trade compression, reconstruction, and the learnability of the coordinates a downstream generator has to work in, which complicates adapting these spaces to different domains or tasks: continued training is expensive, and without a new alignment objective it often fails to improve downstream generation. We instead leave the encoder and decoder fixed and change only the coordinates between them, using an invertible bi-Lipschitz adapter aligned offline to a semantic teacher and frozen during class-conditional flow matching. Invertibility preserves compatibility with the pretrained autoencoder, while the bi-Lipschitz constraint limits how far alignment can distort its latent geometry. This provides a simple way to improve semantic organization while retaining the compression and reconstruction properties of the original latent space. On two medical imaging datasets, constrained latent adaptation improves feature metrics, seed stability and counterfactual edit strength, while remaining competitive on downstream classification. In contrast, unconstrained adaptation can become highly anisotropic and sensitive despite remaining invertible, highlighting the importance of controlling geometric distortion. Our results suggest that carefully constrained post-hoc adaptation offers a parameter-efficient and tuning-free alternative to jointly modifying the representation during generative training.

Primary area:
generative models
Keywords:
Flow Matching, Latent Space Geometry, Generative Modeling for Medical Imaging, Representation Alignment, Invertible Maps

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kkGqOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

In order to train an image flow matching model in latent space, it is common to use a pretrained VAE trained on a large scale image dataset. However, the latent space might not be well-aligned with the target image distribution as it can be significantly different from the distribution used to train the VAE.

The paper proposes to use a simple learned adapter gθg_\theta to modify the behavior of pretrained VAE. It is then composed with the encoder E\mathcal{E} and decoder D\mathcal{D} networks by gθ∘Eg_\theta \circ \mathcal{E} and D∘gθ−1\mathcal{D} \circ g_\theta^{-1} respectively.

This adapter is designed to be an invertible bi-Lipschitz function and trained to minimize the Centered Kernel Alignment with respect to a reference embedding model, the teacher model ϕ\phi. L=1−CKA(P∘gθ∘E(X1),ϕ(X1))L = 1 - CKA(P \circ g_\theta \circ \mathcal{E}(X_1), \phi(X_1)) With X1X_1 sampled from the target image distribution and PP a pooling operation. Once the adapter is trained they concatenate it with a postprocessing operation NN and learn the FM model by transporting the full path instead of doing standard FM in the new latent space.

Trained with MIMIC-CXR and CT-RATE datasets, the paper argues that it is competitive with other latent adapting methods while being typically more stable

Critical strengths

Paper is well presented, with a clear motivation and a clean construction. The need for bi-Lipschitz functions instead of just invertible ones is justified cleanly. The result itself that shows that invertible functions are not enough is very interesting on itself. The proposed adapter strategy through bi-Lipschitz functions is flexible and allows further exploration such as through different losses beyond CKA. The adapter is also lightweight in parameters, making it efficient compared to other adapter methods

Critical weaknesses

I have significant concerns about the adapter network behavior after training. There are reasons to suspect that the network is zeroing-out all residual connections and thus behaving essentially as a simple orthogonal + scaling operation sQsQ. If this is the case, then it undermines the complete construction as the adapter is not learned from the data and barely contributes to the encoding. I'll explain why I think this is the case.

Table 7 shows the certified stretch/compression bounds and compares them with maximum stretching obtained by sampling the Jacobian in different points. It shows a 1.6491.649 certificate vs 1.042–1.0561.042–1.056 for the actual empirical stretching observed. The difference is interesting.

This could be explained by the network zeroing out its residual branches. Under 0 residual this leads to B(x)=cosh(λi)xB(x) = cosh (\lambda_i)x. With 3 residual blocks, we have λi=0.5/3\lambda_i=0.5/3 and $g_\theta(x) = cosh(0.5/3)^3 Q x = 1.042 Q x $ with Q some orthogonal matrix. This matches the empirical observation. If this is the case then the role in the adapter on the quality of the training can only be explained by the scaling and the postprocessing operation N, as the adapter is providing almost no contribution to the transformation. The paper also mentions that during the adapter training, it tends to saturate early, which can be due to this phenomena. This can explain the stability observed as a behavior equivalent to scaling the learning rate.

I think that is crucial in order to justify the construction that the adapter network is contributing significantly to the final transformation hθh_\theta, otherwise the procedure does not hold against simpler alternatives.

Questions for authors

In order to change my assessment I think that there should be clear evidence that the adapter network is not zeroing out its residuals and that it's preferable to a simpler transformation. There are various relevant questions in order to show this

  • How does the gθg_\theta transformation differ from the sQsQ transformation (with ss the associated scaling for the network)? Are they typically close to each other under MSE?
  • How do the metrics in Table 1 perform when using the sQsQ transformation (still composing the N transformation, as gθg_\theta) instead of gθg_\theta ? It should be the case that gθg_\theta performs significantly better.
  • The method also composes with operation N, so it can also be the case that most gains are due to N and not gθg_\theta. I think it is important to ablate that. How does Table 1 perform under gθ=Ig_\theta=I and thus hθ=Nh_\theta=N
  • How does the CKA loss compare under transformations I,sQI,sQ and gθg_\theta?
Reviewer_rMEQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces an invertible bi-Lipschitz adapter that changes the coordinates of a frozen autoencoder’s latent space. The adapter is trained offline to align latent similarity structure with a semantic teacher using linear CKA, then frozen during flow-matching training. A residual construction provides explicit distortion bounds, and a sampled triplet criterion selects the distortion budget. The main generator pushes the entire reference path, including its noise source, through the adapted coordinates. Evaluation on MIMIC-CXR and CT-RATE considers generation quality, seed variability, train-on-synthetic utility, and counterfactual editing.

Critical strengths
  1. A principled and economical approach to latent adaptation. The reversible adapter preserves compatibility with the frozen decoder and retains the original reconstruction mapping under exact inversion. The block bounds and bijectivity argument are justified, and the paper correctly identifies the triplet budget condition as necessary rather than sufficient. This makes the method technically grounded while avoiding full autoencoder retraining.

  2. Useful generation-quality and stability evidence. The main comparison includes three flow-matching seeds and several adaptation baselines. CT-FID improves from 20.23 to 15.46, while the MIMIC-CXR domain-FID standard deviation decreases from 7.50 to 0.69 relative to no adaptation. These results support the usefulness of a small adapter, even though it does not outperform every alternative on every metric.

Critical weaknesses
  1. The tuning-free claim is not adequately supported in the main configuration. Table 5 varies the distortion budget using pool 8 and only one flow-matching/generation seed, whereas the principal results use pool 0 and multiple seeds. Macro F1 ranges from 0.123 to 0.189 in the sweep. Comparing this variation with seed variability under a different configuration does not establish robustness of the proposed default. This matters because selecting a reliable budget without generator tuning is one of the paper’s stated contributions.

  2. The empirical advantage is not fully isolated from the changed training geometry and evaluation protocol. Pushforward and endpoint interpolation differ in their source distributions, intermediate states, and velocity targets; their comparison does not isolate semantic alignment alone. A consistently preprocessed identity or simple constrained-coordinate control would help identify the adapter’s added value. In addition, patient/volume disjointness is not explicit, and Table 1 uses training-reference FID for MIMIC-CXR but test-reference FID for CT-RATE. These details affect how confidently the improvements can be interpreted as generalization rather than favorable measurement or conditioning choices.

Questions for authors
  1. Does the budget-robustness conclusion hold for a small set of budgets across seeds under the default pool-0 configuration, or should the tuning-free claim be restricted?
  2. How are noise and fixed normalization handled in the unadapted baseline? Is there an identity or simple constrained adapter control using the same path and preprocessing?
  3. Are the train, validation, and test sets disjoint by patient or CT volume, and what motivates the different FID reference splits?
Reviewer_bspTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper trains a small invertible, bi-Lipschitz adapter to align frozen autoencoder latents with a semantic teacher using CKA. It then freezes the adapter, pushes the entire flow-matching path through it, and inverts generated latents before decoding. Experiments on MIMIC-CXR and CT-RATE assess generation, train-on-synthetic classification, counterfactual editing, and sensitivity to the teacher, probability path, adapter family, and distortion budget.

Critical strengths

S1. Coherent integration with a practical purpose. The adapter preserves the frozen encoder-decoder reconstruction map under exact inversion while controlling geometric distortion. Applying the same transformation to the source, target, and intermediate path gives a consistent Jacobian-vector-product target. Established invertible-network and CKA ingredients [R1-R3] are combined into a reusable adaptation stage; a new theorem is not necessary for this to be useful. Table 8 reports a 0.69 ms inverse and a 4.3 × 10⁻⁶ round-trip error.

S2. The empirical support is substantial and relevant. Table 1 reports CT-FID improving from 20.23 to 15.46 and DINOv2 FID from 186.9 to 111.9; MIMIC-CXR seed variability also decreases. Three generator seeds, independent evaluation features, TSTR, and teacher/path/family controls support practical benefit. The CT teacher comparison supplies partial evidence that semantic information matters. The paper also acknowledges that stronger counterfactual edits increase off-target changes rather than claiming uniformly better controllability.

[R1] Behrmann et al. (2019). Invertible Residual Networks. [R2] Wang, Dvijotham, and Manchester (2024). Monotone, Bi-Lipschitz, and Polyak-Lojasiewicz Networks. [R3] Kornblith et al. (2019). Similarity of Neural Network Representations Revisited.

Critical weaknesses

W1. The central attribution remains insufficiently resolved. Prior source/target reparameterization [R4] makes the offline semantic adaptation, rather than coordinate changes alone, the distinctive contribution. Equation (7) changes the source, path, and velocity target together, but Sec. 5.1 does not establish whether no adaptation is the identity-map instance with the same source/normalization convention. Teacher comparisons are useful, yet do not replace a matched initialization or shuffled-teacher control or report actual before/after alignment. Consequently, the strong aggregate results may include generic conditioning effects; the current text does not quantify the additional benefit attributable to semantic alignment. This is an unresolved comparison, not a demonstrated implementation error.

[R4] Chen et al. (NeurIPS 2025). CAR-Flow: Condition-Aware Reparameterization Aligns Source and Target for Better Flow Matching.

W2. The evidence for tuning freedom and comparative stability is narrower than the claims. The triplet condition is correctly described as necessary, not sufficient, and Table 5 evaluates one generator seed with pool 8, whereas the main method uses pool 0. It therefore does not establish robustness of the default configuration. Moreover, the PDF omits the generator architecture, optimization settings, and sampling solver/NFE. These details are consequential for the short 10/30-epoch runs and unstable adaptation baselines. The promised supplementary code was unavailable in the supplied attachments, so these concerns may be partly resolvable by identifying existing configurations.

Questions for authors

Q1. Is no adaptation exactly the identity-map instance of Eq. (7), including source preprocessing? Under matched conventions, what alignment and generation improvement remains over an initialized or shuffled-teacher adapter?

Q2. What evidence supports the default pool-0 distortion budget and its stability across generator seeds? Can the tuning-free claim be limited to the configurations actually tested?

Q3. Please identify the generator architecture, optimization and sampling settings, baseline adaptation recipes, and patient/volume split protocol in the supplied code or manuscript. Were the compared methods evaluated at the same generator-training and sampling budgets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.