- Rating
- 1(−1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In order to train an image flow matching model in latent space, it is common to use a pretrained VAE trained on a large scale image dataset. However, the latent space might not be well-aligned with the target image distribution as it can be significantly different from the distribution used to train the VAE.
The paper proposes to use a simple learned adapter to modify the behavior of pretrained VAE. It is then composed with the encoder and decoder networks by and respectively.
This adapter is designed to be an invertible bi-Lipschitz function and trained to minimize the Centered Kernel Alignment with respect to a reference embedding model, the teacher model . With sampled from the target image distribution and a pooling operation. Once the adapter is trained they concatenate it with a postprocessing operation and learn the FM model by transporting the full path instead of doing standard FM in the new latent space.
Trained with MIMIC-CXR and CT-RATE datasets, the paper argues that it is competitive with other latent adapting methods while being typically more stable
Critical strengths
Paper is well presented, with a clear motivation and a clean construction. The need for bi-Lipschitz functions instead of just invertible ones is justified cleanly. The result itself that shows that invertible functions are not enough is very interesting on itself. The proposed adapter strategy through bi-Lipschitz functions is flexible and allows further exploration such as through different losses beyond CKA. The adapter is also lightweight in parameters, making it efficient compared to other adapter methods
Critical weaknesses
I have significant concerns about the adapter network behavior after training. There are reasons to suspect that the network is zeroing-out all residual connections and thus behaving essentially as a simple orthogonal + scaling operation . If this is the case, then it undermines the complete construction as the adapter is not learned from the data and barely contributes to the encoding. I'll explain why I think this is the case.
Table 7 shows the certified stretch/compression bounds and compares them with maximum stretching obtained by sampling the Jacobian in different points. It shows a certificate vs for the actual empirical stretching observed. The difference is interesting.
This could be explained by the network zeroing out its residual branches. Under 0 residual this leads to . With 3 residual blocks, we have and $g_\theta(x) = cosh(0.5/3)^3 Q x = 1.042 Q x $ with Q some orthogonal matrix. This matches the empirical observation. If this is the case then the role in the adapter on the quality of the training can only be explained by the scaling and the postprocessing operation N, as the adapter is providing almost no contribution to the transformation. The paper also mentions that during the adapter training, it tends to saturate early, which can be due to this phenomena. This can explain the stability observed as a behavior equivalent to scaling the learning rate.
I think that is crucial in order to justify the construction that the adapter network is contributing significantly to the final transformation , otherwise the procedure does not hold against simpler alternatives.
Questions for authors
In order to change my assessment I think that there should be clear evidence that the adapter network is not zeroing out its residuals and that it's preferable to a simpler transformation. There are various relevant questions in order to show this
- How does the transformation differ from the transformation (with the associated scaling for the network)? Are they typically close to each other under MSE?
- How do the metrics in Table 1 perform when using the transformation (still composing the N transformation, as ) instead of ? It should be the case that performs significantly better.
- The method also composes with operation N, so it can also be the case that most gains are due to N and not . I think it is important to ablate that. How does Table 1 perform under and thus
- How does the CKA loss compare under transformations and ?