- Rating
- 4(+2)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes a framework for any-to-any multimodal generation. A single conditional flow model learns the distribution of a full-observation latent, given any observed subset of modalities. With all modalities observed, this is deterministic encoding. With none observed, it's unconditional generation. Anything in between is conditional latent inference. This extends UNITE's Generative Encoder from two endpoints to every subset. Each modality has its own generative decoder, and all decoders read from one shared latent sample.
Critical strengths
S1: Careful experimental setup. The CFM/DFM baselines are parameter-matched and get twice the training epochs. S2: Large coherence gains, with a check against mode collapse
Critical weaknesses
W1: It's unclear which part of the method produces the gain over MUNI. Many components are inherited: factorized decoders, target-detached reconstruction, coherence sufficiency and the benchmarks come from MUNI, the generative encoder from UNITE, clean-latent prediction from JiT. The ablations show the one-to-many advantage over MUNI holds even without target detaching, the contrastive loss or partial-data distillation. So the gain seems to come from replacing a Gaussian product-of-experts posterior with a conditional flow over the full-observation latent.
W2: No measure of variance. No seeds, confidence intervals or standard errors are reported. Several margins are small, e.g., A→(T,I) CLIP 26.334 vs. 25.850, and FFHQ64 unconditional age coherence 0.5692 vs. 0.5630.
W3: Missing discussion of diffusion bridge models: The paper doesn't discuss diffusion bridge models, which are the main alternative way to transport between distributions. Most importantly, LDDBM also translates between modalities in a shared latent space. SDB is also relevant. It adds structural constraints to latent bridges so they can learn from semi-paired or unpaired data. MUNITE's self-distillation on partially observed tuples tackles a similar problem.
LDDBM: Berman et al. Towards general modality translation with contrastive and predictive latent diffusion bridge SDB: Kosman et al. Structured Diffusion Bridges: Inductive Bias for Denoising Diffusion Bridges
Questions for authors
- Can you report results over multiple seeds, plus inference wall-clock time against MUNI?
- Can you extend the ablation to help identify which component gives the gain?
4 comments
Response to Reviewer tZor (2/2)
[W2, Q1] Variance and inference time
As is common practice in this line of work (CoDi, OmniFlow, NExT-GPT, FlowBind and MUNI), we reported a single run per method in the paper. Following the reviewer's suggestion, Table R1 reports three training runs for every method we train on image-text-audio.
Table R1. Image-text-audio coherence (paper Table 2), mean std over three training runs.
\begin{array}{l|ccc|ccc} \hline &&1\to N\ \tx{coherence}&&&\tx{Unconditional}&\nl \tx{Method}&T\to(I,A)&I\to(T,A)&A\to(T,I)&(T,I)&(T,A)&(I,A)\nl &\tx{AIS}&\tx{CLAP}&\tx{CLIP}&\tx{CLIP}&\tx{CLAP}&\tx{AIS}\nl \hline \tx{CoDi}&64.12\pm0.60&7.73\pm0.20&23.57\pm0.08&\tx{-}&\tx{-}&\tx{-}\nl \tx{OmniFlow}&77.31\pm0.74&15.38\pm0.65&22.67\pm0.11&21.11\pm0.08&14.20\pm0.24&50.89\pm1.16\nl \hline \tx{CFM}&74.08\pm0.54&21.65\pm0.09&24.06\pm0.07&23.66\pm0.06&20.40\pm0.47&74.53\pm0.44\nl \tx{DFM}&73.31\pm0.63&20.85\pm0.19&24.16\pm0.05&23.79\pm0.05&19.60\pm0.26&72.10\pm1.00\nl \tx{FlowBind}&79.61\pm0.29&26.30\pm0.38&25.13\pm0.22&\tx{-}&\tx{-}&\tx{-}\nl \tx{MUNI}&81.12\pm0.32&32.06\pm0.41&25.82\pm0.07&26.76\pm0.05&27.68\pm0.15&82.46\pm0.31\nl \tx{MUNITE}&\bb{84.31\pm0.92}&\bb{34.76\pm0.15}&\bb{26.41\pm0.04}&\bb{27.33\pm0.10}&\bb{30.31\pm0.55}&\bb{85.97\pm0.51}\nl \hline \end{array}$$ CoDi and OmniFlow show mean $\pm$ std over three evaluation seeds of their released checkpoints. **In every run, MUNITE is higher than MUNI on all six metrics of Table 2, and each margin is several times the run-to-run standard deviation, including $A\to(T,I)$ CLIP at 26.41 $\pm$ 0.04 versus 25.82 $\pm$ 0.07.** On FFHQ64, over three evaluation seeds of the submitted checkpoints, age and gender coherence are on par with MUNI (age 0.568 $\pm$ 0.005 vs. 0.564 $\pm$ 0.007), and the clear gains are in FID (29.8 $\pm$ 0.3 vs. 34.8 $\pm$ 0.5) and RGB-segmentation coherence (0.890 vs. 0.775). We have revised Sec. 1 and 4.3 accordingly. **MUNITE takes $0.98\times$ to $1.38\times$ MUNI's end-to-end inference time per sample (Table R5).** Its latent stage, 50 Euler steps on the pixel benchmarks and 25 on image-text-audio, takes 1.5-6.4 ms per sample against about 0.01 ms for MUNI's posterior sampling, but decoding dominates the total. We have added these results to App. D.3. **Table R5.** End-to-end inference time per sample in milliseconds. $$\def\tx#1{\text{#1}}\def\nl{\newline}\def\ii{\phantom{00}} \begin{array}{l|cccc|c} \hline \tx{Route}&\tx{MUNI}&\tx{CFM}&\tx{DFM}&\tx{MUNITE}&\tx{Latent stage}\nl \hline \textit{PolyMNIST-D-Q}\nl \ii d\to m_0&37.6&41.9&67.1&47.2&2.7\nl \ii d\to(m_0,m_1,m_2)&113.0&41.9&67.2&136.2&2.8\nl \ii\tx{Unconditional}&113.4&41.9&67.6&136.1&2.8\nl \hline \textit{FFHQ64}\nl \ii\tx{(Age, Gender)}\to\tx{(RGB, Seg, Norm)}&419.5&143.9&240.8&415.6&6.4\nl \ii\tx{Unconditional}&422.7&144.0&244.8&415.1&6.1\nl \hline \textit{Image-text-audio}\nl \ii(I,A)\to T&11.0&80.8&70.5&15.2&1.6\nl \ii T\to(I,A)&534.5&560.6&561.6&536.0&1.5\nl \ii\tx{Unconditional}&547.4&628.7&651.9&555.5&1.5\nl \hline \end{array}$$ ## [W3] Relation to diffusion bridges We thank the reviewer for pointing us to these works. **We note that diffusion bridges address a narrower setting than MUNITE.** A bridge transports one source distribution to one target distribution, and LDDBM and SDB learn such a bridge between two modalities in a latent space. Covering n modalities therefore takes a bridge per source and target set, and unconditional generation needs a separate model. Running one bridge per target would also sample the missing variation independently for each target, which is the failure the paper analyzes. **MUNITE instead samples one latent shared by all targets from its conditional distribution, so a single network covers every subset of observed modalities, including none, and the missing variation shared across targets is sampled once.** SDB and MUNITE both learn from incomplete data, but in different settings. SDB learns bridges between two modalities from semi-paired or unpaired data, whereas self-distillation lets tuples with missing modalities train the conditional distribution of the full-observation latent for every observed subset. We have added LDDBM and SDB to the related work in Sec. 3, together with this comparison.Follow-up on Our Response
Dear Reviewer tZor,
We appreciate the time and effort you have put into reviewing our paper. Since the discussion period ends on November 18, we wanted to check whether our response has resolved your concerns. If anything still needs clarification, please let us know and we will respond promptly.
Thank you again for your time.
Thanks for clarifying
I really appreciate the thorough discussion you further provided. I will thus consider changing the score accordingly.
Response to Reviewer tZor (1/2)
Dear Reviewer tZor,
We thank the reviewer for recognizing the "careful experimental setup" and the "large coherence gains, with a check against mode collapse." We have extended the ablations and the variance analysis as asked.
[W1, Q2] Which component produces the gain
First, we would like to emphasize how MUNITE differs from the methods it builds on. UNITE unifies the latent encoder and the generator for single-image generation, where the input is either a complete image or none. MUNITE is, to our knowledge, the first to account for partial input observations in a generative encoder, which arise naturally in the multimodal setting. Depending on which modalities are observed, its single inference model becomes a deterministic encoder or models the conditional or marginal distribution of the full-observation latent. This inference model is the key contribution of the paper, and it resolves a structural limitation of MUNI, whose latent inferred from an observed subset does not guarantee coherence among the generated targets (App. A.3 and Sec. 4). To train this model also on partially observed examples, which provide no full-observation latent as a target, we introduce partial-data self-distillation, and Proposition 1 shows that, under an exact teacher, it gives the same expected gradient as denoising against that latent (Sec. 2.3, App. B).
As suggested, we extended the ablation of Table 14 to PolyMNIST-D-Q and FFHQ64 to show the role of target detaching, the contrastive loss and self-distillation (Table R4).
Table R4. Component ablations on PolyMNIST-D-Q and FFHQ64 (extends paper Table 14).
\begin{array}{ll|cccc|c} \hline \tx{Input}\to\tx{output}&\tx{Metric}&\tx{MUNITE}&\tx{No target}&\tx{No}&\tx{Full-only}&\tx{MUNI}\nl &&&\tx{detach}&\tx{contrastive}&\tx{distill.}&\nl \hline \textit{PolyMNIST-D-Q}\nl \ii\tx{Unconditional}&\tx{FD}\dn&1.52&\bb{0.91}&3.71&0.94&3.04\nl &\tx{Digit coh.}\up&0.983&\bb{0.985}&0.984&0.984&0.580\nl &\tx{Quadrant coh.}\up&\bb{1.000}&\bb{1.000}&\bb{1.000}&\bb{1.000}&0.904\nl \hline \ii d\to m_i&\tx{digit acc.}\up&0.994&0.994&\bb{0.995}&\bb{0.995}&0.984\nl \hline \ii(d,q)\to m_i&\tx{both acc.}\up&0.994&\bb{0.995}&\bb{0.995}&0.994&0.993\nl \hline \ii d\to(m_0,m_1,m_2)&\tx{quadrant coh.}\up&\bb{1.000}&\bb{1.000}&\bb{1.000}&\bb{1.000}&0.253\nl \hline \ii q\to(m_0,m_1,m_2)&\tx{digit coh.}\up&\bb{0.990}&0.988&0.989&0.988&0.107\nl \hline \textit{FFHQ64}\nl \ii\tx{Unconditional}&\tx{RGB FID}\dn&\bb{29.92}&34.18&46.02&39.98&34.62\nl &\tx{Age coh.}\up&0.569&0.568&0.527&\bb{0.580}&0.563\nl &\tx{Gender coh.}\up&0.961&\bb{0.963}&0.957&0.955&0.960\nl &\tx{RGB-Seg coh.}\up&\bb{0.890}&0.875&0.870&0.850&0.776\nl &\tx{RGB-Normal err.}\dn&\bb{0.049}&0.058&0.055&0.063&0.070\nl \hline \ii\tx{Age}\to\tx{RGB}&\tx{age acc.}\up&\bb{0.627}&0.616&0.535&0.622&0.520\nl \hline \ii\tx{Gender}\to\tx{RGB}&\tx{gender acc.}\up&\bb{0.979}&0.976&0.960&0.978&0.968\nl \hline \ii\tx{(Age, Gender)}\to\tx{RGB}&\tx{both acc.}\up&0.590&\bb{0.615}&0.500&0.598&0.510\nl \hline \ii\tx{(Age, Gender)}\to&\tx{RGB-Seg coh.}\up&\bb{0.889}&0.879&0.866&0.853&0.604\nl \ii\phantom{0}\tx{(RGB, Seg, Norm)}&\tx{RGB-Normal err.}\dn&\bb{0.050}&0.058&0.057&0.063&0.111\nl \hline \end{array}$$ **Every ablation keeps the one-to-many coherence on both benchmarks, with digit coherence under quadrant conditioning at 0.988-0.990 on PolyMNIST-D-Q (MUNI 0.107) and RGB-segmentation coherence given age and gender at 0.853-0.889 on FFHQ64 (MUNI 0.604).** The contrastive loss mainly improves fidelity (FD 3.71 and FID 46.0 without it, against 1.52 and 29.9). On PolyMNIST-D-Q, where coherence is near saturation for every variant, partial-data distillation has little room to help, while on FFHQ64 it improves both fidelity and coherence, as restricting distillation to fully observed examples raises FID to 40.0 and lowers RGB-segmentation coherence from 0.890 to 0.850. We have added these ablations to App. E and summarize them in Sec. 4.4.