acceptodds
Under review as a conference paper at ICLR 2027

MUNITE: Unified Multimodal Latent Inference for Any-to-Any Multimodal Generation

Abstract

We introduce MUNITE, a latent-variable framework for flexible any-to-any multimodal generation that treats encoding and latent generation as the same inference problem under different amounts of observed evidence. Given any subset of modalities, MUNITE models the conditional distribution over the latent representation associated with the complete observation. Full observation recovers deterministic encoding, no observation recovers the latent marginal, and intermediate subsets define conditional latent inference, all within a single conditional flow model. A shared latent sample captures variation that must remain consistent across generated targets, while modality-specific generative decoders model the remaining uncertainty independently. To learn these conditional distributions from incomplete training examples, we extend conditional flow matching through self-distillation: predictions conditioned on richer available observations supervise the same model conditioned on smaller subsets at the same intermediate latent state. When the richer-evidence trajectory follows the exact conditional flow, this provides the same expected learning signal as full-target denoising. Across PolyMNIST-D-Q, FFHQ64, and image–text–audio, MUNITE achieves competitive or better generation quality and source–target alignment, with higher joint-generation coherence. In particular, it attains the highest coherence in all one-to-many and unconditional image–text–audio comparisons, showing the effectiveness of unified latent inference across diverse multimodal settings.

Primary area:
generative models
Keywords:
Generative Models, Diffusion Models, Multimodal Generation, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_tZorOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a framework for any-to-any multimodal generation. A single conditional flow model learns the distribution of a full-observation latent, given any observed subset of modalities. With all modalities observed, this is deterministic encoding. With none observed, it's unconditional generation. Anything in between is conditional latent inference. This extends UNITE's Generative Encoder from two endpoints to every subset. Each modality has its own generative decoder, and all decoders read from one shared latent sample.

Critical strengths

S1: Careful experimental setup. The CFM/DFM baselines are parameter-matched and get twice the training epochs. S2: Large coherence gains, with a check against mode collapse

Critical weaknesses

W1: It's unclear which part of the method produces the gain over MUNI. Many components are inherited: factorized decoders, target-detached reconstruction, coherence sufficiency and the benchmarks come from MUNI, the generative encoder from UNITE, clean-latent prediction from JiT. The ablations show the one-to-many advantage over MUNI holds even without target detaching, the contrastive loss or partial-data distillation. So the gain seems to come from replacing a Gaussian product-of-experts posterior with a conditional flow over the full-observation latent.

W2: No measure of variance. No seeds, confidence intervals or standard errors are reported. Several margins are small, e.g., A→(T,I) CLIP 26.334 vs. 25.850, and FFHQ64 unconditional age coherence 0.5692 vs. 0.5630.

W3: Missing discussion of diffusion bridge models: The paper doesn't discuss diffusion bridge models, which are the main alternative way to transport between distributions. Most importantly, LDDBM also translates between modalities in a shared latent space. SDB is also relevant. It adds structural constraints to latent bridges so they can learn from semi-paired or unpaired data. MUNITE's self-distillation on partially observed tuples tackles a similar problem.

LDDBM: Berman et al. Towards general modality translation with contrastive and predictive latent diffusion bridge SDB: Kosman et al. Structured Diffusion Bridges: Inductive Bias for Denoising Diffusion Bridges

Questions for authors
  1. Can you report results over multiple seeds, plus inference wall-clock time against MUNI?
  2. Can you extend the ablation to help identify which component gives the gain?
4 comments
AuthorsOpenReview ↗

Response to Reviewer tZor (1/2)

Dear Reviewer tZor,

We thank the reviewer for recognizing the "careful experimental setup" and the "large coherence gains, with a check against mode collapse." We have extended the ablations and the variance analysis as asked.

[W1, Q2] Which component produces the gain

First, we would like to emphasize how MUNITE differs from the methods it builds on. UNITE unifies the latent encoder and the generator for single-image generation, where the input is either a complete image or none. MUNITE is, to our knowledge, the first to account for partial input observations in a generative encoder, which arise naturally in the multimodal setting. Depending on which modalities are observed, its single inference model becomes a deterministic encoder or models the conditional or marginal distribution of the full-observation latent. This inference model is the key contribution of the paper, and it resolves a structural limitation of MUNI, whose latent inferred from an observed subset does not guarantee coherence among the generated targets (App. A.3 and Sec. 4). To train this model also on partially observed examples, which provide no full-observation latent as a target, we introduce partial-data self-distillation, and Proposition 1 shows that, under an exact teacher, it gives the same expected gradient as denoising against that latent (Sec. 2.3, App. B).

As suggested, we extended the ablation of Table 14 to PolyMNIST-D-Q and FFHQ64 to show the role of target detaching, the contrastive loss and self-distillation (Table R4).

Table R4. Component ablations on PolyMNIST-D-Q and FFHQ64 (extends paper Table 14).

\begin{array}{ll|cccc|c} \hline \tx{Input}\to\tx{output}&\tx{Metric}&\tx{MUNITE}&\tx{No target}&\tx{No}&\tx{Full-only}&\tx{MUNI}\nl &&&\tx{detach}&\tx{contrastive}&\tx{distill.}&\nl \hline \textit{PolyMNIST-D-Q}\nl \ii\tx{Unconditional}&\tx{FD}\dn&1.52&\bb{0.91}&3.71&0.94&3.04\nl &\tx{Digit coh.}\up&0.983&\bb{0.985}&0.984&0.984&0.580\nl &\tx{Quadrant coh.}\up&\bb{1.000}&\bb{1.000}&\bb{1.000}&\bb{1.000}&0.904\nl \hline \ii d\to m_i&\tx{digit acc.}\up&0.994&0.994&\bb{0.995}&\bb{0.995}&0.984\nl \hline \ii(d,q)\to m_i&\tx{both acc.}\up&0.994&\bb{0.995}&\bb{0.995}&0.994&0.993\nl \hline \ii d\to(m_0,m_1,m_2)&\tx{quadrant coh.}\up&\bb{1.000}&\bb{1.000}&\bb{1.000}&\bb{1.000}&0.253\nl \hline \ii q\to(m_0,m_1,m_2)&\tx{digit coh.}\up&\bb{0.990}&0.988&0.989&0.988&0.107\nl \hline \textit{FFHQ64}\nl \ii\tx{Unconditional}&\tx{RGB FID}\dn&\bb{29.92}&34.18&46.02&39.98&34.62\nl &\tx{Age coh.}\up&0.569&0.568&0.527&\bb{0.580}&0.563\nl &\tx{Gender coh.}\up&0.961&\bb{0.963}&0.957&0.955&0.960\nl &\tx{RGB-Seg coh.}\up&\bb{0.890}&0.875&0.870&0.850&0.776\nl &\tx{RGB-Normal err.}\dn&\bb{0.049}&0.058&0.055&0.063&0.070\nl \hline \ii\tx{Age}\to\tx{RGB}&\tx{age acc.}\up&\bb{0.627}&0.616&0.535&0.622&0.520\nl \hline \ii\tx{Gender}\to\tx{RGB}&\tx{gender acc.}\up&\bb{0.979}&0.976&0.960&0.978&0.968\nl \hline \ii\tx{(Age, Gender)}\to\tx{RGB}&\tx{both acc.}\up&0.590&\bb{0.615}&0.500&0.598&0.510\nl \hline \ii\tx{(Age, Gender)}\to&\tx{RGB-Seg coh.}\up&\bb{0.889}&0.879&0.866&0.853&0.604\nl \ii\phantom{0}\tx{(RGB, Seg, Norm)}&\tx{RGB-Normal err.}\dn&\bb{0.050}&0.058&0.057&0.063&0.111\nl \hline \end{array}$$ **Every ablation keeps the one-to-many coherence on both benchmarks, with digit coherence under quadrant conditioning at 0.988-0.990 on PolyMNIST-D-Q (MUNI 0.107) and RGB-segmentation coherence given age and gender at 0.853-0.889 on FFHQ64 (MUNI 0.604).** The contrastive loss mainly improves fidelity (FD 3.71 and FID 46.0 without it, against 1.52 and 29.9). On PolyMNIST-D-Q, where coherence is near saturation for every variant, partial-data distillation has little room to help, while on FFHQ64 it improves both fidelity and coherence, as restricting distillation to fully observed examples raises FID to 40.0 and lowers RGB-segmentation coherence from 0.890 to 0.850. We have added these ablations to App. E and summarize them in Sec. 4.4.
AuthorsOpenReview ↗

Response to Reviewer tZor (2/2)

[W2, Q1] Variance and inference time

As is common practice in this line of work (CoDi, OmniFlow, NExT-GPT, FlowBind and MUNI), we reported a single run per method in the paper. Following the reviewer's suggestion, Table R1 reports three training runs for every method we train on image-text-audio.

Table R1. Image-text-audio coherence (paper Table 2), mean ±\pm std over three training runs.

\begin{array}{l|ccc|ccc} \hline &&1\to N\ \tx{coherence}&&&\tx{Unconditional}&\nl \tx{Method}&T\to(I,A)&I\to(T,A)&A\to(T,I)&(T,I)&(T,A)&(I,A)\nl &\tx{AIS}&\tx{CLAP}&\tx{CLIP}&\tx{CLIP}&\tx{CLAP}&\tx{AIS}\nl \hline \tx{CoDi}&64.12\pm0.60&7.73\pm0.20&23.57\pm0.08&\tx{-}&\tx{-}&\tx{-}\nl \tx{OmniFlow}&77.31\pm0.74&15.38\pm0.65&22.67\pm0.11&21.11\pm0.08&14.20\pm0.24&50.89\pm1.16\nl \hline \tx{CFM}&74.08\pm0.54&21.65\pm0.09&24.06\pm0.07&23.66\pm0.06&20.40\pm0.47&74.53\pm0.44\nl \tx{DFM}&73.31\pm0.63&20.85\pm0.19&24.16\pm0.05&23.79\pm0.05&19.60\pm0.26&72.10\pm1.00\nl \tx{FlowBind}&79.61\pm0.29&26.30\pm0.38&25.13\pm0.22&\tx{-}&\tx{-}&\tx{-}\nl \tx{MUNI}&81.12\pm0.32&32.06\pm0.41&25.82\pm0.07&26.76\pm0.05&27.68\pm0.15&82.46\pm0.31\nl \tx{MUNITE}&\bb{84.31\pm0.92}&\bb{34.76\pm0.15}&\bb{26.41\pm0.04}&\bb{27.33\pm0.10}&\bb{30.31\pm0.55}&\bb{85.97\pm0.51}\nl \hline \end{array}$$ CoDi and OmniFlow show mean $\pm$ std over three evaluation seeds of their released checkpoints. **In every run, MUNITE is higher than MUNI on all six metrics of Table 2, and each margin is several times the run-to-run standard deviation, including $A\to(T,I)$ CLIP at 26.41 $\pm$ 0.04 versus 25.82 $\pm$ 0.07.** On FFHQ64, over three evaluation seeds of the submitted checkpoints, age and gender coherence are on par with MUNI (age 0.568 $\pm$ 0.005 vs. 0.564 $\pm$ 0.007), and the clear gains are in FID (29.8 $\pm$ 0.3 vs. 34.8 $\pm$ 0.5) and RGB-segmentation coherence (0.890 vs. 0.775). We have revised Sec. 1 and 4.3 accordingly. **MUNITE takes $0.98\times$ to $1.38\times$ MUNI's end-to-end inference time per sample (Table R5).** Its latent stage, 50 Euler steps on the pixel benchmarks and 25 on image-text-audio, takes 1.5-6.4 ms per sample against about 0.01 ms for MUNI's posterior sampling, but decoding dominates the total. We have added these results to App. D.3. **Table R5.** End-to-end inference time per sample in milliseconds. $$\def\tx#1{\text{#1}}\def\nl{\newline}\def\ii{\phantom{00}} \begin{array}{l|cccc|c} \hline \tx{Route}&\tx{MUNI}&\tx{CFM}&\tx{DFM}&\tx{MUNITE}&\tx{Latent stage}\nl \hline \textit{PolyMNIST-D-Q}\nl \ii d\to m_0&37.6&41.9&67.1&47.2&2.7\nl \ii d\to(m_0,m_1,m_2)&113.0&41.9&67.2&136.2&2.8\nl \ii\tx{Unconditional}&113.4&41.9&67.6&136.1&2.8\nl \hline \textit{FFHQ64}\nl \ii\tx{(Age, Gender)}\to\tx{(RGB, Seg, Norm)}&419.5&143.9&240.8&415.6&6.4\nl \ii\tx{Unconditional}&422.7&144.0&244.8&415.1&6.1\nl \hline \textit{Image-text-audio}\nl \ii(I,A)\to T&11.0&80.8&70.5&15.2&1.6\nl \ii T\to(I,A)&534.5&560.6&561.6&536.0&1.5\nl \ii\tx{Unconditional}&547.4&628.7&651.9&555.5&1.5\nl \hline \end{array}$$ ## [W3] Relation to diffusion bridges We thank the reviewer for pointing us to these works. **We note that diffusion bridges address a narrower setting than MUNITE.** A bridge transports one source distribution to one target distribution, and LDDBM and SDB learn such a bridge between two modalities in a latent space. Covering n modalities therefore takes a bridge per source and target set, and unconditional generation needs a separate model. Running one bridge per target would also sample the missing variation independently for each target, which is the failure the paper analyzes. **MUNITE instead samples one latent shared by all targets from its conditional distribution, so a single network covers every subset of observed modalities, including none, and the missing variation shared across targets is sampled once.** SDB and MUNITE both learn from incomplete data, but in different settings. SDB learns bridges between two modalities from semi-paired or unpaired data, whereas self-distillation lets tuples with missing modalities train the conditional distribution of the full-observation latent for every observed subset. We have added LDDBM and SDB to the related work in Sec. 3, together with this comparison.
AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer tZor,

We appreciate the time and effort you have put into reviewing our paper. Since the discussion period ends on November 18, we wanted to check whether our response has resolved your concerns. If anything still needs clarification, please let us know and we will respond promptly.

Thank you again for your time.

Reviewer_tZorOpenReview ↗

Thanks for clarifying

I really appreciate the thorough discussion you further provided. I will thus consider changing the score accordingly.

Reviewer_8rLYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

MUNITE unifies full-observation encoding, partial-observation latent inference, and unconditional generation within one conditional flow model, and then generates the requested modalities from a shared latent sample.

Critical strengths
  1. The unified formulation is conceptually clear.
  2. The PolyMNIST experiments directly test the central mechanism.
  3. The theoretical analysis identifies a meaningful distinction between individual accuracy and joint coherence.
Critical weaknesses
  1. For the presentation, there are some areas for improvement. a). L107, what does dw mean? b). The second paragraph of Sec.2.1 is not clear. Hard to understand why the conditional distribution collapses to Q_E(\cdot | X^M)=delta_w (what is delta_w? Also, it should be X^M instead of x^M, right? What is the difference between Q_E and E?) c). L143, what is the Gaussian query and t? what is the epsilon in Eq.3? Many places are not listed here. I would recommend that the author use consistent symbols and try to minimize the number of symbols. It is quite hard to understand the paper.
  2. This area has many baselines (e.g., [1,2]). How does this work perform on challenging benchmarks, e.g., LAION and ImageNet? Why consider only the low-resolution FFHQ64 instead of FFHQ256?
  3. Sec.2.7 explains how direct conditional flow avoids explicit posterior composition, but it remains unclear whether the learned latent conditionals are consistent with the Bayesian posterior implied by the learned prior and factorized decoders.
  4. Eq.2 requires the data modalities to become conditionally independent given W, but the paper does not establish that reconstruction and target detaching enforce this property. Can the authors elaborate on what prevents the learned full-observation latent from omitting dependencies shared among the missing targets?

[1] Le, Duong H., et al. "One diffusion to generate them all." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025. [2] Cui, Jiali, et al. "ShaLa: Multimodal Shared Latent Space Modelling." Proceedings of the AAAI Conference on Artificial Intelligence 2026

Questions for authors

see Weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 8rLY (1/3)

Dear Reviewer 8rLY,

We thank the reviewer for noting that "the unified formulation is conceptually clear" and that the PolyMNIST experiments "directly test the central mechanism."

[W1] Notation

(a) QE(dw)Q_{\mathcal{E}}(dw) denotes a probability measure and plays the role of q(w)dwq(w)dw for a density qq. We use it because the full-observation conditional in (b) is a point mass, which has no density.

(b)

  • δw\delta_w is the Dirac measure at ww. We define the latent of a complete observation as its deterministic encoding w=E(xM)w = \mathcal{E}(x^{\mathcal{M}}), as UNITE does for an observed image, so given xMx^{\mathcal{M}} the latent is this single value and its conditional distribution is δw\delta_w. We have added this definition of δw\delta_w where it first appears in Sec. 2.1.

  • xMx^{\mathcal{M}} refers to the observed value of the random variable XMX^{\mathcal{M}}. We use uppercase letters for random variables and lowercase letters for their values, so QE(⋅∣xM)Q_{\mathcal{E}}(\cdot \mid x^{\mathcal{M}}) is the conditional distribution given this value.

  • E\mathcal{E} denotes our latent inference model, written as the function E(xM)\mathcal{E}(x^{\mathcal{M}}) for a full observation, where it is deterministic. QEQ_{\mathcal{E}} denotes the distributions of the latent induced by this function, with QE(⋅∣xS)Q_{\mathcal{E}}(\cdot \mid x^{\mathcal{S}}) defined in Eq. 14 as the pushforward of the data conditional given xSx^{\mathcal{S}} through E\mathcal{E}.

(c) ϵ∼N(0,ID)\epsilon \sim \mathcal{N}(0, I_D) in Eq. 3 is the Gaussian query, the latent state at t=0t = 0. As in UNITE, it keeps the input format the same at every observation level, and with the full observation the trained model effectively ignores it (Sec. 2.3). t∈[0,1)t \in [0, 1) is the flow time and wtw_t the latent state at time tt. We now define both at their first use in Sec. 2.2.

We have also added a notation table at the start of App. A.

AuthorsOpenReview ↗

Response to Reviewer 8rLY (2/3)

[W2] Baselines and benchmarks

We kindly note that the two models the reviewer mentions address different settings. OneDiffusion covers visual generation and prediction, including joint image-depth generation, and does not generate the text and audio our protocols require. ShaLa learns a shared latent with a diffusion prior, but it is trained on complete modality tuples and its conditional sampler takes one observed modality, so it does not run on the incompletely observed training sets or the subset-conditioned routes of our benchmarks. Learning from incomplete data and conditioning on arbitrary subsets are what MUNITE adds.

We also evaluated additional baselines, NExT-GPT [1], AnyGPT [2] and AudioX [3] on our image-text-audio benchmark, alongside the UnifiedIO2-L [4] scores reported by MUNI (Tables R7a and R7b). When asked for two outputs at once, NExT-GPT returns both for at most a third of the inputs. On single-target routes, the larger systems lead on several metrics, such as I→TI\to T for UnifiedIO2-L and AnyGPT and T→AT\to A for AudioX, while MUNITE leads on A→TA\to T, A→IA\to I and I→AI\to A.

Table R7a. External baselines on the one-to-many routes (paper Table 2), 975 triplets.

\begin{array}{l|ccc} \hline \tx{Method}&T\to(I,A)&I\to(T,A)&A\to(T,I)\nl &\tx{AIS}&\tx{CLAP}&\tx{CLIP}\nl \hline \tx{CoDi}&64.12&7.73&23.57\nl \tx{OmniFlow}&77.31&15.38&22.67\nl \tx{NExT-GPT}&\tx{n/a}\sm{0.0\unicode{x25}}&34.64\sm{32.8\unicode{x25}}&25.62\sm{8.9\unicode{x25}}\nl \hline \tx{MUNI}&81.12&32.06&25.82\nl \tx{MUNITE}&\bb{84.31}&\bb{34.76}&\bb{26.41}\nl \hline \end{array}$$ NExT-GPT is scored only on the inputs for which it returns both requested outputs, and the parentheses give the fraction of such inputs. **Table R7b.** Single-target routes (paper Tables 11-13). $$\def\tx#1{\text{#1}}\def\nl{\newline}\def\sm#1{\ {\scriptsize(#1)}}\def\up{\uparrow}\def\dn{\downarrow} \begin{array}{ll|cccccc|cc} \hline \tx{Route}&\tx{Metric}&\tx{CoDi}&\tx{Omni-}&\tx{Unified-}&\tx{NExT-}&\tx{Any-}&\tx{AudioX}&\tx{MUNI}&\tx{MUNITE}\nl &&&\tx{Flow}&\tx{IO2-L}&\tx{GPT}&\tx{GPT}&&&\nl \hline T\to I&\tx{FID}\dn&20.79&17.63&22.03&15.94\sm{99.2\unicode{x25}}&18.08&\tx{-}&15.60&15.80\nl &\tx{CLIP}\up&29.69&30.75&29.73&28.10\sm{99.2\unicode{x25}}&27.78&\tx{-}&29.74&28.92\nl \hline I\to T&\tx{CIDEr}\up&12.12&26.72&91.36&33.59\sm{100\unicode{x25}}&43.83&\tx{-}&44.27&37.74\nl &\tx{CLIP}\up&26.06&27.49&29.97&29.71\sm{100\unicode{x25}}&29.06&\tx{-}&28.28&27.54\nl \hline T\to A&\tx{FAD}\dn&8.64&3.92&7.13&4.86\sm{95.9\unicode{x25}}&\tx{-}&2.12&3.48&3.19\nl &\tx{CLAP}\up&16.39&29.87&15.22&31.77\sm{95.9\unicode{x25}}&\tx{-}&40.66&30.77&30.92\nl \hline A\to T&\tx{CIDEr}\up&7.76&31.27&15.18&6.62\sm{100\unicode{x25}}&\tx{-}&\tx{-}&51.05&55.70\nl &\tx{CLAP}\up&31.89&43.83&35.48&28.84\sm{100\unicode{x25}}&\tx{-}&\tx{-}&40.08&40.67\nl \hline I\to A&\tx{FAD}\dn&13.99&5.00&\tx{-}&4.09\sm{94.4\unicode{x25}}&\tx{-}&6.18&2.24&2.12\nl &\tx{AIS}\up&59.31&77.08&\tx{-}&74.53\sm{94.4\unicode{x25}}&\tx{-}&82.20&90.38&90.81\nl \hline A\to I&\tx{FID}\dn&46.59&93.10&\tx{-}&46.70\sm{88.3\unicode{x25}}&\tx{-}&\tx{-}&22.72&21.23\nl &\tx{AIS}\up&84.75&64.55&\tx{-}&79.63\sm{88.3\unicode{x25}}&\tx{-}&\tx{-}&94.63&94.30\nl \hline (T,I)\to A&\tx{CLAP}\up&10.76&13.27&\tx{-}&30.82\sm{88.1\unicode{x25}}&\tx{-}&32.09&30.64&30.79\nl &\tx{AIS}\up&64.56&63.93&\tx{-}&83.96\sm{88.1\unicode{x25}}&\tx{-}&89.73&82.30&82.91\nl \hline \end{array}$$ NExT-GPT is scored only on the inputs for which it returns the requested output, and the parentheses give the fraction of such inputs. On the benchmarks, PolyMNIST-D-Q and FFHQ64 are controlled settings in which the attributes shared across modalities are annotated and verifiable, so they isolate the target coherence the paper analyzes. For larger-scale evaluation, we test the same method on the image-text-audio benchmark, with 0.72M training examples including 241K LAION-COCO pairs, and compare it with large pretrained models. We have added the discussion of OneDiffusion and ShaLa to Sec. 3 and these results to App. E. [1] S. Wu, H. Fei, L. Qu, W. Ji, and T.-S. Chua. NExT-GPT: Any-to-Any Multimodal LLM. In ICML, 2024. [2] J. Zhan, J. Dai, J. Ye, et al. AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling. In ACL, 2024. [3] Z. Tian, Z. Liu, Y. Jin, R. Yuan, L. Xue, X. Tan, Q. Chen, W. Xue, and Y. Guo. AudioX: A Unified Framework for Anything-to-Audio Generation. In ICLR, 2026. [4] J. Lu, C. Clark, S. Lee, Z. Zhang, S. Khosla, R. Marten, D. Hoiem, and A. Kembhavi. Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action. In CVPR, 2024.
AuthorsOpenReview ↗

Response to Reviewer 8rLY (3/3)

[W3] Consistency with the Bayes posterior

The learned conditionals coincide with the Bayes posterior implied by the prior and the decoders when the decoders are exact and WW satisfies Eq. 1. This posterior is

Pψ(dw∣xS)∝∏m∈Spψm(xm∣w)QE(dw),P_{\psi}(dw \mid x^{\mathcal{S}}) \propto \prod_{m \in \mathcal{S}} p_{\psi_m}(x^m \mid w) Q_{\mathcal{E}}(dw),

where pψm(xm∣w)p_{\psi_m}(x^m \mid w) is the likelihood of decoder Dψm\mathcal D_{\psi_m}, while the flow is trained toward the pushforward QE(⋅∣xS)Q_{\mathcal{E}}(\cdot \mid x^{\mathcal{S}}) of Eq. 14. Under these two conditions, the generative model gives (W,XM)(W, X^{\mathcal{M}}) the same joint distribution as the data, so the two conditionals are the same object. We have added this statement after Eq. 14.

We also test this empirically by measuring the total variation (TV) between the distribution of the unobserved attribute in generated samples and its data conditional, on PolyMNIST-D-Q in Table 10 and on FFHQ64 in Table R6, where the data conditionals are estimated from the annotations and held-out real faces give the verifier's own error.

Table R6. FFHQ64 conditional consistency.

\begin{array}{l|cccc|c} \hline \tx{Metric}&\tx{MUNI}&\tx{CFM}&\tx{DFM}&\tx{MUNITE}&\tx{Real}\nl \hline \tx{Age}\to\tx{RGB}\nl \ii\tx{TV of gender given age}\dn&0.139&0.135&0.071&\bb{0.057}&0.010\nl \ii\tx{Age accuracy}\up&0.520&0.251&0.400&\bb{0.627}&\tx{-}\nl \hline \tx{Gender}\to\tx{RGB}\nl \ii\tx{TV of age given gender}\dn&0.245&\bb{0.121}&0.210&0.151&0.065\nl \ii\tx{Gender accuracy}\up&0.968&0.694&0.852&\bb{0.979}&\tx{-}\nl \hline \tx{(Age, Gender)}\to\tx{(RGB, Seg, Norm)}\nl \ii\tx{RGB-Seg coherence, shared latent}\up&0.604&\tx{-}&\tx{-}&\bb{0.889}&0.913\nl \ii\tx{RGB-Seg coherence, independent latents}&\tx{-}&\tx{-}&\tx{-}&0.603&\tx{-}\nl \hline \end{array}$$ **On both routes MUNITE has lower missing-attribute TV than MUNI (0.057 vs. 0.139 and 0.151 vs. 0.245) together with higher accuracy on the observed condition (Table R6).** ## [W4] Conditional independence given W (Eq. 1) We thank the reviewer for raising this point. **At its optimum, the reconstruction objective with target detaching makes $W$ satisfy the conditional independence that Eq. 2 requires (Eq. 1).** With full observation, each decoder is trained on $W = \mathcal{E}(x^{\mathcal{M}})$ (Eq. 3). If $X^m$ still depended on the other modalities given $W$, adding their information to $W$ would strictly lower the optimal loss of decoder $m$ without raising the loss of any other decoder, since more information cannot increase a decoder's optimal loss. At the optimum with exact decoders, each $X^m$ must therefore be independent of all other modalities given $W$, which is equivalent to Eq. 1. The argument only adds information from the other modalities, which enters $W$ through their own streams and the latent registers. It therefore holds even when the loss of $m$ does not train the stream of $m$ at all, as under target detaching, which keeps this loss from pushing the private detail of $m$ into $W$. We have stated this result in Sec. 2.2 and added it with its proof as Proposition 2 in App. A.4. Empirically, decoding the three PolyMNIST-D-Q views from one latent sample gives near-perfect coherence, while separately drawn samples fall to the chance level (Table 10). On FFHQ64, independent latents likewise drop RGB-segmentation coherence from 0.889 to 0.603, the level of MUNI, against 0.913 for real faces (Table R6), so the coherence comes from the shared latent sample.
Reviewer_aQ3KOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MUNITE is a shared-latent framework for any-to-any multimodal generation that does not use an LLM backbone. A single conditional flow model takes any subset of observed modalities and models the distribution of the full-observation latent. With all modalities observed this reduces to encoding, with none observed it is unconditional generation, and intermediate subsets give conditional inference, extending UNITE's Generative Encoder. One shared latent sample is decoded by modality-specific generative decoders, following MUNI. To learn from partially observed data, the paper extends conditional flow matching with self-distillation, where predictions from richer observations supervise those from smaller subsets. Proposition 1 shows this matches the expected gradient of full-target denoising under an exact teacher. Training also uses target-detached reconstruction and an auxiliary contrastive loss. Experiments on PolyMNIST-D-Q, a new FFHQ64 benchmark, and a feature-space image–text–audio benchmark compare against MUNI, self-implemented CFM/DFM baselines, FlowBind, CoDi, and OmniFlow. MUNITE reports the best joint coherence, especially for one-to-many and unconditional generation, and competitive single-target quality.

Critical strengths
  1. A clean, principled unification of encoding and conditional generation. One conditional flow model covers deterministic encoding, conditional inference under any observed subset, and unconditional generation, and the self-distillation objective lets partially observed data train these conditionals, with a clear justification (tower property and Proposition 1). This matters because real multimodal corpora are mostly incomplete, and the formulation explains why sharing one sampled full-observation latent across targets is the right way to get joint coherence, not just target-wise accuracy.

  2. Convincing evidence on the one-to-many coherence gap. The PolyMNIST-D-Q analysis isolates the problem cleanly. MUNI matches the correct per-view distribution of the missing attribute but its views agree only at chance level (0.25 and 0.10), whereas MUNITE reaches 1.00 and 0.99. The TV-to-uniform check rules out mode collapse, and decoding from independent latents drops MUNITE back to chance. This controlled evidence is the strongest support for the paper's central claim and is the main reason I see value in the work.

Critical weaknesses
  1. The baseline comparisons do not convincingly support the claimed advantages.
  • CFM and DFM look like weak baselines. They are self-implemented stand-ins for OmniFlow- and NExT-OMNI-style models, and their results are unusually poor, e.g., FD 71.7 vs. 1.5 and digit coherence 0.02 on PolyMNIST-D-Q. This likely reflects that a joint pixel-space Transformer has no generative decoders or pretrained latent space, rather than a real limitation of joint-flow approaches.
  • Text decoding is not matched across methods. On image–text–audio, CFM, DFM, and FlowBind use the frozen text decoder, while MUNI and MUNITE fine-tune it with LoRA. This probably explains much of the large CIDEr gap.
  • The external baselines are not evaluated in the same setting. CoDi and OmniFlow scores are copied from prior work and come from models that generate raw modalities.
  • The MUNI result is partly expected by construction. MUNI samples a PoE posterior without a prior expert, so it can only encode source information.

The margins over the strongest baseline on image–text–audio are also small (e.g., 81.6→83.3 and 25.85→26.33), with no seeds or confidence intervals, and there is no diversity check outside PolyMNIST-D-Q to rule out coherence gains from reduced variation. Since joint coherence is the paper's main claim, these issues directly weaken the evidence for it.

  1. The real-modality evaluation is narrow, and the positioning is incomplete. The only heterogeneous-modality benchmark models fixed 768-d CLIP, CLAP, and Gemma embeddings, and frozen external models do the actual generation. In particular, audio is rendered by AudioLDM, and the CLAP-conditioned renderer is then scored with CLAP-based metrics. Final output quality is therefore capped by these renderers and may favor methods aligned in the same embedding spaces, which makes "any-to-any multimodal generation" an overstatement of what is shown. The paper also motivates itself as an alternative to LLM-backbone any-to-any models, yet does not discuss representative LLM-based and unified systems [1, 2], or recent unified multimodal-conditioned audio generators [3], which offer a stronger alternative to the dated audio rendering used here. Without this discussion or a more end-to-end evaluation, it is hard to judge where MUNITE stands against the approaches it claims to replace.

[1] S. Wu, H. Fei, L. Qu, W. Ji, and T.-S. Chua. NExT-GPT: Any-to-Any Multimodal LLM. In ICML, 2024.

[2] J. Zhan, J. Dai, J. Ye, et al. AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling. In ACL, 2024.

[3] Z. Tian, Z. Liu, Y. Jin, R. Yuan, L. Xue, X. Tan, Q. Chen, W. Xue, and Y. Guo. AudioX: A Unified Framework for Anything-to-Audio Generation. In ICLR, 2026.

Questions for authors

Clarifications:

  1. MUNI and MUNITE use a LoRA-tuned text decoder, but CFM, DFM and FlowBind use the frozen one. Is the CIDEr gap still there if the decoders are matched?

  2. CFM and DFM are much worse than the latent models (e.g. FD 71.7 vs 1.5 on PolyMNIST-D-Q). How were they tuned? Did you try running them in a pretrained latent space rather than on pixels?

  3. For MUNI, is sampling the PoE posterior without a prior expert the default conditional sampler from the original paper?

  4. Many of the gaps are small. Could you give results over several seeds, and some diversity measure for one-to-many / unconditional samples on image–text–audio?

Suggestion:

  1. It would help to discuss the LLM-based any-to-any models the intro positions against, and to try a stronger audio decoder than AudioLDM.
8 comments
AuthorsOpenReview ↗

Response to Reviewer aQ3K (1/5)

Dear Reviewer aQ3K,

We thank the reviewer for the careful reading, and for describing the formulation as "a clean, principled unification of encoding and conditional generation" and the PolyMNIST-D-Q analysis as "the strongest support for the paper's central claim." Below, we respond to each point and describe the additional experiments and revisions.

[W1-1, Q2] Design and tuning of the CFM and DFM baselines

We first note that, on PolyMNIST-D-Q and FFHQ64, MUNITE does not use a pretrained latent space either. It takes raw pixels like CFM and learns its latent together with its decoders (App. C.1). DFM, in contrast, already runs on a pretrained latent, the discrete token space of a frozen VQ-VAE (App. C.3), which is the setting Q2 asks about, and still falls below MUNITE on every PolyMNIST-D-Q metric (Table 1).

Without modality-specific decoders, a single Transformer backbone in CFM and DFM must learn to generate every modality in full, for example the handwriting and background of each PolyMNIST-D-Q view, whereas MUNITE's latent carries the shared attributes and leaves the rest to per-view decoders. For a fair comparison, CFM and DFM use the Transformer blocks, width and learning rate of MUNITE, with more blocks so that their trainable parameters approximately match those of MUNITE including its decoders (Table 7). They also train for twice as many epochs as MUNITE in every experiment (Table 8), since joint Transformer models may converge more slowly than latent models with generative decoders.

CFM and DFM also generate plausible images, though with imperfect digit details (Fig. 4). FD on PolyMNIST-D-Q is computed on the features of the verifier, the classifier used for accuracy and coherence (App. E.1), so it can be sensitive to these details, which would explain the large gap in FD.

We also note that CFM and DFM serve as controlled baselines for joint flow-based approaches and do not directly correspond to OmniFlow or to NExT-OMNI. On image-text-audio, the released OmniFlow and CoDi are already reported next to them in Table 2, and OmniFlow, pretrained on far more data, falls below CFM on five of the six coherence metrics (Tables 2 and R1).

[W1-2, Q1] Matched text decoding

Regarding the text decoder, we would like to note that we follow the experimental setup of MUNI, which is designed to show that encoder-decoder models such as MUNI and MUNITE benefit from decoding each modality with its own generative framework. For text, MUNITE decodes a token sequence directly from the shared latent with an autoregressive text renderer that is adapted with LoRA and trained with the model (L844-846). CFM, DFM and FlowBind instead generate normalized text features with flow models, and training them jointly with an autoregressive text renderer is not straightforward. They therefore decode with a frozen text renderer, which FlowBind fine-tuned on captions from the same datasets we train on (L1011-1013, FlowBind App. C.1).

Still, as the reviewer points out, only MUNI and MUNITE apply LoRA to the text renderer. Although the flow baselines cannot easily be trained jointly with this renderer, we ran an additional experiment for a fully apples-to-apples comparison. To evaluate every method with an identical text renderer, we separately fine-tuned the renderer with LoRA on ground-truth text features of our training data, froze it, and used it for all methods we trained, including MUNI and MUNITE.

Table R2. Metrics on generated text with the matched renderer, mean over three training runs.

\begin{array}{l|cc|cc|cc} \hline \tx{Method}&I\to T&A\to T&I\to(T,A)&A\to(T,I)&\tx{Unc. }(T,I)&\tx{Unc. }(T,A)\nl &\tx{CIDEr}&\tx{CIDEr}&\tx{CLAP}&\tx{CLIP}&\tx{CLIP}&\tx{CLAP}\nl \hline \tx{CFM}&27.05\sm{13.25}&36.26\sm{19.70}&25.65\sm{21.65}&25.38\sm{24.06}&25.08\sm{23.66}&23.03\sm{20.40}\nl \tx{DFM}&22.59\sm{11.08}&38.90\sm{22.37}&24.49\sm{20.85}&25.41\sm{24.16}&25.01\sm{23.79}&22.63\sm{19.60}\nl \tx{FlowBind}&34.39\sm{16.41}&53.12\sm{29.47}&31.55\sm{26.30}&26.49\sm{25.13}&\tx{-}&\tx{-}\nl \tx{MUNI}&\bb{43.72}\sm{44.27}&\bb{60.34}\sm{51.05}&33.39\sm{32.06}&26.24\sm{25.82}&27.13\sm{26.76}&29.14\sm{27.68}\nl \tx{MUNITE}&37.22\sm{37.74}&59.18\sm{55.70}&\bb{36.15}\sm{34.76}&\bb{26.75}\sm{26.41}&\bb{27.70}\sm{27.33}&\bb{31.99}\sm{30.31}\nl \hline \end{array}$$ In parentheses, the scores with the renderers used in the paper. **With this matched renderer, the CIDEr gap remains, and MUNITE still has the highest score on all four coherence metrics that involve generated text (Table R2),** although CFM, DFM and FlowBind roughly double their CIDEr but remain below MUNITE. We have added this comparison to App. E.
AuthorsOpenReview ↗

Response to Reviewer aQ3K (2/5)

[W1-3] Clarification on the evaluation setting of external baselines

We would like to clarify that the external baselines are evaluated in the same evaluation setting as the other methods. Every method is evaluated on its generated images, captions and audio with the same evaluators, as in MUNI and FlowBind, so the protocol measures the quality of any-to-any generation in each output modality. CoDi and OmniFlow are built to generate raw modalities, so the feature-space part of the protocol does not apply to them. We would appreciate any suggestion from the reviewer for a better way to evaluate them.

For this rebuttal, we also re-evaluated both released models ourselves on every route they support with three evaluation seeds (Tables R1 and R7b), and obtained scores close to those reported by MUNI, with a slightly lower FID. We have updated Tables 2 and 11-13 with these measurements in the revision.

[W1-4] MUNI result expected by construction

Here we elaborate on the limitation of MUNI that we identify in the paper. In MUNI, as in multimodal VAEs, one latent is sampled from the posterior of the observed subset and every target is decoded from it, so that what the inputs leave open is decided once for all targets. MUNI's subset posterior, however, is a diagonal Gaussian aggregated from the observed modalities (Sec. 3), and its criterion only asks it to be sufficient for each missing target separately (Eq. 13), so the shared latent need not carry the variation the targets must agree on.

Identifying and resolving this structural limitation is the main contribution of the paper. MUNITE models the conditional distribution of the full-observation latent, so what the inputs leave open is sampled once and shared by all targets (Sec. 2.7, App. A.3). Our one-to-many evaluations on PolyMNIST-D-Q and FFHQ64 are designed to expose this limitation, and MUNITE removes it without losing single-target quality.

AuthorsOpenReview ↗

Response to Reviewer aQ3K (3/5)

[W1-5, Q4] Variance and diversity

As is common practice in this line of work (CoDi, OmniFlow, NExT-GPT, FlowBind and MUNI), we reported a single run per method in the paper. Following the reviewer's suggestion, Table R1 reports three training runs for every method we train on image-text-audio.

Table R1. Image-text-audio coherence (paper Table 2), mean ±\pm std over three training runs.

\begin{array}{l|ccc|ccc} \hline &&1\to N\ \tx{coherence}&&&\tx{Unconditional}&\nl \tx{Method}&T\to(I,A)&I\to(T,A)&A\to(T,I)&(T,I)&(T,A)&(I,A)\nl &\tx{AIS}&\tx{CLAP}&\tx{CLIP}&\tx{CLIP}&\tx{CLAP}&\tx{AIS}\nl \hline \tx{CoDi}&64.12\pm0.60&7.73\pm0.20&23.57\pm0.08&\tx{-}&\tx{-}&\tx{-}\nl \tx{OmniFlow}&77.31\pm0.74&15.38\pm0.65&22.67\pm0.11&21.11\pm0.08&14.20\pm0.24&50.89\pm1.16\nl \hline \tx{CFM}&74.08\pm0.54&21.65\pm0.09&24.06\pm0.07&23.66\pm0.06&20.40\pm0.47&74.53\pm0.44\nl \tx{DFM}&73.31\pm0.63&20.85\pm0.19&24.16\pm0.05&23.79\pm0.05&19.60\pm0.26&72.10\pm1.00\nl \tx{FlowBind}&79.61\pm0.29&26.30\pm0.38&25.13\pm0.22&\tx{-}&\tx{-}&\tx{-}\nl \tx{MUNI}&81.12\pm0.32&32.06\pm0.41&25.82\pm0.07&26.76\pm0.05&27.68\pm0.15&82.46\pm0.31\nl \tx{MUNITE}&\bb{84.31\pm0.92}&\bb{34.76\pm0.15}&\bb{26.41\pm0.04}&\bb{27.33\pm0.10}&\bb{30.31\pm0.55}&\bb{85.97\pm0.51}\nl \hline \end{array}$$ CoDi and OmniFlow show mean $\pm$ std over three evaluation seeds of their released checkpoints. **In every run, MUNITE is higher than MUNI on all six metrics of Table 2, and each margin, including $T\to(I,A)$ AIS and $A\to(T,I)$ CLIP, is several times the run-to-run standard deviation.** The paper rules out collapsed samples on PolyMNIST-D-Q (Table 10), and Table R3 extends the check to image-text-audio as suggested. **MUNITE is at least as diverse as MUNI on every route and target and clearly more diverse in generated text, so its coherence advantage does not come from reduced variation.** CFM, DFM, CoDi and OmniFlow are mostly more diverse but far less coherent (Table R1). Unconditional diversity is close overall (joint Vendi 38.27 for MUNITE and 39.25 for MUNI), with higher image and audio coverage for MUNITE and higher image recall for MUNI (0.63 vs. 0.55). We have added the full unconditional results to App. E. **Table R3.** Diversity of eight samples per input on the one-to-many routes. $$\def\tx#1{\text{#1}}\def\nl{\newline} \begin{array}{lll|ccccccc} \hline \tx{Route}&\tx{Target}&\tx{Measure}&\tx{CoDi}&\tx{OmniFlow}&\tx{MUNI}&\tx{CFM}&\tx{DFM}&\tx{FlowBind}&\tx{MUNITE}\nl \hline T\to(I,A)&I&\tx{Dist.}&0.266&0.261&0.231&0.357&0.365&0.197&0.237\nl &&\tx{Vendi}&2.64&2.62&2.41&3.31&3.38&2.19&2.45\nl &A&\tx{Dist.}&0.229&0.222&0.109&0.247&0.251&0.114&0.112\nl &&\tx{Vendi}&2.33&2.28&1.61&2.42&2.44&1.64&1.63\nl \hline I\to(T,A)&T&\tx{Dist.}&0.414&0.322&0.060&0.420&0.450&0.000&0.227\nl &&\tx{Vendi}&3.75&2.83&1.25&3.75&3.97&1.00&2.24\nl &A&\tx{Dist.}&0.213&0.224&0.107&0.299&0.330&0.112&0.144\nl &&\tx{Vendi}&2.22&2.29&1.60&2.76&2.97&1.63&1.80\nl \hline A\to(T,I)&T&\tx{Dist.}&0.409&0.171&0.037&0.458&0.436&0.000&0.212\nl &&\tx{Vendi}&3.68&1.84&1.14&4.05&3.87&1.00&2.17\nl &I&\tx{Dist.}&0.222&0.285&0.251&0.401&0.405&0.185&0.278\nl &&\tx{Vendi}&2.35&2.78&2.56&3.66&3.69&2.12&2.74\nl \hline \end{array}$$ Dist. is the pairwise cosine distance and Vendi the Vendi score of the generated outputs.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers and AC,

We thank the reviewers for recognizing "a clean, principled unification of encoding and conditional generation" (aQ3K), that the PolyMNIST experiments "directly test the central mechanism" (8rLY), and the "careful experimental setup" with "large coherence gains, with a check against mode collapse" (tZor). In response, we ran the additional experiments and analyses summarized below, which the individual responses describe in detail. We have also uploaded a revised manuscript with the changes highlighted in blue. Line numbers in our responses refer to the original submission.

  • Mean and standard deviation over three training runs of every method we train on image-text-audio (Table R1).
  • Text-generation results with a matched text renderer (Table R2).
  • Sample diversity of generated outputs (Table R3).
  • Component ablations extended to PolyMNIST-D-Q and FFHQ64 (Table R4).
  • Inference time against MUNI (Table R5).
  • A comparison of the conditional samples with the data conditionals on FFHQ64 (Table R6).
  • NExT-GPT, AnyGPT and AudioX as additional baselines on image-text-audio, alongside the UnifiedIO2-L scores reported by MUNI (Table R7).
  • Re-evaluation of the image-text-audio coherence and many-to-one metrics with independent evaluators (SigLIP, MS-CLAP and ImageBind, Table R8).
  • MUNITE with an audio decoder built from AudioX in place of the AudioLDM renderer (Table R9).

In each of three training runs on image-text-audio, MUNITE is higher than MUNI on all six coherence metrics of Table 2. It stays first with a matched text renderer and under independent evaluators, so the advantage does not depend on either choice. In the component ablations of Table 14, which we extended to PolyMNIST-D-Q and FFHQ64 (Table R4), the contrastive loss mainly improves fidelity, and partial-data self-distillation mainly improves coherence on FFHQ64 and image-text-audio, along with fidelity on FFHQ64.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.