acceptodds
Under review as a conference paper at ICLR 2027

LatentUMM: Dual Latent Alignment for Unified Multimodal Models

Abstract

Unified multimodal models (UMMs) achieve strong performance in both understanding and generation by learning a shared latent space, yet they often exhibit functional inconsistency between these two capabilities. We observe that this issue does not stem from a lack of shared representations, but from the absence of explicit alignment between the transformations that map into and out of the latent space. As a result, generation and re-encoding can follow inconsistent trajectories, leading to semantic drift under modality transitions. In this work, we propose LatentUMM, a framework that constructs an enhanced shared latent space to explicitly align these transformations and improve cross-modal consistency. LatentUMM consists of two stages. First, dual latent alignment enforces consistency at both the modality and capacity levels: cross-modal alignment uses a stronger embedding model to impose structured cross-modal semantics, while dual capacity alignment enforces bidirectional consistency under generation and re-encoding. Second, latent dynamics stabilization improves robustness via stochastic latent rollouts and preference optimization, favoring trajectories that better preserve semantic consistency. Experiments show that LatentUMM consistently improves multimodal consistency across diverse architectures. Code is available at: https://anonymous.4open.science/r/LatentUMM.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Unified Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aYQdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes LatentUMM, a post-training framework for improving consistency between understanding and generation in unified multimodal models. The framework combines dual latent alignment with latent dynamics stabilization. Dual latent alignment uses external embedding supervision to align modalities and introduces a capacity-alignment objective intended to preserve semantics under generation and re-encoding. Latent dynamics stabilization samples latent perturbations and applies a preference objective based on their consistency scores. The paper also provides a distributional-alignment analysis.

The evaluation covers multimodal understanding, image generation, image editing, and dedicated consistency benchmarks, with BAGEL as the main backbone and additional experiments on Janus-Pro and Harmon. The reported results suggest improvements across these settings, although methodological and inference-protocol inconsistencies make the source of these improvements difficult to establish.

Critical strengths

(1) The paper studies a meaningful aspect of multimodal unification and evaluates it beyond standalone task scores.

(2) The empirical coverage provides encouraging evidence of applicability across configurations.

Critical weaknesses

(1) The method descriptions in the main text and appendix are inconsistent. Section 3.3 performs alignment through generation and re-encoding, whereas Appendix E.1 reconstructs hidden states through learnable mappings.The latter does not by itself establish improved consistency between generation and understanding. The authors should clarify the actual training objective and demonstrate its incremental benefit beyond teacher supervision and standard post-training.

(2) Training gains are not clearly separated from decoding benefits. Table 4(e) reports MMMU scores of 54.1 with self-consistency decoding versus 51.9 with single-pass decoding, while Appendices E.8–E.9 claim unchanged inference and no additional latency. The authors should reconcile these statements and compare methods under matched decoding budgets to isolate the contribution of training.

Questions for authors

(1) How does the preference objective improve stability? Since gradients flow through both selected scores, the loss can decrease by lowering the worst consistency score. What evidence shows improved average or lower-tail consistency rather than merely a wider score gap?

(2) Does the Wasserstein proof preserve the required pairing? Appendix B’s product coupling does not generally preserve the native–auxiliary dependence that determines the interpolated distributions. What assumptions justify this step, or should the guarantee be restricted to the paired-distance bound in Eq.(23)?

4 comments

W1. Implemented objective versus the depicted image cycle

Section 3.3 presents capacity alignment at the conceptual, image level, and Appendix E.1 gives its implemented hidden-state form; we will align the two in Section 3.3 and Figure 2. The implemented objective is a meaningful constraint rather than a free autoencoding task, for three reasons.

  1. It passes through a constrained space. The projection P_θ in Eq. 32 is the same head that L_target and L_x-modal pull toward detached external embeddings, so a hidden state can only be reconstructed well if it carries the semantics that space encodes.
  2. Generated images enter the objective. When rollouts are active, the image-branch target is the external embedding of the image the model actually generated (Eq. 27).
  3. Consistency is measured on outputs. Unified-Bench, RealUnify and the Table 7 diagnostic generate real images and re-interpret them. On Table 7, error falls from 0.89 to 0.79 at T = 1 and from 1.82 to 1.25 at T = 4. (Table 7 reports the squared L2 distance between normalised embeddings, i.e. 2(1 − cos); we will correct Eq. 10 accordingly.)

Incremental benefit beyond teacher supervision. Table R1a is a stage-wise ablation under one configuration (same initialisation, 10,000 training pairs, data order, seed, LoRA configuration and steps; single-pass decoding). Teacher supervision alone gives almost no gain and lowers MME by 19.8 points. Adding the capacity cycle turns the same supervision into gains on every benchmark, so its contribution is not absorbed by the teacher.

Table R1a. All rows from one training sweep with the same initialisation, 10,000 training pairs, data order, seed, LoRA configuration and steps; full benchmarks; single-pass decoding. For reference, the Table 1 checkpoint, trained with the configuration of Appendices E.1–E.2, scores 1695.8 / 54.1 / 67.2 / 73.6.

MethodMME ↑MMMU ↑MM-Vet ↑MathVista ↑
BAGEL (no post-training)1691.451.965.971.3
External teacher supervision only1671.652.465.771.9
Stage I: + capacity alignment1694.753.766.473.8
Stage II only (preference, no Stage I)1685.152.166.173.3
Full LatentUMM (Stage I + II)1696.153.267.273.6

Table R1 extends the same matched ablation to matched SFT and to the output-level consistency metrics by re-evaluating the same checkpoints. Rows marked † are taken from the paper (Tables 5 and 7), except the SFT consistency errors, which we measured with the same protocol as Table 7; SFT was trained on the same data. All rows are from a single training seed; further seeds will be added in the final version.

Table R1.

ObjectiveUnified-Bench ↑RealUnify ↑Error T=1 ↓Error T=4 ↓
BAGEL (no post-training) †0.83460.38750.891.82
SFT †0.83210.36250.911.88
External teacher supervision only0.83610.39250.871.71
+ capacity alignment0.84370.40000.811.33
Full LatentUMM †0.84620.40500.791.25

On generated-and-reinterpreted outputs, teacher supervision alone barely reduces drift (T = 4 error 1.82 → 1.71), while adding capacity alignment brings it to 1.33; standard post-training (SFT) does not reduce it at all. The incremental benefit of capacity alignment beyond teacher supervision and standard post-training is therefore established directly on generated and re-interpreted outputs.

W2. Training gains versus decoding

Every number for LatentUMM and for every baseline in Tables 1, 2, 3, 5, 6 and 7 is obtained with single-pass decoding, and generation, editing and consistency results never use self-consistency.

We apologise for the confusion caused by Table 4(e), which is mislabeled. Its rows do not change how the trained model is decoded at evaluation, which is always single-pass. They change how the K rollout candidates are turned into a training signal in Stage II, the alternatives discussed in Section 3.4. "Single-pass" uses one rollout per example and therefore no candidate comparison; "simple ensembling" regresses on the average candidate score; and "self-consistency" selects the most and least consistent candidates as a preference pair, which is our default. This is why its last row coincides with Table 1. We will rename the table "Rollout aggregation in Stage II" with matching row labels.

To isolate the training effect under matched inference budgets, Table R2 reports all methods under single-pass and under the same inference-time self-consistency budget, with latency measured on the same hardware.

Table R2. Latency measured on one H100 80GB in bf16, batch size 1, averaged over the four benchmarks.

DecodingModelMME ↑MMMU ↑MMB-EN ↑MMB-CN ↑Latency / sample
Single-passBAGEL1691.451.982.6580.940.84 s
Single-pass+ SFT1690.151.982.6580.830.86 s
Single-pass+ LatentUMM1695.854.184.9984.910.87 s
Self-consistency, 5 samplesBAGEL1693.052.683.0481.374.03 s
Self-consistency, 5 samples+ SFT1691.252.582.9781.104.11 s
Self-consistency, 5 samples+ LatentUMM1696.454.485.1985.084.19 s

Within each decoding setting LatentUMM leads by a similar margin (MMMU +2.2 under single-pass, +1.8 under self-consistency), so the gain comes from training. Self-consistency costs roughly five times the single-pass latency for every model, which is why none of the main-table results use it. The statement in Appendices E.8–E.9 therefore holds: no rollout, sampling or external embedding is used at inference. The LoRA adapters are merged into the base weights at inference, so the model has exactly BAGEL's architecture and cost; the single-pass differences of at most 0.03 s per sample in Table R2 are within run-to-run timing variation. We will make this precise in the appendix.

Q1. How does the preference objective improve stability?

The reviewer is right that, in principle, the pairwise loss could be reduced by lowering the worst score. We measured whether this happens in practice; it does not.

Existing matched ablation. In Table 4(d), the σ = 0 row makes all K candidates identical, so s⁺ = s⁻ and L_pref has zero gradient, with everything else unchanged. Against it, the default model improves MMMU from 52.0 to 54.1 and MMBench-CN from 80.65 to 84.91. (Section 4.3.2 calls the σ = 0 setting "deterministic inference"; this is a wording error, since σ is the training-time rollout perturbation and all rows use the same single-pass decoding.)

Candidate-score distribution. Table R3 compares the full model with and without L_pref on the same held-out prompts and the same K = 10 perturbation seeds, together with an independent image-level loopback score. Scores are × 100; worst and best are per-prompt minimum and maximum, averaged over prompts.

Table R3.

ModelMean s ↑p10 ↑Worst ↑Best ↑Best − worstImage loopback ↑
Full, λ₂ = 078.668.974.282.58.385.9
Full LatentUMM79.871.076.482.96.586.6

The lower tail improves rather than degrades. With prompt-level paired bootstrap 95% CIs, the worst candidate rises by +2.2 [+1.7, +2.7] and p10 by +2.1 [+1.4, +2.8], while the best candidate is unchanged within noise (+0.4 [−0.1, +0.9]). The narrower gap therefore comes from lifting the worst candidates, and the independent image-level loopback moves in the same direction. We will add this analysis to the paper.

Q2. Does the Wasserstein argument preserve the pairing?

The product coupling in Appendix B does not preserve the dependence between ZtZ_t and AtA_t, but it is also unnecessary: using the natural data pairing instead gives the same conclusion directly.

Let Zt,Zi,At,AiZ_t, Z_i, A_t, A_i all be computed from the same data pair (xt,xi)(x_t, x_i). Under this data pairing, (Z~t,Z~i)(\tilde Z_t, \tilde Z_i) of Eq. 13 is itself a coupling of the two interpolated distributions. Hence:

W2(P~t,P~i)≤(E∥Z~t−Z~i∥22)1/2≤(1−α)δpair+αεpairW_2(\tilde P_t, \tilde P_i) \leq \left(\mathbb{E} \lVert \tilde Z_t - \tilde Z_i \rVert_2^2\right)^{1/2} \leq (1-\alpha) \delta_{\mathrm{pair}} + \alpha \varepsilon_{\mathrm{pair}}

where

δpair=(E∥Zt−Zi∥22)1/2,εpair=(E∥At−Ai∥22)1/2\delta_{\mathrm{pair}} = \left(\mathbb{E} \lVert Z_t - Z_i \rVert_2^2\right)^{1/2}, \qquad \varepsilon_{\mathrm{pair}} = \left(\mathbb{E} \lVert A_t - A_i \rVert_2^2\right)^{1/2}

The second inequality is Minkowski's inequality applied to Eq. 15. Whenever εpair<δpair\varepsilon_{\mathrm{pair}} < \delta_{\mathrm{pair}} and α>0\alpha > 0, the bound is strictly below δpair\delta_{\mathrm{pair}}. Because δ=W2(Pt,Pi)≤δpair\delta = W_2(P_t, P_i) \leq \delta_{\mathrm{pair}}, the guarantee concerns the paired gap rather than the marginal gap δ\delta. This is the quantity in Eq. 23, and also the one Lx-modalL_{\text{x-modal}} optimises, since it compares each text with its own image.

Thank you again for your thoughtful and constructive feedback. As the discussion period draws to a close, we would like to kindly follow up to ensure that we have fully addressed your concerns. In response to your suggestions, we have incorporated the requested analyses and further clarified our methodology. If you have any further questions or feedback, we would be grateful if you could let us know. We hope these revisions adequately address your concerns, and would greatly appreciate it if you could reconsider your evaluation in light of these updates.

Reviewer_H38oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to address the inconsistency problem between understanding and generation in multimodal models. And the authors claim that the inconsistency comes not from a lack of shared representations but from misaligned transformations into and out of the latent space. To this end, this work proposed a solution, LatentUWM, it adds a shared latent space defined by an external embedding model i.e., Gemini Embedding, using an L2 cross-modal loss and a latent cycle-reconstruction loss. Additionally, LatentUWM perturbs latents and applies a ranking loss that favors more consistent samples. Results are reported on Bagel, Janus-Pro and Harmon across understanding, generation and consistency benchmarks.

Critical strengths

First of all, the inconsistency problem between understanding and generation in unified multimodal models is quite interesting and well-motivated. It is a real and important challenge for unified multimodal models.

Second, the method is quite simple or easy to use and can generalize to different backbones.

Third, I appreciate the authors' broad experiment evaluation, including understanding, generation, consistency, different backbones), and also includes an SFT control and several ablations.

Critical weaknesses

There are several critical weaknesses for this work:

(1) The ablations contradict the central claim. The paper claims that the inconsistency problem is not because a lack of shared representations. However, in table 4 (a), we found that aligning within the existing latent space gives almost no gain. Also in table 4 (b), using Bagel’s own embeddings gives results identical to the baseline. These results show the method does not depend on one specific external model, but it neither works without one. Figure 3 likewise measures a smaller representation gap.

(2) The main comparison may be unfair. The final results in Table 4e use self-consistency decoding, which is never mentioned in previous sections. With single-pass decoding, MMBench-EN (80.16), MMBench-CN (80.60) and MME (1688.9) fall below the Bagel baseline, and MMMU matches it (51.9). Please clarify which decoding the baselines in Tables 1 and 6 use, and compare all methods under the same setting.

(3) The gains are small and reported without variance. Many improvements are 1–2 points on small benchmarks (MMMU, MMVet, RealUnify), and most of the MathVista gain already comes from SFT. The authors are suggested to report means and standard deviations over multiple runs.

(4) Methodology-wise, there are many existing alignment methods, such as DoraCycle、REPA/VIRAL, RecA. Please sate the difference between LatentUWM and these methods. For experiments, please include compare with RecA in the generation task.

Questions for authors

I mainly have two questions for authors: (1) How do the results compare when all methods use the same decoding? (2) How much of the gain comes from embedding distillation alone (L_x-modal only)?

4 comments

W1 / Q2. Do the ablations contradict the central claim?

First, a clarification on Table 4(b). The "Bagel" row is the base model without post-training, included as a reference; this is why it matches Table 1 exactly. It is not alignment with BAGEL's own embeddings. Alignment inside BAGEL's native space is Table 4(a), first row, together with the teacher-free controls in Appendix E.6. We will relabel the row "BAGEL (no post-training)".

We respectfully believe the ablations support, rather than contradict, the central claim. Our claim is that a shared latent space is necessary but not sufficient. BAGEL already has one, yet its outputs drift under repeated generation and re-interpretation (Table 7: 0.89 at T = 1 rising to 1.82 at T = 4). Aligning inside that native space brings only small gains, such as MMMU 51.9 → 52.3. What is missing is structure that both mappings must agree on, not another shared space; an external semantic reference supplies that structure.

Two further results support this reading. First, the gain is not tied to one teacher: CLIP, SigLIP and Gemini embeddings all improve MMMU (52.8, 54.2, 54.1 against 51.9) and MathVista (73.3, 73.4, 73.6 against 71.3). Second, what matters is the teacher's semantics, not the extra loss. In a controlled run that changes only the target, a randomly permuted teacher target (same distribution and loss scale) reduces T = 2 consistency error by just 1.1%, whereas the correct target reduces it by 21.3% (0.1991 → 0.1566; error = 1 − cos under the Gemini embedding model).

Figure 3 is meant to show representational proximity; the functional evidence is Tables 5 and 7, and Table R1 below. To prevent the reading that shared representations are unimportant, we will clarify in the abstract that the inconsistency "is not resolved by sharing representations alone".

Q2. How much comes from embedding distillation alone? Very little on its own. Table R1a is a stage-wise ablation with the same initialisation, 10,000 training pairs, data order, seed, LoRA configuration, steps and single-pass decoding. Aligning to the external embeddings alone changes MMMU by +0.5 and MathVista by +0.6, and lowers MME by 19.8. The gain appears only when the capacity cycle is added, which turns the same supervision into improvements on every benchmark.

Table R1a. All rows from one training sweep with the same initialisation, 10,000 training pairs, data order, seed, LoRA configuration and steps; full benchmarks; single-pass decoding. For reference, the Table 1 checkpoint, trained with the configuration of Appendices E.1–E.2, scores 1695.8 / 54.1 / 67.2 / 73.6.

MethodMME ↑MMMU ↑MM-Vet ↑MathVista ↑
BAGEL (no post-training)1691.451.965.971.3
External embedding alignment only1671.652.465.771.9
Stage I: + capacity alignment1694.753.766.473.8
Stage II only (preference, no Stage I)1685.152.166.173.3
Full LatentUMM (Stage I + II)1696.153.267.273.6

So the contribution is not embedding distillation, but semantic supervision combined with the bidirectional consistency term. Table R1 extends this matched ablation to the generation and consistency benchmarks by re-evaluating the same checkpoints. Rows marked † are taken from the paper (Tables 2 and 5), where SFT was trained on the same data. All rows are from a single training seed; further seeds will be added in the final version.

Table R1.

ObjectiveDPG ↑Unified-Bench ↑RealUnify ↑
BAGEL (no post-training) †84.10.83460.3875
SFT †83.780.83210.3625
External supervision only (L_x-modal)84.420.83610.3925
+ capacity alignment85.180.84370.4000
Full LatentUMM †85.620.84620.4050

External supervision alone accounts for a small part of the improvement over the base model; most of it appears once capacity alignment is added. We will state this attribution explicitly in the paper.

W2 / Q1. Is the main comparison fair?

Yes. Every number for LatentUMM and for every baseline in Tables 1 and 6 is obtained with single-pass decoding under the same protocol.

We apologise for the confusion caused by Table 4(e), which is mislabeled. Its rows do not change how the trained model is decoded at evaluation, which is always single-pass. They change how the K rollout candidates are turned into a training signal in Stage II, the alternatives discussed in Section 3.4. "Single-pass" uses one rollout per example and therefore no candidate comparison; "simple ensembling" regresses on the average candidate score; and "self-consistency" selects the most and least consistent candidates as a preference pair, which is our default. This is why its last row coincides with Table 1. The table therefore shows that the preference formulation itself matters, and we will rename it "Rollout aggregation in Stage II" with matching row labels.

In addition, the gains on generation, editing and the consistency benchmarks (Tables 2, 3, 5 and 7) cannot come from decoding, because self-consistency is never used there. As the reviewer requested, Table R2 also applies the same inference-time self-consistency budget to every method, with latency measured on the same hardware. The single-pass rows are the Table 1 numbers.

Table R2. Latency measured on one H100 80GB in bf16, batch size 1, averaged over the four benchmarks.

DecodingModelMME ↑MMMU ↑MMB-EN ↑MMB-CN ↑Latency / sample
Single-passBAGEL1691.451.982.6580.940.84 s
Single-pass+ SFT1690.151.982.6580.830.86 s
Single-pass+ LatentUMM1695.854.184.9984.910.87 s
Self-consistency, 5 samplesBAGEL1693.052.683.0481.374.03 s
Self-consistency, 5 samples+ SFT1691.252.582.9781.104.11 s
Self-consistency, 5 samples+ LatentUMM1696.454.485.1985.084.19 s

Under the same self-consistency budget, LatentUMM keeps its lead over BAGEL (MMMU 54.4 against 52.6; MMBench-CN 85.08 against 81.37), so the improvement is a training effect rather than a decoding effect. Under single-pass decoding the three models have the same inference cost: the LoRA adapters are merged into the base weights, so LatentUMM has exactly BAGEL's architecture, and the differences of at most 0.03 s per sample are within run-to-run timing variation.

W3. Small gains and variance

We agree that uncertainty estimates are important and provide them here. Within the discussion period we report evaluation-level uncertainty: Table R5 gives paired bootstrap 95% CIs over evaluation items (10,000 resamples) for the Table 1 checkpoints, against both the base model and matched SFT. This captures uncertainty from the finite evaluation sets but not from training randomness. We will add training-seed variance (3 seeds per method, including matched SFT, reported as mean ± SD) in the final version.

Table R5. Paired differences on the same evaluation items [95% bootstrap CI] (single training run per method; point estimates equal the Table 1 and Table 5 differences).

ComparisonMMMUMM-VetMathVistaRealUnify
LatentUMM − BAGEL+2.2 [+0.3, +4.1]+1.3 [−2.0, +4.6]+2.3 [+0.2, +4.4]+0.0175 [−0.0200, +0.0550]
LatentUMM − matched SFT+2.2 [+0.2, +4.2]+1.4 [−1.9, +4.7]+0.5 [−1.3, +2.3]+0.0425 [+0.0060, +0.0790]

The gains hold up under this test. MMMU improves significantly over both BAGEL and matched SFT, MathVista over BAGEL, and RealUnify over matched SFT, the controlled post-training baseline. All eight point estimates are positive. On MM-Vet, whose 218 GPT-graded items give wide intervals, the difference points the same way but is not resolved at this sample size, and, as the reviewer notes, the MathVista gain over the base model is largely shared with SFT; we report both as they are. We will include these intervals in the paper.

W4. Relation to DoraCycle, REPA/VIRAL and RecA

We will expand the related-work section along the dimensions below.

MethodTraining signalExternal teacherPathway optimisedGenerated images re-interpreted
DoraCycleCycle consistency through text→image→text and image→text→image cycles, to adapt a unified model to a target domainNoBothYes, during training
REPAAligns intermediate diffusion-transformer states with features of a pretrained visual encoder (e.g. DINOv2)Yes, visual onlyGenerationNo
VIRALAligns the MLLM's internal visual representations with features of a vision foundation modelYes, visual onlyUnderstandingNo
RecASelf-supervised reconstruction: regenerates the input image conditioned on the model's own understanding-encoder embeddingsNoGeneration and editingNo
LatentUMMAligns both modalities to one cross-modal target, plus a latent cycle and preference over perturbed latentsYes, cross-modalBothYes, via rollout targets and in evaluation

The distinction we will emphasise: REPA and VIRAL each align one pathway to a single-modality teacher, and RecA strengthens generation through reconstruction. DoraCycle is closest in spirit, but it enforces cycle consistency on decoded outputs to adapt a model to a new domain. LatentUMM instead aligns the latent interface of both pathways to one cross-modal semantic target and stabilises it with latent rollouts, aiming at consistency between understanding and generation rather than domain transfer.

We also ran RecA on BAGEL under the same data and single-pass decoding (Table R6). RecA is slightly stronger on DPG-Bench and WISE, which measure prompt-following fidelity of generation and are what its reconstruction objective targets. LatentUMM leads on UEval and Unified-Bench, which require the generated output to be re-interpreted consistently, and also yields larger understanding gains (Table 1: MMMU 54.1 against 52.3 for RecA). We will report this comparison as is in the final version.

Table R6.

BAGEL post-trainingDPG-Bench ↑WISE ↑UEval ↑Unified-Bench ↑
SFT83.780.40731.10.8321
RecA85.870.42131.40.8418
LatentUMM85.620.41831.80.8462

Thank you again for your thoughtful and constructive feedback. As the discussion period draws to a close, we would like to kindly follow up to ensure that we have fully addressed your concerns. In response to your suggestions, we have incorporated the requested analyses and further clarified our methodology. If you have any further questions or feedback, we would be grateful if you could let us know. We hope these revisions adequately address your concerns, and would greatly appreciate it if you could reconsider your evaluation in light of these updates.

Reviewer_F4hXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed LatentUMM to address the inconsistency between I2T understanding and T2I generation in unified multimodal models. It combines external embedding supervision, cross-modal alignment, a reconstruction-based consistency objective, and preference optimization over perturbed latent representations. The manuscript reports improvements in understanding, generation, and consistency benchmarks, with experiments on BAGEL, Janus-Pro, and Harmon.

Critical strengths
  • The paper is well-organized, with detailed explanations of the preliminary, intuition, and methodology.
  • The problem is well motivated. Evaluating whether a model faithfully interprets its own generated outputs goes beyond measuring understanding and generation separately.
  • The paper presents comprehensive experiments covering understanding, generation, editing, and unification.
Critical weaknesses
  • There seems to be a disconnect between how capacity alignment is framed in the main text and how it’s actually implemented. Section 3.3 describes alignment through a full cycle of image generation and re-encoding, but Appendix E.1 reveals that the actual objective just reconstructs hidden states( Eq. 32–33). Because the model only learns to map hidden states back and forth, it can easily achieve a low reconstruction error without actually fixing semantic mistakes in the generated images or how they are interpreted.

  • I have a concern about how the preference objective behaves in practice. Because minimizing −log⁡σ(smax⁡−smin⁡)-\log\sigma(s_{\max}-s_{\min}) simply widens the margin between the best and worst scores, the model can trivially satisfy this loss by actively degrading the worst candidate rather than improving the best one . Instead of boosting average or worst-case consistency, this could actually work against the paper's goal of making representations robust to perturbations. I suggest the authors could provide a more detailed discussion on this.

  • Table 4 mainly tests hyperparameter variations (embeddings, noise levels, rollouts) rather than isolating the three core objectives. While the appendix adds teacher-free baselines, it changes the LoRA rank and learning rate, and its full-method scores do not match Table 1. Without a strictly matched ablation, it is impossible to tell whether the performance gains come from the proposed consistency losses or simply from distilling an external teacher.

  • While Table 6 tests BAGEL, Janus-Pro, and Harmon on understanding and generation tasks, the consistency benchmarks and repeated-transformation diagnostics are only reported for BAGEL. Stronger single-task metrics do not guarantee better cross-pathway alignment. To substantiate the claim of an architecture-agnostic benefit, the authors should evaluate the other two backbones on the dedicated consistency benchmarks as well.

Questions for authors
  • As mentioned in W1, could the authors clarify why this hidden-state shortcut is enough to genuinely constrain the actual generation and understanding pathways?

  • Could the author provide an analysis on how the full candidate-score distribution changes? It would help establish whether the objective actually stabilizes latent representations.

3 comments

W1 / Q1. Why the hidden-state cycle constrains the real pathways

Section 3.3 presents capacity alignment at the conceptual, image level, and Appendix E.1 gives its implemented hidden-state form. The implemented objective is not a shortcut, for three reasons.

  1. The cycle passes through a constrained space. The projection P_θ in Eq. 32 is the same head that L_target and L_x-modal pull toward detached external embeddings. The cycle therefore asks that each hidden state be recoverable from a representation that must also sit at its semantic target. An unconstrained autoencoder could reach zero reconstruction error, but this one cannot unless the hidden states carry the semantics that the external space encodes.
  2. Generated images enter the objective. When rollouts are active, the target for the image branch is the external embedding of the image the model actually generated (Eq. 27). Semantic errors in generated images therefore produce a training signal, even though no gradient passes through pixels.
  3. The evidence is measured on outputs, not hidden states. Unified-Bench, RealUnify and the Table 7 diagnostic all generate real images and re-interpret them, all under single-pass decoding. On Table 7, error falls from 0.89 to 0.79 at T = 1 and from 1.82 to 1.25 at T = 4, and the gap widens with each step. (Table 7 reports the squared L2 distance between normalised embeddings)

The cycle is what makes the teacher useful. A stage-wise ablation separates the two. Aligning to the external embeddings alone (L_x-modal without the capacity cycle) gives almost nothing and lowers MME by 19.8 points; adding the capacity cycle turns the same supervision into gains on every benchmark. Running the preference stage without Stage I is also clearly weaker than running it on top of Stage I.

Table R1a. Stage-wise ablation on BAGEL (all rows from one training sweep with the same initialisation, 10,000 training pairs, data order, seed, LoRA configuration and steps; full benchmarks; single-pass decoding. For reference, the Table 1 checkpoint, trained with the configuration of Appendices E.1–E.2, scores 1695.8 / 54.1 / 67.2 / 73.6.)

MethodMME ↑MMMU ↑MM-Vet ↑MathVista ↑
BAGEL (no post-training)1691.451.965.971.3
External embedding alignment only1671.652.465.771.9
Stage I: + capacity alignment1694.753.766.473.8
Stage II only (preference, no Stage I)1685.152.166.173.3
Full LatentUMM (Stage I + II)1696.153.267.273.6

The value of the cycle is thus shown on the model's behaviour, not merely by a low reconstruction error, and Table R1 below shows the same pattern on the dedicated consistency metrics.

W2 / Q2. Does the preference loss help the worst candidates?

The reviewer is right that, in principle, the pairwise loss could be reduced by lowering the worst score. In practice this does not happen, and we show it directly.

Existing matched ablation. In Table 4(d), the σ = 0 row makes all K candidates identical, so s⁺ = s⁻ and L_pref contributes zero gradient; everything else, including single-pass decoding, is unchanged. Against this row, the default model improves MMMU from 52.0 to 54.1, MMBench-CN from 80.65 to 84.91, MMBench-EN from 83.45 to 84.99 and MME from 1689.7 to 1695.8. (σ is the training-time rollout perturbation and all rows use the same single-pass decoding.)

Candidate-score distribution (Q2). Table R3 compares the full model with and without L_pref on the same held-out prompts, the same K = 10 perturbation seeds per prompt and the same σ. Scores are × 100; worst and best are per-prompt minimum and maximum over candidates, averaged over prompts; p10 is taken over all candidates. Paired differences use a prompt-level bootstrap.

Table R3.

ModelMean s ↑p10 ↑Worst ↑Best ↑Best − worst
Full, λ₂ = 078.668.974.282.58.3
Full LatentUMM79.871.076.482.96.5
Paired difference [95% CI]+1.2 [+0.8, +1.6]+2.1 [+1.4, +2.8]+2.2 [+1.7, +2.7]+0.4 [−0.1, +0.9]−1.8 [−2.3, −1.3]

The preference loss raises the lower tail rather than lowering it. The worst candidate improves by 2.2 points and p10 by 2.1, both with intervals excluding zero, while the best candidate is unchanged within noise. The gap therefore narrows because the worst candidates are lifted, the opposite of the failure mode the reviewer describes. We will add this analysis to the paper.

W3. Matched ablation of the core objectives

Beyond the hyperparameter study in Table 4, we provide matched ablations of the core objectives. Table R1a above adds the objectives one at a time under a fixed configuration. It shows that the gain does not come from distilling the teacher: teacher alignment alone leaves MMMU at 52.4 and MathVista at 71.9, and lowers MME.

We also tested whether the teacher acts as a generic regulariser. Three variants share initialisation, data, optimiser, LoRA configuration and loss weights, and differ only in the teacher target: none, a randomly permuted target (same distribution and loss scale, no semantic correspondence), or the correct target.

Table R1b. Correct versus random semantic pairing (text→image→text, T = 2; error = 1 − cos(z_prompt, z_caption) under the Gemini embedding model).

Teacher targetConsistency error ↓Change vs. no target
None0.19910.0%
Randomly permuted0.1969−1.1%
Correct pairing0.1566−21.3%

Random pairing gives almost no benefit, while correct pairing reduces the error by about a fifth. The improvement therefore comes from semantic correspondence, not from an extra loss term.

Table R1 completes the matched ablation on the dedicated consistency metrics. It re-evaluates the checkpoints of Tables R1a and R1b, so every new row shares the same 10,000 training pairs and data order, LoRA rank, learning rate and number of steps, a checkpoint-selection rule fixed before evaluation, and single-pass decoding. Rows marked † are taken from the paper (Tables 1, 2, 5 and 7), except the SFT consistency errors, which we measured with the same protocol as Table 7; SFT was trained on the same data. All rows are from a single training seed; we will add further seeds and report mean ± SD in the final version.

Table R1.

ObjectiveUnified-Bench ↑RealUnify ↑Error T=1 ↓Error T=4 ↓MMMU ↑DPG ↑
BAGEL (no post-training) †0.83460.38750.891.8251.984.1
SFT †0.83210.36250.911.8851.983.78
External supervision only0.83610.39250.871.7152.484.42
+ capacity alignment (λ₁)0.84370.40000.811.3353.785.18
Full LatentUMM †0.84620.40500.791.2554.185.62
Control: randomly permuted targets0.83390.38500.891.8051.883.95

On the output-level metrics, external supervision alone barely moves consistency (T = 4 error 1.82 → 1.71), whereas adding capacity alignment brings it to 1.33 and raises Unified-Bench from 0.8361 to 0.8437. Randomly permuted targets stay at the base level, and SFT does not improve consistency at all. Together with R1a and R1b, this shows that the consistency gain comes from the capacity objective with correctly paired semantic targets, not from distilling the teacher or from extra regularisation.

On the two points about Appendix E.6. First, E.6 is a self-contained comparison: the teacher-free controls and the full-method reference they are compared against were trained in the same sweep, with the same 10,000 pairs, data order, seed, LoRA rank 64 and α 128, peak learning rate 5 × 10⁻⁶ and 2,000 steps. No comparison in E.6 therefore mixes configurations, and its conclusion that a native projection cannot replace the semantic teacher does not depend on the setting described in Appendices E.1–E.2. Second, the full-method values in E.6 (1696.1 MME, 53.2 MMMU) differ from Table 1 (1695.8, 54.1) because they come from a different training run: the full-method run of this matched sweep, rather than the Table 1 checkpoint trained with the configuration of Appendices E.1–E.2. Both are scored on the full benchmarks. We will label the two runs explicitly in the appendix.

W4. Consistency beyond BAGEL

We agree that task scores do not establish consistency, and we now evaluate the dedicated consistency metrics beyond BAGEL. Table R4 evaluates the post-trained Janus-Pro checkpoint from Table 6 against its own base model under identical prompts and single-pass decoding.

Table R4.

BackboneModelUnified-Bench ↑RealUnify ↑Error T=1 ↓Error T=4 ↓
Janus-ProBase0.79030.38750.911.87
Janus-Pro+ LatentUMM0.79840.39000.861.61

On Janus-Pro, LatentUMM reduces drift at both horizons, most clearly over repeated transformations (T = 4 error 1.87 → 1.61, −14%, compared with −31% on BAGEL), and improves Unified-Bench (0.7903 → 0.7984). The consistency benefit is therefore not specific to BAGEL.

Thank you again for your thoughtful and constructive feedback. As the discussion period draws to a close, we would like to kindly follow up to ensure that we have fully addressed your concerns. In response to your suggestions, we have incorporated the requested analyses and further clarified our methodology. If you have any further questions or feedback, we would be grateful if you could let us know. We hope these revisions adequately address your concerns, and would greatly appreciate it if you could reconsider your evaluation in light of these updates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.