- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes LatentUMM, a post-training framework for improving consistency between understanding and generation in unified multimodal models. The framework combines dual latent alignment with latent dynamics stabilization. Dual latent alignment uses external embedding supervision to align modalities and introduces a capacity-alignment objective intended to preserve semantics under generation and re-encoding. Latent dynamics stabilization samples latent perturbations and applies a preference objective based on their consistency scores. The paper also provides a distributional-alignment analysis.
The evaluation covers multimodal understanding, image generation, image editing, and dedicated consistency benchmarks, with BAGEL as the main backbone and additional experiments on Janus-Pro and Harmon. The reported results suggest improvements across these settings, although methodological and inference-protocol inconsistencies make the source of these improvements difficult to establish.
Critical strengths
(1) The paper studies a meaningful aspect of multimodal unification and evaluates it beyond standalone task scores.
(2) The empirical coverage provides encouraging evidence of applicability across configurations.
Critical weaknesses
(1) The method descriptions in the main text and appendix are inconsistent. Section 3.3 performs alignment through generation and re-encoding, whereas Appendix E.1 reconstructs hidden states through learnable mappings.The latter does not by itself establish improved consistency between generation and understanding. The authors should clarify the actual training objective and demonstrate its incremental benefit beyond teacher supervision and standard post-training.
(2) Training gains are not clearly separated from decoding benefits. Table 4(e) reports MMMU scores of 54.1 with self-consistency decoding versus 51.9 with single-pass decoding, while Appendices E.8–E.9 claim unchanged inference and no additional latency. The authors should reconcile these statements and compare methods under matched decoding budgets to isolate the contribution of training.
Questions for authors
(1) How does the preference objective improve stability? Since gradients flow through both selected scores, the loss can decrease by lowering the worst consistency score. What evidence shows improved average or lower-tail consistency rather than merely a wider score gap?
(2) Does the Wasserstein proof preserve the required pairing? Appendix B’s product coupling does not generally preserve the native–auxiliary dependence that determines the interpolated distributions. What assumptions justify this step, or should the guarantee be restricted to the paired-distance bound in Eq.(23)?
4 comments
W2. Training gains versus decoding
Every number for LatentUMM and for every baseline in Tables 1, 2, 3, 5, 6 and 7 is obtained with single-pass decoding, and generation, editing and consistency results never use self-consistency.
We apologise for the confusion caused by Table 4(e), which is mislabeled. Its rows do not change how the trained model is decoded at evaluation, which is always single-pass. They change how the K rollout candidates are turned into a training signal in Stage II, the alternatives discussed in Section 3.4. "Single-pass" uses one rollout per example and therefore no candidate comparison; "simple ensembling" regresses on the average candidate score; and "self-consistency" selects the most and least consistent candidates as a preference pair, which is our default. This is why its last row coincides with Table 1. We will rename the table "Rollout aggregation in Stage II" with matching row labels.
To isolate the training effect under matched inference budgets, Table R2 reports all methods under single-pass and under the same inference-time self-consistency budget, with latency measured on the same hardware.
Table R2. Latency measured on one H100 80GB in bf16, batch size 1, averaged over the four benchmarks.
| Decoding | Model | MME ↑ | MMMU ↑ | MMB-EN ↑ | MMB-CN ↑ | Latency / sample |
|---|---|---|---|---|---|---|
| Single-pass | BAGEL | 1691.4 | 51.9 | 82.65 | 80.94 | 0.84 s |
| Single-pass | + SFT | 1690.1 | 51.9 | 82.65 | 80.83 | 0.86 s |
| Single-pass | + LatentUMM | 1695.8 | 54.1 | 84.99 | 84.91 | 0.87 s |
| Self-consistency, 5 samples | BAGEL | 1693.0 | 52.6 | 83.04 | 81.37 | 4.03 s |
| Self-consistency, 5 samples | + SFT | 1691.2 | 52.5 | 82.97 | 81.10 | 4.11 s |
| Self-consistency, 5 samples | + LatentUMM | 1696.4 | 54.4 | 85.19 | 85.08 | 4.19 s |
Within each decoding setting LatentUMM leads by a similar margin (MMMU +2.2 under single-pass, +1.8 under self-consistency), so the gain comes from training. Self-consistency costs roughly five times the single-pass latency for every model, which is why none of the main-table results use it. The statement in Appendices E.8–E.9 therefore holds: no rollout, sampling or external embedding is used at inference. The LoRA adapters are merged into the base weights at inference, so the model has exactly BAGEL's architecture and cost; the single-pass differences of at most 0.03 s per sample in Table R2 are within run-to-run timing variation. We will make this precise in the appendix.
Q1. How does the preference objective improve stability?
The reviewer is right that, in principle, the pairwise loss could be reduced by lowering the worst score. We measured whether this happens in practice; it does not.
Existing matched ablation. In Table 4(d), the σ = 0 row makes all K candidates identical, so s⁺ = s⁻ and L_pref has zero gradient, with everything else unchanged. Against it, the default model improves MMMU from 52.0 to 54.1 and MMBench-CN from 80.65 to 84.91. (Section 4.3.2 calls the σ = 0 setting "deterministic inference"; this is a wording error, since σ is the training-time rollout perturbation and all rows use the same single-pass decoding.)
Candidate-score distribution. Table R3 compares the full model with and without L_pref on the same held-out prompts and the same K = 10 perturbation seeds, together with an independent image-level loopback score. Scores are × 100; worst and best are per-prompt minimum and maximum, averaged over prompts.
Table R3.
| Model | Mean s ↑ | p10 ↑ | Worst ↑ | Best ↑ | Best − worst | Image loopback ↑ |
|---|---|---|---|---|---|---|
| Full, λ₂ = 0 | 78.6 | 68.9 | 74.2 | 82.5 | 8.3 | 85.9 |
| Full LatentUMM | 79.8 | 71.0 | 76.4 | 82.9 | 6.5 | 86.6 |
The lower tail improves rather than degrades. With prompt-level paired bootstrap 95% CIs, the worst candidate rises by +2.2 [+1.7, +2.7] and p10 by +2.1 [+1.4, +2.8], while the best candidate is unchanged within noise (+0.4 [−0.1, +0.9]). The narrower gap therefore comes from lifting the worst candidates, and the independent image-level loopback moves in the same direction. We will add this analysis to the paper.
Q2. Does the Wasserstein argument preserve the pairing?
The product coupling in Appendix B does not preserve the dependence between and , but it is also unnecessary: using the natural data pairing instead gives the same conclusion directly.
Let all be computed from the same data pair . Under this data pairing, of Eq. 13 is itself a coupling of the two interpolated distributions. Hence:
where
The second inequality is Minkowski's inequality applied to Eq. 15. Whenever and , the bound is strictly below . Because , the guarantee concerns the paired gap rather than the marginal gap . This is the quantity in Eq. 23, and also the one optimises, since it compares each text with its own image.
Thank you again for your thoughtful and constructive feedback. As the discussion period draws to a close, we would like to kindly follow up to ensure that we have fully addressed your concerns. In response to your suggestions, we have incorporated the requested analyses and further clarified our methodology. If you have any further questions or feedback, we would be grateful if you could let us know. We hope these revisions adequately address your concerns, and would greatly appreciate it if you could reconsider your evaluation in light of these updates.
W1. Implemented objective versus the depicted image cycle
Section 3.3 presents capacity alignment at the conceptual, image level, and Appendix E.1 gives its implemented hidden-state form; we will align the two in Section 3.3 and Figure 2. The implemented objective is a meaningful constraint rather than a free autoencoding task, for three reasons.
Incremental benefit beyond teacher supervision. Table R1a is a stage-wise ablation under one configuration (same initialisation, 10,000 training pairs, data order, seed, LoRA configuration and steps; single-pass decoding). Teacher supervision alone gives almost no gain and lowers MME by 19.8 points. Adding the capacity cycle turns the same supervision into gains on every benchmark, so its contribution is not absorbed by the teacher.
Table R1a. All rows from one training sweep with the same initialisation, 10,000 training pairs, data order, seed, LoRA configuration and steps; full benchmarks; single-pass decoding. For reference, the Table 1 checkpoint, trained with the configuration of Appendices E.1–E.2, scores 1695.8 / 54.1 / 67.2 / 73.6.
Table R1 extends the same matched ablation to matched SFT and to the output-level consistency metrics by re-evaluating the same checkpoints. Rows marked † are taken from the paper (Tables 5 and 7), except the SFT consistency errors, which we measured with the same protocol as Table 7; SFT was trained on the same data. All rows are from a single training seed; further seeds will be added in the final version.
Table R1.
On generated-and-reinterpreted outputs, teacher supervision alone barely reduces drift (T = 4 error 1.82 → 1.71), while adding capacity alignment brings it to 1.33; standard post-training (SFT) does not reduce it at all. The incremental benefit of capacity alignment beyond teacher supervision and standard post-training is therefore established directly on generated and re-interpreted outputs.