acceptodds
Under review as a conference paper at ICLR 2027

UMM-JEPA: Generative Pretraining via Masked Next-Scale Representation Prediction for Unified Multimodal Models

Abstract

Unified multimodal models can support both visual understanding and generation, yet generative training does not necessarily improve understanding. We argue that this limitation may arise from a mismatch between the representation targeted by generation and the perceptual representation used for understanding. Conventional generation objectives primarily optimize synthesis-oriented spaces such as discrete visual codes or generative latents. We therefore introduce UMM-JEPA, which reformulates generation as progressive prediction of continuous visual-understanding representations. The model constructs a representation hierarchy from coarse to fine, while pixel rendering is decoupled from representation pretraining and attached only afterward. Controlled comparisons show that the representation-prediction objective accounts for most of the understanding improvement, while a data-matched control leaves the group means nearly unchanged. The final UMM-JEPA reaches a 16-benchmark visual mean of 51.71, against 46.36 for the understanding-only Qwen-VL baseline, and the text mean also rises. Compared with discrete-code and latent-flow generation objectives, perceptual representation prediction yields stronger and more consistent transfer to understanding. Linear probes and visual-evidence interventions further reveal richer object and spatial structure and stronger use of image-specific information. These results suggest that generation can benefit understanding when it directly supervises the perceptual representations that understanding consumes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Unified Multimodal Models, JEPA, LLM Pretraining

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_ApJtOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how generative training improves the comprehension capabilities of unified multimodal models. UMM-JEPA targets the frozen continuous visual representations of SigLIP2, and jointly trains a shared multimodal backbone with comprehension tasks via masked multi-scale next-scale prediction. The training leverages coarse-scale contexts derived from real images, and recursively uses predicted representations during inference; pixel rendering is incorporated separately after representation pretraining. The full model additionally introduces auxiliary representation reconstruction on the comprehension side. The authors validate the scheme through comprehension baselines, data matching controls, comparisons between discrete codes and latent-flow generation, as well as component ablation studies. The comprehension gains are mainly observed in specialized visual tasks such as OCR, document understanding and localization. Linear probing and visual intervention further analyze the representation structure and the utilization of image information, and the comprehension advantages persist after subsequent instruction tuning.

Critical strengths
  1. The experiments are well-rounded. The authors supplemented matched data controls and compared several generation objectives, demonstrating that the improvement in comprehension ability is not merely attributable to the use of more image-text data.
  2. Beyond benchmark scores, Table 3 also examines object and spatial information within visual representations. The results are largely consistent with the gains observed on specialized visual tasks, validating the effectiveness of this training paradigm.
Critical weaknesses
  1. The idea of applying JEPA to umm is interesting, yet the distinction between this work and existing perceptual feature prediction methods remains insufficiently clear. MetaMorph [1] already uses cosine loss to predict SigLIP features, and PUMA [2] also implements coarse-to-fine multi-scale prediction. The paper lacks a baseline of single-scale, non-masked SigLIP2 feature prediction. Thus, it is hard to tell what performance vanilla feature regression can achieve under the setup of this paper, and how much additional benefit the multi-scale design brings. Although experiments with a with/without masking comparison are included, explorations on design choices such as masking ratio and masking strategy are limited.
  2. Understanding and generation are evaluated after separate fine-tuning stages. Can the final model retain strong performance on both tasks at the same time? References [1] Tong, Shengbang, et al. "Metamorph: Multimodal understanding and generation via instruction tuning." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2025. [2] Fang, Rongyao, et al. "Puma: Empowering unified mllm with multi-granular visual generation." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.
Questions for authors
  1. Under the same training setup, how well does single-scale SigLIP2 feature prediction without masking perform? How much additional improvement comes from multi-scale prediction? Beyond the with/without masking comparison, are there results showing the effects of masking ratios and strategies?
  2. Can the understanding and generation scores reported in the paper be achieved simultaneously under the same final checkpoint and adapter configuration? If not, does umm-jepa outperform the baseline?
6 comments
AuthorsOpenReview ↗

Response to Reviewer ApJt (1/2)

We are grateful for your constructive feedback. Thank you for recognizing the value of our experimental controls and for noting that our analysis of object and spatial information aligns with the gains on specialized visual tasks. If the additional experiments and clarifications below address your concerns, we would appreciate it if you would reconsider your rating.

W1&Q1: The idea of applying JEPA to umm is interesting, yet the distinction between this work and existing perceptual feature prediction methods remains insufficiently clear. MetaMorph [1] already uses cosine loss to predict SigLIP features, and PUMA [2] also implements coarse-to-fine multi-scale prediction. The paper lacks a baseline of single-scale, non-masked SigLIP2 feature prediction. Thus, it is hard to tell what performance vanilla feature regression can achieve under the setup of this paper, and how much additional benefit the multi-scale design brings. Although experiments with a with/without masking comparison are included, explorations on design choices such as masking ratio and masking strategy are limited.

Under the same training setup, how well does single-scale SigLIP2 feature prediction without masking perform? How much additional improvement comes from multi-scale prediction? Beyond the with/without masking comparison, are there results showing the effects of masking ratios and strategies?

A: To quantify the gains over single-scale feature regression and distinguish them from the effect of masking, we varied both the prediction design and the masking condition. MetaMorph predicts perceptual features, and PUMA uses a coarse-to-fine hierarchy. Our specific question is how much understanding benefits from masked next-scale prediction of the continuous features consumed by the understanding model. In UMM-JEPA the shared backbone predicts the finer SigLIP2 features from coarser states, and pixel rendering is attached after representation pretraining.

SigLIP2 feature predictionTextVisUVisS
Single scale, no mask59.64 ± 0.2448.21 ± 0.0251.10 ± 0.04
Recursive four scales, no mask60.02 ± 0.2248.42 ± 0.0151.38 ± 0.04
Single scale, default masking59.91 ± 0.0948.51 ± 0.0251.98 ± 0.06
Independent four scales, default masking60.01 ± 0.3748.53 ± 0.0652.10 ± 0.06
Recursive four scales, default masking60.61 ± 0.6348.82 ± 0.2453.06 ± 0.38

All entries are means ± sample SD over three runs. Without masking, moving from single-scale to recursive four-scale prediction added +0.38 Text, +0.21 VisU and +0.28 VisS. With default masking, the corresponding gains were +0.70, +0.31 and +1.08. With both masking and the four targets held fixed, cross-scale conditioning added +0.96 VisS over independent prediction. The plain regression baseline already captures a substantial share of the understanding benefit.

Masking ratio and placement. To isolate the effects of masking ratio and placement, we varied one factor at a time. The default recipe samples the partial masking ratio between 25% and 75%. We compared fixed ratios and, separately, changed how positions were selected across scales while keeping the partial ratio at 25%.

Partial masking in UMM-JEPA-coreTextVisUVisS
Default ratio and placement60.61 ± 0.6348.82 ± 0.2453.06 ± 0.38
Fixed 25%, default placement60.39 ± 0.3748.74 ± 0.1752.75 ± 0.30
Fixed 75%, default placement60.34 ± 0.3448.70 ± 0.1452.65 ± 0.27
Fixed 25%, independent placement across scales60.34 ± 0.5948.67 ± 0.1352.57 ± 0.23

The default recipe had the highest observed means among the tested settings. On VisS, fixing the ratio at 25% or 75% cost 0.31 or 0.41 points. At the same 25% ratio, selecting positions independently across scales cost a further 0.18. These are modest differences relative to the run-to-run spread; the experiment supports the default as a working choice without establishing a uniquely optimal masking rule. Section 2 now discusses how our objective relates to MetaMorph and PUMA. Table 10 in Appendix J.1 summarizes the prediction and masking ablations, and Tables 17 and 18 in Appendix J.5 give the benchmark-level results.

AuthorsOpenReview ↗

Response to Reviewer ApJt (2/2)

W2&Q2: Understanding and generation are evaluated after separate fine-tuning stages. Can the final model retain strong performance on both tasks at the same time? References [1] Tong, Shengbang, et al. "Metamorph: Multimodal understanding and generation via instruction tuning." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2025. [2] Fang, Rongyao, et al. "Puma: Empowering unified mllm with multi-granular visual generation." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

Can the understanding and generation scores reported in the paper be achieved simultaneously under the same final checkpoint and adapter configuration? If not, does umm-jepa outperform the baseline?

A: To quantify performance retention and compare the jointly tuned models, we evaluated understanding and generation on the same jointly tuned checkpoint with the same adapter configuration in each run. We compared it both with our task-specific checkpoints, to measure how much performance was retained, and with jointly tuned Qwen-VL, to measure the remaining advantage over the baseline. The single-task rows show the scores of the checkpoints tuned separately for understanding and generation. The joint rows report means ± sample SD over three runs.

CheckpointTextVisUVisSGenEvalDPG
UMM-JEPA, understanding SFT62.7654.1558.39——
UMM-JEPA, generation SFT———81.2083.75
Qwen-VL, joint tuning60.50 ± 0.2051.70 ± 0.2054.30 ± 0.3070.20 ± 0.5072.00 ± 0.40
UMM-JEPA, joint tuning62.50 ± 0.2053.90 ± 0.2058.00 ± 0.3080.80 ± 0.4083.30 ± 0.30

The joint model scored 0.26 / 0.25 / 0.39 points below the understanding-tuned checkpoint on Text / VisU / VisS and 0.40 / 0.45 points below the generation-tuned checkpoint on GenEval / DPG. It therefore retained most of each capability, with a small observed decrease on each metric relative to its separate single-task reference. Against jointly tuned Qwen-VL, the gains were +2.00 / +2.20 / +3.70 on understanding and +10.60 / +11.30 on generation. Section 4.6 reports the jointly tuned checkpoint’s scores. Table 14 in Appendix J.3 includes both the task-specific checkpoints and jointly tuned Qwen-VL.

Dear Reviewer ApJt,

We are grateful for your careful comments. We hope our responses have adequately addressed your concerns. With the response deadline approaching, please let us know if anything is still a concern; we are happy to elaborate. If your concerns have been addressed, we respectfully invite you to update your assessment accordingly. We look forward to further discussions with you.

Reviewer_LqmaOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes UMM-JEPA, a generative pretraining framework for unified multimodal models that aims to improve visual understanding through representation prediction. Instead of predicting discrete visual tokens or generative latents, UMM-JEPA performs masked, coarse-to-fine prediction of continuous perceptual representations, while decoupling pixel rendering from representation pretraining through a lightweight connector and a pretrained diffusion model. Experiments show improvements in visual understanding over understanding-only and alternative generative baselines, while achieving competitive image generation performance. Ablation studies and representation analyses further suggest that the proposed training objective encourages richer visual representations and stronger reliance on visual information during understanding.

Critical strengths
  1. The paper proposes a generative pretraining paradigm that improves visual understanding through perceptual representation prediction. The motivation is clear, and the proposed method provides a meaningful approach to generation-to-understanding transfer learning.
  2. UMM-JEPA achieves consistent gains across visual understanding tasks, with notable improvements over Qwen-VL on ChartQA (41.20 → 56.64, +15.44), DocVQA (31.36 → 43.09, +11.73), and RefCOCO (32.62 → 46.53, +13.91), and maintains competitive image generation performance.
Critical weaknesses

As reported in Section 4.3, UMM-JEPA-core only slightly outperforms UMM-VQ on Visual Understanding (48.55 vs. 48.46) and Visual Specialty (52.62 vs. 51.83). Given these small differences, results from multiple runs would help establish whether the improvements are consistent.

Questions for authors

Please see weakness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Lqma

Thank you for reading our paper so carefully. We appreciate your recognition of the motivation behind our approach and the consistent gains in visual understanding alongside competitive image generation performance.

W1: As reported in Section 4.3, UMM-JEPA-core only slightly outperforms UMM-VQ on Visual Understanding (48.55 vs. 48.46) and Visual Specialty (52.62 vs. 51.83). Given these small differences, results from multiple runs would help establish whether the improvements are consistent.

Q1: Please see weakness.

A: To assess the size and consistency of the visual gains, we compared UMM-JEPA-core with UMM-VQ on seven Visual Understanding and nine Visual Specialty benchmarks. The table reports three-run means in points, with sample SDs where run-level scores are available. The UMM-VQ VisU mean is the average of the seven benchmark means.

GroupUMM-VQUMM-JEPA-coreUMM-JEPA-core − UMM-VQ
VisU48.3948.82 ± 0.24+0.43
VisS51.79 ± 0.0353.06 ± 0.38+1.26 ± 0.41

UMM-JEPA-core had higher mean scores in both groups, with a smaller mean gain on Visual Understanding. The gain on Visual Specialty was positive in all three comparisons. Appendix J.1, Table 9 lists the individual VisS differences. Runs are compared by index because seed matching across methods was undocumented. Section 4.3 reports the group means for both models. Tables 15 and 16 in Appendix J.4 give their benchmark-level scores.

Reviewer_zZvhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes UMM-JEPA, and claims that visual generation can improve understanding by predicting the perceptual representations used for understanding. UMM-JEPA jointly trains a shared backbone with language modeling and masked next-scale prediction of SigLIP2 features, and adds a diffusion renderer after representation pretraining. Experiments compare different generation objectives and report improvements in understanding and image generation.

Critical strengths

The focus of this paper is meaningful. The understanding results after generative pretraining are promising, and the gains remain after instruction tuning. The representation probes provide further evidence that the learned visual states preserve useful object and spatial information.

Critical weaknesses
  1. The specific contribution of masked next-scale prediction needs stronger evidence. Representation Forcing (Wang et al., 2026), already discussed in the paper, also makes visual representations generation targets and reports benefits to both understanding and generation. A comparison using the same SigLIP2 targets without multi-scale dependencies would make the design motivation more convincing.

  2. The practical benefit of a unified model needs a clearer joint evaluation. The generation results in Table 4 involve a pretrained FLUX.2 renderer and further LoRA/instruction tuning, including adaptation of the language backbone, while the understanding results are reported at pretraining or understanding-tuned checkpoints. It remains a gap whether the same final model retains both reported capabilities. I would like to see understanding and generation evaluated on the same final backbone.

Questions for authors

See the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer zZvh

We appreciate your thoughtful feedback. Thank you for highlighting both the persistence of the visual understanding gains after instruction tuning and the evidence from our probes that the learned representations retain object and spatial information. If the additional experiments and clarifications below address your concerns, we would appreciate it if you would reconsider your rating.

W1: The specific contribution of masked next-scale prediction needs stronger evidence. Representation Forcing (Wang et al., 2026), already discussed in the paper, also makes visual representations generation targets and reports benefits to both understanding and generation. A comparison using the same SigLIP2 targets without multi-scale dependencies would make the design motivation more convincing.

A: To isolate the contribution of cross-scale conditioning, we trained a control that predicts the same four scales of SigLIP2 features independently. Both models use perceptual generation targets, as in Representation Forcing. Only UMM-JEPA-core uses coarser visual states to predict finer features. With default masking, the three-run understanding results were:

Prediction with SigLIP2 targetsTextVisUVisS
Single scale, default masking59.91 ± 0.0948.51 ± 0.0251.98 ± 0.06
Four independent scales, default masking60.01 ± 0.3748.53 ± 0.0652.10 ± 0.06
Four recursive scales, default masking (UMM-JEPA-core)60.61 ± 0.6348.82 ± 0.2453.06 ± 0.38

Compared with the independent four-scale model, UMM-JEPA-core gained +0.60 Text, +0.29 VisU and +0.96 VisS. Against single-scale prediction, the gains were +0.70, +0.31 and +1.08. The independent model already learns useful perceptual features; the additional VisS gain measures how much the coarse-to-fine design contributed in this comparison. Section 4.3 reports the gain from recursive conditioning. Table 10 in Appendix J.1 compares the three prediction designs, and Table 17 in Appendix J.5 gives benchmark-level results for the controls.

W2: The practical benefit of a unified model needs a clearer joint evaluation. The generation results in Table 4 involve a pretrained FLUX.2 renderer and further LoRA/instruction tuning, including adaptation of the language backbone, while the understanding results are reported at pretraining or understanding-tuned checkpoints. It remains a gap whether the same final model retains both reported capabilities. I would like to see understanding and generation evaluated on the same final backbone.

A: To test whether one final model retains both capabilities, we evaluated both tasks on the same jointly tuned checkpoint with the same adapter configuration in each run. The single-task rows show the scores of the checkpoints tuned separately for understanding and generation. The joint rows report means ± sample SD over three runs.

CheckpointTextVisUVisSGenEvalDPG
UMM-JEPA, understanding SFT62.7654.1558.39——
UMM-JEPA, generation SFT———81.2083.75
Qwen-VL, joint tuning60.50 ± 0.2051.70 ± 0.2054.30 ± 0.3070.20 ± 0.5072.00 ± 0.40
UMM-JEPA, joint tuning62.50 ± 0.2053.90 ± 0.2058.00 ± 0.3080.80 ± 0.4083.30 ± 0.30

The joint model scored 0.26 / 0.25 / 0.39 points below the understanding-tuned checkpoint on Text / VisU / VisS and 0.40 / 0.45 points below the generation-tuned checkpoint on GenEval / DPG. It therefore retained most of each capability, with a small observed decrease on each metric relative to its separate single-task reference. Against jointly tuned Qwen-VL, the gains were +2.00 / +2.20 / +3.70 on understanding and +10.60 / +11.30 on generation. Section 4.6 reports both capabilities for the same checkpoint, and Table 14 in Appendix J.3 gives the full comparison with task-specific tuning.

Dear Reviewer zZvh,

We are grateful for your careful comments. We hope our responses have adequately addressed your concerns. With the response deadline approaching, please let us know if anything is still a concern; we are happy to elaborate. If your concerns have been addressed, we respectfully invite you to update your assessment accordingly. We look forward to further discussions with you.

Dear Reviewer zZvh,

As the discussion period draws to a close, we would like to thank you again for your thoughtful questions. We hope the comparison of independent and coarse-to-fine prediction clarifies the contribution of cross-scale conditioning, and that the joint-tuning results address your question about retaining both understanding and generation in one checkpoint. If you feel your concerns have been addressed, we would be grateful if you could revisit your rating.

Reviewer_pUWXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes UMM-JEPA, a unified multimodal pretraining framework that aims to improve visual understanding through generative representation learning. The authors hypothesize that the limited transfer from visual generation to understanding arises from a mismatch between synthesis-oriented generation targets and the perceptual representations used for understanding. To address this issue, UMM-JEPA introduces masked next-scale prediction of continuous SigLIP2 representations across a 64, 128, 256, 512 hierarchy. Understanding and representation generation share a Qwen3-based multimodal backbone, while pixel rendering is decoupled from core pretraining and subsequently attached through a trainable connector and a pretrained diffusion renderer.

The authors evaluate the framework on 16 visual understanding benchmarks, four text benchmarks, and two image-generation benchmarks. Comparisons include an understanding-only baseline, a data-matched control, and alternative discrete-code and latent-flow generation objectives. Additional ablations examine attention, masking, and auxiliary reconstruction, while linear probes and visual-evidence interventions investigate changes in learned representations and visual information use.

Critical strengths

1. A well-motivated formulation of generation-to-understanding transfer.

The paper addresses an important question in unified multimodal learning: whether and how generative objectives can improve visual understanding. Its central hypothesis, that the representation targeted by generation influences the transfer to understanding, is meaningful. The proposed framework provides a coherent implementation through shared multimodal parameters, masked coarse-to-fine representation prediction, and decoupled image rendering.

2. Relatively comprehensive experimental evaluation and controlled comparisons.

The paper evaluates understanding across a broad set of benchmarks and includes several useful controls. The additional linear probes and visual-evidence interventions go beyond aggregate performance improvements and provide supporting evidence that representation-generative training changes the learned visual states. The instruction-tuning experiments also help establish that the reported advantages are not confined to the pretraining checkpoint.

Critical weaknesses

1. The empirical evidence does not sufficiently isolate the benefit of the proposed perceptual representation target.

The central claim of the paper is that generation improves understanding more effectively when its prediction target is aligned with the perceptual representation used for understanding. While the experiments show improvements over the understanding-only baseline, the evidence for the particular importance of this target choice is not sufficient.

In Table 2, the UMM-JEPA-core model improves over the data-matched Qwen-VL control by 0.99, 1.28, and 6.97 points on Text, Visual Understanding, and Visual Specialty, respectively. However, the closest alternative generation-objective control, UMM-VQ, already achieves 59.64/48.46/51.83 on these three metrics, compared with 59.89/48.55/52.62 for UMM-JEPA-core. The differences attributable to the proposed representation-prediction paradigm are therefore only +0.25/+0.09/+0.79 points at the group-mean level.

This suggests that much of the improvement over the understanding-only baseline is shared by other generative objectives, rather than uniquely arising from prediction in the perceptual representation space. The final UMM-JEPA further benefits from auxiliary reconstruction, which adds +1.40/+0.42/+1.22 points over UMM-JEPA-core. Consequently, the stronger performance of the final system cannot be attributed solely to its choice of generative prediction target.

Moreover, several of the largest benchmark improvements relative to Qwen-VL, including ChartQA and DocVQA, are also largely reproduced by UMM-VQ and Show-o2. Although some tasks, particularly RefCOCO, show a clearer advantage for the full UMM-JEPA system, the contribution of auxiliary reconstruction makes it difficult to isolate the underlying target-space effect.

The paper also does not report uncertainty estimates or repeated-run variability for the relatively small differences between UMM-JEPA-core and UMM-VQ. Given the scale and complexity of the training procedure, it is unclear whether these differences establish a robust advantage for perceptual representation prediction.

More directly controlled comparisons, such as alternative representation targets under the same architecture, masking, and prediction objective, would help substantiate the central hypothesis.

Overall, I find the evidence convincing that additional generative representation supervision can improve understanding, but less convincing that the specific target-alignment principle proposed in this paper is the primary factor responsible for these improvements.

2. The role of progressive representation generation and its connection to the claimed understanding improvements remain insufficiently established.

The proposed framework combines masked representation prediction, a progressive scale hierarchy, teacher-forced training, and recursive generation at inference. However, the experiments do not adequately isolate the contribution of the progressive prediction mechanism itself.

In particular, the training procedure conditions on ground-truth-derived coarse visual states, whereas inference recursively conditions on model-predicted states. This introduces a train–inference discrepancy that is acknowledged in Section 3.2 but not sufficiently evaluated. The reported scale-wise prediction errors in Appendix H are informative, yet the paper does not directly establish how error propagation across predicted scales affects downstream generation or whether recursive prediction is necessary for the observed understanding gains.

The theoretical analysis also provides limited support for the stronger conclusions. Proposition 1 follows directly from the assumed Lipschitz continuity of the downstream mapping, while Proposition 2 uses the standard conditional-expectation characterization of the optimal squared-loss predictor. Lemma 1 establishes that conditioning on additional ground-truth coarse states cannot increase optimal prediction risk. However, this result does not directly establish the benefit of conditioning on model-predicted states during inference, nor does it demonstrate that progressive prediction is preferable to a simpler non-progressive representation objective.

A more important empirical limitation is the absence of a controlled comparison between the proposed hierarchical prediction mechanism and simpler alternatives, such as single-scale masked representation prediction or independent multi-scale prediction without recursive feedback. Without such controls, it remains difficult to determine whether the improvements arise from the progressive generative formulation or primarily from the additional perceptual representation supervision.

The image-generation results also require careful interpretation. UMM-JEPA uses a pretrained FLUX.2 renderer together with a separately trained connector, diffusion adapters, and generation instruction tuning. Although the comparisons with Qwen-VL and Text-only Qwen use the same rendering recipe, UMM-VQ and Show-o2 retain different native generation pathways. Therefore, the cross-paradigm generation results do not isolate the effectiveness of the proposed representation-prediction mechanism under a common renderer.

Furthermore, Appendix C states that the 512-scale representation is not included in the connector conditioning sequence. This raises questions about the contribution of the final prediction scale to actual image synthesis, despite its prominent role in the proposed hierarchy.

Questions for authors

Table 2 shows that UMM-JEPA-core improves over UMM-VQ by only +0.25/+0.09/+0.79 points on the three reported group means. Could the authors provide uncertainty estimates or additional evidence demonstrating that these improvements are robust? More importantly, could the authors clarify which controlled experiments specifically establish that alignment with the understanding representation?

The theoretical analysis assumes access to ground-truth coarse representations, while inference relies on recursively generated states. Could the authors clarify how Lemma 1 supports the latter setting? Could the authors also quantify the impact of recursive prediction errors across the four scales and compare the proposed approach against single-scale or non-progressive multi-scale representation prediction?

Appendix C states that the 512-scale representation is not included in the conditioning sequence provided to the diffusion renderer. Could the authors clarify the role of this final scale in the generation pipeline and whether removing its prediction loss affects understanding or generation performance? Additionally, since the generation comparisons involve different rendering architectures, could the authors provide further evidence separating the contributions of the learned representation pathway from the pretrained diffusion renderer and its subsequent adaptation?

Finally, the manuscript refers to the additional training budget as both "10B image–text pairs" and "10B image–text tokens" in different sections. Could the authors clarify the exact unit and amount of additional training data used in the controlled comparisons?

5 comments
AuthorsOpenReview ↗

Response to Reviewer pUWX (1/3)

Thank you for your insightful comments. We appreciate your positive assessment of our motivation and framework design, as well as the breadth of our evaluation and the use of controlled comparisons. If the additional experiments and clarifications below address your concerns, we would appreciate it if you would reconsider your rating.

W1&Q1: The empirical evidence does not sufficiently isolate the benefit of the proposed perceptual representation target. The central claim of the paper is that generation improves understanding more effectively when its prediction target is aligned with the perceptual representation used for understanding. ... Overall, I find the evidence convincing that additional generative representation supervision can improve understanding, but less convincing that the specific target-alignment principle proposed in this paper is the primary factor responsible for these improvements.

Table 2 shows that UMM-JEPA-core improves over UMM-VQ by only +0.25/+0.09/+0.79 points on the three reported group means. Could the authors provide uncertainty estimates or additional evidence demonstrating that these improvements are robust? More importantly, could the authors clarify which controlled experiments specifically establish that alignment with the understanding representation?

A: To assess whether the gains recur across training runs, we compared UMM-JEPA-core with UMM-VQ and measured the additional gain from auxiliary reconstruction by comparing UMM-JEPA with UMM-JEPA-core. The table reports mean differences over three training runs, in points, with sample SDs where run-level differences are available. The VisU gain of 0.43 points is the difference between group means. Its sample SD cannot be calculated from the available summaries because the corrected run-level differences are unavailable.

GroupUMM-JEPA-core − UMM-VQUMM-JEPA − UMM-JEPA-core
Text+0.82 ± 0.49+0.59 ± 0.70
VisU+0.43+0.18 ± 0.21
VisS+1.26 ± 0.41+0.53 ± 0.60

UMM-JEPA-core had higher mean scores than UMM-VQ in all three groups. The gains on Text and Visual Specialty were positive in all three comparisons, with the largest mean gain on Visual Specialty. The contribution of auxiliary reconstruction varied by task. On RefCOCO, for example, UMM-VQ, UMM-JEPA-core and UMM-JEPA scored 37.94, 40.40 and 44.63, respectively. Appendix J.1, Table 9 lists the available run-level differences. Runs are compared by index because seed matching across methods was undocumented.

Control for target choice. The UMM-VQ comparison changes both the target and the prediction loss. To examine whether using the understanding encoder’s features as the generation target helps, we kept SigLIP2 on the understanding side and compared DINO and SigLIP2 as generation targets in the recursive four-scale configuration. Scores below are means ± sample SD over three runs.

Generation targetTextVisUVisS
DINO features60.00 ± 0.2548.51 ± 0.0452.02 ± 0.02
SigLIP2 features used by understanding60.61 ± 0.6348.82 ± 0.2453.06 ± 0.38

Using the understanding encoder’s target gave +0.61 / +0.31 / +1.04 points. DINO and UMM-VQ both outperform the data-matched understanding-only control on VisS (52.02 and 51.79 versus 45.68). We therefore attribute the broad gain to additional generative supervision and a smaller, measurable increment to the choice of target in this setup. Since the two encoders also differ in the features they encode, this experiment alone cannot separate target alignment from every other property of those features. The abstract now states that predicting perceptual representations during generative pretraining improves visual understanding. Section 4.3 examines how the choice of prediction target contributes to this improvement. Table 10 in Appendix J.1 compares DINO and SigLIP2 targets.

AuthorsOpenReview ↗

Response to Reviewer pUWX (2/3)

W2&Q2: The role of progressive representation generation and its connection to the claimed understanding improvements remain insufficiently established. ... This raises questions about the contribution of the final prediction scale to actual image synthesis, despite its prominent role in the proposed hierarchy.

The theoretical analysis assumes access to ground-truth coarse representations, while inference relies on recursively generated states. ... Could the authors clarify the exact unit and amount of additional training data used in the controlled comparisons?

A: Propagation of coarse-scale errors. Training supplies coarse context derived from the image. At inference each predicted coarse state is available to subsequent scales. To measure sensitivity to errors in that context, we perturbed the predicted 64-scale state and evaluated the resulting trajectory and rendered images with the sampling seed fixed. Each error entry is the excess over the unperturbed condition at the same scale.

Perturbation at 64Δ error at 64At 128At 256At 512GenEvalDPG
None000081.2383.75
Light0.0400.0190.0100.00681.0283.53
Medium0.1000.0520.0270.01580.1082.81
Heavy0.2000.1140.0650.03977.8280.92

The additional error diminished along the trajectory at all three strengths. The final image remained sensitive to the initial disturbance: the light perturbation cost 0.21 GenEval and 0.22 DPG points, while the heavy perturbation cost 3.41 and 2.83. This quantifies the effect of an imposed coarse-state error; the unperturbed trajectory can still contain its own natural prediction error. Table 11 in Appendix J.2 reports the scale-wise errors and final generation scores.

Scope of Lemma 1. Under squared loss, the optimal expected fine-scale prediction error cannot increase when the predictor receives true coarse features in addition to the caption. This motivates conditioning on coarse structure during training. At inference we substitute an estimated coarse state. The lemma gives no guarantee that this substitution improves prediction or that recursive errors contract. Propositions 1 and 2 describe sensitivity under a Lipschitz assumption and the optimal conditional predictor, respectively. The perturbation test above addresses inference sensitivity empirically. The following training ablation addresses the distinct question of transfer to understanding. Section 3.1 now states the assumption of observed coarse states and explains its implications for inference.

Value of cross-scale conditioning. To distinguish the benefit of predicting perceptual features from the additional benefit of conditioning on coarser states, we kept the SigLIP2 targets and default masking and compared three prediction designs. The table reports means ± sample SD over three runs.

Prediction with SigLIP2 targetsTextVisUVisS
Single scale, default masking59.91 ± 0.0948.51 ± 0.0251.98 ± 0.06
Four independent scales, default masking60.01 ± 0.3748.53 ± 0.0652.10 ± 0.06
Four recursive scales, default masking (UMM-JEPA-core)60.61 ± 0.6348.82 ± 0.2453.06 ± 0.38

With the same four target scales, allowing finer predictions to use coarser states improved Text, VisU and VisS by 0.60, 0.29 and 0.96 points. The single-scale model also benefited from representation supervision. The comparison identifies an incremental benefit for progressive conditioning, most clearly on VisS. Section 4.3 reports this gain, and Table 10 in Appendix J.1 compares all three prediction designs.

What reaches the renderer. To assess whether the generation advantage persists under a common rendering recipe, we compared Qwen-VL and UMM-JEPA using the same FLUX.2 rendering and adaptation procedure. Qwen-VL scored 70.57 / 72.50 on GenEval / DPG and UMM-JEPA scored 81.23 / 83.75. The adapters were fitted separately, so we also tested a fixed trained renderer. Replacing the three conditioning representations with those from a different prompt reduced the scores from 81.23 / 83.75 to 68.40 / 73.00, drops of 12.83 / 10.75. Thus the content of the learned visual conditions affects the output even with the trained rendering path held fixed. This intervention does not partition the total score between representation learning and renderer adaptation. UMM-VQ and Show-o2 use their native decoders and serve as end-to-end generation comparisons. Section 4.5 identifies each rendering pathway, and Table 13 in Appendix J.2 reports the fixed-renderer intervention.

AuthorsOpenReview ↗

Response to Reviewer pUWX (3/3)

Role of 512-scale supervision. The 256-scale states are trained to predict 512-scale features. These predictions supply a training loss; the renderer reads states at 64, 128 and 256. To test whether that supervision improves the states the renderer actually reads, we removed only the 256→512 loss from UMM-JEPA-core and evaluated both understanding and generation. The table reports three-run means.

UMM-JEPA-core training configurationTextVisUVisSGenEvalDPG
Without the 512-scale loss60.5248.8052.7780.0782.71
With the 512-scale loss60.6148.8253.0680.8083.31

The largest understanding difference was on VisS (+0.29); VisU changed by only +0.02. The generation gains were +0.73 GenEval and +0.60 DPG. These are indirect benefits of supervising the states used by the renderer. Section 4.5 specifies the renderer inputs, and Table 12 in Appendix J.2 reports the effect of removing the 512-scale loss.

Training budget. The additional budget in the controlled comparisons was 10 billion image–text tokens on top of 10B text and 45B multimodal tokens, for 65B tokens in total in the generation-bearing runs. The budget is measured in tokens throughout. The caption of Table 1 and Section 4.3 specify the unit for this additional budget.

Comments on the paper

1 comment

Dear Area Chair,

We thank the reviewers for their thoughtful and constructive feedback. We appreciate their recognition of the following strengths:

  1. Well-motivated research question (pUWX, zZvh). Both reviewers recognized the value of studying how generative pretraining can improve visual understanding in unified multimodal models.
  2. Broad evaluation and controlled comparisons (pUWX, ApJt). The reviewers highlighted the breadth of the evaluation, the data-matched control, and the comparisons across generation objectives.
  3. Visual understanding gains and supporting analysis (Lqma, zZvh, ApJt). Lqma highlighted gains on ChartQA, DocVQA, and RefCOCO. zZvh noted that the gains persist after instruction tuning, and both zZvh and ApJt highlighted the evidence from object and spatial probes.

The main concerns and our responses are summarized below.

  1. Contribution of masked next-scale prediction (pUWX, zZvh, ApJt). We added single-scale and independent four-scale controls using SigLIP2 targets and the same default masking. With all four target scales held fixed, conditioning finer predictions on coarser states improved the three-run mean by 0.29 points on Visual Understanding and 0.96 on Visual Specialty.

  2. Consistency of gains and choice of generation target (pUWX, Lqma). Across three runs, UMM-JEPA-core outperformed UMM-VQ by an average of 0.43 points on Visual Understanding and 1.26 points on Visual Specialty. To address pUWX’s further question about target choice, we compared DINO and SigLIP2 as generation targets while keeping the understanding encoder and prediction design fixed. Both models scored above the data-matched understanding-only control (45.68) on Visual Specialty. Using SigLIP2 features from the understanding encoder as the generation target raised the score from 52.02 with DINO targets to 53.06, a gain of 1.04 points.

  3. Understanding and generation in the same checkpoint (zZvh, ApJt). In each run, we evaluated both capabilities using a single jointly tuned UMM-JEPA checkpoint and the same adapter configuration. Across three runs, the mean score on each of the five metrics was within 0.45 points of its task-specific reference. Jointly tuned UMM-JEPA also outperformed jointly tuned Qwen-VL on all five metrics, including gains of 3.70 points on Visual Specialty and 10.60 on GenEval (Appendix J.3, Table 14).


We also addressed two reviewer-specific requests:

  1. Recursive prediction and rendering (pUWX). We clarified the assumptions of Lemma 1, measured how coarse-scale perturbations affect generation, and tested the contribution of the conditioning features with the trained renderer held fixed.
  2. Related methods and masking choices (ApJt). We clarified how our approach relates to MetaMorph and PUMA, added the requested single-scale unmasked baseline, and evaluated masking ratio and placement.

During the rebuttal period, we addressed the reviewers’ concerns through additional experiments and clarifications. We hope our responses will help the reviewers reassess the paper.

We believe these additions substantially improve the clarity and completeness of the paper.

Best regards,

The authors of Submission 42727

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.