- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how generative training improves the comprehension capabilities of unified multimodal models. UMM-JEPA targets the frozen continuous visual representations of SigLIP2, and jointly trains a shared multimodal backbone with comprehension tasks via masked multi-scale next-scale prediction. The training leverages coarse-scale contexts derived from real images, and recursively uses predicted representations during inference; pixel rendering is incorporated separately after representation pretraining. The full model additionally introduces auxiliary representation reconstruction on the comprehension side. The authors validate the scheme through comprehension baselines, data matching controls, comparisons between discrete codes and latent-flow generation, as well as component ablation studies. The comprehension gains are mainly observed in specialized visual tasks such as OCR, document understanding and localization. Linear probing and visual intervention further analyze the representation structure and the utilization of image information, and the comprehension advantages persist after subsequent instruction tuning.
Critical strengths
- The experiments are well-rounded. The authors supplemented matched data controls and compared several generation objectives, demonstrating that the improvement in comprehension ability is not merely attributable to the use of more image-text data.
- Beyond benchmark scores, Table 3 also examines object and spatial information within visual representations. The results are largely consistent with the gains observed on specialized visual tasks, validating the effectiveness of this training paradigm.
Critical weaknesses
- The idea of applying JEPA to umm is interesting, yet the distinction between this work and existing perceptual feature prediction methods remains insufficiently clear. MetaMorph [1] already uses cosine loss to predict SigLIP features, and PUMA [2] also implements coarse-to-fine multi-scale prediction. The paper lacks a baseline of single-scale, non-masked SigLIP2 feature prediction. Thus, it is hard to tell what performance vanilla feature regression can achieve under the setup of this paper, and how much additional benefit the multi-scale design brings. Although experiments with a with/without masking comparison are included, explorations on design choices such as masking ratio and masking strategy are limited.
- Understanding and generation are evaluated after separate fine-tuning stages. Can the final model retain strong performance on both tasks at the same time? References [1] Tong, Shengbang, et al. "Metamorph: Multimodal understanding and generation via instruction tuning." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2025. [2] Fang, Rongyao, et al. "Puma: Empowering unified mllm with multi-granular visual generation." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.
Questions for authors
- Under the same training setup, how well does single-scale SigLIP2 feature prediction without masking perform? How much additional improvement comes from multi-scale prediction? Beyond the with/without masking comparison, are there results showing the effects of masking ratios and strategies?
- Can the understanding and generation scores reported in the paper be achieved simultaneously under the same final checkpoint and adapter configuration? If not, does umm-jepa outperform the baseline?
6 comments
Response to Reviewer ApJt (2/2)
W2&Q2: Understanding and generation are evaluated after separate fine-tuning stages. Can the final model retain strong performance on both tasks at the same time? References [1] Tong, Shengbang, et al. "Metamorph: Multimodal understanding and generation via instruction tuning." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2025. [2] Fang, Rongyao, et al. "Puma: Empowering unified mllm with multi-granular visual generation." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.
Can the understanding and generation scores reported in the paper be achieved simultaneously under the same final checkpoint and adapter configuration? If not, does umm-jepa outperform the baseline?
A: To quantify performance retention and compare the jointly tuned models, we evaluated understanding and generation on the same jointly tuned checkpoint with the same adapter configuration in each run. We compared it both with our task-specific checkpoints, to measure how much performance was retained, and with jointly tuned Qwen-VL, to measure the remaining advantage over the baseline. The single-task rows show the scores of the checkpoints tuned separately for understanding and generation. The joint rows report means ± sample SD over three runs.
| Checkpoint | Text | VisU | VisS | GenEval | DPG |
|---|---|---|---|---|---|
| UMM-JEPA, understanding SFT | 62.76 | 54.15 | 58.39 | — | — |
| UMM-JEPA, generation SFT | — | — | — | 81.20 | 83.75 |
| Qwen-VL, joint tuning | 60.50 ± 0.20 | 51.70 ± 0.20 | 54.30 ± 0.30 | 70.20 ± 0.50 | 72.00 ± 0.40 |
| UMM-JEPA, joint tuning | 62.50 ± 0.20 | 53.90 ± 0.20 | 58.00 ± 0.30 | 80.80 ± 0.40 | 83.30 ± 0.30 |
The joint model scored 0.26 / 0.25 / 0.39 points below the understanding-tuned checkpoint on Text / VisU / VisS and 0.40 / 0.45 points below the generation-tuned checkpoint on GenEval / DPG. It therefore retained most of each capability, with a small observed decrease on each metric relative to its separate single-task reference. Against jointly tuned Qwen-VL, the gains were +2.00 / +2.20 / +3.70 on understanding and +10.60 / +11.30 on generation. Section 4.6 reports the jointly tuned checkpoint’s scores. Table 14 in Appendix J.3 includes both the task-specific checkpoints and jointly tuned Qwen-VL.
Dear Reviewer ApJt,
We are grateful for your careful comments. We hope our responses have adequately addressed your concerns. With the response deadline approaching, please let us know if anything is still a concern; we are happy to elaborate. If your concerns have been addressed, we respectfully invite you to update your assessment accordingly. We look forward to further discussions with you.
Response to Reviewer ApJt (1/2)
We are grateful for your constructive feedback. Thank you for recognizing the value of our experimental controls and for noting that our analysis of object and spatial information aligns with the gains on specialized visual tasks. If the additional experiments and clarifications below address your concerns, we would appreciate it if you would reconsider your rating.
A: To quantify the gains over single-scale feature regression and distinguish them from the effect of masking, we varied both the prediction design and the masking condition. MetaMorph predicts perceptual features, and PUMA uses a coarse-to-fine hierarchy. Our specific question is how much understanding benefits from masked next-scale prediction of the continuous features consumed by the understanding model. In UMM-JEPA the shared backbone predicts the finer SigLIP2 features from coarser states, and pixel rendering is attached after representation pretraining.
All entries are means ± sample SD over three runs. Without masking, moving from single-scale to recursive four-scale prediction added +0.38 Text, +0.21 VisU and +0.28 VisS. With default masking, the corresponding gains were +0.70, +0.31 and +1.08. With both masking and the four targets held fixed, cross-scale conditioning added +0.96 VisS over independent prediction. The plain regression baseline already captures a substantial share of the understanding benefit.
Masking ratio and placement. To isolate the effects of masking ratio and placement, we varied one factor at a time. The default recipe samples the partial masking ratio between 25% and 75%. We compared fixed ratios and, separately, changed how positions were selected across scales while keeping the partial ratio at 25%.
The default recipe had the highest observed means among the tested settings. On VisS, fixing the ratio at 25% or 75% cost 0.31 or 0.41 points. At the same 25% ratio, selecting positions independently across scales cost a further 0.18. These are modest differences relative to the run-to-run spread; the experiment supports the default as a working choice without establishing a uniquely optimal masking rule. Section 2 now discusses how our objective relates to MetaMorph and PUMA. Table 10 in Appendix J.1 summarizes the prediction and masking ablations, and Tables 17 and 18 in Appendix J.5 give the benchmark-level results.