acceptodds
Under review as a conference paper at ICLR 2027

Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System

Abstract

While unified multimodal models (UMMs) jointly perform visual understanding and generation within a single model, functional unification does not guarantee learning synergy: the two objectives may reinforce each other, compete for capacity, or merely coexist. We investigate their relationship at the representation, task, and system levels in a controlled, structurally native setting without pretrained vision priors. At the representation level, we find that each objective provides useful signal to the other: generation enriches the visual features learned for understanding, while understanding strengthens vision–language alignment for generation. However, when both objectives are forced through the same computation path, one tends to dominate. A task-decoupled architecture that specializes conflicting visual computation while preserving semantic interaction avoids this asymmetric degradation. At the task level, through three case studies, we find positive bidirectional transfer when understanding and generation tasks rely on shared knowledge. At the system level, we show that an end-to-end UMM outperforms a matched planner–executor pipeline on complex tasks that explicitly require both image understanding and generation. Together, these results show that the value of UMMs extends beyond a unified interface: appropriate specialization, shared task knowledge, and end-to-end optimization can turn coexistence into synergy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Unified Multimodal Models, Multimodal Large Language Models, Native Multimodal Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CEUeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when visual understanding and generation reinforce or interfere with each other in native unified multimodal models. Through controlled experiments at the representation, task, and system levels, it finds that task-decoupled computation helps balance the two capabilities, related tasks can benefit from joint training, and end-to-end unification can outperform a planner–executor pipeline on reasoning-intensive image editing.

Critical strengths
  1. The paper systematically studies understanding–generation interactions at the representation, task, and system levels, an important research question for unified multimodal models.
  2. The controlled architectural comparisons and task-specific experiments provide useful insights into when joint training leads to synergy or interference.
Critical weaknesses
  1. All experiments use a Qwen3-1.7B backbone. It remains unclear whether the observed interference and benefits of task decoupling persist at larger model scales.
  2. The paper finds that generation supervision improves visual representations for understanding. Related work has also explored this direction: ROSS [1] improves understanding through image reconstruction supervision, while UniMRG [2] enhances spatial understanding through multi-representation generation. Discussing these connections and clarifying the differences would strengthen the paper’s positioning and rigor.
  3. Task-decoupled MoT architectures are already used in leading models such as BAGEL [3] and Cosmos 3 [4]. The finding that separating understanding and generation computation helps mitigate interference is therefore somewhat expected.

References:

[1] Reconstructive Visual Instruction Tuning. 2024.

[2] Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation. 2026.

[3] Emerging Properties in Unified Multimodal Pretraining. 2025.

[4] Cosmos 3: Omnimodal World Models for Physical AI. 2026.

Questions for authors
  1. Do you expect the observed interference and benefits of task decoupling to persist with larger backbones? Could increased model capacity alleviate the trade-offs observed in your experiments?

  2. How does the generation-to-understanding benefit discussed in this paper differ from that explored in existing work?

  3. What new insights does your study provide into task-decoupled MoT architectures beyond existing models such as BAGEL and Cosmos 3?

3 comments
AuthorsOpenReview ↗

Response to Reviewer CEUe

We thank the reviewer for recognizing the importance of the research question and the value of the controlled experiments. We address the questions below.

Q1: Do interference and the benefits of task decoupling persist with larger backbones?

A1: Thank you for this suggestion. We have extended the representation-level study to an 8B backbone (Qwen3-8B), covering dense sharing, modality-decoupled MoT, and task-decoupled MoT. The current checkpoints have been trained for 120k steps, with the results summarized below.

ArchitectureTrainingGeneralOCRV-Centric & SIDPG
DenseUND / GEN64.5468.3857.1477.89
DenseUND+GEN64.6468.7858.6576.79
Modality-decoupled MoTUND / GEN62.1866.9556.8780.67
Modality-decoupled MoTUND+GEN61.1962.3055.8482.23
Task-decoupled MoTUND+GEN65.1469.2559.0182.70

The 8B results reproduce the qualitative findings at 1.7B: the balance between understanding and generation depends on routing, with asymmetric interference under shared visual computation and a more balanced outcome under task decoupling. This shows that the observed behavior extends to a substantially larger backbone. We will continue training to the full schedule and add the complete results, together with additional 8B task- and system-level experiments, to the paper.

Q2: How does the generation-to-understanding finding relate to ROSS and UniMRG?

We thank the reviewer and have added these discussion in Section 6 of our revised paper. The key difference is the nature of the "generation" being studied.

  • ROSS does not perform image generation. It adds a denoiser head that reconstructs frozen VAE latents of the input image, as an auxiliary loss for an MLLM with a pretrained SigLIP/CLIP encoder.
  • UniMRG turns extra visual perception supervision into generation-style tasks: producing depth maps (from Depth Anything V2) and segmentation maps (from SAM) of the input image. These targets are perception labels, not natural images. It is therefore expected that they improve perception-oriented understanding (spatial reasoning, fine-grained perception, hallucination).

In both cases, the generative task is a re-rendering of the input image, designed as an auxiliary signal for understanding.

Our work studies the intrinsic relation between multimodal understanding and standard generation: text-conditioned natural-image generation and editing, trained as a full task, in a native model without pretrained vision encoders, VAEs or perception teachers. This allows us to ask a different question: when do the two genuine capabilities help each other, and when do they conflict? We find that the answer depends on how computation is shared.

Q3: What distinguishes the study from existing task-decoupled models such as BAGEL and Cosmos 3?

A3: BAGEL and Cosmos 3 address a different goal: how to fit understanding and generation into one model without obvious interference. To achieve this, their visual representations are split rather than unified. Understanding uses a semantic ViT, while generation uses the latents of a separate frozen VAE, and each representation is processed by its own tower. This avoids interference, but it also weakens the connection between the two capabilities. The two tasks do not share a visual representation, and neither work shows that understanding and generation actually improve each other. BAGEL compares Dense / MoE / MoT only by training-loss curves, and neither report compares against single-task models.

Our study asks the complementary question: can understanding and generation reinforce each other when they share a unified visual representation? In our native model, both tasks read and write images through the same pixel-patch interface, learned jointly from scratch, with no separate encoder or VAE.

In short, BAGEL and Cosmos 3 show that separate representations can coexist in one model. We show that a unified representation can deliver mutual improvement, and we identify the conditions under which it does. We have added this discussion in Section 6 of our revised paper.

The rebuttal has addressed my concerns, and I maintain my positive score.

Thank you for your constructive feedback and continued positive assessment. We are pleased that our rebuttal addressed your concerns and appreciate your time and careful review.

Reviewer_kLHuOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the synergy and conflict between visual understanding and generation in native unified multimodal models at three levels.

At the representation level, it examines how parameter sharing and routing affect feature transfer and interference across three architectures: dense sharing, modality-decoupled MoT, and task-decoupled MoT.

At the task level, it evaluates whether joint training enables positive bidirectional transfer between tasks that require shared knowledge or capabilities. These include geometry problem solving and diagram generation, SVG code understanding and rendering, and 3D spatial question answering and novel view synthesis.

At the system level, it compares an end-to-end UMM with a modular planner–executor pipeline on reasoning-based image editing tasks that require close interaction between understanding and generation.

Critical strengths
  1. The structurally native, pixel-in, pixel-out setting provides a clean basis for studying the interaction between understanding and generation. Visual inputs are converted into patch tokens using two convolutional layers with a patch size of 32, without a pretrained vision encoder. Visual outputs are generated directly in continuous pixel space through flow matching, without a VAE or discrete tokenizer. All models use Qwen3-1.7B as the language backbone, with causal attention across the sequence and bidirectional attention within images. This design removes confounding factors from external visual encoders and decoders, allowing the interaction between the next-token cross-entropy objective and the flow-matching mean squared error objective to be examined directly.

  2. Prior work such as MoT by Liang et al. and Janus primarily separates computation by modality or uses distinct visual encoders. This paper identifies a more specific distinction between understanding and generation tokens. Clean visual understanding tokens, UND-V, behave more like text tokens and depend heavily on the semantic priors in the pretrained language model’s feed-forward layers. In contrast, noisy visual generation tokens, GEN-V, require denoising computations for high-frequency visual details that can conflict with semantic processing.

  3. The analysis goes beyond measuring joint-training gains. By examining multiple architectures and modality-specific representations, the paper provides insight into both positive transfer and interference between understanding and generation.

Critical weaknesses
  1. The role of the pre-buffer layers is unclear. Sections 2.2 and B.2 describe 16 randomly initialized pre-buffer layers before Qwen3-1.7B, while the caption of Figure 2 states that these layers are omitted for simplicity. In task-decoupled MoT, are these layers shared across tasks or also decoupled? If they are shared, how is interference between understanding and generation handled within them?

  2. Table 1 reports both separate training under UND/GEN and joint training under UND+GEN for dense sharing and modality-decoupled MoT. However, the fifth row reports only UND+GEN for task-decoupled MoT. The corresponding understanding-only and generation-only baselines are missing. These baselines should retain the same architecture and any applicable masked reconstruction components to isolate the effect of joint training.

  3. It is unclear whether parameter counts are matched across architectures. Does MoT double the total parameter count or substantially increase it? If so, could the additional capacity account for some of the observed gains?

  4. The system-level analysis does not sufficiently explain the source of the performance difference. The UMM also reasons about and refines the prompt before generation. How does its resulting description differ from the planner’s output, and how do these differences affect generation quality? The current analysis does not clearly distinguish gains from the understanding branch’s reasoning from gains attributable to joint understanding-generation training.

Questions for authors

Please address the questions and missing comparisons raised in the Critical Weaknesses section.

4 comments
AuthorsOpenReview ↗

Response to Reviewer kLHu (1/2)

We thank the reviewer for the detailed feedback and for highlighting the clean experimental setting and the distinction between understanding and generation tokens. We address each question below.

Q1: Are the pre-buffer layers shared or decoupled in task-decoupled MoT?

A1: The pre-buffer layers are also task-decoupled. Text and clean visual tokens use one pre-buffer branch, while noisy generation tokens use a separate branch with independent parameters. The branches interact through attention. We have clarified this design in Figure 2 and Appendix B.2 of our revised paper.

Q2: What are the corresponding task-decoupled single-task baselines, and how is masked reconstruction controlled?

We thank the reviewer for this point. A useful property of task-decoupled routing is that under single-task training it reduces exactly to architectures that are already in Table 1, because the routing rule only distinguishes UND-V from GEN-V tokens:

  • UND-only. No GEN-V tokens exist, so every token (text and UND-V) goes through the language branch, and the generation branch is never used. This is the computation of the Dense UND-only model (Row 1), with the same randomly initialized pre-buffer.
  • GEN-only. Text goes through the language branch and GEN-V tokens go through the generation branch. The small amount of editing data introduces a routing difference for clean source-image tokens. We tested routing these tokens through the LLM branch and observed slightly worse overall performance, which we attribute to the absence of substantial understanding supervision to support visual processing in that branch. We therefore use the Mod-dec. MoT GEN-only model (Row 3, with the same randomly initialized pre-buffer and generation branch) as the primary generation baseline..

To isolate joint training from the auxiliary masked-reconstruction objective, we also report the task-dec. joint model trained without masked reconstruction.

Task-dec. MoTGeneralOCRV-Centric & SIGenEval2DPGHPSv3Aesthetic
UND-only (= Row 1)67.8271.9660.72––––
GEN-only (= Row 3)–––57.5582.067.725.37
UND+GEN, w/o masked rec.68.21 (+0.39)72.41 (+0.45)61.76 (+1.04)65.64 (+8.09)82.73 (+0.67)7.51 (−0.21)5.34 (−0.03)
UND+GEN, w/ masked rec. (Row 5)69.02 (+1.20)72.09 (+0.13)62.38 (+1.66)63.96 (+6.41)82.31 (+0.25)7.90 (+0.18)5.50 (+0.13)

(Δ relative to the stronger single-task baseline.)

  • Joint training itself, not masked reconstruction, gives the gains. Without masked reconstruction, the task-dec. joint model already improves all three understanding categories and the compositional generation metrics over both single-task models.
  • Masked reconstruction is complementary. It mainly improves General / V-Centric understanding and image quality. As shown in Table 10, adding it to Dense joint training does not reproduce these gains.
AuthorsOpenReview ↗

Response to Reviewer kLHu (2/2)

Q3: Could additional MoT parameters explain the observed gains?

A3: We report the parameter counts and computational costs below. All three architectures have the same active parameters and FLOPs per token, while the two MoT variants also have identical total parameter counts, allowing a direct comparison of their routing choices.

DenseMod-dec. MoTTask-dec. MoT
Total parameters2.88B5.10B5.10B
Active parameters per token (text / UND-V / GEN-V)2.53 / 2.22 / 2.25B2.53 / 2.22 / 2.25B2.53 / 2.22 / 2.25B
Forward GFLOPs per token, linear layers (text / UND-V / GEN-V)5.05 / 4.45 / 4.495.05 / 4.45 / 4.495.05 / 4.45 / 4.49
Attention FLOPs per token per attended key3.6×10⁵3.6×10⁵3.6×10⁵
  • MoT increases total parameters (2.88B → 5.1B) but not active parameters or FLOPs. Every token is processed by exactly one 44-layer branch of identical width (2.215B decoder parameters), and attention is global over the same sequence.
  • Capacity does not explain the gains. Mod-dec. MoT has the same total parameters as Task-dec. MoT, yet joint training degrades its understanding by 2.6–4.7 points. Task-dec. improves both capabilities. The difference is which tokens share a branch, not how many parameters exist. We have added this comparison in Appendix B.2 of our revised paper.

Q4: Does the system-level improvement come from better reasoning or joint understanding–generation training? How do the intermediate descriptions differ?

A4: We thank the reviewer for this question. Both systems produce an explicit edit instruction via reasoning before generating. The question is therefore whether the end-to-end UMM wins because (i) it writes a better instruction (reasoning quality), or (ii) it turns the same instruction into a better image (understanding–generation integration). Without retraining, we separate the two with a 2×2 swap of the existing models. Both the planner and the end-to-end UMM emit the same format (Reasoning Process … Explicit Edit Prompt …), so their outputs are directly interchangeable:

Instruction from plannerInstruction from end-to-end UMM
Generated by executorA: Planner→Executor (Table 5)B: UMM's explicit edit prompt → executor
Generated by end-to-end UMMC: planner's reasoning + prompt inserted as the UMM's think contentD: End-to-end UMM (Table 5)
SystemInstructionGeneratorRISE OverallKRIS Overall
A Planner→Executorplannerexecutor16.6666.48
BUMMexecutor17.5067.15
CplannerUMM18.0567.49
D End-to-end UMMUMMUMM18.8868.33

Interpretation: The results show that end-to-end training improves both instruction quality and image generation:

  1. Better edit instructions. With the executor fixed, using instructions from the UMM improves performance (B > A), showing that the end-to-end UMM produces more effective edit instructions.
  2. Better image generation. With the instruction source fixed, the UMM generates better edits than the executor (C > A and D > B), showing that the benefit also extends to executing the requested changes.

Thanks for your answess. Your rebuttal has addressed my concerns, and I'll maintain my positive score.

Thank you for your thoughtful feedback and positive assessment. We are glad our responses addressed your concerns and appreciate your time and constructive discussion.

Reviewer_adqDOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper systematically investigates the synergy between visual understanding and generation in native unified multimodal models (UMMs). The authors study this relationship at three levels: (1) At the representation level, they compare Dense Sharing, Modality-decoupled MoT, and Task-decoupled MoT to analyze interference during joint training and introduce a task-decoupled architecture. (2) At the task level, they investigate bidirectional transfer through geometry reasoning, SVG understanding and generation, and 3D spatial intelligence. (3) At the system level, they compare an end-to-end UMM with a Planner--Executor pipeline on reasoning-intensive image editing tasks. Experiments suggest that appropriate task decoupling can alleviate interference between understanding and generation, while joint training can facilitate positive bidirectional transfer across related tasks.

Critical strengths
  1. The paper addresses an important research question concerning whether visual understanding and generation genuinely reinforce each other in unified multimodal models.
  2. The investigation spans representation, task, and system levels, providing a systematic empirical analysis of synergy and interference between understanding and generation.
  3. The authors include controlled comparisons such as the Textual-Desc baseline and demonstrate joint-training benefits in geometry reasoning, SVG tasks, and 3D spatial intelligence, providing empirical support for the observed synergy.
Critical weaknesses
  1. Insufficient explanation of the underlying mechanisms.Although the paper identifies asymmetric interference between understanding and generation across different architectures, the fundamental causes of these observations remain insufficiently investigated. For instance, access to pretrained language parameters, initialization of visual branches, and the degree of parameter sharing may all influence the observed results. While the authors provide representation analyses and several ablations, these experiments do not fully explain the mechanisms underlying task interference.
  2. Limited practical guidance from the findings.Although the authors introduce Task-decoupled MoT to alleviate task conflicts, the paper does not clearly establish how task relatedness or interference should determine the choice of parameter-sharing and decoupling strategies. Consequently, how these empirical findings can systematically guide the design of broader unified multimodal architectures remains insufficiently explored.
  3. Limited generalizability of the findings.The main experiments rely on Qwen3-1.7B and a particular visual generation paradigm, with limited validation across model scales and alternative multimodal architectures. Moreover, the benefits of joint training vary considerably across tasks. For example, the improvements in SVG understanding and generation are asymmetric, and some reasoning-intensive image editing subtasks exhibit performance degradation. These observations suggest that the synergy may be task-dependent, leaving the boundaries of the conclusions unclear.
Questions for authors

Can the authors explain why the magnitude of synergy varies across different tasks and discuss whether the findings generalize to other model scales or unified multimodal architectures?

4 comments
AuthorsOpenReview ↗

Response to Reviewer adqD (1/3)

We thank the reviewer for recognizing the importance of the research question, the systematic three-level analysis, and the controlled task-level experiments. We address each point below.

Q1: What explains the asymmetric interference, and how do pretrained language parameters and parameter sharing affect it?

A1: We further investigate the mechanisms underlying asymmetric interference through controlled architectural ablations and gradient-level analyses. The ablations examine how initialization and parameter sharing affect the direction of transfer, while the gradient analyses explain how the two objectives exert unequal influence on shared computation.

1. Architectural ablations: what determines the direction of interference?

The following table reports the change from single-task to joint training under each configuration. ΔUND is the average change across nine understanding benchmarks, and ΔGEN is the change on DPG-Bench.

ArchitectureConfigurationΔUNDΔGEN
DenseRandomly initialized pre-buffer+0.60−1.00
DenseLLM-initialized pre-buffer−3.25−2.35
Modality-decoupled MoTRandomly initialized pre-buffer−3.57+1.03
Modality-decoupled MoTLLM-initialized pre-buffer−3.93+3.21
Modality-decoupled MoTLLM-initialized visual branch−4.16−0.71
Partial MoTOnly 16/44 layers decoupled−1.99−1.94
Task-decoupled MoTUND-V shares the text pathway; GEN-V is specialized+0.88+1.19

These comparisons reveal three points. First, pretrained initialization alone does not resolve understanding degradation in modality-decoupled MoT: the decrease persists even when the visual branch is initialized from the LLM. Second, reducing the extent of sharing through partial decoupling does not automatically yield mutual improvement. Third, separating clean understanding tokens from noisy generation tokens while retaining their connection to text produces positive transfer in both directions.

Thus, initialization influences the outcome, but which token types share computation is central to achieving balanced understanding–generation learning.

2. Gradient analysis: why does the dominant objective differ across architectures?

To better understand the mechanisms of the asymmetric interference, we computed per-layer gradients of the UND loss (CE) and the GEN loss (flow matching) on held-out samples. We split each objective's samples into independent halves, which gives unbiased estimates of the expected (consistent) gradient ‖E∇L‖² and its noise. We then report UND's share of the expected joint gradient, using the training loss weights and data mixture.

We first examine whether the two objectives systematically push shared parameters in opposing directions. In every shared layer and every architecture, the noise-corrected cosine between E∇L_und and E∇L_gen is ≈ 0 (|median| < 0.02). The objectives do not push the shared weights in opposite directions. They compete for ownership of the shared computation.

We then measure which objective dominates each pathway. The table below reports the median per-layer UND share of the expected joint gradient.

Joint modelPre-buffer (L0–15)Pretrained LLM layers (L16–43)Visual branch
Dense, random pre-buffer (60k → 120k → 210k)0.06 → 0.04 → 0.11 (GEN-driven)0.38 → 0.63 → 0.93 (UND-driven)—
Mod-dec. MoT (random / LLM-init pre-buffer)text: 0.94 / 0.97text: 0.89 / 0.920.00–0.11 (GEN-driven)
Task-dec. MoT0.950.940 (by construction)
  • Mod-dec. MoT. In the shared visual branch, UND supplies essentially none of the consistent learning signal. The reason is structural. GEN supervises every visual token with a dense per-pixel target. UND reaches visual tokens only indirectly, through attention from text answer tokens. So the visual computation that UND-V depends on is optimized almost exclusively for generation.
  • Dense. A division of labour emerges in the dense model. GEN claims the randomly initialized pre-buffer, while the pretrained LLM layers become UND-driven.
  • Task-dec. MoT has no visual path to fight over. The text/UND-V branch is UND-driven, and GEN has its own branch.

Together, the ablations and gradient analyses provide a consistent explanation for the asymmetric interference across architectures: routing and initialization shape where each objective dominates, and the capability that relies on computation predominantly shaped by the other objective can degrade. Task decoupling addresses this imbalance by keeping clean understanding tokens with text and assigning noisy generation tokens to a specialized pathway, while preserving semantic interaction through attention. We have included the detailed ablations and per-layer analyses in Section 3.5 and Appendix B.7 of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer adqD (2/3)

Q2: How do the findings guide the design of unified multimodal architectures?

A2: Our findings provide practical guidance at two levels: how to design a UMM architecture that supports both capabilities, and how to train and use that model to benefit from their interaction.

1. Architecture design: specialize conflicting computation while preserving communication.

Our comparisons show that separating text and vision alone does not ensure balanced learning. In modality-decoupled MoT, clean understanding tokens and noisy generation tokens still share a visual pathway, which is predominantly shaped by generation. Task decoupling instead keeps clean visual context with text and assigns noisy generation tokens to a specialized pathway, while retaining attention-based communication. This suggests that routing should reflect the computational roles of tokens, rather than modality alone. The gradient analyses in Q1 provide a diagnostic for this choice: identify pathways where one objective dominates computation needed by another, and consider specializing those pathways.

2. Training and application: build on shared knowledge and jointly model tasks that require both capabilities.

Once an architecture supports balanced understanding and generation, our task- and system-level studies suggest two ways to benefit from unification.

  • To improve a UMM and identify suitable applications, pair understanding and generation tasks that rely on shared knowledge. Geometry reasoning and diagram generation share geometric constraints; SVG understanding and rendering share program–appearance correspondences; spatial understanding and view generation share 3D scene and camera knowledge. Our results show that training these complementary tasks together can improve both capabilities. This also points to promising applications in embodied intelligence, where world modeling and policy learning both require understanding the current state, physical dynamics, and the consequences of actions. A UMM that jointly predicts future observations and learns actions could benefit from this shared knowledge, making joint world modeling and policy learning a natural direction for extending our findings.

  • To apply a UMM, use it to complete complex tasks requiring both visual understanding and generation end-to-end. Such tasks are often handled by pipelines of separate expert models for perception, reasoning, and generation. A UMM can integrate these capabilities and complete the process within one model. In reasoning-intensive image editing, our end-to-end UMM outperforms a planner–executor pipeline initialized from the same checkpoint and trained on the same source examples. This demonstrates the potential of UMMs to simplify multi-model pipelines while improving overall task performance.

AuthorsOpenReview ↗

Response to Reviewer adqD (3/3)

Q3: How generalizable are the findings across model scales, generation paradigms, and tasks?

A3: We thank the reviewer for this question. We address model scale, modeling choices, and task-dependent gains separately below.

1. Additional representation-level experiments with an 8B backbone.

We have extended the representation-level study to an 8B backbone (Qwen3-8B), covering dense sharing, modality-decoupled MoT, and task-decoupled MoT. The current checkpoints have been trained for 120k steps, with the results summarized below.

ArchitectureTrainingGeneralOCRV-Centric & SIDPG
DenseUND / GEN64.5468.3857.1477.89
DenseUND+GEN64.6468.7858.6576.79
Modality-decoupled MoTUND / GEN62.1866.9556.8780.67
Modality-decoupled MoTUND+GEN61.1962.3055.8482.23
Task-decoupled MoTUND+GEN65.1469.2559.0182.70

The 8B results reproduce the qualitative findings at 1.7B: the balance between understanding and generation depends on routing, with asymmetric interference under shared visual computation and a more balanced outcome under task decoupling. This shows that the observed behavior extends to a substantially larger backbone. We will continue training to the full schedule and add the complete results, together with additional 8B task- and system-level experiments, to the paper.

2. Flow matching and the pixel-space interface are deliberate modeling choices.

Our formulation combines autoregressive text modeling with flow matching in pixel space. These are two distinct choices. Flow matching and diffusion are established, widely adopted approaches underlying leading visual generation systems. Studying this family therefore makes our findings directly relevant to a mainstream, high-performing generation paradigm.

The pixel-space interface is chosen specifically for the scientific question we investigate. By removing external pretrained vision encoders and VAEs, we can examine how understanding and generation jointly shape visual representations learned inside the unified model, without conflating this interaction with representations supplied by separately pretrained visual modules. This also avoids requiring a separately trained unified visual tokenizer that must support understanding, generation, and reconstruction/editing simultaneously.

Thus, the native setting provides a controlled basis for studying representation-level interaction. We also plan to extend this investigation to additional formulations, including autoregressive visual generation, to examine how the findings transfer across modeling paradigms.

3. Different tasks naturally exhibit different magnitudes of transfer.

The tasks differ substantially in their prediction targets, required knowledge, and the model’s starting capabilities. Consequently, complementary supervision need not yield equal gains across tasks or in both directions. For SVG, image-to-code prediction and code-to-image generation share program–appearance correspondences but solve different prediction problems. The unequal magnitudes are consistent with different baseline capabilities and opportunities for improvement.

For reasoning-intensive image editing, the end-to-end model improves six of the seven reported subcategories. The only decrease is on the Logical subset of RISEBench, from 11.76 to 9.41. Both models have low absolute performance on this relatively small subset, and the difference corresponds to only two examples. Viewed across the full evaluation, the end-to-end model improves RISEBench from 16.66 to 18.88 and KRIS-Bench from 66.48 to 68.33.

These results support our central finding: joint learning provides benefits across diverse tasks, while the magnitude of transfer depends on the task and the model’s starting capability. The consistent overall improvements, together with the additional 8B results, demonstrate the broader relevance of the study.

AuthorsOpenReview ↗

A Gentle Follow-up for Reviewer adqD

Dear Reviewer adqD,

Thank you again for your thoughtful feedback on our paper. We wanted to kindly follow up and ask whether you have had a chance to read our responses and the revised manuscript.

In response to your comments, we have added architectural ablations and gradient-level analyses to explain the asymmetric interference, clarified the practical implications of our findings, and provided additional experiments with an 8B backbone.

We would appreciate hearing whether these additions address your concerns. If any questions remain, we would be happy to provide further clarification. If our responses have resolved your concerns, we would also appreciate your reconsideration of the assessment.

Thank you for your time and consideration.

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Work and the Rebuttal

Dear Area Chair and Reviewers,

We sincerely thank the reviewers for their thoughtful feedback and the Area Chair for handling our submission. We summarize the recognized strengths and our main rebuttal additions below.

I. Recognized Strengths

  • An important, systematic investigation. Reviewer adqD recognized the “important research question” and “systematic empirical analysis” across representation, task, and system levels. Reviewer CEUe likewise highlighted the importance of the question and the insights from controlled experiments.
  • A clean experimental setting. Reviewer kLHu noted that our native pixel-in, pixel-out setting “provides a clean basis for studying the interaction between understanding and generation,” removing confounding factors from external visual encoders and decoders.
  • Insights beyond joint-training gains. Reviewer kLHu highlighted the distinction between clean understanding tokens and noisy generation tokens. Reviewer adqD acknowledged the controlled task-level comparisons and empirical evidence for synergy.

II. Main Concerns and Our Responses

1. Mechanisms of asymmetric interference (adqD).
We added architectural ablations and gradient analyses across initialization, routing, and training stages. These reveal unequal influence on shared computation: mean gradients are nearly orthogonal, yet generation dominates the shared visual branch in modality-decoupled MoT. Task decoupling separates the competing visual computations while preserving interaction through attention.

2. Architectural controls and capacity (kLHu).
We clarified the independent pre-buffers, single-task references, and reconstruction controls. Both MoT variants have identical total parameters and active computation, yet different transfer outcomes. Task decoupling also improves understanding and compositional generation without reconstruction supervision, supporting the role of routing.

3. Generalization to larger models (adqD, CEUe).
New Qwen3-8B experiments reproduce the main representation-level findings: dense and modality-decoupled sharing exhibit opposite transfer asymmetries, while task decoupling yields balanced improvements. Full-schedule and task/system-level 8B experiments are planned.

4. Sources of system-level gains (kLHu).
A new 2×2 instruction–generator swap experiment shows that end-to-end training improves both capabilities: UMM-generated instructions improve a fixed executor, and the UMM produces better edits with either instruction source. The complete UMM performs best on both RISE and KRIS.

5. Positioning and practical guidance (adqD, CEUe).
We clarified connections to ROSS, UniMRG, BAGEL, and Cosmos 3. Our deliberate native setting studies understanding–generation interaction without external pretrained visual representations. The findings guide both architecture design—specialize computation by token role while preserving communication—and UMM development and application—pair tasks around shared knowledge and solve complex understanding–generation tasks end-to-end.

III. Discussion Status

Reviewers kLHu and CEUe have confirmed that our rebuttal addressed their concerns and maintained their positive assessments. We have provided detailed responses and additional evidence for Reviewer adqD’s concerns and welcome further discussion.

Thank you again for your time and consideration.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.