acceptodds
Under review as a conference paper at ICLR 2027

Relaying Understanding to Generation in Unified Multimodal Models

15 views

Abstract

Unified multimodal models bring visual understanding and generation into a single architecture, yet sharing a backbone alone does not guarantee that knowledge acquired for understanding can be effectively utilized by the generator. A central challenge is therefore not only what the model understands, but also how understanding-side representations are exposed in a form that the generation pathway can readily consume. We address this problem by introducing an explicit understanding-to-generation interface into the topology of unified multimodal models. Specifically, understanding- and generation-oriented computation are separated, while a compact set of learnable relay tokens is inserted between them. The understanding pathway transforms input semantics into these intermediate states, and the generation pathway is constrained to condition on them rather than directly accessing the original input representation. This topology encourages the understanding pathway to organize semantic information into a generation-compatible representation, while providing the generator with a compact and structured conditioning interface. Despite its simplicity, the proposed design consistently improves image generation, with particularly strong gains on compositional semantics such as spatial relations, attribute binding, and actions. Our results suggest that explicitly structuring the topology of information flow between understanding and generation is an effective way to improve how unified multimodal models translate semantic understanding into visual generation.

Primary area:
generative models
Keywords:
Unified Multimodal Model, latent token

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qNmBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RUG, an approach that introduces learnable Relay Tokens to facilitate information transfer from understanding to generation in unified multimodal models. The understanding expert processes these tokens, and the generation expert attends to their layer-wise representations alongside the original text. Experiments on Janus-Pro and BAGEL evaluate generation performance, architectural variants, and the effects of intervening on relay representations.

Critical strengths
  1. The paper studies an important question: how unified multimodal models can better use their understanding capabilities for image generation. The proposed method is relatively simple and retains the original training objectives without requiring additional alignment losses or explicit reasoning supervision.

  2. The controlled architecture comparisons and Relay Token transplantation experiments provide useful evidence about the method’s effectiveness and how relay representations influence generated attributes and spatial relations.

Critical weaknesses
  1. The paper does not report understanding benchmark results before and after fine-tuning, making it difficult to determine whether generation improvements come at the expense of understanding performance. Section 4.1 explicitly mentions evaluation on MME and MMMU, but I could not find the corresponding results. This discrepancy makes the experimental reporting appear incomplete and insufficiently polished.

  2. The additional inference overhead introduced by Relay Tokens is not quantified. Although the paper describes the overhead as negligible, it does not report inference latency, memory usage, or computational cost to substantiate this claim.

  3. The gains are small on several evaluations. Compared with MoT-Vanilla, RUG improves WISE from 27.20 to 27.60 and DPG from 82.65 to 82.71 in Table 2, while the overall ImgEdit score only increases from 3.82 to 3.83 in Table 3. Improvements of this magnitude cannot rule out the influence of random variation.

Questions for authors
  1. Could you report visual understanding benchmark results before and after fine-tuning?

  2. What additional inference latency, memory usage, and computational cost do Relay Tokens introduce compared with MoT-Vanilla under identical settings?

5 comments
AuthorsOpenReview ↗

Response to Reviewer qNmB (Part 1/3)

We thank the reviewer for recognizing the importance of our research question, the simplicity of our method, and the value of our controlled comparisons and relay transplantation experiments.

W1 & Q1. Understanding Performance Before and After Fine-Tuning

A1: We thank the reviewer for pointing out this omission. We now report MME, MMMU, and MMStar for the original BAGEL and the matched fine-tuned MoT-Vanilla and RUG models.

ModelMMMU (%) ↑MME-P ↑MME-C ↑MMStar (%) ↑
BAGEL (original)52.561669.02678.2166.67
MoT-Vanilla (matched fine-tuning)53.001654.09664.6466.80
RUG52.561667.49674.2967.60

RUG improves image generation over both the original model and MoT-Vanilla while maintaining comparable understanding performance. These results indicate that RUG's improvements in image generation do not come at the cost of a substantial decline in multimodal understanding.

Fine-tuning with understanding data does not guarantee improvement on every understanding benchmark. SGT reports a small MMStar decrease (67.46 → 66.93) after SFT using 500K LLaVA-OneVision samples (SGT [[1]]), while TorchUMM reports decreases in both MME components after BAGEL SFT (TorchUMM [[2]]). A possible contributor is a gap in quality or distribution between public fine-tuning datasets and the original models' training data. AIA explicitly notes that its public datasets are lower-quality than Janus-Pro's non-public training data and supplements them with internal samples (AIA [[3]]). This offers a possible explanation for small score changes.

W2 & Q2. Inference Overhead: Latency, Memory, and Computational Cost

A2: We thank the reviewer for this important suggestion. We evaluate original BAGEL, MoT-Vanilla, and RUG using the same prompts, seeds, precision (BF16), batch size (1), resolution (1024 × 1024), and sampling settings. LoRA weights are merged in both models, and RUG uses 128 Relay Tokens. Model inference latency includes VAE decoding but excludes PNG encoding and is averaged over 6 requests per prompt length.

Model / Prompt tokens102550100150200Mean (s)
BAGEL (original, no Relay)14.75114.57714.79314.72614.71814.81614.730
MoT-Vanilla (no Relay)14.74514.55914.78214.71614.71714.78214.717
RUG14.91514.72814.95714.87114.88714.92714.881

RUG increases average inference latency by only a negligible amount over original BAGEL. Memory and computational overhead are also small:

Additional cost of RUGvs. original BAGELvs. MoT-Vanilla
Peak allocated GPU memory≈14.9 MiB≈14.9 MiB
Analytic Transformer FLOPs≈0.406%≈0.406%

After LoRA merging, the only additional model parameters are the 128 Relay Token embeddings: 458,752 parameters (0.459M; approximately 0.003% of either baseline's parameter count).

Reference

  1. Semantic Generative Tuning for Unified Multimodal Models.
  2. TorchUMM: A Unified Multimodal Model Codebase for Evaluation, Analysis, and Post-training.
  3. AIA: Rethinking Architecture Decoupling Strategy In Unified Multimodal Model.
AuthorsOpenReview ↗

Response to Reviewer qNmB (Part 2/3)

W3. Performance Gain on different task

A3.1: We thank the reviewer for raising this concern. We will add further validation and analysis, as providing clear evidence of RUG's effectiveness is important. After this response in A3.2 , we also discuss RUG's broader potential to help understanding and generation learn better together.

For the T2I task, RUG improves GenEval from 72.87% to 77.68% (+4.81 points), with gains in two-object composition (+7.83), spatial relations (+9.00), and color binding (+8.25). GenEval2 also improves in verb understanding (+5.91) and AM (+3.08), supporting the benefit of Relay Tokens for compositional generation.

However, DPG-Bench can understate performance differences. Prior work reports rapid saturation (Deep Fusion [[4]]) and automated scoring errors (UniFusion [[5]]). A published ablation (MOSS-Image [[6]]) shows that GenEval drops from 0.6484 to 0.6244 and OneIG from 0.4363 to 0.3482, while DPG barely changes (82.420 → 82.411). Our Table 2 checkpoints show a similar contrast: the understanding-only baseline which only fine-tuned with vqa dataset and RUG differ by less than 1.5 DPG points, despite gaps of 16.31 GM and 42.33 AM points on GenEval2.

Model (Table 2 checkpoint)GenEval2 GMGenEval2 AMDPG-Bench
Understanding-only2.6122.8081.27
RUG18.9265.1382.71

We therefore place greater weight on the consistent GenEval and GenEval2 gains when assessing RUG's architectural benefits.

In addition, Table 2's small WISE gain (27.20 → 27.60) may partly reflect the limited knowledge and reasoning capacity of Janus-Pro-1B, the small backbone used for our architectural ablations. WISE requires models to infer what to generate using world knowledge and reasoning. We therefore add a matched BAGEL comparison, using the same training recipe (320K generation + 640K VQA) and evaluating all 1,000 WISE_Verified prompts with the same scoring protocol.

Model (BAGEL)WISE (%) ↑
MoT-Vanilla (no Relay)34.90
RUG39.00

The 4.10-point improvement provides further evidence that Relay Tokens benefit generation that requires world knowledge and reasoning.

For the I2I task, the ImgEdit gain over MoT-Vanilla is modest (3.8169 → 3.8331). We therefore extend evaluation to RISEBench, 360 reasoning-oriented editing, covering temporal, causal, spatial, and logical reasoning. At matched 10K checkpoints, using the same Qwen3.5-35B-A3B judge, RUG improves weighted scores across all four reasoning categories, increasing the overall score from 38.01 to 41.56 (+3.55), with the largest gain in causal reasoning (+6.97).

Model (BAGEL)TemporalCausalSpatialLogicalOverall
MoT-Vanilla (no Relay)38.1351.9752.595.9438.01
RUG41.5158.9454.038.5341.56

These results support RUG's ability to improve image editing without dedicated editing data during fine-tuning.

Reference

  1. Deep Fusion: Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis.
  2. UniFusion: Vision-Language Model as Unified Encoder in Image Generation.
  3. MOSS-Image Technical Report.
AuthorsOpenReview ↗

Response to Reviewer qNmB (Part 3/3)

W3. Performance Gain on different task

A3.2: We hope these additional results and explanations make RUG's effectiveness clearer. But our hope for relay tokens goes further. Relay Tokens give the two experts a way to communicate at every layer. The understanding expert can learn to organize what it knows into representations that better support generation, while the generation expert learns to make better use of them. This opens up the possibility of both experts improving together, perhaps easing a difficulty that has long troubled unified models: understanding and generation getting in each other’s way during joint training.

We did not build our motivation around solving this problem, and we do not claim that RUG resolves it. Still, our experiments gave us a reason to be hopeful. In Table 2 and Fig. 3(a), we carefully control the training conditions and compare joint training with models trained only for understanding or only for generation. These separate models give us a reference for how each task learns when the other is absent.

What we observed went beyond our expectations. RUG’s generation loss fell below that of the generation-only model, even though the latter had no understanding objective competing with it and already achieved the lowest generation loss among the baselines. On the understanding side, RUG’s loss was also the closest among jointly trained models to the understanding-only model. For us, this was an encouraging sign that the tension between these two tasks can be eased—that learning them together can create room for one to help the other. We hope to explore this possibility further.

We would also like to share why we ultimately chose the name Relay. To us, a query suggests generation asking understanding for the information it needs, the relay expresses the relationship we hope to build: the understanding expert organizes what it knows into representations that are easier for the generation expert to use, and passes them on through Relay Tokens. Generation can then carry that understanding forward into an image. We hope that what the model understands can ultimately find its way into the world it paints.

Reviewer_jUAnOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RUG introduces learnable Relay Tokens to pass information from the understanding expert to the generation expert within a unified multimodal Transformer. The understanding expert updates relay states from the textual context, and the generation expert reads their layer-specific keys and values alongside the original text. The interface is trained with the underlying understanding and generation objectives using LoRA adaptation. Experiments on Janus-Pro-1B and BAGEL report improved compositional image generation, alongside attention-topology and relay-capacity ablations, qualitative relay-transplantation tests, and an image-editing evaluation.

Critical strengths
  1. Relay Tokens provide a simple interface for transferring understanding-side information to image generation within a unified model.
  2. Experiments on autoregressive Janus-Pro and diffusion-based BAGEL demonstrate applicability to two different generation architectures.
  3. Attention-topology and token-count ablations, together with relay-transplantation examples, help probe the contribution of the relay pathway.
Critical weaknesses
  1. How does RUG differ from query-based interfaces such as MetaQueries and UniLIP, and why are there no controlled comparisons demonstrating the advantages of integrating queries into the model's layer-wise attention?
  2. Does updating the understanding expert degrade its capabilities, and why are understanding benchmark results, including the mentioned MME and MMMU evaluations, absent from the reported results?
  3. Can relays access source-image representations during editing, and could placing them after the full text-and-source-image context improve the negligible gain over matched MoT-Vanilla (3.83 versus 3.82)?
  4. The abstract describes Relay Tokens as the sole conditioning pathway, whereas Section 3.3 and the final Text-plus-Relay configuration permit direct text access, leaving the core mechanism inconsistently defined.
Questions for authors
  1. What substantive methodological advantage over query-based interfaces such as MetaQueries and UniLIP is established by integrating relay tokens into layer-wise attention, and what evidence supports it?
  2. Do the MME and MMMU evaluations mentioned in Section 4.1 show that understanding performance is preserved relative to both the original model and a matched fine-tuned baseline?
9 comments
AuthorsOpenReview ↗

Response to Reviewer jUAn (Part 1/3)

We thank the reviewer for recognizing the simplicity of our interface, its applicability across generation architectures, and the value of our ablation and transplantation analyses.

W1 & Q1. Architectural Differences and Controlled Comparisons with Query-Based Interfaces

A1.1: We thank the reviewer for this question, which encouraged us to discuss the differences between the two architectures in greater depth. It also inspired us to explain why we chose the name Relay Tokens, a point we return to at the end of this response after presenting the additional experimental evidence in A1.2.

MetaQueries and UniLiP adopt a final-layer handoff design: MetaQueries extracts final-layer query features, while UniLiP uses final-layer text and query features. Both deliver these representations to external generators through independent connectors and cross-attention. In contrast, RUG enables layer-wise coupling within MoT, where generation jointly attends to text and Relay K/V from the corresponding understanding layers through unified masked attention.

To compare these information-flow designs, we implement four adapted final-layer controls on the same Janus-Pro-1B backbone with identical training data (320K generation + 640K VQA), examining both cross-attention and MoT-style unified attention for consuming conditioning information.

GenEval accuracy (%)

MethodAttention2K10K20K
MetaQuery-styleCross-attention13.3844.3349.43
UniLiP-styleCross-attention35.3961.4462.23
MetaQuery-styleUnified attention0.7358.3864.34
UniLiP-styleUnified attention51.2364.6964.52
RUG (Text + Relay)Unified attention75.0275.78—

RUG achieves 75.02% at 2K steps, exceeding all final-layer controls even after 20K steps (best: 64.69%). The controls also benefit from MoT-style unified attention over separate cross-attention at later steps. These results suggest that layer-wise coupling, together with unified attention, provides a more effective way to consume understanding-derived information than final-layer handoff in our controlled setting. We do not reproduce the original independent connector, as our comparison focuses on information extraction and consumption within the same backbone rather than the complete external-generator architecture.

Furthermore, to investigate why layer-wise transfer is useful, we extend the Relay Transplantation Test in Fig. 5 by replacing Relay K/V at selected layers while keeping other conditions fixed. Taking left–right spatial relations as an example:

Replaced Relay LayersRelation Transfer
Layer 1✗
Layers 1–7✗
Layers 8–14✓
Layers 15–21✗
Layers 22–28✗
Layers 1–14✓
Layers 15–28✗
All layers (1–28)✓

✓: successfu; ✗: unsuccessful

Notably, layers 8–14 alone transfer the spatial relation consistently, revealing depth-dependent semantic control. This further supports RUG's layer-wise attention topology, which allows generation to access and utilize understanding-organized semantic representations at different depths during image synthesis, rather than depending solely on final-layer conditioning.

AuthorsOpenReview ↗

Response to Reviewer jUAn (Part 2/3)

W1 & Q1. Architectural Differences and Controlled Comparisons with Query-Based Interfaces

A1.2: The idea behind interfaces such as MetaQueries and BLIP3-o is to connect an understanding model with a separate image generator. Query tokens can be viewed as the generator asking the understanding model for the information it needs. The resulting semantic representations can replace the generator's original text conditioning; in MetaQueries, a learned connector and the diffusion model's cross-attention make this handoff possible.

In a MoT architecture such as BAGEL, the generation expert can already access understanding representations directly through attention at corresponding layers. the relay token build on this connection as a layer-wise communication interface alongside the original text. Through this interface, the understanding expert can learn to organize representations for generation, while the generation expert learns to make better use of them. If the two experts can improve together through this exchange, perhaps we can also ease the interference between understanding and generation during joint training.

We did not build our motivation around solving this problem, and we do not claim that RUG resolves it. Still, our experiments gave us a reason to be hopeful. In Table 2 and Fig. 3(a), we carefully control the training conditions and compare joint training with models trained only for understanding or only for generation. These separate models give us a reference for how each task learns when the other is absent.

What we observed went beyond our expectations. RUG’s generation loss fell below that of the generation-only model, even though the latter had no understanding objective competing with it and already achieved the lowest generation loss among the baselines. On the understanding side, RUG’s loss was also the closest among jointly trained models to the understanding-only model. For us, this was an encouraging sign that the tension between these two tasks can be eased—that learning them together can create room for one to help the other. We hope to explore this possibility further.

We would also like to share why we ultimately chose the name Relay. To us, a query suggests generation asking understanding for the information it needs, relay expresses the relationship we hope to build: the understanding expert organizes what it knows into representations that are easier for the generation expert to use, and passes them on through Relay Tokens. Generation can then carry that understanding forward into an image. We hope that what the model understands can ultimately find its way into the world it paints.

AuthorsOpenReview ↗

Response to Reviewer jUAn (Part 3/3)

W2 & Q2. Understanding Preservation and Comparisons with Baselines

A2: We thank the reviewer for pointing out this omission. We now provide MME, MMMU, and MMStar results for both backbones.

BackboneModelMME-PMME-CMMMU (%)MMStar (%)
Janus-Pro-1BMoT-Vanilla1450.64248.5736.1143.67
Janus-Pro-1BRUG1429.93252.1435.3344.20
BAGEL-7BMoT-Vanilla1654.09664.6453.0066.80
BAGEL-7BRUG1667.49674.2952.5667.60

Together with the generation gains reported above, these results indicate that RUG's improvements in image generation do not come at the cost of a substantial decline in multimodal understanding.

W3. Source-Image Access and Image Editing Evaluation

A3: We thank the reviewer for this question. Yes. In ImgEdit, Relay Tokens already follow the source-image representations and editing instruction. Through the understanding expert, they read the layer-wise KV cache containing source VAE and ViT features and the instruction. The generation expert directly accesses both the source context and relay states. Only the target-image tokens being denoised are masked from the relays. For the I2I task, the ImgEdit gain over MoT-Vanilla is modest (3.8169 → 3.8331). We therefore extend evaluation to a popular evaluated bench, RISEBench, 360 reasoning-oriented editing tasks (RISEBench [[1]]), covering temporal, causal, spatial, and logical reasoning. At matched 10K checkpoints, using the same Qwen3.5-35B-A3B judge, RUG improves weighted scores across all four reasoning categories, increasing the overall score from 38.01 to 41.56 (+3.55), with the largest gain in causal reasoning (+6.97).

Model (BAGEL, 10K)TemporalCausalSpatialLogicalOverall
MoT-Vanilla (no Relay)38.1351.9752.595.9438.01
RUG41.5158.9454.038.5341.56

These results support RUG's ability to improve image editing without dedicated editing data during fine-tuning.

W4. RUG’s Architectural Choices and Conditioning Pathways

A4: We thank the reviewer for pointing this out. We compare Relay-only and Text + Relay in Table 2, whose results support our final choice of Text + Relay. We will revise the abstract to clearly state that the final model conditions generation on both text and relay states.

This choice is based on the mean of each bench. Text + Relay achieves a higher average at every evaluated training checkpoint (Figure. 3(c)) and every tested Relay Token count (Figure. 4). This agrees with our above A1 controls, where UniLiP-style (text + query) interfaces outperform MetaQuery-style (query-only) interfaces.

Relay-only still has strengths on individual metrics: it leads on GenEval2 Position in Table 2, and its peak WISE score across checkpoints is 29.30, compared with 29.10 for Text + Relay. We therefore favor Text + Relay for its overall performance while recognizing these Relay-only advantages.

Reference

  1. RISEBench: Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editing.
Reviewer_TRymOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RUG, which uses learnable relay tokens to pass understanding-side representations to the generation expert through layer-wise attention. Experiments on Janus-Pro and BAGEL show improvements in image generation.

Critical strengths
  1. The paper is well written and easy to follow.
  2. The method is simple and requires no auxiliary supervision.
  3. The relay transplantation experiment provides useful analysis.
Critical weaknesses
  1. The experiments do not show whether RUG repairs cases where the model answers correctly but generates incorrectly, which is the core motivation. Transplantation demonstrates that relay states affect generation, but does not distinguish understanding transfer from additional context processing. Given existing learned-query methods (e.g., metaquery), the specific value of RUG’s topology needs stronger justification.
  2. The architectural benefit in the final training setting remains unclear. Table 2 uses filtered data, whereas Table 1 additionally fine-tunes RUG on full BLIP3o-60k without a corresponding matched no-relay result. Gains also vary across tasks: over fine-tuned MoT-Vanilla, editing improves only from 3.82 to 3.83, and DPG from 82.65 to 82.71, which is relatively incremental.
  3. MME/MMMU are listed as evaluations, and the conclusion mentions mixed understanding results, but no scores are reported in the main paper.
Questions for authors

Please refer to the Critical Weaknesses section.

3 comments
AuthorsOpenReview ↗

Response to Reviewer TRym (Part 1/3)

We thank the reviewer for recognizing the clarity of our presentation, the simplicity of our method, and the informative analysis.

W1. Empirical Validation of Our Motivation and Advantages of RUG’s Topology

A1.1: Empirical Validation of Our Motivation

We thank the reviewer for suggesting a quantitative evaluation of our motivation. To examine whether correct understanding translates into successful generation, we report text accuracy to measure correct answers and image accuracy given correct text to measure generation success when understanding is correct. We evaluate both BAGEL checkpoints (10K steps) on all 1,000 WISE_Verified prompts. Text and images are independently evaluated by Qwen3.5-35B-A3B, using WISE explanations for text judgments and the official WISE protocol for image evaluation.

ModelText Answer Accuracy (%)Image Accuracy Given Correct Text (%)
MoT-Vanilla53.6043.28
RUG54.9047.72

Conditional image accuracy is computed over each model's correctly answered prompts, respectively.

MoT-Vanilla generates correct images in only 43.28% of cases with correct text answers, revealing an understanding–generation gap. With comparable text accuracy, RUG improves conditional image accuracy to 47.72% (+4.44 points). This observation suggests that RUG may better utilize the semantic knowledge demonstrated by the understanding pathway to guide visual generation.

A1.2: Advantages of RUG’s Topology

MetaQueries and UniLiP adopt a final-layer handoff design: MetaQueries extract final-layer query features, while UniLiP uses final-layer text and query features. Both deliver these representations to external generators through independent connectors and cross-attention. In contrast, RUG enables layer-wise coupling within MoT, where generation jointly attends to text and Relay K/V from the corresponding understanding layers through unified masked attention.

To compare these information-flow designs, we implement four adapted final-layer controls on the same Janus-Pro-1B backbone with identical training data (320K generation + 640K VQA), examining both cross-attention and MoT-style unified attention for consuming conditioning information.

GenEval accuracy (%)

MethodAttention2K10K20K
MetaQuery-styleCross-attention13.3844.3349.43
UniLiP-styleCross-attention35.3961.4462.23
MetaQuery-styleUnified attention0.7358.3864.34
UniLiP-styleUnified attention51.2364.6964.52
RUG (Text + Relay)Unified attention75.0275.78—

RUG achieves 75.02% at 2K steps, exceeding all final-layer controls even after 20K steps (best: 64.69%). Controls also benefit from MoT-style unified attention over separate cross-attention at later steps. These results suggest that layer-wise coupling, together with unified attention, provides a more effective way to consume understanding-derived information than final-layer handoff in our controlled setting. We do not reproduce the original independent connector, as our comparison focuses on information extraction and consumption within the same backbone rather than the complete external-generator architecture.

Furthermore, to investigate why layer-wise transfer is useful, we extend the Relay Transplantation Test in Figure. 5 by replacing Relay K/V at selected layers while keeping other conditions fixed. Taking left–right spatial relations as an example:

Replaced Relay LayersRelation Transfer
Layer 1✗
Layers 1–7✗
Layers 8–14✓
Layers 15–21✗
Layers 22–28✗
Layers 1–14✓
Layers 15–28✗
All layers (1–28)✓

✓: successfu; ✗: unsuccessful

Notably, layers 8–14 alone transfer the spatial relation consistently, revealing depth-dependent semantic control. This further supports RUG's layer-wise attention topology, which allows generation to access and utilize understanding-organized semantic representations at different depths during image synthesis, rather than depending solely on final-layer conditioning.

AuthorsOpenReview ↗

Response to Reviewer TRym (Part 2/3)

W2. Architectural Benefits Under Matched Training and Across Tasks

A2: We thank the reviewer for this comment. Table 1 provides broader comparisons with existing models trained on full BLIP3o-60k or other high-quality datasets, so we additionally fine-tune RUG on the unfiltered dataset. To supplement Table 1, we report a matched comparison with MoT-Vanilla (no Relay) below.

BackboneMethodGenEvalGenEval2 AMGenEval2 GMWISE
Janus-Pro-1BMoT-Vanilla (no Relay)74.1064.8417.5027.50
Janus-Pro-1BRUG77.5267.3419.7027.70
Improvement+3.42+2.50+2.19+0.20
BAGELMoT-Vanilla (no Relay)82.2073.1324.6536.30
BAGELRUG84.0875.0628.0440.90
Improvement+1.88+1.93+3.39+4.60

RUG outperforms the no-Relay baseline on all metrics across both backbones under this unfiltered-data setting. However, BLIP3o contains near-duplicate GenEval prompt templates (RecA [[1]]). They therefore include a filtered-data ablation to ensure fair comparisons. Our subsequent supplementary experiments follow Table 2's filtered-data setting, with GenEval-template examples and overlapping prompts removed and the backbone, data, and training recipe matched between MoT-Vanilla and RUG.

For the T2I task, under this controlled setting, RUG improves GenEval from 72.87% to 77.68% (+4.81 points), with gains in two-object composition (+7.83), spatial relations (+9.00), and color binding (+8.25). GenEval2 also improves in verb understanding (+5.91) and AM (+3.08), supporting the benefit of Relay Tokens for compositional generation.

However, DPG-Bench can understate performance differences. Prior work reports rapid saturation (Deep Fusion [[2]]) and automated scoring errors (UniFusion [[3]]). A published ablation (MOSS-Image [[4]]) shows that GenEval drops from 0.6484 to 0.6244 and OneIG from 0.4363 to 0.3482, while DPG barely changes (82.420 → 82.411). Our Table 2 checkpoints show a similar contrast: the understanding-only baseline which only fine-tuned with vqa dataset and RUG differ by less than 1.5 DPG points, despite gaps of 16.31 GM and 42.33 AM points on GenEval2.

Model (Table 2 checkpoint)GenEval2 GMGenEval2 AMDPG-Bench
Understanding-only (1K)2.6122.8081.27
RUG (9K)18.9265.1382.71

We therefore place greater weight on the consistent GenEval and GenEval2 gains when assessing RUG's architectural benefits.

For the I2I task, the ImgEdit gain over MoT-Vanilla is modest (3.8169 → 3.8331). We therefore extend evaluation to RISEBench, 360 reasoning-oriented editing tasks (RISEBench [[5]]), covering temporal, causal, spatial, and logical reasoning. At matched 10K checkpoints, using the same Qwen3.5-35B-A3B judge, RUG improves weighted scores across all four reasoning categories, increasing the overall score from 38.01 to 41.56 (+3.55), with the largest gain in causal reasoning (+6.97).

Model (BAGEL, 10K)TemporalCausalSpatialLogicalOverall
MoT-Vanilla (no Relay)38.1351.9752.595.9438.01
RUG41.5158.9454.038.5341.56

These results support RUG's ability to improve image editing without dedicated editing data during fine-tuning.

Reference

  1. RecA: Reconstruction Alignment Improves Unified Multimodal Models.
  2. Deep Fusion: Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis.
  3. UniFusion: Vision-Language Model as Unified Encoder in Image Generation.
  4. MOSS-Image Technical Report.
  5. RISEBench: Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editing.
AuthorsOpenReview ↗

Response to Reviewer TRym (Part 3/3)

W3. Complete Multimodal Understanding Results

A3: We thank the reviewer for pointing out this omission. We now provide MME, MMMU, and MMStar results for both backbones.

BackboneModelMME-PMME-CMMMU (%)MMStar (%)
Janus-Pro-1BMoT-Vanilla1450.64248.5736.1143.67
Janus-Pro-1BRUG1429.93252.1435.3344.20
BAGEL-7BMoT-Vanilla1654.09664.6453.0066.80
BAGEL-7BRUG1667.49674.2952.5667.60

Together with the generation gains reported above, these results indicate that RUG's improvements in image generation do not come at the cost of a substantial decline in multimodal understanding.

Helping generation use what the model already understands is the motivation behind RUG. This also led us to consider a broader possibility for Relay Tokens. Relay Tokens provide a layer-wise communication interface: the understanding expert can learn to organize text into representations that are easier for generation to use, while the generation expert can learn to use those representations more effectively. If both experts can improve together through this exchange, perhaps we can also ease a difficulty that has long troubled unified models: understanding and generation getting in each other’s way during joint training.

We did not build our motivation around solving this problem, and we do not claim that RUG resolves it. Still, our experiments gave us a reason to be hopeful. In Table 2 and Fig. 3(a), we carefully control the training conditions and compare joint training with models trained only for understanding or only for generation. These separate models give us a reference for how each task learns when the other is absent.

What we observed went beyond our expectations. RUG’s generation loss fell below that of the generation-only model, even though the latter had no understanding objective competing with it and already achieved the lowest generation loss among the baselines. On the understanding side, RUG’s loss was also the closest among jointly trained models to the understanding-only model. For us, this was an encouraging sign that the tension between these two tasks can be eased—that learning them together can create room for one to help the other. We hope to explore this possibility further.

We would also like to share why we ultimately chose the name Relay. To us, a query suggests generation asking understanding for the information it needs, relay expresses the relationship we hope to build: the understanding expert organizes what it knows into representations that are easier for the generation expert to use, and passes them on through Relay Tokens. Generation can then carry that understanding forward into an image. We hope that what the model understands can ultimately find its way into the world it paints.

Comments on the paper

1 comment

Dear Reviewers,

We are approaching the deadline of the discussion period (Nov. 18). Please go through authors' rebuttal at your earliest convenience, and engage in a discussion or leave any follow-up comments.

Thank you for your time and participation.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.