- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RUG, an approach that introduces learnable Relay Tokens to facilitate information transfer from understanding to generation in unified multimodal models. The understanding expert processes these tokens, and the generation expert attends to their layer-wise representations alongside the original text. Experiments on Janus-Pro and BAGEL evaluate generation performance, architectural variants, and the effects of intervening on relay representations.
Critical strengths
-
The paper studies an important question: how unified multimodal models can better use their understanding capabilities for image generation. The proposed method is relatively simple and retains the original training objectives without requiring additional alignment losses or explicit reasoning supervision.
-
The controlled architecture comparisons and Relay Token transplantation experiments provide useful evidence about the method’s effectiveness and how relay representations influence generated attributes and spatial relations.
Critical weaknesses
-
The paper does not report understanding benchmark results before and after fine-tuning, making it difficult to determine whether generation improvements come at the expense of understanding performance. Section 4.1 explicitly mentions evaluation on MME and MMMU, but I could not find the corresponding results. This discrepancy makes the experimental reporting appear incomplete and insufficiently polished.
-
The additional inference overhead introduced by Relay Tokens is not quantified. Although the paper describes the overhead as negligible, it does not report inference latency, memory usage, or computational cost to substantiate this claim.
-
The gains are small on several evaluations. Compared with MoT-Vanilla, RUG improves WISE from 27.20 to 27.60 and DPG from 82.65 to 82.71 in Table 2, while the overall ImgEdit score only increases from 3.82 to 3.83 in Table 3. Improvements of this magnitude cannot rule out the influence of random variation.
Questions for authors
-
Could you report visual understanding benchmark results before and after fine-tuning?
-
What additional inference latency, memory usage, and computational cost do Relay Tokens introduce compared with MoT-Vanilla under identical settings?
5 comments
Response to Reviewer qNmB (Part 2/3)
W3. Performance Gain on different task
A3.1: We thank the reviewer for raising this concern. We will add further validation and analysis, as providing clear evidence of RUG's effectiveness is important. After this response in A3.2 , we also discuss RUG's broader potential to help understanding and generation learn better together.
For the T2I task, RUG improves GenEval from 72.87% to 77.68% (+4.81 points), with gains in two-object composition (+7.83), spatial relations (+9.00), and color binding (+8.25). GenEval2 also improves in verb understanding (+5.91) and AM (+3.08), supporting the benefit of Relay Tokens for compositional generation.
However, DPG-Bench can understate performance differences. Prior work reports rapid saturation (Deep Fusion [[4]]) and automated scoring errors (UniFusion [[5]]). A published ablation (MOSS-Image [[6]]) shows that GenEval drops from 0.6484 to 0.6244 and OneIG from 0.4363 to 0.3482, while DPG barely changes (82.420 → 82.411). Our Table 2 checkpoints show a similar contrast: the understanding-only baseline which only fine-tuned with vqa dataset and RUG differ by less than 1.5 DPG points, despite gaps of 16.31 GM and 42.33 AM points on GenEval2.
| Model (Table 2 checkpoint) | GenEval2 GM | GenEval2 AM | DPG-Bench |
|---|---|---|---|
| Understanding-only | 2.61 | 22.80 | 81.27 |
| RUG | 18.92 | 65.13 | 82.71 |
We therefore place greater weight on the consistent GenEval and GenEval2 gains when assessing RUG's architectural benefits.
In addition, Table 2's small WISE gain (27.20 → 27.60) may partly reflect the limited knowledge and reasoning capacity of Janus-Pro-1B, the small backbone used for our architectural ablations. WISE requires models to infer what to generate using world knowledge and reasoning. We therefore add a matched BAGEL comparison, using the same training recipe (320K generation + 640K VQA) and evaluating all 1,000 WISE_Verified prompts with the same scoring protocol.
| Model (BAGEL) | WISE (%) ↑ |
|---|---|
| MoT-Vanilla (no Relay) | 34.90 |
| RUG | 39.00 |
The 4.10-point improvement provides further evidence that Relay Tokens benefit generation that requires world knowledge and reasoning.
For the I2I task, the ImgEdit gain over MoT-Vanilla is modest (3.8169 → 3.8331). We therefore extend evaluation to RISEBench, 360 reasoning-oriented editing, covering temporal, causal, spatial, and logical reasoning. At matched 10K checkpoints, using the same Qwen3.5-35B-A3B judge, RUG improves weighted scores across all four reasoning categories, increasing the overall score from 38.01 to 41.56 (+3.55), with the largest gain in causal reasoning (+6.97).
| Model (BAGEL) | Temporal | Causal | Spatial | Logical | Overall |
|---|---|---|---|---|---|
| MoT-Vanilla (no Relay) | 38.13 | 51.97 | 52.59 | 5.94 | 38.01 |
| RUG | 41.51 | 58.94 | 54.03 | 8.53 | 41.56 |
These results support RUG's ability to improve image editing without dedicated editing data during fine-tuning.
Reference
Response to Reviewer qNmB (Part 3/3)
W3. Performance Gain on different task
A3.2: We hope these additional results and explanations make RUG's effectiveness clearer. But our hope for relay tokens goes further. Relay Tokens give the two experts a way to communicate at every layer. The understanding expert can learn to organize what it knows into representations that better support generation, while the generation expert learns to make better use of them. This opens up the possibility of both experts improving together, perhaps easing a difficulty that has long troubled unified models: understanding and generation getting in each other’s way during joint training.
We did not build our motivation around solving this problem, and we do not claim that RUG resolves it. Still, our experiments gave us a reason to be hopeful. In Table 2 and Fig. 3(a), we carefully control the training conditions and compare joint training with models trained only for understanding or only for generation. These separate models give us a reference for how each task learns when the other is absent.
What we observed went beyond our expectations. RUG’s generation loss fell below that of the generation-only model, even though the latter had no understanding objective competing with it and already achieved the lowest generation loss among the baselines. On the understanding side, RUG’s loss was also the closest among jointly trained models to the understanding-only model. For us, this was an encouraging sign that the tension between these two tasks can be eased—that learning them together can create room for one to help the other. We hope to explore this possibility further.
We would also like to share why we ultimately chose the name Relay. To us, a query suggests generation asking understanding for the information it needs, the relay expresses the relationship we hope to build: the understanding expert organizes what it knows into representations that are easier for the generation expert to use, and passes them on through Relay Tokens. Generation can then carry that understanding forward into an image. We hope that what the model understands can ultimately find its way into the world it paints.
Response to Reviewer qNmB (Part 1/3)
We thank the reviewer for recognizing the importance of our research question, the simplicity of our method, and the value of our controlled comparisons and relay transplantation experiments.
W1 & Q1. Understanding Performance Before and After Fine-Tuning
A1: We thank the reviewer for pointing out this omission. We now report MME, MMMU, and MMStar for the original BAGEL and the matched fine-tuned MoT-Vanilla and RUG models.
RUG improves image generation over both the original model and MoT-Vanilla while maintaining comparable understanding performance. These results indicate that RUG's improvements in image generation do not come at the cost of a substantial decline in multimodal understanding.
Fine-tuning with understanding data does not guarantee improvement on every understanding benchmark. SGT reports a small MMStar decrease (67.46 → 66.93) after SFT using 500K LLaVA-OneVision samples (SGT [[1]]), while TorchUMM reports decreases in both MME components after BAGEL SFT (TorchUMM [[2]]). A possible contributor is a gap in quality or distribution between public fine-tuning datasets and the original models' training data. AIA explicitly notes that its public datasets are lower-quality than Janus-Pro's non-public training data and supplements them with internal samples (AIA [[3]]). This offers a possible explanation for small score changes.
W2 & Q2. Inference Overhead: Latency, Memory, and Computational Cost
A2: We thank the reviewer for this important suggestion. We evaluate original BAGEL, MoT-Vanilla, and RUG using the same prompts, seeds, precision (BF16), batch size (1), resolution (1024 × 1024), and sampling settings. LoRA weights are merged in both models, and RUG uses 128 Relay Tokens. Model inference latency includes VAE decoding but excludes PNG encoding and is averaged over 6 requests per prompt length.
RUG increases average inference latency by only a negligible amount over original BAGEL. Memory and computational overhead are also small:
After LoRA merging, the only additional model parameters are the 128 Relay Token embeddings: 458,752 parameters (0.459M; approximately 0.003% of either baseline's parameter count).
Reference