- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DiFuse investigates whether a unified multimodal model's native diffusion representations can improve spatial understanding. For each image, the frozen generation branch is conditioned on a fixed depth-map instruction, and features are extracted from a selected DiT layer and denoising timestep. The features are compressed, spatially aligned, and injected into visual-token states in the understanding decoder through residual adapters. The adapters and decoder LoRA parameters are trained using ordinary image-text instruction supervision. Experiments on UniWorld-V1 and OmniGen2 compare DiFuse with matched SFT baselines, with stronger spatial gains on UniWorld-V1 and smaller gains on OmniGen2.
Critical strengths
- Reusing native generation features for understanding explores a useful direction of information transfer within unified multimodal models.
- Matching instruction data and training schedules with SFT baselines helps isolate the contribution of the added feature pathway.
- Feature-shuffling controls, design ablations, and evaluation on two backbones provide useful evidence about feature content and spatial correspondence.
Critical weaknesses
- Without comparisons against outputs or intermediate features from dedicated depth-estimation models, the necessity and performance–cost advantages of the diffusion feature pathway remain unclear.
- For uncached images, the extra MLLM conditioning pass and DiT execution up to the selected timestep incur unquantified inference costs whose justification requires end-to-end latency, computation, and memory measurements.
- Why does DiFuse preserve general understanding on UniWorld-V1 but reduce it relative to matched SFT on OmniGen2, and what does this discrepancy imply for its transferability across backbones?
Questions for authors
What evidence demonstrates an accuracy–cost advantage over using outputs or features from a dedicated depth-estimation model when the full conditioning and diffusion feature-extraction costs are included?
5 comments
Response to Reviewer gWP2 (part 2/3)
Weakness 2
For uncached images, the extra MLLM conditioning pass and DiT execution up to the selected timestep incur unquantified inference costs whose justification requires end-to-end latency, computation, and memory measurements.
Thank you for pointing out the missing cost measurements. We have added DiT call counts, latency, peak GPU memory, and cache size.
1. We have measured the new-image and cache-hit request costs of the current explicit-fusion implementation.
We measured UniWorld-V1 on 100 VSR examples, using one A800 80 GB GPU and batch size 1:
| Mode | DiT calls | Latency p50 / p95 (s) | Peak allocated GPU memory (GiB) | Cache per image |
|---|---|---|---|---|
| SFT | 0 | 0.157 / 0.272 | 15.68 | None |
| DiFuse: new image | 28 | 18.091 / 20.004 | 66.74 | Median: 2.345 MiB |
| DiFuse: cache hit | 0 | 0.445 / 0.581 | 16.83 | Existing file reused |
The new-image request includes conditioning, DiT extraction, compression, cache writing and reading, and answering. The 28 calls include a final partial forward pass that stops at the selected layer. Latency is measured after model loading and 3 warm-up requests, excluding a full process cold start. We report the maximum of the peak allocated GPU memory measured during each request. The new-image mode includes the feature-extraction and answering models resident at the same time.
This implementation is used to test whether native generative representations can help understanding, with spatial tasks providing empirical tests of this direction. The added measurements quantify the actual cost of this approach.
2. Caching reduces repeated computation, and future work will explore distillation to remove online DiT feature extraction.
The current implementation tests the role of generative representations in understanding, and cached features can be reused. In future work, we plan to use DiFuse as a teacher and distill its spatial ability into the understanding model. The goal is to use the generation branch only during training, without extracting DiT features at inference.
Response to Reviewer gWP2 (part 3/3)
Weakness 3
Why does DiFuse preserve general understanding on UniWorld-V1 but reduce it relative to matched SFT on OmniGen2, and what does this discrepancy imply for its transferability across backbones?
Thank you for pointing out the difference between the two backbones.
1. The per-task results on OmniGen2 show that the effects of generative representations are selective.
OmniGen2 achieves Avg-G/Avg-S of 67.74/65.87. Relative to SFT, RealWorldQA improves from 64.31 to 66.27, while MME-norm decreases from 76.71 to 75.15 and MMStar from 58.00 to 57.90, showing that different understanding tasks respond differently to the fused features. Recomputed using the updated per-question predictions, the 95% paired bootstrap intervals for the Avg-S/Avg-G differences are [+0.075, +0.926] and [−0.500, +0.607], respectively. The spatial-average improvement receives paired statistical support at the evaluated checkpoint, while the general-average point estimate remains close to SFT.
2. The existing ablations show that fusion configurations change the trade-off between general and spatial performance.
In Table 4, extending the injection range from layers 0–7 to 0–13 in the same UniWorld-V1 model reduces Avg-G from 71.94 to 71.11. This suggests that changes in general performance are related to how features are injected, and that each backbone requires its own adaptation.
The two-backbone experiments support the applicability of this pathway across architectures, while the magnitude of the gains still depends on the backbone and fusion configuration. The existing ablations support configuration sensitivity but cannot by themselves fully explain the difference between the backbones.
Question
What evidence demonstrates an accuracy–cost advantage over using outputs or features from a dedicated depth-estimation model when the full conditioning and diffusion feature-extraction costs are included?
Thank you for suggesting this comparison. Our question is whether a UMM’s existing generative representations can help understanding without introducing an additional spatial model. We therefore focus this round of experiments on internal representation reuse, rather than extending it to external augmentation with a dedicated depth model.
The current experiments report the understanding gains from this pathway and its complete costs, including conditioning and DiT extraction. We do not use these results to claim an accuracy–cost advantage over dedicated depth models. The research scope and main results are described under Weakness 1, and the cost measurements under Weakness 2.
Response to Reviewer gWP2 (part 1/3)
Thank you for the careful review and specific suggestions. We also appreciate your recognition of native generative representation reuse for understanding, the matched SFT setup, and the ablations. We conducted more extensive hyperparameter tuning to address the limited configuration exploration for OmniGen2 in the original submission and updated its results in Table 1 of the revised manuscript. We address each concern below and provide the additional results.
Weakness 1
Thank you for the suggestions on feature sources and comparisons. As you note in Critical Strengths 1, reusing native generative features explores a valuable direction of information transfer within UMMs. We would like to clarify the research scope around this question.
1. Our core question is whether native generative representations can help understanding in UMMs.
We test the role of native generative representations through spatial tasks, using general instruction data without adding dedicated spatial training data. Our contribution therefore includes testing the value of reusing generative representations already available within the model.
2. Table 1 provides empirical evidence for native generative representations helping understanding on spatial tasks.
The updated Table 1 is shown below; the UniWorld-V1 results remain unchanged. Under general instruction tuning, both backbones achieve higher spatial averages than matched SFT, while the updated OmniGen2 general average remains close to SFT. These results provide empirical evidence that native generative representations can improve spatial understanding, while also showing that the magnitude of the gains varies across backbones.