acceptodds
Under review as a conference paper at ICLR 2027

DiFuse: Fusing Native Diffusion Representations for Spatial Reasoning in Unified Multimodal Models

Abstract

Unified multimodal models (UMMs) jointly support visual understanding and image generation, yet information flow between these two capabilities remains largely asymmetric: understanding representations routinely condition generation, whereas generative representations are rarely reused for multimodal reasoning. In this work, we investigate whether intermediate representations from a UMM’s native diffusion branch can enhance spatial reasoning without compromising general multimodal understanding. We propose DiFuse, which selectively incorporates intermediate diffusion representations into the understanding pathway of UMMs. Through systematic layer-wise analysis, we identify which diffusion representations are most informative and where they can be integrated with minimal interference to semantic processing. Guided by these findings, DiFuse spatially aligns diffusion features with visual tokens and injects them into selected multimodal decoder layers via residual fusion. Extensive experiments on UniWorld-V1 show that, DiFuse consistently improves spatial reasoning over a matched SFT baseline while maintaining broadly comparable performance on general multimodal benchmarks. Further analyses show that these gains depend critically on the choice of diffusion representations, injection positions, and the preservation of spatial correspondence between diffusion features and visual tokens. Results on OmniGen2 further demonstrate the transferability of DiFuse across unified multimodal architectures. Overall, our findings show that native diffusion representations can provide complementary spatial cues for multimodal reasoning when selectively integrated in a structure-preserving manner.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Unified Multimodal Models, Generation-to-Understanding, Diffusion Representation, Spatial Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gWP2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DiFuse investigates whether a unified multimodal model's native diffusion representations can improve spatial understanding. For each image, the frozen generation branch is conditioned on a fixed depth-map instruction, and features are extracted from a selected DiT layer and denoising timestep. The features are compressed, spatially aligned, and injected into visual-token states in the understanding decoder through residual adapters. The adapters and decoder LoRA parameters are trained using ordinary image-text instruction supervision. Experiments on UniWorld-V1 and OmniGen2 compare DiFuse with matched SFT baselines, with stronger spatial gains on UniWorld-V1 and smaller gains on OmniGen2.

Critical strengths
  1. Reusing native generation features for understanding explores a useful direction of information transfer within unified multimodal models.
  2. Matching instruction data and training schedules with SFT baselines helps isolate the contribution of the added feature pathway.
  3. Feature-shuffling controls, design ablations, and evaluation on two backbones provide useful evidence about feature content and spatial correspondence.
Critical weaknesses
  1. Without comparisons against outputs or intermediate features from dedicated depth-estimation models, the necessity and performance–cost advantages of the diffusion feature pathway remain unclear.
  2. For uncached images, the extra MLLM conditioning pass and DiT execution up to the selected timestep incur unquantified inference costs whose justification requires end-to-end latency, computation, and memory measurements.
  3. Why does DiFuse preserve general understanding on UniWorld-V1 but reduce it relative to matched SFT on OmniGen2, and what does this discrepancy imply for its transferability across backbones?
Questions for authors

What evidence demonstrates an accuracy–cost advantage over using outputs or features from a dedicated depth-estimation model when the full conditioning and diffusion feature-extraction costs are included?

5 comments
AuthorsOpenReview ↗

Response to Reviewer gWP2 (part 1/3)

Thank you for the careful review and specific suggestions. We also appreciate your recognition of native generative representation reuse for understanding, the matched SFT setup, and the ablations. We conducted more extensive hyperparameter tuning to address the limited configuration exploration for OmniGen2 in the original submission and updated its results in Table 1 of the revised manuscript. We address each concern below and provide the additional results.

Weakness 1

Without comparisons against outputs or intermediate features from dedicated depth-estimation models, the necessity and performance–cost advantages of the diffusion feature pathway remain unclear.

Thank you for the suggestions on feature sources and comparisons. As you note in Critical Strengths 1, reusing native generative features explores a valuable direction of information transfer within UMMs. We would like to clarify the research scope around this question.

1. Our core question is whether native generative representations can help understanding in UMMs.

We test the role of native generative representations through spatial tasks, using general instruction data without adding dedicated spatial training data. Our contribution therefore includes testing the value of reusing generative representations already available within the model.

2. Table 1 provides empirical evidence for native generative representations helping understanding on spatial tasks.

The updated Table 1 is shown below; the UniWorld-V1 results remain unchanged. Under general instruction tuning, both backbones achieve higher spatial averages than matched SFT, while the updated OmniGen2 general average remains close to SFT. These results provide empirical evidence that native generative representations can improve spatial understanding, while also showing that the magnitude of the gains varies across backbones.

BackboneVariantAvg-GAvg-SMME-normMMBMMStarGQARWQAVSREmbSpatialCVB-3D3DSR-C
UniWorld-V1Base71.9271.0882.3583.9362.7460.9269.6787.3269.9781.0845.94
UniWorld-V1SFT71.51 (−0.41)71.88 (+0.80)78.2384.8863.7660.969.887.8170.9981.9246.79
UniWorld-V1DiFuse71.94 (+0.02)73.68 (+2.60)80.8284.5462.9761.769.6789.3672.3984.5848.38
OmniGen2Base66.0465.0176.5577.4956.7260.2259.2280.7761.4674.0843.73
OmniGen2SFT67.69 (+1.65)65.37 (+0.36)76.7178.2658.0061.1764.3182.4162.274.7542.12
OmniGen2DiFuse (updated)67.74 (+1.70)65.87 (+0.86)75.1578.2657.961.1166.2781.9162.6676.1742.74
AuthorsOpenReview ↗

Response to Reviewer gWP2 (part 2/3)

Weakness 2

For uncached images, the extra MLLM conditioning pass and DiT execution up to the selected timestep incur unquantified inference costs whose justification requires end-to-end latency, computation, and memory measurements.

Thank you for pointing out the missing cost measurements. We have added DiT call counts, latency, peak GPU memory, and cache size.

1. We have measured the new-image and cache-hit request costs of the current explicit-fusion implementation.

We measured UniWorld-V1 on 100 VSR examples, using one A800 80 GB GPU and batch size 1:

ModeDiT callsLatency p50 / p95 (s)Peak allocated GPU memory (GiB)Cache per image
SFT00.157 / 0.27215.68None
DiFuse: new image2818.091 / 20.00466.74Median: 2.345 MiB
DiFuse: cache hit00.445 / 0.58116.83Existing file reused

The new-image request includes conditioning, DiT extraction, compression, cache writing and reading, and answering. The 28 calls include a final partial forward pass that stops at the selected layer. Latency is measured after model loading and 3 warm-up requests, excluding a full process cold start. We report the maximum of the peak allocated GPU memory measured during each request. The new-image mode includes the feature-extraction and answering models resident at the same time.

This implementation is used to test whether native generative representations can help understanding, with spatial tasks providing empirical tests of this direction. The added measurements quantify the actual cost of this approach.

2. Caching reduces repeated computation, and future work will explore distillation to remove online DiT feature extraction.

The current implementation tests the role of generative representations in understanding, and cached features can be reused. In future work, we plan to use DiFuse as a teacher and distill its spatial ability into the understanding model. The goal is to use the generation branch only during training, without extracting DiT features at inference.

AuthorsOpenReview ↗

Response to Reviewer gWP2 (part 3/3)

Weakness 3

Why does DiFuse preserve general understanding on UniWorld-V1 but reduce it relative to matched SFT on OmniGen2, and what does this discrepancy imply for its transferability across backbones?

Thank you for pointing out the difference between the two backbones.

1. The per-task results on OmniGen2 show that the effects of generative representations are selective.

OmniGen2 achieves Avg-G/Avg-S of 67.74/65.87. Relative to SFT, RealWorldQA improves from 64.31 to 66.27, while MME-norm decreases from 76.71 to 75.15 and MMStar from 58.00 to 57.90, showing that different understanding tasks respond differently to the fused features. Recomputed using the updated per-question predictions, the 95% paired bootstrap intervals for the Avg-S/Avg-G differences are [+0.075, +0.926] and [−0.500, +0.607], respectively. The spatial-average improvement receives paired statistical support at the evaluated checkpoint, while the general-average point estimate remains close to SFT.

2. The existing ablations show that fusion configurations change the trade-off between general and spatial performance.

In Table 4, extending the injection range from layers 0–7 to 0–13 in the same UniWorld-V1 model reduces Avg-G from 71.94 to 71.11. This suggests that changes in general performance are related to how features are injected, and that each backbone requires its own adaptation.

The two-backbone experiments support the applicability of this pathway across architectures, while the magnitude of the gains still depends on the backbone and fusion configuration. The existing ablations support configuration sensitivity but cannot by themselves fully explain the difference between the backbones.

Question

What evidence demonstrates an accuracy–cost advantage over using outputs or features from a dedicated depth-estimation model when the full conditioning and diffusion feature-extraction costs are included?

Thank you for suggesting this comparison. Our question is whether a UMM’s existing generative representations can help understanding without introducing an additional spatial model. We therefore focus this round of experiments on internal representation reuse, rather than extending it to external augmentation with a dedicated depth model.

The current experiments report the understanding gains from this pathway and its complete costs, including conditioning and DiT extraction. We do not use these results to claim an accuracy–cost advantage over dedicated depth models. The research scope and main results are described under Weakness 1, and the cost measurements under Weakness 2.

Reviewer_NAjQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DiFuse investigates whether intermediate representations from a unified multimodal model’s native diffusion branch can improve spatial reasoning. Given an image and a fixed “generate the depth map” prompt, it extracts DiT features, spatially aligns them with visual tokens, and injects them into the understanding pathway. Relative to a supervised fine-tuning (SFT) baseline trained with the same data and optimization settings, the average score across four spatial benchmarks rises from 71.88 to 73.68 on UniWorld-V1 and from 65.37 to 65.84 on OmniGen2.

Critical strengths
  1. The motivation is meaningful and clearly developed. The introduction explains that understanding representations commonly condition generation in unified models, whereas representations from the generation branch are rarely reused for understanding. This motivates the investigation of native DiT features (§1).

  2. The main experiments include a matched SFT baseline. Within each backbone, SFT and DiFuse use the same instruction data, LoRA configuration, and number of training steps. DiFuse improves all four spatial benchmarks on UniWorld-V1, providing initial evidence of its effectiveness (§4.1–4.2, Table 1).

  3. The analysis covers several design choices. The paper examines the DiT source layer, denoising timestep, decoder injection layers, fusion strength, and spatial correspondence, and tests a second backbone. In particular, shuffling the spatial positions of feature tokens lowers the spatial average from 73.68 to 71.84, supporting the importance of spatial alignment (Tables 3–5).

Critical weaknesses
  1. The gains are modest, and their robustness to training randomness is unknown. DiFuse improves the spatial average over matched SFT by 1.80 points on UniWorld-V1 and only 0.47 points on OmniGen2 (Table 1). The paper reports neither means and standard deviations across training seeds nor paired confidence intervals over evaluation examples. It is therefore difficult to assess the stability of these differences, especially on the second backbone. Deterministic decoding does not address training variability.

  2. The paper does not establish whether intermediate-feature fusion is better than directly supplying spatial information. The DiT features are extracted using a “generate the depth map” prompt, but the paper does not compare DiFuse with a baseline that obtains a depth map and supplies it alongside the original image to the understanding model (Figure 3, §3.2). Such a comparison would test whether the proposed fusion offers an advantage over explicit depth information. A fair implementation would account for the pretrained visual encoder’s input interface—for example, by supplying depth as a second image or using a parameter-matched adapter for an additional channel—and report the associated inference cost.

  3. The contribution of diffusion features remains partly confounded with the additional fusion pathway. Matched SFT scores 71.88 on the spatial average; replacing diffusion features with random features while retaining the fusion pathway yields 72.80, compared with 73.68 for full DiFuse (Tables 1 and 5). Thus, the random-feature control recovers roughly half of the total gain, suggesting that added capacity or training effects may contribute substantially. The fusion adapters contain approximately 208 million trainable parameters on UniWorld-V1 (Appendix Table 8). Capacity-matched comparisons using features from the original visual encoder or other frozen visual representations would better isolate the value of native DiT features.

  4. The trade-off between performance and cost is unclear. In Figure 1(c), VST-SFT achieves a larger spatial gain over the shared base model, accompanied by a modest reduction in general understanding. Its spatial average in Table 2 is 77.17, compared with DiFuse’s 73.68. These methods differ in training and are not a controlled comparison, but the results suggest that DiFuse’s potential advantage lies in its general-versus-spatial performance trade-off rather than absolute spatial performance. The paper also does not report end-to-end latency, peak memory, or cache requirements when processing a new image with the DiT (§3.2, Appendices A.3–A.4), making the practical trade-off difficult to assess.

  5. The presentation could be improved. The motivation in the first part of the introduction is clear, but the abstract and the end of the introduction repeat several experimental conclusions and could be more concise. Text in Figure 1 is difficult to read, and the notation and formatting around Equation 7 in §3.3 need checking. Introducing Figure 2 earlier might also help readers grasp how DiFuse relates to prior approaches.

Questions for authors
  1. What are the mean and standard deviation of the main UniWorld-V1 and OmniGen2 results across multiple training seeds? Are the paired improvements over matched SFT stable across evaluation examples?
  2. Under comparable parameter and inference budgets, how does supplying the original image together with an explicit depth map compare with intermediate-feature fusion? How is the depth map obtained, and what is its quality?
  3. With matched fusion capacity and training settings, how do features from the original visual encoder or other frozen visual representations compare with DiT features and the reported random-feature control?
  4. If preserving general understanding is a principal advantage, how do the authors assess the performance and computational trade-off with VST-SFT shown in Figure 1(c) and Table 2?
8 comments
AuthorsOpenReview ↗

Response to Reviewer NAjQ (part 1/6)

Thank you for the careful review and for recognizing the value of the motivation, matched SFT setup, and design ablations. Following your suggestions, we have added paired statistical analyses, an original visual-encoder feature control, and inference cost measurements. We also conducted more extensive hyperparameter tuning to address the limited configuration exploration for OmniGen2 in the original submission and updated its results in Table 1 of the revised manuscript. We address each point below.

Weakness 1

The gains are modest, and their robustness to training randomness is unknown. DiFuse improves the spatial average over matched SFT by 1.80 points on UniWorld-V1 and only 0.47 points on OmniGen2 (Table 1). The paper reports neither means and standard deviations across training seeds nor paired confidence intervals over evaluation examples. It is therefore difficult to assess the stability of these differences, especially on the second backbone. Deterministic decoding does not address training variability.

The updated Table 1 is shown below; the UniWorld-V1 results remain unchanged.

BackboneVariantAvg-GAvg-SMME-normMMBMMStarGQARWQAVSREmbSpatialCVB-3D3DSR-C
OmniGen2Base66.0465.0176.5577.4956.7260.2259.2280.7761.4674.0843.73
OmniGen2SFT67.69 (+1.65)65.37 (+0.36)76.7178.2658.0061.1764.3182.4162.274.7542.12
OmniGen2DiFuse (updated)67.74 (+1.70)65.87 (+0.86)75.1578.2657.961.1166.2781.9162.6676.1742.74
  1. Paired statistical analysis supports spatial gains on both backbones.

We performed 10,000 paired cluster bootstrap replicates using the saved per-question predictions. We preserved each benchmark’s scoring rules and recomputed the equal-weight average across the four spatial benchmarks.

ComparisonAvg-S difference (points)95% confidence interval
UniWorld-V1: DiFuse − SFT1.8[+1.18, +2.44]
UniWorld-V1: DiFuse − Random0.88[+0.30, +1.46]
OmniGen2: updated DiFuse − SFT0.503[+0.075, +0.926]

Both UniWorld-V1 confidence intervals lie above zero. At the evaluated checkpoints, the gain over matched SFT and the additional gain over Random, which retains the fusion pathway, both receive support from the paired analysis. The updated OmniGen2 achieves Avg-G/Avg-S of 67.74/65.87, and its confidence interval for the Avg-S difference from SFT also lies above zero. Its Avg-G change relative to SFT is +0.047 percentage points, with a 95% paired interval of [−0.500, +0.607]; the general-average point estimate remains close to SFT.

2. Multiple ablation configurations achieve higher spatial averages than SFT.

Across the timesteps, injection ranges, and fusion coefficients reported in Table 4, all configurations achieve Avg-S values of 72.08–73.68, above SFT’s 71.88. The spatial gains are therefore consistent across the tested configurations. Together with the paired analysis, these results provide evidence for the observed gains under configuration changes and sample resampling.

3. We do not add further training-seed runs in this rebuttal because of the cost of full post-training and the rebuttal timeline.

Multi-seed validation would require repeated full post-training runs of SFT and DiFuse on both backbones, at substantial computational cost. In this round, we prioritized the additional feature-source control and paired statistical and inference cost analyses to address the source of the gains, sample uncertainty, and practical costs. These experiments provide additional evidence within their respective scopes, so we do not extend this round to repeated multi-seed training.

AuthorsOpenReview ↗

Response to Reviewer NAjQ (part 2/6)

Weakness 2

The paper does not establish whether intermediate-feature fusion is better than directly supplying spatial information. The DiT features are extracted using a “generate the depth map” prompt, but the paper does not compare DiFuse with a baseline that obtains a depth map and supplies it alongside the original image to the understanding model (Figure 3, §3.2). Such a comparison would test whether the proposed fusion offers an advantage over explicit depth information. A fair implementation would account for the pretrained visual encoder’s input interface—for example, by supplying depth as a second image or using a parameter-matched adapter for an additional channel—and report the associated inference cost.

Thank you for suggesting this comparison.

1. Our core question is whether native generative representations can help understanding in UMMs, with spatial tasks used to test this direction.

As you recognized in Critical Strengths, existing UMMs typically use understanding representations to support generation, while reusing generative representations for understanding remains worth exploring. DiFuse extracts intermediate features from the native diffusion branch and injects them into the understanding pathway to investigate whether the model’s existing generative representations can help understanding, rather than aiming for the best spatial leaderboard performance.

2. Spatial gains under general instruction tuning support the value of reusing native generative representations.

DiFuse and matched SFT use the same general instruction data, without adding dedicated spatial training data. Under this setting, DiFuse improves Avg-S over SFT by 1.80 points on UniWorld-V1, with support from paired statistical analysis. This provides empirical evidence for this direction of information transfer.

We test whether native intermediate generative representations can be used effectively by the understanding branch. An explicit depth map provides spatial information at the output level; supplying it as an additional image is different from the internal representation reuse studied here. We therefore focused the added feature-source controls on this motivation, as described under Weakness 3. We did not extend this round to a budget-matched RGB-plus-depth comparison and do not draw conclusions about the relative merits of the two input forms.

AuthorsOpenReview ↗

Response to Reviewer NAjQ (part 3/6)

Weakness 3

The contribution of diffusion features remains partly confounded with the additional fusion pathway. Matched SFT scores 71.88 on the spatial average; replacing diffusion features with random features while retaining the fusion pathway yields 72.80, compared with 73.68 for full DiFuse (Tables 1 and 5). Thus, the random-feature control recovers roughly half of the total gain, suggesting that added capacity or training effects may contribute substantially. The fusion adapters contain approximately 208 million trainable parameters on UniWorld-V1 (Appendix Table 8). Capacity-matched comparisons using features from the original visual encoder or other frozen visual representations would better isolate the value of native DiT features.

Thank you for pointing out that additional parameters and training may themselves contribute to the gains. We have added an original visual-encoder feature control.

1. DiT features achieve higher general and spatial averages with the same fusion architecture.

The added control compresses features from the frozen original visual encoder to 1,024 dimensions with PCA, uses the same fusion architecture, and trains its own fusion modules and LoRA weights in a separate run. DiFuse’s Avg-G/Avg-S are 0.98/0.75 points higher than this control.

VariantAvg-GAvg-SMME-normMMBMMStarGQARWQAVSREmbSpatialCVB-3D3DSR-C
SFT71.5171.8878.2384.8863.7660.969.887.8170.9981.9246.79
Random features71.9172.880.3684.4564.7360.7469.2888.9572.018248.23
Original visual-encoder features70.9672.9378.5383.9362.6260.8368.8988.7971.8782.6748.38
Aligned DiT features (ours)71.9473.6880.8284.5462.9761.769.6789.3672.3984.5848.38

2. Random-feature and shuffle experiments further support the role of feature content and spatial correspondence.

Random achieves higher Avg-S than SFT, suggesting that additional parameters or training effects may contribute part of the gain. DiFuse and Random are trained independently. DiFuse achieves a further 0.88-point gain, and the paired confidence interval lies above zero (see Weakness 1).

With the Aligned weights fixed, applying Image shuffle or Token shuffle only during evaluation reduces Avg-S from 73.68 to 72.60/71.84. This shows that the trained model’s performance depends on image-specific information and the spatial correspondence of feature tokens.

Reviewer_duMHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DiFuse extracts native diffusion features using a fixed depth prompt, pools and PCA-compresses them, and injects them into a unified model's understanding branch through residual adapters. Only adapters and LoRA are trained to improve spatial understanding.

Critical strengths
  1. Native-branch reuse is well motivated, with clearly specified spatial alignment and zero initialization.
  2. The SFT comparison matches data and LoRA settings: UniWorld-V1 improves on all four spatial benchmarks, raising Avg-S from 71.88 to 73.68.
  3. Random-feature and image/token-shuffling controls, plus a second backbone, are useful; the authors acknowledge the contribution of added capacity.
Critical weaknesses

1. Scope and generalizable insight are limited. The recipe requires an available native diffusion branch, a fixed depth prompt and backbone-specific configurations. The experiments do not sufficiently explain which generative representations reliably help understanding, under what conditions, or whether the benefit specifically requires generative information rather than an additional spatial pathway. UniWorld-V1 gains only 0.88 points over random features, without uncertainty estimates or a clear statement of whether corrupted-feature controls are retrained (Table 5). On OmniGen2, spatial performance improves by only 0.47 points over SFT while general performance falls by 0.67 (Table 1). The evidence therefore mainly supports a working fusion recipe in particular configurations; its broader significance remains underdeveloped.

2. The practical benefit–cost trade-off is unresolved. Extraction follows part of the denoising trajectory (§3.2), and caching does not remove computation for new images. Parameter counts are reported, but actual DiT evaluations, latency, peak memory and cache size are missing, making the practical value of the modest gains difficult to assess.

3. Selection/evaluation separation is unclear. Section 4.3 selects layers, timestep and fusion coefficient using the same named benchmarks as the main results, without identifying an independent development split. Appendix A.4 also leaves PCA fitting data unspecified. These omissions limit the generalization claim, but do not establish test leakage.

Questions for authors
  1. Which data determine hyperparameters and PCA, and are they independent of final evaluation?
  2. Are Table 5 controls separately retrained? Please provide paired uncertainty for real versus random features and clarify the supported applicability. If feasible, a same-adapter native non-diffusion feature control would help identify the source of improvement.
  3. What are the DiT evaluations per new image, cold/warm latency, peak memory and cache size?
5 comments
AuthorsOpenReview ↗

Response to Reviewer duMH (part 1/3)

Weakness 1

Scope and generalizable insight are limited. The recipe requires an available native diffusion branch, a fixed depth prompt and backbone-specific configurations. The experiments do not sufficiently explain which generative representations reliably help understanding, under what conditions, or whether the benefit specifically requires generative information rather than an additional spatial pathway. UniWorld-V1 gains only 0.88 points over random features, without uncertainty estimates or a clear statement of whether corrupted-feature controls are retrained (Table 5). On OmniGen2, spatial performance improves by only 0.47 points over SFT while general performance falls by 0.67 (Table 1). The evidence therefore mainly supports a working fusion recipe in particular configurations; its broader significance remains underdeveloped.

Thank you for your comments. Our core research question is whether a unified multimodal model’s native generative representations can improve understanding. We investigate this through spatial reasoning tasks and train on general instruction data from LLaVA-Mix-665K, without adding dedicated spatial training data. To address your concerns, we have added feature-source controls and paired statistical analyses. We also conducted further hyperparameter tuning to address the limited configuration exploration for OmniGen2 in the original submission and updated the corresponding results in Table 1 of the revised manuscript. We clarify the conclusions supported by these experiments below.

1. The gains depend on the diffusion state and injection location.

On UniWorld-V1, the source-layer screening results do not improve monotonically with diffusion-layer depth (Table 3). The timestep and injection location also affect performance (Table 4): injecting into decoder layers 0–7 gives an Avg-G/Avg-S of 71.94/73.68, whereas extending injection to layers 0–13 reduces these scores to 71.11/73.08. These results inform which representations to select and where to inject them.

2. The added feature-source control shows an additional spatial gain from diffusion representations.

VariantAvg-GAvg-SMME-normMMBMMStarGQARWQAVSREmbSpatialCVB-3D3DSR-C
SFT71.5171.8878.2384.8863.7660.969.887.8170.9981.9246.79
Random features71.9172.880.3684.4564.7360.7469.2888.9572.018248.23
Original visual-encoder features70.9672.9378.5383.9362.6260.8368.8988.7971.8782.6748.38
Aligned DiT features (ours)71.9473.6880.8284.5462.9761.769.6789.3672.3984.5848.38

The added control uses features from the frozen original visual encoder and achieves an Avg-G/Avg-S of 70.96/72.93, compared with 71.94/73.68 for DiFuse. DiFuse is higher by 0.98 and 0.75 points, respectively. It also improves Avg-S over Random by 0.88 points.

3. Paired statistical analysis supports the additional gain over random features.

We performed 10,000 paired cluster bootstrap replicates using the saved per-question predictions. We preserved each benchmark’s scoring rules and recomputed the equal-weight average across the four spatial benchmarks.

ComparisonAvg-S difference (points)95% confidence interval
UniWorld-V1: DiFuse − SFT1.8[+1.18, +2.44]
UniWorld-V1: DiFuse − Random0.88[+0.30, +1.46]

Both intervals lie above zero. This supports an additional gain from diffusion features even when the fusion pathway is retained.

4. Disrupting spatial correspondence reduces the same model’s spatial reasoning performance.

Aligned and Random are trained independently. Image shuffle and Token shuffle use the same Aligned weights and apply perturbations only during evaluation. With parameters and weights unchanged, these perturbations reduce Avg-S from 73.68 to 72.60 and 71.84, respectively. This shows that image-specific information and the spatial correspondence of feature tokens affect the trained model’s performance.

AuthorsOpenReview ↗

Response to Reviewer duMH (part 2/3)

5. Updated results support cross-backbone applicability.

Following this additional tuning, OmniGen2 improves Avg-S over SFT by 0.503 percentage points, with an Avg-G change of +0.047 percentage points. The recomputed 95% paired bootstrap intervals are [+0.075, +0.926] and [−0.500, +0.607], respectively, supporting the spatial-average improvement while the general-average point estimate remains close to SFT. Taken together, the controls and statistical analyses on both backbones support that native generative representations can improve spatial understanding under general instruction tuning, providing a concrete method and empirical evidence for generation helping understanding within unified multimodal models.

BackboneVariantAvg-GAvg-SMME-normMMBMMStarGQARWQAVSREmbSpatialCVB-3D3DSR-C
OmniGen2Base66.0465.0176.5577.4956.7260.2259.2280.7761.4674.0843.73
OmniGen2SFT67.69 (+1.65)65.37 (+0.36)76.7178.2658.0061.1764.3182.4162.274.7542.12
OmniGen2DiFuse (updated)67.74 (+1.70)65.87 (+0.86)75.1578.2657.961.1166.2781.9162.6676.1742.74

Weakness 2

The practical benefit–cost trade-off is unresolved. Extraction follows part of the denoising trajectory (§3.2), and caching does not remove computation for new images. Parameter counts are reported, but actual DiT evaluations, latency, peak memory and cache size are missing, making the practical value of the modest gains difficult to assess.

Thank you for your comments. We have added runtime cost measurements and clarified the research goal of these experiments.

1. We use spatial tasks to test whether native generative representations can help understanding.

We train on general instruction data to study whether the understanding branch can use representations already available in the model’s generation branch. The value of this study therefore includes testing this direction of information transfer and should not be judged solely by the ratio of spatial-score gains to inference time. We also report the actual costs of the current implementation to clarify its practical requirements.

2. The measurements include new-image extraction costs, while caching avoids repeated DiT execution.

We measured UniWorld-V1 on 100 VSR examples, using one A800 80 GB GPU and batch size 1:

ModeDiT callsLatency p50 / p95 (s)Peak allocated GPU memory (GiB)Cache per image
SFT00.157 / 0.27215.68None
DiFuse: new image2818.091 / 20.00466.74Median: 2.345 MiB
DiFuse: cache hit00.445 / 0.58116.83Existing file reused

New-image latency includes conditioning, DiT feature extraction, compression, cache writing and reading, and answer generation. The DiT runs 27 full forward passes, followed by a final partial forward pass that stops at the selected layer. Timing excludes model loading and warm-up. The memory measurements include both the feature-extraction model and the answering model loaded at the same time.

The current experiments use explicit feature fusion to test the role of generative representations. In future work, we plan to use DiFuse as a teacher and distill the acquired spatial ability into the understanding model. The goal is to use the generation branch only during training and the understanding model alone at inference, retaining the gains while reducing online costs.

Weakness 3

Selection/evaluation separation is unclear. Section 4.3 selects layers, timestep and fusion coefficient using the same named benchmarks as the main results, without identifying an independent development split. Appendix A.4 also leaves PCA fitting data unspecified. These omissions limit the generalization claim, but do not establish test leakage.

Thank you for pointing out the missing protocol details. We clarify hyperparameter selection, checkpoint selection, and PCA fitting separately.

1. Checkpoints are determined by the preset training duration, not selected by benchmark scores.

All main experiments evaluate the final checkpoint after 3 training epochs. We do not select the highest-scoring checkpoint from intermediate checkpoints.

AuthorsOpenReview ↗

Response to Reviewer duMH (part 3/3)

2. Hyperparameters are chosen based on the balance between general and spatial performance in the ablations.

We did not use a separate development split. We selected the source layer, timestep, injection range, and fusion coefficient based on the balance between Avg-G and Avg-S on the reported benchmarks. Tables 3 and 4 provide the selection results and show sensitivity to these configurations. Notably, all reported configurations in Table 4 achieve Avg-S values of 72.08–73.68, exceeding SFT’s 71.88. Thus, the observed spatial gains are not limited to the final selected configuration.

3. PCA is fitted on training-side image features and then kept fixed.

PCA is fitted without supervision using only image features from the LLaVA-Mix-665K training set. It does not use final evaluation answers or scores. Once fitted, the PCA projection remains fixed for subsequent training and evaluation.

Question 1

Which data determine hyperparameters and PCA, and are they independent of final evaluation?

Thank you for asking about the experimental protocol. Hyperparameters are selected based on the reported benchmark results, without a separate development split. PCA is fitted on training-side image features, and each evaluated checkpoint is the final checkpoint after 3 training epochs. The selection procedures are detailed under Weakness 3.

Question 2

Are Table 5 controls separately retrained? Please provide paired uncertainty for real versus random features and clarify the supported applicability. If feasible, a same-adapter native non-diffusion feature control would help identify the source of improvement.

Thank you for the suggestions. Aligned and Random are trained independently. Image shuffle and Token shuffle perturb the same Aligned model only during evaluation. DiFuse improves Avg-S over Random by 0.88 points, with a 95% interval of [+0.30, +1.46]. We have also added an original visual-encoder feature control. Full results and the supported applicability are described under Weakness 1.

Question 3

What are the DiT evaluations per new image, cold/warm latency, peak memory and cache size?

Thank you for asking about practical costs. We have added latency, GPU memory, and cache-size measurements for new images and cache hits. Measurements are taken after model loading and warm-up and do not include a full process cold start. The results and DiT call-count details are provided under Weakness 2, point 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.