acceptodds
Under review as a conference paper at ICLR 2027

HiUni: Harmonizing Understanding and Generation in Shared Visual Space

15 views

Abstract

Unified multimodal models (UMMs) aim to support visual understanding and generation within a single framework. However, prevailing approaches often separate their visual representations or backbone computation to accommodate the different requirements of the two tasks, which limits the degree of unification. In this paper, we present HiUni, which revisits this design choice and explores whether a pretrained VLM can be directly extended into a unified model with a high degree of sharing. Specifically, HiUni reuses the same language aligned ViT and dense Transformer backbone for both capabilities, while confining task specific computation to the output side. Yet such a design introduces two key challenges. Direct generation in pixel space is difficult to optimize, while visual compression and semantic abstraction in pretrained VLMs may discard fine spatial details required for synthesis. To address these challenges, HiUni introduces a Pixel Diffusion Head tailored to direct pixel-space optimization and a lightweight HiGate pathway that provides fine-grained spatial details near the output stage. Extensive experiments show that HiUni achieves a similar average understanding score to the pretrained VLM while attaining competitive performance on image generation and editing. Controlled comparisons further demonstrate that a single dense backbone can effectively support both capabilities without task-specific backbone computation, highlighting a simpler path toward unified multimodal modeling.

Primary area:
foundation or frontier models, including LLMs
Keywords:
UMM, Multimodal

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Smg9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HiUni, a unified multimodal model extended directly from a pretrained VLM (Qwen3-VL-4B-Instruct). Understanding and generation share the same language-aligned ViT and the same dense Transformer backbone, and generation-specific computation is confined to the output side.

To handle the difficulty of optimizing diffusion in pixel space, the authors design a Pixel Diffusion Head. It removes block-wise AdaLN, injects the timestep through a time token, and keeps only a single timestep modulation before the output. To compensate for the spatial detail lost through patch merging and deep semantic abstraction, the authors introduce HiGate, which injects raw pixel features of the noisy image into the diffusion head through a token-wise gated residual.

The model is evaluated on understanding, text-to-image generation (including long text rendering in Chinese and English), and image editing. Its average understanding score is close to that of the base VLM, and its generation and editing results are comparable to larger unified models. The authors also compare dense and MoT backbones at the 2B scale and analyze the feature source and injection strength of HiGate.

Critical strengths
  • I think this is a fairly solid empirical study. The three-stage training pipeline, the resolution schedule that grows from 256 to 2048, the per-stage hyperparameters, and the head architecture are all documented in the appendix in good detail. The engineering is clearly mature.
  • The generation ability at the 4B scale is quite good and shows potential for scaling up. With 4B+1.3B parameters, HiUni reaches 47.98 on the Chinese Overall of Qwen-Image-Bench, 0.963 / 0.954 on LongText, and 6.70 / 6.81 on GEdit Overall. This is on par with larger models such as Bagel (7B+7B) and SenseNova-U1 (8B+8B), and I would be curious to see how this route performs at a larger scale.
  • The paper is clearly written and easy to follow.
Critical weaknesses
  • The paper reads more like an empirical study of pixel-space Transfusion design, and I do not see insights that match the "Harmonizing" in the title. To me, it is closer to a technical report for an industrial product and does not offer much novel insight. This direction has already been explored quite systematically: TUNA and Cheers study various unified representations and training recipes for understanding and generation, and architectures such as HiGate are themselves inspired by them. Some of the pixel-space training insights also come from JiT and related work. Feeding noisy pixels directly into the head to recover details has been done before in DiP [1] and DeCo [2]: the Patch Detailer Head in DiP takes both the DiT output and the original noisy pixel patch, and the pixel decoder in DeCo takes the full-resolution noisy image as input, yet the paper neither cites nor discusses these two works. The observation that removing block-wise AdaLN alleviates loss spikes in pixel space is interesting, but it feels more like an engineering measure for stable training than a new understanding of unified modeling.
  • The discussion of related work is missing. RepFusion [3] also feeds noisy visual inputs through the MLLM's own visual pathway and uses the LLM outputs to drive a lightweight denoiser, which is quite close to the core idea of HiUni. Since the two works appeared close in time, I do not count this as a missing citation, but I suggest that the authors discuss the mechanistic difference so that readers can judge the boundary of the contribution.
  • The ablations are too simple and incomplete. All ablations (Table 4, Table 5, Table 11) report only OneIG on the generation side, with no editing metrics such as ImgEdit or GEdit, and neither Qwen-Image-Bench nor LongText. For a unified model that emphasizes both generation and editing, a single benchmark is not enough to support the conclusions about the backbone choice and the HiGate design.
  • The dataset information is too limited. Appendix A.5 only states that the understanding and generation corpora come from proprietary datasets, and that the editing data include private data produced by an internal pipeline in addition to public data. I understand that private data may not be easy to disclose, but some details must be provided, for example whether the authors used data distilled from external models (such as BLIP3o-60k, which is generated by closed-source models). I am also concerned that the improvement over other works in the main experiments may largely come from higher-quality synthetic data.
  • Both the Intro and Sec. 4.3 present the dense vs. MoT comparison as an important argument. Yet in the only controlled comparison with the same initialization and the same recipe (Table 4), MoT-2B beats Dense-2B on both understanding (59.23 vs. 58.55) and generation (OneIG 0.345 vs. 0.291), and the authors still choose dense. Their argument is that, at a similar total parameter count, Dense-4B has clearly stronger understanding (65.04) and "competitive" generation (0.314). However, this overlooks the fact that Dense-4B is initialized from Qwen3-VL-4B while MoT-2B is initialized from Qwen3-VL-2B, so the two start from different places. Much of the understanding gap comes from the stronger base model rather than from the dense design itself, so this is not a fair comparison. Even so, Dense-4B still generates worse than MoT-2B.

[1] DiP: Taming Diffusion Models in Pixel Space, arXiv:2511.18822

[2] DeCo: Frequency-Decoupled Pixel Diffusion for End-to-End Image Generation, CVPR 2026

[3] RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space, arXiv:2606.14700

Questions for authors

See weaknesses.

  • If I understand correctly, the paper maps the timestep through an MLP and uses the output to replace the embedding of the <time> token, which then enters the LLM as one token in the sequence context. JanusFlow uses a similar time token, but Transfusion and BAGEL add the timestep embedding directly to every patch. Why did the authors choose this design? Is there an ablation against other ways of injecting the timestep?
  • Have the authors tried removing the understanding data and training only on generation and editing? This would surely hurt understanding, but could it improve generation and editing? I think this would help clarify whether understanding data is a help or a cost for generation.
9 comments
AuthorsOpenReview ↗

Response to Reviewer Smg9 Part [1]

We sincerely thank the reviewer for the constructive feedback and for recognizing the value of our empirical study, the detailed training pipeline, and the promising generation performance at the 4B scale. We have revised the manuscript to better clarify our contributions, strengthen the architectural comparisons, and provide a more balanced analysis of the trade-offs in unified modeling. Our revisions and additional results are summarized below.


[W1] Insights into harmonizing understanding and generation

We agree that pixel-space diffusion and residual detail injection have precedents. The central question of HiUni is whether a pretrained VLM's semantic visual pathway and dense backbone can serve both understanding and generation, without a separate generative visual encoder or task-specific backbone computation. Our experiments lead to three insights.

First, a shared dense backbone enables effective reuse of pretrained VLM capabilities across understanding and generation. In the controlled 2B comparison, Dense and MoT have similar understanding performance (58.55 vs. 59.23), while MoT is stronger on generation (0.345 vs. 0.291). A 4B shared backbone, initialized from a larger pretrained VLM, offers a practical alternative with stronger understanding and competitive generation under a similar total parameter budget. Joint training also retains comparable understanding to an understanding-only baseline (65.04 vs. 64.58).

Second, semantic modeling and fine-grained visual reconstruction can be effectively decoupled. HiGate supplies raw noisy-pixel features only near the generation output, improving OneIG from 0.267 to 0.314 with little change in understanding (Section 4.4). This decouples the requirements of semantic modeling and fine-detail prediction without introducing task-specific backbone branches.

Third, architectural sharing does not guarantee uniformed positive transfer. Under a fixed total token budget, adding editing supervision increases average understanding from 62.64 to 64.49, while OneIG decreases from 0.538 to 0.528 (Section 4.5).

We therefore use harmonizing to describe supporting distinct tasks within a highly shared architecture, not to claim that all tasks mutually improve. These strengths and limitations provide a more precise characterization of highly shared UMMs. We believe that demonstrating this feasibility and identifying the benefits and limitations of sharing provides useful insights into unified multimodal modeling, laying the foundation for more mutually beneficial interactions between understanding and generation in the future.


[W2] Discussion of related work

We thank the reviewer for highlighting these closely related studies. We have expanded Section 2 to better position HiUni relative to prior work.

TUNA constructs a shared visual representation by cascading a VAE encoder with a representation encoder, while TUNA-2 adopts an encoder-free design based on direct pixel embeddings. Cheers combines a VAE encoder, a VAE decoder, and a ViT as the visual encoder, and injected the output features of VAE into the diffusion head through residuals. In contrast, HiUni focuses on directly reusing the pretrained VLM's language-aligned ViT, patch merger, and dense backbone, and injects raw noisy-pixel features at the output side.

DiP and DeCo demonstrate the benefit of preserving pixel-level details in image generation. DiP uses a Patch Detailer Head, while DeCo introduces a pixel decoder conditioned on semantic features. HiUni does not claim novelty in such mechanism, its distinction lies in integrating pixel-level detail injection into a unified model that shares the same pretrained visual encoder and dense backbone across understanding and generation.

RepFusion is particularly relevant because it also processes noisy visual inputs using a pretrained multimodal model. However, it conditions a separate diffusion Transformer in representation space. In contrast, HiUni processes noisy and clean images through the same visual encoder and dense backbone and predicts pixels through a lightweight task-specific head.

These comparisons clarify that HiUni contributes a novel approach to architectural sharing and pretrained VLM reuse for UMM, rather than claiming novelty in individual generation components. We have added the relevant references and clarified these connections in Section 2.

AuthorsOpenReview ↗

Response to Reviewer Smg9 Part [2]

[W3] Ablations evaluated only on OneIG for generation

We agree that evaluating multiple generation benchmarks provides stronger evidence for comparisons. In our preliminary experiments, we found OneIG to be the most consistent benchmark with generation quality and to provide clear differentiation between early checkpoints. We therefore selected it as the primary generation metric for the sake of brevity. Following the reviewer's suggestion, we have extended the evaluations of the Table 4 and Table 5 to Qwen-Image-Bench, LongText, and OneIG, covering both Chinese and English prompts.

Additional results for Table 4

ModelQwen-Image-Bench ZHQwen-Image-Bench ENLongText ZHLongText ENOneIG ZHOneIG EN
HiUni-Dense-4B17.5017.230.1090.0850.3380.289
HiUni-Dense-2B17.1516.980.1000.0280.3190.262
HiUni-MoT-2B18.5018.370.1570.1410.3580.331
HiUni-Gen-4B23.2222.650.3610.2300.4090.371

Additional results for Table 5

ConfigurationQwen-Image-Bench ZHQwen-Image-Bench ENLongText ZHLongText ENOneIG ZHOneIG EN
Pixel Feature17.5017.230.1090.0850.3380.289
ViT Feature14.1914.060.0480.0200.3040.263
No Gate12.7112.130.0140.0040.2840.249

The additional evaluations confirm the observations in Sections 4.3 and 4.4. HiUni-MoT-2B consistently outperforms HiUni-Dense-2B on generation, while HiUni-Dense-4B achieves substantially stronger understanding performance. The generation-only baseline performs best across all six generation metrics, further revealing the generation trade-off of joint training. For HiGate, Pixel Feature consistently outperforms ViT Feature and No Gate across all six generation metrics.

The Table 4–5 ablations are early-stage checkpoints trained before editing supervision is introduced. We therefore do not treat direct ImgEdit or GEdit evaluation of those checkpoints as a meaningful editing ablation. Instead, Table 3 evaluates the fully trained model, while Section 4.5 and Appendix E test the effects of adding editing supervision.


[W4] Data description and influence

We agree that training-data composition matters for interpreting cross-model results. Figure 3 and Appendix A.5 describe the main data categories, use of public and proprietary sources, and model-generated supervision. We provide further quantitative details here.

Data sources. We do not use BLIP3o-60k or similar public distilled image-generation datasets. The generation corpus mostly comprises real image–text pairs, internally synthesized images represent 3.9% and 5.3% of generation tokens in pretraining and SFT, respectively, and are used mainly for style control. The understanding corpus includes the training splits of datasets such as ScienceQA, Visual Genome, RefCOCO, COCO, and DocVQA, which are not included in the evaluation set. Approximately half of the understanding pre-training data contain model-generated captions, largely from Qwen-series models, and about 97% of understanding SFT examples contain model-generated reasoning trajectories to train a think-mode model. The editing corpus combines public datasets and internally generated edit pairs, as described in Appendix A.5.

Leakage checks. We perform text matching and image near-duplicate detection against the reported evaluation benchmarks. The detected image overlap rates were 1.2% for MMStar, 1.0% for MMBench, and below 0.5% for the other understanding benchmarks, primarily from shared underlying public image sources. For image generation, we detected 20 relative prompts in OneIG-EN (1.8%) and one in LongText-EN (0.6%), with none detected in the other evaluated generation sets.

Controlled Comparison. Since most compared models do not disclose their complete training data and recipe, we do not attribute differences in Tables 1–3 solely to architecture design. Controlled experiments in Section 4.3 use the same underlying training sources and configuration for Dense-2B and MoT-2B, differing in backbone sharing, while the single-task baselines probe task-mixture effects. These comparisons support architectural and optimization conclusions more directly than cross-paper leaderboard rankings.

AuthorsOpenReview ↗

Response to Reviewer Smg9 Part [3]

[W5] Dense versus MoT comparison

Before answering this question, we would like to clarify that our goal is not to establish the superiority of shared backbones over task-specific designs. We have revised the Introduction and Section 4.3 to acknowledge these results explicitly. Here, we would like to reiterate the motivation of HiUni and the purpose of our architectural comparisons.

First, HiUni aims to explore whether a pretrained VLM can support both understanding and generation without introducing task-specific backbone computation, rather than demonstrating that Dense is generally superior to MoT. This highly shared design allows both tasks to reuse the same pretrained backbone, providing a simpler alternative to task-specific architectures.

Second, the ability to inherit stronger pretrained representations is an intended advantage of the shared Dense design. Under a similar total parameter budget, HiUni-Dense-4B can directly reuse the pre-trained Qwen3-VL-4B backbone, whereas HiUni-MoT-2B uses two task-specific branches initialized from Qwen3-VL-2B. We acknowledge that this comparison involves different pre-trained initializations and is not intended to establish the inherent superiority of Dense. Instead, it highlights a practical advantage of backbone sharing that, a larger pre-trained VLM can be reused for both tasks within a similar total parameter budget.

Our central claim is consequently one of feasibility and design flexibility: a single shared backbone is a viable UMM alternative, even though specialized computation can improve generation performance with the same activated parameters.


[Q1] Why use a time token instead of adding timestep embeddings to each image patch?

We thank the reviewer for this question. The time-token conditioning follows prior models such as JanusFlow and Hunyuan-Image-3.0, which incorporate timestep embeddings as tokens into the Transformer sequence. We adopt this approach as a simple implementation choice compatible with our shared backbone. We have clarified the corresponding description and references in Section 3.2.

We have not conducted a controlled comparison with patch-wise timestep addition under otherwise identical settings. Since timestep conditioning is not the focus of HiUni, we do not claim that the time-token design is superior to alternative approaches, but regard it as an established implementation choice.


[Q2] Would removing understanding data improve generation performance?

Yes. To test this, we introduce HiUni-Gen-4B, which uses the same pretrained initialization, visual pathway, backbone, and 0.5B diffusion head as HiUni-Dense-4B, but trains on generation-only data rather than the joint task mixture. The experimental setup is an early comparison before introducing image editing.

As shown in Table 4, generation-only training improves the average OneIG score from 0.314 to 0.390. The additional evaluations in W3 show consistent improvements across all six generation metrics. In particular, OneIG ZH/EN increases from 0.338/0.289 to 0.409/0.371, with similar gains on Qwen-Image-Bench and LongText.

These results reveal two important insights. First, a pretrained VLM can be effectively adapted into a competitive image generation model, highlighting a promising direction for extending VLMs beyond visual understanding. Second, while HiUni focuses on introducing generation capabilities while preserving the understanding performance of pretrained VLMs, the stronger performance of HiUni-Gen-4B highlights the potential for further improving generation through more effective unified optimization, opening up opportunities to advance both capabilities together.


We thank the reviewer again for the valuable suggestions, and hope our additional experiments and revisions have adequately addressed the concerns.

Reviewer_rDKFOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HiUni, a unified multimodal model that supports image understanding, generation, and editing within a shared visual space. Built upon a pretrained Qwen3-VL, HiUni shares the same vision encoder and dense Transformer backbone across tasks, eliminating the need for separate visual encoders or task-specific backbone computation. To enable pixel-space image generation, the authors introduce a Pixel Diffusion Head with simplified timestep modulation and HiGate, a lightweight residual pathway that preserves fine-grained spatial information. Experiments demonstrate that HiUni maintains the understanding performance of the pretrained VLM while achieving competitive image generation and editing results. Additional ablations investigate the trade-offs between shared and task-specific backbones and validate the effectiveness of HiGate.

Critical strengths
  1. HiUni demonstrates that image understanding and generation can share the same vision encoder and dense Transformer backbone without relying on separate visual encoders or task-specific backbone computation. This provides a simpler and promising direction for building unified multimodal models.

  2. HiUni preserves the understanding performance of the pretrained VLM while achieving competitive image generation and editing results. The ablation studies further validate the effectiveness of HiGate and demonstrate the feasibility of the proposed shared architecture.

Critical weaknesses

The paper compares HiUni directly with the original Qwen3-VL, which is already well trained for visual understanding. It would be helpful to also continue training Qwen3-VL on the same understanding data. This would help clarify how much of the performance gap comes from joint training.

Questions for authors

I am curious about how HiUni would compare with Qwen3-VL further trained on the same understanding data without the generation task. Could the authors provide such a comparison?

1 comment
AuthorsOpenReview ↗

Response to Reviewer rDKF

We thank the reviewer for the constructive suggestion and for recognizing the value of investigating this highly shared architecture.

[W1 & Q1] Comparison with understanding-only training

We agree that comparison with the original pre-trained Qwen3-VL does not isolate the effect of introducing image generation. We therefore add HiUni-Und-4B as an understanding-only baseline in Section 4.3 and Table 4. It uses the same Qwen3-VL-4B initialization and underlying understanding data sources as HiUni-Dense-4B, while excluding generation supervision. The controlled comparison uses matched early-stage training settings.

ModelMMBenchRealWorldQAMMStarHallusionBenchAI2DOCRBenchAvg.
HiUni-Dense-4B72.6065.4953.9348.9677.9571.3065.04
HiUni-Und-4B72.5263.7953.3346.9677.8573.0064.58

HiUni-Dense-4B achieves comparable average understanding performance to HiUni-Und-4B (65.04 vs. 64.58) across six benchmarks, outperforming it on five of them. These results suggest that incorporating image generation supervision does not substantially compromise understanding performance under the same training setting. We have revised Section 4.3 to clarify this finding.

We thank the reviewer again for this valuable suggestion, which has helped us provide a more rigorous analysis of the effects of unified multimodal modeling.

Reviewer_d5meOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper is about unified generating/understanding models. HiUni, proposed in this paper, and based on Qwen3-VL-4B in particular, is about taking a pretrained model and extending it into a unified image understanding, generation, and editing model. Results show competitive generation and editing while roughly preserving average understanding performance.

The main contribution of this paper is a particular architectral combination and its empirical validation. Pretrained initialization, shared backbones, pixel space generation, and residual detail are all pathways that have been explored before.

Critical strengths

The design of the proposed model works over three explicitly separated tasks with competitive results across the three of them. The feat of having good results for the new taks while preserving performance on the original understanding task is a strength of this model.

The proposed pixel residual has convincing value (Table 5). A significant increase on OneIG while relatively unchanged understanding.

The evaluation is properly done across the several taks with broad scope.

Critical weaknesses

My main concern is that the exact incremental contirbutons over existing works is not properly established. With a large amount of works on this topic, the scope, goals, and contributions must be explicitly clarified. TUNA already explored shared representation, TUNA2 explored pixel-based unification. HiGate is inspired by previous work as well (Cheers). The specific choice to retain ViT is not properly compared against alternative design options in the existing literature.

The claims of preserved understanding need to be discussed more or with more nuance. Average overall score is does not expose significant degradation across certain tasks like OCR (OCRBench) and a large MMBench drop.

More importantly, in the experiments it is not clear if the inference settings / budgets are similar for competing methods (reasoning and continued decoding). Establishing fairness in evaluation might not be trivial for this line of work but important.

Benefits from the multi-task modeling are not visible in the experiments. Adding generation/editing certainly does not help understanding, but not clear if the understanding backbone is helping generation/editing/

Questions for authors

Check the weakness section, in particular, I would appreciate if the authors can discuss/comment on the proposed ideas within the literature, for example, what measurable advantage does retaining the pretrained ViT and patch merger offer over TUNA (even better if considered under comparable training budgets)?

How about finding out if sharing improve transfer relative to separated backbones under matched task exposure?

5 comments
AuthorsOpenReview ↗

Response to Reviewer d5me Part [1]

We thank the reviewer for the constructive feedback and for recognizing the overall understanding, generation, and editing performance and the effectiveness of HiGate. We have revised the manuscript to better clarify our contributions, strengthen the architectural comparisons, and provide a more balanced analysis of the trade-offs in unified modeling. Our revisions and additional experiments are detailed below.


[W1] Novelty and differentiation from related work

We agree that unified visual representations and residual detail injection have precedents. HiUni's contribution is not the individual introduction of these ideas, but their integration into a pretrained VLM while preserving its language-aligned ViT, patch merger, and dense Transformer backbone across understanding and generation. This choice allows the same pretrained semantic pathway to process clean images for understanding and noisy target images for generation, with pixel-level detail supplied only to the generation head.

We have expanded Section 2 to distinguish this design from relevant alternatives. TUNA constructs shared visual features by cascading a VAE and a representation encoder, TUNA-2 instead removes the visual encoders and uses pixel embeddings. Cheers employs a unified visual tokenizer and gated detail features after VAE, whereas HiUni injects the noisy target's raw pixel patches into the generation head. DiP and DeCo focus on pixel-level generation details rather than sharing the understanding and generation backbone. RepFusion also leverages a pretrained VLM, but uses a separate diffusion Transformer for representation-space generation, rather than predicting pixels through a shared backbone followed by a Pixel Diffusion Head.

The TUNA-2-style comparison reported in Q1 further examines the value of retaining pretrained visual features. We also observe 1.44× higher training throughput than Cheers in matched comparison, which provides an additional efficiency perspective on this design.


[W2] Performance degradation on MMBench and OCRBench

We have revised Section 4.2 to report benchmark-specific results rather than relying on an overall average. We re-evaluated Qwen3-VL-4B-Thinking under the same inference and evaluation settings as HiUni and added the results to Table 1. Relative to this reference, HiUni improves on four of eight benchmarks but trails on MMBench (80.7 vs. 86.2) and OCRBench (74.5 vs. 79.6).

To further investigate the effect of unified modeling, Table 4 includes the understanding-only baseline HiUni-Und-4B, since the original Qwen3-VL training corpus and recipe are unavailable. Under the controlled comparison, HiUni-Dense-4B achieves a similar average understanding score (65.04 vs. 64.58) and better scores on five of six benchmarks. In particular, MMBench is comparable (72.60 vs. 72.52), while OCRBench is slightly lower (71.30 vs. 73.00).

These results suggest that the performance gaps relative to the pretrained VLM cannot be explained by joint training alone. Differences in the training data distribution may also contribute to the observed changes.


[W3] Fairness of evaluation settings

Appendix B.1 now specifies the image-resolution and sequence-length limits, reasoning-mode answer extraction, continued decoding, and benchmark-specific evaluation protocols. We also re-evaluated Qwen3-VL-4B-Thinking with the same evaluation settings as HiUni to make the Table 1 reference more directly comparable.

We distinguish this matched re-evaluation from officially reported numbers for other models, whose inference budgets and implementations may differ. Those official results provide context, not strict controlled architectural comparisons. For architecture and task-mixture analysis, we rely primarily on the matched experimental configurations in Section 4.3 and Table 4, with the different-initialization Dense-4B versus MoT-2B comparison explicitly treated as an additional parameter-allocation reference.

AuthorsOpenReview ↗

Response to Reviewer d5me Part [2]

[W4] Benefits from multi-task modeling

We agree that understanding the interaction between different tasks is important for a shared architecture. Our revised Sections 4.3 and 4.5 separately examine the effect of generation supervision and the effect of adding editing supervision.

First, Table 4 shows that joint training retains comparable understanding performance relative to the understanding-only baseline (65.04 vs. 64.58), while falling below the generation-only baseline on OneIG (0.314 vs. 0.390). Thus, the shared model supports both tasks, although it introduces a trade-off in generation.

Second, we conduct an additional experiment using a VLM + DiT (Head) architecture to investigate whether updating the understanding backbone is helping generation. Under 50k training steps, updating the VLM improves OneIG ZH/EN from 0.337/0.304 to 0.367/0.339 and LongText ZH/EN from 0.087/0.022 to 0.242/0.148, compared with a frozen-VLM baseline that trains only the DiT. These results provide further evidence that jointly training VLM and generation module can benefit image generation.

Finally, we introduce editing data by changing task sampling ratios under a fixed total training-token budget, without changing the underlying understanding and generation data sources. Appendix E shows improvements on four of six understanding benchmarks, particularly AI2D (+6.70) and MMStar (+4.80), as the average increases from 62.64 to 64.49. However, HallusionBench decreases by 4.60 points, and OneIG declines slightly from 0.538 to 0.528.

Together, these results reveal task-dependent interference within the shared model. While we do not observe a direct generation benefit from understanding supervision, editing provides an example of positive transfer to understanding. We have made this distinction explicit in Section 4.5.


[Q1] What is the advantage of retaining the pretrained ViT and patch merger?

We compare HiUni with a TUNA-2-style encoder-free variant constructed for this experiment. Both variants initialize their language backbone from Qwen3-VL-4B and follow the same training data and configuration, consisting of 30K generation-head warmup steps followed by 30K joint pretraining steps. Following the encoder-free design of TUNA-2, the comparison variant directly processes pixel embeddings without the pretrained ViT and patch merger, while HiUni retains the pretrained visual pathway.

ModelQwen-Image-Bench ZHQwen-Image-Bench ENLongText ZHLongText ENOneIG ZHOneIG EN
HiUni (w/ ViT)17.5017.230.1090.0850.3380.289
TUNA-2-style (w/o ViT)11.6612.150.0180.0040.2780.250
ModelMMBenchRealWorldQAMMStarHallusionBenchAI2DOCRBenchAvg.
HiUni (w/ ViT)72.6065.4953.9348.9677.9571.3065.04
TUNA-2-style (w/o ViT)29.9552.4236.6021.7027.9163.0538.61

Retaining the pretrained visual pathway improves all six reported generation metrics and substantially preserves understanding performance under this limited training budget. The largest generation differences occur in text rendering. These results suggest that retaining the pretrained ViT and patch merger provides a stronger starting point for extending a pretrained VLM to image generation under the same training budget. We have also clarified the architectural difference from TUNA-2 in Section 2.

AuthorsOpenReview ↗

Response to Reviewer d5me Part [3]

[Q2] Does sharing improve transfer relative to separated backbones?

Before answering this question, we would like to clarify that our goal is not to establish the superiority of shared backbones over task-specific designs, but to investigate whether a shared dense backbone can effectively support both understanding and generation (Section 4.3).

Under matched 2B initialization, data, and training settings, HiUni-Dense-2B reaches a similar understanding average to HiUni-MoT-2B (58.55 vs. 59.23), while the MoT model performs better on OneIG (0.345 vs. 0.291). This result favors task-specific backbone computation for generation in the tested setting. However, under a comparable total parameter count, Dense-4B achieves stronger understanding (65.04 vs. 59.23) and competitive generation (0.314 vs. 0.345). This comparison illustrates a practical advantage of allocating more parameters with a shared backbone.

Besides, Appendix E shows that editing supervision can benefit some understanding tasks within the shared model, alongside modest generation interference, as discussed in W4. Since editing is usually introduced during the final stage, a fair transfer comparison with MoT would require training a corresponding MoT model through the preceding stages, we have not conducted this comparison and therefore do not claim that sharing provides stronger transfer than separated backbones. Therefore, understanding when shared and separated architectures differ in transfer remains an open question.


We thank the reviewer again for the valuable suggestions. We hope the revised manuscript and additional experiments better address the concerns and clarify the contributions and limitations of HiUni.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response and Summary of Revisions

We sincerely thank all reviewers for their constructive feedback. HiUni investigates whether a pretrained vision-language model (VLM) can be extended to image generation and editing while sharing its language-aligned ViT, patch merger, and dense Transformer backbone across tasks. Our goal is to establish the feasibility and trade-offs of this highly shared design, rather than claim that dense backbones are generally superior to task-routed MoT architectures. We appreciate the reviewers' recognition of the shared architecture (rDKF), competitive understanding, generation, and editing results and the effectiveness of HiGate (d5me), as well as the detailed training pipeline and promising generation performance at the 4B scale (Smg9).

We have revised the manuscript and conducted additional analyses to address the main concerns:

  1. Related work (d5me, Smg9). We have clarified the distinction from TUNA/TUNA-2, Cheers, DiP, DeCo, and RepFusion. HiUni reuses the pretrained VLM's visual pathway and backbone for both tasks, while adding generation-specific processing only near the output stage (Introduction, Section 2).
  2. Understanding evaluation (d5me, rDKF). We now report a Qwen3-VL-4B-Thinking re-evaluation under the same evaluation settings in Table 1, discuss benchmark-specific differences, including the declines on MMBench and OCRBench, and introduce an understanding-only baseline in Table 4.
  3. Architectural comparison (d5me, rDKF, Smg9). We distinguish the matched Dense-2B/MoT-2B comparison from the larger Dense-4B comparison, which uses a different pretrained initialization. Understanding-only and generation-only baselines reveal that a shared dense backbone maintains comparable understanding performance but incurs a generation trade-off (Section 4.3, Table 4).
  4. Cross-task interaction (d5me, Smg9). Under a fixed total training-token budget, introducing editing supervision through a change in task sampling ratios improves several understanding benchmarks while slightly reducing OneIG performance (Section 4.5, Appendix E).
  5. Typo correction. We have corrected the final model's generation-specific parameter count from 1.3B to 0.9B. The controlled ablations in Tables 4–5 use a separately specified 0.5B diffusion head. Neither change affects the reported experimental results.

We additionally provide a TUNA-2-style encoder-free comparison and broader generation ablation results in the reviewer-specific responses. We thank the reviewers again for their valuable feedback and hope these revisions better clarify the contributions, supporting evidence, and positioning of HiUni.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.