acceptodds
Under review as a conference paper at ICLR 2027

UniSpace: Unified Visual Representation and Scalable Multimodal Modeling

Abstract

Semantic vision encoders provide a powerful interface for multimodal understanding, but their final tokens lose fine-grained visual details needed for reconstruction and image editing. We ask whether understanding, generation, and editing can share a visual representation built from a pretrained semantic ViT. A controlled probe shows that replacing only the patch embedding improves recoverability while keeping all Transformer blocks frozen. Motivated by this observation, we introduce Patch Reparameterization: the original semantic pathway is preserved, and a reconstruction-aware patch embedding supplies visual details to the same frozen backbone. We compress and concatenate the two token streams and balance their contributions during flow-matching training. The resulting tokenizers retain aggregate multimodal understanding while enabling high-fidelity reconstruction and image generation. PR-DINOv2 achieves an ImageNet reconstruction FID of 0.14 and a generation FID of 1.87 with classifier-free guidance. We further scale this representation into UniSpace, an 8B Mixture-of-Transformer-Experts model that supports understanding, generation, and editing without a separate VAE pathway. It scores 4.28 on ImgEdit and 0.84 on GenEval, demonstrating a shared ViT-based visual interface at system scale.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
unified visual representation, visual tokenizer, patch reparameterization, multimodal generation, image editing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_877EOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper proposes Patch Reparameterization, adding a trainable reconstruction-aware patch embedding to a frozen semantic ViT.
  • Experiments show UniSpace improves reconstruction, tokenizer-level understanding, and generation and editing performance.
Critical strengths
  • Patch Reparameterization offers a novel and conceptually simple approach to improving reconstruction fidelity while preserving aggregate multimodal understanding performance, without modifying the pretrained Transformer blocks.
  • Balanced flow matching introduces an interesting component-wise formulation that explicitly balances semantic and reconstruction objectives.
  • The evaluation is reasonably comprehensive, with targeted ablations supporting the design choices, while the method’s simplicity facilitates reproduction of its core components.
Critical weaknesses
  • The reconstruction-aware branch requires an additional forward pass through the frozen ViT, despite sharing its parameters. The paper does not quantify the resulting FLOPs, latency, or memory overhead, making it difficult to assess the reconstruction gains relative to their computational cost.
  • Although VTP-L is included, comparisons are not controlled for the generative prior or training budget. More systematic comparisons with VTP, OpenVision-3, and the FLUX.2 VAE under matched settings would help establish whether the proposed representation offers a compelling reconstruction–generation trade-off.
  • Understanding preservation is evaluated only under the frozen-encoder LLaVA-v1.5 setting. This does not establish the method’s value when the vision encoder is jointly fine-tuned with a large-scale MLLM backbone, where strict freezing may constrain performance.
Questions for authors
  • What additional encoding FLOPs, latency, and memory overhead does the reconstruction branch introduce?
  • Under matched generative priors and training budgets, does the method offer a better reconstruction–generation trade-off than VTP, OpenVision-3, and FLUX.2 VAE?
  • Is freezing the ViT essential to the method, and do its benefits persist when the vision encoder is jointly fine-tuned with the MLLM backbone?
Reviewer_mvYgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper is about unified models with a distinct view on starting from a pre trained semantic ViT. THe idea is very simple: learn a new patchifier for the frozen DiT backbone, and concatenate compressed reconstruction features with the semantic tokens with a weighted flow matching loss. This tokenizer is then used in a Qwen model. The idea is simple but well executed with very impressive gains on reconstructions, and more importantly some gains across editing and generation with stable understanding.

Critical strengths

The paper proposes a simple but clear idea, which is executed well and delivers measurable reconstruction, editing and generation gains. Although the generation gains are less clear, editing gains are somewhat strong.

Although many experiments are small reconstruction-based, the authors also explore more realistic generation and editing benchmarks. Replacement and removal are particularly strong. I appreciate the clarity on the unmatched testing settings, which are not ideal but at least the reader knows there is a confounding factor.

Critical weaknesses

The claim on patch embedding bottleneck is probably a bit overstated. Although the random projection experiment is informative and interesting, it probably only speaks about how input parametrization might affect recoverability on a frozen backbone. This is experiment cannot conclude that the information is lost as the patch embedding itsef.

The understanding results do not include how the mdel might perform after large scale joint training which is one of the advertised capabilities of the model. The reported results are on a separated Vicuna/LLava experiment which is a weaker addition to the paper. A simple ablation before/after editing training would be ideal.

Efficiency is worth discussing in this paper, since a double forward pass of the ViT and large decoder are necessary, this is an important discussion point that gives color to the paper.

Attribution of gains is difficult along the paper due to internal data and budget mismatches. Smaller control experiments would be very informative as proper ablations would make the paper stronger.

Questions for authors

As per the weaknesses.

Reviewer_FnGHOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes UniSpace, a framework for multimodal understanding, generation, and editing using a shared ViT-based visual representation. UniSpace combines semantic and reconstruction-oriented features from a shared frozen ViT and integrates this representation into a Qwen3-8B-based MoT model for understanding, generation, and editing. Results are promising, but missing comparisons with relevant unified tokenizers and differences in backbone and training settings limit attribution of the gains to the proposed representation.

Critical strengths
  1. A simple and effective way to recover detail while retaining a pretrained semantic pathway.
  2. Meaningful evidence beyond reconstruction alone.
Critical weaknesses
  1. The paper should compare with HYDRA-X [1] on the editing benchmarks evaluated by both methods and include the relevant HYDRA [2] tokenizer variants in the reconstruction comparisons. The relationship to HYDRA’s observations about substantial differences in reconstructability across ViT layers also needs clarification. In particular, the authors should explain what the patch-embedding intervention establishes beyond this prior layer-wise analysis. Without these comparisons and discussion, the novelty and empirical advantage of the proposed representation remain difficult to assess.

  2. Insufficient full-system understanding evaluation and controlled comparisons. The understanding results evaluate tokenizers within a separate LLaVA setup rather than the final UniSpace model. Full-system MMU evaluation, particularly on challenging tasks such as OCRBench and DocVQA, is needed to substantiate the claimed understanding–generation–editing balance. Furthermore, UniSpace uses Qwen3-8B, whereas the compared Show-o2 [3] and TUNA [4] configurations use Qwen2.5-7B. Differences in backbone, training data, and compute confound attribution of performance gains to the tokenizer. The paper also lacks a controlled comparison demonstrating the benefit of a unified visual interface over separate ViT–VAE spaces within the same MoT architecture.

Questions for authors
  1. Can the authors compare with HYDRA-X on shared editing benchmarks, including ImgEdit and GEdit where applicable, and with the relevant HYDRA tokenizer variants on reconstruction metrics under comparable resolution and evaluation settings?
  2. How does the proposed bottleneck diagnosis relate to HYDRA’s layer-wise reconstruction findings? What additional mechanistic conclusion is established by changing the patch embedding, beyond showing that input parameterization affects detail preservation through the backbone?
  3. How does the final UniSpace checkpoint perform on MMU benchmarks, particularly OCRBench, DocVQA, and other tasks requiring fine-grained visual understanding? Does the tokenizer-level preservation of understanding survive joint system training?
  4. Can the authors isolate the tokenizer’s contribution using the same language backbone, data, and training budget? In particular, how much of the difference from Show-o2 and TUNA is attributable to Qwen3-8B rather than the proposed visual representation?
  5. Within the same MoT architecture, what measurable advantage does the unified tokenizer provide over a dual ViT–VAE interface, given that understanding and generation still use separate experts?

References [1] HYDRA-X: Native Unified Multimodal Models with Holistic Visual Tokenizers. NeurIPS 2026. [2] HYDRA: Unifying Multi-modal Generation and Understanding via Representation-Harmonized Tokenization. NeurIPS 2026. [3] Show-o2: Improved Native Unified Multimodal Models. NeurIPS 2025. [4] TUNA: Taming Unified Visual Representations for Native Unified Multimodal Models. CVPR 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.