acceptodds
Under review as a conference paper at ICLR 2027

Vision as Unified Multimodal Generation

15 views

Abstract

We formulate computer vision as unified multimodal generation, where heterogeneous visual tasks are expressed through the native text and image generation spaces of a unified multimodal model (UMM), without task-specific architectures. With this formulation, the single model **VisionUMM** matches leading task-specialized systems across structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry. Natural-language instructions and optional visual prompts specify the task, target regions or views, and decoding convention. Responses are then generated as text for symbolic records, images for dense spatial targets, or mixed outputs for compositional tasks. To enable large-scale training, we convert heterogeneous computer vision annotations into instruction-response examples compatible with these native generation spaces. This conversion yields **VisionCorpus**, a computer-vision instruction-response corpus spanning text, image, and mixed text-and-image targets. Starting from an off-the-shelf pretrained UMM, VisionUMM is trained primarily on the VisionCorpus, using auxiliary multimodal data as a capability-preserving mixture and requiring no task-specific prediction heads or architectural changes. The resulting model covers detection, OCR, keypoints, segmentation, depth, surface normals, point maps, and camera pose estimation, and can follow language-defined variants that combine category, color, region, and other visual cues. These results suggest unified multimodal generation as a scalable route for integrating computer vision into general-purpose foundation models. The VisionUMM model and VisionCorpus are publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Unified Multimodal Models, Generalist Vision Models, Multimodal Generation, Visual Perception, Computer Vision

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_fwEyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether a unified multimodal model can perform computer vision tasks with vastly different output formats by leveraging its native text and image generation capabilities. The authors propose VisionUMM, a model based on Bagel-7B-MoT that represents structured targets—such as detection boxes, keypoints, and camera poses—as text, and represents depth, normals, segmentation masks, and 3D point maps as images. It expresses region descriptions and their spatial correspondences through mixed text-image outputs, eliminating the need for task-specific prediction heads.

Key contributions include a unified representation scheme covering four categories of vision tasks, a large-scale, instruction-based vision dataset named VisionCorpus, and a jointly trained model supporting these tasks. The paper reports quantitative results for structured understanding, dense geometry, segmentation, and multi-view geometry, while presenting qualitative examples of compositional instructions and novel task formats in the appendix. The results demonstrate that the proposed scheme achieves competitive performance across a wide range of tasks; its contributions lie primarily in system integration, data construction, and extensive empirical validation.

Critical strengths

A single model covers a diverse range of visual tasks and achieves competitive quantitative results. The paper addresses a broad spectrum of tasks—spanning detection, OCR localization, and keypoint estimation to segmentation, depth, surface normals, and multi-view geometry—without relying on task-specific prediction heads. For instance, the Dense200 model achieves an F1@mIoU of 66.8 (Table 1), outperforming listed baselines; segmentation inference scores reach 63.2 on the validation set and 60.7 on the test set (Table 3), surpassing existing methods; and ETH3D reconstruction F1 reaches 72.2 (Table 4), significantly outperforming G2VLM’s 36.7. These results demonstrate that a unified generative interface can support highly competitive visual prediction, rather than merely providing a standardized output format; this serves as the primary empirical evidence supporting the paper's acceptance.

The construction of large-scale data and the detailed documentation of protocols offer significant value to the community. VisionCorpus converts visual supervision from diverse sources into parsable instruction-response samples and addresses practical challenges such as pseudo-labeling, sparse depth completion, instance-color binding, and multi-view coordinate normalization. The paper introduces VisionCorpus-50M, with the appendix detailing data sources, filtering thresholds, templates, and training mixture ratios. This work promises to lower the barrier for researchers integrating traditional visual supervision into unified multimodal models, thereby establishing the paper's value in terms of data and systems, independent of its architectural innovations.

Critical weaknesses
  1. The methodological contributions are limited; the paper's primary value lies in data construction and task coverage. There is already established prior work on representing structured targets (such as bounding boxes and keypoints) via text generation, and representing depth, surface normals, and segmentation results via image generation. Unified-IO 2 [1] supports these diverse output types within a single model, and DICEPTION [2] has utilized pre-trained generative models to perform various visual perception tasks. Consequently, the core concept of mapping visual tasks to text and image generation spaces overlaps significantly with existing research. Regarding implementation, VisionUMM employs the off-the-shelf Bagel model and adopts standard training objectives—text cross-entropy and image rectified-flow. Its main contributions are the construction of the large-scale VisionCorpus, the design of output protocols for various tasks, and the expansion of task coverage. While these data and system-level contributions are valuable, the paper fails to adequately demonstrate what substantive technical advancements or new research insights the proposed representation and training designs offer compared to existing unified generative frameworks. This limits my assessment of the method's innovation.

  2. Validation of the model's compositional task capabilities relies primarily on qualitative examples, which is insufficient to determine whether these capabilities are stable or generalizable. Beyond reporting performance on standard visual tasks, the paper highlights the model's ability to combine visual capabilities based on language instructions—such as predicting depth only for a specified region, generating segmentation masks based on text-provided coordinates, or distinguishing instances by specific colors. However, the supporting evidence consists mainly of a few visualization examples in the appendix, lacking success rates or task metrics on a fixed test set, as well as direct comparisons with the original Bagel model. The failure cases shown in the appendix underscore the importance of this validation: the same compositional instruction can yield different output types depending on CFG settings; the model sometimes fails to accurately adhere to color mappings or correctly enumerate instances; and some results require manual modification of intermediate visualizations to achieve satisfactory outcomes. These phenomena indicate that achieving good results on standard tasks does not guarantee the model's ability to reliably execute novel compositional instructions. The authors acknowledge in Appendix C.4 that it is currently impossible to disentangle the extent to which these behaviors stem from joint training versus the pre-trained model itself. Therefore, while the existing evidence demonstrates that the model possesses a certain degree of compositional capability, it is insufficient to support a definitive assessment of its stability and generalization scope. Given that the ability to perform visual tasks through linguistic composition and expansion is a key value emphasized in this paper, this gap in validation materially impacts my evaluation.

[1] Jiasen Lu et al. Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action. CVPR, pp. 26439–26455, 2024.

[2] Canyu Zhao et al. DICEPTION: A Generalist Diffusion Model for Visual Perceptual Tasks. Advances in Neural Information Processing Systems, Vol. 38, 2025.

Questions for authors

Request for clarification: What is the core methodological innovation of this paper? Compared to existing works such as Unified-IO 2 and DICEPTION, what key new designs have been introduced regarding output representations or training methods? Please detail the specific differences of these designs and the benefits they yield, so that the paper's technical contributions—beyond data construction and task expansion—can be properly assessed.

Request for verification of current conclusions: How stable is the capability for compositional tasks, and what performance gains result from the training described in this paper? For tasks listed in the appendix—such as region-constrained depth estimation and text-based coordinate segmentation—could you report the success rates or task metrics (without manual post-processing) using a fixed set of test samples, prompts, and CFG settings, and compare them against the original Bagel model? This would help evaluate the reliability of the compositional capabilities and determine whether they have been enhanced by the training method proposed in this paper.

4 comments

We thank the reviewer for recognizing the task coverage and value of VisionCorpus.

[W1] Methodological contributions. We agree that the individual output representations and training objectives have precedents. Our contribution is to show how a pretrained unified multimodal model can be adapted to a broad set of vision tasks through the following three choices.

(i) Output representations. We represent sparse records as text, dense maps as images, and semantic–spatial correspondences as interleaved text–image responses. This lets one UMM produce benchmark-compatible outputs through its native pathways, including end-to-end joint generation of masks and semantic labels.

(ii) Unified supervision. VisionCorpus converts heterogeneous annotations into instruction–response examples, harmonizing coordinate systems, geometric conventions, instance–color correspondence, and multi-view normalization. This makes diverse supervision jointly usable within a common generative training framework.

(iii) Joint adaptation. We jointly fine-tune BAGEL's native pathways with next-token prediction and rectified flow, without task-specific heads or decoders. This yields one checkpoint covering four task families (Tables 1–4), without separate task models.

[W2] Compositional perception. We distinguish quantitative evidence for segmentation extensions from exploratory depth examples.

(i) Region-constrained depth. This task is not yet reliably supported. We lack fixed-set measurements of regional depth accuracy and gains over pretrained BAGEL, and do not claim stable performance.

(ii) Text-coordinate segmentation. We test transfer from visual mask prompts to normalized points or boxes in text; both VisionUMM and X-SAM were trained with visual mask prompts. Our custom COCO2017 val protocol aims to cover valid instances, querying each target separately. Scores are binary-mask mIoU (%); SA-23 averages its 23 datasets equally.

CueModelSA-23 pointCOCO2017 val pointCOCO2017 val box
Visual maskX-SAM41.4765.470.0
Visual maskVisionUMM37.0960.973.9
Text coordinatesX-SAM17.7220.0335.18
Text coordinatesVisionUMM28.3535.4037.06

VisionUMM scores above the X-SAM text-coordinate reference in all three settings, although a gap remains relative to its own visual mask scores.

(iii) Color assignments. We compare with pretrained BAGEL on all 2,938 GCG val images, partitioned into three disjoint prompt groups (Supplement C.3.2); region descriptions request at most four regions per image. Both models receive the same images and prompts.

PromptModelImagesColor-bound mIoUGCG mIoUColor accuracyMatch coverage
Color-name mappingBAGEL97915.7928.8248.9321.20
Color-name mappingVisionUMM97936.0850.4569.8353.89
Region descriptionsBAGEL79714.4027.4950.1018.98
Region descriptionsVisionUMM79737.9260.2971.8957.16
Hexadecimal colorsBAGEL1,16210.9324.5134.1816.70
Hexadecimal colorsVisionUMM1,16231.0443.8964.6246.38

Scores are percentages. For both models, RGB pixels are assigned to the nearest requested color, allowing RGB deviations; black is treated as background. Color-bound mIoU averages over all visible requested GT targets per image, then over images; misses score zero. GCG mIoU ignores caption and color identity, averaging greedily matched positive-IoU pairs per image, then over images. It omits unmatched targets, so we also report coverage. Color accuracy and coverage use one-to-one maximum-total-IoU matching, retaining pairs with IoU ≥\geq 0.5: accuracy is the fraction of matches with the requested decoded color; coverage is the fraction of visible GT targets recovered.

VisionUMM improves color-bound mIoU by 20.1–23.5 points and coverage by 29.7–38.2 points over BAGEL. These results provide quantitative evidence that the adapted checkpoint follows the tested region–color instructions more accurately than pretrained BAGEL.

AuthorsOpenReview ↗

Additional exploratory evaluations and qualitative evidence.

We conducted the tests in Supplementary Sections C.1–C.3 after training the checkpoint evaluated in this paper, without dedicated training protocols for these particular variants. They explore recombining input cues and output formats across our four task families: structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry.

To complement the quantitative results, we have prepared an anonymized supplementary gallery, viewable offline via index.html. It contains 100 visualizations for each color-assignment variant—category-to-color-name mappings, descriptive region–color assignments, and hexadecimal color specifications—together with all 200 Dense200 examples and 100 VGD box-prompt examples of reference-guided instance segmentation. Complete per-example JSONL manifests for all five groups are included, with original output filenames preserved.

Panels show the input image and prompt, available reference masks or detection boxes, and the predicted mask; query markers are retained where applicable. Detection boxes are not segmentation ground truth. Dense200 is shown in full; the other examples use reproducible random sampling without filtering by performance. Failed outputs are retained, and predictions were not manually corrected.

To fit the supplementary file-size limit, we include a subset of visualizations and omit separate raw prediction masks. The JPEG panels are also subject to lossy compression.

AuthorsOpenReview ↗

Comparison with Unified-IO 2 and DICEPTION.

We acknowledge the overlap in representing visual perception as generation. The differences concern how outputs are generated and how semantic labels are associated with dense predictions.

Output representations and training. Unified-IO 2 (Sec. 3.1) represents sparse targets with discrete tokens and generates dense targets as discrete image tokens. DICEPTION (Secs. 3.2–3.3) adapts a pretrained diffusion model to produce RGB perception outputs, including keypoint maps and color-coded instance masks. VisionUMM instead adapts a pretrained text-and-image UMM: sparse records use autoregressive text generation, dense maps use latent rectified-flow generation, and tasks requiring both use interleaved responses. We retain the pretrained pathways and objectives. This allows structured records, dense geometry, and semantic–spatial correspondences to coexist within one checkpoint, without task-specific prediction heads or decoders.

Specific case study: Joint generation of segmentation masks and semantic labels.

For instance segmentation, Unified-IO 2 (Appendix D.6) first localizes instances of a target class, then predicts a binary mask conditioned on the class and each detected box. DICEPTION (Secs. 3.2–3.3) produces category-agnostic entity masks and performs category-conditioned instance segmentation with colors separating instances.

VisionUMM jointly generates semantic category labels and masks for thing instances and stuff regions in a single end-to-end interleaved response. The generated legend assigns colors to the predicted regions, explicitly binding each region's semantic label to its mask. Semantic identities are therefore part of the prediction, alongside spatial regions, without separate per-box segmentation queries. For GCG, the same interface jointly generates region descriptions and corresponding masks. This provides an explicit semantic–spatial output interface that supports both panoptic segmentation and open-ended grounded descriptions.

Reviewer_fwEyOpenReview ↗

Response

Thank you for the response. I have no other concerns.

Reviewer_9p5GOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper identifies that an off-the-shelf UMM (unified multimodal model) can be finetuned to be a vision-generalist: it demonstrates that finetuning this model on a large curated dataset can achieve strong performance on several computer vision tasks involving both structured output and dense predictions. The evaluation spans several task families, including “structured visual understanding”, “dense geometric prediction”, “segmentation” and “multi-view geometry”.

Critical strengths
  • This work demonstrates that a UMM can be finetuned into a vision generalist on several task families, and compete with other specialists and generalists. As the paper states, several earlier methods either lack task coverage, or have to use custom decoders for every format. The method is simple but effective; the model performs well on several tasks (the results across the board are very respectable for a generalist).
  • The paper also contributes a large dataset (VisionCorpus), with 50-million examples, which helps reproducibility.
Critical weaknesses
  • L302 mentions that the training preserves the general multimodal capabilities of Bagel (Fig.1 does as well). However, this doesn’t seem to be explored anywhere in the paper (e.g., for VQA).
Questions for authors
  1. How does VisionUMM compare with Bagel on tasks like general Q&A and VQA? Was there any degradation as a result of training?

Minor points

  • I think the abstract and conclusion slightly oversell the abilities of this model (”matches leading task-specialized systems”), when it clearly lags behind in Table 4. I would slightly reword both.
  • MODUS [1] is a concurrent paper that seems to use a similar methodology. It might be useful to position against this.

[1] MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities

1 comment

We thank the reviewer for the positive assessment and for highlighting the potential of a single UMM for diverse visual perception tasks. We address the concerns regarding general capability retention, performance claims, and concurrent work below.

[W1] Preservation of general multimodal capabilities.

We agree that capability retention requires empirical evaluation. VisionUMM is initialized from pretrained BAGEL-7B-MoT and fine-tuned on VisionCorpus with auxiliary multimodal data. We report both BAGEL's official scores and our local evaluations for image understanding and generation.

Image Understanding

ModelMMMUMathVista
BAGEL, reported55.373.1
BAGEL, evaluated51.672.4
VisionUMM49.067.6
Δ\Delta−2.6-2.6−4.8-4.8

Scores are percentages; Δ\Delta is relative to locally evaluated BAGEL and computed before rounding. We evaluate all 900 MMMU validation questions and all 1,000 MathVista testmini questions.

Our MMMU evaluation does not add CoT prompting, whereas BAGEL's reported evaluation uses CoT. Relative to locally evaluated BAGEL, VisionUMM scores 2.6 and 4.8 percentage points lower on MMMU and MathVista, respectively.

Image Generation

ModelGenEvalWISE
BAGEL, reported0.820.52
BAGEL, evaluated0.770.51
VisionUMM0.850.45
Δ\Delta+0.08+0.08−0.06-0.06

Δ\Delta is relative to locally evaluated BAGEL. With prompt rewriting, BAGEL reports 0.88 on GenEval.

Overall, VisionUMM largely preserves the base UMM's core multimodal abilities while expanding to a broad range of visual tasks.

We have added these evaluations at the end of Section 4.2 and clarified the role of auxiliary multimodal supervision in the abstract and Section 4.1.

[Q1] Performance claims relative to task-specialized models.

We have revised the abstract and conclusion to describe VisionUMM as achieving competitive results across the four evaluated perception families.

[Q2] Relationship to concurrent work MODUS.

We thank the reviewer for pointing out the concurrent work MODUS. We acknowledge the methodological overlap: both works build upon pretrained BAGEL and leverage autoregressive prediction and flow matching to unify discrete and continuous multimodal generation.

The two works, however, differ in their primary objectives and contributions. MODUS focuses on general any-to-any modality translation, chained generation, and cross-modal self-verification. In contrast, our work investigates how a unified multimodal model can serve as a general-purpose visual perception system. We have added the MODUS citation and this distinction in Section 2.

Reviewer_xYjNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents VisionUMM, which uses the text and image outputs of a unified multimodal model (UMM) for visual perception. It generates coordinates as text and masks and geometric maps as images. The authors build VisionCorpus and fine-tune Bagel-7B-MoT without adding separate prediction heads for each task. They evaluate localization, segmentation, depth, surface normals, multi-view reconstruction, and camera pose estimation.

Critical strengths
  1. One model performs well across many perception tasks. The comparisons include specialist models and recent generalist models. Combining these capabilities without separate prediction heads is useful, although specialists remain stronger on some tasks.

  2. VisionCorpus is a substantial contribution. The authors convert annotations from different sources into text, images, or both, and carefully document the sources and conversion rules. This provides a useful starting point for training models on several kinds of perception tasks.

Critical weaknesses
  1. The interface still needs task-specific design. VisionUMM removes task-specific prediction heads, but uses coordinate conventions, color palettes, pose tokens, and decoding rules for different tasks. The main advance is making these tasks work in one pretrained UMM; the conceptual change to how perception tasks are represented is more limited.

  2. The instance-segmentation extensions in Appendix C depend on correctly generating a structured legend, which lists instances and their colors. Figures S14 and S16 show incorrect regions or merged instances when the list is wrong or incomplete. This limits the reliability of these task extensions, although the examples do not establish how often such errors occur on the benchmark tasks.

Questions for authors

See weaknesses.

1 comment

We thank the reviewer for the constructive feedback and for recognizing the broad task coverage and value of VisionCorpus. We clarify two main concerns below.

[W1] Task-specific output conventions versus architectural unification.

We agree that different tasks require different output conventions, such as coordinate formats, instance-color mappings, and pose parameterizations. However, we distinguish task-specific output protocols from task-specific model architectures. These conventions specify how predictions are represented and interpreted, but do not introduce separate prediction heads, decoders, or architectural branches. All tasks are handled by the same adapted UMM through its native text and image generation pathways.

Our goal is to unify the prediction and optimization of heterogeneous visual targets within a shared generative model. Symbolic and dense spatial outputs use the UMM's native next-token prediction and image-generation objectives, respectively. Task-specific conventions make these heterogeneous predictions benchmark-readable within a shared model architecture.

[W2] Reliability of instance-color legends.

We agree that incorrect or incomplete legends can lead to missing regions or merged instances (Figs. S14 and S16). On COCO panoptic val (5,000 images), we compare predicted and reference instance counts for each thing class in each image. The smaller count gives the number of matches; extra and missing entries count as false positives and false negatives, respectively. Micro-F1 pools these counts across images and classes, capturing both kinds of error without letting an extra instance in one class offset a missing instance in another.

Legend micro-F1 measures the instance counts listed before mask decoding; mask micro-F1 measures the counts of labeled regions in the final panoptic output. Both measure category-wise instance enumeration. We use X-SAM as a mask baseline because it predicts labeled panoptic regions directly, without generating a color legend. We apply the same counting rule to both models on the same images.

OutputPrecisionRecallMicro-F1Panoptic PQSemantic mIoU
VisionUMM: generated legend68.6878.9273.44––
VisionUMM: final masks68.7178.8873.4548.864.0
X-SAM: final masks75.3785.2980.0254.866.6

All scores are percentages. Precision, recall, and micro-F1 cover thing classes only. We recompute PQ and mIoU over all classes using the saved panoptic and semantic predictions, respectively.

The legend micro-F1 of 73.44 summarizes category-wise instance enumeration. At the image level, 3,145 of 5,000 images (62.90%) have at least one missing or extra thing entry. After mask decoding, micro-F1 changes by just 0.01 points, from 73.44 to 73.45; per-class instance counts are unchanged in 99.26% of images. These results indicate that the generated legend largely determines instance enumeration in our final masks on this benchmark.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.