- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether a unified multimodal model can perform computer vision tasks with vastly different output formats by leveraging its native text and image generation capabilities. The authors propose VisionUMM, a model based on Bagel-7B-MoT that represents structured targets—such as detection boxes, keypoints, and camera poses—as text, and represents depth, normals, segmentation masks, and 3D point maps as images. It expresses region descriptions and their spatial correspondences through mixed text-image outputs, eliminating the need for task-specific prediction heads.
Key contributions include a unified representation scheme covering four categories of vision tasks, a large-scale, instruction-based vision dataset named VisionCorpus, and a jointly trained model supporting these tasks. The paper reports quantitative results for structured understanding, dense geometry, segmentation, and multi-view geometry, while presenting qualitative examples of compositional instructions and novel task formats in the appendix. The results demonstrate that the proposed scheme achieves competitive performance across a wide range of tasks; its contributions lie primarily in system integration, data construction, and extensive empirical validation.
Critical strengths
A single model covers a diverse range of visual tasks and achieves competitive quantitative results. The paper addresses a broad spectrum of tasks—spanning detection, OCR localization, and keypoint estimation to segmentation, depth, surface normals, and multi-view geometry—without relying on task-specific prediction heads. For instance, the Dense200 model achieves an F1@mIoU of 66.8 (Table 1), outperforming listed baselines; segmentation inference scores reach 63.2 on the validation set and 60.7 on the test set (Table 3), surpassing existing methods; and ETH3D reconstruction F1 reaches 72.2 (Table 4), significantly outperforming G2VLM’s 36.7. These results demonstrate that a unified generative interface can support highly competitive visual prediction, rather than merely providing a standardized output format; this serves as the primary empirical evidence supporting the paper's acceptance.
The construction of large-scale data and the detailed documentation of protocols offer significant value to the community. VisionCorpus converts visual supervision from diverse sources into parsable instruction-response samples and addresses practical challenges such as pseudo-labeling, sparse depth completion, instance-color binding, and multi-view coordinate normalization. The paper introduces VisionCorpus-50M, with the appendix detailing data sources, filtering thresholds, templates, and training mixture ratios. This work promises to lower the barrier for researchers integrating traditional visual supervision into unified multimodal models, thereby establishing the paper's value in terms of data and systems, independent of its architectural innovations.
Critical weaknesses
-
The methodological contributions are limited; the paper's primary value lies in data construction and task coverage. There is already established prior work on representing structured targets (such as bounding boxes and keypoints) via text generation, and representing depth, surface normals, and segmentation results via image generation. Unified-IO 2 [1] supports these diverse output types within a single model, and DICEPTION [2] has utilized pre-trained generative models to perform various visual perception tasks. Consequently, the core concept of mapping visual tasks to text and image generation spaces overlaps significantly with existing research. Regarding implementation, VisionUMM employs the off-the-shelf Bagel model and adopts standard training objectives—text cross-entropy and image rectified-flow. Its main contributions are the construction of the large-scale VisionCorpus, the design of output protocols for various tasks, and the expansion of task coverage. While these data and system-level contributions are valuable, the paper fails to adequately demonstrate what substantive technical advancements or new research insights the proposed representation and training designs offer compared to existing unified generative frameworks. This limits my assessment of the method's innovation.
-
Validation of the model's compositional task capabilities relies primarily on qualitative examples, which is insufficient to determine whether these capabilities are stable or generalizable. Beyond reporting performance on standard visual tasks, the paper highlights the model's ability to combine visual capabilities based on language instructions—such as predicting depth only for a specified region, generating segmentation masks based on text-provided coordinates, or distinguishing instances by specific colors. However, the supporting evidence consists mainly of a few visualization examples in the appendix, lacking success rates or task metrics on a fixed test set, as well as direct comparisons with the original Bagel model. The failure cases shown in the appendix underscore the importance of this validation: the same compositional instruction can yield different output types depending on CFG settings; the model sometimes fails to accurately adhere to color mappings or correctly enumerate instances; and some results require manual modification of intermediate visualizations to achieve satisfactory outcomes. These phenomena indicate that achieving good results on standard tasks does not guarantee the model's ability to reliably execute novel compositional instructions. The authors acknowledge in Appendix C.4 that it is currently impossible to disentangle the extent to which these behaviors stem from joint training versus the pre-trained model itself. Therefore, while the existing evidence demonstrates that the model possesses a certain degree of compositional capability, it is insufficient to support a definitive assessment of its stability and generalization scope. Given that the ability to perform visual tasks through linguistic composition and expansion is a key value emphasized in this paper, this gap in validation materially impacts my evaluation.
[1] Jiasen Lu et al. Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action. CVPR, pp. 26439–26455, 2024.
[2] Canyu Zhao et al. DICEPTION: A Generalist Diffusion Model for Visual Perceptual Tasks. Advances in Neural Information Processing Systems, Vol. 38, 2025.
Questions for authors
Request for clarification: What is the core methodological innovation of this paper? Compared to existing works such as Unified-IO 2 and DICEPTION, what key new designs have been introduced regarding output representations or training methods? Please detail the specific differences of these designs and the benefits they yield, so that the paper's technical contributions—beyond data construction and task expansion—can be properly assessed.
Request for verification of current conclusions: How stable is the capability for compositional tasks, and what performance gains result from the training described in this paper? For tasks listed in the appendix—such as region-constrained depth estimation and text-based coordinate segmentation—could you report the success rates or task metrics (without manual post-processing) using a fixed set of test samples, prompts, and CFG settings, and compare them against the original Bagel model? This would help evaluate the reliability of the compositional capabilities and determine whether they have been enhanced by the training method proposed in this paper.
4 comments
Additional exploratory evaluations and qualitative evidence.
We conducted the tests in Supplementary Sections C.1–C.3 after training the checkpoint evaluated in this paper, without dedicated training protocols for these particular variants. They explore recombining input cues and output formats across our four task families: structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry.
To complement the quantitative results, we have prepared an anonymized supplementary gallery, viewable offline via index.html. It contains 100 visualizations for each color-assignment variant—category-to-color-name mappings, descriptive region–color assignments, and hexadecimal color specifications—together with all 200 Dense200 examples and 100 VGD box-prompt examples of reference-guided instance segmentation. Complete per-example JSONL manifests for all five groups are included, with original output filenames preserved.
Panels show the input image and prompt, available reference masks or detection boxes, and the predicted mask; query markers are retained where applicable. Detection boxes are not segmentation ground truth. Dense200 is shown in full; the other examples use reproducible random sampling without filtering by performance. Failed outputs are retained, and predictions were not manually corrected.
To fit the supplementary file-size limit, we include a subset of visualizations and omit separate raw prediction masks. The JPEG panels are also subject to lossy compression.
Comparison with Unified-IO 2 and DICEPTION.
We acknowledge the overlap in representing visual perception as generation. The differences concern how outputs are generated and how semantic labels are associated with dense predictions.
Output representations and training. Unified-IO 2 (Sec. 3.1) represents sparse targets with discrete tokens and generates dense targets as discrete image tokens. DICEPTION (Secs. 3.2–3.3) adapts a pretrained diffusion model to produce RGB perception outputs, including keypoint maps and color-coded instance masks. VisionUMM instead adapts a pretrained text-and-image UMM: sparse records use autoregressive text generation, dense maps use latent rectified-flow generation, and tasks requiring both use interleaved responses. We retain the pretrained pathways and objectives. This allows structured records, dense geometry, and semantic–spatial correspondences to coexist within one checkpoint, without task-specific prediction heads or decoders.
Specific case study: Joint generation of segmentation masks and semantic labels.
For instance segmentation, Unified-IO 2 (Appendix D.6) first localizes instances of a target class, then predicts a binary mask conditioned on the class and each detected box. DICEPTION (Secs. 3.2–3.3) produces category-agnostic entity masks and performs category-conditioned instance segmentation with colors separating instances.
VisionUMM jointly generates semantic category labels and masks for thing instances and stuff regions in a single end-to-end interleaved response. The generated legend assigns colors to the predicted regions, explicitly binding each region's semantic label to its mask. Semantic identities are therefore part of the prediction, alongside spatial regions, without separate per-box segmentation queries. For GCG, the same interface jointly generates region descriptions and corresponding masks. This provides an explicit semantic–spatial output interface that supports both panoptic segmentation and open-ended grounded descriptions.
Response
Thank you for the response. I have no other concerns.
We thank the reviewer for recognizing the task coverage and value of VisionCorpus.
[W1] Methodological contributions. We agree that the individual output representations and training objectives have precedents. Our contribution is to show how a pretrained unified multimodal model can be adapted to a broad set of vision tasks through the following three choices.
(i) Output representations. We represent sparse records as text, dense maps as images, and semantic–spatial correspondences as interleaved text–image responses. This lets one UMM produce benchmark-compatible outputs through its native pathways, including end-to-end joint generation of masks and semantic labels.
(ii) Unified supervision. VisionCorpus converts heterogeneous annotations into instruction–response examples, harmonizing coordinate systems, geometric conventions, instance–color correspondence, and multi-view normalization. This makes diverse supervision jointly usable within a common generative training framework.
(iii) Joint adaptation. We jointly fine-tune BAGEL's native pathways with next-token prediction and rectified flow, without task-specific heads or decoders. This yields one checkpoint covering four task families (Tables 1–4), without separate task models.
[W2] Compositional perception. We distinguish quantitative evidence for segmentation extensions from exploratory depth examples.
(i) Region-constrained depth. This task is not yet reliably supported. We lack fixed-set measurements of regional depth accuracy and gains over pretrained BAGEL, and do not claim stable performance.
(ii) Text-coordinate segmentation. We test transfer from visual mask prompts to normalized points or boxes in text; both VisionUMM and X-SAM were trained with visual mask prompts. Our custom COCO2017 val protocol aims to cover valid instances, querying each target separately. Scores are binary-mask mIoU (%); SA-23 averages its 23 datasets equally.
VisionUMM scores above the X-SAM text-coordinate reference in all three settings, although a gap remains relative to its own visual mask scores.
(iii) Color assignments. We compare with pretrained BAGEL on all 2,938 GCG val images, partitioned into three disjoint prompt groups (Supplement C.3.2); region descriptions request at most four regions per image. Both models receive the same images and prompts.
Scores are percentages. For both models, RGB pixels are assigned to the nearest requested color, allowing RGB deviations; black is treated as background. Color-bound mIoU averages over all visible requested GT targets per image, then over images; misses score zero. GCG mIoU ignores caption and color identity, averaging greedily matched positive-IoU pairs per image, then over images. It omits unmatched targets, so we also report coverage. Color accuracy and coverage use one-to-one maximum-total-IoU matching, retaining pairs with IoU ≥ 0.5: accuracy is the fraction of matches with the requested decoded color; coverage is the fraction of visible GT targets recovered.
VisionUMM improves color-bound mIoU by 20.1–23.5 points and coverage by 29.7–38.2 points over BAGEL. These results provide quantitative evidence that the adapted checkpoint follows the tested region–color instructions more accurately than pretrained BAGEL.