acceptodds
Under review as a conference paper at ICLR 2027

Zero-Shot Perception with Generator

18 views

Abstract

Pretrained on web-scale visual data, generation models acquire rich visual representations and expressive world priors that provide a natural foundation for perception. To unlock these capabilities, we explore perception as generation, leveraging generation paradigm to convert these internal representations directly into visual predictions. Although recent models such as Vision Banana demonstrate strong performance and impressive zero-shot generalization across diverse perception tasks, reliance on proprietary backbones hinders independent investigation and open reproduction. In this work, we present UniVision, an open 8B vision foundation model that unifies zero-shot visual perception and image generation. UniVision generates metric depth, surface normals, and semantic, instance, and referring segmentation as RGB images, which are converted back to target modalities via our noise-tolerant pixel codecs. UniVision achieves advanced zero-shot performance across all five tasks while capturing finer geometric detail than specialized baselines. We show that the core recipe behind these capabilities lies in deterministic, noise-tolerant codecs and scalable data pipelines. Furthermore, based on UniVision, we systematically investigate the perception-as-generation paradigm across both inference and training sides, revealing how inference-time generation techniques and test-time aggregation boost perception accuracy, characterizing task-specific data scaling dynamics, and demonstrating positive data synergy where joint perception training actively enhances generation quality.

Primary area:
generative models
Keywords:
generation model, world model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Sk4xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UniVision fine-tunes the generation expert of a pretrained vision-language/image-generation model to produce RGB encodings of depth, normals and segmentation. Task-specific decoders convert the generated images into predictions. The paper builds a large mixed training collection, studies resolution, guidance and sample aggregation, and tests data scaling and the effect of perception training on text-to-image generation. Some segmentation results additionally use external multimodal models for presence detection or query rewriting and mask selection.

Critical strengths

The empirical scope is substantial, and the paper provides detailed codecs, prompts, data sources and inference protocols. It explicitly acknowledges the perception-as-generation precedent, including Vision Banana [1], DICEPTION and earlier work. The fixed-output codec ablation is useful: it shows a real improvement from the proposed decoding refinements. The data-scaling study controls training-token budgets, and the text-to-image synergy experiment includes controls for both iteration count and text-to-image exposure.

Critical weaknesses

1. Two central task results depend heavily on external proprietary perception systems. This is disclosed in the table and appendix, but it changes what the results establish about UniVision. Table A.2 reports instance cgF1 of 2.30 without Gemini gating and 45.14 with it; image-level presence MCC changes from 0.0442 to 0.8504. Positive-query segmentation remains useful, but the generator largely fails to reject absent targets. The appendix attributes this to training without negative queries. This is a substantial gap in open-vocabulary instance segmentation, not a small optional enhancement.

Similarly, referring cIoU rises from 63.40 to 75.39 with the agent in Table A.3. Appendix C.5 specifies GPT-5.6 Sol with xhigh reasoning, up to two rounds of four 50-step generations, and multimodal selection/rewrite calls. The controller interprets the source image and query, so it supplies perception and grounding capabilities of its own. The resulting system can be useful, but its score does not isolate what the open generator learned. Please put the standalone rows in the main table, train a negative-query control, and separate candidate sampling, selection and rewriting under matched budgets. A comparison using the same controller/candidate budget for an alternative mask generator is also needed. The current all-or-nothing agent ablation cannot attribute the improvement among these steps.

2. The incremental scientific contribution and its computational cost need a sharper evaluation. The overall RGB-output formulation and the depth codec follow existing work, which the authors appropriately cite. The potentially useful additions are the data recipe, decoding refinements and empirical findings. Table A.1 supports the refinements, but does not show whether the more complex decoder is robust to different output resolutions, object sizes and thresholds. The palette, component-size and merge rules introduce several choices; their sensitivity and the validation split used to select them should be reported.

The claimed inference improvements also increase cost. The default uses 50 denoising steps, high-resolution inputs for some tasks, and potentially multiple samples; the referring agent adds external reasoning calls. There is no corresponding latency, memory or cost comparison. Accuracy-versus-compute curves would make these improvements much easier to assess. The architecture also contains a frozen 8B vision-language backbone in addition to the 8B generation expert, so reporting only the latter's parameter count does not describe the full inference system.

Finally, the 1.91-point text-to-image gain over the matched-exposure control is interesting, but is measured with one generation per prompt and one automated judging protocol. Prompt-level confidence intervals, repeat seeds and an independent check would help establish its robustness. Metric-depth performance is also uneven: the average delta1 is 0.842 versus Vision Banana's 0.933, with ETH3D AbsRel 0.664. Figure 3 uses scale-and-shift-aligned depth, as its caption states, so it illustrates structure rather than validating metric calibration. The paper should more clearly distinguish these limitations from its strong relative-depth results.

I lean toward rejection in the current form. The study is useful, but the standalone capabilities, incremental findings and full-system cost need to be established more clearly.

Questions for authors
  1. Can negative-query training remove the dependence on Gemini for instance presence detection? Please include standalone results in the main table.
  2. What are the separate contributions of additional samples, mask selection and query rewriting? How does another mask generator perform with the same controller and budget?
  3. On what held-out data were codec thresholds, resolution, shift and guidance selected? How sensitive are the decoding gains to these choices?
  4. Please report full-system parameters, latency and peak memory, including external calls and sample count, and uncertainty for the claimed text-to-image synergy.

Presentation

The appendix is detailed, but the main text should make the standalone-versus-assisted results more prominent. Table 1 compresses six tasks into small panels, and Figs. 4–5 use small axes and labels. Enlarging these panels and reporting the inference budget beside each result would improve readability. Claims about metric depth should be supported with an unaligned metric visualization alongside the current aligned example.

Reference

[1] Image Generators are Generalist Vision Learners (Vision Banana). https://arxiv.org/html/2604.20329v3

2 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer Sk4x (1/2)

Thank you for the detailed review, and for recognizing the codec ablation on fixed outputs, the token-controlled scaling study and the matched controls in the synergy study. We answer your points in order, with new runs on the reported checkpoint. We would also be grateful if you could read our general response, which shows that the same recipe carries over to video and that UniVision learns a new perception task with a small LoRA.

1. Dependence on external models (Weakness 1, Questions 1 and 2). We agree that the standalone and assisted results must be separated, and the revised Table 1 will show them side by side. The external models never draw a mask: they judge presence (instance) or select and rephrase (referring), and every mask comes from UniVision. Our controls show that the segmentation quality is UniVision's own and that open models can replace the proprietary ones.

Instance (Question 1). The gate leaves mask quality unchanged (pmF1 on present phrases 53.35 without it, 52.82 with it); what it adds is the rejection of absent phrases, which our instance data never asked for because it contains no absent-target queries. The dependence on Gemini is removed by an open gate, and the same gate does not bring other generators to our level (full SA-Co/Gold cgF1, box detectors at their best threshold):

mask generatorGemini gateour open gate
UniVision45.1445.29
Grounding DINO-B + SAM 2.142.4442.82
OWLv2-L + SAM 2.131.0030.89

Our open gate is a Qwen3-VL-30B-A3B-based model fine-tuned for presence on COCO, LVIS, PACO and OpenImages images. Adding absent-target queries to the training data is the direct way to fold this decision into the generator.

Referring (Question 2). We agree that sampling, selection and rewriting should be separated under matched budgets. The agent logs separate them (RefCOCOg val cIoU):

settinggenerationscIoU
UniVision alone163.40
GPT-5.6 selects 1 of 4 samples470.29
best of 4 samples (oracle)472.98
full agent (+ rewrite for 7% of queries)4.375.39
one rewrite by open-weight Kimi K3, one sample175.95

Selection adds 7 points and is bounded by the oracle; rewriting the query adds the rest, and one open-model rewrite with a single sample matches the full agent with a quarter of the generations. With the same rewrite and budget, SAM 3 reaches 67.99 (55.46 with the original expression): the rewrite lifts both generators by the same 12.5 points, and UniVision stays 8 points ahead. The agent's gain therefore comes from clarifying the query, which helps any generator equally, while UniVision's lead comes from its masks. We will add this breakdown and the SAM 3 control to the paper.

2. Robustness of the decoding refinements (Weakness 2, Question 3). We agree that the decoding refinements, one of the contributions you single out, must show that their gain does not hinge on their own settings. We re-decoded saved outputs of the reported checkpoint, without regenerating them, once with naive nearest-color decoding and once with our decoder, and varied each factor you name:

Factor varied (same generated outputs)MetricNaive decodingOur decoder
Output resolution and guidance
Semantic, 1536 / 1728 / 1984 pxCityscapes mIoU64.6 / 64.2 / 62.567.7 / 67.2 / 66.0
Instance, 1.0 / 1.5 / 2.0 MP at guidance 4SA-Co/Gold cgF115.7 / 15.8 / 15.943.6 / 43.8 / 44.0
Instance, 2.0 MP at guidance 2 (reported)SA-Co/Gold cgF117.245.1
Object size: smallest instance < 0.05 / 0.05 to 0.2 / 0.2 to 1 / 1 to 5 / ≥ 5% of the image
Instance, queries whose phrase is presentSA-Co/Gold pmF116.6 / 20.0 / 21.5 / 21.5 / 19.032.4 / 42.5 / 52.3 / 62.8 / 61.7
Palette, component-size and merge rules, one threshold changed at a time
Semantic: seed radius 10 to 40, component floor ×0.5 to ×4, seam fill 0 to 3 pxCityscapes mIoU64.666.5 to 67.8
Instance: activation distance 20 to 40, margin 4 to 12, area floors ×0.5 to ×2SA-Co/Gold cgF117.244.7 to 45.4
Instance: merge rule (absorb and erode) removedSA-Co/Gold cgF117.243.2

Our decoder adds about 3 mIoU and 28 cgF1 at every output resolution and guidance scale, and it helps objects of every size, from +15.8 pmF1 for the smallest to +42.7 for the largest. Its thresholds sit in a flat region: changing any palette, component-size or merge threshold moves the result by at most 1.1 mIoU or 0.4 cgF1, several changes even score slightly higher than the reported values, and removing the merge rule altogether still leaves 26 of the 28 cgF1 points. The decoding gain therefore comes from the design of the decoder rather than from a particular threshold or inference setting. We will add these results to Appendix A.1.

AuthorsOpenReview ↗

Rebuttal for Reviewer Sk4x (2/2)

3. Full-system size, latency and memory (Weakness 2, Question 4). We agree that the cost of the full system should be reported, including samples and external calls. The full system has about 15.9B parameters: the frozen 8.2B language backbone, the 7.0B generation expert with its adapters (the only trained part), and the VAE. At the paper's settings (one GB200 GPU, batch 1, bf16, 50 steps, VAE decoding included), one prediction takes 7.0 s for depth, 3.6 s for normals, 7.4 s for semantic and 8.4 s for instance segmentation, with 31 to 36 GiB peak memory. The referring agent runs 4 generations per expression, 4 more for the 7% of expressions it rewrites (345 of 4,896), and 1.14 GPT-5.6 calls per expression on average.

The 50-step default is conservative. A step sweep on the same checkpoint gives the accuracy versus time trade-off (time per image in brackets):

task50 stepsfewer steps
metric depth, mean δ184.18 (7.0 s)86.67 at 4 steps (0.75 s)
normals, mean error18.79° (3.6 s)18.16° at 1 step (0.24 s)
instance, cgF145.14 (8.4 s)45.01 at 15 steps (3.3 s)
semantic, mIoU67.65 (7.4 s)67.33 at 25 steps (4.0 s); 55.17 at 10 steps

For depth and normals, one to four steps are both faster and more accurate (9× and 15× less time), while segmentation stays within 0.4 points of its 50-step accuracy at 15 to 25 steps, in about half the time. UniVision's accuracy therefore does not depend on an expensive inference budget. We will report the full-system parameter count, put the inference budget (steps, samples, external calls) beside each result in Table 1, add these curves, and add the cost of test-time aggregation (N generations) to Figure 5.

4. Uncertainty of the text-to-image synergy (Weakness 2, Question 4). We agree that one generation per prompt and one judge call for prompt-level uncertainty and an independent check. We computed paired bootstrap intervals over the 1,170 prompts (50,000 resamples) and re-scored the same saved images with a second judge, GPT-5.4, under the unchanged UniGenBench protocol (gain of w/ vision in points, 95% CI):

ComparisonGemini 3 Flash (paper's judge)GPT-5.4 (second judge)
vs T2I control matched in T2I updates, all prompts+1.90 [0.89, 2.94]+1.76 [1.05, 2.47]
same control, Public prompts+3.32 [1.94, 4.71]+2.19 [1.17, 3.23]
same control, Physical AI prompts+0.61 [−0.90, 2.13]+1.36 [0.38, 2.33]

Against the T2I control matched in iterations, the paper's judge gives an even larger gain, +3.76 [2.67, 4.85]. The two judges agree on 85.9% of the 8,193 test points, and the overall gain is significant under both. We will report the overall gain with these intervals in Section 4.5 instead of per-category rankings, and state that the joint run samples text-to-image, vision and editing data in equal thirds.

5. Metric depth (Weakness 2). We agree that relative and metric depth should be presented separately, and the revision does so. UniVision's relative depth is the best among the compared methods on KITTI and ETH3D (δ1 98.0 and 98.9), and its metric gap is a per-image scale error rather than a structural one: dividing out one scalar per image (the median prediction to ground truth ratio, no shift) brings every benchmark to 97.3 to 97.7 δ1.

DatasetMetric δ1 as isδ1 with one scale per imageImages with scale off by more than 25%
NYUv293.597.63.5%
KITTI92.697.74.1%
ETH3D70.697.528.4%
iBims-180.097.317.0%

The ETH3D AbsRel of 0.664 comes mostly from seven images of the "terrains" scene that are predicted 18 to 54 times too far: they carry 68.5% of the total, and AbsRel is 0.213 without them. Inference already narrows the gap: 4 sampling steps give 86.67 instead of 84.18 mean δ1 (ETH3D 75.5 instead of 70.6), and test-time aggregation corrects part of the scale. The metric gap is therefore a calibration error that inference already reduces, while the predicted geometry is accurate. We will separate the relative and metric depth claims in Section 4.2 and add this analysis.

Reviewer_3rrDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work presents UniVision, an open-source variant of generative multi-task perception models. UniVision finetunes a generative model to solve many perception tasks including depth, surface normal, and segmentation. UniVision is finetuned using large scales of data from SA-1B and OpenImages and gets good zero-shot results on many datasets. Authors argue the core insight leading to their success is several "codecs" they introduce to represent the task in an image-to-image manner.

Critical strengths

The paper is very experimentally strong for all settings except instance segmentation. In nearly all cases, the authors report SOTA or close-to-SOTA results.

If the pseudo-labeled data were to be released, this could have a solid impact as it would save future researchers the time and compute of pseudo-labeling their own datasets across several tasks.

Critical weaknesses

The claimed insight of this work appears to be the introduction of a set of codecs. In my opinion, it is not clear what difference these codecs have from those used in Vision Banana. If there is a difference, this should be more clearly stated and the advantage should be shown experimentally. At this stage, it is not clear what difference (other than an open reproduction) there is relative to Vision Banana. If there is, perhaps the paper should focus more on insights gained from this process.

The experimental setup across instance and referring segmentation is very weak. It appears the segmentation metrics are based on SAM 3 and the metrics are not well known across the community. As someone who is experienced with the instance segmentation literature, but has not read the SAM 3 paper in detail, I was somewhat confused at first what the metrics were. Also, conventional benchmarks are often reported using mAP/IoU on datasets such as COCO, ADE20K, LVIS, etc. The lack of conventional benchmarks makes it challenging to compare to related works.

The use of external models in the experiments undermines the trust in the claims. For example, Gemini 3.1 Flash-Lite is used to check if the queried object exists in the image and return no mask if not. However, in theory, the model should return a perfect black image in this case, resulting in no need for an external model. If the model does not do this, it suggests an inherent weakness. Similarly, for referring instance segmentation, GPT 5.6 Sol is used several times in the loop to verify and reprompt the model if output is bad. This also leads to a substantial jump in performance for appendix C.

I feel that I can understand this paper because I am familiar with relevant literature and have worked in this area. But it is very clear to see how someone new to this could get lost reading this, as it feels like you have to "go digging" in the paper for core experimental details.

Questions for authors

I am very interested in segmentation performance on small objects, reporting IoU or mAP. Prior works such as gen2seg (Khangaonkar and Pirsiavash, 2026) have found that the segmentation performance for small objects using a generative prior can be very weak. Do you experience similar weaknesses, and if so, what resolves it?

gen2seg: Generative Models Enable Generalizable Instance Segmentation. Om Khangaonkar and Hamed Pirsiavash. ICLR 2026.

Are the claims of "zero-shot" really valid if many "nearest-neighbors" exist in the labeled data that UniVision finetunes on?

3 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer 3rrD (1/2)

Thanks for your feedback. please find our detailed rebuttal below.

  1. Novelty in codec and experiment.

On codecs, our specific refinement concerns segmentation decoding: we use palette confidence and spatial support to identify reliable regions and resolve uncertain pixels caused by color drift and boundary blending.

Appendix A.1 provides the comparison you ask for. On the same fixed UniVision outputs, without retraining or regenerating predictions, our implementation of Vision Banana’s instance decoder achieves 38.58 cgF1 and 45.35 pmF1, while our full decoder achieves 45.1 cgF1 and 52.8 pmF1. For semantic segmentation, decoding refinements improve Cityscapes mIoU from 64.6 to 67.7. These results establish the benefit of our decoder on UniVision outputs. We agree these results are important and will bring this comparison into the main text and distinguish the inherited codec components from our refinements.

We also want to emphasize that, beyond the codecs and an open recipe for training a strong model, our findings on data scaling and data synergy are an important part of our contribution. These insights help the community better understand how perception-as-generation models benefit from more data and how perception training can improve generation.

Our general response adds two new experiments on novelty, and we would be grateful if you could read it: the UniVision recipe carries over to video without architectural changes, where it outperforms specialized video models on metric depth and surface normals, and UniVision learns a new perception task, image matting, with a small LoRA trained in under 3 GPU hours.

  1. metric

We agree that the paper should define these metrics instead of deferring to the SAM 3 paper, and we explain below how UniVision produces instance masks, which is why these metrics fit it.

How UniVision segments instances. Given an image and a noun phrase, UniVision generates an RGB image in which every instance of the phrase is painted in its own palette color on a black background. Our codec then decodes this image: each pixel is assigned to the nearest palette color, uncertain pixels are resolved from reliable neighboring regions, and each color becomes one instance mask; an all-black image decodes to no mask. The output is therefore a set of masks.

Why mask AP does not apply. Mask AP, the metric of COCO, LVIS and ADE20K instance segmentation, ranks all predicted masks by a confidence score and averages precision over recall as the score threshold varies, so it needs a score for every mask. A generated mask comes without such a score: the generator paints an instance or it does not, and the paint carries no probability. With all masks tied, AP collapses to a single precision-recall point and no longer measures what it is meant to measure.

The metric we use. SA-Co/Gold, the open-vocabulary instance segmentation benchmark released with SAM 3, scores the set of masks that a model returns, which matches our output, and SAM 3 and Vision Banana both report it. Each query pairs an image with a noun phrase, and the model must return every instance of the phrase, or nothing when the phrase is absent, which is the case for about 83% of the 166,766 queries. The official metric, cgF1, is the product of two factors, computed in each of the seven domains and then averaged. pmF1 is the micro F1 of one-to-one mask matches on queries whose phrase is present, averaged over IoU thresholds from 0.50 to 0.95 as in mask AP. IL_MCC is the Matthews correlation of the image-level present or absent decision over all queries, so it penalizes masks returned for absent phrases. For semantic segmentation, we report the conventional mIoU on Cityscapes.

We will add this explanation and the definitions of cgF1, pmF1 and IL_MCC to Section 4.1.

AuthorsOpenReview ↗

Rebuttal for Reviewer 3rrD (2/2)

  1. externel model

We agree that the role of the external models must be separated from what UniVision does itself, and that ideally UniVision would return an empty mask for an absent phrase on its own. It does not yet, because our instance training data contains no absent-target queries, so the model was never asked to return an empty mask; adding such queries is a direct extension. The external model never draws a mask: it only decides whether the phrase is present (instance segmentation), or rephrases the query and picks among UniVision's samples (referring segmentation). Three controls on the reported checkpoint show that the segmentation quality comes from UniVision.

First, the gate does not need a proprietary model: a Qwen3-VL-30B-A3B-based model that we fine-tuned for presence on public COCO, LVIS, PACO and OpenImages images matches Gemini as the gate (UniVision row of the table below).

Second, the same gate does not lift other models to our level. Behind the same gate on the full SA-Co/Gold benchmark (each detector at its best score threshold):

mask generatorGemini gateour open gate
UniVision45.1445.29
Grounding DINO-B + SAM 2.1 masks42.4442.82
OWLv2-L + SAM 2.1 masks31.0030.89

Third, in referring segmentation the same rewritten phrase helps UniVision and SAM 3 alike, and UniVision stays ahead (RefCOCOg cIoU, one prediction per expression):

Mask generatorOriginal expressionOne rewrite by the open-weight Kimi K3
SAM 355.4667.99
UniVision63.4075.95

The gate and the rewrite are therefore light, replaceable steps that open models perform as well as proprietary ones, while the segmentation quality comes from UniVision. We will report the standalone and assisted results side by side in Table 1 and add these controls to the paper.

  1. small object

Yes. Like every segmenter, UniVision is less accurate on small objects than on large ones, in line with what gen2seg found for generative priors. Thank you for pointing us to gen2seg; it is important related work that we missed. It fine-tunes Stable Diffusion and MAE to output instance colorings, close to our palette codec, and shows that generative pretraining transfers to unseen object types. We will add it to the related work.

On small objects, UniVision is not weaker than other open-vocabulary segmenters. We grouped the SA-Co/Gold queries whose phrase is present by the area of their smallest ground-truth instance (2.5K to 10K queries per group) and scored UniVision and two baselines with the official evaluator on the same queries:

smallest instance (% of image)< 0.050.05 to 0.20.2 to 11 to 5≥ 5
pmF1
UniVision, no gate31.842.752.862.962.0
UniVision + Gemini gate (reported setting)32.442.552.362.861.7
UniVision + Gemini gate, naive decoding16.620.021.521.519.0
OWLv2-L + SAM 2.1 masks29.436.642.045.144.8
Grounding DINO-B + SAM 2.1 masks32.940.047.054.660.2
Recall at IoU 0.5
UniVision + Gemini gate40.255.567.080.078.3
same, naive decoding40.655.066.479.278.0

The box detectors get their masks from SAM 2.1, as in the SA-Co/Gold release, and each uses the score threshold that maximizes its overall pmF1. Our OWLv2 run comes close to its published value (41.4 vs 42.0).

All models lose accuracy on small objects. UniVision is above OWLv2 in every size group, level with Grounding DINO-B in the smallest group and clearly above it in the other four. For UniVision the loss is in recall, while precision stays near 65. Our decoder does not cause this loss: naive nearest-color decoding without any area floor has the same recall in every group, while our decoder raises pmF1 in every group, from +15.8 for the smallest objects to +42.7 for the largest. The limit therefore lies in the generated outputs, and since instance accuracy still rises with the input resolution cap, higher-resolution inputs are the most direct remedy. We will add this analysis to the paper.

  1. Zero-shot setting.

We believe the zero-shot claim holds. By “zero-shot”, we mean transfer to downstream benchmarks without using their training splits for UniVision's perception fine-tuning. UniVision is trained on no training pairs from any of the benchmarks we evaluate on, and on no referring expressions from the RefCOCO family; all of its training pairs come from separate data sources. Visually similar images can occur in the training collection of any model trained at this scale, so zero-shot evaluation is defined by whether a benchmark's own training data is used, and under this definition UniVision's results are zero-shot. We will state this definition explicitly in Section 4.1.

Thanks for the detailed rebuttal. At this stage, I am still left wondering “what did I learn from this paper?”. It is hard to find an answer. I agree that the released code data and weights will likely be of use to the community. However, I still do not feel that the paper holds sufficient insight to be accepted to a top tier conference. I encourage the authors to release their work and find public impact on arXiv, but I do not feel that ICLR is the right venue for such a work and thus I keep my score.

Reviewer_GNFAOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper starts from the Cosmos2 Nano image generation model and adapts it for various visual understanding tasks: metric depth, surface normals, semantic/instance/referring segmentation. They do this by keeping the diffusion framework also for understanding, where the all output types are represented in RGB images. They use specific noise insensitive 'codecs' to help the conversion from targets to/from the RGB space (e.g. for segmentation one needs a color palette but the diffusion process will generate pixels which fall outside any predefined palette). They also have a data curation pipeline to add pseudo labels to unannotated images. Results look solid and there are some good ablations. But it is hard to understand what the authors bring to the table in terms of novelty.

Critical strengths
  • Using diffusion for understanding tasks is obtaining increasingly strong results and this paper is an example of this. It outperforms most previous works who have attempted this.
  • Ablations are insightful.
Critical weaknesses

MAJOR: I do not know what this paper bring in terms of novelty compared to Nano Banana, DiCeption, but also "what matters repurposing diffusion [,,,] for general dense perception tasks "https://openreview.net/forum?id=BgYbk6ZmeX , Visual Bridge https://arxiv.org/pdf/2511.07877 , GenRep https://openreview.net/forum?id=yKC3CaFg8K (these are all published works). Normally I feel the community is a bit too obsessed with novelty. But in this case:

  • the authors do not even attempt to distinghuish themselves and only have a mini paragraph on related work.
  • the method section basically explains standard diffusion in 3.1 and 3.2, has something with conversion of targets and back into pure RGB without references to the many related works which do the same (Sec 3.3), and the data curation section (Sec 3.4) is so small it contains no relevant information while pseudo annotations are extremely common nowadays. So while I was initially excited to read this paper, after reading it I have little new knowledge except that diffusion works for image understanding, which was already demonstrated by the Vision Banana paper among an increasing growing body of literature.

Other problems:

  • Table 1 misses many metric names and dataset names.
  • The Data synergy section (4.5) is extremely hard to read. T2I is never introduced so I do not understand what is going on. Also, in the second paragrah a sentence is not even complete.
  • The data scaling is extremely hard to read. Why vary both the number of training tokens and the data size? Just pick one and the graphs can be better grasped. Also, the text here is extremely convoluted and non-smooth.
  • The 'benchmarks' section contains a lot of not-very-relevant information and can better be moved to the supplemantary.
  • Fig 3. would benefit from also including the RGB images.
  • Test-time aggregation is known to work. But while I see the value for segmentation, for depth the averages re-introduce the problem that depth for boundary pixels is not truly defined (light comes from two objects) and while averaging may improve scores due to the used metrics, it also creates inconsistent depth maps with floating pixels.
  • Figure 6 is too small. Consider showing only 2 or 4 examples and move the rest to the supplementary.
Questions for authors

What is the difference between this models and models which came before it? Generally, the related work section should triple in size to discuss this more thoroughly while several other sections in this paper can easily be moved to the supplementary.

Will code and model weights be made available? From the introduction, this seems to be the main motivation of the paper and it would be beneficial to the community.

For me this is currently a true borderline paper. Currently I'm leaning towards reject because I have learned little and writing needs to be much improved for an top-tier conference paper. Having a better understanding of why this method is better than others and releasing the model weights and training code to the community may convince me to raise my score, since I do acknowledge getting such models to work this well requires serious amd difficult work.

2 comments

Thank you for your feedback and for raising this concern about novelty. We believe our work differs from VisionBanana and other related work in several important ways, especially when we consider what it takes to train a large model for perception tasks:

  • Training recipe and data pipeline. These works do not provide detailed accounts of their training recipes and data pipelines. Yet these details are central to training large models, and we believe sharing them is valuable to the community.
  • Data scaling. Scaling up training data is an important part of large-model training, and zero-shot performance depends on generalization. This raises a natural question: do we see similar scaling effects when we formulate perception as generation? And do the scaling curves differ across perception tasks? Section 4.6 explores these questions and presents findings that, to our knowledge, have not been reported in these works.
  • Data synergy. Taking this a step further, we would expect perception training data to help shape the model’s internal representations. Could this also improve performance on other generation tasks? This matters to us because our unified model is designed for both perception and generation. Our data synergy section explores this question, which we believe remains underexplored in related work.
  • Yes, we will release the weights of a larger model with better perception-as-generation performance, and we agree that this will be useful to the community. Beyond model performance, we care about understanding how to train such models, how they benefit from more data, and how perception and generation can benefit each other. We see these insights as an important part of our contribution.

We would also be grateful if you could read our general response, which addresses novelty with two new experiments: the UniVision recipe carries over to video without architectural changes, where it outperforms specialized video models on metric depth and surface normals, and UniVision learns a new perception task, image matting, with a small LoRA trained in under 3 GPU hours.

Writing, Table 1 and figures (other problems). Thank you for these concrete suggestions. We agree with each of them and will make the following changes in the revision:

  • Related work and method. The related work will be expanded to discuss GenPercept, DICEPTION, GenRep, Visual Bridge and Vision Banana in detail. To make room, Sections 3.1 and 3.2 will be shortened to the essentials, Section 3.3 will cite prior RGB encodings and state which codec parts are inherited and which are ours, and Section 3.4 will summarize the curation pipelines and data sources that are currently only in the appendix.
  • Table 1. Every panel will name its dataset and metric, for example Cityscapes mIoU, RefCOCOg cIoU and SA-Co/Gold cgF1.
  • Data synergy (Section 4.5). We will define text-to-image (T2I) at its first use, describe the four training conditions in full sentences, and complete the unfinished sentence in the second paragraph.
  • Data scaling (Section 4.6). We vary both axes because data size and training compute are confounded: at a fixed token budget, a smaller data fraction means more passes over the same samples. Comparing data fractions at the same budget isolates the effect of data, and following one curve shows the effect of more training. The revised section explains this choice, keeps the curves, and states the three findings in plain sentences: more data mainly helps metric depth and referring segmentation, 10% of the data comes within one point of the full data for semantic and instance segmentation, and relative depth and normals need only about 20K samples.
  • Benchmarks paragraph. The main text will keep only the datasets and metrics, and the protocol details will move to the appendix.
  • Figure 3. We will add the RGB input images.
  • Figure 6. We will show two of its six examples at a larger size and move the other four to the appendix.
Reviewer_GNFAOpenReview ↗

Maintain reject

I thank the authors for their response. However, reviewer 3rrD raised similar concerns as I did. And while some of these concerns are addressed in the response, it still falls short of ICLR standards. Generally this paper goes in the right direction but I feel a substantial rewrite and refocus is needed to show better to the audience what exactly this paper is about and what it shows. This feeling strengthed by the sentiments in the other reviews. I therefore maintain my score: Weak reject.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.