- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
UniVision fine-tunes the generation expert of a pretrained vision-language/image-generation model to produce RGB encodings of depth, normals and segmentation. Task-specific decoders convert the generated images into predictions. The paper builds a large mixed training collection, studies resolution, guidance and sample aggregation, and tests data scaling and the effect of perception training on text-to-image generation. Some segmentation results additionally use external multimodal models for presence detection or query rewriting and mask selection.
Critical strengths
The empirical scope is substantial, and the paper provides detailed codecs, prompts, data sources and inference protocols. It explicitly acknowledges the perception-as-generation precedent, including Vision Banana [1], DICEPTION and earlier work. The fixed-output codec ablation is useful: it shows a real improvement from the proposed decoding refinements. The data-scaling study controls training-token budgets, and the text-to-image synergy experiment includes controls for both iteration count and text-to-image exposure.
Critical weaknesses
1. Two central task results depend heavily on external proprietary perception systems. This is disclosed in the table and appendix, but it changes what the results establish about UniVision. Table A.2 reports instance cgF1 of 2.30 without Gemini gating and 45.14 with it; image-level presence MCC changes from 0.0442 to 0.8504. Positive-query segmentation remains useful, but the generator largely fails to reject absent targets. The appendix attributes this to training without negative queries. This is a substantial gap in open-vocabulary instance segmentation, not a small optional enhancement.
Similarly, referring cIoU rises from 63.40 to 75.39 with the agent in Table A.3. Appendix C.5 specifies GPT-5.6 Sol with xhigh reasoning, up to two rounds of four 50-step generations, and multimodal selection/rewrite calls. The controller interprets the source image and query, so it supplies perception and grounding capabilities of its own. The resulting system can be useful, but its score does not isolate what the open generator learned. Please put the standalone rows in the main table, train a negative-query control, and separate candidate sampling, selection and rewriting under matched budgets. A comparison using the same controller/candidate budget for an alternative mask generator is also needed. The current all-or-nothing agent ablation cannot attribute the improvement among these steps.
2. The incremental scientific contribution and its computational cost need a sharper evaluation. The overall RGB-output formulation and the depth codec follow existing work, which the authors appropriately cite. The potentially useful additions are the data recipe, decoding refinements and empirical findings. Table A.1 supports the refinements, but does not show whether the more complex decoder is robust to different output resolutions, object sizes and thresholds. The palette, component-size and merge rules introduce several choices; their sensitivity and the validation split used to select them should be reported.
The claimed inference improvements also increase cost. The default uses 50 denoising steps, high-resolution inputs for some tasks, and potentially multiple samples; the referring agent adds external reasoning calls. There is no corresponding latency, memory or cost comparison. Accuracy-versus-compute curves would make these improvements much easier to assess. The architecture also contains a frozen 8B vision-language backbone in addition to the 8B generation expert, so reporting only the latter's parameter count does not describe the full inference system.
Finally, the 1.91-point text-to-image gain over the matched-exposure control is interesting, but is measured with one generation per prompt and one automated judging protocol. Prompt-level confidence intervals, repeat seeds and an independent check would help establish its robustness. Metric-depth performance is also uneven: the average delta1 is 0.842 versus Vision Banana's 0.933, with ETH3D AbsRel 0.664. Figure 3 uses scale-and-shift-aligned depth, as its caption states, so it illustrates structure rather than validating metric calibration. The paper should more clearly distinguish these limitations from its strong relative-depth results.
I lean toward rejection in the current form. The study is useful, but the standalone capabilities, incremental findings and full-system cost need to be established more clearly.
Questions for authors
- Can negative-query training remove the dependence on Gemini for instance presence detection? Please include standalone results in the main table.
- What are the separate contributions of additional samples, mask selection and query rewriting? How does another mask generator perform with the same controller and budget?
- On what held-out data were codec thresholds, resolution, shift and guidance selected? How sensitive are the decoding gains to these choices?
- Please report full-system parameters, latency and peak memory, including external calls and sample count, and uncertainty for the claimed text-to-image synergy.
Presentation
The appendix is detailed, but the main text should make the standalone-versus-assisted results more prominent. Table 1 compresses six tasks into small panels, and Figs. 4–5 use small axes and labels. Enlarging these panels and reporting the inference budget beside each result would improve readability. Claims about metric depth should be supported with an unaligned metric visualization alongside the current aligned example.
Reference
[1] Image Generators are Generalist Vision Learners (Vision Banana). https://arxiv.org/html/2604.20329v3
2 comments
Rebuttal for Reviewer Sk4x (2/2)
3. Full-system size, latency and memory (Weakness 2, Question 4). We agree that the cost of the full system should be reported, including samples and external calls. The full system has about 15.9B parameters: the frozen 8.2B language backbone, the 7.0B generation expert with its adapters (the only trained part), and the VAE. At the paper's settings (one GB200 GPU, batch 1, bf16, 50 steps, VAE decoding included), one prediction takes 7.0 s for depth, 3.6 s for normals, 7.4 s for semantic and 8.4 s for instance segmentation, with 31 to 36 GiB peak memory. The referring agent runs 4 generations per expression, 4 more for the 7% of expressions it rewrites (345 of 4,896), and 1.14 GPT-5.6 calls per expression on average.
The 50-step default is conservative. A step sweep on the same checkpoint gives the accuracy versus time trade-off (time per image in brackets):
| task | 50 steps | fewer steps |
|---|---|---|
| metric depth, mean δ1 | 84.18 (7.0 s) | 86.67 at 4 steps (0.75 s) |
| normals, mean error | 18.79° (3.6 s) | 18.16° at 1 step (0.24 s) |
| instance, cgF1 | 45.14 (8.4 s) | 45.01 at 15 steps (3.3 s) |
| semantic, mIoU | 67.65 (7.4 s) | 67.33 at 25 steps (4.0 s); 55.17 at 10 steps |
For depth and normals, one to four steps are both faster and more accurate (9× and 15× less time), while segmentation stays within 0.4 points of its 50-step accuracy at 15 to 25 steps, in about half the time. UniVision's accuracy therefore does not depend on an expensive inference budget. We will report the full-system parameter count, put the inference budget (steps, samples, external calls) beside each result in Table 1, add these curves, and add the cost of test-time aggregation (N generations) to Figure 5.
4. Uncertainty of the text-to-image synergy (Weakness 2, Question 4). We agree that one generation per prompt and one judge call for prompt-level uncertainty and an independent check. We computed paired bootstrap intervals over the 1,170 prompts (50,000 resamples) and re-scored the same saved images with a second judge, GPT-5.4, under the unchanged UniGenBench protocol (gain of w/ vision in points, 95% CI):
| Comparison | Gemini 3 Flash (paper's judge) | GPT-5.4 (second judge) |
|---|---|---|
| vs T2I control matched in T2I updates, all prompts | +1.90 [0.89, 2.94] | +1.76 [1.05, 2.47] |
| same control, Public prompts | +3.32 [1.94, 4.71] | +2.19 [1.17, 3.23] |
| same control, Physical AI prompts | +0.61 [−0.90, 2.13] | +1.36 [0.38, 2.33] |
Against the T2I control matched in iterations, the paper's judge gives an even larger gain, +3.76 [2.67, 4.85]. The two judges agree on 85.9% of the 8,193 test points, and the overall gain is significant under both. We will report the overall gain with these intervals in Section 4.5 instead of per-category rankings, and state that the joint run samples text-to-image, vision and editing data in equal thirds.
5. Metric depth (Weakness 2). We agree that relative and metric depth should be presented separately, and the revision does so. UniVision's relative depth is the best among the compared methods on KITTI and ETH3D (δ1 98.0 and 98.9), and its metric gap is a per-image scale error rather than a structural one: dividing out one scalar per image (the median prediction to ground truth ratio, no shift) brings every benchmark to 97.3 to 97.7 δ1.
| Dataset | Metric δ1 as is | δ1 with one scale per image | Images with scale off by more than 25% |
|---|---|---|---|
| NYUv2 | 93.5 | 97.6 | 3.5% |
| KITTI | 92.6 | 97.7 | 4.1% |
| ETH3D | 70.6 | 97.5 | 28.4% |
| iBims-1 | 80.0 | 97.3 | 17.0% |
The ETH3D AbsRel of 0.664 comes mostly from seven images of the "terrains" scene that are predicted 18 to 54 times too far: they carry 68.5% of the total, and AbsRel is 0.213 without them. Inference already narrows the gap: 4 sampling steps give 86.67 instead of 84.18 mean δ1 (ETH3D 75.5 instead of 70.6), and test-time aggregation corrects part of the scale. The metric gap is therefore a calibration error that inference already reduces, while the predicted geometry is accurate. We will separate the relative and metric depth claims in Section 4.2 and add this analysis.
Rebuttal for Reviewer Sk4x (1/2)
Thank you for the detailed review, and for recognizing the codec ablation on fixed outputs, the token-controlled scaling study and the matched controls in the synergy study. We answer your points in order, with new runs on the reported checkpoint. We would also be grateful if you could read our general response, which shows that the same recipe carries over to video and that UniVision learns a new perception task with a small LoRA.
1. Dependence on external models (Weakness 1, Questions 1 and 2). We agree that the standalone and assisted results must be separated, and the revised Table 1 will show them side by side. The external models never draw a mask: they judge presence (instance) or select and rephrase (referring), and every mask comes from UniVision. Our controls show that the segmentation quality is UniVision's own and that open models can replace the proprietary ones.
Instance (Question 1). The gate leaves mask quality unchanged (pmF1 on present phrases 53.35 without it, 52.82 with it); what it adds is the rejection of absent phrases, which our instance data never asked for because it contains no absent-target queries. The dependence on Gemini is removed by an open gate, and the same gate does not bring other generators to our level (full SA-Co/Gold cgF1, box detectors at their best threshold):
Our open gate is a Qwen3-VL-30B-A3B-based model fine-tuned for presence on COCO, LVIS, PACO and OpenImages images. Adding absent-target queries to the training data is the direct way to fold this decision into the generator.
Referring (Question 2). We agree that sampling, selection and rewriting should be separated under matched budgets. The agent logs separate them (RefCOCOg val cIoU):
Selection adds 7 points and is bounded by the oracle; rewriting the query adds the rest, and one open-model rewrite with a single sample matches the full agent with a quarter of the generations. With the same rewrite and budget, SAM 3 reaches 67.99 (55.46 with the original expression): the rewrite lifts both generators by the same 12.5 points, and UniVision stays 8 points ahead. The agent's gain therefore comes from clarifying the query, which helps any generator equally, while UniVision's lead comes from its masks. We will add this breakdown and the SAM 3 control to the paper.
2. Robustness of the decoding refinements (Weakness 2, Question 3). We agree that the decoding refinements, one of the contributions you single out, must show that their gain does not hinge on their own settings. We re-decoded saved outputs of the reported checkpoint, without regenerating them, once with naive nearest-color decoding and once with our decoder, and varied each factor you name:
Our decoder adds about 3 mIoU and 28 cgF1 at every output resolution and guidance scale, and it helps objects of every size, from +15.8 pmF1 for the smallest to +42.7 for the largest. Its thresholds sit in a flat region: changing any palette, component-size or merge threshold moves the result by at most 1.1 mIoU or 0.4 cgF1, several changes even score slightly higher than the reported values, and removing the merge rule altogether still leaves 26 of the 28 cgF1 points. The decoding gain therefore comes from the design of the decoder rather than from a particular threshold or inference setting. We will add these results to Appendix A.1.