- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes DVD, which encodes 2D boxes, 2D masks and 3D boxes into a small number of representation tokens and adds these tokens to the MLLM vocabulary. In this way, DVD unifies 2D and 3D representations without fixed range constraints.
Critical strengths
S1. The paper's motivation and direction are important and interesting, and the topic has the potential for broad impact.
S2. DVD is very effective at reducing the number of tokens needed to represent perception outputs.
Critical weaknesses
W1. On the 3D task, text coordinates outperform DVD (Table 8), which conflicts with the paper's core motivation. The paper's conclusions may be overstated.
W2. I have serious doubts about the fairness of the experiments. DVD's training signal comes from datasets in the same domains as the evaluation benchmarks, whereas the generalist MLLMs it is compared with are trained with denser and broader supervision. Given this, I would like to know whether DVD suffers from catastrophic forgetting. The authors should evaluate it on more benchmarks from different domains and task types, to check whether DVD's performance drops relative to other MLLMs.
W3. The reported performance is limited. InternVL3.5-8B outperforms DVD on RefCOCO+ val (87.9 vs. 87.7; Table 3). On referring segmentation, DVD-2B scores lower than LISA-7B and PerceptionGPT on RefCOCO val (69.9 vs. 74.9 / 75.1 cIoU; Table 4). In addition, DVD-8B performs worse than DVD-2B on several benchmarks, including RefCOCO val, RefCOCOg val, KITTI and ARKitScenes. The paper does not analyze this, and it raises concerns about the method's robustness and generalization.
W4. Compared with the ambition of its motivation, the paper's experiments are insufficient, and it lacks interpretability evidence to show that the design is reliable.
Questions for authors
Could the authors provide more efficiency analysis?
4 comments
Part 2/2
W4: Experiments are insufficient, and it lacks interpretability evidence
We conduct more experiments and analysis to provide the reliability evidence.
(1) The controlled comparison of different representations. Please refer to Figure 4 and Table 10 in the revised paper.
| Representation | Tokens | Latency | 3D AP3D | Tokens | latency | 2D P@50 | Tokens | Latency | RES cIoU |
|---|---|---|---|---|---|---|---|---|---|
| Text | 74 | 5493 ms | 38.0 | 15 | 729 ms | 84.5 | 512 | 25078 ms | 49.2 |
| Special Token | 15 | 925 ms | 25.1 | 4 | 265 ms | 82.0 | 128 | 6213 ms | 51.9 |
| DVD (ours) | 8 | 620 ms | 33.7 | 2 | 146 ms | 86.3 | 64 | 1672 ms | 62.0 |
(2) Reconstruction quality actually transfers to downstream predictions. The geometric-loss ablation in (2) shows the geometric loss improves the reconstrcution and prediction on three tasks. Please refer to Table 6 in the revised paper.
| Task | w/o geo | w/ geo |
|---|---|---|
| AP3D@15 | 31.8 | 33.7 |
| 2D P@50 | 85.6 | 86.3 |
| RES cIoU | 57.1 | 62.0 |
(3) The detailed efficiency gains. Please refer to Q1. The stage-wise profiling in Q1 shows DVD largely reduces the latency of llm decoding by the more efficient representation.
(4) OOD results of DVD. We retrained an autoencoder without using any outdoor data, and then tested it on the KITTI dataset, which is not included in the training process. Indoor distances are typically less than 10 meters.
| Data | SUN-RGBD | KITTI |
|---|---|---|
| All | 98.9 | 99.6 |
| w/o Outdoor | 97.9 | 63.9(OOD) |
Distance breakdown: Near (<10m), Mid (10–35m), Far (>35m) on the unseen dataset:
| Setting | KITTI N/M/F |
|---|---|
| All | 99.7 / 98.0 / 83.6 |
| w/o Outdoor | 89.5 / 56.9 / 26.8 |
Without outdoor training data, the codebook still reconstructs completely unseen large-scale outdoor scenes at 63.9 AP3D@15. The accuracy drop concentrates at far ranges (>35 m), while near-range reconstruction remains high (89.5 at <10 m). These results indicate that DVD retains a certain degree of transferability to previously unseen outdoor scenes. Please refer to Table 11 in the revised paper.
Q1: More efficiency analysis
To provide more efficiency analysis, we breakdown the latency into vision encoder, prefill, decoding, and de-tokenization. DVD significantly improve the efficiency by introducing negligible addtional de-tokenization latnecy. Please refer to Table 5 in the revised paper.
| Method | Task | tokens | vision encoder | llm prefill | llm decoding | de-tokenization | All |
|---|---|---|---|---|---|---|---|
| text | det3d | 74 | 57.9 | 38.1 | 5397 | 0.1 | 5493 |
| text | det2d | 15 | 17.3 | 25.8 | 686 | 0.1 | 729 |
| text | seg | 512 | 17.5 | 25.8 | 25034 | 0.2 | 25078 |
| tok4k | det3d | 15 | 57.9 | 38.0 | 829 | 0.1 | 925 |
| tok4k | det2d | 4 | 16.9 | 25.5 | 222 | 0.1 | 265 |
| tok4k | seg | 128 | 17.5 | 25.6 | 6170 | 0.1 | 6213 |
| vae | det3d | 8 | 57.9 | 38.0 | 517 | 7.4 | 620 |
| vae | det2d | 2 | 17.1 | 26.1 | 99 | 3.6 | 146 |
| vae | seg | 64 | 17.7 | 26.0 | 1624 | 4.0 | 1672 |
Dear Reviewer gD3h,
Thank you for recognizing our "topic has the potential for broad impact." and "DVD is very effective at reducing the number of tokens needed to represent perception outputs." We hope our rebuttal has addressed your concerns with new evidence and we have revised our paper to include more analysis and experiments.
As the discussion period ends soon, we would greatly appreciate knowing whether these new results address your concerns. We remain happy to provide further experiments or details. Thank you.
Dear Reviewer gD3h,
Thank you again for your time in this discussion. As the November 18 discussion deadline approaches, we would greatly appreciate it if you could revisit our latest response. We sincerely hope our response can address your remaining concerns.
Thank you for your time.
Part 1/2
We thank the detailed reviews, which will significantly improve our paper. We respond to these reviews point by point. Moreover, we have uploaded the revised paper for your reference.
W1: On the 3D task, text coordinates outperform DVD (Table 8), which conflicts with the paper's core motivation
We would like to clarify that this trade-off primarily exists in 3D space. On the contrary, it simultaneously reduces the number of tokens and improves accuracy on 2D tasks. DVD also substantially outperforms special tokens in accuracy while using fewer tokens.
W2: The fairness of the experiments
We would like to clarify that the main table does not exclusively compare against general-purpose models: the 3D grounding includes VST-3B; 2D grounding includes Rex-Omni; and the RES includes methods tailored for perception tasks, such as Text4Seg. Furthermore, these methods differ from DVD in backbones and training data, the main table illustrates the performance achievable within a unified DVD framework. A fair comparison between different representations is instead provided through controlled experiments.
We also evaluate general QA against the official Qwen3-VL-8B checkpoint. DVD-8B uses 665K additional general-purpose QAs during training.
Since our goal is to address the high-cost problem in perception tasks based on MLLMs, we have not incorporated an excessive amount of general-purpose QA data. Including such data in the training set can restore the general capabilities of an MLLM.
W3: The reported performance is limited
(1) InternVL3.5-8B outperforms DVD on RefCOCO+ val (87.9 vs. 87.7; Table 3)
In terms of overall performance across 8 splits for 2D grounding, DVD achieves an average score of 90.0, compared to 89.7 for InterVL3.5-8B. This indicates that the overall performance of DVD remains superior to that of InterVL3.5-8B.
(2) On referring segmentation, DVD-2B scores lower than LISA-7B and PerceptionGPT on RefCOCO val.
We would like to clarify that comparing DVD-2B with larger models such as LISA-7B is not equitable. These models differ significantly in scale. And our DVD-8B achieves clear performance improvements compared with these methods.
(3) DVD-8B performs worse than DVD-2B on several benchmarks, including RefCOCO val, RefCOCOg val, KITTI and ARKitScenes.
Averaging across multiple datasets is a more robust way to compare performance. We therefore additionally report task averages as the robust result in the revised paper. The average performance of DVD-2B is 33.9, 90.3, 68.2 for 3D grounding, 2D grounding, 2D RES, respectively. The average performance of DVD-8B is 34.8, 90.0, 73.5 for 3D grounding, 2D grounding, 2D RES, respectively. On average, scaling the backbone helps the geometry-heavy tasks, while 2D grounding is essentially saturated.