acceptodds
Under review as a conference paper at ICLR 2027

DVD: Dynamic Vector Decoding for Efficient MLLM-based Perception

Abstract

Multimodal large language models have made remarkable progress in bridging vision and language, facilitating various perception tasks essential for human-machine interaction, robotics, and autonomous driving. However, existing MLLM-based perception methods predominantly rely on text-based coordinate representation, which suffers from excessive token overhead, or fixed-range quantization, which suffers from range and precision constraints, especially for 3D domains with unbounded spatial range and high localization accuracy requirements. To address these challenges, we propose a dynamic vector decoding method named DVD, which unifies the representation of 2D and 3D perception tasks. Specifically, we first transform diverse perceptual representation (i.e., 2D bounding boxes, 2D masks, and 3D bounding boxes) into 1D vector sequences, which are then mapped to compact discrete tokens in the high-dimensional space. Then, a lightweight de-tokenizer enables seamless integration with MLLMs by decoding output tokens back to original 2D and 3D perceptual representations. Extensive experiments on 2D and 3D perception benchmarks including RefCOCO series, SUN-RGBD, KITTI, Hypersim, nuScenes demonstrate that DVD achieves superior performance in 2D and 3D tasks and reduces significantly the token overhead and inference latency. DVD provides an efficient and general framework for integrating perception capabilities into MLLMs, overcoming the inherent limitations of existing methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Perception, Multimodal Large Language Model, Efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gD3hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DVD, which encodes 2D boxes, 2D masks and 3D boxes into a small number of representation tokens and adds these tokens to the MLLM vocabulary. In this way, DVD unifies 2D and 3D representations without fixed range constraints.

Critical strengths

S1. The paper's motivation and direction are important and interesting, and the topic has the potential for broad impact.

S2. DVD is very effective at reducing the number of tokens needed to represent perception outputs.

Critical weaknesses

W1. On the 3D task, text coordinates outperform DVD (Table 8), which conflicts with the paper's core motivation. The paper's conclusions may be overstated.

W2. I have serious doubts about the fairness of the experiments. DVD's training signal comes from datasets in the same domains as the evaluation benchmarks, whereas the generalist MLLMs it is compared with are trained with denser and broader supervision. Given this, I would like to know whether DVD suffers from catastrophic forgetting. The authors should evaluate it on more benchmarks from different domains and task types, to check whether DVD's performance drops relative to other MLLMs.

W3. The reported performance is limited. InternVL3.5-8B outperforms DVD on RefCOCO+ val (87.9 vs. 87.7; Table 3). On referring segmentation, DVD-2B scores lower than LISA-7B and PerceptionGPT on RefCOCO val (69.9 vs. 74.9 / 75.1 cIoU; Table 4). In addition, DVD-8B performs worse than DVD-2B on several benchmarks, including RefCOCO val, RefCOCOg val, KITTI and ARKitScenes. The paper does not analyze this, and it raises concerns about the method's robustness and generalization.

W4. Compared with the ambition of its motivation, the paper's experiments are insufficient, and it lacks interpretability evidence to show that the design is reliable.

Questions for authors

Could the authors provide more efficiency analysis?

4 comments

We thank the detailed reviews, which will significantly improve our paper. We respond to these reviews point by point. Moreover, we have uploaded the revised paper for your reference.

W1: On the 3D task, text coordinates outperform DVD (Table 8), which conflicts with the paper's core motivation

We would like to clarify that this trade-off primarily exists in 3D space. On the contrary, it simultaneously reduces the number of tokens and improves accuracy on 2D tasks. DVD also substantially outperforms special tokens in accuracy while using fewer tokens.

RepresentationTokensLatency3D AP3DTokenslatency2D P@50TokensLatencyRES cIoU
Text745493 ms38.015729 ms84.551225078 ms49.2
Special Token15925 ms25.14265 ms82.01286213 ms51.9
DVD (ours)8620 ms33.72146 ms86.3641672 ms62.0

W2: The fairness of the experiments

We would like to clarify that the main table does not exclusively compare against general-purpose models: the 3D grounding includes VST-3B; 2D grounding includes Rex-Omni; and the RES includes methods tailored for perception tasks, such as Text4Seg. Furthermore, these methods differ from DVD in backbones and training data, the main table illustrates the performance achievable within a unified DVD framework. A fair comparison between different representations is instead provided through controlled experiments.

We also evaluate general QA against the official Qwen3-VL-8B checkpoint. DVD-8B uses 665K additional general-purpose QAs during training.

MethodMMBenchPOPEMMEpRealWorldQAAvg.
Qwen3-VL-8B88.689.01723.569.283.2
DVD-8B70.289.01326.150.068.9

Since our goal is to address the high-cost problem in perception tasks based on MLLMs, we have not incorporated an excessive amount of general-purpose QA data. Including such data in the training set can restore the general capabilities of an MLLM.

W3: The reported performance is limited

(1) InternVL3.5-8B outperforms DVD on RefCOCO+ val (87.9 vs. 87.7; Table 3)

In terms of overall performance across 8 splits for 2D grounding, DVD achieves an average score of 90.0, compared to 89.7 for InterVL3.5-8B. This indicates that the overall performance of DVD remains superior to that of InterVL3.5-8B.

(2) On referring segmentation, DVD-2B scores lower than LISA-7B and PerceptionGPT on RefCOCO val.

We would like to clarify that comparing DVD-2B with larger models such as LISA-7B is not equitable. These models differ significantly in scale. And our DVD-8B achieves clear performance improvements compared with these methods.

(3) DVD-8B performs worse than DVD-2B on several benchmarks, including RefCOCO val, RefCOCOg val, KITTI and ARKitScenes.

Averaging across multiple datasets is a more robust way to compare performance. We therefore additionally report task averages as the robust result in the revised paper. The average performance of DVD-2B is 33.9, 90.3, 68.2 for 3D grounding, 2D grounding, 2D RES, respectively. The average performance of DVD-8B is 34.8, 90.0, 73.5 for 3D grounding, 2D grounding, 2D RES, respectively. On average, scaling the backbone helps the geometry-heavy tasks, while 2D grounding is essentially saturated.

W4: Experiments are insufficient, and it lacks interpretability evidence

We conduct more experiments and analysis to provide the reliability evidence.

(1) The controlled comparison of different representations. Please refer to Figure 4 and Table 10 in the revised paper.

RepresentationTokensLatency3D AP3DTokenslatency2D P@50TokensLatencyRES cIoU
Text745493 ms38.015729 ms84.551225078 ms49.2
Special Token15925 ms25.14265 ms82.01286213 ms51.9
DVD (ours)8620 ms33.72146 ms86.3641672 ms62.0

(2) Reconstruction quality actually transfers to downstream predictions. The geometric-loss ablation in (2) shows the geometric loss improves the reconstrcution and prediction on three tasks. Please refer to Table 6 in the revised paper.

Taskw/o geow/ geo
AP3D@1531.833.7
2D P@5085.686.3
RES cIoU57.162.0

(3) The detailed efficiency gains. Please refer to Q1. The stage-wise profiling in Q1 shows DVD largely reduces the latency of llm decoding by the more efficient representation.

(4) OOD results of DVD. We retrained an autoencoder without using any outdoor data, and then tested it on the KITTI dataset, which is not included in the training process. Indoor distances are typically less than 10 meters.

DataSUN-RGBDKITTI
All98.999.6
w/o Outdoor97.963.9(OOD)

Distance breakdown: Near (<10m), Mid (10–35m), Far (>35m) on the unseen dataset:

SettingKITTI N/M/F
All99.7 / 98.0 / 83.6
w/o Outdoor89.5 / 56.9 / 26.8

Without outdoor training data, the codebook still reconstructs completely unseen large-scale outdoor scenes at 63.9 AP3D@15. The accuracy drop concentrates at far ranges (>35 m), while near-range reconstruction remains high (89.5 at <10 m). These results indicate that DVD retains a certain degree of transferability to previously unseen outdoor scenes. Please refer to Table 11 in the revised paper.

Q1: More efficiency analysis

To provide more efficiency analysis, we breakdown the latency into vision encoder, prefill, decoding, and de-tokenization. DVD significantly improve the efficiency by introducing negligible addtional de-tokenization latnecy. Please refer to Table 5 in the revised paper.

MethodTasktokensvision encoderllm prefillllm decodingde-tokenizationAll
textdet3d7457.938.153970.15493
textdet2d1517.325.86860.1729
textseg51217.525.8250340.225078
tok4kdet3d1557.938.08290.1925
tok4kdet2d416.925.52220.1265
tok4kseg12817.525.661700.16213
vaedet3d857.938.05177.4620
vaedet2d217.126.1993.6146
vaeseg6417.726.016244.01672

Dear Reviewer gD3h,

Thank you for recognizing our "topic has the potential for broad impact." and "DVD is very effective at reducing the number of tokens needed to represent perception outputs." We hope our rebuttal has addressed your concerns with new evidence and we have revised our paper to include more analysis and experiments.

As the discussion period ends soon, we would greatly appreciate knowing whether these new results address your concerns. We remain happy to provide further experiments or details. Thank you.

Dear Reviewer gD3h,

Thank you again for your time in this discussion. As the November 18 discussion deadline approaches, we would greatly appreciate it if you could revisit our latest response. We sincerely hope our response can address your remaining concerns.

Thank you for your time.

Reviewer_AMXNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DVD flattens 2D boxes, mask contours, and 3D boxes into one-dimensional sequences, compresses them through a shared VQ codebook, and extends an MLLM vocabulary to predict the resulting tokens. A lightweight decoder reconstructs geometry, assisted during training by a Chamfer-based loss. The goal is unified perception with fewer output tokens and lower decoding latency.

Critical strengths
  1. The unified representation has clear engineering value. One codebook and decoding framework serves 2D boxes, masks, and 3D boxes through an ordinary autoregressive vocabulary.
  2. There is a direct representation comparison. Figure 4, pp. 8–9, compares text, scalar special tokens, and DVD under matched training settings; the paper does not lack controlled experiments altogether. For 3D boxes, token count falls from 74 to 8 and latency from 4165 ms to 499 ms. For masks, the corresponding values are 512 to 64 tokens and 24410 ms to 1715 ms. The speed ratios are arithmetically consistent.
  3. Reconstruction ablations support geometric supervision. Table 6 improves 3D AP from 80.7 to 90.8 and mask cIoU from 74.8 to 84.8. Additional experiments examine code usage, compression, and multi-task learning.
Critical weaknesses
  1. The "eliminating range and precision limits" claim is overstated. A fixed 4096-entry codebook, fixed eight-token 3D output, and fixed decoder still yield a finite, quantized output set—this is data-adaptive nonuniform quantization, not elimination of quantization. The paper's own matched comparison is the clearest evidence: textual 3D output reaches 37.5 AP versus DVD's 34.1 (pp. 8–9), and the main-table SOTA folds in a full training budget. These support an accuracy–efficiency trade-off, not "eliminated precision loss." The claim should be narrowed, or errors reported outside the training range and across scales.
  2. The decisive comparison does not isolate the shared high-dimensional codebook. The 4000-bin scalar baseline scales and clips 3D centers, sizes, and rotations into a shared range, but the exact scaling, clipping, and quantization-only reconstruction error are unclear (p. 9); its 17.5 AP more likely reflects a poorly calibrated baseline than an intrinsic limit of the 1D representation. A per-parameter calibrated scalar baseline, with reconstruction and prediction errors separated, is needed. In addition, the 3D main-table baselines (general MLLMs) are not trained on the same in-domain data, so "surpassing Qwen3-VL-8B" cannot be attributed to the representation alone.
  3. The fixed-length representation imposes a hard ceiling on mask quality. Token count is fixed per task (masks always 64 points, K=2) and does not adapt to contour complexity. Table 9 shows mask reconstruction cIoU falling from 92.7 (K=1) to 84.8 (K=2), so compression alone caps quality near 84.8—yet the paper calls this "negligible reconstruction error." The ceiling shows up in results: DVD-2B's RES clearly lags (RefCOCO val 69.9, testA 70.7 vs 77–79 for LISA/PerceptionGPT/Text4Seg), and only a larger backbone partly recovers. A 64-point ordered polygon also struggles with holes, disconnected regions, and fine concavities. The reconstruction ceiling and the cost of fixed length for complex masks should be reported.
Questions for authors
  1. How do a fixed codebook and fixed length support the "unlimited range / eliminated precision limits" claim? Can you report reconstruction error outside the training coordinate range and across object scales?
  2. What are each scalar parameter's scaling, clipping, and quantization error? What net benefit remains against a calibrated scalar baseline, and are the 3D main-table baselines trained on the same in-domain data?
  3. What is the mask reconstruction ceiling at 64 points / K=2, and the error on complex or holed masks? Can you add a variable- or adaptive-length comparison (and provide the image/scene splits for both training stages)?
6 comments

We thank the detailed reviews, which will significantly improve our paper. We respond to these reviews point by point. Moreover, we have uploaded the revised paper for your reference.

W1 & Q1: The "eliminating range and precision limits" claim is overstated and OOD results

DVD achieved a data-driven learnable quantization rather than an elimination of quantization. We have removed the relevant statements regarding "eliminating limitations on range and precision".

We would like to clarify that the trade-off primarily exists in the 3D domain. For 2D Grounding and RES, DVD does not involve "trading accuracy for efficiency." Instead, it achieves both a reduction in token count and an improvement in accuracy.

RepresentationTokensLatency3D AP3DTokenslatency2D P@50TokensLatencyRES cIoU
Text745493 ms38.015729 ms84.551225078 ms49.2
Special Token15925 ms25.14265 ms82.01286213 ms51.9
DVD (ours)8620 ms33.72146 ms86.3641672 ms62.0

We also added OOD experiments for reconstruction: we retrained an autoencoder without using any outdoor data, and then tested it on the KITTI dataset.

DataSUN-RGBDKITTI
All98.999.6
w/o Outdoor97.963.9(OOD)

Distance breakdown: Near (<10m), Mid (10–35m), Far (>35m):

SettingKITTI N/M/F
All99.7 / 98.0 / 83.6
w/o Outdoor89.5 / 56.9 / 26.8

Without outdoor training data, the codebook still reconstructs completely unseen large-scale outdoor scenes at 63.9 AP3D@15. These results indicate that DVD retains a certain degree of transferability to previously unseen outdoor scenes. We have included these resutls in the Table 11.

W2 & Q2: The decisive comparison does not isolate the shared high-dimensional codebook

We implemented an adaptive quantization method that employs per-parameter calibration. Specifically, we select different quantization spaces for each parameter based on priors from the training set. We perform quantization on the GT coordinates, which improves 3D AP3D@15 (quantization-only) from 89.78 to 96.18. We report the prediction results using the same representation. AP3D@15 is only 25.1 and still lower than DVD’s 33.7.

TaskcalibratedSpecial tokenDVD
AP3D@1525.123.533.7
2D P@5082.081.986.3
RES cIoU51.946.762.0

We would like to clarify that the main table does not exclusively compare against general-purpose models: the 3D grounding includes VST-3B; 2D grounding includes Rex-Omni; and the RES includes Text4Seg. Furthermore, these methods differ in backbones and training data, the main table illustrates the performance achievable within a unified DVD framework. A fair comparison between different representations is instead provided through controlled experiments text and special token representations.

The numbers of the generalist MLLMs (e.g., Qwen3-VL) are quoted from their official technical reports, where their training data is undisclosed.

W3: The fixed-length representation imposes a hard ceiling on mask quality

We have revised the generalized claim of "negligible reconstruction error".

We would like to clarify that comparing DVD-2B with larger models (e.g., LISA-7B) is not equitable. And our DVD-8B achieves clear performance improvements compared with these methods. Moreover, the comparisons in the main table illustrates achievable performance. A fair comparison is provided through the controlled experiments.

Regarding the complex masks, We reported the reconstruction results separately for each mask structure, based on their respective shapes.

MaskcIoU
All84.8
multiple components84.3
single component84.9
w/ holes83.0
w/o holes85.1
concave structure82.9
convex structure89.2

The results indicate that the bottleneck for fixed-length representations primarily lies in complex contour shapes. In contrast, the impact of multiply connected regions is minimal. We have included these resutls in the Table 16.

Q3: mask reconstruction ceiling and variable- or adaptive-length comparison

With the default settings of 64 points and K=2, the reconstruction upper bound is 84.8 cIoU.

First, we clarify that variable-length techniques are orthogonal to our method. Moreover, we implemented the adaptive encoding: each mask is encoded using by adaptively selecting 64, 128 or 256 points with the best reconstruction performance.

Methodaverage token/maskcIoU
fixed 64 points64.0 points84.8
adaptive94.7 points90.1

This demonstrates that an adaptive strategy based on complexity can achieve superior encoding performance within the same framework. However, the design of an adaptive masking method and the DVD method are two orthogonal methods.

Dear Reviewer AMXN,

Thank you for recognizing our "The unified representation has clear engineering value." We hope our rebuttal has addressed your concerns with new evidence and we have revised our paper to include more analysis and experiments.

As the discussion period ends soon, we would greatly appreciate knowing whether these new results address your concerns. We remain happy to provide further experiments or details. Thank you.

Reviewer_AMXNOpenReview ↗

Response

The additional experiments address some of my previous concerns, but several issues remain insufficiently resolved.

  1. The calibrated quantization baseline does not fully isolate the contribution of the shared high-dimensional codebook. The observed improvements may also arise from learned vector quantization and joint coordinate modeling, making it difficult to identify the source of the performance gains.
  2. The OOD results reveal a substantial degradation, particularly at long distances (83.6 to 26.8 AP3D@15). While these experiments are useful, they also highlight the limited generalization of the learned quantization scheme beyond the training distribution. This limitation should be discussed more explicitly.
Reviewer_6ogYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Dynamic Vector Decoding (DVD), a unified tokenization framework for efficient MLLM-based 2D and 3D perception. The method converts bounding boxes and segmentation masks into one-dimensional vector sequences, which are compressed into discrete tokens using a shared VQ-based codebook and reconstructed through a lightweight decoder. A task-agnostic geometric loss is further introduced to improve reconstruction fidelity. Experiments across multiple grounding and segmentation benchmarks demonstrate competitive perception performance with substantial reductions in output token counts and inference latency.

Critical strengths
  1. Practical and effective representation compression. The paper addresses an important inefficiency in MLLM-based perception, where autoregressive generation of numerical coordinates or dense masks can introduce substantial inference overhead. DVD significantly reduces token counts across 2D grounding, 3D grounding, and referring segmentation, achieving notable latency improvements. This constitutes a meaningful practical contribution.

  2. Broad empirical validation across heterogeneous perception tasks. The proposed framework is evaluated on multiple indoor and outdoor 3D grounding datasets, as well as standard 2D grounding and segmentation benchmarks. The experiments demonstrate that a single model can handle these heterogeneous outputs while achieving competitive accuracy. The controlled comparisons between text, special-token, and DVD representations further support its favorable accuracy–efficiency trade-off.

Critical weaknesses
  1. Insufficient justification of the claimed high precision and unrestricted spatial range. The paper repeatedly emphasizes that DVD overcomes the range and precision limitations of conventional quantization. However, its representation still relies on a finite discrete codebook, and therefore cannot fundamentally eliminate quantization constraints. No systematic evaluation is provided on out-of-distribution coordinate ranges, extreme object scales, or stringent localization thresholds. Moreover, the controlled comparison in Table 8 shows that DVD underperforms text-based coordinates on 3D grounding (34.1 vs. 37.5 AP3D), indicating a remaining accuracy–efficiency trade-off. These observations weaken one of the paper's central claims.

  2. Limited methodological novelty and insufficient isolation of unified codebook benefits. The proposed representation primarily combines established VQ-based tokenization, one-dimensional sequence encoding, and Chamfer-distance supervision. Although multi-task autoencoder training improves reconstruction quality, the experiments do not demonstrate that a unified codebook is superior to task-specific codebooks under matched capacity and token budgets. Furthermore, the geometric-loss ablations primarily assess codec reconstruction rather than downstream MLLM performance, making it difficult to determine how much of the final accuracy improvement comes from the proposed representation design.

Questions for authors
  1. Given that DVD uses a finite codebook, in what sense does it overcome the range and precision constraints of quantization? Could the authors provide evaluations on out-of-distribution coordinate ranges, different object scales, and stricter 3D localization thresholds, together with direct geometric reconstruction errors?

  2. Can the authors compare the proposed unified codebook against separate task-specific codebooks under matched parameter counts, codebook capacity, and output token budgets? This would help establish whether shared discrete representations are necessary for the reported improvements.

  3. Could the authors compare DVD with more competitive coordinate quantization and learned compression alternatives, rather than relying primarily on uniform fixed-range quantization? In particular, would the accuracy–efficiency advantage persist under adaptive or scale-aware coordinate encoding?

  4. Can the authors report downstream MLLM performance with and without the geometric loss, rather than only autoencoder reconstruction results? Additionally, how does the fixed-length boundary representation perform on complex masks containing multiple components, holes, or fine structures?

7 comments

We thank the detailed reviews, which will significantly improve our paper. We respond to these reviews point by point. Moreover, we have uploaded the revised paper for your reference.

W1 & Q1: Insufficient justification of the claimed high precision and unrestricted spatial range. OOD evaluation.

We thank the reviewer for pointing out this issue. We have revised the relevant claim to "data-driven adaptive quantization" and "mitigating the bottlenecks of fixed-range scalar quantization," while removing phrases such as "unlimited range" and "eliminated precision limitations."

We condcut OOD experiments for codec reconstruction: we retrained an autoencoder without using any outdoor data, and then tested it on the KITTI dataset. This creates a distribution shift. We have included these resutls in the Table 11.

DataSUN-RGBDKITTI
All98.999.6
w/o Outdoor97.963.9(OOD)

Distance breakdown: Near (<10m), Mid (10–35m), Far (>35m):

SettingKITTI N/M/F
All99.7 / 98.0 / 83.6
w/o Outdoor89.5 / 56.9 / 26.8

Without outdoor training data, the codebook still reconstructs completely unseen large-scale outdoor scenes at 63.9 AP3D@15. These results indicate that DVD retains a certain degree of transferability to previously unseen outdoor scenes.

We would like to clarify that the trade-off primarily exists in the 3D domain. For 2D Grounding and RES, DVD does not involve "trading accuracy for efficiency." Instead, it achieves both a reduction in token count and an improvement in accuracy.

RepresentationTokensLatency3D AP3DTokenslatency2D P@50TokensLatencyRES cIoU
Text745493 ms38.015729 ms84.551225078 ms49.2
Special Token15925 ms25.14265 ms82.01286213 ms51.9
DVD (ours)8620 ms33.72146 ms86.3641672 ms62.0

W2 & Q2 & Q4: The novelty of the method is limited, and the gains from the unified codebook and geometric loss have not been isolated.

We acknowledge that VQ, 1D sequence encoding, and Chamfer supervision are not novel components. However, DVD is not merely another general-purpose VQ algorithm and it serves as a unified discrete representation and interfacing framework designed for the geometric outputs of MLLMs. Its value lies in transforming heterogeneous geometric annotations into a token-based interface.

To isolate the effectiveness of shared codebook, we conduct a controlled experiment with equivalent training settings. The codebook was partitioned into three disjoint subsets and the model was trained jointly across all tasks.

Taskpartitioned codebook(~1365 per task)shared codebook(4096)
AP3D@1589.390.8
2D P@50100.0100.0
RES cIoU77.084.8

Independent codebook makes the codebook consistently worse on all three tasks compared to shared codebook. We have included these resutls in the Table 8.

We have added a comparison of downstream MLLM performance with and without the geometric loss. Both MLLM training setups are identical, differing only in the codec used.

Taskw/o geow/ geo
AP3D@1531.833.7
2D P@5085.686.3
RES cIoU57.162.0

Removing geometric loss leads to consistent performance declines across all three downstream tasks, demonstrating that improvements in reconstruction quality effectively translate to better downstream predictions. We have included these results in the Table 6.

Q3: Compared with stronger coordinate quantization methods

We first clarify the implementation of our original special tokens: 3D centers are scaled by 200m and sizes by 50m, then clipped to the range [-1,1], linearly mapped to [0,1], and quantized into 4000 bins.

Additionally, we implemented an adaptive quantization method that employs per-parameter calibration. Specifically, we select different quantization spaces for each parameter based on priors from the training set.

TaskcalibratedSpecial tokenDVD
AP3D@1525.123.533.7
2D P@5082.081.986.3
RES cIoU51.946.762.0

Employing stronger quantization methods fails to significantly narrow the gap with DVD (25.1 vs. 33.7).

Q4: The performance of fixed-length boundary representation on complex masks

Regarding the complex masks, We reported the reconstruction results separately for each mask structure, based on their respective shapes.

MaskcIoU
All84.8
multiple components84.3
single component84.9
w/ holes83.0
w/o holes85.1
concave structure82.9
convex structure89.2

The results indicate that the bottleneck for fixed-length representations primarily lies in complex contour shapes. In contrast, the impact of multiply connected regions is minimal. We have included these results in the Table 16.

Dear Reviewer 6ogY,

Thank you for recognizing our "Practical and effective representation compression." and "Broad empirical validation across heterogeneous perception tasks." We hope our rebuttal has addressed your concerns with new evidence and we have revised our paper to include more analysis and experiments.

As the discussion period ends soon, we would greatly appreciate knowing whether these new results address your concerns. We remain happy to provide further experiments or details. Thank you.

Thank you for the detailed response and the additional experiments. I appreciate the authors' efforts to address my concerns and improve the manuscript. However, I believe some of my original concerns remain only partially addressed.

  1. Does the advantage of the shared codebook persist when compared with task-specific codebooks that have matched per-task capacity, rather than only matched total capacity? The current partitioned setup reduces each task's accessible codebook from 4096 to approximately 1365 entries.

  2. Can the authors clarify whether the benefits of shared codebook learning also translate to downstream MLLM performance, and further discuss the substantial degradation in OOD long-range reconstruction?

  3. Could the authors clarify why the Special Token baseline results differ between the W1 and Q3 response tables?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.