acceptodds
Under review as a conference paper at ICLR 2027

Towards Efficient and Effective Scene Visual Representation for 3D Spatial Scene Understanding

Abstract

Despite recent advances in multimodal large language models (MLLMs), 3D spatial reasoning remains challenging because spatial evidence is often redundant, occluded, or fragmented across densely sampling views. Existing video-based 3D MLLMs typically adapt video MLLMs to 3D tasks while retaining dense visual tokens, resulting in heavy inference costs. Recent post-hoc token compression methods can alleviate these costs but inevitably compromise performance. In this paper, we propose a unified framework that integrates aggressive visual token compression and multi-teacher distillation into a single stage of 3D task fine-tuning, dubbed SAVR. Specifically, we compress visual tokens to a prescribed token budget before they enter the LLM, enabling the model to adapt directly to compact visual tokens. To preserve informative visual cues under aggressive compression, we distill complementary priors from diverse vision foundation models into visual token representations at the final LLM layer. A learnable adaptive weighting mechanism further modulates the contributions of individual teachers to mitigate potentially noisy supervision and facilitate selective knowledge transfer. By coupling token compression with task adaptation and complementary expert distillation, our method leads to compact, task-relevant visual representations without requiring a separate post-hoc compression stage and maintains relatively vanilla performance. Extensive experiments across multiple 3D vision and spatial reasoning benchmarks demonstrate SoTA token efficiency with negligible performance degradation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Representation, 3D Scene Understanding, Distillation, Spatial Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_e4GqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies object-centric 3D multimodal large language models by decomposing their visual interface into three components: object proposals, the underlying 3D representation, and object-token construction. By holding the proposal set fixed, the authors compare dense scene representations, independently encoded object representations, and detector-query representations. They also compare several object-token aggregation mechanisms, including mean/max pooling, MLP-based aggregation, single-query attention, and Perceiver IO. The main empirical conclusion is that the choice of pretrained 3D representation has a substantially larger effect than the complexity of the object-token construction mechanism. Strong frozen point representations combined with simple mean pooling perform competitively across five ScanNet-based benchmarks. The strongest configuration achieves the best reported result on eight of the nine comparable metrics in Table 1.

Critical strengths
  1. The object-token-based, object-centric representation formulation presented in this paper leads to substantial performance improvements in 3D understanding. By separating the object proposals from the representation supplied to the language model, the proposed formulation enables these two factors to be studied independently. Its effectiveness is further demonstrated across multiple downstream tasks.

  2. The paper is well written, clear, and easy to follow.

  3. The ablation studies are well designed. In particular, the object-token construction function gθg_\theta is evaluated through comprehensive ablations.

Critical weaknesses
  1. The core formulation consists of three components: the object proposals MM, the 3D representation encoder ϕ\phi, and the object-token construction function gθg_\theta. However, both the object proposal generators and the 3D representation encoders are adopted from existing open-source works, while gθg_\theta is investigated using several relatively standard aggregation mechanisms. Consequently, the algorithmic novelty of the paper is limited, and the work does not introduce a novel theory or learning principle for representation learning.
Questions for authors
  1. Can the authors provide an experiment using a representation model whose pretraining data excludes ScanNet, or evaluate on a genuinely different scene distribution?

  2. What are the proposal recall and oracle grounding performance at IoU thresholds of 0.25 and 0.5?

  3. Are the small differences among object-token construction methods stable across multiple random seeds?

  4. How much of the improvement from the Vicuna-7B configuration to the Qwen3.5-9B configuration is attributable to language-model capacity rather than the 3D representation?

Reviewer_MzoTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SAVR is a framework for learning compact visual representations in video-based 3D MLLMs. It incorporates visual token compression directly into 3D task fine-tuning, using parameter-free adaptive average pooling to reduce each frame’s features to a specified token budget. Training and inference therefore use the same compact representation, avoiding the mismatch introduced by post-hoc token pruning. To help preserve useful information under aggressive compression, SAVR draws on three frozen teachers during training: VGGT supplies geometric cues, C-RADIO provides dense semantic features, and Wan2.1 contributes spatiotemporal priors. The teacher features are pooled to match the student’s spatial token grid and used to supervise the hidden states corresponding to visual tokens in the final LLM layer, with separate projection layers aligning the student representations to each teacher’s feature space. Learnable uncertainty parameters adjust the contribution of each teacher’s distillation loss to balance these different sources of supervision. At inference, the teachers and all distillation-specific components are removed, leaving the student model to operate on compressed visual tokens. Experiments across 3D scene understanding and spatial reasoning benchmarks show that SAVR substantially reduces visual token counts while largely preserving the performance of the uncompressed baseline.

Critical strengths
  1. The paper tackles a practical challenge in video-based 3D MLLMs: processing multi-view scene inputs produces large numbers of visual tokens and incurs substantial computational cost. Incorporating token compression into task-specific fine-tuning lets the model learn from compact visual representations, helping reduce the mismatch between training and inference.
  2. The approach is straightforward and well suited to efficient inference. Visual tokens are compressed through adaptive average pooling, while the teachers, distillation projectors, and learnable weighting parameters are required only during training. The resulting model can therefore benefit from the knowledge of several vision foundation models without running them at inference.
  3. The reported results show a favorable trade-off between performance and computation, even at low token budgets. SAVR substantially reduces the number of visual tokens and the reported LLM computational cost while retaining performance close to the uncompressed baseline overall, with improvements on some individual metrics.
  4. The evaluation uses two base models and spans several tasks, including 3D visual grounding, 3D dense captioning, question answering, and spatial reasoning. The ablation studies examine the effects of pooling-based compression, teacher supervision, and learnable loss weighting, offering initial support for the contribution of each component.
Critical weaknesses
  1. The methodological contribution appears more incremental than the paper’s framing suggests, particularly statements such as “we first introduce a unified framework.” Each component has a close precedent: average-pooling-based compression in DeCo [1], training on compressed inputs with distillation in EPIC [2], VGGT-based supervision of visual hidden states in 3D MLLMs in 3DRS, and uncertainty-based loss weighting in Kendall et al. (2018). A more precise discussion of these connections would help clarify what new insights emerge from combining these components in this setting, beyond demonstrating the effectiveness of the overall system.
  2. The ablations show that additional supervision improves performance, but offer less convincing evidence for the proposed explanations of teacher complementarity and conflict mitigation. Adding teachers sequentially does not distinguish gains from complementary knowledge from those attributable to teacher quality or changes in the overall strength of supervision. Similarly, LAW’s improvement of approximately 0.43 percentage points does not, on its own, establish that the mechanism detects unreliable or conflicting supervision, especially without estimates of variability or a comparison against carefully tuned fixed weights.
  3. The claim of negligible performance degradation should be qualified. The reported relative score of 99.63% averages across nine metrics and can obscure declines in individual capabilities. For example, CIDEr scores decrease on both Scan2Cap and ScanQA, while the Room Size score on VSI-Bench drops from 62.08 to 54.09. Although the limitations section acknowledges potential information loss, attributing these declines to small objects and fine-grained attributes remains a plausible interpretation rather than a directly tested explanation.
  4. The efficiency evaluation provides stronger evidence for reductions in token counts and LLM computation than for improvements in end-to-end efficiency. Visual encoding and output generation also contribute to runtime, so the reported computational savings do not necessarily translate into proportional speedups. In addition, extracting features from multiple teachers offline introduces preparation and storage costs that should be accounted for when discussing overall training efficiency.
  5. Several implementation details need clarification to support reproducibility. Appendix A.4 describes pooling as occurring after the multimodal projector, whereas Algorithm 1 appears to place it before the projector. Appendix B.1 does not sufficiently specify the number of training samples or how the subsets were selected, and it contains an unfinished reporting instruction. The Scan2Cap metric headers in Table 3 and inconsistencies in notation also need correction. [1] Yao, Linli, et al. “DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models.” arXiv preprint arXiv:2405.20985, 2024. [2] Wen, Zichen, et al. “Efficient Multi-modal Large Language Models via Progressive Consistency Distillation.” Advances in Neural Information Processing Systems, vol. 38, 2025.
Questions for authors
  1. Could you clarify the main methodological differences between SAVR and DeCo, EPIC, and 3DRS? In particular, what does 3D task adaptation under a constrained token budget contribute, and what further benefits come from multiple teachers and LAW? Given that Table 3 already evaluates several related component configurations, please explain how these comparisons support the claimed novelty of the full approach.
  2. What evidence supports the claims that the teachers provide complementary information and that LAW mitigates conflicting supervision? Additional comparisons using individual teachers or removing one teacher at a time, while controlling the overall strength of supervision, would help assess complementarity. For LAW, could you include a comparison against reasonably tuned fixed weights and report the variability of the observed gain?
  3. Could you examine where aggressive compression leads to the largest performance losses, particularly in generation metrics and Room Size estimation? Representative failure cases or a focused analysis by object size or attribute type would help evaluate the proposed explanation. If such analyses are not feasible, could you qualify the causal interpretation and the claim of negligible degradation to better reflect the available evidence?
  4. Could you report end-to-end latency, peak memory usage, and throughput under matched hardware, input, and generation settings? Please also provide the time and storage costs of teacher-feature extraction, so readers can assess the practical trade-off between additional training preparation and reduced inference computation.
  5. Could you clarify whether pooling is applied before or after the multimodal projector and explain how 3D positional encodings are handled after compression? Please also specify the per-task training sample counts and the size and selection procedure of the spatial-reasoning training subset, and correct the metric headers and inconsistent notation. These details would make the reported results easier to reproduce.
5 comments
AuthorsOpenReview ↗

Response to Reviewer MzoT (Part I)

Thank you for the detailed and constructive comments. We have addressed all your questions below with additional experiments to provide further explanation and clarification.


[W1, Q1]: How does SAVR differ from previous works, such as DeCo, EPIC and 3DRS?

DeCo [1] uses parameter-free 2D adaptive average pooling as the projector of a generic image MLLM (CLIP ViT-L/14 with Vicuna-7B) to reduce the number of visual tokens. It conducts evaluations only on 2D image-text benchmarks. We also utilize adaptive average pooling, but the differences lie in the setting and in how aggressive the token compression can go:

  • First, DeCo compresses the tokens of a single image for 2D image-text tasks. SAVR compresses the tokens of multi-view frames for 3D spatial tasks, including 3D visual grounding, 3D dense captioning, 3D question answering, and spatial reasoning tasks.
  • Secondly, DeCo degrades clearly under aggressive compression. In its compression study (DeCo, Figure 6), visual-localization [email protected] degrades from about 51% with all 576 tokens to about 40% with 144 tokens and about 32% with 64 tokens. At 11% of the tokens, it only retains roughly 62% of its vanilla accuracy. DeCo raises the limitation that average pooling causes severe visual information deficiency at increasingly high compression ratios. However, our SAVR is able to maintain relatively vanilla model performance and induce no extra inference overhead even under an aggressive token compression ratio, without any post-hoc token selection-then-compression method when compared with existing token compression methods.

EPIC [2] is a self-distillation method that trains a LLaVA-1.5 to adapt token pruning inside the LLM (FastV, DART, or random pruning) and evaluates the model on 2D image-text benchmarks. EPIC uses one shared MLLM that acts as both teacher and student, while the student undergoes more aggressive and stronger visual token compression and the teacher operates under moderate compression, forcing the student to match the teacher's output distribution during training. Due to this self-distillation mechanism, EPIC can not activate the informative cues if the teacher is not able to highlight or student easily loses that under the aggressive visual token compression. Moreover, the target is the output of the same MLLM that the teacher cannot supply visual information that the MLLM itself has lost or can not encode. Our proposed SAVR instead builds on the dense, per-patch representations of vision foundation models, e.g., VGGT, C-RADIO and Wan2.1. The teachers are frozen expert models that differ from the MLLM and from each other: VGGT (3D geometry), C-RADIOv3 (semantics) and Wan2.1 (video generation). They encode heterogeneous and complementary visual or context information, which is then aligned with their features and pooled over the same region to be distilled to the student for the 3D task. At 64 token retention (11.1%), the distillation term of EPIC raises the average score from 58.1 / 58.7 to 59.4 (EPIC Tables 3–4, TCD / LCD), while the full-token model scores 61.4, leading to 26–39% of the remaining gap. However, our SAVR improves the averaged 3D tasks performance up to 99.63%, relatively close to the vanilla model. Moreover, EPIC employs token compression within LLM layers before the pruning layer, which incurs substantial computation overhead, while our SAVR does not.

3DRS shows that a feed-forward 3D geometry model (VGGT or FLARE) can improve a 3D MLLM performances. 3DRS explores only whether 3D supervision improves a full-token 3D model while leaving several questions that SAVR studies. Firstly, 3DRS processes all visual tokens of its dense sampled video frames and does not penalize visual token efficiency. Secondly, 3DRS compares 3D teachers (e.g., VGGT, FLARE) with 2D teachers (DINOv2, MAE, SigLIP), and concludes that 3D geometry teachers are best. But we explore several representative vision foundation models that contain heterogeneous and complementary information, e.g., VGGT provides geometry-aware and cross-view information, C-RADIO contributes general dense semantic cues, and Wan2.1 supplies spatio-temporal and generative geometry priors. Our SAVR instead penalizes visual token inefficiency under the dense multi-view input and investigates the redundancy and compression capacity in terms of visual token space.

AuthorsOpenReview ↗

Response to Reviewer MzoT (Part II)

Kendall et al. (2018). LAW is motivated by the heterogeneous yet complementary teachers that we use. VGGT, C-RADIOv3, and Wan2.1 are trained for different goals that differ strongly in which visual regions they highlight and where they raise noise (Figure 2): 1) VGGT gives a smooth and low-frequency map that follows the scene layout, which is conducive to scene reconstruction while diverging the noise across large object areas such as the floor and curtain, 2) C-RADIOv3 responds to object instances due to the vision-language pretraining while showing noise in blocky, patch-scale texture on the background. 3) Wan2.1 generates a high-frequency map that follows edges and thin structures, which is beneficial for scene generation, while noise is a band along the image border and speckle on the floor texture.

Therefore, these three alignment targets differ in how hard and how noisy they are to fit, while a fixed weighting scheme is difficult to adaptively resemble multi-teacher vision foundation model knowledge, while our LAW help to harmonize this heterogeneous knowledge into a 3D model under aggressive token compression while maintaining relatively vanilla model performance without inducing any inference overhead.


[W2, Q2]: More ablations in terms of complementary knowledge from multi-teachers and effectiveness of LAW.

Complementary information from teachers and effectiveness of LAW. We further conduct a qualitative experiment in Figure. 5 while also plotting the dynamic loss weights during training in Figure. 4 in our revised paper. From the Figure. 5, we find that:

  • VGGT gives a smooth and low-frequency map that follows the scene layout, which is conducive to scene reconstruction and grounding tasks while diverging the noise across large object areas such as the floor and curtain.
  • C-RADIOv3 responds to object instances due to the vision-language pretraining, which is good at semantic understanding while showing noise in blocky, patch-scale texture in the background.
  • Wan2.1 generates a high-frequency map that follows edges and thin structures, which is beneficial for scene generation and provides spatio-temporal and generative geometry priors, while noise is a band along the image border and speckle on the floor texture.

This explains why the selected multi-teacher vision foundation models present heterogeneous yet complementary supervision signals for the 3D model. From the Figure. 4, the proposed LAW indeed enforce the model adaptively learn to abstract the knowledge distillation from various teacher models as the training goes that helps to mitigate the potential noise or conflicts.

More ablations for complementarity and Fixed-weights Experiments. All rows use 8.2% tokens, 32 uniformly sampled frames, and identical data and fine-tuning on Video-3D LLM.

SetupRelative Score
Pooling + fine-tuning (baseline)96.39
+ VGGT only98.48
+ VGGT + Wan2.198.90
+ VGGT + Wan2.1 + C-RADIO99.21
+ VGGT + C-RADIO99.03
+ Wan2.1 + C-RADIO98.67
+ all three, fixed VGGT:Wan2.1:C-RADIO=1:1:199.21
+ all three, fixed VGGT:Wan2.1:C-RADIO=0.5:1:198.39
+ all three, fixed VGGT:Wan2.1:C-RADIO=1:0.5:199.00
+ all three, fixed VGGT:Wan2.1:C-RADIO=1:1:0.598.87
+ all three, fixed VGGT:Wan2.1:C-RADIO=0.5:0.5:0.599.10
+ all three, fixed VGGT:Wan2.1:C-RADIO=0.1:0.1:0.198.15
+ all three, LAW (SAVR)99.63

To establish the effectiveness of our proposed LAW, we further conduct ablative studies above. From the experiments, we can see that removing one teacher at a time while controlling the overall strength of supervision presents consistent improvements over the baseline. And VGGT boost the overall performances mostly while the other two teachers combination presents a bit inferior when removing VGGT, which we analysis the overall training signal can only rely on Wan2.1's generative geometry priors, weaker than geometry model (VGGT). Moreover, when lowering the weighting of distillation signals greatly, the 3D model performance degrades significantly, from 99.21% -> 99.10 -> 98.15 corresponding to λ_dist=1.0, λ_dist=0.5 and λ_dist=0.1, respectively. Since the core objective comes from the language next-token prediction part L_task in Eq. 11, we keep the distillation not larger than 1.0. Overall, these additional ablative studies demonstrate that multi-teachers from representative vision foundation models provide heterogeneous yet complementary visual signals, while LAW helps with conflict mitigation.

AuthorsOpenReview ↗

Response to Reviewer MzoT (Part III)

[W3, Q3]: Largest performance losses under aggressive token compression, particularly in generation metrics and Room Size estimation.

1. Generation metrics: testing the small-object / attribute explanation

  • Our SAVE under aggressive token compression at an 8.2% token budget obtains retention from 101.6% to 102.9% on the four grounding metrics, which presents improvements in grounding ability, while leading to 100.4% on Scan2Cap B-4 and 99.6% on SQA3D, compared with the vanilla model.
  • It is lower on the generation metrics: 96.2% on Scan2Cap CIDEr, and 95.9% / 95.7% on ScanQA CIDEr / EM.

Aggressive visual token compression mainly destroys fine-grained object discrimination, i.e., identifying small objects among similar or larger neighbours, and their color. CIDEr in Scan2Cap or ScanQA particularly penalizes these errors, such as the question examples Q:"What is to the right of the water bottle on the counter?" with GT:"paper towel roll; cylindrical white paper towel roll" and wrong prediction:"coffee maker / coffee maker", and Q:"What color is the laptop computer on the left edge of a messy desk?" with GT:"trash can; red backpack" and wrong prediction:"window / window". Under extreme token compression, SAVR inevitably degrades by 3.0 points on Scan2Cap CIDEr, whereas FastV and VisionTrim result in a more serious degradation of 29.6–29.8 points (shown in Table 1). This demonstrates that our proposed SAVR significantly alleviates these failures.

2. Room Size (VSI-Bench, VG LLM base)

From our experimental results, we find that this degradation comes with aggressive visual token compression as well. At a 10% token budget, VisionZip and FastVID retain only 75.4% / 73.7% of Room Size, while our SAVR still maintains 87.24% of the vanilla in terms of Room Size. We analyze that the extreme few visual tokens lose scene-scale extent cues severely, such as wall and floor boundaries that are often homogeneous and spread over many frames, but the model is not able to obtain complete visual cues for the whole scene under then aggressive visual token compression.


[W4, Q4]: End-to-end latency and the cost of teacher extraction and per-budget training.

(a) Latency Evaluation Environment Setup:

  • ScanQA val, one GH20 GPU, batch size 1, 32 uniformly sampled frames.
  • Every setting generates exactly 32 tokens, so decoding length is identical.
  • 50 warm-up runs are excluded. We report the mean over 100 timed samples, timed with CUDA events.
  • Only the visual-token budget differs between settings.
SettingVisual tokensVision (ms)Compression (ms)LLM prefill (ms)TTFT (ms)E2E, 32 tokens (ms)Peak mem (GiB)LLM TFLOPsTotal TFLOPs
Vanilla (100%)6272200.66.4244.3451.41294.618.7595.66116.99
SAVR (51.0%)3200200.76.1128.4335.2 (1.3×)1191.0 (1.1×)18.7549.18 (1.95×)70.51 (1.66×)
SAVR (25.0%)1568200.75.670.7277.0 (1.6×)1115.5 (1.2×)18.7524.49 (3.91×)45.82 (2.55×)
SAVR (8.2%)512200.75.538.9245.1 (1.8×)1102.2 (1.2×)18.758.5 (11.25×)29.83 (3.92×)

LLM TFLOPs (prefill) follows the same protocol as Table 1. Total TFLOPs = LLM TFLOPs + vision encoder and projector (SigLIP on 32 frames, 21.33 TFLOPs, identical for all settings).

Performing our proposed SAVR under aggressive token compression at 8.2% token budget can speed up the overall end-to-end wall-clock latency by up to 1.2 compared with the vanilla model, while significantly boosting LLM prefill 6.3× faster (244.3 → 38.9 ms), Time-To-First-Token (TTFT) up to 1.8, and LLM TFLOPs up to 11.25, respectively. Note that our method can achieve much more speedup on short-answer scenarios thanks to efficient TTFT and enable more efficient throughtout due to much less token sequence within heavy LLM. And our method can be deployed without any extra online token selection-then-compression strategy or expert models (e.g., heavy geometry encoder) compared with other token compression methods, leading to simple, efficient, and effective inference.

We have added these latency comparisons into our revision in Sec. 4.5 Efficiency Analysis.

Reviewer_crLKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SAVR uses fixed-grid average pooling per frame and budget-specific fine-tuning with training-only distillation from VGGT, Wan2.1, and C-RADIO. For 32-frame inputs, it reports 8.2% visual-token retention, 8.5 versus 95.66 LLM-inference TFLOPs, and a 99.63% Relative Score averaged over nine metric ratios to Vanilla.

Critical strengths
  1. The strong average performance under severe compression makes the approach practically interesting.
  2. A pooling-only fine-tuning control and teacher ablations help attribute gains; teachers add no inference cost.
Critical weaknesses
  1. Most main-table competitors are training-free, whereas SAVR receives budget-specific fine-tuning and distillation. The comparison does not isolate pooling against other selectors under matched training resources.
  2. LLM-only FLOPs do not establish end-to-end latency. GPU-hours and storage for offline teacher extraction and per-budget training are not reported; the aggregate score also masks declines on Scan2Cap and ScanQA generation metrics.
Questions for authors
  1. How does a content-aware selector perform under the same data, budget, fine-tuning, and distillation?
  2. What are the measured end-to-end latency and the GPU-hours/storage for teacher extraction and per-budget training?
4 comments
AuthorsOpenReview ↗

Response to Reviewer crLK (Part I)

Thank you for the detailed and constructive comments. We have addressed all your questions below with additional experiments to provide further explanation and clarification.


[W1, Q1]: How does a content-aware selector perform under the same data, budget, fine-tuning, and distillation?

We further conduct an experiment under the matched training resources by replacing our SAVR's content-agnostic spatial pooling with VisPruner (ICCV'25), a content-aware selector that keeps the visual tokens that receive the most attention in the vision encoder, plus a set of diverse tokens, while keeping others identical to our SAVR. At the same 8.2% visual-token budget, our SAVR achieves a Relative Score of 99.63%, compared with 96.22% for VisPruner with the same fine-tuning and distillation protocol. SAVR scores higher on all nine metrics in this comparison.

  • the same training data, initialization, and schedule.
  • the same 8.2% budget under 32 uniformly sampled views.
  • the same multi-teacher distillation with the same adaptive loss weighting.

For distillation, the teacher features are gathered with the same indices as the selected tokens following VisPruner, so each student token is aligned with the teacher feature at its own matched position.

MethodTokensScanRefer Acc @0.25 / 0.5Multi3DRef F1 @0.25 / 0.5Scan2Cap B-4 / CScanQA C / EMSQA3D EMRel.
Vanilla100%58.00 / 51.4057.17 / 51.9539.63 / 80.41103.23 / 29.7559.81100%
VisPruner, training-free (Table 1)10.0%55.24 / 48.7352.06 / 46.5336.73 / 62.4490.67 / 26.5055.8490.14%
VisPruner + same fine-tuning & distillation8.2%57.41 / 50.2056.33 / 51.1238.02 / 72.1697.39 / 28.2058.3596.22%
Pooling only, no other components (Table 3 baseline)8.2%57.01 / 50.6056.25 / 50.5638.10 / 73.1699.67 / 28.0658.0596.39%
SAVR8.2%59.59 / 52.8758.11 / 52.7839.80 / 77.3798.97 / 28.4759.5999.63%

Rel. is the visual token retention relative to Vanilla, averaged over the nine metrics, as in Table 1 in our manuscript.

From our experiments above. we find:

  1. Under the matched training resources, the VisPruner-based method improves from 90.14% in a training-free manner to 96.22% (+6.08 points). Employing token compression during training is able to boost performance on the 3D task. However, our proposed SAVR still obviously scores 99.63% against 96.22% for the trained VisPruner with the context-aware token selector. As shown in Table 1 in our manuscript, our SAVR shows an average 9.5-point superiority to the training-free VisPruner.
  2. The Table 3 baseline uses adaptive average pooling only and no other SAVR component. It reaches 96.39%, performing close to the trained VisPruner (96.22%) with multi-teacher distillation training. We analyze that one possible explanation is spatial locality and coverage. Adaptive average pooling combines student and teacher features across each region, whereas the VisPruner-based method uses the features at the retained patch locations only. Moreover, to ensure the multi-teacher distillation matches the student, under our used adaptive average pooling operator, each teacher target is averaged over the same region as its token, and teacher features at every position supervise the student naturally, while only the teacher features at the selected positions are distilled to the corresponding student under context-aware selection. This enables our proposed SAVR with adaptive average pooling to obtain denser and more complete multi-teacher knowledge distillation than content-aware selection.

We greatly thank and appreciate the reviewer's constructive and insightful suggestion to help us consolidate the essential effectiveness of our method in depth. We will add the controlled VisPruner-based comparison into our revision and perform the discussion of this.

AuthorsOpenReview ↗

Response to Reviewer crLK (Part II)

[W2, Q2]: End-to-end latency and the cost of teacher extraction and per-budget training.

To measure the end-to-end latency, the teacher-extraction cost, and the per-budget training cost, we conduct evaluations and report the corresponding results as follows:

(a) Latency Evaluation Environment Setup:

  • ScanQA val, one GH20 GPU, batch size 1, 32 uniformly sampled frames.
  • Every setting generates exactly 32 tokens, so decoding length is identical.
  • 50 warm-up runs are excluded. We report the mean over 100 timed samples, timed with CUDA events.
  • Only the visual-token budget differs between settings.
SettingVisual tokensVision (ms)Compression (ms)LLM prefill (ms)TTFT (ms)E2E, 32 tokens (ms)Peak mem (GiB)LLM TFLOPsTotal TFLOPs
Vanilla (100%)6272200.66.4244.3451.41294.618.7595.66116.99
SAVR (51.0%)3200200.76.1128.4335.2 (1.3×)1191.0 (1.1×)18.7549.18 (1.95×)70.51 (1.66×)
SAVR (25.0%)1568200.75.670.7277.0 (1.6×)1115.5 (1.2×)18.7524.49 (3.91×)45.82 (2.55×)
SAVR (8.2%)512200.75.538.9245.1 (1.8×)1102.2 (1.2×)18.758.5 (11.25×)29.83 (3.92×)

LLM TFLOPs (prefill) follows the same protocol as Table 1. Total TFLOPs = LLM TFLOPs + vision encoder and projector (SigLIP on 32 frames, 21.33 TFLOPs, identical for all settings).

Performing our proposed SAVR under aggressive token compression at 8.2% token budget can speed up the overall end-to-end wall-clock latency by up to 1.2 compared with the vanilla model, while significantly boosting LLM prefill 6.3× faster (244.3 → 38.9 ms), Time-To-First-Token (TTFT) up to 1.8, and LLM TFLOPs up to 11.25, respectively. Note that our method can achieve much more speedup on short-answer scenarios thanks to efficient TTFT. And our method can be deployed without any extra online token selection-then-compression strategy or expert models (e.g., heavy geometry encoder) compared with other token compression methods, leading to simple, efficient, and effective inference.

We have added these latency comparisons into our revision in Sec. 4.5 Efficiency Analysis.

(b) Teacher-feature extraction (one-time and offline).

TeacherExtraction time (single GH20 GPU, hours)Storage
VGGT4.66345 GB
C-RADIOv3-g13.41144 GB
Wan2.1-VACE-1.3B7.46393 GB
Total25.53 GPU-hours882 GB

Teacher features for Video-3D LLM experiments are extracted once for the training set and reused by every budget and every training run offline. The teachers and corresponding projectors can be totally removed at inference, so they induce no inference latency overhead. Our method can also support online multi-teacher distillation, which was conducted for VG LLM experiments since the training sets SPAR-7M and LLaVA-Hound are processed with much sparser sampling video frames and do not result in heavy training overhead.

(c) Per-budget training.

Each budget is one fine-tuning run of one epoch on 8 × GH20 GPUs:

SettingTraining time (h)GPU-hours
Baseline [fine-tuning without distillation]27[27 × 8]
SAVR (8.2%)30[30 × 8]
SAVR (25.0%)31[31 × 8]
SAVR (51.0%)35[35 × 8]
  • Training is cheaper at smaller budgets because fewer visual tokens will be processed by the heavy LLM.
  • At 8.2%, SAVR's training takes 30 hours against 27 hours for the baseline with 8 GH20 GPUs.

Though our SAVR under aggressive token compression with an 8.2% token budget takes an extra 11.11% training time compared with the Vanilla model, our method is able to boost inference significantly and maintain relative vanilla model performance within a unified single stage. This showcases much more superior efficiency and effectiveness to existing token compression methods that need to fine-tune the video LLM first and then perform online complex token selection and compression.

AuthorsOpenReview ↗

Response to Reviewer crLK (Part III)

(d) The aggregate score masks declines on generation metrics.

  • Our SAVE under aggressive token compression at 8.2% token budget obtains retention from 101.6 to 102.9% on the four grounding metrics, which presents improvements for the grounding ability, while leading to 100.4% on Scan2Cap B-4 and 99.6% on SQA3D, compared with the vanilla model.
  • It is lower on the generation metrics: 96.2% on Scan2Cap CIDEr, and 95.9% / 95.7% on ScanQA CIDEr / EM.

Aggressive visual token compression mainly destroys fine-grained object discrimination, i.e., identifying small objects among similar or larger neighbours, and their color. CIDEr in Scan2Cap or ScanQA particularly penalizes these errors, such as the question examples Q: "What is to the right of the water bottle on the counter?" with GT: "paper towel roll; cylindrical white paper towel roll" and wrong prediction: "coffee maker / coffee maker", and Q: "What color is the laptop computer on the left edge of a messy desk?" with GT: "trash can; red backpack" and wrong prediction: "window / window". Under extreme token compression, SAVR degrades by 3.0 points on Scan2Cap CIDEr, whereas FastV and VisionTrim result in a more serious degradation of 29.6–29.8 points (shown in Table 1). This demonstrates that our proposed SAVR significantly alleviates these failures.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer crLK,

As the discussion period draws to a close, we would appreciate it if you could review our rebuttal and share any further feedback you may have. We have provided additional experiments and clarifications in response to your questions.

Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

We greatly and sincerely thank reviewers for their detailed and constructive comments and questions. We have addressed the feedback in our individual responses to Reviewers crLK and MzoT and uploaded a revised manuscript, with all changes highlighted in blue. Table, figure, and section numbers in our responses refer to the revised manuscript unless otherwise noted. We welcome further discussion and would be happy to clarify any points or address any concerns that remain unresolved.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.