acceptodds
Under review as a conference paper at ICLR 2027

GeoGR: Spatial–Semantic Token Compression and Group Routing for Efficient 3D VLMs

Abstract

Multi-view representations enable pretrained image and video vision-language models to understand 3D scenes, but their long visual token sequences make inference costly. Redundancy arises both from repeated observations of the same scene regions across views and from processing all visual tokens at every LLM layer, regardless of their relevance to the current computation. We propose GeoGR, a two-stage framework that jointly compresses scene representations and visual computation. At the projector level, GeoSemZip consolidates cross-view observations into 3D voxel tokens and combines semantic saliency with spatial coverage to construct a compact scene representation. Unselected voxel features are aggregated into contextual anchors to retain scene information that would otherwise be discarded. At the LLM level, we identify an effective visual computation window and observe that query-conditioned token relevance remains locally stable but changes over longer layer intervals. Guided by these findings, GroupRoute combines late entry and early exit with recoverable group-wise skipping. Low-relevance tokens bypass complete LLM layers while retaining their hidden states and can rejoin the active group at subsequent anchor layers selected under a computation budget. For compression-aware post-training, we construct a compact dataset of 67K samples by retaining 30% of the original training pool. With this protocol, GeoGR retains 95.2% and 96.8% of the dense post-trained models' performance on LLaVA-OneVision-7B and Video-3D LLM-7B, respectively, while accelerating LLM prefill by and . These results demonstrate the consistent effectiveness of GeoGR across 3D VLMs with and without explicit 3D spatial encoding, highlighting a favorable performance-efficiency trade-off.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MLLMs, Vision Token Pruning, Token Compression, Operation Compression, Efficiency and Compression, Interpretability and Analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_woYfOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on reducing the computational costs of 3D VLMs from two directions, i.e., token reduction and layer skipping. In detail, it utilizes 3D voxel tokens to replace the multi-view tokens, which can merge the repeated tokens in multiple views and reduce the number of vision tokens. Then, it uses late entry and early exit with recoverable group-wise layer skipping to reduce the token computational costs. The experimental results demonstrate a 6× speedup in inference.

Critical strengths
  • The paper is well-written, with a clear presentation of its experimental findings, motivation, and methodology. The analysis in Sec. 3 is particularly informative.
  • The paper offers sufficient results to prove the efficiency of the proposed method.
Critical weaknesses

The novelty of the proposed method appears limited:

  • Using 3D voxel representations to reduce the number of tokens is common in multi-view tasks, as demonstrated by Scaffold-GS [A] and DTC [B].
  • The GroupRoute is similar to HiDrop.

For the Saliency-aware selection in Line 263, higher attention weights do not necessarily indicate greater token importance. For example, vision foundation models such as DINOv2 can assign high attention scores to task-irrelevant regions.

[A] Lu, Tao, et al. Scaffold-gs: Structured 3d gaussians for view-adaptive rendering. CVPR 2024.

[B] Huang, Hsiang-Wei, et al. Zero-shot 3d question answering via voxel-based dynamic token compression.CVPR 2025.

Questions for authors
  • In the equation in Line 258, different operations are used to aggregate features and saliency scores within each voxel, which may cause a mismatch between the aggregated features and their corresponding saliency scores.
  • There is no experimental results to demonstrate that saliency- and coverage-aware selection outperforms random selection.
7 comments
AuthorsOpenReview ↗

Response to Reviewer woYf (Part 1/4)

We thank you for your thorough review and appreciate both the positive feedback and the constructive comments. Please find our responses to your points below. In the revised manuscript, newly added or substantially modified text is highlighted in blue, while previously existing passages that already address your concerns are highlighted in yellow for ease of reference. Please let us know if you have any further questions or suggestions.

W1: The novelty of the proposed method appears limited: Using 3D voxel representations to reduce the number of tokens is common in multi-view tasks, as demonstrated by Scaffold-GS and DTC. The GroupRoute is similar to HiDrop.

A1: We respectfully disagree with the assessment of limited novelty. As summarized in the Global Response, GeoGR jointly addresses representational redundancy across views and computational redundancy across LLM layers in multi-view 3D VLMs. Its technical contributions include spatial–semantic compression that preserves salient content, spatial coverage, and information from unselected voxels, together with recoverable group-wise skipping motivated by our analysis of evolving token relevance across layers. Below, we clarify these contributions and present the supporting evidence.

  • Spatial–semantic Representation Compression (GeoSemZip):
    • Voxelization is a well-established technique for 3D representation [1-3]. Scaffold-GS [4] uses voxelized anchors to organize view-adaptive 3D Gaussians for rendering, while DTC [5] uses voxelization for cross-view visual token compression. Voxelization itself is not the defining innovation of either method; their distinctive contributions lie in the mechanisms built on this common geometric foundation.

    • The distinction between DTC and GeoSemZip lies in how they compress and preserve scene information using this geometric foundation. DTC progressively enlarges voxels and repeatedly performs similarity-based token matching to remove redundancy. GeoSemZip instead consolidates cross-view observations into voxels and allocates a fixed token budget to high-saliency dominant voxels and contextual anchors sampled in Morton order from the remaining voxels. It then aggregates features from unselected voxels into these anchors through residual merging. This design jointly preserves salient content, spatial coverage, and information from unselected regions, while avoiding iterative voxel resizing and matching.

    • Our original submission already reports comparisons between GeoSemZip and DTC at approximately 10%, 20%, and 30% projector token budgets in Appendix D.2, Table 12. We present the key results below for ease of reference. GeoSemZip achieves higher performance retention at all three budgets.

      Table R1. Performance retention on LLaVA-OneVision-7B across projector token budgets. All configurations are post-trained on LLaVA-OneVision-7B, with retention normalized to Dense (100%).

      Projector token budgetDTC (%)GeoSemZip (%)Gain (pp)
      ≈10%92.6194.00+1.39
      ≈20%94.3897.26+2.88
      ≈30%98.2599.33+1.08
    • Our updated latency measurements further demonstrate the practical efficiency advantage of GeoSemZip. With the same general inference optimizations applied to both methods, GeoGR achieves substantially lower projector-stage latency than DTC on both models.

      Table G1 (excerpt from the Global Response). Projector-stage latency after inference optimizations (ms), including token compression, measured on NVIDIA RTX PRO 5000 Blackwell.

      ModelDTCGeoGR (with GeoSemZip)
      LLaVA-OneVision-7B1068.112.0
      Video-3D LLM-7B998.220.3
    • Together, these results demonstrate the benefits of GeoSemZip in both performance retention and practical efficiency.

[1] Liu Z, Tang H, Lin Y, et al. Point-voxel cnn for efficient 3d deep learning[J]. Advances in neural information processing systems, 2019, 32.

[2] Graham B, Engelcke M, Van Der Maaten L. 3d semantic segmentation with submanifold sparse convolutional networks[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Ieee, 2018: 9224-9232.

[3] Chen Y, Liu J, Zhang X, et al. Voxelnext: Fully sparse voxelnet for 3d object detection and tracking[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023: 21674-21683.

[4] Lu T, Yu M, Xu L, et al. Scaffold-gs: Structured 3d gaussians for view-adaptive rendering[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024: 20654-20664.

[5] Huang H W, Chen F C, Chai W, et al. Zero-shot 3d question answering via voxel-based dynamic token compression[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 19424-19434.

AuthorsOpenReview ↗

Response to Reviewer woYf (Part 2/4)

W1: The novelty of the proposed method appears limited: Using 3D voxel representations to reduce the number of tokens is common in multi-view tasks, as demonstrated by Scaffold-GS and DTC. The GroupRoute is similar to HiDrop.

A1 (Continued)

  • Layer-wise Visual Computation (GroupRoute):

    • As summarized in the Global Response, the key distinction from HiDrop lies in how visual tokens are processed within the visual computation window. Our analysis in Sec. 3 shows that visual token relevance is locally stable, but currently low-relevance tokens may become important at later layers. Building on HiDrop’s Late Entry and Early Exit [6], GroupRoute introduces recoverable group-wise skipping within a visual computation window adapted to multi-view 3D VLMs. Unlike HiDrop’s permanent pruning, low-priority tokens temporarily bypass attention and FFN computation while retaining their hidden states. At regrouping anchors, all retained visual tokens are updated and regrouped, allowing previously skipped tokens to resume computation.

    • To isolate the contribution of recoverable skipping, we additionally disable GeoSemZip and Late Entry/Early Exit and compare RGS with permanent pruning at matched average active visual token budgets per layer. On LLaVA-OneVision, RGS improves performance retention over the best tested pruning variant by 0.57/1.44 points when 50%/10% of visual tokens remain active after each anchor, respectively, supporting the value of retaining tokens for later regrouping. The table below further shows that full GeoGR outperforms HiDrop on both models after post-training at comparable average active-token budgets.

      Table G2. Isolated RGS evaluation and comparison with HiDrop. Scores report average performance retention relative to the corresponding Dense model (100%), gains are in percentage points.

      SettingModelOurs (%)Comparison Methods (%)Gain (pp)
      RGS only, 50% activeLLaVA-OneVision-7B100.38Permanent pruning: 99.81+0.57
      RGS only, 10% activeLLaVA-OneVision-7B97.30Permanent pruning: 95.86+1.44
      Full GeoGRLLaVA-OneVision-7B95.18HiDrop: 88.84+6.34
      Full GeoGRVideo-3D LLM-7B96.80HiDrop: 95.21+1.59
  • Joint Representation and Computation Compression:

    • GeoSemZip and GroupRoute address two distinct sources of redundancy: repeated scene observations across views and unnecessary visual computation across decoder layers. GeoSemZip consolidates cross-view evidence into a compact scene representation, while GroupRoute selectively processes the retained tokens according to their query relevance across layers. Together, they reduce both the number of tokens entering the LLM and the computation spent on those tokens. Compared with projector-only compression at a comparable average active-token budget, this joint design allows more scene tokens to be retained at the projector and processed selectively across layers.

    • Our original submission reports this comparison in Appendix E.2, Table 18, summarized below in Table G3. GeoGR achieves 95.18% retention with 638 average active visual tokens per layer, compared with 94.00% at 628 tokens for GeoSemZip alone and 94.65% at 2,128 tokens for GroupRoute alone, supporting the complementarity of the two stages.

      Table G3. Component comparison on LLaVA-OneVision-7B. Visual tokens are averaged over all decoder layers. GroupRoute alone includes Late Entry/Early Exit and RGS but disables GeoSemZip.

      ConfigurationProjector keep ratioAvg. active visual tokens/layerScanQA CIDErScanQA EMSQA3D EMAvg. (%)
      GeoSemZip only10%62887.4425.4156.6994.00
      GroupRoute only100%2,12888.4826.0755.8494.65
      GeoGR (both)30%63888.2526.2056.6195.18

[6] Wu H, Fan Y, Dai J, et al. Hidrop: Hierarchical vision token reduction in mllms via late injection, concave pyramid pruning, and early exit[C]//International Conference on Learning Representations. 2026, 2026: 97296-97316.

AuthorsOpenReview ↗

Response to Reviewer woYf (Part 3/4)

W2: For the Saliency-aware selection in Line 263, higher attention weights do not necessarily indicate greater token importance. For example, vision foundation models such as DINOv2 can assign high attention scores to task-irrelevant regions.

A2: We agree that high attention weights alone do not guarantee greater token importance. GeoSemZip uses vision-encoder attention as a visual-saliency cue and complements it with spatial coverage and residual merging to preserve information from regions with low attention scores.

  • Visual-saliency Estimation: For each visual token, we average its received self-attention across heads and visual query positions in the last executed vision-encoder layer. We pool these scores to the projector grid and take the maximum within each voxel. This question-independent score reflects the attention a region receives during visual encoding. Encoder attention is also used for visual token selection in VisionZip [7] and VisPruner [8].

  • Spatial Coverage and Residual Merging: GeoSemZip uses attention to select dominant tokens and allocates the remaining budget to spatially distributed contextual anchors, sampled in Morton order from the remaining voxels independently of their saliency scores. This spatial-coverage branch helps retain regions that saliency-only selection may overlook. Residual merging further aggregates features from unselected voxels into these anchors, preserving additional scene context without increasing the token budget.

  • Component Ablation: Table 4 of our original submission evaluates these components at a fixed 30% token budget. Saliency-only selection retains 96.99% of Dense performance. Including cross-view voxel consolidation and spatial coverage increases retention to 98.19%, and adding residual merging further raises it to 99.33%. These results support combining attention-based selection with mechanisms that preserve complementary spatial context.

Q1: In the equation in Line 258, different operations are used to aggregate features and saliency scores within each voxel, which may cause a mismatch between the aggregated features and their corresponding saliency scores.

A3: The pooled feature and the saliency score summarize different properties of the same voxel. The feature integrates visual observations across views, while the score aggregates their saliency for voxel selection. This score is not an attention score recomputed from the averaged voxel feature.

  • Why We Use Different Pooling Operations: Mean feature pooling integrates observations of the same voxel across views and reduces dependence on any single observation. For saliency, visibility and occlusion can vary across views. A region may be clearly visible and highly salient in one view while receiving lower scores in others. Max saliency pooling preserves that strong response for voxel selection. The two operations therefore serve complementary purposes: fusing cross-view features and retaining the strongest observed saliency cue.

  • Ablation at a Fixed Token Budget: We evaluate all four combinations of feature and saliency pooling at the same 30% token budget (Table R2). Mean feature pooling with max saliency pooling achieves the highest performance retention (99.33%), compared with mean/mean (99.06%), max/max (98.29%), and max/mean (98.11%), with feature pooling listed first. Holding feature pooling fixed at mean, switching saliency pooling from mean to max improves retention by 0.27 percentage points. Holding saliency pooling fixed at max, switching feature pooling from max to mean improves retention by 1.04 percentage points. These comparisons support using mean pooling for cross-view feature fusion and max pooling for voxel selection in GeoSemZip.

    Table R2. Feature and saliency pooling at a 30% token budget on LLaVA-OneVision-7B. Retention is normalized to Dense (100%).

    Feature poolingSaliency poolingScanQA CIDErScanQA EMSQA3D EMAvg. retention
    MeanMax (GeoSemZip)93.9527.5157.6399.33%
    MeanMean93.1527.7657.1299.06%
    MaxMax91.6927.2957.6398.29%
    MaxMean92.0527.1757.3798.11%

[7] Yang S, Chen Y, Tian Z, et al. Visionzip: Longer is better but not necessary in vision language models[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 19792-19802.

[8] Zhang Q, Cheng A, Lu M, et al. Beyond text-visual attention: Exploiting visual cues for effective token pruning in vlms[C]//2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025: 20857-20867.

Reviewer_Q2rgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GeoGR combines geometry-guided voxel compression (GeoSemZip) with query-conditioned, recoverable LLM-layer skipping (GroupRoute). After post-training on 66,940 selected examples, it retains 95.18%/96.80% of the corresponding subset-trained dense models' aggregate performance on LLaVA-OneVision/Video-3D LLM at about 9% of LLM-side visual FLOPs, with 2.4×/2.3× measured TTFT speedups.

Critical strengths
  1. Results on two backbones include measured latency gains, supporting the practical value of the system.
  2. Ablations support complementary compression and query-conditioned grouping (95.18% versus 92.73% average retention under the same schedule).
Critical weaknesses
  1. At the same checkpoint and matched average active-token budget, recoverable skipping beats the best tested permanent pruning by only 0.15 average percentage points (Table 6); uncertainty and cross-backbone evidence are missing.
  2. Table 3 includes visual encoding and compression time, but the availability and any upstream estimation cost of depth, intrinsics, and poses remain unclear, especially for RGB-only LLaVA.
Questions for authors
  1. Can you provide paired scene-level uncertainty or independent post-training seeds, plus a matched Video-3D LLM comparison for skipping?
  2. Are geometric inputs supplied or estimated at inference, and is any estimation cost included in TTFT?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Q2rg (Part 1/3)

We thank you for your thorough review and appreciate both the positive feedback and the constructive comments. Please find our responses to your points below. In the revised manuscript, newly added or substantially modified text is highlighted in blue, while previously existing passages that already address your concerns are highlighted in yellow for ease of reference. Please let us know if you have any further questions or suggestions.

W1: At the same checkpoint and matched average active-token budget, recoverable skipping beats the best tested permanent pruning by only 0.15 average percentage points (Table 6); uncertainty and cross-backbone evidence are missing.

Q1: Can you provide paired scene-level uncertainty or independent post-training seeds, plus a matched Video-3D LLM comparison for skipping?

A1: We have extended the budget-matched comparison of recoverable group-wise skipping (RGS) and permanent pruning to Video-3D LLM and report mean ± standard deviation across seeds on both models. We also evaluate RGS in isolation at matched active-token budgets and separately compare recoverable skipping with permanent pruning using the same anchor positions and keep-ratio settings.

  • Cross-model Comparison and Seed Variability:

    • We extend the original budget-matched comparison on LLaVA-OneVision (Table 6 of the original submission) to Video-3D LLM across all five benchmarks. GeoSemZip and Late Entry/Early Exit remain enabled in all configurations. At a matched budget of approximately 671 active visual tokens per layer, GeoGR retains 96.80% of Dense performance, compared with 96.38% when RGS is replaced by the best tested permanent-pruning configuration. This gives a gain of 0.42 percentage points on Video-3D LLM, compared with 0.15 percentage points in the original LLaVA-OneVision comparison.

    • RGS itself is training-free and is applied only at inference. GeoGR’s compression-aware post-training adapts the model to GeoSemZip and Late Entry/Early Exit. For the seed comparisons, we keep the resulting checkpoint fixed for each backbone and vary the evaluation seed to assess evaluation variability. As shown in Table R1, GeoGR with RGS achieves mean retention of 95.14 ± 0.03% on LLaVA-OneVision and 96.83 ± 0.03% on Video-3D LLM, exceeding the strongest pruning mean by 0.10 and 0.63 percentage points, respectively.

      Table R1. Matched-budget comparison within the full GeoGR pipeline. Entries report single-run retention (mean ± SD across seeds), in %. All configurations use GeoSemZip and Late Entry/Early Exit. Permanent pruning replaces RGS in the ablation variants.

      SettingsLLaVA-OneVision (%)Video-3D LLM (%)
      GeoGR with pruning at 994.65 (94.71 ± 0.05)95.75 (95.77 ± 0.04)
      GeoGR with pruning at 1395.03 (95.04 ± 0.06)96.38 (96.20 ± 0.03)
      GeoGR with pruning at 9 and 1394.73 (94.73 ± 0.06)96.26 (96.12 ± 0.01)
      GeoGR (RGS at 9 and 13)95.18 (95.14 ± 0.03)96.80 (96.83 ± 0.03)
  • Isolated Evaluation of RGS:

    • We believe the 0.15-percentage-point gain in Table 6 of the original submission may partly reflect the compression already introduced by GeoSemZip and Late Entry/Early Exit. GeoSemZip reduces the number of visual tokens, while Late Entry/Early Exit narrows their computation window, potentially leaving fewer opportunities for skipped tokens to contribute again. To examine RGS without these constraints, we disable both components on LLaVA-OneVision and compare RGS with permanent pruning at matched average active-token budgets (Table R2).

    • At the 50% active ratio, RGS retains 100.38% of Dense performance, compared with 99.81% for the best tested budget-matched pruning configuration (+0.57 percentage points). At the 10% active ratio, this advantage increases to 1.44 points (97.30% vs. 95.86%). These results support the benefit of preserving skipped tokens for later regrouping, even without GeoSemZip or Late Entry/Early Exit, with a larger gain at the lower active ratio.

      Table R2. Isolated RGS evaluation on LLaVA-OneVision. “Active ratio” refers to RGS after anchors 9 and 13; pruning at layer 9 is budget-matched. “Tokens” denotes average active visual tokens per layer.

      MethodActive ratioTokensAvg. Retention (%)Gain (pp)
      Permanent pruning0.504,25699.81—
      RGS0.504,256100.38+0.57
      Permanent pruning0.102,64395.86—
      RGS0.102,64497.30+1.44
AuthorsOpenReview ↗

Response to Reviewer Q2rg (Part 2/3)

W1: At the same checkpoint and matched average active-token budget, recoverable skipping beats the best tested permanent pruning by only 0.15 average percentage points (Table 6); uncertainty and cross-backbone evidence are missing.

Q1: Can you provide paired scene-level uncertainty or independent post-training seeds, plus a matched Video-3D LLM comparison for skipping?

A1 (Continued):

  • Comparison under Identical Parameter Settings:
    • The 0.15-point gap may also partly reflect our use of closely matched average active-token budgets. RGS preserves skipped tokens and selects from the full recoverable pool at each anchor, whereas permanent pruning selects only from the remaining tokens, causing removals to accumulate. Consequently, using the same anchors and keep ratios results in fewer active tokens for permanent pruning. Matching the RGS budget therefore requires relaxing the pruning configuration, allowing the pruning baseline to retain more tokens than under identical parameter settings.

    • Table R3 illustrates this difference within the full GeoGR pipeline, with GeoSemZip and Late Entry/Early Exit enabled. We replace RGS with permanent pruning at the same anchors (layers 9 and 13), using a 50% keep ratio at each anchor. Under these settings, permanent pruning processes approximately 437 vs. 638 active visual tokens per layer on LLaVA-OneVision and 453 vs. 671 on Video-3D LLM. Its performance retention is 93.66% vs. 95.18% and 94.75% vs. 96.80%, respectively.

      Table R3. Recoverable skipping and permanent pruning with the same 50% keep ratio at each anchor (layers 9 and 13).

      ModelMethodTokensAvg. retention
      LLaVA-OneVision-7BGeoGR (RGS)63895.18%
      LLaVA-OneVision-7BGeoGR with permanent pruning43793.66%
      Video-3D LLM-7BGeoGR (RGS)67196.80%
      Video-3D LLM-7BGeoGR with permanent pruning45394.75%

Taken together, the budget-matched comparisons on both backbones and the isolated RGS experiments support the benefit of preserving skipped tokens for later regrouping.

AuthorsOpenReview ↗

Response to Reviewer Q2rg (Part 3/3)

W2: Table 3 includes visual encoding and compression time, but the availability and any upstream estimation cost of depth, intrinsics, and poses remain unclear, especially for RGB-only LLaVA.

Q2: Are geometric inputs supplied or estimated at inference, and is any estimation cost included in TTFT?

A2: As clarified in the Global Response, we directly use depth maps, camera intrinsics, and poses supplied by ScanNet [1], without online depth or pose estimation. All processing between the vision encoder and the LLM, including feature projection, visual pooling, 3D back-projection, voxelization, and token compression, is included in the projector-stage timing and TTFT.

  • Inference Optimizations: As detailed in the Global Response, we optimized visual encoding, attention computation, visual pooling, LM Head projection, and RoPE reuse. These general optimizations are applied uniformly to all compared methods on both models. GeoGR additionally benefits from reusing routing indices and LayerNorm/Q/K results. Inference consistency checks were performed, with trained models, token budgets, and compression settings unchanged between evaluation and timing.

  • Updated Efficiency Results: For GeoGR, TTFT decreases from 476.7 to 324.3 ms on LLaVA-OneVision-7B and from 556.6 to 359.9 ms on Video-3D LLM-7B, corresponding to reductions of 32.0% and 35.3%, respectively. Relative to optimized Dense, GeoGR achieves 6.8× and 6.7× LLM Prefill speedups and 2.6× and 2.3× TTFT speedups, while retaining 95.2% and 96.8% of Dense performance, respectively. GeoGR improves performance retention over SeGPruner by 3.5/1.0 percentage points with 7.2%/5.0% lower TTFT, and over DTC by 2.6/1.4 points with 76.6%/72.8% lower TTFT, achieving the best performance–efficiency trade-off among the compared methods.

    Table G1. Efficiency before and after inference optimizations on NVIDIA RTX PRO 5000 Blackwell. Arrows denote before → after; speedups in parentheses are relative to Dense under the corresponding optimization setting. KV Cache denotes visual KV memory and remains unchanged. Avg. reports the original benchmark performance normalized to Dense (100%).

    ModelMethodKV Cache (MB)Encoder (ms)Projector (ms)LLM Prefill (ms)TTFT (ms)Avg. (%)
    LLaVA-OneVision-7BDense359.7311.2 → 232.673.7 → 7.3635.4 (1.0×) → 587.3 (1.0×)1020.2 (1.0×) → 827.3 (1.0×)100.0
    LLaVA-OneVision-7BSeGPruner36.7307.5 → 231.8101.3 → 34.891.5 (6.9×) → 82.5 (7.1×)500.7 (2.0×) → 349.6 (2.4×)91.7
    LLaVA-OneVision-7BDTC40.3302.0 → 226.81080.6 → 1068.196.7 (6.6×) → 88.4 (6.6×)1480.5 (0.7×) → 1384.3 (0.6×)92.6
    LLaVA-OneVision-7BGeoGR36.6311.2 → 225.478.5 → 12.086.8 (7.3×) → 86.4 (6.8×)476.7 (2.1×) → 324.3 (2.6×)95.2
    Video-3D LLM-7BDense359.7386.8 → 251.982.0 → 15.1578.1 (1.0×) → 578.4 (1.0×)1047.1 (1.0×) → 845.5 (1.0×)100.0
    Video-3D LLM-7BSeGPruner36.7381.3 → 256.7105.1 → 38.583.7 (6.9×) → 83.4 (6.9×)570.6 (1.8×) → 378.8 (2.2×)95.8
    Video-3D LLM-7BDTC40.3367.2 → 240.91016.2 → 998.287.2 (6.6×) → 86.4 (6.7×)1470.8 (0.7×) → 1325.2 (0.6×)95.4
    Video-3D LLM-7BGeoGR38.5381.3 → 252.687.0 → 20.387.6 (6.6×) → 86.8 (6.7×)556.6 (1.9×) → 359.9 (2.3×)96.8

[1] Dai A, Chang A X, Savva M, et al. Scannet: Richly-annotated 3d reconstructions of indoor scenes[C]//2017 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, 2017: 2432-2443.

Reviewer_uJC1OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper represent GeoGR, a training framework for efficient 3D VLM. GeoGR is composed of two main modules: GeoSemZip and GeoRoute. First, GeosemZIP compress multi-view tokens based on 3D voxel. Second, GeoRoute reduce computation cost with grouping. For additional contribution, this paper represent distilled dataset.

Critical strengths
  • The paper provides clear motivation for its main components through Fig. 2 and Fig. 4.
  • The paper shows that addressing representation-level (cross-view) redundancy and computation-level (layer-wise) redundancy is complementary.
  • GeoSemZip alone retains 99.33% of the dense model’s performance at a 30% token budget, i.e., nearly lossless compression.
  • Compared with other compression methods, GeoGR achieves higher performance under similar computation and memory budgets.
Critical weaknesses
  • Incremental Novelty Late Entry / Early Exit is adopted from HiDrop, and the voxel-based cross-view merging is similar to DTC.
  • Comparison Methods Table 2 omits baselines that compress within the LLM (e.g., HiDrop) and compares only against projector-level methods.
  • Limited Generalization of Experiments Both evaluated models share the same SigLIP + Qwen2-7B architecture and belong to a similar model family. This is insufficient to support the claim that GeoGR is effective for models both with and without explicit 3D spatial encoding.
Questions for authors
  • The window, token budget, ρ, and voxel size seem to be tuned on ScanQA/SQA3D, the same benchmarks used for evaluation. Was a separate validation split used?
  • Could the authors compare against LLM-level compression methods (e.g., FastV/PyramidDrop, HiDrop) under the same FLOPs and post-training settings?
  • GeoSemZip alone retains 99.33% at a 30% budget, but adding Late Entry drops it to 93.99% even with [5, 28]. Is this loss due to layer skipping or to the second-stage post-training?
5 comments
AuthorsOpenReview ↗

Response to Reviewer uJC1 (Part 1/5)

We thank you for your thorough review and appreciate both the positive feedback and the constructive comments. Please find our responses to your points below. In the revised manuscript, newly added or substantially modified text is highlighted in blue, while previously existing passages that already address your concerns are highlighted in yellow for ease of reference. Please let us know if you have any further questions or suggestions.

W1: Incremental Novelty. Late Entry / Early Exit is adopted from HiDrop, and the voxel-based cross-view merging is similar to DTC.

A1: We respectfully disagree with the assessment of incremental novelty. As summarized in the Global Response, GeoGR jointly addresses representational redundancy across views and computational redundancy across LLM layers in multi-view 3D VLMs. Its technical contributions include spatial–semantic compression that preserves salient content, spatial coverage, and information from unselected voxels, together with recoverable group-wise skipping motivated by our analysis of evolving token relevance across layers. Below, we clarify these contributions and present the supporting evidence.

  • Spatial–semantic Representation Compression (GeoSemZip): Voxelization is a well-established technique for 3D representation [1-3] and is not specific to DTC [4]. The distinction between DTC and GeoSemZip lies in how they compress and preserve scene information using this geometric foundation. DTC progressively enlarges voxels and repeatedly performs similarity-based token matching to remove redundancy. GeoSemZip instead consolidates cross-view observations into voxels and allocates a fixed token budget to high-saliency dominant voxels and contextual anchors sampled in Morton order from the remaining voxels. It then aggregates features from unselected voxels into these anchors through residual merging. This design jointly preserves salient content, spatial coverage, and information from unselected regions, while avoiding iterative voxel resizing and matching.
    • Our original submission already reports comparisons between GeoSemZip and DTC at approximately 10%, 20%, and 30% projector token budgets in Appendix D.2, Table 12. We present the key results below for ease of reference. GeoSemZip achieves higher performance retention at all three budgets.

      Table R1. Performance retention on LLaVA-OneVision-7B across projector token budgets. All configurations are post-trained on LLaVA-OneVision-7B, with retention normalized to Dense (100%).

      Projector token budgetDTC (%)GeoSemZip (%)Gain (pp)
      ≈10%92.6194.00+1.39
      ≈20%94.3897.26+2.88
      ≈30%98.2599.33+1.08
    • Our updated latency measurements further demonstrate the practical efficiency advantage of GeoSemZip. With the same general inference optimizations applied to both methods, GeoGR achieves substantially lower projector-stage latency than DTC on both models.

      Table G1 (excerpt from the Global Response). Projector-stage latency after inference optimizations (ms), including token compression, measured on NVIDIA RTX PRO 5000 Blackwell.

      ModelDTCGeoGR (with GeoSemZip)
      LLaVA-OneVision-7B1068.112.0
      Video-3D LLM-7B998.220.3
    • Together, these results demonstrate the benefits of GeoSemZip in both performance retention and practical efficiency.

[1] Liu Z, Tang H, Lin Y, et al. Point-voxel cnn for efficient 3d deep learning[J]. Advances in neural information processing systems, 2019, 32.

[2] Graham B, Engelcke M, Van Der Maaten L. 3d semantic segmentation with submanifold sparse convolutional networks[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Ieee, 2018: 9224-9232.

[3] Chen Y, Liu J, Zhang X, et al. Voxelnext: Fully sparse voxelnet for 3d object detection and tracking[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023: 21674-21683.

[4] Huang H W, Chen F C, Chai W, et al. Zero-shot 3d question answering via voxel-based dynamic token compression[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 19424-19434.

AuthorsOpenReview ↗

Response to Reviewer uJC1 (Part 2/5)

W1: Incremental Novelty. Late Entry / Early Exit is adopted from HiDrop, and the voxel-based cross-view merging is similar to DTC.

A1 (Continued)

  • Layer-wise Visual Computation (GroupRoute):

    • As summarized in the Global Response, the key distinction from HiDrop lies in how visual tokens are processed within the visual computation window. Our analysis in Sec. 3 shows that visual token relevance is locally stable, but currently low-relevance tokens may become important at later layers. Building on HiDrop’s Late Entry and Early Exit [5], GroupRoute introduces recoverable group-wise skipping within a visual computation window adapted to multi-view 3D VLMs. Unlike HiDrop’s permanent pruning, low-priority tokens temporarily bypass attention and FFN computation while retaining their hidden states. At regrouping anchors, all retained visual tokens are updated and regrouped, allowing previously skipped tokens to resume computation.

    • To isolate the contribution of recoverable skipping, we additionally disable GeoSemZip and Late Entry/Early Exit and compare RGS with permanent pruning at matched average active visual token budgets per layer. On LLaVA-OneVision, RGS improves performance retention over the best tested pruning variant by 0.57/1.44 points when 50%/10% of visual tokens remain active after each anchor, respectively, supporting the value of retaining tokens for later regrouping. The table below further shows that full GeoGR outperforms HiDrop on both models after post-training at comparable average active-token budgets.

      Table G2. Isolated RGS evaluation and comparison with HiDrop. Scores report average performance retention relative to the corresponding Dense model (100%), gains are in percentage points.

      SettingModelOurs (%)Comparison Methods (%)Gain (pp)
      RGS only, 50% activeLLaVA-OneVision-7B100.38Permanent pruning: 99.81+0.57
      RGS only, 10% activeLLaVA-OneVision-7B97.30Permanent pruning: 95.86+1.44
      Full GeoGRLLaVA-OneVision-7B95.18HiDrop: 88.84+6.34
      Full GeoGRVideo-3D LLM-7B96.80HiDrop: 95.21+1.59
  • Joint Representation and Computation Compression:

    • GeoSemZip and GroupRoute address two distinct sources of redundancy: repeated scene observations across views and unnecessary visual computation across decoder layers. GeoSemZip consolidates cross-view evidence into a compact scene representation, while GroupRoute selectively processes the retained tokens according to their query relevance across layers. Together, they reduce both the number of tokens entering the LLM and the computation spent on those tokens. Compared with projector-only compression at a comparable average active-token budget, this joint design allows more scene tokens to be retained at the projector and processed selectively across layers.

    • Our original submission reports this comparison in Appendix E.2, Table 18, summarized below in Table G3. GeoGR achieves 95.18% retention with 638 average active visual tokens per layer, compared with 94.00% at 628 tokens for GeoSemZip alone and 94.65% at 2,128 tokens for GroupRoute alone, supporting the complementarity of the two stages.

      Table G3. Component comparison on LLaVA-OneVision-7B. Visual tokens are averaged over all decoder layers. GroupRoute alone includes Late Entry/Early Exit and RGS but disables GeoSemZip.

      ConfigurationProjector keep ratioAvg. active visual tokens/layerScanQA CIDErScanQA EMSQA3D EMAvg. (%)
      GeoSemZip only10%62887.4425.4156.6994.00
      GroupRoute only100%2,12888.4826.0755.8494.65
      GeoGR (both)30%63888.2526.2056.6195.18

[5] Wu H, Fan Y, Dai J, et al. Hidrop: Hierarchical vision token reduction in mllms via late injection, concave pyramid pruning, and early exit[C]//International Conference on Learning Representations. 2026, 2026: 97296-97316.

AuthorsOpenReview ↗

Response to Reviewer uJC1 (Part 3/5)

W2: Comparison Methods. Table 2 omits baselines that compress within the LLM (e.g., HiDrop) and compares only against projector-level methods.

Q2: Could the authors compare against LLM-level compression methods (e.g., FastV/PyramidDrop, HiDrop) under the same FLOPs and post-training settings?

A2: We thank the reviewer for suggesting a direct comparison with LLM-level compression methods. We have added comparisons with FastV [6], PyramidDrop [7], DART [8], and HiDrop [5] on both LLaVA-OneVision-7B and Video-3D LLM-7B at comparable LLM-side visual FLOP budgets. All post-trained methods use the same training settings. We also include training-free results for FastV, PyramidDrop, and DART.

On LLaVA-OneVision, GeoGR achieves 95.18% performance retention, exceeding the strongest post-trained LLM-level baseline, PyramidDrop (88.87%), by 6.31 percentage points. On Video-3D LLM, GeoGR achieves 96.80% retention, outperforming the strongest post-trained LLM-level baseline, HiDrop (95.21%), by 1.59 percentage points.

Table R2. LLM-level compression on LLaVA-OneVision-7B. TF: training-free; PT: post-training. Tokens denotes the average number of active visual tokens per LLM layer.

SettingMethodTokensVisual TFLOPsScanQA CIDErScanQA EMSQA3D EMAvg. retention
TFFastV6418.5382.4723.9851.7887.69%
TFPyramidDrop6478.6177.7522.6751.8984.55%
TFDART6458.5875.0221.5850.0781.23%
PTFastV6418.5376.0522.7452.4384.34%
PTPyramidDrop6478.6181.5523.9154.4888.87%
PTDART6458.5878.3423.4952.6086.14%
PTHiDrop6439.6280.5924.0254.7988.84%
PTGeoGR (ours)6388.6588.2526.2056.6195.18%

Table R3. LLM-level compression on Video-3D LLM-7B. TF: training-free; PT: post-training. Tokens denotes the average number of active visual tokens per LLM layer.

SettingMethodTokensVisual TFLOPsScanQA CIDErScanQA EMSQA3D EMScan2Cap [email protected]Scan2Cap [email protected]ScanRefer [email protected]ScanRefer [email protected]Multi3DRefer [email protected]Multi3DRefer [email protected]Avg. retention
TFFastV6759.9579.9621.6750.5333.6141.5754.8048.9347.4943.4582.14%
TFPyramidDrop6749.9779.6821.6551.8631.5039.4854.9749.0743.0839.5779.86%
TFDART6759.9581.1222.7450.6133.5742.8254.6648.5843.0739.3981.04%
PTFastV6759.9583.4323.5153.9135.0149.5055.8049.5754.6949.5588.45%
PTPyramidDrop6749.9790.3025.4555.7838.5366.8655.9449.7054.5149.4593.74%
PTDART6759.9585.7624.5853.7736.4057.0155.0648.7454.5449.5090.17%
PTHiDrop67310.0491.8126.0557.2038.5567.7256.4450.1855.6550.5595.21%
PTGeoGR (ours)6719.0994.8526.2256.4639.4573.2756.8750.6955.8950.8496.80%

[5] Wu H, Fan Y, Dai J, et al. HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exit. ICLR, 2026.

[6] Chen L, Zhao H, Liu T, et al. An Image Is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models. ECCV, 2024, pp. 19–35.

[7] Xing L, Huang Q, Dong X, et al. PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction. arXiv preprint arXiv:2410.17247, 2024.

[8] Wen Z, Gao Y, Wang S, et al. Stop Looking for “Important Tokens” in Multimodal Language Models: Duplication Matters More. EMNLP, 2025, pp. 9961–9980.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global Response (Part 1/3)

We thank all reviewers for their thorough reviews and constructive feedback. We appreciate their recognition of the well-motivated design and informative analyses underlying GeoGR (uJC1, woYf). These analyses offer the community insights into representational and computational sparsity in 3D VLMs. The reviewers also highlight the complementarity of our two compression stages (uJC1, Q2rg) and GeoGR’s favorable performance–efficiency trade-off (all reviewers).

Across the reviews, we identified a few shared themes, which we address below along with additional points we would like to highlight:

  • Practical Latency Improvements through Inference Optimizations

    • We optimized inference through layer-wise visual encoding, frame-wise attention computation, channels-last interpolation for visual pooling, last-position LM Head projection, and per-prefill RoPE reuse. GeoGR additionally benefits from reusing routing indices and LayerNorm/Q/K results. For fair comparisons, these general optimizations are applied uniformly to GeoGR and all baselines on both models. Equivalent-inference checks were performed during optimization, with the trained models, token budgets, and compression settings unchanged between evaluation and timing. Implementation details of these optimizations are provided in Appendix D.1 and latency results are summarized in Table G1.

    • These optimizations reduce Dense TTFT by 192.9 ms (18.9%) on LLaVA-OneVision-7B and 201.6 ms (19.3%) on Video-3D LLM-7B, reaching 827.3 ms and 845.5 ms, respectively.

    • For GeoGR, TTFT decreases from 476.7 to 324.3 ms on LLaVA-OneVision-7B and from 556.6 to 359.9 ms on Video-3D LLM-7B, corresponding to reductions of 32.0% and 35.3%, respectively. Relative to optimized Dense, GeoGR achieves 6.8× and 6.7× LLM Prefill speedups and 2.6× and 2.3× TTFT speedups, while retaining 95.2% and 96.8% of Dense performance, respectively. GeoGR improves performance retention over SeGPruner by 3.5/1.0 percentage points with 7.2%/5.0% lower TTFT, and over DTC by 2.6/1.4 points with 76.6%/72.8% lower TTFT, achieving the best performance–efficiency trade-off among the compared methods.

    • Geometric Inputs and Timing Scope (Reviewer: Q2rg): Our experiments use depth maps, camera intrinsics, and poses supplied by ScanNet [1]. No online depth or pose estimation is performed, so there is no estimation overhead to include in TTFT. Here, “Projector” denotes all processing after the vision encoder and before the LLM, including feature projection, visual pooling, 3D back-projection, voxelization, and token compression. This entire stage is included in TTFT.

      Table G1. Efficiency before and after inference optimizations on NVIDIA RTX PRO 5000 Blackwell. Arrows denote before → after; speedups in parentheses are relative to Dense under the corresponding optimization setting. KV Cache denotes visual KV memory and remains unchanged. Avg. reports the original benchmark performance normalized to Dense (100%).

      ModelMethodKV Cache (MB)Encoder (ms)Projector (ms)LLM Prefill (ms)TTFT (ms)Avg. (%)
      LLaVA-OneVision-7BDense359.7311.2 → 232.673.7 → 7.3635.4 (1.0×) → 587.3 (1.0×)1020.2 (1.0×) → 827.3 (1.0×)100.0
      LLaVA-OneVision-7BSeGPruner36.7307.5 → 231.8101.3 → 34.891.5 (6.9×) → 82.5 (7.1×)500.7 (2.0×) → 349.6 (2.4×)91.7
      LLaVA-OneVision-7BDTC40.3302.0 → 226.81080.6 → 1068.196.7 (6.6×) → 88.4 (6.6×)1480.5 (0.7×) → 1384.3 (0.6×)92.6
      LLaVA-OneVision-7BGeoGR36.6311.2 → 225.478.5 → 12.086.8 (7.3×) → 86.4 (6.8×)476.7 (2.1×) → 324.3 (2.6×)95.2
      Video-3D LLM-7BDense359.7386.8 → 251.982.0 → 15.1578.1 (1.0×) → 578.4 (1.0×)1047.1 (1.0×) → 845.5 (1.0×)100.0
      Video-3D LLM-7BSeGPruner36.7381.3 → 256.7105.1 → 38.583.7 (6.9×) → 83.4 (6.9×)570.6 (1.8×) → 378.8 (2.2×)95.8
      Video-3D LLM-7BDTC40.3367.2 → 240.91016.2 → 998.287.2 (6.6×) → 86.4 (6.7×)1470.8 (0.7×) → 1325.2 (0.6×)95.4
      Video-3D LLM-7BGeoGR38.5381.3 → 252.687.0 → 20.387.6 (6.6×) → 86.8 (6.7×)556.6 (1.9×) → 359.9 (2.3×)96.8

[1] Dai A, Chang A X, Savva M, et al. Scannet: Richly-annotated 3d reconstructions of indoor scenes[C]//2017 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, 2017: 2432-2443.

AuthorsOpenReview ↗

Global Response (Part 2/3)

  • Technical Contributions and Novelty (Reviewers: uJC1, woYf)
    • Spatial–semantic Representation Compression (GeoSemZip): Voxelization is a well-established technique for 3D representation [2-4], and we do not claim novelty in voxelization itself. Both DTC [5] and GeoSemZip build on voxelization to further compress visual tokens. DTC progressively enlarges voxels and repeatedly performs similarity-based token matching to remove redundancy. GeoSemZip instead consolidates cross-view observations into voxels and, under a fixed token budget, preserves key voxels based on semantic saliency and selects contextual anchors to ensure spatial coverage of the scene. It then recovers information from unselected voxels through residual aggregation into these anchors, without increasing the token count. This design avoids DTC’s iterative voxel resizing and matching process. With the same general inference optimizations applied to both methods (Table G1), GeoGR’s projector-stage latency, including compression, is 12.0/20.3 ms, compared with 1068.1/998.2 ms for DTC on LLaVA-OneVision-7B and Video-3D LLM-7B, respectively. The full GeoGR framework also retains 95.2%/96.8% of Dense performance, compared with 92.6%/95.4% for DTC, demonstrating a better performance–efficiency trade-off.

    • Layer-wise Visual Computation (GroupRoute):

      • Our analysis in Sec. 3 shows that visual token relevance is locally stable, but currently low-relevance tokens may become important at later layers. Building on HiDrop’s Late Entry and Early Exit [6], GroupRoute introduces recoverable group-wise skipping within a visual computation window adapted to multi-view 3D VLMs. Unlike HiDrop’s permanent pruning, low-priority tokens temporarily bypass attention and FFN computation while retaining their hidden states. At regrouping anchors, all retained visual tokens are updated and regrouped, allowing previously skipped tokens to resume computation.

      • To isolate the contribution of recoverable skipping, we disable GeoSemZip and Late Entry/Early Exit and compare RGS with permanent pruning at matched average active visual token budgets per layer. On LLaVA-OneVision, RGS improves performance retention over the best tested pruning variant by 0.57/1.44 points when 50%/10% of visual tokens remain active after each anchor, supporting the value of retaining tokens for later regrouping. Table G2 further shows that full GeoGR outperforms HiDrop on both models after post-training at comparable budgets.

        Table G2. Isolated RGS evaluation and comparison with HiDrop. Scores report average performance retention relative to the corresponding Dense model (100%), gains are in percentage points.

        SettingModelOurs (%)Comparison Methods (%)Gain (pp)
        RGS only, 50% activeLLaVA-OneVision-7B100.38Permanent pruning: 99.81+0.57
        RGS only, 10% activeLLaVA-OneVision-7B97.30Permanent pruning: 95.86+1.44
        Full GeoGRLLaVA-OneVision-7B95.18HiDrop: 88.84+6.34
        Full GeoGRVideo-3D LLM-7B96.80HiDrop: 95.21+1.59
    • Joint Representation and Computation Compression:

      • GeoSemZip and GroupRoute address two distinct sources of redundancy: repeated scene observations across views and unnecessary visual computation across decoder layers. GeoSemZip consolidates cross-view evidence into a compact scene representation, while GroupRoute selectively processes the retained tokens according to their query relevance across layers. Together, they reduce both the number of tokens entering the LLM and the computation spent on those tokens. Compared with projector-only compression at a comparable average active-token budget, this joint design allows more scene tokens to be retained at the projector and processed selectively across layers.

      • Our original submission reports this comparison in Appendix E.2, Table 18, summarized below in Table G3. GeoGR achieves 95.18% retention with 638 average active visual tokens per layer, compared with 94.00% at 628 tokens for GeoSemZip alone and 94.65% at 2,128 tokens for GroupRoute alone, supporting the complementarity of the two stages.

        Table G3. Component comparison on LLaVA-OneVision-7B. Visual tokens are averaged over all decoder layers. GroupRoute alone includes Late Entry/Early Exit and RGS but disables GeoSemZip.

        ConfigurationProjector keep ratioAvg. active visual tokens/layerScanQA CIDErScanQA EMSQA3D EMAvg. (%)
        GeoSemZip only10%62887.4425.4156.6994.00
        GroupRoute only100%2,12888.4826.0755.8494.65
        GeoGR (both)30%63888.2526.2056.6195.18
AuthorsOpenReview ↗

Global Response (Part 3/3)

  • Technical Contributions and Novelty (Reviewers: uJC1, woYf) (continued)
    • Open Post-training Dataset for 3D VLMs: We will publicly release our 67K-sample dataset for compression-aware post-training of 3D VLMs. Using PRISM [7], we select the least redundant 30% of examples from each of five existing training sources, covering 3D question answering, captioning, and grounding. Dense models post-trained on this subset retain 95.69%/98.72% of their full-data post-training performance on LLaVA-OneVision-7B and Video-3D LLM-7B, respectively, supporting reproducible adaptation with far less data (Table 1; Appendix D.2, Table 12).

[2] Liu Z, Tang H, Lin Y, et al. Point-voxel cnn for efficient 3d deep learning[J]. Advances in neural information processing systems, 2019, 32.

[3] Graham B, Engelcke M, Van Der Maaten L. 3d semantic segmentation with submanifold sparse convolutional networks[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Ieee, 2018: 9224-9232.

[4] Chen Y, Liu J, Zhang X, et al. Voxelnext: Fully sparse voxelnet for 3d object detection and tracking[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023: 21674-21683.

[5] Huang H W, Chen F C, Chai W, et al. Zero-shot 3d question answering via voxel-based dynamic token compression[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 19424-19434.

[6] Wu H, Fan Y, Dai J, et al. Hidrop: Hierarchical vision token reduction in mllms via late injection, concave pyramid pruning, and early exit[C]//International Conference on Learning Representations. 2026, 2026: 97296-97316.

[7] Bi J, Jin Z, Wang Y, et al. Prism: Self-pruning intrinsic selection method for training-free multimodal data selection[C]//Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing. 2026: 14557-14580.

  • Cross-Model Validation and Scope of Conclusions (Reviewer uJC1, Q2rg)

    • Different Spatial Representations and Training Backgrounds (Reviewer uJC1):

      • Both models use SigLIP and Qwen2-7B, but differ in their native spatial representations. LLaVA-OneVision [8] independently encodes each RGB view and feeds the resulting visual tokens into the LLM without explicit 3D positional embeddings. Video-3D LLM [9] uses depth maps, camera intrinsics, and poses to map visual patches into a shared 3D coordinate system, then fuses their 3D positional embeddings with visual features before passing them to the LLM. In both settings, GeoSemZip uses geometry to consolidate cross-view tokens within voxels and select contextual anchors for spatial coverage.
      • LLaVA-OneVision is originally trained for single-image, multi-image, and video understanding and receives additional 3D task post-training in our experiments. Video-3D LLM builds on LLaVA-Video-7B and is jointly fine-tuned on 3D question answering, captioning, and grounding. We evaluate Video-3D LLM on all five benchmarks and LLaVA-OneVision on ScanQA and SQA3D, as LLaVA-OneVision does not natively support object-conditioned 3D captioning or 3D visual grounding. GeoGR retains 95.2% of Dense performance on LLaVA-OneVision and 96.8% on Video-3D LLM, measured on their respective benchmark suites, supporting its effectiveness in both evaluated spatial-encoding settings.
    • Cross-Model Validation of Recoverable Skipping (Reviewer Q2rg): We extend the original comparison on LLaVA-OneVision (revised Table 7) to Video-3D LLM. With average active visual token budgets matched within each model, we assess the incremental benefit of recoverable skipping in the full pipeline with GeoSemZip and Late Entry/Early Exit. RGS improves performance retention over the best tested permanent-pruning configuration by 0.15 points on LLaVA-OneVision and 0.42 points on Video-3D LLM (Table G4), supporting its benefit in both evaluated spatial-encoding settings. For RGS-only ablations, please refer to Table G2.

      Table G4. Comparison of recoverable skipping and permanent pruning within the full compression pipeline. Average active visual token budgets are matched within each model: approximately 638 tokens per layer on LLaVA-OneVision and 671 on Video-3D LLM. Scores report average performance retention relative to each Dense model (100%).

      MethodAnchorsLLaVA-OneVision-7B Avg. (%)Video-3D LLM-7B Avg. (%)
      Permanent pruning994.6595.75
      Permanent pruning1395.0396.38
      Permanent pruning9, 1394.7396.26
      RGS (ours)9, 1395.1896.80

[8] Li B, Zhang Y, Guo D, et al. Llava-onevision: Easy visual task transfer[J]. arXiv preprint arXiv:2408.03326, 2024.

[9] Zheng D, Huang S, Wang L. Video-3d llm: Learning position-aware video representation for 3d scene understanding[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 8995-9006.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.