acceptodds
Under review as a conference paper at ICLR 2027

Geometrize What Really Matters: Toward Lighter and Faster Spatial MLLMs

Abstract

Spatial multimodal large language models augment vision-language reasoning with explicit geometric representations, enabling understanding of object locations, distances, directions, and cross-view relations. This capability, however, introduces redundancy at two levels: dense multi-view observations must first pass through expensive geometric encoders, and the resulting geometry-enhanced visual tokens then increase language-model prefill computation and KV-cache storage. Existing pruning methods typically optimize only the latter stage after geometry has already been computed, while geometric diversity alone may fail to preserve evidence that is critical to the frozen model. We present GRAFT (Geometry Routing Across Frames and Tokens), a training-free hierarchical routing framework that allocates geometric computation and evidence along the inference pipeline. Before full geometric encoding, GRAFT uses shallow features from a frozen geometry backbone to select views with reliable cross-view relations and temporal coverage, reducing the observations processed by the full geometry encoder. After visual-geometric encoding, it estimates model sensitivity to construct a compact semantic evidence set and exploits aligned 3D coordinates to correct spatially invalid token substitutions under an exact budget. Extensive experiments on three benchmarks demonstrate that GRAFT consistently maintains competitive spatial reasoning accuracy while reducing both geometric encoding and language inference costs. On VSI-Bench, retaining only 5% of visual tokens preserves 94% of the accuracy, while reducing inference latency by 82% and peak memory usage by 55%. Anonymously released code: https://anonymous.4open.science/r/graft-88DA

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Spatial MLLM, Spatial Understanding, Token Pruning, Frame Selection, Model Lightweighting

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_Gb2pOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GRAFT, a training-free framework for improving the efficiency of spatial multimodal large language models through frame selection and visual token pruning. It addresses computational costs at two stages: geometric encoding of candidate views and language-model processing of geometry-enhanced visual tokens. Before full geometric encoding, GRAFT uses shallow features from a frozen VGGT backbone to select frame pairs within temporal groups, combining reciprocal feature matching with image-plane displacement to balance correspondence reliability, viewpoint variation, and temporal coverage. After encoding the selected frames, it constructs an initial token set using semantic coverage and token importance estimated through perturbations of the frozen PatchMerger. It then refines this set using 3D coordinates, accepting token swaps that reduce spatial mismatch while preserving the exact token budget and keeping both mean and worst-case semantic coverage errors within their initial bounds. The selected original fused tokens are passed directly to the language model, without updating model parameters or modifying their feature values.

Critical strengths
  1. The framework addresses both geometric encoding and language-model computation through a coherent two-stage design. Selecting frames before full geometry estimation reduces upstream costs, while subsequent token selection shortens the language-model input. This extends efficiency improvements beyond downstream token pruning and targets both major computational bottlenecks in the evaluated system.
  2. The geometric refinement formulation explicitly balances spatial consistency with semantic coverage. It evaluates spatial mismatch through semantic representative assignments and accepts token exchanges only when both mean and worst-case semantic coverage constraints are satisfied. This provides a principled refinement procedure that preserves the token budget while improving the defined geometric objective.
  3. GRAFT operates entirely within a frozen model, requiring no additional training and preserving the selected token representations. The paper also explains how token ordering, attention masks, and positional information remain aligned after pruning. These integration details make the approach practical to implement and support reproducibility beyond the description of its selection criteria.
  4. The experimental evaluation supports the usefulness of the design under aggressive compression. Results across spatial reasoning and question-answering benchmarks, together with an additional backbone evaluation, demonstrate its practical potential. Component ablations further support the contributions of frame selection and constrained geometric refinement, providing evidence for the main methodological choices.
Critical weaknesses
  1. Does the proposed frame-routing stage offer a sufficiently better performance–cost trade-off than uniform frame sampling followed by the same token-routing procedure? The existing ablation suggests a modest performance gain, but the cost of obtaining that gain remains unclear.
  2. What accounts for Stage II’s efficiency advantage over other methods at the same token budget? Since GRAFT performs full visual and geometric encoding before token selection, it is unclear which operations produce the reported savings beyond shorter LLM inputs.
  3. How does GRAFT’s methodological contribution differ from the closest existing mechanisms? In particular, what does its initialization add to ZOO-Prune’s sensitivity estimation and sensitivity–diversity selection, beyond temporal grouping? A targeted comparison with CoverPrune or 3DZip would help establish the value of hard semantic constraints over joint semantic–geometric objectives.
  4. Do the proposed semantic and geometric objectives reliably capture the evidence needed for spatial reasoning? The refinement guarantees concern feature-space coverage and predicted coordinates; they do not directly establish preservation of question-relevant information or physically valid representative relationships. Could the authors examine how often exchanges occur, how much they improve the geometric objective, and whether those improvements correspond to better answers? Such analysis would strengthen the proposed explanation beyond the observed task gains.
Questions for authors
  1. Could you report total latency and peak memory for the “uniform sampling + token routing” configuration in Table 4 under the same measurement protocol as full GRAFT? The existing results already establish the performance difference; the missing cost comparison would clarify when Stage I’s benefit justifies its additional computation. This would directly evaluate the proposed frame-routing stage without introducing another selection method or benchmark.
  2. Could you provide a Stage II timing and FLOP breakdown at a representative low-token budget, separating visual and geometric encoding, sensitivity estimation, geometric refinement, and LLM prefill? Please specify whether intermediate features are reused across stages and whether all routing operations are included. What explains the substantially larger reduction in TTFT than in LLM prefill time? Providing the same breakdown for one competing method with similar prefill latency would help determine whether the reported execution paths are comparable.
  3. Could you clarify how the initialization in Equations (7)–(9) relates to ZOO-Prune, particularly whether the no-refinement configuration in Table 6 corresponds to its selection rule applied within GRAFT’s temporal groups?
Reviewer_5WRsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduce a method called GRAFT, a training-free inference acceleration framework for spatial multimodal language models. It introduces two stages of routing. First, shallow VGGT features are used to select temporally distributed frame pairs based on reciprocal feature correspondences, similarity, and image-plane displacement, reducing the need for full geometric processing during candidate-frame selection. Second, after encoding the selected frames, GRAFT constructs a compact token set using PatchMerger sensitivity and semantic coverage. It then performs budget-preserving token swaps to reduce geometric mismatch while keeping the defined mean and worst-case semantic coverage errors within their initialization bounds. The selected original fused tokens are passed to the language model without additional feature aggregation. Experiments on three spatial reasoning benchmarks report strong accuracy retention across multiple compression budgets, together with reductions in preprocessing-plus-TTFT and peak memory.

Critical strengths
  • The training-free framework addresses both expensive frame selection and long visual-token sequences.

  • It retains strong accuracy under aggressive compression, with results on three benchmarks and useful component ablations.

Critical weaknesses
  • The reported Stage II speedup needs clearer explanation, especially when competing methods have similar LLM prefill times.

  • Uniform sampling with the same token routing achieves nearly the same accuracy, but its total computational cost is not reported.

  • The independent benefit of PatchMerger sensitivity is not isolated, and its perturbation settings lack sensitivity analysis.

Questions for authors
  • What accounts for the additional Stage II efficiency gains beyond the reduction in LLM prefill?

Could you provide module-level latency and FLOPs for the semantic encoder, VGGT aggregation, geometry prediction heads, sensitivity estimation, token routing, and LLM prefill in Table 3 and Figure 3? Please clarify input shapes, module invocation counts, feature caching or reuse across stages, and the boundaries used for each timing measurement.

  • How does full GRAFT compare with uniform sampling plus the same token routing in total cost?

Could you report accuracy, total preprocessing-plus-TTFT, and peak memory for uniform sampling plus token routing, the default sampler plus token routing, and full GRAFT under matched settings? Given the 49.6 versus 50.1 scores in Table 4, this comparison would clarify the practical benefit of the proposed frame-selection stage.

  • How does performance change with uniform sensitivity weights or different perturbation settings?

Why was the perturbation parameter set to 0.01, and is it stable across different values?

If this method works, it's worth trying 128 frames directly (or even the original video) instead of sticking to 16 frames. Could you run a comparison on the 128-frame setup?

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to Reviewer BYEF

[Q1, W1] How does the model perform when the geometry branch is disabled and only semantic tokens are retained?

We thank the reviewer for raising this important question. We agree that demonstrating the contribution of geometric information to spatial reasoning requires evidence beyond the high accuracy achieved with fused tokens at high compression ratios. We therefore conducted an additional experiment in which we disabled the VGGT geometry branch and used only the Qwen2.5-VL visual encoder to extract patch features. During inference, we bypassed the VGGT forward pass and semantic–geometric fusion and fed these semantic tokens directly to the LLM as visual input. The results are shown below.

MethodChoiceNumer.Avg. Score
Baseline K=3168K=316852.152.652.4
GRAFT K=160K=16050.250.150.1
GRAFT with the geometry branch disabled K=160K=16046.348.447.4

Full GRAFT scores 50.1 while retaining only 160 tokens, and the score drops to 47.4 when the geometry branch is disabled. For Room Size, the scores with full GRAFT and with the geometry branch disabled are 42.3 and 39.8, respectively. For Relative Direction, the corresponding scores are 54.9 and 40.4. These results provide direct evidence that our method efficiently retains geometric information for spatial reasoning at high compression ratios.

[Q2, W2] What are the performance and efficiency advantages of Frame Routing over the official Spatial-MLLM sampler?

We thank the reviewer for pointing out the choice of baseline for the frame routing comparison in the original manuscript. Our claim for shallow frame routing in Stage I under the full-token setting concerns its ability to reduce computational cost while preserving as much useful spatial evidence as possible. Specifically, Stage I reduces geometric encoding latency from 14.80 s to 2.41 s and computational cost by 85.6%. More importantly, at K=160, the overall score rises from 49.3 with the official sampler combined with Stage II to 50.1 with full GRAFT. These results show that the primary benefit of shallow frame routing is a better performance–efficiency trade-off.

Both selection strategies use the same pre-cached pool of 128 candidate frames and the same inference configuration. They differ only in the frame selection mechanism. The official sampler performs full geometric encoding before selecting views, whereas GRAFT uses shallow cross-view features from VGGT for frame routing. Both strategies therefore use the same base model, number of candidate frames, and final frame budget, allowing a direct comparison of the computational costs and performance of the two frame selection mechanisms.

[Q3] Can GRAFT transfer effectively to VLM-3R and retain spatial reasoning performance?

We thank the reviewer for the valuable suggestion regarding cross-model validation. To further evaluate the generalizability of GRAFT, we conducted additional experiments on VLM-3R and compared it with two competitive methods, Geo3DPruner and ScoRe. The results are summarized below:

MethodInput FramesAvg. Score
VLM-3R32 frames, K=6,27260.9
VLM-3R + GRAFT16 frames, K=32056.04
VLM-3R + Geo3DPruner16 frames, K=32046.32
VLM-3R + ScoRe16 frames, K=32052.26

The results demonstrate that GRAFT effectively transfers to VLM-3R, retaining approximately 92% of the original model's overall performance with only 5% of the visual tokens. Under identical frame and token budgets, GRAFT outperforms Geo3DPruner and ScoRe by 9.7 and 3.8, respectively. These findings highlight GRAFT's generalizability across spatial MLLMs and its ability to preserve spatial reasoning performance under aggressive token compression.

AuthorsOpenReview ↗

Follow-up on Our Response to Reviewer BYEF

Dear Reviewer BYEF,

Thank you again for your constructive suggestions. Our response reports a geometry-branch ablation at K=160, clarifies the comparison with the official sampler, and adds VLM-3R experiments with matched frame and token budgets across pruning methods.

We would appreciate your thoughts on this additional evidence and any remaining concerns about geometry use or transferability. If these results resolve your main concerns, we would be grateful if you could reconsider your assessment and rating.

Best regards,

The Authors

The rebuttal addresses several of my concerns. The results suggest that the contribution of geometry to spatial reasoning is limited, while the efficiency improvements remain useful. Overall, I find the paper’s contribution meaningful, and I will maintain my score of Weak Accept.

AuthorsOpenReview ↗

Summary of Rebuttal

Dear Area Chair,

Thank you for overseeing our submission. We summarize GRAFT's contributions and the evidence added during rebuttal, including our new responses to Reviewer 5WRs.

GRAFT formulates efficient spatial MLLM inference as hierarchical geometric-resource allocation across views and tokens. It combines shallow cross-view selection before full geometric encoding with geometric token refinement under explicit semantic coverage constraints. This addresses both upstream geometry computation and downstream language-model input, while keeping model parameters and retained token representations unchanged. Across three benchmarks and four token budgets, GRAFT and its Stage-II-only variant rank among the top two configurations. On VSI-Bench, GRAFT retains 94% of the dense model's score with 5% of its visual tokens, scoring 50.1 versus 46.9 for the strongest compared pruner.

  1. Frame-routing value and variability (5WRs, Gb2p, BYEF). We compared all three samplers using the same 128 candidates, 16 selected frames, and downstream router at K=160. Total model latency is 16.25 s for the default sampler, 1.44 s for uniform sampling, and 2.87 s for GRAFT. Across three seeds on full VSI-Bench, GRAFT scores 50.04±0.39 versus 49.22±0.35 for uniform sampling (mean±sample SD). Uniform remains the fastest option; GRAFT provides a higher mean score for 1.43 s additional latency, while reducing latency by approximately 82% relative to the default sampler. Feeding all 128 frames into Stage II scores 49.03 versus 50.1 for selected views at the same token budget; 5WRs explicitly considered this question resolved.
  2. Efficiency accounting (5WRs, Gb2p). We provided module-level profiling and clarified that Stage I includes the 0.99 s required to construct the selected frames' geometry cache. Including cache construction and the 197 ms sensitivity step, the sum of stage medians gives 2.86 s for GRAFT versus 16.04 s for dense Spatial-MLLM. The comparison with VisionTrim shows nearly identical LLM prefill times; the Stage II latency gap primarily reflects cross-stage geometry reuse in the measured implementations. We clarified this boundary and acknowledge that comparing router costs under identical caching requires separate profiling.
  3. Sensitivity contribution and transfer (5WRs). The requested uniform-weight ablation fixes frames, token budget, temporal groups, and refinement rules: sensitivity weighting improves VSI-Bench from 46.3 to 50.1. The same h=0.01 was fixed before evaluation and used across VSI-Bench, ScanQA, and SQA3D, as specified in the paper. We also corrected the interpretation of the step-size sweep: above h=0.01, larger steps reduce multiple-choice accuracy while numerical scores increase.
  4. Contribution beyond sensitivity-based initialization (Gb2p). We explicitly acknowledge that initialization builds on ZOO-Prune. GRAFT adds temporal balancing and geometric refinement, keeping the sensitivity-weighted mean semantic coverage error and worst-case radius within their initialization bounds. Table 6 reports 48.0 without refinement, 48.3 with unconstrained swaps, and 50.1 with both constraints. This supports the value of preserving semantic coverage when improving spatial representativeness. Additional logs show accepted swaps in all 12 analyzed scenes, averaging 14.17 per scene.
  5. Geometry use and broader applicability (BYEF). Disabling the geometry branch at K=160 reduces VSI-Bench from 50.1 to 47.4, including a drop from 54.9 to 40.4 on Relative Direction. Beyond the paper's VG-LLM evaluation, added VLM-3R experiments yield 56.04 for GRAFT versus 52.26 for SCoRe and 46.32 for Geo3DPruner at matched 16-frame/320-token budgets. These results support useful geometric evidence retention under compression and transfer beyond Spatial-MLLM. BYEF subsequently acknowledged that the rebuttal addressed several concerns and maintained Weak Accept.

We thank the Area Chair for their time and thoughtful consideration. We respectfully ask that the paper’s contributions be assessed in light of the additional experiments and clarifications provided during rebuttal.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.