- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GRAFT, a training-free framework for improving the efficiency of spatial multimodal large language models through frame selection and visual token pruning. It addresses computational costs at two stages: geometric encoding of candidate views and language-model processing of geometry-enhanced visual tokens. Before full geometric encoding, GRAFT uses shallow features from a frozen VGGT backbone to select frame pairs within temporal groups, combining reciprocal feature matching with image-plane displacement to balance correspondence reliability, viewpoint variation, and temporal coverage. After encoding the selected frames, it constructs an initial token set using semantic coverage and token importance estimated through perturbations of the frozen PatchMerger. It then refines this set using 3D coordinates, accepting token swaps that reduce spatial mismatch while preserving the exact token budget and keeping both mean and worst-case semantic coverage errors within their initial bounds. The selected original fused tokens are passed directly to the language model, without updating model parameters or modifying their feature values.
Critical strengths
- The framework addresses both geometric encoding and language-model computation through a coherent two-stage design. Selecting frames before full geometry estimation reduces upstream costs, while subsequent token selection shortens the language-model input. This extends efficiency improvements beyond downstream token pruning and targets both major computational bottlenecks in the evaluated system.
- The geometric refinement formulation explicitly balances spatial consistency with semantic coverage. It evaluates spatial mismatch through semantic representative assignments and accepts token exchanges only when both mean and worst-case semantic coverage constraints are satisfied. This provides a principled refinement procedure that preserves the token budget while improving the defined geometric objective.
- GRAFT operates entirely within a frozen model, requiring no additional training and preserving the selected token representations. The paper also explains how token ordering, attention masks, and positional information remain aligned after pruning. These integration details make the approach practical to implement and support reproducibility beyond the description of its selection criteria.
- The experimental evaluation supports the usefulness of the design under aggressive compression. Results across spatial reasoning and question-answering benchmarks, together with an additional backbone evaluation, demonstrate its practical potential. Component ablations further support the contributions of frame selection and constrained geometric refinement, providing evidence for the main methodological choices.
Critical weaknesses
- Does the proposed frame-routing stage offer a sufficiently better performance–cost trade-off than uniform frame sampling followed by the same token-routing procedure? The existing ablation suggests a modest performance gain, but the cost of obtaining that gain remains unclear.
- What accounts for Stage II’s efficiency advantage over other methods at the same token budget? Since GRAFT performs full visual and geometric encoding before token selection, it is unclear which operations produce the reported savings beyond shorter LLM inputs.
- How does GRAFT’s methodological contribution differ from the closest existing mechanisms? In particular, what does its initialization add to ZOO-Prune’s sensitivity estimation and sensitivity–diversity selection, beyond temporal grouping? A targeted comparison with CoverPrune or 3DZip would help establish the value of hard semantic constraints over joint semantic–geometric objectives.
- Do the proposed semantic and geometric objectives reliably capture the evidence needed for spatial reasoning? The refinement guarantees concern feature-space coverage and predicted coordinates; they do not directly establish preservation of question-relevant information or physically valid representative relationships. Could the authors examine how often exchanges occur, how much they improve the geometric objective, and whether those improvements correspond to better answers? Such analysis would strengthen the proposed explanation beyond the observed task gains.
Questions for authors
- Could you report total latency and peak memory for the “uniform sampling + token routing” configuration in Table 4 under the same measurement protocol as full GRAFT? The existing results already establish the performance difference; the missing cost comparison would clarify when Stage I’s benefit justifies its additional computation. This would directly evaluate the proposed frame-routing stage without introducing another selection method or benchmark.
- Could you provide a Stage II timing and FLOP breakdown at a representative low-token budget, separating visual and geometric encoding, sensitivity estimation, geometric refinement, and LLM prefill? Please specify whether intermediate features are reused across stages and whether all routing operations are included. What explains the substantially larger reduction in TTFT than in LLM prefill time? Providing the same breakdown for one competing method with similar prefill latency would help determine whether the reported execution paths are comparable.
- Could you clarify how the initialization in Equations (7)–(9) relates to ZOO-Prune, particularly whether the no-refinement configuration in Table 6 corresponds to its selection rule applied within GRAFT’s temporal groups?
Response to Reviewer BYEF
[Q1, W1] How does the model perform when the geometry branch is disabled and only semantic tokens are retained?
We thank the reviewer for raising this important question. We agree that demonstrating the contribution of geometric information to spatial reasoning requires evidence beyond the high accuracy achieved with fused tokens at high compression ratios. We therefore conducted an additional experiment in which we disabled the VGGT geometry branch and used only the Qwen2.5-VL visual encoder to extract patch features. During inference, we bypassed the VGGT forward pass and semantic–geometric fusion and fed these semantic tokens directly to the LLM as visual input. The results are shown below.
Full GRAFT scores 50.1 while retaining only 160 tokens, and the score drops to 47.4 when the geometry branch is disabled. For Room Size, the scores with full GRAFT and with the geometry branch disabled are 42.3 and 39.8, respectively. For Relative Direction, the corresponding scores are 54.9 and 40.4. These results provide direct evidence that our method efficiently retains geometric information for spatial reasoning at high compression ratios.
[Q2, W2] What are the performance and efficiency advantages of Frame Routing over the official Spatial-MLLM sampler?
We thank the reviewer for pointing out the choice of baseline for the frame routing comparison in the original manuscript. Our claim for shallow frame routing in Stage I under the full-token setting concerns its ability to reduce computational cost while preserving as much useful spatial evidence as possible. Specifically, Stage I reduces geometric encoding latency from 14.80 s to 2.41 s and computational cost by 85.6%. More importantly, at K=160, the overall score rises from 49.3 with the official sampler combined with Stage II to 50.1 with full GRAFT. These results show that the primary benefit of shallow frame routing is a better performance–efficiency trade-off.
Both selection strategies use the same pre-cached pool of 128 candidate frames and the same inference configuration. They differ only in the frame selection mechanism. The official sampler performs full geometric encoding before selecting views, whereas GRAFT uses shallow cross-view features from VGGT for frame routing. Both strategies therefore use the same base model, number of candidate frames, and final frame budget, allowing a direct comparison of the computational costs and performance of the two frame selection mechanisms.
[Q3] Can GRAFT transfer effectively to VLM-3R and retain spatial reasoning performance?
We thank the reviewer for the valuable suggestion regarding cross-model validation. To further evaluate the generalizability of GRAFT, we conducted additional experiments on VLM-3R and compared it with two competitive methods, Geo3DPruner and ScoRe. The results are summarized below:
The results demonstrate that GRAFT effectively transfers to VLM-3R, retaining approximately 92% of the original model's overall performance with only 5% of the visual tokens. Under identical frame and token budgets, GRAFT outperforms Geo3DPruner and ScoRe by 9.7 and 3.8, respectively. These findings highlight GRAFT's generalizability across spatial MLLMs and its ability to preserve spatial reasoning performance under aggressive token compression.