acceptodds
Under review as a conference paper at ICLR 2027

Token-Adaptive Routing of Multi-Level Geometry for Spatial Reasoning

Abstract

Spatio-temporal reasoning in vision-language models requires visual representations that preserve physical geometry rather than merely semantic appearance. Recent multimodal models incorporate geometric information through structural branches, 3D-aware supervision, reasoning-stage fusion, or long-horizon memory. While these approaches demonstrate the importance of geometry for spatial intelligence, many combine geometric features with visual representations through shared or stage-wise rules. This leaves a finer-grained question: which levels of geometric evidence should each visual token use for its spatial context? To address this question, we introduce GeoWeaver, a token-adaptive geometric grounding framework that selects multi-level evidence before language decoding. GeoWeaver constructs a multi-level geometry bank from a frozen geometry encoder, and each visual token sparsely selects evidence from this bank. The selected evidence is incorporated into visual tokens via a residual grounding operation prior to language modeling, yielding geometry-grounded representations for downstream reasoning. Across six reported spatial reasoning benchmarks, GeoWeaver attains a 64.8 mean. In a matched 32-frame ReVSI comparison, it exceeds SpatialStack by 2.9 points, while its four-task general multimodal mean improves on the base model (68.98 versus 68.41). On VSI-Bench, token-adaptive allocation reaches 72.7, compared with 71.30 for global layer weighting and 70.44 for uniform averaging, supporting the value of selecting geometric evidence at token granularity. Code and models will be released.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Spatio-temporal reasonin, Token-adaptive geometric grounding, Multi-level geometry bank

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Q7bXOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces GeoWeaver, a token-adaptive geometric grounding framework for spatio-temporal reasoning. It uses a frozen geometry encoder to construct a multi-level feature bank, from which a token-level router selects and combines two spatially aligned geometric features for each visual token. The selected features are integrated into visual representations before language decoding, enabling different regions to leverage different levels of geometric information. GeoWeaver outperforms SpatialStack on VSI-Bench and under a matched 32-frame ReVSI setting. Ablation studies further show that token-adaptive selection provides modest improvements over uniform averaging and global layer weighting.

Critical strengths
  1. A clear and well-motivated mechanism. The paper investigates whether different visual tokens benefit from features extracted at different layers of the geometry encoder, with controlled comparisons against uniform averaging and global layer weighting. Ablations help isolate the contribution of token-adaptive selection.

  2. Strong results against a closely related baseline. GeoWeaver outperforms SpatialStack on VSI-Bench and all seven ReVSI subtasks under a matched 32-frame setting. The shared backbone and geometry encoder, along with closely aligned training recipes, make the comparison particularly meaningful for evaluating the overall methods.

Critical weaknesses
  1. The contribution of token-adaptive routing remains unclear. GeoWeaver's improvements over SpatialStack involve changes in both the fusion stage and layer selection, while the direct evidence for routing is mainly a 1.4-point gain over global weighting on VSI-Bench. Thus, although the overall framework is effective, the specific contribution of token-adaptive selection remains difficult to isolate.

  2. Limited evidence of generalizability and advantages over prior methods. Routing ablations are restricted to VSI-Bench, while the six-benchmark evaluation covers only the full framework. The paper could better clarify its contributions relative to SpatioLM [1], which leverages intermediate visual features, and GeoThinker [2], which selectively integrates geometric information. Given differences in backbones and evaluation settings, the comparison should emphasize methodological distinctions rather than raw performance differences.

[1] Wu et al., "SpatioLM: Towards General Physical Spatial Intelligence in Vision-Language Models"

[2] Li et al., "Thinking with Geometry: Active Geometry Integration for Spatial Reasoning"

Questions for authors
  1. While GeoWeaver is evaluated on ReVSI and SPAR-Bench, the benefit of token-adaptive selection over global layer weighting is demonstrated only on VSI-Bench. Does token-adaptive routing yield similar improvements on ReVSI and SPAR-Bench?

  2. Minor presentation issues. On page 2, “The main contributions of this work are:” appears twice consecutively. In Table 5, the Avg. column uses inconsistent decimal precision (e.g., 69.725 versus mostly one or two decimal places).

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Dear Reviewer Q7bX,

We thank you for the thoughtful review and for recognizing token-adaptive selection.

W1. “contribution of token-adaptive routing”

We address this through the motivation for regional geometry selection and experiments that isolate its contribution. VGGT depths respond differently to the same scene (Fig. 1(b)), while native 2D visual tokens describe different image regions. This motivates learning a separate 2D–3D coupling for each region, which GeoWeaver implements by using the native visual token to select and combine two aligned features from twelve frozen VGGT layers through a weighted residual update. The joint scene representation then gives LLM attention visual semantics and geometry to use together with the language question.

Token-wise and shared Top-2 use the same fusion stage and two-layer budget, directly testing whether each region benefits from its own choice. Shared Top-2 averages twelve-way probabilities over the input's visual tokens and frames, selects the two largest entries and renormalizes their weights for all locations.

BenchmarkToken Top-2Shared Top-2Gain
VSI-Bench72.6970.66+2.03
ReVSI, 32 frames55.0553.02+2.03
SPAR-Bench69.0867.01+2.07

All eight VSI categories improve, led by route planning (+3.14) and relative distance (+2.41).

The complete grounding operation also raises native-token pooled R@1 from 20.75% to 24.34% on the cross-view retrieval probe over 30 ScanNet scenes. Together, the controlled allocation gains and improved geometric correspondence support adaptive geometry as a way to strengthen the joint 2D–3D representation used by the LLM for spatial understanding.

W2/Q1. “similar improvements on ReVSI and SPAR-Bench”

To test whether the routing benefit extends beyond VSI-Bench, we compare token-wise selection with shared dense and uniform aggregation on ReVSI and SPAR-Bench. Shared dense uses input-averaged router probabilities as twelve weights shared by all visual positions, while uniform averaging gives all twelve layers equal weight.

BenchmarkToken Top-2Shared denseUniformTop-2 minus shared dense
ReVSI, 32 frames55.0553.2453.56+1.81
SPAR-Bench69.0867.2467.46+1.84

The gains against shared Top-2 and all-layer aggregation show that region-specific allocation is useful under both sparse and dense alternatives. This extends the benefit of adaptive geometric coupling beyond VSI-Bench.

W2. “methodological distinctions rather than raw performance differences”

The key differences are the geometry source, selection granularity, and fusion stage. SpatioLM constructs geometry-aware representations from intermediate native-vision features through its Spatio-Vision Module and injects them into LLM blocks. GeoThinker uses frame-constrained cross-attention and importance gating inside selected decoder blocks. GeoWeaver instead uses each native visual token to choose aligned features across the depths of an external frozen VGGT bank before language decoding, so its selection determines which geometric evidence each region receives and incorporates into its visual representation.

Q2. “Minor presentation issues”

We removed the duplicated contribution lead-in on page 2. The decoder-side fusion Avg. in the revised PDF is 69.73, the eight-category mean (69.725) rounded to two decimal places.

If you have any other concerns, please let us know. Thanks for your time again.

Reviewer_Q7bXOpenReview ↗

Reply

Thank you for the additional experiments and clarifications. My concerns have been addressed. I recommend including these additions in the final version, and I maintain my positive score.

AuthorsOpenReview ↗

Comment by authors

Dear Reviewer Q7bX,

We are pleased to hear that our response has adequately addressed your concerns. We will add all the experiments to the final version of the manuscript. We sincerely appreciate your dedicated time and effort in offering invaluable feedback. Please let us know if there are any aspects you would like us to clarify or improve further. We would be happy to provide additional explanations.

Best,

The authors

Reviewer_gL4EOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GeoWeaver is a geometric-grounding module designed to improve spatial reasoning in multimodal language models. A frozen VGGT encoder provides aligned features from several intermediate layers, after which a learned router selects and combines two geometry layers for each native visual token. The resulting representation is added through a residual update before language decoding, so the method does not lengthen the visual-token sequence. Reported results cover several spatial and general multimodal benchmarks, including strong gains on VSI-Bench, SPAR-Bench, and a frame-matched ReVSI comparison. Controlled experiments vary the geometry bank, its size, the number of selected layers, allocation granularity, and the integration design, while a cross-view correspondence probe examines whether the grounded tokens retain geometric information.

Critical strengths
  1. Sparse token-wise selection over a frozen, multi-level geometry bank is a simple and coherent design. It gives each spatial location its own layer mixture without increasing the visual-token sequence length, offering a clear alternative to applying one shared mixture everywhere.

  2. Breadth of evidence is a notable asset: the experiments span six spatial benchmarks, several ReVSI frame budgets, four general multimodal benchmarks, and two model scales. Many of the reported improvements are substantial rather than marginal.

  3. Allocation studies directly test the main routing claim. On VSI-Bench, token-adaptive allocation reaches 72.7, versus 71.30 for global layer weights and 70.44 for uniform averaging under the same grounding architecture. Selecting two layers also performs best among the tested bank sizes.

Critical weaknesses
  1. Run-to-run stability is not established because downstream results lack variation across independently trained seeds. This omission is especially important for the allocation experiment that isolates the main claimed contribution, where token-adaptive routing improves over global layer weighting by only 1.40 points.

  2. Question dependence is absent from the router even though the method is described as query-conditioned. Routing uses only the native visual token before the language question is introduced, so an image region receives the same geometry-layer selection for depth, direction, size, or temporal-order questions. This narrows the claimed form of adaptivity.

  3. Mechanistic attribution remains incomplete. Although the allocation-granularity ablation is informative, comparisons among input-level, decoder-side, and pre-reasoning designs jointly alter placement and the allocation rule, as the paper acknowledges. In addition, the routing visualizations cover only a few examples and do not reveal consistent functional roles for selected depths across tasks.

Questions for authors
  1. Which exact splits are used for SPAR and VSI appearance-order data during training and evaluation, and what scene-, video-, or question-level deduplication separates instruction tuning from the reported benchmarks?

  2. Why does routing depend only on the visual token instead of jointly incorporating the language instruction, and have the authors tested question-conditioned routing when one region supports different spatial queries?

  3. How much end-to-end latency and memory overhead does GeoWeaver add relative to the unaugmented Qwen3.5 base model once VGGT preprocessing and device transfer are included?

  4. When the geometry encoder, token grid, or visual domain changes, how well does the learned router transfer, and do its layer preferences remain meaningful without retraining the complete grounding module?

2 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Dear Reviewer gL4E,

We sincerely thank you for the thoughtful review.

W1/W3. “Run-to-run stability” / “Mechanistic attribution”

  • We examine both the source of the gain and its stability across independent training runs. Token Top-2 outperforms shared Top-2 when both use two layers at the same fusion stage. Shared Top-2 selects one pair from input-averaged routing probabilities, renormalizes its weights, and applies it across all tokens.
BenchmarkToken Top-2Shared Top-2Gain
VSI-Bench72.6970.66+2.03
ReVSI-3255.0553.02+2.03
SPAR-Bench69.0867.01+2.07

Token Top-2 improves all three benchmarks and all eight VSI categories. Three-seed stability (mean ± sample SD). We trained token Top-2, global weighting, and uniform averaging with seeds 0–2, changing data subsampling and router/projector initialization. Paired gains are calculated within each seed.

MethodVSI-BenchReVSI-32SPAR-Bench
Token Top-272.66 ± 0.2455.10 ± 0.2669.04 ± 0.23
Global weighting71.13 ± 0.1553.12 ± 0.3367.03 ± 0.14
Uniform averaging70.47 ± 0.4352.78 ± 0.3966.71 ± 0.21
Paired Top-2 − Global+1.53 ± 0.24+1.98 ± 0.23+2.01 ± 0.37

Token Top-2 − Global gains for each seed:

SeedVSI-BenchReVSI-32SPAR-Bench
0+1.40+1.71+2.03
1+1.38+2.10+1.63
2+1.81+2.12+2.36

All nine paired gains are positive. On seed-0 VSI, 10,000 paired-question bootstraps give 95% intervals [0.63, 2.14] versus global and [1.28, 3.19] versus uniform.

On 30 ScanNet scenes, Table S3 reports cross-view R@1 rising from 20.75% to 24.34% after grounding, with VGGT depths spanning 31.70%–43.36%. The regional choices in Fig. S3 further illustrate how depth preferences vary with local visual content. These results support enriching each visual region with adaptive geometry to improve spatial understanding, with gains that persist across independent training runs.

W2/Q2. “Why does routing depend only on the visual token”

Our goal is to build a joint 2D–3D scene representation, enriching visual semantics with geometric structure before question-specific reasoning. The router therefore selects geometric evidence for each native visual token (Eq. (5)). Joint visual/question routing was not evaluated. The LLM then combines the question with these enriched scene tokens (Eq. (10)), using both semantic and geometric evidence to understand the queried targets and their spatial relations. The routing “query” in Fig. 2 is the native visual token.

Q1. “Which exact splits are used”

SPAR instruction-tuning records and held-out SPAR-Bench tests are disjoint. Training used 140,566 SPAR-234K records (60% of that training source) and 1,909 VSI appearance-order records (50% of that source). The full appearance-order source spans 166 ScanNet videos and has no scene-ID or scan-family overlap with local VSI-Bench test annotations.

Q3. “end-to-end latency and memory overhead”

We assess practical overhead across the complete inference pipeline, including sampling, preprocessing, device transfer, VGGT where used, generation, and answer decoding. On one A800-80GB with BF16, batch size 1, and 32 output tokens, we measured warmed end-to-end latency and peak allocated GPU memory on matched 8- and 32-frame inputs (three videos, five repeats each).

Model8 frames (s / GiB)32 frames (s / GiB)
Qwen3.5-4B2.169 ± 0.205 / 8.9894.581 ± 1.747 / 10.382
SpatialStack-5B2.540 ± 0.199 / 15.5056.354 ± 1.730 / 22.088
GeoWeaver-5B2.528 ± 0.207 / 15.5056.445 ± 1.762 / 22.503

Relative to Qwen3.5-4B, GeoWeaver adds 0.359 s (+16.6%) and 6.516 GiB at 8 frames, and 1.864 s (+40.7%) and 12.121 GiB at 32 frames. Its measured end-to-end cost remains close to SpatialStack while allowing each region to select geometry evidence.

Q4. “how well does the learned router transfer”

We examine whether learned layer preferences remain useful when token grids and sampled frames change. Tests on 10 VSI-Bench clips and 22 questions covering all eight subtasks show that layer preferences remain broadly consistent.

PerturbationTop-2 overlap (%)Dense JS (nats)ΔQA macro (pp)
Resize ×0.9784.220.000410−3.33
Resize ×1.0384.330.000390−0.42
Resampling, changed frames77.190.000852+0.42

Top-2 overlap measures retained layers, and JS compares twelve-way probabilities, both averaged across videos. We align resized grids by normalized coordinates and resampled frames by optical flow (60.16% mean coverage), preserving 32 frames and temporal order.

The measured overlaps and benchmark gains show that learned depth preferences continue to guide local geometry selection after these visual changes.

If you have any other concerns, please let us know. Thanks again for your time

AuthorsOpenReview ↗

Kind reminder

Dear Reviewer gL4E,

Thank you again for your time and effort in reviewing our paper. We would like to kindly follow up on our rebuttal and hope that our responses have adequately addressed your concerns.

As the rebuttal discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our responses when convenient. If any questions or concerns remain, we would be more than happy to provide further clarification.

Thank you again for your valuable feedback and consideration!

Best regards,

The Authors

Reviewer_YbmcOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GeoWeaver adds geometry to a vision-language model's visual tokens before they reach the LLM. For each visual token, a lightweight router picks two of layers 12–23 of a frozen VGGT and weights them with a softmax. It then adds those features from the same position back to the token through a zero-initialized projection. The 5B model reports 72.7 on VSI-Bench, against 67.5 for SpatialStack-5B, 71.30 for global layer weighting and 70.44 for uniform averaging.

Critical strengths
  • a smooth storyline from vgllm -> spatialstack -> geoweaver, for single layer feature pickup to multilayer to adaptive multilayer. and the title "weave" is a straightforward naming.
  • clean experiment setup over the vsibench and clear enhancement.
Critical weaknesses

The routing story is incomplete.

  • The motivation for routing rests mainly on qualitative results. In contrast, works such as Perception Encoder (Meta, 2025) give a clear, quantified link between the type of vision task and which layer's features work best for it. This paper has no comparable profiling. It does not show which two layers the router typically selects for a given task, or how often each layer is selected for one type of QA within the test set.
  • The paper also lacks a sensitivity analysis of the router. A robust router should produce similar routing preferences as long as the input semantics do not change much. No experiment demonstrates this kind of determinism or robustness.
Questions for authors
  • profile the router preference or distribution for different subtasks
  • profile the router preference or distribution for the same VQA suite but with textual or visual perturbation, including but not limited to prompt rephrasing, frame resampling and rescaling.
2 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Dear Reviewer Ybmc,

We sincerely thank you for the careful review and questions about routing behavior.

W1/Q1. “preference or distribution for different subtasks”

  • We examine what different geometry depths provide and how the router selects among them. The router combines a common layer preference with regional variation: across 282 clips, L21+L23 is most frequent in every full QA group at 33.0–39.5% of tokens, while 33 different pairs occur overall (Table S9). Each clip counts once per group and can join several groups, which describe the associated visual inputs because routing does not read the question.

  • Figure 1(b) and Appendix Figure C show different spatial responses across depths, quantified through cross-view retrieval on 30 ScanNet scenes in Table S3. Across L12–L23, pooled R@1 ranges from 31.70% to 43.36%, with L14 best at R@1 (43.36%) and L23 best at R@5 (83.14%). Grounding raises native-token R@1 from 20.75% to 24.34%, showing that geometry enrichment improves the visual representation's correspondence information.

  • The selected examples below each pool 600 tokens from two frames in different videos, chosen for layer-pair diversity and regional contrast.

Selected-frame subtask examplesMost frequent pair (% of tokens)Most selected layers (Top-2 inclusion, %)
Object countingL12+L21 (24.0)L21 85.3, L12 38.2, L14 26.2
Absolute distanceL14+L21 (23.8)L21 91.7, L12 30.8, L14 30.3
Object sizeL12+L14 (20.7)L21 70.5, L12 47.7, L14 36.2
Room sizeL20+L21 (20.0)L21 73.0, L12 42.3, L20 30.8
Relative distanceL20+L21 (26.8)L21 73.8, L12 42.3, L20 41.8
Relative directionL12+L20 (21.0)L21 62.8, L12 51.7, L20 42.0
Route planningL21+L23 (25.3)L21 72.0, L12 38.2, L20 36.5
Appearance orderL20+L21 (23.5)L21 74.5, L12 41.7, L20 38.0
  • Figure S2 contrasts the full-group distributions and selected examples, and Figure S3 shows regional selection within frame E2. L20+L21 leads in the top-left quadrant at 60.0%, while L12+L21 leads in the bottom-right at 48.8%, directly illustrating different choices across one image.

  • We compare adaptive and shared Top-2 under the same fusion stage and two-layer budget. Shared Top-2 averages twelve-way probabilities across an input's tokens and frames, selects two layers, and renormalizes their weights for use at every location.

BenchmarkAdaptive Top-2Shared Top-2Gain
VSI-Bench72.6970.66+2.03
ReVSI-3255.0553.02+2.03
SPAR-Bench69.0867.01+2.07
  • These gains show that regional allocation improves reasoning at the same selection budget. Adaptive selection lets each native 2D visual token gather geometry for its region, enriching the representation that the LLM uses for spatial understanding and reasoning.

W1/Q2. “prompt rephrasing, frame resampling and rescaling”

  • We test whether routing preferences remain consistent when question wording or visual sampling changes. We evaluated 10 clips and 22 paired questions covering all eight subtasks. Because Eq. (5) routes from visual tokens alone, prompt rephrasing with identical processed visuals gives identical dense probabilities and ranked Top-2 choices across all 211,200 token comparisons.

  • Both resize settings change all 10 token grids while preserving the whole image and matching normalized spatial positions. Resampling retains 32 frames, temporal order, and appearance-order evidence, with forward/backward-consistent flow matches covering 60.16% of candidate tokens in changed frames on average.

Visual perturbationMean Top-2 overlapDense JS (nats)QA macro change (points)
Resize ×0.9784.22%0.000410−3.33
Resize ×1.0384.33%0.000390−0.42
Resampling, changed frames77.19%0.000852+0.42
  • Overlap counts shared selected layers divided by two, and JS compares the full twelve-way probabilities before selection, both averaged across videos. QA changes use the eight-category macro score over all 22 questions and the complete perturbed inputs. Dense depth preferences remain close under these measured changes, while local Top-2 choices adapt to the changed visual input. The results quantify routing sensitivity alongside the corresponding QA changes.

If you have any other concerns, please let us know. Thanks again for your time

AuthorsOpenReview ↗

Kind reminder

Dear Reviewer Ybmc,

Thank you again for your time and effort in reviewing our paper. We would like to kindly follow up on our rebuttal and hope that our responses have adequately addressed your concerns.

As the rebuttal discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our responses when convenient. If any questions or concerns remain, we would be more than happy to provide further clarification.

Thank you again for your valuable feedback and consideration!

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.