acceptodds
Under review as a conference paper at ICLR 2027

DK-VLM: Geometry as Position for Spatial Reasoning in Vision-Language Models

Abstract

Vision-language models still struggle to understand 3D spatial relations across views and update spatial judgments as the observer moves. A common approach fuses 3D reconstruction features into visual tokens, but aligning these features with pretrained semantic representations remains challenging under limited spatial supervision. We propose DK-VLM with Dual-Kernel Attention, which computes attention from both a visual token's input position and its geometric position in the scene. A semantic kernel retains the pretrained model's native attention and positional encoding, while a separate geometric kernel uses estimated 3D points and camera poses to directly modulate attention through point displacements and relative view transformations. Then, a learned gate fuses their outputs. Geometry thus enters attention as position rather than as fused reconstruction features. We further introduce Reference-Frame Composition (RC) supervision to train the model to update its reference frame across successive judgments. It provides stepwise turn and heading targets derived from verified single-step annotations. Among the compared methods, DK-VLM-8B achieves the highest average score across six spatial benchmarks and the highest overall scores on VSI-Bench (74.33) and ReVSI-32 (58.28). RC supervision further improves multi-step spatial reasoning, while the full model preserves general video understanding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
spatial reasoning, vision-language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_SwbdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DK-VLM, a geometry-aware vision-language model for spatial reasoning. Its Dual-Kernel Attention retains native semantic attention and positional encoding while introducing a separate geometric kernel conditioned on estimated 3D points and camera poses. Learned gates combine the two outputs. Training additionally incorporates pairwise geometric supervision and Reference-Frame Composition, which constructs stepwise turn and heading targets from existing annotations. Experiments on two Qwen backbones evaluate spatial reasoning, multi-step composition, and general video understanding.

Critical strengths
  1. Well-motivated geometric integration. The paper addresses a meaningful distinction between a visual token’s input position and its physical location across views. The parallel attention design offers a coherent way to incorporate point displacements and camera transformations while retaining the backbone’s native positional computation. Reference-Frame Composition provides complementary supervision for spatial judgments involving successive changes in perspective.

  2. Broad evaluation with useful controls. The experiments cover seven spatial benchmarks and two general video benchmarks. Beyond competitive benchmark results, the same-data SFT comparison and capacity-preserving identity-transform ablation provide useful evidence for the proposed geometric modeling: DK-VLM achieves 74.33 on VSI-Bench, versus 69.20 for the SFT baseline and 70.02 without geometric transforms. RC also improves several public route-planning and perspective-dependent tasks, supporting its value beyond the constructed chain evaluation.

Critical weaknesses
  1. Incremental methodological novelty. The geometric kernel builds directly on GTA, with Point GTA using patch-level scene points as translation anchors and Camera GTA using camera poses. The parallel branch, learned gating, and auxiliary geometric prediction objective provide a sensible adaptation to pretrained VLMs, but the underlying geometric operation is inherited. RC similarly reorganizes existing angular annotations into short supervision chains. The contribution therefore lies primarily in architectural integration and supervision construction, with a relatively modest conceptual advance over existing geometric attention.

  2. Unverified alignment premise. A central motivation is that reconstruction-feature fusion suffers from semantic–geometric representation misalignment, yet this limitation is not directly demonstrated. The benchmark comparisons and geometric ablations establish the usefulness of DK-VLM, but do not isolate its advantage over feature fusion under matched conditions. A comparison using the same backbone, geometry frontend, training data, and adaptation budget would better substantiate the proposed integration strategy. The existing evidence leaves the claimed alignment advantage insufficiently supported.

  3. Unisolated compositional benefits. RC produces encouraging improvements on public spatial tasks, but its specific contribution remains difficult to separate from additional task-relevant supervision. Adding 150K examples changes both training volume and exposure to sequential spatial questions. The backbone comparison does not isolate the importance of intermediate reference-frame targets. Controls using matched additional single-step data or the same chains with final-target supervision would clarify this distinction. Furthermore, the four-step evaluation follows the same construction procedure, providing limited evidence of compositional generalization beyond short turn and heading sequences.

  4. Unquantified computational cost. The small trainable-parameter fraction provides an incomplete picture of the method’s practical requirements. DK-VLM requires a geometry frontend and adds full attention over geometry-valid visual tokens at five decoder layers. These operations introduce costs that depend on the number of frames and visual tokens. Without end-to-end latency, peak memory, or inference FLOPs relative to competing approaches, it remains unclear how the spatial accuracy gains trade off against computational overhead and whether the approach scales practically to longer videos.

Questions for authors

See the weaknesses.

3 comments

W1: Novelty relative to GTA. GTA encodes camera transforms in attention; we keep this as Camera GTA and add Point GTA, which uses each patch's estimated 3D point as the translation anchor. Removing Point GTA costs 2.5 points on VSI-Bench (Table 6). Our main contribution is the adaptation you mention: GTA was designed for training from scratch, and in a pretrained VLM the same operator scores 39.14 when it replaces mRoPE inside the native attention, far below SFT at 59.48, but 62.78 as a separate branch beside it. The dual kernel injects geometry as positions in attention rather than through the feature fusion common in spatial VLMs, and in the matched comparison under W2 it outperforms feature fusion. Taking explicit points and poses as input also lets the geometry frontend be swapped without retraining.

RC does build on existing annotations and needs no relabeling. What it adds is a different target: each step is defined relative to the heading reached at the previous step, so a chain supervises successive reference-frame updates, which single-step questions do not. In the controlled experiment under W3, the same relations as single-step questions reach 2.3% chain accuracy, and RC raises it to 10.9%.

W2: Alignment premise. We ran the comparison you suggested, with the same backbone, frontend, data and training steps: DK-VLM scores 62.78 on VSI-Bench, feature fusion 61.33 and SFT 59.48 (table in our reply to 8Sow, W1/Q1).

W3: Isolating the compositional benefit. We ran the single-step control you suggested. On the same base data, we add the same number of single-step questions or RC chains, built from the same relations with the same answer format; the setup and tables are in our reply to 8Sow, W2/Q2. RC reaches 10.9% chain accuracy, against 2.3% for single-step data. The single-step model often gives the wrong number of steps, but even among answers of the correct length, RC is right twice as often. RC also scores highest on VSI-Bench and on route planning, which requires a sequence of turns.

Four steps is a length not seen in training. In the paper, RC reaches 20.6% chain accuracy on four-step chains, against 0.0% without RC; in the controlled experiment, only RC answers any four-step chain correctly, 4.4% against 0% for the other two arms. The four-step set uses the same construction as training and covers turn and heading targets, which we now state in Appendix A.8.

W4: Cost. One H100, each method's official protocol, 32 frames (Spatial-MLLM: 128 frames to VGGT, 16 selected for the LLM); reported VSI-Bench scores:

ModelVSI-BenchLatency (s)Memory (GiB)Memory growth, 8→64 frames (GiB)
Qwen3-VL-8B57.903.0119.2+4.9
DK-VLM-8B74.333.1422.3+4.5
GeoThinker-8B72.603.0931.7+17.4
VG-LLM-8B62.203.8329.5+21.4
VLM-3R-7B60.909.0323.4+10.8
Spatial-MLLM-4B48.412.1144.9–

DK-VLM gains 16.4 points over Qwen3-VL-8B at nearly the same latency and 3.1 GiB more memory, and it is more accurate than GeoThinker with similar latency and less memory. Of the two costs you point out, the geometry frontend runs in parallel with tokenization, and the full attention at five layers adds only 2.6% to the prefill compute. For longer videos, from 8 to 64 frames DK-VLM's memory grows no faster than Qwen's, while the memory increase of the other geometry-aware methods is 2–5× larger. At 64 frames its latency is 5.61 s against 4.80 s for Qwen: VGGT-Ω's global attention grows quadratically with the number of frames, so the frontend is no longer fully hidden behind tokenization.

Thank you again for your constructive feedback on DK-VLM. We have added the matched feature-fusion comparison, the single-step control for RC, and inference-cost measurements to the revised manuscript, and clarified our contributions relative to prior work in our response.

As the discussion period draws to a close, we would appreciate any further feedback on our responses and additional results. We would be happy to clarify any remaining questions.

Reviewer_nHoEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DK-VLM, which incorporates 3D geometry into vision-language models through a geometric attention branch alongside the native semantic attention. It also proposes Reference-Frame Composition supervision to improve multi-step spatial reasoning. DK-VLM-8B achieves the highest average score across six spatial benchmarks among the compared methods, while maintaining comparable general video understanding performance.

Critical strengths

It's a good idea to introduce geometry-only attention and force the model to learn it. The dual-kernel design incorporates geometric relations while preserving the pretrained attention mechanism.

Strong benchmark results and component ablations support the effectiveness of the proposed approach. It improves Qwen3-VL a lot on VSI-Bench and ReVSI-32.

Critical weaknesses

The paper mainly compares with Qwen3-VL which is a little bit outdated for ICLR 2027. It's worth running one frontier model to show spatial understanding is still a problem.

For frontier LLMs, large-scale SFT make the instruction following ability worse. So it limits the proposed methods from being integrated to frontier models like the next Qwen models. In my opinion, framing it as a multimodal pretraining problem might be a better idea, since it involves architecture changes.

Questions for authors

See weakness.

4 comments

W1: More frontier models. We evaluated a newer open model under our protocol and collected recent results reported by others:

ModelVSI-BenchSource
DK-VLM-8B74.33ours (paper)
InternVL3.5-241B-A28B69.5InternVL3.5 report
Qwen3.8-27B63.28our evaluation, same protocol as DK-VLM
Qwen3.5-35B-A3B58.1EASI leaderboard
Qwen3-VL-8B57.9EASI paper
GPT-555.0EASI paper
GPT-5.254.5EASI leaderboard
Kimi-K2.553.6Spatial-TTT
Gemini-3-Pro52.5SenseNova-SI
Seed-2.050.7Spatial-TTT

Qwen3.8-27B is an open Qwen model released in August 2026. Under the same 32-frame protocol it is 11.1 points below DK-VLM-8B and lower on all eight categories (e.g., relative direction 68.99 vs. 90.48, route planning 51.55 vs. 60.31). On a fixed random subset of 500 questions, turning on Qwen3.8's thinking mode raises it from 63.50 to 64.42, while DK-VLM-8B scores 77.93 on the same subset. We have added Qwen3.8-27B to Table 1 and this comparison to Appendix A.13.5.

W2: Instruction following. On IFEval, with official rule-based scoring and the same protocol for all models:

ModelPrompt-level strictInstruction-level strict
Qwen3-VL-8B (original)84.189.3
DK-VLM-8B79.385.9
GeoThinker-8B40.952.2

DK-VLM is 4.8 points lower than the original at the prompt level, while GeoThinker-8B, fine-tuned from the same backbone, is 43.2 points lower. DK-VLM's drop comes mostly from length constraints. Its answers are shorter, because the spatial QA data it is fine-tuned on mostly have short answers, such as an option letter or a number. IFEval is text-only, so the geometric branch does not run here: the drop comes from the fine-tuning data, not from adding the geometric branch.

Pre-training. We agree that adding the geometric kernel during pre-training is a promising direction. In this work we add it after pre-training, which lets us apply it to existing models such as Qwen2.5-VL and Qwen3-VL without retraining them from scratch. We now mention pre-training as future work in the conclusion.

Thank you again for your constructive feedback on DK-VLM. We have added the Qwen3.8-27B comparison and IFEval results to the revised manuscript, and discussed incorporating the geometric kernel during pre-training as future work.

As the discussion period draws to a close, we would appreciate any further feedback on our responses and additional results. We would be happy to clarify any remaining questions.

Thanks for the rebuttal, I'm happy with the new results and am going to keep my score.

Thank you for your positive feedback and for taking the time to review our rebuttal.

Reviewer_8SowOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DK-VLM, a framework for improving spatial reasoning in vision-language models by incorporating 3D geometry into attention as positional information rather than fusing reconstruction features into visual token content. Its central component, Dual-Kernel Attention, combines two parallel branches at selected decoder layers. The semantic branch retains the backbone’s native attention and positional encoding, while the geometric branch uses estimated 3D points and camera poses to model point displacements and relative transformations between views. Layer-specific learned gates combine their outputs. The geometric inputs are obtained from a frozen geometry frontend, without requiring ground-truth depth or camera poses at inference. Training includes an auxiliary pairwise objective that supervises distances and directions between visual tokens. To support reasoning across changing viewpoints, the authors also introduce Reference-Frame Composition supervision, which connects verified single-step spatial relations into sequences with explicit turn and heading targets. The model is jointly trained on standard instruction data, these compositional examples, and the pairwise geometric objective. Experiments using Qwen2.5-VL and Qwen3-VL backbones report improved performance across spatial reasoning benchmarks.

Critical strengths
  1. The paper makes a substantive adaptation of geometric attention to pretrained VLMs, supported by relevant controls. Replacing the geometric transforms with identity operations while retaining the full branch suggests that the gains extend beyond additional model capacity. Comparisons with additive positional embeddings and direct single-kernel GTA replacement further support the separate-kernel design under the tested adaptation setting.
  2. The empirical gains extend across several spatial reasoning benchmarks. DK-VLM improves over the reported GeoThinker results on VSI-Bench, ReVSI-32, and the aggregate benchmark comparison. ReVSI provides useful evidence beyond the original VSI evaluation, while the category-level results allow readers to assess both the improvements and the capabilities that remain challenging.
  3. Reference-Frame Composition offers a concrete supervision strategy for learning successive spatial judgments. Its benefits appear on public route-planning, multi-step reasoning, and perspective-dependent tasks, extending beyond the authors’ constructed chain evaluation. The documented convention normalization, reference-frame corrections, consistency checks, and provenance validation also make the procedure more reproducible and potentially reusable.
  4. Using explicit points and camera poses creates a practical interface between the geometry frontend and the VLM. The paper tests this design by replacing the default frontend with several alternatives at inference without retraining, with performance remaining close to the original configuration. This provides empirical support for frontend interchangeability and for using explicit geometric states rather than frontend-specific latent features.
Critical weaknesses
  1. The results support the proposed architecture more clearly than the explanation that it reduces semantic–geometric alignment difficulty. GeoThinker’s feature-fusion interface is not tested under matched training and frontend conditions, while the single-kernel GTA comparison covers only one adaptation regime. The broader advantage of positional geometry over well-adapted feature fusion therefore remains uncertain.
  2. The RC experiments do not fully isolate sequential reference-frame composition from additional supervision and training compute. The evaluation also pools targets with different compositional demands and includes very few longer action chains. Although the public-task gains support RC’s usefulness, they provide less direct evidence for the claimed compositional mechanism.
  3. The quantitative claims need qualification. Aggregate improvements coexist with weaker results on some benchmarks and metric-estimation tasks. The architecture–RC interaction is modest and lacks variability estimates. Likewise, comparable performance to a matched SFT baseline on two video benchmarks does not establish broader preservation of the original backbone’s general capabilities.
  4. Deployment costs remain insufficiently characterized. Although training resources and parameter counts are disclosed, both the geometry frontend and the additional full visual attention remain active at inference. End-to-end latency and peak-memory measurements, especially across increasing frame counts, are needed to assess this trade-off.
Questions for authors
  1. Could you clarify whether the evidence supports reduced semantic–geometric alignment difficulty beyond the specific injection strategies tested? Please explain how the single-kernel GTA baselines handle text tokens, invalid geometry, attention masks, and trainable projections. Available training diagnostics could help separate positional disruption from other adaptation effects. A matched feature cross-attention comparison would strengthen the connection to GeoThinker; otherwise, could the claim be limited to the evaluated configurations?
  2. Could you compare RC training with an equal-budget set of verified single-step targets from the same annotations, using comparable response formats and optimization steps? This would help isolate the benefit of composing relations into chains. Please also report the chain evaluation separately by target type, including format-validity rates, and clarify whether held-out scenes and relations are excluded from the entire training mixture rather than only the RC subset.
  3. Could you provide uncertainty estimates for the principal improvements, particularly the relatively small architecture–RC interaction, and discuss the capabilities that do not improve despite the higher aggregate score? Please also reconcile GeoThinker’s reported overall score with its reproduced component scores and clarify their provenance. Finally, could the general-capability claim be restricted to the tested video benchmarks and matched SFT baseline, unless broader comparisons with the original backbone are available?
5 comments

W1/Q1: Alignment. To test the effectiveness of injecting geometry as position, we compared it with feature fusion and other injection strategies in the setting of Appendix A.12: the same Qwen3-VL-8B, VGGT-Ω frontend, 10% of the training data and the same steps, without the pairwise loss. The feature-fusion row is a new experiment.

ConfigurationNative PE keptGeometryVSI-Bench
Qwen SFTyes–59.48
Feature fusionyesas features61.33
Additive 3D PE, 5 layersyesas positions, added59.74
Single-kernel GTA, 3 layersreplacedas positions45.98
Single-kernel GTA, 5 layersreplacedas positions39.14
DK-VLM, 5 layersyesas positions, separate kernel62.78

With the same geometry from the same frontend, injecting it with our dual kernel gains 3.3 points over SFT, while injecting it as features through GeoThinker-style gated cross-attention gains 1.9. Among the positional variants, our dual kernel works best.

Single-kernel baseline. It uses GTA in place of the positional encoding in the corresponding channels of the native attention: tokens with valid geometry attend to each other with full attention, as in DK-VLM's geometric kernel, while text and geometry-invalid tokens keep mRoPE and the causal mask. It adds no parameters; only the same LoRA is trained.

Training diagnostics. At step 0, the single-kernel model differs from the original Qwen3-VL-8B only in the replaced attention, and its loss is already about 2.1, against about 0.9 for SFT and DK-VLM. After training it is still about 0.9, while the other two reach 0.3–0.4. Replacing the native attention thus disrupts the pretrained model from the outset, and the gap persists after training with the same budget.

W2/Q2: RC vs. extra supervision. We ran the suggested control. All arms are trained on a 10% sample of our non-RC training data. One arm adds 15K single-step questions and another adds 15K RC chains, built from the same relations with the same template and array answer format and trained for the same number of steps. Chain accuracy on the paper's chain evaluation, and the share of answers that are valid arrays and have the requested number of steps, in %:

Arm2 steps3 steps4 steps (unseen)All lengthsValid arrayRequested no. of steps
Base only2.20.00.00.849.712.2
+ single-step5.30.90.02.399.740.0
+ RC15.011.94.410.999.797.0

The single-step arm learns the answer format as well as RC but often gives the wrong number of steps, and even among chains of the correct length, RC is right twice as often, 11.3% against 5.6%. Only RC answers any four-step chain correctly, and it leads on every target type:

TargetBase only+ single-step+ RC
VSI turn angle0.41.35.4
VSI action1.23.615.5
SPAR local turn0.82.112.5
SPAR heading0.82.511.7

On VSI-Bench, RC scores highest, both overall and on route planning:

Base only+ single-step+ RC
VSI-Bench62.5162.6863.44
Route planning47.4246.3949.48

Held-out scenes. In the paper, the evaluation scenes are excluded from the RC data only, so the models with and without RC have both seen them in other training questions; in the controlled experiment above, these scenes and all relations in them are excluded from all training data, and RC still leads.

W3/Q3 (1): Uncertainty. To check that the gains are not specific to the 288 videos in VSI-Bench, we resampled the videos with replacement 10,000 times, using per-question predictions from our inference code, and report paired 95% confidence intervals:

Comparison (VSI-Bench)Difference [95% CI]
DK-VLM vs. original Qwen3-VL-8B+16.32 [+14.30, +18.33]
DK-VLM vs. GeoThinker-8B (re-run)+2.26 [+1.02, +3.55]
Architecture × RC interaction, VSI-Bench (Table 7)+0.65 [+0.17, +1.41]

The intervals for the gains over the backbone and over GeoThinker-8B, and for the architecture × RC interaction, all lie above zero, i.e., these gains are statistically significant.

(2) Capabilities that do not improve. On VSI-Bench, DK-VLM-8B is above GeoThinker-8B on six of the eight categories and slightly below on absolute distance, 60.48 against 62.49, and room size, 72.78 against 75.31; in our paired comparison, the 95% intervals of both differences include zero, so neither difference is statistically significant. Across the six spatial benchmarks in Table 2, it is below GeoThinker-8B only on MMSI-Bench and CV-Bench, by 0.11 and 0.58 points.

(3) GeoThinker numbers. The overall score (72.60) is from the GeoThinker paper, which gives no per-category scores for 8B; the starred per-category scores in Table 1 come from our re-run of GeoThinker's released weights with its official code (mean 72.11). We now explain this in the caption of Table 1.

(4) General capability. As suggested, we compared DK-VLM with the original Qwen3-VL-8B on more general video and image benchmarks, both run through the same pipeline:

BenchmarkQwen3-VL-8BDK-VLM-8B
Video-MME (w/o subs)66.166.0
MVBench66.568.3
MLVU (dev, MC)66.668.8
LongVideoBench (val)62.362.8
MMStar64.263.0
MMBench-EN dev (Circular)85.785.7

DK-VLM stays close to or above the original model on all of them.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the reviewers for detailed and constructive reviews. Three points came up in more than one review: a matched comparison with feature fusion, separating Reference-Frame Composition (RC) from extra supervision, and inference cost. New experiments:

  1. Dual kernel vs. feature fusion with the same backbone, VGGT-Ω frontend, data and budget: DK-VLM gains 3.3 points over SFT on VSI-Bench, feature fusion 1.9.

  2. Controlled RC experiment. On the same base data, we add the same number of single-step questions or RC chains, built from the same relations. RC reaches 10.9% chain accuracy vs. 2.3% with single-step data and 0.8% for the base, and only RC answers any unseen four-step chain correctly. RC also scores highest on VSI-Bench and on its route-planning category.

  3. Cost: latency and peak memory against Qwen3-VL-8B and four geometry-aware methods (GeoThinker, VG-LLM, VLM-3R, Spatial-MLLM) on one H100, up to 64 frames.

  4. General capability against the original Qwen3-VL-8B: four video benchmarks, two image benchmarks and IFEval.

  5. Confidence intervals for the main results, including a re-run of GeoThinker-8B for a paired comparison.

  6. Newer models: recent proprietary models and our own evaluation of Qwen3.8-27B (released Aug 2026).

We have uploaded a revised PDF with changes in blue. New evaluations were run on H100 and RTX 4090 GPUs rather than the H200s used in the paper, so scores differ slightly, by less than 0.4 points (e.g., 74.38 vs. 74.33 for DK-VLM on VSI-Bench).

AuthorsOpenReview ↗

Summary of the Discussion

We thank all reviewers and the AC. All three reviewers recommended acceptance. During the discussion, Reviewer Swbd confirmed that our response resolved their questions, and Reviewer nHoE stated they were happy with the new results. Following Reviewer Swbd’s suggestion, we updated the revised PDF to move key results into the main paper: the controlled comparison between RC and single-step supervision (Section 4.3), inference cost (Section 4.6), and alternative geometry frontends and input frames (Section 4.7). Our responses to Reviewer 8Sow are in the thread above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.