acceptodds
Under review as a conference paper at ICLR 2027

Remembering isn't Updating: Motion-Guided Spatial Reasoning in Vision-Language Models

Abstract

Spatial reasoning requires understanding spatial relationships and keeping them consistent across changing viewpoints. Yet retaining past observations does not ensure that a vision-language model can interpret them in the reference frame required by a question. We propose FrameShift, a geometry-guided method for spatial reasoning that represents camera motion in both the initial and final reference frames. This dual-reference representation preserves complementary spatial information about the same trajectory. A learned readout uses the question to combine evidence from the two reference frames. FrameShift then combines question relevance with geometric confidence to control the contribution of spatial evidence. The resulting spatial tokens are integrated with visual and language representations for answer generation. We further introduce geometric supervision and consistency objectives that connect continuous spatial information with language predictions. FrameShift improves overall accuracy on SAW-Bench by 7.19 percentage points and relative-direction accuracy by 22.30 points over the base model. These results highlight the value of representing the same trajectory in complementary reference frames. They support using motion and question context together to interpret spatial relationships in the appropriate coordinate system as the viewpoint changes. Code is available at https://anonymous.4open.science/r/FrameShift-0368.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language Models, Multimodal Learning, Spatial Reasoning, Embodied Intelligence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_j6eWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FrameShift to improve spatial reasoning in vision-language models under changing viewpoints. It represents camera motion in complementary reference frames and uses a question-conditioned adapter to integrate geometric evidence into a frozen VLM. Experiments show improved spatial question-answering performance, particularly in relative-direction reasoning, alongside more accessible spatial information in intermediate model representations.

Critical strengths

The paper addresses a meaningful problem with a geometrically motivated approach to incorporating camera motion. Direction-reasoning gains are substantial. The diagnostics include useful controls, and the appendix transparently reports unfavorable results and generalization limitations.

Critical weaknesses
  1. Insufficient isolation of the proposed design’s contribution. The comparison adds both SLAM geometry and task-specific supervision to the original backbone. Without pose-as-text or simple geometry-token baselines using matched resources, the gains cannot be clearly attributed to the dual-reference representation and question-conditioned readout. The ablations are inference-time interventions without retraining, rather than independently optimized alternatives.

  2. Table 7 shows that disabling soft gating increases Relative Direction accuracy from 40.41% to 51.20%, a gain of 10.79 percentage points. Without matched gate-on/off results across all tasks or an analysis of confidence calibration, it remains unclear whether gating provides a compensating benefit elsewhere.

  3. SAW gains come entirely from Relative Direction; other tasks collectively decrease by 2.99 points. The 256-query hidden-target gain has a confidence interval including zero, while full ARKit hidden accuracy decreases from 31.44% to 25.76%. The full-test overall difference also remains statistically inconclusive.

Questions for authors
  1. Could you compare against pose-as-text and simple geometry-token baselines with matched poses and supervision, alongside retrained single-reference and CE-only variants?

  2. Why does disabling gating improve 8B direction accuracy by 10.79 points? Does gating protect other tasks, and is confidence calibrated against pose quality or downstream errors?

  3. How sensitive is FrameShift to estimated-pose errors and frame sampling? What explains the low ARKit visible-target accuracy, and were these questions checked for answerability from the actual inputs?

1 comment

We sincerely thank Reviewer j6eW for the constructive feedback and clear experimental suggestions on matched baselines, gating calibration, pose robustness, and ARKit answerability.

Q1: Matched Retrained Baselines and CE-Only Variant

A1: Under identical poses, QA data, and update steps on a frozen VLM (C1), raw pose-as-text fails (30.52% overall, 17.87% Direction), while dual-reference with CE alone reaches 38.61% Direction (+9.95% over single-reference and +9.71% over simple MLP tokens), and adding auxiliary losses lifts all five other tasks to reach 39.26% overall.

Pose-as-text + LoRA reaches 42.59% overall (38.73% Direction) only by modifying the LLM's internal weights (3.83M LoRA parameters vs. 3.53M input-adapter parameters for FrameShift and 1.13M for the MLP). On 32-frame videos, pose text also expands inputs from 2,523 to 4,009 tokens (152 to 1,634 in numerical QA) and slows initial model input processing by 33% (840 ms vs. 563 ms; peak VRAM: 18.77 GB vs. 18.43 GB), whereas our full-schedule frozen-VLM model (gate-off) reaches 42.83% overall and 51.20% Direction.

Q2: Soft Gating Suppression and Confidence Calibration

A2: Our question gate (grelg_{\text{rel}}) cleanly separates motion tasks (~1.00) from static Spatial Memory (0.04), and VGGT tracking confidence (ctrackc_{\text{track}}) reflects pose accuracy well (AUROC 0.807 on 6,000 ARKit clips), but their product is not calibrated to predict whether the VLM answers correctly (AUROC 0.481). Because low ctrackc_{\text{track}} values (0.26-0.35) unnecessarily weakened valid tokens on 8B, removing this multiplier raises Direction to 51.20% and overall SAW accuracy to 42.83% with no average loss on other tasks (C2). We will clarify this in §5.4.

Q3: Robustness to Pose Errors and Sparse Frame Sampling

A3: FrameShift is highly robust to both SLAM pose noise and sparse frame sampling: replacing VGGT poses with ground truth (-0.78%), injecting up to 15∘15^\circ rotation or 15% translation noise, or subsampling down to 4 or 8 frames barely changes performance.

Subsampled Input (N = 268, >8 Original Frames)4-Frame Delta (%)8-Frame Delta (%)
Visual + Geometry-1.12-1.12
Visual Only-1.12-1.12
Geometry Only+0.37-2.61

On 256 hidden-target queries (with clear angle definitions and upright images), ground-truth poses give 37.11% vs. 37.89% for VGGT poses. Adding random Gaussian noise (first pose fixed) maintains steady accuracy at 5∘/15∘5^\circ / 15^\circ rotation SD (39.06% / 37.50%) and 5% / 15% path-length translation SD (37.89% / 38.28%), and 4- or 8-frame subsampling shifts accuracy by at most 2.61%. Thus, FrameShift tolerates real-world SLAM noise well.

Q4: Low ARKit Visible-Target Accuracy and Input Answerability

A4: Low default visible-target accuracy (5.45% Base, 6.10% FrameShift, even though 98.44% of the 1,671 labels are "front") comes from undefined "front" angle boundaries in the prompt and tilted handheld camera views, rather than coordinate bugs.

On 256 visible-target queries, specifying exact horizontal angle ranges lifts Base from 16.41% to 37.89%, and rotating tilted images upright via gravity raises Base to 50.78%. FrameShift scores 26.95% here for a simple, fixable reason: ARKit uses the exact same 12 question templates for visible and hidden objects, so our text-only gate activates (grel=1.00g_{\text{rel}} = 1.00) even when the target is directly in view in the last frame, letting past camera motion bias the prediction. Re-checking all 6,000 episodes confirmed zero coordinate bugs (with occasional motion blur and partial views in raw clips). Making the gate visually aware of in-view targets is a natural extension we will note alongside the full ARKit analysis.

Reviewer_qQMJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents FrameShift, a geometry-guided approach to improving spatial reasoning in vision-language models. The main idea is that remembering earlier visual observations is not sufficient for reasoning about spatial relationships after the viewpoint changes. FrameShift uses VGGT-SLAM 2 to estimate camera trajectories and represents the motion in both the initial and final reference frames. A lightweight, question-conditioned adapter combines these representations and injects spatial tokens into a frozen Qwen3-VL model. The adapter is trained using procedural trajectory QA and a small set of real-video QA examples, together with several auxiliary geometric objectives. Experiments on SAW-Bench show improvements in overall accuracy, particularly on Relative Direction questions. The paper also includes representation probing, geometric interventions, and component ablations to investigate the contribution of motion information.

Critical strengths
  1. The distinction between remembering a view and interpreting it after camera motion provides a clear framing of the problem. Supplying explicit pose information through a lightweight adapter while keeping the VLM frozen is also a practical design. The Relative Direction gains are substantial, rising from 18.11% to 40.41% for the 8B model, with a smaller but consistent improvement for the 4B model, demonstrating effectiveness on camera-relative spatial reasoning.

  2. The controlled experiments verify that the model actively uses the supplied geometric input. The matched backbone comparisons, representation probes, and interventions with shuffled or stale poses and zero-valued spatial tokens all confirm that motion information plays a real role, making the direction-reasoning results credible.

Critical weaknesses
  1. The experimental results do not support the object-level spatial updating capability emphasized throughout the paper. The proposed representation captures camera motion without maintaining object-level spatial states, and much of the evaluation tests camera-reference direction reasoning rather than updating the locations of previously observed objects. This gap is particularly evident in the ARKit results. While the main text reports a 7.03-point improvement on the 256-episode hidden-target diagnostic, its confidence interval includes zero. More importantly, Appendix A.7 shows that FrameShift performs worse than the baseline on the complete 6,000-episode ARKit evaluation, with hidden-target accuracy dropping from 31.44% to 25.76% and overall accuracy from 24.20% to 20.28%. This discrepancy undermines the paper's central motivation and shows that the current method supports camera-relative direction reasoning rather than updating spatial relationships for previously observed objects.

  2. The experiments do not adequately justify the proposed architectural choices. The improvements on SAW-Bench are entirely concentrated in Relative Direction, while the 8B model degrades on four of the other five tasks. Furthermore, Appendix Table 7 shows that removing soft gating increases Relative Direction accuracy from 40.41% to 51.20%, directly contradicting the claim that gating effectively regulates geometric evidence, and the paper provides no gate-off results on the remaining tasks to show a preservation trade-off. In addition, the dual-reference representation is only evaluated via inference-time pruning of a fixed model, rather than against simpler alternatives trained under the same conditions, such as a single-reference adapter or direct pose conditioning. Without retrained baselines, it remains unclear how much of the gain requires the specific design of FrameShift.

Questions for authors
  1. What accounts for the discrepancy between the two ARKit evaluations? Why does FrameShift improve on the 256-episode hidden-target diagnostic but degrade in hidden-target accuracy on the complete 6,000-episode test? Please explain the protocol differences and provide a breakdown of the failure cases.

  2. The paper motivates spatial updating using objects that were previously visible but are later occluded, yet many evaluated direction questions involve relationships between camera viewpoints. How much of the reported gain comes from camera-pose reasoning versus reasoning about previously observed objects? In particular, what is the performance on objects first observed at intermediate frames and subsequently occluded?

  3. Removing soft gating improves the 8B model's Relative Direction accuracy by 10.79 percentage points. Why does the learned gate suppress valid geometric information to this extent? Please report the results of disabling the gate across all six SAW-Bench tasks to clarify whether the gate provides a cross-task trade-off or simply limits performance.

  4. How does FrameShift compare with simpler baselines that incorporate the same camera-motion information? Please provide results for a single-reference adapter trained from scratch under the same conditions, as well as a baseline that supplies the estimated camera poses directly as text prompts or simpler spatial tokens.

  5. FrameShift-8B loses accuracy on Self-Localization, Route Shape, Reverse Route Plan, and Spatial Memory, even though these tasks also depend on camera motion. Do these regressions stem from the narrow direction-focused training data or from how the spatial tokens interact with the frozen VLM?

3 comments

We sincerely thank Reviewer qQMJ for the insightful distinction between camera-viewpoint reasoning and hidden-object tracking, and for suggesting controlled retrainings and full six-task gating experiments that substantially strengthened the paper.

Q1: Discrepancy Between the 256-Episode and 6,000-Episode ARKit Evaluations

A1: We agree with your assessment: the full 6,000-episode ARKit benchmark (hidden targets: 31.44% to 25.76%; overall: 24.20% to 20.28%) shows that camera-motion tokens alone do not track hidden objects, and its contrast with the 256-episode subset (+7.03%) comes from clip duration and scene differences rather than label balance or output format.

The 256-episode subset uses 10 longer validation videos (16 frames, median 29.8s), whereas the 6,000-episode test covers 20 separate scenes with much shorter clips (2-25 frames, median 4.9s). Checking class-balanced accuracy (33.73% to 29.99%) and swapping single-letter vs. full-QA output formats confirms neither factor changes the trend; instead, on very short clips without an object-level map, trajectory tokens tend to bias predictions toward "back" (particularly on "right" targets). Moving this full 6,000-episode analysis into the main text will make the scope boundary completely transparent.

Q2: Camera-Pose Reasoning vs. Objects Observed at Intermediate Frames

A2: On SAW-Bench, gains come from relating camera viewpoints (+186 net correct across 834 Relative Direction questions), whereas objects seen briefly midway through a clip and later occluded show no clear gain even with ground-truth poses (+0.39%).

ARKit Midway-Observed Targets (N = 256)Base (%)FrameShift (%)Delta
Raw RGB + estimated poses33.2025.39-7.81
Upright RGB & padding + GT poses36.7237.11+0.39

As noted in §4.3, FrameShift encodes camera motion rather than an object map. Testing 256 balanced ARKit cases where an object is unseen in frame 1, appears in frame 2, and leaves view before the last frame confirms that tracking unseen objects requires explicit object memory in addition to ego-motion. We appreciate your feedback and will revise our introduction and claims to focus strictly on camera-viewpoint spatial updating, where FrameShift delivers large gains (+33.09% over Base on SAW Direction and +13.92% zero-shot on MindCube cross-view).

Q3: Why Removing Soft Gating Improves 8B Direction by 10.79%

A3: Thank you for prompting this six-task analysis (C2): our question gate (grelg_{\text{rel}}) works properly (~1.00 on motion tasks vs. 0.04 on static Spatial Memory), and the suppression came solely from low raw VGGT tracking confidence (mean 0.35 on Direction). Turning that multiplier off raises FrameShift-8B to 51.20% on Direction and 42.83% overall without lowering average accuracy on the other five tasks (36.94% vs. 37.19%). We will highlight this finding and report 4B and 8B separately in §5.4.

Q4: Retrained Single-Reference and Simpler Pose Baselines

A4: Under matched from-scratch training on a frozen VLM (C1), dual-reference CE (38.61% Direction) strongly outperforms frozen pose-as-text (17.87%, +20.74%), a single-reference adapter (28.66%, +9.95%), and simple MLP pose tokens (28.90%, +9.71%). Pose-as-text only works when LoRA modifies the LLM's internal weights (42.59% overall, 38.73% Direction), whereas our full-schedule frozen-VLM FrameShift-8B (gate-off) reaches 42.83% overall and 51.20% Direction without altering the backbone.

Q5: Cause of Regressions on Non-Direction SAW-Bench Tasks

A5: Feeding all-zero spatial tokens matches the base model on non-Direction tasks (40.18% vs. 39.94% Base), proving our 8-token interface does not disturb the frozen VLM; moreover, adding auxiliary geometric losses (C1) lifts all five non-Direction tasks and beats Base-8B on four of the six tasks.

Specifically, the earlier drop on non-Direction tasks came from our real-video QA set focusing heavily on start-to-end direction (450 direction vs. 62 non-motion examples). As Table C1 shows, regularizing with our auxiliary geometric losses counteracts this skew, surpassing Base-8B on Route Shape (43.41% vs. 39.74%), Reverse Route (25.76% vs. 24.02%), and Affordance (53.70% vs. 53.09%) in addition to Direction.

Reviewer_qQMJOpenReview ↗

Q2, Q4, and Q5: Addressed; Q1 and Q3: Remaining Considerations

Thank you for the detailed responses and additional experiments. Overall, I believe the authors have largely addressed Q2, Q4, Q5. My remaining considerations mainly concern Q1 and Q3, as discussed below.

Q1:

The authors have provided a plausible explanation for the discrepancy between the 256-episode diagnostic and the full 6,000-episode ARKit evaluation. The former uses longer videos (median 29.8s), while the latter consists of much shorter clips (median 4.9s) from different scenes. However, these differences make it difficult to determine the main cause of the performance discrepancy. More importantly, FrameShift performs worse on hidden-target queries in the full evaluation (31.44% to 25.76%). This does not support the original motivation of object-level spatial updating. I appreciate the authors' revised positioning, but the paper should clearly distinguish camera-viewpoint reasoning from object-level spatial updating.

Q3:

The authors provide a plausible explanation for why removing the confidence-based gate improves performance. In particular, disabling the gate leads to a substantial improvement in Relative Direction accuracy (40.41% to 51.20%) and overall accuracy (38.34% to 42.83%), while maintaining similar average performance on the other five tasks. This raises questions about the practical necessity of the proposed gating mechanism. Although the authors attribute the improvement to low VGGT tracking confidence, it remains unclear whether the current gating design provides any consistent benefit over simply removing it.

Thank you for the careful follow-up and for recognizing that our responses have largely addressed Q2, Q4, and Q5. We clarify the remaining two points below.

Q1. Distinguishing camera-viewpoint reasoning from object-level spatial updating

We agree that this distinction should be clearer. FrameShift supplies a frozen VLM with explicit camera-motion information to improve spatial reasoning across viewpoints. As stated in §4.3, its geometric representation encodes camera trajectories without maintaining an explicit object map. The full ARKit hidden-target result (31.44% → 25.76%) must therefore remain a limitation regarding object-level updating; we do not treat this capability as an established contribution.

Our previous analysis identified differences in both clip duration and scenes between the 256-query diagnostic and the 6,000-query full test. This comparison does not isolate which factor drives the reversal. We will make the contribution boundary explicit in the abstract, introduction, and conclusion, and present both ARKit evaluations together in the main text.

Q3. Gating introduces task trade-offs; removing it preserves the core architecture

For 8B, gate-off indeed achieves higher overall accuracy (38.34% → 42.83%) and Relative Direction accuracy (40.41% → 51.20%). At the same time, gate-on achieves higher accuracy on four of the five remaining tasks in this evaluation:

Non-Direction taskNGate-onGate-off
Self-Localization20039.50%36.50%
Route Shape54637.18%41.03%
Reverse Route Plan22918.78%17.03%
Spatial Memory10046.00%42.00%
Spatial Affordance16253.09%50.62%

The previously reported 36.94% and 37.19% are pooled accuracies over these 1,237 questions: gate-off gains 21 correct answers on Route Shape while losing 18 across the other four tasks, yielding a net gain of three. The similar pooled scores therefore mask task-specific trade-offs. On the 4B checkpoint, Direction accuracy is 30.94% with gating versus 29.62% without it, further showing that the observed benefit varies across tasks and checkpoints.

Gate-off only fixes the final scalar to g=1g=1; dual-reference encoding and question-conditioned readout remain intact. Thus, the 51.20% result still comes from a FrameShift variant retaining these modules. We will report both 8B settings in the main results and present scalar gating as an optional component with explicit task trade-offs, without claiming a universal advantage for the current gate.

Reviewer_rW1aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FrameShift, where shared video frames provide visual tokens and estimated camera poses, and the question-conditioned adapter encodes geometry in initial and final reference frames into spatial tokens to be prepended to the multi-modal VLM input.

Critical strengths

The paper is well motivated by how camera motion trajectories can help in spatial reasoning, and is generally well written. The question-conditioned readout is novel and reasonably motivated by different kinds of spatial reasoning capabilities require different kinds of camera trajectory information. It is also validated by the ablation study.

Critical weaknesses
  1. This paper uses a relatively diverse training data composition, but the evaluation is limited to a single benchmark (SAW-Bench). Please consider "Spatial Mental Modeling from Limited Views" and "Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces"
  2. Baselines comparisions are limited. Only VLMs are evaluated and compared against FrameShift, how about other spatial representation and memory methods mentioned in 2.2?
Questions for authors

Could you please demonstrate this proposed training and the resulting model can generalize to different tasks and kinds of questions and outperform spatial reasoning baselines in Section 2.2?

1 comment

We sincerely thank Reviewer rW1a for the encouraging feedback on our paper's clarity, motivation, and question-conditioned readout, as well as for recommending two valuable benchmarks.

Q1: Generalization to MindCube and VSI-Bench Across Question Types

A1: Tested strictly zero-shot with no retraining, FrameShift-8B improves MindCube accuracy by +8.00% overall (and by +13.92% on cross-view questions), while preserving base performance on VSI-Bench (53.86 vs. 54.11).

BenchmarkMetricNBaseFrameShiftDelta
MindCube (Spatial Mental Modeling)Accuracy (%)1,05030.0038.00+8.00
VSI-Bench (Thinking in Space)Official Score5,13054.1153.86-0.25

Both models use the exact same frames, official prompts, and decoding (all views on MindCube tinybench; 32 uniformly sampled frames on VSI-Bench). Consistent with our design, FrameShift brings strong gains whenever questions connect different camera views: on MindCube, cross-view relation queries ("meanwhile") jump from 22.38% to 36.30% and rotation improves from 31.50% to 32.50% (while imagined "what-if" movement without actual camera motion shifts from 62.75% to 52.94%). On VSI-Bench, where tasks focus mainly on static scene memory, object counts, and room/object sizes rather than viewpoint changes, base accuracy is cleanly preserved.

Q2: Comparison with Spatial Representation and Memory Methods in §2.2

A2: FrameShift and spatial-memory VLMs address complementary needs: on scene-memory tasks (VSI-Bench), VSI-trained VLM² leads (68.29 vs. 53.86 zero-shot), whereas for feeding 3D camera motion into a frozen VLM (C1), FrameShift outperforms frozen pose-as-text by +20.74% and simpler geometry adapters by +9.71% to +9.95% on Direction.

Specifically, VLM² fine-tunes a Qwen2-7B backbone with dedicated visual memory on in-domain VSI training data to remember static scenes, while FrameShift is a lightweight 3.53M-parameter plug-in on a frozen VLM with zero VSI training. Meanwhile, geometric methods in §2.2 (such as DROID-SLAM and DUSt3R) output 3D poses or point clouds and need an adapter like FrameShift to answer language questions, where our dual-reference design clearly beats alternative pose interfaces (C1). Combining explicit scene memory with FrameShift's viewpoint adapter is an exciting direction we will discuss alongside these results in §5.

Comments on the paper

1 comment
AuthorsOpenReview ↗

To PCs, SACs, ACs, and Reviewers

Dear Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers,

We sincerely thank our Reviewers (rW1a, qQMJ, j6eW) for the constructive discussion, and the ACs and PCs for overseeing the review process. Below is a summary of our core contributions to VLM spatial reasoning and our rebuttal resolutions.

Core Insight & Contributions

  • Core Insight ("Remembering isn't Updating"): For spatial reasoning across changing viewpoints, retaining past video frames does not ensure a VLM can update spatial relationships into the coordinate frame required by a question after camera motion; trajectories need complementary reference frames.
  • Our Approach (FrameShift): For motion-guided spatial reasoning, we propose a 3.53M-parameter plug-in for frozen VLMs (Qwen3-VL 4B/8B) that encodes SLAM trajectories in initial and final reference frames, uses a question-conditioned readout to fuse them into 8 spatial tokens, and aligns 3D motion with language via auxiliary geometric losses, without altering LLM weights or text length.
  • Key Results: On SAW-Bench (8B), FrameShift boosts overall spatial reasoning accuracy by +7.19 pp (38.34%, up to 42.83% with gate g=1) and Relative Direction by +22.30 pp (18.11% → 40.41% gate-on; 51.20% / +33.09 pp gate-off). Zero-shot transfer to MindCube (N=1,050) yields +8.00% overall (30.00% → 38.00%) and +13.92% on cross-view queries (22.38% → 36.30%), while preserving static memory on VSI-Bench (N=5,130; 53.86 vs. 54.11 Base).

Reviewer Concerns & Our Resolutions

1. Isolating Architectural Gains vs. Matched Baselines (rW1a, qQMJ, j6eW)

  • Concern: Do gains come from our architecture or simply from adding SLAM poses and supervision?
  • Resolution: Trained from scratch under the exact same frozen-VLM setup (Table C1), our dual-reference adapter (38.61% Direction, CE-only) far outperforms frozen pose-as-text (17.87%, +20.74%), a retrained single-reference adapter (28.66%, +9.95%), and simple MLP tokens (28.90%, +9.71%). Pose-as-text only works when LoRA modifies the LLM (38.73% Direction, 42.59% overall), trailing frozen FrameShift-8B (51.20% Direction, 42.83% overall) while inflating 32-frame tokens (2,523 → 4,009) and slowing prefill by 33% (840 vs. 563 ms).

2. Cross-Benchmark Generalization & Spatial Memory Models (rW1a)

  • Concern: Generalization beyond SAW-Bench and comparison with spatial memory methods (§2.2).
  • Resolution: Zero-shot tests (no retraining) confirm strong gains on MindCube (+8.00% overall, +13.92% cross-view) and preserved accuracy on static-memory VSI-Bench (53.86 vs. 54.11). Full-model fine-tuning for static scene recall (e.g., VLM²) and our 3.53M frozen-VLM plug-in for viewpoint updating are complementary and can be combined.

3. Soft Gating Trade-Offs & Multi-Task Performance (qQMJ, j6eW)

  • Concern: Why turning off soft gating lifts 8B Direction (40.41% → 51.20%), and why some non-Direction tasks drop.
  • Resolution: Question gate g_rel separates motion (~1.00) from static memory (0.04) and c_track predicts pose quality (AUROC 0.807), but their product is uncalibrated for VLM accuracy (AUROC 0.481), so conservative c_track (mean 0.35) over-dampens valid tokens on 8B. Across all 6 SAW-Bench tasks, gate-on protects 4 of 5 non-Direction tasks (Self-Localization 39.50% vs. 36.50%, Reverse Route 18.78% vs. 17.03%, Memory 46.00% vs. 42.00%, Affordance 53.09% vs. 50.62%), while gate-off (g=1) excels on Direction (51.20%) and Route Shape (41.03% vs. 37.18%); we will report both in the main paper. All-zero tokens match Base on non-Direction tasks (40.18% vs. 39.94%, proving no interface disruption), and auxiliary geometric losses (Table C1) lift all 5 non-Direction tasks, beating Base-8B on 4 of 6 tasks.

4. Scope on Occluded Objects & ARKit Robustness (qQMJ, j6eW)

  • Concern: The 256- vs. 6,000-episode ARKit hidden-target gap (31.44% → 25.76%), low visible accuracy, and pose-noise/subsampling robustness.
  • Resolution: Camera motion helps on longer clips (256-subset, median 29.8s), but trajectories alone cannot track occluded objects on short clips (6,000-test, median 4.9s) or midway-occluded targets without an explicit object map. Following reviewers' advice, we narrowed our claims strictly to camera-viewpoint spatial updating and moved the 6,000-episode analysis to the main text. We also verified zero coordinate bugs across all 6,000 episodes. Low default visible accuracy stems from undefined "front" angle bounds and tilted views (specifying bounds and gravity alignment lifts Base from 16.41% to 50.78%), and the model remains robust to Gaussian pose noise (up to 15° rotation / 15% translation SD) and 4/8-frame subsampling (≤ 2.61% shift).

Thanks again to the Reviewers, ACs, and PCs for your guidance in improving our work!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.