- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes FrameShift to improve spatial reasoning in vision-language models under changing viewpoints. It represents camera motion in complementary reference frames and uses a question-conditioned adapter to integrate geometric evidence into a frozen VLM. Experiments show improved spatial question-answering performance, particularly in relative-direction reasoning, alongside more accessible spatial information in intermediate model representations.
Critical strengths
The paper addresses a meaningful problem with a geometrically motivated approach to incorporating camera motion. Direction-reasoning gains are substantial. The diagnostics include useful controls, and the appendix transparently reports unfavorable results and generalization limitations.
Critical weaknesses
-
Insufficient isolation of the proposed design’s contribution. The comparison adds both SLAM geometry and task-specific supervision to the original backbone. Without pose-as-text or simple geometry-token baselines using matched resources, the gains cannot be clearly attributed to the dual-reference representation and question-conditioned readout. The ablations are inference-time interventions without retraining, rather than independently optimized alternatives.
-
Table 7 shows that disabling soft gating increases Relative Direction accuracy from 40.41% to 51.20%, a gain of 10.79 percentage points. Without matched gate-on/off results across all tasks or an analysis of confidence calibration, it remains unclear whether gating provides a compensating benefit elsewhere.
-
SAW gains come entirely from Relative Direction; other tasks collectively decrease by 2.99 points. The 256-query hidden-target gain has a confidence interval including zero, while full ARKit hidden accuracy decreases from 31.44% to 25.76%. The full-test overall difference also remains statistically inconclusive.
Questions for authors
-
Could you compare against pose-as-text and simple geometry-token baselines with matched poses and supervision, alongside retrained single-reference and CE-only variants?
-
Why does disabling gating improve 8B direction accuracy by 10.79 points? Does gating protect other tasks, and is confidence calibrated against pose quality or downstream errors?
-
How sensitive is FrameShift to estimated-pose errors and frame sampling? What explains the low ARKit visible-target accuracy, and were these questions checked for answerability from the actual inputs?
We sincerely thank Reviewer j6eW for the constructive feedback and clear experimental suggestions on matched baselines, gating calibration, pose robustness, and ARKit answerability.
Q1: Matched Retrained Baselines and CE-Only Variant
A1: Under identical poses, QA data, and update steps on a frozen VLM (C1), raw pose-as-text fails (30.52% overall, 17.87% Direction), while dual-reference with CE alone reaches 38.61% Direction (+9.95% over single-reference and +9.71% over simple MLP tokens), and adding auxiliary losses lifts all five other tasks to reach 39.26% overall.
Pose-as-text + LoRA reaches 42.59% overall (38.73% Direction) only by modifying the LLM's internal weights (3.83M LoRA parameters vs. 3.53M input-adapter parameters for FrameShift and 1.13M for the MLP). On 32-frame videos, pose text also expands inputs from 2,523 to 4,009 tokens (152 to 1,634 in numerical QA) and slows initial model input processing by 33% (840 ms vs. 563 ms; peak VRAM: 18.77 GB vs. 18.43 GB), whereas our full-schedule frozen-VLM model (gate-off) reaches 42.83% overall and 51.20% Direction.
Q2: Soft Gating Suppression and Confidence Calibration
A2: Our question gate (grel) cleanly separates motion tasks (~1.00) from static Spatial Memory (0.04), and VGGT tracking confidence (ctrack) reflects pose accuracy well (AUROC 0.807 on 6,000 ARKit clips), but their product is not calibrated to predict whether the VLM answers correctly (AUROC 0.481). Because low ctrack values (0.26-0.35) unnecessarily weakened valid tokens on 8B, removing this multiplier raises Direction to 51.20% and overall SAW accuracy to 42.83% with no average loss on other tasks (C2). We will clarify this in §5.4.
Q3: Robustness to Pose Errors and Sparse Frame Sampling
A3: FrameShift is highly robust to both SLAM pose noise and sparse frame sampling: replacing VGGT poses with ground truth (-0.78%), injecting up to 15∘ rotation or 15% translation noise, or subsampling down to 4 or 8 frames barely changes performance.
On 256 hidden-target queries (with clear angle definitions and upright images), ground-truth poses give 37.11% vs. 37.89% for VGGT poses. Adding random Gaussian noise (first pose fixed) maintains steady accuracy at 5∘/15∘ rotation SD (39.06% / 37.50%) and 5% / 15% path-length translation SD (37.89% / 38.28%), and 4- or 8-frame subsampling shifts accuracy by at most 2.61%. Thus, FrameShift tolerates real-world SLAM noise well.
Q4: Low ARKit Visible-Target Accuracy and Input Answerability
A4: Low default visible-target accuracy (5.45% Base, 6.10% FrameShift, even though 98.44% of the 1,671 labels are "front") comes from undefined "front" angle boundaries in the prompt and tilted handheld camera views, rather than coordinate bugs.
On 256 visible-target queries, specifying exact horizontal angle ranges lifts Base from 16.41% to 37.89%, and rotating tilted images upright via gravity raises Base to 50.78%. FrameShift scores 26.95% here for a simple, fixable reason: ARKit uses the exact same 12 question templates for visible and hidden objects, so our text-only gate activates (grel=1.00) even when the target is directly in view in the last frame, letting past camera motion bias the prediction. Re-checking all 6,000 episodes confirmed zero coordinate bugs (with occasional motion blur and partial views in raw clips). Making the gate visually aware of in-view targets is a natural extension we will note alongside the full ARKit analysis.