acceptodds
Withdrawn from ICLR 2027

Query-Framed Spatial Reasoning via Spatial Latent Transformation in Video MLLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Answering spatial questions from video requires using observations made at different positions and viewing directions. Remembering what was seen is not enough when a question asks for a relation from another viewpoint. Learning a scene representation is one way to support such reasoning. We investigate a direct alternative: adjust the observed visual evidence for answering, without requiring a video multimodal large language model (Video MLLM) to first generate a separate scene representation. We propose QUASAR (QUery-frAmed SpAtial Reasoning) framework, centered on Spatial Latent Transformation. Given estimated observation geometry and an explicit reference type and numerical pose, it adds bounded spatial corrections to the original visual features. The MLLM reads the full question and answers from this adjusted evidence; geometry remains required at inference. Spatial supervision, including four learned spatial summary tokens, accompanies supervised fine-tuning, followed by Group Relative Policy Optimization. QUASAR exceeds the original Qwen3-VL-8B baseline by 5.54 points on VSI-Bench and 19.88 points on VSTI-Bench; the historical VSI comparison uses different visual inputs and answer options. On both benchmarks, it also outperforms the Visual-only baseline (SFT+GRPO), which shares the backbone and data recipe but differs in pre-GRPO SFT exposure. Fixed-model interventions further support use of the visual corrections and, on parsable target-camera questions, the numerical reference. These results support direct visual-evidence adjustment as a viable route to spatial answering.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video spatial reasoning, multimodal large language models, reference-frame conditioning, geometry-aware visual representations

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NiDhOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper is based on the hypothesis that when video language models answer spatial questions (like whether an object is to the left or right of another), simply remembering past video frames isn't enough. As the camera moves, relative left-and-right visual positions shift depending on where the observer stands and faces. The authors calls this the Observation-to-Query (O2Q) Bottleneck.

And propose a solution they call QUASAR.

Instead of forcing the model to first generate a separate, 3D scene representation before answering, QUASAR takes a direct route: it uses estimated camera positions and the question's reference view to apply small, bounded mathematical corrections directly onto the original visual features. The LLM then answers the question straight from these adjusted visual features.

The paper contributes

  1. A spatial latent transformation module that calculates norm-bounded spatial tweaks using camera geometry and explicit reference coordinates, writing them directly into the visual features.
  2. An answer-path spatial supervision: A training technique using four spatial summary tokens and geometric supervision to keep spatial context accessible along the answer path.

And the results report

  • Direct visual tweaks win: Adjusting visual features directly outperforms visual-only baselines (in most cases) without requiring explicit 3D scene reconstruction.
  • Benchmark gains: Built on Qwen3-VL-8B, QUASAR achieves +5.54 points on VSI-Bench and +19.88 points on VSTI-Bench over the base model.
Critical strengths

The main strength of this paper, I think, is its direct visual adjustment approach.

Instead of forcing the model to build a complex 3D map of a room before answering a question, which may be slow, and memory-intensive, QUASAR applies small mathematical tweaks straight onto the raw visual features in place.

Also, great seeing the code and scripts to run it.

Critical weaknesses

There are 2 weakness of this paper I believe are critical, and one stylistic comment

  1. Tables 1 and 2 are not apples-to-apples comparisons. They compare QUASAR (built on Qwen3-VL-8B) against a mix of models using different backbones, input resolutions, and training recipes. There are no controlled, equal-budget target comparisons. More importantly, QUASAR is not the top performer; external memory models like Q-GeoMem and VLM^2 outperform QUASAR on VSI-Bench. Yes, they use external spatial memory, and I don’t think that disqualifies them.

  2. The method does not seem to generalize well. While it helps with left/right spatial relative questions, it degrades performance on temporal tasks like tracking object appearance order (Table 12) and causes negative transfer on external benchmarks like CV-Bench (Table 23) and ViewSpatial-Bench (Table 20). An analysis of why, at least, would be nice.

  3. [Style] Table 4 is supposedly very important, but it’s confusing and verbose (so is Figure 2). The Appendix reads like a dump of all experiments the authors made to get to the paper and is quite challenging to browse and grasp

Questions for authors

Please prioritize the weaknesses I raised above.

Other questions that could affect my assessment

  1. Why 4 latent updates and 4 summary tokens? Any ablations on that? Could be useful for people looking to adopt such an approach to understand the trade-offs.
  2. Why do the visual feature edits actively break basic temporal tasks like tracking object appearance order and require manually disabling writeback to avoid trailing baselines on CV-Bench?
  3. Why were QUASAR and the visual-only baseline evaluated at different SFT step checkpoints before GRPO? How much of the performance gap is just down to unequal training exposure?
Reviewer_nVuoOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the QUASAR framework, designed to address the mismatch between the observation viewpoint and the answer reference in video spatial question-answering. The method utilizes estimated observation geometry, along with a given reference type and numerical pose, to generate bounded residuals via a spatial latent variable transformation module and write them back to the original visual features. The model then combines the adjusted visual information with the complete question to generate an answer. Experiments demonstrate that QUASAR achieves performance improvements on VSI-Bench and VSTI-Bench.

Critical strengths
  1. The research question is valuable, and the overall framework is relatively clear. Distinguishing between the camera pose at the time of observation and the reference required for answering is a key challenge in video spatial reasoning.

  2. The experiments demonstrate competitive performance. Compared to the baseline, QUASAR achieves improvements of 1.75 and 3.93 percentage points on VSI-Bench and VSTI-Bench, respectively.

Critical weaknesses
  1. I view the work presented in this paper as an incremental innovation. The paper’s main components—including conditional re-encoding, geometric residual injection, and spatial auxiliary supervision—have all been explored in previous work. The authors may need to clarify what the core differences are between this paper and prior work, and whether the presented experiments support their claims.

  2. Further ablation studies are needed to evaluate the updated strategy. Should the authors include comparisons with alternative update methods—such as simple conditional projection, analytical coordinate transformations, or single-path residuals—against the full SLT to demonstrate which components contribute stable and independent gains?

  3. The validation of reference-conditioned reasoning remains insufficient. I would like to see that, when the reference specified in the question and the numerical pose change consistently, the model gives the correct answer under the new reference. The existing experiments support the usefulness of reference matching, but do not yet establish its independent benefit over general geometric feature enhancement or the correctness of reasoning under a changed reference.

Questions for authors

See weakness.

My main concern is what distinguishes this paper from general geometric feature enhancement approaches, and whether the paper’s conclusions support this view. Based on that, if the author resolves my issue, I will raise my rating.

Reviewer_ay6BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this paper, the authors propose a new framework QUASAR to deal with spatial reasoning, where a video MLLM needs to answer a question requiring viewpoint-dependent knowledge. In QUASAR, a Spatial Latent Transformation uses estimated geometry to add bounded corrections to existing visual features, allowing the video MLLM to answer without generating a separate scene representation. Experiments on VSI-Bench and VSTI-Bench show that the proposed framework improve performance on vision-only baseline.

Critical strengths
  • The motivation is clear that observation-to-query bottleneck identifies why remembering visual evidence is insufficient when spatial relations must be judged from another viewpoint.

  • Reference-conditioned bounded feature corrections preserve and augment original visual features with spatial information directly related to question answering.

Critical weaknesses
  • The proposed method seems contradict with the motivation that there is no need to generate a separate scene representation. The proposed framework already leverages CUT3R to construct geometry and poses for the scene which is a kind of representation of the video scene.

  • The performance of the model is not impressive on the benchmarks. Although the final model outperforms the vision-only sft baseline, the performance is lower than other methods with spatial memory methods.

  • Some notations are missing in the paper and make it hard to understand comprehensively. For example, in Section 3.2, the meaning of semantic-anchor, scene-canonical and how they are obtained is not clarified. In Section 3.3, there is clarify of how the point, confidence, validity are obtained since if the authors are not familiar with CUT3R, they will be confused with the function of CUT3R

Questions for authors

Please refer to weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.