- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents Spatial Latent Reasoning (SLR), a framework designed for pointing-gesture visual grounding in embodied reference understanding. Built upon continuous latent reasoning paradigms (such as Coconut and Latent Visual Reasoning), SLR introduces a 5-step recurrent latent sequence before autoregressively decoding the bounding box. Auxiliary readouts and annotations are used exclusively during training; during inference, the model generates latent states recurrently and greedily predicts the bounding box. The authors report results on EgoPoint-Ground (Standard, Hard-Similar, and Hard-Complex) and YouRefIt using Qwen2.5-VL-7B, Qwen3-VL-8B, and Qwen3.5-4B backbones.
Critical strengths
- Integrating human pointing gestures with multi-modal continuous reasoning is a practical and interesting challenge for embodied interaction.
- The authors conduct evaluations across several recent MLLM backbones (Qwen2.5-VL, Qwen3-VL, Qwen3.5-4B) and report both standard and challenging split metrics.
Critical weaknesses
-
The authors motivate parity pooling using 2D polyphase decomposition from classical multirate signal processing. However, this conceptual mapping to vision transformer representations is fundamentally flawed. In multirate signal processing, polyphase components operate on raw spatial grids where Nyquist sampling and spatial shift properties hold. In contrast, the tokens in have already undergone deep self-attention layers with broad receptive fields and positional embeddings. Consequently, adjacent patch tokens in a ViT are already globally contextualized and spatially mixed. Splitting tokens by index parity does not extract distinct phase frequencies; it merely acts as an arbitrary checkerboard subsampling heuristic.
-
Incoherent and contradictory ablation study results. In Table 4 (Hard-Similar), vanilla SFT achieves 0.361 mIoU, while Spatial-only achieves only 0.286 mIoU (a drop of -7.5%). Why does supervising pointing geometry alone lead to catastrophic degradation in distinguishing visually similar objects? In Table 4 (Hard-Complex), Target-only achieves 0.597 mIoU, while the full SLR model (Spatial + Parity) achieves only 0.551 mIoU (a drop of -4.6%). In other words, adding the proposed spatial ray state actively harms performance in complex scenes.
Questions for authors
- Why does LVR perform so poorly on Qwen3.5-4B (0.525 mIoU vs. 0.750 for SFT)? Was LVR trained with equivalent effective batch sizes, learning rates, and target alignment weights ?
- How do you reconcile the claim that encodes pointing geometry with the empirical observation in Figure 5 that assigns almost negligible attention mass to image tokens? If you zero out the image attention for at test time, does final grounding accuracy change?
2 comments
Continued Official Comment for Reviewer UnNW
Q4: LVR adaptation and comparison with SFT
We adapt LVR’s first-stage training to PointVG. Projected visual tokens are selected using the annotated target box and ordered row by row, with a median of 48 tokens per sample. The preceding hidden state predicts the current teacher-forced region embedding using raw feature MSE with weight 0.1, alongside next-token prediction. At inference, LVR receives only the image and question and performs four recurrent steps after the latent-start marker.
Training starts from the original Qwen3.5-4B weights with a base learning rate of . Following the original paper, LVR uses an effective batch size of 24, 2,500 optimizer updates, cosine decay after 3% warm-up, and constrained bounding-box decoding. The training data, optimization budget, and evaluation protocol are the same as those used for the other methods.
The discrepancy between teacher-forced region tokens during training and model-generated states during inference contributes to the performance gap. Another contributing factor is the degradation of visual information with longer latent-token sequences.
Q5: Low image-attention mass of the spatial state
Thank you for highlighting this observation. Figure 5 averages direct attention over all 16 heads in layers 20, 24, 28, and 32. It measures routing in these selected layers, rather than the total visual information contained in the final state.
Visual information can reach the spatial state through earlier layers, residual connections, or contextual representations that already contain visual information. Averaging may also obscure specialized heads. The current visualization does not identify which pathway dominates. Moreover, attention weights alone do not account for value vectors, output projections, or residual accumulation (Abnar and Zuidema, 2020).
The left panel is normalized within image tokens, whereas the right panel reports attention mass across image, context, and previous-state tokens. A concentrated image heatmap can therefore coexist with low total image-attention mass.
We treat Figure 5 as descriptive evidence. Fingertip and angular errors assess geometric encoding, while intervention experiments assess its influence on localization.
Q6: Test-time intervention on image attention
Thank you for suggesting this experiment. We will use the same validation-selected Qwen3.5-4B Full SLR checkpoint as in the main experiments, freeze all parameters, and evaluate Standard and both hard subsets.
When generating the spatial state , we will mask connections from its producer query position to image tokens in all attention heads of layers 20, 24, 28, and 32, then regenerate the subsequent states and answer.
We will compare three conditions: no masking, image-connection masking, and masking non-image connections with comparable attention mass. All conditions will use greedy decoding, a 96-token output limit, and disabled caching.
We will report mIoU and per-image paired IoU differences relative to the unmasked baseline, and add the results to the main text as soon as they become available.
We have revised the presentation and are working intensively to complete the additional experiments. We have devoted substantial effort to this work and sincerely appreciate your careful consideration of our responses.
Comment:
We sincerely thank the reviewer for the careful reading of our paper and the thoughtful, constructive feedback. Your comments have helped us clarify the theoretical motivation, experimental design, and interpretation of our results. We address each concern below.
Q1: Connection between parity pooling and 2D polyphase decomposition
Thank you for highlighting the distinction between index-based sampling phases and the physical or frequency phases of the original image. Our connection concerns the former.
Let the visual encoder and merger output a feature grid V∈RHg×Wg×d. Despite global contextual mixing, each output token retains a defined grid position. We can therefore define four sampling phases:
Pab(V)[i,j]=V[2i+a,2j+b],a,b∈{0,1}.
This follows the classical polyphase construction of interleaved subsequences indexed by their remainders. Applying it to vector-valued features requires neither independence nor strictly local information at each position (Smith, 2011).
SLR then averages each phase within the target-region mask Mb:
\frac{\sum_{i,j}M_b(2i+a,2j+b)V_{2i+a,2j+b}} {\sum_{i,j}M_b(2i+a,2j+b)}.$$ Empty groups are excluded from the alignment loss. The indices refer to the **post-merge visual grid**, not the full sequence containing text and special tokens. The branches indeed perform interleaved subsampling; “polyphase” describes their complete partition by sampling phase. Our contribution is to use these region-conditioned phase means for continuous-state supervision. Global feature mixing does not invalidate the indexing identity, although it does not establish independent semantics, anti-aliasing, or translation invariance. Averaging is lossy, and its supervision value is evaluated empirically. ### Q2: Why Spatial-only underperforms SFT on Hard-Similar Thank you for pointing out this result. Spatial-only achieves **0.286 mIoU**, compared with **0.361 for SFT**, showing that a single spatial state with geometric supervision does not improve this subset by itself. Spatial-only still receives the original image and bounding-box answer supervision. Its auxiliary labels constrain fingertip position and direction, but do not specify the target’s appearance, identity, or full extent. A direction may therefore remain compatible with several similar candidates. A plausible explanation is that the added geometric constraint changes the representation or optimization trade-off without sufficient benefit for instance selection. Joint supervision reaches **0.421 mIoU**, exceeding Target-only at **0.369**, consistent with combining geometry and target visual supervision. The full model also adds a recurrent state, so fixed-budget controls are needed to isolate the spatial loss. Our conclusion is therefore that the joint configuration benefits Hard-Similar. Further matched-budget comparisons and analyses of object-selection versus box-localization errors will help explain the Spatial-only decrease. ### Q3: Why Full SLR underperforms Target-only on Hard-Complex We acknowledge this limitation. On Hard-Complex, adding the spatial state reduces mIoU from **0.597 to 0.551**, a decrease of 4.6 percentage points. The fixed geometry–vision configuration is therefore not equally effective across scene conditions. The 2D finger-base-to-fingertip labels do not directly provide depth, occlusion relationships, or complete relations among competing objects. Geometric uncertainty and trade-offs between auxiliary supervision and localization may affect performance. We have not yet separated these factors through stratified evaluation or gradient analysis. This ablation also changes both the state count and supervision configuration. Target-only performs best on Hard-Complex, while Full SLR performs better on Standard and Hard-Similar. We will make this scene-dependent behavior explicit in the paper.