acceptodds
Withdrawn from ICLR 2027

Spatial Latent Reasoning for Embodied Reference Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pointing-gesture visual grounding requires connecting hand geometry with the visual identity and extent of a referred object. A central challenge for continuous latent reasoning is how to organize these complementary cues into useful intermediate supervision. We propose Spatial Latent Reasoning (SLR), a framework that structures this supervision around an ordered sequence of geometric and visual states. A spatial ray state is supervised by fingertip position and pointing direction, followed by four states aligned with target-region features. To construct the visual targets, we introduce parity pooling, which applies polyphase grouping to average region tokens on four interleaved spatial supports. All states are generated recurrently during training and inference; auxiliary annotations are required only during training. On EgoPoint-Ground, the framework improves mIoU over same-backbone supervised fine-tuning by 2.8, 17.5, and 21.1 percentage points on Qwen3.5-4B, Qwen2.5-VL-7B, and Qwen3-VL-8B, respectively, with improvements on both hard subsets. On YouRefIt, it achieves 77.6% precision at IoU 0.5, a numerical margin of 5.2 percentage points over the reported state of the art under differing evaluation protocols. Ablations support joint geometric and visual supervision on the standard and similar-object sets, and favor parity over three alternative pooling operators on the standard set. These results support task-structured supervision for continuous pointing grounding. We will release the code and supporting materials.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Continuous latent reasoning, Multimodal large language models, Spatial supervision, Parity pooling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UnNWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Spatial Latent Reasoning (SLR), a framework designed for pointing-gesture visual grounding in embodied reference understanding. Built upon continuous latent reasoning paradigms (such as Coconut and Latent Visual Reasoning), SLR introduces a 5-step recurrent latent sequence before autoregressively decoding the bounding box. Auxiliary readouts and annotations are used exclusively during training; during inference, the model generates latent states recurrently and greedily predicts the bounding box. The authors report results on EgoPoint-Ground (Standard, Hard-Similar, and Hard-Complex) and YouRefIt using Qwen2.5-VL-7B, Qwen3-VL-8B, and Qwen3.5-4B backbones.

Critical strengths
  1. Integrating human pointing gestures with multi-modal continuous reasoning is a practical and interesting challenge for embodied interaction.
  2. The authors conduct evaluations across several recent MLLM backbones (Qwen2.5-VL, Qwen3-VL, Qwen3.5-4B) and report both standard and challenging split metrics.
Critical weaknesses
  1. The authors motivate parity pooling using 2D polyphase decomposition from classical multirate signal processing. However, this conceptual mapping to vision transformer representations is fundamentally flawed. In multirate signal processing, polyphase components operate on raw spatial grids where Nyquist sampling and spatial shift properties hold. In contrast, the tokens in V=Eϕ(I)V = \mathcal{E}_\phi(I) have already undergone deep self-attention layers with broad receptive fields and positional embeddings. Consequently, adjacent patch tokens in a ViT are already globally contextualized and spatially mixed. Splitting tokens by index parity does not extract distinct phase frequencies; it merely acts as an arbitrary checkerboard subsampling heuristic.

  2. Incoherent and contradictory ablation study results. In Table 4 (Hard-Similar), vanilla SFT achieves 0.361 mIoU, while Spatial-only achieves only 0.286 mIoU (a drop of -7.5%). Why does supervising pointing geometry alone lead to catastrophic degradation in distinguishing visually similar objects? In Table 4 (Hard-Complex), Target-only achieves 0.597 mIoU, while the full SLR model (Spatial + Parity) achieves only 0.551 mIoU (a drop of -4.6%). In other words, adding the proposed spatial ray state actively harms performance in complex scenes.

Questions for authors
  1. Why does LVR perform so poorly on Qwen3.5-4B (0.525 mIoU vs. 0.750 for SFT)? Was LVR trained with equivalent effective batch sizes, learning rates, and target alignment weights λ\lambda?
  2. How do you reconcile the claim that hsph_{sp} encodes pointing geometry with the empirical observation in Figure 5 that hsph_{sp} assigns almost negligible attention mass to image tokens? If you zero out the image attention for hsph_{sp} at test time, does final grounding accuracy change?
2 comments

Comment:

We sincerely thank the reviewer for the careful reading of our paper and the thoughtful, constructive feedback. \texttt{We sincerely thank the reviewer for the careful reading of our paper and the thoughtful, constructive feedback. }Your comments have helped us clarify the theoretical motivation, experimental design, and interpretation of our results. We address each concern below.

Q1: Connection between parity pooling and 2D polyphase decomposition

Thank you for highlighting the distinction between index-based sampling phases and the physical or frequency phases of the original image. Our connection concerns the former.

Let the visual encoder and merger output a feature grid V∈RHg×Wg×dV\in\mathbb{R}^{H_g\times W_g\times d}. Despite global contextual mixing, each output token retains a defined grid position. We can therefore define four sampling phases:

Pab(V)[i,j]=V[2i+a,2j+b],a,b∈{0,1}.\mathcal{P}_{ab}(V)[i,j] = V[2i+a,2j+b], \quad a,b\in\{0,1\}.

This follows the classical polyphase construction of interleaved subsequences indexed by their remainders. Applying it to vector-valued features requires neither independence nor strictly local information at each position (Smith, 2011).

SLR then averages each phase within the target-region mask MbM_b:

\frac{\sum_{i,j}M_b(2i+a,2j+b)V_{2i+a,2j+b}} {\sum_{i,j}M_b(2i+a,2j+b)}.$$ Empty groups are excluded from the alignment loss. The indices refer to the **post-merge visual grid**, not the full sequence containing text and special tokens. The branches indeed perform interleaved subsampling; “polyphase” describes their complete partition by sampling phase. Our contribution is to use these region-conditioned phase means for continuous-state supervision. Global feature mixing does not invalidate the indexing identity, although it does not establish independent semantics, anti-aliasing, or translation invariance. Averaging is lossy, and its supervision value is evaluated empirically. ### Q2: Why Spatial-only underperforms SFT on Hard-Similar Thank you for pointing out this result. Spatial-only achieves **0.286 mIoU**, compared with **0.361 for SFT**, showing that a single spatial state with geometric supervision does not improve this subset by itself. Spatial-only still receives the original image and bounding-box answer supervision. Its auxiliary labels constrain fingertip position and direction, but do not specify the target’s appearance, identity, or full extent. A direction may therefore remain compatible with several similar candidates. A plausible explanation is that the added geometric constraint changes the representation or optimization trade-off without sufficient benefit for instance selection. Joint supervision reaches **0.421 mIoU**, exceeding Target-only at **0.369**, consistent with combining geometry and target visual supervision. The full model also adds a recurrent state, so fixed-budget controls are needed to isolate the spatial loss. Our conclusion is therefore that the joint configuration benefits Hard-Similar. Further matched-budget comparisons and analyses of object-selection versus box-localization errors will help explain the Spatial-only decrease. ### Q3: Why Full SLR underperforms Target-only on Hard-Complex We acknowledge this limitation. On Hard-Complex, adding the spatial state reduces mIoU from **0.597 to 0.551**, a decrease of 4.6 percentage points. The fixed geometry–vision configuration is therefore not equally effective across scene conditions. The 2D finger-base-to-fingertip labels do not directly provide depth, occlusion relationships, or complete relations among competing objects. Geometric uncertainty and trade-offs between auxiliary supervision and localization may affect performance. We have not yet separated these factors through stratified evaluation or gradient analysis. This ablation also changes both the state count and supervision configuration. Target-only performs best on Hard-Complex, while Full SLR performs better on Standard and Hard-Similar. We will make this scene-dependent behavior explicit in the paper.
AuthorsOpenReview ↗

Continued Official Comment for Reviewer UnNW

Q4: LVR adaptation and comparison with SFT

We adapt LVR’s first-stage training to PointVG. Projected visual tokens are selected using the annotated target box and ordered row by row, with a median of 48 tokens per sample. The preceding hidden state predicts the current teacher-forced region embedding using raw feature MSE with weight 0.1, alongside next-token prediction. At inference, LVR receives only the image and question and performs four recurrent steps after the latent-start marker.

Training starts from the original Qwen3.5-4B weights with a base learning rate of 10−510^{-5}. Following the original paper, LVR uses an effective batch size of 24, 2,500 optimizer updates, cosine decay after 3% warm-up, and constrained bounding-box decoding. The training data, optimization budget, and evaluation protocol are the same as those used for the other methods.

The discrepancy between teacher-forced region tokens during training and model-generated states during inference contributes to the performance gap. Another contributing factor is the degradation of visual information with longer latent-token sequences.

Q5: Low image-attention mass of the spatial state

Thank you for highlighting this observation. Figure 5 averages direct attention over all 16 heads in layers 20, 24, 28, and 32. It measures routing in these selected layers, rather than the total visual information contained in the final state.

Visual information can reach the spatial state through earlier layers, residual connections, or contextual representations that already contain visual information. Averaging may also obscure specialized heads. The current visualization does not identify which pathway dominates. Moreover, attention weights alone do not account for value vectors, output projections, or residual accumulation (Abnar and Zuidema, 2020).

The left panel is normalized within image tokens, whereas the right panel reports attention mass across image, context, and previous-state tokens. A concentrated image heatmap can therefore coexist with low total image-attention mass.

We treat Figure 5 as descriptive evidence. Fingertip and angular errors assess geometric encoding, while intervention experiments assess its influence on localization.

Q6: Test-time intervention on image attention

Thank you for suggesting this experiment. We will use the same validation-selected Qwen3.5-4B Full SLR checkpoint as in the main experiments, freeze all parameters, and evaluate Standard and both hard subsets.

When generating the spatial state hsph_{\mathrm{sp}}, we will mask connections from its producer query position to image tokens in all attention heads of layers 20, 24, 28, and 32, then regenerate the subsequent states and answer.

We will compare three conditions: no masking, image-connection masking, and masking non-image connections with comparable attention mass. All conditions will use greedy decoding, a 96-token output limit, and disabled caching.

We will report mIoU and per-image paired IoU differences relative to the unmasked baseline, and add the results to the main text as soon as they become available.

We have revised the presentation and are working intensively to complete the additional experiments. We have devoted substantial effort to this work and sincerely appreciate your careful consideration of our responses. We would be grateful if you would consider raising your score.\texttt{We would be grateful if you would consider raising your score.}

Reviewer_DhoMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Spatial Latent Reasoning (SLR) for pointing-based visual grounding. It introduces a geometry-supervised latent state followed by four parity-pooled visual states, enabling continuous reasoning without test-time auxiliary annotations. Experiments on EgoPoint-Ground and YouRefIt demonstrate improvements over several baselines.

Critical strengths
  1. Task-specific supervision: The combination of pointing geometry and target-region features is well motivated. The method is clearly formulated and requires no auxiliary annotations during inference.
  2. Promising empirical results: SLR consistently improves mIoU over same-backbone SFT across three backbones and multiple evaluation sets. The component and pooling ablations provide useful empirical evidence.
Critical weaknesses
  1. The paper title does not appear to follow the small-caps typography required by the ICLR 2027 template. The authors should ensure that the final manuscript conforms to the official formatting requirements.
  2. Insufficient validation of the proposed latent reasoning mechanism: Although SLR introduces task-specific geometric and visual supervision, its fundamental components are closely related to existing latent reasoning methods (e.g., LVR and RIS). The current ablations do not sufficiently isolate the contribution of the ordered recurrent latent states from that of auxiliary supervision. Furthermore, using four visual states provides only a marginal improvement over one state in the target-only setting (0.764 vs. 0.761 mIoU), leaving the necessity of the proposed state organization unclear.
  3. Limited generality of component improvements and unclear baseline performance: On Hard-Complex, Target-only outperforms the full SLR model (0.597 vs. 0.551 mIoU), suggesting that geometric supervision does not consistently improve grounding and requiring further explanation. Additionally, LVR performs substantially worse than standard SFT on Qwen3.5-4B (0.525 vs. 0.750 mIoU). More details on its adaptation and training protocol would help establish the reliability of this comparison.
Questions for authors
  1. Can the authors provide matched controls with the same latent-state budget but without auxiliary supervision or with different supervision orders, to clarify whether the improvements stem from the proposed structured latent reasoning mechanism?
  2. Why does Target-only outperform the full SLR model on Hard-Complex (0.597 vs. 0.551 mIoU)? Does this suggest that spatial supervision may negatively affect performance in certain scenarios?
  3. Why does LVR perform substantially worse than standard SFT (0.525 vs. 0.750 mIoU)? Could the authors clarify whether the training data, optimization budgets, and evaluation protocols were comparable across these baselines?
2 comments

We sincerely thank the reviewer for the time and effort devoted to reviewing our work and for the thoughtful, constructive feedback. We address each question below.

Q1: Title formatting

Thank you for pointing this out. We checked the submitted PDF and confirmed that the title uses regular capitalization. The \normalfont command overrides the small caps applied by the official style. We will remove this override, use the official \maketitle, and check the first page after recompilation.

Q2: Contribution of the ordered recurrent latent states

SLR builds on LVR’s continuous-state feedback and visual-feature alignment. RIS also combines spatial and semantic supervision in latent states. For pointing-based grounding, we introduce a specific division of roles: the first state learns an explicit pointing ray, followed by four states aligned with interleaved target-region features. The target states can therefore use the preceding geometric state when modeling the referent’s appearance and extent. During both training and inference, all states fed into subsequent steps are generated by the model.

The component ablations examine the individual and joint contributions of geometric and target visual supervision, while the pooling ablations compare different target-feature constructions. Together, these experiments support our supervision design.

To further distinguish the sources of improvement, we vary two factors separately: auxiliary losses with fixed slots and order, and supervision order with fixed losses and state budget. We are also completing the Ray + Target versus Hand-mask + Target comparison across Standard, Hard-Similar, and Hard-Complex. We will report these additional results as soon as they become available.

Q3: Fixed-budget controls without auxiliary supervision and with different orders

Thank you for this suggestion. We agree that auxiliary supervision and supervision order should be examined under the same state budget.

Specifically, we fix five model-generated latent states and compare three settings:

  • Answer cross-entropy only, without auxiliary supervision.
  • The original order: spatial → four target visual states.
  • The alternative order: four target visual states → spatial.

All groups use the same training budget and checkpoint-selection protocol, with supervision assigned according to each state’s role. We are conducting these comparisons and will report the results as soon as they are available.

Q4: Motivation for four target visual states

Four visual states naturally correspond to the four interleaved supports produced by the 2D parity partition, providing a structured supervision scheme.

Our motivation is to organize variable-sized target regions into four phase-specific feature summaries. This provides grouped visual supervision for the continuous states and reduces reliance on a single aggregated target. We will further assess the stability of the results through repeated, multi-seed experiments and evaluate robustness using targeted perturbation tests.

Q5: Lower performance on Hard-Complex

In Table 4, Full SLR has an mIoU 4.6 percentage points below Target-only. In the per-image comparison, Full improves IoU on 48 images, reduces it on 45, and leaves three unchanged. The average decrease among degraded examples is larger than the average increase among improved examples.

All 96 annotated forward rays intersect the target boxes. However, even when a ray passes through the correct target, it may also pass through other objects in a cluttered scene. Moreover, checking the annotations does not establish the accuracy of the model’s estimated direction. The current evidence therefore does not determine the specific cause of the decrease.

Q6: LVR adaptation, training budget, and evaluation protocol

We adapt LVR’s first-stage training to PointVG. During training, projected visual tokens are selected using the annotated target box and ordered row by row, with a median of 48 tokens per sample. The preceding hidden state predicts the current teacher-forced region-token embedding using raw feature MSE with a weight of 0.1, alongside next-token prediction. At inference, LVR receives only the image and question and performs four recurrent steps after the latent-start marker.

Training starts from the original Qwen3.5-4B weights with a base learning rate of 10−510^{-5}. Following the original paper, LVR uses an effective batch size of 24, 2,500 optimizer updates, cosine decay after 3% warm-up, and constrained bounding-box decoding. The training data, optimization budget, and evaluation protocol are the same as those used for the other methods.

The discrepancy between teacher-forced region tokens during training and model-generated states during inference contributes to the performance gap. Another contributing factor is the degradation of visual information with longer latent-token sequences.

AuthorsOpenReview ↗

Continued Comments to Reviewer DhoM

Thank you for your comments and questions. We have corrected the writing issues in the manuscript and are working diligently to complete the additional experiments. We will provide the corresponding results tables as soon as they are available. We have devoted substantial effort to this work and sincerely appreciate your careful consideration of our responses. If our clarifications and additional results adequately address your concerns, we would be grateful if you would consider raising your score.\texttt{If our clarifications and additional results adequately address your concerns, we would be grateful if you would consider raising your score.}

Reviewer_1zKLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. The paper proposes a recurrent latent reasoning framework for visual grounding using pointing-gesture. It utilizes finger pointing ray as an auxiliary geometric guidance for spatial latent and visual target features extracted using parity pooling for subsequent latent supervision.
  2. It claims above supervision acts as a complementary information to improve spatial grounding for visually similar and complex objects.
  3. Evaluation conducted on EgoPoint-Ground and YouRefIt shows improved performance compared to existing methods. Ablations conducted on additional two separate hard evaluation sets also support the claimed improvements.
Critical strengths
  1. The paper is well written and easier to understand.
  2. Supervising using point ray derived from finger base to finger tip seems intuitive in case of first person view.
  3. Parity pooling is an innovative approach due to its adaptive nature in variable sized target ROI as it assigns each recurrent state a consistent sampling rule irrespective of region size.
Critical weaknesses
  1. Early improper definition or nonreferenced keywords such as four states, latent budgets etc breaks the coherence of the paper even though its meaning is cleared after reading subsequent sections of the paper.
  2. Adding auxiliary supervision of finger tip position and direction seems plausible only for non-occluded cases and orthogonal viewing axis. Does the evaluation set, Hard-Complex, account for those cases? If so, add details about those evaluation datasets. Similarly, its effect on very closed objects when point ray is visually indistinctive.
  3. Ray states alone doesn't improve much as shown in ablation studies. Some experiments to show how it acts as a complementary information could strengthen the claim of the paper.
  4. Ablations of some other supervision corresponding to hand such as hand segmentation instead of pointing ray could be useful to differentiate the importance of pointing ray vs semantic mask as it can also enforce geometric supervision implicitly.
  5. What is the idea of using the generated state as h(ab)=z(2+2a+b)? Is there some relation between order of recurrent latent states and this formula or just the idea constraining it? If so, need to show how different formulations or order impact the target support?
  6. The conclusion section looks incomplete.
Questions for authors
  1. It will be good to see how the pointing ray acts as a complementary signal and its interpretation for hard-similar or complex cases.
  2. Ablations of some other modality i.e finger or hand mask to distinguish it from the proposed geometric approach.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Continued Comments for Reviewer 1zKL

Q4: Complementarity between ray and target visual supervision

Ray supervision provides a starting point and direction for target search. Target visual supervision encourages learning the referent’s appearance and extent through features pooled from four interleaved supports.

On Hard-Similar, Full SLR achieves 0.421 mIoU, compared with 0.369 for Target-only and 0.286 for Spatial-only. Relative to Target-only, IoU improves on 52 images. Twelve examples cross the IoU 0.7 threshold from incorrect to correct, while overall [email protected] increases from 0.344 to 0.396. Full SLR also improves Standard mIoU from 0.764 to 0.778.

These results support complementarity in similar-object scenes. Target-only remains stronger on Hard-Complex, showing that the benefit is conditional.

We are currently conducting fixed-budget experiments with K=5K=5, comparing Full SLR, Target-only, and Ray-only under matched training conditions. These controls separate auxiliary-supervision effects from differences in recurrent computation. We will report the results as soon as they become available.

Q5: Ray supervision versus hand-mask supervision

Hand or finger masks capture shape and spatial occupancy, providing indirect gesture information. Ray supervision explicitly specifies the fingertip location and the direction extending outward from it, which directly relate to target search.

An earlier control supervised the first state using features pooled within the hand bounding box, retaining the same four target visual states. This hand-region appearance control achieved 0.745 mIoU on Standard.

We are currently conducting a matched five-state ablation comparing ray supervision with hand pseudo-mask supervision generated by SAM 2.1, using the hand bounding box as its prompt. The four target-alignment objectives, training samples, and training budget are held constant. We will report the results as soon as they become available.

Q6: Mapping parity groups to recurrent latent states

The mapping

hab=z2+2a+b,a,b∈{0,1},h_{ab}=z_{2+2a+b}, \qquad a,b\in\{0,1\},

assigns the parity groups to the target-state slots in row-major order:

(00,01,10,11)⟼(z2,z3,z4,z5).(00,01,10,11)\longmapsto(z_2,z_3,z_4,z_5).

The first slot, z1z_1, is reserved for the spatial ray state.

Permuting this assignment preserves each group’s spatial support but changes its position in the autoregressive sequence. Row-major ordering is a fixed convention; unchanged sampling supports do not imply unchanged performance. We do not claim that this ordering is optimal or that performance is invariant to alternative orders.

Q7: Completing the conclusion

We will expand the conclusion to summarize the method, results, and applicability. The proposed text is:

We presented Spatial Latent Reasoning (SLR) for pointing-based visual grounding. SLR generates one continuous state supervised by fingertip position and pointing direction, followed by four visual states aligned with interleaved target-region features. At inference, all five states are generated from the image and query without auxiliary annotations. On Qwen3.5-4B, SLR improves EgoPoint-Ground Standard mIoU from 0.750 to 0.778 and achieves 77.6% [email protected] on YouRefIt. Joint supervision outperforms either single-cue configuration on Hard-Similar, whereas Target-only performs best on Hard-Complex. These results indicate that the benefits of geometric and target visual supervision depend on scene conditions.

Thank you for your comments and questions. We have corrected the writing issues in the manuscript and are working diligently to complete the additional experiments. We will provide the corresponding results tables as soon as they are available. We have devoted substantial effort to this work and sincerely appreciate your careful consideration of our responses. If our clarifications and additional results adequately address your concerns, we would be grateful if you would consider raising your score.\texttt{If our clarifications and additional results adequately address your concerns, we would be grateful if you would consider raising your score.}

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.