acceptodds
Under review as a conference paper at ICLR 2027

Look, Reason, Draw a Region: Recursive Refinement for Spatial Context Prediction

Abstract

Spatial context prediction requires visual grounding and spatial reasoning to identify vacant regions relative to objects in a scene. Spatial relationships and physical constraints remain challenging for large vision-language models (VLMs). Recursive reasoning has demonstrated strong performance on discrete and abstract reasoning tasks, motivating its use for improving spatial context prediction in natural images. Weight sharing enables higher accuracy with a small trainable parameter count. We introduce Spatial-TRM, a compact recursive model with frozen vision-language features, a convex region decoder, and just 11.2M trainable parameters. This weight-tied reasoner supports six spatial relationships, three reference frames, and open-vocabulary anchor objects. On the RoboSpatial-Home benchmark, Spatial-TRM achieves 29.51% context accuracy, a gain of 9.02 percentage points (44.0% relative) over a separately trained single-step model with the same parameter count. Including the frozen encoders, the model has 877.8M total parameters, approximately 15× fewer than the reported 13B size of RoboPoint. These results show that repeated refinement improves spatial context prediction from pretrained vision-language features at a fixed trainable parameter count on this benchmark.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
spatial reasoning, spatial context prediction, recursive refinement, visual grounding, vision-language learning, parameter-efficient learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zuknOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a methodology for addressing the problem of spatial context prediction, which can be formulated as as identifying vacant regions within a scene by analyzing the relative spatial relationships between objects. This task requires both the ability to ground vacant spaces and the capacity for spatial reasoning. To this end, the paper introduces a Spatial-TRM adopting recursive reasoning mechanism. As a compact recursive model, it utilizes a relatively small number of trainable parameters (11.2M in total model). Through this approach, the Spatial-TRM achieve superior performance compared to existing methods on the RoboSpatial-Home benchmark dataset.

Critical strengths
  • To address the problem of spatial context prediction, this study effectively combines several existing approaches including representation learning (w/ DINOv3 ViT-L/16) and recursive refinement (Tiny Recursive Model; TRM. Therefore, the proposed method contributes to the achievement of superior performance compared to open-weight models and other previous studies.
  • By training only a very small fraction (11.2M) of the total model size, we achieved spatial context prediction performance that surpasses that of open-weight models.
Critical weaknesses
  • In the manuscript, there are some sentences including "dino.txt" in lines 151, 157, and 783. Although the authors stated that they used generative AI to improve language editing, clarity, and readability, this appears to have actually hindered readability.
  • Table 2 should include information on the encoder used for each model, so as to clearly demonstrate which part of the model contributed to the performance improvements.
  • Some tables mentioned in the paper are not actually provide. For example, Table 19 (referenced at line 376) and Table 4 (referenced at line 375). Please correct this if I have simply overlooked them.
  • In Table 2, the number of parameters are not marked on some of open-weight and commercial models. This prohibits the clear comparison between other models provided in Table 1.
  • Contrary to other mathematical notations, the pixel indicator p is not italic in line 222.
Questions for authors
  • In Figure 1, the explanation regarding the information intended to be conveyed to the reader through the mask on the right is insufficient. While the caption briefly refers to these as "target points," it appears that the white areas in the figure do not actually correspond to "vacant regions." Further clarification on this point is required.
  • In line 258, what is the ACT? Please provide the full name of ACT.
2 comments
AuthorsOpenReview ↗

Response to Reviewer zukn

Thank you for these detailed comments. We have clarified the terminology, comparison tables and figure explanation.

W1: Meaning of dino.txt

The notation was unclear. dino.txt is the name of the text-aligned DINO model introduced in DINOv2 Meets Text (Jose et al., 2025). We now typeset it in monospace and explain in Section 4.1 that we use its frozen text tower with DINOv3 image features.

W2: Encoder details and their contribution

We added a compact encoder table in Appendix E, Table 5, with links to released configurations and model cards. The main comparison, now Table 3, points to this appendix.

Encoder labels alone cannot attribute differences between models, because training data, language backbones and inference procedures also differ. Table 4 therefore provides an encoder ablation within Spatial-TRM, keeping the remaining architecture and training configuration fixed:

EncoderContext accuracy (%)Total parameters
DINOv3 ViT-L/1629.51877.8M
CLIP ViT-L/1426.23438M
SigLIP-SO400M/1416.39888M

The CLIP configuration reduces total parameters by 50.1%, at a cost of 3.28 percentage points. Separately, the recursion ablation holds the frozen encoders and trainable parameter count fixed, distinguishing the refinement gain from a change in encoder.

W3: References to Tables 19 and 4

These refer to tables in external sources. We now identify them explicitly as Table 19 of the Gemini Robotics 1.5 report and Table 4 of the RoboSpatial paper, so they cannot be mistaken for missing tables in our manuscript.

W4: Parameter counts

Every open-weight model in Table 3 now has a parameter count. Spatial-TRM has 877.8M total parameters: 866.6M frozen and 11.2M trainable. The caption distinguishes this full-system count from the baselines’ nominal model sizes and notes that the SpaceTools count excludes its external tools. A dash denotes an undisclosed proprietary-model size.

W5: Pixel notation

The pixel variable p is now italicized consistently in the mathematical notation.

Q1: Interpretation of Figure 1

We corrected the image crop and clarified the caption. The white area is the convex hull of the annotated target points for this query. It is a supervision target, rather than a model prediction or a complete map of vacant space. The query asks for space to the bed’s left in the object’s reference frame.

Section 3 now explains that hulls constructed from sparse points can omit valid space and span occupied areas. The mask should therefore not be interpreted as a guarantee that every white pixel is physically vacant.

Q2: ACT

ACT stands for Adaptive Computation Time. It is defined at its first use in Section 4.2, and the revision also spells it out in Section 4.4.1.

Thank you for the detailed response and revisions. The authors’ responses have addressed most of my concerns, and the additional encoder table and ablation studies help clarify the model comparisons.

I appreciate the authors’ efforts to improve the manuscript and will raise my score accordingly.

Reviewer_GZg2OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

To address the limitations of vision-language models (VLMs) in understanding spatial relationships and identifying free space around objects, Spatial-TRM introduces recursive refinement into spatial context prediction. It uses a frozen vision-language encoder for feature extraction, cross-attention to localize reference objects, and a shared-weight recursive reasoning module to iteratively refine spatial representations. A convex-region decoder then predicts target regions. On RoboSpatial-Home, Spatial-TRM achieves 29.51% accuracy with 11.2 million trainable parameters, outperforming a single-step model with a comparable parameter count by 9.02 percentage points.

Critical strengths

None

Critical weaknesses
  1. Limited evaluation and baseline comparisons. Evaluation uses only 122 test samples from RoboSpatial-Home, with no validation on other datasets, making generalizability and robustness difficult to establish. Baseline comparisons mainly rely on results reported in prior work, without direct comparisons against recent, competitive spatial reasoning models under consistent settings. Thus, the method’s effectiveness and competitiveness remain insufficiently demonstrated.
  2. Insufficient visualizations and ablations. Limited qualitative examples and ablation experiments make prediction reliability difficult to assess. Independent ablations of key components, including the convex-region decoder and relational contrastive loss, are also missing, leaving their individual contributions and the necessity of the overall design unclear.
Questions for authors
  1. Could the method be compared with more recent, competitive visual spatial reasoning models under a unified evaluation setting?
  2. Could additional ablation studies clarify the contribution of each key component to the overall performance improvement?
3 comments
AuthorsOpenReview ↗

Response to reviewer GZg2

Thank you for identifying these gaps. We have expanded the direct comparisons, component ablations and qualitative analysis, and made the limits of the evaluation explicit.

W1/Q1: Competitive baselines under a common evaluation protocol

We added seven direct baseline evaluations using RGB images and text, the same RoboSpatial-Eval script, a one-point scoring rule, and output coordinates normalized to [0, 1].

ModelContext accuracy (%)
Qwen3-VL-2B7.38
Cosmos-Reason2-2B15.6
Qwen3-VL-4B22.95
Spatial-TRM29.51
Embodied-R1-3B-v133.6
Embodied-R1.5-8B-SFT50.82
RoboRefer-2B-SFT55.74
RoboBrain2.5-4B59.02

Table 3 marks these direct evaluations separately from literature scores. Spatial-TRM exceeds the two Qwen variants and Cosmos with 877.8M total parameters, including frozen encoders, but trails the stronger spatial models. Training data and architectures still differ, so these comparisons do not isolate architecture alone.

W1: Evaluation scale and generalization

The main benchmark still contains only 122 context questions; one answer changes accuracy by approximately 0.82 percentage points. We therefore keep the main conclusion specific to this benchmark and the controlled recursion comparison.

Appendix D adds Where2Place evaluation. Spatial-TRM scores 7.7% across all 100 questions, using manually extracted anchor objects. We evaluate ten points per question: one decoder interior point and nine evenly sampled foreground points. The score is the fraction inside the target mask, averaged across questions. Supported and unsupported relations use the same pipeline; this metric averages point correctness rather than accepting any single successful point.

Of the 100 questions, 45 use relations outside the training vocabulary. The remaining 55 score 9.8%. Removing positional qualifiers from the anchor descriptions leaves 41 questions, with 12.20% (50/410 points). These subsets describe differences in task coverage; they do not isolate their causal effects.

The low overall score exposes limitations beyond the trained task and limits claims of broad generalization. The 999-sample RoboSpatial validation set provides additional diagnostics but was used for checkpoint selection, so we do not count it as an independent test benchmark. Appendix A.3 also clarifies that the 29.51% checkpoint was selected using validation metrics, not the 31.97% benchmark peak.

W2/Q2: Component ablations

Table 1 now reports the following ablations:

VariantContext accuracy (%)Change (percentage points)
Full model29.51—
Without cross-attention22.95−6.56
Without relational contrastive loss19.67−9.84
Separately trained single-step model20.49−9.02
Feedforward model, bypassing the reasoning core17.21−12.30
Linear decoder with highest-logit answer point21.31−8.20

The single-step control retains the same frozen encoders and 11.2M trainable parameters, isolating the benefit of repeated refinement at fixed parameter count. The other architectural ablations can change model capacity.

The decoder comparison requires an additional distinction. With the same highest-logit point-selection rule, the full decoder configuration scores 24.59%, versus 21.31% for the linear variant: a 3.28-point gain. Using the full decoder’s interior point adds 4.92 points, reaching 29.51%. The complete 8.20-point difference therefore combines decoder configuration and point selection. Trainable parameters also differ, at 11.2M versus 8.9M, so this comparison does not isolate convexity alone.

Mask overlap remains similar: IoU is 0.11 versus 0.10, and Dice is 0.16 for both. We therefore distinguish the point-accuracy improvement from evidence about region quality.

W2: Qualitative reliability

Figure 4 adds examples with correct answer points and failures involving anchor orientation, object extent, object height and grounding. Sections 5–7 explain that a correct representative point does not establish that the full predicted region is vacant or physically valid. The new examples and ablations provide a clearer account of both the model’s gains and its remaining failure modes.

I thank the authors for their response, but I do not think it adequately addresses my concerns. I also noticed an additional issue: the main benchmark appears to contain too few questions. Overall, I do not believe the paper is ready for publication, and I will therefore maintain my original score.

AuthorsOpenReview ↗

Response to reviewer GZg2

Thank you for taking the time to review our response and for answering. We understand your concern about the evaluation scale and agree that 122 questions provide limited evidence for broad generalization. The additional Where2Place results also highlight limitations, which we now discuss explicitly.

Your feedback helped us strengthen the experimental analysis. We added seven direct baseline comparisons under a common evaluation protocol, component ablations, and further qualitative examples. These additions give a clearer picture of where the approach helps and where it falls short, including its gap to stronger spatial models.

The focused finding we hope to convey is that recursive refinement improves accuracy from 25/122 to 36/122 correct answers with the same frozen encoders and 11.2M trainable parameters. We have narrowed the conclusions to reflect this evidence and made the remaining limitations more explicit.

We may still be missing part of your concern. Could you help us understand what you feel remains missing from the new comparisons or ablations? We would be glad to clarify these points or address any specific gap we have overlooked. Thank you again for your time and consideration.

Reviewer_AeScOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Spatial-TRM proposes a small model to predict a free region in an image from a referring instruction. It combines frozen image and text encoders, repeated updates through a shared network (TRM) similar to a loop transformer, with a decoder that produces a convex region. On RoboSpatial-Home, it improves point-based accuracy from 20.49% to 29.51% over a separately trained single-step model with the same parameter count.

Critical strengths
  • Clear improvement from repeated refinements. The single-step and recursive models share the same architecture and parameter count and are trained separately. The 9.02% point improvement shows meaningful support for repeated computation within this framework.

  • A compact model with few trainable parameters. The complete system contains 877.8M parameters, with only 11.2M trained for the task. The CLIP version further reduces the total to 438M while retaining 26.23% accuracy -- a useful tradeoff between model size and performance.

  • Useful analysis of evaluation sensitivity. The paper shows that checking more representative points raises accuracy even when the predicted regions remain unchanged. This points out the limitation of point-based evaluation in this task.

Critical weaknesses
  • Missing comparisons with other small models. The controlled comparisons mainly involve the authors’ own variants, while Table 2 collects external scores obtained under different settings. Relevant candidates include RoboRefer-2B-SFT with RGB-only input, Cosmos-Reason2-2B, and Qwen3-VL-2B or 4B. Although these models are larger than Spatial-TRM, evaluating one or two with the same RGB input and scoring protocol would provide useful baselines.

  • Limited novelty. The recursive mechanism follows TRM, the convex representation builds on CvxNet, and the training data follows the RoboSpatial pipeline. The main contribution is adapting and combining these ideas for spatial region prediction. I consider this an incremental novelty whose value depends largely on the ablations.

  • Limited region quality. Predicted regions sometimes overlap with objects. The convex training masks can include occupied space, and the decoder cannot represent holes or separate free areas.

  • Limited evaluation scale (minor). The main benchmark contains only 122 context questions.

Questions for authors
  • Could you compare against an existing small model, e.g., a 2B VLM with the same settings?

  • What are the inference latency and peak memory use of the single-step and recursive models on the same hardware?

2 comments
AuthorsOpenReview ↗

Response to Reviewer AeSC

Thank you for the constructive feedback. We have added direct baseline comparisons, component ablations and computational measurements, and clarified the contribution and limitations.

W1/Q1: Comparisons with existing small models

We evaluated all four suggested models ourselves using RGB images and text, the same RoboSpatial-Eval script, and a one-point scoring rule. Output coordinates are normalized to [0, 1] before evaluation.

ModelContext accuracy (%)
Qwen3-VL-2B7.38
Cosmos-Reason2-2B15.6
Qwen3-VL-4B22.95
Spatial-TRM29.51
RoboRefer-2B-SFT55.74

Spatial-TRM exceeds the two Qwen variants and Cosmos, while RoboRefer achieves substantially higher accuracy. Spatial-TRM contains 877.8M total parameters, including its frozen encoders, with 11.2M trainable parameters. Table 3 distinguishes these direct evaluations from literature scores. The common input modality and scoring rule improve comparability, although training data and architectures still differ.

W2: Contribution and novelty

We agree that TRM, CvxNet and the RoboSpatial data pipeline are prior components. Our contribution is adapting latent recursive refinement to spatial-context region prediction from frozen visual and text features, and quantifying its benefit at a fixed trainable parameter count.

The recursive model answers 36/122 questions correctly (29.51%), compared with 25/122 (20.49%) for a separately trained single-step model with the same encoders and 11.2M trainable parameters. This is 11 additional correct answers: +9.02 percentage points, or 44.0% relative. The control is trained without recursion; it is not a recursive checkpoint evaluated with fewer steps. Additional computation is required, as quantified below.

Table 1 also reports cross-attention, decoder and relational-contrastive-loss ablations. These strengthen the empirical contribution while preserving appropriate credit to the underlying methods.

W3: Region quality and occupied space

We agree that region quality remains a limitation. The full model achieves IoU 0.11 and Dice 0.16, versus 0.10 and 0.16 for the linear-decoder variant. These low, similar overlap scores do not establish improved region quality despite the gain in point accuracy.

Two limitations are now explicit. First, convex hulls constructed from sparse target points can include occupied space. The mask loss supervises agreement with these targets, but provides no independent occupancy labels for obstacles inside them. Second, a single convex region cannot represent holes or disconnected valid areas. Sections 3, 5 and 7 distinguish agreement with the annotations from the physical validity of the entire predicted region.

W4: Evaluation scale

The main test remains small: each of its 122 questions contributes approximately 0.82 percentage points. Appendix D adds Where2Place evaluation, with 7.7% over all 100 questions and 9.8% over the 55 questions using trained relations. For the 41-question subset excluding positional anchor qualifiers, the result is 12.20% (50/410 points).

We manually extract anchor objects. Each question is scored by the fraction of ten points inside the target mask: one decoder interior point and nine evenly sampled foreground points. Supported and unsupported relations use the same pipeline. These subset comparisons are descriptive and do not isolate causal effects.

The low Where2Place score limits claims of broader generalization. The 999-sample validation set is used for checkpoint selection and is not an independent test benchmark. Appendix A.3 also reports checkpoint sensitivity: the validation-selected checkpoint scores 29.51%, below the 31.97% benchmark peak; the peak was not used for selection.

Q2: Inference latency and peak memory

Table 2 reports measurements on one NVIDIA GH200, with batch size 1, bfloat16 precision, compilation enabled, 10 warm-up iterations and 50 timed iterations. Timing includes the image and text encoders, run once per sample; their embeddings are reused across refinement steps.

ConfigurationMean ± std latency (ms)Peak memory
Single-step21.20 ± 0.771.85 GB
Recursive, one deep-supervision step27.71 ± 0.231.85 GB
Recursive, four deep-supervision steps60.38 ± 0.501.85 GB

The full recursive model therefore takes 2.85× the single-step latency, for a 9.02-percentage-point accuracy gain. Measured peak memory is unchanged at the reported precision.

Thank you for adding the baseline comparisons and latency evals, which address my concerns partially. However, I still find the method novelty limited, and the practical contribution depends on what the task requires: if the goal is to find a point, RoboRefer-2B is more accurate. If the goal is to identify an entire usable region, the annotations are pretty coarse (convex hulls around sparse points). I felt the paper isn't ready to be published, and I would retain my original assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.