acceptodds
Withdrawn from ICLR 2027

SpatialMind: Test-time spatial pointing via observation dependency graph

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

This paper presents a test-time framework for spatial pointing with frozen vision-language models. The presented method features a set of zoom procedures, which we call the registry, and treats their outputs as dependent observations. This casts test-time pointing as selection and acquisition. For selection, we propose SpatialMind (SM), a relational answer model that estimates the posterior of correctness of each observation. SM draws on the frozen model’s prefill states and messages over dependency relations, starting from a boosted-tree prior. For acquisition, we propose Adaptive SpatialMind (ASM), a practical extension of SM for reduced call budgets. ASM keeps SM unchanged and adds a cost-conditioned policy that decides what to acquire next and when to stop. Our analysis shows that SM learns to update its prior by Bayes’ rule in log-odds over dependent observations, and that ASM includes full acquisition and every fixed schedule as special cases. The proposed framework is applied to three frozen backbones on five public pointing benchmarks. The experiments demonstrate the accuracy of SM and the cost efficiency of ASM against both training-free and learned test-time methods.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
spatial pointing, vision-language models, test-time scaling, visual grounding, relational message passing, cost-sensitive acquisition, robotics perception

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_C2jBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SpatialMind treats pointing with a frozen VLM as candidate selection and observation acquisition. A registry of dependent zoom procedures produces candidate points; a learned graph model combines a boosted-tree prior, prefill features, and typed messages to score their correctness. Adaptive SpatialMind adds a cost-conditioned acquisition and stopping policy. Experiments span three backbones and five benchmarks, with image-grouped cross-fitting and held-out-benchmark evaluation. The paper reports accuracy and call-efficiency gains and motivates the method through Bayesian scoring and policy-class arguments. The main unresolved issue is whether graph correction itself explains the improvement over a matched selector.

Critical strengths
  1. A useful formulation of dependent visual observations. The registry deliberately generates different evidence through self-centered and anchor-centered zoom chains. Separating candidate coverage from selection clarifies why obtaining more correct candidates need not improve the returned answer. Reusing the answer model over partial histories gives a practical connection between selection and adaptive acquisition, rather than treating the two as unrelated procedures.

  2. Credible system-level gains under a careful evaluation protocol. Image-grouped cross-fitting, nested training of learned components, held-out-benchmark evaluation with overlap removal, clustered resampling, and multiplicity adjustment are substantial strengths. Table 3 reports 70.71% accuracy for SpatialMind versus 65.74% for the plain call and 69.23% for registry voting; the 1.48-point margin over voting has a 95% interval of [0.88, 2.08]. Adaptive SpatialMind reaches 69.05% with 2.93 backbone calls versus 7.91 for full acquisition. These results support the utility of the overall system under the stated call-count measure.

Critical weaknesses
  1. The learned baseline does not isolate the observation-graph contribution. The internal prior in Appendix D.1 uses 36 features, including procedure success rates and retrieval-conditioned rates, and is trained over partial histories with its own fitting choices. The boosted-tree baseline in Appendix E instead uses 22 features, default classifier settings, and full histories. SpatialMind additionally receives prefill states. Therefore, the 70.71% versus 69.08% comparison changes the prior, information available, and relational model together. The zero-correction internal prior is an especially direct missing control, since Proposition 1 explicitly states that it is contained in the model class. A same-feature scorer without dependency relations would then distinguish the benefit of those relations from the richer inputs. This attribution matters because the observation dependency graph is the central methodological claim.

  2. The Bayesian interpretation exceeds what is established for the actual objective. Proposition 1 characterizes the population optimum of binary cross-entropy, while Equation 12 combines BCE with a normalized pairwise ranking loss. Appendix B proves a useful two-node ordering property, but ranking consistency is not calibrated posterior estimation: the pairwise optimum depends on joint label probabilities. The combined objective therefore does not inherit the stated posterior interpretation without further conditions. Likewise, exact messages in an idealized Markov chain and optimality over all history-dependent policies do not establish exact inference or optimality for the implemented network and approximate policy iteration. These results remain useful motivation, but the abstract's claim that the method learns Bayes' rule should be qualified accordingly.

Questions for authors
  1. What accuracy does the exact internal boosted prior achieve with the graph correction disabled, under identical folds, retrieval features, partial-history training, and fitting choices? With prefill inputs also held fixed, how does a scorer without parent/child relations perform? These controls would most directly affect my recommendation.

  2. Which result applies to posterior calibration under the complete Equation 12 objective, rather than to its BCE component or two-node ordering alone? Please either establish the needed conditions or narrow the Bayesian claim. A BCE-only comparison and held-out calibration measurements would be informative if posterior estimation remains a central claim.

  3. At comparable mean call cost, how does the acquisition policy compare with a training-selected fixed schedule using the same answer model? The paper already computes fixed-schedule values during policy training; reporting this control would clarify how much benefit comes from adapting acquisition to each query.

Reviewer_FC81OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies test-time spatial pointing with frozen vision-language models. A seven-procedure registry generates dependent predictions through self-zoom and landmark-based zoom chains. SpatialMind (SM) ranks their points using a boosted-tree prior corrected by relation-aware message passing. Adaptive SpatialMind (ASM) learns which procedure to call next and when to stop. Experiments with three backbones on five pointing benchmarks report improved accuracy and cost-adjusted performance.

Critical strengths
  1. The registry increases candidate coverage over repeated sampling at comparable call counts (Table 2). Treating the resulting predictions as dependent observations is a useful way to formulate the selection problem.

  2. The evaluation includes three backbones, five benchmarks, image-grouped splits and held-out-benchmark tests. SM improves over the registry vote by 1.48 points at identical call counts, and ASM offers a promising accuracy–cost trade-off (Table 3).

Critical weaknesses
  1. The experiments do not isolate the relational correction. The boosted-tree baseline uses 22 features and is trained on full histories (Appendix E), while SM's own tree prior uses 36 features, including retrieval-based success priors, and is trained on partial histories (Appendix D.1). The 1.63-point difference in Table 3 therefore cannot be attributed to typed message passing. Results for the exact tree prior and ablations of the relation types and prefill states are needed. The Bayesian interpretation is also stronger than what Proposition 1 establishes: it characterizes cross-entropy minimization, whereas the actual loss in Eq. (12) also includes pairwise ranking.

  2. ASM is not compared with fixed-depth zoom schedules or simple stopping rules using the same registry and answer model. This matters because Table 5 shows that early zoom steps are often more useful than deeper ones. A shallow fixed schedule might explain much of the reported call reduction. Proposition 3 concerns an optimal policy over the general policy class, not the fitted ASM policy, so it does not resolve this empirical question.

Questions for authors
  1. How does SM compare with its exact boosted-tree prior, without the relational correction? What is the effect of removing typed edges or prefill states?

  2. Can the authors compare ASM against the best validation-selected fixed schedule and a simple confidence-based stopping rule, using the same registry and answer model?

  3. Are the learned probabilities σ(si)\sigma(s_i) calibrated on held-out queries, given the pairwise term in Eq. (12)?

Reviewer_LsMpOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a test-time framework for spatial pointing. The general idea is to first get a set of zoom with a VLM and estimates the posterior of correctness of each observation. The method reaches 70.71 accuracy with 7.91 mean calls.

Critical strengths
  1. The experiments are comprehensive and the choice of benchmarks is reasonable.
Critical weaknesses
  1. The contribution of this paper is unfortunately a bit limited. As honestly reported in the related works, i.e., App. A, there are too many training-free VLM papers that explore visual search, zoom and visual prompting. And a "learned answer model that also uses the prefill states already produced by these calls" does not introduce enough novelty.

  2. The uniqueness of the proposed SM method makes it hard to deploy into real-world products, since it requires a separate small model that only targets the pointing task.

Questions for authors

Have the authors made their code available?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.