acceptodds
Withdrawn from ICLR 2027

FrameBind: Directional Grounding for Allocentric Spatial Reasoning with VLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-Language Models (VLMs) have made rapid progress on spatial reasoning benchmarks, but it remains unclear whether these gains reflect grounded geometric understanding or semantic shortcuts. We study allocentric relative-direction tasks, where the models are prompted to determine the direction a target object is with respect to reference object or to determine the object in a given direction. On controlled scenes we find a pronounced answer-format asymmetry: models appear to be more accurate at selecting an object than at naming the direction, even when the image and underlying relation are held fixed. We trace this gap to a strong camera-centric bias; in our experiments, only 3.6% of a representative model's direction answers match the requested reference frame, while 95% match the camera frame. Component diagnostics show that VLMs localize objects reliably but cannot recover a reference object's orientation or extend partial orientation cues into a usable frame. Only a complete labeled coordinate display can restore performance, likely because it reduces the task to image-plane localization. Motivated by this diagnosis, we introduce FrameBind, a training-free, tool-assisted pipeline that overlays a reference-centered coordinate frame on the original RGB image rather than replacing it with a symbolic abstraction. Additionally, we introduce an Object-Map Inversion that converts direction queries into sparse, direction-to-object maps to exploit the object-answer preference. Across four benchmarks, FrameBind improves both answer formats over its Qwen3-VL-8B backbone, while object-map inversion adds further gains on controlled scenes.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spatial Reasoning, Allocentric Analysis, Egocentric Bias Mitigation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_iiihOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks why VLMs fail at "from object X's view, where is Y?" questions. On 4COMFORT, Qwen3-VL-8B gets 3.6% of direction questions right and mostly answers by image left/right. Probes with ground-truth cues suggest the models localize objects fine but can't use orientation unless all four axes are drawn for them. FrameBind draws those axes on the image using SAM3, Depth Anything 3 and OrientAnything-v2. It can also ask the VLM for a direction-to-object map and read the answer off it. It beats its Qwen3-VL backbone on all four test sets.

Critical strengths
  1. The ground-truth cue experiments are the most useful part of the paper. Giving the model only a front arrow, or front and left arrows, barely changes object accuracy (35.7% and 26.3%, compared with 26.9% with no cue), while four labeled axes raise it to 80.0%. The arrow-length and label-position sweeps in Fig. 4 then show that accuracy goes up as the arrow tips get closer to the correct objects. Taken together, this is a convincing case that the model uses the overlay as a pointer in the image and does not actually reason in the reference object's frame, which is worth knowing for anyone who designs visual prompts for spatial questions.
  2. The paired object-answer and direction-answer questions are a simple and effective diagnostic. Because both questions describe the same relation in the same image, the confusion matrices for Qwen3-VL (Fig. 2) and the other eight models (App. D) show directly how often a model falls back on image left/right. Appendix A also describes how each dataset was built in enough detail that the evaluation sets could be rebuilt.
Critical weaknesses
  1. FrameBind already computes the reference frame and object centres (Eq. 13), allowing direct answers from the dominant signed coordinate component, as in Kubric labeling (Eq. 6). However, the paper lacks this geometry-only baseline, ablations with ground-truth orientation or boxes, OrientAnything-v2 heading errors, and the map acceptance/fallback rates mentioned in Appendix B.4. It remains unclear whether the improvements come from VLM reasoning or the geometric tool outputs.
  2. SymPL, identified as the closest approach, is excluded with only a brief explanation (Sec. 5.2), while the authors' symbolic layout (Sec. 6) is not an equivalent substitute. The only evaluated tool-assisted baseline, APC-Vis, performs near chance, with unspecified backbone and settings. Comparisons are therefore mainly against unassisted VLMs, making it difficult to establish FrameBind's advantage over existing tool-assisted methods.
  3. In 4COMFORT, the camera rotates with the reference, consistently placing it down-left and making image-based left/right answers incorrect by construction. The reported 95.4% matches the image-left/right responses in Fig. 2a (6,032/6,320), while the 3.6% accuracy reflects this fixed setup rather than a general failure. On COMFORTer, Qwen3-VL's direction accuracy is near chance (24.2%), with only a 3.4-point gap to object accuracy. Object accuracy on 4COMFORT is also near chance (26.9%), weakening the claimed object-answer advantage motivating map inversion. The abstract and conclusion should limit these claims to the evaluated settings.

Minor: The paper reports no confidence intervals, even though the Kubric and 3DSR sets are small (218 and 119 items) and the +0.8 inversion gain on 3DSR amounts to a single question. Runtime is not reported, even though each query runs several foundation models and at least two VLM calls.

Questions for authors

Q1. What accuracy does the rule-based readout from the same tool outputs (Eq. 13 with the dominant-component rule) reach on each dataset and answer format?

Q2. What are the map acceptance rate, the accuracy of accepted maps, and the number of fallbacks per dataset?

Q3. How does FrameBind perform with ground-truth orientation or boxes, and with the camera-up control instead of the fitted ground plane? Why does direct direction answering reach 59.90 on COMFORTer but only 30.38 on 4COMFORT, which has the simpler layout?

Reviewer_RyuMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper aims to address the problem of allocentric spatial reasoning in VLMs/MLLMs, in particular in a form where the model has to answer questions of spatial relations between objects from perspective of another object (rather than the camera viewpoint). This is a well known issue, especially in smaller models, that the understanding of spatial directions in an allocentric setting is significantly lower than in egocentric situation (i.e. if the coordinates are as seen from the camera viewpoint). The authors propose a benchmark set dedicated to VLM evaluation, as well as a method based on overlaying input images with directional marks to improve the VLM performance without additional training. The paper shows results on basis of several popular smaller models, evaluated on the proposed benchmark as well as other existing evaluation sets.

Critical strengths
  • The studied problem is an important issue, relevant to VLM development, with strong applications in embodied AI and related settings.
  • Proposed 4COMFORT benchmark is technically valid, and can be useful to diagnose allocentric reasoning in VLMs
  • The FrameBind method improves the performance of evaluated models as a zero-shot adaptation, without requiring any fine-tuning
  • The paper includes an evaluation of common VLMs on the task, as well showcases the proposed method performance on both the proposed task and other existing benchmarks, with favorable results.
  • The paper is overall easy to read and well organized.
Critical weaknesses
  • The method idea is not novel - the proposed FrameBind method is mostly a task-specific extension of standard visual prompting (e.g. Yang, Jianwei, et al. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. 2023). This is a well-known approach to improve perception performance of vision-language models.
  • At the same time one could argue, that providing visual guidelines for the VLM as an overlay on the image only simplifies the problem instead of actually addressing the reasoning limitations of the model itself.
  • Object-answer and direction-answer questions change the supplied information and answer choices, so their performance gap does not isolate an output-format effect. Likewise, map inversion changes the inference procedure compared to baseline, so the comparison isn't completely fair either.
  • In the 4COMFORT benchmark the reference object always faces the same image plane direction, while targets occupy four orthogonal positions. This is a weaker evidence of general allocentric reasoning than stated objective (but it remains a useful controlled diagnostic overall).
  • The authors do not provide data or code at the time of evaluation, therefore I cannot ascertain the quality and soundness of the evaluation. This is problematic for a paper that provides a benchmark as one of the main contributions.
  • The paper doesn't mention any human validation of the benchmark data, nor provides any human baseline numbers.
  • FrameBind is evaluated on a single backbone, generalization of this technique beyond this setting is unknown (although following the above mentioned visual prompting paper it's likely it will work with other models as well).
Questions for authors
  • Did you validate the final benchmark questions (or a radom subset) with human annotators? This includes checking if the tests have only one clear answer.
  • How well does the FrameBind generalize beyond the controlled scene layouts? Can you evaluate on a set with balanced reference headings, varied camera viewpoints, and targets outside the four principal directions?
  • Can you provide the dataset and code for review?
  • Does the FrameBind method generalizes to other models? Please provide evaluation on other common VLM models, including at least one state-of-the-art medium or large model.
Reviewer_3VzTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies object-relative spatial reasoning through paired questions that either name a target and ask for its direction, or name a direction and ask for an object. Controlled diagnostics reveal strong camera/image-plane-aligned responses and substantial sensitivity to the question formulation. The proposed training-free FrameBind pipeline uses segmentation, depth, and orientation estimators to overlay a reference-centered coordinate frame on the original RGB image. A target-conditioned sparse direction-to-object map can then be inverted to answer direction questions. Experiments on two COMFORT-derived collections, a selected Kubric subset, and a small 3DSR subset show improvements over the Qwen3-VL-8B backbone, with particularly large gains on the COMFORT-derived datasets.

Critical strengths
  1. The paper offers a useful diagnostic decomposition rather than relying only on aggregate task accuracy. It distinguishes object localization, orientation recovery, and use of supplied coordinate cues, and the full-versus-partial overlay interventions expose substantial differences in performance. The paired-query setup also makes a practically important point: success at selecting an object does not automatically demonstrate reliable reference-relative direction prediction. The manuscript appropriately acknowledges in Section 5.3 that these pairs change the supplied information and candidate domain, so this is question-formulation sensitivity rather than a pure output-format intervention.

  2. FrameBind is a concrete, training-free intervention with large measured gains on the COMFORT-derived evaluations: on COMFORTer, object accuracy rises from 27.55% to 67.75%, and direct direction accuracy from 24.15% to 59.90%, reaching 67.75% with map inversion. Preserving the original image is a sensible way to retain candidate appearance and location when extracted scene representations are incomplete. The authors also candidly document the limitations of the small natural-image evaluation, candidate generation, estimated geometry, and the representation comparison, which makes the evidence easier to interpret.

Critical weaknesses
  1. The strongest diagnostic interpretation is not adequately separated from the restricted data and prompting choices. In Table 1 and Appendix A.4, all 441 orientation-probe images have the same projected ground-truth heading, down-left; 4COMFORT also fixes the camera-reference relationship. These results demonstrate a striking failure on that configuration, but do not by themselves establish a general inability to recover orientation or establish the mechanism behind the object-direction gap. COMFORTer varies viewpoints, but the corresponding balanced orientation and cue-utilization diagnostics are not reported there. Moreover, the inverse questions change both conditioning information and answer candidates, and a target-conditioned map can legally contain only the target and three empty entries, so its gain does not isolate an object-selection mechanism. The abstract/conclusion should be restricted to the observed behavior unless balanced-heading, matched-prompt, and matched-cue controls substantiate the broader explanation.

  2. The evidence does not yet isolate the value of the proposed inference design from the extra geometric information and alternative reformulations. Table 2 omits a direct geometric classifier using the same estimated reference frame and object centers, and no numerical camera-up versus fitted-plane ablation is provided despite the control being defined in Appendix B.2. Figure 6 is explicitly only a representation study: its heading-up symbol layout with the original question is not the full SymPL method, which also uses spatial partitions and localization reformulation. A stronger matched-information comparison is therefore needed to establish the incremental contribution. The map evaluation is also incomplete: Section 4 promises map-only accuracy, valid-map coverage, and fallback counts in Appendix B.4, but B.4 contains no such numbers. This matters particularly outside the controlled scenes: inversion adds only five correct answers on 218 Kubric relations and one on 119 3DSR relations, with no paired, image/video-clustered uncertainty estimates.

Questions for authors
  1. Does the orientation/cue-utilization diagnosis persist when reference headings and camera viewpoints vary independently, with balanced projected headings and matched wording/label placement? Appendix C.3 defines a mirror control; please report its results or clarify whether it was evaluated.

  2. Using the same estimated masks, depth, orientation, and candidate centers, how does FrameBind compare with direct geometric relation classification and a symbolic alternative that also reformulates the query as localization? Please state whether the optional geometric prior in Appendix B.3 was enabled for Table 2, and report the camera-up versus estimated-plane comparison.

  3. Please provide the promised accepted-map coverage, conditional map accuracy, repair/fallback counts, and paired uncertainty estimates clustered by source image or video. Does the inversion improvement survive a prompt-matched direction baseline or an object-mapping control that tests whether the benefit actually requires predicting other objects?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.