- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks why VLMs fail at "from object X's view, where is Y?" questions. On 4COMFORT, Qwen3-VL-8B gets 3.6% of direction questions right and mostly answers by image left/right. Probes with ground-truth cues suggest the models localize objects fine but can't use orientation unless all four axes are drawn for them. FrameBind draws those axes on the image using SAM3, Depth Anything 3 and OrientAnything-v2. It can also ask the VLM for a direction-to-object map and read the answer off it. It beats its Qwen3-VL backbone on all four test sets.
Critical strengths
- The ground-truth cue experiments are the most useful part of the paper. Giving the model only a front arrow, or front and left arrows, barely changes object accuracy (35.7% and 26.3%, compared with 26.9% with no cue), while four labeled axes raise it to 80.0%. The arrow-length and label-position sweeps in Fig. 4 then show that accuracy goes up as the arrow tips get closer to the correct objects. Taken together, this is a convincing case that the model uses the overlay as a pointer in the image and does not actually reason in the reference object's frame, which is worth knowing for anyone who designs visual prompts for spatial questions.
- The paired object-answer and direction-answer questions are a simple and effective diagnostic. Because both questions describe the same relation in the same image, the confusion matrices for Qwen3-VL (Fig. 2) and the other eight models (App. D) show directly how often a model falls back on image left/right. Appendix A also describes how each dataset was built in enough detail that the evaluation sets could be rebuilt.
Critical weaknesses
- FrameBind already computes the reference frame and object centres (Eq. 13), allowing direct answers from the dominant signed coordinate component, as in Kubric labeling (Eq. 6). However, the paper lacks this geometry-only baseline, ablations with ground-truth orientation or boxes, OrientAnything-v2 heading errors, and the map acceptance/fallback rates mentioned in Appendix B.4. It remains unclear whether the improvements come from VLM reasoning or the geometric tool outputs.
- SymPL, identified as the closest approach, is excluded with only a brief explanation (Sec. 5.2), while the authors' symbolic layout (Sec. 6) is not an equivalent substitute. The only evaluated tool-assisted baseline, APC-Vis, performs near chance, with unspecified backbone and settings. Comparisons are therefore mainly against unassisted VLMs, making it difficult to establish FrameBind's advantage over existing tool-assisted methods.
- In 4COMFORT, the camera rotates with the reference, consistently placing it down-left and making image-based left/right answers incorrect by construction. The reported 95.4% matches the image-left/right responses in Fig. 2a (6,032/6,320), while the 3.6% accuracy reflects this fixed setup rather than a general failure. On COMFORTer, Qwen3-VL's direction accuracy is near chance (24.2%), with only a 3.4-point gap to object accuracy. Object accuracy on 4COMFORT is also near chance (26.9%), weakening the claimed object-answer advantage motivating map inversion. The abstract and conclusion should limit these claims to the evaluated settings.
Minor: The paper reports no confidence intervals, even though the Kubric and 3DSR sets are small (218 and 119 items) and the +0.8 inversion gain on 3DSR amounts to a single question. Runtime is not reported, even though each query runs several foundation models and at least two VLM calls.
Questions for authors
Q1. What accuracy does the rule-based readout from the same tool outputs (Eq. 13 with the dominant-component rule) reach on each dataset and answer format?
Q2. What are the map acceptance rate, the accuracy of accepted maps, and the number of fallbacks per dataset?
Q3. How does FrameBind perform with ground-truth orientation or boxes, and with the camera-up control instead of the fitted ground plane? Why does direct direction answering reach 59.90 on COMFORTer but only 30.38 on 4COMFORT, which has the simpler layout?