acceptodds
Withdrawn from ICLR 2027

Refine Before You Regress: Persistent Belief Refinement for Geometric Reasoning with Frozen VLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Iterative geometric reasoning raises a basic design question: should the numerical answer also serve as the state for subsequent refinement? We study this question through full-scene projective cuboid prediction from grayscale observations of a single RGB image with frozen vision–language models (VLMs). We introduce persistent belief refinement, a training-free inference protocol that maintains persistent object identities and revisable hypotheses about object support, shape, camera and scene relations, and uncertainty, while generating cuboid coordinates only at the final readout. A recurrent-sufficiency analysis motivates this separation by characterizing when a terminal output can replace a richer state without changing subsequent task-level behavior. On 90 held-out SUN RGB-D scenes, the protocol improves coverage-aware Cuboid-F1 AUC by under Raw observations and under coarse-to-fine observations relative to coordinate regeneration, with both protocols using six logical calls. Both configurations also substantially reduce false positives. On a 20-scene subset, adding coordinate feedback while retaining the belief-state schema and persistence rules yields lower F1 at unchanged aggregate recall under approximately matched token usage. A 17-scene cross-backbone comparison finds higher Cuboid-F1 AUC for the complete belief-refinement protocol in all three tested VLM families. These findings support separating recurrent hypothesis refinement from terminal numerical prediction in frozen-VLM cuboid inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, spatial reasoning, iterative inference, structured state representations, persistent belief refinement, training-free inference, projective cuboid prediction

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_AS3gOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a workflow that can enhance the capability of a frozen VLM to predict eight projected image-space corners for a cuboid-like object. The workflow maintains revisable spatial hypotheses across iterations and generates numerical cuboids only at the final readout. The experiments compare the method in this paper with a method that repeatedly regenerates coordinates, examine the effects of memory, numerical feedback, and observation order, and assess the workflow across different frozen VLM backbones.

Critical strengths

S1. The proposed pipeline improves geometric prediction and reduces false positives.

This method continuously maintains and revises structured spatial hypotheses and offers a viable approach to improving geometric prediction with frozen VLMs. Compared with the Raw configuration that repeatedly generates coordinates, the proposed pipeline raises the F1 score from 51.5% to 69.0% and reduces the number of false positives per image from 3.11 to 0.90. These results support the practical value of the pipeline in the present task.

S2. Simple and training-free design.

The method uses a frozen VLM without requiring task-specific fine-tuning or additional geometry-specific networks. Its key idea is to separate spatial hypothesis refinement from numerical coordinate prediction: instead of repeatedly regenerating cuboid coordinates, the model maintains and revises structured spatial hypotheses, including object shape, camera viewpoint, and uncertainty, across iterations, and produces the final coordinates only at the last step. This design is intuitive and easy to implement, allowing existing VLMs to perform more structured geometric reasoning without modifying their architectures or training additional models.

Critical weaknesses

W1. The main comparisons do not match token budgets.

The main experiments match the number of model calls, but the proposed pipeline uses substantially more tokens than the compared baselines. Although the authors acknowledge this limitation, a comparison under comparable token budgets would help clarify the trade-off between performance and token cost. This makes it difficult to separate the benefits of the workflow design from those of additional inference resources.

W2. The methodological novelty over existing spatial-representation and iterative-refinement approaches remains unclear.

Although the paper discusses related work, including Thinking in Space, Theory of Space, and Self-Refine, the distinction between the proposed persistent belief refinement and existing approaches to maintaining explicit spatial representations or iteratively revising model outputs remains insufficiently validated. In particular, it is unclear whether the performance gains arise specifically from separating the recurrent state from the final numerical output, or primarily from structured memory and object-persistence rules. Stronger controlled comparisons would help establish the method's distinct contribution.

W3. Limited scope of geometric reasoning.

Despite being framed as geometric reasoning, the task is restricted to predicting projected cuboid corners for objects in a single image. This is closer to image-space geometric localization than general 3D reasoning, as the predicted 2D corners do not uniquely determine metric 3D positions, dimensions, or orientations. Consequently, the experiments provide limited evidence that the proposed persistent belief refinement generalizes beyond this specific prediction task. It remains unclear whether the proposed approach offers similar benefits for standard 3D object detection or more general geometric reasoning problems.

Questions for authors

1)Are there any additional results comparing the proposed pipeline with the coordinate-regeneration baselines under comparable token budgets? The proposed method uses substantially more tokens than the coordinate-regeneration baselines despite using the same number of logical model calls. This difference in inference budget may contribute to the observed performance gains, making it difficult to isolate the benefits of the proposed workflow itself. A comparison under comparable token budgets would help clarify whether persistent belief refinement remains advantageous when inference resources are controlled.

2)How should the scope of the geometric reasoning claim be interpreted? The evaluation focuses on predicting projected cuboid corners from a single image, which does not uniquely determine metric 3D geometry. Do the authors intend the proposed method as a task-specific protocol for projective cuboid prediction or as a more general geometric reasoning framework? What evidence in the current results supports the latter interpretation?

3)The workflow uses one state initialization, four state updates, and a terminal numerical readout. Have fewer or more state updates been evaluated while keeping the initialization and readout procedures unchanged? Why were four updates chosen, and do they offer the best trade-off between prediction performance and token cost?

Reviewer_p5xhOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes “persistent belief refinement”, a training-free object grounding method that allows multiple passes of a model to give better predictions. The paper mainly proposes in theory that: If two internal states produce the same current answer, but would produce different answers after the same subsequent observation, then the current answer alone cannot preserve both states’ future behavior. So keeping the fields describing the object’s image region, shape, viewpoint, relationships and uncertainties while doing the next pass is useful. Experimental results shows much higher precision but much lower recall.

Critical strengths
  1. The method is indeed a technical novelty. Bringing more information to the next pass, especially the relationships and uncertainty, is a sound contribution that may be helpful to academia.
  2. There are improvements on part of the reported metrics.
Critical weaknesses
  1. Overcomplexity of the writing. The contribution should be framed as a technical one supported by the implementation of multiple passes in the task of object grounding, instead of the flashy and incomprehensible “recurrent sufficiency” and “output-state aliasing”. As a reviewer, honestly I am very disappointed when digging into your theory and finding that it is actually straightforward, only wrapped up by intentionally complicated language.
  2. Coarseness of the whole paper writing. Take Figure 1 as an example: would any reviewer or AC or reader expect to see such an AI-drawn, low-dpi, copy-pasted, and uninformative figure, which is unfortunately ineligible for a slide for the weekly report to a PI, as the main pipeline illustration?
  3. Experimental design is coarse. Across 3 tables, authors only do experiments on 1 dataset, and what’s more, it’s a subset of 90. Ablation studies are done on 17 scenes. Therefore, a reader should not view their results as solid or generalizable.
  4. Even on this small set, the result is controversial. Precision grows at the expense of Recall, which makes the true improvement from the pipeline doubtful.
Questions for authors
  1. Seriously, what is the percentage contribution of all human authors in this research?
  2. Please state clearly, in one paragraph, what is the main theoretical contribution of the paper is, for other reviewers’ information.
  3. Please explain why experiments are conducted on such a narrow choice of datasets and scenes.
Reviewer_J4DbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free belief refinement method for geometric reasoning with frozen VLMs. Instead of repeatedly predicting cuboid coordinates, the method maintains structured spatial hypotheses across iterations and generates coordinates only at the final step. Experiments on SUN RGB-D show improved Cuboid-F1 AUC and fewer false positives compared with coordinate-regeneration baselines. Additional experiments examine different inference strategies and VLM backbones.

Critical strengths
  1. The paper presents a clear and well-motivated approach to iterative geometric reasoning. Maintaining structured spatial hypotheses instead of repeatedly generating coordinates is an interesting training-free strategy.
  2. The method achieves notable improvements in Cuboid-F1 AUC and substantially reduces false positives on SUN RGB-D. The reported confidence intervals further support the experimental results.
  3. The paper includes useful experiments on memory design, coordinate feedback, and observation order. The cross-backbone evaluation also provides evidence that the method can benefit different VLMs.
Critical weaknesses
  1. The main comparisons use the same number of inference calls but different token budgets. The Belief methods consume substantially more tokens than the coordinate-based baselines, making it difficult to rule out the effect of additional computation.
  2. The proposed framework combines structured belief states, object persistence, state normalization, and deferred readout. The current ablation studies do not sufficiently isolate the contribution of each component.
  3. The novelty of the proposed framework is not sufficiently established. While the paper compares against free-text memory, it remains unclear how structured belief refinement differs from existing state-based iterative reasoning methods, or whether the improvements depend on the specific belief representation.
  4. The recurrent-sufficiency analysis provides a theoretical motivation for separating reasoning states from numerical outputs, but its connection to the empirical improvements is not directly established. In particular, the paper does not demonstrate output-state aliasing in the evaluated VLMs.
Questions for authors
  1. Can the authors provide comparisons under matched token budgets to better assess whether the improvements come from the proposed method rather than additional computation?
  2. Can the authors further isolate the contributions of structured belief states, object persistence, state normalization, and deferred readout through additional ablation studies?
  3. What distinguishes the proposed belief refinement framework from existing state-based iterative reasoning methods? Would a simpler structured-memory approach achieve similar results?
  4. Can the authors provide empirical evidence of output-state aliasing in the evaluated VLMs and clarify how it relates to the observed performance improvements?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.