acceptodds
Withdrawn from ICLR 2027

ReasonFit: Coupling VLM Structural Reasoning with Differentiable Multi-view Fitting for Primitive-Based Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recovering a part-structured 3D model from multi-view images requires deciding which parts are present and estimating their geometry. These decisions depend on each other: an omitted part can distort its neighbours, while a restrictive primitive can fragment a single component. We present ReasonFit, a closed-loop framework that couples vision-language structural reasoning with differentiable geometric fitting. Within this loop, a vision-language model (VLM) proposes part structure and qualitative shape constraints, while differentiable fitting estimates geometry from multi-view appearance and silhouettes. We use deformable SuperFrustum primitives to capture curved profiles and varying cross-sections within a single part, with VLM shape cues guiding the allowed deformations. In turn, the fitted assembly and unexplained image regions guide the VLM in distinguishing missing parts from fitting errors and revising the structure. The updated assembly is jointly refined while preserving the fitted state of retained parts. Experiments across diverse objects show that ReasonFit produces more compact reconstructions than scaffold-based fitting. Evaluations of geometric fidelity, coverage, and dedicated part recovery further reveal the trade-off between compactness and preserving distinct components.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Structured 3D Reconstruction, Primitive-based Modeling, Vision-Language Models, Analysis-by-Synthesis, Multi-view Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Ciq4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper reconstructs part-structured 3D models made of a few parametric primitives from calibrated multi-view images and masks, without 3D supervision. A VLM proposes the part inventory, relations and qualitative shape cues. A compiler grounds these in a visual hull to initialize deformable SuperFrustum primitives and to decide which deformations each part may use. Differentiable fitting with surface-bound 2D Gaussians and mask-ray constraints then estimates the geometry, and the VLM reviews the fitted assembly to add missing parts or reinitialize collapsed ones before joint refinement. On a hand-built Synthetic set, ShapeNet and GSO, it reports lower CD with fewer primitives than PartGS and DBW.

Critical strengths
  1. The closed loop is a sensible way to use a VLM in reconstruction. The VLM only makes discrete structural decisions and never has to predict metric coordinates, and the fitted geometry is fed back to it as evidence. Turning qualitative cues into deformation permissions is a simple and useful way to limit fitting ambiguity.
  2. The qualitative results are convincing for curved and slender parts. A mug handle or a set of table feet are fitted as single primitives where PartGS and DBW fragment or merge them, and ReasonFit uses fewer primitives than both baselines on all three datasets.
Critical weaknesses
  1. The evaluation is small and favors the proposed primitive. The three datasets contain 92 objects in total, and part-level metrics are only reported on the authors' Synthetic set, which is built from the kinds of shapes SuperFrustum represents well. On ShapeNet and GSO, the CD gains over the better baseline are small.
  2. The ablations are too small to support the central claim. The VLM-revision ablation uses only 3 objects and the component ablation 5, so the large gain attributed to revision isn't established.
  3. There is no variant that runs the same fitting without VLM initialization, and only one VLM is tested, so it's unclear how much of the gain comes from the VLM rather than the primitive and fitting design.
  4. SuperFrustum is borrowed from ResFit, while the appearance model follows PartGS. Comparisons with related image-based primitive methods, such as ISCO and GaussianBlock, are also missing.
Questions for authors
  1. How does the pipeline perform without the VLM, i.e., SuperFrustum fitting with residual birth and pruning from a generic initialization?
  2. Could you run the revision ablation on the full Synthetic set, and report how often the VLM's edits were correct?
  3. ShapeNet has part annotations (e.g., PartNet). Could you report dedicated recovery there, on objects not built from the same primitive types?
  4. How does the method behave with a different VLM?
Reviewer_3ryMOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper describes a procedure that combines VLM and classical optimization to decompose objects from multiple posed images. They use a VLM to propose primitives and counts, then run an appearance-based optimization using multiple posed views to obtain the parameters of each primitive and location and pose. They evaluate mainly on their synthetic dataset and show qualitative examples on simple synthetic shapes. They also show results on ShapeNet, a dataset of synthetic, simple shapes, and GSO, which contains slightly harder shapes but with significant performance drop.

Critical strengths

The paper proposes a complete procedure and designs shape primitives for decomposition of synthetic objects. The paper combines VLMs with classical optimization.

Critical weaknesses

The paper omit important baselines, such as recent VLMs: astra, fable - even older generations like opus, which should be able to solve this task reasonably without classical optimization in the loop. The paper focuses evaluation on simple, synthetic shapes and do not show any generalization on real, complex objects. The evaluation on GSO shows quite significant chamfer distance drop. It would be helpful to show non-cherry-picked qualitative examples on real objects. It is unclear what the novelty or technical contribution of this paper is. VLMs can decompose shapes, but that is not a new finding. Classical optimization's effectiveness depends greatly on how good the initialization is, which is done by VLM off-the-shelf.

Questions for authors

Could you provide a few (on the order of 10-20) non-cherry-picked qualitative examples on real objects? Could you provide some metrics on more complex datasets like Objaverse? A uniform random subset is ok. Could you provide comparison with GPT astra or Claude fable without optimization, and show how your method provides better results?

Reviewer_UjjAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ReasonFit reconstructs a labelled assembly of deformable SuperFrustum primitives from calibrated multi-view RGB images and foreground masks. A VLM proposes part identities, counts, relations, and shape cues. A structural compiler grounds these descriptions in a silhouette-derived visual hull to initialize geometry and determine allowed deformations. Image-based fitting jointly optimizes the primitives and their surface-bound Gaussians using appearance and implicit mask-ray constraints. The fitted assembly is then inspected by the VLM to propose missing parts or reinitialize collapsed parts, followed by joint refinement that preserves the fitted state of retained primitives. The evaluated revision policy does not delete parts. Across 41 synthetic, 31 ShapeNet, and 20 GSO objects, the full method improves Chamfer distance and uses fewer primitives than PartGS and DBW. A separate paired continuation experiment examines the contribution of structural revision.

Critical strengths

The method produces more compact reconstructions while improving geometric accuracy across all three evaluated datasets (Table 1). For example, on Synthetic, Chamfer distance decreases from 0.94 for PartGS to 0.83, while the primitive-to-reference-part ratio decreases from 1.10 to 0.92. The curved-handle and slender-support examples in Figure 4 help explain why a more expressive, structurally guided representation can improve both quantities.

The structural-revision experiment includes a useful control for additional optimization: both continuations share the fitted checkpoint, optimizer-reset procedure, and 5K-iteration budget. Revision improves dedicated recovery from 53.8% to 86.5% relative to no-edit continuation (Table 2). This is concrete evidence that changing the part inventory can help beyond simply fitting longer, although the scope of this experiment is limited.

Critical weaknesses

My main concern is that the demonstrated operating conditions are too restrictive to establish sufficient practical value. Every evaluation uses 100 rendered, calibrated views and foreground masks per object, including the evaluation on GSO (Section 4.1 and Appendix B.1). There is no evaluation on captured photographs, no view-count study, and no evidence showing how performance changes with imperfect masks or camera estimates. I am not assuming that the algorithm inherently requires exactly 100 views; the problem is that the paper does not establish a less demanding usable setting. The reported improvements are meaningful within this controlled protocol, but they leave the usefulness of the complete reconstruction workflow unresolved. A larger feedback ablation under the same input conditions would not by itself address this concern.

A secondary concern is the limited evidence for the general benefit of the reasoning-fitting feedback. Its direct comparison with no-edit continuation covers only ball_table, crab, and mug, with two fitting seeds and shared initial VLM descriptions (Appendices B.4–B.5). The larger main comparison changes several components together and therefore does not isolate feedback. The paired experiment is encouraging, but it does not establish how consistently revision helps across different structures or fresh VLM calls. This is a limitation of the evidence, rather than a demonstrated error in the mechanism.

Questions for authors
  1. Validation request: Can you demonstrate a practically usable input regime? A compact view-count comparison on a representative subset and a small evaluation using captured images with estimated cameras and masks would directly address my central concern. These results could be placed in the supplement. What reconstruction quality and dedicated part recovery remain under those conditions?

  2. Clarification: What is the end-to-end wall-clock cost per object, including visual-hull construction, initial VLM reasoning and localization, revision calls and retries, and fitting? The 30K-iteration budget and two valid correspondence responses do not specify the total workflow cost. Please also clarify the initial-fit versus post-revision iteration split used in the main comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.