acceptodds
Under review as a conference paper at ICLR 2027

GATOR: Generative and Agentic 3D Object Reconstruction from Casual Images

Abstract

Reconstructing complete, scene-aligned 3D objects from casual images requires integrating sparse, uncertain observations and inferring surfaces hidden by occlusion and clutter. We present GATOR, a generative and agentic framework that recovers textured object assets and their scene-relative pose from one or more images. Our local modality mixer couples patch-aligned RGB, target-mask, and pointmap features before cross-view reasoning, preserving scene context while distinguishing the target from its surroundings. Text-guided semantic conditioning complements these spatial cues with category names and object captions through stage-specific adapters for structure, geometry, and appearance generation. The generated asset initializes a multimodal agent, providing instance-specific geometry and pose for targeted structural and texture refinement through an observation-guided edit-render-review loop. Across synthetic objects, cluttered tabletops, and indoor scenes, GATOR achieves strong geometric and appearance fidelity while recovering scene-relative pose from sparse observations. Time-budget comparisons and scene-level simulation further demonstrate the reconstruction efficiency and simulation readiness.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D reconstruction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_N2PpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes GATOR, a generation and agentic refinement framework for 3D object generation. Unlike previous approaches, GATOR uses mask and pointmap information for conditioning. After converting each to embedding by DINOv3 or trainable pointmap encoder, the embeddings are merged through modality mixer which consists of patch-wise attention. Additionally, 3D projected features are used for sparse latent generation. The research further enhances generation performance using text conditioning. It then adopts agentic refinement on the generation objects based on GPT-6-Astra. As a result, GATOR outperformed previous approaches and an agentic-only approach in object and scene reconstruction tasks. Each conditioning and refinement method was demonstrated to contribute on performance improvement.

Critical strengths
  1. Utilization of mask and pointmap information unlike previous methods helps improving generation performance, as evidenced by experiments. Local modality mixer encodes the condition correctly for this purpose.

  2. Generate-and-refine framework outperforms agent-only approach. This result may imply importance of initialization for agentic generation.

Critical weaknesses
  1. The experiment is unfair. While many baselines only use images (and related inputs such as depths, masks, ...) GATOR can use text-condition to improve the result quality. Image-only version of GATOR should also be compared. Furthermore, GATOR utilizes agentic framework while other methods don't. If other combinations (e.g. TRELLIS.2 + agentic refinement) are better than GATOR, sec 3.2 - sec.3.4 can be demonstrated to be worse than previous approaches.

  2. Simply using GPT-6-Astra (with prompt designing) for refinement doesn't seem to be a significant academic contribution, especially compared to the other contributions that the authors are proposing. Moreover, text-embedding of existing embedder is simply provided to the denoiser, which doesn't look to be something new. However, the paper is introducing them as two out of four main contributions.

  3. Definitions of some notations are missing, making the paper difficult to follow. For instance, definitions of x\mathbf{x} and q\mathbf{q} in line 218, jj in line 259, and FDF^D and FNF^N in equation 2 are unclear.

Questions for authors
  1. Is adding condition on noise (line 267-268) natural operation for flow matching? Why is addition used instead of cross-attention?

  2. For the modality mixer, why is patch-wise attention used instead of full attention? Brief explanation would improve the paper. If available, experiments comparing proposed patch-wise and full attention can support the paper.

5 comments
AuthorsOpenReview ↗

Response to Reviewer N2Pp (Part 1 / 3)

We thank Reviewer N2Pp for the careful reading and the precise questions.

  • Without text, pointmaps or the agent, GATOR leads the baselines, and TRELLIS.2 refined by our agent still trails GATOR's generator alone (W1).
  • To our knowledge GATOR is the first to combine a 3D generative model with agentic refinement for reconstruction from casual captures, and its text conditioning is designed per stage (W2).
  • We define each symbol that the review lists and will add the definitions to the paper (W3).
  • Addition matches cross-attention at lower cost, and patch-wise attention beats full attention in the modality mixer (Q1 and Q2).

W1 Is the comparison fair when GATOR uses text and an agent?

First, ShapeR uses the same input modalities as GATOR. In Table R1 it receives the same images, masks and cameras, points from the same pointmaps and a caption.

Second, these inputs need no additional sensors and are cheap to obtain. Masks, cameras, pointmaps and captions can all be estimated from the same casual images (Sec. 3.1), and GATOR is designed as a multimodal model that makes full use of them for in-the-wild captures.

Third, GATOR still leads without text, pointmaps or agent refinement. GATOR is trained with all modalities, and modality dropout (Sec. B.1) lets the same model run with some of them missing at inference. Table R1 first removes text and then also pointmaps, and it compares GATOR with the competitive Pixal3D and ShapeR. All methods receive the same images, masks and cameras, and the checkmarks show which ones also use pointmaps and text. We run ShapeR with and without its caption, and without it ShapeR uses its learned null text condition.

Table R1. Reconstruction of the HANDAL objects without the agent.

MethodPointmapTextCD↓NC↑F1↑
Pixal3D✗✗0.01400.7610.828
ShapeR✓✓0.03020.6580.581
ShapeR✓✗0.02920.6580.592
GATOR✓✓0.00780.8270.932
GATOR✓✗0.00790.8300.932
GATOR✗✗0.00800.8310.928

GATOR leads both baselines on all three metrics in every input setting. GATOR without text and pointmaps also beats the best baseline in all four geometry metrics (CD 0.0080 against 0.0140 on Pixal3D).

Moreover, TRELLIS.2 with our agent refinement still trails GATOR's generator alone. GATOR w/o agent refinement also already beats every baseline, including GPT-6-Astra reconstructing from scratch, in all four geometry metrics on all five datasets. Table 6 in the appendix reports this for each tabletop dataset. Following your suggestion, we also combine our same agent refinement step with TRELLIS.2 on four-view Toys4K reconstructions. We will add the “GATOR w/o agent” rows to Table 1.

Table R2. Four-view Toys4K reconstruction before and after the same agent. † Run with MultiDiffusion as in Table 1.

MethodCD↓NC↑F1↑IoU2D↑PSNR↑SSIM↑LPIPS↓
TRELLIS.2†0.01430.7510.6280.74715.730.8550.231
TRELLIS.2† + agent0.01270.7600.6420.76116.570.8660.215
GATOR w/o agent0.00830.8020.7650.82116.850.8630.204
GATOR0.00860.8010.7570.81917.600.8730.191

The agent improves TRELLIS.2 on all seven metrics. Yet TRELLIS.2 + agent still trails GATOR w/o agent on six of the seven metrics and the full GATOR on all seven. The generator of Secs. 3.2 to 3.4 therefore gives GATOR its lead, and the agent improves both generators.

AuthorsOpenReview ↗

Response to Reviewer N2Pp (Part 2 / 3)

W2 Are agentic refinement and text conditioning significant contributions?

Yes. Our contribution is to combine a generator built for casual captures with agentic refinement, and as the review notes, this combination outperforms the agent-only approach. Existing agentic 3D methods create or edit assets by writing or modifying modeling programs, as in 3D-GPT (Sun et al., 2023), SceneCraft (Hu et al., ICML 2024), LL3M (Lu et al., 2025), VIGA (Yin et al., 2026) and Articraft (Zhou et al., 2026). In GATOR the agent instead refines the complete and scene-aligned asset that the generator recovers from casual captures (Secs. 3.2 to 3.4), in an edit-render-review loop against the input views (Sec. 3.5). To our knowledge GATOR is the first to combine a 3D generative model with agentic refinement for reconstruction from casual captures and to evaluate what the combination gains. The generated asset beats a from-scratch agent in all four geometry metrics on all five datasets, and the agent improves both GATOR and TRELLIS.2 (W1). In addition, during our rebuttal, we extensively study the refinement stage using different agents, including both close and open weight agents. There was a lack of such comprehensive study on the latest agents on 3D object reconstruction tasks, and thus we hope sharing such study and our code, weights, harness can benefit the community.

Text conditioning is also designed per stage rather than simply provided to the denoiser. Text is still under-explored in generative 3D reconstruction. ShapeR (Siddiqui et al., 2026) uses captions but adds caption tokens only to its denoiser and generates no texture. GATOR conditions every stage through stage-specific adapters, with a pooled SigLIP2 category embedding for structure generation and T5Gemma2 caption (with finer-grained descriptions) tokens for the geometry and appearance flows (Secs. 3.3 and 3.4). This conditioning favors category-consistent completion when views are sparse or occluded. Table R3 compares the two representations in the structure and geometry stages. Each variant changes one stage, keeps the other stages of the main model and is trained like the Table 4 ablations.

Table R3. Text conditioning of the structure and geometry stages on the HANDAL objects without the agent.

StageText conditioningCD↓F1↑IoU2D↑
StructurePooled SigLIP2 (ours)0.00880.9120.716
T5Gemma2 tokens0.00910.9060.709
GeometryT5Gemma2 tokens (ours)0.00780.9320.739
Pooled SigLIP20.00780.9320.736

In coarse structure generation, pooled SigLIP2 embedding concentrates on encoding category information for better overall structure generation, as reflected on all three metrics. In the geometry stage, T5Gemma2 embedding of finer-grained descriptions slightly improves the IoU2D.


W3 Missing definition of some notations

We will add these clarifications to the revised paper.

  • x\mathbf{x} (line 218) is a 3D point in the world frame of the input views.
  • q\mathbf{q} (line 218) is the same point in GATOR's normalized coordinates, shifted and scaled by the object anchor but not rotated (Eq. 3 in Sec. A.1).
  • qj\mathbf{q}_j (line 259) is the normalized coordinate of the j-th active voxel of the generated structure.
  • FvDF_v^D (Eq. 2) is the DINOv3 feature map of the masked image of view v.
  • FvNF_v^N (Eq. 2) is the NAF-upsampled version of FvDF_v^D (Chambon et al., 2025).
AuthorsOpenReview ↗

Response to Reviewer N2Pp (Part 3 / 3)

Q1 Is additive conditioning natural for flow matching, and why not cross-attention?

Yes. The projected feature is added to the voxel tokens inside the velocity network, not to the noisy latent, so the rectified flow itself is unchanged (Eq. 4). In each block a voxel's token receives cross-attention to global image and text tokens plus a learned projection of its own feature fj\mathbf{f}_j (Eq. 6 in Sec. A.3). GATOR therefore uses both, cross-attention for global context in unobserved regions and addition for local detail. Because fj\mathbf{f}_j is sampled at the voxel's own projection, it is already aligned with that voxel, and addition passes it on directly. Cross-attention would instead have to search all image tokens for the same correspondence.

Adding spatially aligned conditions to the noisy latent or to the network's features is common in diffusion and flow models. ControlNet (Zhang et al., ICCV 2023), Animate Anyone (Hu, CVPR 2024) and VACE (Jiang et al., ICCV 2025) all do so.

Table R4. Geometry without the agent on the HANDAL objects.

ConditioningCD↓NC↑F1↑IoU2D↑
Addition (ours)0.00780.8300.9330.739
Cross-attention0.00790.8280.9320.737

Following your suggestion, we studied a cross-attention variant in which every voxel attends to all image patch tokens as in TRELLIS.2, and Table R4 above compares it with our addition. Addition matches the cross-attention variant, slightly ahead on all four metrics, and is cheaper in memory and runtime. For a typical object with four views, one training step through the 30 conditioning layers takes 2.3 GiB and 79 ms with cross-attention to all patch tokens, against 1.2 GiB and 37 ms for our conditioning (global cross-attention plus addition). For the full blocks this is 18% more memory and 26% more time. Each view adds 1,029 image tokens, so with eight views cross-attention grows to 3.4 GiB and 120 ms, while our conditioning costs the same as with four.


Q2 Why patch-wise attention instead of full attention in the modality mixer?

The RGB, mask and pointmap tokens of a patch already correspond, so attention within the patch suffices, and it is also more accurate than full attention (Table R5). Within a patch, attention only has to weigh the three cues against each other, for example estimated depth against appearance at a mask boundary. Patches still share global context. The frozen DINOv3 features of RGB and masks already carry context from the whole image, and the structure latents then cross-attend to the mixed tokens of all patches and views (Sec. 3.3). Full attention would repeat this global step and would have to learn such correspondences from scratch, which the alternative shared patch grid already provides, at a cost that grows quadratically with the number of views, patches and modalities.

Adding the mixer with its patch-position and ray encoding lowers ADD-SB from 0.057 to 0.037 in Table 4. Table R5 isolates the attention scope with a variant that applies full self-attention to every token of every view following your suggestion.

Table R5. Structure generation with patch-wise or full attention in the modality mixer on the HANDAL objects without the agent.

Mixer attentionADD-SB↓[email protected]↑CD↓F1↑IoU2D↑
Patch-wise (ours)0.032598.180.01080.8750.706
Full0.040796.360.01160.8630.681

Our current patch-wise attention is better on all five metrics, significantly so on placement (ADD-SB) and IoU2D. We will add this explanation and Table R5 to the revised paper.


We would be glad to discuss any remaining concerns.

Reviewer_KhcqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a pipeline that reconstructs complete textured and scene-aligned objects from sparse images of cluttered scenes. Specifically, it builds on TRELLIS, and mixes patch-aligned RGB, mask, and pointmap features before structure generation, uses stage-specific text conditioning, and projects image features onto the generated structure for geometry and appearance generation. Then GPT-6 is used to edit the object together with Blender. Experiments validate the effectiveness of the proposed method.

Critical strengths
  1. The paper tries to solve a practical problem, i.e., recovering the full 3D asset from so-called casual images.
  2. The use of GPT-6 is timely since it was released several days before the submission deadline.
  3. The whole pipeline was shown to be effective via different evaluation setups.
Critical weaknesses
  1. The novelty is kind of limited. All the components are partially established, like generative backbone, local feature fusion, text adapters, and a prompted agent.
  2. The agentic component is largely heuristic to me. The whole process relies on the agent's judgments and prompt instructions, with limited explicit geometric verification.
  3. The agent's independent benefit is weakly supported. From Table 4, we can see that refinement sometimes worsens several HANDAL geometry metrics. So it is clear enough that why we need the complex agent improvement.
  4. Using the SOTA GPT-6 is timely and interesting, while it also leads to another question. If the VLM model becomes stronger and stronger, is the current pipeline still meaningful?

Overall, I am currently leaning towards rejection, while I am open to being persuaded.

Questions for authors

Please refer to the weakness part.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Khcq (Part 1 / 3)

We thank Reviewer Khcq for the detailed assessment and for being open to discussion.

  • GATOR's novelty lies in its local modality mixer, its stage-specific text conditioning and its agentic refinement of a generated asset, and each one measurably improves reconstruction (W1).
  • An explicit geometric verifier inside the agent's loop changes no metric significantly (W2).
  • The agent repairs local defects such as thin structures and lettering, and in Table 4 it improves three of the four HANDAL geometry metrics (W3).
  • Stronger VLMs plug into GATOR's agent stage and improve its inputs, while its generator gives them a starting point that even GPT-6-Astra from scratch does not reach (W4).

W1 Limited novelty

GATOR builds on established components, and its novelty lies in three designs for casual captures that prior methods lack. Each one changes how the components work together and measurably improves reconstruction.

  • Local modality mixing. ShapeR combines separately encoded tokens only inside its denoiser, and RecGen simply adds mask and pointmap embeddings to its image tokens. In contrast, our GATOR's mixer attends among the aligned RGB, mask and pointmap tokens of each patch before cross-view reasoning (Sec. 3.2), so it can cross-check estimated depth against appearance and the mask. It lowers ADD-SB from 0.057 to 0.037 in Table 4.
  • Stage-specific text conditioning. ShapeR adds caption tokens only to its shape denoiser. Differently, our GATOR conditions structure, geometry and appearance through stage-specific adapters (Secs. 3.3 and 3.4), and text improves all nine metrics in Table 4.
  • Agentic refinement of a generated asset. Agentic 3D methods create or edit assets by writing modeling programs, and prior generative reconstruction methods do not refine their outputs with an agent. To our knowledge GATOR is the first to combine the two for reconstruction. The generated asset anchors the agent to instance-specific geometry, appearance and pose, and the agent repairs fine structure and texture (W3).

GATOR also makes full use of inputs that come cheaply from the same casual images. Unlike ShapeR, which relies on SLAM points pre-segmented for the target, GATOR needs only inputs that can be estimated from the images (Sec. 3.1), and training on perturbed and image-estimated inputs (Sec. B.1) makes it robust to their errors. With only estimated cameras, depth and masks on ScanNet++ it still leads every baseline on all six metrics (Table 3). This robustness on casual captures makes GATOR practical on challenging real data.


W2 Is the agent largely heuristic, with limited geometric verification?

No. The agent's edits are already grounded in the observations. Prompt 1 in appendix instructs the agent to keep the asset's origin, orientation and scale and to preserve reliable geometry. After each repair it renders the candidate, compares it with the input images and keeps the earlier version if the new one regresses (Sec. 3.5).

We also studied with an explicit verifier inside its loop and found it does not necessarily lead to improvement as shown in the table below. In particular, we add an explicit verifier that checks every refinement step's updated version against the input masks and depth (mask IoU, observed-surface IoU and depth chamfer distance) and rejects it when these scores worsen beyond thresholds. When the latest step output does not pass the verification, the reverted version from the last step becomes the output. Otherwise, the agent keeps the refinement loop until the same runtime budget. Over the 360 objects of HANDAL, LM-O, HB and ScanNet++, the verifier rejects 19 of the 1,167 step outputs it checks (1.6%).

Table R1. GPT-6-Astra refinement without and with the per-step geometry verifier inside the loop.

DatasetGeometry verifierADD-SB↓[email protected]↑CD↓NC↑F1↑
TabletopNone0.020798.440.00860.8640.915
Per step0.020698.440.00860.8640.915
ScanNet++None0.041295.590.03150.7250.602
Per step0.041595.590.03140.7240.599

The verifier is only as reliable as the masks and depth it checks against. Tabletop objects are seen from many sides by orbiting cameras, and the verifier rejects only 1 of their 415 step outputs. In cluttered ScanNet++ rooms objects are often occluded, seen from few viewpoints or cut off by the frame, so their estimated masks and depth are less accurate. The other 18 rejections occurred there, and 10 of the 19 rejected step outputs were closer to the ground truth than the mesh the agent kept. An explicit verifier therefore adds handcrafted thresholds without improving on the agent's own review.

AuthorsOpenReview ↗

Response to Reviewer Khcq (Part 2 / 3)

W3 Is the agent's independent benefit supported?

Yes. The agent repairs local defects that the generator leaves, and these repairs matter for downstream use even when averaged metrics change little. In Fig. 11 it recovers a bus's lattice railing, fixes garbled lettering and completes a chair's missing legs, while leaving an accurate asset largely unchanged. In Fig. 5 it recovers holes and thin structures. Refined assets also behave better in simulation, where three generated chairs in a ScanNet++ meeting room lean or topple by 40 to 79 degrees while their refined versions stay within 1.3 degrees of upright (Sec. D.6 and Fig. 12). The project page linked in Fig. 1 (https://gator3d.pages.dev/) shows interactive 3D viewers. Such improvements make the 3D reconstruction more suitable for downstream simulation applications while they are uneasy to quantify or reflect in the metrics.

In Table 4, refinement improves CD, NC, F1, PSNR and LPIPS on HANDAL. The IoU2D and SSIM drops come mainly from one woven strainer that appears in four HANDAL scenes (Table R2).

Table R2. Table 4 before and after refinement, split into the woven strainer in its four scenes and the other 51 HANDAL objects.

ObjectsRefinementCD↓NC↑F1↑IoU2D↑PSNR↑SSIM↑LPIPS↓
Woven strainerBefore0.02160.7730.7640.85416.860.8980.197
After0.01620.6620.8110.65515.080.7650.257
Other objectsBefore0.00680.8330.9430.73019.100.9270.095
After0.00660.8420.9450.72119.960.9300.084

This strainer is the left object in Fig. 1. Its wire bowl with mesh structure is see-through in the photos, but its GT mesh given by the dataset has a wrongly closed solid bowl. The agent rebuilds the bowl as an open weave, which the inaccurate GT penalizes in IoU2D, NC and appearance even though CD improves in all four scenes. On the other 51 objects every metric except IoU2D improves, and a different strainer in Fig. 5 improves on all seven metrics.

Our agent refinement design also benefits other generative models. When directly applied to TRELLIS.2, it improves four-view Toys4K reconstructions on all seven metrics (Table R3).

Table R3. Four-view Toys4K reconstruction before and after the same agent. † Run with MultiDiffusion as in Table 1.

MethodCD↓NC↑F1↑IoU2D↑PSNR↑SSIM↑LPIPS↓
TRELLIS.2†0.01430.7510.6280.74715.730.8550.231
TRELLIS.2† + agent0.01270.7600.6420.76116.570.8660.215
GATOR w/o agent0.00830.8020.7650.82116.850.8630.204
GATOR0.00860.8010.7570.81917.600.8730.191
AuthorsOpenReview ↗

Response to Reviewer Khcq (Part 3 / 3)

W4 Is the pipeline still meaningful as VLMs become stronger?

Yes. Stronger VLMs plug into GATOR's agent stage and improve its inputs, while the generator still gives them a grounded and efficient starting point.

  • New agents plug in without retraining. The agent stage is model-agnostic, so each new VLM upgrades refinement directly. Claude Opus 5.5 already runs with our prompt and tools unchanged and improves appearance more than the locally hosted open-weight Kimi K3.1 Pro (Table R4). This demonstrates how stronger VLMs can benefit our agent refinement step.
  • Our generator remains the efficient starting point. An agent models an object through graphics programs, which can recover recognizable structure yet reduce parts to simplified shapes (Fig. 1). The long chain of tool calling, heavy graphics pipeline, and multi-loop edit and review process also lead to slowness. In contrast, our generator recovers complete, scene-aligned geometry and appearance from the observations in about half a minute, while further model optimization can make our specialized model inference even faster, remaining a more efficient choice than using general, strong, and large VLM-based agents.
  • Stronger VLMs can also improve our generator. Stronger agents can make 3D assets to supply the training data to strengthen our future generator models. Therefore, our model can co-evolve with the advancements of stronger VLMs.

Table R4 compares the refinement by three agents on all real-world objects. Tabletop averages HANDAL, LM-O and HB as in Table 2, and "Best baseline" is the best value of any baseline in Tables 2 and 3 for each metric, including GPT-6-Astra from scratch.

Table R4. Generator-only results and refinement by three agents.

DatasetMethodCD↓NC↑F1↑IoU2D↑PSNR↑LPIPS↓
TabletopBest baseline0.01560.7990.7800.81317.160.160
GATOR w/o agent0.00910.8550.9090.85317.310.146
+ GPT-6-Astra0.00950.8590.9040.84117.970.140
+ Claude Opus 5.50.00900.8550.9080.84617.630.145
+ Kimi K3.1 Pro0.00910.8550.9090.85317.360.146
ScanNet++Best baseline0.03350.7160.5810.716——
GATOR w/o agent0.03080.7260.6060.735——
+ GPT-6-Astra0.03140.7260.6030.733——
+ Claude Opus 5.50.03120.7250.6040.732——
+ Kimi K3.1 Pro0.03080.7260.6060.735——

Current results already point this way. Even GPT-6-Astra reconstructing from scratch trails GATOR's generator alone in all four geometry metrics on all five datasets (Table R5, and Table 6 for each tabletop dataset). The generator alone is also about 18x faster. In Table R5, Tabletop averages HANDAL, LM-O and HB as in Table 2. Time is the mean runtime per object and GATOR runs on one A100 GPU. We will add the “GATOR w/o agent” rows to Tables 1-3.

Table R5. Geometry and runtime of GATOR w/o agent and GPT-6-Astra from scratch.

DatasetMethodCD↓NC↑F1↑IoU2D↑Time↓
Toys4KGPT-6-Astra0.01280.7540.6140.7749.5 min
GATOR w/o agent0.00830.8020.7650.8210.5 min
TabletopGPT-6-Astra0.01560.7990.7730.7919.5 min
GATOR w/o agent0.00910.8550.9090.8530.5 min
ScanNet++GPT-6-Astra0.03350.7160.5810.7169.5 min
GATOR w/o agent0.03080.7260.6060.7350.5 min

Stronger VLMs should therefore increase the benefit of refinement rather than replace the generator.


We would be glad to discuss any remaining concerns.

Reviewer_6cpzOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an agentic method to reconstruct 3D from casual images, which is well motivated. It uses a lot of embeddings like DINOv3, Siglip, gemma, point encoder etc, which I believe all help to improve their performance.

Critical strengths
  1. This method combines the agentic power with sparse transformer type learning in the Trellis-style well. I think it advances this field in turning into more VLM style 3D reconstruction.
  2. the multimodal mixing is interesting
  3. introducing gpt6 astra improves performance.
  4. This work supports multiview.
Critical weaknesses
  1. I think using GPT6-Astra may be treated cautiously here. It can only be accessed via API. I highly suggest authors experimenting with some open-sourced model deployed locally like KimiK3,2.6 or other models. The access to API is a hard requirement for reproducing the results in this paper
  2. GPT6-Astra seems already quite good at texture, looks like the improvement mainly from geometry. I think authors should first try agentic training-free methods with GPT6 for example using the author review loop for it itself, to try to refine geometry.
  3. I think author should also try other LLMs, like Claude, Gemini, etc, to test whether their approach can be generalized to other LLMs.
  4. Relying on too many embeddings. What will the effect be if we remove some of them?
Questions for authors

It would be great if authors can show some curve of increasing number of views and performance. It would be great if authors can try to incorporate this model to a joint layout model as well.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 6cpz (Part 1 / 2)

We thank Reviewer 6cpz for the support and the concrete suggestions. We are glad that the reviewer sees GATOR as advancing VLM-style 3D reconstruction.

  • Without any API, GATOR's generator alone beats every baseline in all four geometry metrics, and the refinement transfers unchanged to Claude Opus 5.5 and also runs with the locally deployed open-weight Kimi K3.1 Pro (W1 & W3).
  • A training-free GPT-6-Astra agent with the same harness is our from-scratch baseline, and GATOR leads it in 21 of the 22 metrics in Tables 1 to 3 (W2).
  • The extra embeddings encode inputs that are cheap to obtain from the same images and improve all seven metrics, and without them GATOR still beats the best baseline in geometry (W4).
  • Accuracy rises quickly as views are added, and two-view GATOR already outperforms eight-view Pixal3D in geometry (Q1).

W1 Is access to the API a hard requirement for reproducing the results?

No. GATOR outperforms the baselines without any API, and its agent stage can also run with an open-weight model deployed locally. GATOR's generator alone (GATOR w/o agent) runs locally and beats every baseline, including GPT-6-Astra reconstructing from scratch, in all four geometry metrics on all five datasets. Table 6 in the appendix reports it for each tabletop dataset, and Table R1 adds ScanNet++. We will add the “GATOR w/o agent” rows to Tables 1-3.

To test other agents, we reran the refinement on all real-world objects. As the reviewer suggested, the agents include Claude Opus 5.5 and the locally deployed open-weight Kimi K3.1 Pro. In Table R1, Tabletop averages HANDAL, LM-O and HB as in Table 2, and "Best baseline" is the best value of any baseline in Tables 2 and 3 for each metric, including GPT-6-Astra from scratch.

Table R1. Generator-only results and refinement by three agents.

DatasetMethodCD↓NC↑F1↑IoU2D↑PSNR↑LPIPS↓
TabletopBest baseline0.01560.7990.7800.81317.160.160
GATOR w/o agent0.00910.8550.9090.85317.310.146
+ GPT-6-Astra0.00950.8590.9040.84117.970.140
+ Claude Opus 5.50.00900.8550.9080.84617.630.145
+ Kimi K3.1 Pro0.00910.8550.9090.85317.360.146
ScanNet++Best baseline0.03350.7160.5810.716——
GATOR w/o agent0.03080.7260.6060.735——
+ GPT-6-Astra0.03140.7260.6030.733——
+ Claude Opus 5.50.03120.7250.6040.732——
+ Kimi K3.1 Pro0.03080.7260.6060.735——

Kimi K3.1 Pro edits geometry more conservatively and slightly raises PSNR. It makes only minor changes to the generator's geometry and repairs textures and materials on 33 of the 360 assets, so its gains are smaller than those of the two frontier agents. Even so, the fully local pipeline with Kimi beats the best baseline in every metric of Table R1, and stronger open-weight agents can replace Kimi without retraining GATOR.

Recent 3D works also build on closed frontier models. Examples include LEGO-Anything (Li et al., 2026) and 3DCodeVerse (Gao et al., 2026) with GPT-6-Astra, Code-as-Room (Yang et al., 2026) with GPT-5.5, S-Agent (Dai et al., 2026) with GPT-5.4 and WorldClaw (Guo et al., 2026) with Claude Opus 4.8. In particular, our framework only uses agents for local minor refinement, while our generative network's initial reconstruction already achieves strong results. Furthermore, for reproducibility, we will release GATOR's code and weights with the agent's prompt (Prompt 1) and harness.


W2 Can a training-free GPT-6-Astra loop refine geometry on its own?

Our from-scratch GPT-6-Astra baseline (Sec. 4.1) is such a training-free agent. It uses the same input observations, harness and runtime budget as our pipeline but starts without a mesh (Prompt 2 in Sec. A.4). This harness includes the refinement tools of our agent stage, which can repair geometry. This agent is one of the most competitive baselines and is strong in appearance as the reviewer observes. GATOR leads it in 21 of the 22 metrics in Tables 1 to 3.

AuthorsOpenReview ↗

Response to Reviewer 6cpz (Part 2 / 2)

W3 Can the approach generalize to other LLMs?

Yes. Claude Opus 5.5 shows refinement capabilities similar to GPT-6-Astra (Table R1). Both agents improve appearance on the tabletop datasets while keeping geometry close to the generator's, and on ScanNet++ they perform nearly identically. Claude edits more conservatively, leaving 67 of the 360 assets unchanged against 5 for GPT-6-Astra, so its appearance gains are smaller.


W4 Relying on too many embeddings. What if some embeddings are removed?

These embeddings encode inputs that are cheap to obtain from the same images and improve GATOR further, and GATOR leads the baselines in geometry even without them. Besides RGB, every input can be estimated from the images themselves, such as cameras and pointmaps with VGGT-Ω, and the text is a category name or a VLM caption (Sec. 3.1). GATOR is trained with all modalities, and modality dropout (Sec. B.1) lets the same model drop text or pointmaps at test time without retraining. Removing text drops the SigLIP2 and T5Gemma2 embeddings, and removing pointmaps drops the pointmap encoder.

Table R2. GATOR w/o agent with inputs removed at inference, averaged over HANDAL, LM-O and HB as in Table 2.

Inputs at inferenceCD↓NC↑F1↑IoU2D↑PSNR↑SSIM↑LPIPS↓
All (RGB, pointmap, text)0.00910.8550.9090.85317.310.8910.146
No text0.00910.8520.9060.85417.120.8900.148
No pointmap0.01020.8370.8840.84017.000.8890.152
No pointmap, no text0.01020.8420.8850.84116.860.8870.154

Pointmaps mainly improve geometry and text mainly improves appearance, so together they improve all seven metrics. Even without both, GATOR beats Table R1's best baseline in all four geometry metrics (CD 0.0102 against 0.0156).


Q1 Performance versus number of views

Accuracy rises quickly as views are added, and with only two views GATOR already outperforms Pixal3D with eight in every geometry metric. Table 7 (Sec. D.3) varies the number of views for GATOR w/o agent on LM-O and HB. Its CD falls from 0.0109 with two views to 0.0096 with eight, and with two views it is already about half that of Pixal3D with eight (0.0201).

Table 7 covers tabletop scenes, so we add ScanNet++, where objects in cluttered rooms are seen in one to ten views selected for visibility and viewpoint diversity (Sec. D.4). As in Table 7, Table R3 gives GATOR w/o agent results on one, two, four and six of these views, here on the 55 objects that have at least six.

Table R3. GATOR w/o agent on the 55 ScanNet++ objects with at least six views, given nested subsets of one to six of their views.

ViewsCD↓NC↑F1↑IoU2D↑
10.04340.6520.5160.613
20.03600.6750.5840.651
40.03140.6890.6240.674
60.03130.6940.6280.680

Each step up to four views improves F1 and IoU2D significantly, and six views bring no significant change over four. The 143 objects with at least four views follow the same trend. We will add view-count curves to the appendix.


Q2 Joint layout model

GATOR's posed outputs already compose into scenes and can serve as input to a joint layout model. GATOR predicts each object's scene-relative pose, so its outputs directly compose into scenes without a separate layout step (Fig. 1 right), with an [email protected] of 95.63% on ScanNet++ (Table 3). A joint layout model that resolves interpenetration and contact could take these posed assets as input. We demonstrate a scene-level improvement with GPT-6-Astra in Sec. D.6 (Fig. 12) and on the project page. There, the reconstructed objects of a ScanNet++ meeting room are simulated together, and three chairs that lean or topple before refinement stay upright after it.


We would be glad to discuss any remaining questions.

Thank you again for your support and the concrete suggestions. As the discussion period is coming to an end, we would like to check whether our response addresses your concerns. We would be glad to answer any further questions. If our response resolves your concerns, we would be grateful if you would consider reflecting this in your final rating.

The authors answer my question quite well, and I improve my rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Rebuttal Summary

We thank the reviewers for their constructive and detailed reviews. They found that GATOR tackles a practical and well-motivated problem (6cpz, Khcq), is effective across evaluation setups and supports multiple views (6cpz, Khcq) and advances VLM-style 3D reconstruction with a timely use of GPT-6 (6cpz, Khcq). They also found that its local modality mixer uses masks and pointmaps to improve generation (6cpz, N2Pp), that agentic refinement improves performance (6cpz) and that generate-and-refine outperforms an agent-only approach (N2Pp).


Below we summarize additional experiments and clarification:

  • Generator alone (6cpz, Khcq, N2Pp). Without the agent refinement or any API, GATOR outperforms every baseline overall, including GPT-6-Astra reconstructing from scratch on real-world datasets, as shown in Table 2 and 3 in our updated manuscript. Our responses below show averaged results on Toys4K, tabletop datasets and ScanNet++.
  • Other agents including open models (6cpz). With the prompt and harness unchanged in our agent refinement step, Claude Opus 5.5 shows refinement capabilities similar to GPT-6-Astra. We also tested with self-hosting the open-weight Kimi K3.1 Pro which edits geometry more conservatively and slightly raises PSNR.
  • Modality ablation (6cpz, N2Pp). Modality dropout lets one model run without text or pointmaps. Without both, GATOR still beats every baseline in all four geometry metrics on the tabletop datasets, and adding them improves all seven metrics.
  • TRELLIS.2 equipped with our agent refinement (N2Pp). With the same agent refinement proposed in our framework, GATOR outperforms TRELLIS.2 on all seven metrics.
  • Further analyses (6cpz, Khcq, N2Pp). An explicit geometric verifier changes no metric significantly, ScanNet++ accuracy changes little beyond four views and additional ablations support design choices of the generator.

Revisions (in red in the uploaded PDF)

  • Generator-only rows in Tables 1, 2 and 3 (6cpz, Khcq, N2Pp).
  • Definitions of x and q in Sec. 3.1 and of qjq_j, FvDF_v^D and FvNF_v^N in Sec. 3.4 (N2Pp).
  • The rationale for patch-wise attention in the modality mixer, with its comparison to full attention (Sec. D.8 and Table 8) (N2Pp).
  • The view-count curves on ScanNet++ (Sec. D.7 and Fig. 13) (6cpz).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.