- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes GATOR, a generation and agentic refinement framework for 3D object generation. Unlike previous approaches, GATOR uses mask and pointmap information for conditioning. After converting each to embedding by DINOv3 or trainable pointmap encoder, the embeddings are merged through modality mixer which consists of patch-wise attention. Additionally, 3D projected features are used for sparse latent generation. The research further enhances generation performance using text conditioning. It then adopts agentic refinement on the generation objects based on GPT-6-Astra. As a result, GATOR outperformed previous approaches and an agentic-only approach in object and scene reconstruction tasks. Each conditioning and refinement method was demonstrated to contribute on performance improvement.
Critical strengths
-
Utilization of mask and pointmap information unlike previous methods helps improving generation performance, as evidenced by experiments. Local modality mixer encodes the condition correctly for this purpose.
-
Generate-and-refine framework outperforms agent-only approach. This result may imply importance of initialization for agentic generation.
Critical weaknesses
-
The experiment is unfair. While many baselines only use images (and related inputs such as depths, masks, ...) GATOR can use text-condition to improve the result quality. Image-only version of GATOR should also be compared. Furthermore, GATOR utilizes agentic framework while other methods don't. If other combinations (e.g. TRELLIS.2 + agentic refinement) are better than GATOR, sec 3.2 - sec.3.4 can be demonstrated to be worse than previous approaches.
-
Simply using GPT-6-Astra (with prompt designing) for refinement doesn't seem to be a significant academic contribution, especially compared to the other contributions that the authors are proposing. Moreover, text-embedding of existing embedder is simply provided to the denoiser, which doesn't look to be something new. However, the paper is introducing them as two out of four main contributions.
-
Definitions of some notations are missing, making the paper difficult to follow. For instance, definitions of and in line 218, in line 259, and and in equation 2 are unclear.
Questions for authors
-
Is adding condition on noise (line 267-268) natural operation for flow matching? Why is addition used instead of cross-attention?
-
For the modality mixer, why is patch-wise attention used instead of full attention? Brief explanation would improve the paper. If available, experiments comparing proposed patch-wise and full attention can support the paper.
5 comments
Response to Reviewer N2Pp (Part 2 / 3)
W2 Are agentic refinement and text conditioning significant contributions?
Yes. Our contribution is to combine a generator built for casual captures with agentic refinement, and as the review notes, this combination outperforms the agent-only approach. Existing agentic 3D methods create or edit assets by writing or modifying modeling programs, as in 3D-GPT (Sun et al., 2023), SceneCraft (Hu et al., ICML 2024), LL3M (Lu et al., 2025), VIGA (Yin et al., 2026) and Articraft (Zhou et al., 2026). In GATOR the agent instead refines the complete and scene-aligned asset that the generator recovers from casual captures (Secs. 3.2 to 3.4), in an edit-render-review loop against the input views (Sec. 3.5). To our knowledge GATOR is the first to combine a 3D generative model with agentic refinement for reconstruction from casual captures and to evaluate what the combination gains. The generated asset beats a from-scratch agent in all four geometry metrics on all five datasets, and the agent improves both GATOR and TRELLIS.2 (W1). In addition, during our rebuttal, we extensively study the refinement stage using different agents, including both close and open weight agents. There was a lack of such comprehensive study on the latest agents on 3D object reconstruction tasks, and thus we hope sharing such study and our code, weights, harness can benefit the community.
Text conditioning is also designed per stage rather than simply provided to the denoiser. Text is still under-explored in generative 3D reconstruction. ShapeR (Siddiqui et al., 2026) uses captions but adds caption tokens only to its denoiser and generates no texture. GATOR conditions every stage through stage-specific adapters, with a pooled SigLIP2 category embedding for structure generation and T5Gemma2 caption (with finer-grained descriptions) tokens for the geometry and appearance flows (Secs. 3.3 and 3.4). This conditioning favors category-consistent completion when views are sparse or occluded. Table R3 compares the two representations in the structure and geometry stages. Each variant changes one stage, keeps the other stages of the main model and is trained like the Table 4 ablations.
Table R3. Text conditioning of the structure and geometry stages on the HANDAL objects without the agent.
| Stage | Text conditioning | CD↓ | F1↑ | IoU2D↑ |
|---|---|---|---|---|
| Structure | Pooled SigLIP2 (ours) | 0.0088 | 0.912 | 0.716 |
| T5Gemma2 tokens | 0.0091 | 0.906 | 0.709 | |
| Geometry | T5Gemma2 tokens (ours) | 0.0078 | 0.932 | 0.739 |
| Pooled SigLIP2 | 0.0078 | 0.932 | 0.736 |
In coarse structure generation, pooled SigLIP2 embedding concentrates on encoding category information for better overall structure generation, as reflected on all three metrics. In the geometry stage, T5Gemma2 embedding of finer-grained descriptions slightly improves the IoU2D.
W3 Missing definition of some notations
We will add these clarifications to the revised paper.
- (line 218) is a 3D point in the world frame of the input views.
- (line 218) is the same point in GATOR's normalized coordinates, shifted and scaled by the object anchor but not rotated (Eq. 3 in Sec. A.1).
- (line 259) is the normalized coordinate of the j-th active voxel of the generated structure.
- (Eq. 2) is the DINOv3 feature map of the masked image of view v.
- (Eq. 2) is the NAF-upsampled version of (Chambon et al., 2025).
Response to Reviewer N2Pp (Part 3 / 3)
Q1 Is additive conditioning natural for flow matching, and why not cross-attention?
Yes. The projected feature is added to the voxel tokens inside the velocity network, not to the noisy latent, so the rectified flow itself is unchanged (Eq. 4). In each block a voxel's token receives cross-attention to global image and text tokens plus a learned projection of its own feature (Eq. 6 in Sec. A.3). GATOR therefore uses both, cross-attention for global context in unobserved regions and addition for local detail. Because is sampled at the voxel's own projection, it is already aligned with that voxel, and addition passes it on directly. Cross-attention would instead have to search all image tokens for the same correspondence.
Adding spatially aligned conditions to the noisy latent or to the network's features is common in diffusion and flow models. ControlNet (Zhang et al., ICCV 2023), Animate Anyone (Hu, CVPR 2024) and VACE (Jiang et al., ICCV 2025) all do so.
Table R4. Geometry without the agent on the HANDAL objects.
| Conditioning | CD↓ | NC↑ | F1↑ | IoU2D↑ |
|---|---|---|---|---|
| Addition (ours) | 0.0078 | 0.830 | 0.933 | 0.739 |
| Cross-attention | 0.0079 | 0.828 | 0.932 | 0.737 |
Following your suggestion, we studied a cross-attention variant in which every voxel attends to all image patch tokens as in TRELLIS.2, and Table R4 above compares it with our addition. Addition matches the cross-attention variant, slightly ahead on all four metrics, and is cheaper in memory and runtime. For a typical object with four views, one training step through the 30 conditioning layers takes 2.3 GiB and 79 ms with cross-attention to all patch tokens, against 1.2 GiB and 37 ms for our conditioning (global cross-attention plus addition). For the full blocks this is 18% more memory and 26% more time. Each view adds 1,029 image tokens, so with eight views cross-attention grows to 3.4 GiB and 120 ms, while our conditioning costs the same as with four.
Q2 Why patch-wise attention instead of full attention in the modality mixer?
The RGB, mask and pointmap tokens of a patch already correspond, so attention within the patch suffices, and it is also more accurate than full attention (Table R5). Within a patch, attention only has to weigh the three cues against each other, for example estimated depth against appearance at a mask boundary. Patches still share global context. The frozen DINOv3 features of RGB and masks already carry context from the whole image, and the structure latents then cross-attend to the mixed tokens of all patches and views (Sec. 3.3). Full attention would repeat this global step and would have to learn such correspondences from scratch, which the alternative shared patch grid already provides, at a cost that grows quadratically with the number of views, patches and modalities.
Adding the mixer with its patch-position and ray encoding lowers ADD-SB from 0.057 to 0.037 in Table 4. Table R5 isolates the attention scope with a variant that applies full self-attention to every token of every view following your suggestion.
Table R5. Structure generation with patch-wise or full attention in the modality mixer on the HANDAL objects without the agent.
| Mixer attention | ADD-SB↓ | [email protected]↑ | CD↓ | F1↑ | IoU2D↑ |
|---|---|---|---|---|---|
| Patch-wise (ours) | 0.0325 | 98.18 | 0.0108 | 0.875 | 0.706 |
| Full | 0.0407 | 96.36 | 0.0116 | 0.863 | 0.681 |
Our current patch-wise attention is better on all five metrics, significantly so on placement (ADD-SB) and IoU2D. We will add this explanation and Table R5 to the revised paper.
We would be glad to discuss any remaining concerns.
Response to Reviewer N2Pp (Part 1 / 3)
We thank Reviewer N2Pp for the careful reading and the precise questions.
W1 Is the comparison fair when GATOR uses text and an agent?
First, ShapeR uses the same input modalities as GATOR. In Table R1 it receives the same images, masks and cameras, points from the same pointmaps and a caption.
Second, these inputs need no additional sensors and are cheap to obtain. Masks, cameras, pointmaps and captions can all be estimated from the same casual images (Sec. 3.1), and GATOR is designed as a multimodal model that makes full use of them for in-the-wild captures.
Third, GATOR still leads without text, pointmaps or agent refinement. GATOR is trained with all modalities, and modality dropout (Sec. B.1) lets the same model run with some of them missing at inference. Table R1 first removes text and then also pointmaps, and it compares GATOR with the competitive Pixal3D and ShapeR. All methods receive the same images, masks and cameras, and the checkmarks show which ones also use pointmaps and text. We run ShapeR with and without its caption, and without it ShapeR uses its learned null text condition.
GATOR leads both baselines on all three metrics in every input setting. GATOR without text and pointmaps also beats the best baseline in all four geometry metrics (CD 0.0080 against 0.0140 on Pixal3D).
Moreover, TRELLIS.2 with our agent refinement still trails GATOR's generator alone. GATOR w/o agent refinement also already beats every baseline, including GPT-6-Astra reconstructing from scratch, in all four geometry metrics on all five datasets. Table 6 in the appendix reports this for each tabletop dataset. Following your suggestion, we also combine our same agent refinement step with TRELLIS.2 on four-view Toys4K reconstructions. We will add the “GATOR w/o agent” rows to Table 1.
The agent improves TRELLIS.2 on all seven metrics. Yet TRELLIS.2 + agent still trails GATOR w/o agent on six of the seven metrics and the full GATOR on all seven. The generator of Secs. 3.2 to 3.4 therefore gives GATOR its lead, and the agent improves both generators.