acceptodds
Under review as a conference paper at ICLR 2027

FIRE3D: Feed-forward Interactive 3D Scene Reconstruction Within A Minute

Abstract

We present FIRE3D, a unified framework that transforms a single RGB image or casual video into interactable 3D scene assets for games and interactive applications in under a minute for up to twelve textured instances including preprocessing. At the core of FIRE3D is a feed-forward inference pipeline that predicts a compositional scene representation from posed RGB-D observations estimated from the RGB capture, including the 6-DoF pose, bounding box, mesh, and texture for detected objects. By modeling the scene as a collection of discrete entities, FIRE3D produces amodal object assets that can be independently edited and used in interactive applications. Our framework requires no test-time optimization, runs substantially faster than the compared reconstruction systems, and provides object-level completeness beyond existing feed-forward 3D approaches. We demonstrate leading detection accuracy, strong geometry reconstruction, and competitive rendering quality on the evaluated benchmarks, with substantial runtime gains.

Primary area:
generative models
Keywords:
Computer Vision, Reconstruction, Generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Aa4uOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a systematic approach for simultaneous scene layout and object estimation. The contributions lie in three folds: (1) the pipeline that parses the scene into object poses and instances and then feeds the features into a latent diffusion network for object reconstruction. (2) an HC-VAE that compacts the latents without loss of quality. (3) a comprehensive dataset for training such a network.

Experiments show the approach's effectiveness in both versatility and accuracy.

Critical strengths
  • A systematic contribution that leads to solid progress toward automatic composite scene reconstruction.

  • A compact latent that reduces computation, with potentially broader applications in generative 3D methods.

  • A dataset for following works to build on, suppose it will be released.

Critical weaknesses

I do not see a "critical weakness" but do have concerns about the design choice.

  • The model adopts a two-stage approach where object proposals with lower confidence will be filtered out before 3D reconstruction. Will some critical but difficult-to-localize objects be omitted? What if you adopt an end-to-end approach where the object features are directly sent as the condition?

  • The technical pipeline is a composition of a DETR and a TRELLIS-like model, which itself is relatively incremental.

Questions for authors
  • It will be nice to see how performance varies with different compression rates in Table A6.

  • As HC-VAE is trained from scratch on top of SC-VAE, how much data is minimally required to match the performance of SC-VAE as indicated in Table A6?

  • There is a lack of a real-world dataset for 3D reconstruction quantitative comparison. The comparison under ScanNet++ is suggested.

  • Some related research works on amodal scene reconstruction should be discussed:

World Tracing: Generative Pixel-Aligned Geometry Beyond the Visible

LaRI: Layered Ray Intersections for Single-view 3D Geometric Reasoning

NOVA3R: Non-pixel-aligned Visual Transformer for Amodal 3D Reconstruction

VolFill: Single-View Amodal 3D Scene Reconstruction with Volumetric Flow Matching

Know Your Neighbors: Improving Single-View Reconstruction via Spatial Vision-Language Reasoning

Reviewer_yKdnOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors present FIRE3D, which targets reconstructing object-level 3D scenes from a single RGB image or a casual video. The core differences compared to other methods are the prompt-free feed-forward framework and speed. With Pi3X-estimated point clouds and DINOv3 features, a perception network takes the lifted voxel features as input to predict the segmentation and object poses, and flow-matching models then generate each object. An HC-VAE is built on SC-VAE to improve the inference speed at a sacrifice in accuracy. They curate a large-scale dataset to train the model. Experiments show competitive reconstruction quality within a minute.

Critical strengths
  1. The system is practical and useful. It integrates perception and reconstruction models and accelerates object-level reconstruction, producing simulator-usable indoor scenes in about one minute.

  2. HC-VAE is an intuitive and effective design. It trades some accuracy for a large speedup.

  3. Across the evaluated benchmarks, FIRE3D achieves the highest detection mAP and strong geometric reconstruction while running substantially faster than the compared methods.

Critical weaknesses
  1. It is questionable whether the framework is truly "unified": the perception and generation modules are trained separately and communicate only through instance point clouds and poses, which is very similar to a pipeline with an external parser.

  2. The perception network relies mostly on well-known building blocks. Its gains are large on synthetic benchmarks but small on the real-world AEO benchmark (mAP 0.25 vs. 0.23). Since neither its design choices nor the data choices (e.g., Flux.2 augmentation) are ablated, the source of the perception gains is unclear.

  3. Tab. 1 lists the input as an RGB image or video, but the network actually takes Pi3X-estimated posed RGB-D.

  4. The effect of HC-VAE on fine detail is not adequately evaluated: SceneGen appears to capture finer structures than FIRE3D in Fig. 7.

Questions for authors
  1. Could you separate the contributions of the architecture and the training data for the perception model?

  2. How robust is the generator to noisy and heavily occluded instance point clouds?

  3. How were the HC-VAE latent resolution and channel width chosen? Did you evaluate other configurations, or 2D feature planes (e.g., a triplane) instead of sparse 3D convolutions?

  4. Could you provide additional analysis or comparisons of how much fine detail is lost through HC-VAE.

  5. How do you verify that the Flux.2-edited frames remain pixel-aligned with the original depth and instance labels, and consistent across frames?

  6. The submission does not appear to include the AI statement required by the ICLR 2027 AI Policy for Authors. Could you add it?

  7. The reconstructed textures appear rather synthetic; is their fidelity sufficient for downstream robot simulation?

Reviewer_rfjHOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FIRE3D proposes an object-centric 3D scene reconstruction system from a single image or video. It estimates posed RGB-D observations, predicts object instances and poses, and generates separate textured meshes. A compact latent representation enables batched reconstruction without per-scene optimization.

The authors curate a large-scale training corpus and evaluate OBB results, instance segmentation results, overall mesh geometry.

FIRE3D outperforms Boxer in detection AP on all three tested datasets, although the improvement is much larger on synthetic scenes than on real AEO data. It also outperforms the ShapeR pipeline under inferred perception on the synthetic benchmarks, while ShapeR remains stronger on its real-world geometry benchmark.

Critical strengths

Oriented bounding box detection remains a valuable task in 3D, especially for accurate simulation reconstruction and other applications, and this work produces promising results compared to SOTA method on this.

Although more direction comparision, such as compare with the dataset in Boxer paper would be more convicing.

Critical weaknesses
  • The paper does not clearly state which categories it supports.
  • Evaluation on Boxer’s original benchmark datasets would provide a more convincing comparison, which is currently missing.
Questions for authors
  1. What object categories does this support?
  2. Have the author compared this with the latest Agent Astra detections?
  3. How many object categories can it support per scene?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.