- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CoVoGen reconstructs separate object meshes from sparse posed RGB-D views and instance masks matched across views. It first predicts shared coarse scene occupancy using a depth-conditioned adaptation of GenRecon. A learned membership classifier selects coarse voxels for each object. These voxels, neighboring geometry, and aligned observations guide local object refinement. A frozen TRELLIS.2 geometry generator then produces meshes in world coordinates. The main two-view experiments report improved object and scene F-scores, while ablations suggest that completed scene structure is particularly helpful for heavily occluded objects.
Critical strengths
- A useful connection between scene completion and object generation. The method combines shared scene structure with pretrained object priors to generate separate meshes in world coordinates. The visibility-based ablation is particularly informative: scene-derived conditioning improves object Mean F by 17.75 percentage points for objects with less than 25% visible surface, supporting its value under heavy occlusion.
- Broad evaluation with useful checks. The paper evaluates four datasets, including held-out real scenes, compares several view budgets against ShapeR, and reports paired scene-bootstrap confidence intervals. F-score evaluation includes failed outputs, and additional captured-depth measurements help assess consistency with real observations.
Critical weaknesses
- The key technical advance over prior work remains unclear. HIVE-3D already follows a coarse-scene generation, component extraction, and voxel-conditioned refinement pipeline. GenRecon provides the scene-generation foundation, while 3D-Fixer already performs coarse-to-fine object completion in place. CoVoGen mainly adds depth-aware conditioning, learned membership, and local spatial refinement, but the discussion and experiments do not sufficiently establish what these designs contribute beyond the closest prior work.
- Robustness and spatial consistency in complex scenes are insufficiently evaluated. Object extraction depends on predicted scene occupancy, and no mesh is generated when an object receives no supporting voxels. The paper provides limited analysis of recovery when small objects, thin structures, or substantial parts are missing. Although refinement uses neighboring geometry, it does not directly evaluate whether shape changes introduce intersections, lost contacts, or floating objects. A reasonable coarse layout therefore does not establish the physical plausibility of the final scene. Multi-room examples are provided, but background geometry is excluded from the final evaluation, and complex backgrounds and larger scenes lack systematic validation.
Questions for authors
- Please clarify the paper’s main technical innovation.
- How does performance vary with object size, crowding, and coarse-support recall? When valid masks and depth observations are available but the predicted support is empty or severely incomplete, is there a fallback mechanism? Can the method recover thin structures or substantially missing geometry?
- Refinement and mesh generation change individual object shapes. How are non-penetration, contact, and support relationships preserved? Are there before-and-after measurements of intersections, lost contacts, or floating objects?
- The architectural backgrounds in the multi-room examples use reference geometry. What is the current scope of the method for complex backgrounds and larger scenes? What are the main limitations and possible extensions regarding background generation, scene chunking, and computational cost?
5 comments
Thanks for the additional experiments and explanations. They answer some of my questions. I now see this as a borderline paper, between 2 and 3.
- I do not find the overall approach particularly elegant, with several stages that need to be trained separately. CoVoGen is faster than ShapeR, but the gains are not enough to change my view. Since the methods use different base models and training setups, it is hard to tell how much of the improvement comes from the proposed design. Texturing has not been evaluated, and it is unclear how much extra adaptation it would require.
- I also remain concerned about the quality of mesh. The real-scene example looks better than ShapeR, but still has visible fragments and other shape errors. Limited observations and a gap between the training data and real scenes may partly explain this, but producing reliable geometry under these conditions is an important part of the task.
- Some of the limitations discussed earlier also remain.
Follow-up on attribution, texturing, and geometry quality
Thank you for the reassessment and for clarifying the remaining concerns.
Modularity and contribution. The scene stage provides structural support for occluded objects, while local refinement combines this support with spatially aligned observations and pretrained object priors. Our contribution is this conditioning interface between completed scene occupancy and instance-specific refinement. The modular formulation connects global scene completion with local object priors while keeping the final geometry generator frozen. Efficiency is an additional practical benefit of this design.
Evidence under a fixed model. To examine the contribution of scene-derived target support with the backbone fixed, we use a shared-checkpoint ablation on 40 scenes/211 targets. Replacing visible-depth target support with completed-scene target support improves final Mean F from 79.91% to 86.05%, with the same neighboring support, trained models, two RGB-D inputs, initialization, sampling and postprocessing. This provides diagnostic evidence for the value of scene-derived structure within the same trained model, complementing the system-level comparison with ShapeR.
Texturing without additional training. We have additionally conducted a qualitative texturing test using frozen TRELLIS.2 on both two-storey examples, covering 64 objects. It operates on the existing postprocessed meshes using one observed image crop per object, with no additional training or fine-tuning. Figures T1–T2 in TEXTURING_SUPPLEMENT.pdf, included in the updated supplementary ZIP, show the full-scene views and local texture details. These retain the original dense-view geometry settings and object positions; gray architecture is reference geometry. The examples demonstrate integration without learned adaptation.
The real-scene comparison shows improved scene structure under matched observations, with fine-surface fidelity remaining an important direction for further improvement.
Response to Reviewer XYrQ
W1/Q1:
Our contribution is the conditioning interface from completed shared occupancy to instance-specific, spatially aligned object refinement:
The hierarchy and backbones are inherited; the contribution lies in constructing and transferring these conditions.
W2/Q2:
We paired CoVoGen and ShapeR on all 1,129 scenes/6,880 targets, retaining unknown groups and all eight generation failures scored as F=0. Differences below are CoVoGen minus ShapeR in object Mean F percentage points. Size bins are within-dataset tertiles of reference-box diagonal:
There is no automatic fallback for empty support: no mesh is generated, affecting 4/6,613 synthetic targets. Nonempty support undergoes ordinary refinement, but missing parts or an inaccurate frame can persist.
W2/Q3:
Shared coordinates and target/neighbor occupancy provide spatial conditioning, without hard non-penetration, contact or support constraints. The two-view diagnostic covers the shared 40 scenes/211 targets/876 pairs. “Coarse” decodes membership occupancy; “final” uses saved final meshes. Both receive postprocessing, so differences include decoding/resampling effects as well as refinement.
Crossings measure triangle-surface intersections, not penetration volume. Contact is not guaranteed: at 1% of the reference-object diagonal, four of eight initially valid FRONT floor contacts remain near and four become crossing; one of seven initially near-floor SAGE targets moves below it.
W2/Q4:
We generate target-object meshes in shared world coordinates. In the rebuttal visual supplement (p. 1, Fig. Q4.1), colored objects are generated; gray walls, floors and stairs are reference geometry. Architectural RGB-D observations can condition generation, but complete architectural meshes are outside the current output scope.
Both two-storey/four-room cases retain all 32 targets at their original positions, using 364/387 RGB-D views. They demonstrate dense-view feasibility.
Overlapping chunk predictions are merged on a common lattice before instance assignment. Figure Q4.2 (p. 1) shows the vertical partition using 2.85 m cubes and 1.425 m stride:
Timings use one RTX PRO 5000 72 GB. Architecture generation and adaptive chunk scheduling are possible extensions.
The cited visual supplement is REBUTTAL_VISUAL_SUPPLEMENT.pdf in the updated supplementary ZIP.