- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces OPERA, a framework designed to improve single-view 3D reconstruction by integrating object-specific perceptual knowledge into the multi-view diffusion sampling process. The core idea is to employ Generation-Perception Alignment Modules (GPAMs) to map intermediate generative features from a diffusion model into the target feature spaces of pre-trained 2D perception foundation models. The gradients derived from these perception models are then utilized to guide the sampling process at test time. The authors demonstrate their method on top of existing baselines.
Critical strengths
-
The paper is well-written. The architecture of the GPAMs and the overall guidance pipeline are clearly explained. The implementation provided in the appendix ensure its reproducibility.
-
The idea of leveraging the generalized 2D priors from perception foundation models to constrain and improve 3D generation is well-motivated. Align disparate feature spaces using GPAMs is technically sound and well designed.
Critical weaknesses
-
The framework is essentially a test-time "classifier guidance" mechanism. It relies on external 2D priors to patch existing baselines rather than solving root 3D reconstruction challenges.
-
The gradient-based optimization at each diffusion step causes an impractical inference time (~4 minutes per object), making it uncompetitive with modern feed-forward models for real-world applications.
-
The quantitative improvements, particularly on the Wonder3D baseline, are minor and fail to justify the massive computational overhead introduced by the guidance mechanism.
-
The objects shown in the current figures lack geometric complexity. How does the framework perform on inputs with intricate structures or severe occlusions? Providing such examples would better validate the necessity of the perceptual guidance.
Questions for authors
-
Could you provide a deeper comparison between OPERA and existing standard "classifier guidance" techniques? What makes the GPAM approach fundamentally different or superior to standard feature-matching guidance in diffusion models?
-
The gains on Wonder3D are somewhat marginal. How does OPERA perform when applied to more recent, inherently more consistent 3D generation models? Does the guidance still provide meaningful improvements, or do the base model's native capabilities saturate the benefits of 2D perceptual guidance?
2 comments
Answers to Reviewer upPh's questions - updated
Q1. Differences from conventional classifier guidance
Conventional classifier guidance typically employs a pretrained classifier to provide gradients associated with a target class label during diffusion sampling. However, our task is fundamentally different: we aim to generate multiple perceptually consistent views from a single input image for subsequent 3D reconstruction, rather than generate a 2D image conditioned on a class label.
In particular, a class label provides only coarse semantic information and does not capture the detailed perceptual characteristics of the specific object being reconstructed. OPERA instead exploits richer perceptual representations extracted from pretrained foundation models.
Furthermore, extending perceptual guidance to multi-view generation requires addressing the relationships between representations across different views. As discussed in our response to Reviewer 1, direct feature matching using an L1/L2 objective is suitable for front-view guidance, but does not explicitly capture cross-view representational relationships. We therefore introduce CKA-based alignment to support multi-view perceptual guidance.
Another important distinction is the role of GPAM. Rather than requiring the external perception models to provide guidance gradients at every sampling step, GPAM learns to predict the corresponding perceptual representations from intermediate diffusion features. The perceptual targets are extracted once from the input image and remain fixed throughout sampling, avoiding repeated evaluation of the pretrained perception models.
Thus, the contribution of OPERA is not merely the use of an external classifier or feature-matching loss, but the formulation of a foundation-model-driven perceptual alignment and guidance framework for multi-view generation.
Q2. Generalisation to more recent 3D generation models
Please refer to our response to Reviewer 1 regarding the differences in improvement margins between Wonder3D and Era3D.
We appreciate the reviewer's question about whether perceptual guidance can benefit more recent 3D generation models with stronger native capabilities. To investigate this, we have conducted an additional experiment by integrating our proposed perceptual guidance into the structure flow-matching sampling process of TRELLIS.2. We evaluate the improvement on an evaluation set of 1,250 objects from Toys4K, the benchmark dataset used in the original TRELLIS.2 paper.
The results demonstrate consistent improvements across all four evaluation metrics. The table reports paired comparison p-values against TRELLIS.2.
| Setting | [email protected] ↑ | p | PSNR ↑ | p | SSIM ↑ | p | LPIPS ↓ | p |
|---|---|---|---|---|---|---|---|---|
| TRELLIS.2 | 0.416 | — | 23.89 | — | 0.9436 | — | 0.0728 | — |
| TRELLIS.2 + Ours | 0.427 | 0.0001 | 24.06 | <0.0001 | 0.9441 | 0.0132 | 0.0702 | <0.0001 |
In particular, our method improves geometric F1 from 0.416 to 0.427, while also improving PSNR, SSIM, and LPIPS. These results are important because they demonstrate that the benefits of perceptual guidance are not restricted to the diffusion-based multi-view generation pipelines originally investigated in OPERA. Instead, the proposed approach can also be extended to flow-matching-based 3D generation, improving both geometric reconstruction and rendering quality.
This experiment provides additional evidence that foundation-model perceptual guidance is complementary to the native generative capabilities of more recent 3D generation models, rather than having its benefits saturated by stronger generative backbones.
Responses to Reviewer upPh's comments
W1. OPERA as a test-time classifier guidance mechanism
We respectfully clarify that OPERA is not simply an application of conventional test-time classifier guidance or an attempt to patch existing 3D reconstruction methods using external 2D priors.
A fundamental challenge in generative single-view 3D reconstruction is that the input image provides only partial observations of an object. Consequently, existing generative approaches can produce plausible but semantically or perceptually inconsistent views, leading to hallucinated structures and errors in the subsequent 3D reconstruction.
Our motivation is to address this limitation by leveraging the rich perceptual knowledge encoded in pretrained foundation models to improve the understanding of the object being reconstructed. Specifically, OPERA introduces GPAM to predict perceptual representations from intermediate diffusion features and uses these predictions to guide multi-view sampling.
Importantly, unlike conventional classifier guidance, which typically relies on gradients from a pretrained classifier with respect to a class-label objective, OPERA exploits foundation-model perceptual representations as guidance signals. Furthermore, GPAM learns to predict these representations during training, eliminating the need to run the external perception models during inference.
Therefore, OPERA is better understood as a perception-guided multi-view generation framework that enhances existing diffusion-based single-view 3D reconstruction pipelines, rather than a conventional classifier-guidance mechanism.
W2. Computational overhead
We would like to clarify that the reported inference time of approximately 4 minutes refers to the entire reconstruction pipeline, rather than the additional computational overhead introduced by OPERA.
Specifically, the baseline multi-view generation process using Era3D/Wonder3D takes approximately 1.8 minutes, while the subsequent 3D reconstruction using Wang et al. (2021) takes approximately 2 minutes. The additional perception-guided sampling introduced by OPERA takes only approximately 10 seconds.
Therefore, OPERA introduces a relatively small computational overhead compared with the overall reconstruction time. Importantly, the pretrained perception models are not required during inference, as GPAM directly predicts the perceptual representations from intermediate diffusion features.
We will clarify this breakdown in the revised manuscript to avoid confusion between the total reconstruction time and the additional overhead of our method.
W3. Improvements on Wonder3D
We acknowledge that the improvement margins vary across the two backbones. As discussed in our response to Reviewer 1, the effectiveness of perceptual guidance is influenced by the capabilities of the underlying generative backbone.
Era3D generally produces higher-quality and higher-resolution multi-view images than Wonder3D, providing richer visual details and more reliable geometric cues for subsequent reconstruction. Consequently, the two backbones have different baseline performance levels and different potential for improvement.
Nevertheless, OPERA is designed as a plug-in framework that can be applied to different diffusion-based multi-view generation backbones without fine-tuning the underlying generative models. The different improvement margins should therefore be interpreted in the context of the respective backbone capabilities, rather than as evidence that OPERA is inherently dependent on Era3D.
W4. Geometric complexity of evaluated objects
We would like to clarify that our evaluation is not limited to the relatively simple objects shown in the main paper. In particular, the 100 objects from OmniObject3D used in our evaluation include objects with more intricate geometric structures and varying levels of shape complexity.
To provide a more comprehensive assessment, we will include additional qualitative visualisations from the OmniObject3D evaluation, showing reconstructed objects from multiple viewpoints. These examples will help demonstrate the behaviour of OPERA on objects with more complex geometric structures and provide a broader view of its reconstruction capabilities.