acceptodds
Under review as a conference paper at ICLR 2027

Beyond RGB: Polarization-Guided Surface-Aware 3D Generation

Abstract

Single-view 3D generation relies on strong RGB generative priors. However, 3D inference from RGB appearance alone remains geometrically ambiguous: a model may produce a visually plausible object while failing to recover local surface orientation and fine-scale geometry, particularly in weakly textured or highly reflective regions. Unlike RGB appearance, the angle of linear polarization (AoLP) is related to surface orientation and provides a complementary cue to local geometry. We propose polarization-guided surface-aware 3D generation, which retains the prior of an RGB-conditioned generator while using polarization to refine its modeling of local surfaces. A surface-aware feature fusion module jointly encodes semantic and appearance information from RGB and orientation cues from AoLP, yielding a normal-aware condition for the denoising process of a pretrained 3D generator. To support training and evaluation, we also construct OBJMV3D with 60,000 paired groups of RGB images, AoLP observations, surface normals, and high-quality 3D assets. Experiments on this benchmark show that polarization-derived surface cues improve fine-scale geometry while preserving overall generation quality, with more pronounced geometric benefits where RGB evidence is limited by reflectance or weak texture.

Primary area:
generative models
Keywords:
3Dgeneration Multimodal Polarization

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_wE5LOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes polarization-guided single-view 3D generation, which uses the angle of linear polarization (AoLP) as a complementary surface-orientation cue to an RGB image. A surface encoder fuses frozen DINOv3 RGB features with a trainable AoLP U-Net branch under normal supervision. The resulting surface tokens condition a pretrained Step1X-3D generator, which is adapted via LoRA with a latent consistency loss and a multi-view rendered-normal loss. The authors also build OBJMV3D, a synthetic dataset of 60K Objaverse assets with rendered RGB, AoLP, normals, and text, and evaluate on 100 held-out synthetic objects.

Critical strengths

None

Critical weaknesses
  • Limited technical insight. Feeding AoLP into the network as an additional input alongside RGB and other conditions is a common and fairly straightforward practice in the community. The paper does not analyze how AoLP orientation evidence leads to better 3D results. For example, it never examines how the AoLP branch resolves the pi-ambiguity and the diffuse–specular ambiguity it describes in Sec. 3.2, or which regions benefit from it. The authors themselves acknowledge (Appendix A) that there is no comparison against normals predicted from RGB by an external estimator. It is therefore unclear whether the gain comes from polarization specifically or simply from normal conditioning.
  • The value of the OBJMV3D contribution is limited and hard to assess. OBJMV3D is built directly from existing Objaverse assets, and its curation and cleaning are described only briefly and qualitatively (Sec. 4). The paper does not describe how materials were assigned for polarimetric rendering, which directly determines whether the synthetic AoLP is physically meaningful. It does not report the distribution of reflective or weakly textured objects, even though these are the cases the paper targets. With the current information, the quality of the dataset cannot be judged, which limits this contribution.
  • The experiments do not demonstrate superiority over modern foundation models. The method is only demonstrated on toy examples with simple structures, such as the skull, cube, penguin, and bottle in Fig. 5, all of which are synthetic renderings. Quantitatively, the model is worse than its own backbone Step1X-3D in Chamfer Distance (0.0089 vs. 0.0064, Table 1). Its advantage is reported only in CLIP-SIM, which measures semantic consistency rather than geometric accuracy. Besides, no real polarimetric captures are evaluated. In the era of large foundation models, these results cannot show that the proposed method offers a clear advantage.
Questions for authors
  1. Please analyze how AoLP contributes beyond simple input fusion. For instance, show per-region error on reflective or texture-less areas. Also add the missing control that replaces the AoLP branch with normals predicted from RGB by an off-the-shelf estimator.
  2. Please evaluate on more complex objects and on real polarimetric captures. Also explain why the model underperforms its own backbone Step1X-3D in Chamfer Distance, and report geometry-focused metrics (e.g., normal consistency) in addition to CLIP-SIM.
Reviewer_gb18OpenReview ↗
Rating
2(−2)
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a single-view 3D generation framework that incorporates Angle of Linear Polarization (AoLP) alongside RGB images to provide additional local surface orientation cues. The method first introduces an RGB–AoLP Surface Encoder to extract normal-aware features, which are then injected into a pretrained Step1X-3D generator through LoRA adaptation. During fine-tuning, the authors introduce a Multi-view Rendered Normal Loss in addition to latent consistency supervision to improve the geometric quality of generated meshes.

Critical strengths
  1. Well-motivated use of additional geometric information. Incorporating polarization cues to address the geometric ambiguity of RGB-only 3D generation is an interesting and reasonable direction, particularly for weakly textured and reflective surfaces.

  2. Reasonable integration with pretrained models. The surface-aware conditioning and LoRA adaptation provide a practical way to incorporate additional geometric information while largely preserving the capabilities of the pretrained 3D generator.

Critical weaknesses
  1. Limited practical applicability and lack of real-world validation. The method requires an additional AoLP observation, which is not available for most images and requires specialized capture hardware or a polarization capture procedure. This substantially limits its applicability compared with RGB-only 3D generation. More importantly, all experiments use synthetic polarization data with perfectly aligned RGB–AoLP pairs. The paper provides no evidence that the method generalizes to real polarization measurements.

  2. Missing important baselines and insufficient geometric evaluation. The paper omits several recent 3D generation methods, including TRELLIS.2, Hunyuan3D 2.1, Direct3D-S2, and UltraShape 1.0. More importantly, the proposed method achieves a worse Chamfer Distance than its own pretrained backbone, Step1X-3D.

  3. Insufficient evidence for polarization-specific improvements. The authors motivate their approach by emphasizing weakly textured and highly reflective surfaces. Although some reflective objects are shown, the evaluation does not specifically measure improvements on these challenging regions.

  4. Presentation and clarity issues. In Figure 2, the GT 3D Mesh is incorrectly shown passing through a "Mesh Decoder" to obtain the GT 3D Latent, whereas this should be a Mesh Encoder.

trellis2: Native and compact structured latents for 3d generation.

hunyuan3d 2.1: From images to high-fidelity 3d assets with production-ready pbr material.

Direct3D-S2: Gigascale 3d generation made easy with spatial sparse attention.

Ultrashape 1.0: High-fidelity 3d shape generation via scalable geometric refinement

Questions for authors
  1. Surface normal prediction accuracy. How accurate is the proposed RGB–AoLP Surface Encoder compared with existing off-the-shelf monocular normal estimation methods? The paper does not report quantitative normal prediction accuracy. If comparable normals can already be estimated from RGB, what additional benefit does AoLP provide? I strongly encourage the authors to include an RGB-only normal conditioning baseline under the same generator training setup.
Reviewer_2tfaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces polarization information into single-view 3D generation to improve surface geometry. It combines RGB and polarization features with surface-aware supervision and introduces a synthetic multimodal dataset.

Critical strengths
  1. Incorporating polarization information into generative 3D reconstruction is an interesting and potentially useful direction.
  2. The proposed dataset and component ablations provide useful experimental support.
Critical weaknesses
  1. It is difficult to distinguish the benefits of polarization information from those of additional surface supervision.
  2. The method does not consistently outperform the original baseline in Chamfer Distance, which weakens the geometric improvement claim.
Questions for authors
  1. Could the authors compare polarization conditioning with RGB-predicted normal conditioning?
  2. Could additional surface-normal metrics be reported to better demonstrate improvements in local geometry?
Reviewer_sNEiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2(−1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies polarization-guided 3D generation conditioned on an RGB image, a pixel-aligned angle-of-linear-polarization (AoLP) map, and text. The method trains a surface-aware encoder with normal supervision, injects its features into a pretrained Step1X-3D backbone, and adds a rendered-normal objective during adaptation. The paper also introduces OBJMV3D, a synthetic dataset of 60,000 objects and 360,000 view tuples. The approach is promising, but the evaluation does not yet establish what polarization contributes or substantiate the claimed improvements in local surface geometry.

Critical strengths

The paper addresses a relevant limitation of image-conditioned 3D generation: recovering surface details that RGB appearance does not constrain well. Combining polarization-sensitive features with a strong generative prior is a useful direction, and the two-stage training strategy is clearly motivated. The dataset and qualitative examples provide a starting point for further investigation. The method achieves the highest CLIP similarity among the reported comparisons, while ablations that remove surface conditioning and normal-rendering supervision support the usefulness of the combined design. The appendix also openly acknowledges important limitations of the evaluation.

Critical weaknesses

1:The evidence for improved surface geometry is not yet convincing. Table 1 reports a Chamfer Distance of 0.0089 for the proposed method and 0.0064 for Step1X-3D. A higher global distance can coexist with better local detail, but neither CLIP similarity nor selected visualizations directly measures that improvement. Since local surface recovery is central to the paper, the authors should report normal consistency, angular error, and metrics sensitive to fine geometric detail. Results broken down by material and surface type would help show where polarization helps and whether local gains come at the expense of overall shape accuracy.

2:The ablations do not isolate the contribution of polarization. Removing surface conditioning also removes the additional feature pathway and its normal-supervised representation. An RGB-only surface encoder with matched architecture, supervision, and adaptation budget is essential. A control using RGB-predicted normals would also be informative. The appendix acknowledges this limitation, but the current evidence cannot attribute the gains specifically to polarization.

3:Treatment of polarization ambiguity needs stronger validation. AoLP is periodic and becomes unreliable when polarization is weak. Its relationship to orientation also differs between diffuse and specular components. The authors should explain how AoLP is encoded and test sensitivity to noise, material variation, and misalignment between RGB and AoLP.

4:Evaluation on 100 synthetic objects rendered under the data-generation protocol provides limited evidence of generalization. Object-level separation should be complemented by tests under different illumination and material distributions, or a small real-capture set, to better support the method's practical motivation.

5:The dataset and conditioning protocols need clarification. The test data are separate from the adaptation data, but possible overlap between the Objaverse-derived test assets and the pretrained backbone's training data remains unclear. The authors should document overlap and near-duplicate checks where possible and state what they cannot verify. Training captions are generated from six views using Qwen2.5VL, whereas inference captions use a single RGB image and Spark. The authors should examine the effect of this change in conditioning.

6:Several details are needed to reproduce the training objective. The authors should explain how gradients pass through latent-to-mesh decoding and normal rendering, including mesh extraction, visibility handling, and foreground masking. The timestep-dependent reconstruction mapping in Eq. (4) also needs clarification. Loss weights, LoRA configuration, training duration, and batch size should be specified. These details are needed to identify which components receive geometry supervision, reproduce the optimization, and assess the added computational cost relative to adapting the backbone without this objective.

7:The reported geometry and appearance metrics favor different methods, so the comparison protocol needs tighter standardization. The authors should report surface-sampling and alignment settings for Chamfer Distance, conditioning and adaptation budgets, and uncertainty across stochastic generations. Qualitative comparisons should also use matched rendering poses, as acknowledged in the appendix.

Questions for authors

Can the authors quantify the claimed local surface improvements and explain how they coexist with the higher Chamfer Distance relative to Step1X-3D? Would the gains hold against an RGB-only surface encoder trained with identical normal supervision and adaptation resources? Please clarify the differentiability of the normal-rendering objective, potential overlap with the backbone's pretraining data, and the effect of changing the caption-generation protocol. How robust is the method to polarization noise, misalignment, and unseen materials or illumination?

1 comment

Thank you for the author's detailed response and supplementary experiments.

Solved question: Compared with the LoRA baseline of only RGB, it better demonstrates the unique value of polarization information; The newly added indicators of surface distance and normal angle also address my concerns about local geometric quality.

There are still limitations: Although the new results have improved, compared to the original Step1X-3D, the Chamfer Distance is higher, which is still a trade-off. The generalization and robustness of this method in scenarios with different real materials, lighting conditions, and RGB AoLP registration deviations still require more quantitative experimental verification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.