acceptodds
Under review as a conference paper at ICLR 2027

V23D: LEVERAGING VIDEO DIFFUSION PRIORS FOR HIGH-FIDELITY MULTI-VIEW 3D GENERATION

Abstract

Single-image 3D generation must infer self-occluded regions from incomplete evidence. Orbit videos offer complementary views, but a shared keyframe subset can omit cues essential to individual regions. We present V23D, a two-phase videoprior-to-3D framework that combines geometry-aware orbit generation with regionspecific conditioning. In Phase I, we adapt Wan2.2 with noise-decoupled LoRA and a Geometry-aware Orbit Adapter (GOA) to generate 49-frame turntable priors, reducing FVD from 205.8 to 176.8. In Phase II, we introduce a Regional Source Adapter (RSA) that connects the complete orbit to coarse-to-fine 3D generation. The sparse-structure stage records regional source-frame indices and their original attention masses; the shape stage uses independent current queries to re-read the referenced patches, while receiving sparse support separately. This design preserves source access across generation stages rather than passing only fixed fused features. On HY3D-Bench-test, V23D achieves 0.2418 ULIP-I and 0.3827 Uni3D-I, outperforming the evaluated baselines. Compared with the fourview OFA configuration, RSA improves F-score@1% from 57.4% to 61.1%, and achieves 62.7% on Toys4K in cross-dataset evaluation. Matched full-orbit comparisons further favor RSA over all-frame aggregation, instantaneous regional routing, and fixed-feature transfer, supporting the value of cross-stage source preservation and state-dependent patch re-reading.

Primary area:
generative models
Keywords:
video gen, 3d gen, SFT

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_o9yKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents V23D, a framework for single-image 3D generation using video diffusion priors. It combines a Geometry-aware Orbit Adapter for orbit video generation with a Regional Source Adapter for incorporating multi-view information into 3D generation. Experiments demonstrate improvements over the evaluated baselines.

Critical strengths
  1. The paper is clearly written and easy to follow.
  2. The paper addresses the ambiguity of occluded geometry in single-image 3D generation and the uneven distribution of useful information across views.
  3. The motivation is clear, and the overall framework is logically coherent.
  4. The results provide empirical support for the effectiveness of the proposed framework.
Critical weaknesses
  1. There is a gap between RSA training and inference. RSA is trained using GT-rendered 24-view images, but inference uses 49 generated video frames. Generated videos may contain artifacts and cross-view inconsistencies. The authors should clarify this setting and analyze its impact on performance.
  2. The novelty of GOA is unclear. Equation (2) appears to use a conventional residual conditioning mechanism, and the improvement over the LoRA baseline in Table 3 is limited. More ablations are needed to verify the contribution of GOA and its auxilary losses.
  3. The novelty and generalizability of RSA require further validation. Related ideas, such as token-wise view routing and attention-based multi-view feature fusion, have been explored in prior work. The authors should better clarify how RSA differs from these methods, particularly in its cross-stage source-frame selection and patch re-reading mechanism. Moreover, RSA is evaluated only within TRELLIS.2, and its applicability to other 3D generation architectures remains unclear.
  4. The comparison with related video generation methods is incomplete. Since V23D generates multi-view-consistent orbital videos from a single image, comparisons with methods such as SV3D and Hi3D would help better demonstrate its advantages, particularly for downstream 3D generation.
Questions for authors

Please see the weaknesses.

2 comments

Thanks for the detailed rebuttal. My concerns are largely addressed. I am happy to maintain my positive rating. The authors should incorporate the relevant explanations and clarifications provided in their response into the revised manuscript to improve its completeness and clarity.

Thank you sincerely for your time and your review comments; we have completed the final revisions to the main text.

Reviewer_MDacOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

V23D uses an adapted Wan2.2 model to generate a 49-frame video around an object, then passes video information to TRELLIS.2 through RSA. The coarse stage selects reference views for each region, and the refinement stage reads details from those views. Results on two datasets and targeted comparisons support the design.

Critical strengths
  1. RSA separates choosing reference views from reading their details. This fits the need to first generate a rough structure and then refine it (§3.3).
  2. Table 5 compares alternatives such as using all frames, selecting views again, and passing fixed features between stages. These comparisons help explain the role of each design choice. Table 2 also includes cross-dataset evaluation on Toys4K.
Critical weaknesses
  1. The advance appears limited, and fixing the source views needs stronger justification. Relative to the regional routing and cross-stage reuse discussed in §2, RSA mainly differs by saving coarse-stage view selections and re-reading their features during refinement. Table 5 supports this design, but an important view missed at the coarse stage cannot be added later through RSA. This still limits its ability to recover missing local information (§3.3).
  2. Consistent information from several views may still receive less weight. Equations 4–6 keep the original weights of the selected views without renormalizing them. When several similar views share the attention, the video's influence on refinement can therefore weaken even if those views agree. The paper should explain why spreading attention should mean making less use of the video information.
  3. Similar appearance does not guarantee a match to the same surface. RSA mainly matches by content similarity, with zero spatial bias. Camera information records the source but does not directly constrain matching (§3.3; Appendix A.1). For repeated parts or conflicting generated views, the method mainly relies on learned features and the 3D prior, without a dedicated way to resolve these geometric conflicts.
Questions for authors
  1. What is RSA's main methodological advance over regional routing and cross-stage reuse? Why should fixed coarse-stage view selections meet the needs of later refinement?
  2. If several views provide redundant but consistent information, why should spreading attention across them reduce the video's influence?
  3. How does RSA handle repeated parts or conflicting views? Please enlarge the key comparisons in Figure 6 so readers can see which geometric problems it improves.
Reviewer_ujEmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies 3D generation from a single image. It first adapts Wan2.2 with LoRA and GOA to generate a 49-frame orbit video. Then RSA uses these views to guide TRELLIS.2. The sparse-structure stage selects source frames and records their attention masses for each region. The shape stage keeps these choices and reads the image patches again using its current queries. The paper reports improvements on HY3D-Bench and Toys4K, with several ablation studies.

Critical strengths

RSA has a clear technical idea, and Table 5 provides useful controls. Its F-score improves from 59.5% with current-state routing and 59.0% with fixed-content transfer to 61.1%. These comparisons support the benefit of source transfer and patch re-reading. The fixed-content control also shares RSA's sparse structure and support.

The improvement is consistent across two datasets. Compared with OFA, RSA improves F-score from 57.4% to 61.1% on HY3D-Bench and from 60.1% to 62.7% on Toys4K. This gives useful evidence beyond the visual examples.

Critical weaknesses

Some details needed for reproduction are still unclear. These include the RSA training schedule for II-A/II-B, the generated/rendered view mixture, and the mapping from shape coordinates to coarse regions. Appendix B.1 mainly describes OFA training. The paper also does not report seed counts, failure counts, or uncertainty for the modest gain over the strongest full-orbit control. These issues limit confidence in the exact improvement, but the existing ablations still support the main design.

Questions for authors

Pls clarify the RSA training settings, generated/rendered view mixture, source-cache construction, and coordinate mapping. Which settings differ from the OFA recipe in Appendix B.1?

Were the official HY3D-Bench splits used in full, and which Toys4K subset was evaluated? Please clarify overlap checks with Phase-I training assets, geometry evaluation settings, and attempted/completed sample counts.

How consistent is the Table 5 improvement across assets and seeds? If available, please provide paired uncertainty for E4 versus E2/E3. Also, which generated views are given to Hunyuan3D-2MV in Table 2?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewers,

Thank you again for your time and thoughtful feedback. As the rebuttal period closes today, we wanted to check whether any points in our responses remain unclear or would benefit from further clarification. We would be happy to address any remaining questions.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.