- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper presents a post-training framework to cover and unify different types of existing 3D generation foundation approaches. Specifically, the authors argue that existing post-training approaches have several gaps. For instance, they are limited in terms of the type of input conditioning (either text or image conditioning), rely on 2D rendering-based rewards, and are sensitive to the reward aggregation step required to aggregate rewards from different viewpoints of a 3D object during the post-training stage. To address these issues and bring all existing 3D generation approaches under a unified umbrella, the paper proposes a two-stage RL post-training framework. In the first stage, the authors develop two main reward pipelines designed specifically for text and image conditioning. In the second stage, the authors select the best existing RL post-training approach (FDFO) from among several existing methods. This is done by applying these post-training methods to a 3D generation baseline and evaluating them using different automatic metrics. The paper also presents a novel tree-structured rollout technique for multi-stage 3D generation methods, such as TRELLIS, which consists of two stages for generating geometry and appearance, respectively. This helps the RL post-training framework jointly assign credits/rewards to each stage of the 3D generation model and update both stages, leading to better results.
Critical strengths
-
The paper is well-written and easy to follow.
-
The paper presents extensive ablations and experiments to demonstrate the effectiveness of the different proposed components.
-
The multi-stage adaptation (Section 5), in which a binary tree-structured rollout is proposed, is particularly novel. It provides a simple but effective way to post-train both the geometry and appearance (texture) backbones of multi-stage 3D generation models, such as TRELLIS. This approach could also be applied beyond this work and represents a valuable idea for the 3D community.
Critical weaknesses
-
In the Introduction, the paper argues that existing post-training approaches rely only on 2D rendering-based rewards in their reward pipelines. While this is a valid point, the same issue also seems to exist in the proposed approach. The only component that takes the actual 3D structure into account is the Plausibility Reward proposed in Equation (2). Even this reward may not properly represent the geometry of highly complex meshes with a large number of vertices and triangles, as the paper samples only 8192 points from the mesh surface, which could be an issue for high-complexity meshes. Moreover, although checking the plausibility of a 3D asset through its 3D point cloud representation may be valid for geometry validation, it does not account for the relationship between the 3D geometry and its texture (appearance). For example, not all existing post-training methods rely solely on 2D renderings of 3D assets. The work in [1], for instance, addresses this issue for 3D texture generation models by proposing 3D-aware rewards that relate the texture to the geometry. Could the authors clarify this point and explain whether there are other components in the proposed pipeline (that I might have missed) that directly address the issue of relying on 2D rendering-based rewards, which the paper also highlights as a limitation of existing methods?
-
While the paper presents a general post-training framework with extensive experiments, and I appreciate the authors’ efforts to unify different approaches for post-training 3D generation, most of the proposed components have already been introduced and are well-established in the literature. For example, the reward components and post-training policy optimization algorithms are all existing methods that are leveraged in this work to unify post-training for 3D generation. This unification is, by itself, a valuable result. However, in my humble opinion, it may not constitute a sufficiently strong contribution for a top-tier conference such as ICLR. The main novel component of this work, which I also appreciated in my comments above, is the proposed binary tree-structured rollouts. However, even this contribution alone may not be sufficiently compelling to support the paper as a standalone research contribution.
-
Regarding the proposed text-to-3D reward, although the paper mentions that existing rewards in the literature are sensitive to the choice of reward aggregation across viewpoints, the proposed approach adopts the same strategy by applying max aggregation, without discussing the potential consequences of this choice. I would expect a major disadvantage of this strategy to be that it does not take into account the information that can be obtained from other views. Although max aggregation selects the best-rendering view, the post-training approach could also learn from less optimal rendering viewpoints and incorporate those signals into the post-training procedure. This is one of the main advantages of algorithms such as DiffusionNFT, whose results are also reported in Table 1 of the paper. Overall, while max aggregation may outperform other aggregation techniques, such as mean or max-mean, as reported in Table 14, I believe that max aggregation alone does not resolve the issue that the paper correctly identifies regarding the sensitivity of existing rewards to the choice of aggregation across viewpoints. Instead, true 3D-aware rewards, such as those proposed in [1], could provide a way to address this aggregation issue by properly relating the appearance of an object to its geometry.
-
In Figures 4 and 5 (and also the figures in appendix and attached supplementary video), which present the main qualitative comparisons, in some cases, the differences between the results of the proposed post-trained models and their pretrained counterparts are not particularly obvious or convincing. For example, in the case of the blue character in Figure 5, almost all generated 3D renderings appear to adhere reasonably well to the reference image. The same observation applies to the dragon character. Similarly, looking at Tables 2 and 3, which present the main quantitative comparisons, the improvements achieved by the proposed post-training approach over the pretrained counterparts do not appear to be particularly significant, especially in the case of TRELLIS in both tables. This seems inconsistent with the paper's claim in the Introduction that the proposed approach achieves strong superiority over the baselines. Furthermore, in the Abstract, the authors argue that existing text-to-3D generation methods achieve only marginal improvements. However, based on the qualitative and quantitative comparisons presented in the paper, the proposed method appears to face a similar issue at least in terms of the improvements over its pretrained counterparts. I believe the authors should better clarify and substantiate these claims.
-
Minor comment: Tables 2 (in the main paper) and 9 (in the appendix) present the same information, but there are also minor discrepancies in the reported numbers. Could the authors clarify the purpose of including two tables for the same experiments and explain the differences in the reported results?
Reference
[1] Zamani, Amirhossein, et al. "End-to-End Fine-Tuning of 3D Texture Generation using Differentiable Rewards." 2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2026.
Questions for authors
-
The proposed tree-structured rollouts are an interesting idea for incorporating both the geometry and texture backbones in a multi-stage 3D generation model. However, I would expect this approach to become computationally expensive as the number of stages in the 3D generation model or the group size in the post-training algorithm increases. Could the authors discuss this potential limitation and provide additional results on the computational cost of the proposed approach?
-
Could the authors more clearly identify and explain the novel components of the proposed framework, distinguishing them from existing methods and techniques adopted from literature?
6 comments
Response to Reviewer 8dLK - Part 2
W2/Q2: Most components are adopted from prior work, and the novel contributions are insufficiently distinguished.
We agree that RL algorithms and the individual reward models are adopted from prior work, and we do not claim these components themselves as novel. However, the contributions of Next3DGen extend beyond assembling existing components. The RL algorithms and individual reward models are modular and can be replaced with compatible alternatives. Our contributions lie in developing a framework that extends reward-based post-training to previously unsupported 3D generation settings, and in systematically validating the corresponding conditioning-specific reward, rendering, and optimization designs. To the best of our knowledge, Next3DGen is the first framework to address reward-based post-training for image-to-3D foundation models. Prior 3D post-training methods support only text-conditioned generation, leaving the distinct challenges of image-to-3D post-training unexplored.
Conditioning-specific reward design. For text-to-3D generation, we develop a principled reward pipeline based on three explicit and complementary criteria. We systematically study reward composition and reward-model selection rather than relying on either a single signal or a complex combination of semantically overlapping rewards, such as the seven-reward combination used in AR3D-R1 [3]. The results in Table 4 and Figure 6 demonstrate the superiority of our reward design over existing text-to-3D reward pipelines. For image-to-3D generation, we develop a distinct reward design that improves unobserved regions while preserving fidelity to the input image and avoids conflicting supervision between alignment and quality rewards.
Conditioning-specific rendering pipelines. For text-to-3D generation, we use viewpoints distributed over an icosahedron to provide broader elevation coverage than prior rendering configurations, an approach whose effectiveness is also validated in Table 12. For image-to-3D generation, we develop a dedicated rendering pipeline that estimates and corrects the unknown reference camera pose and then constructs the corresponding antipodal view, with its efficacy validated in Table 13. Together, these views provide broad azimuthal coverage with minimal overlap, allowing both reward objectives to be optimized harmoniously.
Systematic evaluation of efficient RL algorithms. We evaluate recent RL post-training algorithms for modern large-scale 3D generators and select an efficient algorithm based on this comparison. The contribution is not the RL algorithm itself, but the controlled evaluation and practical characterization of RL algorithms for large-scale 3D post-training.
Multi-stage post-training. As recognized by the reviewer, we introduce tree-structured rollouts that jointly optimize multiple stages of a 3D generator through stage-specific credit assignment. To the best of our knowledge, Next3DGen is also the first reward-based post-training framework to support multi-stage 3D generators.
Therefore, our contribution lies not in the RL algorithms, any individual reward model, or a simple assembly of existing components, but: (1) extending reward-based post-training to image-to-3D foundation models; (2) developing transparent, conditioning-specific reward and rendering pipelines that outperform existing designs; (3) systematically evaluating efficient RL algorithms for large-scale 3D generators; and (4) enabling joint post-training of multi-stage generators through tree-structured rollouts. Together, these contributions establish a unified framework for 3D generation post-training across conditioning modalities and architectures. We will revise the contribution summary to more clearly distinguish these framework-level contributions from adopted components.
Response to Reviewer 8dLK - Part 3
W3: Max aggregation ignores information from less optimal viewpoints and does not resolve sensitivity to view aggregation.
We agree that less optimal viewpoints may contain useful information. However, max aggregation does not imply that only the selected view matters. Our contribution is to identify an empirically effective way to leverage mature 2D reward models for 3D post-training while complementing their missing information with object-level 3D evaluation.
Our 2D rewards assess the aesthetic quality and prompt alignment of a 3D object through rendered views. However, these properties cannot be represented equally well across all viewpoints. Side and rear views often have lower reward upper bounds because they contain fewer distinctive semantic features and are less visually informative than frontal views. Naively optimizing rewards across all views can therefore encourage the model to duplicate high-reward features across other viewpoints, resulting in reward hacking and Janus-like artifacts. To leverage the capabilities of well-established 2D reward models while mitigating reward hacking, we use max aggregation, whose effectiveness is demonstrated by our experiments. This choice is further supported by our comparison with prior approaches: Nabla-R2D3 uses mean aggregation, while AR3D-R1 jointly supplies multiple views to a VLM. Both produce Janus-like artifacts, such as a teapot with duplicated spouts, as shown in Figure 6.
We agree that lower-scoring viewpoints may contain useful information and that rendering-based rewards alone are insufficient. Even with additional views, some regions may remain occluded, and certain structural defects may not be reliably captured by 2D reward models. We therefore complement the view-based rewards with an object-level Plausibility Reward based on a 3D multimodal model. This reward evaluates the complete XYZRGB point-cloud representation for global plausibility and semantic consistency with the input, including regions not selected by max aggregation. Our ablation in Table 5 shows that combining object-level plausibility evaluation with view-based rewards improves performance.
Therefore, max aggregation applies 2D aesthetic and alignment rewards to the view where these properties can be assessed most faithfully. The Plausibility Reward evaluates the object as a whole to assess its global structural and semantic plausibility. It is particularly suitable for evaluating views with inherently lower aesthetic-reward upper bounds, where plausibility, rather than visual appeal, is the relevant objective.
Response to Reviewer 8dLK - Part 1
We thank the reviewer for the detailed feedback and for recognizing the clarity, extensive experiments, and novel tree-structured rollout of our work. However, the novelty assessment focuses primarily on the tree-structured rollout and overlooks our other contributions. In addition, the assessment of our empirical improvements focuses on selected examples without fully considering the broader quantitative and qualitative evidence. Below, we address these points with direct references to the manuscript and clarify details that were insufficiently explicit.
W1: Reliance on 2D rendering-based rewards, limited representation of complex geometry, and geometry-appearance consistency.
We respectfully clarify that the paper does not consider the use of 2D rewards itself a limitation. Instead, our observation is that prior pipelines either rely exclusively on rendering-based signals, as in Nabla-R2D3 [2], or combine multiple signals without systematically examining their individual roles, as in AR3D-R1 [3]. Next3DGen therefore combines complementary view-based and object-level signals through a systematic and empirically validated design.
To answer the reviewer’s question, Plausibility Reward is the component that directly evaluates the 3D object. It complements the view-based aesthetic and alignment rewards by assessing global geometry and appearance plausibility. We apologize that the manuscript did not explicitly state that the reward uses an XYZRGB colored point cloud, rather than an XYZ-only point cloud. Each sampled surface point contains its 3D coordinates and corresponding RGB appearance, both of which PointLLM [4] can understand. The reward therefore evaluates appearance in spatial association with the generated 3D surface. We will revise Section 4.1 and Equation (2) to explicitly define P(O_i) as a normalized XYZRGB point cloud.
Regarding the use of 8,192 points, we acknowledge that fixed-size sampling may omit fine geometric or textural details in highly complex objects. However, the Plausibility Reward is intended to assess global object plausibility and semantic consistency with the prompt, rather than reconstruct every vertex, triangle, or texture texel. For this purpose, 8,192 XYZRGB surface samples provide broad object coverage while remaining compatible with the fixed-size input of the 3D multimodal model. While evaluating highly complex mesh geometry is important, we leave the development of such a specialized reward for future work.
We further clarify that [1] addresses a related but different setting: texture generation on a given, fixed mesh. Its geometry-aware rewards assess whether the generated texture follows precomputed properties, such as curvature directions and symmetry planes. However, they assume a fixed, valid mesh and therefore neither evaluate the underlying geometry nor explicitly assess semantic alignment between the generated texture and the conditioning prompt. Meanwhile, our Plausibility Reward evaluates the plausibility of both geometry and appearance, as well as their semantic alignment with the input prompt.
Moreover, [1] relies on a differentiable UV-based texture-synthesis pipeline and is therefore not directly applicable to representations without separate UV maps, such as 3DGS. In contrast, our Plausibility Reward operates on an XYZRGB point cloud obtainable from heterogeneous representations, including meshes and 3DGS, enabling consistent object-level evaluation across the generators considered in Next3DGen. We appreciate the complementary contribution of [1] and will discuss it in the revised manuscript.