acceptodds
Under review as a conference paper at ICLR 2027

Hybrid3D-RL: Hybrid-Space Reinforcement Learning for High-Fidelity 3D Generation

Abstract

Reinforcement-learning post-training has become a standard approach for improving image and video generation, yet its extension to high-fidelity 3D generation remains challenging. Reward signals computed only from rendered images cannot fully characterize geometric quality, while direct supervision with explicit 3D metrics is ill-posed for single-image generation because unobserved regions have no uniquely determined geometry. We introduce Hybrid3D-RL, a hybrid-space post-training framework for VoxSet-based 3D generation. Its first component, the Implicit Latent Reward Model (I-LRM), reuses the geometric prior of a pre-trained generator to assess mesh quality directly from latent representations. We train this model without human annotation using preference pairs constructed through multi-granularity VAE re-encoding. Its second component, HRPO (Hybrid-space Ranking Preference Optimization), separates online rollout from reward evaluation in a modular training pipeline. HRPO combines an explicit appearance reward measured from a camera-corrected rendering with the implicit latent reward, and optimizes stochastic flow trajectories using relative advantages and reference-policy regularization. Experiments demonstrate reliable latent quality discrimination and consistent improvements in image-to-3D generation across quantitative and qualitative evaluations.

Primary area:
generative models
Keywords:
3D Generation, Reinforment Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_73YLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies RL post-training for a two-stage image-to-3D generator, focusing on its VoxSet-based refinement model. It introduces an Implicit Latent Reward Model (I-LRM) that reuses the first eight blocks of a pretrained DiT together with a learned reward head. Preference labels are constructed automatically by ranking VAE reconstructions obtained at different latent granularities. The proposed HRPO pipeline combines this latent score with ID-Sim and DINOv3 similarity scores computed between the input image and a rendering from the corrected conditioning-camera view. It then applies Flow-GRPO-style updates with regularization against a frozen reference policy. The evaluation includes held-out synthetic preference pairs, three image-shape alignment metrics on 90 test images, reward-component ablations, and user studies. The full system improves over the pretrained model on all three reported alignment metrics.

Critical strengths

1.The reward design addresses a genuine ambiguity in single-image 3D generation. By treating observed-view consistency and learned whole-shape preferences as complementary sources of supervision, the method avoids imposing a single reference completion on unobserved regions.

2.The full method improves Uni3D, OpenShape, and ULIP-2 scores over the pretrained model and outperforms both single-reward variants on all three metrics.

Critical weaknesses

1.Although the paper describes camera correction using recorded camera parameters, how does the explicit reward handle residual pose misalignment between the generated mesh and the reference image? Moreover, this reward evaluates the rendered image as a whole and may not direct optimization toward local defects. For example, if an owl mesh has poorly reconstructed toes, the reward could increase by further improving the already reasonable feathers without correcting the toes. This could lead to a mismatch with human judgments.

2.The central claim about the learned reward has not been sufficiently validated beyond the process used to generate its labels. Section 3.2 defines the preferred samples through VAE reconstruction granularity, while Section 4.2 evaluates on held-out pairs produced by the same procedure. The reported accuracy of 88.12% therefore establishes agreement with this proxy labeling rule, but does not necessarily demonstrate reliable quality rankings of actual generator outputs. In addition, the human evaluation protocol needs further clarification, including the number of participants and the blinding procedures, so that its fairness and reliability can be assessed.

Questions for authors

1.How does the explicit reward handle residual pose misalignment between the generated mesh and the reference image, even after camera correction? Since it scores the rendered image as a whole, how does it guide optimization toward local defects, such as poorly generated fingers?

2.Were the source meshes split into training and held-out sets before VAE reconstruction and pair construction, ensuring that the 1,000 held-out pairs contain no source meshes used to train I-LRM? What granularity settings were used, and how were token counts and voxel support handled at scoring time? Can you evaluate I-LRM's agreement with independent human judgments on actual policy-generated pairs while controlling for granularity-related cues, and assess visible-view consistency separately from whole-shape quality?

3.Clarify the provenance of the 90 test images and their separation from the training data. For Figures 8 and 9, provide the complete evaluation protocols and the numbers of inputs and judgments underlying the reported results, including the number of raters, task assignment, presentation order, blinding procedures, and views shown to participants.

4.Equation (9) follows Flow-GRPO, whereas Table 5 compares online hybrid-reward training with offline DPO trained on granularity-derived preferences without the hybrid reward. How much of the improvement can be attributed to the optimization algorithm itself, rather than to differences in feedback or online data collection?

Reviewer_9Ex5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Hybrid3D-RL, a reinforcement learning post-training framework for improving high-fidelity 3D generation. The proposed reward consists of two components. The Implicit Latent Reward Model (I-LRM) evaluates geometric quality directly in the VAE latent space without decoding the generated mesh. It is trained on automatically constructed preference pairs obtained through multi-granularity VAE re-encoding, avoiding expensive human annotations. The Explicit Appearance Reward evaluates the consistency between the generated mesh and the conditioning image. These two rewards are combined to guide online flow-based policy optimization.

Critical strengths
  1. Novel latent reward design. The proposed I-LRM is an effective approach to evaluating 3D geometric quality directly in latent space, avoiding mesh decoding and rendering.

  2. Preference data construction. Generating preference pairs through multi-granularity VAE re-encoding is a practical way to obtain large-scale training data without manual quality annotations.

  3. Impressive qualitative improvements. The visual comparisons demonstrate noticeable improvements in geometric details, structural completeness, and surface quality.

Critical weaknesses
  1. Insufficient validation of I-LRM generalization. In Section 4.2, I-LRM is evaluated on held-out pairs from the same Mesh Quality Comparison Data construction pipeline used for training. Therefore, the reported ranking accuracy mainly demonstrates its ability to distinguish VAE reconstructions at different compression granularities, rather than its ability to assess the quality of meshes generated by actual 3D generative models. I would strongly encourage the authors to evaluate I-LRM on meshes generated by different models and measure its ranking agreement with human judgments of geometric quality. This would provide more convincing evidence of the reward model's generalization.

  2. Unclear end-to-end efficiency benefit of latent reward evaluation. A key motivation of I-LRM is to evaluate geometric quality without decoding meshes. However, the complete HRPO pipeline still requires mesh decoding and rendering for the Explicit Appearance Reward on each candidate. Therefore, the extent to which decoding-free latent evaluation reduces the overall RL training cost is unclear.

Questions for authors
  1. Real-world generalization. How does the post-trained model perform on real-world captures?

  2. Computational cost. What are the total GPU hours and computational resources required for I-LRM training and HRPO post-training?

  3. Noise-level training for I-LRM. I-LRM is trained on latents with different noise levels, but HRPO evaluates only clean terminal latents (τ=0\tau=0). What is the motivation for training across the entire noise schedule?

Reviewer_TJtfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper post-trains the Stage-2 flow model of an image-to-3D pipeline. I-LRM reuses pretrained DiT features and learns a latent reward from automatically constructed preferences between VAE reconstructions at different granularities. HRPO combines this terminal latent reward with camera-aligned rendered-image rewards through Flow-GRPO-based updates to improve geometry and image consistency.

Critical strengths

The reward design is well motivated. The image reward checks only the visible view, and the learned prior covers the full shape, so unseen regions are not forced toward one fixed completion. Preference pairs are built automatically from VAE reconstructions, so they need no manual labels and can scale. The paper compares against the same pretrained model and ablates each reward component. All three automatic metrics improve, and some qualitative examples show sharper local detail.

Critical weaknesses
  1. The reward model is not validated on generated samples. Tables 1 and 3 test I-LRM on pairs built by the same rule used for training. This shows it can separate these synthetic pairs, not that it ranks the policy's own outputs correctly. Since HRPO scores only clean final latents, the relevant test is agreement with human judgments on generated samples. The model may also rely on differences in token count or voxel support, which may not carry over to ranking candidates within an online group. This is not controlled.

  2. LPO [1] already uses pretrained diffusion features as a noise-aware latent reward model, including on DiT/flow models and with a GRPO variant. The paper does not discuss it. By the paper's own description, HRPO follows Flow-GRPO. The paper should state what is new (the automatic 3D preference pairs, the 3D-specific changes or the hybrid reward) and support each claim with evidence. PRFL [2] is also closely related: its reward model uses the first eight DiT blocks, a learnable query for aggregation and a scalar reward head.

  3. Better full-shape geometry is not well-proved. Against the same pretrained model, Uni3D, OpenShape and ULIP-2 improve by 0.0073, 0.0073 and 0.0055 on 90 images. No paired confidence intervals or seed variation are given. These metrics measure image-pointcloud feature alignment. They do not directly show better local geometry, unseen surfaces or structural completeness. This makes the user studies important, but the paper does not report participant or object counts, the full protocol, randomization method or uncertainties.

  4. The DPO comparison does not isolate the optimizer. In Table 5, DPO trains offline on VAE-reconstruction pairs, while HRPO trains online with hybrid rewards. Both sampling and supervision change, so the table compares whole setups, not optimizers. HRPO's additions should be tested with the same backbone, rewards and budget, matching preference supervision if possible.

[1]. Tao Zhang et al. Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization (LPO). NeurIPS 2025.

[2]. Xiaoyue Mi et al. Video Generation Models Are Good Latent Reward Models (PRFL). CVPR 2026 (Findings).

Questions for authors
  1. At the clean endpoint (\tau = 0), how well does I-LRM agree with blinded human judgments on policy-generated candidates? Please use objects not seen in training, and control for token count and voxel support.
  2. What is new beyond LPO and standard Flow-GRPO, and how does I-LRM differ from PRFL? Can each addition be isolated, for example with Flow-GRPO under the same rewards and budget, or with DPO on preferences from the same scorer?
  3. For both user studies, please give the full protocol, number of votes/participants and uncertainties.
  4. For the test set, please give per-image paired statistics and results over several seeds.
  5. Please also check the ULIP-2 numbers in Table 2, which clearly split into two distinct groups, and confirm that every method was evaluated in the same way.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.