acceptodds
Withdrawn from ICLR 2027

Efficient Post-Training for 3D-Consistent Video Generation via Optimal Transport Reward Balancing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning (RL) can improve the 3D awareness of video diffusion models without architectural modifications. However, existing methods face three challenges: reconstruction rewards favor limited camera motion, heterogeneous reward distributions cause imbalanced optimization, and policy updates over many denoising steps are computationally expensive. We propose 3D-Balanced-R1 to address these issues. First, a visibility score measures inter-frame overlap and reweights the reconstruction reward according to reconstruction difficulty. Sec- ond, optimal transport aligns different rewards to a common Gaussian distribu- tion, reducing domination by individual rewards or outliers. Third, we update the policy at only two denoising steps, using an early step for camera control and a late step for video quality and 3D consistency. On Wan2.1-1.3B, 3D-Balanced- R1 achieves a 3.53× training speedup over World-R1. With the same number of training epochs and comparable video quality, 3D-Balanced-R1 improves the pre- trained model’s 3D score by 9.6%, compared with 6.3% for World-R1. Under a similar training-time budget, the gain increases to 14.3%, or 2.3× that of World- R1. Similar improvements are also observed on the larger Wan2.2-5B model.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video diffusion, 3D, RL

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Pm8LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes 3D-Balanced-R1 for video generation rl. Beyond World-R1, it mainly introduces i) visibility-based reconstruction reward re-weighting; ii) reward renorm based on optimal transport and iii) early-late two stage policy updates. Experiments on two wan dit backbones show improved 3d score and camera following, and better training efficiency.

Critical strengths
  1. The proposed approaches are intuitive and interpretable, with experiment results supporting the effectiveness.
  2. The comparison with LTX-2.5-22B is a welcome addition.
  3. The presentation, including the figure visualization and math formulation, is concise and clear, making it easy for readers to follow.
Critical weaknesses
  1. The methodological contribution is somewhat incremental. World-R1 already discusses the near-static hacking problem and some mitigation approaches in Appendix D.7. Therefore, the main contribution here is the proposed visibility correction rather than identifying the issue. In addition, given that World-R1 has discussed the mitigations, the authors are suggested to present the potential issues with World-R1’s method and explain why the proposed modeling of S_vis is superior.
  2. The videos in the supplementary show visual trade-offs that are not sufficiently discussed. In “A steampunk city square,” the proposed method successfully follows “Camera orbit right,” but the requested “richly printed poster illustration” style is less pronounced than in the baseline. Plus, in “silk and tile” and “rail terminal”, the visual quality also appears noticeably worse. We know that rl can lead to similar de-stylization issues, but ‘ours’ appears more severe than World-R1.
  3. The early-late ablation compares the proposed two-step update with a single-step baseline, which may be unfair since the number of steps differs. It would be more appropriate to compare {early, late} v.s. {random, random} under the same two-step sampling budget (here random mean randomly sample a step in all possible timesteps)
  4. (minor) The timestep notation is inconsistent: Section 3 defines t in[0,1], whereas Section 4.3 uses t=T-s as an integer transition index.
Questions for authors
  1. Can S_vis distinguish the reduced overlap caused by camera movement from that caused by geometric inconsistency (for example, a relatively large object moves or just disappears inexplicably)? If inconsistent frames also receive a visibility bonus, could this weaken the reconstruction reward’s ability to penalize them?
Reviewer_nz6aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces 3D-Balanced-R1, an RL post-training framework for improving 3D-aware video generation. The method combines three components: visibility-dependent reweighting of reconstruction rewards, component-wise Gaussian quantile normalization interpreted as one-dimensional optimal transport, and policy updates at two sampled denoising transitions, one early and one late. Experiments on Wan2.1-1.3B and Wan2.2-5B report improved camera-trajectory alignment and aggregate 3D scores. The method also achieves a reported 3.53× training speedup over the evaluated World-R1 configuration, enabling additional training within a comparable computational budget.

Critical strengths
  • Reconstruction rewards can favor limited viewpoint changes, creating tension with camera-motion requirements. Addressing this incentive while reducing the cost of video RL is a useful research direction.

  • Table 2 separates rollout generation from policy updates and explains how retaining fewer transitions reduces computation and enables a larger rollout batch. The paper also provides equal-epoch comparisons and progress against GPU-hours. Gains appear on both backbone scales and remain evident under the alternative VGGT-Ω pose estimator. The equal-epoch results indicate that these improvements are not solely attributable to longer training. The paper studies all three proposed components and reports VBench results beyond the optimized rewards. It also acknowledges tradeoffs in subject and background consistency rather than presenting uniform improvement.

Critical weaknesses
  1. Section 5.1 defines Recon using the visibility multiplier in Eq. 10, and W3D includes this modified score. Consequently, lower inter-frame overlap can increase Recon without improving the original reconstruction fidelity. Applying the same metric to every method may be consistent, but its interpretation as improved 3D consistency remains limited. Unweighted reconstruction results and an independent geometric-consistency evaluation are needed. TrajΩ provides useful independent evidence for camera following, but does not resolve this geometry-specific concern.
  2. VGGRPO already constructs geometric rewards through cross-view reprojection and depth comparison over valid projected pixels (Sec. 3.3, Eqs. 14–15). The distinction here appears to be using the non-overlap fraction to reweight an existing reconstruction reward. Although VGGRPO is cited, the paper does not directly establish the benefit of this modification over reprojection-consistency rewards. A controlled comparison using the same geometry estimator and training setup would clarify the contribution.
Questions for authors
  1. Can the authors report the unweighted reconstruction score and visibility statistics separately, together with an independent geometric-consistency measure? Please also confirm that every baseline and the “No Vis.” variant use exactly the same evaluation formula.
  2. How does early–late sampling compare with two uniformly sampled transitions under the same rollout and update budget? What grouping, stochastic-sampling, and gradient-rectification settings are used in the single-step baseline?
  3. What precisely replaces OT in the “No OT” variant? With group centering, batch scaling, reward weights, and clipping controlled, does Gaussian quantile normalization outperform clipped standardization or uniform-rank normalization?
Reviewer_Pex5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes 3D-Balanced-R1, an RL post-training method for text-to-video diffusion models. It multiplies a reconstruction reward by a factor based on reduced inter-frame visibility, maps each reward component to Gaussian quantiles using one-dimensional optimal transport, and updates only two denoising transitions, sampled from early and late portions of generation. Experiments on Wan2.1-1.3B and Wan2.2-5B compare against their pretrained backbones and World-R1 on 300 test prompts, supplemented by component ablations, training-time measurements, VBench results, and qualitative examples. The reported results support substantial training savings and improved camera-trajectory following; the evidence for improved geometric consistency is less conclusive.

Critical strengths
  1. The visibility-based intervention addresses a concrete failure mode and has meaningful camera-control evidence. Reconstruction rewards can favor high-overlap videos with little camera movement, and Section 4.1 gives a simple intervention for that incentive. In Table 1, at equal epochs on Wan2.1-1.3B, trajectory alignment improves from 0.426 for World-R1 to 0.472; the alternative VGGT-Ω estimate also improves from 0.427 to 0.475. The trend extends to Wan2.2-5B. Moreover, removing visibility in Table 3 reduces Traj from 0.480 to 0.431 and TrajΩ from 0.484 to 0.441. This evidence makes the camera-control gain credible beyond a change to the reconstruction metric, and the qualitative comparisons in Figures 1, 4, and 7–9 illustrate the intended behavior.

  2. The efficiency improvement is substantial and measured at a useful system level. Table 2 compares one epoch of 384 rollouts on eight GPUs: total time falls from 385.83 to 109.27 minutes, with policy-update time falling from 269.61 to 16.30 minutes. Section 5.2 explains both the reduction from 24 to two optimized transitions and the larger feasible rollout batch. The equal-epoch results in Table 1 are valuable because they show that the reported performance benefit does not depend entirely on generating more training samples. Table 4 also provides evidence that useful general video quality survives post-training, while transparently showing the consistency/dynamics trade-off.

Critical weaknesses
  1. The principal evaluation does not disentangle geometric consistency from the visibility bonus. Section 5.1 defines Recon using Equation (10), namely the original reconstruction score multiplied by a factor that increases as overlap decreases; W3D then adds trajectory alignment. Even if this formula is applied identically to every method, a higher Recon can result from greater viewpoint change without improved reconstruction fidelity. Tables 1 and 3 report neither the original reconstruction score nor the visibility contribution separately, so the headline W3D improvement cannot establish how much geometric consistency improved, particularly at comparable camera motion. TrajΩ is a useful independent camera estimate, but does not resolve this geometric-consistency question. The qualitative appendix and VBench results likewise do not provide a quantitative check of geometry at comparable motion. This matters because preserving or improving geometry while increasing movement is the central claim, rather than movement alone.

  2. The ablations establish useful configurations but do not adequately isolate the claimed optimization mechanisms. For OT, Table 3 shows a trade-off: Recon rises from 0.845 to 0.932 while HPSv3 falls from 10.390 to 10.229, with only a small change in TrajΩ (0.483 to 0.484). The increase in the chosen Total score is encouraging, but does not establish that Gaussian distribution matching offers an advantage over a differently weighted, component-standardized and clipped reward combination. Figure 3 shows the transformation, but no measurements of the claimed HPSv3 dominance or its reduction during optimization. Similarly, Section 5.3 compares early/late sampling against a single-step variant; this changes both timestep placement and the number of optimized transitions. Without a two-step control, the specific benefit of the early/late split is unresolved. These controls matter to the contribution beyond combining visibility rewards with sparse policy updates. Appendix A adds qualitative results but does not address these attribution issues.

Questions for authors
  1. Clarification and evidence needed for the current consistency claim: Are all rows of Tables 1 and 3 evaluated with exactly the same visibility reweighting? What values of λ_vis and τ_vis are used? Please report the original reconstruction score and visibility contribution separately, and provide a geometric-consistency comparison that controls for the amount of camera motion or overlap. Would the conclusion that geometry improves remain under that comparison?

  2. Evidence needed for the current reward-balancing claim: Does the “No OT” variant retain exactly the component-wise centering, batch scaling, and weights in Equations (13)–(14)? Can a standardized/clipped alternative with adjusted reward weights achieve a comparable geometry–quality trade-off? A direct measurement of each reward's contribution to the combined advantage during training would also clarify the claimed domination mechanism.

  3. Evidence needed for the current sampling claim: At the same number of optimized transitions and training budget, how does early/late sampling compare with sampling two transitions uniformly across the denoising trajectory? Please also specify the actual timestep weights ω_t in Equation (15), since these can affect the comparison. This would determine whether the gain attributed to the split comes from timestep placement or simply from optimizing two transitions instead of one.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.