acceptodds
Under review as a conference paper at ICLR 2027

LoGo: Local-Global Rewards for Consistent Long-Horizon Video Generation

Abstract

Camera-controlled video models are rapidly advancing toward long generation horizons and complex camera control. A key failure mode is 3D inconsistency: as the camera moves, objects lose permanence and scene structures shift. Existing post-training techniques, which assign a single scalar reward to the entire generation, are poorly suited to correcting these inconsistencies over long horizons. We introduce LoGo, which blends global and spatially localized rewards for camera-controlled video models. The local reward provides fine-grained credit assignment, which substantially improves 3D consistency, while the global reward preserves camera following and video quality. Across three base models, LoGo shows a clear advantage on DL3DV and TrajectoryBench, a new benchmark for long-horizon, complex-camera-control generation that current evaluations lack. LoGo effectively reduces local object shifts, artifacts, and global scene changes, illustrating the importance of credit assignment in post-training video models.

Primary area:
generative models
Keywords:
Video generaion, post-training, world model, camera-controlled video generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gjBsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies 3D inconsistency in camera-controlled video of a few hundred frames: objects drift, appear or disappear, pick up local artifacts, or the scene layout changes. The claim is that geometry-reward post-training such as VideoGPA and World-R1 gives each video one scalar, so a local failure is hidden by the rest of the clip (the paper's term is "washes out").

LoGo reconstructs the generated video with VGGT, voxelizes the fused point cloud (edge length 0.1 times the P90 scene depth), and scores each voxel by the mean RGB and depth reprojection error of the pixels that fall in it (Eq. 1). The group-normalized local score is mixed 1:1 with a video-level reprojection reward in DiffusionNFT's optimality-probability space (Eq. 2). Voxel scores are then averaged onto the latent patches they project to (Eq. 3). The training schedule also inserts HPSv3 and a camera-error reward. The paper adds TrajectoryBench: 2,000 image-plus-trajectory examples, with easy/medium/hard splits, indoor/outdoor/door-transition scenes, and up to 7–8 camera motions in 240 frames, including revisits.

Lingbot2, UniWorld and Lyra2 (all 14B) are post-trained on 1,700 DL3DV scenes and tested on TrajectoryBench and a camera-conditioned DL3DV hold-out. Reported metrics are reprojection PSNR from VGGT and from Depth Anything V3, Gaussian-splatting PSNR, MVCS, epipolar error, RPE, and VideoReward VQ. LoGo improves the consistency metrics over the base models, the authors' reimplementations of VideoGPA and World-R1, and a Global-only ablation. For Lingbot2 the paper reports +2.2 dB PSNR-D and a 26% lower epipolar error on TrajectoryBench, and +2.7 dB and 37% on DL3DV. The gap is larger on harder trajectories (Table 3) and longer videos (Fig. 8). Ablations vary the localization (voxel, frame, 2D patch, window), the reward recipe, the voxel size, and the RL algorithm (Flow-GRPO, DRaFT).

Critical strengths

S1. The comparison that isolates the method is the authors' own Global-only run. It uses the same reward model, the same RL algorithm and the same training recipe.

  • LoGo is better on every consistency metric for all three base models (Tables 1–2).
  • Voxel localization beats frame-level, 2D-patch and window localization (Table 4; PSNR-D 17.2 vs. 15.7 and epipolar error 1.381 vs. 1.726 against Global-only).
  • The advantage grows with difficulty (Table 3, Lingbot2 PSNR-D: +1.0 dB on Easy vs. +2.5 dB on Hard) and with length (Fig. 8.).

The added cost is 9 seconds in a 1,920 second step (Efficiency paragraph in §3; 64 H100s). The same reward is applied to three 14B models: Lingbot2 is autoregressive with a KV cache, Lyra2 generates chunks with a 3D cache, and UniWorld is bidirectional.

S2. TrajectoryBench is a harder camera-control test than WorldScore. The tasks leave the first view, revisit a region, pass through a door, and put several camera motions in one 240-frame video. There are difficulty splits, stylized scenes, and paired standard errors in the appendix. DL3DV is also evaluated with camera conditioning. The authors say they will release code, checkpoints and the benchmark. WorldScore uses 80 frames and one camera command, so it does not produce the long single-clip failures in Fig. 1. It is not the only revisit evaluation already available: the introduction cites MIND (Ye et al., 2026), which tests revisit consistency with closed-loop actions rather than an open-loop camera trajectory.

Critical weaknesses

The tables do not yet show that a viewer would see more consistent video, or that the gain over prior methods comes from the spatial reward.

The training reward is a VGGT(-Ω) RGB-D reprojection error. PSNR-V is the same kind of score. PSNR-D and MVCS are also reprojection metrics, and RPE uses VGGT poses. HPSv3 in Table 5 is one of the interleaved training rewards. Section 3.2 and Fig. 3a/c show that blurring and desaturating can lower this reward. VR-VQ is the reported quality number that is not a training reward, and there is no human study. Epipolar error does not use that reconstruction. It drops clearly for Lingbot2. On DL3DV the other two models move less: UniWorld and Lyra2 gain +0.6 dB and +0.8 dB PSNR-D, and epipolar error falls by 5% and 24% (Table 2).

Section 4.2 says VideoGPA and World-R1 were built on WAN, an 80-frame model without camera control, and that the authors extend them to this setting. It does not say whether that extension uses the same data, the same number of steps, or the HPSv3 and camera rewards used for LoGo and Global-only. On Lingbot2 (Table 1), Global-only is still a scalar reprojection reward, but it is already at PSNR-D 17.1 and RPE 11.2°, against 16.1 and 13.1° for VideoGPA. In Table 5, adding interleaving alone moves RPE from 11.2° to 8.9°. The gap to the prior methods can therefore come from DiffusionNFT and the training schedule. Table 9 points the same way: with Flow-GRPO and DRaFT (UniWorld) the margin over Global-only is about 0.1–0.2 dB, and on the Flow-GRPO hard split the epipolar error goes from 1.500 to 1.553. The caption still says the advantage persists for every post-training algorithm.

Questions for authors
  1. Baseline setup. Section 4.2 says VideoGPA and World-R1 are extended to the camera-controlled setting. Did that extension, and Global-only, use the same data, step budget, and HPSv3/camera interleaving as LoGo? If not, what happens if the VideoGPA or World-R1 reward is trained inside the same DiffusionNFT run and the same schedule?

  2. Evidence outside the training reward. The training reward, PSNR-V, and RPE depend on VGGT; PSNR-D uses Depth Anything V3. Section 3.2 shows that blur and desaturation can lower the VGGT reward. Is there a viewer comparison on consistency and image quality, or a metric the reward does not optimize (frame sharpness or saturation, or RPE from a pose estimator other than VGGT)?

  3. Shared voxel grid (Eq. 2). What are eˉi,local\bar e_{i,\text{local}} and Zei,localZ_{e_{i,\text{local}}}? The 16 rollouts are separate VGGT reconstructions. How do they share one voxel grid, in coordinate frame and scale, and how is a voxel scored when only some rollouts observe it?

6 comments
AuthorsOpenReview ↗

Response to Reviewer gjBs

We thank reviewer gjBs for your review! Thank you for highlighting the importance of our global-only ablation and the value of TrajectoryBench. Our goal is to improve 3D consistency while preserving other aspects of video generation such as video quality and camera control. To achieve this, LoGo has three components that work together: credit assignment provides good consistency, but when used alone can induce reward hacking; local-global blending extends the Pareto frontier in the quality-consistency tradeoff; reward interleaving combats reward hacking. We address your question below:

VideoGPA and World-R1 are extended to the camera-controlled setting. Did that extension, and Global-only, use the same data, step budget, and HPSv3/camera interleaving as LoGo?

The baseline methods and global-only are all trained on the same data and till convergence (thus step budget is not a limit). Reward interleaving, along with localized credit assignment and local-global blending, are three components of LoGo's method (Sec 3.1-3.3). For a fair comparison with existing methods, we stay faithful to the design choice of each method regarding post-training effectiveness and video quality. For example, VideoGPA is a two-stage method with rollout, DPO data pair selection via specific heuristics, and DPO training. Its recipe contains "preserving the base model’s visual quality" (Section 1, VideoGPA paper) by doing data filtering (Section 3.4, VideoGPA paper). We preserve these design choices for a fair recipe comparison.

Local-global blending and reward interleaving are important parts of the LoGo recipe (Table 5). These components need to work alongside with credit assignment to make LoGo effective. We do provide ablations to isolate out the effect of credit assignment in the LoGo vs. global-only comparison (Table 1). Global-only and LoGo are trained on exact same settings (same interleaving, same data etc.), both till convergence. The advantage of LoGo beyond Global-only shows that our advantage to baselines is not just in DiffusionNFT and reward interleaving. In fact, as shown in the Pareto plot (Fig. 3b), reward interleaving combats reward hacking rather than increases consistency. All components of LoGo (Sec 3.1,3.2,3.3) work together to make it an effective method.

AuthorsOpenReview ↗

Response to Reviewer gjBs

Evidence outside the training reward. The training reward, PSNR-V, and RPE depend on VGGT; PSNR-D uses Depth Anything V3. Section 3.2 shows that blur and desaturation can lower the VGGT reward. Is there a viewer comparison on consistency and image quality, or a metric the reward does not optimize (frame sharpness or saturation, or RPE from a pose estimator other than VGGT)?

We totally agree that evaluating metrics should fall outside the training reward, and we do exactly this. We would first like to clarify a misunderstanding.

Clarification: Our goal is to improve 3D consistency while preserving other aspects of video generation, including video quality and camera control. Blur and desaturation do not “lower the VGGT reward” as you mention --- on the opposite, they increase the VGGT reward at the expense of video quality (Sec 3.2, L187-192), i.e. they are "reward hacking" behaviors. The design of LoGo, which includes local and global blending and reward interleaving, aims to combat this behavior. VR-VQ, a video quality metric, is exactly the metric to catch this, and it is not used in any training.

Metrics not used in training: LoGo’s evaluation contains various metrics that are not used in training.

For 3D consistency, which is LoGo’s main goal, we only train with VGGT reward and evaluate 3 other independent metrics that are not used in training. PSNR-D comes from Depth Anything V3, a model that is never used in training. PSNR-GS comes from Gaussian reconstruction, which is an alternative 3D reconstruction method that is different from depth estimation, and thus PSNR-GS is an unbiased metric completely unused in training. Finally, epipolar error is based on geometry: epipolar constraint rather than a reconstruction model, which is also outside of the training signal.

For video quality, we use VR-VQ, a video quality metric not used in training which can capture aspects such as blur, desaturation, or otherwise aesthetic degradation. LoGo maintains and largely improves VR-VQ (Table 1, Table 2), showing that it successfully combats reward hacking and preserves (even improves) video quality, in addition to improving 3D consistency.

For camera pose, we used RPE from VGGT in the main tables since it is the most accurate pose estimation model. We additionally show camera RPE from another pose estimator, Depth Anything V3, which highly agrees with the VGGT: on average, its rotation varies from VGGT less than 1.6% and translation less than 3.6%. We show the full VGGT and DA3 metrics below:

FamilyModelRot (VGGT)Rot (DA3)Trans (VGGT)Trans (DA3)
LingBotBase13.213.50.1080.111
LingBotVideoGPA13.113.10.1070.111
LingBotWorld-R113.113.20.1080.110
LingBotGlobal11.211.20.1020.105
LingBotLoGlo8.98.80.1030.107
UniWorldBase13.814.10.1320.136
UniWorldVideoGPA13.814.00.1320.135
UniWorldWorld-R113.113.40.1290.133
UniWorldGlobal13.313.50.1260.130
UniWorldLoGlo14.014.10.1300.133
LyraBase7.98.30.1000.106
LyraVideoGPA7.77.70.0990.104
LyraWorld-R17.77.90.1010.105
LyraGlobal7.67.70.0990.104
LyraLoGlo7.57.70.1010.106

Human evals: In addition to all the non-training metrics we evaluate, we additionally add human evaluation of LoGo vs. other baseline methods in 3D consistency on 150 video pairs based on stratified random sampling (to cover each difficulty level proportionally). We show the win rate of LoGo vs. each method (ties excluded) ranked by three independent annotators below. LoGo shows significant advantage, with 2.6x to 7.6x win rates.

Win rate
LoGo vs. Base3.4 : 1
LoGo vs. VideoGPA2.6 : 1
LoGo vs. World-R17.6 : 1

The tables do not yet show that a viewer would see more consistent video

Our metrics measure 3D consistency by measuring depth reprojection error (PSNR-V & -D), gaussian reconstruction error (PSNR-GS), MVCS, and epipolar error. In addition to the quantitative analysis, we also provide qualitative comparisons in Fig 6 and in video format in the supplementary zip folder. The purpose of the qualitative comparisons is to demonstrate the advantage of LoGo in generating more consistent videos – for example, maintaining the subway entrance scene rather than changing it into a different room (keyframes in Fig 6 first row, video in videos/baseline_comparison/0/) or removing hallucinated column-like artifacts (keyframes in Fig 6 second row, video in videos/baseline_comparison/3/).

AuthorsOpenReview ↗

Response to Reviewer gjBs

Shared voxel grid (Eq2). What are eˉi,local\bar{e}_{i, local} and Zei,localZ_{e_{i,local}}? The 16 rollouts are separate VGGT reconstructions. How do they share one voxel grid, in coordinate frame and scale, and how is a voxel scored when only some rollouts observe it?

eˉi,local\bar{e}_{i, local} is the mean of voxel ii's voxel error across rollouts and Zei,localZ_{e_{i,local}} is the normalization factor based on cross-rollout standard deviation - this is the standard normalization step to transform voxel errors into [0,1], the “optimality probability” used in DiffusionNFT, as defined in Sec 3.3 in the DiffusionNFT paper. Since the initial (conditioning) view is shared across rollouts, we use the camera frame of that view as the coordinate frame and voxel size of 0.1*P90 depth of the shared conditioning frame, so that various rollouts are comparable. If a voxel is observed by less than 16 rollouts but no less than 2, it is normalized within the subset of rollouts that observe it. If it is observed by less than 2 rollouts, it is skipped since standard deviation for normalization cannot be calculated (the corresponding cells later receive a neutral score of 0.5 during optimization). This affects less than 1% of the cells in practice.

Reviewer_XTmoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes LoGo, a post-training method for improving 3D consistency in camera-controlled video generation. The method uses spatially localized rewards in the 3D voxels unprojected by VGGT to reflect local geometry quality and combines them with a global video-level reward for credit assignment during post-training. The proposed reward is combined with other reward functions and shows it can improve the 3D consistency. Experiments are conducted on Lingbot2, Lyra2, and UniWorld, with evaluation on DL3DV and the proposed TrajectoryBench.

Critical strengths

The paper is well-written and easy to follow.

The method is clear and reasonable: the idea of using local reward to reflect local geometry quality make sense to me. The proposed method that leverages VGGT-omega to unproject and then computes voxel level geometry error is simple and straightforward but turns out to be useful. The motivation for moving from a single video-level reward to localized credit assignment also sounds natural.

The performance gains across different base models are noticeable. The local-only and global-only ablations also help explain why combining the two rewards is useful.

Critical weaknesses

I do not have major concerns for this paper. I only have a few minor concerns:

First, the evaluation horizon is relatively short for a paper focused on long-horizon generation. TrajectoryBench mainly evaluates up to 240 frames, and the 400-frame experiment only reports epipolar error for Lingbot2. Since models such as Lingbot2 and other world model papers (e.g. DreamX-World) already support minute-scale generation, I would like to see results on one-minute or longer rollouts, especially with revisits after a long horizon.

Second, the comparison does not cover enough recent models. Although Lingbot2 is included as a base model, DreamX-World and HY-WorldPlay 1.5 are missing from the benchmark comparisons. It will be better to also eval these recent models.

Third, it will be better if authors could provide a visualization of reconstructed point cloud or Gaussian splats to directly show the geometry consistency improvement before and after post-train with LoGo.

Questions for authors

Please see the weakness section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer XTmo

We thank reviewer XTmo for your review! Thank you for mentioning that our method of using local reward is clear and reasonable, and the performance gains across multiple models are noticeable.

Minute-scale generation

Thank you for the question! Our goal is to extend the generation horizon given complex trajectories that include rotations, turning one direction and back, and revisits from various angles. Trajectory complexity, which TrajectoryBench highlights, is a key aspect, in addition to the raw number of frames generated. In fact, models like DreamX-World and Lingbot2, although featuring minute-scale demos with very simple (e.g. predominantly forward/backward) trajectories, struggle greatly with more complex trajectories even at 240 frames.

Specifically, DreamX-World’s four minute-long demos are mostly forward motions which do not test 3D consistency. With TrajectoryBench’s trajectories that include turns and revisits, DreamX-World struggles at 240 frames, obtaining only 13.7dB in the “hard” split, lower than all base models we use. For comparison, Lyra2 base model obtains 17.8dB in this split (raised to 19.4dB by LoGo). We show visualizations of DreamX-World on TrajectoryBench in Fig.11 in the updated Appendix.

For Lingbot2, most of the long demos are in game-like environments, and heavily feature backward/forward motion without revisits. Lingbot2 also struggles on TrajectoryBench at 240 frames, obtaining a PSNR of only 15.3dB in the “hard” split, and also shown qualitatively in Fig.6. LoGo improves its performance on "hard" split to 17.6dB.

The goal of TrajectoryBench is to highlight the challenge brought by complex camera trajectories, in addition to raw frame count. As shown in our result, the benchmark is very challenging to today’s state-of-the-art models, and these models degrade significantly going from 80 to 240 frames (Table 3) from easy to hard. LoGo is our method to effectively extend the generation horizon under these challenging trajectories.

Although Lingbot2 is included as a base model, DreamX-World and HY-WorldPlay 1.5 are missing from the benchmark comparisons. It will be better to also eval these recent models.

We evaluate and post-train state-of-the-art camera-controlled models: Lingbot2 (July 2026), Lyra 2 (Apr 2026), Uniworld-View (Aug 2026). As requested, we additionally include models DreamX-World (June 2026) and an older model released last year, HY-World 1.5 (Dec 2025). Due to time constraints of the rebuttal, we evaluate the new models on a subset of 400 scenes of TrajectoryBench, proportionally covering each difficulty level. The results are as follows:

modelPSNR-VPSNR-DPSNR-GSMVCSEpipolarRPE-RotRPE-TransVR-VQ
Lyra-2 base18.3917.7318.970.8971.557.930.100-0.235
Lyra-2 LoGo19.6618.9619.970.9051.357.510.101-0.194
LingBot base16.9316.0216.940.8421.5113.250.1080.214
LingBot LoGo19.0518.2318.430.8941.118.860.1030.370
UniWorld base17.6116.8017.500.8811.2613.750.1320.390
UniWorld LoGo18.7717.9918.630.9011.1114.020.1300.408
DreamX-World14.5214.0516.270.8581.6022.710.2040.374
HY-WorldPlay 1.516.6016.0816.790.777*3.3224.970.179-0.071

Both DreamX-World and HY-WorldPlay 1.5 are weaker in 3D consistency than the base models we use, and additionally struggle with camera following. HY-WordPlay 1.5 significantly struggles when the camera moves away from the initial view, and often breaks down into artifacted patterns. This happens to 70% of the hard trajectories based on human evaluation of 40 random scenes. This behavior is caught by the high epipolar error. * denotes that since the artifacts break the depth scale in the MVCS metric, we use per-frame depth scaling to calculate MVCS for HY-WorldPlay 1.5 so that non-artifact frames still receive reasonable depth. In LoGo, we choose base models that are state-of-the-art in camera-controlled generation, and show effective improvement across Lyra2, Lingbot2 and UniWorld. We are happy to include these two additional models as baselines in the paper.

it will be better if authors could provide a visualization of reconstructed point cloud or Gaussian splats to directly show the geometry consistency improvement before and after post-train with LoGo.

Thank you for the suggestion! We visualize reconstructed point clouds in Fig.12 in the updated Appendix. Without LoGo post-training, the scenes show inaccurate, overlapping 3D points due inconsistency (first two rows), or multiple duplicate reconstructed items (duplicate door frames in the third row, duplicate monuments in the last row). LoGo effectively solves these issues and results in clean, coherent reconstructions.

Thank you for your response. I will maintain my score.

Reviewer_n4vmOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

LoGo studies post-training camera-controlled video generators for long-horizon 3D consistency. It aggregates reconstruction errors in world-space voxels, maps localized rewards back to video latents, combines them with a global consistency reward, and interleaves geometry, aesthetic, and camera-following objectives. The paper also introduces TrajectoryBench for more complex camera trajectories. Experiments on three base models and two benchmarks show consistent geometric improvements, with an explicit analysis of the consistency–quality trade-off.

Critical strengths
  1. Spatial credit assignment is a well-motivated response to a concrete weakness of scalar video rewards. The comparison with global, window, framewise, and patchwise alternatives provides useful evidence that the 3D localization matters (Table 4). The paper also explicitly studies the quality degradation caused by local-only optimization rather than treating a higher geometric score as sufficient.
  2. The evaluation is reasonably broad and supports the practical relevance of the approach. Gains appear across three generators and both TrajectoryBench and DL3DV, and the use of DA3-based metrics and Gaussian reconstruction complements the VGGT-derived reward. For example, Lingbot2 improves from 16.0 to 18.2 PSNR-D on TrajectoryBench, compared with 17.1 for the global-only variant. The harder-trajectory breakdown and reported paired standard errors are valuable.
Critical weaknesses
  1. The exact localization recipe needs to be reconciled between the mathematical description and the supplied implementation. Appendix A.2 describes averaging the rewards of intersecting voxels for a latent patch. In the supplied voxel scorer, the default latent-resolution path assigns a cell to its dominant voxel, whereas the alternative patch path uses pixel-count-weighted aggregation. The code also includes neutral treatment of unmatched cells and a within-rollout median/MAD contrast term. These are meaningful choices for localized credit assignment, and the paper does not clearly identify which combination produces each main result. The shared first-camera coordinate frame and depth scaling are documented in the code, but their robustness to independently reconstructed rollouts also deserves clarification.
  2. The causal interpretation of local–global blending is not fully isolated from optimization strength. Mixing two normalized but imperfectly correlated signals changes their variance and therefore the effective update magnitude. Table 5 demonstrates a useful empirical trade-off, but does not establish whether the global signal itself is responsible for preserving quality, rather than some of the benefit coming from weaker updates. This limits the explanation of why the combined recipe works, even though it does not negate the reported gains.
Questions for authors
  1. Which voxel-to-latent aggregation, temporal pooling, within-rollout normalization, and unmatched-voxel settings produce Tables 1–5? Please reconcile the default implementation with Eq. (3) in Appendix A.2 and provide the corresponding configuration for each ablation.
  2. How much cross-rollout voxel coverage is available after first-camera anchoring and depth normalization, especially for expansion and hallucinated objects? What fraction of latent cells receives a neutral reward, and how sensitive are the gains to reconstruction/registration error?
  3. Does local-only optimization with matched advantage variance, learning rate, or parameter-update magnitude recover the quality benefit attributed to global blending? A controlled comparison would materially strengthen the interpretation of Table 5.
2 comments
AuthorsOpenReview ↗

Response to Reviewer n4vm

We thank reviewer n4vm for the detailed review and questions! Thank you for saying that our spatial credit assignment is a well-motivated response to a concrete weakness of scalar video rewards, and that our evaluation is broad. We answer your questions below:

Configuration clarification

Thank you for the detailed review! The code we attached during submission time includes additional options which we explored in earlier, small-scale settings but which were no longer relevant in the scaled-up setting we report in the paper. LoGo follows one recipe, used across all tables and experiments in the paper, following Equation 1,2 in the main text. During training, config settings are passed in via command line arguments. We add the following details to answer your specific questions around implementation.

Voxel to latent aggregation: each latent patch gets a pixel-weighted average of voxels that it maps to, as detailed in the updated appendix Equation 3. During training, we pass in the actual latent dimension of the videos, in our case 60x104 (for 480x832 resolution videos and 8x VAE spatial compression). For Uniworld this is defined as NFT_VOXEL_PATCH=60x104 (or converted from env variable “patch”: [60,104] in models/uniworld/train.sh), and the weighted averaging logic is in models/uniworld-view/rl/scoring/nft_voxel.py L420-431. For Lingbot2, this is set via NFT_VOXEL_PATCH=60x104, and the weighted average logic is defined in models/lingbot-world-v2/wan/rl/scoring/nft_voxel.py L410-424. For Lyra2, this is set via NFT_VOXEL_PATCH=latent which converts to (60,104) in models/lyra2/Lyra-2/lyra_2/_src/rl/scoring/nft_voxel.py L108-109, and the weighted averaging logic in models/lyra2/Lyra-2/lyra_2/_src/rl/scoring/nft_voxel.py L276-307.

Temporal pooling: “rollout” is used, i.e. the reward is always normalized per voxel, as Equation 2 specifies.

Within-rollout normalization: This is not used in LoGo. This was a variant we explored for small group sizes, as described in L168-170. However, for our large scale experiments where group sizes are larger, this was not used.

Unmatched voxel settings: We skip voxels that are covered by less than 2 rollouts, since the standard deviation needed for normalization cannot be calculated with less than 2 rollouts. In practice, this affects less than 1% of cells. This is now specified in the updated appendix A.2 (L732), and it is defined in code in models/uniworld-view/rl/scoring/nft_voxel.py L 339-340 for UniWorld, models/lingbot-world-v2/wan/rl/scoring/nft_voxel.py L347-348 for Lingbot2, and models/lyra2/lyra-2/lyra_2/_src/rl/scoring/nft_voxel.py L375-376 for Lyra2.

How much cross-rollout voxel coverage is available after first-camera anchoring and depth normalization, especially for expansion and hallucinated objects? What fraction of latent cells receives a neutral reward?

We align the 3D space generated across rollouts by leveraging the first frame shared across all rollouts which is passed as conditioning signal. We take the camera frame of the first frame, and use a voxel size based on the P90 depth of this shared frame.

We note that for alignment, what matters is the space coverage of the generated video rather than the details of generated content (such as hallucinated objects). For example, two rollouts generating the same room, one with a hallucinated lamp and one without, cover the same space and thus are well-aligned, although the hallucinated rollout might receive a lower score. To quantitatively show this, we measure the cross-rollout overlap percentage: only less than 1% of cells receive a "neutral" reward, and over 90% of cells are covered by more than half of the rollouts. Even for the expansion split, only less than 2% of cells receive a neutral reward.

how sensitive are the gains to reconstruction/registration error?

Our work targets post-training for video generation, a generative setting where there is no single ground truth. To address this, we use VGGT-Omega, a state-of-the-art large-scale depth prediction model. Models of this kind have improved rapidly in recent years and now effectively reconstruct 3D point clouds of scenes from video. However, they may carry their own biases. To guard against this, our evaluation includes metrics built on an independent depth estimator (DA3), as well as metrics based on a different 3D reconstruction approach (Gaussian splatting instead of depth). These results show that our method does not overfit to the particular behaviors of VGGT-Omega.

AuthorsOpenReview ↗

Response to Reviewer n4vm

Does local-only optimization with matched advantage variance, learning rate, or parameter-update magnitude recover the quality benefit attributed to global blending?

We believe the advantage from local-global blending does not come from weaker updates, based on the pareto analysis in the paper, a gradient norm comparison, our “train till convergence” setup, and an additional advantage variance scaling experiment we perform during rebuttal and show below.

Pareto analysis: Our analysis in the 3D consistency vs. video quality tradeoff is shown in Fig. 3b of the main paper. If, as you say, LoGo has “benefit coming from weaker updates”, then limiting the training steps of local-only reward would have the same performance as LoGo. But Fig.3b shows this is not true - the dots in Fig.3b along the “local” curve come from different optimization steps. By limiting optimization with local-only reward, you can only move along the same curve - both without interleaved reward (solid, triangle) and with interleaved reward (dotted, circle). However, LoGo is not limited to this curve but pushes the Pareto frontier. The fact that LoGo exhibits a better Pareto frontier shows that blending’s advantage is not just due to optimization strength.

Grad norm comparison: We use the same learning rate for local-only and local-global blended settings. We compare the gradient norm during training to understand the effect on optimization strength, shown below (± denotes 90% confidence interval):

Local-onlyLocal-global blending
0.79 ± 0.150.83 ± 0.15

There is no significant difference in grad norm in local-only vs. local-global blending, which suggests “optimization strength” stays similar regardless of blending.

Train till convergence: For all our experiment settings, we train the models till convergence (L265), and use AdamW as the optimizer. If the advantage of local-global blended reward results in a weaker update, it should converge later, requiring more optimization steps, but should not affect the results at convergence.

Advantage variance and scaling experiment: We audit the standard deviation of advantage across local, global, and blended (± denotes 90% confidence interval). The 90% confidence interval of the two settings overlap, suggesting the difference is not significant.

Local-onlyLocal-global blending
0.78 ± 0.110.65 ± 0.06

Despite this, we perform an additional experiment to scale up the advantage of the blended version by 1.2x to match the mean advantage standard deviation of local-only. Due to the compute constraint during rebuttal, we cannot finish full training, but we can do the following analysis within a smaller number of steps: during optimization, how much does the gradient and parameter update change due to the 1.2x advantage scaling? Is this change more significant than the noise floor (i.e. simply changing the random seed)? To answer this, we report gradient norm ratio, as well as difference in the parameter update (parameter after vs. before the step, which takes into account AdamW’s effect), both in norm and in direction via cosine similarity. We report the mean per-step ratio/cosine similarity. We compare the difference caused by the 1.2x advantage scaling vs. changing the random seed below.

SettingGrad norm ratioUpdate norm ratioUpdate cosine similarity
1.2x scaling vs. original1.00051.00080.9997
Different seed vs. original (noise floor)1.00651.00140.9851

The change in grad norm, parameter update norm, and parameter update direction caused by the slight scaleup of advantage are all less than the change caused by random seed, which is reasonable because 1.2x advantage scaling is a small adjustment. This further suggests that the optimization strength due to advantage variance of local-global blend is not critical to our results.

We are happy to respond if you have further questions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.