acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Diffusion for Camera-controlled Autoregressive Video Generation

Abstract

We address the task of long-video, camera-controlled generation from a single image. The dominant autoregressive paradigm suffers from two key limitations: (1) error accumulation over time, as distant frames are conditioned on previously generated outputs rather than ground truth, leading to drift; and (2) ambiguity in how to incorporate camera control. Standard solutions—such as self-forcing and conditioning on explicit 6-DOF poses—either incur significant computational overhead or produce inconsistent scene scale in generated videos. We propose a simple and lightweight framework that combines two models, WanAR and FixAR, built on online reconstruction and hierarchical denoising. To enforce consistency with the input camera scale, we draw from feed-forward novel-view synthesis and reformulate video generation as a warping-and-inpainting problem. To mitigate drift, we introduce a hierarchical denoising strategy in which different stages condition on different temporal contexts. Specifically, we first generate frames conditioned on the immediate past, and then refine them using a second denoiser conditioned on a temporally subsampled long-range history to enforce global consistency. This fine-to-coarse scheme enables each generated frame to effectively condition on the full temporal context without incurring a prohibitive cost. Experiments on RealEstate10k and DL3DV demonstrate that WanAR+FixAR outperforms prior methods for camera-controlled long-video generation, producing results that are both temporally stable and consistent with the scale of the input camera trajectories.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Camera-controlled generation, Long video generation, Video diffusion models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_aJghOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This papers proposes a pipeline for camera-controlled autoregressive video generation. It uses an off-the-shelf metric depth estimation model to estimate the depth for the first frame, propagates the estimated geometry to the rest of the frames, and warp the rest of the frames. The camera-controlled problem is therefore reframed as a video inpainting task using a first AR. It further uses another AR to fix the inconsistency between frames, and a third AR to interpolate between the refined frames to recover the original frame rate. The results show that the proposed pipeline can achieve good camera control compared to the baselines.

Critical strengths
  1. The paper is easy to read, and the motivation of the every proposed component is easy to follow.
  2. The proposed method achieves good camera control compared to the baseline models.
Critical weaknesses
  1. I have concerns over the efficiency of the pipeline. It seems that WarpAR is doing the heavy lift of the camera condition part for the video generation, but with limited accuracy since only the key frame has explicit geometry, and the rest of the frames are inpainted without geometry-guidance. This leads to downstream ARs to fix the error. Each AR has the same architecture, and therefore the same computational cost, as the main WarpAR. This makes me wonder if the WarpAR can be substantially improved, so that the subsequent consistency fixes do not have to be heavy. Also, multiple components could potentially make the system fragile if upstream AR fails.
  2. The geometry part is inferred with an off-the-shelf metric depth model, which could bring another source of error if its estimation is not accurate. Besides, it brings me concern that if it's fair since other baselines do not have the off-the-shelf metric depth model as source of input.
Questions for authors
  1. Could the authors provide a robustness analysis of how often each component fails, and if the subsequent component can recover the error or not?
  2. Could you propose potential solutions to reduce the error for the WarpAR estimation itself before any downstream fixes?
3 comments
AuthorsOpenReview ↗

Official Comment by Authors

Q1: How robust is each component, can downstream stages recover upstream errors, and how efficient is the pipeline?

WanAR (referred to as WarpAR in the review) provides camera accuracy. In the downstream stages, FixAR addresses visual drift accumulated during WanAR rollouts while preserving camera accuracy, and InterpolatAR restores the original frame rate. The newly added camera metrics in revised Tables 1 and 2 further support accurate camera control across both datasets. Revised Table 3 ablation demonstrates the robustness of downstream FixAR through five runs with different seeds: camera metrics remain similar, while mean FID (36.11 → 33.88), KID (0.0022 → 0.0017), and FVD (451.21 → 423.86) consistently improve. It also shows that our FixAR outperforms SeedVR2, a state-of-the-art diffusion-based video restoration baseline, across all reported metrics.

Regarding pipeline efficiency, our full pipeline has moderate runtime and GPU memory usage, comparable to many evaluated baselines (Appendix .2, Table 5).

Q2: How could errors in WanAR be reduced before downstream correction?

We agree that reducing visual drift within WanAR itself is a promising direction. Potential solutions include scaling its model capacity and training data, or adapting diffusion-forcing or self-forcing strategies to improve robustness to imperfect generated histories. These approaches involve tradeoffs in training cost and train-test distribution mismatch, as discussed in “Autoregressive long video generation” in Related Work.

Q3: Is using an off-the-shelf depth estimator fair to the baselines?

Many baselines, including ViewCrafter, TrajectoryCrafter, CogNVS, and VACE, also use off-the-shelf depth estimators in our evaluation. Appendix Table 7 further shows that changing the depth estimator has little effect in our ablation, supporting the fairness of the comparison and limited sensitivity to the particular estimator.

Reviewer_aJghOpenReview ↗

Reply to the authors

Thank the authors for providing the rebuttal. While I don't have other questions for the authors, I believe this paper achieves good camera control while having "fundamental limitations of geometry-based warping", acknowledge by reviewer 7vzY as well. I will remain my score as Weak Accept.

Thank you again for your constructive feedback and for taking the time to consider our responses.

Reviewer_7vzYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a hierarchical diffusion framework for camera-controlled autoregressive long-video generation. The framework consists of three stages: WanAR, which reformulates camera-controlled generation as a depth-based warping-and-inpainting task; FixAR, which refines sparsely sampled keyframes using long-range temporal context to mitigate autoregressive drift; and InterpolatAR, which reconstructs intermediate frames from the refined keyframes. The authors argue that this hierarchical formulation enables accurate camera control and long-term consistency without expensive self-forcing training. Experimental results on RealEstate10K and DL3DV demonstrate competitive performance against existing video generation and novel-view synthesis methods, with particularly strong qualitative results.

Critical strengths
  1. The proposed framework is practical and well-motivated. Combining warping-and-inpainting with hierarchical temporal refinement is a reasonable and effective strategy for addressing both camera-control accuracy and long-horizon autoregressive drift. Although the individual components are relatively straightforward, their integration is practically meaningful.
  2. The experimental results are competitive across RealEstate10K and DL3DV. The qualitative results are particularly convincing, demonstrating visually consistent and high-quality generation compared to several strong baselines.
Critical weaknesses
  1. The evaluation of camera-control accuracy is insufficient. Despite accurate camera control being a central contribution, the paper primarily relies on image reconstruction and perceptual metrics rather than directly measuring camera-pose adherence. This is particularly problematic for large camera rotations, where most of the target view consists of previously unobserved regions. In such cases, PSNR/SSIM and especially mPSNR cannot reliably distinguish plausible scene generation from accurate camera control.
  2. The proposed autoregressive warp-and-inpaint framework inherits fundamental limitations of geometry-based warping. Starting from the second rollout, depth is repeatedly estimated from the last generated frame and used to warp subsequent views. Consequently, hallucinated content, inaccurate depth estimates, and warping artifacts can accumulate across rollout boundaries. Such issues are particularly concerning for small objects, thin structures, and disoccluded regions. The paper lacks a detailed analysis of these failure modes and their effects on long-term viewpoint consistency.
  3. The actual contribution of FixAR remains insufficiently demonstrated. Although FixAR improves visual quality, it is unclear whether it also improves adherence to the specified camera trajectories, particularly given the potential accumulation of viewpoint errors across repeated warp-and-inpaint rollouts. FixAR operates on generated frames without explicit camera-pose conditioning, and the quantitative improvements in Table 3 are relatively modest, particularly for PSNR and SSIM. Moreover, the paper provides neither quantitative camera-pose error measurements nor qualitative evidence specifically demonstrating that FixAR corrects accumulated viewpoint inconsistencies.
  4. The depth-scale alignment protocol is insufficiently clarified. Appendix .4 states that MoGe depth predictions are aligned with ground-truth depth for each rollout. However, it is unclear how this ground-truth information is obtained during inference, particularly when later rollouts are conditioned on generated frames. Since scale consistency is a central motivation of the paper, the precise inference protocol and its implications for fair comparison with baselines should be explicitly clarified.
Questions for authors
  1. Could the authors provide explicit camera-pose evaluation metrics, such as rotation and translation errors, by estimating camera trajectories from generated videos?
  2. Could the authors provide a per-rollout analysis of camera-control accuracy and error accumulation, particularly around rollout boundaries?
  3. Does FixAR actually correct camera-pose inconsistencies, or does it primarily improve visual appearance? Could the authors provide quantitative evidence distinguishing these two effects over long autoregressive sequences?
  4. Could the authors clarify whether ground-truth depth is accessed during inference for the rollout-wise scale alignment described in Appendix .4? If so, how is this information obtained, and is equivalent information provided to the baselines?
  5. Since WanAR generates more frames than those ultimately retained by FixAR, could the authors clarify the necessity of these intermediate frames? A comparison with directly generating sparse keyframes would help justify the computational efficiency of the hierarchical pipeline.
  6. Citation formatting is inconsistent throughout the manuscript, with several textual citations incorrectly placed in contexts requiring parenthetical citations.
3 comments
AuthorsOpenReview ↗

Official Comment by Authors

Q1: Can you provide explicit camera-pose evaluation metrics?

We now report explicit camera metrics (R_dist and T_dist) throughout the quantitative quality and camera-control evaluations, including Tables 1 (RealEstate10K) and 2 (DL3DV). We use VGGT to estimate camera trajectories from both generated and ground-truth videos, then compute the camera metrics. Our WanAR + FixAR achieves lower camera errors than every baseline on DL3DV and every baseline except DFoT on RealEstate10K. DFoT is an outlier because relative-pose evaluation can mask scale inconsistency; mPSNR better reveals this issue in covisible regions. See revised Table 1, Table 2, Section 4.1, “Metrics,” and Section 4.2.

Q2: How do camera accuracy and accumulated errors evolve across rollouts, particularly at boundaries?

Appendix .6, Figure 8 now reports frame-indexed camera, reconstruction, and visual-quality metrics on RealEstate10K. Our WanAR + FixAR slows degradation and consistently maintains better reconstruction quality across the sequence, alongside strong camera accuracy, with the DFoT exception discussed above. We observe gradual error accumulation rather than pronounced jumps at rollout boundaries.

For detailed analysis of failure modes, Appendix .7, Figure 9 further examines 56-second generation and camera revisits. Errors in generated content and estimated depth can propagate through subsequent warps. FixAR mitigates this drift but cannot fully eliminate it over extended horizons. Revisits can also produce plausible but different content, reflecting the absence of persistent scene memory already acknowledged in our limitations section. Appendix .7 discusses these remaining limitations and possible extensions.

Q3: Does FixAR correct camera-pose inconsistencies or primarily improve visual appearance?

FixAR primarily improves visual quality while maintaining the camera accuracy provided by WanAR. Revised Table 3 separates these effects using five runs with different seeds, reporting means and standard deviations. Camera metrics remain similar. Meanwhile, mean FID, KID, and FVD improve from 36.11 → 33.88, 0.0022 → 0.0017, and 451.21 → 423.86. These results support robust drift mitigation while preserving camera accuracy. For further details, including the comparison with SeedVR2, please see our response to Reviewer 1, Q2, and revised Table 3, Figure 4, and Section 4.3.

Q4: Is ground-truth depth accessed during rollout-wise scale alignment, and is calibration equivalent for baselines?

Current evaluation protocol. Yes, we use reference depth at warp-source keyframes, including later generated anchors, only to derive scale factors for predicted depth, and baselines receive equivalent calibration. A single multiplicative scale is re-estimated for each anchor; warping uses the rescaled MoGe-2 prediction, and the reference depth map is not supplied directly to the generator. All geometry-based baselines receive the same procedure on their own predicted depths; Plücker-conditioned methods use ground-truth camera poses. This ensures fair baseline treatment, with related reference-geometry calibration practices established in MegaScenes and WRIVA/ULTRRA. See revised Appendix .4, “Our evaluation protocol.”

Chained alignment without future ground-truth depth. Following your suggestion, we additionally test a protocol that uses reference depth only to align the first frame. Later anchors inherit scale by reprojecting the previous anchor’s aligned 3D points into the new camera and fitting the new predicted depth to this sparse reference over reliable overlapping pixels. Table 8 shows very similar WanAR results: R_dist changes from 1.54° to 1.60°, and T_dist from 0.032 to 0.029, alongside similar reconstruction and visual quality. These results support the same conclusion about WanAR’s camera accuracy without access to future ground-truth depth. Given the limited rebuttal time, the main-paper tables retain the current evaluation protocol; Table 8 reports this additional WanAR ablation.

Q5: Why generate intermediate WanAR frames instead of directly generating sparse keyframes?

Intermediate generation provides updated scene geometry for reaching distant viewpoints. Appendix .5 directly compares single-rollout sparse WanAR against our autoregressive pipeline. Warping only the initial image to distant target views can produce entirely empty inputs, removing geometric camera guidance and making inpainting harder. Autoregressive WanAR instead updates the source view and geometry using the latest generated anchor (Figure 7). Our WanAR + FixAR improves every reported metric over direct sparse generation (Table 9). Thus, intermediate generation maintains evolving geometric context, while sparse FixAR processing reduces the cost of refinement.

Q6: Citation formatting is inconsistent.

Thank you for pointing this out. We have corrected citations throughout the manuscript.

I thank the authors for their detailed responses and additional experiments. My concerns have been satisfactorily addressed, particularly through the additional long-term rollout analysis, the clarification of FixAR's role in mitigating visual drift rather than correcting camera inaccuracies, and the detailed explanation and ablation of the ground-truth depth alignment protocol. While these additions do not necessarily demonstrate that all methodological limitations have been fully resolved, the revised manuscript now clearly acknowledges and discusses them. I believe the remaining limitations do not outweigh the paper's strengths and contributions, and I will therefore raise my score.

Thank you again for your thoughtful feedback and for taking the time to review our responses and additional experiments.

Reviewer_kPFXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses camera-controlled video generation from a single image, focusing on camera adherence and error accumulation during autoregressive generation. It proposes a three-stage framework: WanAR uses depth-based warping and inpainting to generate video chunks; FixAR corrects temporally subsampled keyframes using broader temporal context; and InterpolatAR reconstructs the intermediate frames. FixAR is trained on precomputed examples of actual autoregressive drift. Experiments on RealEstate10K and DL3DV report improvements in reconstruction accuracy and video quality, supported by qualitative comparisons and component ablations.

Critical strengths
  • Well-motivated framework. Separating local generation, broader-context correction, and interpolation provides a sensible approach to camera-controlled video generation. Training the correction model on precomputed autoregressive outputs exposes it to realistic artifacts without requiring online rollouts during every correction-model training iteration.

  • Broad empirical evaluation. Comparisons with eight baselines across two datasets show strong reconstruction and FVD results, complemented by useful supplementary visual comparisons. The matched-backbone camera-conditioning experiment and additional keyframe and depth-estimator ablations provide further supporting evidence.

Critical weaknesses
  1. The role of ground-truth geometry at inference is unclear. The introduction and Fig. 6 describe aligning predicted depth to ground-truth geometry during rollouts, but the reference geometry, alignment transformation, and frequency of re-estimation are insufficiently specified. In particular, it is unclear whether this relies only on initially available geometry or uses reference depths associated with later frames. The latter would introduce information beyond the stated single-image input. The paper should also clarify whether baselines receive equivalent scale calibration. Without these details, it is difficult to separate the benefits of geometric conditioning from those of ground-truth-assisted alignment.

  2. The independent contribution of FixAR needs stronger validation. Table 3 shows consistent but small improvements in reconstruction metrics: PSNR increases from 16.63 to 16.71, SSIM from 0.519 to 0.521, and LPIPS decreases from 0.342 to 0.338. Although FID and FVD also improve, no variability estimates are provided to assess the robustness of these gains. Moreover, the absence of mPSNR or another camera-consistency metric leaves unclear whether refinement preserves geometric accuracy. Clarifying whether interpolation and sampling settings are held fixed, and comparing against a generic refinement baseline, would help isolate the value of training FixAR on actual autoregressive drift.

  3. The demonstrated generation horizons provide limited support for the longer-horizon claims. While the paper refers to “minute-long” generation, the main quantitative experiments cover approximately 28 seconds on RE10K and 20 seconds on DL3DV, involving only one or two FixAR calls. These results demonstrate generation beyond the base model’s temporal window, but provide limited evidence that repeated hierarchical correction remains effective over substantially longer sequences. Longer rollouts, with reconstruction and camera-consistency errors reported as a function of rollout length, would better establish scalability. Dedicated revisit or loop-closure evaluations would also help characterize long-term scene consistency, particularly given the acknowledged absence of persistent scene memory.

Questions for authors
  1. What reference geometry is used for depth alignment during rollouts? Does it require ground-truth depths from later frames, and do baselines receive equivalent scale calibration?

  2. Are interpolation and sampling settings held fixed in Table 3? How does FixAR affect camera consistency, and how does it compare with a generic refinement baseline?

  3. How does performance change over longer sequences with more FixAR iterations, particularly when the camera revisits previously observed regions?

3 comments
AuthorsOpenReview ↗

Official Comment by Authors

Q1: Does rollout-wise depth alignment use later-frame ground truth, and do baselines receive equivalent calibration?

Current evaluation protocol. Yes, we use reference depth at warp-source keyframes, including later generated anchors, only to derive scale factors for predicted depth, and baselines receive equivalent calibration. A single multiplicative scale is re-estimated for each anchor; warping uses the rescaled MoGe-2 prediction, and the reference depth map is not supplied directly to the generator. All geometry-based baselines receive the same procedure on their own predicted depths; Plücker-conditioned methods use ground-truth camera poses. This ensures fair baseline treatment, with related reference-geometry calibration practices established in MegaScenes and WRIVA/ULTRRA. See revised Appendix .4, “Our evaluation protocol.”

Chained alignment without future ground-truth depth. Following your suggestion, we additionally test a protocol that uses reference depth only to align the first frame. Later anchors inherit scale by reprojecting the previous anchor’s aligned 3D points into the new camera and fitting the new predicted depth to this sparse reference over reliable overlapping pixels. Table 8 shows very similar WanAR results: R_dist changes from 1.54° to 1.60°, and T_dist from 0.032 to 0.029, alongside similar reconstruction and visual quality. These results support the same conclusion about WanAR’s camera accuracy without access to future ground-truth depth. Given the limited rebuttal time, the main-paper tables retain the current evaluation protocol; Table 8 reports this additional WanAR ablation.

Q2: Are interpolation and sampling settings fixed, does FixAR preserve camera accuracy, and how does it compare with generic refinement?

Yes, interpolation and sampling settings are held fixed in Table 3, following the inference setup in Appendix .1. FixAR’s primary role is to mitigate autoregressive drift while preserving WanAR’s camera accuracy. To assess robustness, we reran WanAR and our WanAR + FixAR with five different seeds and now report means and standard deviations, including mPSNR, R_dist, and T_dist. FixAR maintains similar camera metrics, while consistently reducing mean FID (36.11 → 33.88), KID (0.0022 → 0.0017), and FVD (451.21 → 423.86). We also apply SeedVR2, a state-of-the-art generic diffusion-based video restoration method, to WanAR outputs. It sharpens details but leaves accumulated structural drift largely unresolved. This supports the value of training FixAR on actual autoregressive drift rather than generic restoration degradations. See revised Table 3, Figure 4, and Section 4.3.

Q3: How does performance change over longer sequences and when the camera revisits earlier views?

We added 56-second RealEstate10K examples with camera revisits (Appendix .7, Figure 9). Our WanAR + FixAR retains plausible visual content through the extended sequences, although later frames show more artifacts than the first 28 seconds. FixAR effectively mitigates drift but does not fully eliminate it over extended horizons. Since our main quantitative experiments cover 28 seconds on RealEstate10K and 20 seconds on DL3DV, we have removed the blanket “minute-long generation” claim. On revisits, generated content can remain plausible while differing from earlier observations, reflecting the absence of persistent scene memory, as already acknowledged in our limitations section. Our focus is camera control and drift mitigation; memory is a complementary direction. Appendix .7 discusses a growing 3D scene cache and longer-context FixAR as future extensions.

Thank you very much to the author for responding to my comments; my concerns have been addressed. Taking both the paper and the author's response into account, I will raise my score.

Thank you again for your helpful feedback and for taking the time to review our responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.