acceptodds
Under review as a conference paper at ICLR 2027

Warp-as-History: Generalizable Camera-Controlled Video Generation from One Training Video

Abstract

Camera-controlled video generation has made substantial progress, enabling generated videos to follow prescribed viewpoint trajectories. However, existing methods usually learn camera-specific conditioning through camera encoders, control branches, or attention and positional-encoding modifications, which often require post-training on large-scale camera-annotated videos. Training-free alternatives avoid such post-training, but often shift the cost to test-time optimization or extra denoising-time guidance. We propose Warp-as-History, a simple interface that turns camera-induced warps into camera-warped pseudo-history with target-frame positional alignment and visible-token selection. Given a target camera trajectory, we construct camera-warped pseudo-history from past observations and feed it through the model's visual-history pathway. Crucially, we align its positional encoding with the target frames being denoised and remove warped-history tokens without valid source observations. Without any training, architectural modification, or test-time optimization, this modification reveals a non-trivial zero-shot capability of a frozen video generation model to follow camera trajectories. Moreover, lightweight offline LoRA finetuning on only one camera-annotated video further improves this capability and generalizes to unseen videos, improving camera adherence, visual quality, and motion dynamics without test-time optimization or target-video adaptation. With systems-level optimizations, the method supports real-time interactive inference on a single H200 GPU. Extensive experiments on diverse datasets confirm the effectiveness of our method.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Camera Controlled Video Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_iRYXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose Warp-as-History, a method that converts camera-induced warps into camera-warped pseudo-history with target-frame positional alignment and visible-token selection to route camera control through a model's native visual-history pathway. This simple interface reveals a non-trivial zero-shot capability in frozen video generation models to follow prescribed camera trajectories without requiring architectural changes or test-time optimization. Applying lightweight offline LoRA finetuning on just a single separate video further stabilizes this camera-following behavior and allows it to generalize effectively to unseen videos.

Critical strengths
  • Unlike existing methods that require large-scale datasets, the proposed approach achieves competitive performance using only a single training video and lightweight offline LoRA finetuning.
  • By routing camera-warped pseudo-history through the model's native visual-history pathway with PE adjustment, the method unlocks a zero-shot camera-control capability in the frozen Helios model.
Critical weaknesses
  • Overall, the manuscript suffers from significant writing and organizational issues. The logical flow is difficult to follow, paragraph transitions are weak, and intra-paragraph cohesion needs substantial improvement. In particular, the Introduction section requires a major overhaul to clearly set up the work.
  • There is an apparent contradiction in the core pitch: the introduction claims that Warp-as-History achieves generalizable camera control without any training or inference-time optimization methods, yet the methodology later states that offline LoRA fine-tuning is performed on a single camera-annotated video. This conceptual inconsistency needs to be reconciled.
  • Please specify which individual video is used for the offline LoRA fine-tuning. Furthermore, what is the underlying mechanism or justification that enables a model fine-tuned on just a single video to achieve universal generalizability?
  • Regarding Table 3, evaluating video generation metrics against baselines that do not share the same backbone offers limited utility. These metrics should instead be compared against the distilled Helios model without any fine-tuning applied to provide a fair and meaningful baseline.
Questions for authors
  • Line 458 states: 'The fixed car-roundabout source was chosen before the retrospective sweep.' Please clarify the meaning of this sentence and provide sufficient context to explain its relevance to the discussion."
  • Is the DAVIS dataset considered an appropriate or standard benchmark for evaluating camera-controllable video generation methods, given the specific domain?
5 comments
AuthorsOpenReview ↗

Response to Reviewer iRYX

Thank you for the detailed comments. We agree that our organization and presentation did not make several important distinctions sufficiently visible, and we address each concern below.

Writing and organization. We accept this criticism and will substantially rewrite the Introduction. It will follow a clearer sequence: (1) limitations of existing camera-conditioning approaches; (2) our observation that pretrained history-conditioned models contain latent camera-following behavior; (3) zero-shot Warp-as-History as a diagnostic of this behavior; and (4) the final one-video offline LoRA model, with fixed weights at test time. We will also use zero-shot, one-shot, offline finetuning, and test-time optimization consistently throughout.

The training claim. We would like to clarify that the paper does not claim that the final one-shot model requires no training. “Without any training” refers only to the zero-shot observation on the frozen model. The Introduction explicitly states: “Although this zero-shot effect is not robust enough to serve as a final method on its own,” followed by, “We further enhance this capability with lightweight offline LoRA finetuning on only one single camera-annotated video.” Thus, zero-shot is diagnostic evidence of a latent capability; the final model uses offline LoRA on one separate video, with no test-time optimization, per-video fitting, or target-video adaptation. We apologize that our presentation did not make this separation sufficiently prominent.

Source video and generalization mechanism. The sole source is DAVIS car-roundabout, as stated in Section 4.1, and it is disjoint from the evaluation videos. Our claim is not that one video teaches universal video generation or camera control from scratch. The pretrained base model already acquires rich appearance, motion, and viewpoint generation capabilities. Warp-as-History reuses its native history-conditioning interface to supply geometric evidence; one-video LoRA then learns to interpret and stabilize this evidence, balancing it against the generative prior. The zero-shot behavior supports this activation interpretation. We will avoid “universal generalizability” and restrict our claims to the unseen videos, trajectories, and backbones actually evaluated.

Same-backbone comparison. We agree that cross-backbone metrics alone cannot isolate our contribution. Importantly, the requested comparison already appears in Appendix Table 12: Text-only Base, Zero-shot Full, and One-shot Full all use Helios-Distilled and include FID, FVD, DOVER, and VBench metrics. Table 1 provides a similar text-only comparison on WorldScore. On DAVIS, Text-only Base→One-shot Full improves PSNR (14.38→15.21), LPIPS (0.4199→0.3794), FID (71.55→68.18), and FVD (80.87→57.95), while Dynamic Degree increases from 0.091 to 0.714. However, DOVER (0.463→0.448), Imaging Quality (65.21→62.20), and some consistency metrics decrease. We therefore do not claim lossless improvement on every quality axis; the results show a control–quality trade-off. We regret not foregrounding these controls and will move them into the main comparison, treating external cross-backbone results only as system-level references.

“Chosen before the retrospective sweep.” This means that car-roundabout was fixed for the main experiments before we conducted the multi-source sensitivity sweep in Tables 9–11. We did not replace it afterward with a better-performing source based on downstream results. The statement was intended to rule out post-hoc selection of the best source, not to claim that car-roundabout was optimal. We will state this timeline and selection process directly.

DAVIS. We agree that DAVIS is a widely used video-object-segmentation dataset, not a standard camera-control benchmark. Since existing camera-control evaluations predominantly emphasize static scenes, we use DAVIS as a complementary dynamic-foreground stress test, not as a replacement for WorldScore or RE10K. Its estimated camera trajectories also introduce pseudo-ground-truth noise and monocular geometry errors. We apologize for insufficiently explaining this motivation and will clearly state both DAVIS’s complementary role and its limitations.

Reviewer_iRYXOpenReview ↗

Reply by Reviewer iRYX

The rebuttal addresses my concerns partly. Specifically, I now understand the generalization claim / backbone-dependent comparison / evalset points after reading the author's rebuttal. None of updates are applied in the manuscript. Similarly, authors agree significant modification of writing and organization is needed. However, I do not see any modifications applied on the draft. Authors can correct me if I am wrong. Thus, my most concerning point is not addressed at all. Moreover, similar to what Reviewer sxPw pointed out, statements on latency is quite misleading on the manuscript even though the authors answered clearly on the rebuttal. I strongly believe that the manuscript be updated to address reviewers' concerns.

AuthorsOpenReview ↗

Response to Reviewer iRYX

Thank you for pointing this out. You are correct: our previous rebuttal described the intended revisions, but we did not upload an updated manuscript. That was our oversight, and we apologize for the discrepancy between our response and the PDF.

We have now uploaded a revised manuscript, with the changes highlighted in blue. Specifically, we have (1) substantially reorganized and rewritten the Introduction to clarify the zero-shot observation versus the final one-video LoRA model; (2) added matched Helios-Distilled comparisons to Tables 2 and 3 and explicitly discussed the resulting quality trade-offs; (3) revised the camera-accuracy and generalization claims; (4) clarified the source-selection timeline and the complementary role of DAVIS; and (5) corrected the latency statements by distinguishing the standard A800 measurements from the separately optimized H200 configuration.

Thank you again for encouraging us to address these concerns directly in the manuscript.

Reviewer_sxPwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Warp-As-History, a method for camera-controlled video generation. Previous works either (i) train existing and/or additional model weights on a camera-annotated video dataset or (ii) perform test-time optimization to control the video generation for a specific camera trajectory. While these methods either sacrifice large-scale training time or lengthy inference time, Warp-As-History does not require either. This work constructs a camera-warped pseudo-history to guide the video generation process in a zero-shot manner. Fine-tuning with one training video allows generalizable camera control.

Critical strengths
  • Novel idea: Injecting warping information as camera control in the history instead of with a learnable layer is interesting and preserves the existing model architecture.
Critical weaknesses
  • No proper or correct motion in videos: Looking at the supplementary videos, the method does not produce proper or correct motion. In the 30s jeep video, the motion is completely unnatural. The vehicle doesn't have correct vehicle dynamics, and as the camera moves closer, the car melts, deforms, and breaks down into blurry noise instead of moving correctly through the scene. Similarly, in the interactive angel demo, as soon as the camera turns and zooms, the character and scene distort heavily with no realistic motion. The model struggles with proper motion dynamics and consistency over time.
  • Artifacts for dynamic objects: In the comparison videos with ViewCrafter, GEN3C, and Hunyuan (like the skater, dog, and swan), the method often copies the warping border artifacts and there is ghosting or weird motion when the objects move. The rigid warping doesn’t work with dynamic objects.
  • Scaling analysis does not show clear improvement: The authors added a sweep from 1 to 12 videos (Tables 6–8), but there is no clear trend that more videos help. On RE10K (Table 8), using 5 or 12 videos actually makes FID and FVD worse. So it is not clear if fine-tuning with 1 video is actually a general recipe or if it is just very brittle to which video is used.
  • Inference overhead and reliance on 3D model: The method relies completely on an external reconstruction model (pi3) to render the warps, so reconstruction mistakes are copied into the video. Also, Table 15 shows that sampling time increases by over 7.5 seconds per chunk, which is a big slowdown for something that claims to be interactive.
  • No supplementary website: There is only a zip with separate videos and no website to compare the results side by side with baselines.
Questions for authors

The paper proposes a novel idea how to use warping conditioning for camera-controlled video generation. However, the visual results are still not that great and have noticeable issues.

I would like authors to address following questions:

  • In the 30s jeep video, why is there no proper or correct motion, and why does the car melt and deform so badly instead of moving naturally?
  • In dynamic scenes like the skater and dog, why does the method produce ghosting and warping border artifacts?
  • Why does scaling the training data from 1 to 12 videos not show a clear monotonic improvement, and why does it make RE10K metrics worse?
  • Table 15 shows an extra 7.5s sampling time per chunk. How does this work for real-time interactive generation?

For now, I am negative and lean towards rejection, but happy to reconsider my score based on a rebuttal.

3 comments
AuthorsOpenReview ↗

Response to Reviewer sxPw

Thank you for carefully examining our supplementary videos. Your comments also reveal unclear video labeling and insufficient separation of our inference configurations and small-data conclusions. We apologize for these presentation issues and clarify each point below.

Jeep video. The 30-second comparison contains three results: “Ours” on the left, WorldPlay in the middle, and LingBot on the right. The severe vehicle melting/deformation described in the review primarily occurs in the middle WorldPlay result; LingBot also degrades substantially over time, whereas our result remains relatively stable. Because method names are mainly embedded in the video and Figure 6 cannot show the full rollout, the results could easily be misattributed. This is our presentation shortcoming. We will label each result prominently and state its ordering in the captions and supplementary descriptions.

Angel video. We agree that visible distortion remains. This interactive demo replaces the original VAE decoder with lightweight TAEHV for faster decoding; we have observed that this can reduce quality and exacerbate local distortions. Long-rollout identity drift and degradation are also existing limitations of the Helios base model, rather than effects that appear only after adding Warp-as-History. Thus, this demo reflects both the accelerated decoder’s quality–speed trade-off and base-model limitations, and should not be attributed entirely to camera control. We will state the configuration explicitly and acknowledge long-horizon subject preservation as a limitation.

Dynamic scenes and warp copying. We agree that zero-shot conditioning can over-copy rigid warps, treating dynamic foregrounds as rigid geometry and causing ghosting or boundary artifacts. This motivates one-video LoRA finetuning. The resulting model does not use the warp as a hard reconstruction target: it remains native, text-guided score sampling, with warp history providing soft geometric evidence. Finetuning teaches the model to balance reliable camera cues against its generative prior, allowing correction/completion of dynamic or occluded content. The training target remains the native video-generation objective, not the warp. We do not claim that all artifacts are eliminated, but will distinguish zero-shot over-copying from the one-shot model’s soft use of warp evidence.

Data scaling and source sensitivity. Tables 6–8 test sensitivity; they do not claim monotonic gains from more videos. In this low-resource post-training regime, LoRA is intended to activate camera-following behavior already supported by pretraining, not learn a new video distribution. Once one video with clear camera motion provides the warp-history correspondence, adding a few heterogeneous videos need not improve every metric and may alter the prior–control balance. We therefore conclude only that no consistent benefit is observed under the current recipe, not that more data cannot help elsewhere. Tables 9–11 already vary the source with a fixed recipe: sources with clear viewpoint changes and reliable warps generally work, whereas difficult cases such as water can produce unreliable warps. The key factor is camera evidence/warp quality, not simply video count.

Runtime. Table 15 profiles the standard, higher-quality A800 configuration, not the optimized single-H200 interactive configuration in Appendix G. The latter uses FlashAttention-3, patch_mid warp-history downsampling, fewer denoising steps, fast rendering/preprocessing, a resident autoregressive state, and TAEHV decoding. Thus, the 7.5-second A800 overhead does not measure the optimized H200 system. The “less than one second” statement refers only to history packing, not full generation; we will correct this wording and the configuration descriptions. We will release all inference code, configurations, and real-time scripts once double-blind constraints permit.

Project page. The MP4s already contain concatenated baseline comparisons, but inconsistent naming/labeling makes inspection difficult. We will provide a unified page with complete comparisons, explicit labels, camera trajectories, and clearly separated standard and real-time configurations.

Our contribution is a history-interface for camera control that preserves the pretrained generative prior, not a solution to every base-model long-horizon or dynamics limitation. We will revise the claims and presentation accordingly.

Thanks for the response. I still have concerns over the artifacts and will discuss with other reviewers what they think.

Thank you for considering our response and for continuing the discussion. We agree that visual artifacts remain, and we do not claim that they are fully solved.

We would only like to point to 165_1782639437.mp4 in the supplementary ZIP, which may provide broader visual context. It contains over 20 examples, each rolled out for more than 30 seconds. The first frames were generated using GPT-Image/Nano Banana, and the camera trajectories were procedurally randomized according to general exploration rules, rather than manually tuned for individual scenes. We hope these additional examples are useful for assessing the method’s behavior across diverse inputs. We will make the remaining artifacts and limitations explicit in the revision.

Reviewer_pwLvOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Warp-as-History, a method for camera-controlled video generation that routes camera-induced geometric warps through a pretrained video model's native history-conditioning pathway rather than through dedicated camera encoders, control branches, or test-time optimization. Given a target camera trajectory, the method reconstructs the scene, projects it to target cameras to produce a 2D warp video, and then inserts this warp into the model's history stream with two key modifications: (1) target-frame positional alignment (assigning warp tokens the RoPE indices of the target latents being denoised), and (2) visible-token selection (dropping warp tokens without valid source observations, leaving disocclusions to the generative prior).

The central claim is that this interface reveals a latent camera-following capability in frozen history-conditioned video models (zero-shot), and that LoRA finetuning on a single camera-annotated video stabilizes and improves this behavior, generalizing to unseen videos without test-time optimization. Experiments are conducted on WorldScore, RE10K, and DAVIS against Gen3C, Voyager, ViewCrafter, and HyWorldPlay, plus a cross-backbone transfer to HunyuanVideo-1.5/minWM.

Critical strengths

Conceptual reframing of camera control as history conditioning. The insight that pretrained history-conditioned models already contain a camera-follow prior, and that warps can be routed through the existing history interface with only positional alignment and visibility filtering, is elegant and removes the need for camera-specific modules or large-scale camera-annotated data. This is the paper's most consequential contribution and is well-supported by the zero-shot ablation (Table 4, Figure 5), where alignment and visibility dropping each produce clear, monotonic improvements over the naive warp baseline.

Extremely low data and compute footprint with cross-backbone transfer. One video, ~1 hour on a single A800, no test-time optimization, and the interface transfers to a second backbone (HunyuanVideo-1.5/minWM, Table 16) with large gains over a camera-trained PRoPE baseline. Even if the absolute numbers don't beat large-scale baselines, the data-efficiency result is a meaningful and practically relevant finding.

Critical weaknesses

The "competitive with SOTA" claim is not supported by the camera-following metrics. On RE10K, Ours (one-shot) has PSNR 17.15 vs Gen3C 20.10, R-Err 1.28 vs 0.62, T-Err 0.0454 vs 0.0158 — a substantial gap on the very metric (camera adherence) the method targets. On DAVIS, R-Err 2.97 vs Gen3C 2.24. The paper's framing emphasizes quality/consistency wins (FID/FVD/DOVER) while camera-following is consistently behind. This is consequential because the abstract and introduction ("competitive with recent state-of-the-art camera-control baselines... with comparable camera adherence") misrepresent the evidence; camera adherence is not comparable, it is worse. The paper would be stronger if it claimed "quality-preserving low-resource camera control" rather than parity.

Source-video selection and the "one-shot" framing. Appendix D shows the fixed car-roundabout source was chosen before the sweep, but Table 9 shows large variance across sources (mean rank 2.13 for train vs 10.88 for surf), and the best source (train) is not the one used. The method's headline result therefore depends on a source-selection procedure that is acknowledged but not characterized as part of the method. Combined with the non-monotonic few-shot scaling (Table 6: 1 video → 16.02 PSNR, 3 videos → 15.89, 5 → 15.04), the "one video is enough" claim is really "one well-chosen video is enough," which is a weaker and less clean story.

Questions for authors

Camera-following gap. Can you provide a direct comparison of R-Err/T-Err under an identical evaluation protocol (same Pi3X stride, same frame count, same subset) for Ours vs Gen3C on RE10K? The current table mixes protocols across methods (Appendix C notes "matched baseline outputs" but the RE10K stride-4 detail is only stated for camera metrics). If the gap persists under matched protocols, the "comparable camera adherence" claim should be revised.

Source selection as method component. Given Table 9's variance, what is the practical guidance for a user with a new scene and no source sweep? Is there a cheap diagnostic (e.g., the reported invisibility ratio / rotation statistics) that predicts which single video will work, and does using the best source (train) change the headline numbers enough to alter the comparison to baselines?

1 comment
AuthorsOpenReview ↗

Response to Reviewer pwLv

Thank you for the constructive comments. We agree with both major concerns: our camera-adherence claim was too strong, and our discussion of source-video sensitivity and practical source selection was insufficient. We respond to each point below.

Camera-following accuracy and the SOTA claim. We agree that the current results do not support “comparable camera adherence.” On RE10K, our R-Err/T-Err are 1.28/0.0454, compared with Gen3C’s 0.62/0.0158, and our PSNR is also lower. On DAVIS, our R-Err likewise trails Gen3C. Advantages in perceptual quality or consistency should not obscure this camera-accuracy gap. We will remove “comparable camera adherence” and reposition the paper as quality-preserving, low-resource camera control. Our intended contribution is to activate effective cross-scene camera control through the native history interface, using one training video, no camera-specific control branch, and no test-time optimization—not to claim parity with Gen3C’s absolute camera accuracy after large-scale training.

“One video is enough” and source sensitivity. We agree that this should not be interpreted as “any arbitrary video works equally well.” Source videos differ in camera motion, foreground dynamics, and reconstruction difficulty, which can affect finetuning. The more precise claim is that one source video with clear camera motion and a reasonably reliable warp can suffice to activate and stabilize camera following on unseen test videos, while source quality still affects performance. Tables 9–11 are intended to analyze this sensitivity rather than present only the best source. We will revise the text to distinguish “one suitable video can suffice” from “all videos are equivalent,” and discuss source variance and failure cases more directly.

Evaluation protocol for Ours vs. Gen3C. We confirm that the existing RE10K comparison already uses an identical protocol: the same 99 sequences, 33 frames, Pi3X frame stride 4, and the same camera/DOVER/VBench evaluation procedure. Thus, the Table 2 gap is not caused by different subsets, frame counts, or strides. We apologize that the appendix did not state this prominently enough. We will place these protocol details next to Table 2 and revise the camera-adherence claim as described above, rather than attributing the gap to evaluation differences.

Practical source selection. Car-roundabout was selected directly during our initial experiments, without comparing downstream performance across candidate videos or selecting the best among multiple LoRA checkpoints. It simply contained clear camera motion and was then fixed for the main experiments. The source sweep in Tables 9–11 was conducted only afterward; we did not use it to replace the main source retrospectively.

The subsequent sweep shows that most sources with clear viewpoint changes, visible parallax, and reasonably reliable warps can activate effective camera control. Most viable sources produce results within comparable ranges, without a consistent source ordering across all metrics. Thus, the main result does not depend on a uniquely optimal source found through extensive search. Nevertheless, sources such as water, involving large non-rigid motion, reflections, or reconstruction failures, can yield unreliable warps and substantially degrade finetuning.

For users unable to run a source sweep, our current guidance is a simple manual heuristic, not a validated automatic predictor: choose clips with clear camera translation/rotation and parallax; avoid nearly static cameras; prefer stable reconstruction, moderate disocclusion, and clean warp boundaries; and avoid large water/reflection regions, severe non-rigid foreground motion, or obvious reconstruction failures.

Finally, Tables 10–11 already contain the best-performing train source. Relative to car-roundabout, it improves some metrics—for example, DAVIS R-Err decreases from 3.03 to 2.72 and RE10K LPIPS from 0.2353 to 0.2282—but RE10K T-Err slightly increases from 0.0456 to 0.0481, and camera accuracy remains behind Gen3C. Hence, using the best source does not alter the overall comparison or provide evidence that the headline result depended on post-hoc source selection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.