acceptodds
Withdrawn from ICLR 2027

First-Frame-Guided Video Editing with a Visual Autoregressive Model

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

While text-to-video generative models have advanced video editing, maintaining spatiotemporal consistency in direct video manipulation remains challenging. As a practical alternative to direct video editing, first-frame guided video editing has emerged as a new paradigm, which leverages the superior fidelity of image editing by propagating edited attributes from the first frame to the full sequence. Existing first-frame guided video editing methods are primarily built on diffusion and flow matching models, which struggle to disentangle appearance from motion. To tackle this problem, we propose a novel framework that explicitly exploits the inherent inductive bias of visual autoregressive (VAR) models. Unlike diffusion and flow-matching models that adapt appearance and motion on a shared fixed-resolution latent space, VAR exposes an explicit coarse-to-fine hierarchy, making motion easier to disentangle from the appearance during adaptation. Exploiting this structure, we restrict motion adaptation to low scales while preserving high-scale appearance refinement, achieving high-fidelity first-frame guided video editing with robust source motion preservation. Extensive evaluations demonstrate that our approach achieves leading input-frame consistency while maintaining strong source-motion preservation across both local and global edits.

Primary area:
generative models
Keywords:
first-frame-guided video editing, visual autoregressive model, generative model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uSHVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a video propagation (first-frame-guided video editing) method based on a VAR (specifically InfinityStar in the implementations). The proposed LoRA-based approach is trained in two stages, where Stage 1 trains an Appearance LoRA in the T2I setting and Stage 2 trains a Motion LoRA in the I2V setting with Appearance LoRA frozen (the first 6 resolution scales are used based on a motion reconstruction analysis). In addition to the BCE loss, a flow-guided regularization term using optical flow is employed to improve motion in the generated video. For experiments, comparisons (quantitative, qualitative and a user study) and an ablation study is provided.

Critical strengths

The idea of disentangling to two LoRAs (Appearance LoRA and Motion LoRA) looks reasonable, though I am unconvinced by the results. I appreciate that video results were provided in the supplementary materials.

Critical weaknesses
  1. My biggest concern is the quality of the visual results. For example, the following regions in the supplementary videos (viewed with index.html) have very obvious high-frequency flickering artifacts that are (usually) not observed in other compared methods (though other methods may have other issues):
  • Ground region in both the top and bottom examples in the 7th row
  • Tree region in the 9th row
  • Tree region in the 10th row
  • Tree region in the 11th row
  • Gravel region in the 12th (last) row
    (Note that I counted each white block as a row)

Artifacts can be seen in the same samples in Fig 3 as well, for instance, in the tree region in the bus example on the top left and the tree region in the bike example on the bottom right (trees are unnatural and temporally inconsistent). These issues are especially important since the authors claim to overcome issues like temporal flickering (l081-083) of diffusion-based approaches. It also makes me wonder whether the proposed method is indeed working.

As a side note, the edited first frames in Fig 1 are also questionable: Edit #1 shows very different (overly smooth) water texture compared to the source video; Edit #2 does not look like impressionist style even though the text specifies impressionist style. I understand that the first frames are not generated by the proposed method, but I think this is still worth mentioning since they still influence the final results.

  1. Another major concern is the quantitative results: (i) In Table 2 for local edits, the proposed method lags behind other methods in terms of visual quality and temporal consistency metrics and only slightly outperforms the next best method in terms of subject and background input consistency; (ii) Ablation studies in Table 4, Table 5 and Table 6 do not really support the selected components (specifically, 3 out of 6 metrics support w/o regularization in Table 4; 4 out of 6 metrics support Joint Training in Table 5; 4 out of 6 metrics support k_c=9 in Table 6)

Also, since many commercial models (e.g., Runway Aleph 2) support video propagation, it would be useful to note their performance, though it's not necessary to beat them as an academic paper.

[Not critical but worth noting] Many references just link arxiv versions of the paper even though they have been accepted at peer-reviewed venues. The authors are suggested to carefully review each reference and note the correct venue.

Some examples are:

  • "Dynamic Concepts Personalization from Single Videos" was presented at SIGGRAPH 2025
  • "VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models" was published at TPAMI
  • "I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Models" was published at SIGGRAPH Asia 2024
Questions for authors

N/A

3 comments

Appreciate the authors providing a response. I have read the responses and went through the updated parts in the PDF revision as well as the supplementary material.

  1. Visual quality is still my biggest concern. In the provided "Backbone Comparison", although similar artifacts exist in the reconstruction results, the proposed method worsens the distortion and flickering in the tree regions. This is critical as the proposed method aims to handle temporal dynamics. Furthermore, the two examples provided as "Backbone Comparison" only cover one sample out of the five that I pointed out in my original comment. Moreover, if the original backbone has such artifacts, I would consider this as a wrong backbone choice since such artifacts are not usually observed in the other compared methods in the supplementary videos as mentioned in my original comment. It is hard to make any concrete judgements or analyses with such temporal quality. Also: When I checked the revised draft, the paper is still misleading regarding "temporal flickering" issues in existing models and that the proposed method overcomes such issues. My comment regarding the first frame edits in Fig 1 has not been addressed.

  2. Table 2: I believe visual quality and temporal consistency metrics are also important for evaluating video editing models. I acknowledge that a method does not need to beat all methods in every metric, however, the performance gaps on the Subject and Background consistency metrics (pointed out by the authors) are also small as mentioned in my original comment.

  3. Ablation study: The new Table 5 also raises similar concerns as the gain in motion fidelity seems to be at the expense of other quality criteria. The values 0.7100 and 0.7533 mentioned in the authors' comment does not appear in Table 5. Also, I believe the authors mean Table 7 when they mention Table 6 in their comment.

Overall, I feel the submission is far from ready to be published and will keep my original score.

AuthorsOpenReview ↗

Response to Reviewer uSHV

Thank you for taking the time to read our rebuttal and for providing further detailed feedback. We deeply respect your rigorous evaluation.

Regarding your remaining concerns, we would like to provide a brief final clarification along with additional user study results:

1. Visual Quality and Flickering: We appreciate your check. To provide a brief technical clarification regarding the backbone comparison: the relatively clear regions in the backbone's output are simply ground-truth (GT) tokens passed directly through the VAE without any inference. In contrast, the regions where actual inference takes place (e.g., k=0,k=6k=0, k=6) are severely degraded in the backbone itself, to the point of being visually indistinguishable. Thus, the artifacts and flickering are inherent limitations of the backbone's inference process, rather than new distortions introduced by our proposed method. Nevertheless, we fully respect your feedback. Since our framework relies on this backbone, the final output still suffers from these artifacts. We agree that claiming to have "overcome" temporal flickering was misleading. Following your advice, we will revise our manuscript to remove these claims and explicitly discuss these visual distortions and backbone dependencies as a primary limitation.

2. Metrics (Table 2): We respect your perspective on the marginal performance gaps in certain metrics. However, as can be observed from the other baselines in Table 2, we believe the field has reached a stage where it is highly uncommon for a single method to overwhelmingly dominate every evaluation metric. Trade-offs are often inevitable. Rather than claiming absolute superiority across all metrics, our primary objective is to demonstrate the underlying potential of the Visual Autoregressive (VAR) framework. We believe our results show that VAR can achieve a highly competitive and well-balanced performance, proving its viability as a strong alternative to the currently dominant diffusion-based methods.

3. Ablation Study & Typos: We sincerely apologize for the confusion. During the revision process, adding a new table caused the subsequent table numbers to shift, and we unfortunately failed to reflect this updated numbering in our previous comment (mistakenly referring to Table 6 instead of Table 7). The specific values (0.7100 and 0.7533) were cited from an internal log rather than the final Table 5. We appreciate you catching this oversight.

4. Additional User Study on Visual Quality: To further address your concerns regarding overall visual quality, we have conducted an additional human evaluation focusing exclusively on this aspect. We are currently recruiting 20 participants and present the interim average rankings from the 15 participants who have completed the survey (lower rank is better):

MethodLocal Editing (Scale 1~5) ↓Global Editing (Scale 1~3) ↓
Ours1.971.29
LoRAEdit2.01-
I2VEdit4.212.33
GWTF2.962.37
VACE3.84-

While human evaluators likely noticed the temporal flickering in our results, they consistently ranked our method higher. This is because baseline methods often suffered from more severe degradation, such as heavy structural artifacts or color saturation. Because our approach is elegantly simple—utilizing LoRA without requiring external masks or heavy auxiliary signals—it introduces very little additional corruption to the source. The remaining temporal flickering is largely bottlenecked by the backbone; just as early diffusion models rapidly improved in quality over successive generations, we firmly believe the visual quality and temporal stability of our framework will naturally scale and improve as underlying VAR architectures advance.

We deeply appreciate your rigorous critiques, which have clearly highlighted the limitations of our current approach and helped us significantly improve the manuscript. We hope these final technical clarifications and the new user study results alleviate your remaining concerns. In light of our commitment to transparently reflecting these discussions in the final version, we respectfully ask if you might reconsider your final evaluation.

Thank you again for your time and constructive engagement.

Appreciate the authors providing an additional user study. I agree that the quality of VAR backbones will improve over time. However, the results of LiveEdit (https://live-edit.github.io/) and EditStream (https://real-time-video-research.github.io/editstream/), which are all autoregressive video editing models, have much better visual quality and do not exhibit such significant flickering. Note that EditStream also handles edit propagation (first-frame guided video editing) like this paper. One difference is that both models train from a bidirectional DiT and distill the model into an autoregressive student. I believe the authors' method could potentially benefit from such approaches of building on top of stronger video editing backbones. (I do note that this is not really a "fair" comparison since the tested videos are different, but a high-level visual inspection.)

Furthermore, I realize that such VAR-based video editing models are not analyzed in the current paper (in the introduction, related work, or experiments), although they are very (perhaps the most) related to this work. Hence, after taking another look at the submission, I do not feel that this work is placed in the right context.

Reviewer_VKJhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work addresses first-frame-guided video editing, where edits applied to the first frame are propagated across the video sequence while preserving the original source motion. Existing diffusion and flow-matching methods struggle to disentangle appearance and motion within a fixed-resolution latent space. The authors leverage the native coarse-to-fine hierarchical structure of Visual Autoregressive (VAR) models: motion adaptation is restricted to coarse low-scale token layers, while fine high-scale layers handle appearance refinement. They adopt disentangled appearance and motion LoRAs for this separation. Evaluations include quantitative metrics, ablation studies, and human user studies, demonstrating competitive first-frame consistency and source-motion preservation on both local and global editing tasks.

Critical strengths
  1. The work empirically validates VAR as an alternative backbone for first-frame-guided video editing, reporting competitive quantitative, qualitative and user-study results against diffusion baselines.
  2. The coarse-to-fine hierarchy of VAR is used to separate motion and appearance adaptation by restricting motion learning to coarse layers. This architecture-inspired disentanglement strategy is conceptually clear.
Critical weaknesses
  1. Novelty is limited. The paradigm of first-frame editing with disentangled appearance/motion LoRAs already exists in diffusion-based prior work. The primary change is switching the model backbone to VAR; no fundamentally new editing mechanism is introduced.
  2. Presentation issues reduce clarity. Notation problems include reused k and undefined symbol c. No full inference pipeline flowchart is provided, only the LoRA training diagram. The presented qualitative outputs and supplementary video results are blurry, and the competing baseline LoRAEdit achieves better visual quality.
  3. Incomplete experimental validation weakens core claims. There is no ablation study for individual contributions of the two LoRAs. Evidence from Table 1 alone is insufficient to justify the k<6 selection. The user study in Table 3 excludes visual quality metrics, and marginal differences in Table 4 cannot reliably validate the regularization’s effectiveness.
Questions for authors
  1. Why is backward flow used in Eq.5 instead of forward flow?
  2. Table 1 relies on motion reconstruction to select k<6. Can you provide more evidence to validate this choice?
  3. Why did your user study only measure consistency and fidelity and omit visual quality?
  4. Table 4 presents marginal differences across ablations. How do you confirm the regularization yields statistically significant gains instead of random variance? Fig.4 can be cherry-pick, are the scenarios (without regularization) shown typical?
Reviewer_hgzTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method to handle first-frame-guided video editing via the visual autoregressive model. It utilizes the pre-trained VAR model, InfinityStar, to achieve the editing in a disentangled way. It first trains the appearance LoRA in the T2I stage using the source image, and then freeze the appearance LoRA, train the motion LoRA in the I2V stage using the source video. During inference, it injects the edited first frame into the T2I stage, disables the appearance LoRA, and injects the source video into the first 6 steps together with the edited first image information in the I2V stage to preserve motion. It achieves better appearance and motion disentanglement during editing with better performance.

Critical strengths

The paper uses the pre-trained 2-stage VAR model to handle first-frame-guided video editing, utilizing the appearance-motion disentanglement characteristics to achieve better editing performance. It conducts extensive experiments to prove its effectiveness and superior performance. The results shown in the supplement look good, which is the main reason for my current rating.

Critical weaknesses
  1. Novelty seems limited. First-frame-guided editing, disentanglement of motion and appearance, and LoRA-based edits are all old concepts. It seems that the main novelty of this paper is to apply these well-known concepts to the two-stage VAR, which has not been tried on this task before.

  2. Although the results shown in the supplement look good, the experiments in the paper are still insufficient to prove that the two-stage VAR achieves good motion-appearance disentanglement. Table 1. only shows the motion reconstruction analysis given the injection scales, but the appearance leakage remains uncertain. More experiments measuring the extent of appearance leakage should be added to enhance the paper.

Questions for authors

NA

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.