- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies how small perturbations of the conditioning image affect long rollouts of distilled autoregressive image-to-video models. Native Self-Forcing shows growing sensitivity and HiAR is less sensitive but not robust. The authors fine-tune HiAR for 200 updates with "teacher-guided on-policy adversarial fine-tuning": the student rolls out on an adversarially perturbed image, and a frozen copy of the original model, given the same query and raw history but the clean image, provides a per-node prediction target. A cheap 7-block single-step surrogate generates attacks, refreshed every six updates, and one in three updates uses clean inputs to anchor normal behavior. Against model-specific white-box PGD, mean latent response falls 22–31%; VBench-I2V scores under attack improve. A control that instead uses the teacher's independent clean-trajectory prediction lowers response further but blurs outputs badly.
Critical strengths
- The context-matched vs clean-trajectory target control is the most valuable finding. It shows that a lower sensitivity metric can hide a large quality loss, a useful caution for robustness work on generators.
- Evaluation pipeline is well designed: own-model attacks, PGD40 and late-target PGD10 in addition to PGD20, a held-out confirmation set disjoint by parent and frame hash, paired whole-source bootstrap intervals, and explicit reporting of a source that gets worse.
Critical weaknesses
- The threat model is not motivated. Who perturbs the conditioning image, and why? In practice, imperceptible perturbations of I2V inputs are mostly used defensively to stop misuse of personal photos. Robustness to them weakens those protections; this is a dual-use issue the ethics statement only touches.
- The fine-tuned model drifts substantially on clean inputs. Its clean drift N = 0.49 exceeds the original model's response to random 8/255 noise, and random-input Q rises 39%. Lower sensitivity may partly come from relying less on fine image detail rather than from true robustness.
- Scope is narrow: one base model , one data domain, one threat model, one training seed, and attacks that maximize latent MSE only. Attacks are not converged. Fig. 6 shows the PGD objective still rising at 40 steps, so the reported reductions are finite-budget numbers.
- Simple defenses are missing as baselines: input purification (JPEG, blur, diffusion purification) or standard adversarial training with a clean-output target.
- The paper is very hard to read. Dense, nonstandard terms ("source eviction", "parent-disjoint", "raw-state construction", "causal replay metadata", "ancestors of inherited cache entries") appear without definitions, and Sec. 2 needs several passes.
Questions for authors
- Threat model and dual use. Please describe a concrete scenario in which an adversary perturbs the user's own conditioning image. How does your fine-tuned model change the effectiveness of protective perturbations such as I2VGuard or Anti-I2V? If it neutralizes them, this belongs in the main paper, not a single ethics sentence.
- Robustness vs ignoring the image. Does the adapted model still respond correctly to meaningful image edits? A test of sensitivity to semantic edits would separate invariance to imperceptible noise from reduced use of the conditioning image.
- VBench-I2V numbers. The clean scores of original HiAR in Table 2 are exactly round while every other cell has finer precision. Were these computed in your setup with the same seeds as "Ours", or copied from elsewhere?
- Attack strength. Since PGD has not converged at 40 steps, please report results with many more steps, multiple restarts, and EOT over generation noise. Also try attacks with other objectives, e.g., targeted changes or VBench-quality degradation.
2 comments
Thank you again for your suggestions. We have further evaluated our on-policy adversarial fine-tuning (OAT) on Self-Forcing, adding evidence beyond HiAR and measuring subject fidelity alongside latent response.
We initialized the student from the official Self-Forcing DMD checkpoint (self_forcing_dmd.pt, EMA weights) and used an independent frozen copy as the teacher. We used Self-Forcing's blockwise autoregressive generation while retaining OAT's objective and fixed 200-update budget. A matched random-perturbation control used the same initialization, training data, update count, teacher-target construction, and optimizer, replacing only the adversarial training perturbations with random perturbations at the same budget.
We evaluated 16 image sources with two generation seeds each (32 videos per model and input condition), using paired prompts/noise and approximately 20-second videos. We averaged the two seeds within each source, then averaged across sources. Each model received its own prefix PGD20 attack at an ℓ∞ budget of 8/255 and step size 1/255, maximizing latent MSE on blocks 1–7 relative to its own clean output with the same prompt and noise. Full/late S used the same clean reference over blocks 1–27/21–27; the late result therefore measures the long-term effect of the prefix attack. Subject similarity is the DINOv2 cosine similarity between the tracked video subject and the input subject.
| Model | Full S ↓ | Late S ↓ | Clean late subject similarity ↑ | Attacked late subject similarity ↑ |
|---|---|---|---|---|
| Self-Forcing | 2.148 | 2.782 | 0.659 | 0.542 |
| + random-perturbation training | 2.019 | 2.651 | 0.650 | 0.547 |
| + OAT | 1.697 | 2.327 | 0.638 | 0.558 |
OAT reduced full/late S by 21.0%/16.4% relative to Self-Forcing, and by 15.9%/12.2% relative to matched random-perturbation training. The attack-induced drop in late subject similarity was 0.117 for Self-Forcing, 0.103 for random training, and 0.080 for OAT. Thus, OAT achieved both a smaller attack-induced drop and higher subject similarity under attack, beyond the gains from random perturbation augmentation.
OAT's clean late subject similarity was 0.021 lower than Self-Forcing's, while its attacked similarity was 0.016 higher. The lower latent response therefore accompanies better subject fidelity under attack, with a measurable clean-fidelity cost. Visual comparisons showed reduced road-texture distortion and better retention of some clothing attributes, although long-video content errors remained.
Our earlier response also reported semantic-edit tests, purification comparisons, and stronger attacks on HiAR: OAT retained a 25.77% reduction in S with PGD200 plus two PGD100 restarts, and a 28.57% reduction with generation-noise EOT evaluated on held-out noise draws. The new Self-Forcing experiment further broadens the model coverage.
Due to the time required for a full VBench-I2V evaluation, complete scores for these Self-Forcing experiments are not yet available in this rebuttal; we will report them in the revised manuscript.
We will include the additional results and observed trade-offs in the revision. We would appreciate your updated assessment in light of these results and our earlier response.
W1/Q1. Motivation and dual use
We consider video generation from authorized product or character images. A third party handling these images could deliberately make small pixel changes that alter the generated subject, requiring regeneration or manual review. OAT ('Ours' in the paper) aims to keep the video faithful to the reference image. It may also weaken perturbations added by image owners to prevent unauthorized animation. We will clarify this scenario and discuss dual use in the revision.
W2/Q2. Meaningful image edits
Yes. OAT still follows edits to the input image. We tested 16 images (eight color/shape edits and eight object additions/removals), two seeds per model/image, and ~20-second videos. Original/edited inputs shared prompts and noise, and prompts did not reveal edits. For color/shape edits, DINOv2 similarity of tracked subjects to the edited reference rose from 0.746 to 0.814 (+0.068) for HiAR and 0.738 to 0.803 (+0.065) for OAT. Clothing colors and shapes followed edits throughout the videos. Removing a green bowl produced four bowls in place of five, with the bowl staying absent and other colors/layout largely preserved.
W3. Seeds and perturbation budgets
We reproduced the improvement in all three training runs. We repeated training with two new seeds, using the same HiAR weights and fixed 200-update recipe. On 160 confirmation sources, under each checkpoint's own prefix PGD20 (20-step projected-gradient descent), OAT reduced full/late latent response S by 22.11% ± 2.11 / 20.16% ± 2.25 percentage points (mean ± sample standard deviation across three training seeds).
OAT also reduced attack-induced changes at smaller and larger perturbation budgets. On 80 development sources, the fixed OAT checkpoint trained at 8/255 reduced full/late S by 33.17%/28.79% at 4/255, 30.20%/26.47% at 8/255, and 19.30%/15.17% at 16/255. Model-specific prefix PGD20 used steps of 1/255, 1/255, and 2/255, with paired prompts/noise. Prefix attacks target blocks 1–7, with full/late evaluation on blocks 1–27/21–27.
W4. Purification and target choice
All five preprocessing methods helped, and OAT did better at preserving the subject under attack. We re-optimized PGD20 through each full pipeline, targeting blocks 21–27 at ℓ∞ = 8/255. JPEG70 and diffusion used true forward passes and identity BPDA. Resizing, blur and noise+upscaling used direct gradients. Noise+upscaling and diffusion used four-draw EOT with fixed purifier noise. Late-segment DINOv2 distances to the input subject were 0.247 for JPEG, 0.261 for resizing, 0.270 for blur, 0.266 for noise plus upscaling, and 0.235 for diffusion purification, versus 0.294 for HiAR and 0.218 for OAT. JPEG+OAT further reduced it to 0.209, with clean distance changing from 0.192 to 0.195.
Matching teacher targets to the student's current context helped preserve detail. With an independent clean-trajectory target, S fell from 1.3539 to 0.9500, but clean/attacked MUSIQ image-quality scores also fell from 75.69/76.26 to 55.81/48.59, with blur or ghosting in later frames.
Q3. Table 2 provenance and precision
All four rows were computed in our own evaluation under the official VBench-I2V protocol, with paired seeds across HiAR/OAT and clean/attacked conditions. We generated five ~20-second videos per image–prompt pair, using the official image subsets for each metric. Each model received its own PGD20 under the same budget and procedure.
The scoring procedure can produce round numbers. We display two decimal places. For example, Dynamic Degree is the fraction of videos classified as having motion. With 246 inputs × five videos, 50.00%, 49.19%, 40.98%, and 48.05% are arithmetically consistent with 615, 605, 504, and 591 out of 1230 after rounding. Trailing zeros also occur in continuous-metric entries in Ours clean and HiAR attacked rows.
W3/Q4. Stronger attacks and generation randomness
OAT still produced smaller output changes when we used more attack steps and random restarts. On 16 fixed development sources, both fixed checkpoints received their own prefix PGD200 plus two independent PGD100 restarts at 8/255 with step size 1/255. For each source and model, we selected the quantized candidate with the highest prefix objective across all three trajectories and evaluated all 27 blocks. OAT's S reduction on these sources was 29.30% with PGD20, 26.89% with PGD200, and 25.77% including restarts (HiAR/OAT S = 3.4827/2.5851).
OAT also reduced output changes when we evaluated with fresh generation noise. EOT (Expectation over Transformation)-PGD20 averaged each model's prefix objective over four generation-noise draws and evaluated four separate held-out draws. We compared each attacked video with a clean video from the same model and noise draw. Full-video S was 0.9981 for HiAR and 0.7129 for OAT (28.57% lower).
W5. Clarity
We apologize for the unclear terminology and thank you for pointing it out.