acceptodds
Under review as a conference paper at ICLR 2027

Predictability in Video Diffusion: Early Attention Predicts Visual Failures

Abstract

Text-to-video diffusion models require costly multi-step denoising, yet generation failures are usually identified only after the final video is complete. We find that intermediate video-to-text attention contains early information about whether individual visual constraints will be satisfied. Motivated by this insight, we introduce VideoProbe, a lightweight spatiotemporal probe that predicts token-level failures using only intermediate cross-attention and the denoising timestep. We also construct AttnPulse-10K, a benchmark that aligns multi-step attention from 10,000 Wan2.2 trajectories with uncertainty-masked token supervision derived from the completed videos. On Wan2.2, a single probe trained across nine denoising steps achieves mean AUPRC and generalizes without retraining to unseen denoising steps, including the first denoiser forward ( AUPRC). Shuffling attention across samples reduces AUPRC to , supporting the conclusion that matched attention carries predictive information. Independent experiments on Wan2.1 and HunyuanVideo-1.5 further show that early attention predicts final failures across different video generators and attention architectures. These early token-level predictions could support seed routing, prompt refinement alerts, and RL rollout mining.

Primary area:
generative models
Keywords:
video diffusion models, text-to-video generation, early failure prediction, cross-attention probing, prompt-specific diagnosis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Zt7WOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether visual failures in text-to-video diffusion models can be predicted before generation is complete. The authors find that intermediate video-to-text attention patterns contain early signals of whether individual prompt constraints will be satisfied, with successful constraints generally exhibiting more concentrated spatial attention. Based on this insight, they introduce VideoProbe, a lightweight spatiotemporal model that predicts token-level failures from intermediate attention and denoising timesteps, alongside AttnPulse-10K, a benchmark of 10,000 generation trajectories. Experiments across Wan2.2, Wan2.1, and HunyuanVideo-1.5 demonstrate consistent early predictability, achieving 0.340 AUPRC on Wan2.2 and maintaining performance at unseen denoising steps, including the very first step.

Critical strengths
  1. This paper propose a novel setting: to identify whether a generation would fail during the diffusion process;
  2. The whole story is complete: from proposing a new problem, prior-analysis, to formulate a method to solve this, and experiments to prove the method can work;
Critical weaknesses
  1. The biggest weakness is the paper only identify the failure, but fail to provide a method to rectify this;
Questions for authors
  1. Can authors clarify why identify the failure during the diffusion is improtant? (to stop early, have another try?)
  2. The methods need to re-train for different models? Otherwise, switching different diffusion models, need to retrain this prob-network would not be efficient.
Reviewer_5HRmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

1. A clear question with several sensible design choices. Attention-only input without token identity, a cross-trajectory shuffle control, evaluation at unseen steps, three generators with two attention architectures, and a carefully specified labelling schema (dependency rules, uncertainty masking, deterministic validation). The planned release of AttnPulse-10K with frozen splits and raw evaluator outputs is a genuine resource.

2. Honest reporting. The authors state plainly that dynamic constraints are near chance and that no downstream benefit has been measured.

Critical strengths

1. A clear question with several sensible design choices. Attention-only input without token identity, a cross-trajectory shuffle control, evaluation at unseen steps, three generators with two attention architectures, and a carefully specified labelling schema (dependency rules, uncertainty masking, deterministic validation). The planned release of AttnPulse-10K with frozen splits and raw evaluator outputs is a genuine resource.

2. Honest reporting. The authors state plainly that dynamic constraints are near chance and that no downstream benefit has been measured.

Critical weaknesses

1. The central claim is not separated from a simpler explanation: attention geometry encodes which token it is, and prompt identity predicts failure. The paper frames the result as attention carrying early information about the trajectory's outcome, but several of its own observations point to prompt-side priors. (a) Performance is essentially flat from step 0 (the input is pure noise) to step 49 (AUPRC 0.32 / 0.35 / 0.32, Table 5), and the independently trained per-step probes in Table 13 are best at step 0; if the probe read evolving trajectory state, later steps should be more informative. (b) In the pilot (Table 14) a text-only model reaches 0.49 AUPRC / 0.85 AUROC versus 0.27 / 0.67 for attention-only, and adding attention to text hurts. (c) AttnPulse-10K has one trajectory per prompt (10,000 trajectories from 1,000 + 3,000 + 6,000 prompts), so the cross-trajectory shuffle destroys prompt identity together with trajectory state and cannot distinguish the two hypotheses. The decisive experiments are missing: a text-only and text+attention baseline on the main benchmark, and above all a within-prompt test (several seeds per prompt; predict which seed fails, or shuffle attention among seeds of the same prompt). The motivating application, seed routing, is exactly the within-prompt problem, and nothing in the paper measures it. Until this is resolved, "early attention predicts visual failures" may largely mean "the prompt predicts visual failures".

2. Unvalidated supervision and modest utility. All labels come from a single VLM at temperature 0; no agreement with human annotators is reported beyond four hand-picked examples. Since the benchmark is a stated contribution, the label noise ceiling must be measured. In terms of utility, AUPRC 0.34 at prevalence 0.147 (F1+ 0.42, IoU 0.30) and chance-level AUROC (0.50 to 0.54) for action/motion/interaction/spatial constraints leave unclear which decisions the signal could support; the three applications are only sketched.

Questions for authors

Q1. How many unique prompts and how many seeds per prompt are in each of the three datasets? If it is one trajectory per prompt, please add a within-prompt experiment (e.g. 300 to 500 prompts x 4 seeds): can the probe rank seeds of the same prompt by failure risk, and what is AUPRC when attention is shuffled only among seeds of the same prompt?

Q2. Please report text-only and text+attention baselines on the full AttnPulse-10K test split under the main protocol. Does attention add information beyond the prompt, and how much?

Q3. Please report human-Gemini agreement on a random sample of constraints (e.g. 300), and probe performance against the human labels.

Q4. Why is step-0 attention, computed from pure Gaussian noise and the prompt, as predictive as step 12? What in the step-0 attention map varies across trajectories other than prompt and noise?

Q5. What fraction of constraints is labelled "uncertain" per category, and does excluding them bias the test set toward easy cases?

Q6. All experiments use 50-step sampling. Since deployed fast generators typically run 4 to 8 distilled few steps, does the signal exist at step 1 or 2 of a few-step Wan variant, and what compute saving would seed routing offer in that regime?

Reviewer_DQhYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates early failure prediction in text-to-video diffusion models. It finds that video-to-text attention during early denoising already contains information about whether the final video will satisfy specific visual constraints. Based on this observation, the paper introduces VideoProbe, a lightweight spatiotemporal probe that uses intermediate cross-attention and the denoising timestep to predict failures associated with individual prompt tokens.

Critical strengths
  • The research problem is practically important, as identifying failures early in the denoising process could reduce wasted inference computation.
  • VideoProbe provides token-level predictions, enabling it to identify which object, attribute, numerical, or action constraint is likely to fail and offering more fine-grained diagnostic information.
Critical weaknesses
  • The methodological novelty is limited, as the work primarily extends cross-attention-based probing from 2D image generation to spatiotemporal video generation.
  • The annotations rely on Gemini 3.1 Pro Preview, without human validation of label reliability.
  • Experiments mainly cover early architectures such as Wan2.2, while newer models such as LTX, FLUX, and MiniMax H3 are not evaluated.
  • The probe does not generalize across model architectures; each video generator requires a separately trained VideoProbe.
  • On Wan2.2, the mean AUPRC is approximately 0.34. Although this improves over the 0.147 baseline, the performance remains insufficiently reliable.

Minor concerns

  • Some results are strongly affected by class distribution. For example, dynamic attributes achieve an AUPRC of 0.9675, but their failure rate is already 95%, indicating little improvement over predicting every instance as a failure.
  • The paper does not directly quantify how much computation VideoProbe saves, how much it improves generation success rates, or how many potentially successful trajectories would be incorrectly terminated.
Questions for authors

see Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.