acceptodds
Under review as a conference paper at ICLR 2027

Dual-Support Online Policy Optimization for Few-Step Autoregressive Video Models

Abstract

Few-step causal autoregressive (AR) video models enable efficient streaming generation through timestep distillation and KV-cached causal inference, making online optimization directly on the final deployed model increasingly practical. However, post-distillation optimization introduces a deployment-support mismatch: the training updates may be estimated at timesteps and causal contexts that differ from those actually encountered by the few-step streaming policy at inference. We identify two complementary sources of this mismatch. The distilled timestep support is the discrete timestep set retained by the few-step sampler, while the streaming context support is the distribution of causal contexts induced by the model's own autoregressive rollout. Based on this observation, we propose a dual-support online policy optimization framework that performs relative forward-process updates only on distilled timesteps and optimizes candidate continuations under policy-generated long-horizon contexts. Historical KV states are detached so that optimization remains local while context exposure remains long-horizon. Across multiple distilled AR video backbones and generation horizons, our method consistently improves preference-oriented quality and motion dynamics while preserving the original few-step inference procedure. Our results show that effective post-distillation alignment depends not only on the optimization objective, but also on matching the state distribution encountered by the final generative policy at inference.

Primary area:
generative models
Keywords:
Autoregressive Video Generation, Policy Optimization, Few-Step Video Diffusion

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_1REmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on deployment-support mismatch; one side related to training on timesteps that 4-step sampler (due to distilled model) can't encounter, and one side related to training under reset contexts. The propose a dual-support online policy optimization method that performs relative forward-process updates on distilled timesteps and optimizes continuations of the candidates under policy-generated contexts in long-horizon.

Critical strengths

Experiments are sufficiently detailed; including ablations such as in Table 5, we can clearly see the ablation of both components that contributes independently, supporting author's hypothesis that they based the paper on.

Matched-budget comparison in Table 4 also further supports the author's claims.

Critical weaknesses

Table S3 says "These values are predicted placeholders and must be replaced by measured results before submission.".

This is absolutely unacceptable. The reviewers cannot know if those results are predicted placeholders or real results; and whether any other table has such errors without explicitly noting placeholder warning.

Questions for authors

Are the results in Table S3 real or spaceholder? And are there any other tables or results that we need to be aware of in terms of placeholders?

Reviewer_SHLfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies reward-based post-training of timestep-distilled causal autoregressive video generators. Its central claim is that optimization should account for two deployment-specific distributions: the small set of denoising timesteps retained by the few-step sampler, and the causal contexts produced by the model's own streaming rollout.

The method applies a DiffusionNFT-style relative forward-regression loss only at retained timesteps. It samples candidate continuations under a shared prefix generated by a lagged behavior policy, detaches historical KV states, and updates the active clip or window. HPSv3 and VideoAlign provide visual, motion, and text-alignment rewards; a disagreement-gated velocity-space reference penalty stabilizes optimization.

Experiments show higher training-reward scores and several independent VBench improvements while keeping the deployed few-step inference procedure.

Critical strengths
  1. The paper tests a useful deployment-aware design with a direct two-factor ablation. Table 5 compares dense versus distilled timesteps and reset versus streaming contexts. VBench-Long rises from 82.95 for dense timesteps with reset contexts to 83.43 with timestep restriction alone, 83.58 with streaming contexts alone, and 84.24 with both. This supports the practical value of both components. Improvements on multiple backbones and generation horizons make the result potentially useful beyond one checkpoint.
  2. The approach preserves efficient inference and makes the training trade-off explicit. Detached historical KV states permit local updates under contexts accumulated over a long rollout. Table 5 reports peak-memory values of 96.3 for clip-wise backpropagation and 54.4 for shared-context sampling with detached history, with closely matched reported quality. Table 1 reports unchanged inference throughput for each paired backbone. These are useful implementation details for practical streaming-model alignment.
Critical weaknesses
  1. The evidence for improved human preference and long-term consistency is incomplete, and the magnitude of some independent gains is uncertain. HPSv3 and VideoAlign are training rewards, so improvement on these metrics does not independently establish better human judgments. Table S3 shows provisional placeholders rather than measured results. The long-horizon claim also needs qualification: Table 3's Causal Forcing consistency score decreases from 95.88 to 95.63, and the paper itself notes that clip-wise CLIP does not establish global consistency. Consequently, the current evidence supports reward alignment and some benchmark improvements more strongly than broad human-preference or temporal-consistency claims.
Questions for authors
  1. Independent validation: Has the planned human evaluation actually been completed? If so, please provide measured results, the exact baseline and video horizon, evaluation instructions, and prompt-level uncertainty. Otherwise, please remove Table S3's predicted results and state clearly which human-preference claims are supported only by reward-model proxies. This would directly affect my assessment of the empirical contribution.
  2. Implemented objective: Which reward produces the advantages in the reported experiments: the equally weighted aggregate in Section 3.4 or the primary reward in Algorithm 1? Are reward components normalized before aggregation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.