acceptodds
Withdrawn from ICLR 2027

FrameMorrow: Future-Guided Frame Selection with Prospective Tokens for Long-Horizon Video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-horizon video generation requires models to effectively leverage an increasingly long generation history. As the generated history grows, retaining all previous content becomes increasingly expensive and redundant, making effective historical selection essential. Existing approaches often determine historical relevance from the current content. However, information that is relevant to the present is not necessarily useful for future generation, while seemingly less relevant history may become important later. Our key insight is that historical information should be selected according to its relevance to future information needs. Capturing these needs does not require generating the full future. A compact representation of what becomes important next is sufficient to guide historical selection. Building on this insight, we propose FrameMorrow, a prospective frame selector that predicts a small set of prospective tokens representing future information needs and uses them to identify relevant information from history. FrameMorrow selects explicit historical frames rather than model-specific internal states, enabling plug-and-play integration across diverse generators, including closed-source models, with little additional inference cost. We evaluate FrameMorrow across five benchmarks and 11 generative models spanning long-video generation, interactive generation, and action-conditioned world models. Extensive experiments demonstrate improvements in long-range consistency, visual quality, and action alignment across diverse generation settings.

Primary area:
generative models
Keywords:
Autoregressive Video Generation, Frame Selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hq21OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FrameMorrow, a historical-frame selector for long-horizon video generation. Given the observed history, recent context, and currently available rollout condition, a lightweight causal Transformer autoregressively predicts a small set of prospective tokens. These tokens act as queries for ranking historical frames. During training, the selector learns from rankings derived from DINOv2 similarities between historical frames and realized future observations. Future observations are not required at inference. The selected frames are supplied to frozen generators through their existing reference or memory-conditioning mechanisms. The paper evaluates the approach across 11 generators spanning long-video generation, interactive generation, closed-source generation, and action-conditioned world models. It also provides selection baselines, design ablations, a diagnostic relating relevance scores to generation utility, and a user study.

Critical strengths

A practical selection interface with demonstrated reuse across generators.
FrameMorrow returns explicit historical frames rather than generator-specific internal states. The paper shares a selector checkpoint across compatible backbones within each condition modality and demonstrates integration with several frozen generators, including closed-source reference-conditioning interfaces. This provides meaningful evidence for the practical value of separating historical-frame selection from the generator.

Substantial evidence supporting the supervision strategy and design choices.
The paper compares selection strategies under matched historical candidates, frame budgets, and conditioning interfaces. Figure 6 examines future supervision and autoregressive versus parallel prospective queries.Appendix E.5 directly investigates whether future-grounded relevance corresponds to downstream generation utility: the predicted ranking achieves a Spearman correlation of 0.42, compared with 0.10 for current-context similarity. Although this diagnostic concerns individual-frame utility in a condition-free setting, it provides useful support for the proposed training proxy.

Critical weaknesses
  1. The conceptual contribution relative to closely related work remains insufficiently distinguished.
    As acknowledged in Section 2, Future Forcing constructs future-query proxies, PaFu-KV distills salience from a bidirectional teacher, and MemFlow retrieves history using the upcoming chunk prompt. Thus, anticipating future relevance is not by itself a sufficiently distinctive contribution. The generator-external, explicit-frame formulation is a meaningful difference, but the paper should more precisely establish what the learned prospective-token formulation adds beyond these related ideas. The existing ablations support its effectiveness within the proposed system, but do not fully resolve this question of conceptual differentiation.

  2. The evidence for the incremental advantage over strong selection baselines is not yet sufficiently conclusive.
    In Table 5, FrameMorrow improves consistency over prompt matching by 0.95 points in the single-shot setting and 0.32 points in the multi-shot setting. These gains are encouraging, but their uncertainty is not reported. Furthermore, although the paper states that direct scoring uses the same inputs, its architecture, capacity, and training objective are not sufficiently specified to assess how cleanly this comparison isolates the prospective-token formulation. Improvements over native generators also include the benefit of adding historical references, as Appendix C.3 correctly notes, and therefore do not by themselves establish the additional value of the proposed selection mechanism.

Questions for authors

The following questions concern the interpretation of the current method and results.

  1. Clarification of the contribution:
    Relative to Future Forcing and PaFu-KV, what is the central methodological distinction of the prospective-token formulation beyond its generator-external, explicit-frame interface? Which existing results most directly support this distinction?

  2. Clarification of the direct-scoring control:
    Table 5 states that direct scoring uses the same inputs. Please specify its architecture, trainable parameter count, and training objective. Does it use the same future-grounded ranking targets and optimization settings as FrameMorrow?

  3. Uncertainty in the central comparisons:
    Could the authors quantify the uncertainty of the Table 5 differences against prompt matching and direct scoring, preferably using paired resampling of the existing per-case results? This would help determine how much weight to place on the reported incremental gains.

Reviewer_LjcsOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper address the long context window problem in video generation via token pruning by keeping only top-k relevant candidates.

Critical strengths

The paper confirms that using top-k history frame selection can better retain inter-frame consistency, comparing to a few sampling and retrieval-style heuristics. The Dynamic Degree experiments rules out the direct relevant concerns that using DINOv2 may cause: though similarity is susceptible to select similar contents, it is observed that there may not be videos gradually frozen to a standing still position, or entering some local loop.

Critical weaknesses

The selection is based on DINOv2 similarity, which makes sense for video that have a stable theme subject, but may not hold for videos that have floating changes, for example, the forward view of cars / bicycles.

Questions for authors

Have you measured the rank correlation between the DINOv2-based teacher relevance scores and the generator’s attention to historical frames during future generation? In particular, does this correlation differ between scene-revisit and continuously changing scenes?

Reviewer_Yd7ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FrameMorrow, which uses prospective tokens to select history frames. The future frames are conditioned on the selected frames and the next prompt or action. The selector learns from similarity-based rankings of the GT future frames. Experiments show improvements in consistency, quality, and alignment across different video renderers, including closed-source models.

Critical strengths
  • This system design is plug-and-play and does not need to fine-tune the video model (only a lightweight selector).

  • The selector improves performance across different video models and also supports closed-source models.

  • The writing is clear and good.

Critical weaknesses
  • The distinction between the existing work ("current-oriented" or "history-based") and this work (“future-oriented” retrieval) is overstated: prior methods also select history to improve future generation, using current context as a proxy for future utility. The contribution is more precisely a different supervision and relevance estimation strategy, rather than a fundamental shift in objective.

  • The method is trained to retrieve historical frames that visually resemble the GT future, making its motivation most compelling when upcoming content reuses previously observed evidence, as in the revisit showcases in the paper. When a transition introduces entirely new scenes or subjects, relevant historical evidence may not exist, yet the selector still returns a fixed number of frames. The paper does not sufficiently establish whether its benefits extend beyond history-reuse scenarios, limiting support for its broader claims.

Questions for authors

For the ablation on tokens and direct scoring, could you clarify the detailed settings for direct scoring? Does it use the same supervision and training settings as with prospective tokens?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.