acceptodds
Under review as a conference paper at ICLR 2027

StreamPilot: Agentic Inference for Long-Horizon Streaming Video Generation

Abstract

Streaming video generation models produce videos autoregressively. This leads to two challenges that all streaming video generation models share. The first is the drift: errors in the generated content accumulate until the picture collapses. The second is long-horizon consistency: as the video grows and the prompt keeps changing, the model does not remember what it generated, and may forget relevant content or recall the wrong content. We propose StreamPilot, a training-free agentic inference framework in which a VLM agent that reads frames fills the long-term cache during inference. At each instruction change and at fixed intervals, the agent reads the current instruction and the latest frames, decides which earlier chunk comes back as a reference and checks each new chunk before it is stored. References are stored as latents and recomputed where they are loaded. Without any training, StreamPilot achieves better long-horizon consistency than the base models on three streaming video models and four public benchmarks, showing that an agent can manage the memory of a streaming video model at inference time. On Causal Forcing, it raises the share of MemFlow clips showing the requested scene from 29.0% to 73.5% and cuts the videos that collapse after a scene returns from 58% to 13%. It also lets Self Forcing surpass LongLive in memory without any training, although LongLive is tuned for long videos.

Primary area:
generative models
Keywords:
streaming video generation, long-horizon video generation, video diffusion, inference-time adaptation, memory management

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pkpGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces StreamPilot, a training-free agentic inference framework designed to improve long-horizon consistency and reduce error accumulation in autoregressive streaming video generation. Instead of relying on fixed memory-selection strategies, StreamPilot employs a vision-language model (VLM) agent to dynamically manage long-term memory by selecting relevant past video chunks, verifying newly generated content, and recomputing stored latent representations within the current context. Experiments across multiple streaming video models, including LongLive, Self Forcing, and Causal Forcing, demonstrate substantial improvements in scene consistency, identity preservation, and generation stability.

Critical strengths
  1. I think this problem is Practical and important, the method demonstrate strong improvements on multiple backbones, and have provided extensive ablation studies and experiments;
Critical weaknesses
  1. StreamPilot cites IAMFlow but does not directly compare against it under matched conditions, despite both targeting multi-prompt streaming generation;

  2. FreeMem: Enhancing Consistency in Long Video Generation via Tuning-Free Memory (AAAI 2026): FreeMem already demonstrates that maintaining and updating historical noise, token features, and attention caches can improve long-video consistency without additional training.

Questions for authors

Please find the details in major weaknesses.

Reviewer_qFc7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies how an autoregressive video generator should manage historical information as generation continues and instructions change. The paper distinguishes two related problems: drift, where errors propagate through conditioning on previously generated content, and long-term inconsistency. The authors proposed solution is a training-free framework that uses a pre-trained vision-language model as an online memory controller. The paper is well written and easy to follow.

Critical strengths

1:The paper identifies a meaningful distinction between remembering more history and remembering appropriate history. 2: The framework provides a concrete interface between semantic decisions and low-level cache operations.

Critical weaknesses

1: The memory mechanism has limited ways to stop harmful historical conditioning. KEEP and WAIT preserve existing references, and the action space lacks an explicit operation to clear or disable a slot. Furthermore, after three unsuccessful checks, a candidate is still stored and loaded as unreliable. The reliability flag therefore does not prevent its immediate influence on generation. This design is particularly restrictive when an instruction requires a genuinely new scene or when every available candidate is unsuitable. An explicit CLEAR action, confidence-dependent reference strength, or a policy that leaves failed candidates outside the active cache would be valuable additions. These alternatives should be evaluated on hard scene changes and conflicting identities.

2: The method proposed in this paper offers limited novelty, as many of its components would readily occur to practitioners with routine experience using agents. Moreover, in real-world applications, the approach has clear limitations when generating long-form videos in real time, making deployment challenging.

Questions for authors

see the weakness above

Reviewer_aA35OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces StreamPilot where off-the-shelf VLMs manage long term memory and agentically decide which past content to store/retrieve. Authors also introduce reference-slot cache connecting these decisions into streaming video models with a sink and a sliding window attention. Every component of this works training free which is interesting, and experiments show that the long horizon consistencies are enhanced compared to several baselines.

Critical strengths
  • Roles of KV-cache and attention context managing for causal streaming models have been actively studied recently. Many works try to have a fixed rule for managing it, but having an agent directly in the loop sounds interesting. The suggested interface (reference-slot cache) looks reasonable and improvements on the 1.3B models and the fact it’s training-free approach are both good strengths.

  • I appreciate the authors' efforts to examine what the agent is doing. Authors include several controlled experiments and a transparent failure (negative results) and limitations in the appendix, which helps understand the actual scope of the method.

Critical weaknesses
  • One of the concern is the evaluation setup. The generator is frozen, and the agent mainly selects previously generated contents and puts it into the attention context. It’s clear providing a relevant reference would improve consistency, but I’m less sure how this establishes the benefit of agentic memory management over simple reference selection (I think there aren’t that much complex agentic process VLM is going through here, so we could potentially do a bit more engineering with CLIP/MiniLM approaches as mentioned in the paper – which would practically be better than having a 26B Gemma running outside) More importantly, the main evaluator is the same VLM used by the agent. I think what we are essentially doing is very cyclic: we are asking agent to arrange a context from the instructions and ask the same agent whether it is aligned. Since authors also adopt Qwen-based evals it’s not super critical, but I would still appreciate blinded human study on representative cases.

  • Having agent in the loop is an interesting idea, but at the same time, it’s an idea easy to think about and implement quickly. To make the paper stronger, I would also be curious in testing these on state-of-the-art models with state-of-the-art VLMs. For example, we have some streaming variants of Wan 14B variants / Minimax H3 and we can always ask off-the-shelf closed proprietary models (GPT/Claude/Gemini) which can probably arrange things much better. Given that the method is training-free, testing them would make the paper sound stronger. Given that agentic things slow down the process anyway, I don’t think real time or 30 FPS things are the deciding factors for this paper.

Questions for authors

Please check the weakness section for the questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.