acceptodds
Withdrawn from ICLR 2027

StreamReal: Reference-Guided Subject Switching in Streaming Video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reference-guided subject switching requires a causal streaming generator to adopt subjects specified by new visual references during an ongoing rollout while preserving scene and temporal continuity. This is challenging because accumulated history entangles preceding-subject information with context needed for continuation. Comparisons on an adapted causal generator show that retaining the historical cache or recaching it under updated text yields weak alignment with new references, whereas clearing it improves responsiveness over subsequent chunks at the cost of aesthetics and motion smoothness. We introduce StreamReal, which organizes causal history to distinguish subject-dependent information from context that remains relevant across reference updates. To establish reference conditioning, Few-Step Conditioning Alignment aligns a pretrained reference pathway with a frozen few-step causal generator, followed by reference-conditioned Distribution Matching Distillation warm-up to adapt the reference branch to self-generated histories. During streaming, Memory Flow maintains a cross-chunk semantic index associating historical information with the evolving subject. Upon a reference update, Memory-Guided History Routing uses this index to regulate access to native historical states, suppressing preceding-subject evidence while preserving context for coherent continuation. We introduce a Subject Switch Benchmark spanning diverse subject categories and switching positions. Evaluations on OpenS2V-Eval and this benchmark show that StreamReal combines subject fidelity and visual quality with a favorable balance between switching responsiveness and temporal continuity.

Primary area:
generative models
Keywords:
streaming video generation, multi-subject customization, interactive generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5gMiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces StreamReal, a method designed to switch subjects based on new reference images during streaming video generation while maintaining scene and temporal continuity. The approach adapts existing autoregressive video models to incorporate reference image conditioning and employs Memory Flow to establish historical indexing between the subject and the background, selectively masking cache entries related to the old subject during the switch. Experimental results demonstrate that this method improves reference consistency for new subjects and mitigates the loss of continuity typically caused by clearing historical data.

Critical strengths
  • MF is reasonably designed: maintaining separate subject and background memory summaries supports selective suppression of outdated subject information and appears helpful in reducing interference from redundant history.
  • The semantic index is decoupled from the native KV cache. MF organizes historical information, while MHR controls access during subject switching, providing a modular design that integrates with existing causal generators.
  • The progressive training pipeline—FCA followed by DMD adaptation to self-generated histories—is coherent, addressing reference-conditioning alignment and the mismatch between training and inference histories.
Critical weaknesses
  1. No video demos are included in the provided supplementary material. For a video generation paper, demonstrations are essential for assessing visual quality, temporal continuity, and the effectiveness of subject switching. The authors should provide representative examples, including failure cases.

  2. The meaning and practical motivation of “subject switching” require further clarification. Introducing or removing subjects during an ongoing rollout seems useful for customized generation. However, it is less clear when users would need to replace an existing subject while preserving the ongoing scene, and how this differs from a video editing task. The authors should clarify the task definition and provide concrete use cases in L52–L61.

  3. Several relevant works on video customization and streaming video editing are missing from the discussion:

    • [1] Li W, Zhang Y, Long F, et al. ReactID: Synchronizing Realistic Actions and Identity in Personalized Video Generation. ICLR, 2026: 62463–62485.
    • [2] Xu X, Li Y, You S, et al. Smrabooth: Subject and Motion Representation Alignment for Customized Video Generation. CVPR, 2026: 16130–16141.
    • [3] Deng Y, Yin Y, Guo X, et al. MagRef: Masked Guidance for Any-Reference Video Generation with Subject Disentanglement. arXiv:2505.23742, 2025.
    • [4] Wang X, Zhao C, Zhan F, et al. LiveEdit: Towards Real-Time Diffusion-Based Streaming Video Editing. ECCV, 2026: 631–649.

    The authors should discuss the methodological relationships and differences between their approach and [1]–[3]. An experimental comparison with [4] would also be valuable where the task settings are compatible; otherwise, the differences should be explicitly explained.

  4. The writing is somewhat disorganized. The introduction would benefit from a concise overview of the training and inference workflows and a clearer statement of the core contributions. Figure 3 is overly complex, making the methodological rationale difficult to follow and giving the impression of a collection of engineering components rather than a clearly motivated framework. Terminology should also be consistent: “Few-Step Conditioning Alignment” in L198 is referred to as “Few-Step Alignment” in Figure 3.

  5. The authors should quantitatively evaluate the accuracy of MF’s predicted token labels. In particular, they should demonstrate whether (rk,mk)(r_k,m_k) reliably captures historical relevance and subject association, and whether the resulting subject labels align with actual subject regions. The computational and memory overhead of MF should also be reported through a controlled comparison with and without the module, including its effect on generation latency during ordinary rollout and subject switching.

Questions for authors

I hope the author can address the weaknesses I raised above.

Reviewer_Qg6ZOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies reference-guided subject switching in streaming video generation and introduces the first framework StreamReal for this setting. The method first adapts reference conditioning to a frozen causal generator using FCA and DMD, then employs MF to maintain a semantic index of subject and background information in the generation history. It further introduces MHR to mask historical information associated with the previous subject during switching. Experiments on standard subject-conditioned video generation and a newly constructed subject-switching benchmark show a better trade-off between switching responsiveness and temporal continuity.

Critical strengths
  1. The paper introduces the first framework for reference-guided subject switching in causal streaming video generation, a task that may has potential practical applications. The authors also construct a corresponding Subject Switch Benchmark, which may serve as a useful resource for the research community.
Critical weaknesses
  1. The paper suffers from serious weaknesses in writing and presentation, making the proposed method difficult to understand. For example, in the inference portion of Figure 3, the logical relationships of the Memory Flow module are unclear, and several modules on the right lack explicit connections to the rest of the pipeline, making them appear isolated. Overall, the figure is cluttered and overly complex, obscuring the core idea of the method. The paper also fails to clearly highlight its main technical contributions. The method comprises numerous modules and stages without clearly identifying which components constitute the key innovations, making it seem to be a overly complex engineering pipeline. Similarly, the related work subsection “Causal video generation and history management” briefly lists several prior studies without clearly explaining how the proposed approach differs from or advances them.

  2. The experimental validation is insufficient, and the qualitative results are also unsatisfactory. Although Table 2 demonstrates benefits of the full framework over three history strategies, it does not highlight the necessity of learned MF compared with more straightforward segmentation or label-propagation approaches. Table 3 do not clearly specifies which modules are enabled in each variant and do not validate the contribution of the modules within the proposed framework. Consequently, it remains unclear whether the gains can be attributed specifically to the proposed MF. Furthermore, the sunglasses update in Figure 1 appears to introduce changes in the subject’s facial appearance (man to woman) beyond the accessory itself, raising concerns about subject fidelity during switching. The paper also does not discuss the limitations of the proposed method.

Questions for authors
  1. The paper’s presentation is disorganized. In particular, Figure 3 is overly complex, with a cluttered layout and unclear logical relationships among its modules. Could the authors improve the presentation and clarify how the components connect and interact?

  2. The paper’s contributions and novelty are unclear. Which modules of the proposed framework are novel, or does the framework primarily constitute an engineering integration of existing components? The writing should be revised to clearly identify the novel components and highlight the paper’s technical contributions.

Reviewer_EXcpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

StreamReal studies changing reference subjects during causal video generation while retaining useful scene history. It adapts a VACE reference branch to frozen LongLive/Causal Forcing backbones through Few-Step Conditioning Alignment (FCA) and reference-conditioned DMD warm-up. Memory Flow (MF) learns a subject/background index of committed tokens. At an update, Memory-Guided History Routing (MHR) masks historical subject keys for the first updated chunk, then retires obsolete history and reseeds memory. Evaluation includes 180 OpenS2V-Eval cases and a new benchmark of 54 reference pairs at three switching positions, totaling 162 single-switch cases. Results support strong fixed-reference generation and improved switching relative to retaining, clearing, or recaching history; repeated updates are illustrated qualitatively.

Critical strengths

(1) A consequential task with a controlled switching comparison. Keep, Clear, Recache, and StreamReal share the adapted LongLive generator and receive identical condition updates (Section 4.1; Appendix A.1). Table 2 reports substantially smaller reference-alignment losses for StreamReal: CLIP-I-Seg changes by -0.0941, versus -0.2895 for Keep and -0.1666 for Clear. Figure 5 also shows a large alignment improvement in the first updated chunk, so the evidence is not limited to eventual recovery. This supports selective history management as a practical way to reconcile responsiveness and continuity.

(2) Effective reference conditioning for fast causal generation. On OpenS2V-Eval, the Causal Forcing variant leads the compared methods on all five subject-fidelity metrics, including DINO-I-Seg of 0.5086 versus Phantom's 0.4662, while reporting 0.180 versus 1.955 seconds/frame (Table 1a). The combination is practically attractive. This strength concerns the reference-conditioning pipeline; the reused backbone T2V scores in Table 1b are not independent evidence for MF/MHR.

Critical weaknesses

(1) The core semantic supervision cannot be reconstructed from the supplied description. Section 3.3 defines the reader outputs as attention to labeled history and the subject fraction of that attention (p. 5), but the training paragraph only invokes a frozen teacher and a token-label loss (p. 6). It does not identify the label-producing procedure, specify the target equations or loss, or explain the conversion from these attention quantities to valid subject labels. Appendix A.1 repeats the teacher statement without resolving it. Grounding DINO/SAM2 initialization (p. 7) explains seed regions, but not the subsequent training targets. The relevance/validity rule and numerical subject threshold used by Eq. (9) are also unspecified. These choices determine which historical evidence is suppressed, so this is a reproducibility gap in the principal contribution.

(2) The experiments establish the full system's advantage more clearly than the role of learned semantic memory or the limits of continuity after updates. Table 3 concerns fixed-reference fidelity; MHR is explicitly inactive during ordinary generation (p. 7), so those results do not isolate MF/MHR. The switching comparison changes selective masking, learned indexing, and history retirement together, without a control that retains the same update policy while replacing the learned index. Moreover, Table 2 excludes the switching chunk: Figure 5(e) shows motion smoothness falling from approximately 0.99 to 0.82 at that chunk before recovering. The improvement over Clear is convincing in subsequent chunks, but the initial metric decrease warrants interpretation.

The 162 quantitative cases comprise 54 reference pairs, each tested at three positions with one switch per rollout (Appendix A.1). Repeated updates are illustrated qualitatively, without a controlled return to an earlier subject after a delay. This matters because p. 7 explicitly retires obsolete target KV entries, labels, and summaries after the first updated chunk, then reseeds MF. The exact retirement rule and visible-history/summary budget are unspecified, and no return-gap or cache-budget analysis establishes what context survives repeated updates. Reproducing subject A from its reference in an A-to-B-to-A sequence would establish successful reconditioning; retention of earlier generated scene or subject state requires a separate check against the earlier video. These limits should be characterized or the continuity claim bounded; the omission alone does not demonstrate that recall fails.

Questions for authors

(1) Clarification required for reproducibility: What teacher and exact computations produce the two reader targets? Please specify the token-label loss, relevant feature/attention extraction, validity rule, and routing thresholds, including how segmentation seed labels initialize these quantities.

(2) Decision-relevant evidence request: Does learned MF improve switching over the same masking and history-retirement policy driven by direct segmentation or an index without cross-chunk memory? A comparison isolating this change would establish the contribution of the proposed memory mechanism beyond the full-system comparison.

(3) Decision-relevant clarification and evidence request: What exactly is retired after the first updated chunk, and what bounds the native KV cache and visible MF summaries? Please characterize repeated updates, including A-to-B-to-A returns at varying gaps and history budgets, or explicitly delimit unsupported continuity behavior. Separate identity recovered from the resupplied reference from preservation of earlier generated context not specified by the new input. Report immediate-transition behavior as well as subsequent recovery, including the visible implications of the approximately 0.17 first-chunk smoothness drop in Figure 5. This request concerns the claimed interactive streaming behavior, rather than an unrestricted world-reconstruction task.

(4) Related-work and comparison clarification: Please cite and discuss MAGREF and Learning Zero-Shot Subject-Driven Video Generation Using 1% Compute, positioning the reference-adaptation component relative to learned reference conditioning and subject disentanglement. How does fixed-reference quality compare with these alternatives on compatible OpenS2V subsets, with model scale, reference inputs, and subject-count support disclosed? This comparison would clarify the practical value of the adapted causal S2V generator; it should be kept distinct from testing the switching mechanism, for which the shared-generator controls are appropriate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.