acceptodds
Under review as a conference paper at ICLR 2027

StreamingR2V: Towards Long and Interactive Reference-to-Video Generation

Abstract

Reference-to-video (R2V) generation renders user-specified subjects faithfully, but only within a short clip whose reference set is fixed before generation begins. In practice, however, subjects often enter or leave partway through a shot, and users need interactive control over the cast as the video unfolds over a long horizon. We therefore extend R2V to (SR2V), which keeps the horizon open and lets references be added or removed as generation proceeds, and identify two challenges that both peak at the moment of interaction. First, history frames carry the identities already rendered, which override the expressiveness of newly inserted references. Second, the chunk where the reference set changes must render a subject unseen in history, which can raise its error, and may accumulate more acutely during generation. We accordingly propose with two complementary designs. The reconstructs each history token from the references and subtracts the reference-explainable identity from its value, leaving the attention map untouched. The adds history-error banks indexed by rollout depth, approximating error accumulation at one forward pass per iteration. On an SR2V benchmark with diverse events of subject entry and exit, StreamingR2V matches or exceeds leading commercial models and several baselines across a wide range of metrics, especially on long-term consistency and interaction compliance. The anonymous demo is available at: https://sr2v-demo.github.io/.

Primary area:
generative models
Keywords:
Video Generation, Personalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PgukOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets at Streaming Reference-to-Video (R2V), which is an autoregressive R2V generation in which reference images can be added or removed during rollout. The authors noticed two problems in this topic: (a) identities already rendered into history typically overpower a newly inserted reference; (b) the chunk where the reference set changes is especially error-prone.

Therefore, the authors propose two modules to address these problems:

  • Reference reflective value gate: it reconstructs each history value token in every self-attention layer, and then gates the reconstruction per token.
  • Virtual rollout error chain: it extends SVI's error recycling, and then injects a residual tube into the history. This can approximate multi-chunk error accumulation at one forward pass per iteration.

Based on the reported results, StreamingR2V is claimed to be best or near-best on most metrics (R2V metrics, timing metrics, interaction-compliance rates).

Critical strengths

[S1]

I like the idea of combining autoregressive video generation and R2V generation, which are typically treated as two independent topics. The paper really fills a gap between R2V and long/interactive generation.

Moreover, to adapt this new topic, the authors propose an evaluation protocol that actually targets it. Specifically: (a) R2V metrics are scored per condition group, so a subject is only required while its reference is active; (b) multi-subject matching in OpenS2V-Eval is fixed with Hungarian assignment, and missing subjects score zero instead of being dropped; and much more reasonable metrics.

A new topic + the corresponding evaluation protocol make this paper comprehensive.

[S2]

Based on the reported numbers, the proposed modules look promising. Specifically, from Table 3, it seems this cheap / drop-in module can (a) significantly raise post-interaction Face-Sim + DINO-I, (b) cut the before/after gap by about almost half (Face-Sim) to two-thirds (DINO-I), and (c) also lift pre-interaction fidelity. These numbers are consistent with the argument that drifted identity in history shouldn't serve as an identity condition.

Critical weaknesses

[W1]

I think the experiments are not enough to show that the proposed mechanisms are necessary and better than some simpler alternatives. They also don't show the proposed modules work for the claimed reasons. It is especially important because these two modules are the technical contribution.

For example, I can easily think of these alternatives which are untested in the paper:

  • For the continuity trade-off, you could try to down-weight or mask history attention. Also, sanitized values could actually change hidden states, so later-layer attention is not actually untouched.
  • You can try to mask subject regions in history using the SAM3 masks already in the pipeline.
  • Stronger reference guidance.
  • An R2V baseline that also conditions on history frames.

For the "Error chain", the l-max ablation in the paper is non-monotonic. l-max = 2 is worse than l-max = 1 on post-interaction Face-Sim & DINO-I. Since depth is not an input to the model, the chain mainly reshapes the mixture of injected errors. The "virtual rollout" approximation is never compared with real inference-time residuals.

Lastly, on the mechanism side, you can try to visualize the "gate" and "learned y values". Table 2 averages post-interaction fidelity over all subjects in the new set, so the claim that the gate specifically frees the newly inserted subject is not directly shown.

[W2]

I noticed that the interaction-compliance evidence is weaker than presented. This is consequential because interaction compliance and long-horizon consistency are the headline claims. There are a couple of concerns around this:

  • The benchmark and metrics are self-constructed. Also, event counts are small (34 entries and 14 exits at 200 frames) so a single exit moves XSR by about 0.07.
  • From my understanding, no non-commercial method sees a new reference before its cue. Two-stage pipelines invoke it only at the boundary, and SFT & StreamingR2V follow the schedule. However, at 200 frames, these baselines score PR of 0.08-0.13, and VACE+SVI scores 0.579.
  • I am also confused on the PR definition. The paper defines PR as a fraction of entry events. ESR and XSR come out as multiples of 1/34 and 1/14, but the reported PR values do not. It seems like PR is computed differently than described.
Questions for authors

[Q1]

I am not sure how the authors produce non-zero PR for methods that never receive the new reference before its cue. Could the authors explain this? Also, could you report:

  • the detector's false-positive rate on clips where a subject is known to be absent
  • PR and RR under a sweep of tau, separately for the face and DINO backends
  • how often a "premature" match is actually an existing subject

[Q2]

In Algorithm 1, is a tail produced at depth l-max dropped, or written back into the deepest bank? Is the l-max = 1 row literally SVI's self-recycling bank? Given the non-monotonic l-max results, how stable are Tables 3 and 7 across different random seeds?

[Q3]

Can Table 2 be split into newly inserted vs. already-present subjects, and into entry vs. exit boundaries? Can the Table 3 ablation also report ESR/PR/XSR/RR? For exits, the departing subject's reference is removed, so the gate has nothing to explain away its identity in history. How are exits represented in the curated training data, and does either module affect XSR or RR?

[Q4]

I think it is better to reconcile Figure 3 with Table 1 for MAGREF+SVI. How were the 600- and 1000-frame points computed, given that the benchmark has 200/400/800/1200-frame samples? Why is 1200 omitted, and why are VACE+SVI and Phantom+SVI not plotted?

2 comments

Thanks to the authors for the detailed rebuttal. I have carefully read the rebuttal & the other reviewers' comments. First, I noticed that some of my concerns are shared by the other reviewers.

  • Similar to my [W1, Q2], reviewer u72D also asked for an attention down-weighting baseline & whether l-max = 1 reproduces the original SVI error bank.
  • Similar to my [Q3], reviewer KCiH also noted that Table 2 mixes persistent & newly inserted subjects.
  • Similar to my [W2, Q1], reviewer KCiH also questioned the interaction metrics (including threshold sensitivity & human validation of the detector)
  • Similar to my error-chain points in [W1, Q2], reviewer KCiH also noted that the chain is never compared with actual rollout errors.

The rebuttal has addressed my [W1] & most of my questions. However, there are still two remaining concerns:

  • Since PR region include the post-cue tolerance window, an entry that is on time but abrupt counts as "premature", and the value depend on the pre-cue length. I think the authors should discuss this clearly & ideally report the pre-cue and tolerance-window parts separately. Without it, the gap to two-stage baselines is easy to over-read.
  • Also regarding error chain, the end-to-end gain is clear. But the post-cue DINO-I drop from l-max = 1 to 2 is still unexplained, and there is still no check against real rollout errors.

Overall, the rebuttal makes me more confident in the results. I agree with the other reviewers that the task is well motivated. Considering that two remaining concerns might affect how the interaction-compliance & error-chain claims should be read, I will keep my rating as weak accpetance.

Dear Reviewer Pguk,

Thank you for your response and for recognizing that our rebuttal has resolved most of your concerns. We address your remaining concerns below.

1. More Detailed PR Analysis

PR covers both the pre-cue region and the tolerance window that follows the cue, so we now report the two parts separately, over all 120 samples of the benchmark (τ=0.3\tau=0.3). The sampled frames of the pre-cue region and the tolerance window are in a ratio of about 3:1.

Table H: PR split into the pre-cue region and the tolerance window.

MethodPR_precuePR_windowPR
VACE+SVI0.1550.6730.285
SVI+R2V SFT0.1010.1390.111
StreamingR2V0.0320.0530.037

We have the following findings:

  • The gap does not depend on the window or the pre-cue length. StreamingR2V is lowest in both parts (0.032 vs. 0.155 / 0.101 before the cue; 0.053 vs. 0.673 / 0.139 in the window). As PR is their frame-weighted mean, the ranking holds for any pre-cue length.

  • The tolerance window exposes abrupt entry. The tolerance-window part of PR reflects an important failure that the other metrics hardly capture: the new subject is already fully visible right after the cue, instead of entering the scene gradually. Such an abrupt entry conflicts with the nature of streaming reference-to-video, yet ESR still counts it as a success. Every method has a higher PR in the window than before the cue. The increase is sharpest for VACE+SVI (0.673 vs. 0.155), whose new subject comes from a freshly generated clip and thus exhibits exactly this failure, while StreamingR2V stays lowest (0.053). In short, the window part shows how smoothly a new subject enters, and StreamingR2V introduces new subjects most smoothly.

We will add Table H and the definitions of both parts to the appendix of the revised paper.

2. Error Chain

The drop at ℓmax=2\ell_{max}=2. This drop appears only in post-cue DINO-I (0.612 to 0.578), while post-cue FS stays within its seed variation and both pre-cue metrics improve. ℓmax=2\ell_{max}=2 is also the least stable setting across seeds (post-cue FS std 0.044, vs. at most 0.02 elsewhere). Our design does not rely on this setting, since ℓmax=3\ell_{max}=3 and ℓmax=4\ell_{max}=4 both exceed ℓmax=1\ell_{max}=1 on all four metrics. We will add this discussion to the revised paper.

Check against real rollout errors. We clarify that the chain is a training-time strategy: each stored tube is a real model residual, but the history it is injected into is not a real rollout. Furthermore, a step-wise comparison with real inference-time residuals (as raised in your original W1) would need a ground-truth video at inference, which R2V does not have: the only inputs are scheduled reference images and prompts, and the references in our benchmark are not taken from any video. Real rollout errors are therefore observable only through their effect on the generated long videos, where the chain helps (Tab. 3 and Tab. 7 of the paper; Tables B and G of our first-round rebuttal). We will update this discussion in our revised paper.

We hope the clarifications above address your remaining concerns. Again, we sincerely thank you for your constructive suggestions and your continued positive evaluation of our submission.

Best regards,

Authors of Submission 753

Reviewer_KCiHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies long video generation in which reference subjects enter or leave during generation. StreamingR2V combines a gate that removes reference-related information from history values with depth-indexed banks that recycle prediction errors during training. Built on Wan2.1-I2V-14B, it improves consistency and reference compliance on 120 cases spanning 200–1,200 frames. The appendix adds ablations, runtime measurements, and a human preference study.

Critical strengths
  1. The task is useful and clearly motivated. Dynamically changing references addresses a limitation of fixed-reference generation. The examples include multiple subjects, camera movement, entry, and exit.

  2. The comparison includes a jointly trained SVI+R2V baseline using the same data. Table 3 separates the gate and error chain, with both contributing to the result.

  3. The evaluation includes interaction events, longer sequences, parameter studies, and a 30-person preference study. The gate also adds relatively little inference overhead.

Critical weaknesses
  1. The before/after identity scores do not isolate the claimed effects. Appendix A.2.2 compares different reference sets across an interaction, so the gap mixes subject difficulty with actual degradation. Table 2 cannot separately establish preservation of existing subjects and fidelity of newly inserted ones. Reporting these groups separately can clarify the conclusion.

  2. The interaction metrics allow some failures to escape. Equation (8) averages over matched frames, while entry success requires only three consecutive detections somewhere after the cue. A subject that appears late or briefly can therefore receive favorable scores. The appendix also reports no detector agreement with human judgments or sensitivity to the shared 0.30 similarity threshold.

  3. The error chain helps empirically, but its interpretation as rollout-error modeling is less established. Errors pass between unrelated clips, without comparison against actual rollout errors. Tables 4 and 7 also leave unclear whether total bank capacity is matched when depth changes. Algorithm 1 needs to specify what happens when the deepest sampled bank writes to depth ℓmax⁡+1\ell_{\max}+1.

  4. The human study covers only 200-frame videos and excludes SVI+R2V, the most directly comparable baseline. Commercial models receive all references upfront, while two-stage baselines restart at interactions. These differences make the joint baseline especially important for assessing long, interactive generation.

Questions for authors

As shown in weakness.

Reviewer_u72DOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Streaming Reference-to-Video (SR2V), enabling long-horizon video generation with dynamically added or removed reference subjects. It proposes a reference reflective value gate to reduce identity interference from historical frames and a virtual rollout error chain to mitigate autoregressive error accumulation. Experiments on a newly curated SR2V benchmark demonstrate improvements in subject consistency, interaction compliance, and long-horizon video quality over several baselines.

Critical strengths
  • The paper addresses an important limitation of existing R2V methods by enabling interactive reference control during long-horizon generation. The motivation is clear, and the proposed setting has practical relevance.

  • The two proposed components address complementary challenges in reference conditioning and autoregressive generation. The paper provides strong quantitative improvements, long-horizon evaluations, and human studies.

Critical weaknesses
  • Sec. 1 argues that modifying attention weights degrades temporal continuity. However, no direct comparison with simpler attention down-weighting alternative is provided to support this design choice.

  • Table 7 labels the depth-1 setting as original SVI, but SVI recycles errors from corrupted histories into the same bank, whereas the proposed method explicitly separates rollout depths. It is unclear whether the depth-1 setting reproduces the original SVI procedure, which affects the interpretation of this ablation.

Questions for authors
  • Could the authors provide an attention down-weighting baseline to support the claim that modifying attention weights harms temporal continuity?

  • Could the authors clarify whether the depth-1 setting in Table 7 reproduces the original SVI error bank, or instead uses Algorithm 1 restricted to a single depth?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.