acceptodds
Under review as a conference paper at ICLR 2027

Refrain: Learning Multi-Shot Consistency from Inconsistent References for Video Generation

Abstract

A comic-book character, a photorealistic cityscape, an oil-painted coffee cup — such stylistically diverse references provide visual guidance across shots and frames, yet generating a coherent multi-shot video that unifies their visual styles while preserving subject identities and scene content remains challenging. Existing methods maintain cross-shot continuity through additional attention control or memory processing. However, they struggle to resolve disparate per-shot styles or handle reference–target duration mismatches within a single framework. To tackle this, we introduce **Refrain**, a unified model that learns cross-shot consistency through a self-supervised reconstruction task: reference clips from the same original video are deliberately perturbed with distinct visual edits, challenging the model to recover the unedited source footage under a shared style prompt. In this way, Refrain learns to suppress conflicting stylistic artifacts while faithfully retaining reference content and motion dynamics. Operating on the principle (**Any Reference, Any Duration, One Timeline)**, Refrain maps image and video references onto a shared temporal canvas without auxiliary conditioning modules. For evaluation, experiments show that standard CLIP or DINO metrics frequently overlook stylistic clashes or misinterpret intended narrative cuts (such as changes in subjects or scenes) as inconsistency. We therefore propose **Shot-Debiased Consistency (SDC)**, which uses an extensive real-video bank to establish a baseline for style variation associated with content changes within shots. By discounting this baseline divergence, SDC evaluates cross-shot style shifts against real-world distributions, where smaller residual differences indicate higher stylistic continuity. We further construct **DurMix-300K**, a dataset pairing stylistically altered clips of varying lengths with their original video sequences, to train the model. Extensive experiments demonstrate that Refrain achieves state-of-the-art performance in cross-shot consistency, while excelling at shot timing, reference adherence, and text alignment.

Primary area:
generative models
Keywords:
Multi-shot Video Generation, Reference Editing, Diffusion Generative Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HUpDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Refrain, a multi-shot video generation model built on HunyuanVideo-1.5 that keeps a consistent style across shots even when references conflict in style. In training, references from different shots are edited independently, and the model learns to reconstruct the original video under a shared style prompt. The authors also propose SDC, a cross-shot style consistency metric, and DurMix-300K, a dataset of edited references.

Critical strengths
  • The design is unified and efficient. One backbone handles four tasks without extra modules, and the cost stays the same as the base model.
  • The baselines are recent and strong, including MiniMax-H3, and the baseline settings are described in detail.
  • The training task is simple and clever. Editing references independently and reconstructing the original video gives free supervision for cross-shot style consistency.
Critical weaknesses
  • The human study only covers Refrain's own results. It is used to validate the SDC metric, but it does not compare Refrain with any baseline. A user study that compares the methods is needed.
  • I have some doubts about the motivation. Even if references with different styles can hurt the result, a simple solution exists: first apply an image or video style transfer model to the references, so that they match the target style, and then generate. This cascaded approach is more complex, but it may work just as well. The paper does not compare with or discuss this option.
  • The architectural novelty is limited. Sparse temporal placement is close to VideoCanvas, VACE and SparseCtrl. The shot-aware RoPE is taken from MultiShotMaster. Channel concatenation with a mask is a standard base-model design.
  • The visual quality of the videos in the supplementary material is somewhat low. I understand this may come from the HunyuanVideo-1.5 base model. However, many stronger base models are now available. Why choose HunyuanVideo-1.5? If the design only works on this base model, its upper bound may be low. Newer models such as LTX-2.3 or MiniMax-H3 can generate video with audio, and they can also generate video only. I would like the authors to explain this choice.
  • The SDC metric is reasonable to some extent, but the overall gains are not large. The task setting is also narrow. Overall, the technical contribution is limited.
Questions for authors
  • Can you add a baseline that first applies style transfer to the reference images or clips? Most of the current baselines are general models without special training for this task, so they are not expected to handle style conflicts.
  • Can you run a user study that compares Refrain with the baselines on style consistency and overall quality?
8 comments
AuthorsOpenReview ↗

Response (Part 1/4)

Thank you for your feedback. We have revised the paper and added the requested comparisons, as detailed below.

W1 / Q2. Comparative user study.

We ran the requested comparison on 100 test requests with six professional raters, blind to method identity (Table 3, Section 4.2, Appendix F). Raters saw the shot descriptions, the references, and four anonymized outputs side by side, and scored six dimensions from 1 to 5 with the rubric in Figures 6–7.

MethodStyleReferenceTextVisualMotionOverall
Phantom2.182.803.073.503.682.81
StoryMem3.753.303.273.633.803.35
MiniMax-H34.034.274.304.344.384.25
Refrain4.254.193.993.914.053.98

Refrain receives the highest style-consistency rating of the four methods and is rated higher than StoryMem and Phantom on all six dimensions. Against MiniMax-H3, the reference-fidelity ratings are close (4.19 vs. 4.27), and MiniMax-H3 is rated higher in text adherence, visual quality, motion, and overall preference.

The lower visual-quality and motion ratings help explain the overall-preference gap despite the higher style-consistency rating. Refrain matches its HunyuanVideo-1.5 base in VideoAlign VQ and aesthetics and improves motion quality (W4), and its visual quality stays at the level of the chosen backbone.

W2 / Q1. Motivation and the cascade baseline.

We added this baseline. The cascade reaches a similar style gain, but it needs about 7 NVIDIA H20 GPU-minutes of preprocessing per reference and lowers both reference-fidelity measures. Refrain reaches this style consistency in a single generation pass. We implemented the cascade with the editing pipeline that produced DurMix-300K. Each test reference is restyled toward the request's shared style (FLUX.1 Kontext on the first frame, depth-conditioned Wan-2.1 propagation), passes the same filters, and is given to MiniMax-H3 or Refrain (Appendix H, Table 20). We report the requests with video references. Reference fidelity is measured against the original references and, for the cascade, also against the restyled references the generator received.

MethodSDC ↑Ref-Subj (orig.) ↑Ref-Bg (orig.) ↑Ref-Subj (restyled) ↑Ref-Bg (restyled) ↑TA ↑MAE (s) ↓
Refrain0.3230.6450.653––−0.2040.197
Cascade + Refrain0.3240.5690.5740.6270.638−0.2960.227
MiniMax-H30.2680.5470.525––−0.2880.268
Cascade + MiniMax-H30.3260.5020.4850.5560.560−0.3590.285
  • Content is lost in the editing stage. Restyling raises the SDC of MiniMax-H3 from 0.268 to 0.326, similar to Refrain (0.323), but lowers reference-subject similarity from 0.547 to 0.502 and background similarity from 0.525 to 0.485, even after the DurMix-300K filters. The editor changes content along with style. In Figure 9(a), it removes the identity cues of the veiled character, and Cascade + MiniMax-H3 generates a different person. In Figure 9(b), both MiniMax-H3 variants render the close-up shot-3 reference as a distant figure.
  • Fidelity against the restyled references. Scored against the references it actually received, Cascade + MiniMax-H3 reaches 0.556 / 0.560 (subject / background), below Refrain on the original references (0.645 / 0.653). TA also falls from −0.288 to −0.359, and transition MAE rises from 0.268 s to 0.285 s. The restyled references pull the output away from the shot descriptions.
  • Cost. Restyling the 639 image and video references took about 73 NVIDIA H20 GPU-hours including regeneration, about 7 GPU-minutes per reference, most of it in Wan-2.1 propagation (median 3.8 GPU-minutes per video reference). Of these references, 100 failed the filters and were regenerated, 67 of them twice (Appendix H).
  • Cascade + Refrain. Restyling the references before Refrain leaves SDC nearly unchanged (0.324 vs. 0.323) but lowers reference-subject similarity from 0.645 to 0.569. Refrain is trained to render conflicting references in the shared style, and this training already provides the style gain that the cascade brings to MiniMax-H3. The generated outputs show lower fidelity to the original references after restyling.
AuthorsOpenReview ↗

Response (Part 2/4)

W3. Architectural novelty.

Thank you for raising this point. The A²O interface builds on prior placement and positional designs, and, as you noted, it keeps the design unified and efficient. The novelty of Refrain lies in the problem it addresses and in how it learns that problem from ordinary videos.

The problem. A multi-shot video is generated from several image or video references whose styles conflict with each other and with the requested style, while each reference keeps its content and the cuts follow the requested times. Table 1 compares this setting with multi-shot, subject-to-video, memory-based, and temporal-placement generation:

SettingReferencesDuration ≠ intervalConflicting stylesShared styleCut timing
Multi-shot (HoloCine, MultiShotMaster)none / images–✗✓✓
Subject-to-video (Phantom, RefAlign)images✗✗✗✗
Memory multi-shot (StoryMem)images✗✗✓✗
Temporal placement (VACE, VideoCanvas)images + video✓✗✗✗
Oursimages + video✓✓✓✓

The cascade experiment (W2) also motivates this setting. Restyling the references before generation reaches a similar SDC but loses reference content, TA, and timing. Refrain resolves the conflict during generation and keeps the references.

The insight. Paired data for this problem are hard to collect. Even our 86.9M-video calibration bank with the IP and Stock collections (W5) contains few real clip pairs with matching content and conflicting styles. We build such pairs from coherent multi-shot videos. Editing the references of different shots independently creates the conflict, and the original video remains the reconstruction target. The model learns to keep the content of each reference and to replace its appearance with the shared style, without manual labels. Setting reference durations shorter or longer than their intervals extends the same supervision to duration mismatches. The pipeline yields DurMix-300K at scale, and the same data also improve MiniMax-H3 through a LoRA (W4).

The comparisons below test the editing strategy, the conditioning interface, and the transfer of this supervision to MiniMax-H3:

ComparisonVariantSDC ↑
Training edits, same model (Table 4a)Shared edits0.301
Independent edits (Refrain)0.315
Injection, same data (Table 4c)Channel0.315
Sequence0.315
DurMix fine-tuning on MiniMax-H3 (Appendix L)MiniMax-H3, zero-shot0.295
MiniMax-H3 + DurMix-300K LoRA0.310

The training-edit comparison uses inconsistent test references, and the MiniMax-H3 LoRA keeps similar reference fidelity (details under W4). The training improves SDC with both conditioning interfaces and transfers to MiniMax-H3 through a LoRA, without architectural change.

The duration supervision also holds across reference lengths. Table 26 groups the requests with video references by reference duration relative to the assigned interval, and Refrain leads MiniMax-H3 on SDC, reference subject, and transition timing in all four groups:

Reference vs. interval (Table 26)ShorterEqualLongerMixed
SDC ↑, Refrain0.3340.3290.3290.307
SDC ↑, MiniMax-H30.3140.2490.2760.254
Ref-Subj ↑, Refrain0.6780.6490.6850.608
Ref-Subj ↑, MiniMax-H30.5820.5450.5640.521
MAE (s) ↓, Refrain0.1660.2080.2060.204
MAE (s) ↓, MiniMax-H30.2380.3180.2550.244
AuthorsOpenReview ↗

Response (Part 3/4)

W4. Visual quality and the choice of HunyuanVideo-1.5.

Thank you for this question. DurMix-300K and the training objective also apply to MiniMax-H3, a backbone with higher visual quality, and raise its SDC. We first explain the choice of HunyuanVideo-1.5 and its effect on visual quality, and then show the results on MiniMax-H3.

Backbone choice. We chose HunyuanVideo-1.5 for its training and inference efficiency (Table 2, Appendix K):

HunyuanVideo-1.5 (Refrain)MiniMax-H3Ratio
Parameters8.49B33.12B3.9×
Median time per training update14.2 s (full parameters)74.8 s (rank-32 LoRA)5.3×
Denoising computation per request324 PFLOPs551 PFLOPs1.7×

Both training times are measured on 64 NVIDIA H20 GPUs with 8-way sequence parallelism and a global batch of 8.

HunyuanVideo-1.5's conditioning channels also suit the A²O canvas, which adds only 150,528 parameters.

Visual quality. We scored all methods (Table 18, Appendix G):

MethodVideoAlign VQ ↑VideoAlign MQ ↑Aesthetic ↑
HunyuanVideo-1.5−0.492−0.8280.506
Refrain−0.492−0.7360.506
Phantom−0.114−0.5480.536
RefAlign−0.320−0.7480.522
StoryMem−0.550−0.5860.530
VACE−0.632−0.7860.486
MiniMax-H3−0.451−0.6610.534

Visual quality tracks the base model. Refrain equals HunyuanVideo-1.5 in VQ and aesthetics and scores higher in motion quality. In VQ it ranks above StoryMem and VACE and below Phantom, RefAlign, and MiniMax-H3. Phantom and RefAlign use 14B backbones and generate each shot as a separate single-shot clip. Refrain generates the 241-frame multi-shot sequence jointly, which allows cuts at the requested times (MAE 0.199 s).

Results on MiniMax-H3. We fine-tuned MiniMax-H3 on 10K DurMix-300K pairs through its native sequence-reference interface with a rank-32 LoRA (166M parameters) and no architectural change (Table 4c, Appendix L):

MiniMax-H3SDC ↑Ref-Subj ↑TA ↑
Zero-shot0.2950.559−0.267
+ DurMix-300K LoRA0.3100.558−0.280

The LoRA is trained for 1,000 steps with noise levels sampled from a logit-normal distribution and a self-distillation term toward the released model. It raises SDC with reference fidelity and cut timing at the zero-shot level (MAE 0.237 s vs. 0.242 s), and SDC improves at both training lengths in Appendix L. LTX-2.3 conditions on references through a sequence interface similar to MiniMax-H3, and the MiniMax-H3 results indicate how the same data transfer to this class of backbones. We plan to fine-tune MiniMax-H3 on the full DurMix-300K.

Reviewer_jM4TOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Refrain jointly addresses heterogeneous reference style conflicts, duration mismatches, and shot transition timing. It introduces a self-supervised "per-shot independent editing → original reconstruction" training task, paired with a sparse unified timeline and a novel SDC metric. Table 1 shows an SDC of 0.315, with notable improvements in reference subject/background metrics. SDC also correlates well with human judgments in controlled settings. I am positively inclined, but the independence of the proposed metric and the generalization capability require further validation.

Critical strengths

The self-supervised task directly targets the real-world conflict scenario. Table 3 honestly discloses performance degradation under shared-editing tests, acknowledging the method's boundaries. A2O elegantly unifies four conditional tasks, variable-length references, and shot temporal dynamics without requiring a separate reference network. The SDC metric attempts to explicitly disentangle style differences caused by content changes, supported by large-scale controlled experiments and human annotation.

Critical weaknesses
  1. Circularity of the SDC Metric: SDC is a novel metric tailored precisely to the paper’s main claim. Despite ablations and human validation, relying solely on a custom metric to declare SOTA is risky. Please incorporate independent metrics (e.g., expert style rating, subject attribute error rate, perceptual quality, shot transition structure) and report Pearson/Spearman correlations among them.
  2. Class Imbalance in Human Validation: In Table 5, out of 120 clips, only 8 reached consensus as "inconsistent," with another 8 having no consensus. Although the AUC is 0.969, such extreme scarcity of negative samples raises stability concerns. Please expand negative examples of complex real-world style conflicts, and report stratified AUC, inter-annotator agreement (e.g., Fleiss’ Kappa), and cross-source validation.
  3. Generalization of Training Edit Distribution: Independent editing training excels on independent test sets (0.315 vs. 0.301) but underperforms shared-editing training on shared tests (0.299 vs. 0.304). Please evaluate performance curves on real user-uploaded heterogeneous footage, unseen edit types, varying edit intensities, and clean references. Compare hybrid training strategies to assess distributional robustness.
Questions for authors

Does SDC maintain stable correlation with professional editors' ratings on independently collected real-world mixed footage?

3 comments
AuthorsOpenReview ↗

Response (Part 1/3)

Thank you for your feedback. We have expanded the evaluation and address each point below.

W1. Circularity of SDC, independent metrics, and their correlations.

We added the suggested independent measures, which do not use SDC's calibration (Table 3, and Table 19 in Appendix G), and report their correlations below:

MethodSDC ↑Human style ↑VLM style ↑Attribute error ↓Cut P ↑Cut R ↑Cut F1 ↑MAE (s) ↓
Refrain0.3154.254.7727.6%0.8730.8340.8530.199
MiniMax-H30.2954.034.4536.2%0.7110.7150.7130.242
StoryMem0.1753.754.6869.1%————
Phantom0.1302.183.6551.4%————
RefAlign0.125—3.4247.4%————
  • Human style is the mean of six professional raters in the comparative study (100 requests, side by side, blind to method and SDC).
  • VLM style is a blind Qwen3-VL-32B judge using the human rubric with two frames per shot. Its agreement with the raters is listed below.
  • Attribute error is the fraction of wrong subject attributes (hair, clothing, accessories, category) per reference, excluding deliberately edited attributes.
  • Shot transition structure. TransNetV2 detects the cuts in each video, and a detected cut is correct when it lies within 0.25 s of a requested cut, with each requested cut matched at most once. Cut precision, recall, and F1 are pooled over all requests. Transition MAE (Table 2, Appendix C) measures the timing error of matched cuts and adds a 1 s penalty for each missing or extra cut. These measures apply to methods that generate the full sequence. Shot-wise baselines generate each shot separately and concatenate them at the requested times.

Agreement between measures. Refrain obtains the highest SDC and style ratings, the lowest attribute error, and the most accurate cuts. At the method level, SDC orders Refrain, MiniMax-H3, StoryMem, and Phantom as the six raters do (SDC 0.315 > 0.295 > 0.175 > 0.130, human style 4.25 > 4.03 > 3.75 > 2.18). The VLM judge places StoryMem (4.68) above MiniMax-H3 (4.45). At the video level, following the side-by-side protocol, we compare the outputs within each request. Pairwise agreement is how often a measure orders two methods of the same request as the raters do:

Comparative study (100 requests × 4 methods)Spearman ρPairwise agreement
SDC vs. six-rater style mean0.5376.6%
VLM judge vs. six-rater style mean0.5877.4%

SDC needs only frame features from CSD, VGG, and DINOv2, matches the human method ranking, and agrees with the raters on method pairs nearly as often as the 32B VLM judge.

Independence from the calibration bank. We rebuilt the calibration from five banks of 0.22M to 86.9M real videos. For the largest bank, we newly collected two video sets that share no video with the 2.14M paper bank, IP (17.2M clips of films, series, anime, and games) and Stock (67.6M everyday web videos), and added them to it. The content–style slope stays between 0.388 and 0.411, mean real-video SDC stays stable at 0.337 and 0.326 on the paper and 86.9M banks (Table 14), and the ranking of all seven methods is the same under every bank (Kendall τ = 1.0, Table 15):

Calibration bankVideosSlope κ̂RefrainMiniMax-H3StoryMemPhantomRefAlignVACEHY1.5
DurMix original0.22M0.3880.2910.2720.1600.1140.1100.0940.080
DurMix shared edits0.22M0.3930.3060.2860.1670.1270.1230.1050.087
DurMix independent edits0.22M0.3930.3050.2850.1670.1260.1220.1040.086
Paper bank2.14M0.3990.3150.2950.1750.1300.1250.1080.090
Paper + IP + Stock86.9M0.4110.2770.2600.1580.1020.0990.0840.076
AuthorsOpenReview ↗

Response (Part 2/3)

W2. Class imbalance, agreement, stratified AUC, and cross-source validation.

We added more negative examples, stratified AUC, inter-annotator agreement, and cross-source validation through an additional single-clip study with six professional raters recruited for it (Appendix F, module A, rubric in Figure 6). It covers 150 clips: real two-shot splices from our sources and from the aforementioned IP and Stock collections, natural real multi-shot clips, generated clips from four generators, clips shared with the original 120-clip study, and test–retest repeats. Figure 8 in Appendix F shows examples.

The splices include text-matched and random shot pairs from the same or different collections, which lets content relatedness and style compatibility vary independently. Text-matched shots can still clash in style, and shots from different sources can still be compatible.

  • More negatives and higher agreement. The added study has six times as many consensus-inconsistent clips, and its raters agree more closely:
Original 120-clip study (Appendix E)Added study (Appendix F)
Clips rated by six raters120150
Consensus-inconsistent clips848
Krippendorff's ordinal α0.310.68
Fleiss' κ, five levels / three classes0.19 / 0.340.35 / 0.50
  • SDC and human ratings (Table 7, Appendix F). Of the 150 rated clips, 8 test–retest repeats and 2 clips without a detectable cut are excluded. For the comparative study, ρ is computed within request, matching its side-by-side protocol.
Rater studySpearman ρAUC
Single-clip study (Appendix F), all scored clips0.680.90
Single-clip study (Appendix F), real clips including IP and Stock splices0.770.97
Single-clip study (Appendix F), real clips from IP and Stock only0.790.98
Comparative study (Table 3), generated videos0.530.74

For reference, a single rater agrees with the mean of the other five at ρ = 0.66 within request.

AuthorsOpenReview ↗

Response (Part 3/3)

W3. Generalization of the training edit distribution.

We added hybrid training, clean references, varying edit intensities, unseen instructions, and unedited real footage.

Clean and shared-edit references. The hybrid table below includes the shared-edit test of Table 4a, and the intensity table includes clean references (α=0\alpha = 0).

Hybrid training (Appendix I, Table 21). This setting draws shared or independent edits with probability 0.5 per video.

TrainingIndependent-edit testShared-edit test
Independent (ours)0.3150.299
Shared0.3010.304
Hybrid 1:10.3000.303

Independent editing gives the best result on independent edits, and on shared-edit tests the three settings differ by at most 0.005. The gain comes from the independent supervision, in which references of one video carry different edits and the model learns to reconcile them. In the 1:1 mixture, half of the videos carry one shared edit and provide no such conflict, and the results fall to those of shared-edit training on both test sets. We keep independent editing as the training distribution.

Edit-intensity curve, clean references, and unseen edits (Appendix I, Figure 10, Table 22, examples in Figure 11). For each video reference, every frame is an RGB blend of the original reference RR and its benchmark edit R~\widetilde R, namely (1−α)R+αR~(1-\alpha)R + \alpha\widetilde R, re-encoded before generation. α=0\alpha = 0 is the clean reference and α=1\alpha = 1 the benchmark edit. Because these tests change only video references, they use the requests with video references. On these requests, the benchmark edits give 0.323, 0.301, and 0.306, against 0.315, 0.301, and 0.300 on the full benchmark.

Test referencesIndependent (ours)SharedHybrid
Clean (α=0\alpha = 0)0.3500.3470.336
α=0.25\alpha = 0.250.3340.3190.325
α=0.50\alpha = 0.500.3240.3060.310
α=0.75\alpha = 0.750.3240.3050.315
Benchmark (α=1\alpha = 1)0.3230.3010.306
Unseen instructions0.3180.3140.312
  • Intensity. For independent-edit training, SDC drops from 0.350 to 0.334 at α=0.25\alpha = 0.25 and then stays between 0.323 and 0.324 up to α=1\alpha = 1. Shared-edit training drops from 0.347 to 0.301. At nonzero blend intensities, the hybrid setting lies between the two.
  • Unseen edits. We used 20 instructions newly written and absent from all training data (e.g., gouache, colored pencil, 1980s camcorder, misty blue-hour dawn, faded instant film). Refrain reaches SDC 0.318, against 0.323 on the benchmark edits of the same requests.

Real heterogeneous footage (Appendix I, Table 23, Figure 12). Paired user footage with conflicting styles is hard to collect, and we use real clips from IP and Stock as a proxy for user-uploaded material. We paired clips from different source videos with similar shot descriptions but different visual styles. These unedited pairs form 50 three-shot requests without ground-truth target videos. The two clips are the references of shots 1 and 3, and shot 2 between them is generated from its description alone. In the second example of Figure 12, Refrain renders the cartoon character of shot 3 in the realistic look and colors of shot 1. In the third example, it renders the real dark horse of shot 3 in the animated style of shot 1. Under the main protocol, Refrain reaches higher SDC and preserves the real references more closely:

MetricRefrainMiniMax-H3
SDC ↑0.2170.198
Ref-DINO ↑0.7650.674
Ref-CLIP ↑0.8640.844

Q. Does SDC keep a stable correlation with professional ratings on independently collected real mixed footage?

SDC keeps a stable correlation in the added professional-rater study, which includes real footage from IP and Stock.

On the 90 real clips of the added study, SDC agrees with the six raters at Spearman ρ = 0.77 and AUC 0.97. On the 54 IP and Stock clips alone, it reaches ρ = 0.79 and AUC 0.98. Figure 8 in Appendix F shows style-conflicting splices with low SDC and low ratings, and natural multi-shot clips rated as consistent.

Reviewer_RU7GOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses a specific setting of reference-conditioned multi-shot video generation, where the reference media differ substantially in visual style from the target video. The authors develop a data construction pipeline that synthesizes stylistically modified reference images and videos from target videos. They also design a model architecture tailored to these reference inputs and report results on a held-out test set, outperforming other state-of-the-art open-source methods. In addition, the authors propose an evaluation metric specifically for this setting.

Critical strengths

The paper studies the proposed setting in depth and provides a comprehensive solution covering data construction, model design, and evaluation.

Critical weaknesses

The paper’s main novelty lies in its dataset and benchmark contributions. However, the proposed task setting is not clearly defined or sufficiently explained. The data construction pipeline is also not described comprehensively or in sufficient detail in the main text. Meanwhile, considerable space is devoted to the model architecture, although its design largely reuses existing approaches. Several design choices also lack sufficient ablation studies. For example, it is unclear whether concatenating context along the channel dimension offers advantages over concatenation along the sequence dimension, as used in MiniMax-H3. In my view, reframing the paper around data construction and evaluation for this specific task would substantially strengthen it.

Questions for authors

see weekness

4 comments
AuthorsOpenReview ↗

Response (Part 1/2)

Thank you for your feedback. We have revised the paper in response and address each point below.

W1. Task definition.

We have added a formal Task Definition at the start of Section 3. The Introduction summarizes it, and Table 1 compares it with existing settings.

A request specifies KK shots, a shared style prompt ss, and M≥0M \ge 0 image or video references. Shot kk has a description yky_k and spans [tk,tk+1)[t_k, t_{k+1}) on the output timeline. Reference RiR_i is transformed and placed on the same timeline at [di,di+Di)[d_i, d_i + D_i), although its length ∣Ri∣|R_i| can be shorter or longer than DiD_i. Reference appearances may conflict with the style specified by ss or with one another. The output should satisfy three requirements:

  1. Style. Every shot is rendered in the shared style ss.
  2. Content. Inside its interval, the output keeps the content of RiR_i (subjects, layout, motion) and takes its appearance from ss.
  3. Timing. Cuts occur at the requested times tkt_k.

Relation to existing settings (Table 1). Multi-shot generation (HoloCine, MultiShotMaster) is conditioned on text, and MultiShotMaster also on subject and background images. Subject-to-video (Phantom, RefAlign) and memory-based methods (StoryMem) take image references without cut timing. Temporal-placement methods (VACE, VideoCanvas) place images or variable-length clips at given times, and VACE also supports repainting and restyling. In our setting, references may conflict in style with each other, and the output should share one style across shots.

SettingReferencesDuration ≠ intervalConflicting stylesShared styleCut timing
Multi-shot (HoloCine, MultiShotMaster)none / images–✗✓✓
Subject-to-video (Phantom, RefAlign)images✗✗✗✗
Memory multi-shot (StoryMem)images✗✗✓✗
Temporal placement (VACE, VideoCanvas)images + video✓✗✗✗
Oursimages + video✓✓✓✓

W2. Data construction pipeline in the main text.

We have added a Data Construction Pipeline paragraph to Section 3.1 and reported the statistics in Section 4.1. The pipeline has six stages:

  1. Sources and split. Web videos annotated by Gemini-3 and split by source video into T2V, reference-editing, and test sets.
  2. Reference selection from distinct shots.
  3. First-frame editing with FLUX.1 Kontext under 61 instructions in 8 categories, drawn independently per reference (Eq. 1).
  4. RGB extension for references longer than the available footage. SkyReels-V3 first completes the RGB control, and depth is then re-estimated over the completed clip.
  5. Propagation through the clip with Depth Anything v2 and depth-conditioned Wan-2.1, which keeps the original motion. Shorter references are cropped in latent space.
  6. Filtering. Qwen3-VL-235B screens each clip on three source frames and three edited frames, and re-checks rejected clips on six of each. TransNetV2 removes clips with unintended cuts.

The resulting statistics (Section 4.1, Appendix A):

StatisticValue
Source videos annotated by Gemini-32.15M
T2V / reference-editing / test videos1.85M / 297K / 7.4K
Edit instructions61 in 8 categories
Retained reference conditions309,352
References shorter / longer than their targets200,265 / 109,087
Videos with several references that receive different edit categories80.03%
Reference generation costabout 126 h on 864 NVIDIA H20 GPUs
Edit category (Table 9)InstructionsConditions
Style transfer20125,457
Subject appearance831,655
Object material or colour724,544
Lighting and weather631,175
Background retexturing628,718
Spatial relation621,266
Camera or lens filtering422,672
Motion change423,865

The generator receives the edited references and the text without the edit instructions, and reconstructing the original video supervises it to keep each reference's content while replacing its appearance. Table 2 shows that the model keeps this content (reference subject 0.671, against 0.559 for MiniMax-H3). Appendix A also gives an annotation example and the duration construction. Section 4.1 introduces two separately collected video sets (IP and Stock), used only to validate SDC and to test cross-source references.

AuthorsOpenReview ↗

Response (Part 2/2)

W3. Space devoted to the architecture.

Following your suggestion, the revised paper places greater emphasis on the task, data construction, and evaluation. The contributions lead with self-supervised data construction and DurMix-300K, Section 3.1 describes the pipeline, and Section 4.4 collects the SDC validation across calibration sources and professional-rater studies.

We present A²O as the interface that connects this supervision to one generation model. Image and video references, including clips shorter or longer than their assigned intervals, guide joint generation on one timeline across T2V, V2V, TI2V, and TIV2V. It builds on sparse temporal placement as in VideoCanvas and VACE and on the shot-aware RoPE offsets of MultiShotMaster, and adds 150,528 parameters. Section 3.2 specifies how references of different lengths occupy the target timeline.

These comparisons test the editing strategy, the conditioning interface, and the transfer of DurMix supervision to MiniMax-H3:

ComparisonVariantSDC ↑
Training edits, same model (Table 4a)Shared edits0.301
Independent edits (Refrain)0.315
Injection, same data (Table 4c)Channel0.315
Sequence0.315
DurMix fine-tuning on MiniMax-H3 (Appendix L)MiniMax-H3, zero-shot0.295
MiniMax-H3 + DurMix-300K LoRA0.310

The training-edit comparison uses inconsistent test references. Table 4b further compares shot coordinates and placement:

CoordinatesPlacementMAE (s) ↓SDC ↑
Shot-awareSparse0.1990.315
Shot-awareResample0.2330.310
NoneSparse0.2810.304
NoneResample0.2930.289

W4. Channel vs. sequence concatenation (as in MiniMax-H3).

Channel concatenation reaches the same SDC as sequence concatenation with 56% of the time per denoising step. We trained a sequence-concatenation variant and evaluated it on the same benchmark (Table 4c, Section 4.3, Appendix J). It is trained on the same data with identical settings and appends the reference latents as extra tokens at their shot-aware coordinates.

InjectionSDC ↑Ref-Subj ↑Ref-Bg ↑TA ↑MAE (s) ↓
Channel (Refrain)0.3150.6710.674−0.2210.199
Sequence0.3150.6660.670−0.2500.209

The two variants give nearly the same results. Sequence concatenation takes 1.8× the time per denoising step. For a 241-frame target with three 49-frame references:

CostChannelSequence
Latent frames6161 + 3 × 13 = 100
Tokens1.00×1.64×
Self-attention FLOPs1.0×about 2.7×
Denoising computation per request324 PFLOPs811 PFLOPs
Measured time per denoising step1.0×1.8×
Measured time per training update1.0×2.0×

Channel concatenation keeps the sequence length fixed for any number and length of references, whereas the cost of sequence concatenation grows with each added reference.

We also fine-tuned MiniMax-H3, which uses sequence injection, on DurMix-300K through its native interface with a rank-32 LoRA (166M parameters) trained for 1,000 steps on 10,211 DurMix pairs (Table 4c, Appendix L):

MiniMax-H3SDC ↑Ref-Subj ↑MAE (s) ↓
Zero-shot0.2950.5590.242
+ DurMix-300K LoRA0.3100.5580.237

Appendix L reports two training lengths, and SDC improves at both. DurMix-300K improves SDC under both conditioning families.

W5. Reframing around data construction and evaluation.

We followed this suggestion and have revised the contributions to emphasize data construction and evaluation:

  1. Self-supervised data construction and DurMix-300K. References from different shots are edited independently, reference and target durations are decoupled, and the original coherent video serves as supervision. This pipeline builds DurMix-300K.
  2. A²O unified generation. Four tasks, variable-length references in both directions, and joint generation on one timeline, which lets the same supervision support all four conditioning tasks. Table 4 compares the editing scheme, shot coordinates, placement, and injection.
  3. SDC and its validation. SDC evaluates style under content change. We validate it with calibration banks that add separately collected sources and styles (up to 86.9M videos, identical method ranking, Table 15), with professional ratings of real footage that include independently collected splices (Table 7), and with a comparative study of four generators (Table 3).

Dear Reviewer RU7G,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period will end in about 2 days, we gently request that you read the rebuttal and check whether your concerns are addressed.

Thanks again for your time and help in improving the quality of this paper.

Dear Reviewer RU7G,

Thank you again for your review. As the discussion period ends soon, we would be grateful to know whether our revisions (task definition, data pipeline in the main text, channel vs. sequence ablation, and reframed contributions) address your concerns. We are happy to clarify anything further.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We genuinely thank all reviewers for their constructive comments, which have helped us improve the quality and clarity of our work. In response to the reviews, we have strengthened the paper along three axes: (i) testing the motivation and the transfer of our training data through a cascade baseline and DurMix-300K fine-tuning on MiniMax-H3, (ii) supporting the reliability of SDC with new human studies, independent metrics, and calibration banks of up to 86.9M videos, and (iii) clarifying the task definition, the data construction pipeline, and the framing of contributions. All changes are marked in blue in the revised manuscript.

Additional experiments

  1. Comparative user study (HUpD). Six professional raters compared Refrain, MiniMax-H3, StoryMem, and Phantom on 100 requests, blind to method identity. Refrain receives the highest style-consistency rating (Section 4.2, Table 3, Appendix F).
  2. Cascade baseline (HUpD). References are first restyled toward the shared style and then given to MiniMax-H3 or Refrain. The cascade reaches a similar SDC but lowers reference fidelity and requires about 7 GPU-minutes of preprocessing per reference (Appendix H).
  3. Channel vs. sequence injection (RU7G). We added a sequence-injection variant trained on the same data. It brings no SDC gain over channel injection (0.315 for both) and runs 1.8× slower per denoising step, with a token count that grows with each reference (Section 4.3, Table 4c, Appendix J).
  4. Fine-tuning MiniMax-H3 on DurMix-300K (HUpD). A LoRA trained on 10K DurMix-300K pairs raises the SDC of MiniMax-H3 from 0.295 to 0.310 with similar reference fidelity and cut timing (Section 4.3, Table 4c, Appendix L).
  5. Independent measures (jM4T, HUpD). Human and VLM style ratings, subject attribute error, and cut precision, recall, and F1. Refrain leads on all of them, and SDC orders the methods as the human raters do (Table 3, Appendix G).
  6. Additional professional-rater study (jM4T). 150 clips with 48 consensus-inconsistent clips, real splices from two newly collected sources (IP and Stock), stratified AUC, and inter-annotator agreement. On real IP and Stock clips, SDC reaches Spearman ρ = 0.79 and AUC 0.98 (Section 4.4, Table 7, Appendix F).
  7. Calibration independence (jM4T, HUpD). SDC is recalibrated on five banks of up to 86.9M videos, and the ranking of all seven methods stays the same (Kendall τ = 1.0, Tables 14 and 15).
  8. Generalization of the training edits (jM4T, HUpD). Hybrid training, clean references and graded edit intensities, 20 unseen edit instructions, and unedited real footage from IP and Stock (Appendix I).
  9. Visual quality and backbone choice (HUpD). VideoAlign and aesthetic scores for all methods, and the training and inference cost of HunyuanVideo-1.5 against MiniMax-H3 (Appendices G and K).

Revisions to the manuscript

  1. Task definition (RU7G). A formal definition at the start of Section 3, with a comparison to existing settings in Table 1.
  2. Data construction (RU7G). The pipeline is described in Section 3.1, with details in Appendix A and statistics in Section 4.1.
  3. Framing of contributions (RU7G, HUpD). The contributions now lead with self-supervised data construction and DurMix-300K, and A²O is presented as the interface connecting this supervision to one generation model, building on prior placement and positional designs (Sections 1 and 3).
  4. SDC validation (RU7G, jM4T). Validation across calibration banks and professional-rater studies is consolidated in Section 4.4.
  5. Transfer to MiniMax-H3 (HUpD). Appendix L now specifies how Refrain's training recipe is used in the MiniMax-H3 fine-tuning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.