- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Refrain, a multi-shot video generation model built on HunyuanVideo-1.5 that keeps a consistent style across shots even when references conflict in style. In training, references from different shots are edited independently, and the model learns to reconstruct the original video under a shared style prompt. The authors also propose SDC, a cross-shot style consistency metric, and DurMix-300K, a dataset of edited references.
Critical strengths
- The design is unified and efficient. One backbone handles four tasks without extra modules, and the cost stays the same as the base model.
- The baselines are recent and strong, including MiniMax-H3, and the baseline settings are described in detail.
- The training task is simple and clever. Editing references independently and reconstructing the original video gives free supervision for cross-shot style consistency.
Critical weaknesses
- The human study only covers Refrain's own results. It is used to validate the SDC metric, but it does not compare Refrain with any baseline. A user study that compares the methods is needed.
- I have some doubts about the motivation. Even if references with different styles can hurt the result, a simple solution exists: first apply an image or video style transfer model to the references, so that they match the target style, and then generate. This cascaded approach is more complex, but it may work just as well. The paper does not compare with or discuss this option.
- The architectural novelty is limited. Sparse temporal placement is close to VideoCanvas, VACE and SparseCtrl. The shot-aware RoPE is taken from MultiShotMaster. Channel concatenation with a mask is a standard base-model design.
- The visual quality of the videos in the supplementary material is somewhat low. I understand this may come from the HunyuanVideo-1.5 base model. However, many stronger base models are now available. Why choose HunyuanVideo-1.5? If the design only works on this base model, its upper bound may be low. Newer models such as LTX-2.3 or MiniMax-H3 can generate video with audio, and they can also generate video only. I would like the authors to explain this choice.
- The SDC metric is reasonable to some extent, but the overall gains are not large. The task setting is also narrow. Overall, the technical contribution is limited.
Questions for authors
- Can you add a baseline that first applies style transfer to the reference images or clips? Most of the current baselines are general models without special training for this task, so they are not expected to handle style conflicts.
- Can you run a user study that compares Refrain with the baselines on style consistency and overall quality?
8 comments
Response (Part 2/4)
W3. Architectural novelty.
Thank you for raising this point. The A²O interface builds on prior placement and positional designs, and, as you noted, it keeps the design unified and efficient. The novelty of Refrain lies in the problem it addresses and in how it learns that problem from ordinary videos.
The problem. A multi-shot video is generated from several image or video references whose styles conflict with each other and with the requested style, while each reference keeps its content and the cuts follow the requested times. Table 1 compares this setting with multi-shot, subject-to-video, memory-based, and temporal-placement generation:
| Setting | References | Duration ≠ interval | Conflicting styles | Shared style | Cut timing |
|---|---|---|---|---|---|
| Multi-shot (HoloCine, MultiShotMaster) | none / images | – | ✗ | ✓ | ✓ |
| Subject-to-video (Phantom, RefAlign) | images | ✗ | ✗ | ✗ | ✗ |
| Memory multi-shot (StoryMem) | images | ✗ | ✗ | ✓ | ✗ |
| Temporal placement (VACE, VideoCanvas) | images + video | ✓ | ✗ | ✗ | ✗ |
| Ours | images + video | ✓ | ✓ | ✓ | ✓ |
The cascade experiment (W2) also motivates this setting. Restyling the references before generation reaches a similar SDC but loses reference content, TA, and timing. Refrain resolves the conflict during generation and keeps the references.
The insight. Paired data for this problem are hard to collect. Even our 86.9M-video calibration bank with the IP and Stock collections (W5) contains few real clip pairs with matching content and conflicting styles. We build such pairs from coherent multi-shot videos. Editing the references of different shots independently creates the conflict, and the original video remains the reconstruction target. The model learns to keep the content of each reference and to replace its appearance with the shared style, without manual labels. Setting reference durations shorter or longer than their intervals extends the same supervision to duration mismatches. The pipeline yields DurMix-300K at scale, and the same data also improve MiniMax-H3 through a LoRA (W4).
The comparisons below test the editing strategy, the conditioning interface, and the transfer of this supervision to MiniMax-H3:
| Comparison | Variant | SDC ↑ |
|---|---|---|
| Training edits, same model (Table 4a) | Shared edits | 0.301 |
| Independent edits (Refrain) | 0.315 | |
| Injection, same data (Table 4c) | Channel | 0.315 |
| Sequence | 0.315 | |
| DurMix fine-tuning on MiniMax-H3 (Appendix L) | MiniMax-H3, zero-shot | 0.295 |
| MiniMax-H3 + DurMix-300K LoRA | 0.310 |
The training-edit comparison uses inconsistent test references, and the MiniMax-H3 LoRA keeps similar reference fidelity (details under W4). The training improves SDC with both conditioning interfaces and transfers to MiniMax-H3 through a LoRA, without architectural change.
The duration supervision also holds across reference lengths. Table 26 groups the requests with video references by reference duration relative to the assigned interval, and Refrain leads MiniMax-H3 on SDC, reference subject, and transition timing in all four groups:
| Reference vs. interval (Table 26) | Shorter | Equal | Longer | Mixed |
|---|---|---|---|---|
| SDC ↑, Refrain | 0.334 | 0.329 | 0.329 | 0.307 |
| SDC ↑, MiniMax-H3 | 0.314 | 0.249 | 0.276 | 0.254 |
| Ref-Subj ↑, Refrain | 0.678 | 0.649 | 0.685 | 0.608 |
| Ref-Subj ↑, MiniMax-H3 | 0.582 | 0.545 | 0.564 | 0.521 |
| MAE (s) ↓, Refrain | 0.166 | 0.208 | 0.206 | 0.204 |
| MAE (s) ↓, MiniMax-H3 | 0.238 | 0.318 | 0.255 | 0.244 |
Response (Part 3/4)
W4. Visual quality and the choice of HunyuanVideo-1.5.
Thank you for this question. DurMix-300K and the training objective also apply to MiniMax-H3, a backbone with higher visual quality, and raise its SDC. We first explain the choice of HunyuanVideo-1.5 and its effect on visual quality, and then show the results on MiniMax-H3.
Backbone choice. We chose HunyuanVideo-1.5 for its training and inference efficiency (Table 2, Appendix K):
| HunyuanVideo-1.5 (Refrain) | MiniMax-H3 | Ratio | |
|---|---|---|---|
| Parameters | 8.49B | 33.12B | 3.9× |
| Median time per training update | 14.2 s (full parameters) | 74.8 s (rank-32 LoRA) | 5.3× |
| Denoising computation per request | 324 PFLOPs | 551 PFLOPs | 1.7× |
Both training times are measured on 64 NVIDIA H20 GPUs with 8-way sequence parallelism and a global batch of 8.
HunyuanVideo-1.5's conditioning channels also suit the A²O canvas, which adds only 150,528 parameters.
Visual quality. We scored all methods (Table 18, Appendix G):
| Method | VideoAlign VQ ↑ | VideoAlign MQ ↑ | Aesthetic ↑ |
|---|---|---|---|
| HunyuanVideo-1.5 | −0.492 | −0.828 | 0.506 |
| Refrain | −0.492 | −0.736 | 0.506 |
| Phantom | −0.114 | −0.548 | 0.536 |
| RefAlign | −0.320 | −0.748 | 0.522 |
| StoryMem | −0.550 | −0.586 | 0.530 |
| VACE | −0.632 | −0.786 | 0.486 |
| MiniMax-H3 | −0.451 | −0.661 | 0.534 |
Visual quality tracks the base model. Refrain equals HunyuanVideo-1.5 in VQ and aesthetics and scores higher in motion quality. In VQ it ranks above StoryMem and VACE and below Phantom, RefAlign, and MiniMax-H3. Phantom and RefAlign use 14B backbones and generate each shot as a separate single-shot clip. Refrain generates the 241-frame multi-shot sequence jointly, which allows cuts at the requested times (MAE 0.199 s).
Results on MiniMax-H3. We fine-tuned MiniMax-H3 on 10K DurMix-300K pairs through its native sequence-reference interface with a rank-32 LoRA (166M parameters) and no architectural change (Table 4c, Appendix L):
| MiniMax-H3 | SDC ↑ | Ref-Subj ↑ | TA ↑ |
|---|---|---|---|
| Zero-shot | 0.295 | 0.559 | −0.267 |
| + DurMix-300K LoRA | 0.310 | 0.558 | −0.280 |
The LoRA is trained for 1,000 steps with noise levels sampled from a logit-normal distribution and a self-distillation term toward the released model. It raises SDC with reference fidelity and cut timing at the zero-shot level (MAE 0.237 s vs. 0.242 s), and SDC improves at both training lengths in Appendix L. LTX-2.3 conditions on references through a sequence interface similar to MiniMax-H3, and the MiniMax-H3 results indicate how the same data transfer to this class of backbones. We plan to fine-tune MiniMax-H3 on the full DurMix-300K.
Response (Part 1/4)
Thank you for your feedback. We have revised the paper and added the requested comparisons, as detailed below.
W1 / Q2. Comparative user study.
We ran the requested comparison on 100 test requests with six professional raters, blind to method identity (Table 3, Section 4.2, Appendix F). Raters saw the shot descriptions, the references, and four anonymized outputs side by side, and scored six dimensions from 1 to 5 with the rubric in Figures 6–7.
Refrain receives the highest style-consistency rating of the four methods and is rated higher than StoryMem and Phantom on all six dimensions. Against MiniMax-H3, the reference-fidelity ratings are close (4.19 vs. 4.27), and MiniMax-H3 is rated higher in text adherence, visual quality, motion, and overall preference.
The lower visual-quality and motion ratings help explain the overall-preference gap despite the higher style-consistency rating. Refrain matches its HunyuanVideo-1.5 base in VideoAlign VQ and aesthetics and improves motion quality (W4), and its visual quality stays at the level of the chosen backbone.
W2 / Q1. Motivation and the cascade baseline.
We added this baseline. The cascade reaches a similar style gain, but it needs about 7 NVIDIA H20 GPU-minutes of preprocessing per reference and lowers both reference-fidelity measures. Refrain reaches this style consistency in a single generation pass. We implemented the cascade with the editing pipeline that produced DurMix-300K. Each test reference is restyled toward the request's shared style (FLUX.1 Kontext on the first frame, depth-conditioned Wan-2.1 propagation), passes the same filters, and is given to MiniMax-H3 or Refrain (Appendix H, Table 20). We report the requests with video references. Reference fidelity is measured against the original references and, for the cascade, also against the restyled references the generator received.