- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SEGUE for one-shot transition effect generation. Given a reference video showing a transition, the model transfers that effect to a new pair of video clips or images without requiring a text description of the effect. The method uses neutral text prompts and one-way attention during training, followed by null-reference guidance at inference. The authors also build a dataset with 1,692 transition classes and evaluate the method on held-out effects and endpoints. SEGUE clearly outperforms the adapted baselines in both automatic metrics and human evaluation, especially when the effect description is removed from the prompt.
Critical strengths
-
The task is practical and clearly motivated. Existing methods often depend on a text description of the effect, which cannot fully describe its motion or timing. SEGUE instead learns from the reference video itself. The idea of using a dissolve as the negative reference is simple but appears effective.
-
The experiments are fairly comprehensive. The paper includes held-out effects and endpoints, comparisons with several related methods, automatic metrics, and a human study. The performance gap is large, particularly with neutral prompts, which supports the claim that SEGUE makes better use of the visual reference.
Critical weaknesses
-
I found that it is quite hard to tell which parts of SEGUE are actually responsible for its improvement. The method includes neutral-prompt training, one-way attention, and null-reference guidance, but the main ablation mostly focuses on removing the guidance. The “Plain LoRA” baseline in the appendix is not explained clearly enough, and the paper does not separately remove the neutral prompt or the attention mask. The comparison between the dissolve null and an empty null is also only qualitative.
-
The evaluation of the main two-sided transition task is quite small. It contains only 38 test samples, while most of the evaluation focuses on the single-sided visual-effect-transfer setting. I am also unsure how the proposed transition-fidelity metric is computed for EffectData classes, since the dataset description appears to keep only one reference video per effect while the metric excludes the reference itself. The paper does not report how well this new metric correlates with human judgments at the sample level.
Questions for authors
- Could the authors separately ablate neutral-prompt training and the one-way attention mask? At the moment, it is hard to know which training choice contributes most to the result.
- What exactly is included in the “Plain LoRA” baseline in Table 6? Does it use effect prompts, full attention, or null-reference guidance?
- How is transition fidelity computed for EffectData effects when there is only one reference video for each effect and the reference itself is excluded from the comparison set?
- Does the transition-fidelity metric correlate with human preferences on individual samples? This would help establish that the large automatic improvement reflects a real perceptual improvement rather than a property of the metric.
- Why is the two-sided TEG evaluation limited to 38 samples? Since this is the main new task introduced by the paper, results on a larger two-sided set would make the overall claim more convincing.
6 comments
Response to Reviewer icXV (2/3)
W1.2 and Q2: What Plain LoRA is
What exactly is included in the "Plain LoRA" baseline in Table 6? Does it use effect prompts, full attention, or null-reference guidance?
Plain LoRA (now Table 12) was our first experiment, where we tested and verified text dependence. It fine-tunes the same LTX-2 backbone with LoRA (rank 32, learning rate 2e-4, 5,000 steps) on 139 Higgsfield clips covering 26 effects. It is trained with neutral prompts, and it uses full attention, with no null-reference guidance. We have added this baseline definition to Appendix E (paragraph Plain LoRA, Table 12).
With neutral-prompt training, Plain LoRA becomes less dependent on text, with a gap of only 7.6 on seen samples (Table R9). However, on held-out effect classes, TF drops to 61.9, and the gap grows to 19.6, showing poor generalization and greater reliance on text. This motivated our cross-paired dataset and the other components of SEGUE, which achieve 93.4 TF with a gap of just 1.2 on the same classes.
Table R9 reports Transition Fidelity with neutral (N) and effect (E) prompts, along with the gap between them (|Δ|), across the three tiers of our full evaluation set, both tasks pooled.
Table R9. Plain LoRA vs. SEGUE by tier (N, E: TF with the neutral / effect prompt; |Δ|: their gap)
| Tier | Plain LoRA N | E | Δ | SEGUE N | E | Δ |
|---|---|---|---|---|---|---|
| Seen effects | 85.2 | 92.8 | 7.6 | 96.1 | 96.7 | 0.6 |
| Held-out instances | 72.6 | 84.3 | 11.7 | 87.9 | 89.9 | 2.0 |
| Held-out effect classes | 61.9 | 81.5 | 19.6 | 93.4 | 94.6 | 1.2 |
W1.3: Dissolve null vs. empty null, in numbers
The comparison between the dissolve null and an empty null is also only qualitative.
We compared both nulls on 68 held-out class samples (Table R10; added as Table 11, App. E). They achieve similar TF, but the dissolve produces smoother transitions near the target endpoints and less copying of scenes and objects from the reference endpoints. This is consistent with the reference-copying failure shown for the empty null in Fig. 13.
Table R10. Negative references for NRG
| Negative reference | TF ↑ | Motion fid. ×100 ↑ | Ref. sim. ×100 ↑ | Seam-free % ↑ | Ref.-endpoint sim. ×100 ↓ |
|---|---|---|---|---|---|
| NRG with each negative (w = 6) | |||||
| Dissolve (ours) | 91.7 | 22.0 | 96.4 | 95.6 | 29.1 |
| Empty (reference dropped, Fig. 13) | 91.6 | 22.7 | 96.5 | 91.2 | 31.2 |
W2.1 and Q5: Size of the two-sided evaluation
Why is the two-sided TEG evaluation limited to 38 samples? …
Two-sided creative transitions with several real videos per class came only from Higgsfield, and most of its held-out classes are single-sided, as are the EffectData effects. We further expanded the two-sided evaluation to all tiers of our full evaluation set (148 creative transition videos over 12 classes) and to 100 videos from 10 held-out procedural shader families (248 two-sided videos in total), and provided the results in Table R11. Supplementary Part 3 shows SEGUE outputs on six of the held-out shader families (3.5–3.16). We also want to clarify that related works use test sets of 48 samples (Video-As-Prompt) and 56 reference videos (refVFX), making our previous evaluation set in a comparable range (Table R12).
Table R11. Expanded two-sided evaluation
| Set | Videos | Classes | Base LTX-2 | VACE | Wan2.1 FLF2V | refVFX | SEGUE |
|---|---|---|---|---|---|---|---|
| Held-out classes (two-sided set of Table 3 in the paper) | 76 | 5 | 49.8 ± 2.7 | 51.9 ± 2.5 | 56.6 ± 2.3 | 49.8 ± 2.8 | 97.0 ± 0.7 |
| Held-out-instance classes | 64 | 7 | 59.7 ± 3.0 | 57.2 ± 2.8 | 62.6 ± 3.0 | 54.2 ± 2.5 | 97.4 ± 0.5 |
| Seen classes | 8 | 2 | 58.1 ± 5.9 | 50.6 ± 9.4 | 66.4 ± 7.4 | 38.7 ± 4.6 | 97.8 ± 1.0 |
| All two-sided samples | 148 | 12 | 54.5 ± 2.0 | 54.1 ± 1.8 | 59.8 ± 1.8 | 51.1 ± 1.8 | 97.3 ± 0.4 |
| Held-out shader families | 100 | 10 | 61.3 ± 2.0 | 78.7 ± 1.4 | 80.0 ± 1.2 | 78.0 ± 1.2 | 97.8 ± 0.5 |
Table R12. Evaluation set sizes of related work
| Method | Task | Test set | Human study |
|---|---|---|---|
| SEGUE (ours) | Reference + both endpoints, or start only | 38 two-sided + 183 single-sided held-out samples, 2 seeds (442 videos); 384 samples over all tiers | 29 raters, 197 pairs on 76 samples; VLM judge on 625 pairs |
| Video-As-Prompt | Reference-based effect | 48 (24 conditions × 2) | 20 researchers |
| refVFX | Reference-based effect | 56 reference videos, 28 unseen effects | over 500 annotators |
| VFXMaster | Reference-based effect | not stated (15 test + 20 out-of-domain effects) | 50 participants |
Response to Reviewer icXV (3/3)
W2.2 and Q3: TF for EffectData effects
How is transition fidelity computed for EffectData effects when there is only one reference video for each effect …?
As the reviewer points out, TF requires comparison with other clips of the same effect. To make this evaluation reliable, we selected held-out EffectData effects from the largest classes, each containing at least 45 clips. The model receives one clip as its reference, while TF compares the output against a fixed set of eight other clips from the same class, excluding the input reference (now stated in App. C). “One reference video per effect” in App. A refers only to the model’s input.
W2.3 and Q4: Sample-level agreement of TF with humans
Does the transition-fidelity metric correlate with human preferences on individual samples? …
Yes. TF achieves the highest agreement with both human preference (86.7%) and VLM judgments (87.3%) among all evaluated metrics (Table R13). This holds for both two-sided and single-sided transitions. For comparison, the VLM judge agrees with human raters 89.9% of the time. These results support TF’s ability to reflect human preferences on individual sample pairs (Table R13; added as Table 5, App. C). Examples where TF and the raters disagree are in Supplementary Part 1 (1.1–1.5).
Table R13. Agreement of each metric with the human preference and the VLM judge
| Metric | Agreement with human preference | Agreement with VLM judge |
|---|---|---|
| Transition fidelity | 86.7% (two-sided 89.5%, single-sided 85.3%) | 87.3% (two-sided 92.7%, single-sided 84.7%) |
| Reference similarity | 81.9% (two-sided 80.7%, single-sided 82.5%) | 85.1% (two-sided 87.3%, single-sided 84.0%) |
| Motion fidelity | 70.4% (two-sided 66.7%, single-sided 72.4%) | 71.3% (two-sided 67.9%, single-sided 73.1%) |
| Action KL | 59.3% (two-sided 59.6%, single-sided 59.2%) | 60.9% (two-sided 65.5%, single-sided 58.8%) |
To further evaluate how well TF captures known transition effects, we run retrieval and separation tests on 222 Higgsfield videos covering 39 effect classes. All metrics are used without training or tuning for these tests, and no SEGUE-generated videos are involved. Retrieval accuracy measures how often a video’s closest match shows the same effect. Separation measures how clearly each metric distinguishes same-effect pairs from different-effect pairs, using Cohen’s d. TF achieves the highest retrieval accuracy (73%) and the second-highest separation (1.39), supporting its ability to recognize and distinguish effects across different videos (Table R14; added as Table 6, App. C).
Table R14. Effect retrieval and separation on real Higgsfield videos
| Metric | Retrieval accuracy ↑ | Separation (Cohen’s d) ↑ |
|---|---|---|
| Transition Fidelity | 73% | 1.39 |
| Reference similarity | 64% | 1.54 |
| Action KL | 32% | 0.60 |
| Motion fidelity | 8% | 0.43 |
Response to Reviewer icXV (1/3)
Videos for W2.1 are in the updated supplementary material (open index.html → Rebuttal material, Part 3).
We thank the reviewer for recognizing the practical motivation, the effectiveness of the dissolve negative, and the strong results across automatic metrics and human evaluation, particularly without effect descriptions. We appreciate the detailed suggestions, which have helped us strengthen the ablations, expand the two-sided evaluation and clarify the baselines and metric.
We have provided the full ablation study (W1.1, Q1); clarified plain LoRA (W1.2, Q2); compared the dissolve and empty nulls in numbers (W1.3); expanded our evaluation set and compared its size with related work (W2.1, Q5); EffectData fidelity uses the effect's other clips (W2.2, Q3); and transition fidelity agrees with the raters on individual samples (W2.3, Q4).
Evaluation setting. The new experiments use a class-balanced subset of our evaluation set: 134 samples (all 26 seen, 40 held-out instance, 68 held-out class; tiers as in App. G). Each table states which tier it uses.
Metrics. Transition fidelity (TF) measures whether the output reproduces the reference’s transition effect, scaled so that real videos score at the top. Motion fidelity, reference similarity, and action KL compare the output with the reference. Seam-free rate is the percentage of clips that have smooth connections at the boundaries between the generated transition and the given endpoints (Table 2 and App. C). Reference-endpoint similarity measures copying of scenes and objects from the reference endpoints into the generated output (Table R10).
W1.1 and Q1: Separate ablations of the neutral prompt, one-way attention and NRG
We trained all four combinations on the same data for half the full training schedule:
We evaluated each on held-out effect classes with both neutral (N) and effect (E) prompts. Table R8 reports Transition Fidelity and the gap between the two prompts (|Δ|), which measures text dependence.
Neutral training prompts. These have the largest effect, reducing |Δ| from 37.9 to 16.7 with full attention and from 29.3 to 14.5 with one-way attention.
One-way attention. It further reduces text dependence under both training prompts: from 37.9 to 29.3 with effect prompts and from 16.7 to 14.5 with neutral prompts.
Combined. The SEGUE recipe achieves the highest neutral-prompt TF (73.9) and the smallest gap (14.5) among the four ablation models. Full training improves TF to 86.9 and reduces the gap to 3.6. NRG further improves these to 91.7 and 1.7, respectively.
This table is also in the revised paper (Table 8, Appendix E).
Table R8. Prompt and attention ablation on held-out class samples (N, E: TF with the neutral / effect prompt; |Δ|: their gap)