acceptodds
Under review as a conference paper at ICLR 2027

SmartDirector: Keyframe-Conditioned Cinematic Video Generation with Narrative Pacing Control

Abstract

The narrative quality of a video fundamentally determines its perceptual value. Although existing video generation methods can produce visually appealing content, they predominantly rely on sparse conditioning signals such as text prompts or first/last frames, which limits precise control over narrative structure and temporal pacing. In this paper, we propose SmartDirector, a framework that enhances the narrative capacity of video generation models through multiple keyframes. SmartDirector supports flexible generation scenarios including single-shot generation and multi-shot narrative synthesis. The framework operates in two stages: Director-Gen generates a low-resolution video conditioned on the provided keyframes, and Director-SR refines the output by exploiting high-resolution keyframes as semantic anchors to recover fine-grained details. To enable robust multi-keyframe training, we construct a data pipeline that curates single-shot and multi-shot sequences from movies. Director-Gen improves all six metrics over a same-backbone pairwise-generation baseline on the common 220-example KeyFrame-Compass subset. We further evaluate Director-SR independently on four standard video super-resolution benchmarks and assess the complete two-stage pipeline through a controlled Gen-only versus Gen+SR comparison.

Primary area:
generative models
Keywords:
video generation, keyframe conditioning, controllable generation, multi-shot generation, video super-resolution

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_94h1OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SmartDirector generates single-shot and multi-shot videos from keyframes at specified temporal positions. Director-Gen encodes video chunks independently at keyframe boundaries, then jointly models them using global attention and boundary-aware temporal coordinates. Director-SR uses high-resolution keyframes to refine the generated video. Evaluation on 220 KeyFrame-Compass cases shows improved keyframe execution over a baseline using the same backbone. Additional experiments assess super-resolution and a separate proprietary implementation.

Critical strengths
  1. The combination of independent chunk encoding, global attention, and MC-RoPE addresses a concrete representation mismatch caused by causal temporal compression. The components have clear roles and provide a practical way to enforce multiple visual anchors while sharing context across the video.
  2. Against the same-backbone pairwise baseline, Director-Gen improves all six reported metrics; KPA rises from 0.590 to 0.905 and KFS from 0.634 to 0.783. Component ablations also consistently favor the full model. These comparisons provide substantive evidence for the generation contribution.
Critical weaknesses
  1. Accurate keyframe timing does not establish natural event progression or transitions between anchors. The main evaluation emphasizes keyframe execution and overall visual quality, without directly establishing the pacing quality of the MiniMax H3 implementation. The pacing study in Appendix B uses a different proprietary backbone, so its conclusions cannot be directly attributed to the main model. This limits support for a central claim.
  2. The added value of Director-SR is not isolated. In Table 4, refinement increases resolution but reduces KFS from 0.783 to 0.749, SVQ from 0.856 to 0.824, and DVQ from 0.774 to 0.762. Resolution-normalized evaluation may miss genuine detail gains, but these results do not establish improved perceptual quality. Table 3 quotes SparkVSR results under incompletely matched reference conditions, and Director-SR lacks an ablation without keyframe guidance. Its specific contribution therefore remains uncertain.
Questions for authors
  1. For the main MiniMax H3 implementation, do you have pacing judgments or additional results, such as PAK and RU, that distinguish natural transitions from brief keyframe flashes or prolonged static holds?
  2. What are the source, number, and positions of the high-resolution reference frames in Table 3, and are they comparable to SparkVSR? On generated inputs, can a comparison with simple upsampling and Director-SR without keyframe guidance establish the benefit at a common output resolution?
  3. In Table 2, why does standard RoPE reduce HR from 1.000 to 0.767 when keyframe latents remain fixed? Can you separate reconstruction at the specified timestamps from failures introduced by shot assignment or keyframe matching windows?
2 comments

We thank the reviewer for the detailed reading and for the concrete questions on pacing and on isolating the SR stage. We answer the three weaknesses in order and will carry the changes into the revision.

Q1 · Pacing evidence for H3 (W1)

The evaluation reports six metrics fixed in advance — HR, KFS, KOC, KPA, SVQ, DVQ — of which KPA is the only timing-aware measure; PAK and RU are not computed. KPA establishes that the keyframe event occurs at the requested position within the benchmark's tolerance, so a misplaced or absent keyframe cannot obtain 0.905. It says nothing about the rate of in-between motion: a method may hold a frame nearly static for most of an interval and still satisfy KPA, and only a duration or onset measurement separates a natural transition from a brief flash or a prolonged hold. KPA is a timing metric, not a pacing metric.

The closest evidence is the supplementary material, whose clips are generated from keyframes spaced unevenly in time: the in-between motion is paced by that spacing rather than a fixed rate. The clips are an illustration rather than a measurement, because no established metric exists for how motion should be distributed in time and no reproducible model exposes keyframe timing as a control.

The only pacing-specific values come from the Narrative Pacing dimension of the blind human study in Appendix B: 50.2% of 317 responses prefer SmartDirector versus 13.3% for Dreamina, and 65.8% versus 12.0% on multi-shot cases. That study uses the 32B internal backbone, not H3, so a revision would retitle to "…with Temporal Control" and demote Appendix B to portability evidence.

Q2 · Isolating Director-SR (W2)

  1. Reference conditions. All four benchmarks are public (UDM10, SPMCS, YouHQ40, RealVSR). Following SparkVSR, the reference is a single image obtained by upsampling the first low-resolution frame with PiSA-SR; in the reviewer's terms, the source is the first frame of the low-resolution video, the position is frame 0, and the number is one.

  2. Provenance. Our row follows this protocol; the SparkVSR row is the value reported in its paper. The reference configuration is therefore matched, and the revised caption will state both facts.

  3. Table 4. We agree that it cannot establish a perceptual gain. KeyFrame-Compass resizes before SVQ and DVQ, so the near-constant values indicate preservation of control and quality rather than a sharper 1080p image; our evidence for detail recovery is Fig. 4 and the LPIPS column of Table 3.

  4. The two controls. The right controls are bicubic upsampling at 1080p and Director-SR without keyframe guidance; without the second, keyframe guidance is not isolated, and we will add both.

PipelineHRKFSKPASVQ
Director-Gen, 480p1.0000.7830.9050.856
+ Director-SR, 1080p1.0000.7490.9000.824

On the KFS decrease after SR. The decrease follows from the design, not from an artifact. Director-SR is conditioned on a high-resolution reference and moves the video away from the low-resolution input wherever that input is wrong; that is the function of the stage, and the reason the faces and text in Fig. 4 are recovered. KFS and SVQ compare the refined video against the reference and, at resized resolution, against a quality model, so a refiner that corrects identity drift and distorted faces does not always raise a pixel-faithful similarity score and can lower it when the correction departs from the low-resolution content. The same mechanism explains the PSNR/SSIM losses against SparkVSR alongside the LPIPS gains.

Q3 · Positional indexing and HR (W3)

Fixing a latent does not fix the time the model assigns to it, nor the content it must carry. With r = 4 a regular latent spans four frames and a boundary latent one, yet native indexing advances both by the same step. The model cannot distinguish them and must compromise at every boundary between the two densities; the coordinate is also over-extended by 0.75 steps per boundary — three steps, or twelve frames, for five keyframes — so the modeled duration drifts from the real one, and frames next to a boundary decode as mixtures with their neighbors. This is why SVQ and DVQ move with the coordinate scheme, and why the effect is not a small positional shift. For HR and KFS, the presence of keyframe content is not in question because the latents are clamped; what is in question is whether the event detection finds it at the requested time with sufficient similarity, and the aggregate metrics cannot indicate which of the two failed.

On separating reconstruction from matching-window failures, the aggregate metrics cannot do so on their own, although one dissociation in Table 2 is informative: direct insertion has a higher KPA than standard RoPE (0.582 vs 0.525) but a lower HR (0.661 vs 0.767), so the two failure modes move the metrics in different directions.

Thank you to the authors for their detailed response. After carefully reading the rebuttal, I find that it has largely addressed my previous concerns. Taking into account the comments from the other reviewers, I have decided to maintain my current score.

Reviewer_QRG9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SmartDirector generates single-shot and multi-shot video from several keyframes placed at chosen times. It argues that inserting keyframe latents into a causal 3D VAE sequence creates artifacts. The proposed Multi-Chunk VAE instead splits the video at each keyframe so that every keyframe opens its own chunk and is encoded on its own. The keyframe latents stay clean (timestep zero), and full attention plus MC-RoPE, which gives boundary latents a 1/4 temporal increment, keep the chunks coherent. A second stage, Director-SR, is a Wan-2.2-5B super-resolution model guided by high-resolution keyframes. Director-Gen fine-tunes the 33B MiniMax H3 FL2VA backbone. On 220 KeyFrame-Compass cases it beats a pairwise same-backbone baseline on all six metrics. Director-SR is tested on four VSR benchmarks against SparkVSR numbers copied from that paper.

Critical strengths
  1. The diagnosis about causal VAEs is concrete and useful to practitioners, and the fix is simple. The ablations (direct insertion, standard RoPE, chunk-local attention) separate each component on the same evaluation set. The KPA drop from 0.905 to 0.525 with standard RoPE is a clear result.
  2. The evaluation protocol is careful: the 220-case subset is fixed in advance, a single request is made per case, and reproduced results are kept separate from copied ones.
Critical weaknesses
  1. The novelty is incremental. Clean conditioning latents with per-frame timesteps already exist (PuSA, and keyframe conditioning in LTX-style models), and chunked encoding plus a RoPE offset is a modest step on top. “Narrative pacing control” in the title is supported only by KPA, with no user study or pacing-specific evaluation.
  2. Most comparisons are not controlled. The only same-backbone baseline generates each keyframe pair independently, which is weak by design, while PuSA and Frame Guidance run on much smaller backbones than the 33B H3. Director-SR uses several high-resolution keyframes, but the SparkVSR numbers it is compared with use only the first frame and come from a different run. Hard-inserting keyframes also makes HR and KOC close to 1.0 almost automatically.
Questions for authors
  1. Can you compare against a stronger same-backbone baseline that conditions H3 on all keyframes in one pass by direct insertion with vectorized timesteps, applied at inference only and then with fine-tuning?
  2. Under matched settings, how does Director-SR compare with SparkVSR given the same number of reference keyframes?
  3. Is there any human or pacing-specific evaluation, for example the timing of in-between motion, to support the title’s pacing claim?
2 comments

We thank the reviewer for separating what is a matter of framing from what is a matter of evidence, and for the specific concern about the SR comparison. We take the two weaknesses and three questions in turn and will align the text with the points below.

W1 · Novelty and the pacing claim

Clean conditioning latents with per-frame timesteps (PuSA) and keyframe conditioning in the LTX family are prior art, and we cite them. Our contribution addresses a different issue: on a causal VAE the chunk boundary is itself the obstacle, since a keyframe cannot be placed mid-latent and a clamped latent acquires a content density that no longer matches its temporal extent. We resolve both with two modifications, and a controlled comparison shows the naive alternative degrades.

We agree the title's "narrative pacing control" rests on KPA plus one dimension of a human study on a different backbone (Appendix B, 32B internal model). The title will become "…with Temporal Control", Appendix B will be demoted to portability evidence, and §1 will state that KPA measures event placement rather than the rate of in-between motion.

W2 · Controlled comparisons

  • Which comparisons are controlled. Only the same-backbone comparison is evidence about the design; PuSA and Frame Guidance, reproduced on the same 220 cases but far smaller backbones, belong in a context block.

  • Director-SR. Following SparkVSR, the reference is a single image obtained by upsampling the first low-resolution frame with PiSA-SR, not several references. Training supports multiple keyframes (§3.3), but we use one for a like-for-like comparison; our row follows this protocol, the SparkVSR row is reported in its paper.

  • HR and KOC. Both are near their ceiling by construction under hard clamping, since the keyframe latents are fixed at τ = 0; the caption will say so, and we treat KFS and KPA as informative.

  • Hard insertion. We agree, and this is why the informative values in the table are KFS and KPA rather than HR.

Q1 · Stronger same-backbone baseline

The fine-tuned version is Table 2's "Direct insertion": keyframe latents clamped clean at τ = 0 with vectorized timesteps and native indexing, trained on the same corpus for the same 30,000 steps. The only change is that the sequence is not split at keyframes, which makes it direct insertion into one latent sequence, i.e. the variant applied with fine-tuning. It obtains 0.661 / 0.359 / 0.858 / 0.582 / 0.537 / 0.518, below the full model on all six metrics and below the un-fine-tuned baseline on SVQ (0.537 vs 0.629); even when trained for it, it does not recover placement (KPA 0.582 vs 0.905). Of the two variants, the fine-tuned one is the more informative: an inference-only version could fail simply because the backbone has never seen clean latents there, whereas training removes that confound. We have not included it in this period.

Q2 · Director-SR versus SparkVSR

All four benchmarks are public. Following SparkVSR, the reference is the first low-resolution frame upsampled with PiSA-SR, and the same LQ inputs and metric scripts are used; our row is produced under this protocol, while the SparkVSR row is the value reported in its paper. The methods differ in what the reference is used for: pixel-level VSR is anchored to the low-resolution input and reproduces its errors, so an identity drift is carried through, whereas Director-SR builds its endpoint from the reference latents (Eq. 5) and corrects errors downstream.

BenchmarkLPIPS (ours / SparkVSR)
UDM100.202 / 0.355
SPMCS0.224 / 0.339
YouHQ400.137 / 0.350
RealVSR0.146 / 0.217

PSNR/SSIM are lower on UDM10 and RealVSR, which is expected: a generative refiner trades pixel fidelity for corrected content. This mirrors the practical use of the stage: one frame is super-resolved with an off-the-shelf model to anchor identity. Conditioning on the low-resolution frame itself was less effective, since the model then follows the blurred content.

Q3 · Pacing-specific evaluation

No human study on H3 and no quantitative pacing metric are reported. The closest evidence is the supplementary material, whose keyframes are spaced unevenly in time (0.5–3.2 s), with the in-between motion paced by that spacing rather than a fixed rate; on the same KeyFrame-Compass cases Seedance 2.0 at matched duration filled the interval at a near-constant pace. This difference in rate is what the title means, but it is an illustration rather than a measurement, because no established metric exists for how motion should be distributed in time and no reproducible model exposes keyframe timing as a control.

The only pacing-specific values come from the 32B internal implementation (Appendix B): 50.2% of 317 responses prefer SmartDirector against 13.3% for Dreamina, and 65.8% vs 12.0% on multi-shot cases — a different backbone, so the title will be scoped accordingly.

Thanks for your rebuttal. I raised my review rating by one level.

Reviewer_iZYXOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

SmartDirector is a two-stage framework for multi-keyframe video generation. Director-Gen fine-tunes MiniMax H3 FL2VA to generate 480p video from keyframes at given positions, and Director-SR upsamples it to 1080p using the high-resolution keyframes as references. The two-stage frameworks are widely used, in a lot of formats, and the low-rez followed by high-rez one is the most widely used one. So there is nothing new when it comes to the two-stage design.

The core idea is to encode each keyframe as the first frame of its own VAE chunk, which avoids breaking the causal VAE, plus a small RoPE rescaling at chunk boundaries. Beyond these two modifications, the method relies on the backbone’s native components and established conditioning and VSR practice, so the technical novelty is limited.

Critical strengths

The paper is a system paper that tackles a simple problem solved by two simple implementation tricks: the Multi-Chunk VAE and MC-RoPE. However, the novelty is very limited.

Critical weaknesses

1- Limited novelty. The contribution reduces to chunk-wise VAE encoding and the RoPE adjustment. The proposed framework is very simple; a two-stage design is widely used in many formats, the predominant one is low resolution, followed by a high-resolution one. The two new components introduced in sections 3.2 and 3.3 are very simple. The dataset pipeline and collection would be considered novel if they add substantial value to the community or tackle new tasks that were not possible before this data, which is not the case in the paper. The Director-SR follows existing keyframe-referenced VSR, such as SparkVSR.

2- Gains attribution. It’s unclear whether the same-backbone baseline was fine-tuned on the same internal data and within the same budget. The large jump in per-frame quality (SVQ 0.629→0.856) suggests the gains come mainly from fine-tuning on proprietary data rather than from the design.

3- Unreliable ablation. In the leave-one-out ablation, every variant performs worse than the untuned baseline on SVQ and KFS. For example, changing only the boundary RoPE drops SVQ from 0.856 to 0.556.

Questions for authors

1- Please clarify whether the pairwise MiniMax H3 FL2VA baseline was fine-tuned on the same internal corpus with the same 30k-step budget as Director-Gen. If not, please report that run. This is the single most important experiment for separating the method’s contribution from the data and fine-tuning.

2- What training budget did the ablations use? Please give the budget for each Table 2 variant and confirm whether it matches the full model. Several single-component removals fall below the untuned baseline on SVQ and KFS. For example, the Standard RoPE variant drops SVQ from 0.856 to 0.556. Please explain how changing the temporal coordinate of a few boundary latents can degrade per-frame visual quality this much, and report variance across at least three seeds.

3- Can you provide a cumulative ablation? Please start from the fine-tuned pairwise baseline and add Multi-Chunk VAE, full attention, and MC-RoPE one at a time. This would show how much each component actually contributes on top of the data effect.

4- Please clarify any novel parts that you think I missed or undervalued. I am happy to discuss more about novelty.

These are the main points that if they are tackled proporly I will be inclinded to raise my score.

5 comments

We thank the reviewer for a direct review and for identifying the two points that decide the assessment: the training status of the baseline and whether the ablation is informative. We answer the four questions in order and will carry these clarifications into the revision.

Q1 · Baseline fine-tuning (W2)

The baseline was not fine-tuned. It is the frozen MiniMax H3 FL2VA used in its native first–last-frame mode: each adjacent-keyframe interval is generated from its two endpoints, and the clips are concatenated. §4.1 and §A.3 describe the inputs shared with Director-Gen rather than the training procedure; both will be corrected.

The gaps in Table 1 (KPA 0.590→0.905, KFS 0.634→0.783) therefore reflect both joint multi-keyframe modeling and domain fine-tuning. The design effect can be separated inside the ablation family, where all four runs share corpus, budget (30,000 steps, 64×H20), optimizer, sampling, seed policy and the 220 cases, and differ in one component: SVQ is 0.537 (direct insertion), 0.556 (standard RoPE) and 0.571 (chunk-local attention), against 0.856 for the full model and 0.629 for the un-fine-tuned baseline.

The differences between them are therefore due to the design: fine-tuning alone does not improve visual quality here, as all three variants fall below the un-fine-tuned model, whereas placement tracks the coordinate scheme (0.525→0.905). PuSA and Frame Guidance use far smaller backbones.

A second full-budget run of the pairwise baseline was not possible and will be reported in the revision.

Q2 · Ablation budget, indexing, and variance (W3)

All three variants used the same corpus, 30,000 steps, 64×H20 allocation, optimizer, sampling and seed policy, and 220 cases, so their scores reflect the removed component. Each case is evaluated once; the differences we rely on are 0.19–0.35 and consistent in sign across all six metrics, with seed variance unavailable in this period.

The boundary coordinate decides how much content each latent carries. With r = 4 a regular latent spans four frames and a boundary latent one, yet native indexing advances both by the same step. The model cannot tell them apart, so the frames decoded around a boundary become mixtures of the keyframe and its neighbors. The coordinate is also over-extended by 0.75 steps per boundary — 3 steps, about 12 frames, for five keyframes — so the modeled duration drifts. The first effect shows up in the per-frame metrics, the second in KPA.

Q3 · Cumulative ablation

Read in this order, the rows of Table 2 add exactly one component each.

RowChunkingIndexingAttentionKPA
Direct insertion—nativefull0.582
Standard RoPE✓nativefull0.525
Chunk-local attention✓MC-RoPEchunk-local0.550
Full Director-Gen✓MC-RoPEfull0.905

HR, KFS, KOC, SVQ and DVQ rise at every step; KPA is the exception, and it is the most informative entry: adding chunks while keeping native indexing lowers KPA below direct insertion (0.582→0.525), MC-RoPE alone recovers only part of it (0.550), and the large gain appears only when boundary-aware coordinates and cross-chunk attention are both present. The components are therefore coupled rather than additive, explaining why all three variants fall below the un-fine-tuned baseline on SVQ: each removes half the design and must place hard-clamped keyframes on a mismatched representation.

Q4 · Novelty and undervalued parts (W1)

Our contribution is system-level. Inside Director-Gen, under a causal 3D VAE, inserting a latent at an arbitrary position changes its content density — one frame for a boundary latent, r = 4 for a regular one — so the insertion either breaks the causal chain (direct insertion) or silently re-scales the temporal axis (native indexing). Multi-Chunk VAE with MC-RoPE is the minimal pair that keeps insertion exact on a single global axis, and Table 2 shows either half alone is insufficient. The data pipeline records multi-shot sequences with shot-time metadata, per-character anchors and camera descriptions in one sample (Fig. 3, Appendix C), which makes keyframe-positioned training feasible. Together they let a director place anchors at chosen times and have the backbone follow them rather than drift or distort them.

Director-SR builds on standard keyframe-referenced VSR; its role is generative correction against high-resolution anchors. Pixel-level VSR (SeedVR, FlashVSR, SparkVSR) restores appearance, but because it stays anchored to the low-resolution input it cannot recover the detail that low-resolution generation has already lost. Director-SR's endpoint is built from the reference latents (Eq. 5), so it pulls the sequence back toward the anchored identity and corrects errors introduced downstream of the reference; Fig. 4 shows the distorted face and text persisting in the low-resolution video and in SparkVSR while ours recovers them.

Reviewer_iZYXOpenReview ↗

Remaining Concerns

Thanks for the clarification. My main concern is still Q1: the baseline was frozen, so the Table 1 gains mix your design with fine-tuning on internal data. The ablations can’t separate the two, since each one is fine-tuning plus an incomplete design, and all three beat the frozen baseline on DVQ.

The key experiment needs no training: run the trained Director-Gen in pairwise mode (each interval from its two endpoint keyframes, then concatenate) on the same 220 cases and report all six metrics.

On novelty: Multi-Chunk VAE restarts the causal chain at each keyframe, and MC-RoPE makes the time coordinate match real frame time. But LTX-Video already places separately encoded keyframes at frame-accurate RoPE positions with a first-frame fix, and SparkVSR already fuses HR keyframe latents into LR latents, which is essentially Eq. 5.

Keyframes are clamped and decoded chunk by chunk, so RoPE can’t move them. Why then does Standard RoPE drop HR to 0.767 and KPA to 0.525? Please report where the keyframes actually appear in the outputs, and HR/KPA computed with ground-truth windows.

RoPE acts only through relative offsets, which MC-RoPE leaves unchanged within a chunk. Under chunk-local attention it should therefore do nothing, so the 0.550 row can’t be credited to MC-RoPE, and that “cumulative” step changes two components, not one.

If possible, also report per-frame quality (e.g., MUSIQ) and training-loss curves for the Table 2 runs; the pairwise experiment is what would most directly change my score.

I am willing to raise my score if the author clarifies these points clearly.

AuthorsOpenReview ↗

Response (1/2): attribution and the pairwise control

We thank the reviewer for the second-round assessment and for identifying the control that settles the attribution question. This response is in two parts because of the comment length limit; Part 2 follows and covers the RoPE analysis, the keyframe matching protocol, and the supplementary metrics.

1. Attribution: the ablations and the pairwise control

The baseline is frozen, and §4.1 lists the inputs it receives rather than its training; the revision will state this explicitly and correct the baseline description in Appendix A.3, which currently says its training data is identical to Director-Gen's.

The ablation family evaluates the contribution of individual components under the matched fine-tuning setup; it is not a training-only control. The three variants share the corpus, schedule, optimizer and 30,000 steps and differ in one component. All three fall below the frozen model on SVQ (0.537, 0.556, 0.571 against 0.629) and KFS (0.359, 0.431, 0.450 against 0.634), as expected for incomplete designs. DVQ moves the other way: it rises with training rather than design, which is why all three exceed the frozen model on that metric.

The pairwise control generates each interval from its two endpoint keyframes and concatenates the intervals, with the trained Director-Gen on the same 220 cases, under the same weights, keyframe settings and evaluation protocol.

SettingHRKFSKOCKPASVQDVQ
Frozen baseline0.9100.6340.9940.5900.6290.412
Director-Gen, pairwise0.9230.6291.0000.6510.7380.676
Director-Gen, joint1.0000.7830.9990.9050.8560.774

We acknowledge that the data has an effect: the pairwise setting already improves on the frozen baseline, mainly in motion quality, with DVQ rising from 0.412 to 0.676. The architecture has a larger effect, and it lands on keyframe execution: KFS rises from 0.629 to 0.783 and KPA from 0.651 to 0.905, the two metrics that measure how faithfully and how accurately the keyframes are realized. The data therefore supplies part of the quality, but the architecture supplies the keyframe control and contributes more.

2. Novelty relative to LTX-Video and SparkVSR

We agree that LTX-Video already supports multi-keyframe conditioning at specified frame indices: each conditioning item is independently encoded by the causal VAE and assigned temporal coordinates from its latent coordinates and the requested frame index. For non-initial items the encoded latents are appended as additional conditioning tokens with coordinates adjusted to the target frame index and treated as fixed conditioning; the initial frame is handled separately by inserting the conditioning latent into the initial target latent grid with a causal temporal-coordinate correction.

The difference is where a non-initial keyframe lives. LTX-Video appends the separately encoded keyframe as an additional conditioning token outside the target latent grid, so it reaches the output through attention; the initial frame follows a separate path into the initial target latent grid. In SmartDirector every keyframe is a latent of the target sequence itself: we partition the target video into VAE chunks at keyframe positions, so each keyframe is the first latent of its chunk, and the chunks are processed jointly with full spatio-temporal attention. This representation contains two kinds of temporal unit, a boundary latent representing one frame and a regular latent representing r = 4 frames, and MC-RoPE assigns the boundary latent a 1/r increment, preserving one global temporal coordinate across the concatenated chunks. Because the keyframe is a reference token rather than a pinned latent, its exact position and rendering are decided by the generation model and are not precisely controlled; SmartDirector pins the keyframe into the grid, so the frame at that index is the keyframe by construction.

SparkVSR incorporates separately encoded high-resolution keyframe latents as sparse reference inputs alongside the low-resolution video latents. In contrast, Eq. 5 constructs the conditioned endpoint by replacing the low-resolution latents at the keyframe indices with the corresponding high-resolution keyframe latents, and trains the flow-matching trajectory towards the high-resolution video. The distinction is therefore how high-resolution anchors enter the generation process: reference conditioning versus explicit endpoint construction in the flow-matching trajectory. This formulation places the high-resolution anchors directly in the conditioned endpoint, and Fig. 4 illustrates the resulting refinement behaviour. The same principle extends across our two stages: the keyframe is pinned as a latent in generation and substituted into the endpoint in super-resolution, so the two stages form a single, self-consistent design.

Comments on the paper

1 comment
Area_Chair_LPC4OpenReview ↗

Please engage into the discussion with authors and fellow reviewers

Dear reviewers,
The authors have already provided their responses. Please check whether the authors' rebuttal has addressed your concerns.
Please engage into the discussion with authors and fellow reviewers.
Thanks!
Best,
AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.