acceptodds
Withdrawn from ICLR 2027

On Time: Learning to Follow Event Timelines in Long Video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

On-time long video generation requires actions to follow prescribed start and end times, preserve intended overlaps, and remain associated with the correct participants throughout an extended rollout. We introduce OnTime, a framework for this task that combines structured event timelines with a few-step autoregressive generator trained on short sequences. Its core mechanism, Rolling Timeline Conditioning (RTC), preserves global action intervals while assigning persistent scene and identity descriptions bounded local support, so their encoded durations do not grow with the full rollout. Event cross-attention retains the global timeline while cached visual history uses window-relative positions. We train the causal generator through teacher-guided initialization and Self-Forcing. In 60-second rollouts from 10-second training sequences, our model leads the compared systems in event-level visual quality and InternVideo2 alignment, while LongLive leads full-video Temporal-Following Score. Our code will be released.

Primary area:
generative models
Keywords:
generative models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jDyJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies how to make long videos follow a specified schedule of events. Building on existing TIE and RoTE, the author introduces Rolling Timeline Conditioning (RTC). RTC keeps scene and character descriptions available throughout generation, assigns them to the current local time window, and includes nearby actions while preserving their original start and end times.

Critical strengths

RTC addresses the problem in extending temporal conditioning to longer videos. It keeps scene and character descriptions available without making their assigned duration grow with the full video. Actions retain their original intervals, including those that continue across generation blocks. The complete system shows good long video results. The examples and the small human study favor OnTime in visual quality and continuity. Producing these longer sequences from short-sequence training is useful.

Critical weaknesses

The contribution is somewhat incremental, and RTC’s effect has not been isolated. TIE and its RoTE encoding come from prior work, with RTC as the main addition. The experiments compare complete systems, making it hard to tell how much RTC contributes. A comparison within the same model would help show what RTC itself adds. The evidence for improved action timing is mixed. The human study favors OnTime on a small, selected set of videos, while the broader timeline-following evaluation favors LongLive. The event-clip results do not settle this issue. For example, watering could look correct between 10 and 20 seconds even if it starts early or continues too long.

Questions for authors

Could the authors clarify how much RTC contributes to the reported improvements, given that much of the framework comes from prior work? It would also help to explain the mixed timing results and provide stronger evidence that OnTime follows the requested schedule more accurately. Please see the weaknesses for the specific concerns.

2 comments

We thank Reviewer jDyJ for the constructive comments on isolating RTC and evaluating action boundaries.

Isolating RTC within the same model. We evaluate the same trained OnTime model on all 128 60-second timelines with and without RTC (where w/o RTC disables rolling conditioning and uses full-video persistent intervals), reporting all Table 1 metrics alongside our full-video temporal localization metrics:

SettingVQ ↑MQ ↑TA ↑Overall ↑InternVideo2 ↑TFS ↑AP (%) ↑Time mIoU ↑Loc. mIoU ↑
OnTime w/o RTC-0.20650.2103-0.8622-0.85850.02410.303763.440.14660.2702
OnTime w/ RTC (Ours)0.08790.5376-0.59710.02840.01940.371478.850.24540.3529
Δ\Delta (w/ RTC −- w/o RTC)+0.2944+0.3273+0.2651+0.8869-0.0047+0.0677+15.41+0.0988+0.0827

RTC improves visual quality, motion, and timeline adherence (+0.8869 VideoReward Overall, +0.0677 TFS, +15.41% AP, +0.0988 Time mIoU). Without RTC, all action tokens remain active across the video and leak beyond their target intervals, keeping cropped-clip InternVideo2 slightly higher (-0.0047) while degrading temporal localization (Time mIoU 0.1466 vs. 0.2454).

Action-boundary accuracy and mixed timing results. Cropped-clip metrics do not penalize actions that start early or continue too long, whereas full-video TFS (Eq. 8) assigns 60% weight to frame, global, and consistency terms (F‾,G,C\overline{F}, G, C). Static LongLive obtains a higher TFS (0.4165) by generating largely static videos (MQ = 0.1297 vs. OnTime's 0.5376), avoiding drift at the cost of missing scheduled transitions (LongLive Interactive, which switches prompts, scores 0.3310 vs. OnTime's 0.3714).

To directly test start/end boundary accuracy over the full 60s video, we evaluate Action Presence (AP), Time mIoU (temporal IoU between predicted and requested action intervals, scoring missing actions as 0), and Loc. mIoU (temporal IoU over localized actions), which penalize both early onsets and late endings:

MethodAP (%) ↑Time mIoU ↑Loc. mIoU ↑
Self-Forcing55.910.09100.1814
LongLive58.180.09690.1842
LongLive Interactive70.250.20240.3300
OnTime w/o RTC63.440.14660.2702
OnTime w/ RTC (Ours)78.850.24540.3529

OnTime leads all baselines in both action presence (78.85%) and interval overlap (0.2454 Time mIoU, 0.3529 Loc. mIoU), more than doubling static LongLive's Time mIoU (0.0969) and aligning with the human timing evaluation in Table 2.

AuthorsOpenReview ↗

Final Remarks

Dear Reviewer jDyJ,

Before the discussion period ends, we wanted to check whether our new within-model RTC ablation and full-video action-boundary metrics (AP, Time mIoU, Loc. mIoU) have addressed your questions on isolating RTC and action timing. Please let us know if anything remains unclear.

Reviewer_WskDOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies long video generation from event timelines with explicit participants and start and end times. OnTime combines TIE's interval conditioning with self forcing and it can generate 60-second videos after training on 10-second sequences.

Critical strengths
  1. I think the idea makes sense for this very specific setting. You start from a pretrained TIE teacher, then distill it to a causal model, then force the model to look at a sliding window during each block's generation.
Critical weaknesses
  1. There is very limited visualization in the paper and no supporting materials, so it's very hard to understand how well this method worked.
  2. I have concerns about the novelty. It basically applies self forcing + TIE. The contribution seems marginal.
  3. Also, restricting the context window limits the work's upper bound. It's not an extendable method.
Questions for authors

See weakness

2 comments

We thank Reviewer WskD for the helpful feedback.

Visualizations (W1). We expanded the qualitative results in the revised manuscript (Figure 2) and Appendix G (Figures 6–11) with additional 60s comparisons covering single- and multi-participant timelines as well as overlapping events.

Novelty beyond Self-Forcing + TIE (W2). Directly applying Self-Forcing to TIE fails when extrapolating from 10s training sequences to 60s rollouts due to a conflict between interval duration encoding and streaming generation. Full-video [0,60s][0, 60\text{s}] persistent context stretches RoTE's duration modulation D(r)D(r) to 6×6\times the 10s training range, while clipping intervals to local blocks breaks cross-block action timing. RTC (Section 2.3, Eqs. 2–5) resolves this by bounding persistent context to local support (≤9.5s\le 9.5\text{s}) while preserving unclipped global intervals [sj,ej][s_j, e_j] for actions (leveraging Corollary 1's shift invariance). Evaluating the same trained OnTime model on all 128 60s timelines with and without RTC (Table 1 metrics plus full-video action presence AP and boundary overlap Time/Loc. mIoU) confirms RTC's impact:

SettingVQ ↑MQ ↑TA ↑Overall ↑InternVideo2 ↑TFS ↑AP (%) ↑Time mIoU ↑Loc. mIoU ↑
OnTime w/o RTC-0.20650.2103-0.8622-0.85850.02410.303763.440.14660.2702
OnTime w/ RTC (Ours)0.08790.5376-0.59710.02840.01940.371478.850.24540.3529
Δ\Delta (w/ RTC −- w/o RTC)+0.2944+0.3273+0.2651+0.8869-0.0047+0.0677+15.41+0.0988+0.0827

Context window and scalability (W3). Bounding the active context is standard in streaming long-video generation: LongLive Interactive also restricts visual context to a short sliding KV window and restricts text context to a single local prompt per segment via sequential prompt updates and KV recaching. Conversely, feeding an unrestricted global prompt hits UMT5's hard 512-token limit on 60s multi-participant timelines (Appendix D.1), forcing single-prompt baselines to compress or truncate descriptions and diluting attention across dozens of inactive events (Figure 3). Compared with LongLive Interactive's discrete per-segment prompts, RTC keeps active text tokens, persistent duration modulation, and KV cache size bounded (O(1)O(1)) while still preserving unclipped global intervals [sj,ej][s_j, e_j] for overlapping and cross-block actions, enabling rather than limiting scaling from 10s training to 60s+ complex timelines.

AuthorsOpenReview ↗

Final Remarks

Dear Reviewer WskD,

With the discussion deadline approaching, we wanted to follow up to see if our expanded 60s visualizations in Figure 2 and Appendix G (W1), within-model RTC ablation (W2), and context-scalability discussion (W3) have resolved your concerns. Please let us know if any questions remain.

Reviewer_UY53OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OnTime, a framework for following event timelines in long autoregressive video generation. Its core mechanism, Rolling Timeline Conditioning (RTC), preserves global action intervals while giving persistent scene and identity descriptions local temporal support. The authors combine RTC with teacher-guided initialization and Self-Forcing, training on 10-second sequences and evaluating 60-second rollouts. OnTime leads the compared methods in event-level visual quality and InternVideo2 alignment, while LongLive achieves a higher full-video Temporal-Following Score.

Critical strengths
  1. The paper addresses an important problem: precise event timing is essential for controllable streaming video generation.
  2. Extending timeline conditioning to autoregressive streaming generation is a reasonable direction. Training the TIE-initialized model through a full Self-Forcing pipeline provides a relevant long-video evaluation.
Critical weaknesses
  1. The prompt-enrichment (PE) component and agent-based workflow are not described in sufficient detail. The paper also does not train a dedicated PE model for structured-caption generation. Since prompt structure may strongly affect video quality, the training and test prompt distributions should be better aligned.
  2. The methodological novelty appears limited: the main change is adapting RoTE for streaming inference, followed by applying the established Self-Forcing pipeline to TIE-pretrained weights. The paper should clarify what new insight or capability this combination provides.
  3. Qualitative results show visible degradation in OnTime’s video quality, particularly background color shifts and streaks, while other baselines appear to preserve background consistency better.
Questions for authors
  1. Does OnTime support multiple characters within a single event? Could you show results for more complex events?
  2. How does quality change beyond 60 seconds? Could you provide examples with overlapping events or more complex timelines?
  3. Why does OnTime score lower than LongLive on full-video Temporal-Following, despite its stronger event-level results? Does this suggest difficulty following long, complex, or overlapping instructions?
2 comments

We thank Reviewer UY53 for the constructive comments and questions.

Prompt-enrichment (PE) details (W1). As described in Section 2.1 and Appendix A, our pipeline is a structured decomposition protocol rather than free-form caption rewriting. It separates persistent scene and identity context from timed atomic actions [si,ei][s_i, e_i], checking object/hand state consistency and UMT5's 512-token budget within each rolling 10s window. Training a dedicated PE model is unnecessary because a frontier LLM agent performs this structured decomposition directly.

Methodological novelty beyond RoTE + Self-Forcing (W2). Directly combining TIE (RoTE) with Self-Forcing fails when extrapolating from 10s training clips to 60s rollouts. In TIE, keys are modulated by interval half-duration r=(e−s)/2r = (e - s)/2 via D(r)D(r) with entries sinc(ωmr)/Cr\mathrm{sinc}(\omega_m r)/C_r (Section 2.2, Appendix B.2). By Corollary 1 (Eq. 5), shifting an action later in a long video preserves its fixed-feature attention logit only when its duration 2r2r stays unchanged. Extending persistent scene/identity descriptions to [0,60s][0, 60\text{s}] stretches rr to 6×6\times the 10s training range and distorts D(r)D(r); conversely, clipping all intervals to local blocks erases start/end progress for cross-block actions. RTC (Eq. 4) resolves this via asymmetric conditioning: bounding persistent descriptions to local support [ℓk,hk][\ell_k, h_k] (≤9.5s\le 9.5\text{s}) while keeping unclipped global intervals [sj,ej][s_j, e_j] for actions. Evaluating the same trained OnTime model on all 128 60s timelines with and without RTC (Table 1 metrics plus full-video action presence AP and boundary overlap Time/Loc. mIoU) confirms RTC's impact:

SettingVQ ↑MQ ↑TA ↑Overall ↑InternVideo2 ↑TFS ↑AP (%) ↑Time mIoU ↑Loc. mIoU ↑
OnTime w/o RTC-0.20650.2103-0.8622-0.85850.02410.303763.440.14660.2702
OnTime w/ RTC (Ours)0.08790.5376-0.59710.02840.01940.371478.850.24540.3529
Δ\Delta (w/ RTC −- w/o RTC)+0.2944+0.3273+0.2651+0.8869-0.0047+0.0677+15.41+0.0988+0.0827

Background color shifts and streaks (W3). Because our causal generator is trained only on 10s sequences and rolled out autoregressively to 60s, error accumulation across blocks can occasionally cause gradual background color drift or streak artifacts late in the rollout.

Multiple characters and overlapping events (Q1). OnTime supports both multiple characters and overlapping events on a shared global timeline (Section 2.1, Eq. 3). Corresponding 60s qualitative results have been added to the revised manuscript (Figure 2) and Appendix G (Figures 7–11).

Beyond 60s (Q2). Because RTC keeps persistent support (≤9.5s\le 9.5\text{s}), active token count (≤512\le 512), and KV cache size constant regardless of total duration TT, generation remains stable across 60s. Beyond 60s, visual quality gradually degrades compared to initial frames due to inevitable autoregressive error accumulation.

Lower full-video TFS than LongLive (Q3). In Table 1, LongLive Interactive (which actively switches prompts) also scores lower on TFS (0.3310), whereas OnTime (0.3714) outperforms both Self-Forcing (0.3585) and LongLive Interactive (0.3310). Static LongLive reaches 0.4165 because it generates largely static, low-motion videos (MQ = 0.1297 vs. OnTime's 0.5376); barely moving avoids late-rollout drift and inflates the frame, global, and consistency terms (F‾,G,C\overline{F}, G, C, accounting for 60% of TFS in Eq. 8). On direct temporal localization (see General Response, Item 2), static LongLive drops to 58.18% AP and 0.0969 Time mIoU, while OnTime leads all baselines (78.85% AP, 0.2454 Time mIoU).

AuthorsOpenReview ↗

Final Remarks

Dear Reviewer UY53,

As the discussion period draws to a close, we wanted to check whether our rebuttal and revised PDF have addressed your concerns on prompt decomposition (W1), the within-model RTC ablation (W2), 60s multi-participant visualizations (Q1), and full-video boundary metrics (Q3). Please let us know if you have any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and Summary of Revisions

We thank Reviewers UY53, WskD, and jDyJ for their constructive feedback. We are glad that the reviewers recognized the importance of precise event timing for controllable streaming video generation (UY53), the sound design of distilling an interval-conditioned teacher into a causal generator with rolling blockwise conditioning (UY53, WskD), and RTC's effectiveness in bounding persistent scene/identity durations while preserving cross-block action intervals to generate coherent 60s rollouts from 10s training sequences (jDyJ). Below we summarize the new experiments and manuscript revisions addressing the shared questions across reviews:

1. Isolating RTC within the same model (Reviewers UY53, WskD, jDyJ). Comparing OnTime with and without RTC using the same trained model on the 128-condition 60s benchmark shows consistent gains across perceptual quality, timeline following, and action boundary localization (AP, Time mIoU, and Loc. mIoU defined in Item 2):

SettingVQ ↑MQ ↑TA ↑Overall ↑InternVideo2 ↑TFS ↑AP (%) ↑Time mIoU ↑Loc. mIoU ↑
OnTime w/o RTC-0.20650.2103-0.8622-0.85850.02410.303763.440.14660.2702
OnTime w/ RTC (Ours)0.08790.5376-0.59710.02840.01940.371478.850.24540.3529
Δ\Delta (w/ RTC −- w/o RTC)+0.2944+0.3273+0.2651+0.8869-0.0047+0.0677+15.41+0.0988+0.0827

2. Direct start/end boundary evaluation (Reviewers UY53, jDyJ). Because cropped-clip scores do not penalize early onsets or late endings, we evaluate full-video temporal localization via Action Presence (AP), Time mIoU (scoring missing actions as 0), and Loc. mIoU (over localized actions). OnTime leads all methods (78.85% AP, 0.2454 Time mIoU, 0.3529 Loc. mIoU), ahead of LongLive Interactive (70.25%, 0.2024, 0.3300). Static LongLive drops to 58.18% AP and 0.0969 Time mIoU, confirming that its higher TFS (0.4165) comes from low-motion stability (MQ = 0.1297 vs. OnTime's 0.5376) rather than accurate timing.

3. Expanded qualitative comparisons (Reviewers UY53, WskD). Across seven 60s single-, two-, and three-participant timelines with concurrent actions and object handoffs (Figure 2; Appendix G, Figures 6–11), OnTime preserves participant identities and completes late-stage transitions where baselines stall in early poses, miss scheduled interactions, or alter participant count.

Summary of Manuscript Revisions (for Reviewers and AC). We have uploaded a revised PDF (strictly within the 9-page main-text limit, with all revisions marked in blue) and summarize every modification below:

  • Main text (Section 3.3 & Figure 2, pp. 7–8): Replaced Figure 2 and updated the Qualitative comparison of long video rollouts paragraph with a four-method 60s comparison (Self-Forcing, LongLive, LongLive Interactive, and OnTime) on a two-participant timeline featuring concurrent phone and note-taking actions, a cross-desk paper handoff, laptop opening, and a shared high-five.
  • Appendix (Appendix G, Figures 6–11, pp. 23–25; Appendix Contents, p. 13): Added a new appendix section with six full-width 60s qualitative rollout comparisons spanning single-, two-, and three-participant timelines, overlapping intervals, and multi-person object handoffs.
  • Preserved content: All other sections, equations, and quantitative tables (Tables 1–5) remain identical to the original submission; the new within-model RTC ablation and full-video action-boundary evaluations (Items 1–2 above) are reported directly in this rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.