acceptodds
Under review as a conference paper at ICLR 2027

TIE: Time Interval Encoding for Video Generation over Events

Abstract

Director-style prompting, robotic action prediction, and interactive video agents demand temporal grounding over concurrent events – a regime in which 68% of general clips and over 99% of robotics/gameplay clips contain overlapping events, yet existing multi-event generators rest on a single-active-prompt assumption. However, modern video generators, such as Diffusion Transformers (DiT), represent time as discrete points through point-wise positional encodings. This formulation creates a fundamental dimension mismatch: temporally extended intervals and overlapping events are mathematically unrepresentable to the attention mechanism. In this paper, we propose Time Interval Encoding (TIE), a principled, plug-and-play interval-aware generalization of rotary embeddings that elevates time intervals to first-class primitives inside DiT cross-attention. Rather than introducing another heuristic interval embedding, we show that, within RoPE-compatible bilinear attention, TIE is characterized by two basic principles: Temporal Integrability, which requires an event to aggregate positional evidence over its full duration, and Duration Invariance, which removes the trivial bias toward longer intervals. Under a uniform kernel, this characterization yields an efficient closed-form sinc-based solution that preserves the standard attention interface and naturally attenuates boundary noise through interval integration. Empirically, TIE preserves the visual quality of the base DiT model while substantially improving temporal controllability. In our experiments on the OmniEvents dataset, it improves human-verified Temporal Constraint Satisfaction Rate from 77.34% to 96.03% and reduces temporal boundary error from 0.261s to 0.073s, while also improving trajectory-level temporal alignment metrics.

Primary area:
generative models
Keywords:
Video Generation Model, Generative Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pbGwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Time Interval Encoding (TIE) for video generation with concurrent and overlapping events. Its uniform-kernel instance, RoTE, integrates rotary features over event intervals and applies self-overlap calibration, yielding a closed-form sinc-modulated encoding. The authors evaluate temporal controllability and visual quality on OmniEvents, with additional benchmark comparisons and ablations.

Critical strengths
  • The method is well-motivated and compact. The point-to-interval formulation explicitly incorporates event duration while preserving the standard attention interface. Its closed form and reduction to RoPE in the point-wise limit make the design technically coherent and potentially reusable.

  • The empirical improvements are meaningful. In Table 2, RoTE improves StoryEval over finetuning from 39.3% to 49.4% at 5 seconds and from 38.8% to 52.3% at 10 seconds. Appendix F also provides useful comparisons with alternative interval encodings, kernels, and normalization schemes.

Critical weaknesses
  • The human temporal metrics require clearer interpretation. Table A10 reports identical occurrence and TCSR scores of 96.03%, alongside lower order and overlap accuracy. Since TCSR includes relation constraints, the exact constraint sets and aggregation across videos and annotators are needed to reconcile these figures; the reported numbers alone do not establish an error. Moreover, missing-event penalties make it difficult to separate improved event coverage from more accurate temporal localization.

  • The advantage over strong interval alternatives needs uncertainty analysis. Endpoint averaging nearly matches RoTE on nDTW (0.2782 versus 0.2805), although RoTE achieves better FVD (Table A7). Without run-to-run variability estimates or matched human temporal evaluations of these alternatives, the reliability and practical significance of the smaller gains remain unclear.

  • Comparative noise robustness is insufficiently tested. Figure 7 compares perturbed RoTE with a clean-timestamp finetuned baseline. This demonstrates useful performance under noise, but does not establish lower noise sensitivity than competing encodings. Matched perturbations are needed to compare each method’s degradation from its own clean baseline.

Questions for authors
  1. Could the authors provide per-type TCSR constraint counts and clarify aggregation across videos and annotators? Are absolute boundary errors computed before or after averaging signed annotations?

  2. Can the authors report uncertainty and matched human temporal evaluations for RoTE versus center–length or endpoint averaging, including boundary errors on events generated by both methods?

  3. Are timestamp perturbations applied during training, inference, or both? How do the competing methods degrade under the same perturbations?

1 comment

Q1. Constraint counts and aggregation. For a start-time constraint, we assign a score of one if the event occurs and its absolute start-time error is at most 0.25 seconds, and zero otherwise. We score the end time separately. The human evaluation results are mainly performed on GamesEvents dataset, its event types are clear and fixed. However, the actual timestamp annotation may be hard to determine. Annotators can clearly distinguish whether an event actually occurred or if the sequence is incorrect, and they can flag instances where there is a visible temporal misalignment; however, they may be less sensitive to minor temporal discrepancies. The aggregation is the arithmetic mean of each annotators' scores. Missing-event penalties takes about 79% of the temporal error gain of TIE against Finetuned. Event Occurrence and TCSR score are identical here since all flagged temporal misalignment is less than 0.25s. However, this may involve issues such as annotator fatigue, insensitivity to the video timing, and the influence of default options.

Q2. Coverage, localization and uncertainty.

DoTE human evaluation results:

MethodsEvent Occurrence ↑Temporal Error ↑Order Acc ↑Overlap Acc ↑Overall TCSR ↑Temporal IoU ↑
TIE96.030.07592.2988.6496.0395.84
DoTE94.900.09090.4487.6694.6294.60

Q3. Matched noise robustness. The noise experiment perturbs timestamps at inference. Each method is trained on the original annotations. For each validation prompt, we apply the same perturbation to internal event boundaries across methods, preserve event order, hold clip endpoints fixed, and use the same generation seed.

Noise level σFVD ↓RoTE nDTW ↑RoTE EMD ↓
TIE clean87.530.3080.124
TIE + 0.290.220.3000.127
TIE + 0.494.700.2980.130
TIE + 0.697.810.2860.132
TIE + 0.899.910.2790.137
TIE + 1.0100.10.2740.138
DoTE clean102.520.2950.128
DoTE + 0.2101.390.2840.133
DoTE + 0.4102.710.2780.137
DoTE + 0.6104.220.2690.141
DoTE + 0.8107.740.2630.143
DoTE + 1.0111.220.2620.144
Reviewer_3EkbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses video generation with multiple events, where each event is given a time interval and events may overlap in time. The authors argue that point-wise positional encodings such as RoPE do not explicitly represent the temporal support of an event interval. To address this, they propose Time Interval Encoding (TIE), a framework defined by two principles. Temporal Integrability defines the attention score for an interval as the average of point-wise RoPE scores over the interval. Duration Invariance normalizes this score through a self-overlap condition so that its scale does not depend on the interval length. As an instance of TIE, they derive RoTE, which has a closed form and is applied to the event text tokens in cross-attention. The authors also introduce OmniEvents, a dataset of real-world, game, and robotics videos with interval-level event annotations, along with human-verified metrics for temporal control. Fine-tuned on Wan2.2, RoTE improves temporal control over a timestamp-finetuned baseline while maintaining visual quality.

Critical strengths

S1. Well-motivated problem with a clear and practical formulation of interval encoding

Controlling when each event occurs, including events that overlap in time, is an important and practical capability for video generation. Prior approaches mainly handle sequential events, so the problem setting is well motivated.

The paper points out that point-wise positional encodings do not explicitly represent the temporal support of event intervals, and formulates interval encoding through two principles, Temporal Integrability and Duration Invariance. This leads to RoTE, a closed-form encoding that stays compatible with RoPE. The framing is clear and easy to follow. RoTE can also be applied to the cross-attention of an existing backbone without architectural changes, and the paper provides a theoretical analysis of its properties, including robustness to boundary noise (Theorem A.4).

Why it is consequential: This is the main contribution of the paper. It offers a principled and simple way to encode event intervals, which is needed for overlapping events. If the concerns in W1 and W2 are addressed, this would be a solid contribution to temporal control in video generation.

S2. Temporal metrics, benchmark, and extensive experiments

The paper proposes several human-verified metrics, such as TCSR, Temporal Error, and Order and Overlap Accuracy, that evaluate different aspects of temporal control. Since there are few established benchmarks for this task, this is a useful contribution.

OmniEvents covers diverse domains, including real-world, game, and robotics videos, with interval-level event annotations. If released as stated, it would be a useful resource for future work on temporal control.

The appendix covers a wide range of experiments, including encoding variants, design choices such as timestamp scaling, convergence behavior, robustness to boundary noise, and both T2V and I2V settings. Many of these address questions that naturally arise from the method.

Why it is consequential: Since temporal control has few established benchmarks, the dataset and metrics can be useful to the community independently of the proposed method. Their value depends on the release and on the reliability of the metrics (see W2).

Critical weaknesses

W1. Insufficient comparisons to support the main claims

The paper argues that point-wise positional encodings do not explicitly represent the temporal support of event intervals. To verify this claim, the proposed encoding must be compared with prior methods that encode event timing in RoPE. However, the experiments mainly compare against the Finetuned baseline, which concatenates timestamps into the prompt but keeps the original temporal encoding.

MinT (CVPR 2025) is one such method. It assigns each event caption a single position in RoPE after rescaling each event to a fixed length and placing the events one after another, and the related work groups it with sequential approaches. Yet MinT is compared only in Table A2 with numbers taken from its original paper, where the backbone, resolution, and training data differ. Such a comparison cannot isolate the effect of the encoding. Although MinT is not open-sourced, its encoding is described in its paper, and several encodings are already implemented in Table A6, so a controlled comparison seems feasible.

In addition, the core design of the method is to integrate RoPE over the event interval. The self-overlap normalization follows from this choice, as it compensates for the attenuation caused by the integration. To show that integration matters, it should be compared with alternatives that encode the same interval information in other ways. The appendix includes such alternatives, such as center–length encoding (Table A6) and start–end averaging (Table A7), along with DoTE in Table 3. Among these, Appendix F.1 identifies center–length encoding as the strongest alternative on nDTW, EMD, and FVD. RoTE improves nDTW and FVD over it, but the EMD values are similar, and center–length encoding scores higher on CLIP-Event, which the main paper presents as a temporal alignment metric (nDTW 0.2805 vs. 0.2690, FVD 116.64 vs. 125.89, EMD 0.1304 vs. 0.1356, CLIP-Event 0.2091 vs. 0.2115 for RoTE vs. center–length). It would be helpful to understand whether the benefit of integration lies mainly in visual quality or also in temporal alignment.

Finally, the human-verified metrics are reported only for RoTE and Finetuned. Similarly, Theorem A.4 argues that RoTE is less sensitive to boundary noise than point-wise RoPE and DoTE, but the robustness experiment (Sec. 4.5) compares RoTE only with Finetuned.

Why it is consequential: Without comparisons under the same setting, the gains cannot be attributed to encoding intervals rather than to encoding event timing in RoPE at all, nor to integration rather than to other ways of encoding the same interval. This directly concerns the main claim and the main technical contribution of the paper. (See Q1–Q3.)

W2. Reliability of the evaluation

Since precise temporal control is the main goal of the paper, the reliability and precision of the temporal metrics are central to its claims.

(a) Reliability of the annotations. The human-verified metrics ask annotators to judge event boundaries at a very fine granularity. For TCSR, the paper's primary metric, each start and end is counted as satisfied only if it falls within ±0.25 s of the target. Such judgments can be hard to make consistently. For example, for "the dog suddenly barks and leaps into the air" (Figure A4), the end could be either the peak of the jump or the landing. As a minor point, the annotation interface in Figure A8 seems to show "Happens" and "Accurate" selected for each event. If these are default values, this could bias the results upward.

(b) Design of the human-verified metrics. Designing temporal metrics for a task with few established benchmarks is a useful contribution. However, the current metrics may not fully capture how well the generated intervals match the targets. First, the ±0.25 s tolerance is the same for all events regardless of length. Figure A8 shows events as short as 0.25 s, and for such events, an event placed entirely outside its target interval can still fall within the tolerance at both boundaries. Second, since each event has a target interval, the prompt also determines how long two events should co-occur. For example, events at 0–6 s and 2–9 s should co-occur for 4 s. However, Overlap Accuracy appears to count a pair as correct even if the observed intervals overlap only for a brief moment.

(c) Definition of the trajectory-level metrics. Most ablation and analysis results, including Tables A6–A8, the convergence curves (Figure 5), and the robustness experiment (Figure 7), rely on nDTW, EMD, and CLIP-Event. However, these metrics are only briefly described, and their computation, such as which features form the trajectories and what the reference videos are, does not appear to be specified in the paper. In addition, nDTW aligns the two sequences by stretching or compressing time, so a video with the correct event order but shifted timing may still receive a high score.

(d) Interpretation of CLIP-Event. CLIP-Event, which the main paper presents as a temporal alignment metric (Figure 3b), is lower for RoTE than for the alternatives in Tables A6 and A7. Appendix F.1 describes this as a tradeoff with "this semantic metric," but this seems inconsistent with how the metric is used in the main paper. If CLIP-Event reflects temporal alignment, these results suggest that RoTE does not consistently improve temporal alignment over the alternatives.

Why it is consequential: The main claim on concurrent events rests on TCSR and Overlap Accuracy, and most ablation and analysis results rest on the trajectory-level metrics. If these metrics are not reliable or precise enough, the reported gains, including those discussed in W1, are hard to interpret. (See Q4–Q7.)

Questions for authors

I recognize that controlling the timing of multiple, possibly overlapping events is an important problem, and I appreciate the clear formulation and the extensive experiments in this paper. Since the main methodological contribution is to encode an event interval by integrating RoPE over it, my assessment depends largely on how well the experiments support this choice. My main questions concern two points: how the method compares with other ways of encoding event timing, and how reliable the temporal evaluation is. The answers to the questions below could affect my assessment.

[R] = additional result requested, [C] = clarification.

1. Comparisons (W1)

  • Q1 [R]. Could the authors compare RoTE with MinT's encoding, reimplemented on Wan2.2 under the same setting? Reporting the results separately for sequential and overlapping events would also be helpful.
  • Q2 [R]. Could the authors report the human-verified metrics for the strongest alternatives, such as center–length encoding and DoTE, and include DoTE in the robustness experiment (Sec. 4.5)? The latter would directly test Theorem A.4.
  • Q3 [C]. How were the temporal bias and temporal mask variants in Table A6 implemented, and do they follow specific prior methods?

2. Evaluation (W2)

  • Q4 [R/C] (a). What is the inter-annotator agreement on the boundary annotations, for example the standard deviation of the annotated start and end times across annotators? Also, are "Happens" and "Accurate" default values in the annotation interface?
  • Q5 [R] (b). How do the methods compare under a measure that reflects how much the intervals overlap, such as the temporal IoU between the target and annotated intervals? This can be computed from the existing annotations.
  • Q6 [C] (c). How are nDTW, EMD, CLIP-Event, and CLIP-Video computed, including the features and reference videos? CLIP-Video in particular is used as a visual quality metric (L362) but is not defined. In addition, since nDTW aligns sequences by stretching or compressing time, is it suitable for measuring precise timing?
  • Q7 [C] (d). How should CLIP-Event be interpreted, and why does RoTE score lower on it in Tables A6 and A7 while achieving the best score in Table 3?

3. Qualitative results

The supplementary videos show only results of the proposed method, without the prompts or target intervals, which makes it difficult to judge whether each event occurs within its target interval.

  • Q8 [R]. Could the authors supplement the supplementary videos with more examples, including the prompts and target intervals? Comparisons with Finetuned could also be considered, if possible.

Suggestions (not affecting the assessment)

  • In the main text, Definition 3.1 only requires a positive constant C(μI)C(\mu_I) for Duration Invariance, which is also satisfied by C=1C = 1, i.e., no normalization. The self-overlap condition that determines this constant appears only later (L186–187) and in Definition A.1. Including it in Definition 3.1 would make the principle clearer.
  • It would be helpful to further explain why the interval score is defined as an average of point-wise scores, and why the calibration is set at the interval center with a target value of 1.
  • The claim that TIE preserves visual quality relies on VideoScore and on FID and FVD, whose reference set comes from the fine-tuning data. VBench quality metrics are reported only for the prompt-extension setting (Table A3). Reporting them for the main comparison with Finetuned would make this claim more convincing.
1 comment

Q1. Matched comparison with MinT. We reimplemented MinT’s fixed-length sequential time mapping on the same Wan backbone as TIE. Both methods use the same sequential Pexels split, trainable branch, optimization budget, checkpoint step, validation prompts, 480p, and sampling seeds. Since MinT does not support overlapping event prompt, we filter out 55,000 clips of sequential event prompt in the PexelsEvents (with one global description and a series of sequential event prompts).

MetricBaselineMinT-styleTIE
FVD ↓893.06805.06812.35
FID ↓139.52130.45130.21
nDTW ↑0.022010.025530.02635
EMD ↓0.419120.403750.38553
CLIP-Event ↑0.211000.212860.21479
CLIP-Video ↑0.579330.596000.59349
Mean training loss, last 100 steps ↓0.2488450.2450560.242550

On the 480p sequential datasets, TIE's performance is close to the MinT's. We noticed that TIE converges slightly faster than MinT.

Q2 & Q4-Q5. DoTE, human evaluation and temporal IoU. The human evaluation results are mainly performed on GamesEvents dataset, its event types are clear and fixed. However, the actual timestamp annotation may be hard to determine. Annotators can clearly distinguish whether an event actually occurred or if the sequence is incorrect, and they can flag instances where there is a visible temporal misalignment; however, they may be less sensitive to minor temporal discrepancies. "Happens" and "Accurate" are default terms.

DoTE human evaluation results:

MethodsEvent Occurrence ↑Temporal Error ↑Order Acc ↑Overlap Acc ↑Overall TCSR ↑Temporal IoU ↑
TIE96.030.07592.2988.6496.0395.84
DoTE94.900.09090.4487.6694.6294.60

DoTE noise perturbation results:

Noise level σFVD ↓RoTE nDTW ↑RoTE EMD ↓
TIE clean87.530.3080.124
TIE + 0.290.220.3000.127
TIE + 0.494.700.2980.130
TIE + 0.697.810.2860.132
TIE + 0.899.910.2790.137
TIE + 1.0100.10.2740.138
DoTE clean102.520.2950.128
DoTE + 0.2101.390.2840.133
DoTE + 0.4102.710.2780.137
DoTE + 0.6104.220.2690.141
DoTE + 0.8107.740.2630.143
DoTE + 1.0111.220.2620.144

Q3. Temporal bias and mask. We implemented both as controls in the same cross-attention layer. Let dd be the distance from a frame’s query time to the event token’s target interval. The soft variant adds −λd-\lambda d to the attention logit. The hard variant leaves in-interval logits unchanged and applies a large negative bias outside the interval. Untimed context tokens receive no interval penalty. We will give these formulas in the appendix and identify the variants as our implementations.

Q6–Q7. Metric definitions and the CLIP-Event ranking. In the new validation comparison, each generated video is paired with the held-out source video for the same prompt. nDTW aligns normalized CLIP frame features using cosine distance and converts the mean cost along the warping path into a similarity. A correctly ordered action can therefore score well despite a timing shift. EMD measures the cost of transporting frame features, including a penalty for displacement in time. CLIP-Video averages generated/reference frame similarity at corresponding sampled times. CLIP-Event compares frames inside each requested interval with that event’s caption, then averages over events and clips.

These definitions clarify the apparent discrepancy. RoTE has the highest CLIP-Event among the methods in the main ablation, while center–length and endpoint averaging score higher in the appendix comparisons. We will report both findings and describe CLIP-Event consistently as semantic agreement within a requested window.

Q8. Qualitative comparisons and visual quality. We will show RoTE and the timestamp-finetuned baseline side by side with the same prompt, target intervals and playback clock (e.g. example1.mp4: [Event(0.0, 5.0, "A flock of black geese flies across the sky."), Event(0.0, 3.0, "A man stands still on the dirt path."), Event(3.0, 8.5, "The man raises his hand to shield his eyes from the sun."), Event(7.0, 10.0, "A passenger train comes into view."), Event(8.5, 10.0, "The man lowers his hand.")]). We will include an omitted event, a timing error and an overlapping pair so readers can judge when each action actually occurs. The requested VBench evaluation for the main comparison remains pending.

Reviewer_W9hQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a new method for temporally controlling video generation in diffusion transformer (DiT)- based video models. Its core contribution is a generalization of the rotary embeddings for cross-attention. It allows frames corresponding to different time steps to attend to different prompts with each prompt covering a frame interval. Unlike prior work addressing this problem, the present paper enables attention to multiple prompts in parallel, each of which can correspond to a different interval and thus cover different frames.

Critical strengths
  • Overall, I believe that the paper considers an important problem. Improving fine-grained control of videos is important for many practical applications.
  • The theoretical formulation of the method is interesting.
Critical weaknesses
  • The writing is poor and starts on a bad note. For example, what is "director-style prompting"? Even a reader who worked on controllable video editing cannot be assumed to know this terminology, let alone the broader vision community or the even broader ICLR community. Another example: what do the authors mean when they write that duration invariance "calibrates expected self-overlap at the interval center"?

  • In its current form, the work lacks several details important for reproducibility, such as AdamW optimizer parameters, which parameters were tuned (full fine-tune or only part of the network), and more. I recommend that the authors prompt their favorite LLM to see which information is missing for full reproducibility.

  • Poor figure presentation quality. Most figures include fonts that are near unreadably small, and it is hard to understand what they are visualizing at times.

Questions for authors
  • How does the TCSR metric work? The authors state that it "summarizes the total temporal alignment degree, together with breakdowns into Event Occurrence, Temporal Error, Order Accuracy and Overlap Accuracy," but they do not specify how these are measured. The appendix (which should be referred to from the main paper) has some more information on this, but I am still not sure how exactly I would evaluate "c is satisfied" in practice. Is this part of the data labeling?
1 comment

We thank Reviewers W9hQ for their sincere suggestions:

Clarity and scope. Thank you for pointing out that the opening terminology assumes familiarity with the task. By “director-style prompting,” we mean specifying both what should happen and when. For example, a prompt can ask a person to open a door while a dog runs past, with a separate interval for each action. We will introduce the task with a concrete example and enlarge the labels in the figures. We will also narrow our representational claim. A scalar timestamp gives an event a position; RoTE explicitly specifies how every time within the event interval contributes to cross-attention.

What duration invariance means. Averaging rotations over an interval can attenuate the attention response because different phases partly cancel. To calibrate this effect, we take the same unit vector, encode it at the interval midpoint and over the full interval, and measure their inner product. We choose the scale so that this self-match, averaged over vector directions, equals the pointwise self-match. The midpoint provides a symmetric reference for the uniform kernel. Duration still changes the frequency response. We will put this condition in the main definition, since requiring a positive normalization constant alone does not determine its value.

Reproducibility. We will specify the trainable parameters (usually they are full DiT finetuning), AdamW settings (usually we use the default AdamW parameters), data split (the three datasets are reported, and test are performed on additional 500/106/150 for Pexels/Games/Robotics dataset, which were not included in the training split), stopping step (6500 steps for 720p training, 20000steps for 480p training) and inference settings for each experiment.

How TCSR is measured. The prompt supplies the target intervals. Blinded annotators watch the generated video, mark whether each event occurs, and record its observed start and end offsets. Start and end are checked separately against the requested boundaries. Order is checked from the event midpoints, and overlap is checked on pairs requested to overlap. For a start-time constraint, we assign a score of one if the event occurs and its absolute start-time error is at most 0.25 seconds, and zero otherwise. We score the end time separately. We will give a worked example in the main text and report the satisfied and eligible counts separately for each constraint type. We aggregate across annotators by taking the arithmetic mean of their responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.