acceptodds
Under review as a conference paper at ICLR 2027

DynaSense: Learning to Perceive Dynamic Changes in Videos for Temporal Grounding

Abstract

Current video Multi-modal Large Language Models (Video-MLLMs) for Video Temporal Grounding (VTG) rely on RGB-centric static visual representations, which capture visual content but offer limited modeling of visual state evolution over time. Consequently, these approaches fail to perceive dynamic visual changes at the temporal boundaries of events, where fine-grained temporal transitions are critical for accurate localization. To address this limitation, we propose DynaSense, a Video-MLLM that improves temporal grounding by explicitly modeling dynamic representations of temporal changes within videos. DynaSense models continuously evolving videos by representing visual state changes at each temporal step relative to the preceding step. It first aligns visual features at the current timestamp with the most relevant references from the preceding timestamp within a position-free embedding space. Subsequently, it encodes the resulting temporal change as the difference between the corresponding visual features after incorporating 2D spatial positional encodings. This dynamic representation captures both appearance differences and spatial displacements, providing temporally discriminative evidence for localizing event boundaries. Furthermore, we introduce video state change recognition dataset (VideoSCR), a video state-change recognition dataset constructed via an automated annotation pipeline, to enhance the perception and recognition of video dynamics. DynaSense consists of two training stages. The first performs state-change question answering on VideoSCR to facilitate the learning of dynamic video features, and the second introduces video temporal grounding data to strengthen the capability for temporal localization. Experiments across multiple VTG benchmarks demonstrate that DynaSense consistently improves temporal grounding accuracy and can be readily integrated into other Video-MLLMs to enhance their temporal grounding capabilities.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Video Temporal Grounding, Video-LLM, Temporal State-Change Recognition, Dynamic Visual Representations

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LHCtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DynaSense improves video temporal grounding by explicitly modeling changes between neighboring frames through local patch matching and position-aware feature differencing. It processes appearance and dynamic information using a shared visual backbone. The authors also introduce VideoSCR for learning before-and-after state descriptions, followed by temporal grounding training. Experiments report improvements over the underlying video-language models and transfer to several temporal understanding tasks.

Critical strengths
  1. DynaSense constructs dynamic representations directly from native patch embeddings and processes appearance and dynamic inputs through a shared visual backbone. This avoids requiring a separately pretrained motion encoder. The combination of matched appearance differences and positional displacement cues also provides an interpretable rationale for capturing changes relevant to event boundaries.

  2. DynaSense consistently improves temporal grounding over its corresponding Qwen3-VL baselines across the evaluated model sizes and reported TimeLens subsets. It also improves average grounding performance on E.T.Bench, with gains on several complementary temporal tasks. These results support the practical value of the complete approach and suggest that its benefits extend beyond the training task.

Critical weaknesses
  1. The central matching mechanism is insufficiently justified and validated.

DynaSense makes several consequential structural choices: hard nearest-neighbor assignment rather than soft aggregation, unrestricted many-to-one correspondences, mandatory assignment without a confidence-based rejection criterion, matching independent of the language query, and a fixed (3x3) search window. The first four choices are specified in the method but are not individually motivated or evaluated against representative alternatives. Consequently, it remains unclear how these decisions affect correspondence quality and downstream temporal grounding, or whether they are necessary for the reported improvements.

For the search window, the authors provide a plausible motivation based on computational cost, locality, and avoiding spurious matches. However, Table 4 evaluates window sizes only on Charades-TimeLens using the 2B model with reduced training data. Although (3x3) performs best, with no monotonic degradation among the larger windows. These results support the selected configuration in this setting, but do not establish its robustness across datasets or motion regimes. Moreover, the explanation that larger windows introduce spurious correspondences is not supported by a direct assessment of matching accuracy.

The computational motivation also remains unquantified: the paper reports no FLOPs, latency, or memory measurements across window sizes, preventing an assessment of the accuracy-cost tradeoff. Sparse adaptive mechanisms, such as deformable attention, offer a relevant alternative by learning a limited set of sampling locations instead of restricting candidates to a fixed neighborhood. Their suitability for this architecture would require evaluation, but the paper neither discusses this alternative nor provides a comparison under comparable computational and training budgets.

  1. The rationale for the proposed architecture remains insufficiently convincing.

The paper motivates DynaSense through the limitations of appearance-based representations in capturing fine-grained temporal changes. However, it addresses this limitation through explicit local matching and position-enriched differencing while retaining framewise attention in the visual backbone. A natural alternative would be to introduce temporal attention into visual encoding and learn interactions across frames directly. The paper does not establish what advantage its specific matching-and-differencing construction provides over this alternative.

Reusing a pretrained visual backbone is a reasonable practical motivation, but it does not preclude adding lightweight temporal attention layers or temporal adapters to the same backbone. Such a comparison would also avoid the confounding effects of replacing the entire encoder with a differently pretrained video model. Nevertheless, the paper provides no controlled comparison against this type of baseline. The reported improvements therefore do not resolve whether the proposed construction offers a distinctive benefit over a conventional learned temporal module.

Without this evidence, I find it difficult to understand why the proposed architecture should be preferred, or which aspect of its explicit correspondence-and-difference formulation is responsible for an advantage over standard temporal modeling.

Questions for authors
  1. What motivates hard, many-to-one, unconditional, and query-independent matching, and can you provide ablations demonstrating the effects of these choices relative to representative alternatives?

  2. Does the preference for a (3x3) window generalize across datasets and motion regimes, and can you provide matching-quality and computational measurements to substantiate the proposed explanation for this choice?

  3. How does DynaSense compare with temporal attention layers or temporal adapters added to the same backbone under identical training conditions and comparable visual token budgets and additional trainable capacity?

7 comments

Thank you for your review and suggestions. We recognize the heavy reviewing workload and the role of AI tools in preliminary assessment. However, evaluations should reflect independent judgment and careful verification, grounded in the paper's actual contributions and experimental evidence, rather than adopting AI-generated content without scrutiny. Respect for others' scholarly work should be a basic principle of peer review. Several concerns appear to diverge from the motivation and scope of our work; we respond below based on our understanding of the questions.

Our core contribution is to construct explicit dynamic representations directly in the feature space of a pretrained visual encoder, improving Video-MLLMs' perception of fine-grained video state changes and temporal grounding. DynaSense first establishes visual correspondences between adjacent frames through position-free local patch matching. It then differences matched features enriched with spatial positional encodings, preserving both content changes and spatial displacement information. A shared visual backbone and independent projection modules process the static and dynamic representations. We also construct VideoSCR and adopt two-stage training to transfer state-change understanding to temporal grounding. Experiments show consistent improvements across model scales and benchmarks, together with stronger zero-shot generalization.

1. Response to Weakness 1 / Question 1

Our central claim is that local correspondence and position-aware differencing construct change representations in the native visual feature space that benefit temporal grounding. Hard nearest-neighbor matching provides a deterministic reference for differencing; many-to-one matching avoids an additional one-to-one constraint; and retaining a reference for every patch enables change features to be computed at every location. Query relevance is modeled by the downstream LLM. Tables 3–6 present extensive ablations covering the combination of VideoSCR and dynamic representations, local matching window size, the order of matching and differencing relative to positional encoding, and shared versus independent mergers.

2. Response to Weakness 1 / Question 2

We randomly sampled 1,000 examples from the Charades-TimeLens test set and measured inference costs for Qwen3-VL and DynaSense at both 2B and 8B scales.

Table III. Inference cost comparison.

MetricQwen3-VL-2BDynaSense-2BQwen3-VL-8BDynaSense-8B
Generation latency (ms)761.761039.111407.782063.40
TTFT (ms)414.22664.29933.821558.23
Time/token (ms)21.4823.1229.1031.15
Peak GPU memory9.589.5924.6924.88
Visual tokens13575.0820942.9913575.0820942.99
Sampled frames59.1959.1959.1959.19

With identical sampled frame counts, DynaSense uses approximately 54.3% more visual tokens. Average generation latency increases by 36.4% and 46.6% at 2B and 8B, respectively, while time to first token (TTFT) increases by 60.4% and 66.9%. Time per output token increases by only 7.6% and 7.0%, with small changes in peak GPU memory. These results suggest that the additional overhead is concentrated in the prefill stage.

We also analyze the time complexity of dynamic feature construction. Let TT denote the number of temporal blocks, NN the patches per block, dd the feature dimension, and w×ww\times w the local matching window:

  • Patch matching: O((T−1)Nw2d)\mathcal{O}((T-1)Nw^2d).
  • Feature differencing: O(TNd)\mathcal{O}(TNd).

With a fixed 3×33\times3 matching window, the overall complexity of dynamic feature construction is O(TNd)\mathcal{O}(TNd). This excludes the downstream processing cost of the additional visual tokens.

We acknowledge the computational cost of the current method and will explore sparse dynamic token selection and compression to reduce inference latency.

3. Response to Weakness 2 / Question 3

We wish to clarify that explicit dynamic representation construction and learnable temporal interaction modeling are distinct approaches. Temporal attention implicitly learns temporal relations through interactions across frame features. DynaSense instead constructs dynamic representations containing content changes and spatial displacement information through local correspondence and feature differencing, and reuses the pretrained ViT to encode them. Cross-frame changes are already explicitly encoded during dynamic feature construction, and the downstream LLM jointly processes visual information across time. Due to computational constraints, the ViTs in common Video-LLMs [1–5] typically do not perform long-range cross-frame interactions. Local temporal processing is mainly limited to adjacent frame pairs, while longer-range temporal modeling is primarily handled by the LLM.

[1] Qwen Team. Qwen3-VL Technical Report.

[2] Wang et al. InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency.

[3] V Team et al. GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning.

[4] Clark et al. Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding.

[5] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. LLaVA-Video: Video Instruction Tuning With Synthetic Data. Transactions on Machine Learning Research (TMLR), 2025.

Thank you for your response.

First, I would like to state that I did not rely on AI as the basis for my assessment of this paper. I wrote my review in my mother language and used an AI tool only to translate it into English, so some parts may have been described in terms or expressions that differ from those used in the paper. I may also have misunderstood parts of what the authors intend to convey. I will state this use of an AI tool for translation in the "AI Documentation For Reviewing" field.

I also apologize that my initial review was not concise and contained a relatively large amount of information. The conference emphasizes conciseness in reviews, and I overlooked this in my initial review. I had many questions about the paper, and in compressing them into the "one or two critical weaknesses" that the conference asks for, some of the detailed explanation was lost. I apologize for this.

The core issue I want to raise is that the design of "Video Dynamic Feature Matching" (Sec. 3.2) is not sufficiently justified. The authors' response resolved many of my concerns, but the following points remain open.

1. Hard nearest-neighbor matching (Eq. 2). Eq. (2) selects only the single most similar patch in the preceding frame via the arg min of the L1 distance. What is the reason for this? Instead of selecting one, what if the nine candidates in the 3×3 window were averaged with weights given by their similarity to the current patch? The rebuttal explains that hard nearest-neighbor matching serves as a deterministic reference for differencing, but a similarity-weighted average is equally deterministic, so the question remains unanswered in both the paper and the rebuttal.

2. Insufficient validation of the 3×3 local matching window. I raise this because of the statement in L421–423 that larger windows introduce spurious matches. Table 4 is reported only on Charades-TimeLens with DynaSense-2B trained on VideoSCR-28K and TimeLens-20K, so it is unclear whether 3×3 remains the best choice on other datasets or on videos with large motion. Substantiating this claim requires either a direct assessment of matching accuracy or validation on more diverse datasets. The paper only compares grounding performance across window sizes, and neither the paper nor the rebuttal addresses this question.

3. Validity of the matching when no true match exists in the window. The proposed matching is likely to work well in ordinary cases, but it becomes problematic when the true match of a patch is not inside the local window of the preceding frame, for example with newly appearing objects, fast object motion, or scene cuts. Because each current patch retains a match (L231–232), such patches still receive a reference. The rebuttal touches on this by explaining that keeping a reference for every patch makes it possible to compute change features at every location. In my view, however, this answers "why is every patch given a match?" rather than "what happens when the match is wrong?". I would like to hear the authors' view on this.

Finally, I gave this paper an initial rating of 1 because I considered the experimental evidence supporting the design of "Video Dynamic Feature Matching", one of the core components of the paper, to be far too limited. Since this is a core technique on which the paper is built, I believe its design needs to be validated rigorously. I am aware, however, that this is not the paper's only contribution. Apart from this component, I find the design sound and consider this a good paper. The rebuttal has already resolved my concerns to some extent, and if the authors address the remaining concerns above, I will reflect this in my final assessment.

Plus, I raise the score to 2 because the author has partially resolved my concerns.

Reviewer_CL4POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DynaSense to improve video temporal grounding through explicit modeling of visual changes. It introduces dynamic features from adjacent frames and a two-stage training strategy using the newly constructed VideoSCR dataset.

Critical strengths

1.The method captures visual changes using a shared vision backbone, avoiding a separate motion encoder.

2.The method improves over the underlying Qwen3-VL models at both 2B and 8B scales.

Critical weaknesses
  1. Insufficient validation of VideoSCR. The paper provides no completed human-review results or train–test overlap audit, leaving the quality and independence of the synthetic training data unclear.

  2. Incomplete evidence for the core design. Ablations are limited to a reduced 2B setting and omit the dynamic branch without VideoSCR. The proposed matching/differencing order also shows no mIoU advantage over its alternatives.

  3. Limited methodological novelty. The method mainly combines local patch matching, feature differencing, and a dual-stream interface. Its advance over simpler temporal-difference representations is not convincingly established.

  4. Modest experimental gains. DynaSense improves over Qwen3-VL but remains below TimeLens-8B on both main grounding subsets. Controlled ablation gains are relatively small, and improvements on E.T.Bench are uneven across tasks.

Questions for authors
  1. Can you provide source-wise VideoSCR statistics, human-audited annotation accuracy and boundary errors, and source-video/near-duplicate overlap checks against TimeLens-Bench and E.T.Bench? If overlaps exist, how do the results change after removing them?

  2. Under the full 8B setting, how does DynaSense compare with an RGB-only model using identical training data, optimization steps, and total visual token budgets?

  3. How reliable is local matching on pre-ViT patch embeddings under occlusion, repetitive textures, and large motion at 2 fps? What supports interpreting absolute positional embedding differences as meaningful displacement cues?

  4. How does the method distinguish query-relevant transitions from camera motion or unrelated changes? Please provide duration-stratified [email protected] and boundary errors with uncertainty estimates, particularly given the tied mIoU and slightly worse [email protected] of the proposed design in Table 5.

4 comments

Thank you for your careful review. We address your concerns below.

1. VideoSCR sources and quality control

To avoid evaluation-data leakage, we selected sources outside the datasets or splits used for evaluation. Table I summarizes the composition.

Table I. VideoSCR data sources.

Video sourceVideos/clipsState-change examplesAnnotation
LLaVA-Video: liwei_youtube_videos17,60330,834Generated
LLaVA-Video: NExT-QA549837Generated
LLaVA-Video: Perception Test (train split)114204Generated
GEB+8,27028,862Adapted original annotations

After rechecking the counts, VideoSCR contains 60,737 state-change examples, including 31,875 synthetic examples. The reported 48K was a counting error, which we will correct in the revised manuscript.

The two TimeLens-Bench subsets evaluated in our main comparison originate from Charades-STA and ActivityNet Captions. Table II lists the sources of the E.T.Bench grounding tasks.

Table II. Data sources for E.T.Bench grounding tasks.

Grounding taskSource datasetExamples
TVGQVHighlights500
TVGCharades-STA500
EPMEgo4D-NLQ500
TALPerception Test (val split)500
TALTHUMOS'14146
TALTHUMOS'15292
EVSSumMe25
EVSTVSum50
VHDQVHighlights500
VHDYouTube Highlights354

Ten reviewers checked whether changes were visible, subjects and before/after descriptions were consistent, and timestamps aligned with the transitions. Uncertain examples were rechecked and then corrected or rejected. Screening records show 60,737 accepted examples (90.9%) and 6,080 rejected or requiring revision (9.1%). Common reasons include scene cuts (2,322), no visible change (2,183), malformed JSON (982), uncertain wording (44), and missing fields (3).

2. Controlled ablations at 8B

We agree that an 8B comparison with identical training data, optimization steps, and total visual-token budgets would provide stronger evidence. Compute and time constraints currently prevent us from completing this experiment. Table 3 provides evidence at 2B: using the same VideoSCR and TimeLens datasets, adding DynaSense improves mIoU from 45.7 to 47.6 and [email protected] from 50.5 to 54.2.

3. Local matching reliability and positional differences

For repetitive textures, we favor smaller spatial displacement when candidates have similar matching costs. Matching is performed after 3D patch embedding. At 2 fps with a temporal patch size of 2, neighboring temporal blocks are approximately one second apart. Spatial patches are 16×1616\times16, so the 3×33\times3 candidate window covers a 48×4848\times48-pixel region in the resized input. Constraints such as min_tokens, max_tokens, and total_pixels typically resize videos to resolutions between 320×192320\times192 and 896×480896\times480. Resizing reduces the corresponding displacement on the patch grid, helping cover local motion. Large displacements beyond the window and occlusion require further study.

Unlike motion-representation methods such as ReMoRa and EMA, we construct change representations directly in the native patch-feature space and reuse the ViT, without a separate motion encoder or optical-flow estimation. The learnable absolute positional embeddings in the Qwen3-VL ViT encode spatial location, and we use their differences as implicit, learnable displacement cues.

4. Query-relevant changes and Table 5

The matching stage neither explicitly distinguishes change categories nor filters changes by the query. VideoSCR teaches the model to understand a subject's before/after states. Stage 2 then trains the LLM to combine appearance, change information, and the query for temporal grounding.

In Table 5, our configuration achieves higher [email protected]/0.5, identical reported mIoU (47.6), and slightly lower [email protected] (27.5 versus 27.8/27.9). This indicates that the gains mainly concern recall at moderate IoU thresholds, rather than improved boundary precision for every example. Ambiguous local matches or unrelated changes may affect strict boundary predictions for some examples, but this explanation remains to be verified.

We computed duration-stratified [email protected] for DynaSense-8B on Charades-TimeLens. Table III shows an overall upward trend with event duration. [email protected] is 6.10% for events lasting at most one second, compared with 50.56% and 56.91% for the two groups above ten seconds. High-precision grounding remains more challenging for short events, possibly because they provide less temporal context and even small boundary errors can sharply reduce IoU.

Table III. Duration-stratified [email protected] for DynaSense-8B on Charades-TimeLens.

GT duration dd (s)ExamplesShareSuccessful predictions[email protected]
0<d≤10 < d \le 159017.54%366.10%
1<d≤21 < d \le 270921.08%24734.84%
2<d≤52 < d \le 51,02730.54%31530.67%
5<d≤105 < d \le 1055416.47%24644.40%
10<d≤2010 < d \le 2036010.70%18250.56%
d>20d > 201233.66%7056.91%
All examples3,363100.00%1,09632.59%

5. Benchmark comparison

TimeLens is a strong grounding baseline that combines high-quality grounding data, timestamp encoding, and thinking-free RLVR with temporal-IoU rewards. DynaSense focuses on improving video representations through state-change learning and two-stage SFT. TimeLens-8B scores higher on the two subsets in Table 1. On zero-shot E.T.Bench, with neither model fine-tuned on E.T.Instruct, DynaSense-8B achieves higher F1 on four of five tasks: TAL, EVS, VHD, and EPM improve by 4.0, 8.5, 5.0, and 0.1 percentage points, respectively. Mean grounding F1 increases from 36.6 to 39.2 (+2.6), supporting DynaSense's stronger zero-shot generalization in this evaluation.

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper!

Dear Reviewer,

As the ICLR 2027 rebuttal period closes tonight, we would greatly appreciate any feedback on our responses. We hope we have adequately addressed your concerns, but if any questions or doubts remain, please feel free to raise them so we can provide further clarification before the deadline.

Thank you for your time and consideration!

Reviewer_5WiEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DynaSense, a Video-MLLM framework that improves video temporal grounding by explicitly modeling dynamic changes between consecutive frames. The method constructs dynamic features through position-free patch matching and position-aware feature differencing, without requiring an additional motion encoder. The authors also introduce VideoSCR, a state-change recognition dataset, and adopt a two-stage training strategy to enhance temporal understanding. Experiments on multiple benchmarks demonstrate consistent improvements over the underlying Qwen3-VL baselines.

Critical strengths

1.The paper addresses an important limitation of existing Video-MLLMs in capturing fine-grained temporal changes for accurate event boundary localization. 2.The proposed dynamic representation is intuitive and can be integrated into pretrained visual backbones without introducing a separate motion encoder. 3.The method achieves consistent improvements over Qwen3-VL across different model scales, and the ablation studies provide useful insights into the proposed components.

Critical weaknesses

1.The main experiments compare DynaSense after two-stage fine-tuning with the original Qwen3-VL baselines. A more controlled comparison would help clarify the contribution of the dynamic representations. 2.In Table 5, different matching and differencing strategies achieve identical mIoU, making the advantage of the proposed operation order less convincing. 3.Although the framework avoids an additional motion encoder, the dynamic stream introduces extra visual tokens and computation. The practical overhead is not quantitatively evaluated.

Questions for authors

1.Could the authors provide a baseline fine-tuned on the same training data without the proposed dynamic representation to better isolate its contribution? 2.Could the authors further explain why the proposed matching and differencing order is preferable, given the relatively small differences observed in Table 5? 3.Could the authors provide a brief analysis of the additional inference latency or computational overhead introduced by the dynamic stream?

3 comments

We sincerely thank you for your thorough review and insightful suggestions. We acknowledge the specific concerns you have raised regarding novelty, fairness, and methodology. We have carefully revisited these issues and provide our detailed responses below.

1. Ablation of the dynamic representation

As shown in Table I, both configurations use the same two-stage training strategy and datasets: VideoSCR for state-change learning in Stage 1 and TimeLens for temporal grounding fine-tuning in Stage 2. The only model configuration difference is whether the DynaSense dynamic representation is used.

Table I. Ablation of our method.

VideoSCRDynaSense[email protected][email protected][email protected]mIoU
✓×66.050.525.645.7
✓✓68.154.227.547.6

Adding the dynamic representation improves the four metrics by 2.1, 3.7, 1.9, and 1.9 percentage points, respectively.

2. Performance differences in Table 5

In Table II, DynaSense achieves the highest [email protected] and [email protected], indicating an advantage at low and moderate IoU thresholds. However, all three variants have identical reported mIoU, and DynaSense has slightly lower [email protected]. We hypothesize that this pattern is related to differencing features after adding learnable positional embeddings, which turns spatial position information into implicit displacement cues. In future work, we will explore differencing before positional encoding while explicitly encoding spatial displacement.

Table II. Ablation of matching and differencing order.

Method[email protected][email protected][email protected]mIoU
DynaSense*68.154.227.547.6
Matching after PE67.754.127.847.6
Differencing before PE67.253.327.947.6

3. Inference latency and computational overhead of the dynamic branch

We randomly sampled 1,000 examples from the Charades-TimeLens test set and measured inference costs for Qwen3-VL and DynaSense at both 2B and 8B scales.

Table III. Inference cost comparison.

MetricQwen3-VL-2BDynaSense-2BQwen3-VL-8BDynaSense-8B
Generation latency (ms)761.761039.111407.782063.40
TTFT (ms)414.22664.29933.821558.23
Time/token (ms)21.4823.1229.1031.15
Peak GPU memory9.589.5924.6924.88
Visual tokens13575.0820942.9913575.0820942.99
Sampled frames59.1959.1959.1959.19

With the same number of sampled frames, DynaSense uses approximately 54.3% more visual tokens. Average generation latency increases by 36.4% and 46.6% at 2B and 8B, respectively, while time to first token (TTFT) increases by 60.4% and 66.9%. In contrast, time per output token increases by only 7.6% and 7.0%, with small changes in peak GPU memory. These results suggest that the additional overhead is concentrated in the prefill stage.

We also analyze the time complexity of dynamic feature construction. Let TT denote the number of temporal blocks, NN the patches per block, dd the feature dimension, and w×ww\times w the local matching window:

  • Patch matching: O((T−1)Nw2d)\mathcal{O}((T-1)Nw^2d).
  • Feature differencing: O(TNd)\mathcal{O}(TNd).

With a fixed 3×33\times3 window, the overall complexity of dynamic feature construction is O(TNd)\mathcal{O}(TNd). This analysis excludes the downstream processing cost of the additional visual tokens.

We acknowledge the efficiency trade-off and will explore sparse dynamic token selection and compression to reduce inference latency.

Thank you again for your valuable suggestions. We hope these clarifications address your concerns.

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper!

Dear Reviewer,

As the ICLR 2027 rebuttal period closes tonight, we would greatly appreciate any feedback on our responses. We hope we have adequately addressed your concerns, but if any questions or doubts remain, please feel free to raise them so we can provide further clarification before the deadline.

Thank you for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.