- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DynaSense improves video temporal grounding by explicitly modeling changes between neighboring frames through local patch matching and position-aware feature differencing. It processes appearance and dynamic information using a shared visual backbone. The authors also introduce VideoSCR for learning before-and-after state descriptions, followed by temporal grounding training. Experiments report improvements over the underlying video-language models and transfer to several temporal understanding tasks.
Critical strengths
-
DynaSense constructs dynamic representations directly from native patch embeddings and processes appearance and dynamic inputs through a shared visual backbone. This avoids requiring a separately pretrained motion encoder. The combination of matched appearance differences and positional displacement cues also provides an interpretable rationale for capturing changes relevant to event boundaries.
-
DynaSense consistently improves temporal grounding over its corresponding Qwen3-VL baselines across the evaluated model sizes and reported TimeLens subsets. It also improves average grounding performance on E.T.Bench, with gains on several complementary temporal tasks. These results support the practical value of the complete approach and suggest that its benefits extend beyond the training task.
Critical weaknesses
- The central matching mechanism is insufficiently justified and validated.
DynaSense makes several consequential structural choices: hard nearest-neighbor assignment rather than soft aggregation, unrestricted many-to-one correspondences, mandatory assignment without a confidence-based rejection criterion, matching independent of the language query, and a fixed (3x3) search window. The first four choices are specified in the method but are not individually motivated or evaluated against representative alternatives. Consequently, it remains unclear how these decisions affect correspondence quality and downstream temporal grounding, or whether they are necessary for the reported improvements.
For the search window, the authors provide a plausible motivation based on computational cost, locality, and avoiding spurious matches. However, Table 4 evaluates window sizes only on Charades-TimeLens using the 2B model with reduced training data. Although (3x3) performs best, with no monotonic degradation among the larger windows. These results support the selected configuration in this setting, but do not establish its robustness across datasets or motion regimes. Moreover, the explanation that larger windows introduce spurious correspondences is not supported by a direct assessment of matching accuracy.
The computational motivation also remains unquantified: the paper reports no FLOPs, latency, or memory measurements across window sizes, preventing an assessment of the accuracy-cost tradeoff. Sparse adaptive mechanisms, such as deformable attention, offer a relevant alternative by learning a limited set of sampling locations instead of restricting candidates to a fixed neighborhood. Their suitability for this architecture would require evaluation, but the paper neither discusses this alternative nor provides a comparison under comparable computational and training budgets.
- The rationale for the proposed architecture remains insufficiently convincing.
The paper motivates DynaSense through the limitations of appearance-based representations in capturing fine-grained temporal changes. However, it addresses this limitation through explicit local matching and position-enriched differencing while retaining framewise attention in the visual backbone. A natural alternative would be to introduce temporal attention into visual encoding and learn interactions across frames directly. The paper does not establish what advantage its specific matching-and-differencing construction provides over this alternative.
Reusing a pretrained visual backbone is a reasonable practical motivation, but it does not preclude adding lightweight temporal attention layers or temporal adapters to the same backbone. Such a comparison would also avoid the confounding effects of replacing the entire encoder with a differently pretrained video model. Nevertheless, the paper provides no controlled comparison against this type of baseline. The reported improvements therefore do not resolve whether the proposed construction offers a distinctive benefit over a conventional learned temporal module.
Without this evidence, I find it difficult to understand why the proposed architecture should be preferred, or which aspect of its explicit correspondence-and-difference formulation is responsible for an advantage over standard temporal modeling.
Questions for authors
-
What motivates hard, many-to-one, unconditional, and query-independent matching, and can you provide ablations demonstrating the effects of these choices relative to representative alternatives?
-
Does the preference for a (3x3) window generalize across datasets and motion regimes, and can you provide matching-quality and computational measurements to substantiate the proposed explanation for this choice?
-
How does DynaSense compare with temporal attention layers or temporal adapters added to the same backbone under identical training conditions and comparable visual token budgets and additional trainable capacity?
7 comments
Part II
3. Response to Weakness 2 / Question 3
We wish to clarify that explicit dynamic representation construction and learnable temporal interaction modeling are distinct approaches. Temporal attention implicitly learns temporal relations through interactions across frame features. DynaSense instead constructs dynamic representations containing content changes and spatial displacement information through local correspondence and feature differencing, and reuses the pretrained ViT to encode them. Cross-frame changes are already explicitly encoded during dynamic feature construction, and the downstream LLM jointly processes visual information across time. Due to computational constraints, the ViTs in common Video-LLMs [1–5] typically do not perform long-range cross-frame interactions. Local temporal processing is mainly limited to adjacent frame pairs, while longer-range temporal modeling is primarily handled by the LLM.
[1] Qwen Team. Qwen3-VL Technical Report.
[2] Wang et al. InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency.
[3] V Team et al. GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning.
[4] Clark et al. Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding.
[5] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. LLaVA-Video: Video Instruction Tuning With Synthetic Data. Transactions on Machine Learning Research (TMLR), 2025.
Thank you for your response.
First, I would like to state that I did not rely on AI as the basis for my assessment of this paper. I wrote my review in my mother language and used an AI tool only to translate it into English, so some parts may have been described in terms or expressions that differ from those used in the paper. I may also have misunderstood parts of what the authors intend to convey. I will state this use of an AI tool for translation in the "AI Documentation For Reviewing" field.
I also apologize that my initial review was not concise and contained a relatively large amount of information. The conference emphasizes conciseness in reviews, and I overlooked this in my initial review. I had many questions about the paper, and in compressing them into the "one or two critical weaknesses" that the conference asks for, some of the detailed explanation was lost. I apologize for this.
The core issue I want to raise is that the design of "Video Dynamic Feature Matching" (Sec. 3.2) is not sufficiently justified. The authors' response resolved many of my concerns, but the following points remain open.
1. Hard nearest-neighbor matching (Eq. 2). Eq. (2) selects only the single most similar patch in the preceding frame via the arg min of the L1 distance. What is the reason for this? Instead of selecting one, what if the nine candidates in the 3×3 window were averaged with weights given by their similarity to the current patch? The rebuttal explains that hard nearest-neighbor matching serves as a deterministic reference for differencing, but a similarity-weighted average is equally deterministic, so the question remains unanswered in both the paper and the rebuttal.
2. Insufficient validation of the 3×3 local matching window. I raise this because of the statement in L421–423 that larger windows introduce spurious matches. Table 4 is reported only on Charades-TimeLens with DynaSense-2B trained on VideoSCR-28K and TimeLens-20K, so it is unclear whether 3×3 remains the best choice on other datasets or on videos with large motion. Substantiating this claim requires either a direct assessment of matching accuracy or validation on more diverse datasets. The paper only compares grounding performance across window sizes, and neither the paper nor the rebuttal addresses this question.
3. Validity of the matching when no true match exists in the window. The proposed matching is likely to work well in ordinary cases, but it becomes problematic when the true match of a patch is not inside the local window of the preceding frame, for example with newly appearing objects, fast object motion, or scene cuts. Because each current patch retains a match (L231–232), such patches still receive a reference. The rebuttal touches on this by explaining that keeping a reference for every patch makes it possible to compute change features at every location. In my view, however, this answers "why is every patch given a match?" rather than "what happens when the match is wrong?". I would like to hear the authors' view on this.
Finally, I gave this paper an initial rating of 1 because I considered the experimental evidence supporting the design of "Video Dynamic Feature Matching", one of the core components of the paper, to be far too limited. Since this is a core technique on which the paper is built, I believe its design needs to be validated rigorously. I am aware, however, that this is not the paper's only contribution. Apart from this component, I find the design sound and consider this a good paper. The rebuttal has already resolved my concerns to some extent, and if the authors address the remaining concerns above, I will reflect this in my final assessment.
Plus, I raise the score to 2 because the author has partially resolved my concerns.
Part I
Thank you for your review and suggestions. We recognize the heavy reviewing workload and the role of AI tools in preliminary assessment. However, evaluations should reflect independent judgment and careful verification, grounded in the paper's actual contributions and experimental evidence, rather than adopting AI-generated content without scrutiny. Respect for others' scholarly work should be a basic principle of peer review. Several concerns appear to diverge from the motivation and scope of our work; we respond below based on our understanding of the questions.
Our core contribution is to construct explicit dynamic representations directly in the feature space of a pretrained visual encoder, improving Video-MLLMs' perception of fine-grained video state changes and temporal grounding. DynaSense first establishes visual correspondences between adjacent frames through position-free local patch matching. It then differences matched features enriched with spatial positional encodings, preserving both content changes and spatial displacement information. A shared visual backbone and independent projection modules process the static and dynamic representations. We also construct VideoSCR and adopt two-stage training to transfer state-change understanding to temporal grounding. Experiments show consistent improvements across model scales and benchmarks, together with stronger zero-shot generalization.
1. Response to Weakness 1 / Question 1
Our central claim is that local correspondence and position-aware differencing construct change representations in the native visual feature space that benefit temporal grounding. Hard nearest-neighbor matching provides a deterministic reference for differencing; many-to-one matching avoids an additional one-to-one constraint; and retaining a reference for every patch enables change features to be computed at every location. Query relevance is modeled by the downstream LLM. Tables 3–6 present extensive ablations covering the combination of VideoSCR and dynamic representations, local matching window size, the order of matching and differencing relative to positional encoding, and shared versus independent mergers.
2. Response to Weakness 1 / Question 2
We randomly sampled 1,000 examples from the Charades-TimeLens test set and measured inference costs for Qwen3-VL and DynaSense at both 2B and 8B scales.
Table III. Inference cost comparison.
With identical sampled frame counts, DynaSense uses approximately 54.3% more visual tokens. Average generation latency increases by 36.4% and 46.6% at 2B and 8B, respectively, while time to first token (TTFT) increases by 60.4% and 66.9%. Time per output token increases by only 7.6% and 7.0%, with small changes in peak GPU memory. These results suggest that the additional overhead is concentrated in the prefill stage.
We also analyze the time complexity of dynamic feature construction. Let T denote the number of temporal blocks, N the patches per block, d the feature dimension, and w×w the local matching window:
With a fixed 3×3 matching window, the overall complexity of dynamic feature construction is O(TNd). This excludes the downstream processing cost of the additional visual tokens.
We acknowledge the computational cost of the current method and will explore sparse dynamic token selection and compression to reduce inference latency.