acceptodds
Under review as a conference paper at ICLR 2027

TopoAtom: Query-Conditioned Semantic Atomization for Zero-Shot Video Temporal Grounding

Abstract

Zero-shot video temporal grounding (ZS-VTG) increasingly leverages frozen multimodal models to localize natural-language queries without task-specific training. Yet existing approaches largely build localization upon predefined temporal primitives, such as uniformly sampled frames or fixed-length clips, and subsequently measure their relevance to the query. Such fixed granularity overlooks a fundamental property of videos: semantics are structurally organized rather than uniformly distributed along the timeline. We introduce **TopoAtom**, a training-free framework that replaces predefined temporal primitives with adaptively discovered, query-conditioned semantic atoms. Based on pairwise clip distances, we build an -inspired connectivity filtration, whose component-merging process is compactly represented by a minimum spanning tree and its induced single-linkage hierarchy. Starting from the whole video, TopoAtom traverses this hierarchy top-down and recursively splits a component only when a finer descendant exhibits stronger query correspondence; otherwise, the component is retained as a semantic atom. These adaptive semantic atoms provide a structure-aware foundation for candidate generation. We further calibrate each candidate using a topology-derived persistence score, moving beyond sole reliance on query relevance by explicitly incorporating inherent connectivity and compactness. Experiments on Charades-STA, ActivityNet-Captions, and QVHighlights consistently demonstrate the effectiveness of TopoAtom.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video temporal grounding, zero-shot video temporal grounding, persistent homology

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_eXFWOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TopoAtom, a training-free video temporal grounding method that constructs variable-length semantic atoms based on video structure and query relevance. It builds a visual similarity hierarchy and applies query-conditioned pruning to select atoms. Consecutive atoms are combined into candidate segments, which are ranked using query relevance and candidate persistence. The method achieves improvements across multiple benchmarks and frozen backbones, supported by ablations and theoretical analysis.

Critical strengths
  1. The paper provides thorough experimental and theoretical analyses. It includes component ablations, candidate oracle performance, hyperparameter sensitivity, runtime measurements, and qualitative examples. The appendix further analyzes the mathematical properties of the proposed mechanisms, providing useful insights into the method.

  2. The method demonstrates consistent improvements across multiple benchmarks and frozen backbones. Experiments on Charades-STA, ActivityNet-Captions, QVHighlights, and the additional long-video evaluation on TACoS support its effectiveness across different datasets and settings.

Critical weaknesses
  1. The computational overhead is not sufficiently evaluated. TopoAtom encodes multiple clips and grouped segments with Qwen, potentially introducing substantial computational cost. The claim that node similarities can be computed in a few batched forward passes (Section 3.3) does not necessarily imply low computation or memory usage. Appendix A.4 reports runtime but lacks detailed encoding settings, batching configurations, and GPU memory measurements, making practical efficiency and long-video scalability difficult to assess.

  2. The core selection mechanisms lack sufficient justification for their grounding benefits. It is unclear why comparing a node's query similarity with those of its descendants should select appropriate temporal granularity rather than excessively splitting segments. Joint encoding produces scores that differ from averaged clip similarities, but this alone does not explain why the pruning rule should favor semantically appropriate segments. Similarly, the paper does not sufficiently explain why structural coherence should improve query-relevant localization or which limitations of relevance-only ranking it addresses. These gaps limit understanding of why the proposed mechanisms work.

Questions for authors
  1. How significant is the computational overhead of the proposed method? Since TopoAtom requires additional segment encoding with Qwen, could the authors clarify the number of encoded segments, batching configurations, and peak GPU memory usage?

  2. Why should query-conditioned pruning select appropriate temporal granularity? Joint encoding produces similarity scores that differ from those of individual clips, but it remains unclear when a semantically complete segment would score higher than its descendants. What prevents the method from excessively splitting segments?

  3. Why does the pruning mechanism work well for queries about both background context and foreground interactions, as illustrated in Section 4.4? The paper does not clearly explain which aspects of the pruning rule lead to better localization than TAG in these examples.

  4. What limitations of relevance-only ranking does candidate persistence address? The paper introduces candidate persistence to complement query relevance, but does not clearly explain what errors arise from relevance-only ranking or why structural coherence should help correct them. While Table 4 reports changes in candidate rankings and Appendix A.3 presents persistence distributions, could the authors clarify when and why structural coherence helps or hurts localization accuracy?

  5. Why are different baselines used in Tables 1 and 2? Could the authors clarify the baseline selection criteria, including which results are reproduced and why some relevant baselines are absent from QVHighlights?

4 comments
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer eXFW,

Thank you again for your questions about the design and evaluation.

We expanded the explanation of when query-conditioned pruning preserves a broader segment or splits it further (Sections 3.3 and 4.4). We also added examples of when candidate persistence helps or hurts (Fig. 5) and reported batch sizes, encoded-segment counts, latency, and peak GPU memory for long videos (Table 10). Our reply also explains how we selected baselines for each benchmark.

If any concern remains, we would be glad to address it in the remaining discussion period.

Dear Reviewer eXFW,

Thank you again for the time and effort. As the rebuttal period draws to a close, we wanted to briefly follow up. If you have any remaining questions about our response or the revised manuscript, we’d be happy to clarify or discuss them.

If the authors provide a latency comparison with other VTG methods under comparable conditions, I would feel more confident in maintaining my weak accept rating. However, I suspect that some existing methods may have substantially lower latency, potentially below one second, compared with the reported latency of over 10 seconds for TopoAtom. I find the proposed idea interesting, but its computational cost raises concerns about practical applicability. Further efficiency improvements may be needed, and I encourage the authors to explicitly discuss this limitation.

AuthorsOpenReview ↗

Follow-up on the inference latency

Thank you for your following up. We chose TAG as a strong open-source baseline because both methods organize video content into temporal units for grounding. TAG uses query-independent clustering with a fixed cluster count, while TopoAtom uses the query to adapt the granularity of video parsing. This comparison helps quantify the inference cost associated with our design.

Charades-STA. Figure 7 reports total latency and a breakdown by stage. TopoAtom with Qwen3-VL-Embedding 8B takes 5.1 s, compared with 4.0 s for TAG. Most of the difference comes from feature encoding.

Long-video comparison. The latencies above 10 s were measured on ActivityNet-Captions videos longer than two minutes and on TACoS. We added TAG results on the same videos in Table 10.

DatasetMethodBatch sizeLatency (s)Peak GPU memory (GB)Avg. atoms or clustersAvg. encoded inputs
ActivityNet-Captions (>2 min)TopoAtom (8B)410.322.629.479.2
ActivityNet-Captions (>2 min)TAG2566.614.591188.1
TACoSTopoAtom (8B)216.922.735.9135.5
TACoSTAG25610.914.892032.6

For TopoAtom, batch size and encoded inputs refer to hierarchy-induced video segments. For TAG, they refer to individual frames sampled at 3 fps. TAG uses nine clusters in both evaluations.

Limitations and future directions. The additional time is an overall reasonable trade-off for the query-adaptive parsing. Note that for a video with TT base clips, TopoAtom encodes at most 2T−12T-1 distinct video segments to score the hierarchy, including the base clips (added in Section 3.3). We also added a discussion at the end of Section 4.4 stating that the latency may limit applicability in some scenarios that extremely sensitive to the latency. We also outline a direction for improving efficiency by screening hierarchy nodes with inexpensive cues before joint encoding.

Reviewer_T5kvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TopoAtom proposes a training-free framework for zero-shot video temporal grounding, addressing the limitations of uniformly sampled frames or fixed-length video clips. The method constructs an H_0-based hierarchical video representation using a MST, discovers query-conditioned semantic atoms through adaptive pruning, and introduces a topology-inspired candidate persistence score to incorporate structural connectivity and separation into temporal localization. Extensive experiments on Charades-STA, ActivityNet-Captions, and QVHighlights demonstrate strong performance, achieving the best results on 15 out of 16 metrics. Ablation studies validate the effectiveness of semantic atomization and candidate persistence, while experiments across different VLM backbones further demonstrate the robustness of the proposed framework across the diverse models.

Critical strengths
  1. Query-adaptive semantic atomization provides a meaningful methodological contribution.
  2. The proposed Candidate Persistence has a clear theoretical foundation and interpretability.
Critical weaknesses
  1. Insufficient analysis of the query-adaptive pruning strategy. Although the proposed H₀ merge hierarchy is mathematically equivalent to single-linkage hierarchical clustering, the key novelty of the semantic atomization module lies in its query-adaptive pruning strategy. Table 3 compares different Atom Mining strategies, including temporal-adjacent graph construction (TG-Only) and TAG-style clustering, but does not directly isolate the contribution of query-adaptive pruning. Consequently, it remains unclear how much of the performance improvement comes specifically from query-conditioned partitioning rather than hierarchical grouping itself.

  2. Significant degradation in oracle performance. Although semantic atomization improves the final grounding accuracy, Table 3 shows a substantial decrease in oracle performance. Specifically, oracle [email protected] drops from 99.27 to 89.17, while oracle [email protected] drops from 98.29 to 74.07 after introducing atom mining. This suggests that the adaptive atomization process may sacrifice fine-grained boundary coverage, potentially limiting the achievable localization accuracy. Although the reduction may partly result from the smaller proposal space after atomization, the paper does not sufficiently disentangle this factor from the loss of boundary precision, which is particularly important for temporal grounding tasks requiring accurate localization boundaries.

Questions for authors
  1. The contribution of query-adaptive pruning. Could the authors provide a controlled comparison between query-adaptive pruning and conventional query-agnostic pruning strategies, such as fixed distance thresholds or fixed numbers of clusters, while keeping the MST hierarchy strategy, backbone, and candidate scoring pipeline unchanged? It would also be helpful to report results under comparable atom counts or proposal budgets to determine whether query-adaptive pruning itself provides a meaningful advantage.

  2. The reduction in oracle proposal quality. Could the authors further analyze why atom mining significantly reduces oracle performance, particularly at higher IoU thresholds? Additionally, could a local boundary refinement strategy based on the original clips recover the lost oracle coverage while preserving the benefits of semantic atomization?

  3. Candidate persistence is presented as a major contribution, but its advantage over simpler structural scoring functions has not been sufficiently established. According to Table 4, with γ = 0.5, candidate persistence changes 42.7% of Top-1 predictions, while 19.6% are improved and 19.0% are worsened. Although the overall performance improves, these results raise questions about when the persistence score provides reliable structural information and whether its specific formulation is necessary. Could the authors provide additional ablations comparing the proposed persistence score with simpler structural alternatives, such as internal compactness alone, external separation alone, and the unnormalized difference between the two? It would also be helpful to analyze under which conditions candidate persistence improves or worsens localization, and whether similar improvements are observed on datasets beyond Charades-STA.

  4. Incomplete mathematical definitions and inconsistencies in experimental results. How are candidates covering the entire video handled when the background set in Eq. (9) is empty and the external attachment scale is undefined? Additionally, could the authors clarify the discrepancy between the mIoU reported in Table 3 (46.10) and Table 4 (47.10) for the experiment without candidate persistence?

4 comments
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer T5kv,

Thank you again for suggesting the pruning controls and local boundary refinement.

The revision compares query-adaptive pruning with two query-independent cuts on the same hierarchy (Table 3). We also included a local refinement experiment that recovers some oracle boundary coverage (Table 11). We also compare candidate persistence with simpler structural scores (Table 9) and further illustrate when it helps or hurts (Fig. 5).

If any concern remains, we would be glad to address it in the remaining discussion period.

Dear Reviewer T5kv,

Thank you again for the time and effort. As the rebuttal period draws to a close, we wanted to briefly follow up. If you have any remaining questions about our response or the revised manuscript, we’d be happy to clarify or discuss them.

I appreciate the authors' detailed rebuttal and additional experiments. The new controlled comparisons address my main concern regarding query-adaptive pruning. The explanation of the oracle recall trade-off is also reasonable, although some limitations remain. Overall, I am raising score of 3.

Thank you again for your time and feedback. Your insightful comments helped us improve the paper.

Reviewer_3QraOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TopoAtom, a training-free framework for zero-shot video temporal grounding that addresses the limitations of predefined temporal units, such as uniformly sampled frames or fixed-length clips. The authors argue that video semantics are non-uniformly distributed over time and introduce query-conditioned semantic atomization to adaptively discover temporal units based on video structure and query relevance. Specifically, TopoAtom constructs an H0-inspired connectivity hierarchy using pairwise clip similarities and a minimum spanning tree, followed by query-adaptive top-down pruning to identify semantic atoms at different temporal granularities. The framework further introduces a topology-derived candidate persistence score to incorporate structural coherence into candidate ranking alongside query relevance.

Critical strengths

1. Meaningful problem formulation and well-motivated approach. The paper identifies a meaningful limitation of existing zero-shot video temporal grounding methods that rely on predefined temporal units. The idea of adapting temporal granularity according to both video semantics and the language query is intuitive and provides an interesting perspective on how video content should be organized for temporal grounding.


2. Effective training-free framework with query-conditioned semantic atomization. The proposed framework leverages pretrained vision-language representations to discover query-conditioned semantic atoms without task-specific training. By combining hierarchical video structure analysis, query-adaptive pruning, and topology-aware candidate scoring, the method provides a structured approach to temporal localization beyond fixed-length clips.


  1. Strong empirical performance and comprehensive ablation studies. TopoAtom achieves competitive or superior performance compared with the evaluated zero-shot VTG baselines across QVHighlights, Charades-STA, and ActivityNet-Captions. The component-wise ablations provide useful evidence for the effectiveness of semantic atomization and candidate persistence. Furthermore, evaluations across different frozen VLM backbones, including InternVideo and Qwen3-VL-Embedding variants, demonstrate that the improvements are not restricted to a single pretrained model.
Critical weaknesses

1. Unclear trade-off between query-adaptive atomization and precise candidate boundary coverage.

Although the proposed semantic atomization improves final grounding performance, Table 3 shows a substantial reduction in oracle candidate quality compared with fixed-clip enumeration ([email protected]: 98.29 to 74.07, and mIoU: 96.35 to 77.03). While this reduction may partly result from the smaller candidate space, the high-IoU oracle performance of TopoAtom is also only marginally higher than that of the temporal-adjacent graph and TAG-style clustering variants. This raises questions about how effectively the discovered semantic atoms preserve precise event boundaries. In particular, the pruning criterion in Eq. (5) is based on relative query similarity rather than boundary alignment, and it remains unclear how these criteria relate to each other. A more detailed analysis of atom granularity, ground-truth boundary coverage, and the effect of query-adaptive pruning would strengthen the central motivation of the paper.


2. Insufficient isolation of the effects of candidate-overlap post-processing.

While the paper provides component-wise ablations for atom mining, contrastive grounding scores, and candidate persistence, the final prediction procedure additionally incorporates overlap-based score aggregation (Eq. 12 in Appendix A.1). Since semantic atomization changes the number, duration, and overlap distribution of candidate segments, this post-processing mechanism may interact significantly with the proposed atomization strategy. Without an explicit ablation, it is difficult to determine how much of the performance improvement is attributable to semantic atom discovery itself versus its interaction with candidate-overlap aggregation. A controlled comparison with and without Eq. (12), under both fixed-clip and atom-based candidate generation, would clarify these contributions.


3. Limited computational scalability analysis for longer videos.

Although the paper reports inference latency on Charades-STA, the computational scalability of TopoAtom to substantially longer videos remains unclear. The framework involves pairwise clip-distance computation, additional VLM encoding of hierarchy-induced video segments, and quadratic enumeration of atom-level candidates. While the proposed approach is training-free, these operations may introduce substantial inference overhead as video duration increases. Reporting inference latency, peak GPU memory usage, and the number of encoded segments and generated atoms on longer benchmarks already considered in the paper, such as ActivityNet-Captions and TACoS, would help establish the practical scalability of the method.

Questions for authors
  1. Query-adaptive atomization and boundary coverage Table 3 shows that atomization improves final grounding performance but substantially reduces oracle candidate quality. Could the authors further analyze this trade-off, particularly how query-adaptive pruning affects the preservation of ground-truth temporal boundaries? Additionally, a controlled comparison between query-adaptive and query-independent pruning (e.g., cutting the same H0H_0 merge hierarchy H\mathcal{H} at a fixed distance threshold ϵ\epsilon or matching the average number of atoms MM without query conditioning) under the exact same scoring procedure would help isolate the true contribution of query conditioning.

  1. Contribution of candidate-overlap post-processing.
    Could the authors provide an ablation with and without the overlap-based score aggregation in Eq. (12)? In particular, comparing its effect under both fixed-clip and atom-based candidate generation would help clarify whether the improvements primarily arise from semantic atomization or its interaction with the post-processing mechanism.

  1. Computational scalability on longer videos.
    Could the authors report inference latency, peak GPU memory usage, and the number of encoded segments and generated atoms on longer-video benchmarks, such as ActivityNet-Captions and TACoS? This would help assess how the computational cost of hierarchy construction, additional VLM encoding, and candidate enumeration scales with video duration.
3 comments
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer 3Qra,

Thank you again for your thoughtful questions about boundary coverage, score aggregation, and long-video inference.

We added fixed-distance and fixed-count cuts on the same hierarchy (Table 3), aggregation ablations for TopoAtom and TAG (Table 7), and latency and peak memory on long videos (Table 10).

If any concern remains, we would be glad to address it in the remaining discussion period.

Thank you for the clarifications and additional experiments. The rebuttal sufficiently addresses my main concerns regarding query-conditioned atomization, overlap-based aggregation, and long-video inference costs. I will maintain my Weak Accept recommendation.

Thank you for your time and effort. We are glad the clarifications addressed your concerns. Your feedback helps us make the paper clearer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.