acceptodds
Withdrawn from ICLR 2027

Admission Control for Training-Free Zero-Shot Temporal Action Localization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Zero-shot temporal action localization is usually benchmarked with detectors fine-tuned on in-domain segments, and those gains do not survive a change of domain. We work out of domain throughout: a frozen image–text encoder is applied directly to test videos, and no weight is ever updated. On that encoder we build a chain of four components. A cross-video memory collects per-class evidence across the test stream, gated by confidence, and blends it into the frame relevance curve. A class-conditioned admission rule, built from a per-class text contrast direction, decides which frames the memory accepts. A video-level reliability term reads the same contrast at video granularity and rescales a video's scores against the rest of the stream. A decode threshold then places extents at a quantile of each video's own relevance curve. The resulting method improves over the state of the art on THUMOS14 and ActivityNet, at both seen/unseen class splits.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
zero-shot learning, action boundary detection, VLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dt8EOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript proposes a memory-based framework for training-free zero-shot temporal action localization. It integrates cross-video memory, class-conditioned admission control, video-level reliability estimation, and adaptive decoding threshold to leverage test-time information without parameter updates. Extensive experiments on THUMOS14 and ActivityNet validate its effectiveness.

Critical strengths
  1. The experimental evaluation is comprehensive. Extensive ablation studies provide detailed insights into the contribution of each proposed component.
  2. The related work discussion is thorough and well-referenced. Moreover, the method demonstrates consistent performance gains over prior approaches across multiple benchmarks.
Critical weaknesses
  1. The four components address different aspects of the localization pipeline, but their connections are not clearly established. As a result, the framework appears more like a combination of individual improvements than a cohesive methodological contribution.

  2. The idea of leveraging test-time memory for zero-shot TAL has already been explored in [1]. Although this paper introduces parameter-free cross-video memory and additional mechanisms, its core motivation substantially overlaps with prior work. The authors should better clarify the fundamental differences from [1] beyond implementation details.

  3. Key parameters, such as k=3k=3, λdur=0.3\lambda_{\mathrm{dur}}=0.3, and w=1.2w=1.2, are mainly evaluated individually in the appendix. It remains unclear whether the framework is robust to their combined variations, particularly when generalized to other datasets.

  4. In Table 3, class-conditioned admission provides only a marginal improvement on ActivityNet (+0.06 mAP). Similarly, the seven-prompt ensemble in Table 12 offers only a small gain over the best individual prompt, while several configurations in Table 16 achieve comparable performance. These observations suggest that the independent contributions of some components may be limited.

  5. Table 22 shows that using mismatched or permuted class-to-direction assignments actually outperforms the correctly matched directions. This suggests that the improvements may primarily arise from a generic action–background contrast rather than class-specific modeling, weakening the central justification for the proposed admission mechanism.

[1] Jiang H, Tang H, Mu X, et al. Memory Matters: Boosting Training-Free Zero-Shot Temporal Action Localization with a Learnable Lookup Table. CVPR, 2026: 9762–9772.

Questions for authors
  1. Clarify the core methodological novelty compared with [1] and explain how the four components form a unified framework.
  2. Further justify the necessity of each component, particularly those providing only marginal improvements.
  3. Explain the substantial performance differences between THUMOS14 and ActivityNet, especially the limited effectiveness of class-conditioned admission on ActivityNet.
  4. Clarify why mismatched or permuted class-to-direction assignments outperform the correctly matched directions in Table 22.
  5. Provide further evidence of robustness to different hyperparameter combinations.
Reviewer_2WjhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AdCon, a training-free framework for zero-shot temporal action localization (ZS-TAL) that operates on a frozen vision-language model without requiring annotated training videos or gradient-based optimization. The framework integrates four components: (1) a cross-video memory that accumulates class-specific visual evidence from the test stream; (2) a class-conditioned admission mechanism that uses textual contrast directions to regulate which frames enter the memory; (3) a video-level reliability mechanism that adjusts segment confidence scores based on the dispersion of contrast responses; and (4) a quantile-based decoding strategy that determines temporal action boundaries from each video's relevance distribution. Experiments are conducted on THUMOS14 and ActivityNet v1.3 under two seen/unseen class partition protocols. The authors report improvements over several reproduced training-free and test-time adaptation baselines and provide extensive ablation studies, sensitivity analyses, and supplementary investigations of the proposed components.

Critical strengths
  1. The paper addresses an important limitation of existing zero-shot temporal action localization approaches, namely their dependence on in-domain annotations or test-time optimization. Maintaining a frozen vision-language backbone while accumulating useful information from unlabeled test videos offers a computationally attractive alternative. The proposed framework is lightweight and avoids gradient updates, making it potentially useful in settings where retraining is expensive or labeled target-domain data are unavailable.

  2. The paper provides a relatively comprehensive experimental evaluation, including comparisons across two benchmarks, two seen/unseen partition protocols, component-wise ablations, hyperparameter sensitivity studies, alternative memory read-outs, test-video arrival-order analysis, and a limited cross-backbone validation. The appendices are particularly detailed in distinguishing different reference configurations and investigating the contributions of individual design choices. These analyses provide useful empirical evidence beyond simply reporting final benchmark performance.

Critical weaknesses
  1. Although the framework integrates four components, their methodological novelty and necessity remain insufficiently established. Several design choices appear to be extensions or combinations of existing techniques, including prototype memory, semantic contrast, score calibration, and quantile-based decoding. More importantly, Table 22 shows that using another class's contrast direction (14.498 mAP) outperforms the correctly matched class-conditioned direction (14.309 mAP), questioning the necessity of class-specific conditioning. Similarly, Table 29 indicates that the proposed adaptive reliability mechanism performs almost identically to dataset-specific fixed-sign alternatives. These observations weaken the claimed contributions of the corresponding components. More rigorous comparisons with simpler alternatives and clearer justification of each component's distinctive role are needed.

  2. The claimed performance superiority is weakened by substantial discrepancies between published and reproduced baseline results. As shown in Table 43, AdCon achieves 17.3 mAP on ActivityNet (50/50), below the originally published FreeZAD (18.7) and AdaZAD (20.0) results, despite outperforming their reproduced versions. Although the authors acknowledge this discrepancy, its implications for the state-of-the-art claim remain unresolved. Furthermore, the framework relies on benchmark-specific configurations, including video-mean subtraction, duration priors, and temporal gap-bridging strategies. These choices raise concerns about whether the reported improvements arise from the proposed mechanisms or dataset-specific assumptions. Stronger evaluations under unified configurations and unseen data distributions are necessary to substantiate the claimed generalizability.

Questions for authors
  1. Appendix C.3, Table 22 shows that both a different class's contrast direction and a fixed permutation of class-to-direction assignments outperform the correctly matched class-conditioned direction. Could the authors explain what essential role class-specific semantics play in the proposed admission mechanism? Would a single shared action-versus-background contrast direction achieve comparable performance? Please provide a controlled comparison using a global contrast direction, correctly matched class-conditioned directions, and class-permuted directions on both benchmarks. This distinction is important for establishing whether the claimed contribution is genuinely class-conditioned admission or a more general semantic correction mechanism.

  2. Appendix C.4, Table 29 shows that the proposed adaptive reliability-sign rule performs almost identically to the best dataset-specific fixed sign. Could the authors demonstrate that the sign changes meaningfully between individual videos within the same dataset, rather than simply recovering a dataset-level polarity? Please report the distribution of predicted active-frame fractions, the proportion of videos assigned each sign, and the relationship between the proposed reliability statistic and actual segment correctness or localization accuracy. An evaluation under varying action-density distributions would be particularly useful.

  3. Given the substantial differences between the published and reproduced ActivityNet results for FreeZAD and AdaZAD in Appendix D.14, how can the authors justify the general state-of-the-art claim? Can they further clarify which baselines were evaluated using identical class splits and per-video detection outputs, and which comparisons rely on previously published aggregate values? In particular, Appendix D.12 reports bootstrap significance against published baselines, and it would be important to explain how the required baseline-level prediction distributions were obtained. Please provide the paired per-split results, confidence intervals, and complete evaluation settings where possible.

  4. The proposed decoder uses a fixed quantile of 0.60, while ActivityNet additionally employs a 1600-frame gap bridge and a single-instance cap. How robust is the approach when the proportion, number, and duration of action instances differ substantially from those observed in THUMOS14 and ActivityNet? Since the quantile threshold selects approximately 40% of frames before merging, does this assumption systematically overestimate activity in sparse-action videos? Please evaluate the method with adaptive thresholds or alternative density-independent decoding approaches and clarify whether benchmark annotation statistics were used to choose dataset-specific configurations.

  5. The cross-video memory relies on pseudo-labels and accumulated test-stream statistics, potentially exposing it to class imbalance, distribution shifts, and erroneous memory updates. Although arrival-order permutations are considered in Appendix D.10, how does the framework behave under nonstationary class distributions, concentrated pseudo-label errors, or an extended cold-start period? In addition, Appendix C.3.1, Table 23 identifies applying the contrast to both the relevance curve and buffer admission as the deployed configuration, whereas the inference pseudocode in Appendix A.4 appears to apply the contrast only to admission. Could the authors reconcile these descriptions and specify the exact implementation corresponding to the reported results?

  6. Although the supplementary material contains extensive experiments, the main paper provides relatively limited experimental analysis, with most important investigations relegated to the appendix. In particular, the discussion of the experimental results is rather brief and does not sufficiently analyze the underlying reasons for the observed performance improvements, the differences across datasets, or the limitations of the proposed components. Moreover, Page 8 contains substantial unused whitespace, particularly between Tables 1 and 2, indicating inefficient utilization of the available page space. The authors are strongly encouraged to revise the layout and incorporate more substantive experimental evidence into the main paper, such as detailed component comparisons, qualitative localization results, failure-case analyses, and additional discussions of the ablation findings. These revisions would substantially improve the completeness, clarity, and persuasiveness of the empirical evaluation.

Reviewer_eUhEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The resarch problem addressed is practically meaningful, and the proposed method has low computational cost. The use of historical test videos helps reduce the cost of additional training.

Critical strengths

The resarch problem addressed is practically meaningful, and the proposed method has low computational cost. The use of historical test videos helps reduce the cost of additional training.

Critical weaknesses

The experiments do not provide convincing evidence of the effectiveness of the proposed approach compared to existing approaches. There are so many missing details that it is difficult to draw many conclusions.

Questions for authors
  1. Is it necessary to construct memory on the datasets THUMOS and ActivityNet? What is the size of the memory you constructed? Why is it necessary to simultaneously introduce class-conditioned memory admission, video-level reliability, and quantile decoding? Does each design address a distinct problem, or do they perform repeated calibration on the same relevance score?
  2. The abstract and conclusion claim that the proposed method outperforms state-of-the-art approaches on two datasets under two class splits. Under the authors’ unified reproduction pipeline and evaluation settings, ADCON achieves improvements over the reproduced baselines. However, these gains are insufficient to support the unrestricted claim in the abstract that it comprehensively surpasses published state-of-the-art methods.
  3. The paper does not accurately delineate the differences between its core innovations and prior works. LLT already maintains high-confidence candidates using historical test videos to realize cross-video knowledge accumulation. Therefore, cross-video memory itself cannot be regarded as a unique contribution of this work. Moreover, OZ-TAL determines whether the current frame enters memory based on foreground and background text similarity. In contrast, this paper mainly describes its mechanism as using class-agnostic background to correct the relevance curve, without sufficient discussion of its memory admission mechanism.
  4. The description of the paper’s contributions is somewhat overstated. The authors fail to abstract the specific, genuine contributions of this work from the practical technical implementation.
  5. The comparison metrics for component ablation in Table 3 are inconsistent, which hinders the evaluation of the individual gain of each module. The first two items in the table correspond to removing components from the MEMORY+ADMISSION setup, while the latter two items add reliability calibration and quantile decoding to this setup, respectively. This is not a unified ablation study obtained by sequentially removing components from the full ADCON model. It is recommended to report sequential ablations on the complete model and clarify the interaction effects among components.
  6. Table 2 and Table 3 mark the final configuration as applying text contrast at both the relevance curve and memory admission stages. However, Equation (10) in the main text and the full pseudocode in the appendix only explicitly demonstrate the correction of the admission curve. It is necessary to clarify which implementation corresponds to the results in the main tables. In addition, the population memory stores at most 256 snippets per class, yet the eviction strategy once the capacity is exceeded, its relationship with the cache of 4 video candidates per class, and whether the corresponding snippets are synchronously deleted after video candidates are replaced, are not specified.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.