acceptodds
Under review as a conference paper at ICLR 2027

LLM-GUIDED DISTRIBUTION PROPOSAL REFINEMENT FOR WEAKLY SUPERVISED TEMPORAL ACTION LOCALIZATION

Abstract

Weakly supervised temporal action localization aims to localize action instances in untrimmed videos using only the video-level label. However, the semantic mismatch between weak supervision and background frames may yield inaccurate action scores, which in turn causes inaccurate action boundaries. To address this, we introduce an LLM-guided distribution proposal refinement model, which explicitly models subaction semantics via four mechanisms. First, to reduce the semantic mismatch in weak supervision, we design an LLM-guided proposal learning module. It uses LLM prompts to extract the textual semantics of start, middle, and end subactions. A proposal structure loss ensures that subactions are properly ordered. Second, to alleviate ambiguous prediction among subactions, we design a contrastive proposal learning module. The discriminativeness of subaction representations is optimized via intra-action and inter-action contrastive losses. Third, to alleviate ambiguous boundary localization caused by smooth transitions, we design a proposal transition learning module. The score changes at the boundary are encouraged to be separable via a boundary transition loss. Fourth, to alleviate ambiguous boundary localization among gradual transition segments, we propose a proposal distribution refinement module. The boundary is refined to the location by considering the difference between left- and right-window distributions (measured via the first-order Wasserstein distance). Experiments show that our method achieves state-of-the-art performance on THUMOS'14 and ActivityNet 1.3 datasets.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Weakly Supervised Temporal Action Localization, LLM, Subaction Semantics, Proposal Distribution Refinement

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2zKHOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper targets weakly‑supervised temporal action localization and proposes the LDPR model. It leverages LLMs to decompose each action into start, middle and end sub‑actions. Four modules are designed for sub‑action temporal structure, representation discriminability, boundary transition and boundary refinement based on Wasserstein distance. The model achieves competitive results on THUMOS’14 and ActivityNet v1.3. Its main merit is mitigating semantic mismatch via LLM‑driven fine‑grained sub‑action text. However, this work contains multiple concerning flaws including insufficient experimental validations, weak motivation for several designs, non‑trivial inference overhead and incomplete analysis.

Critical strengths
  1. This paper addresses the common boundary offset problem in weakly‑supervised temporal action localization. The idea of using LLMs to produce start‑middle‑end sub‑action descriptions for fine‑grained semantic guidance is reasonable.

  2. The paper provides comprehensive ablation studies covering modules, hyper‑parameters and loss combinations. It also reports boundary offset as an auxiliary evaluation metric.

Critical weaknesses
  1. Sub‑action texts are generated offline by an LLM. The paper does not test robustness against noisy or incorrect LLM outputs. Unstable LLM generations may hurt overall performance in real‑world usage.

  2. Table 6 shows heavy per‑video inference latency for the full model. The PDR module runs on CPU and no acceleration solution is provided. Practical deployment will be difficult and limitations from high latency are not discussed.

  3. Boundary refinement depends on a fixed window‑size hyper‑parameter. Though ablation is provided, generalization across videos with variable action speed and action duration is not discussed.

  4. Evaluations are limited to two public datasets. Experiments on extra benchmarks or cross‑domain settings are absent. It remains unclear whether the LLM‑guided sub‑action pipeline can generalize to other action datasets.

  5. Table 6 shows heavy per‑video inference latency for the full model. The PDR module runs on CPU and no acceleration solution is provided. Practical deployment will be difficult and limitations from high latency are not discussed.

  6. The straight‑through estimator is adopted for proposal structure loss. Does it bring gradient bias? Have you monitored whether sub‑action temporal assignment stays stable during training?

  7. The work uses GPT‑5 for prompt generation while feeding text into frozen GPT‑2 Large encoder. Justification for this hybrid setup is missing. The authors do not report results using GPT‑5 as the text encoder directly.

Questions for authors

see the above weakness

Reviewer_DaEGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LDPR for weakly supervised temporal action localization. It uses a large language model to generate start, middle, and end descriptions for each action, then learns subaction representations through a frozen text encoder and visual–text cross-attention. The method combines a structure loss, a contrastive loss, a boundary-transition loss, and Wasserstein-distance-based boundary refinement. During training, it progressively updates candidate segments generated by an existing weakly supervised method. On THUMOS’14 and ActivityNet v1.3, it achieves average mAP scores of 54.5% and 28.5%, respectively.

Critical strengths
  1. The paper addresses the difficulty of using video-level supervision to guide predictions for transitional frames near action boundaries. The temporally structured subaction descriptions provide a reasonable semantic prior, and the large language model is used only to generate descriptions offline, so it need not be called during inference.
  2. The experiments cover two datasets and include a relatively comprehensive analysis of the proposed components. Tables 4–7 examine refinement during training and inference, different transition measures, individual modules, and combinations of losses. The results support the effectiveness of the overall pipeline in improving detection performance.
Critical weaknesses
  1. The structural objective admits a degenerate solution, which weakens the evidence for the claimed three-stage representation. Under Equation (1), assigning every snippet in a candidate segment to the middle stage yields zero structural loss: all adjacent transitions are self-transitions, and the middle-stage non-emptiness penalty is zero. Moreover, Section 3.4 masks the corresponding transition-loss terms when the start or end stage is empty, and Section 3.5 performs boundary refinement only for non-empty start/end stages. Although the contrastive objectives may provide additional constraints, the paper does not clarify whether they remain effective when one or more stages are absent. The authors should report dataset-level stage-occupancy statistics and explain how degenerate stage assignments are prevented in practice.
  2. The experiments provide limited evidence for the contribution of the LLM-generated temporal semantics. The LPL ablation introduces both text-conditioned processing and structural learning, while the text-encoder comparison does not isolate the effect of the description content. Controls such as duplicating descriptions across classes, shuffling the subaction descriptions, or using simple stage prompts would help determine whether the gains depend on a meaningful temporal-language prior. In addition, if predictions and ground-truth segments are rematched after refinement and candidate filtering, the set of samples included in the boundary-offset metric may change; in that case, the metric would not directly measure boundary improvements on the same set of candidates.
Questions for authors
  1. What mechanism prevents the model from predicting middle for every snippet? Please report the proportion of candidate segments containing all three stages, the frequency with which the start or end stage is empty, and how the contrastive loss is handled when positive or negative samples are unavailable.
  2. Could you add controlled experiments on the text content while keeping the network architecture and training schedule unchanged? The results would materially affect my assessment of the core contribution of LLM guidance.
  3. Could you evaluate boundary changes while fixing the candidate-to-ground-truth assignments, and report the matching coverage? Please also clarify whether the matching is one-to-one in both directions—that is, whether each prediction is matched to at most one ground-truth segment and vice versa.
  4. Table 6 shows that adding PDR in Step 2 increases inference latency per video from 17.54 ms to 576.386 ms, while average mAP rises from 53.4 to 54.1. This performance–cost trade-off should be discussed explicitly.
  5. Table 5 states only that the offset metric is computed for candidates with IoU ≥ 0.1, but it does not specify the candidate-filtering and prediction-to-ground-truth matching protocol. If candidates are filtered or matched separately before and after refinement, the set of samples included in the offset calculation may change. In that case, the metric may not reflect boundary improvements on the same set of candidates.
Reviewer_XwgfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies weakly supervised temporal action localization, where training uses video-level labels but the model must predict action boundaries in untrimmed videos. LDPR decomposes each action proposal into start, middle, and end subactions. GPT-5 generates the subaction descriptions offline from a shared prompt template; the descriptions are encoded by a frozen GPT-2 Large text encoder. The method combines a proposal-structure loss, intra- and inter-action contrastive learning, a transition loss, and a boundary refinement step using the first-order Wasserstein distance between left- and right-window score distributions. Experiments are conducted on THUMOS’14 and ActivityNet v1.3, with mAP reported at multiple temporal IoU thresholds.

Critical strengths
  1. The paper addresses a recognizable weakness of video-level supervision: it tends to emphasize action cores and provides ambiguous supervision around transitions. Modeling start, middle, and end subactions is a direct attempt to address that issue.
  2. The method is described in operational detail. In particular, the paper specifies GPT-5 and the shared offline prompt template for subaction generation, as well as the frozen GPT-2 Large encoder.
Critical weaknesses
  1. The reported ablations do not, from the described setup, isolate the semantic value of GPT-5-generated descriptions from the value of a three-part decomposition and the accompanying losses. A controlled comparison using manually written, templated, or shuffled subaction descriptions would clarify whether the LLM-generated content itself matters.
  2. Table 1 compares methods with different feature and proposal setups, while the paper’s ablations use two-stream RGB/optical-flow features on THUMOS’14. The main-text interpretation would be stronger if it clearly separated apples-to-apples comparisons from comparisons that inherit results or features from prior work. In particular, clarify which raw proposals and features are used for every baseline and for LDPR.
  3. The proposed subaction descriptions are generated from action names rather than annotated video segments. It would help to evaluate whether these descriptions are visually appropriate for the corresponding class and whether ambiguous action names produce systematic errors. This is especially relevant because the text is used to guide temporal localization.
Questions for authors
  1. Could the authors report controlled results using manual/template and shuffled subaction descriptions, keeping the rest of LDPR fixed?
  2. For each Table 1 comparison, which feature backbone and proposal source are used? Which numbers are reproduced under the authors’ setup and which are taken from prior papers?
  3. How sensitive are results to the GPT-5-generated descriptions? Have the authors tested alternative generations or paraphrases for the same class?
  4. Could the authors provide a qualitative or quantitative check of whether the start/middle/end descriptions correspond to visually observable subactions for the benchmark classes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.