acceptodds
Withdrawn from ICLR 2027

PAEM: Progression-Aware Event Matching for Spatio-Temporal Video Grounding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Spatio-temporal video grounding aims to localize the spatio-temporal tube corresponding to a natural-language query in an untrimmed video. Existing methods exploit query-related semantic cues to guide spatial and temporal localization. However, semantic relevance alone may be insufficient to determine the complete extent of a queried event, as salient local cues can lead to incomplete event spans, while broader context may still fail to distinguish candidates that share substantial content but differ in event structure. We argue that event progression, encompassing how an event develops over time and connects to surrounding activities, can provide a structural basis for distinguishing semantically similar candidates. In this paper, we propose Progression-Aware Event Matching (PAEM), which explicitly incorporates event progression into spatio-temporal matching. Specifically, PAEM combines stage-aware context matching and state-aware process matching to jointly capture contextual stage relations and internal event progression. To capture the stage relations, we conduct stage-aware context matching by aggregating query-relevant responses before, within, and after each candidate span. Meanwhile, state-aware process matching characterizes internal state dynamics through latent responses and conditional transitions. The two matching scores jointly determine the target entity and its temporal interval. Experiments across 4 vision-language backbones show m_vIoU improvements of 3.6%–9.9% ↑ over corresponding baselines on HCSTVG-v1/v2, demonstrating PAEM's effectiveness. When transferred to related grounding tasks in a zero-shot setting, PAEM yields consistent improvements over the corresponding baselines, with gains of up to 9.4% ↑ across the primary metrics, highlighting its generalizability.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spatio-temporal video grounding, video-language understanding, latent event progression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mgKbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a novel framework for spatio-temporal video grounding that utilizes event progression as the structural basis for localization, constructing an entity-time lattice via Grounding DINO and SAM 2.1. The model integrates dual branches, Stage-Aware (combining partial optimal transport with stage probabilities) and State-Aware (modeling monotonic hidden state progressions with hierarchical path marginalization) for joint scoring, optimized under expected vIoU objectives.

Critical strengths
  1. The method decouples event completeness into external PRE/EVENT/POST stage constraints and internal monotonic hidden-state paths, transforming abstract structural completeness into explicit, computable mathematical scores.
  2. The evaluation spans four mainstream vision-language backbones and multiple downstream tasks, delivering consistent and substantial gains across all benchmarks.
Critical weaknesses
  1. The current experiments do not sufficiently disentangle the performance gains brought by event progression modeling from those introduced by the new localization training objectives. Furthermore, direct empirical validations are lacking regarding the model's sensitivity to action ordering and the semantic interpretability of the latent states.
  2. The framework relies heavily on pre-extracted entity trajectories, yet it lacks an analysis of sensitivity to trajectory quality as well as runtime and GPU memory overheads. Crucial implementation details—such as the number of latent states, inference across temporal windows, and score calibration across candidate intervals of varying lengths—remain insufficiently clear.
Questions for authors
  1. Does the progression structure improve performance beyond the localization losses? Please provide a standard temporal-network baseline with the same inputs, losses, and comparable parameter count.
  2. Can PAEM reliably distinguish between queries or video segments that contain identical actions occurring in different orders? In addition, could you provide an alignment analysis between latent state transitions and actual sub-action boundaries to substantiate the interpretation of "event progression modeling"?
    3.How does performance vary when switching detection/tracking backbones, or under scenarios with false negatives, identity switches, and broken trajectories? During training, how does the model handle candidates that have very low spatio-temporal overlap with the ground-truth target?
    4.Could you provide the specific value of the latent state number MM, the exact candidate interval generation procedure, the inference mechanism across 24-frame windows, and the latency/peak GPU memory scaling with respect to video length, entity count, and MM?
Reviewer_gYGtOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Progression-Aware Event Matching (PAEM) for spatio-temporal video grounding (STVG), which explicitly incorporates event progression into the candidate matching process. The method consists of two complementary modules: (1) Stage-Aware Context Matching, which evaluates each candidate span against PRE/EVENT/POST temporal stages along the entity trajectory, and (2) State-Aware Process Matching, which models internal event evolution through latent state responses and conditional state transitions. Three video understanding tasks include spatial grounding, temporal grounding, and referring video object segmentation are used for evaluation.

Critical strengths
  1. The generalization ability of the proposed method for ST-Align, MeViS, Charades-STA, and TVGBench.

  2. The two-module design provides a coherent decomposition: stage-aware context matching captures trajectory-level temporal structure, while state-aware process matching models within-event progression, and the unified inference formulation demonstrates their complementarity.

Critical weaknesses
  1. Novelty is a big issue. The similar idea has been exploited by incorperating event for more accurate grounding [1,2]. This is also discussed in the related work section. Taking these works into consideration, the difference is very incremental.

[1] Conditional Multi-Event Temporal Grounding in Long-Form Video, neurips [2] Modularized Dynamic-Granularity Video LLM for Multi-Event Long Video Understanding, cvpr

  1. Meanwhile, event alignment has been exploited in [1]. [1] Event-Oriented State Alignment Network for Weakly Supervised Temporal Language Grounding, entropy

  2. The performance is heavily dependent on pre-extracted entity tubes. PAEM relies entirely on offline Grounding DINO + SAM2.1-Small to produce candidate entity tubes.

  3. In table 2, with Qwen2.5-VL-7B, PAEM achieves better segmentation performance than Qwen3-VL-8B. Considering Qwen3-VL-8B is much stronger than Qwen2.5-VL-7B, the performance promotion gap is questionable.

Questions for authors

The computation cost should be reported and compared.

Reviewer_3LkiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Progression-Aware Event Matching (PAEM) for spatio-temporal video grounding. Entity tubes are pre-extracted with Grounding DINO and SAM2.1. MLLM visual tokens are pooled inside each tube’s box at each frame to form an entity-time lattice, and every (tube, start, end) candidate is scored. The score has two parts. Stage-aware context matching combines a partial-optimal-transport entity relevance score with PRE/EVENT/POST frame probabilities. State-aware process matching aligns the span to M ordered latent states with stay/advance transitions, aggregating over paths with log-mean-exp. The highest-scoring candidate’s tube boxes form the output.

Critical strengths

S1. The latent-state module has a measurable, isolated temporal effect. Removing it costs 6.1/6.8 mtIoU on HCSTVG-v1/v2 (Table 4f), and removing the transitions alone costs 4.3/4.9. This is the paper’s direct evidence that modeling ordered within-event states improves temporal localization, and it is the component whose contribution is cleanly shown.

S2. Consistent gains over matched baselines. Gains appear on four backbones with identical training data, LoRA settings, and proposal sets.

Critical weaknesses

W1. The core components have direct precedents that are either uncited or cited without a stated delta.

  • Candidate selection. Selecting tube × span candidates with per-box feature sequences and vIoU-based supervision was done for HC-STVG by MMN (arXiv:2109.04872). MMN linked per-frame detections into candidate tubes, built a moment map for each tube, and trained with vIoU-based supervision. With an MLLM, STVG-R1 (arXiv:2602.11730, cited) already tracks detected objects with SAM2 and has the model predict the temporal interval together with an object ID. PAEM uses the same benchmarks, most of the same baseline rows, and the same merged HCSTVG + VidSTG training data as STVG-R1. It positions itself only generically, though, and never notes that STVG-R1 already selects among tracked tubes.
  • Stage-aware matching. Scoring span completeness through PRE/EVENT/POST stages closely parallels SSN (arXiv:1704.06228). SSN models each action as starting, course, and ending stages that jointly determine completeness, with a classifier that separates complete instances from background or incomplete proposals. SSN is not cited. Can authors specify the differences from the work?
  • State-aware matching. In PAEM, M latent states where each frame either stays in its state or advances to the next is the topology of the hybrid RNN-HMM (arXiv:1906.01028). That model treats subactions as latent states and, between consecutive frames, allows only the same subaction or the next one. It is not cited. Can authors specify the differences from the work?
  • What is new: transitions conditioned on the query and on frame differences, plus count-normalized marginalization. The latter is worth 0.6/0.5 mvIoU (Table 4i).
  • Why consequential: Contributions 1–2 present stage completeness and latent progression as new criteria. The actual contribution is their combination for MLLM-feature STVG, and that combination needs isolated evidence (W2).

W2. Event progression explains only the temporal gains; most of the mvIoU gain comes from entity selection, and the strongest comparisons are missing.

  • What the ablation shows. Without the state module (Table 4f), temporal localization rests on stage alignment and entity anchoring. Its mtIoU equals the LM baseline’s (52.6 vs. 52.4 on v1; 57.2 vs. 57.9 on v2), even though stage alignment is meant to determine event extent. Yet mvIoU still rises by 5.2/4.2, so 65%/56% of the full gain (8.0/7.5) arrives with no temporal improvement. That pattern points to better entity selection (removing Entity Context Anchoring costs 3.5/3.3 mvIoU, Table 4d), not to event progression. The state module is the only component with a demonstrated temporal effect (−6.1/−6.8 mtIoU when removed).
  • Same-backbone comparisons (Qwen2.5-VL-7B). PAEM leads STVG-R1 by 2.5/1.2 mvIoU (41.6 vs. 39.1; 42.0 vs. 40.8). STVG-o1 (arXiv:2511.21375), cited in Sec. 2.1 but absent from Table 1, reports 44.1 mvIoU on HCSTVG-v1 with Qwen2.5-VL-7B, above all four PAEM variants (best: 43.2). Similarly, Time-R1, which the paper cites as the source of TVGBench, is omitted from Table 3. As reported in STVG-R1, Time-R1 reaches 29.4 [email protected] zero-shot on TVGBench, above PAEM’s 29.0.
  • Why consequential: the paper attributes its gains to event progression. The evidence supports that only for the state module’s temporal effect, and the comparisons that would place PAEM against the strongest prior work are absent.
Questions for authors

Experiments that could help clarify more about the work:

  • A.1.2 states that proposals and identifier mappings are shared with the LM-only baselines. How does the baseline use them: does it output a proposal ID and an interval, or generate boxes? Separately, what does PAEM achieve with neither the stage branch nor the state branch?
  • Please add STVG-o1 and STVG-R1 to Table 1 under matched backbones, and report results on VidSTG, which is used in training but never evaluated.

Clarifications:

  • Tubes and inference:

    • What is Grounding DINO prompted with: the query text, or generic categories?
    • What fraction of test queries have any tube matching the ground-truth target?
    • How are 20 s videos handled at inference, given training uses 24-frame (12 s) windows?
  • On MeViS:

    • How does single-tube selection handle multi-object expressions?

Minor (no response needed):

  • M, the λ weights, ε, ρ, η, τ, and β are not reported.
  • Ablation rows (b) and (i) do not say what replaces the removed component.
  • H_v is query-conditioned under causal attention only if the query precedes the video; please state the input order.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.