acceptodds
Under review as a conference paper at ICLR 2027

What is the Event and When does it Happen?

Abstract

Video Moment Retrieval fundamentally demands mapping a continuous visual stream onto discrete temporal intervals. This distinctive characteristic requires a discriminative latent space capable of determining what event is being described, when it occurs, and which candidate is most plausible. However, existing approaches remain largely optimized through localization-oriented objectives that supervise final temporal predictions, leaving the latent space only indirectly constrained with respect to such discrimination. In this paper, we explicitly decompose multi-dimensional discriminability into a triad of core axes: semantic, temporal, and relational discrimination. Building on this view, we introduce M-R1, a reinforcement learning-based framework that rethinks moment retrieval as the joint resolution of semantic, temporal, and ranking ambiguities. Specifically, we first propose MM-GRPO, which encourages the model to identify modality-specific semantic evidence and selectively regulate its participation in cross-modal interactions. We further devise TA-MOD, which structures temporal information flow within the fused representation via query-conditioned affinity modulation to preserve temporal discriminability. Finally, we present Rank-GRPO, which models inter-candidate relations and optimizes the relative ordering of competing moments. Experimental results on large-scale VMR benchmarks demonstrate significant improvements, with M-R1 achieving average performance gains of 5.4% on multi-moment retrieval and 32.4% on short-moment localization over previous state-of-the-art methods, showing its robustness in challenging scenarios.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Moment Retrieval

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GuCyOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes M‑R1 for video moment retrieval. It decomposes model discriminability into semantic, temporal and relational dimensions. Three components MM‑GRPO, TA‑MOD and Rank‑GRPO are built upon GRPO reinforcement learning. The paper reports experimental gains across several public benchmarks. Its strengths lie in the problem observation about insufficient latent‑space constraints and adapting GRPO to non‑LLM moment retrieval models. It also includes ablation studies and diagnostic analyses. However, several core claims lack solid evidence. Performance improvements are modest. Some technical details and comparisons against prior work are insufficient.

Critical strengths
  1. This paper observes that existing video moment retrieval methods mainly supervise final temporal outputs, while latent discriminative representations are weakly constrained.

  2. Extensive comparisons against existing baselines are conducted on QVHighlights, Charades‑STA, and TACoS. Additional evaluations for short‑moment retrieval are supplemented.

Critical weaknesses
  1. The three‑way decomposition into semantic, temporal, and relational discrimination is subjective in this paper, and there is no solid evidence proving that these dimensions are orthogonal and independent.

  2. Overall performance gains brought by reinforcement learning components are moderate in this paper. Improvements shrink further when strong InternVideo2 features are used. It is hard to attribute gains purely to the proposed discriminative objectives.

  3. MM‑GRPO uses a specific set of counterfactual corruptions. No alternative counterfactual designs are compared. We cannot tell whether current perturbation choices are optimal.

  4. TA‑MOD only modifies temporal affinity weights, and its standalone performance gain in ablation is very small. The actual utility of this module remains questionable.

  5. Related work in this paper fails to draw clear distinctions from prior GRPO‑based video works including Video‑R1 and Time‑R1. Direct technical comparisons are inadequate.

  6. Rank‑GRPO uses stochastic ranking sampling during training while inference stays fully deterministic. There exists train‑inference mismatch. The paper provides no analysis of potential harms caused by this gap.

  7. The method heavily relies on pre‑extracted video features. As acknowledged in the appendix, performance gains degrade under weaker input features. No mitigation solution is presented in this paper.

Questions for authors

See the above weakness

8 comments
AuthorsOpenReview ↗

Response to Reviewer GuCy (Part 3)


W3. Alternative Counterfactual Designs for Video

We apologize for not including explicit comparisons with alternative counterfactual designs in the original manuscript. To examine whether the effectiveness of MM-GRPO comes simply from perturbing temporal information or from how that information is perturbed, we first consider global temporal shuffling as a direct and deliberately strong counterfactual.

Video CounterfactualSetting[email protected][email protected]Avg. mAP
Global ShuffleVideo Only71.8757.5552.77
OursVideo Only73.7458.7753.54
Δ (Global Shuffle − Ours)−1.87−1.22−0.77

Global shuffling results in substantially lower performance than our structured counterfactual. This result reflects an important property of MR, where the model must identify a temporally coherent event within an ordered video rather than merely match the query to unordered visual content. Although global shuffling strongly perturbs temporal information, it also destroys the temporal ordering and local continuity required to localize a coherent event. More importantly, such excessive corruption weakens its utility as a counterfactual in MM-GRPO. A useful counterfactual should remain sufficiently comparable to the original sample so that the difference between them provides a meaningful discriminative learning signal. When the underlying structure is indiscriminately destroyed, the counterfactual can become trivially distinguishable from the original, allowing the objective to rely on coarse corruption cues rather than the intended discriminative relation. This suggests that an effective counterfactual should introduce a meaningful discriminative contrast while preserving sufficient structure for comparison with the original sample.

Based on this observation, we further examine whether the effectiveness of the counterfactual depends on the specific form of perturbation by considering alternative perturbation strategies. We consider three controlled alternatives that affect different aspects of the input. Feature noise perturbs the visual features while leaving the temporal structure unchanged, allowing us to test whether generic feature corruption alone is sufficient. Temporal dropout removes a subset of video clips along the temporal axis, allowing us to examine the effect of partial temporal evidence removal. Context masking instead suppresses the surrounding temporal context, allowing us to examine whether selectively removing contextual information is sufficient.

Video CounterfactualPerturbation CharacteristicSetting[email protected][email protected]Avg. mAP
Feature NoiseNon-temporal feature corruptionVideo Only73.0358.3252.89
Temporal DropoutPartial temporal evidence removalVideo Only73.2358.7152.91
Context MaskingTemporal context removalVideo Only73.2358.7753.52
OursStructured video counterfactualVideo Only73.7458.7753.54

Feature noise that perturbs only the values of visual features without altering the temporal structure of the sequence yields the lowest overall performance among the three alternatives. In contrast, both temporal dropout and context masking, which directly intervene on temporal information, consistently achieve stronger results. This pattern highlights the importance of temporal structure in MR and suggests that perturbations operating on the temporal dimension provide a more informative learning signal than generic feature-level corruption. Among the temporal alternatives, temporal dropout and context masking remain competitive, but both construct the counterfactual primarily by reducing the information available to the model. Such removal can still provide useful supervision, but the resulting difference is partly caused by the loss of information itself rather than by a structured change in temporal relations. In contrast, our counterfactual preserves most of the original visual evidence while modifying its temporal structure, allowing the model to learn from structural differences rather than from missing information.

AuthorsOpenReview ↗

Response to Reviewer GuCy (Part 4)

W3. Alternative Counterfactual Designs for Language and Multimodal Combination

We additionally compare alternative designs for the linguistic counterfactual. MR queries describe an event through multiple linguistic elements, including the entities involved and the action that characterizes what occurs. Prior VMR studies distinguish action-related verbs and object-related nouns as different semantic components of a query [3] and show that verb-related expressions are closely associated with visual dynamics [4]. Motivated by this distinction, we compare our action removal with object removal as a controlled alternative.

Linguistic CounterfactualSetting[email protected][email protected]Avg. mAP
Object RemovalText Only73.1658.2652.81
OursText Only73.6158.5253.60
Δ (Object Removal − Ours)−0.45−0.26−0.79

Action removal consistently yields stronger performance than object removal. This result suggests that the effectiveness of the linguistic counterfactual depends on how directly the perturbed component is related to the event being grounded. Action removal alters cues associated with the event dynamics while preserving the remaining linguistic context, producing a counterfactual that remains close to the original query but changes information relevant to distinguishing the target moment. This provides a more informative discriminative signal for temporal grounding than perturbing object information.

We further examine whether this difference persists when the linguistic counterfactual is combined with our video counterfactual. We keep the video counterfactual fixed and replace only action removal with object removal, allowing us to isolate the effect of the linguistic counterfactual under the same multimodal setting.

Multimodal CounterfactualSetting[email protected][email protected]Avg. mAP
Our Video CF + Object RemovalVideo + Language73.6858.1953.96
OursVideo + Language73.8759.0354.00
Δ (Object Removal − Ours)−0.19−0.84−0.04

When combined with the same video counterfactual, action removal continues to provide stronger performance than object removal. Since the video perturbation is fixed, this comparison further isolates the effect of the linguistic counterfactual design. Notably, the difference is most pronounced in [email protected] in the combined setting, indicating that the choice of linguistic counterfactual remains consequential even after being combined with the video counterfactual.

Overall, the additional experiments consistently favor our counterfactual design across both individual and combined settings. The video comparisons show that neither generic feature corruption, information removal, nor excessive temporal disruption provides a stronger alternative, while the language comparisons consistently favor action removal over object removal. Importantly, the same tendency is maintained when the video and language counterfactuals are combined. While these experiments do not establish global optimality over every possible perturbation, our design achieves the strongest overall performance among the alternatives evaluated, demonstrating that the selected perturbations are not arbitrary but are supported by both the structural properties of MR and controlled empirical comparisons.

Reference

[3] Fast video moment retrieval, J Gao et al., ICCV '21

[4] Towards Generalisable Video Moment Retrieval: Visual-Dynamic Injection to Image-Text Pre-Training, D Luo et al., CVPR '23

AuthorsOpenReview ↗

Response to Reviewer GuCy (Part 5)


W4. Utility of TA-MOD.

TA-MOD is designed to improve fine-grained temporal discrimination by modulating temporal affinities. We isolate its contribution by toggling temporal affinity modulation in the same frozen model, with all other components unchanged.

Setting[email protected][email protected][email protected][email protected]
TA-MOD OFF73.9458.1973.4755.75
TA-MOD ON74.0659.3573.6956.53
Δ+0.12+1.16+0.22+0.78

The larger gains in [email protected] and [email protected] indicate that TA-MOD is especially effective for more precise temporal localization. These metrics require tighter agreement between the predicted and ground-truth moments, supporting the role of TA-MOD in improving fine-grained temporal discrimination.

To understand why TA-MOD leads to these improvements, alongside the CRAM analysis reported in the paper, we conduct a more detailed analysis of the final layer of the temporal encoder. We decompose cross-region temporal attention according to its direction. FG→BG leakage measures the attention assigned by temporal positions inside the GT moment to positions outside the GT moment, while BG→FG leakage measures the reverse interaction.

Final temporal encoder layerTA-MOD OFFTA-MOD ONMean relative reductionQueries reducedRatio
FG→BG leakage0.542400.4825710.97%1,487 / 1,49899.27%
BG→FG leakage0.256520.232399.61%1,445 / 1,49896.46%

These results show that TA-MOD consistently reduces cross-region temporal attention between the GT foreground and background in both directions. Since temporal self-attention determines how each temporal position aggregates contextual information, reducing this cross-region leakage preserves clearer distinctions between the target moment and its surroundings, which is important for distinguishing closely overlapping temporal candidates. Together, the controlled ON/OFF comparison and the final-layer attention analysis clarify the utility of TA-MOD. By suppressing cross-region temporal mixing and preserving fine-grained temporal distinctions, the modulation demonstrates a systematic and functionally meaningful effect on temporal localization.


W5. Distinction from Prior GRPO-based Video Methods.

Policy and Action Space

The fundamental distinction lies in how the policy and its action space are formulated. Video-R1 and Time-R1 are built upon autoregressive MLLMs/LVLMs that model a conditional probability distribution over generated responses, πθ(o∣V,Q)\pi_\theta(o \mid V, Q) This generative formulation provides a stochastic policy from which multiple response sequences can be sampled as GRPO rollouts. In contrast, conventional non-LLM VMR models are discriminative predictors that deterministically map a video-query pair to temporal moment candidates and their corresponding scores or logits, fθ(V,Q)→{(cn,zn)}n=1Nf_\theta(V,Q) \rightarrow \{(c_n,z_n)\}_{n=1}^{N} Unlike autoregressive generative models, these deterministic retrieval models do not inherently define a probability distribution over generated response sequences or provide stochastic rollouts for GRPO. M-R1 therefore explicitly formulates stochastic policies over retrieval decisions by defining temporal-mask selection and candidate ordering as policy actions. This enables the construction of structured retrieval rollouts for group-relative optimization, extending GRPO from autoregressive response generation to discriminative moment retrieval.

Optimization Target and Learning Signal

This difference in policy formulation also changes the optimization target of GRPO. In Video-R1 and Time-R1, group-relative optimization operates on generated response sequences, reinforcing responses according to their correctness, temporal prediction quality, and task-specific rewards. The resulting policy updates therefore reinforce higher-reward responses through group-relative advantages. In contrast, M-R1 performs group-relative optimization directly over structured retrieval decisions. MM-GRPO optimizes temporal-mask selection to encourage semantic discrimination, while Rank-GRPO optimizes candidate ordering to strengthen relative preferences among competing moments. Consequently, the learning signals are defined over how the model identifies relevant temporal evidence and discriminates among retrieval candidates, rather than over the quality of generated responses.

We believe that these distinctions extend beyond differences in reward design or downstream tasks. While Video-R1, Time-R1, and M-R1 share the underlying principle of group-relative policy optimization, M-R1 reformulates the policy, action space, and optimization target around discriminative retrieval decisions. This reformulation further leads to retrieval-specific designs for rollout construction, reward evaluation, and policy learning, including distinct counterfactual and ranking-based objectives. Therefore, M-R1 differs from prior GRPO-based video methods in its overall RL formulation.

Reviewer_y8x3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes M-R1, a reinforcement learning-based framework for video moment retrieval (VMR), aiming to improve semantic, temporal, and relational discrimination. It introduces three components: (1) MM-GRPO, which employs multimodal counterfactual perturbations and stochastic mask optimization to enhance semantic discrimination; (2) TA-MOD, which modulates temporal affinities using query-conditioned relevance to preserve temporal boundaries; and (3) Rank-GRPO, which optimizes candidate rankings through group-relative rewards. The framework adopts a two-stage training strategy combining supervised learning and reinforcement learning. Experiments on QVHighlights, Charades-STA, and TACoS demonstrate competitive performance and improvements over the baseline, particularly in multi-moment retrieval and short-moment localization.

Critical strengths

(1) The design of MM-GRPO is interesting. Instead of relying solely on localization accuracy, the authors introduce both visual temporal perturbations and linguistic action removal to construct counterfactual rewards. By combining the original localization IoU with the performance difference under perturbations, the method encourages the model to make retrieval decisions that are sensitive to event structure and action-related information. (2) The paper effectively adapts GRPO to deterministic VMR models by introducing stochastic policies for temporal mask selection and candidate ranking. This design enables diverse rollouts and group-relative optimization without relying on autoregressive generation, providing a practical way to incorporate reinforcement learning into non-LLM VMR frameworks. (3) The method is evaluated on three widely used VMR benchmarks: QVHighlights, Charades-STA, and TACoS. It achieves competitive results across different settings, with notable improvements in multi-moment retrieval and short-moment localization. The ablation studies further demonstrate the effectiveness of combining the proposed components.

Critical weaknesses

None

Questions for authors

(1) Can the authors provide additional evaluation on a more diverse benchmark such as ActivityNet Captions, or conduct an action-level analysis separating temporally sensitive actions from relatively static events? Such results would help substantiate the claimed improvements in action-semantic discrimination. (2) How do the authors distinguish genuine sensitivity to action semantics from prediction changes caused by temporal permutation itself? Could the authors introduce semantics-preserving temporal perturbations or static-cue-based controls to verify that the counterfactual IoU degradation reflects meaningful semantic discrimination rather than general sensitivity to input rearrangement? (3) Could the authors provide the explicit mathematical construction of the pairwise modulation matrix (D_{ij})? Furthermore, can they analyze the relationship between the original temporal affinity and query-relevance variation to demonstrate why the additional modulation is necessary?

4 comments
AuthorsOpenReview ↗

Response to Reviewer y8x3 (Part 1)

We sincerely appreciate the reviewer's positive evaluation and are pleased that the contributions of M-R1 were well understood.

Q1. Action-Level Analysis of Temporally Sensitive Events

We are also conducting experiments on ActivityNet Captions. However, additional setup is required, and the dataset scale and video lengths make evaluation time-consuming. As these experiments are still ongoing, we first address this question through action-level analysis on QVHighlights and Charades-STA.

To examine whether M-R1 improves retrieval for temporally sensitive actions, we categorize queries by temporal directionality, based on whether reversing the temporal order of the ground-truth event would invalidate its action description. We use spaCy to extract verb lemmas and particles, followed by rule-based classification. Semantically weak verbs (e.g., be, get) and context-dependent verbs are excluded when their meanings cannot be reliably determined. The initial classifications are then cross-checked using Qwen3-2B and manually verified. We define two categories:

  1. DYN (Direction-sensitive): Actions involving directional motion, state transitions, or temporal ordering (e.g., open/close, sit down/stand up, put on/take off).
  2. STA (Relatively direction-insensitive): Persistent states or ongoing activities whose descriptions remain relatively stable under temporal reversal (e.g., holding, wearing, talking, dancing).

Unclassifiable queries are excluded. On QVHighlights validation, we identify 146 DYN and 975 STA queries out of 1,550, with 429 remaining unclassified. On Charades-STA test, we identify 1,543 DYN and 1,121 STA queries. For both benchmarks, we report M-R1 results averaged over three independent runs.

DatasetCategoryMethod[email protected][email protected]mIoUmAP
QVHighlightsDYNQD-DETR55.4843.1553.8439.63
TR-DETR65.0748.6359.5546.08
UVCOM64.3850.0560.0246.78
LA-DETR60.2749.3256.9045.24
R²-Tuning63.0150.6858.7147.79
FlashVTG65.0747.9561.0647.42
Ours68.2651.8363.1850.56
QVHighlightsSTAQD-DETR65.4450.4659.3943.24
TR-DETR68.0053.0362.1645.87
UVCOM66.2653.8561.6046.83
LA-DETR66.0551.4961.1948.95
R²-Tuning69.2353.0362.7948.45
FlashVTG74.3658.5667.8353.80
Ours75.7661.5469.4956.07
Charades-STADYNFlashVTG67.6245.6856.64—
Ours71.5551.2659.92—
Charades-STASTAFlashVTG68.6948.4459.01—
Ours72.7653.8562.18—

The results show that M-R1 consistently outperforms the compared methods across both DYN and STA queries on QVHighlights, with corresponding improvements over FlashVTG on Charades-STA. DYN queries involve actions whose identities depend on temporal direction, where similar visual appearances may correspond to different actions depending on the order of state changes. Retrieving such events therefore requires distinguishing temporally ordered evidence rather than relying solely on static visual cues. On QVHighlights, the improvements over FlashVTG are numerically larger for DYN queries across all reported metrics, suggesting particular benefits for temporally direction-sensitive events. The consistent improvements on DYN queries across both benchmarks provide further evidence of M-R1's effectiveness in retrieving such events, while the gains on STA queries indicate that its benefits also extend to persistent states and ongoing activities.

AuthorsOpenReview ↗

Response to Reviewer y8x3 (Part 2 - Last Part)

Q2. Distinguishing Action Semantics from Temporal Permutation Sensitivity.

We first briefly revisit the motivation and reward design of MM-GRPO. As illustrated in Fig. 2, existing VMR methods retain similar predictions even after action-related query semantics are removed. Motivated by this observation, MM-GRPO aims to reduce such dependence by encouraging retrieval decisions grounded in the temporal structure of the video and the action semantics of the query. Importantly, MM-GRPO does not simply optimize the model to reduce IoU under counterfactual inputs. Rather, as defined in Eq. (4), it evaluates the quality of an original prediction jointly with its ability to distinguish the original event from counterfactual variations. A high IoU on the original input alone is insufficient for a high reward. If the model achieves similarly accurate localization even after the event structure or action-related information is perturbed, the reward is suppressed. Through this mechanism, MM-GRPO encourages the model to rely less on shortcut cues and more on the multimodal semantic evidence that distinguishes the queried event.

We appreciate the reviewer's question regarding whether such counterfactual discrimination reflects genuine sensitivity to action semantics rather than general sensitivity to temporal permutation. To further investigate this distinction, we conduct an additional Temporal Reversal × Action Query Inversion analysis on 49 action-query pairs, each manually verified to represent semantically opposite actions. For each pair, we evaluate the original and inverse action queries (e.g., opening a door and closing a door) using both the original and temporally reversed video feature sequences. For each temporal condition, we compare the relevance scores assigned to the two action queries for the corresponding target moment. We evaluate action-query discrimination using three metrics:

  • OQA (Original Query Preference Accuracy) measures how often the original action query is preferred over its inverse under the original temporal order.

  • RQA (Reversed Query Preference Accuracy) measures how often the inverse action query is preferred over the original query under the reversed temporal order.

  • SFA (Semantic Flip Accuracy) measures how often both conditions are satisfied within the same sample, evaluating whether the preferred action query changes consistently with temporal reversal.

MethodOQA ↑RQA ↑SFA ↑
Baseline61.242.928.6
Ours65.361.234.7

The results show that M-R1 achieves a more pronounced improvement over the baseline under temporal reversal, while its higher SFA indicates greater consistency with the expected action-query preferences across both temporal conditions. Building on the observations in Fig. 2, which reveal that localization predictions may remain unchanged despite changes to event-defining information, our analysis examines whether the relative preference between semantically opposing action queries changes in the expected direction under temporal reversal. The improvements in RQA and SFA provide further evidence that M-R1 better distinguishes semantically opposing action queries under temporal reversal, supporting the discriminative objective of MM-GRPO and demonstrating improved action-query discrimination within the non-LLM VMR setting.

Q3. Mathematical Construction and Analysis of the Pairwise Modulation Matrix.

Temporal self-attention can mix foreground and background features because visually similar clips may receive high affinity despite differing relevance to the query, as illustrated in Fig. 4 of our manuscript. TA-MOD addresses this leakage using query-conditioned FG/BG evidence from the supervised temporal mask head. Given the foreground probability rt=σ(ℓt)r_t=\sigma(\ell_t), we construct DijD_{ij} from the strongest relevance transition between clips ii and jj as Dij=max⁡min⁡(i,j)≤t<max⁡(i,j)∣rt+1−rt∣D_{ij}=\max_{\min(i,j)\leq t<\max(i,j)}|r_{t+1}-r_t| for i≠ji\neq j, with Dii=0D_{ii}=0. We then modulate the original affinity as A~ij=Aij−αDij\tilde A_{ij}=A_{ij}-\alpha D_{ij}. This suppresses attention across strong FG/BG transitions even when the original affinity is high. To verify the effectiveness of TA-MOD, we conduct a controlled ON/OFF comparison with all other model components unchanged.

MetricTA-MOD OFFTA-MOD ONMean relative reduction
FG→BG leakage0.542400.4825710.97%
BG→FG leakage0.256520.232399.61%
MetricTA-MOD OFFTA-MOD ONΔ
[email protected]58.1959.35+1.16
[email protected]55.7556.53+0.78

These results suggest that query-conditioned FG/BG cues complement feature affinity, as visually similar clips may lie across query-relevance boundaries. Modulating affinity across these boundaries reduces cross-region feature mixing, while the localization gains indicate improved temporal precision.

The author addressed most of my concerns. I am keeping my score as is.

We would like to express our sincere gratitude for your thoughtful review and for taking the time to carefully consider our responses and additional experiments. We are particularly grateful for your recognition that we have addressed most of your questions. Your insightful questions encouraged us to conduct further analyses, examine our methodology more carefully, and strengthen the explanations and experimental evidence supporting our work. We truly appreciate the time, effort, and expertise you have devoted to this review process, as your feedback has been valuable in helping us improve both the technical depth and clarity of our research. We fully respect your decision regarding the final evaluation and sincerely appreciate your constructive engagement throughout this process. If you have any further questions, concerns, or suggestions regarding our work, please do not hesitate to reach out to us at any time. We would be more than happy to provide additional clarification or engage in further discussion. Thank you once again for your thoughtful consideration, valuable feedback, and the opportunity to improve our work through your review.

Reviewer_XgLTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes M-R1, a reinforcement-learning framework for non-LLM video moment retrieval. MM-GRPO, TA-MOD, and Rank-GRPO target semantic, temporal, and relational discrimination on top of FlashVTG. Inference remains deterministic. Experiments cover QVHighlights, Charades-STA, and TACoS.

Critical strengths
  1. The full model improves every reported QVHighlights metric over FlashVTG under both feature settings. In Table 3, stage 1 mainly raises [email protected] from 57.29 to 58.35, while adding Rank-GRPO to stage 1 mainly raises Avg. mAP from 52.61 to 54.21.

  2. Table 1 separates SlowFast+CLIP from InternVideo2. Section 4.2 correctly states the InternVideo2 test results: best on [email protected], [email protected], and [email protected], and second-best on [email protected].

Critical weaknesses
  1. The abstract reports gains of 5.4% on multi-moment retrieval and 32.4% on short-moment localization over previous state-of-the-art methods, but it does not define the baseline, metrics, or averaging. None of the relative changes among the three methods in Table 4 equals 32.4%. On Avg. mAP with InternVideo2, SG-DETR† and ME-DETR† are higher than M-R1† on both test and val; SDST† is also higher on val.

  2. Section 4.4 uses the prediction change in Figure 2 as evidence of semantic discrimination. In that example, M-R1’s Self-IoU is 0.000, while the edited query still contains “makeup on face.” The paper states that TA-MOD reduces CRAM for 95.9% of validation queries, but it does not give the average reduction. Stage 1 also lowers Avg. mAP from 52.84 to 52.61.

  3. Stage 1 adds TA-MOD and mask supervision together, and Table 3 has no row that separates them. The paper does not compare the Stage 2 rewards with a supervised loss that uses the same signals.

Questions for authors

Please refer to the Critical Weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer XgLT (Part 1)

Thank you for your thoughtful review and constructive comments. We appreciate your positive assessment of our results and the helpful feedback to further improve the clarity of the paper. Below, we respond to each of your concerns in detail.

W1. Clarification of Performance Gains in the Abstract.

We thank the reviewer for pointing out the ambiguity in the performance claims in our abstract. We apologize for not explicitly specifying the comparison baselines, evaluation metrics, and averaging protocols.

For Multi-Moment retrieval, the originally reported 5.4% improvement was calculated using our reproduced experiments. To reflect recent research progress, we considered methods published from 2025 onward and selected five methods with publicly available implementations, namely MS-DETR, FlashVTG, SDST, SG-DETR, and CG-DETR. We evaluated these methods on the QVHighlights validation set and calculated the relative improvement in Avg. mAP of M-R1 over each method. To avoid potential discrepancies between our reproduced results and those officially reported in the original papers, we recalculate this improvement using the officially reported Avg. mAP values of the five methods. Specifically, we calculate the relative improvement for each method as

Gi=54.58−BiBi×100,G_i = \frac{54.58-B_i}{B_i}\times 100,

where BiB_i denotes the officially reported Avg. mAP of the corresponding method, and 54.58 is the Avg. mAP of M-R1. We then take the arithmetic mean of the five relative improvements, obtaining 4.29%. To ensure a consistent and transparent comparison, we will replace the originally reported 5.4% with 4.29% in the revised abstract.

For Short-moment localization, the originally reported 32.4% improvement was calculated against our directly reproduced FlashVTG baseline, which achieved an Avg. R1 of 9.59, compared with 12.70 for M-R1. However, the original abstract did not explicitly identify this baseline, making the comparison protocol unclear. To provide a more transparent and comprehensive comparison, we additionally evaluate officially released checkpoints of existing methods on the short-moment subset of the QVHighlights validation set. We report Avg. R1 and Avg. mAP, calculated by averaging the corresponding metrics across IoU thresholds from 0.5 to 0.95. The relative improvements are calculated with respect to the corresponding checkpoint-based results.

MethodAvg. R1Avg. mAPR1 GainmAP Gain
QD-DETR5.858.42+117.09%+123.52%
TR-DETR5.579.93+128.01%+89.53%
UVCOM5.9712.68+112.73%+48.42%
R2-Tuning5.7812.60+119.72%+49.37%
FlashVTG9.2315.65+37.59%+20.26%
LA-DETR11.0715.27+14.72%+23.25%
SG-DETR6.7420.29+88.43%-7.24%
SDST12.7018.380.00%+2.39%
M-R112.7018.82--

M-R1 achieves the highest Avg. R1, tied with SDST, and the second-highest Avg. mAP among the evaluated methods. The arithmetic mean of the relative gains across the eight checkpoint-based comparisons is 77.29% in Avg. R1 and 43.69% in Avg. mAP. Notably, M-R1 also outperforms LA-DETR, which specifically targets short-moment retrieval through MomentMix augmentation and a length-aware decoder, achieving relative gains of 14.72% in Avg. R1 and 23.25% in Avg. mAP. These results further demonstrate the competitive performance of M-R1 in short-moment localization.

We acknowledge that the original presentation did not sufficiently distinguish reproduced results from evaluations using officially released checkpoints. We sincerely apologize again for this lack of clarity and for any confusion it may have caused. We will carefully revise Table 4 and the final part of the Abstract to improve the clarity and consistency of the reported results. These clarifications do not change the demonstrated improvements of M-R1 in multi-moment retrieval and short-moment localization, or its competitive performance compared with recent methods. We will also ensure that the comparison sources, baselines, evaluation metrics, and averaging protocols are clearly presented.

AuthorsOpenReview ↗

Response to Reviewer XgLT (Part 2)

W2-1. Semantic Discrimination in Figure 2

The video in Fig. 2 depicts a makeup vlog, including intervals where the woman talks while applying makeup, applies makeup without talking, and talks without applying makeup. The original query specifies the co-occurrence of talking and makeup application, whereas the corrupted query removes the action predicates while retaining the contextual cue "makeup on face." We do not claim that the changed prediction is correct, as no ground-truth annotation exists for the corrupted query. Fig. 2 is intended to illustrate sensitivity to changes in action-related semantics, rather than localization correctness under corruption.

To further substantiate the analyses presented in the main paper and more directly examine whether the observed sensitivity is accompanied by improved action-semantic discrimination, we additionally evaluate the DYN (Direction-sensitive) category, which covers actions involving directional motion, state transitions, and temporal ordering. We identify DYN queries using rule-based filtering of spaCy-extracted verb lemmas and particles, followed by Qwen3-2B cross-checking and manual verification. Generic or context-dependent verbs are excluded to reduce ambiguity. This yields 146 DYN queries on QVHighlights (val) and 1,543 on Charades-STA (test).

DatasetMethod[email protected][email protected]mIoUmAP
QVHighlightsBaseline65.0747.9561.0647.42
M-R168.2651.8363.1850.56
Charades-STABaseline67.6245.6856.64—
M-R171.5551.2659.92—

For a more targeted diagnostic, we further select 49 manually verified action-query pairs from the DYN queries, focusing on clearly distinguishable opposing action semantics. We evaluate three metrics. OQA (Original Query Preference Accuracy) measures how often the original action query is preferred over its inverse under the original temporal order. RQA (Reversed Query Preference Accuracy) measures how often the inverse action query is preferred over the original under reversed temporal order. SFA (Semantic Flip Accuracy) measures the proportion of samples satisfying both conditions.

MethodOQA ↑RQA ↑SFA ↑
Baseline61.242.928.6
M-R165.361.234.7

M-R1 consistently improves DYN retrieval performance across both datasets and achieves higher OQA, RQA, and SFA on the 49-pair diagnostic set. We emphasize that although "makeup on face" remains in the corrupted query, this contextual cue does not preserve the original query's requirement for the co-occurrence of talking and makeup application. Accordingly, the Self-IoU of 0.000 in Fig. 2 illustrates the model's sensitivity to the altered query, rather than establishing localization correctness or semantic understanding on its own. The additional DYN and temporal-reversal results provide complementary quantitative evidence supporting improved action-semantic discrimination.

AuthorsOpenReview ↗

Response to Reviewer XgLT (Part 3)

W2-2. CRAM Reduction and Stage-1 Performance

The previously reported 95.9% refers to the proportion of queries exhibiting reduced CRAM at the first temporal encoder layer, immediately following the cross-modal encoder. To address the reviewer's concern with additional quantitative evidence and examine whether reduced cross-region attention remains observable throughout temporal modeling, we further measure foreground-to-background and background-to-foreground attention leakage at the final temporal encoder layer.

Final Temporal Encoder LayerTA-MOD OFFTA-MOD ONMean Relative ReductionQueries Reduced
FG→BG leakage0.542400.4825710.97%1,487/1,498 (99.27%)
BG→FG leakage0.256520.232399.61%1,445/1,498 (96.46%)

The mean relative reductions are averaged across individual queries. These results indicate that TA-MOD suppresses cross-region attention in both directions, with the effect remaining evident at the final temporal encoder layer. This is consistent with TA-MOD's objective of limiting excessive information mixing between the target moment and its surrounding context, thereby preserving fine-grained temporal distinctions.

Regarding Stage 1, we acknowledge that Avg. mAP decreases slightly from 52.84 to 52.61, despite an improvement in [email protected]. However, preserving temporal distinctions and optimizing overall retrieval performance are related but distinct objectives. Reducing temporal information mixing can improve the localization of event boundaries without necessarily improving the relative scoring and ordering of competing moment candidates. Thus, the Stage-1 results highlight the importance of addressing these complementary aspects of retrieval rather than relying on temporal modulation alone.

To further examine TA-MOD's contribution within the complete framework, we conduct a controlled comparison by enabling and disabling TA-MOD while keeping the remaining components unchanged.

Setting[email protected][email protected][email protected][email protected]Avg. mAP
TA-MOD OFF73.9458.1973.4755.7554.44
TA-MOD ON74.0659.3573.6956.5354.58
Δ+0.12+1.16+0.22+0.78+0.14

The improvements are more pronounced under stricter IoU thresholds, consistent with TA-MOD's role in preserving fine-grained temporal distinctions. The improvement in Avg. mAP further supports its positive contribution within the complete framework. Together, these findings reinforce our motivation to address semantic, temporal, and relational discrimination as complementary objectives. TA-MOD regulates temporal information mixing, while MM-GRPO and Rank-GRPO complement it by optimizing semantic discrimination and inter-candidate ranking, respectively. The results suggest that temporal discrimination alone is insufficient to optimize all retrieval metrics, whereas jointly addressing these complementary dimensions leads to more consistent performance improvements.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Acknowledgment to Everyone Involved in the Review Process

We would like to express our sincere gratitude to all the reviewers, the Area Chair, the Senior Area Chair, and the Program Chair for their time, careful consideration, and thoughtful engagement with our work throughout the review and rebuttal process. We deeply appreciate the effort each reviewer has devoted to examining our manuscript, identifying areas that would benefit from further clarification, and providing detailed and constructive feedback. We recognize that a thorough review requires considerable time and attention, and we are genuinely grateful for the care and expertise reflected in the comments we have received. The questions and concerns raised during this process have been particularly valuable in helping us critically reassess our methodology, strengthen our experimental analysis, and clarify the motivation behind our design choices. In preparing our responses, we have sought not only to address the specific points raised but also to reflect carefully on how our work can be presented with greater precision, transparency, and scientific rigor. We also sincerely apologize for the time it has taken us to provide our responses. We conducted and carefully analyzed a substantial number of additional experiments to address the reviewers' concerns as thoroughly and accurately as possible, which required more time than anticipated. We greatly appreciate your patience and understanding throughout this process, and we are sincerely grateful for the opportunity to improve our work through your thoughtful feedback.

We remain open to any further comments or suggestions from the reviewers and would greatly appreciate the opportunity to address them. We will carefully consider any additional feedback and incorporate the relevant clarifications, experimental findings, and improvements into the final version of our manuscript. We sincerely hope that our responses and additional analyses have helped address the concerns raised, and we remain committed to strengthening the quality, clarity, and rigor of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.