- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes M‑R1 for video moment retrieval. It decomposes model discriminability into semantic, temporal and relational dimensions. Three components MM‑GRPO, TA‑MOD and Rank‑GRPO are built upon GRPO reinforcement learning. The paper reports experimental gains across several public benchmarks. Its strengths lie in the problem observation about insufficient latent‑space constraints and adapting GRPO to non‑LLM moment retrieval models. It also includes ablation studies and diagnostic analyses. However, several core claims lack solid evidence. Performance improvements are modest. Some technical details and comparisons against prior work are insufficient.
Critical strengths
-
This paper observes that existing video moment retrieval methods mainly supervise final temporal outputs, while latent discriminative representations are weakly constrained.
-
Extensive comparisons against existing baselines are conducted on QVHighlights, Charades‑STA, and TACoS. Additional evaluations for short‑moment retrieval are supplemented.
Critical weaknesses
-
The three‑way decomposition into semantic, temporal, and relational discrimination is subjective in this paper, and there is no solid evidence proving that these dimensions are orthogonal and independent.
-
Overall performance gains brought by reinforcement learning components are moderate in this paper. Improvements shrink further when strong InternVideo2 features are used. It is hard to attribute gains purely to the proposed discriminative objectives.
-
MM‑GRPO uses a specific set of counterfactual corruptions. No alternative counterfactual designs are compared. We cannot tell whether current perturbation choices are optimal.
-
TA‑MOD only modifies temporal affinity weights, and its standalone performance gain in ablation is very small. The actual utility of this module remains questionable.
-
Related work in this paper fails to draw clear distinctions from prior GRPO‑based video works including Video‑R1 and Time‑R1. Direct technical comparisons are inadequate.
-
Rank‑GRPO uses stochastic ranking sampling during training while inference stays fully deterministic. There exists train‑inference mismatch. The paper provides no analysis of potential harms caused by this gap.
-
The method heavily relies on pre‑extracted video features. As acknowledged in the appendix, performance gains degrade under weaker input features. No mitigation solution is presented in this paper.
Questions for authors
See the above weakness
8 comments
Response to Reviewer GuCy (Part 4)
W3. Alternative Counterfactual Designs for Language and Multimodal Combination
We additionally compare alternative designs for the linguistic counterfactual. MR queries describe an event through multiple linguistic elements, including the entities involved and the action that characterizes what occurs. Prior VMR studies distinguish action-related verbs and object-related nouns as different semantic components of a query [3] and show that verb-related expressions are closely associated with visual dynamics [4]. Motivated by this distinction, we compare our action removal with object removal as a controlled alternative.
| Linguistic Counterfactual | Setting | [email protected] | [email protected] | Avg. mAP |
|---|---|---|---|---|
| Object Removal | Text Only | 73.16 | 58.26 | 52.81 |
| Ours | Text Only | 73.61 | 58.52 | 53.60 |
| Δ (Object Removal − Ours) | −0.45 | −0.26 | −0.79 |
Action removal consistently yields stronger performance than object removal. This result suggests that the effectiveness of the linguistic counterfactual depends on how directly the perturbed component is related to the event being grounded. Action removal alters cues associated with the event dynamics while preserving the remaining linguistic context, producing a counterfactual that remains close to the original query but changes information relevant to distinguishing the target moment. This provides a more informative discriminative signal for temporal grounding than perturbing object information.
We further examine whether this difference persists when the linguistic counterfactual is combined with our video counterfactual. We keep the video counterfactual fixed and replace only action removal with object removal, allowing us to isolate the effect of the linguistic counterfactual under the same multimodal setting.
| Multimodal Counterfactual | Setting | [email protected] | [email protected] | Avg. mAP |
|---|---|---|---|---|
| Our Video CF + Object Removal | Video + Language | 73.68 | 58.19 | 53.96 |
| Ours | Video + Language | 73.87 | 59.03 | 54.00 |
| Δ (Object Removal − Ours) | −0.19 | −0.84 | −0.04 |
When combined with the same video counterfactual, action removal continues to provide stronger performance than object removal. Since the video perturbation is fixed, this comparison further isolates the effect of the linguistic counterfactual design. Notably, the difference is most pronounced in [email protected] in the combined setting, indicating that the choice of linguistic counterfactual remains consequential even after being combined with the video counterfactual.
Overall, the additional experiments consistently favor our counterfactual design across both individual and combined settings. The video comparisons show that neither generic feature corruption, information removal, nor excessive temporal disruption provides a stronger alternative, while the language comparisons consistently favor action removal over object removal. Importantly, the same tendency is maintained when the video and language counterfactuals are combined. While these experiments do not establish global optimality over every possible perturbation, our design achieves the strongest overall performance among the alternatives evaluated, demonstrating that the selected perturbations are not arbitrary but are supported by both the structural properties of MR and controlled empirical comparisons.
Reference
[3] Fast video moment retrieval, J Gao et al., ICCV '21
[4] Towards Generalisable Video Moment Retrieval: Visual-Dynamic Injection to Image-Text Pre-Training, D Luo et al., CVPR '23
Response to Reviewer GuCy (Part 5)
W4. Utility of TA-MOD.
TA-MOD is designed to improve fine-grained temporal discrimination by modulating temporal affinities. We isolate its contribution by toggling temporal affinity modulation in the same frozen model, with all other components unchanged.
| Setting | [email protected] | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|
| TA-MOD OFF | 73.94 | 58.19 | 73.47 | 55.75 |
| TA-MOD ON | 74.06 | 59.35 | 73.69 | 56.53 |
| Δ | +0.12 | +1.16 | +0.22 | +0.78 |
The larger gains in [email protected] and [email protected] indicate that TA-MOD is especially effective for more precise temporal localization. These metrics require tighter agreement between the predicted and ground-truth moments, supporting the role of TA-MOD in improving fine-grained temporal discrimination.
To understand why TA-MOD leads to these improvements, alongside the CRAM analysis reported in the paper, we conduct a more detailed analysis of the final layer of the temporal encoder. We decompose cross-region temporal attention according to its direction. FG→BG leakage measures the attention assigned by temporal positions inside the GT moment to positions outside the GT moment, while BG→FG leakage measures the reverse interaction.
| Final temporal encoder layer | TA-MOD OFF | TA-MOD ON | Mean relative reduction | Queries reduced | Ratio |
|---|---|---|---|---|---|
| FG→BG leakage | 0.54240 | 0.48257 | 10.97% | 1,487 / 1,498 | 99.27% |
| BG→FG leakage | 0.25652 | 0.23239 | 9.61% | 1,445 / 1,498 | 96.46% |
These results show that TA-MOD consistently reduces cross-region temporal attention between the GT foreground and background in both directions. Since temporal self-attention determines how each temporal position aggregates contextual information, reducing this cross-region leakage preserves clearer distinctions between the target moment and its surroundings, which is important for distinguishing closely overlapping temporal candidates. Together, the controlled ON/OFF comparison and the final-layer attention analysis clarify the utility of TA-MOD. By suppressing cross-region temporal mixing and preserving fine-grained temporal distinctions, the modulation demonstrates a systematic and functionally meaningful effect on temporal localization.
W5. Distinction from Prior GRPO-based Video Methods.
Policy and Action Space
The fundamental distinction lies in how the policy and its action space are formulated. Video-R1 and Time-R1 are built upon autoregressive MLLMs/LVLMs that model a conditional probability distribution over generated responses, This generative formulation provides a stochastic policy from which multiple response sequences can be sampled as GRPO rollouts. In contrast, conventional non-LLM VMR models are discriminative predictors that deterministically map a video-query pair to temporal moment candidates and their corresponding scores or logits, Unlike autoregressive generative models, these deterministic retrieval models do not inherently define a probability distribution over generated response sequences or provide stochastic rollouts for GRPO. M-R1 therefore explicitly formulates stochastic policies over retrieval decisions by defining temporal-mask selection and candidate ordering as policy actions. This enables the construction of structured retrieval rollouts for group-relative optimization, extending GRPO from autoregressive response generation to discriminative moment retrieval.
Optimization Target and Learning Signal
This difference in policy formulation also changes the optimization target of GRPO. In Video-R1 and Time-R1, group-relative optimization operates on generated response sequences, reinforcing responses according to their correctness, temporal prediction quality, and task-specific rewards. The resulting policy updates therefore reinforce higher-reward responses through group-relative advantages. In contrast, M-R1 performs group-relative optimization directly over structured retrieval decisions. MM-GRPO optimizes temporal-mask selection to encourage semantic discrimination, while Rank-GRPO optimizes candidate ordering to strengthen relative preferences among competing moments. Consequently, the learning signals are defined over how the model identifies relevant temporal evidence and discriminates among retrieval candidates, rather than over the quality of generated responses.
We believe that these distinctions extend beyond differences in reward design or downstream tasks. While Video-R1, Time-R1, and M-R1 share the underlying principle of group-relative policy optimization, M-R1 reformulates the policy, action space, and optimization target around discriminative retrieval decisions. This reformulation further leads to retrieval-specific designs for rollout construction, reward evaluation, and policy learning, including distinct counterfactual and ranking-based objectives. Therefore, M-R1 differs from prior GRPO-based video methods in its overall RL formulation.
Response to Reviewer GuCy (Part 3)
W3. Alternative Counterfactual Designs for Video
We apologize for not including explicit comparisons with alternative counterfactual designs in the original manuscript. To examine whether the effectiveness of MM-GRPO comes simply from perturbing temporal information or from how that information is perturbed, we first consider global temporal shuffling as a direct and deliberately strong counterfactual.
Global shuffling results in substantially lower performance than our structured counterfactual. This result reflects an important property of MR, where the model must identify a temporally coherent event within an ordered video rather than merely match the query to unordered visual content. Although global shuffling strongly perturbs temporal information, it also destroys the temporal ordering and local continuity required to localize a coherent event. More importantly, such excessive corruption weakens its utility as a counterfactual in MM-GRPO. A useful counterfactual should remain sufficiently comparable to the original sample so that the difference between them provides a meaningful discriminative learning signal. When the underlying structure is indiscriminately destroyed, the counterfactual can become trivially distinguishable from the original, allowing the objective to rely on coarse corruption cues rather than the intended discriminative relation. This suggests that an effective counterfactual should introduce a meaningful discriminative contrast while preserving sufficient structure for comparison with the original sample.
Based on this observation, we further examine whether the effectiveness of the counterfactual depends on the specific form of perturbation by considering alternative perturbation strategies. We consider three controlled alternatives that affect different aspects of the input. Feature noise perturbs the visual features while leaving the temporal structure unchanged, allowing us to test whether generic feature corruption alone is sufficient. Temporal dropout removes a subset of video clips along the temporal axis, allowing us to examine the effect of partial temporal evidence removal. Context masking instead suppresses the surrounding temporal context, allowing us to examine whether selectively removing contextual information is sufficient.
Feature noise that perturbs only the values of visual features without altering the temporal structure of the sequence yields the lowest overall performance among the three alternatives. In contrast, both temporal dropout and context masking, which directly intervene on temporal information, consistently achieve stronger results. This pattern highlights the importance of temporal structure in MR and suggests that perturbations operating on the temporal dimension provide a more informative learning signal than generic feature-level corruption. Among the temporal alternatives, temporal dropout and context masking remain competitive, but both construct the counterfactual primarily by reducing the information available to the model. Such removal can still provide useful supervision, but the resulting difference is partly caused by the loss of information itself rather than by a structured change in temporal relations. In contrast, our counterfactual preserves most of the original visual evidence while modifying its temporal structure, allowing the model to learn from structural differences rather than from missing information.