- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper argues that high accuracy in Video Temporal Grounding (VTG) does not necessarily imply reliance on relevant visual evidence. The authors propose a fixed-answer audit that compares the ground-truth answer's log-likelihood when the annotated event frames are removed versus when an equal number of other frames are removed. This difference is defined as evidence dependence (ψ), and its change after training is measured by Γ.
The authors show that SFT and IoU-based RL improve VTG accuracy but do not meaningfully increase Γ. They then propose a training objective that explicitly increases evidence dependence and validate it across multiple base models. Additional experiments involving frame removal, query changes, different datasets, and unseen video perturbations demonstrate corresponding changes in model behavior.
Critical strengths
-
A well-controlled evaluation of evidence dependence The fixed-answer audit measures changes in the likelihood of the same answer, reducing ambiguity associated with regenerating predictions after modifying the input. The matched placebo comparison also helps distinguish sensitivity to general video corruption from selective dependence on relevant event frames The finding that existing RL models improve their ability to follow relocated events without increasing Γ is particularly interesting, as it highlights the distinction between locating an event and depending on its visual evidence.
-
Convincing training ablations and behavioral validation Figure 3 and Table 2 show that simply training a model to reduce its confidence when event frames are removed can make it overly sensitive to unrelated frame modifications. The proposed placebo contrast and bound effectively address this issue. Furthermore, the query-crossed experiment in Figure 5(a) shows that the frames a model depends on change when the query refers to a different event. Although this behavior is not consistent across all models, the results support the claim that the learned dependence is related to the query's meaning.
Critical weaknesses
-
The practical benefit of further increasing Γ and its relationship to actual predictions remain unclear. Table 19 shows that existing models already experience substantial accuracy drops under evidence masking, while their performance remains largely unchanged under placebo masking. For example, TimeZero achieves 70.0% accuracy on clean videos, 21.8% under evidence masking, and 70.9% under placebo masking. This suggests that existing models already make substantial use of relevant event frames. While the authors show that Γ does not increase under accuracy-oriented training, the additional benefit of increasing Γ in models that already use visual evidence remains unclear. The proposed method changes model behavior when evidence is removed, but evidence of improved localization accuracy or generalization is limited. For example, on QVHighlights, the proposed Qwen-based model (α=0.5) increases Γ but improves accuracy by only 0.1 percentage points, compared with 6.8 points for SFT (Table 17). Additionally, ψ measures the likelihood of a single annotated answer, whereas actual predictions also depend on the probabilities of alternative intervals. Analyzing the relative likelihoods of the ground-truth answer and competing predictions would help clarify how ψ relates to evidence use in actual temporal localization decisions.
-
Insufficient discussion of related work and the distinction from existing approaches. The paper shows that evidence-dependence training can lead models to abstain when relevant evidence is missing. However, related work on identifying and rejecting irrelevant queries, including RaTSG (NeurIPS 2024), NA-VMR (WACV 2025), and RA-RFT (CVPR 2026), is not sufficiently discussed. Admittedly, this paper focuses on removing event frames, whereas these studies mainly address queries that do not match the video. Nevertheless, the mismatched-query evaluation in Table 21 directly connects to this line of research. It would therefore be valuable to clarify whether training models to reject irrelevant queries also increases evidence dependence, and what distinguishes the proposed approach from these methods.
Questions for authors
Q1. What additional benefits does increasing Γ provide? Existing models already show substantial reliance on relevant event frames (Table 19). How does further increasing Γ improve VTG performance or prediction reliability, particularly when compared with models at similar accuracy or answer coverage?
Q2. Can the relationship between ψ and actual predictions be further examined? Could the authors compare the relative likelihoods of the ground-truth interval and competing predicted intervals under evidence and placebo masking, keeping candidate answers fixed across conditions? This would help clarify how well ψ reflects evidence dependence in actual temporal localization decisions.
Q3. Does refusal-aware training also increase evidence dependence? Please clarify the relationship to RaTSG, NA-VMR, and RA-RFT. If feasible, applying the same audit to a model trained with RA-RFT would help determine whether refusal-aware training also increases Γ and what distinguishes the proposed approach.
Q4. Do the main audit results generalize to other evaluation samples? The main audit uses the first 200 test questions, on which the base model achieves higher accuracy than on the full test set. Does the conclusion that accuracy-oriented training does not increase Γ hold on other randomly sampled subsets?
6 comments
Reply to Reviewer 3 (2/2): refusal-aware training; other samples; presentation
W2 and Q3. RaTSG, NA-VMR and RA-RFT
Thank you for these references. The revision adds them to the related work. Of the two we could audit, RA-RFT leaves Γ unchanged and the released NA-VMR model raises it, so refusal-aware training need not raise Γ. The three methods train on queries that do not match the video and learn to reject them. Our queries always match the video. We change the video, compare the likelihood of the same answer when the event frames or as many other frames are removed, and our training uses no mismatched query and no rejection label.
RA-RFT is a generative MLLM like ours, so we audited it and its base Time-R1 with the same protocol. Its Γ is −0.003 [−0.022, +0.016] on 600 new questions. When every model is forced to give an interval, the evidence-minus-placebo difference in [email protected] is the same as its base's (difference 0.0 [−1.7, +1.7] points), while RA-RFT gives no interval for 84% of mismatched queries under its released prompt (Time-R1 under the same prompt: 72%; this counts any answer without an interval, unlike the verbatim refusals of Table 21). IoU-GRPO with our objective shows the reverse, with Γ +0.88 and no interval for 5.7% of mismatched queries (its base: 29%). Rejection and Γ can therefore come apart.
NA-VMR is a clip-feature model, so we ported the audit to feature space (replacing clip features with those of another video) and read the released UniVTG and UniVTG-NA checkpoints. There the negative-aware model does depend more on the event clips. Its Γ, computed on the mean log foreground probability of the event clips, is +0.55 [+0.47, +0.63] on the 1,883 questions both checkpoints answer correctly (+0.35 over the 3,625 questions whose event spans at most half the video), and its rejection head fires 27.8 points more often when the event clips are replaced than when other clips are (15.4 over all questions). The two released checkpoints also differ in learning rate and starting point, so we attribute this to the released model and make no claim about negative training alone. RaTSG is also feature-based. Trained on queries paired with other videos, it scores whether a query is unrelated to the video, next to a foreground score for each clip, and declines when that score is high. The score asks whether the queried event is in the video at all, and Γ asks how much the answer to a matching query rests on the event's frames, so a model can be strong on one and weak on the other, as RA-RFT is. The audits appear in a new appendix section.
Q4. Other samples
Reviewer 1 asked the same question, and the full table is in our reply to them (part 2/2). We drew 600 new questions at random from test videos outside the 79 of the first 200. They cover 484 videos, and the base reaches [email protected] 0.507 on them, as on the full split on this stack. The conclusion that accuracy-oriented training does not raise Γ holds on them. SFT gives −0.07 [−0.10, −0.04] and mDPO-style −0.01. IoU-GRPO gives +0.04 and −0.07 at 2 fps (two seeds) and 0.00 with the paper's 16 frames, the public TimeZero and Time-R1 checkpoints give −0.02 and +0.05, and the public TimeLens-8B gives −0.14 against its Qwen3-VL-8B base. Our objective gives +1.50 and +1.49 for two seeds, and +1.89 and +0.91 with α = 0.25 and 1.0. On TimeLens-8B it gives +1.42. On three disjoint random subsets of 200 of these questions, SFT gives −0.06, −0.08 and −0.08 and our model +1.45, +1.43 and +1.64.
Presentation
In the revision, Section 2 carries one question through ψ, the placebo and Γ as a running example. The notation table moves forward and keeps only the symbols the main text uses. Every abbreviation and unit (coverage, resolution, 16 frames and 2 fps, nats per token and points) is defined where it first appears, and each table caption names its sample and units. The training objective is stated in words before its equation.
If the RA-RFT and NA-VMR audits and the new sample address your concerns, we hope you will take them into account in your assessment. We are glad to run more checks during the discussion.
Reply to Reviewer 3, round 2: where your points are in the revised PDF
The revised PDF and the anonymous release are now up (see the general comment). Numbers below refer to the revised PDF.
- W1 and Q1: a higher Γ is not a gain in accuracy, and the last paragraph of Section 4 now states the accuracy cost, including on QVHighlights. What it adds is that more of a model's correct answers rest on the event. In Table 38 (F.17), IoU-GRPO with our objective matches IoU-GRPO's accuracy gain, and its evidence share, the part of the placebo-masked accuracy lost to the evidence mask, rises over the base by 12.8 and 15.8 points for two seeds. Accuracy training alone changes it by +3.4 [−1.8, 8.7] for IoU-GRPO, an interval that includes zero, and by −7.5 [−11.9, −2.9] for the SFT checkpoint. The end of F.17 covers TimeZero, a base that already uses the event. There our objective leaves accuracy where it was (+0.5 [−0.7, 1.5]), raises Γ to +1.40 (Table 3) and lowers the rate of answering an all-black video from 0.968 to 0.303 (Table 10).
- Q2: F.16 and Table 37. At the same accuracy gain, IoU-GRPO with our objective lies above IoU-GRPO alone against both fixed competitors, clearly against the shifted interval (+10.70 against +1.87) and with overlapping intervals against the base's own prediction (+1.42 against +0.09).
- W2 and Q3: a paragraph in Section 6, and the audits of RA-RFT in F.14 (Table 35) and NA-VMR in F.15 (Table 36). RA-RFT gives no interval for 84% of mismatched queries against 72% for Time-R1, and also for more of the questions' own queries (19.8% against 11.3%), while its Γ stays at zero, so rejection training and evidence dependence come apart. F.15 adds the mask operator. On the questions both checkpoints answer correctly, the set on which Γ is defined, the released NA-VMR model depends more on the event clips under both edits (+0.55 under swap, +0.28 under the mask); over all audited questions it does so under swap only.
- Q4: F.3 and Table 24, 600 questions from videos outside the first 200, with three disjoint subsets. Section 2 points to them through Appendix F.
- Presentation: a running example and a notation table in Section 2, the objective in words before its equation in Section 4, and samples and units in the captions of Tables 1–3 and Figures 1, 2 and 4.
If anything is still unclear, please let us know.
Reply to Reviewer 3 (1/2): what a higher Γ adds; relative likelihood against competing intervals
Thank you for the review and the concrete suggestions. We ran the RA-RFT audit you suggested and also audited NA-VMR (part 2/2). All comparisons are paired within one stack, as in the general response.
W1 and Q1. What a higher Γ adds when models already use the event
We agree that base models use the event frames, and a higher Γ does not add accuracy by itself. The paper measures the change Γ, and the base already has ψ > 0 (p3 l.139–140). Your TimeZero example fits this reading. Masking the event removes about the same share of the accuracy TimeZero keeps under the placebo mask as for its Qwen2.5-VL base (69% for the base, difference +1.3 [−4.2, +6.9] points, Table 1), although TimeZero is about 20 points more accurate. Two comparisons at matched accuracy or coverage show what a higher Γ adds.
The first is on TimeZero, a model that already uses the event. On Charades-STA our objective leaves TimeZero's accuracy (+0.5 [−0.7, +1.5]) and coverage (0.999) unchanged, raises Γ to +1.40 (Table 3) and the share of correct answers lost to the evidence mask by 10.8 points, and lowers the rate of answering a fully black video from 0.968 to 0.303 (Tables 7, 10, 19). On QVHighlights it costs 3.8 points (Table 17). Added to IoU-GRPO and trained from TimeZero, it gives +1.0 [+0.0, +2.0] points of accuracy with Γ +1.04 [+0.96, +1.12], and masking the event lowers its [email protected] by 4 [1, 7] points more than the placebo mask does, relative to TimeZero (first 1,000 questions).
The second compares pairs of models whose accuracy gains lie within about one point of each other. "Correct under the mask" is, among questions that both the model and its base answer correctly, the share the model still answers correctly after the event frames are masked. Both behavioural columns use the first 1,000 test questions.
At equal accuracy gain, the model with the higher Γ returns fewer correct answers once the event is removed, so more of its correct answers come from the event frames. Accuracy training alone moves this share the other way. Rejection of mismatched queries rises against the SFT checkpoint but not against IoU-GRPO, so we do not attribute it to Γ.
On the questions that a constant window fitted to the training labels cannot answer, our models keep Γ of +1.35 to +2.71 and SFT stays at +0.07, and our accuracy gain comes from these questions (reply to Reviewer 2, W1).
On QVHighlights your numbers are right. Our objective alone does not raise accuracy there, and the revision says so in the main text. In the re-run it gains +0.4 [−1.4, +2.3] points against +2.0 for SFT, where Table 17 gives +0.1 and +6.8 (base 0.319 in the paper, 0.407 in the re-run). Added to IoU-GRPO it keeps the QVHighlights gain (+11.2 against +12.2) and raises Γ there (+0.61 against +0.09).
Q2. Relative likelihood against fixed competing intervals
We added this readout, and its result is mixed. For each question the candidate texts are fixed across conditions. We measure how much more the annotated interval's log-likelihood advantage over a competitor shrinks under the evidence mask than under the placebo mask, relative to the base (first 200 questions). The values are summed over answer tokens, so they are not on the per-token scale of Γ.
Against the base model's own prediction, the kind of competing prediction you suggested, the separation is small. Our 4,800-pair model is above SFT on the same base (+2.42 against +0.75), though their intervals overlap slightly, and the interval of our α = 0.5 model overlaps SFT's. Against the shifted interval our models separate clearly, but IoU-GRPO alone also reaches +1.87, so this row partly reflects accuracy training. We report both rows in the revision.