- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
VAD-TTRL adapts video LLMs without anomaly labels, temporal boundaries, or annotated explanations. It forms a consensus over structured outputs covering the label, temporal interval, spatial location, and explanation. Target-drop, target-keep, and off-target interventions then assess visual support, followed by gated GRPO updates with structured rewards.
On Vad-Reasoning, Qwen2.5-VL-7B improves from 0.761 to 0.829 accuracy and from 0.567 to 0.657 mIoU. The post-trained Vad-R1 improves from 0.875 to 0.886 accuracy and from 0.713 to 0.729 mIoU. The evaluation covers six backbones, transfer to VANE-Bench, additional videos excluded from adaptation, and LLM-based explanation assessment.
Critical strengths
- Broad evaluation, including videos excluded from adaptation. Detection accuracy and mIoU improve on all six backbones. On additional sets of 500 and 1,000 videos not used for adaptation, accuracy rises from 0.754 and 0.766 to 0.806 and 0.819, respectively. These results support transfer beyond the adaptation set.
- A quantitative audit of incorrect consensus. The full configuration achieves 89.0% accepted pseudo-label accuracy at 72.7% update coverage. Among 230 proposals with unanimous 8/8 agreement, 23 are still incorrect, directly supporting the motivation that agreement does not ensure correctness. GPT-4o explanation assessment in the appendix complements token-overlap metrics.
Critical weaknesses
-
The gate admits proposals that are insensitive to the target interval.
Equations (4)–(7) on p. 5 average necessity and sufficiency. If the full-video, target-drop, target-keep, and off-target views all have anomaly probability 0.9, necessity is 0, while sufficiency and the control score are both 1. The resulting exceeds the default threshold of 0.20. A proposal satisfying the other conditions can therefore pass visual verification even when removing the target interval has no effect.
The appendix counterbalances answer ordering and includes matched interventions and temporal-support filtering, but these do not exclude this case. Please report the frequency and accuracy of accepted proposals with near-zero necessity. Comparing raw-confidence filtering, the current verifier, and a verifier requiring positive necessity under the same consensus conditions would establish the added value of video interventions. This counterexample identifies a boundary of the gate; its practical frequency remains to be measured.
-
The evidence weight does not necessarily control update strength.
From Appendix E, the reward can be written as , where trajectories from the same video share and denotes format validity. If format rewards are identical within the group, , reward variance is nonzero, and is negligible relative to the reward standard deviation, centering and standardization cancel . The equations therefore do not generally support the interpretation that weaker evidence produces weaker updates.
The weight can still affect gradients when format rewards differ, is not negligible, or an additional weight is applied outside the normalized objective. Hard gating also retains its filtering effect. Moreover, the label term and visual term share the same indicator, so removing the visual term also changes the effective label weight. Please hold proposals and training trajectories fixed, compare normalized advantages and gradient norms, and clarify whether or is applied outside the loss.
-
Performance is highly sensitive to training-group size, and the final configuration needs clarification.
In Table 7, changing from to reduces anomaly recall from 0.745 to 0.118 and F1 from 0.802 to 0.209. With , recall is only 0.049. The proposed explanation, reinforcement of correlated trajectories, needs a controlled analysis of the trajectories and update budget. Please also explain how the default group size, thresholds, and weights are selected without labels.
Several details affect reproducibility: Are the 439 videos reevaluated after the full adaptation pass, or is each video predicted immediately after its update? What are the intervention strength , normal-risk coefficient , temporal-support threshold, and fallback rules? In Table 8, should reject all videos under exact softmax probabilities with finite logits, although numerical saturation to 1 could create exceptions. Since this row differs from the frozen model, accepted-sample counts, parameter changes, and prediction settings are needed. VANE transfer also lacks a direct comparison with the same frozen Qwen backbone, and end-to-end runtime should be reported.
Questions for authors
- How is visual evidence considered verified when identical anomaly probabilities across all intervention views can still pass the gate? What fraction of accepted proposals has near-zero necessity, and how does raw-confidence filtering perform?
- With identical format rewards, , nonzero reward variance, and negligible , GRPO normalization cancels . Is or applied outside the normalized loss? Can normalized advantages and gradients be compared on fixed trajectories?
- What explains the recall collapse at when trajectories and update budget are controlled? What independent validation information or unlabeled criterion determines the default thresholds, weights, and group size?
- How many videos are accepted and updated with in Table 8? Are the prediction procedure and initial model identical to those of the frozen baseline?
- Please clarify evaluation timing, verification parameters, and end-to-end cost, and report the same frozen backbone on VANE. How much improvement comes from the additional generation and verification calls without parameter updates?
8 comments
Response 2/6 : (W2+Q2)
W2 / Q2: The evidence weight does not necessarily control update strength. Is it applied outside the normalized loss, and can advantages and gradients be compared on fixed trajectories?
We thank you for this careful analysis; it is correct. In our implementation, no evidence weight ( or ) is applied outside the normalized GRPO objective, so the cancellation you describe holds whenever the terms that do not depend on the evidence are identical within a group and is negligible.
How verification acts on the update. Verification mainly acts through the hard gate: for a rejected proposal, the loss is multiplied by zero, so that group contributes no GRPO or KL gradient. We also correct one phrase in the paper, that rejected proposals "produce no policy update". Because the run uses AdamW (weight decay 0.01), the optimizer step still runs, so the gate removes the GRPO/KL gradient contribution rather than every weight-decay or optimizer-state change.
Fixed-trajectory comparison. As you suggested, we replayed all accepted groups with the same proposals, the same selected trajectories and the same reward code, and compared the reward computed with the actual visual evidence against the same reward with the evidence set to 1:
| Fixed-trajectory statistic | Result |
|---|---|
| Accepted groups / fixed completions | 319 / 1,276 |
| Consensus reconstruction mismatches | 0 |
| Groups where the evidence-independent terms are identical | 202 / 319 |
| Groups with unchanged reward ranking | 307 / 319 (96.24%) |
| Mean cosine between normalized advantages | 0.9899 |
| Mean coefficient norm, actual evidence | 0.4298 |
| Mean coefficient norm, evidence = 1 | 0.4310 |
| Relative difference | −0.26% |
Here, the coefficient norm is the gradient with respect to the per-sequence log-probabilities, before the shared model Jacobian (not the full parameter-gradient norm). The two norms are almost equal (−0.26%), which confirms your analysis empirically: inside group normalization, the soft evidence weight does not set the magnitude of accepted updates. It can still change the relative ranking within a group when the other terms differ (117 of the 319 groups).
Shared indicator and . We agree that the label and visual terms share the same agreement indicator, so setting (Table 11) also changes the effective weight on agreement, from to . We now describe that ablation as measuring their combined effect.
Revision. The revised PDF adds a paragraph to Appendix E.5 that derives this cancellation, explains how the evidence still acts (through the hard gate; through the balance between task and format rewards when format validity differs; and through , which re-weights the label term and does not cancel), states that no evidence weight is applied outside the normalized objective, and notes the shared indicator. Section 5.1 and Appendix E.3 now describe evidence weighting as shaping the relative contribution of the reward terms rather than the strength of the update. We have also made the notation in Appendices D and E consistent with the main text (e.g., the rationale reward is now written with and throughout). Thus, the hard gate controls whether a group contributes policy gradients, while the soft evidence shapes rewards within the group. This refines our interpretation of the mechanism and does not change the reported run or results.
Response 3/6 : (W3+Q3)
W3 / Q3: Sensitivity to the training-group size, and how the defaults were selected without labels.
We thank you for these direct questions and answer each in turn.
Final configuration. All reported results use , , , , format weight 0.20, KL coefficient 0.04, learning rate and one epoch, for every backbone and dataset.
What explains the recall collapse at when trajectories and update budget are controlled?
We have not yet run the fully controlled comparison, so we do not claim a causal explanation. What we can state is the following:
- The visual gate is applied per video, before the training group is sampled, so and use the same gate criteria and the same single epoch; they differ mainly in how many trajectories enter each update.
- Recall drops whenever , for both (0.118) and (0.049), while all settings with keep much higher recall (0.500–0.745). The effect therefore follows the size of the update group rather than whether the group equals the whole proposal pool.
- The collapse is directional, not random. Settings that remove a core part of the design move in the same direction, with precision rising and anomaly recall falling: (0.923 / 0.118), no visual gate (0.977 / 0.211), (0.930 / 0.392) and (0.944 / 0.417), against the default (0.869 / 0.745).
- This matches our analysis of the accepted proposals, in which 34 of the 35 incorrect pseudo-labels accepted by the verifier were Abnormal videos accepted as Normal: the base model's main error is a bias toward Normal.
Our working hypothesis is therefore that larger update groups admit more trajectories agreeing with a dominant Normal consensus and so amplify this bias. Training with the default group is otherwise well behaved: the KL divergence from the reference stays below 0.018 throughout adaptation (Figure 4), and also remains well above the frozen model. We present this explanation as a hypothesis, recommend a small update group ( in our experiments), and list the missing controlled analysis as a limitation.
What independent validation information or unlabeled criterion determines the defaults?
No labeled validation data and no unlabeled selection criterion were used. Benchmarks that annotate labels, temporal intervals, spatial locations and reasoning together are scarce, and holding out a labeled split would contradict the label-free setting. The defaults were fixed once from what each value controls: gives a diverse proposal pool at moderate cost; keeps forming the consensus separate from reinforcing it; requires a clear majority (at least six of eight) without unanimity; is a minimum support floor rather than a classification threshold; the reward weights follow task priority; and the learning rate and KL coefficient follow the standard GRPO setting of Shao et al. (2024). The sweeps in Tables 7–11 are sensitivity analyses carried out afterwards, not a selection procedure.
As evidence that this configuration is not fitted to one setting, it was reused unchanged on all six backbones in Table 2, improving accuracy and mIoU in every case, and it transfers to the 1,755 unseen videos of the Vad-Reasoning SFT split (Acc 0.765 → 0.817, mIoU 0.559 → 0.648; Appendix C.2). We agree that label-free model selection remains open. Signals available during adaptation, such as the gate acceptance rate or the fraction of Abnormal predictions, are natural candidates, and Appendix H now lists label-free hyperparameter selection as future work.
Response 1/6 : (W1+Q1)
Thank you for this detailed and constructive review, and for recognising the breadth of the evaluation, the transfer beyond the adaptation set, and our quantitative analysis of incorrect consensus. Your comments helped us sharpen several parts of the paper; we respond point by point below. All changes in the revised PDF are highlighted in blue.
W1 / Q1: The gate admits proposals that are insensitive to the target interval. How often does this happen, and how does raw-confidence filtering compare?
How identical probabilities can pass the gate. We agree with your observation. Equations (4)–(7) combine necessity, sufficiency and the off-target control into one score, so a confident proposal whose removal does not lower the Abnormal probability (necessity = 0) can still pass the 0.20 threshold through sufficiency and the control term. The gate therefore does not establish that the interval is strictly necessary; it checks that the interval keeps positive visual support without relying on off-target context. This also explains why no Abnormal proposal reaches an evidence score above 0.5 (Table R2): when necessity is near zero, the averaged term (Δnec+Ssuf)/2 is at most about 0.5.
How often it happens, and how accurate these proposals are. As you suggested, we measured this on the submitted run over the Vad-Reasoning test set (Huang et al., 2025), using ground truth only for this analysis. Of the 97 Abnormal proposals that pass the consensus conditions, the verifier accepts 92. Of these, 48 (52%) have necessity = 0, so the case you describe is common in practice. However, 47 of these 48 are correct: the admitted zero-necessity proposals are almost all genuine anomalies rather than false positives.
Comparison with a positive-necessity verifier and raw-confidence filtering, under the same consensus conditions:
Requiring positive necessity gives a fully correct Abnormal subset, but keeps only 44 of the 97 proposals. Over the whole test set it reduces update coverage from 319 to 271 and lowers pseudo-label accuracy from 89.03% to 87.45%, because it removes 47 correct proposals along with one incorrect one. At matched coverage, the verifier is slightly more precise than raw confidence (91 vs. 90 correct).
Where the interventions add value. On Abnormal proposals the interventions add little over raw confidence, because these proposals are already highly precise after consensus. Their main contribution is on the Normal side: of the 21 proposals rejected by verification, 16 were predicted Normal (14 of them incorrect, 2 correct) and 5 predicted Abnormal (4 incorrect, 1 correct).