- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how negative construction and judgment interfaces affect video-verifier evaluation. It introduces endpoint-matched trajectory negatives and paired truth-switch videos, then examines direct and CoT judgments on controlled tasks and natural videos. The results show that verifier performance and configuration selection depend on the negative type and interface used.
Critical strengths
The evaluation problem is relevant, and the motivation, contributions, and experiments are clearly connected.
The truth-switch design fixes candidate text while changing the correct answer through video evidence.
Candidate-order controls, evidence ablations, and statistical uncertainty analyses strengthen the reported observations.
Critical weaknesses
Internal-trajectory judgments combine spatial grounding, object correspondence, temporal ordering, and state updating; their contributions are insufficiently disentangled.
Direct–CoT comparisons conflate reasoning with prompting, output budgets, parsing, and response biases.
Natural-video negatives lack sufficiently documented independent validation of ambiguity and sampled-evidence visibility.
The broad capability framing exceeds the diagnostic depth; verifier-selection implications remain limited to offline comparisons.
The presentation is conceptually dense, while some related-work distinctions and operational definitions could be sharper.
Endpoint matching removes endpoint shortcuts but does not isolate temporal verification. Internal-trajectory judgments jointly require spatial localization, object correspondence, event ordering, state updating, and text–video alignment. Without controls separating these demands, the performance drop cannot be attributed to a specific verification deficit, limiting the interpretation of the capability analysis.
Questions for authors
Could you add an independent human validation study for the natural-video comparisons? Annotators should verify candidate truth values, identify the contradictory evidence, and assess whether it is visible in the sampled frames. Agreement and ambiguity rates would help establish evaluation validity, while controlled spatial, temporal, and semantic probes are still needed to localize model failures.
Could you introduce staged controls using verified object locations, event sequences, and state trajectories to distinguish perceptual, spatial, temporal, and reasoning failures? The text-record test bypasses several intermediate stages.
The revised pdf is uploaded; promised results included.
The revised PDF is up, and the tables promised above are in it: Section 5.1 and Appendix H.1 for the validation study, Section 4.3 and Appendix H.4 for the staged controls (Table 51 takes each demand you listed in turn), and Section 2 and Table 70 for the definitions. Two statements in our reply need a qualifier.