- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies composed video retrieval (CVR), where a target video is retrieved using a reference video and modification text. Through short-text substitution and single-modality retrieval experiments, the authors show that some existing benchmarks allow high performance without fully using reference-video details or combining both inputs. They introduce CORE-CVR, with 2,500 queries and 9,984 candidate videos, to evaluate the preservation of static appearance and action-execution information. Two types of hard negatives require models to distinguish candidates that satisfy the modification but fail to preserve reference details from those that preserve the details but violate the modification. The main contributions are benchmark diagnosis and dataset construction rather than a new retrieval method.
Critical strengths
-
The paper identifies an important limitation of existing CVR evaluation. The single-modality and short-text experiments show that high retrieval scores do not necessarily indicate that a model uses both the reference video and the modification text. This provides a clear motivation for a benchmark that tests compositional understanding more directly.
-
The hard-negative design is well motivated and supported by validation. The two negative types separately test attention to reference-video details and compliance with the modification text, with Table 6 isolating their effects at a fixed gallery size. Checks using independent encoders and human judgments on 600 action triplets further strengthen confidence in the benchmark construction.
Critical weaknesses
-
The novelty and presentation are limited. The paper identifies shortcomings in existing CVR datasets and constructs a new benchmark from existing videos and automatically generated modification texts, but the dataset is relatively small, no new retrieval method is proposed, and the evaluation relies heavily on custom-defined metrics. Consequently, the contribution reads more like an experimental report than a substantial methodological or benchmark advance. The writing further obscures its value: the introduction is repetitive, “fine-grained action execution” is insufficiently defined, and oracle coverage, retention, and single-model R@1 are presented without a clear distinction between the conclusions each supports.
-
The experimental evidence does not fully support the benchmark’s central claims, and query imbalance limits its reliability. Short-text substitution does not establish the necessity of fine-grained reference information, because C2T failures may result from description generation or text–video alignment. For example, object-preservation coverage decreases from 26.6% with a ten-word limit to 18.6% without a word limit, complicating the proposed interpretation. More directly, Table 5 reports 29.9% reference-only oracle coverage for Result Change, compared with 31.0% composed-query coverage; only 7.4% of composed-query successes are missed by both unimodal settings. Although acknowledged in Appendix B.2, this challenges the claim that both inputs are necessary across the benchmark and calls for stronger modification negatives or narrower claims. Finally, Temporal-CVR contains 754 action-preservation queries but only 10 action-replacement queries, weakening per-type reliability and making aggregate results disproportionately reflect frequent categories.
Questions for authors
- Why are the query types so imbalanced, particularly the inclusion of only 10 action-replacement queries? Would macro-averaged results change the overall conclusions?
- Have you tested single-frame inputs or shuffled frame order on Temporal-CVR? These controls would help establish whether the queries require temporal information.
- All videos are limited to 1–10-second clips, so the benchmark does not evaluate event localization, long-range temporal dependencies, or multi-event composition in longer videos.
- The four source datasets differ substantially in video style, camera viewpoint, action labels, and caption format. For example, SSv2 uses templated descriptions, while the other datasets include egocentric videos or natural-language narrations. The paper does not report the source distribution for each subtask or provide a cross-dataset analysis, so models may exploit dataset-specific cues instead of understanding the details that should be preserved.
- The paper does not evaluate any model trained or adapted on CORE-CVR. Could the authors provide an in-domain baseline to clarify whether the low performance reflects the intended difficulty of the benchmark or simply domain shift from existing training datasets?