acceptodds
Withdrawn from ICLR 2027

CORE-CVR: Benchmarking Fine-grained Reference Understanding in Composed Video Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Composed Video Retrieval (CVR) retrieves a target video using a reference video and a modification text as a composed query. We reveal two issues with existing CVR benchmarks. First, they rarely require fine-grained understanding of the reference video. On three widely used benchmarks, 65.0–91.3% of the samples can still be retrieved when the composed query is replaced with a text of at most ten words. Second, they contain shortcuts that allow the target to be retrieved from a single query modality in 69.3–99.0% of the samples. Motivated by these findings, we introduce CORE-CVR, a new CVR benchmark in which both fine-grained reference information and the modification text are required to identify the target. CORE-CVR consists of Static-CVR, which preserves the fine-grained appearance of objects or backgrounds, and Temporal-CVR, which preserves the fine-grained execution of actions. We further introduce Reference Negatives and Modification Negatives, which can be rejected only by using the fine-grained reference information and the modification text, respectively. On CORE-CVR, only 22.1% and 16.6% of the samples are retrieved by short text and a single input, respectively. The best model achieves only 27.7% R@1, leaving substantial room for improvement.

Primary area:
datasets and benchmarks
Keywords:
Composed Video Retrieval, Multimodal Retrieval, Video-Language Retrieval, Video Retrieval, Benchmarking, Dataset Construction, Fine-grained Video Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PGZzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies composed video retrieval (CVR), where a target video is retrieved using a reference video and modification text. Through short-text substitution and single-modality retrieval experiments, the authors show that some existing benchmarks allow high performance without fully using reference-video details or combining both inputs. They introduce CORE-CVR, with 2,500 queries and 9,984 candidate videos, to evaluate the preservation of static appearance and action-execution information. Two types of hard negatives require models to distinguish candidates that satisfy the modification but fail to preserve reference details from those that preserve the details but violate the modification. The main contributions are benchmark diagnosis and dataset construction rather than a new retrieval method.

Critical strengths
  1. The paper identifies an important limitation of existing CVR evaluation. The single-modality and short-text experiments show that high retrieval scores do not necessarily indicate that a model uses both the reference video and the modification text. This provides a clear motivation for a benchmark that tests compositional understanding more directly.

  2. The hard-negative design is well motivated and supported by validation. The two negative types separately test attention to reference-video details and compliance with the modification text, with Table 6 isolating their effects at a fixed gallery size. Checks using independent encoders and human judgments on 600 action triplets further strengthen confidence in the benchmark construction.

Critical weaknesses
  1. The novelty and presentation are limited. The paper identifies shortcomings in existing CVR datasets and constructs a new benchmark from existing videos and automatically generated modification texts, but the dataset is relatively small, no new retrieval method is proposed, and the evaluation relies heavily on custom-defined metrics. Consequently, the contribution reads more like an experimental report than a substantial methodological or benchmark advance. The writing further obscures its value: the introduction is repetitive, “fine-grained action execution” is insufficiently defined, and oracle coverage, retention, and single-model R@1 are presented without a clear distinction between the conclusions each supports.

  2. The experimental evidence does not fully support the benchmark’s central claims, and query imbalance limits its reliability. Short-text substitution does not establish the necessity of fine-grained reference information, because C2T failures may result from description generation or text–video alignment. For example, object-preservation coverage decreases from 26.6% with a ten-word limit to 18.6% without a word limit, complicating the proposed interpretation. More directly, Table 5 reports 29.9% reference-only oracle coverage for Result Change, compared with 31.0% composed-query coverage; only 7.4% of composed-query successes are missed by both unimodal settings. Although acknowledged in Appendix B.2, this challenges the claim that both inputs are necessary across the benchmark and calls for stronger modification negatives or narrower claims. Finally, Temporal-CVR contains 754 action-preservation queries but only 10 action-replacement queries, weakening per-type reliability and making aggregate results disproportionately reflect frequent categories.

Questions for authors
  1. Why are the query types so imbalanced, particularly the inclusion of only 10 action-replacement queries? Would macro-averaged results change the overall conclusions?
  2. Have you tested single-frame inputs or shuffled frame order on Temporal-CVR? These controls would help establish whether the queries require temporal information.
  3. All videos are limited to 1–10-second clips, so the benchmark does not evaluate event localization, long-range temporal dependencies, or multi-event composition in longer videos.
  4. The four source datasets differ substantially in video style, camera viewpoint, action labels, and caption format. For example, SSv2 uses templated descriptions, while the other datasets include egocentric videos or natural-language narrations. The paper does not report the source distribution for each subtask or provide a cross-dataset analysis, so models may exploit dataset-specific cues instead of understanding the details that should be preserved.
  5. The paper does not evaluate any model trained or adapted on CORE-CVR. Could the authors provide an in-domain baseline to clarify whether the low performance reflects the intended difficulty of the benchmark or simply domain shift from existing training datasets?
Reviewer_gjL8OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper diagnoses two critical flaws in existing Composed Video Retrieval (CVR) benchmarks (such as WebVid-CoVR, Dense-WebVid-CoVR, and CoVR-R): most queries can be solved by replacing the reference video with a short text description of 10 words or fewer, or by relying solely on a single modality (unimodal shortcut). To resolve this, the authors formalize two properties—Reference Video Utility and Compositional Necessity—and introduce a new benchmark, CORE-CVR (2,500 queries across 9,984 clips curated from SSv2, HD-EPIC, VISOR, and Ego-Exo4D). CORE-CVR is divided into Static-CVR (preserving fine-grained object or background appearance mined via SAM 3.1 and DINOv3) and Temporal-CVR (preserving fine-grained action execution mined via an attentive pooler trained on V-JEPA 2). Each query is paired with Reference Negatives (R-Negs) and Modification Negatives (M-Negs). Empirical evaluations across multimodal embedding models and specialized CVR methods show that CORE-CVR substantially suppresses shortcuts and poses a tough challenge, with the best model (WeMM-Embedding) achieving only 27.7% Recall at 1.

Critical strengths

Rigorous Diagnostic Framework and Literature Positioning: Extending recent shortcut analyses in Composed Image Retrieval (Attimonelli et al., 2026) to the video domain, the proposed C2T Retention and unimodal oracle coverage metrics cleanly expose how existing benchmarks (WebVid-CoVR, FineCVR-1M, CoVR-R) fail to test true multimodal composition. Unlike EgoCVR (Hummel et al., 2024) which relies on scene-level distractors from the same video, or TF-CoVR (Gupta et al., 2025) which specifies temporal edits via text labels, CORE-CVR enforces fine-grained visual and motion preservation directly from the reference video.

Well-Engineered Hard Negatives and Action Representation Validation: The pairing of R-Negs and M-Negs directly operationalizes Reference Video Utility and Compositional Necessity, and the ablation in Table 6 clearly proves that each negative type suppresses its targeted unimodal shortcut. Furthermore, validating the V-JEPA 2 attentive pooler against 600 human-annotated triplets (achieving 74.5% agreement compared to 80.7% human leave-one-out agreement) provides a solid foundation for automated temporal mining.

Critical weaknesses

Unverified Noise from Blind Modification Text Generation and Deictic Phrasing: The modification texts are generated by an LLM using only video captions and metadata, without direct visual access to the videos. Because source captions in datasets like EPIC-KITCHENS and SSv2 are often sparse, unmentioned visual discrepancies between the reference and target can easily make the generated instruction ambiguous or contradictory. Moreover, forbidding the LLM from naming the preserved action in Temporal-CVR forces unnatural, overly deictic phrasing (referring to actions purely by temporal position), and the paper lacks human evaluation verifying how often humans can actually solve CORE-CVR queries.

Temporal-CVR is heavily skewed toward Action Preservation, and relying on fixed cosine thresholds of DINOv3 and V-JEPA 2 for both positive and negative mining risks baking backbone-specific inductive biases into the ground truth.

Questions for authors

Since the LLM generates modification texts solely from captions without seeing the videos, how often do unannotated visual details create false positives or unsolvable queries?

Why does Action Replacement contain only 10 queries (with 0% model accuracy), and why does Result Change retain such a high Reference-only shortcut (29.9% RQ coverage)?

Reviewer_4jphOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates Composed Video Retrieval (CVR)—the task of retrieving a target video based on a reference video combined with a text-based modification. The authors observe that in some existing benchmarks, target videos can be retrieved using only short text prompts or single-modality inputs, failing to rigorously test a model’s ability to comprehend the details of a reference video or effectively integrate multi-modal information. To address this, the paper introduces CORE-CVR, a dataset comprising 2,500 queries and 9,984 candidate videos. It is divided into two categories: "Static-CVR," which focuses on preserving the appearance of objects or backgrounds, and "Temporal-CVR," which emphasizes the specific details of how actions are performed. The authors also introduce "Reference Negative Samples" (R-Negs) and "Modification Negative Samples" (M-Negs) to increase retrieval difficulty by penalizing models that overlook either the details of the reference video or the conditions specified in the modification text. The benchmark’s characteristics are analyzed through experiments involving short-text substitution, single-modality queries, ablation studies on negative samples, and evaluations of various models. The best-performing model achieved an R@1 of 27.7%, demonstrating that this benchmark poses a significant challenge for existing methods.

Critical strengths
  1. The paper distinguishes between "whether the reference video provides information that cannot be easily substituted" and "whether it is necessary to combine both types of inputs," offering greater explanatory power than simply reporting retrieval accuracy. Comparing short-text substitution, single-modality queries, and combined queries reveals potential "shortcuts" in existing benchmarks, providing valuable insights for the design of future benchmarks.

  2. R-Negs and M-Negs address different input dependencies. Table 6, which presents results from replacing negative samples while keeping the candidate pool size constant, aligns with the authors' design expectations. Furthermore, the authors validate attribute similarity using an encoder that was not involved in the mining process and evaluate action representations through 600 human-judged triplets—demonstrating a commitment to ensuring the reliability of the automated construction pipeline.

Critical weaknesses
  1. There is a lack of direct human quality verification for the final benchmark samples. The human-subject experiments in the appendix assess the alignment between action representations and human similarity judgments, but they do not directly verify the correctness of the final queries, targets, and negative samples. Furthermore, a 74.5% agreement rate regarding action similarity cannot be equated to the annotation accuracy of the final benchmark.

  2. The diagnostic metrics provided are insufficient to substantiate claims that "reference information is irreplaceable" or that "compositional input is essential." Failures in C2T (Caption-to-Video) retrieval could stem from omissions during text generation, length constraints, or inadequate text-video alignment capabilities; the failure of a unimodal model does not necessarily imply that unimodal information is inherently insufficient. While the authors acknowledge some of these limitations in Section 4.2, the claims made in the abstract and conclusion remain overly strong. Notably, in the "one-out-of-three" selection experiment, the 10-word C2T accuracy for CORE-CVR is only 18.7%—falling below the 33.3% chance level. This warrants deeper analysis: the result may reflect a systematic bias in the selection of negative samples, rather than merely a lack of reference details.

  3. Some subtasks fail to fully demonstrate the claimed compositionality, and the sample distribution is imbalanced. As shown in Table 5, for the "Result Change" category, the coverage rate is 29.9% for queries using only the reference video and 31.0% for compositional queries, whereas the "CQ-only" (compositional query-only) subset shows a coverage of only 7.4%. This indicates that the vast majority of samples retrievable via compositional queries can also be retrieved using unimodal (reference-only) queries. Although the authors acknowledge this issue in the appendix, it nonetheless undermines the broader conclusion that the benchmark satisfies the requirement for "compositional necessity." Additionally, the "Action Replacement" subtask contains only 10 queries, which is insufficient for a robust evaluation.

  4. The study does not adequately disentangle performance degradation caused by challenges in fine-grained understanding, domain shift, or model configuration. Existing CVR (Compositional Video Retrieval) models are primarily trained on different datasets—with some methods utilizing only the middle frames of reference videos—whereas this new benchmark incorporates distinct video sources and demands temporal reasoning. Consequently, the observed low accuracy cannot be attributed solely to a deficiency in fine-grained compositional understanding.

  5. There is a clear discrepancy between the threshold for background similarity and the reported mean value. Appendix A.1.3 (page 14) specifies that for background-preserved pairs, the DINOv3 similarity must be at least 0.90; however, Table 4 reports a mean DINOv3 background similarity of 0.88 for CORE-CVR.

  6. The protocol for replacing negative samples from the shared candidate pool is not described in sufficient detail. The paper states that a single video segment can serve multiple queries; yet, Table 6 analyzes the respective roles of R-Negs and M-Negs by replacing them. Since the same segment may play different roles across different queries, it remains unclear whether replacing it also affects the targets or negative samples associated with other queries.

  7. Relying on a caption similarity threshold to "guarantee that the modification criteria are not met" is logically insufficient. In Appendix A.1.4 (page 14), the authors ensure that M-Negs do not satisfy the modification requirements by restricting the similarity between candidate captions and target captions to no more than 0.80.

Questions for authors

See Critical Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.