- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces AVEmbedBench, a retrieval benchmark of about 275K text-to-video queries for evaluating audio-visual joint retrieval in omni-embedding models. Each query combines a visual constraint and a speech-semantic constraint, and each target video is paired with two hard negatives that match only one modality. The authors build the benchmark with an automated pipeline: filtering for clips with weakly correlated audio and visual content, generating joint queries with an LLM, and mining modality-specific hard negatives. Evaluating six omni-embedding models, they report a consistent bias toward visual similarity and argue it is partly caused by interference from joint audio-visual encoding.
Critical strengths
- The paper addresses a real gap. Existing audio-visual benchmarks evaluate generative models through QA, and this one targets retrieval ranking in embedding models.
- The benchmark is large (275,409 instances over 637,481 clips), and the hard-negative design discourages unimodal shortcuts.
- Provide analysis from the curated benchmark to understand the root cause of modality bias in Omni-embedding models for cross-modality retrieval tasks.
Critical weaknesses
- The writing is difficult to read; the authors overuse bold text in the academic writing, and some acronyms do not have definitions, like AVHBench in the introduction section.
- The data construction pipeline is vague; for example, what are the rule bases applied to filter out data in step 1, and what is the value of cosine similarity used to choose videos which have minimal audio-visual correlation.
- Queries and hard negatives are produced and filtered automatically (an LLM keep/reject label; embedding-based negative mining), and no human verification is reported. It is unclear whether the negatives are truly mismatched in the intended modality or whether some are valid matches.
- The visual-bias claim relies on , but the two negative types are mined asymmetrically, so the gap could partly reflect the benchmark construction. The “interference from joint encoding” explanation rests on one audio-only vs. joint comparison, with no further controls. The benchmark also covers only text-to-video retrieval based on speech semantics, not non-speech audio.
Questions for authors
- Q1: Please specify the rules used for filtering in Stage 1 and the cosine-similarity threshold (or percentile) used to select audio-visual decoupled clips. How sensitive are the benchmark composition and the reported model rankings to this threshold?
- Q2: Could the observed gap between ORv and ORa be an artifact of this construction, for example the difficulty of the negatives or how concrete the visual and speech parts of the queries are, rather than a property of the models? Is there any human validation, or a control such as swapping the negative-mining procedure, that supports attributing the gap to the models?