acceptodds
Under review as a conference paper at ICLR 2027

AVEmbedBench: A Benchmark for Audio-Visual Joint Retrieval in Omni-Embedding Models

Abstract

Recent omni-embedding models support both visual and audio inputs, yet whether they effectively integrate both modalities for retrieval remains unclear. Existing audio-visual benchmarks primarily evaluate generative models through question answering. However, their reliance on textual responses and focus on answer accuracy rather than retrieval rankings limit their applicability to evaluating joint retrieval in embedding models. To address these limitations, we introduce AVEmbedBench, to our knowledge the first benchmark specifically designed to evaluate audio–visual joint retrieval in omni-embedding models. In AVEmbedBench, Each query combines visual and audio-semantic constraints that the target video must jointly satisfy. To discourage unimodal shortcuts, we select clips with weakly correlated visual and audio semantics and pair each target with two hard negatives, one matching only the visual constraint and the other only the audio constraint. To construct the benchmark at scale, we develop an automated pipeline for audio-visual decoupled video filtering, joint query generation, and modality-specific hard negative mining. Evaluations of recent omni-embedding models reveal a consistent visual bias: models favor visual similarity despite audio mismatch, underscoring the gap between multimodal input support and effective joint retrieval, and motivating more balanced audio–visual integration.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_agqJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces AVEmbedBench, a retrieval benchmark of about 275K text-to-video queries for evaluating audio-visual joint retrieval in omni-embedding models. Each query combines a visual constraint and a speech-semantic constraint, and each target video is paired with two hard negatives that match only one modality. The authors build the benchmark with an automated pipeline: filtering for clips with weakly correlated audio and visual content, generating joint queries with an LLM, and mining modality-specific hard negatives. Evaluating six omni-embedding models, they report a consistent bias toward visual similarity and argue it is partly caused by interference from joint audio-visual encoding.

Critical strengths
  • The paper addresses a real gap. Existing audio-visual benchmarks evaluate generative models through QA, and this one targets retrieval ranking in embedding models.
  • The benchmark is large (275,409 instances over 637,481 clips), and the hard-negative design discourages unimodal shortcuts.
  • Provide analysis from the curated benchmark to understand the root cause of modality bias in Omni-embedding models for cross-modality retrieval tasks.
Critical weaknesses
  • The writing is difficult to read; the authors overuse bold text in the academic writing, and some acronyms do not have definitions, like AVHBench in the introduction section.
  • The data construction pipeline is vague; for example, what are the rule bases applied to filter out data in step 1, and what is the value of cosine similarity used to choose videos which have minimal audio-visual correlation.
  • Queries and hard negatives are produced and filtered automatically (an LLM keep/reject label; embedding-based negative mining), and no human verification is reported. It is unclear whether the negatives are truly mismatched in the intended modality or whether some are valid matches.
  • The visual-bias claim relies on ORv>ORaORv > ORa, but the two negative types are mined asymmetrically, so the gap could partly reflect the benchmark construction. The “interference from joint encoding” explanation rests on one audio-only vs. joint comparison, with no further controls. The benchmark also covers only text-to-video retrieval based on speech semantics, not non-speech audio.
Questions for authors
  • Q1: Please specify the rules used for filtering in Stage 1 and the cosine-similarity threshold (or percentile) used to select audio-visual decoupled clips. How sensitive are the benchmark composition and the reported model rankings to this threshold?
  • Q2: Could the observed gap between ORv and ORa be an artifact of this construction, for example the difficulty of the negatives or how concrete the visual and speech parts of the queries are, rather than a property of the models? Is there any human validation, or a control such as swapping the negative-mining procedure, that supports attributing the gap to the models?
Reviewer_t8T2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a retrieval benchmark requiring joint visual and speech-semantic constraints, with 275,409 queries and 637,481 unique clips. Each positive is paired with negatives intended to satisfy only the visual or speech constraint. An automated pipeline filters clips, generates queries, and mines negatives. Six omni-embedding models are evaluated, motivating claims of visual bias and interference with speech semantics during joint encoding.

Critical strengths
  1. The evaluation objective is valuable. Testing whether models satisfy both constraints goes beyond merely supporting multimodal inputs or reporting general video-retrieval averages. Separating within-triplet ranking from full-gallery retrieval is also informative.

  2. The scale and model coverage have potential resource value. The benchmark spans varied scenes and open/commercial models. The p. 8 ablation holds audio queries and the filtered candidate set fixed, providing a reasonable starting point for investigating joint-encoding effects.

Critical weaknesses
  1. Table 2 is internally inconsistent. Under the single-positive definitions (p. 6), metrics using the same query ranks and averaging weights must satisfy (all values in %):MRR ≤ R@1 + (R@5 − R@1)/2 + (R@10 − R@5)/6 + (100 − R@10)/11.Ranks 2–5, 6–10, and ≥11 have maximum reciprocal ranks of 1/2, 1/6, and 1/11. Yet Table 2 (p. 7) reports MRR above this bound: 91.09 > 90.535 (Nemotron, with hard negatives), 61.61 > 60.787 (LCO, with), and 65.84 > 64.688 (LCO, without). Rounding cannot explain these gaps. The authors must clarify query sets, aggregation, and metric definitions, and recompute the metrics from consistent per-query ranks before these results can support the benchmark's conclusions.

  2. Label validity and negative comparability lack independent verification. The automated pipeline reports no independent human audit of video and audio (pp. 4–6), leaving uncertain whether each negative satisfies exactly one constraint. Differences in validity or difficulty between negative types could also produce ORv>ORa, so this result alone does not establish intrinsic visual bias. Stratified human audits and difficulty-matched controls are needed.

  3. Reproducibility and interpretation need clarification. Candidate pools, filtering thresholds, frame/audio sampling, model versions, and additional-positive handling are underspecified (pp. 5–8). Table 4 is consistent with joint-encoding interference but cannot rule out sequence-length, preprocessing, or score-calibration effects. The claim of no AI use in dataset construction (p. 10) conflicts with Qwen-based query generation/filtering (p. 5) and should be corrected.

Questions for authors
  1. Can you provide per-query positive ranks, recompute Table 2, and explain the three upper-bound violations? Please confirm identical queries, candidate sets, rankings, and averaging weights for Recall and MRR.

  2. Can you provide a stratified human audit of video and audio, reporting the fractions of positives satisfying both constraints and of each negative type satisfying exactly one? Does ORv>ORa persist after label verification and difficulty matching?

  3. Please specify candidate pools, filtering thresholds, frame/audio sampling, model versions, and additional-positive handling. How does Table 4 control sequence length, preprocessing, and score calibration? Please also clarify and correct the discrepancy between the p. 10 AI-use statement and Qwen query generation/filtering on p. 5.

Reviewer_pvChOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces AVEmbedBench, a benchmark for text-to-video retrieval in omni-embedding models where each query combines a visual description with a description of the spoken content, and the target video must satisfy both. Clips are drawn from InternVideo2_Vid_Text and filtered for low cosine similarity between their visual caption and speech transcript, computed with Harrier-OSS-27B. Qwen3.6-27B then rewrites the caption and transcript into a joint query and labels each query as keep or reject.

For each kept query, two hard negatives are mined with FAISS indices over caption and transcript embeddings: one that matches the visual part but not the speech, and one that matches the speech but not the visual part. The resulting benchmark has 275,409 queries over 637,481 clips. Six omni-embedding models (five open-source models and Gemini Embedding 2) are evaluated zero-shot with R@K, MRR, a joint ranking accuracy (JRA), and modality-specific outranking rates.

Critical strengths
  1. The benchmark targets a real gap. Existing audio-visual benchmarks evaluate generative models through question answering, and existing embedding benchmarks do not test whether a retrieval embedding combines evidence from both streams. Pairing each query with one negative that matches only the visual part and one that matches only the speech part is a clean and interpretable design, and the JRA and outranking metrics make the failure modes directly readable.

  2. The scale and model coverage are useful. The benchmark is large, includes a commercial model alongside recent open-source omni-embedding models, and the observation that adding the visual stream lowers retrieval for a speech-only query (Table 4) is an interesting result in its own right, if it holds up under the controls discussed below.

Critical weaknesses
  1. Several reported numbers are mathematically inconsistent, and the benchmark itself is not validated.

(a) Internal inconsistencies in Tables 2 and 3.

  • MRR exceeds the maximum value allowed by the reported recalls. Since a positive at rank 2 to 5 contributes at most 1/2, at rank 6 to 10 at most 1/6, and beyond rank 10 at most 1/11, MRR cannot exceed R@1 + (R@5 − R@1)/2 + (R@10 − R@5)/6 + (1 − R@10)/11. For Omni-Embed-Nemotron with hard negatives, this bound is 90.53, but the reported MRR is 91.09. For LCO-Embedding-Omni, the bound is exceeded in both settings (60.79 vs. 61.61 and 64.69 vs. 65.84).
  • For Omni-Embed-Nemotron, removing the hard negatives raises R@1 by 4.14 points. Each query that moves to rank 1 gains at least 1/2 in reciprocal rank, so MRR should rise by at least 2.07, yet it rises by 0.68.
  • In Table 3, 1 − JRA cannot exceed ORv + ORa unless ties occur, but it does for LCO-Embedding-Omni (32.62 vs. 29.02), Jina-Omni (37.21 vs. 34.25), and Gemini Embedding 2 (10.26 vs. 9.56).
  • The text states that Omni-Embed-Nemotron achieves the highest JRA (88.82), while Table 3 lists 89.74 for Gemini Embedding 2. These inconsistencies make the reported numbers hard to trust and suggest the metrics may have been computed on different query sets or with an implementation error.

(b) No human validation. Every stage of the pipeline is automatic: the source captions and transcripts in InternVideo2_Vid_Text are model-generated, the decoupling filter and the negative mining use Harrier-OSS-27B embeddings, and query quality is judged by Qwen3.6-27B. There is no human check of how often the positive actually satisfies both constraints, or of how often a "hard negative" is in fact a valid match. The second point is a real risk, because visual captions such as "a woman stands in a classroom" are generic, and a negative selected for matching the visual part can easily match the speech part as well. Without an estimate of the label noise, a model's failure on a given query cannot be distinguished from an error in the benchmark.

(c) Underspecified evaluation protocol. The paper does not state the size of the candidate gallery for each query. It is unclear whether all 637,481 clips are ranked for every query, whether the negatives of other queries are part of the gallery, and whether the "full candidate pool" used for NSR@1 is the same pool. Given R@1 near 90% for the strongest models, the gallery size is essential for interpreting the results. The filtering threshold in Stage 1, the rule-based filters, and the corpus used for negative mining are also not specified.

  1. The scope is narrower than claimed, and the main finding has an alternative explanation.

(a) "Audio" means speech content only. Both the query construction and the negatives are defined entirely through ASR transcripts, so the benchmark does not cover environmental sounds, music, or other non-speech audio. These are what most audio-visual work, and the hallucination benchmarks the paper contrasts itself with, focus on. The task is closer to joint visual-and-transcript retrieval, and a cascaded baseline (a captioner plus ASR plus a text embedder) would likely perform very well. Such a baseline would help calibrate how hard the benchmark is and whether end-to-end omni-embedding is the right comparison. The title and abstract should reflect this scope.

(b) The visual-bias conclusion is confounded by how the negatives are mined. We pick each audio-negative from the top-5 visually similar clips and each visual-negative from the top-5 speech-similar clips, all in the Harrier embedding space of the source annotations. Nothing ensures that the two negative types are equally close to the query in their matched modality. Visual captions in the source data tend to be short and generic while transcripts are specific, so audio-negatives may simply be much closer to the visual part of the query than visual-negatives are to the speech part. In that case, ORv > ORa would reflect the difficulty of the negatives rather than a property of the models. The paper should report the matched-modality similarity of each negative type (ideally with human judgments) or balance the two types before attributing the asymmetry to visual bias.

(c) The interference analysis in Table 4 lacks the symmetric control. A text query that describes only the speech is naturally a worse match for an embedding that also encodes unrelated visual content, so a drop from audio-only to joint encoding is expected to some degree for any fused embedding. To support the claim that visual information specifically interferes with speech semantics, the paper would need the analogous experiment for the visual clause (visual-only vs. joint embedding). It would also need to show that the drop is larger for audio than for vision.

Questions for authors

Clarifications:

  1. How large is the candidate gallery for each query? Are all 637,481 clips ranked for every query, are the hard negatives of other queries included, and is the "full candidate pool" in NSR@1 the same pool?

  2. Can you explain the inconsistencies noted above between MRR and R@K in Table 2, and between JRA and ORv + ORa in Table 3? Were all metrics computed on the same set of queries, and how are ties handled?

  3. What similarity threshold and rule-based filters are used in Stage 1, and how many clips remain after each step? Are hard negatives mined from the 490K filtered clips or from the full InternVideo2_Vid_Text corpus?

  4. In negative mining, the candidates are ranked by their similarity to the positive video in the other modality. Is any minimum dissimilarity enforced, or is the least similar of the top-5 always selected?

Suggestions that would affect my assessment:

  1. A human-verified subset (e.g., a few hundred queries) reporting how often the positive satisfies both constraints and how often each negative type truly violates one of them.

  2. The distribution of matched-modality similarity for audio-negatives and visual-negatives, or a difficulty-balanced subset, to show that the ORv > ORa asymmetry is not caused by the mining procedure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.