acceptodds
Desk rejected at ICLR 2027

OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

With the rapid development of multimodal large language models (MLLMs), evaluating their video understanding capabilities has become increasingly important. Existing video understanding benchmarks cover only part of the evaluation dimensions, limiting systematic diagnosis of omni-modal video understanding. We introduce OmniCompass, a holistic benchmark for omni-modal video grounding and understanding. OmniCompass defines two evaluation tracks: OmniCompass-VL assesses vision-language models on tasks requiring visual evidence, while OmniCompass-AV assesses omni-modal models on tasks requiring joint audio-visual evidence. Each track includes three complementary tasks: temporal grounding (TG), spatial grounding (SG), and question answering (QA), assessing when a referenced target appears, where it is, and what happens. QA covers five perception and three reasoning categories in OmniCompass-VL, as well as three perception and two reasoning categories in OmniCompass-AV. OmniCompass-VL contains 820 videos and 2,460 instances across various domains and three duration types: short, medium, and long. A narrative- and audio-rich subset forms OmniCompass-AV with 240 videos and 720 instances. Each instance is evaluated under two paired settings: text-only and text+image, specifying the same target through language alone and visual-language guidance. All instances are manually annotated with multi-round quality verification. Evaluations of 50 MLLMs show that open-source models still lag behind proprietary counterparts overall, particularly in audio-visual understanding. Precise temporal and spatial grounding remains challenging even for leading models, and their predictions still exhibit inconsistencies across the paired target specifications.

Primary area:
datasets and benchmarks
Keywords:
Video Benchmark, Omni-modal Video Understanding, Vision-language Models, Video Groudning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_zr1nOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces OmniCompass, a benchmark combining temporal grounding, spatial grounding, and question answering under paired textual and visual target specifications. The visual track contains 820 videos and 2,460 task instances, while an audio-visual track uses a 240-video subset with 720 separately constructed instances. Cross-Specification Consistency scores reward correctness across both specifications and, for grounding, agreement between predictions. Evaluation of 50 models, together with frame-budget, timestamp, modality, and reference-type analyses, identifies substantial grounding limitations that aggregate QA accuracy alone would obscure.

Critical strengths
  1. The paired design tests a useful form of robustness: whether the same target and answer remain recoverable when the identifying cue changes. Appendix B.3 provides a concrete example: Kimi-K3 has nearly identical QA accuracies under the two specifications, 73.0% and 72.9%, but only 67.6% joint correctness. This makes the benchmark informative beyond a conventional single-prompt leaderboard.

  2. The benchmark combines substantial manual verification with task-specific analysis. The annotation and repair records in Section 3.2 and Tables D1–D2 support the care taken in construction. The audio ablation in Figure 6 is particularly useful: adding audio improves paired QA scores by 18.6–28.1 percentage points across the four tested models, providing empirical support for the audio component rather than relying solely on the task taxonomy.

Critical weaknesses
  1. The evaluation denominator is not specified sufficiently clearly. Appendix A uses model-specific or shared “eligible pairs,” and Figure A3 reports shared subsets of 592 TG, 673 SG, and 689 QA pairs, compared with 820 nominal instances per task. Appendix B.3 reports 808 QA pairs for Kimi-K3. The supplied evaluation protocol does not clearly explain these exclusions or the handling of failed and unparseable outputs. This matters because model-dependent exclusions could affect both rankings and paired consistency scores; the paper needs to distinguish operational coverage from accuracy on successful responses.

  2. The evidence supports robustness across two specifications more directly than it isolates a distinct consistency failure. CS combines correctness and agreement, and Appendix B.3 appropriately notes that the extra prediction-overlap constraint accounts for only 0.3–1.8 points in grounding. In addition, replacing a textual description with an image changes the identifying information, and the experiments use default model settings without a reported same-specification repeatability control. Consequently, the observed differences cannot yet be cleanly separated into specification sensitivity, unequal cue informativeness, and ordinary response variability. This limits the diagnostic interpretation, although it does not negate the benchmark's utility.

Questions for authors
  1. Clarification: What exactly makes a pair eligible, how many pairs enter each main-table score, and how are API failures and malformed answers scored? Would counting all unsuccessful outputs against a fixed denominator change the main rankings?

  2. Clarification: Do the existing runs contain repeated predictions under an unchanged specification? If so, how does that variability compare with the Text versus Text+Image differences? If not, can the consistency claims be scoped explicitly to the observed paired-setting robustness?

5 comments
AuthorsOpenReview ↗

Response to Reviewer zr1n (W1 Part 1: Protocol & VL Denominators)

Thank you for your recognition of our work—specifically our paired target-specification design that reveals instance-level robustness beyond single-prompt leaderboards, and our substantial manual verification alongside empirical audio-ablation analyses—and for raising constructive questions. Your feedback has helped us improve the transparency and precision of our paper. Below, we provide point-by-point responses to your comments.


Response to Weakness 1: Evaluation Denominators & Operational Coverage

Thank you for this important suggestion. We have audited all evaluation logs and explicitly separated operational coverage (service availability) from conditional accuracy on valid responses.

Nominal task sizes are 820 per VL task and 240 per AV task. We strictly distinguish model capability failures from external API/network interruptions: whenever an API returns a response—even if empty, malformed, or truncated—it remains in the denominator and receives 0 (penalizing instruction-following and formatting failures). Only platform safety-filter refusals or network/transport timeouts are excluded from the single-setting conditional denominator, and an item enters the paired CS\text{CS} denominator when both Text and Text+Image receive valid responses:

Response OutcomeSingle-Setting Denom.Paired CS\text{CS} Denom.Fixed Denom. (820 / 240)
Completed & parseableIncluded (scored)Included if both validIncluded (scored)
Completed, empty / malformed / truncatedIncluded (0)Included (0 on that side)Included (0)
Platform safety-filter refusalExcluded (tracked separately)ExcludedIncluded (0)
Network / transport timeoutExcluded (tracked separately)ExcludedIncluded (0)

OmniCompass-VL Actual Denominators (Nominal = 820; Format: T / TI / CS)

ModelTG (T / TI / CS)SG (T / TI / CS)QA (T / TI / CS)
Gemini-3.8-Flash819 / 819 / 819819 / 818 / 818819 / 819 / 819
Gemini-3.1-Pro819 / 819 / 819819 / 818 / 818819 / 819 / 819
Seed-2.1-Pro-260915820 / 820 / 820820 / 820 / 820820 / 820 / 820
Seed-2.1-Turbo-260628820 / 820 / 820820 / 820 / 820820 / 820 / 820
Seed-2.0-Pro-260215820 / 820 / 820820 / 820 / 820820 / 820 / 820
Qwen3.8-Max808 / 808 / 808808 / 808 / 808808 / 808 / 808
Qwen3.8-Flash810 / 813 / 810811 / 812 / 808815 / 809 / 809
GPT-6-Astra749 / 746 / 742745 / 749 / 742748 / 747 / 742
GPT-5.6-Sol820 / 820 / 820820 / 820 / 820820 / 820 / 820
GPT-5.6-Terra820 / 820 / 820820 / 820 / 820820 / 820 / 820
GPT-5.6-Luna820 / 820 / 820820 / 820 / 820820 / 820 / 820
GPT-5.5744 / 740 / 731741 / 740 / 731738 / 746 / 732
Kimi-K3808 / 808 / 808807 / 807 / 806808 / 808 / 808
Qwen3.5-397B-A17B807 / 808 / 807808 / 807 / 807808 / 807 / 807
Qwen3.5-122B-A10B808 / 808 / 808808 / 808 / 808808 / 808 / 808
Qwen3-VL-235B-A22B-Instruct808 / 808 / 808808 / 807 / 807808 / 808 / 808
GLM-5.3-Flash819 / 820 / 819820 / 820 / 820820 / 820 / 820
DeepSeek-V4.1-Flash820 / 820 / 820820 / 820 / 820820 / 820 / 820
MiMo-V2.5763 / 751 / 749782 / 776 / 771793 / 789 / 786
Qwen3.8-27B815 / 815 / 815815 / 814 / 814815 / 815 / 815
Qwen3.6-35B-A3B815 / 815 / 815815 / 815 / 815815 / 815 / 815
Qwen3.6-27B815 / 815 / 815815 / 815 / 815815 / 815 / 815
Qwen3.5-35B-A3B815 / 815 / 815815 / 815 / 815815 / 815 / 815
Qwen3.5-27B815 / 815 / 815815 / 815 / 815815 / 815 / 815
Qwen3.5-9B820 / 820 / 820820 / 820 / 820820 / 820 / 820
Qwen3.5-4B820 / 820 / 820820 / 820 / 820820 / 819 / 819
Qwen3-VL-30B-A3B-Instruct815 / 815 / 815815 / 815 / 815815 / 815 / 815
Qwen3-VL-8B-Instruct820 / 820 / 820820 / 820 / 820820 / 820 / 820
Qwen3-VL-4B-Instruct820 / 820 / 820820 / 820 / 820820 / 820 / 820
InternVL3.5-38B820 / 820 / 820820 / 820 / 820820 / 820 / 820
Keye-VL-2.0-30B-A3B768 / 762 / 720815 / 806 / 802810 / 815 / 806
TimeLens2-8B820 / 820 / 820820 / 820 / 820820 / 820 / 820
TimeLens-8B820 / 820 / 820820 / 820 / 820820 / 820 / 820
LLaVA-OneVision-2-8B820 / 820 / 820820 / 820 / 820820 / 820 / 820
LLaVA-Video-7B820 / 820 / 820820 / 820 / 820820 / 820 / 820
LLaVA-OneVision-7B820 / 820 / 820820 / 820 / 820820 / 820 / 820
AuthorsOpenReview ↗

Response to Reviewer zr1n (W1 Part 2: AV Denominators & W2)

Response to Weakness 1 (Continued): AV Denominators & Shared Subsets

OmniCompass-AV Actual Denominators (Nominal = 240; Format: T / TI / CS)

ModelTG (T / TI / CS)SG (T / TI / CS)QA (T / TI / CS)
Gemini-3.8-Flash239 / 239 / 239239 / 239 / 239239 / 239 / 239
Gemini-3.1-Pro239 / 239 / 239239 / 239 / 239239 / 239 / 239
Qwen3.8-Omni-Flash228 / 228 / 228228 / 227 / 227228 / 228 / 228
Qwen3.5-Omni-Plus228 / 228 / 228228 / 227 / 227228 / 228 / 228
Qwen3.5-Omni-Flash240 / 240 / 240240 / 238 / 238239 / 239 / 238
Seed-2.0-Lite-260428240 / 240 / 240240 / 240 / 240240 / 240 / 240
Seed-2.0-Mini-260428240 / 240 / 240240 / 240 / 240240 / 240 / 240
MiMo-V2.5206 / 187 / 184219 / 208 / 207191 / 201 / 188
Qwen3-Omni-30B-A3B240 / 240 / 240240 / 240 / 240240 / 240 / 240
Qwen2.5-Omni-7B240 / 240 / 240240 / 240 / 240240 / 240 / 240
Qwen2.5-Omni-3B240 / 240 / 240240 / 240 / 240240 / 240 / 240
MiniCPM-o-4.5-8B240 / 240 / 240240 / 240 / 240240 / 240 / 240
video-SALMONN 2+ 72B240 / 240 / 240240 / 240 / 240240 / 240 / 240
video-SALMONN 2+ 7B240 / 240 / 240240 / 240 / 240240 / 240 / 240
Ola-7B240 / 240 / 240240 / 240 / 240240 / 240 / 240
VideoLLaMA2.1-7B-AV240 / 240 / 240240 / 240 / 240240 / 240 / 240
VITA-1.5-7B240 / 240 / 240240 / 240 / 240240 / 240 / 240

Most models—including all locally hosted open-source models and the Seed and GPT-5.6 series—achieve 100% operational coverage (820/820 and 240/240), while a few APIs have minor safety-filter or timeout exclusions (e.g., Kimi-K3 has 808/808/808 on QA and 807/807/806 on SG due to slightly differing safety filters across branches). The 592, 673, and 689 pairs in Figure A3 represent the strict common intersection across the 10 compared models so cross-model comparisons use identical items. We will release per-item status flags and operational coverage alongside conditional accuracy in our benchmark release.


Response to Weakness 2: Scoping Claims to Paired-Setting Robustness & Separating Variability

Thank you for this precise conceptual framing. We fully agree and have updated the paper to scope our core claim explicitly as paired-setting robustness across target specifications, rather than an isolated "pure consistency" defect. Specifically, we disentangle the three contributing factors:

  1. Separating Correctness, Agreement, and CS: Reporting single-setting accuracy, raw prediction Agreement, and CS side-by-side clarifies error sources. In grounding, while the prediction-overlap term (aia_i) adds a 0.3–1.8 point strictness penalty on aggregate averages, min⁡(giT,giTI)\min(g_i^T, g_i^{TI}) penalizes instance-level flips across specifications (making CS 4.3–4.6 points lower than the weaker single-setting mean). In QA, for Kimi-K3 (N=808N=808: 546 both-correct, 44 Text-only correct, 43 Text+Image-only correct, 175 both-wrong), CSQA\text{CS}^{\text{QA}} (67.57%) prevents the 21.66% (175/808) shared-error cases from being rewarded as reliable agreement.
  2. Real-World Cue Modality Shift: Because Text and Text+Image convey target identity via verbal description versus visual reference image, their difference reflects task retention under realistic specification shifts (verbal vs. visual pointer) rather than a pure image-addition ablation.
  3. Isolating Repeatability Noise via Three-Run Controls: Using three independent runs of Gemini-3.8-Flash under identical inputs (detailed in Question 2 below), macro run-to-run variance is minimal (1.33, 1.79, 1.11 points on VL; 1.52, 2.45, 1.04 on AV). Subtracting this same-specification repeatability baseline on TG (1−tIoU×1001 - \text{tIoU} \times 100), cross-specification divergence still exceeds same-specification run-to-run divergence by +9.39 points on VL and +5.58 points on AV, proving genuine specification sensitivity beyond stochastic generation noise.
AuthorsOpenReview ↗

Response to Reviewer zr1n (Q1 & Q2)

Response to Question 1: Fixed-Denominator Sensitivity Analysis

Thank you for this question. As detailed in Weakness 1, an item enters a model's paired CS\text{CS} denominator when both Text and Text+Image return valid responses (with completed-but-empty/unparseable outputs retained and scored 0).

To test ranking sensitivity, we recomputed overall CSAvg\text{CS}^{\text{Avg}} across all 36 VL and 17 AV models using the fixed nominal denominator (820 VL / 240 AV), scoring all safety-filter refusals and transport timeouts as 0:

TrackModelsFull-Leaderboard Spearman ρ\rhoTop-5 Models & Relative Order
OmniCompass-VL360.9928100% unchanged (Gemini-3.8-Flash #1)
OmniCompass-AV170.9926100% unchanged (Gemini-3.8-Flash #1)

OmniCompass-VL (Top-5 Fixed-Denominator Check)

ModelConditional CSAvg\text{CS}^{\text{Avg}}Fixed-Denom. CSAvg\text{CS}^{\text{Avg}}Cond. RankFixed Rank
Gemini-3.8-Flash42.8442.78#1#1
Seed-2.1-Pro-26091542.0242.02#2#2
Seed-2.1-Turbo-26062840.9140.91#3#3
Seed-2.0-Pro-26021540.5240.52#4#4
Qwen3.8-Max40.2239.63#5#5

OmniCompass-AV (Top-5 Fixed-Denominator Check)

ModelConditional CSAvg\text{CS}^{\text{Avg}}Fixed-Denom. CSAvg\text{CS}^{\text{Avg}}Cond. RankFixed Rank
Gemini-3.8-Flash57.4557.21#1#1
Qwen3.8-Omni-Flash48.6446.14#2#2
Qwen3.5-Omni-Plus44.3242.03#3#3
Seed-2.0-Lite-26042839.9439.94#4#4
Gemini-3.1-Pro39.8739.71#5#5

Across the full leaderboard, 19/36 VL and 14/17 AV models have zero rank change (all 100%-coverage models have identical scores), while minor shifts occur only for APIs with higher refusal/timeout rates (max shift: 4 places on VL for GPT-6-Astra #7$\rightarrow$#11 and GPT-5.5 #12$\rightarrow$#16; 2 places on AV for MiMo-V2.5 #7$\rightarrow$#9). For Kimi-K3 on VL QA, CSQA\text{CS}^{\text{QA}} is 67.57% (546/808) conditionally versus 66.59% (546/820) fixed (-0.98 pp). Reporting both metrics jointly captures intrinsic capability and operational availability.


Response to Question 2: Same-Specification Repeatability & Scoping Claims

Thank you for this constructive suggestion. We fixed a stratified 64-video representative subset in each track (VL covering all 7 domains, short/medium/long durations, and all 8 VL QA types; AV covering movies, TV dramas, variety shows, news, and all 5 AV QA types) and ran three independent repeated evaluations of Gemini-3.8-Flash under unchanged inputs (identical 64 timestamped frames, full audio for AV, reference images, prompts, and temperature=0; scores report CS×100\text{CS} \times 100):

OmniCompass-VL (Three Repeated Runs)

ModelTaskRun 1Run 2Run 3Mean Pairwise Diff.
Gemini-3.8-FlashTG18.5818.6016.601.33
Gemini-3.8-FlashSG30.6330.6733.321.79
Gemini-3.8-FlashQA65.0063.3363.331.11

OmniCompass-AV (Three Repeated Runs)

ModelTaskRun 1Run 2Run 3Mean Pairwise Diff.
Gemini-3.8-FlashTG32.8630.7230.581.52
Gemini-3.8-FlashSG42.5241.4645.132.45
Gemini-3.8-FlashQA75.0073.4473.441.04

Macro run-to-run variability is minimal (mean pairwise differences of 1.33, 1.79, 1.11 on VL and 1.52, 2.45, 1.04 on AV). Comparing instance-level TG divergence (1−tIoU×1001 - \text{tIoU} \times 100) across the exact same videos and runs, cross-specification divergence (Text vs. Text+Image) exceeds same-specification run-to-run divergence by +9.39 points on VL and +5.58 points on AV.

Accordingly, we adopt both parts of your recommendation: we explicitly scope our core claim to paired-setting robustness across realistic target-specification shifts, while demonstrating quantitatively that cross-specification variation substantially exceeds ordinary repeatability noise.

Reviewer_LZADOpenReview ↗
Rating
3
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OmniCompass, a benchmark for evaluating MLLMs on video grounding and understanding. It includes two tracks: OmniCompass-VL and OmniCompass-AV. OmniCompass-VL evaluates understanding based on visual information and contains 820 videos and 2,460 task instances. OmniCompass-AV evaluates joint audio-visual understanding and contains 240 videos and 720 task instances.

Both tracks cover three tasks: temporal grounding, spatial grounding, and question answering. Video lengths range from 30 seconds to 88 minutes.

Each task instance is presented under two different target specifications. In the Text setting, the target is described entirely in language. In the Text+Image setting, the model receives a shorter textual query together with a reference image that identifies the target. Both settings refer to the same target and share the same ground truth.

The authors propose Cross-Specification Consistency (CS) scores to measure both the correctness and the consistency of model predictions. CS rewards predictions that are correct and consistent across the two settings.

The authors evaluate 50 MLLMs and find that:

  • models are considerably better at answering questions than at precisely grounding events;
  • proprietary models generally outperform open-source models, and the gap is especially large on audio-visual tasks;
  • providing reference images does not consistently improve performance;
  • adding frame-level timestamps generally improves performance in the ablation experiments.
Critical strengths
  • A clean paired design for target specification. Every instance has a Text version and a Text+Image version that share the same video, target and ground truth. This lets the authors compare the two specifications instance by instance, not just as aggregate averages. The design surfaces a finding that models with similar aggregate accuracy often disagree on individual instances.
  • OmniCompass fills a gap in audio-visual grounding evaluation. The existing audio-visual benchmarks in Table 1 (WorldSense, Video-MME-v2, OmniVideoBench, etc.) cover QA only, with no TG or SG. OmniCompass-AV tests when, where and what on the same videos.
  • Substantial manual annotation with transparent quality control. Authors claim that all instances are annotated by hand. The process includes a pilot calibration round (60 videos, 180 instances), peer cross-verification and final adjudication by a lead expert, for about 1,800 human-hours in total.
Critical weaknesses
  • The spatial grounding protocol may ask for boxes at frames the model never sees. For SG, the model gets an interval and must predict a box at both endpoints. However, inputs are 32–64 frames sampled uniformly over the entire video, and nothing in the paper says the endpoint frames are added.
  • AV-SG does not test audio, and overall audio dependence is not verified. In AV-SG, the time interval is given, and the target is identified by a visual description (Text) or a reference image (Text+Image). Audio is therefore largely redundant. The higher SG scores on AV than on VL (53.5 vs. 32.3 best CS_SG) likely come from large, mostly static targets (30.7% vs. 13.2% of frame area), not from audio-visual integration. Since AV-SG feeds into the AV average, it muddies what the AV track’s overall score means. For AV QA, the vision-only setting already reaches 41–56 CS_QA, and the text-only setting reaches 19–30, well above chance. This suggests many items do not strictly require audio and that the answer options carry language priors.
  • Statistical power and uncertainty. Each video contributes one instance per task (820 per VL task, 240 per AV task; 35–56 per AV QA type), and no confidence intervals or significance tests are reported. Several conclusions rest on small differences. For example, Kimi-K3’s Text vs. Text+Image QA accuracies differ by 0.1 points with only 87 discordant pairs (B.3). With roughly this many discordant pairs, a paired McNemar test needs a difference of about 2.3 points to reach significance on VL QA, and more on AV. Per-model statements such as “only Qwen3.8-27B improves with Text+Image,” the Qwen3.5 397B vs. 122B CS_TG comparison (23.1 vs. 24.3), and small gaps between per-category rankings in Fig. 4 are therefore likely within noise.
Questions for authors
  1. SG input frames. Are the frames at the two SG endpoint timestamps included in the model input? Or does the model see only the 32–64 uniformly sampled frames? If they are not included, how does SG performance change when the two endpoint frames are inserted (oracle setting)?

  2. Sampling limits on TG. For about half of the VL TG instances, at most about one sampled frame falls inside the ground-truth interval. Could you report:

    • the best tIoU achievable when predictions can only use sampled timestamps;
    • TG results broken down by the number of sampled frames inside the ground-truth interval?

    This would show how much of the low TG performance comes from model capability and how much from the protocol.

  3. Audio dependence of the AV track. In Fig. 6, Vision-only already reaches 41–56 CS_QA. What fraction of AV items can be solved without audio? In AV-SG, the interval is given and the target is described visually. What role is the audio expected to play there? Would the AV conclusions still hold on the subset of items that require audio?

9 comments
AuthorsOpenReview ↗

Response to Reviewer LZAD (W1)

Thank you for your recognition of our work—specifically our clean paired target-specification design enabling instance-level comparison across Text and Text+Image, our contribution to filling the gap in audio-visual grounding evaluation across TG, SG, and QA, and our substantial manual annotation with transparent quality control (~1,800 human-hours with pilot calibration, peer cross-verification, and expert adjudication)—and for raising insightful questions. Your feedback has helped us strengthen our paper. Below, we provide point-by-point responses to your comments.


Response to Weakness 1: Endpoint Frame Visibility in Spatial Grounding

Thank you for this suggestion. Our main protocol uses question-agnostic uniform sampling with explicit per-frame timestamps (consistent with V-STaR) so that Text and Text+Image are evaluated on identical frames. Under Uniform-64, at least one sampled frame falls inside the SG interval for 502/820 (61.22%) VL and 120/240 (50.00%) AV instances (rising to 657/820 [80.12%] and 179/240 [74.58%] at 128 frames).

To ensure both endpoint frames are directly visible, we ran an endpoint-visible experiment (+Boundary Frames): keeping the original 64 uniform frames (Uniform 64), we inserted the exact start and end frames of the query interval without ground-truth boxes (AV retains full audio). Table entries report [email protected] (T / TI) | msIoUm_{\text{sIoU}} (T / TI) | CSSG\text{CS}^{\text{SG}} (×100\times 100):

OmniCompass-VL

ModelSetting[email protected] (T / TI)msIoUm_{\text{sIoU}} (T / TI)CSSG\text{CS}^{\text{SG}}
Seed-2.1-Pro-260915Uniform 6440.85 / 38.9040.59 / 39.5634.89
+Boundary Frames82.68 / 79.5170.00 / 68.0664.10 (+29.21)
Gemini-3.8-FlashUniform 6437.48 / 41.5637.74 / 41.8132.28
+Boundary Frames59.76 / 66.7154.75 / 58.6443.30 (+11.02)
Qwen3.5-397B-A17BUniform 6428.22 / 29.6233.44 / 33.7028.03
+Boundary Frames65.43 / 64.8856.35 / 56.8851.07 (+23.04)
Kimi-K3Uniform 6429.99 / 31.6033.35 / 35.2128.45
+Boundary Frames61.34 / 61.9554.61 / 54.9747.97 (+19.52)
Qwen3.8-FlashUniform 6422.56 / 24.3828.84 / 29.2822.63
+Boundary Frames51.46 / 49.3946.14 / 46.6638.55 (+15.92)

OmniCompass-AV

ModelSetting[email protected] (T / TI)msIoUm_{\text{sIoU}} (T / TI)CSSG\text{CS}^{\text{SG}}
Gemini-3.8-FlashUniform 6462.76 / 63.1859.79 / 59.2953.46
+Boundary Frames79.92 / 73.2273.07 / 68.5761.40 (+7.94)
Qwen3.8-Omni-FlashUniform 6460.09 / 59.4756.82 / 55.9650.86
+Boundary Frames85.46 / 84.4475.13 / 73.5270.00 (+19.14)
Seed-2.0-Lite-260428Uniform 6456.25 / 52.9253.32 / 51.8245.49
+Boundary Frames80.00 / 80.4269.52 / 69.6763.07 (+17.58)

This comparison cleanly isolates sampling coverage from spatial localization capability: providing exact endpoint frames eliminates cross-frame interpolation uncertainty (+7.94 to +29.21 CSSG\text{CS}^{\text{SG}}), yet even when both endpoint frames are 100% visible, leading models on VL still reach only 38.55–64.10 CSSG\text{CS}^{\text{SG}}, confirming that fine-grained target matching and box regression remain key bottlenecks.

AuthorsOpenReview ↗

Response to Reviewer LZAD (W2)

Response to Weakness 2: Role of AV-SG, Domain Differences, and Audio Dependence

Thank you for this thoughtful analysis. OmniCompass-AV evaluates whether omni-modal models can localize events in time (TG), ground targets in space (SG), and answer semantic questions (QA) under unified audio-visual input, reporting both per-task scores and CSAvg\text{CS}^{\text{Avg}} to support fine-grained diagnosis alongside holistic comparison.

1. Audio's Role in AV-SG (Speaker-Entity Disambiguation): Given the temporal interval, AV-SG isolates target identity disambiguation and spatial localization. Comparing +Vision and +Vision+Audio item-by-item across all 240 AV-SG instances for the four models in Figure 6 (Gemini-3.8-Flash, Qwen3.8-Omni-Flash, Seed-2.0-Lite-260428, MiMo-V2.5), 59/240 instances (24.58%) show clear CSSG\text{CS}^{\text{SG}} gains with audio in at least three models—specifically 49 items involving the target speaking, conversing, or singing, and 4 items involving a mentioned or listening person. For example, when localizing "the drinking man listening to another person introduce the room", adding audio shifts Qwen3.8-Omni-Flash from boxing the standing speaker to the seated listener (0.00 →\rightarrow 71.56 CSSG\text{CS}^{\text{SG}}; Gemini, Seed, and MiMo gain +19.44, +3.04, and +37.19 points). Thus, audio disambiguates whom to ground in multi-person scenes, while visual perception governs box precision.

2. Cross-Track Target Scale vs. Within-Item Causal Ablation: OmniCompass-AV comprises 240 videos curated from movies (70), TV dramas (70), variety shows (50), and news (50), which naturally feature closer human-centric shots (30.7% mean target area vs. 13.2% in VL). Because cross-track absolute scores reflect domain composition, we isolate audio's causal contribution via controlled within-item modality ablations on the exact same videos and targets.

3. Controlled Six-Setting QA Ablation (CSQA×100\text{CS}^{\text{QA}} \times 100): Using 64 timestamped frames and ASR transcripts generated by Seed-2.0-Lite with manual item-by-item human verification (shared across models):

Input SettingGemini-3.8-FlashQwen3.8-Omni-FlashSeed-2.0-Lite-260428
Text-only30.1326.8718.75
+Vision55.6544.9341.25
+Transcript57.7456.8353.75
+Audio60.6756.3956.25
+Vision+Transcript69.0471.8164.58
+Vision+Audio78.2473.1363.75

Adding audio over +Vision lifts CSQA\text{CS}^{\text{QA}} by +22.59, +28.19, and +22.50 points, while adding vision over +Transcript still lifts CSQA\text{CS}^{\text{QA}} by +11.30, +14.98, and +10.83 points—proving genuine cross-modal synergy rather than text priors or speech-only shortcuts. Moreover, on Gemini-3.8-Flash Emotion Recognition, +Vision+Audio reaches 82.05, surpassing +Vision (46.15) by +35.90 and +Vision+Transcript (69.23) by +12.82, confirming that prosody and non-verbal acoustic cues cannot be replaced by transcripts.

AuthorsOpenReview ↗

Response to Reviewer LZAD (W3 & Q1)

Response to Weakness 3: Statistical Power and Uncertainty

Thank you for this insightful suggestion. We added quantitative uncertainty analyses across two levels: (1) three-run same-specification repeatability on fixed 64-video subsets (detailed in Reviewer 7MqS Q1), confirming minimal run-to-run variance (mean pairwise CS\text{CS} diff.: 1.33, 1.79, 1.11 on VL; 1.52, 2.45, 1.04 on AV); and (2) 10,000-iteration video-level paired bootstrap (95% CIs) across all 36 VL and 17 AV models, plus exact McNemar tests on QA:

  • Kimi-K3 Marginal Accuracy vs. Instance-Level Flips: Text and Text+Image QA accuracies are 73.0% and 72.9% (exact McNemar p=1.0p=1.0), confirming no significant marginal difference. Yet paired analysis reveals 10.8% instance-level correctness flips and 67.6% CSQA\text{CS}^{\text{QA}}, directly illustrating why OmniCompass's paired design uncovers sensitivity hidden by aggregate averages.
  • Qwen3.8-27B Cross-Task Trade-Off: Its overall CSAvg\text{CS}^{\text{Avg}} shift is +0.4 (95% CI [−0.7,1.6][-0.7, 1.6]), which is not statistically significant overall; however, per-task breakdown reveals a structural trade-off (+1.9 CSTG\text{CS}^{\text{TG}} and +1.6 CSSG\text{CS}^{\text{SG}} vs. -2.2 pp QA accuracy), confirming that visual references aid spatial/temporal target matching while omitting some textual details useful for QA.
  • Qwen3.5 397B vs. 122B Scaling Boundary: Their CSTG\text{CS}^{\text{TG}} difference is -1.3 (95% CI [−2.7,0.2][-2.7, 0.2], spanning zero), whereas 397B achieves statistically significant gains of +7.2 CSSG\text{CS}^{\text{SG}} (95% CI [5.9,8.6][5.9, 8.6]) and +5.2 CSQA\text{CS}^{\text{QA}} (95% CI [2.1,8.4][2.1, 8.4]), pinpointing where model scale improves performance.
  • Figure 4 Categories & Macro Conclusions: We added per-category sample sizes and 95% CIs in the revision, framing close category ranks descriptively while focusing on consistent cross-model trends (e.g., Visual Counting lowest, Causal Reasoning highest). All core macro gaps—proprietary vs. open-source tiers, QA vs. TG/SG gaps, and +22.5 to +28.2 AV QA audio gains—remain highly significant under 10,000 bootstrap resamples.

Response to Question 1: SG Input Frames & Endpoint-Visible Evaluation

Uniform sampling does not guarantee inclusion of the two exact SG endpoint frames. Following your suggestion, we evaluated an endpoint-visible setting (64+Boundary) by inserting the exact start and end frames of the query interval into Uniform 64 without ground-truth boxes (AV retains full audio). Table entries report msIoUm_{\text{sIoU}} (Text, Text+Image) and CSSG\text{CS}^{\text{SG}} (×100\times 100):

OmniCompass-VL

ModelSettingText (msIoUm_{\text{sIoU}})Text+Image (msIoUm_{\text{sIoU}})CSSG\text{CS}^{\text{SG}}
Seed-2.1-Pro-260915Uniform 6440.5939.5634.89
64+Boundary70.0068.0664.10 (+29.21)
Gemini-3.8-FlashUniform 6437.7441.8132.28
64+Boundary54.7558.6443.30 (+11.02)
Qwen3.5-397B-A17BUniform 6433.4433.7028.03
64+Boundary56.3556.8851.07 (+23.04)
Kimi-K3Uniform 6433.3535.2128.45
64+Boundary54.6154.9747.97 (+19.52)
Qwen3.8-FlashUniform 6428.8429.2822.63
64+Boundary46.1446.6638.55 (+15.92)

OmniCompass-AV

ModelSettingText (msIoUm_{\text{sIoU}})Text+Image (msIoUm_{\text{sIoU}})CSSG\text{CS}^{\text{SG}}
Gemini-3.8-FlashUniform 6459.7959.2953.46
64+Boundary73.0768.5761.40 (+7.94)
Qwen3.8-Omni-FlashUniform 6456.8255.9650.86
64+Boundary75.1373.5270.00 (+19.14)
Seed-2.0-Lite-260428Uniform 6453.3251.8245.49
64+Boundary69.5269.6763.07 (+17.58)

Inserting exact endpoint frames eliminates cross-frame interpolation errors (+7.94 to +29.21 CSSG\text{CS}^{\text{SG}}), yet even with 100% visible endpoints, VL CSSG\text{CS}^{\text{SG}} remains 38.55–64.10, confirming that fine-grained target matching and box regression remain challenging.

Reviewer_7MqSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose the OmniCompass benchmark for evaluating omni-modal video grounding and understanding. It consists of a vision-language (VL) track of 820 videos and an audio-visual (AV) track of 240 videos, each having three tasks - temporal grounding, spatial grounding and QA. Each of them are further evaluated under Text and Text + Image target specifications, referring to the same video, ground truth and target. A Cross-Specification Consistency Score (CS) is also introduced to jointly measure task accuracy and prediction accuracy. 50 MLLMs are evaluated and the experiments show that temporal grounding remains a significant challenge, providing additional reference images does not consistently improve performance, explicitly providing frame-level temporal information improves performance for temporal grounding and incorporating audio substantially improves question answering on the AV track.

Critical strengths

S1. The paper studies an important gap between answering questions about videos and accurately grounding the corresponding evidence. Evaluating temporal grounding, spatial grounding, and question answering under a single benchmark is valuable, as they probe complementary aspects of video understanding that may not be captured by QA accuracy alone.

S2. A useful aspect of the benchmark is that it evaluates whether models can identify and reason about the same target when it is described using text or text + reference image. Both settings use the same video, task, and ground truth indicating how changing the target specification affects model performance.

S3. Careful dataset construction and comprehensive quality checks involving 1800 human annotation hours, expert review and multiple rounds of verification.

S4. The analyses of frame budgets, timestamps, input modalities, and reference specifications are interesting and useful.

Critical weaknesses

W1. The consistency score does not clearly separate correctness from consistency. Reporting correctness and agreement separately would help clarify whether lower CS means inaccurate predictions or actual disagreement between target specifications.

W2. Appendix C.4 states that the three tasks may refer to different targets within the same video. Therefore, strong QA performance and weak grounding do not indicate that a model understands an event but cannot locate it. An analysis involving the same target or event across tasks would better demonstrate the diagnostic value of the unified benchmark.

W3. The models are benchmarked using only 32 or 64 uniformly sampled frames. Appendix B.1 shows that denser frame sampling improves grounding performance. This raises the question of whether poor grounding is due to model limitations or insufficient visual evidence, especially for short events in long videos.

Questions for authors

Minor Concerns

  1. The related-work discussion could acknowledge prior benchmarks involving composite queries (image+text). Specifically, Localizing Events in Videos with Multimodal Queries (Zhang et al., 2024) and CoVR (Ventura et al., 2024). Although they differ in task formulation, they can motivate the concept of image-augmented target specification.

  2. Spelling error;Line 194 of the draft -- "35extitText" instead of "Text"

  3. Text and Text+Image differ in both textual detail and image availability, as confirmed in Appendix B.2. Thus, their performance differences cannot be attributed to adding a reference image alone.

Questions

  1. Can the authors report prediction correctness and agreement separately? Are evaluations deterministic? How much cross-specification variation remains when compared with repeated runs using the same target specification?

  2. Joint evaluation for TG, SG and QA ideally using cases involving the same target or event.

  3. What percentage of TG events are represented in the sampled frames fed to the model? How does performance change with this evidence coverage?

I give a rating of Weak Reject. However I am open to increasing my rating if the authors convincingly address the above mentioned concerns during the rebuttal.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 7MqS (W1)

Thank you for your recognition of our work—specifically our unified evaluation of complementary tasks (TG, SG, and QA), our paired target-specification design across Text and Text+Image, our 1,800-hour multi-round quality verification, and our ablation analyses—and for raising insightful questions. Your feedback has helped us strengthen our paper. Below, we provide point-by-point responses to your comments.


Response to Weakness 1: Separating Correctness from Agreement

Thank you for this suggestion. While the main paper reports setting-specific scores and Appendix B.3 analyzes joint correctness combinations, reporting single-setting accuracy, raw prediction Agreement (percentage of paired items where Text and Text+Image produce identical outputs, including identical errors), and CS side-by-side clarifies the exact sources of error:

OmniCompass-VL (QA Decomposition, %)

ModelText Acc.Text+Image Acc.AgreementCSQA\text{CS}^{\text{QA}}
Gemini-3.8-Flash73.2676.5685.3569.11
Seed-2.1-Pro-26091573.4174.2783.5467.44
Qwen3.8-Max72.4072.5286.6367.20
Kimi-K373.0272.9085.5267.57
GPT-6-Astra69.1465.6378.1759.43
Qwen3.5-397B-A17B67.7869.6478.5660.59
GLM-5.3-Flash68.2968.6675.2459.39
Qwen3.8-27B64.7962.5877.0655.71

OmniCompass-AV (QA Decomposition, %)

ModelText Acc.Text+Image Acc.AgreementCSQA\text{CS}^{\text{QA}}
Gemini-3.8-Flash80.3380.7594.1478.24
Qwen3.8-Omni-Flash75.8878.9588.6073.25
Qwen3.5-Omni-Plus71.0573.2588.1668.42
Seed-2.0-Lite-26042871.2569.1782.5063.75
Gemini-3.1-Pro72.3871.1380.3364.85
Qwen3.5-Omni-Flash63.4564.2982.3557.98

Both tracks show why reporting all three metrics together is essential:

  1. Similar Marginal Accuracies Mask Instance-Level Flips: On VL, Seed-2.1-Pro differs by only 0.86 percentage points marginally, yet 12.80% of items flip correctness across specifications (Qwen3.8-Max differs by only 0.12 points marginally, yet 10.52% flip).
  2. Raw Agreement Rewards "Consistently Wrong" Predictions: On AV, Gemini-3.8-Flash reaches 94.14% Agreement, yet 15.90% of all items are cases where both settings select the exact same wrong option (19.74% for Qwen3.5-Omni-Plus).

The same principle governs TG and SG via min⁡(giT,giTI,ai)\min(g_i^T, g_i^{TI}, a_i): CS avoids rewarding consistent errors while penalizing instance-level drift, offering a unified metric for reliable cross-specification execution.

AuthorsOpenReview ↗

Response to Reviewer 7MqS (W2 & W3)

Response to Weakness 2: Same-Target / Same-Event Cross-Task Diagnosis

Thank you for this insightful suggestion. Following your recommendation, we curated the exact cross-task subsets where TG, SG, and QA target the same entity or continuous interaction event within the same video: 289/820 (35.24%) triplets in OmniCompass-VL and 42/240 (17.50%) in OmniCompass-AV (full tables in our response to Reviewer gARS Weakness 3).

On these matched-target/event triplets, the stark gap between QA and grounding persists: on VL (N=289N=289), Kimi-K3 achieves 62.98 CSQA\text{CS}^{\text{QA}} versus 20.44 CSTG\text{CS}^{\text{TG}} and 27.18 CSSG\text{CS}^{\text{SG}}; on AV (N=42N=42), Qwen3.8-Omni-Flash achieves 86.49 CSQA\text{CS}^{\text{QA}} versus 16.70 CSTG\text{CS}^{\text{TG}}. Conditioning further on triplets where QA is answered correctly under BOTH Text and Text+Image:

  • On 181 VL triplets where Kimi-K3 answers QA correctly in both settings, its localization scores remain only 23.07 CSTG\text{CS}^{\text{TG}} and 29.86 CSSG\text{CS}^{\text{SG}}.
  • On 32 AV triplets where Qwen3.8-Omni-Flash answers QA correctly in both settings, its CSTG\text{CS}^{\text{TG}} is only 18.28 (CSSG\text{CS}^{\text{SG}}: 68.30).

This directly proves that current MLLMs often answer questions correctly via global context or shortcuts without accurately localizing the underlying spatiotemporal evidence—demonstrating the core diagnostic value of unifying TG, SG, and QA. We will release these triplet IDs with the dataset.


Response to Weakness 3: Model Limitations vs. Sampling Density

Thank you for raising this question. In our protocol, every frame is paired with its exact timestamp in seconds so models can interpolate continuous boundaries. To directly evaluate sensitivity to visual sampling density, we conducted controlled frame-budget experiments across 32, 64, and 128 uniformly sampled frames on six VL models and three AV models (using identical valid video sets per track/task and full audio for AV; scores report CS×100\text{CS} \times 100):

OmniCompass-VL (Equal Frame Budgets)

Model32f (TG / SG / QA)32f CSAvg\text{CS}^{\text{Avg}} (Rank)64f (TG / SG / QA)64f CSAvg\text{CS}^{\text{Avg}} (Rank)128f (TG / SG / QA)128f CSAvg\text{CS}^{\text{Avg}} (Rank)
Gemini-3.8-Flash22.41 / 28.39 / 65.3238.70 (#1)27.98 / 32.66 / 70.1943.61 (#1)35.50 / 44.19 / 75.3551.68 (#1)
Qwen3.5-397B-A17B17.46 / 23.41 / 58.6433.17 (#2)24.01 / 28.79 / 62.4038.40 (#2)27.51 / 32.79 / 65.3241.87 (#2)
Qwen3.8-27B16.68 / 17.86 / 57.6630.73 (#3)23.93 / 20.73 / 61.0035.22 (#3)28.67 / 22.41 / 66.3039.12 (#3)
Qwen3.6-35B-A3B15.03 / 21.59 / 52.5129.71 (#4)19.15 / 26.44 / 56.2733.95 (#4)22.53 / 30.32 / 62.9538.60 (#4)
Qwen3.5-9B14.23 / 18.22 / 51.3927.95 (#5)19.51 / 22.36 / 53.9031.92 (#5)23.37 / 22.76 / 55.8533.99 (#5)
Qwen3-VL-8B-Instruct12.30 / 16.71 / 52.3727.13 (#6)15.64 / 18.63 / 55.2929.86 (#6)18.59 / 19.22 / 55.7131.17 (#6)

OmniCompass-AV (Equal Frame Budgets)

Model32f (TG / SG / QA)32f CSAvg\text{CS}^{\text{Avg}} (Rank)64f (TG / SG / QA)64f CSAvg\text{CS}^{\text{Avg}} (Rank)128f (TG / SG / QA)128f CSAvg\text{CS}^{\text{Avg}} (Rank)
Gemini-3.8-Flash41.29 / 48.58 / 69.1253.00 (#1)39.78 / 53.17 / 78.8057.25 (#1)39.16 / 59.61 / 74.6557.81 (#1)
Qwen3.8-Omni-Flash19.78 / 42.77 / 67.2843.28 (#2)21.50 / 50.63 / 74.1948.77 (#2)22.79 / 54.65 / 72.8150.08 (#2)
Seed-2.0-Lite-2604289.89 / 40.17 / 62.6737.58 (#3)10.92 / 44.45 / 64.9840.11 (#3)12.53 / 33.96 / 62.2136.23 (#3)

Combining these results with our event-coverage stratification (detailed in Question 3) yields two key conclusions:

  1. Model Rankings Are 100% Invariant Across Frame Budgets: Scaling from 32 to 128 frames lifts absolute grounding scores (e.g., Gemini-3.8-Flash gains +13.09 CSTG\text{CS}^{\text{TG}} and +15.80 CSSG\text{CS}^{\text{SG}} on VL; Qwen3.5-397B-A17B gains +10.05 CSTG\text{CS}^{\text{TG}} and +9.38 CSSG\text{CS}^{\text{SG}}), yet the overall CSAvg\text{CS}^{\text{Avg}} and per-task rankings of all six VL models and three AV models remain 100% invariant across 32, 64, and 128 frames.
  2. Fine-Grained Temporal Localization Remains a Core Model Bottleneck: Even at 128 frames (covering 85.4% of VL events) or on high-coverage events with ≥4\ge 4 sampled frames inside the ground-truth interval (see Question 3), leading models still plateau at 35.50–42.88 CSTG\text{CS}^{\text{TG}}. Thus, low grounding scores stem primarily from limited fine-grained temporal boundary localization rather than sampling sparsity alone.
AuthorsOpenReview ↗

Response to Reviewer 7MqS (M1–M3 & Q1)

Response to Minor Concerns 1, 2, and 3

  • Minor Concern 1 (Related Work on Composite Queries): Thank you for pointing out ICQ (Zhang et al., 2025) and CoVR (Ventura et al., 2024). Both motivate multimodal image+text target specification; OmniCompass builds on this paradigm by pairing Text and Text+Image queries on the exact same video, target, and ground truth across TG, SG, and QA in both VL and AV tracks to evaluate cross-specification robustness via CS. We have updated Section 2 (Related Work) to cite and discuss both works.
  • Minor Concern 2 (Typo at Line 194): Thank you for catching this; we have fixed this LaTeX formatting typo.
  • Minor Concern 3 (Textual Detail vs. Reference Image): Replacing explicit textual identifiers (e.g., "Team USA player No. 7") with a visual pointer ("the player in the reference image") is an intentional design to prevent text shortcuts—if full textual descriptions remained, models could bypass the reference image entirely. This mirrors real-world usage where users specify targets either verbally or via an example image under identical visual/audio evidence and ground truth.

Response to Question 1: Correctness Decomposition, Determinism, & Same-Specification Repeatability

Thank you for this thoughtful question. We provide quantitative evidence across three levels:

  1. Independent Decomposition of Correctness, Agreement, and CS: As shown in Weakness 1, reporting single-setting accuracy, Agreement, and CS side-by-side exposes instance-level behavior hidden by marginal averages (e.g., Kimi-K3 on VL QA achieves 73.02% Text and 72.90% Text+Image accuracy, 85.52% Agreement, and 67.57% CSQA\text{CS}^{\text{QA}}—revealing both 10.8% instance-level correctness flips and 17.95% shared-error agreements).
  2. Pipeline Determinism & Same-Specification Three-Run Baseline: Our frame sampling, timestamp alignment, regex parsing, and metric computation pipeline is 100% deterministic. Locally hosted models use greedy decoding (temperature=0), while main-table proprietary APIs use official defaults. To quantify intrinsic API generation variance even at temperature=0, we fixed a representative 64-video subset in each track (covering all 7 domains, short/medium/long durations, and all 8 VL / 5 AV QA types) and ran three independent repeated evaluations of Gemini-3.8-Flash under identical 64 timestamped frames, audio, reference images, and prompts (temperature=0; scores report CS×100\text{CS} \times 100):

OmniCompass-VL (Three Repeated Runs)

ModelTaskRun 1Run 2Run 3Mean Pairwise Diff.
Gemini-3.8-FlashTG18.5818.6016.601.33
Gemini-3.8-FlashSG30.6330.6733.321.79
Gemini-3.8-FlashQA65.0063.3363.331.11

OmniCompass-AV (Three Repeated Runs)

ModelTaskRun 1Run 2Run 3Mean Pairwise Diff.
Gemini-3.8-FlashTG32.8630.7230.581.52
Gemini-3.8-FlashSG42.5241.4645.132.45
Gemini-3.8-FlashQA75.0073.4473.441.04

Macro-level run-to-run variance is minimal (only 1.33, 1.79, and 1.11 points on VL; 1.52, 2.45, and 1.04 points on AV)—an order of magnitude smaller than inter-model gaps and the 20–30+ point gap between QA and grounding. 3. Cross-Specification Variation Substantially Exceeds Repeatability Noise: Measuring instance-level prediction divergence on TG as (1−tIoU)×100(1 - \text{tIoU}) \times 100 across the exact same videos and three runs, cross-specification divergence (Text vs. Text+Image) exceeds same-specification run-to-run divergence by +9.39 points on VL and +5.58 points on AV. Thus, OmniCompass captures genuine cross-specification sensitivity well beyond stochastic generation variance.

Reviewer_gARSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OmniCompass introduces a benchmark for evaluating omni-modal video grounding and understanding. It contains 820 videos and 2,460 task instances in the VL track, with a 240-video subset and 720 instances in the AV track. Both tracks cover temporal grounding (TG), spatial grounding (SG), and question answering (QA). Each task provides paired Text and Text+Image target specifications with shared ground truth. The authors introduce Cross-Specification Consistency (CS) to jointly measure correctness and consistency across the two settings. Experiments on 50 models reveal weak fine-grained grounding, inconsistent performance across target specifications, and substantial gaps between proprietary and open-source models, especially on the AV track.

Critical strengths
  1. Broad evaluation coverage. OmniCompass integrates TG, SG, and QA across visual and audio-visual settings.
  2. Substantial human annotation effort. The benchmark spans seven domains and diverse video durations, with approximately 1,800 human-hours devoted to annotation and multi-stage quality verification.
Critical weaknesses
  1. Insufficient validation of genuine audio-visual integration. The AV evaluation lacks audio-only and transcript-based baselines, making it difficult to distinguish genuine cross-modal reasoning from speech-based understanding. The authors also do not verify that all AV questions require both audio and visual evidence.

2.Potentially unfair spatial grounding evaluation. SG requires bounding boxes at specified temporal boundaries, but models receive only uniformly sampled frames. The exact boundary frames may be absent, making it difficult to distinguish localization failures from missing visual evidence.

  1. Limited task integration and incomplete related work. TG, SG, and QA can target different events within the same video, preventing direct evaluation of whether answers are supported by correctly grounded evidence. Closely related benchmarks, including NExT-GQA, VideoZeroBench and CaST-Bench, are not adequately discussed.

  2. Limited analysis of fine-grained audio-visual grounding. The experiments do not assess audio-visual temporal alignment or speaker-to-entity binding, despite their importance for omni-modal understanding. The limited benefit of audio for spatial grounding also remains largely unexplained.

Questions for authors

1.Fairness of model comparisons. Given the substantial impact of frame budgets shown in Appendix B.1, how reliable are the model rankings under unequal frame budgets? Can the authors compare representative models using the same number of frames?

2.Human performance baseline. How well do humans perform on TG, SG, and QA under the same evaluation protocol? Can the authors report human performance on a representative subset of both VL and AV tracks to establish task difficulty and contextualize the performance gaps between humans and current models?

7 comments
AuthorsOpenReview ↗

Response to Reviewer gARS (W1)

Thank you for your recognition of our work—specifically our broad evaluation coverage integrating TG, SG, and QA across visual and audio-visual settings, and our substantial human annotation effort (~1,800 human-hours with multi-stage quality verification)—and for raising insightful questions. Your feedback has helped us strengthen our paper. Below, we provide point-by-point responses to your comments.

Response to Weakness 1: Validation of Genuine Audio-Visual Integration

Thank you for this suggestion. Extending Figure 6 (Text-only, +Vision, +Vision+Audio), we added +Audio, +Transcript, and +Vision+Transcript baselines on identical valid instances across three representative models: Gemini-3.8-Flash (Gemini), Qwen3.8-Omni-Flash (Qwen), and Seed-2.0-Lite-260428 (Seed). Visual inputs use 64 timestamped frames, and transcripts are generated via ASR by Seed-2.0-Lite-260428 across all 240 AV videos followed by manual item-by-item human verification (shared across all models). Text+Image includes the reference image (except Text-only). Scores report CS×100\text{CS} \times 100 and their unweighted mean (CSAvg\text{CS}^{\text{Avg}}).

TaskTemporal GroundingSpatial GroundingQuestion AnsweringCSAvg\text{CS}^{\text{Avg}}
SettingsGeminiQwenSeedGeminiQwenSeedGeminiQwenSeedGeminiQwenSeed
Text-only0.390.770.6128.5230.7115.8230.1326.8718.7519.6819.4511.73
+Vision11.2710.7710.2052.8051.0945.8755.6544.9341.2539.9135.6032.44
+Transcript2.865.181.5322.9125.8716.2657.7456.8353.7527.8429.2923.85
+Audio39.8238.0125.9318.8711.166.2960.6756.3956.2539.7935.1929.49
+Vision+Transcript11.4916.2513.2246.4448.5447.1269.0471.8164.5842.3245.5341.64
+Vision+Audio40.6422.0010.5753.4651.0845.4978.2473.1363.7557.4548.7439.94

Key Findings:

  1. Audio-Visual Synergy & Item-Level Recovery in QA: Even with human-verified transcripts (+Transcript), adding vision (+Vision+Transcript) lifts QA CS by +11.30 (Gemini-3.8-Flash), +14.98 (Qwen3.8-Omni-Flash), and +10.83 (Seed-2.0-Lite-260428; all paired 95% CIs positive). Joint +Vision+Audio outperforms +Vision by +22.59, +28.19, +22.50 and +Audio by +17.57, +16.74, +7.50 (all six paired 95% CIs above zero), while exceeding +Vision+Transcript by +9.21 on Gemini-3.8-Flash (95% CI [4.6,13.8][4.6, 13.8]). On questions each model fails under +Vision (paired correctness in both settings), adding audio recovers 60/106 (56.6%), 70/125 (56.0%), and 71/141 (50.4%) items; in the text-target branch, the three models also answer 14, 18, and 14 items correctly under joint input after failing with either modality alone.
  2. Non-Verbal Acoustic Cues: On Gemini-3.8-Flash Emotion Recognition, +Vision+Audio reaches 82.05, surpassing +Vision (46.15) by +35.90 and +Vision+Transcript (69.23) by +12.82, confirming that prosody and non-verbal cues cannot be recovered from transcripts alone.
  3. Cross-Task Integration Asymmetry in TG: +Audio far outperforms +Transcript (39.82 vs. 2.86, 38.01 vs. 5.18, 25.93 vs. 1.53) because continuous audio preserves when speech occurs. Adding vision further lifts Gemini-3.8-Flash to 40.64, whereas Qwen3.8-Omni-Flash (22.00) and Seed-2.0-Lite-260428 (10.57) drop under joint input—exposing a diagnostic cross-task asymmetry in how current models fuse dense frames with continuous audio.
  4. Speaker Disambiguation in SG: While aggregate SG means remain close across the full 240-video set, selecting items with strictly positive gain (ΔCSSG>0\Delta\text{CS}^{\text{SG}} > 0 in at least three of four models, no extra effect-size threshold) isolates 59/240 (24.58%) cases with recurring speaker-listener disambiguation gains (see Weakness 4).

As part of our 3,180-instance benchmark alongside OmniCompass-VL (2,460 instances), OmniCompass-AV (240 videos, 720 instances) jointly evaluates TG, SG, and QA on audio-visual videos, providing a comprehensive diagnostic profile across speech-timestamp alignment, speaker disambiguation, and audio-visual reasoning.

AuthorsOpenReview ↗

Response to Reviewer gARS (W2)

Response to Weakness 2: Endpoint Frame Visibility in Spatial Grounding

Thank you for raising this point. Our main protocol uses question-agnostic uniform sampling to maintain a consistent observation protocol across TG, SG, and QA under a fixed visual budget (consistent with IV-Bench, WorldSense, Video-MME-v2, and V-STaR).

To separate frame coverage from boundary-frame localization, we ran an endpoint-visible experiment (+Boundary Frames): keeping the original 64 uniform frames (Uniform 64), we inserted the exact start and end frames of the query interval (without ground-truth boxes; AV retains full audio). Table entries report [email protected] (Text / Text+Image) | msIoUm_{\text{sIoU}} (Text / Text+Image) | CSSG\text{CS}^{\text{SG}}.

OmniCompass-VL

ModelSetting[email protected] (T / TI)msIoUm_{\text{sIoU}} (T / TI)CSSG\text{CS}^{\text{SG}}
Seed-2.1-Pro-260915Uniform 6440.85 / 38.9040.59 / 39.5634.89
+Boundary Frames82.68 / 79.5170.00 / 68.0664.10 (+29.21)
Gemini-3.8-FlashUniform 6437.48 / 41.5637.74 / 41.8132.28
+Boundary Frames59.76 / 66.7154.75 / 58.6443.30 (+11.02)
Qwen3.5-397B-A17BUniform 6428.22 / 29.6233.44 / 33.7028.03
+Boundary Frames65.43 / 64.8856.35 / 56.8851.07 (+23.04)
Kimi-K3Uniform 6429.99 / 31.6033.35 / 35.2128.45
+Boundary Frames61.34 / 61.9554.61 / 54.9747.97 (+19.52)
Qwen3.8-FlashUniform 6422.56 / 24.3828.84 / 29.2822.63
+Boundary Frames51.46 / 49.3946.14 / 46.6638.55 (+15.92)

OmniCompass-AV

ModelSetting[email protected] (T / TI)msIoUm_{\text{sIoU}} (T / TI)CSSG\text{CS}^{\text{SG}}
Gemini-3.8-FlashUniform 6462.76 / 63.1859.79 / 59.2953.46
+Boundary Frames79.92 / 73.2273.07 / 68.5761.40 (+7.94)
Qwen3.8-Omni-FlashUniform 6460.09 / 59.4756.82 / 55.9650.86
+Boundary Frames85.46 / 84.4475.13 / 73.5270.00 (+19.14)
Seed-2.0-Lite-260428Uniform 6456.25 / 52.9253.32 / 51.8245.49
+Boundary Frames80.00 / 80.4269.52 / 69.6763.07 (+17.58)

This comparison cleanly separates frame coverage under standard uniform sampling from boundary-frame localization when exact queried boundary images are supplied (+7.94 to +29.21 CSSG\text{CS}^{\text{SG}}). Even when exact boundary frames are supplied, leading models still reach only 38.55–64.10 CSSG\text{CS}^{\text{SG}} on VL and 61.40–70.00 on AV, showing substantial remaining localization difficulty.

AuthorsOpenReview ↗

Response to Reviewer gARS (W3)

Response to Weakness 3: Same-Target/Event Cross-Task Evaluation & Related Work

Thank you for this valuable suggestion. We have updated Section 2 (Related Work) to cite and discuss NExT-GQA, VideoZeroBench, and CaST-Bench.

1. Same-Target / Same-Event Cross-Task Subset Analysis: OmniCompass balances broad narrative diversity across full videos with direct cross-task evidence verification. Based on all task queries, reference images, and annotations (without using model outputs), we identified 289/820 (35.24%) video triplets in OmniCompass-VL and 42/240 (17.50%) in OmniCompass-AV where TG, SG, and QA target the exact same entity or continuous interaction event. Evaluating existing predictions yields (CS×100\text{CS} \times 100, Full / Subset):

OmniCompass-VL (Same-Target/Event Subset, N=289N=289 vs. Full Set)

ModelCSTG\text{CS}^{\text{TG}}CSSG\text{CS}^{\text{SG}}CSQA\text{CS}^{\text{QA}}CSAvg\text{CS}^{\text{Avg}}
Gemini-3.8-Flash27.14 / 28.5532.28 / 30.8869.11 / 66.0942.84 / 41.84
Seed-2.1-Pro-26091523.72 / 24.2034.89 / 32.8367.44 / 67.4742.02 / 41.50
Qwen3.8-Max26.72 / 24.5026.73 / 25.0967.20 / 62.6340.22 / 37.41
Kimi-K322.64 / 20.4428.45 / 27.1867.57 / 62.9839.55 / 36.87
GPT-6-Astra26.28 / 22.7232.73 / 28.5459.43 / 51.0739.48 / 34.11
Qwen3.5-397B-A17B23.06 / 21.1228.03 / 25.6960.59 / 52.9437.23 / 33.25
GLM-5.3-Flash18.30 / 17.2714.58 / 12.7259.39 / 55.0230.76 / 28.34
Qwen3.8-27B15.98 / 15.2217.24 / 16.8955.71 / 49.8329.64 / 27.31

OmniCompass-AV (Same-Target/Event Subset, N=42N=42 vs. Full Set)

ModelCSTG\text{CS}^{\text{TG}}CSSG\text{CS}^{\text{SG}}CSQA\text{CS}^{\text{QA}}CSAvg\text{CS}^{\text{Avg}}
Gemini-3.8-Flash40.64 / 37.0953.46 / 68.0278.24 / 80.9557.45 / 62.02
Qwen3.8-Omni-Flash21.81 / 16.7050.86 / 67.8473.25 / 86.4948.64 / 57.01
Qwen3.5-Omni-Plus13.99 / 7.8850.55 / 70.0168.42 / 75.6844.32 / 51.19
Seed-2.0-Lite-26042810.57 / 6.2845.49 / 61.1863.75 / 73.8139.94 / 47.09
Gemini-3.1-Pro9.01 / 5.2245.76 / 61.5464.85 / 71.4339.87 / 46.06
Qwen3.5-Omni-Flash5.68 / 5.8734.68 / 42.5457.98 / 70.0032.78 / 39.47

The stark gap between QA and grounding persists on matched-target/event triplets (e.g., Gemini-3.8-Flash scores 66.09 QA vs. 28.55 TG and 30.88 SG on VL; Qwen3.8-Omni-Flash scores 86.49 QA vs. 16.70 TG on AV). Across all 36 VL and 17 AV models, Spearman rank correlations between the full set and this subset reach 0.9918 and 0.9755, with Top-6 rankings 100% unchanged. We will release these triplet mappings with the dataset.

2. Comparison with NExT-GQA, VideoZeroBench, and CaST-Bench:

BenchmarkTarget Spec.EvidenceTasksAvg. Len. (Scope)#Vid. / #Inst.
NExT-GQATextVisualQA + Temp. Evid.39.5s (Mainly Short)990 / 5,553
VideoZeroBenchTextVisual + Audio (part.)QA + Spatio-Temp. Verif.667.1s (Mainly Long)138 / 500
CaST-BenchTextVisualCausal QA + Evid. Chains13.68s (Short)1,015 / 2,061
OmniCompass-VLPaired Text / Text+ImageVisualTG, SG, QA354.0s (Short/Med/Long)820 / 2,460
OmniCompass-AVPaired Text / Text+ImageVisual + AudioTG, SG, QA427.5s (Short/Med/Long)240 / 720

Unlike prior benchmarks that rely on single-modality text queries or focus on QA with evidence verification, OmniCompass uniquely unifies paired Text / Text+Image target specifications, both VL and native AV tracks, and complete manual TG, SG, and QA annotations across 30s–88.4min videos.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.