- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces OmniCompass, a benchmark combining temporal grounding, spatial grounding, and question answering under paired textual and visual target specifications. The visual track contains 820 videos and 2,460 task instances, while an audio-visual track uses a 240-video subset with 720 separately constructed instances. Cross-Specification Consistency scores reward correctness across both specifications and, for grounding, agreement between predictions. Evaluation of 50 models, together with frame-budget, timestamp, modality, and reference-type analyses, identifies substantial grounding limitations that aggregate QA accuracy alone would obscure.
Critical strengths
-
The paired design tests a useful form of robustness: whether the same target and answer remain recoverable when the identifying cue changes. Appendix B.3 provides a concrete example: Kimi-K3 has nearly identical QA accuracies under the two specifications, 73.0% and 72.9%, but only 67.6% joint correctness. This makes the benchmark informative beyond a conventional single-prompt leaderboard.
-
The benchmark combines substantial manual verification with task-specific analysis. The annotation and repair records in Section 3.2 and Tables D1–D2 support the care taken in construction. The audio ablation in Figure 6 is particularly useful: adding audio improves paired QA scores by 18.6–28.1 percentage points across the four tested models, providing empirical support for the audio component rather than relying solely on the task taxonomy.
Critical weaknesses
-
The evaluation denominator is not specified sufficiently clearly. Appendix A uses model-specific or shared “eligible pairs,” and Figure A3 reports shared subsets of 592 TG, 673 SG, and 689 QA pairs, compared with 820 nominal instances per task. Appendix B.3 reports 808 QA pairs for Kimi-K3. The supplied evaluation protocol does not clearly explain these exclusions or the handling of failed and unparseable outputs. This matters because model-dependent exclusions could affect both rankings and paired consistency scores; the paper needs to distinguish operational coverage from accuracy on successful responses.
-
The evidence supports robustness across two specifications more directly than it isolates a distinct consistency failure. CS combines correctness and agreement, and Appendix B.3 appropriately notes that the extra prediction-overlap constraint accounts for only 0.3–1.8 points in grounding. In addition, replacing a textual description with an image changes the identifying information, and the experiments use default model settings without a reported same-specification repeatability control. Consequently, the observed differences cannot yet be cleanly separated into specification sensitivity, unequal cue informativeness, and ordinary response variability. This limits the diagnostic interpretation, although it does not negate the benchmark's utility.
Questions for authors
-
Clarification: What exactly makes a pair eligible, how many pairs enter each main-table score, and how are API failures and malformed answers scored? Would counting all unsuccessful outputs against a fixed denominator change the main rankings?
-
Clarification: Do the existing runs contain repeated predictions under an unchanged specification? If so, how does that variability compare with the Text versus Text+Image differences? If not, can the consistency claims be scoped explicitly to the observed paired-setting robustness?
5 comments
Response to Reviewer zr1n (W1 Part 2: AV Denominators & W2)
Response to Weakness 1 (Continued): AV Denominators & Shared Subsets
OmniCompass-AV Actual Denominators (Nominal = 240; Format: T / TI / CS)
| Model | TG (T / TI / CS) | SG (T / TI / CS) | QA (T / TI / CS) |
|---|---|---|---|
| Gemini-3.8-Flash | 239 / 239 / 239 | 239 / 239 / 239 | 239 / 239 / 239 |
| Gemini-3.1-Pro | 239 / 239 / 239 | 239 / 239 / 239 | 239 / 239 / 239 |
| Qwen3.8-Omni-Flash | 228 / 228 / 228 | 228 / 227 / 227 | 228 / 228 / 228 |
| Qwen3.5-Omni-Plus | 228 / 228 / 228 | 228 / 227 / 227 | 228 / 228 / 228 |
| Qwen3.5-Omni-Flash | 240 / 240 / 240 | 240 / 238 / 238 | 239 / 239 / 238 |
| Seed-2.0-Lite-260428 | 240 / 240 / 240 | 240 / 240 / 240 | 240 / 240 / 240 |
| Seed-2.0-Mini-260428 | 240 / 240 / 240 | 240 / 240 / 240 | 240 / 240 / 240 |
| MiMo-V2.5 | 206 / 187 / 184 | 219 / 208 / 207 | 191 / 201 / 188 |
| Qwen3-Omni-30B-A3B | 240 / 240 / 240 | 240 / 240 / 240 | 240 / 240 / 240 |
| Qwen2.5-Omni-7B | 240 / 240 / 240 | 240 / 240 / 240 | 240 / 240 / 240 |
| Qwen2.5-Omni-3B | 240 / 240 / 240 | 240 / 240 / 240 | 240 / 240 / 240 |
| MiniCPM-o-4.5-8B | 240 / 240 / 240 | 240 / 240 / 240 | 240 / 240 / 240 |
| video-SALMONN 2+ 72B | 240 / 240 / 240 | 240 / 240 / 240 | 240 / 240 / 240 |
| video-SALMONN 2+ 7B | 240 / 240 / 240 | 240 / 240 / 240 | 240 / 240 / 240 |
| Ola-7B | 240 / 240 / 240 | 240 / 240 / 240 | 240 / 240 / 240 |
| VideoLLaMA2.1-7B-AV | 240 / 240 / 240 | 240 / 240 / 240 | 240 / 240 / 240 |
| VITA-1.5-7B | 240 / 240 / 240 | 240 / 240 / 240 | 240 / 240 / 240 |
Most models—including all locally hosted open-source models and the Seed and GPT-5.6 series—achieve 100% operational coverage (820/820 and 240/240), while a few APIs have minor safety-filter or timeout exclusions (e.g., Kimi-K3 has 808/808/808 on QA and 807/807/806 on SG due to slightly differing safety filters across branches). The 592, 673, and 689 pairs in Figure A3 represent the strict common intersection across the 10 compared models so cross-model comparisons use identical items. We will release per-item status flags and operational coverage alongside conditional accuracy in our benchmark release.
Response to Weakness 2: Scoping Claims to Paired-Setting Robustness & Separating Variability
Thank you for this precise conceptual framing. We fully agree and have updated the paper to scope our core claim explicitly as paired-setting robustness across target specifications, rather than an isolated "pure consistency" defect. Specifically, we disentangle the three contributing factors:
- Separating Correctness, Agreement, and CS: Reporting single-setting accuracy, raw prediction Agreement, and CS side-by-side clarifies error sources. In grounding, while the prediction-overlap term () adds a 0.3–1.8 point strictness penalty on aggregate averages, penalizes instance-level flips across specifications (making CS 4.3–4.6 points lower than the weaker single-setting mean). In QA, for Kimi-K3 (: 546 both-correct, 44 Text-only correct, 43 Text+Image-only correct, 175 both-wrong), (67.57%) prevents the 21.66% (175/808) shared-error cases from being rewarded as reliable agreement.
- Real-World Cue Modality Shift: Because Text and Text+Image convey target identity via verbal description versus visual reference image, their difference reflects task retention under realistic specification shifts (verbal vs. visual pointer) rather than a pure image-addition ablation.
- Isolating Repeatability Noise via Three-Run Controls: Using three independent runs of Gemini-3.8-Flash under identical inputs (detailed in Question 2 below), macro run-to-run variance is minimal (1.33, 1.79, 1.11 points on VL; 1.52, 2.45, 1.04 on AV). Subtracting this same-specification repeatability baseline on TG (), cross-specification divergence still exceeds same-specification run-to-run divergence by +9.39 points on VL and +5.58 points on AV, proving genuine specification sensitivity beyond stochastic generation noise.
Response to Reviewer zr1n (Q1 & Q2)
Response to Question 1: Fixed-Denominator Sensitivity Analysis
Thank you for this question. As detailed in Weakness 1, an item enters a model's paired denominator when both Text and Text+Image return valid responses (with completed-but-empty/unparseable outputs retained and scored 0).
To test ranking sensitivity, we recomputed overall across all 36 VL and 17 AV models using the fixed nominal denominator (820 VL / 240 AV), scoring all safety-filter refusals and transport timeouts as 0:
| Track | Models | Full-Leaderboard Spearman | Top-5 Models & Relative Order |
|---|---|---|---|
| OmniCompass-VL | 36 | 0.9928 | 100% unchanged (Gemini-3.8-Flash #1) |
| OmniCompass-AV | 17 | 0.9926 | 100% unchanged (Gemini-3.8-Flash #1) |
OmniCompass-VL (Top-5 Fixed-Denominator Check)
| Model | Conditional | Fixed-Denom. | Cond. Rank | Fixed Rank |
|---|---|---|---|---|
| Gemini-3.8-Flash | 42.84 | 42.78 | #1 | #1 |
| Seed-2.1-Pro-260915 | 42.02 | 42.02 | #2 | #2 |
| Seed-2.1-Turbo-260628 | 40.91 | 40.91 | #3 | #3 |
| Seed-2.0-Pro-260215 | 40.52 | 40.52 | #4 | #4 |
| Qwen3.8-Max | 40.22 | 39.63 | #5 | #5 |
OmniCompass-AV (Top-5 Fixed-Denominator Check)
| Model | Conditional | Fixed-Denom. | Cond. Rank | Fixed Rank |
|---|---|---|---|---|
| Gemini-3.8-Flash | 57.45 | 57.21 | #1 | #1 |
| Qwen3.8-Omni-Flash | 48.64 | 46.14 | #2 | #2 |
| Qwen3.5-Omni-Plus | 44.32 | 42.03 | #3 | #3 |
| Seed-2.0-Lite-260428 | 39.94 | 39.94 | #4 | #4 |
| Gemini-3.1-Pro | 39.87 | 39.71 | #5 | #5 |
Across the full leaderboard, 19/36 VL and 14/17 AV models have zero rank change (all 100%-coverage models have identical scores), while minor shifts occur only for APIs with higher refusal/timeout rates (max shift: 4 places on VL for GPT-6-Astra #7$\rightarrow$#11 and GPT-5.5 #12$\rightarrow$#16; 2 places on AV for MiMo-V2.5 #7$\rightarrow$#9). For Kimi-K3 on VL QA, is 67.57% (546/808) conditionally versus 66.59% (546/820) fixed (-0.98 pp). Reporting both metrics jointly captures intrinsic capability and operational availability.
Response to Question 2: Same-Specification Repeatability & Scoping Claims
Thank you for this constructive suggestion. We fixed a stratified 64-video representative subset in each track (VL covering all 7 domains, short/medium/long durations, and all 8 VL QA types; AV covering movies, TV dramas, variety shows, news, and all 5 AV QA types) and ran three independent repeated evaluations of Gemini-3.8-Flash under unchanged inputs (identical 64 timestamped frames, full audio for AV, reference images, prompts, and temperature=0; scores report ):
OmniCompass-VL (Three Repeated Runs)
| Model | Task | Run 1 | Run 2 | Run 3 | Mean Pairwise Diff. |
|---|---|---|---|---|---|
| Gemini-3.8-Flash | TG | 18.58 | 18.60 | 16.60 | 1.33 |
| Gemini-3.8-Flash | SG | 30.63 | 30.67 | 33.32 | 1.79 |
| Gemini-3.8-Flash | QA | 65.00 | 63.33 | 63.33 | 1.11 |
OmniCompass-AV (Three Repeated Runs)
| Model | Task | Run 1 | Run 2 | Run 3 | Mean Pairwise Diff. |
|---|---|---|---|---|---|
| Gemini-3.8-Flash | TG | 32.86 | 30.72 | 30.58 | 1.52 |
| Gemini-3.8-Flash | SG | 42.52 | 41.46 | 45.13 | 2.45 |
| Gemini-3.8-Flash | QA | 75.00 | 73.44 | 73.44 | 1.04 |
Macro run-to-run variability is minimal (mean pairwise differences of 1.33, 1.79, 1.11 on VL and 1.52, 2.45, 1.04 on AV). Comparing instance-level TG divergence () across the exact same videos and runs, cross-specification divergence (Text vs. Text+Image) exceeds same-specification run-to-run divergence by +9.39 points on VL and +5.58 points on AV.
Accordingly, we adopt both parts of your recommendation: we explicitly scope our core claim to paired-setting robustness across realistic target-specification shifts, while demonstrating quantitatively that cross-specification variation substantially exceeds ordinary repeatability noise.
Response to Reviewer zr1n (W1 Part 1: Protocol & VL Denominators)
Thank you for your recognition of our work—specifically our paired target-specification design that reveals instance-level robustness beyond single-prompt leaderboards, and our substantial manual verification alongside empirical audio-ablation analyses—and for raising constructive questions. Your feedback has helped us improve the transparency and precision of our paper. Below, we provide point-by-point responses to your comments.
Response to Weakness 1: Evaluation Denominators & Operational Coverage
Thank you for this important suggestion. We have audited all evaluation logs and explicitly separated operational coverage (service availability) from conditional accuracy on valid responses.
Nominal task sizes are 820 per VL task and 240 per AV task. We strictly distinguish model capability failures from external API/network interruptions: whenever an API returns a response—even if empty, malformed, or truncated—it remains in the denominator and receives 0 (penalizing instruction-following and formatting failures). Only platform safety-filter refusals or network/transport timeouts are excluded from the single-setting conditional denominator, and an item enters the paired CS denominator when both Text and Text+Image receive valid responses:
OmniCompass-VL Actual Denominators (Nominal = 820; Format: T / TI / CS)