acceptodds
Under review as a conference paper at ICLR 2027

Corrupted Signal, Silent Verdict

Abstract

AV evaluation benchmarks construct negatives by running source clips through an editing pipeline and use unedited originals as clean controls, so negatives and controls differ in both acoustic content and processing history. We introduce a register-only control: a clip that traverses the full pipeline — encode, decode, re-encode at matching settings — but receives no acoustic corruption. A commercial API judge flags register-only clips above clean ( pp, , ) yet flags corrupted clips (four of five types) no higher than register-only (). Verdicts from a local judge, Qwen2.5-Omni-3B, do not distinguish the three conditions (clean , ; register-only ; corrupted ; all ), and its continuous decision logit is near chance (corrupted vs. register-only AUC  [, ]), while linear probes on the same model's hidden states separate matched items at – (AUC –) across all five corruption types. Across both judges, verdicts track processing provenance or nothing — not the acoustic manipulation — while the manipulation is linearly decodable from the representations. Benchmarks of this construction should report clean false-alarm rates and include a register-only arm.

Primary area:
interpretability and explainable AI
Keywords:
audio-visual evaluation, MLLM-as-a-judge, linear probing, signal detection theory, representation-behavior gap, processing-matched controls, omni-modal models, probing methodology, benchmark construction, Qwen2.5-Omni

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_stwkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is very difficult to follow and understand.

But seems like it investigates whether AV LLMs detect acoustic corruptions or artifacts introduced by the editing pipeline. They are using AVE dataset and 5 corruptions (muted, hallucinated, room mismatch, ambience and foley swap) and Qwen2.5 Omni and Gemini API. to evaluate it. Their setup and conclusion looks unclear. However it seems like the AV LLM judges are not reliably detecting actual audio-visual inconsistencies.

Critical strengths

The problem seems to be interesting and valid. The artifacts introduced during benchmark construction may introduce biasness in the evaluation of audio-visual LLMs.

Critical weaknesses
  1. Clarity: The paper is very difficult to follow. The motivation, experimental setup and outcome is not clearly written.
  2. Undefined terminology and insufficient explanations: Several symbols, abbreviations and experimental details are not sufficiently defined. For eg. "manifest_v5.jsonl", "speech_content_swap", "speech_desync". What does register-only mean? etc.
  3. Limited experiments: Would other AVLLMs (other variants of Qwen, Gemma, etc.) give same conclusion? Can we have more dataset (how about VGGsound, audioset etc.) and corruption diversity ?
Questions for authors

Please refer to the weakness section.

Reviewer_FKeMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that AV judge benchmarks confound acoustic corruption with processing history, since corrupted clips go through an editing pipeline and clean controls do not. The authors add a register only control and test two judges, Gemini 3.5 Flash Lite and Qwen2.5 Omni 3B.

Gemini flags register only clips more than clean ones but does not flag corrupted clips more than register only ones. Qwen's verdicts and logit do not separate any condition, yet linear probes on its hidden states separate corrupted from register only items for all five corruption types. The paper concludes that verdicts follow provenance rather than content, and recommends that benchmarks report clean false alarm rates and include a register only arm.

Critical strengths
  1. The register only control is a simple and sensible addition to benchmark design. The paper also shows that the choice of control is not neutral; for room and foley, the C1 and C2 baselines give clearly different probe estimates (Table 4). This is a good argument for actually running such controls rather than assuming their effect, and it is the main reason I see value in the paper despite my concerns below.

  2. The paper withdraws corruption types when its own gates fire, reports how layer selection inflates the readout gap, labels inconclusive types as inconclusive, and reports a pre registered hypothesis that its own data falsified. That honesty makes the work easier to trust and build on, and makes the weaknesses below easier to identify and fix.

Critical weaknesses
  1. The control does not isolate provenance. For room, foley and ambience, the register only arm still alters the audio (matched room impulse response, substituted foley, spliced ambience), and the hallucinated configuration is not described. Only muted (0 dB gain) is clearly a pure pipeline pass, and no perceptual or physical check is reported for these clips. Table 4 supports this reading: since C1 already uses the same AAC pipeline, the positive Δd′\Delta d' for room and foley points to an acoustic difference rather than a codec footprint. These are the two types whose register only arm applies a real acoustic change, while muted sits at −0.07-0.07. The direct test, running the judges on the C1 arm (n=498n = 498), is not reported. As provenance is the central claim, this drives my soundness score.

  2. The headline result is fragile and of unclear relevance. The claim rests on one contrast (Reg minus Clean, p=0.014p = 0.014). The p values in Table 3 match a pooled test while its CIs are unpooled. With the unpooled SE, p=0.023p = 0.023, and after correcting for the three J1 contrasts it is 0.0700.070. The J1 "Reg minus Corr" CI is also sign flipped (it should be [−3.2,+8.2][-3.2, +8.2]). The Appendix G checks do not fix this: the mixed model uses a type random effect (k=5k = 5) rather than a source clip one, and the sign test's five types share one clean denominator. The ambience session gap (6.9 pp) is also larger than the headline effect (5.0 pp). Finally, both judges are near non detectors (Gemini 9.3% hits vs 6.9% false alarms; Qwen flags 72% of clean clips), and no existing benchmark is tested, so the recommendations rest on analogy.

in general i would say the idea is useful, but the central claim needs new experiments rather than revisions to the text.

Questions for authors
  1. What are the Gemini and Qwen flag rates on the C1 arm (n=498n = 498)? This directly tests the provenance interpretation.
  2. Can you report a physical distance (CLAP or spectral) and a small listening test between register only clips and originals for each type? Does the muted only elevation in Table 7 hold with more items?
  3. Were the clean, register only and corrupted Gemini arms collected in the same API session? If not, the reason for excluding ambience applies to the headline contrast too.
  4. Does the clean arm (n=582n = 582, more than the 308 AVE source clips) include VoxCeleb2 clips from withdrawn types? If so, it is not source matched to the corrupted items.
  5. Does the effect hold with source clip random effects, and with at least one judge that detects the corruptions above chance?

Testing on a public benchmark such as AV-SyncBench would strengthen the paper but is not required for my score.

Reviewer_odmkOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes "register-only control" for AV evaluation benchmarks, i.e. a clip that has traversed the full re-encode/decode pipeline but without acoustic corruption.

Critical strengths

I'm not sure I can find clear strengths. It may have to do with bad writing: paper focuses too much on technical fluff, without taking time to explain why the work matters. The main contribution sounds reasonable but it is also very obvious and it is not clear to me why was it necessary in the first place.

Critical weaknesses

I don't think the paper has enough contribution novelty to justify publication to ICLR. The idea of "register-only control" may be fine for a smaller conference dedicated to the topic of AV evaluation, but it is far too simple and straightforward contribution for a major conference.

Furthermore, the paper is currently not clearly written and is not in a state publishable by ICLR. The paper reads as a (not very well written) tech report rather than a self-contained, well motivated paper with clear scientific impact. Some examples: different, not intuitive abbreviations (C1, J1), references to code files (manifest_v5.jsonl, PREREGISTRATION_CAUSAL.md), functions / variables, etc.

Questions for authors

Authors need to make a clear case for why this work matters to a broader scientific community. Right now the work lacks a clearly formulated story: why does this problem exists? why nobody solved it before? Show examples where the lack of "register-only control" caused issues. I'm really not convinced that this problem existed in the first place and authors do not make a clear case for it nether.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.