Corrupted Signal, Silent Verdict
Abstract
AV evaluation benchmarks construct negatives by running source clips through an editing pipeline and use unedited originals as clean controls, so negatives and controls differ in both acoustic content and processing history. We introduce a register-only control: a clip that traverses the full pipeline — encode, decode, re-encode at matching settings — but receives no acoustic corruption. A commercial API judge flags register-only clips above clean ( pp, , ) yet flags corrupted clips (four of five types) no higher than register-only (). Verdicts from a local judge, Qwen2.5-Omni-3B, do not distinguish the three conditions (clean , ; register-only ; corrupted ; all ), and its continuous decision logit is near chance (corrupted vs. register-only AUC [, ]), while linear probes on the same model's hidden states separate matched items at – (AUC –) across all five corruption types. Across both judges, verdicts track processing provenance or nothing — not the acoustic manipulation — while the manipulation is linearly decodable from the representations. Benchmarks of this construction should report clean false-alarm rates and include a register-only arm.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.