acceptodds
Under review as a conference paper at ICLR 2027

CabinBench: Benchmarking What Long-Audio Agents Actually Heard

Abstract

Audio understanding has advanced on clips of seconds, where the answer and its evidence arrive in the same model call. Long recordings break that pairing: the evidence for a question may last three seconds inside an hour, so a correct answer may have been heard or guessed, and benchmarks that hand the model the whole recording and score only the answer cannot say which. A vehicle cabin is a natural testbed: over a trip of tens of minutes, several occupants talk over road noise, phone calls and navigation prompts cut in, and plans change, so understanding the scene means finding, tracking, and attributing evidence. CabinBench is, to our knowledge, the first benchmark that scores the listening process itself rather than the answer alone: the agent decides where to listen under a budget, must cite audio for every claim, and is credited only for citations that the harness’s delivery log confirms. Its recordings are synthesized from fictional scripts by design, which makes the ground truth fully controllable: every question carries an exact evidence contract, and any sentence can be removed, replaced, or moved to produce a counterfactual recording whose truth is known. On 200 recordings and 2,608 questions, a 30B-parameter open omni-modal model answers 16 to 20% of questions yet establishes its evidence in under 1% of runs; most of its correct answers never heard the sentence that settles the question, and a tenth of its submissions cite audio it was never given. Controllable data, an agent in the loop, and process-level scoring turn listening from an assumption into a measurement.

Primary area:
datasets and benchmarks
Keywords:
long-form audio, audio agents, benchmark, counterfactual evaluation, evidence grounding, answer abstention, multimodal agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bGw7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CabinBench, which jointly evaluates answers, evidence citations, and records of agents' actual audio access in long-audio question answering. The dataset consists of synthesized Chinese conversations set in a vehicle cabin. Agents select audio segments within a listening budget and are evaluated against a predefined evidence contract. The experiments examine gaps between answering correctly, accessing relevant evidence, and citing it correctly, as well as the effects of the listening budget on search and abstention behavior.

Critical strengths
  1. The use of harness logs to verify audio delivery makes it possible to identify citations to segments an agent did not access. This provides an auditable basis for studying how long-audio agents obtain evidence.

  2. The paper breaks down correct answers according to whether the relevant evidence was accessed and cited, and reports failure stages and search trajectories. These analyses offer more insight into where systems fail than an ECS score alone.

  3. Preserving sentence, speaker, and timing information during synthesis makes it feasible to define evidence and introduce controlled modifications. The analyses using stricter answer matching and excluding items with option bias also provide useful checks on the robustness of the main findings.

Critical weaknesses
  1. One concern is that ECS may be overly sensitive to citation boundaries. Under the stated formula, citations to required counter-evidence must fully satisfy both temporal matching and speaker attribution, so a citation extending slightly beyond the correct sentence may fail. An oracle-listen control that supplies the correct evidence segments, together with sensitivity analyses of boundary tolerance and scoring thresholds, would help clarify whether low scores primarily reflect difficulties in understanding the evidence or sensitivity to temporal precision.

  2. The current unanswerable questions all involve missing information, and correct abstention does not require any minimum search coverage. A fixed response stating the corresponding abstention reason could therefore pass these items without examining the audio. This limits what the current evaluation can tell us about evidence-based abstention after search. Including potentially confusable cases involving missing and conflicting information, and specifying verifiable search requirements for abstention, would help address this limitation.

  3. A correct answer may still be supported by audio the agent has accessed, even if the designated gold sentence was not among the segments it heard. The appendix's checks for answer strings and aliases are useful, but they may miss paraphrases or indirect inferences. Human reassessment of these responses against the audio actually accessed would help determine whether the answers are supported. Paired testing of the existing counterfactual siblings would also provide a more direct check of whether answers change when the critical evidence changes.

  4. The informative behavioral findings come mainly from one Qwen agent, while the other model largely fails to complete the protocol or chooses to abstain. Including an agent from another model family, along with a simple sequential-scanning strategy using the same backbone, would help distinguish the effects of model capability, search strategy, and protocol adaptation.

Questions for authors

I would appreciate the authors' clarification of the points raised in the weaknesses above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer bGw7

Thank you for your supportive review and insightful analysis. Your observations on citation boundary sensitivity, abstention coverage requirements, alternative supporting evidence, and backbone versus strategy confounds were spot-on, and we designed targeted experiments to test each one.

[W1] Oracle-listen control and citation boundary sensitivity

We ran a Sentence Oracle across all 2,406 answerable questions that supplies the gold sentence with ±2 seconds of context. Giving the agent the exact clip raises answer accuracy from 15.8% to 81.59%, confirming that low autonomous scores stem from failing to locate the evidence rather than failing to understand the speech.

Strict ECS under the sentence oracle is 6.15% on 148 of 2,406 items, and 79.98% of the failures occur on 1,806 runs that answered correctly but scored below the 0.8 temporal support threshold. Because the oracle clip includes ±2 seconds of padding, its median duration is 9.22 seconds compared to 5.20 seconds for the tight gold sentence; simply echoing the delivered window fails the 0.8 F1 gate in 92.2% of cases. Sensitivity sweeps on these oracle submissions confirm your hypothesis:

Scoring Variant on Sentence OracleAnswerable ECSGain vs Strict
Frozen v0.1 Strict Scorer6.15%—
Boundary Tolerance ±1 s22.49%+16.34 pp
Boundary Tolerance ±2 s25.44%+19.29 pp
Support Threshold F1 ≥ 0.627.56%+21.41 pp
Joint Relaxation40.65%+34.50 pp

In our pre-delivered Fixed setup—where exact gold spans are provided without extra padding—strict answerable ECS reaches 43.93% for Qwen and 22.90% for Ming. We keep the v0.1 scorer frozen for reproducibility while reporting these tolerance curves alongside strict ECS.

[W2] Abstention coverage requirements and unanswerable types

Your observation is spot-on. Because v0.1 absent contracts check the refusal reason without enforcing a minimum search coverage floor, a dummy policy that immediately abstains with missing_evidence at zero audio cost passes all 202 absent items, earning 7.75% overall ECS across the 2,608 questions. While no evaluated model used zero-listen abstention, separating answerable ECS from unanswerable abstention prevents passive refusal from inflating overall scores. Moreover, our counterfactual Remove branch tests abstention after search: when the decisive sentence is spliced out of 9 recordings, Qwen3-Omni abstains on only 1 of 9 items, showing strong resistance to abstaining once surrounding context is heard. We will report answerable and unanswerable metrics separately and enforce a minimum coverage gate in the next release.

[W3] Alternative heard evidence and paired counterfactual siblings

We tested both mechanisms directly:

  • Non-gold reassessment: Across 1,333 correct runs from five setups, 756 lacked complete delivery of the designated gold set. Auditing 32 keyword-enriched submissions against the delivered audio showed that 16 had fully sufficient alternative evidence later in the dialogue, especially when speakers repeated a confirmed plan, 12 had partial clues, and 4 were unsupported guesses. Auditing 6 human-sufficient runs that failed only the support gate uncovered 2 valid unindexed alternative spans.
  • Counterfactual siblings: Across 9 counterfactual families and 35 runs, replacing the decisive spoken fact causes the agent to update its answer to the new value on 6 of 8 verified edits (75.0%) with 75.0% citation overlap. Moving the decisive sentence to a new timestamp yields 6 of 7 correct answers (85.7%) and shifts citations to overlap the new measured position in 3 of 7 cases (42.9%).

[W4, Q1] Disentangling model capability, search strategy, and protocol adaptation

We separated backbone capability, search strategy, and tool formatting across three controls:

  • Protocol vs. comprehension: In autonomous search, Ming reaches 1.95% accuracy with 62.00% valid submissions. Pre-delivering gold segments in one turn lifts valid submissions to 90.27%, answer accuracy to 34.16%, and strict ECS to 22.90%.
  • Search policy: On Qwen3-Omni across 7,824 runs, Sequential and Uniform schedules lift raw accuracy from 16.04% to 22.82% and 24.02%, overcoming the agent's bias toward the first five minutes.
  • Modality interface: Passing the exact same uniform intervals through Qwen3-ASR-1.7B into the Qwen3-30B text backbone adds 6.21 percentage points over native audio, reaching 30.62% versus 24.41%.
Reviewer_iSB9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CabinBench to evaluate whether long-audio agents can find and cite evidence rather than merely produce correct answers. The benchmark contains 200 synthetic Mandarin in-cabin recordings and 2,608 questions. Agents request audio segments under a listening budget and submit answers with timestamped citations. The evaluator checks answer correctness, evidence coverage, speaker/source attribution, and whether the cited audio was actually delivered. Experiments on two Qwen models show a large gap between answer accuracy and evidence-constrained success.

Critical strengths
  1. Predicting transcription failures before decoding is a useful goal. The predictor is lightweight and does not modify the underlying Audio LLM.
  2. The results show clear gains over the evaluated baselines. Cross-domain evaluation and cross-model label transfer provide useful additional evidence.
Critical weaknesses
  1. The model evaluation is too limited. Only two models from the Qwen family are tested, and Qwen2.5 largely fails to complete the protocol. Stronger models and simple search baselines are needed to establish the benchmark’s difficulty and usefulness.
  2. The paper does not test whether models can answer correctly when directly given the relevant audio clips. This would help show whether the low scores come from difficulty finding the evidence or understanding it.
  3. Audio-side validation is insufficiently established. Questions and evidence contracts are derived from scripts, while timestamps come from forced alignment. Model-based event review does not fully establish that the final audio contains all required words, speaker identities, and source cues. A human audit of answerability and evidence alignment would strengthen confidence in the dataset.
Questions for authors
  1. How well do models perform when given the gold evidence segments? Which failures remain when search is removed?

  2. Can you include a broader model comparison and simple sequential or uniformly sampled listening baselines under the same budget?

  3. Has a human audit checked the final audio, speaker/source attribution, and evidence timestamps? What are the observed error rates?

1 comment
AuthorsOpenReview ↗

Response to Reviewer iSB9

Thank you for your constructive review. Just to clarify a quick mix-up in the strength comments: CabinBench evaluates long-audio evidence-seeking agents rather than transcription-failure prediction. Your questions on model diversity, gold-evidence controls, and audio validation go straight to the core of our benchmark, and we address each with new experiments below.

[W1, Q2] Broader model comparison and simple listening baselines

We expanded our evaluation with the Ming omni-modal family, an ASR+LLM cascade, and two non-agentic listening schedules—Sequential scanning from the start and Uniform interval sampling—under the 0.25T budget across all 2,608 questions:

System / StrategyRaw AnswerOfficial BoundAnswerable ECSValid Sub.
Qwen3-Omni Agentic16.04%14.59%0.17%99.85%
Qwen3-Omni Sequential22.82%8.02%0.00%97.66%
Qwen3-Omni Uniform24.02%14.34%0.04%98.43%
ASR + Text-LLM Uniform30.63%——100.00%
Ming Agentic Search2.66%1.95%0.37%62.00%
Ming Fixed Pre-delivery34.16%34.16%22.90%90.27%

This comparison highlights a key limitation of current audio agents. In autonomous mode, Qwen3-Omni clusters its listens near the start of the recording, hearing a median of only 16% of the audio. Simple Sequential and Uniform schedules force broader coverage across the timeline, lifting raw answer accuracy from 16.04% to 22.82% and 24.02%. Meanwhile, Ming struggles to sustain multi-turn JSON tool calls during autonomous search, completing valid submissions on 62.00% of runs, but jumps to 90.27% valid submissions and 34.16% accuracy when gold segments are pre-delivered. Converting the uniformly sampled audio into text via Qwen3-ASR-1.7B adds another 6.21 percentage points on the 2,368 strictly matched items, showing that native audio reasoning and multi-turn tool control each introduce distinct bottlenecks.

[W2, Q1] Performance when directly given the gold evidence segments

To separate search failures from comprehension failures, we evaluated Qwen3-Omni across all 2,406 answerable questions under a Window Oracle providing a 286-second surrounding segment, and a Sentence Oracle providing the exact gold sentence with 2 seconds of padding on each side.

Delivering the gold window lifts answer accuracy from 15.8% to 47.1%, and delivering the gold sentence raises accuracy to 81.59% with 1,963 of 2,406 questions answered correctly. Models clearly understand the cabin dialogue once handed the relevant clip; finding that clip in a two-hour recording is the primary bottleneck. Among the 2,258 runs that miss strict ECS under the sentence oracle, we traced the exact first failing gate:

First Failing Gate under Sentence OracleRunsShare
Correct answer, temporal/attribution support < 0.81,80679.98%
Incorrect answer or claim value41118.20%
False abstention on answerable item271.20%
Missing required counter-evidence90.40%
Malformed or rejected submission50.22%

Nearly 80% of remaining oracle failures get the answer right but miss the strict 0.8 temporal F1 threshold because the delivered clip includes ±2 seconds of padding—making it 1.77 times the gold sentence length—and models tend to echo the padded window rather than trimming to exact utterance boundaries. Relaxing boundary tolerance by 2 seconds raises oracle ECS from 6.15% to 25.44%, and joint threshold relaxation reaches 40.65%.

[W3, Q3] Human and reference audits of the final audio

We conducted targeted audits across the rendered waveforms and contracts to verify dataset fidelity:

  • Audio intelligibility: Across 356 audited clips, independent audio evaluations agreed with reference intelligibility labels at 97.5% and 96.9%, finding only 3.1% masked by background noise.
  • Evidence sufficiency and timestamps: Of 34 sufficiency disputes, 20 involved multi-span questions where a single span is only half the puzzle; among the 14 single-span clips, 7 needed wider context. In a check of 134 clips for forced-alignment drift, dual passes flagged only 1 shared instance where the window caught an adjacent utterance.
  • Unanswerable items and synthesis artifacts: Re-auditing all 202 absent questions confirmed 170 as strictly unanswerable, with 28 borderline cases where a related topic is mentioned without the requested detail, and only 3 items where the answer could be inferred. Scanning all 67,752 turns for TTS stage-direction leakage identified just 11 affected turns across 10 recordings; excluding them shifts model accuracy by at most 0.4 percentage points.
Reviewer_cN5ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper presents a new benchmark that evaluates whether an Omni / Audio LLMs actually listens carefully at the content of a long conversation and is not merely arriving at an answer either by guessing or influenced, biased on earlier statements.

The dataset is built purely by synthesizing different scripts being played out in a fake car trip environment with corresponding noise characteristics. The scripts are generated using Deepseek, noise characteristics generated with Stable Audio 3 and the speech rendered using MOSS-TTSD via different speaker references sampled from AISHELL-5. A total of 200 synthesized continuous recordings, each in Mandarin with 2608 questions designed around the clips.

The benchmark itself tests not only for whether the answer is correct but also cite the interval in the audio that carries the content within the specified budget along with speaker and source type. It also evaluates for counter-evidence that was superseded.

The benchmark operates an agentic loop that "listen" - picks an audio clip interval within the long recording via tool-call, "record" - writes the findings, corresponding interval and the status (supported, superseded, confirmed) and "submits" either the answer (along with corresponding clip id, interval, support, superseded status) or abstains.

The primary metric is defined as "Evidence Constrained Success" which credits only when answer is correct, correct interval is cited along with correct speaker, citation delivered to model and is within the budget. Two candidate omni models Qwen2.5 and Qwen3 are evaluated with the benchmark. The results show glaring gaps between answering and listening.

Critical strengths

The paper touches on a clear and well motivated issue that plagues most multi-modal LLMs especially on long recordings based on the premise that a correct answer is a weak evidence that the answer is grounded in the audio. The formulation of the benchmark is interesting and the metrics defined are well positioned to probe candidate LLMs. Harness design is principled to provide diagnostics on failure modes of the models.

Critical weaknesses

The paper's use of the term "listening" is vague and convoluted in it's use in different context. Claims such as "most correct answers never heard the sentence" and "listening is now a measurement" should be restated and clarified.

Some concepts although discussed were never tested or quantified: Counterfactual siblings, shared-index and open-system tracks, the ambiguous and conflicted unanswerable types.

Experiments are run on a single model family, Qwen3-Omni and Qwen2.5 Omni both ranking low. Missing references or baselines including (1) oracle upper bound (acknowledged by authors), (2) a cascaded system - ASR + text-LLM, (3) popular long audio models (Gemini 3 Pro, GPT-4o etc), (4) human reference baseline.

Lack of human validation on the benchmark dataset (every step is model/data driven) is a concern and raises questions on the relevance to real-world scenarios.

Currently, paper claims ECS measures "listening" better - but that claim lacks evidence that ECS tracks human judgements of perception/grounding. This needs a human study to determine human raters blind to ECS reach same ratings as delivered by ECS.

ECS <= 0.9% for every evaluated model so the benchmark cannot rank models meaningfully and the human correlation cannot be inferred without more diverse systems.

Questions for authors

Do we have any human validity study of the scoring? i.e., human agreement with the evidence contracts, human judgments on the correlation of submissions being grounded against ECS metrics?

1 comment
AuthorsOpenReview ↗

Response to Reviewer cN5R

Thank you for recognizing our benchmark's motivation and diagnostic harness. Your feedback helped us sharpen our terminology and run extensive baselines that separate audio comprehension from agentic search and temporal grounding.

[W1] Operationalizing "listening"

We agree that "listening" should refer strictly to observable behavior in the harness log. In the revision, we replace shorthand like "never heard the sentence" with exact delivery-log descriptions of whether the agent accessed the designated gold interval, overlapped it in citations, or found alternative supporting dialogue. At 0.25T, 62.1% of Qwen3-Omni's 448 correct answers were submitted without receiving the designated gold sentence. Auditing 32 keyword-enriched correct submissions lacking the primary gold span showed that 16 had full support from other delivered dialogue, 12 had partial clues, and 4 had no supporting audio.

[W2] Counterfactual siblings and benchmark tracks

We evaluated our counterfactual sibling suite across 9 question families and 35 audio branches where the decisive sentence is retained, replaced, removed, or moved in time, and verified the rendered waveforms.

BranchVerified EditsAnswer / AbstainCitation HitStrict Support
Base Unchanged9 / 977.8%77.8%33.3%
Replace Fact8 / 975.0%75.0%25.0%
Remove Fact9 / 911.1%——
Move in Time7 / 885.7%42.9%0.0%

Agents adapt their answers when a spoken fact is replaced, reaching 75.0% accuracy on 6 of 8 verified edits, and maintain 85.7% accuracy on 6 of 7 verified edits when the fact is moved to a new timestamp. When the decisive fact is removed, agents abstain on only 1 of 9 items, defaulting to plausible guesses. We will also clarify in Sections 3 and 4 that the shared-index track, open-system track, and ambiguous or conflicted unanswerables are framework specifications for future releases, whereas v0.1 evaluates the raw-listening track and absent unanswerables.

[W3] Oracle upper bounds, ASR cascades, and additional model families

We added four reference baselines across our question pool:

  • Oracle upper bounds: Supplying a 286-second window lifts Qwen3-Omni's accuracy on 2,406 answerable questions from 15.8% to 47.1%, and supplying the exact gold sentence raises accuracy to 81.59%.
  • Pre-delivered fixed evidence: When gold clips are pre-delivered without search, Qwen achieves 63.01% answer accuracy and 43.93% strict ECS, proving that the contracts are solvable once the search bottleneck is removed.
  • Second model family: Evaluating the Ming omni-modal family shows a jump from 1.95% accuracy in autonomous mode to 34.16% accuracy and 22.90% strict ECS under pre-delivered gold evidence, as removing multi-turn search lifts its valid submission rate from 62.00% to 90.27%.
  • ASR + LLM cascade: Pairing Qwen3-ASR-1.7B with the Qwen3-30B text backbone on the same 0.25T intervals improves raw accuracy by 6.21 percentage points over native audio across 2,368 matched items, reaching 30.62% versus 24.41%, and reaches 78.10% on full transcripts.

[W4] Audio and dataset validation

Synthesizing conversations from structured timelines guarantees exact speaker, source, and timestamp ground truth while preventing data contamination. Across 356 audited clips, independent audio judges agreed with reference intelligibility labels on 97.5% and 96.9% of clips, with only 3.1% masked by cabin noise. Among 34 sufficiency disputes, 20 reflected multi-span questions where a single clip is incomplete on its own. Scanning all 67,752 dialogue turns found that spoken stage directions touched only 4.22% of questions, and excluding those recordings alters model accuracy by at most 0.4 percentage points.

[W5, Q1, W6] Human alignment of ECS and why the autonomous floor diagnoses agent limits

Comparing strict ECS against reference judgments on 35 evaluable citations shows 100% precision: all 5 ECS-passing submissions were independently judged semantically sufficient with correct speaker attribution. ECS accepts 20.8% of sufficient submissions because its 0.8 temporal F1 gate penalizes loose padding and incomplete multi-span citations. Crucially, the sub-1% autonomous ECS score reflects current models' weak multi-step search and clock reading rather than a benchmark flaw: the exact same scorer yields 43.93% ECS for Qwen and 22.90% for Ming when gold clips are pre-delivered, and 40.65% on the sentence oracle under relaxed tolerances. Pairing strict ECS with component metrics and tolerance curves enables clear ranking across every agent stage.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.