- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CabinBench, which jointly evaluates answers, evidence citations, and records of agents' actual audio access in long-audio question answering. The dataset consists of synthesized Chinese conversations set in a vehicle cabin. Agents select audio segments within a listening budget and are evaluated against a predefined evidence contract. The experiments examine gaps between answering correctly, accessing relevant evidence, and citing it correctly, as well as the effects of the listening budget on search and abstention behavior.
Critical strengths
-
The use of harness logs to verify audio delivery makes it possible to identify citations to segments an agent did not access. This provides an auditable basis for studying how long-audio agents obtain evidence.
-
The paper breaks down correct answers according to whether the relevant evidence was accessed and cited, and reports failure stages and search trajectories. These analyses offer more insight into where systems fail than an ECS score alone.
-
Preserving sentence, speaker, and timing information during synthesis makes it feasible to define evidence and introduce controlled modifications. The analyses using stricter answer matching and excluding items with option bias also provide useful checks on the robustness of the main findings.
Critical weaknesses
-
One concern is that ECS may be overly sensitive to citation boundaries. Under the stated formula, citations to required counter-evidence must fully satisfy both temporal matching and speaker attribution, so a citation extending slightly beyond the correct sentence may fail. An oracle-listen control that supplies the correct evidence segments, together with sensitivity analyses of boundary tolerance and scoring thresholds, would help clarify whether low scores primarily reflect difficulties in understanding the evidence or sensitivity to temporal precision.
-
The current unanswerable questions all involve missing information, and correct abstention does not require any minimum search coverage. A fixed response stating the corresponding abstention reason could therefore pass these items without examining the audio. This limits what the current evaluation can tell us about evidence-based abstention after search. Including potentially confusable cases involving missing and conflicting information, and specifying verifiable search requirements for abstention, would help address this limitation.
-
A correct answer may still be supported by audio the agent has accessed, even if the designated gold sentence was not among the segments it heard. The appendix's checks for answer strings and aliases are useful, but they may miss paraphrases or indirect inferences. Human reassessment of these responses against the audio actually accessed would help determine whether the answers are supported. Paired testing of the existing counterfactual siblings would also provide a more direct check of whether answers change when the critical evidence changes.
-
The informative behavioral findings come mainly from one Qwen agent, while the other model largely fails to complete the protocol or chooses to abstain. Including an agent from another model family, along with a simple sequential-scanning strategy using the same backbone, would help distinguish the effects of model capability, search strategy, and protocol adaptation.
Questions for authors
I would appreciate the authors' clarification of the points raised in the weaknesses above.
Response to Reviewer bGw7
Thank you for your supportive review and insightful analysis. Your observations on citation boundary sensitivity, abstention coverage requirements, alternative supporting evidence, and backbone versus strategy confounds were spot-on, and we designed targeted experiments to test each one.
We ran a Sentence Oracle across all 2,406 answerable questions that supplies the gold sentence with ±2 seconds of context. Giving the agent the exact clip raises answer accuracy from 15.8% to 81.59%, confirming that low autonomous scores stem from failing to locate the evidence rather than failing to understand the speech.
Strict ECS under the sentence oracle is 6.15% on 148 of 2,406 items, and 79.98% of the failures occur on 1,806 runs that answered correctly but scored below the 0.8 temporal support threshold. Because the oracle clip includes ±2 seconds of padding, its median duration is 9.22 seconds compared to 5.20 seconds for the tight gold sentence; simply echoing the delivered window fails the 0.8 F1 gate in 92.2% of cases. Sensitivity sweeps on these oracle submissions confirm your hypothesis:
In our pre-delivered Fixed setup—where exact gold spans are provided without extra padding—strict answerable ECS reaches 43.93% for Qwen and 22.90% for Ming. We keep the v0.1 scorer frozen for reproducibility while reporting these tolerance curves alongside strict ECS.
Your observation is spot-on. Because v0.1 absent contracts check the refusal reason without enforcing a minimum search coverage floor, a dummy policy that immediately abstains with
missing_evidenceat zero audio cost passes all 202 absent items, earning 7.75% overall ECS across the 2,608 questions. While no evaluated model used zero-listen abstention, separating answerable ECS from unanswerable abstention prevents passive refusal from inflating overall scores. Moreover, our counterfactual Remove branch tests abstention after search: when the decisive sentence is spliced out of 9 recordings, Qwen3-Omni abstains on only 1 of 9 items, showing strong resistance to abstaining once surrounding context is heard. We will report answerable and unanswerable metrics separately and enforce a minimum coverage gate in the next release.We tested both mechanisms directly:
We separated backbone capability, search strategy, and tool formatting across three controls: