acceptodds
Under review as a conference paper at ICLR 2027

LISTEN-to-Reason: Listen with Experts, Retrieve over a Graph, Reason with LLMs

20 views

Abstract

Large audio-language models (LALMs) fuse an audio encoder into a large language model (LLM) through multi-stage training. This coupling means that a new domain or a stronger LLM requires retraining, and their answers cannot be traced to what the model heard: a chain-of-thought is a post-hoc account. We propose Listen-to-Reason (L2R), an interpretable-by-design pipeline that passes audio to the LLM through an explicit, human-readable tree: small heads on frozen expert encoders map each chunk of a clip to semantically meaningful nodes on the tree (for speech, music and environmental sound), and a frozen text-only LLM answers from these nodes and an ASR transcript without hearing the clip. Every answer can therefore be traced to the nodes and transcript it read, and the nodes are causal: replacing the deciding node with a distractor overturns 78% of correct answers on SAKURA. With a 7B reader, L2R outperforms all LALMs we compare against on SAKURA and trails them by 6-12 points on MMAU and MMAR, despite training about 1,400 fewer parameters on orders of magnitude less audio data. However, because any LLM can serve as the reader, we show that a stronger reader narrows this gap without retraining any audio component. A new domain is added with one small head: with five labelled clips per species, it outperforms QLoRA fine-tuning of an LALM on the same clips by 13-26 points. Examples and an interactive view of the tree are available at https://anonymous.4open.science/w/listen-to-reason/index.html.

Primary area:
interpretability and explainable AI
Keywords:
Audio language models, Audio question answering, Interpretability, Concept bottleneck, Knowledge tree, Few-shot domain adaptation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yYxjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Listen-to-Reason (L2R) which separates audio perception from language reasoning. Small heads on frozen audio encoders map clips to named concepts in a hierarchical tree. A frozen text LLM then answers questions using these concepts and an ASR transcript. The system supports replacing the reader and adding domain-specific heads without retraining the existing components. It performs well on SAKURA but falls behind the compared audio-language models on MMAU, MMAR, and MMAU-Pro.

Critical strengths
  1. The modular design has clear practical benefits. Audio features can be cached and reused across readers. The experiments with 26 LLMs demonstrate that the reader can be replaced without retraining the audio components. New domains can also be added with small heads.

  2. The intermediate evidence is easy to inspect and edit. The node interventions go beyond showing example explanations: replacing or correcting a node often changes the answer. This provides useful evidence that the reader actually uses the supplied concepts.

Critical weaknesses
  1. The benefit of the tree itself is not established. The reader receives a text list of selected nodes rather than traversing the graph. However, there is no direct comparison with a flat concept predictor using the same encoders and training data, or with the closely related SAR-LM approach discussed in the paper. The input ablations show that audio labels help, but do not show why the proposed hierarchy is needed.

  2. The interpretability claims are stronger than the evidence supports. A readable input reveals what evidence was available, but does not fully explain how the LLM reached its answer. The single-node interventions focus on SAKURA single-hop questions, where the edited attribute often directly names the answer. They show dependence on that label, but provide limited evidence about more complex reasoning.

  3. The novelty relative to prior work needs clearer discussion. [1] also converts inferred speech states into text/label for a language-only LLM. This is closely related to L2R's use of explicit concepts to connect audio perception and language reasoning.

[1] X. Zhou, J. Lian, H. Hong, X. Yang, and G. Anumanchipalli, "Speech World Model: Causal State-Action Planning with Explicit Reasoning for Speech," ICLR 2026.

Questions for authors

NA

6 comments
AuthorsOpenReview ↗

Weakness 1. Value of the hierarchy and comparison with SAR-LM

Thank you for highlighting the distinction between evidence transparency and reasoning interpretability, and the need to clarify L2R’s contribution relative to related work.

Please see General Response §2 for the flat-predictor comparison and §3 for controlled SAR-LM comparisons.

The tree organizes which acoustic concepts are predicted and passed to the LLM, which reads them as text. This structure supports selecting relevant branches, adding concept heads, and locating or correcting individual nodes.

AuthorsOpenReview ↗

Weakness 2. Interpretability and interventions beyond single-hop questions

Please see General Response §4 for multi-hop and MMAU/MMAR interventions, unrelated-edit controls, and improvements from correcting nodes.

By separating audio perception from the LLM, L2R also makes it easy to apply interpretability methods to the reader. Our Gemma experiments with sparse autoencoders demonstrate this by inspecting and editing internal concept features without retraining audio components. Our main contribution, however, remains audio-bottleneck interpretability; these initial experiments show how the design supports further analysis of LLM reasoning.

AuthorsOpenReview ↗

Weakness 3. Relationship to Speech World Model

Thank you for highlighting this closely related work. Speech World Model also exposes intermediate states and supports both a language-only reader and a multimodal reader receiving raw speech alongside those states. The main differences are:

  • Reader training: Speech World Model instruction-tunes its LLM/SLM on inferred states. L2R keeps the reader frozen and evaluates 26 interchangeable readers without retraining the audio side.
  • Representation and scope: Speech World Model models dependencies among four fixed speech-state modules: context, mental state, speech act, and intent. L2R organizes acoustic concepts across speech, environmental sound, and music in an extensible hierarchy.
  • Adaptation: L2R demonstrates adding new domains through small heads while keeping existing components and the reader unchanged. For SWM, adaptation would require fine-tuning the language model.

The revised Related Work section clarifies this relationship.

Reviewer_kxrmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

L2R employs an audio concept tree to decouple perception and reasoning. Router networks learn to predict these audio concepts from the frozen audio encoder embeddings. Then, a frozen text LLM is used to answer questions using the audio concepts and the ASR transcripts. The reader language model can be switched out without finetuning the audio branch, and new domains can be added by training small heads on suitable frozen encoders. Using Qwen2.5-7B as the reader, L2R outperforms the compared audio LLMs on SAKURA but underperforms them on the other three benchmarks. Two species identification tasks demonstrate low-cost adaptation. The node interventions further show that audio concepts affect reader outputs.

Critical strengths
  1. L2R decouples perception from reasoning through readable audio evidence, and node interventions show that the intermediate concepts strongly influence answers, supporting interpretability.
  2. The small domain heads outperform Qwen2.5-Omni-7B fine-tuned with QLoRA on both species-identification tasks while requiring substantially fewer trainable parameters.
  3. Experiments with 26 frozen text readers demonstrate that the LLM can be replaced without retraining the audio components.
Critical weaknesses
  1. The 7B configuration trails Qwen2.5-Omni-7B on three of the four main reasoning benchmarks, limiting its advantage for general audio reasoning.
  2. The reported inference computation and runtime are higher than those of Qwen2.5-Omni-7B despite a similar total parameter count, limiting its deployment efficiency.
  3. The paper lacks numerical comparisons with closely related symbolic audio reasoning methods such as SAR-LM, leaving the added value over existing text-based pipelines unclear.
Questions for authors
  1. How does Qwen2.5-Omni-7B compare between direct audio input and text-only input of L2R’s tree and transcript, keeping the questions, options, and scoring fixed? Using the same LLM weights would help isolate L2R’s effect on accuracy and total inference cost, including its audio preprocessing.
  2. How much does using the tree hierarchy improve accuracy over predicting the same concepts independently, with the same encoders, training data, reader, and number of output concepts?
  3. Are the development samples used to select attributes and prompts separate from the final test samples, and if not, what are the results after removing the overlap?
9 comments
AuthorsOpenReview ↗

Weakness 1. General audio-reasoning performance

Thank you for the careful assessment and for suggesting controlled comparisons to clarify the contribution of L2R’s evidence representation and hierarchy.

Please see General Response §§1–2 for the tree’s contribution beyond transcripts, reader scaling, and remaining performance limitations. We observe that the tree, when used together with transcription, gives better performance compared to transcription used alone.

L2R also provides interpretable and editable audio evidence, few-shot adaptation using as few as five labeled clips per species, and lightweight training of only 7.7M parameters with limited training data. Its modular design allows us to replace the reader with a stronger LLM without retraining any components. As shown in the reader-scaling study (Table 1, Figure 2, and Appendix F, Table 8), stronger readers can better use the tree’s evidence and narrow the performance gaps, although gains vary across benchmarks.

As demonstrated in §4.3, L2R can adapt to new domains with few-shot data while keeping existing components frozen. Its tree helps locate missing or incorrect concepts, and its modular design supports adding concept heads, encoders, or tools to address these limitations.

AuthorsOpenReview ↗

Weakness 2. Inference cost

Please see General Response §5 for runtime and compute measurements, including audio preprocessing, and the optimized configuration. We observe that the modular nature of the tree enables reducing the computations. Please see the response for details.

AuthorsOpenReview ↗

Weakness 3. Comparison with SAR-LM

Please see General Response §3 for controlled comparisons using three readers, including Qwen3-30B-A3B. We observe that on the majority of the datasets, L2R obtains higher performance compared to SAR-LM, and offers a more inspectable and adaptable pipeline.

Reviewer_s6QYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the interpretability of audio-language models, whose internal processing of sound is largely opaque and whose generated explanations may not faithfully reflect how they reach their answers. It introduces LISTEN-TO-REASON (L2R), a modular architecture that converts audio into a structured, human-readable description before passing it to a language model. Rather than processing audio directly, the language model reasons entirely over explicit acoustic concepts and speech transcripts.

The system organizes acoustic information into a hierarchical knowledge tree and uses specialized, largely frozen audio encoders to identify relevant sounds and their properties. This representation makes the evidence available to the language model inspectable, editable, and extensible to new sound categories without retraining the entire system.

Experiments show that L2R achieves competitive performance on audio question-answering tasks with substantially fewer trainable parameters, although it trails conventional audio-language models on several benchmarks. Crucially, controlled interventions demonstrate that changing the extracted acoustic concepts directly changes the model's answers, providing evidence that the explicit representations play a causal role in its decisions.

The paper's central contribution is an interpretable, modular alternative to end-to-end audio-language reasoning, in which acoustic evidence is explicitly represented and can be inspected, modified, and tested, rather than remaining hidden within the model's internal states.

Critical strengths
  1. Interpretability-by-design that is TESTED, not asserted. Every answer is conditioned solely on human-readable nodes + transcript, and the authors verify both that the nodes fit the audio (3 audio-LLM judges with a <1% node-overlap swap control, higher on 97–99% of items) and that the answer causally depends on them (swap breaks 78.3%, inject repairs 90.9%; per-attribute up to 99.8% for gender). Consequential because the paper's own motivation, that LALM reasoning traces are post-hoc and unfaithful, is a real open problem, and L2R is a rare audio system where the explanation IS the computation and faithfulness is measured.

  2. Modularity + genuine data/parameter efficiency for adaptation. Any of 26 frozen readers (0.5–72B) reads the same cached context, so stronger LLMs can be adopted with zero audio-side retraining, and a new domain is one 0.17–0.34M-param head trained in <1 min that beats QLoRA fine-tuning of Omni-7B (birds: 79.6 vs 54.1 at 5 clips/species, 120–240× fewer trained params). This matters for the safety-critical voice-health settings the paper motivates, where repeatedly fine-tuning a large LALM is costly and opaque.

Critical weaknesses
  1. The central artifact, the knowledge tree, contributes little beyond the transcript/blind reader on the harder benchmarks, which bounds the contribution to attribute-identity QA. In the input ablation (Table 1), tree-only barely exceeds the blind reader on MMAU-Pro (39.3 vs 38.9) and MMAR (40.3 vs 37.8); the large tree gain is on SAKURA (74.7 vs 32.6), exactly the identity attributes the tree encodes. L2R trails every LALM on MMAU/MMAR/MMAU-Pro, and App H localizes the losses to counting, order/timing, scene and music-theory questions. Reported honestly, but the "interpretable reasoning over audio" framing overreaches relative to where the bottleneck actually helps: it is a strong interpretable identity-attribute extractor, not yet an interpretable reasoner.

  2. Framing/comparison nuances inflate the headline advantage. (a) L2R readers are scored by forced-choice option-likelihood (Eq 4) rather than free generation; if the LALM baselines are scored differently, part of both the SAKURA win and the MMAU/MMAR gaps could be protocol, not capability. (b) The "7.7M params / ~1,400× fewer" headline counts only TRAINABLE parameters while the system leans on many large frozen pretrained models (Whisper-large-v3, CLAP, emotion2vec, BEATs, BirdNET, readers up to 72B), and the authors concede inference is actually MORE expensive than a comparable dense LALM (§4.5: 5.3 vs 2.7 TMACs, 0.57 vs 0.12 s). (c) Qwen3-Omni-30B both annotates the tree and serves as one of three relatedness judges, a mild circularity. These are presentation/clarity issues, but should be tightened before the claims are taken at face value.

Questions for authors

Clarifications that could change the assessment:

  1. How are the LALM baselines scored: free generation parsed to an option, or the same option-likelihood scoring (Eq 4) used for L2R readers? If different, how much of the SAKURA win and the MMAU/MMAR gap is protocol rather than capability?

  2. Does excluding the annotator model (Qwen3-Omni-30B) from the relatedness-judge panel change the direction or magnitude of Figure 4?

  3. The node-intervention test edits the single clause that names the answer on single-hop SAKURA items; a change in the answer then seems near-tautological. What does 78.3%/90.9% establish beyond "the reader reads the clause you changed," and can you report an analogous causal test on multi-hop or MMAU/MMAR items?

  4. You state no benchmark audio is used for training; please confirm the deduplication against the benchmark clips, especially for AudioSet/FSD50K-derived items that may overlap MMAU sources. Future-work (non-blocking): event-level nodes and signal-level experts (diarization, beat tracking) for counting/order; a distilled/parallel front end to cut the ASR-dominated inference cost; adaptation on domains that require reasoning, not just identification.

7 comments
AuthorsOpenReview ↗

Weakness 1. Tree contribution and interpretability

Thank you for the constructive feedback and for recognizing L2R’s modularity, editable evidence, and low-cost adaptation.

Please see General Response §1 for tree–transcript complementarity and reader scaling, where we show that using the tree together with transcription improves over transcription-only performance.

Also in our general response §4 we provide additional intervention experiments to establish the importance of the extracted features for the reader and Gemma experiments using sparse autoencoders, where we show that it is possible to also incorporate SAEs to provide explainability for a reader such as Gemma.

AuthorsOpenReview ↗

Weakness 2(a) / Q1. Scoring consistency

You are correct: L2R used option-likelihood scoring with a blind prior, while the LALM baselines used parsed generation. We reran L2R using free generation with the same parser as the baselines:

BenchmarkL2R: submitted resultsL2R: free generationOmni-7B: free generation
SAKURA80.179.873.2
MMAU68.368.075.1
MMAR51.651.763.1
MMAU-Pro45.745.157.2

L2R’s accuracy changes by at most 0.6 points. The SAKURA advantage and the gaps on the other benchmarks remain. In the revised manuscript, Table 1 reports matched-generation results, and the evaluation protocol clarifies the scoring procedure.

AuthorsOpenReview ↗

Weakness 2(b). Parameters and inference cost

Please see General Response §5 for trainable versus total parameters, inference costs, and efficiency improvements.

L2R is by-design modular, which enables swapping out expensive decoders. Please see the general response for the detailed analysis.

Comments on the paper

7 comments
AuthorsOpenReview ↗

General Response 5 - Training versus inference efficiency

5. Training efficiency and inference cost — s6QY Weakness 2b; kxrm Weakness 2

Our main efficiency claim concerns training and adaptation. L2R trains only 7.7M parameters*reusing pretrained components. Readers can be replaced without retraining, and our adaptation experiments use as few as five labeled clips per species. Including frozen components, L2R has 10.5B total parameters versus 8.9B for Omni-7B, and Table 18 lists the parameters of every L2R component.

However, L2R has a modular design, and the main paradigm can still be applied with more streamable/efficient components.

For instance, the ASR module is a significant bottleneck. On one A100 with 10-second clips, L2R uses 5.26 TMACs versus 2.73 TMACs for Omni-7B. Running ASR only where speech is served reduces L2R's overall compute to 3.84 TMACs.

The following breakdown uses conditional ASR, averaged over eight clips per audio type. Timings exclude model loading and initial warm-up (Table 19):

MeasurementSpeechSoundMusicOverlap
L2R time (s)1.410.240.380.77
Omni-7B time (s)0.160.160.150.15
L2R compute (TMACs)5.781.893.674.03
Omni-7B compute (TMACs)2.672.702.722.82

ASR is the main speech bottleneck, taking 0.99 s of the 1.41 s total. On sound clips, L2R uses less compute than Omni-7B, although it remains slower.

We further experimented with reducing the costs using a more efficient model, Whisper-large-v3-turbo, and question-dependent encoder selection (Table 19):

BenchmarkAccuracy: original → optimizedTime/item (s): original → optimizedTMACs/item: original → optimized
SAKURA79.8 → 80.10.44 → 0.293.79 → 3.42
MMAU68.0 → 66.80.93 → 0.605.82 → 5.49
MMAR51.7 → 52.31.54 → 0.918.37 → 7.69

These benchmark measurements use actual clip lengths, rather than fixed 10-second clips.

Runtime decreases by 34–41%, with accuracy changes between −1.2 and +0.6 points.

Beyond the ASR components, we can also replace the heavy batch-mode encoders with streamable versions. For instance, we test MAEST in place of MuQ-MuLan (for music), reducing encoder size from 663M to 86M parameters but only losing 0.6 points on MMAU and 0.8 on MMAR. Further work will address this accuracy–cost trade-off.

The results presented above are added in Appendix N (Tables 17–120) of the revised manuscript.

AuthorsOpenReview ↗

General Response 4 - Evidence interventions and reader analysis

4. Evidence interventions and reader analysis — s6QY Weakness 1/Question 3; yYxj Weakness 2

The reviewers raised two concerns: whether concept interventions extend beyond single-hop lookup, and whether readable evidence explains the LLM’s reasoning. We address these through two experiments.

Interventions beyond single-hop questions. We provide new results on multi-hop intervention presented in Section 4 and Table 2 in the revised manuscript. We change one concept at a time on SAKURA multi-hop questions and selected MMAU/MMAR questions about concepts represented in the tree:

  • Relevant edit: Replace a concept needed to answer the question with an incorrect value.
  • Unrelated edit: Change a concept unrelated to the question as a control.

The question, transcript, and all other concepts stay unchanged. We provide an experiment where the table below shows how often an originally correct answer changes:

QuestionsChange after a relevant editChange after an unrelated edit
SAKURA multi-hop58.5%2.3%
MMAU selected subset62.9%4.8%
MMAR selected subset46.4%3.8%

Higher percentages indicate greater dependence on the edited concept. Relevant edits change answers much more often than unrelated edits.

Supplying the correct concept also improves accuracy on these subsets: 73.8 → 81.7 on SAKURA, 70.0 → 92.7 on MMAU, and 50.0 → 69.6 on MMAR. These results support L2R’s use for targeted debugging and editing beyond single-hop questions.

Extending interpretability to the reader. L2R lets us replace the reader without retraining audio components. We used Gemma-3-27B** with Gemma Scope 2 [1], whose sparse autoencoders help identify and edit features inside the LLM.

Gemma scores 83.0/68.6/55.2/49.6 on SAKURA/MMAU/MMAR/MMAU-Pro, compared with 79.8/68.0/51.7/45.1 for Qwen2.5-7B under the same evaluation protocol.

Below, we provide sparse autoencoder (SAE) latent variable interpretability study to show that LLM-reader can in fact be further explained with an SAE.

StudyResultConclusion
Find internal features that activate strongly for one concept and rarely for others22 of 24 tested SAKURA concepts have a feature with selectivity above 0.8We can identify features associated with particular animals, emotions, or languages.
Change the internal features associated with a concept to those of an alternative concept, keeping the input text unchangedOn 273 multi-hop questions, 54.6% of answers follow the alternative conceptChanging these internal features influences the reader’s answer.

Our main contribution remains audio-bottleneck interpretability. Note that the modular design also lets us replace the reader with an LLM supported by interpretability tools, as demonstrated by these preliminary Gemma experiments above. Choosing and fully evaluating methods for explaining LLM reasoning is outside this paper’s scope, but such methods can be readily integrated into L2R.

[1] McDougall et al. Gemma Scope 2 — Technical Paper, 2025.

AuthorsOpenReview ↗

General Response 3 - Comparison with SAR-LM

3. Comparison with SAR-LM — kxrm Weakness 3; yYxj Weakness 1

Below we provide an experiment where we reproduced SAR-LM’s released pipeline within 0.8 points of its reported open-model results. To perform a fair comparison, we replaced only SAR-LM’s extracted evidence with L2R’s tree traversals and transcript, keeping SAR-LM’s reader, prompt, decoding, and scoring fixed between two methods. Here are our new results:

ReaderEvidenceSAKURAMMAUMMAR
Qwen2.5-7BSAR-LM48.462.647.8
Qwen2.5-7BL2R77.265.846.6
Qwen2.5-Omni-7BSAR-LM49.063.949.3
Qwen2.5-Omni-7BL2R74.767.348.8
Qwen3-30B-A3BSAR-LM51.166.754.4
Qwen3-30B-A3BL2R81.468.556.5

L2R improves SAKURA by 25.7–30.3 points and MMAU by 1.8–3.4 points, with comparable MMAR results. Beyond these performance comparisons, we emphasize that L2R expands its vocabulary through small heads for concepts not already covered in the tree (as shown in Section 4.3). The tree also supports interpretable evidence, selective processing, and easier debugging and editing. We added the experiment above for the SAR-LM baseline in Table 1, with reproduction details in appendix I

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.