- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Listen-to-Reason (L2R) which separates audio perception from language reasoning. Small heads on frozen audio encoders map clips to named concepts in a hierarchical tree. A frozen text LLM then answers questions using these concepts and an ASR transcript. The system supports replacing the reader and adding domain-specific heads without retraining the existing components. It performs well on SAKURA but falls behind the compared audio-language models on MMAU, MMAR, and MMAU-Pro.
Critical strengths
-
The modular design has clear practical benefits. Audio features can be cached and reused across readers. The experiments with 26 LLMs demonstrate that the reader can be replaced without retraining the audio components. New domains can also be added with small heads.
-
The intermediate evidence is easy to inspect and edit. The node interventions go beyond showing example explanations: replacing or correcting a node often changes the answer. This provides useful evidence that the reader actually uses the supplied concepts.
Critical weaknesses
-
The benefit of the tree itself is not established. The reader receives a text list of selected nodes rather than traversing the graph. However, there is no direct comparison with a flat concept predictor using the same encoders and training data, or with the closely related SAR-LM approach discussed in the paper. The input ablations show that audio labels help, but do not show why the proposed hierarchy is needed.
-
The interpretability claims are stronger than the evidence supports. A readable input reveals what evidence was available, but does not fully explain how the LLM reached its answer. The single-node interventions focus on SAKURA single-hop questions, where the edited attribute often directly names the answer. They show dependence on that label, but provide limited evidence about more complex reasoning.
-
The novelty relative to prior work needs clearer discussion. [1] also converts inferred speech states into text/label for a language-only LLM. This is closely related to L2R's use of explicit concepts to connect audio perception and language reasoning.
[1] X. Zhou, J. Lian, H. Hong, X. Yang, and G. Anumanchipalli, "Speech World Model: Causal State-Action Planning with Explicit Reasoning for Speech," ICLR 2026.
Questions for authors
NA
6 comments
Weakness 2. Interpretability and interventions beyond single-hop questions
Please see General Response §4 for multi-hop and MMAU/MMAR interventions, unrelated-edit controls, and improvements from correcting nodes.
By separating audio perception from the LLM, L2R also makes it easy to apply interpretability methods to the reader. Our Gemma experiments with sparse autoencoders demonstrate this by inspecting and editing internal concept features without retraining audio components. Our main contribution, however, remains audio-bottleneck interpretability; these initial experiments show how the design supports further analysis of LLM reasoning.
Weakness 3. Relationship to Speech World Model
Thank you for highlighting this closely related work. Speech World Model also exposes intermediate states and supports both a language-only reader and a multimodal reader receiving raw speech alongside those states. The main differences are:
- Reader training: Speech World Model instruction-tunes its LLM/SLM on inferred states. L2R keeps the reader frozen and evaluates 26 interchangeable readers without retraining the audio side.
- Representation and scope: Speech World Model models dependencies among four fixed speech-state modules: context, mental state, speech act, and intent. L2R organizes acoustic concepts across speech, environmental sound, and music in an extensible hierarchy.
- Adaptation: L2R demonstrates adding new domains through small heads while keeping existing components and the reader unchanged. For SWM, adaptation would require fine-tuning the language model.
The revised Related Work section clarifies this relationship.
Weakness 1. Value of the hierarchy and comparison with SAR-LM
Thank you for highlighting the distinction between evidence transparency and reasoning interpretability, and the need to clarify L2R’s contribution relative to related work.
Please see General Response §2 for the flat-predictor comparison and §3 for controlled SAR-LM comparisons.
The tree organizes which acoustic concepts are predicted and passed to the LLM, which reads them as text. This structure supports selecting relevant branches, adding concept heads, and locating or correcting individual nodes.