- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper investigates whether Qwen3-Omni contains verbalizable representations of audio content before generating an answer. It applies a logit lens at audio-token positions, which cannot attend to subsequent questions or options, and evaluates the readouts through waveform swaps, activation patching, and audio-versus-caption comparisons. The results reveal sound-dependent semantic signals, multilingual readouts, and emotion-related content. The authors interpret these findings as evidence for a verbalizable middle-layer workspace in an audio LLM.
Critical strengths
-
Restricting the readout to audio-token positions provides a clear safeguard against directly copying concepts from subsequent questions or answer options.
-
The waveform-swap control provides useful evidence that the readout depends on the specific audio. The mismatched-audio condition also reduces the length confound associated with silence.
-
The qualitative visualizations are informative, and the paper combines several complementary analyses. The appendix explicitly acknowledges important limitations, including restricted causal coverage and uncertainty about the multi-hop examples.
Critical weaknesses
-
The causal evidence is not specific to the verbalized concepts. Patching the entire residual stream restores all information at the selected positions, so successful restoration does not identify which decoded concepts contribute to the answer. Although the authors acknowledge this limitation, the connection between the interpretable readouts and the model’s actual computation remains insufficiently tested.
-
The multi-hop interpretation lacks direct validation. In the Nixon example, an unsuccessful transcription does not establish that the intermediate concepts were absent from the audio, and the model can answer correctly without audio. These observations leave partial recognition and associative retrieval as plausible explanations, while the proposed event-to-actor chain is not tested through targeted interventions.
-
A distinct middle-layer function is not clearly established. The bands are imported from prior text-model work, while the sound-dependent readout extends beyond the designated workspace and the real-versus-mismatch gap is largest in the motor band. This does not rule out a workspace, but stronger comparisons with equal-sized bands and interventions across both audio and text positions are needed to distinguish workspace-specific processing from general information propagation.
-
The strongest evidence has limited task coverage. The central readout experiment uses 140 selected items, and activation patching covers only ten sound-track clips. Moreover, Appendix A.1 reports that the real-versus-mismatch effect is mainly driven by sound-source identification, with no comparable advantage on speech-content clips, leaving the extension to speech reasoning less well supported.
Questions for authors
- Can selectively removing or replacing a decoded concept change the answer in the predicted direction, without replacing the entire residual state?
- Can the authors clarify the final-layer readout/generation mismatch in Appendix D.8, including whether comparisons use token IDs, identical input positions, and the same decoding settings?
12 comments
Reply to Reviewer eYKv [Part 2]
Q2: The multi-hop interpretation lacks direct validation. In the Nixon example, an unsuccessful transcription does not establish that the intermediate concepts were absent from the audio, and the model can answer correctly without audio. These observations leave partial recognition and associative retrieval as plausible explanations, while the proposed event-to-actor chain is not tested through targeted interventions.
A2: Thank you for this careful point. We would like to clarify that the event is spoken and recognized, while the actor is never spoken, so the actor is inferred from what is heard. On the two explanations you name: the event is recognized from the audio, which is the chain's first step, and the actor read at the audio positions is retrieved from the heard event, not from the question or the options. These positions precede the question, and with the prompt unchanged, silencing the heard event words removes Nixon from the readout, while silencing the same duration of other speech never does (table below). That the model can also answer from the question alone concerns the answer once the question is given; the chain is read at the audio positions, which precede the question, and there the readout of both the actor and the answer concept depends on the heard event words.
To further address your concern about what the Nixon clip contains, we have transcribed it with two independent speech recognizers, Whisper large-v3 and wav2vec 2.0 large, which agree word for word (wav2vec 2.0 spells "US" as "ES"): "The Watergate scandal brought down a US president." Aligned with the readout of Figure 2, the event concepts reach rank 1 where their words are spoken, while Nixon, never spoken, is inferred (Appendix E.5).
To further address your concern about whether this order holds beyond one clip, we have tested it on every speech clip whose correct answer is never spoken (Appendix E.13):
| Speech clips whose correct answer is never spoken (n=198) | Clips where both become readable | Heard word first vs. answer first | p |
|---|---|---|---|
| All clips | 89 | 76 vs. 8 | 5.0 × 10⁻¹⁵ |
| Event-Based Knowledge Retrieval, the category of the Nixon clip | 15 | 15 vs. 0 | 6.1 × 10⁻⁵ |
Transcripts by Whisper large-v3; readable: rank 10 or better at the audio positions; two-sided sign test on which becomes readable first, clips where both do so at the same depth left out.
To further address your concern about a targeted intervention on the event-to-actor chain, we have silenced the heard event words in the Nixon clip's waveform, against silencing the same duration of other speech in 50 random draws (Appendix E.16):
| Nixon clip: waveform | Answer concept: best rank at the audio positions | Nixon: best rank at the audio positions |
|---|---|---|
| Unedited | 5 | 1 |
| Heard event words "scandal brought" silenced | 205 | 950 |
| Same duration of other speech silenced (50 random draws) | 1–3 | 1 |
Answer: the correct option's first token. Best rank over the audio positions and all depths; the prompt is unchanged; one-sided permutation test over the 50 draws, p=0.02 for both.
These results verify that the event is heard and the actor, never spoken, is inferred from it: silencing the heard event words removes the actor and the answer concept from the readout, while silencing the same duration of other speech never does, and a heard word becomes readable at a shallower depth than the inferred answer on most clips where both become readable. Following your suggestion, we have presented the clip as a case study with its transcript in Section 4.2 and reported these analyses in Appendices E.5, E.13 and E.16.
Reply to Reviewer eYKv [Part 3]
Q3: A distinct middle-layer function is not clearly established. The bands are imported from prior text-model work, while the sound-dependent readout extends beyond the designated workspace and the real-versus-mismatch gap is largest in the motor band. This does not rule out a workspace, but stronger comparisons with equal-sized bands and interventions across both audio and text positions are needed to distinguish workspace-specific processing from general information propagation.
A3: Thank you for this insightful comment. We would like to clarify that, with the bands fixed a priori from the text-model work, what distinguishes the middle band is where the answer reads the audio: the sound content is readable in every band, as you observe, but blocking the answer's attention to the audio in the workspace band removes the answer on many clips, while blocking it in the motor band never does.
To further address your concern about equal-sized bands and interventions at both audio and text positions, we have added these comparisons (Appendices E.2, E.3 and E.11):
| Band | Real − Mismatch, equal-size window of 10 depths (p) | Answer restored by patching the band: audio positions | text positions | Answer kept with its attention to the audio blocked: in the band | in the equal-size window |
|---|---|---|---|---|---|
| Sensory | +5.0 (0.0036) | — | 42.4% | 90.7% | 94.0% |
| Workspace | +8.0 (6.4 × 10⁻⁷) | 92.5% | 100.0% | 55.2% | 86.7% |
| Motor | +8.1 (5.1 × 10⁻⁹) | 2.0% | — | 100.0% | 100.0% |
Windows: depths 4–13, 23–32 and 39–48; real and mismatched audio read at the same number of audio positions; balanced accuracy, exact two-sided McNemar. Patching: the 255 clips answered correctly with the audio and wrongly with it zeroed. Blocking: the 248 of these that the unedited pass answers correctly; every text position is kept from attending to the audio positions inside the given depths, and nothing else changes; a lower value means the answer reads the audio more there.
These results verify that, with depths matched, blocking the middle window removes the answer more often than blocking the early or the late window (exact two-sided McNemar, p=0.003 and 2.3 × 10⁻¹⁰), and that the information the answer needs reaches the answer's positions by the end of the middle band, where the answer's reading of the audio peaks, while the content itself propagates to the output. Following your suggestion, we have reported these analyses in Appendices E.2, E.3 and E.11, and stated in Section 4.7 that the answer reads the audio in the middle band.
Reply to Reviewer eYKv [Part 1]
We sincerely thank you for the careful and constructive review. We greatly appreciate your recognition that "restricting the readout to audio-token positions provides a clear safeguard against directly copying concepts from subsequent questions or answer options," that "the waveform-swap control provides useful evidence that the readout depends on the specific audio," and that "the paper combines several complementary analyses."
We understand your concerns focus primarily on whether the decoded concepts themselves are used, the validation of the Nixon chain, what distinguishes the middle band, and the task coverage of the strongest evidence, together with the final-layer comparison of Appendix D.8.
To address these thoroughly, we have (1) edited only the decoded answer concept, against placebo edits of the same size, which moves the answer in the predicted direction (Appendices E.4 and E.12); (2) transcribed the Nixon clip with two speech recognizers, silenced its heard event words, which removes the inferred actor from the readout, and tested its order on every speech clip whose answer is never spoken (Appendices E.5, E.13 and E.16); (3) compared equal-size bands and intervened at both the audio and the text positions, which shows that the answer reads the audio in the middle band (Appendices E.2, E.3 and E.11); (4) extended the readout to all four-option MMAU-mini clips and the patching to every one of them that needs the audio, on all three tracks, including speech-content questions (Appendices E.1 to E.3); and (5) compared the final-layer readout with greedy generation on token IDs, which agree on every clip (Appendix E.10). All additions have been integrated into our revised manuscript. Below, we address each specific question systematically.
A1: Thank you for this important point. We would like to clarify that the causal evidence now reaches the decoded concept itself: editing only the answer concept's lens direction at the audio positions, with the rest of the state unchanged, moves the answer to the predicted option significantly more often than placebo edits of the same size.
To further address your concern about whether the decoded concepts contribute to the answer, we have added a concept-level edit on the clips that need the audio (Appendix E.12):
The component of the audio-position states along the direction from the answer concept to a randomly drawn distractor's concept is shifted by α times the state's norm; the placebo edit does the same along the direction between two unrelated words. Of the 255 clips answered correctly with the audio and wrongly with it zeroed, the 205 whose answer and distractor begin with different tokens and that the unedited pass answers correctly; exact two-sided McNemar.
These results verify that the decoded concept carries part of what the answer uses, in the direction the lens reads it. Following your suggestion, we have stated in Section 3.5 that full-state patching establishes the use of the audio content, and reported the concept-level edits in Appendices E.4 and E.12.