acceptodds
Under review as a conference paper at ICLR 2027

Can We Read the Mind of an Audio LLM? A Verbalizable, Multilingual Middle-Layer Workspace

Abstract

Recent interpretability work shows that language models maintain a small, privileged set of verbalizable representations in their middle layers—readable as words, reused across internal computation, and broadcast to later processing—that behaves like a global workspace, so far characterized for text inputs. We show that an audio language model forms such representations from sound. Reading an off-the-shelf Qwen3-Omni model with a logit lens at the audio-token positions, which precede and cannot see the question, we find answer-relevant concepts readable as words from the early and middle layers on, before any token is emitted. A waveform-swap control fixes the question and options and swaps only the audio (the real clip, a mismatched clip, or silence): the clip-level readout is best with the real clip, weaker with a mismatched clip, and at chance with silence, so it is driven by the sound rather than by the lens's preference for some option tokens. Traced across depth, this sound-driven signal is null at the input, becomes significant about a tenth of the way up, and persists to the output. Patching the audio-position activations into a run whose audio was removed restores the answer from the early and middle bands but not from the late layers, so the audio content is used, and committed before the late layers. The readable content is conceptual rather than a re-transcription: one concept surfaces in several languages and scripts, and a speaker's emotion is read at the audio positions although the model's own transcript and caption usually do not name it. On a clip about a president forced to resign, Watergate and scandal are followed by the answer, Nixon, at positions that cannot see the printed options. In this model, the verbalizable representations that text models form thus also form from sound.

Primary area:
interpretability and explainable AI
Keywords:
audio language models, interpretability, logit lens, global workspace, multilingual representations, multimodal LLMs, speech understanding, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_eYKvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper investigates whether Qwen3-Omni contains verbalizable representations of audio content before generating an answer. It applies a logit lens at audio-token positions, which cannot attend to subsequent questions or options, and evaluates the readouts through waveform swaps, activation patching, and audio-versus-caption comparisons. The results reveal sound-dependent semantic signals, multilingual readouts, and emotion-related content. The authors interpret these findings as evidence for a verbalizable middle-layer workspace in an audio LLM.

Critical strengths
  1. Restricting the readout to audio-token positions provides a clear safeguard against directly copying concepts from subsequent questions or answer options.

  2. The waveform-swap control provides useful evidence that the readout depends on the specific audio. The mismatched-audio condition also reduces the length confound associated with silence.

  3. The qualitative visualizations are informative, and the paper combines several complementary analyses. The appendix explicitly acknowledges important limitations, including restricted causal coverage and uncertainty about the multi-hop examples.

Critical weaknesses
  1. The causal evidence is not specific to the verbalized concepts. Patching the entire residual stream restores all information at the selected positions, so successful restoration does not identify which decoded concepts contribute to the answer. Although the authors acknowledge this limitation, the connection between the interpretable readouts and the model’s actual computation remains insufficiently tested.

  2. The multi-hop interpretation lacks direct validation. In the Nixon example, an unsuccessful transcription does not establish that the intermediate concepts were absent from the audio, and the model can answer correctly without audio. These observations leave partial recognition and associative retrieval as plausible explanations, while the proposed event-to-actor chain is not tested through targeted interventions.

  3. A distinct middle-layer function is not clearly established. The bands are imported from prior text-model work, while the sound-dependent readout extends beyond the designated workspace and the real-versus-mismatch gap is largest in the motor band. This does not rule out a workspace, but stronger comparisons with equal-sized bands and interventions across both audio and text positions are needed to distinguish workspace-specific processing from general information propagation.

  4. The strongest evidence has limited task coverage. The central readout experiment uses 140 selected items, and activation patching covers only ten sound-track clips. Moreover, Appendix A.1 reports that the real-versus-mismatch effect is mainly driven by sound-source identification, with no comparable advantage on speech-content clips, leaving the extension to speech reasoning less well supported.

Questions for authors
  1. Can selectively removing or replacing a decoded concept change the answer in the predicted direction, without replacing the entire residual state?
  2. Can the authors clarify the final-layer readout/generation mismatch in Appendix D.8, including whether comparisons use token IDs, identical input positions, and the same decoding settings?
12 comments
AuthorsOpenReview ↗

Reply to Reviewer eYKv [Part 1]

We sincerely thank you for the careful and constructive review. We greatly appreciate your recognition that "restricting the readout to audio-token positions provides a clear safeguard against directly copying concepts from subsequent questions or answer options," that "the waveform-swap control provides useful evidence that the readout depends on the specific audio," and that "the paper combines several complementary analyses."

We understand your concerns focus primarily on whether the decoded concepts themselves are used, the validation of the Nixon chain, what distinguishes the middle band, and the task coverage of the strongest evidence, together with the final-layer comparison of Appendix D.8.

To address these thoroughly, we have (1) edited only the decoded answer concept, against placebo edits of the same size, which moves the answer in the predicted direction (Appendices E.4 and E.12); (2) transcribed the Nixon clip with two speech recognizers, silenced its heard event words, which removes the inferred actor from the readout, and tested its order on every speech clip whose answer is never spoken (Appendices E.5, E.13 and E.16); (3) compared equal-size bands and intervened at both the audio and the text positions, which shows that the answer reads the audio in the middle band (Appendices E.2, E.3 and E.11); (4) extended the readout to all four-option MMAU-mini clips and the patching to every one of them that needs the audio, on all three tracks, including speech-content questions (Appendices E.1 to E.3); and (5) compared the final-layer readout with greedy generation on token IDs, which agree on every clip (Appendix E.10). All additions have been integrated into our revised manuscript. Below, we address each specific question systematically.

Q1: The causal evidence is not specific to the verbalized concepts. Patching the entire residual stream restores all information at the selected positions, so successful restoration does not identify which decoded concepts contribute to the answer. Although the authors acknowledge this limitation, the connection between the interpretable readouts and the model’s actual computation remains insufficiently tested.

A1: Thank you for this important point. We would like to clarify that the causal evidence now reaches the decoded concept itself: editing only the answer concept's lens direction at the audio positions, with the rest of the state unchanged, moves the answer to the predicted option significantly more often than placebo edits of the same size.

To further address your concern about whether the decoded concepts contribute to the answer, we have added a concept-level edit on the clips that need the audio (Appendix E.12):

Replace the answer concept by a distractor's (audio positions, depths 0–38; n=205)Answer moved to the distractor: concept editplacebo editp
α = 0.19.3%2.0%2.7 × 10⁻⁴
α = 0.221.0%6.3%2.3 × 10⁻⁷
α = 0.448.3%18.0%1.7 × 10⁻¹³

The component of the audio-position states along the direction from the answer concept to a randomly drawn distractor's concept is shifted by α times the state's norm; the placebo edit does the same along the direction between two unrelated words. Of the 255 clips answered correctly with the audio and wrongly with it zeroed, the 205 whose answer and distractor begin with different tokens and that the unedited pass answers correctly; exact two-sided McNemar.

These results verify that the decoded concept carries part of what the answer uses, in the direction the lens reads it. Following your suggestion, we have stated in Section 3.5 that full-state patching establishes the use of the audio content, and reported the concept-level edits in Appendices E.4 and E.12.

AuthorsOpenReview ↗

Reply to Reviewer eYKv [Part 2]

Q2: The multi-hop interpretation lacks direct validation. In the Nixon example, an unsuccessful transcription does not establish that the intermediate concepts were absent from the audio, and the model can answer correctly without audio. These observations leave partial recognition and associative retrieval as plausible explanations, while the proposed event-to-actor chain is not tested through targeted interventions.

A2: Thank you for this careful point. We would like to clarify that the event is spoken and recognized, while the actor is never spoken, so the actor is inferred from what is heard. On the two explanations you name: the event is recognized from the audio, which is the chain's first step, and the actor read at the audio positions is retrieved from the heard event, not from the question or the options. These positions precede the question, and with the prompt unchanged, silencing the heard event words removes Nixon from the readout, while silencing the same duration of other speech never does (table below). That the model can also answer from the question alone concerns the answer once the question is given; the chain is read at the audio positions, which precede the question, and there the readout of both the actor and the answer concept depends on the heard event words.

To further address your concern about what the Nixon clip contains, we have transcribed it with two independent speech recognizers, Whisper large-v3 and wav2vec 2.0 large, which agree word for word (wav2vec 2.0 spells "US" as "ES"): "The Watergate scandal brought down a US president." Aligned with the readout of Figure 2, the event concepts reach rank 1 where their words are spoken, while Nixon, never spoken, is inferred (Appendix E.5).

To further address your concern about whether this order holds beyond one clip, we have tested it on every speech clip whose correct answer is never spoken (Appendix E.13):

Speech clips whose correct answer is never spoken (n=198)Clips where both become readableHeard word first vs. answer firstp
All clips8976 vs. 85.0 × 10⁻¹⁵
Event-Based Knowledge Retrieval, the category of the Nixon clip1515 vs. 06.1 × 10⁻⁵

Transcripts by Whisper large-v3; readable: rank 10 or better at the audio positions; two-sided sign test on which becomes readable first, clips where both do so at the same depth left out.

To further address your concern about a targeted intervention on the event-to-actor chain, we have silenced the heard event words in the Nixon clip's waveform, against silencing the same duration of other speech in 50 random draws (Appendix E.16):

Nixon clip: waveformAnswer concept: best rank at the audio positionsNixon: best rank at the audio positions
Unedited51
Heard event words "scandal brought" silenced205950
Same duration of other speech silenced (50 random draws)1–31

Answer: the correct option's first token. Best rank over the audio positions and all depths; the prompt is unchanged; one-sided permutation test over the 50 draws, p=0.02 for both.

These results verify that the event is heard and the actor, never spoken, is inferred from it: silencing the heard event words removes the actor and the answer concept from the readout, while silencing the same duration of other speech never does, and a heard word becomes readable at a shallower depth than the inferred answer on most clips where both become readable. Following your suggestion, we have presented the clip as a case study with its transcript in Section 4.2 and reported these analyses in Appendices E.5, E.13 and E.16.

AuthorsOpenReview ↗

Reply to Reviewer eYKv [Part 3]

Q3: A distinct middle-layer function is not clearly established. The bands are imported from prior text-model work, while the sound-dependent readout extends beyond the designated workspace and the real-versus-mismatch gap is largest in the motor band. This does not rule out a workspace, but stronger comparisons with equal-sized bands and interventions across both audio and text positions are needed to distinguish workspace-specific processing from general information propagation.

A3: Thank you for this insightful comment. We would like to clarify that, with the bands fixed a priori from the text-model work, what distinguishes the middle band is where the answer reads the audio: the sound content is readable in every band, as you observe, but blocking the answer's attention to the audio in the workspace band removes the answer on many clips, while blocking it in the motor band never does.

To further address your concern about equal-sized bands and interventions at both audio and text positions, we have added these comparisons (Appendices E.2, E.3 and E.11):

BandReal − Mismatch, equal-size window of 10 depths (p)Answer restored by patching the band: audio positionstext positionsAnswer kept with its attention to the audio blocked: in the bandin the equal-size window
Sensory+5.0 (0.0036)—42.4%90.7%94.0%
Workspace+8.0 (6.4 × 10⁻⁷)92.5%100.0%55.2%86.7%
Motor+8.1 (5.1 × 10⁻⁹)2.0%—100.0%100.0%

Windows: depths 4–13, 23–32 and 39–48; real and mismatched audio read at the same number of audio positions; balanced accuracy, exact two-sided McNemar. Patching: the 255 clips answered correctly with the audio and wrongly with it zeroed. Blocking: the 248 of these that the unedited pass answers correctly; every text position is kept from attending to the audio positions inside the given depths, and nothing else changes; a lower value means the answer reads the audio more there.

These results verify that, with depths matched, blocking the middle window removes the answer more often than blocking the early or the late window (exact two-sided McNemar, p=0.003 and 2.3 × 10⁻¹⁰), and that the information the answer needs reaches the answer's positions by the end of the middle band, where the answer's reading of the audio peaks, while the content itself propagates to the output. Following your suggestion, we have reported these analyses in Appendices E.2, E.3 and E.11, and stated in Section 4.7 that the answer reads the audio in the middle band.

Reviewer_uwGkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether the verbalizable middle-layer workspace reported for text LMs also forms in an audio LLM. The authors apply the logit lens at the audio-token positions of Qwen3-Omni-30B, which causally precede the written question, and read answer-relevant concepts before any output token. The main methodological contribution is a waveform-swap control: with the prompt byte-identical, the real clip yields higher readout accuracy than a same-category mismatched clip, and silence falls to chance, attributing the readout to the sound rather than to the lens's option-token prior. Activation patching shows audio content is causally used and committed before the motor band. Further results show the readout is multilingual and that paralinguistic content (emotion) is readable at audio positions.

Critical strengths
  1. The waveform-swap control. Prior lens studies of audio LLMs (Semantic Hub, AudioLens) cannot separate what the sound supplies from the lens's preference for certain option tokens. The real/mismatch/silence design with a byte-identical prompt closes this gap, and the mismatched condition additionally controls the position-count confound of silence.

  2. The paper pairs readouts with causal evidence and honest scoping. Activation patching shows the audio content is used, not merely decodable, and that commitment happens before the motor band.

Overall, this is a careful, control-driven analysis of a question that matters for the oversight of audio LLMs. We should encourage such papers.

Critical weaknesses
  1. Some claims in this paper rest on single clips. The multi-hop chain (the Nixon clip) and the semantic-association findings are each demonstrated on a few clips. These cases are illustrative and intriguing, but as presented they are anecdotes rather than population-level evidence; more cases of this kind, together with quantitative evaluation, would be needed to establish these claims.

  2. If I understand the design correctly, the claim made in Section 4.6, "the audio carries what its caption lacks", is somewhat stronger than what the evidence shows. The experiments demonstrate that emotion and sound-source information are verbally readable from the internal states at the audio positions. They do not, however, establish that this information is expressible from audio yet inexpressible in words, because the caption baseline was generated under an instruction to omit emotion. The asymmetry between the audio mind and the caption mind may therefore reflect the captioning instruction rather than a genuine limit of what words can carry. A more informative comparison would be against the model's best effort to describe the clip in words, affect included.

Questions for authors
  1. Section 4.4 uses the 10 clips answered correctly with audio and incorrectly without. What is the denominator of this selection? If only 10 of several hundred MMAU clips truly require the audio, that is a striking property of the benchmark and deserves discussion.

  2. Regarding the design of Section 4.6 discussed in the weaknesses, could you provide a more detailed explanation for the current setup, or how the experiment could be improved?

5 comments
AuthorsOpenReview ↗

Reply to Reviewer uwGk [Part 1]

We sincerely thank you for the careful and constructive review. We greatly appreciate your recognition that "the real/mismatch/silence design with a byte-identical prompt closes this gap," that "the paper pairs readouts with causal evidence and honest scoping," and that "this is a careful, control-driven analysis of a question that matters for the oversight of audio LLMs."

We understand your concerns focus primarily on the claims that rest on single clips, the strength of the claim of Section 4.6 and the design behind it, and the denominator of the patching set.

To address these thoroughly, we have (1) tested the multi-hop pattern of the Nixon clip on every speech clip whose answer is never spoken and the association pattern on every four-option clip, both of which hold across clips (Section 4.2, Appendices E.13 and E.15); (2) clarified that Section 4.6 compares what the audio positions carry with the model's own words, and retitled it and the matching contribution bullet accordingly (Sections 1 and 4.6); (3) run the audio-necessity screen and activation patching over all four-option MMAU-mini clips, which gives the denominator across the benchmark and shows the causal pattern of Table 2 on all three tracks (Section 4.4, Appendix E.2); and (4) compared the audio with the model's best-effort description on MMSU emotion-recognition clips, where the model answers significantly better from the audio (Section 4.6, Appendix E.7). All additions have been integrated into our revised manuscript. Below, we address each specific question systematically.

AuthorsOpenReview ↗

Reply to Reviewer uwGk [Part 2]

Q1: Some claims in this paper rest on single clips. The multi-hop chain (the Nixon clip) and the semantic-association findings are each demonstrated on a few clips. These cases are illustrative and intriguing, but as presented they are anecdotes rather than population-level evidence; more cases of this kind, together with quantitative evaluation, would be needed to establish these claims.

A1: Thank you for this important point. We would like to clarify that the Nixon chain and the association families are case studies of what the readout contains (further cases, including the treaty and archaeology multi-hop clips, in Appendix D.5), and that both patterns hold across clips, measured quantitatively.

To further address your concern about the multi-hop chain, we have tested the order of the Nixon clip on every speech clip whose correct answer is never spoken (Appendix E.13):

Speech clips whose correct answer is never spoken (n=198)Clips where both become readableHeard word first vs. answer firstp
All clips8976 vs. 85.0 × 10⁻¹⁵
Event-Based Knowledge Retrieval, the category of the Nixon clip1515 vs. 06.1 × 10⁻⁵

Transcripts by Whisper large-v3; heard words: transcript words in neither the question nor the options; readable: rank 10 or better at the audio positions; two-sided sign test on which becomes readable first, clips where both do so at the same depth left out.

To further address your concern about the semantic-association findings, we have tested them on every four-option clip: the model, with no audio, is asked for the word it most strongly associates with the correct option, and we read the top words of its answer distribution that are neither printed nor spoken at the audio positions of the real clip and of a mismatched clip, with the prompt unchanged (Appendix E.15):

Related words neither printed nor spokenClipsReal clip more vs. mismatched clip morep
Pre-specified filter948334 vs. 1871.2 × 10⁻¹⁰
Stricter, outside the top-100 neighbours of every answer word948315 vs. 1811.9 × 10⁻⁹
Stricter, outside the top-1,000 neighbours of every answer word927229 vs. 1404.2 × 10⁻⁶

Related words: the first five dictionary words of the model's text-only answer distribution, in probability order, whose first four letters begin no word of the question, the options or the clip's Whisper large-v3 transcript, and that lie outside the top-100 lens-direction neighbours of the answer's first token; the stricter rows also drop shared stems. Readable: among the top 10 lens tokens at some audio position and depth. Per clip, the readable related words are counted with the real and with the mismatched clip; two-sided sign test over clips, ties left out.

These results verify that, on most clips where both become readable, a heard word becomes readable at a shallower depth than the inferred answer, as on the Nixon clip, and that the readout reaches unspoken related words of what is heard across the benchmark. Following your suggestion, we have labelled the Nixon chain and the association families as case studies in Section 4.2 and reported these tests in Appendices E.13 and E.15.

AuthorsOpenReview ↗

Reply to Reviewer uwGk [Part 3]

Q2: If I understand the design correctly, the claim made in Section 4.6, "the audio carries what its caption lacks", is somewhat stronger than what the evidence shows. The experiments demonstrate that emotion and sound-source information are verbally readable from the internal states at the audio positions. They do not, however, establish that this information is expressible from audio yet inexpressible in words, because the caption baseline was generated under an instruction to omit emotion. The asymmetry between the audio mind and the caption mind may therefore reflect the captioning instruction rather than a genuine limit of what words can carry. A more informative comparison would be against the model's best effort to describe the clip in words, affect included.

A2: Thank you for this careful reading. We would like to clarify that Section 4.6 compares what is read at the audio positions with the model's own words about the clip, its verbatim transcript and its emotion-free caption: the emotion word read at the audio positions is mostly absent from both (Table 1).

To further address your concern about the captioning instruction, we have run the comparison you propose, against the model's best-effort description with affect included, on every emotion-recognition clip of MMSU [1] (Appendix E.7):

Input the question is answered from (MMSU emotion recognition, n=117)AccuracyBalanced accuracyExact McNemar p vs. audio
Audio64.1%63.9%—
Best-effort description, affect included36.8%35.4%1.8 × 10⁻⁶
Verbatim transcript32.5%31.5%5.7 × 10⁻⁸
Emotion-free caption34.2%33.1%7.9 × 10⁻⁸
Silence23.1%22.5%1.2 × 10⁻⁹

The best-effort description is the model's own account of the clip in words, asked to include every speaker's emotion and tone of voice; it names an emotion or tone on every clip. The four options are affect labels, so the question gives nothing away.

These results verify that, on these clips, the advantage of the audio does not come from the instruction to omit emotion. Following your suggestion, we have retitled Section 4.6 as "What the Audio Positions Carry beyond the Model's Own Transcript and Caption" and rephrased the matching contribution bullet as "it carries more than the model's own words" (Sections 1 and 4.6).

Q3: Section 4.4 uses the 10 clips answered correctly with audio and incorrectly without. What is the denominator of this selection? If only 10 of several hundred MMAU clips truly require the audio, that is a striking property of the benchmark and deserves discussion.

A3: Thank you for this insightful question. We would like to clarify that the denominator of the ten clips is 27: among the first 128 MMAU-mini clips, 31 have a correct answer that is a single vocabulary token; 27 of these are answered correctly with the audio, 17 of the 27 also with the audio zeroed, and the remaining 10 form the patching set.

To further address your concern about the denominator across the benchmark, we have run the same screen over all four-option MMAU-mini clips and patched every clip that needs the audio (Appendix E.2):

All four-option MMAU-mini clipsSoundSpeechMusicAll
Clips332282334948
Correct with the audio272224251747
Correct with the audio and also with it zeroed188116188492
Correct with the audio, wrong with it zeroed (need the audio)8410863255
Of these, answer restored by patching the workspace band at the audio positions94.0%91.7%92.1%92.5%
Of these, answer restored by patching the motor band at the audio positions0.0%2.8%3.2%2.0%

Each clip is run with its audio and with its audio features zeroed. Patching copies the clean run's states of one band into the run with the audio zeroed, at the audio positions.

These results verify that the clips that need the audio come from all three tracks, and that on them the causal pattern of Table 2 holds on every track. This is indeed a striking property of the benchmark: most clips answered correctly with the audio are also answered correctly with it zeroed, since the question and options often name or suggest the answer (for example, "the source of the moo sound"). Accuracy on such items says little about listening, which is why our readout is taken at positions that cannot see the question and is attributed to the sound only through the waveform-swap control. Following your suggestion, we have discussed this property and reported the screen and the patching in Appendix E.2, with a pointer from Section 4.4.

Reviewer_6TGGOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Prior interpretability work focuses on decoding text and words from intermediate layers of language and vision models. This paper extends that work to audio models. In the authors’ experimental setup, they take a model (Qwen3-Omni-30B-A3B-Instruct) and give it an audio clip and a question and answer options. Causal attention is used, so the audio token positions cannot see the question and answer options. The authors show that information that is relevant to the answer options is decodable at audio positions. They show that this decidability is genuinely due to the audio input, i.e., decodability is less possible with mismatched audio. And they attempt to show that the decodability casually influences the ultimately chosen answer.

Critical strengths
  • The paper’s method is clear and easy to follow
  • The decision to include a null comparison is good, i.e., comparing with silent/mismatched audio
  • The findings with respect to the decoded words seem interpretable
Critical weaknesses
  • I believe the text relies too heavily on AI for writing. This is not a bad thing in itself, since AI is a very valuable tool for research. But the result is that many claims are overstated or vaguely stated. For example, the opening sentence, “An audio language model has been a black box: we see what it says, never what it thinks” does not seem justified. It ignores the field of interpretability, of which this paper builds on. Another example of unclear text includes lines line 95 and line 340 (see below). That said, the method itself is clearly described (mentioned above).
  • This sort of interpretability work, even for audio models, has already been explored in “Do Audio-Visual Large Language Models Really See and Hear?” https://arxiv.org/pdf/2604.02605. This paper also decodes intermediate audio representations using the model’s language decoder. This paper does something similar as well: “Transcribe, Translate, or Transliterate: An Investigation of Intermediate Representations in Spoken Language Models” https://arxiv.org/pdf/2510.02569
  • The insight about emotion and two minds are interesting. But Figure 5 and figure 6 make these points with only three and one examples respectively? It would be good to quantify these effects across more examples.
Questions for authors

Major

  • Section 3.5: I don’t follow the reasoning here. You want to test if concepts are used. So you corrupt features at different bands and check when the answer degrades? But the corruption is done by zeroing out features. So you’re removing concept data, but you’re also removing ALL data. How would you remove just concept data only?

Minor

  • Line 95: Minor point, but only the first of these bullets is a contribution. The other three bullets are findings/interpretations.
  • Line 340: “The pattern recurs, specifically and sparsely” — What do you mean by “sparsely”? The word does not appear in the paragraph body.
  • Equation 3: Why take the min over tokens? I thought there was only one token (line 195). If there really are multiple tokens, are you worried about irrelevant tokens dominating?
  • Line 194: why only take the first token? Isn’t that unlikely to hold the answer?
  • 201: “some option tokens are readable at the audio positions whatever the audio contains“ It’s unclear to me how this can happen? How can the audio positions with no information contain relevant decodable tokens?
  • Line 202: “Answer positions are also imbalanced” – why not balance them?
  • Line 223: Why not make the silence longer so it matches in time?
  • Figure 2, can you align the audio transcript with the y-axis, so we can see what words triggered the concepts?
9 comments
AuthorsOpenReview ↗

Reply to Reviewer 6TGG [Part 1]

We sincerely thank you for the careful and constructive review. We greatly appreciate your recognition that "the paper's method is clear and easy to follow," that "the decision to include a null comparison is good, i.e., comparing with silent/mismatched audio," and that "the findings with respect to the decoded words seem interpretable."

We understand your concerns focus primarily on the precision of the writing, the positioning against the two prior works you point to, the quantification of the emotion and two-mind effects beyond Figures 5 and 6, and how concept information alone could be removed, together with several points of clarification.

To address these thoroughly, we have (1) revised the opening sentence, the contribution list and the heading at line 340 for clarity (Sections 1 and 4.2); (2) clarified how both prior works differ from ours, cited both (Section 2), and compared their readouts with ours on the same clips, where ours gains the most from the real clip over a mismatched one (Appendix E.6); (3) split the emotion tally by task and compared answering emotion questions from the audio and from the model's own description, which shows that the audio carries emotion beyond the model's own words (Appendices E.7 and E.9); (4) edited only the decoded concept's lens directions, against placebo edits of the same size, which moves the answer in the predicted direction (Sections 3.5 and 4.7, Appendices E.4 and E.12); (5) rescored every four-option clip with all tokens of each option and by answer position, where real audio stays above both controls (Appendix E.14); (6) added a duration-matched silence condition, which reads out like the 2 s silence (Appendix E.8); and (7) aligned the Figure 2 clip with a two-recognizer transcript, which shows that the event words are spoken and the actor is not (Section 4.2, Appendix E.5). All additions have been integrated into our revised manuscript. Below, we address each specific question systematically.

Q1: I believe the text relies too heavily on AI for writing. This is not a bad thing in itself, since AI is a very valuable tool for research. But the result is that many claims are overstated or vaguely stated. For example, the opening sentence, “An audio language model has been a black box: we see what it says, never what it thinks” does not seem justified. It ignores the field of interpretability, of which this paper builds on. Another example of unclear text includes lines line 95 and line 340 (see below). That said, the method itself is clearly described (mentioned above).

A1: Thank you for this candid comment and for noting that the method is clearly described. We would like to clarify that we have rewritten each sentence you point to for clarity and precision. Following your suggestion:

  • The opening sentence now reads: "Interpretability methods have made the intermediate computation of text language models partly legible, and first lens studies of audio models read what their audio representations contain; much less is known about what an audio language model represents while it listens, before it answers." It cites the first lens studies of audio models, including the two you point to.
  • The contribution list (line 95) now states one methodological contribution and three findings.
  • The heading at line 340 now reads "The pattern recurs on other clips".
  • The Nixon chain, the further cases and the association families are labelled as case studies (Section 4.2), and Section 4.6 is retitled "What the Audio Positions Carry beyond the Model's Own Transcript and Caption".

As our AI use statement notes, AI tools were used only to polish the language.

AuthorsOpenReview ↗

Reply to Reviewer 6TGG [Part 2]

Q2: This sort of interpretability work, even for audio models, has already been explored in “Do Audio-Visual Large Language Models Really See and Hear?” https://arxiv.org/pdf/2604.02605. This paper also decodes intermediate audio representations using the model’s language decoder. This paper does something similar as well: “Transcribe, Translate, or Transliterate: An Investigation of Intermediate Representations in Spoken Language Models” https://arxiv.org/pdf/2510.02569

A2: Thank you for pointing us to these two relevant works. We would like to clarify that Selvakumar et al. [1] study audio-visual LLMs on video with sound, asking how the audio and visual streams fuse, and Ògúnrèmí et al. [2] study the speech adapter of spoken language models, asking what its output encodes, whereas our paper studies an audio language model answering questions on sound, speech and music, with no video, and which concepts it forms from the sound while it listens, before it answers. Like other audio-lens studies, both read audio representations through the model's vocabulary, which Section 2 credits, but our work differs from both in two ways: (i) the question: [1] reads, layer by layer, the sound content of the clip during captioning and finds that, when audio and vision conflict, vision suppresses these audio cues in the output, and [2] finds, from the nearest decoder token to the adapter output, whether it carries the meaning or the phonetics of the speech, whereas we read, across depth, inferred and associated concepts that are never spoken, as well as paralinguistic ones such as emotion; and (ii) the control: [1] swaps audio tracks to build audio-visual conflicts, but, to our knowledge, neither work has a control that separates what the sound supplies from the lens's preference for some option tokens, which our waveform swap with a byte-identical prompt provides.

To further address your concern about how our readout relates to theirs, we have applied both readouts and ours to the same model, clips and prompts (Appendix E.6):

Readout at the audio positionsDepths readClips where an option is readRealMismatchOurs − this readout, real audio
Nearest input-embedding token of the adapter output (adapted from [2])input11.0%25.8%25.5%+12.3 (p < 10⁻⁴)
Top-1 logit-lens token (adapted from [1])all 4919.5%28.2%25.1%+9.8 (p < 10⁻⁴)
Top-10 logit-lens tokens (a more lenient variant)all 4956.6%34.0%27.0%+4.0 (p = 5.5 × 10⁻⁴)
Ours: best rank of each optionall 49100.0%38.0%27.4%—

Balanced accuracy. Each token readout predicts the option whose first token it reads most often (a random guess where it reads none of the four), ours the best-ranked option. Last column: paired sign-flip permutation test over clips.

These results verify that our readout is the most accurate on real audio and gains the most from the real clip over a mismatched one. Following your suggestion, we have cited and discussed both works in Section 2 and reported this comparison in Appendix E.6.

AuthorsOpenReview ↗

Reply to Reviewer 6TGG [Part 3]

Q3: The insight about emotion and two minds are interesting. But Figure 5 and figure 6 make these points with only three and one examples respectively? It would be good to quantify these effects across more examples.

A3: Thank you for this suggestion. We would like to clarify that both effects are already counted across clips in Table 1, beyond the examples drawn in Figures 5 and 6. For the two minds, answering from the audio beats answering from the model's own caption on every MMAU-mini track, and where the two minds disagree on the speaker's role, the audio mind forms the correct role more often:

Two minds (Table 1)nAudioOwn caption
MMAU-mini, sound33380.8%69.7%
MMAU-mini, speech33374.5%68.5%
MMAU-mini, music33475.4%69.8%
Speaker-role clips where the two minds disagree: correct role formed2788.9%70.4%

Answer accuracy, with the question and options identical in both conditions; last row: input-position readout.

To further address your concern about the emotion effect, we have split the emotion tally of Table 1 by task (Appendix E.9):

Emotion clipsnStrong affect readout at the audio positionsof which absent from the transcriptEmotion word in the transcriptEmotion word in the caption
Dissonant Emotion Interpretation35272258
Emotion Flip Detection20121115
All (Table 1)553933613

Transcript and caption are the model's own verbatim transcript and its emotion-free caption of the same clip.

To further address your concern about quantifying the emotion effect in behaviour, we have answered the emotion-recognition questions of MMSU [3] from each input (Appendix E.7):

MMSU emotion recognition (n=117)AccuracyPaired test vs. audio
Audio64.1%—
Model's own best-effort description, affect included36.8%p=1.8 × 10⁻⁶
Silence23.1%p=1.2 × 10⁻⁹

The four options are affect labels, so the question gives nothing away; the description is the model's account of the clip in words, asked to include every speaker's emotion and tone. Exact two-sided McNemar.

These results verify that both effects hold across clips: the emotion read at the audio positions is mostly absent from the model's own words, and the model answers significantly better from the audio than from its own best-effort description. Following your suggestion, we have added the split by task (Appendix E.9) and the MMSU comparison (Appendix E.7), referenced from Section 4.6.

Reviewer_qvFROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether verbalizable internal representations, previously studied in text language models, also arise from audio. The authors apply a logit lens to audio-token positions in Qwen3-Omni that precede the written question and answer options. Real, mismatched, and silent audio conditions are compared to establish whether the readout depends on the sound. Activation patching tests whether audio-derived states affect the answer. Additional examples examine multilingual readouts, emotion, and intermediate concepts in apparent reasoning chains. The authors interpret these observations as evidence for a verbalizable middle-layer workspace.

Critical strengths
  1. The choice of readout positions and the waveform-swap control are well motivated. Since the audio positions precede the question and options, their hidden states cannot directly copy information from that text. Comparing the same readout under real and mismatched audio then provides evidence that the signal depends on the clip. The improvement from 32.2% to 40.0% balanced accuracy in Table 3 is a useful result, particularly given the answer-position imbalance.

  2. The multilingual and emotion examples are interesting cases of information becoming readable before an answer is generated. Comparing the audio readout with the model’s own transcript and caption helps illustrate what those text outputs may omit. I also appreciate the reporting of negative results and limitations, including the nonsignificant audio–transcript difference on the emotion slice and the weak real–mismatch effect on speech-content questions. These qualifications make the empirical findings easier to assess.

Critical weaknesses
  1. The evidence does not yet establish what is distinctive about the proposed middle-layer workspace. Figure 8 shows that the sound-dependent signal appears early and persists through the final layers. In Table 4, the real–mismatch accuracy gap is larger in the motor band than in the workspace band: approximately 20.7 versus 12.9 percentage points. The bands also contain different numbers of layers, which matters when the readout takes the best rank across both layers and positions. Appendix A.1 acknowledges that a size-controlled comparison is missing. I do not expect this paper to test every property of global-workspace theory, but the present results do not clearly distinguish the proposed account from ordinary audio-semantic representations becoming readable across depth. This matters for novelty, since the Semantic Hub Hypothesis and AudioLens already study vocabulary readouts of audio representations.

  2. The causal experiment does not establish that the concepts exposed by the lens are used in the proposed reasoning process. Equation (7) restores the full residual states at all audio positions throughout a band. This establishes a role for audio information, but it does not isolate the readable concepts from the other information carried by those states. The ten tested clips are all from the sound track, and patching the workspace at text positions also restores nine of them. Meanwhile, the Nixon example used to motivate the hidden reasoning chain is answered correctly without audio and is therefore excluded from this test. Its failed model-generated transcription also leaves open whether the event words were actually spoken, as Appendix A.1 acknowledges. The paper consequently has evidence for audio-dependent decodability and for the causal usefulness of audio states, but the connection between these two findings remains untested.

Questions for authors
  1. Additional analysis: Does the depth-dependent real–mismatch effect remain after matching the number of layers and audio positions used by each readout? Please also provide the sample counts and effect sizes separately for sound-source and speech-content questions. These results would help assess whether the middle band has a distinct role.

  2. Clarification: Is there any intervention in the current experiments that isolates the lens-readable concepts rather than restoring complete residual states? If so, please identify the result and explain how it rules out the answer being recovered through other information in the patched states.

  3. Clarification: Which of Watergate, scandal, and Nixon are actually present in the Nixon clip? A verified transcript of the relevant segment would help. Given that the question is answered correctly without audio, what evidence connects the displayed concept sequence to the computation producing this answer?

7 comments
AuthorsOpenReview ↗

Reply to Reviewer qvFR [Part 1]

We sincerely thank you for the careful and constructive review. We greatly appreciate your recognition that "the choice of readout positions and the waveform-swap control are well motivated," that the gain in balanced accuracy "is a useful result, particularly given the answer-position imbalance," and that "the multilingual and emotion examples are interesting cases of information becoming readable before an answer is generated."

We understand your concerns focus primarily on what distinguishes the middle band, how the readable concepts connect to the causal evidence, the matched and per-question-type analyses you request, whether any intervention isolates the readable concepts, and the content of the Nixon clip.

To address these thoroughly, we have (1) blocked the answer's attention to the audio band by band, which shows that the answer reads the audio in the middle band (Appendix E.11); (2) clarified our distinction from the Semantic Hub Hypothesis and AudioLens and compared our readout with top-token logit-lens readouts on the same clips, where ours is the most accurate (Appendix E.6); (3) patched every clip that needs the audio among all four-option MMAU-mini clips, at the audio and at the text positions, which shows the causal pattern of Table 2 on all three tracks (Appendix E.2); (4) compared equal-size bands with depths and audio positions matched and broken out sound-source and speech-content questions, where real audio stays significantly above a mismatched clip throughout (Appendix E.3); (5) edited only the answer concept's lens directions against placebo edits of the same size, which moves the answer in the predicted direction (Appendices E.4 and E.12); and (6) transcribed the Nixon clip with two speech recognizers, which show that Watergate and scandal are spoken and Nixon is not, silenced its heard event words, which removes the inferred actor from the readout, and tested its order across clips (Appendices E.5, E.13 and E.16). All additions have been integrated into our revised manuscript. Below, we address each specific question systematically.

AuthorsOpenReview ↗

Reply to Reviewer qvFR [Part 2]

Q1: The evidence does not yet establish what is distinctive about the proposed middle-layer workspace... (i.e., Critical Weakness 1)

A1: Thank you for this important point. We would like to clarify that what distinguishes the middle band is where the answer reads the audio: the sound content is readable across depth, as you observe, also with depths and audio positions matched (+5.0, +8.0 and +8.1 points in equal-size windows of the three bands, all p < 0.004; Appendix E.3), but blocking the answer's attention to the audio inside the workspace band removes the answer far more often than in any other band, while blocking it inside the motor band, where you note the larger real–mismatch gap, leaves the answer in place on every clip.

To further address your concern about what is distinctive about the middle band, we have blocked the attention from the answer's positions to the audio positions band by band, and in equal-size windows of ten depths (Appendix E.11):

BandAnswer kept, whole band blockedAnswer kept, equal-size window blocked
Sensory (depths 1–16; window 4–13)90.7%94.0%
Workspace (depths 17–38; window 23–32)55.2%86.7%
Motor (depths 39–48; window 39–48)100.0%100.0%

The 248 clips that need the audio (correct with it, wrong with it zeroed) and that the unedited pass answers correctly; with the audio present, every text position after the audio is kept from attending to the audio positions inside the given depths. Lower means the answer reads the audio more there.

These results verify that the answer reads the audio in the middle band and no longer reads it in the motor band, where the content is still readable: blocking the workspace band removes the answer significantly more often than blocking the motor band, and the middle window more often than the early or the late window (exact two-sided McNemar). Following your suggestion, we have reported the blocking in Appendix E.11 and stated in Section 4.7 that the answer reads the audio in the middle band.

Comparison with the Semantic Hub Hypothesis and AudioLens. We would like to clarify that the Semantic Hub Hypothesis [1] shows that audio content is lens-readable in the model's dominant language and AudioLens [2] tracks how surface attributes such as gender and emotion evolve across layers, whereas we ask whether the workspace organization of text models, a verbalizable middle band carrying task-relevant, inferred content, appears for audio. Like both studies, we read audio representations through a vocabulary projection, which Section 2 credits, but our work differs from both in two ways: (i) what is read: [1] reads audio content, [2] surface attributes, whereas we read task-relevant concepts, including inferred ones never spoken; and (ii) the control: to our knowledge, neither has a control that separates what the sound supplies from the lens's preference for some option tokens, which our waveform swap with a byte-identical prompt provides.

To further address your concern about novelty, we have compared our readout with top-token logit-lens readouts, the top-1 token and a more lenient top-10 variant, on the same model, clips and prompts (Appendix E.6):

Readout at the audio positions (all 49 depths)RealMismatchOurs − this readout, real audio
Top-1 logit-lens token28.2%25.1%+9.8 (p < 10⁻⁴)
Top-10 logit-lens tokens34.0%27.0%+4.0 (p = 5.5 × 10⁻⁴)
Ours: best rank of each option38.0%27.4%—

Balanced accuracy; a clip whose readout names no option counts as a random guess; last column: paired sign-flip permutation test.

These results verify that ranking every option is more accurate than reading the top tokens and gains the most from the real clip over a mismatched one. Following your suggestion, we have reported this comparison in Appendix E.6, referenced from Section 2.

AuthorsOpenReview ↗

Reply to Reviewer qvFR [Part 3]

Q2: The causal experiment does not establish that the concepts exposed by the lens are used in the proposed reasoning process. Equation (7) restores the full residual states at all audio positions throughout a band. This establishes a role for audio information, but it does not isolate the readable concepts from the other information carried by those states. The ten tested clips are all from the sound track, and patching the workspace at text positions also restores nine of them. Meanwhile, the Nixon example used to motivate the hidden reasoning chain is answered correctly without audio and is therefore excluded from this test. Its failed model-generated transcription also leaves open whether the event words were actually spoken, as Appendix A.1 acknowledges. The paper consequently has evidence for audio-dependent decodability and for the causal usefulness of audio states, but the connection between these two findings remains untested.

A2: Thank you for this careful summary. We would like to clarify that each point you raise is addressed: patching covers every clip that needs the audio, on all three tracks and at both the audio and the text positions (below); an intervention edits only the readable concept (below); and the Nixon clip has a two-recognizer transcript: "The Watergate scandal brought down a US president," so the event words are spoken and Nixon is not (Appendix E.5).

To further address your concern about the ten sound-track clips and the text positions, we have run the audio-necessity screen that selected the clips of Table 2 over all four-option MMAU-mini clips and patched every clip that needs the audio, band by band, at the audio and at the text positions (Appendix E.2):

Restored band and positionsSound (n=84)Speech (n=108)Music (n=63)All (n=255)
Workspace band, audio positions94.0%91.7%92.1%92.5%
Motor band, audio positions0.0%2.8%3.2%2.0%
Sensory band, text positions35.7%33.3%66.7%42.4%
Workspace band, text positions100.0%100.0%100.0%100.0%

Clips that need the audio: correct with the audio and wrong with it zeroed. Restored: the run with the audio zeroed answers correctly after the clean run's states of one band are copied in at the given positions; text positions are all positions after the audio.

These results verify that the causal pattern of Table 2 holds on all three tracks, and, since the text positions have already attended to the audio in the clean run, that the content the answer needs reaches the answer's positions by the end of the middle band. Following your suggestion, we have reported the screen and this patching in Appendix E.2, with a pointer from Section 4.4.

To further address your concern about connecting the readable concepts with their causal use, we have edited only the decoded answer concept, replacing it by a distractor's, against placebo edits of the same size, on the clips that need the audio (Appendix E.12):

Replace the answer concept by a distractor's (n=205)αAnswer moved to the distractor: concept editplacebo editp
Audio positions, depths 0–380.19.3%2.0%2.7 × 10⁻⁴
0.221.0%6.3%2.3 × 10⁻⁷
0.448.3%18.0%1.7 × 10⁻¹³
Audio and text positions, depths 17–380.114.6%2.4%4.2 × 10⁻⁷
0.229.8%6.3%3.5 × 10⁻¹¹
0.453.2%16.1%1.9 × 10⁻¹⁵

Of the 255 clips answered correctly with the audio and wrongly with it zeroed, the 205 whose answer and distractor begin with different tokens and that the unedited pass answers correctly. The component of each edited state along the direction from the answer concept to a randomly drawn distractor's is shifted by α times the state's norm; the placebo edit does the same along the direction between two unrelated words; exact two-sided McNemar.

These results verify that the readable concept and the content the answer uses are carried in the same audio-position states: moving only the concept moves the answer in the predicted direction.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Global Response [Part 1]

Dear Area Chairs and Reviewers,

Thank you for your time and effort in reviewing our work. Below is our global response:

We sincerely thank all reviewers for their thoughtful and constructive feedback. We are particularly encouraged by the recognition of our readout at the audio-token positions as a clear safeguard against copying concepts from the subsequent question or options; of the waveform-swap control, with a byte-identical prompt, as well motivated and as separating what the sound supplies from the lens's preference for certain option tokens; of the clarity of our method; and of the work as a careful, control-driven analysis of a question that matters for the oversight of audio LLMs. We greatly appreciate the time all reviewers invested in providing constructive suggestions that have significantly strengthened our work.

After careful consideration of the reviewers' comments, we identified the main concerns focusing on: (1) whether the readable concepts themselves are used by the model (Reviewers eYKv, qvFR, 6TGG); (2) the validation of the Nixon chain and of the case studies (Reviewers eYKv, uwGk, qvFR, 6TGG); (3) what distinguishes the middle band (Reviewers eYKv, qvFR); (4) the coverage of the strongest evidence (Reviewers eYKv, uwGk, qvFR); (5) the claim and design of the two-mind comparison (Reviewers uwGk, 6TGG); (6) the positioning against prior lens readouts of audio (Reviewers 6TGG, qvFR); and (7) the precision of the writing and points of clarification (Reviewers 6TGG, eYKv).

We have comprehensively addressed these concerns through clarifications and extensive additional experiments in our revision, with the same model, conditions and tests, over all four-option MMAU-mini clips where the question calls for it. The results are reported in Appendix E (Supplementary Experiments), and the main direct tests are summarized, with a summary figure, in Sec. 4.7. Below, we summarize the clarifications and additional experiments contributed during the rebuttal to address the reviewers' concerns:

Whether the Readable Concepts Are Used (Sec. 3.5, Sec. 4.7, App. E.4, App. E.12): To address concerns about whether the readable concepts are used, we clarify that a decoded concept can be edited alone: a token's lens logit is proportional to the projection of the state onto one direction, so we edit only that component and leave the rest of the state unchanged. To further address this concern, we have replaced only the decoded answer concept by a distractor's, against a placebo edit of the same size (Appendix E.12):

Replace the answer concept by a distractor's (audio positions, depths 0–38; n=205)Answer moved to the distractor: concept editplacebo editp
α = 0.19.3%2.0%2.7 × 10⁻⁴
α = 0.221.0%6.3%2.3 × 10⁻⁷
α = 0.448.3%18.0%1.7 × 10⁻¹³

Shift of α times the state's norm along the direction from the answer concept to a randomly drawn distractor's; placebo: the same along the direction between two unrelated words. Clips answered correctly with the audio and wrongly with it zeroed whose answer and distractor begin with different tokens and that the unedited pass answers correctly; exact two-sided McNemar.

These results verify that the readable concept carries part of what the answer uses, in the direction the lens reads it; removing it, on the 26 clips whose answer is clearly readable, also lowers the answer's margin significantly more than removing placebo directions in the workspace band (two-sided sign test; Appendix E.4).

AuthorsOpenReview ↗

Global Response [Part 2]

The Nixon Chain and the Case Studies (Sec. 4.2, Sec. 4.7, App. E.1, App. E.5, App. E.13, App. E.15, App. E.16): To address concerns about the Nixon chain, we clarify that the event is spoken and the actor is not, so the actor is inferred from what is heard: two independent speech recognizers agree word for word (up to the spelling of US): "The Watergate scandal brought down a US president." We have aligned this transcript with Figure 2 (Appendix E.5). To further address this concern, we have silenced the heard event words in the Nixon clip's waveform, against silencing the same duration of other speech in 50 random draws (Appendix E.16):

Nixon clip: waveformAnswer concept: best rank at the audio positionsNixon: best rank at the audio positions
Unedited51
Heard event words "scandal brought" silenced205950
Same duration of other speech silenced (50 random draws)1–31

Answer: the correct option's first token. Best rank over the audio positions and all depths; the prompt is unchanged; one-sided permutation test over the 50 draws, p=0.02 for both.

We have also tested the depth order of Figure 2 on every speech clip whose correct answer is never spoken (Appendix E.13):

Speech clips whose correct answer is never spoken (n=198)Clips where both become readableHeard word first vs. answer firstp
All clips8976 vs. 85.0 × 10⁻¹⁵
Event-Based Knowledge Retrieval, the category of the Nixon clip1515 vs. 06.1 × 10⁻⁵

Readable: rank 10 or better at the audio positions; two-sided sign test on which becomes readable first, clips where both do so at the same depth left out.

To further address the concern about the case studies, we have tested the association pattern on every four-option clip, with words the model itself associates with the correct option, neither printed nor spoken (Appendix E.15):

Related words neither printed nor spoken (all four-option MMAU-mini clips)Real clip more vs. mismatched clip morep
Pre-specified filter334 vs. 1871.2 × 10⁻¹⁰
Stricter: no shared stem with any printed or spoken word, outside the top-100 neighbours of every answer word315 vs. 1811.9 × 10⁻⁹

Per clip, how many of these words are among the top 10 lens tokens at the audio positions with the real and with a mismatched clip, the prompt unchanged; two-sided sign test, ties left out.

These results verify that silencing the heard event words removes the inferred actor from the readout of the Nixon clip, that, on most clips where both become readable, a heard word becomes readable at a shallower depth than the inferred answer, as on the Nixon clip, and that the readout reaches unspoken related words of what is heard. We have labelled the Nixon chain, the further cases and the association families as case studies.

What Distinguishes the Middle Band (Sec. 4.4, Sec. 4.7, App. E.2, App. E.3, App. E.11): To address concerns about the middle band, we clarify that what distinguishes it is where the answer reads the audio: the content is readable in every band, but the answer takes it from the audio positions in the middle band. To further address this concern, we have compared equal-size bands, patched each band at the audio and the text positions, and blocked the answer's attention to the audio band by band (Appendices E.2, E.3 and E.11):

BandReal − Mismatch, equal-size window of 10 depths (p)Answer restored by patching the band: audio positionstext positionsAnswer kept with its attention to the audio blocked: in the bandin the equal-size window
Sensory+5.0 (0.0036)—42.4%90.7%94.0%
Workspace+8.0 (6.4 × 10⁻⁷)92.5%100.0%55.2%86.7%
Motor+8.1 (5.1 × 10⁻⁹)2.0%—100.0%100.0%

Windows: depths 4–13, 23–32 and 39–48, audio positions matched; balanced accuracy. Patching: the 255 clips answered correctly with the audio and wrongly with it zeroed; blocking: the 248 of these answered correctly before blocking (lower = the answer reads the audio more there; workspace vs. motor band p=7.7 × 10⁻³⁴, middle vs. early and late window p=0.003 and 2.3 × 10⁻¹⁰). Exact two-sided McNemar.

These results verify that the answer's reading of the audio peaks in the middle band and is complete by its end, while the content stays readable to the output.

AuthorsOpenReview ↗

Global Response [Part 3]

Coverage of the Strongest Evidence (Sec. 4.4, App. E.1, App. E.2, App. E.3): To address concerns about coverage, we clarify that the readout holds over all four-option MMAU-mini clips and the patching over every one of them that needs the audio: over all four-option MMAU-mini clips, real audio is read out above both a mismatched clip and silence (exact two-sided McNemar; Table 8). To further address the concern about the ten patching clips, we have run the audio-necessity screen over all four-option clips and patched every clip that needs the audio (Appendix E.2):

Clips that need the audio (correct with it, wrong with it zeroed)SoundSpeechMusicAll
Clips8410863255
Answer restored by patching the workspace band at the audio positions94.0%91.7%92.1%92.5%

Over all four-option MMAU-mini clips.

To further address the concern about speech-content questions, we have run the waveform-swap control by question type (Appendix E.3):

Question typeDepths readnReal − Mismatchp
Sound: source identificationmotor band94+10.60.024
Sound: source identificationall 4994+19.01.1 × 10⁻⁴
Speech: content (what is said)motor band105+14.50.0046
Speech: content (what is said)all 49105+15.00.0029

Balanced accuracy; exact two-sided McNemar.

These results verify that the readout and the patching extend beyond the 140 core items and the ten sound clips to all three tracks, including speech-content questions.

The Two-Mind Comparison (Sec. 4.6, App. E.7, App. E.9): To address concerns about the two-mind comparison, we clarify that Sec. 4.6 compares what the audio positions carry with the model's own words, and have retitled it as what the audio positions carry beyond the model's own transcript and caption. To further address the concern about the caption baseline, we have answered the MMSU [5] emotion-recognition clips from the audio and from the model's best-effort description, affect included (Appendix E.7):

Input the question is answered from (117 MMSU emotion-recognition clips)AccuracyExact McNemar p vs. audio
Audio64.1%—
Best-effort description, affect included36.8%1.8 × 10⁻⁶
Verbatim transcript32.5%5.7 × 10⁻⁸
Emotion-free caption34.2%7.9 × 10⁻⁸
Silence23.1%1.2 × 10⁻⁹

Each question asks how the speaker feels and the options are affect labels; the description names an emotion or tone on every clip.

These results verify that the model recovers the speaker's emotion from the audio significantly better than from its own best-effort description; we have also split the emotion tally of Table 1 by task (Appendix E.9).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.