- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how irrelevant historical speech affects Speech-LLM answers. Through causal interventions on two models, the authors show that historical interference can persist through intermediate hidden states, even after direct access to the source is blocked. They further identify distinct source-role and answer-value influences, suggesting that correct source identification alone is insufficient to prevent memory interference.
Critical strengths
The work focuses on an important challenge in Speech-LLM memory. The authors provide controlled experiments across two models, offering evidence of historical interference. The combination of attention masking, state transfer, and subspace interventions provides causal insights beyond behavioral evaluation.
Critical weaknesses
My main concern lies in the experimental design and interpretation. The authors do not clearly explain why the role subspace is constructed before the value subspace, as reversing this order may lead to different results. The role subspace is also derived from hidden-state differences between different questions, but these differences may reflect variations in question wording, semantics, or answer content rather than source-role information alone. Additionally, the role intervention changes the model's answer preferences, but this does not necessarily mean that the model has correctly identified the relevant source. Since the role subspace is derived from hidden-state differences between questions. Without directly verifying whether the model identifies the correct source, it remains unclear whether historical interference persists despite correct source identification.
Questions for authors
Could you test whether reversing the construction order of the role and value subspaces changes the results?
Could you use different question templates and paraphrases to verify that the role subspace captures source-role information rather than question wording or answer content?
Thank you for the specific questions. We tested both construction orders and frozen-basis wording controls, and measured source associations independently. Question/content contrasts below correspond to the original role/value terminology. References are to the revised manuscript.
1. Why question-first, and what changes on reversal?
Question-first asks whether historical content outside the fitted question directions still affects the answer: the joint-minus-role-only test in original Section 3.3, Eq.(3). We fit question differences with history fixed, remove their projection from content differences, then fit the residual. Reversal changes this reference comparison, not an assumed processing order (Appendix D.1, Eq.10, p.27).
The subspace materials contrast an original arrangement and its update; the receiving question asks about the original. Both orders were evaluated at all 36 layers on 512 test scenarios, with bases fitted on 128 development scenarios. At layer 23, question-only mean shifts under question-first/content-first are 2.154/1.993 logits in MOSS and 0.811/0.234 in MiMo (Table 17, p.30).
Why do effects change? Let A/B denote question-first/content-first, PO=PQO+PCO, and dQ,dC the fixed donor contrasts. The joint edit is eO=PQOdQ+PCOdC. Eq.(11), p.28 gives, before rounding:
eB−eA=(PB−PA)dC+(PQB−PQA)(dQ−dC).
The first term reflects a change in the retained joint span. The second reflects which donor contrast supplies coefficients within it. Even an unchanged joint span need not preserve the edit. MOSS's joint-span overlap is 0.984, yet its joint margin shift changes from 2.719 to 2.663. Question-only edits instead differ by (PQB−PQA)dQ. After subsequent computation, output changes depend on the final-state difference along the answer-readout direction (Eq.12). These tests retain projected magnitudes, so direction and amplitude both change; the output effects are not separately attributed to the two terms above.
The shared finding is additional content influence. At layer 23, joint-minus-question-only effects are 0.565/0.670 logits in MOSS and 0.405/0.511 in MiMo. This mean effect favors the unasked answer at all five prespecified reporting layers, in both models and orders, with every pointwise interval above zero (Table 16, p.29; complete-audio-group resampling).
2. Do the directions reflect wording or source role?
With bases frozen from the original templates, we test receiver paraphrases, donor paraphrases, and same-event wording differences that preserve the requested event and answer. Tests cover two paraphrases per task family, all 512 scenarios, both orders, and five layers (Appendix D.2-D.3, Tables 18-20, pp.30-33).
At layer 23, question-first receiver effects for original/p1/p2 wording are 2.154/2.117/1.969 logits in MOSS and 0.811/0.726/0.813 in MiMo. Same-event rewording also contributes: the MOSS original-event p2 control gives 0.114 [0.079, 0.150] logits.
The effect therefore transfers across tested phrasings, but paraphrases preserve both event and answer, whereas switching events can change both. We use question/content contrasts operationally, not as pure semantic variables. Source associations are tested independently.
3. What verifies source associations on original errors?
In Section 4.1 (Figure 3 and Table 4, pp.5-6), three linear probes read layer-17 states: one identifies the requested utterance, and two identify each utterance's associated option. They are fitted on development data and frozen. The model answers without their predictions.
All three are correct on 488/617 MOSS and 206/218 MiMo distractor errors (79.09%/94.50%). Swapping utterance order while fixing questions/options reverses source assignments. All three probes remain correct in both orders on 85.23%/97.46% of 4,096 pairs; question-to-source random-label controls are near 50% (Appendix A.3, pp.15-17). An unchanged question-to-position mapping cannot satisfy both orders.
Among correct/error pairs where all three probes are correct in both unmodified runs, transferring wrong-run option Keys and Values to the correct partner reproduces the specific wrong answer in 190/215 MOSS and 83/118 MiMo pairs (Section 4.4; Table 13, p.22). Only the distractor differs; question/options stay fixed. Transfers act on later reads of option vectors, not entire answer states.
Thus, the associations needed for a correct answer remain recoverable while later option reads can favor the distractor. This does not assert that the decoder already selected the correct source, and does not depend on pure role/value separation.