The Right Source Is Not Enough: Understanding History Interference in Speech-LLM Memory

Speech large language models (Speech-LLMs) rely on audio history to support multi-turn conversations. Existing work on conversational memory has largely examined whether past information is retained and can be retrieved when needed. We study a different failure mode in which irrelevant historical audio changes an answer even when the relevant event remains identifiable. Using controlled multi-source histories and causal interventions on two Speech-LLMs, we find that irrelevant history systematically shifts predictions toward values mentioned in another event. Internal interventions show that the queried event remains represented and causally effective, ruling out source identification as the main explanation. The causal path further depends on answer format. In open-ended QA, history-value errors largely disappear once later computation can no longer access the irrelevant audio, whereas many multiple-choice errors persist because historical influence has already entered question-candidate states. Layer-wise interventions reveal a non-serial decision process. Sensitivity to answer-value information is strongest in intermediate layers, while sensitivity to source role becomes stronger deeper in the network. On natural failure cases, strengthening the correct source role and neutralizing the competing historical value can each recover errors, with their joint intervention producing the strongest recovery. These results identify historical interference as competition between source-role and answer-value influences during downstream decision making, rather than a general long-context degradation or a failure of source identification alone.