acceptodds
Under review as a conference paper at ICLR 2027

Long-Term Memory-Guided Enhancement for Target Perception in Audio-Language Models

Abstract

Audio large language models (ALLMs) can reason about the content of audio recordings to perform complex tasks. However, these capabilities usually collapse in real-world environments when background noise and competing sources mix the target sound. Inspired by long-term memory in human listening, we propose Long-Term Memory-Guided Audio Enhancement (LTM-AE) to improve selective target perception by refining the audio representations of ALLMs without training. LTM-AE extracts representations in hidden states from separate clean reference recordings as long-term memory for each category, guiding enhancement toward a user-specified listening target. We reconstruct incoming audio tokens in the selected category long-term memory and interpolate the reconstructions with the original tokens before language backbone decoding. This interpolation controls the influence of stored auditory experience while keeping all ALLM parameters fixed. Diagnostic readouts across twenty sound categories and three ALLMs show that LTM-AE strengthens responses to a specified target amid three interfering sources. Averaged over constrained and free-form classification, accuracy gains over raw mixtures range from 29.53 to 46.15 percentage points across multiple open source models. For speech content recovery, LTM-AE with an additional learned token-level gate reduces Qwen2-Audio's word error rate from 23.07% to 14.77%. This work takes an initial step toward using principles of human long-term memory to enhance ALLMs for real-world listening.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Audio Large Language Models, Robustness, Long-Term Memory

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_BEawOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Long-Term Memory-Guided Audio Enhancement (LTM-AE), which builds a category-specific PCA memory from clean audio tokens and uses a user-specified target category to select a memory at inference time. It reconstructs and interpolates audio tokens before they enter a frozen audio-language model. Experiments on 20 sound categories and three models report gains in constrained and free-form classification, as well as target-conditioned retrieval. A separate speech-transcription experiment adds a learned token-level gate and reports improved WER on Qwen2-Audio.

Critical strengths
  1. The method is clearly specified and does not update the audio-language model’s parameters. The rank and interpolation weight are calibrated on a separate validation set, and the paper reports results across multiple memory sizes. The authors also discuss inference cost and compact memory storage.
  2. The empirical study covers three model interfaces and 20 sound categories. It includes class-wise analyses, a PCA/SVD solver cross-check, and an absent-category control. Improvements in language outputs provide useful evidence beyond representation-level retrieval.
Critical weaknesses
  1. The main interpretive question is whether LTM-AE improves target perception or steers the representation toward the selected category. The enhancement uses that category’s PCA memory, and calibration minimizes error against paired clean target embeddings. Retrieval is then measured against clean category examples, so it should be viewed as representation-alignment evidence rather than independent confirmation of recovered perception. This does not negate the language-output gains, but the paper lacks wrong-category-memory and simpler matched controls, such as a category mean/prototype or a dimension-matched subspace. The absent-category control tests piano memory on 100 examples with Qwen2-Audio only.
  2. The main benchmark is a controlled synthetic setting: three-second, mono mixtures at a fixed −10 dB target-to-interference ratio, with the target category known and present. The results establish effectiveness in this setting but leave performance on natural mixtures and absent or mis-specified targets unclear. The speech result is a separate learned extension: ungated LTM-AE worsens WER, while the gate improves it in a single-model experiment.
Questions for authors

Q1: Do wrong-category memories or simpler matched category-conditioned controls produce similar gains? Results from these controls would help determine how much of the improvement is specific to the proposed PCA memory.

Q2: Is the clean retrieval gallery source-group-disjoint from the memory, validation, and evaluation recordings? For speech transcription, are training, selection, and test sets disjoint by speaker and original recording?

Reviewer_U234OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents LTM-AE (Long-Term Memory-Guided Audio Enhancement) as a training-free method to improve the ability of ALLMs by perceiving target sounds in complex, multi-source audio mixtures. Inspired by human long-term auditory memory, LTM-AE extracts hidden state representations from clean reference recordings of specific sound categories and stores their mean and principal components. At inference time, when a user specifies a target category, incoming mixed audio tokens are projected onto this stored memory subspace and interpolated with the original tokens before being passed to the frozen language backbone. The rank and interpolation weight are calibrated offline using paired clean/mixed validation data.

Critical strengths
  1. It is training-free without fine-tuning the massive language or audio backbones.
  2. The experiments are conducted across three distinct ALLMs.
  3. In the diagnostic readouts, three distinct evaluation protocols provide insight into where the method helps.
Critical weaknesses
  1. The MultiPerception benchmark relies on artificially synthesized mixtures, at a fixed -10 dB target-to-interference ratio with exactly 3 interferers. This is quite far from the real-world acoustic environments.
  2. The paper assumes that the target category is known so that the correct memory matrix can be selected. It does not solve the blind source separation. The paper also lacks analysis on what happens if the specified category is incorrect or ambiguous.
  3. The baseline is limited. The paper does not compare against other enhancement methods and target-sound extraction methods. It is difficult to assess whether the proposed method is superior to simply cleaning the waveform first. Furthermore, the dataset for speech transcription is too small.
Questions for authors
  1. It is mentioned that calibration minimizes reconstruction error on "paired validation clean / mixed token pairs". Have you tested how sensitive the chosen rank and interpolation weights are?
  2. What happens to the output if the user specifies the wrong target category?
1 comment
AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer,

Thank you again for your valuable comments. As the discussion period is nearing its end, we would like to kindly ask whether our responses have adequately addressed your concerns. Please let us know if any further clarification is needed. If our responses have resolved your concerns, we would sincerely appreciate your consideration in updating your evaluations accordingly.

Reviewer_ggStOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Long-Term Memory-Guided Audio Enhancement (LTM-AE), a method for improving target sound perception in audio large language models (ALLMs) under acoustic interference. The approach constructs category-specific memories by applying PCA to audio token representations extracted from clean reference recordings. At inference time, a specified target category selects the corresponding memory, and the mixture embeddings are projected onto the learned subspace and interpolated with their original representations. The ALLM parameters remain frozen.

The authors evaluate LTM-AE on MultiPerception, a synthetic benchmark covering 20 sound categories and mixtures containing one target and three interfering sources at a fixed target-to-interference ratio of −10 dB. Experiments with Qwen2-Audio, Kimi-Audio, and Step-Audio-2-mini show improvements in constrained classification, free-form classification, and representation-level retrieval. An additional speech transcription experiment introduces a learned token-level gate, reducing Qwen2-Audio's WER from 23.07% to 14.77% on 100 held-out LibriSpeech mixtures.

Critical strengths
  1. Simple, interpretable, and computationally efficient representation-level intervention. LTM-AE provides a straightforward mechanism for incorporating category-specific acoustic information into frozen ALLMs. The projection and interpolation formulation is mathematically transparent, and the additional inference complexity is (O(Tdk)). The method is also evaluated across three models with different audio interfaces, suggesting that the general approach is not tied to one particular architecture. This is a practical strength for investigating lightweight adaptation of existing audio-language models.

  2. Substantial empirical improvements with detailed class-level reporting. The evaluation spans 20 speech and non-speech categories and three diagnostic protocols. The improvements are substantial across all three ALLMs, including gains in generated language outputs rather than representation-level retrieval alone. The paper also provides extensive per-class results and multiple memory sizes in the appendix, making the reported aggregate improvements relatively transparent and easier to inspect.

Critical weaknesses
  1. Ground-truth category conditioning confounds the evaluation.

LTM-AE uses the true target category to select its PCA memory, while all three readouts evaluate alignment with that same category. The raw-mixture baseline receives no equivalent target cue. Thus, the reported gains may partly reflect category-prior injection rather than improved acoustic perception. This concern is supported by enhanced mixtures occasionally outperforming clean audio (Table 1) and large retrieval gains that do not consistently translate into classification accuracy. Moreover, the piano target-absent control (Appendix D.4) is weak because Qwen2-Audio also fails to identify piano when it is actually present (Table 5). Essential controls include category-mean-only projection, mismatched-category memories, and target-absent mixtures across multiple categories.

  1. Missing matched baselines and insufficient evidence of content preservation.

The baseline is asked to identify the "main sound" without a target cue, although the designated target is deliberately quieter than three interfering sources. Comparisons with equally target-conditioned prompting, simpler embedding interventions, or target-conditioned separation methods are missing. More importantly, the only experiment requiring recovery of information beyond the category label, speech transcription, shows that ungated LTM-AE worsens WER from 23.07% to 33.75%. The learned gate improves WER to 14.77%, but without a parameter-matched gate-only baseline, this improvement cannot be attributed to PCA memory. The 20-example gate-selection set and lack of uncertainty estimates further weaken this conclusion.

Additional concerns: Figures 3 and 5 use test-set best-of-NN results; several categories may contain dataset-specific recording signatures; evaluation covers only synthetic mixtures at one interference level; and the novelty relative to classical subspace enhancement and representation steering is insufficiently discussed.

Questions for authors
  1. What are the accuracies across all three models and readouts when all audio tokens are replaced with the target-category mean (k=0,α=1k=0,\alpha=1)? Additionally, when an incorrect or absent category's memory is applied, how often does the model predict that category?

  2. How does LTM-AE compare with baselines receiving the same target-category information, such as target-conditioned prompting or text-queried audio separation (e.g., AudioSep or CLAPSep)?

  3. Can the authors report the selected (kc,αc)(k_c,\alpha_c) and calibration grid GG, and clarify whether the retrieval gallery is source-group-disjoint from evaluation recordings?

  4. What WER does a parameter-matched gate trained with identical supervision but without PCA memory achieve? Can the authors report variability across seeds? Given that ungated LTM-AE worsens WER from 23.07% to 33.75%, this control is important for attributing the gated improvement to the proposed memory mechanism.

3 comments
AuthorsOpenReview ↗

[1/2]Response to Reviewer ggSt

Thank you for your detailed suggestions on the method and evaluation.

To examine whether category priors explain the gains, we added structural controls and baselines with matched target information. We also examined parameter selection and data partitioning.

For W1 & Q1

Our motivation draws on how humans use listening experience to interpret current sounds. For example, a music expert familiar with an instrument may identify an ambiguous recording of it more accurately than an ordinary listener hearing the same recording. The expert does not receive a clearer signal, but prior memory can help interpret incomplete acoustic cues. LTM-AE similarly uses category memory to interpret the current audio. Moving representations toward the target category is therefore consistent with its design and does not by itself imply that the model is directly repeating a known answer. The additional category prior may also explain why recognition accuracy for some categories exceeds that of clean audio without memory intervention.

We added category-mean replacement (k=0, α=1) and a shared category-independent subspace using the corresponding category’s k and α. LTM-AE outperforms both controls on both classification metrics, supporting the role of category-specific variation combined with the current input. The table reports classification accuracy (%).

MethodConstrainedFree-form
Clean target66.9032.05
LTM-AE47.2026.25
Raw mixture9.604.80
Shared subspace5.790.51
Category mean only5.000.00

The key question is whether the gains require the current input and category memory to work together. If supplying a category prototype were sufficient, mean replacement should achieve comparable classification performance. Instead, its constrained and free-form accuracies are only 5.00% and 0.00%, versus 47.20% and 26.25% for full LTM-AE. This difference supports a contribution from input-dependent memory reconstruction beyond the tested category-prototype replacement, consistent with our design of combining acoustic evidence with memory.

We additionally assigned an absent category to each of the same 2,000 mixtures and supplied it through either an incorrect-category prompt or incorrect-category memory. Both conditions use the same assigned category per sample, absent from the target and all three interferers. The table measures the rate of predicting that injected category.

ConditionIncorrect-category following rate (%)
Incorrect-category prompt99.45
Incorrect-category memory39.75

For W2 & Q2

We added direct prompting and AudioSep with the same target-category information, together with fixed-direction representation steering and signal-domain log-spectral PCA. Classification accuracies (%) are shown below.

MethodConstrainedFree-form
Target-category prompt99.8378.37
AudioSep23.4811.51
Fixed-direction steering10.865.15
Signal-domain log-spectral PCA6.711.54
LTM-AE47.2026.25

LTM-AE outperforms AudioSep on both classification metrics. It also outperforms fixed-direction steering and the tested spectral PCA control. Direct prompting obtains higher scores, but its prompt contains the category being scored. To test whether these scores reflect recognition of the audio, we supplied an incorrect category absent from each mixture and measured how often the model reported it.

ConditionIncorrect-category following rate (%)
Incorrect-category prompt99.45
Incorrect-category memory39.75

In constrained classification, target-category prompts lead the model to report the specified category almost always, whether that category is correct or incorrect. In particular, the model reports the prompted category in 99.45% of cases even when it is absent from the audio. Thus, the high accuracy with correct prompts largely reflects following the supplied category and does not establish reliable recognition of the acoustic target. Incorrect-category memory produces a lower following rate of 39.75% in the same control.

Whereas fixed-direction steering shifts representations along a predefined direction, LTM-AE uses the current audio to determine how category memory contributes to reconstruction. Compared with enhancement methods that first process the waveform, LTM-AE uses category memory within the audio-language model’s internal representations to improve target perception. The controls above support this design. LTM-AE achieves higher accuracy on both classification metrics than the tested fixed-direction steering and signal-domain PCA methods.

AuthorsOpenReview ↗

[2/2]Response to Reviewer ggSt

For W2 & Q4

To isolate the contribution of PCA memory, we compared vanilla gate against gate + PCA memory under the same gate architecture. All other experimental conditions were held fixed, with PCA memory as the only changed factor. The table reports Qwen2-Audio WER as mean and sample standard deviation over three training seeds (%, lower is better).

MethodWER (%)
vanilla gate18.14 ± 2.82
gate + PCA memory14.89 ± 0.95

Adding PCA memory lowers WER in all three training seeds. Mean WER decreases from 18.14% to 14.89%, a reduction of 3.25 percentage points or 17.90% relative. This shows that memory provides an additional improvement under the same gate and training conditions, so the speech gains are not attributable solely to the gate or additional supervision.

For Q3

For classification, the calibration grid is k∈{8,16,32,64,128} and α∈{0.0,0.1,…,1.0}, selected by the lowest paired clean/mixture token MSE on validation data. All 20 Qwen2-Audio categories select k=8. The complete distribution of α is given below.

Categorykα
Dog vocalization81.0
Classical guitar80.8
All remaining 18 categories80.9

The retrieval gallery is source-group-disjoint from other partitions within each category, with no identical recordings found across partitions.

For Additional concerns

To address test-set best-of-N selection in Figures 3 and 5, we added results selecting N∈{20,50,100} for each category solely by validation MSE. Of the 20 categories, 3 select N=20. Another 6 select N=50. The remaining 11 select N=100. Constrained and free-form accuracies are 48.85% and 26.98%. Selection does not use test accuracy.

We also held the target and interference clips fixed and extended the target-to-total-interference ratio to −5 dB and 0 dB. We also evaluated +5 dB. The table reports classification accuracy (%), with each cell showing raw mixture → LTM-AE.

Target-to-total-interference ratioConstrainedFree-form
−10 dB9.60 → 47.204.80 → 26.25
−5 dB14.75 → 49.857.75 → 27.71
0 dB21.39 → 53.8010.85 → 28.76
+5 dB29.41 → 57.9615.43 → 30.21

LTM-AE improves both classification metrics at all four interference levels. We use synthetic mixtures to independently control the target sound and interfering sources. We also control the target-to-total-interference ratio. All methods process identical mixtures. In natural recordings, source composition and relative levels are generally difficult to control precisely, and corresponding isolated clean-target references are difficult to obtain. Controlled mixing reduces these confounds and allows a more direct assessment of the method under different interference conditions. The supplementary experiments follow this controlled design and do not yet include evaluation on naturally recorded mixtures.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer,

Thank you again for your valuable comments. As the discussion period is nearing its end, we would like to kindly ask whether our responses have adequately addressed your concerns. Please let us know if any further clarification is needed. If our responses have resolved your concerns, we would sincerely appreciate your consideration in updating your evaluations accordingly.

Reviewer_19yhOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LTM-AE, which enhances a specified sound within an audio mixture by modifying an audio-language model’s internal representations. Embeddings obtained from clean recordings from the target class are used to identify the principal subspace for the class using PCA. In the mixture data, components of the mixture tokens that fall on this subspace are enhanced. The degree of enhancing is calibrated using paired clean and mixed recordings. The rest of the processing, whether it is labelling, or description or transcription, proceeds from these enhanced embeddings.

Experiments across three models and twenty categories show substantial classification and retrieval gains, though ASR requires additional trained token-level gates (without which performance actually degrades).

Critical strengths
  1. The proposed method hearkens back to model-based methods of the 2000s. It proposes a simple, elegant reusable technique for enhancing a target class by boosting components in its principal subspace.

  2. The method is generic and applies to any model, and potentially applies to any task.

  3. It does not require any modification or tuning of the model itself.

  4. For ASR, with the learned token-level gate, transcription WER falls by 40%!

Critical weaknesses
  • The baseline comparison is not truly informative. In the raw-mixture baseline the model will naturally gravitate towards the dominant sources, since it receives no target cue. What the proposed method is doing is actually to 'guide' the model towards the target sound. It does not necessarily mean that sound is enhanced. To give an example, if we had two speakers, one in the foreground, and one in the background, raw-mixture processing would capture the foreground speaker. An example of the background speaker would permit the model to 'steer' towards it. This does not necessarily mean that speaker's speech will be correctly transcribed -- it has been steered towards, not enhanced. This goes against the paper’s central claim. But this argument is supported by the point below.

  • When applied to speech, the transcription WER actually goes up from 23.07% to 33.75%. Improvement to 14.77% requires a supervised token-level gate, arguing that my first point is valid.

  • I would want to see a cleaner experiment, or some other evidence of enhancement.

Questions for authors
  1. Do you have comparisons in which the raw-mixture baseline or another enhancement method receives the same target-category information as LTM-AE? Without such comparisons, how do you distinguish improved perception from category-conditioned steering?

  2. Have you compared the category-specific PCA filter with simpler controls, such as interpolation toward the category mean alone or projection onto a shared, category-independent subspace? These would clarify whether the stored within-category variation is essential to the reported gains.

  3. For transcription, do you have a matched baseline trained with the same clean-token and transcript supervision but without category-specific memory? Given that ungated LTM-AE worsens WER, this comparison would help establish whether memory contributes beyond the additional supervised enhancement.

1 comment
AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer,

Thank you again for your valuable comments. As the discussion period is nearing its end, we would like to kindly ask whether our responses have adequately addressed your concerns. Please let us know if any further clarification is needed. If our responses have resolved your concerns, we would sincerely appreciate your consideration in updating your evaluations accordingly.

Comments on the paper

1 comment

Dear Area Chair and Senior Area Chair,

Reviewer BEaw (2→3) raised their score after the calibrated mean-interpolation experiment and our clarifications on incorrect targets and speaker overlap. The table summarizes the results and clarifications relevant to concerns shared across reviews. Parentheses indicate reviewer scores.

Reviewer concernRaised byRebuttal result
Can movement toward the category mean explain the gains?BEaw (2→3), ggSt (2), 19yh (3)Calibrated mean interpolation reaches 98.65% retrieval accuracy, but only 5.46% constrained and 1.64% free-form accuracy. LTM-AE reaches 47.20% and 26.25% on the two classification tasks.
What happens with an incorrect target?BEaw (2→3), ggSt (2), U234 (2)Across 2,000 mixtures, the model predicts the absent category 39.75% of the time with incorrect memory and 99.45% with an incorrect prompt.
How does target-conditioned separation compare?ggSt (2), U234 (2), 19yh (3)AudioSep reaches 23.48% constrained and 11.51% free-form accuracy, compared with 47.20% and 26.25% for LTM-AE.
Do gains hold at other interference levels?BEaw (2→3), ggSt (2), U234 (2)LTM-AE improves both classification measures over raw mixtures at −10, −5, 0, and +5 dB.
Does memory improve transcription under matched supervision?ggSt (2), 19yh (3)Adding PCA memory lowers WER in all three seeds, from 18.14% to 14.89% on average.
Do the speech partitions share speakers?BEaw (2→3)The partitions are utterance-disjoint. Of 100 test utterances, 56 come from speakers seen during gate training and 44 from unseen speakers.

We responded in detail to ggSt (2) and U234 (2). Neither reviewer posted a follow-up assessment. We would appreciate your consideration of these results when weighing their original concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.