acceptodds
Under review as a conference paper at ICLR 2027

EchoDistill: Robust Large Audio Language Models via Noisy-to-Clean Self-Distillation

Abstract

Large Audio Language Models (LALMs) remain vulnerable to acoustic noise, which can obscure task-relevant evidence and produce unreliable responses. We propose EchoDistill, a noisy-to-clean self-distillation framework that uses clean audio as privileged information during post-training. A noisy-input student samples candidate responses reflecting its inference-time behavior, while a frozen copy of the same backbone processes the corresponding clean audio. EchoDistill combines masked response-token distillation, task-gated consistency shaping, and teacher-referenced group-relative optimization to align noisy-input generation with clean-conditioned semantics. Only the student is retained at inference time, introducing no additional inference cost. Across three LALM backbones and three audio domains at -10dB, EchoDistill improves average noisy-input accuracy by 1.63 percentage points over the strongest baseline. On Qwen2.5-Omni, it raises noisy-input accuracy from 59.33% to 62.94%, while clean-audio accuracy increases from 76.56% to 77.56%. Replacing matched audio with random, shuffled, or silent inputs reduces accuracy by 3.08-6.42 points, confirming that matched acoustic evidence contributes to its predictions. Additional evaluations show improvements on held-out additive noises and external benchmarks, while revealing that these gains do not reliably extend to non-additive distortions. These results demonstrate robust post-training improvements under severe additive noise without sacrificing clean-audio capability across diverse tasks. Our codes are available at https://anonymous.4open.science/r/EchoDistill-45DA.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Audio Language Model, Self-Distillation, Robustness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rX2zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EchoDistill improves audio-language-model robustness using privileged clean audio during post-training. A noisy-input student generates candidate responses, while a frozen copy of the backbone evaluates the same response prefixes under clean audio. Training combines response-token distillation, correctness-gated consistency shaping, and a teacher-referenced clipped group-relative objective. Only the student is retained at inference. Experiments cover three backbones and three audio domains under severe additive noise, with additional clean-retention, generalization, and audio-intervention analyses. The paper reports a 1.63-percentage-point average advantage over the strongest signal/representation baseline, with substantially heterogeneous gains across backbones.

Critical strengths
  1. The evaluation distinguishes correctness from clean–noisy agreement. Controlled noisy-data SFT, standard GRPO+KL, component ablations, and paired clean-retention results provide a stronger assessment than agreement metrics alone. Using only the student at inference is also practically attractive.

  2. The additional controls meaningfully qualify the claims. Audio replacement experiments establish dependence on matched evidence, while held-out noises, external benchmarks, non-additive distortions, and uncertainty estimates expose important boundaries. The paper generally avoids treating positive point estimates as uniformly established improvements.

Critical weaknesses
  1. The empirical advantage is heterogeneous and sometimes small. Section 4.2 reports gains over the strongest corresponding baseline of only 0.04 percentage points on Step-Audio-2-mini and 0.76 points on MiniCPM-o, compared with 3.43 points on Qwen2.5-Omni. The advantage over standard GRPO+KL is also uncertain in several generalization settings. For example, the independent noise-resampling evaluation in Table 17 yields a pooled improvement of 1.17 points with a 95% confidence interval of [−1.33, 3.67]. These results support improvements in particular settings but leave the breadth and consistency of the advantage less firmly established.

  2. The incremental benefit of the additional optimization components needs stronger validation. Table 3 provides useful component ablations and favorable point estimates for the full method, but these comparisons are based on a single training run. The reported evaluation-sample confidence intervals do not capture variability across independently trained checkpoints. It therefore remains unclear whether the additional components provide a reproducible benefit over standard distillation and GRPO+KL under comparable hyperparameter tuning and compute budgets. Repeated training runs and clearer reporting of baseline tuning and training costs would strengthen the justification for the added complexity.

Questions for authors
  1. Can the full method and the strongest controlled optimizer be compared across multiple training seeds? The existing confidence intervals address evaluation-sample uncertainty but not checkpoint variability.

  2. What training compute, teacher-scoring budget, and hyperparameter-search budget were used for EchoDistill and the controlled baselines? Were checkpoints selected using the same validation criterion?

  3. Does the advantage of the teacher-referenced clipped objective persist across reasonable clipping settings and repeated runs, compared with standard GRPO+KL using the same teacher access?

Reviewer_gTgoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies the robustness of large audio language models under acoustic noise and proposes EchoDistill, a noisy-to-clean self-distillation framework. A student processes noisy audio and samples candidate responses, while a frozen copy of the same backbone processes the corresponding clean audio to provide next-token distributions along the student-generated prefixes. Training combines masked response-token distillation, task-gated consistency reward shaping, and a teacher-referenced group-relative clipped objective. Only the student is retained at inference time. Experiments cover three backbones and three audio domains under severe additive noise, supplemented by controlled training baselines, component ablations, generalization evaluations, and acoustic-evidence interventions. On the fixed-denominator Qwen2.5-Omni evaluation at −10 dB, EchoDistill improves noisy-input accuracy from 59.33% to 62.94% and clean-input accuracy from 76.56% to 77.56%. Further evaluations examine transfer to unseen additive noises and external benchmarks, as well as performance under non-additive distortions.

Critical strengths
  1. A well-motivated use of privileged clean-audio supervision along student-generated trajectories. Evaluating the clean-conditioned teacher on noisy-student-generated prefixes provides a direct way to guide decisions encountered during noisy-input generation. The combination of token-level supervision and task-gated consistency shaping is an interesting design, and retaining only the student at inference makes the approach practically appealing.

  2. A broad experimental investigation with useful diagnostic controls. Beyond the main comparison across three backbones and three domains, the paper includes noisy-data training baselines, component ablations, SNR sweeps, held-out noise categories, external benchmarks, non-additive distortions, teacher-error audits, and audio interventions. These experiments address several plausible alternative explanations for the observed improvements.

Critical weaknesses
  1. The practical gains vary substantially across backbones, while controlled validation is concentrated on one model. In Table 1, average accuracy improves over Initial by only 0.21 and 0.85 percentage points on Step-Audio-2-mini and MiniCPM-o-2.6, respectively, compared with 3.64 points on Qwen2.5-Omni. Qwen’s gains are particularly large in the Sound domain, while several individual settings favor existing methods. The causes of this variation remain insufficiently explored. Moreover, the controlled noisy-data baselines, optimization ablations, and fixed-denominator clean-retention audit focus on Qwen. The consistency curves for the other backbones do not establish the components’ effects on task accuracy or clean-input performance. Even on Qwen, the advantage over Standard GRPO+KL is less conclusive in the SNR sweep and independent noise-resampling experiment. These results leave the effectiveness of the full formulation across backbones insufficiently established.

  2. The demonstrated generalization benefits remain limited in scope. For held-out additive noises, the pooled improvement is 0.81 percentage points, with both the pooled and category-specific confidence intervals including zero. Among the three external benchmarks in Table 4, only DREAM-TTS shows accuracy improvements whose reported intervals exclude zero. Gains over Initial also diminish as SNR increases and become negative at 10 dB, while the non-additive stress test reveals a further boundary of the current training setup. Although the authors acknowledge these limitations, the evidence for broader robustness beyond severe additive noise remains weak.

Questions for authors
  1. What explains the substantial variation in gains across backbones and domains? The teacher-reliability audit in Table 13 provides a useful analysis for Qwen2.5-Omni, but it does not explain the cross-backbone and cross-domain pattern in Table 1. Could the authors extend this analysis to examine whether gains track the initial clean–noisy accuracy gap, clean-teacher correctness, or student–teacher disagreement across these settings? In particular, explaining the large gain on Qwen’s Sound domain and the small gains on Step-Audio-2-mini would clarify under what circumstances the proposed method is most useful.

  2. Do the component-level benefits and clean-input retention observed on Qwen2.5-Omni extend to another backbone? Table 1 provides task-accuracy results for all three backbones, but the controlled component comparisons and paired clean/noisy accuracy audit are limited to Qwen2.5-Omni. Please prioritize a corresponding evaluation on Step-Audio-2-mini, where the aggregate gain over Initial is small. Comparing Initial, Standard GRPO+KL, and the full EchoDistill model on the same fixed set of clean/noisy examples, with paired accuracy differences and confidence intervals, would help establish whether the method provides benefits beyond the main backbone. Including EchoDistill-StdRatio and EchoDistill-Ungated would further clarify whether the contributions of the teacher-referenced objective and task-positive gate also transfer across backbones.

  3. What motivates the training/held-out noise partition, and how sensitive are the conclusions to that choice? The paper specifies ten training noise categories and lists six held-out categories, but the selection rationale and their acoustic relationship are unclear. Please explicitly list the training categories and explain how the two groups differ in acoustic characteristics, recording sources, and mixing procedures. An alternative category split or a targeted leave-category-out evaluation would strengthen the evidence that transfer extends beyond the particular partition used in the paper.

  4. How does EchoDistill advance beyond prior clean-teacher/noisy-student distillation methods? Related approaches have been explored by Markov and Matsui (Interspeech 2016), Mošner et al. (ICASSP 2019), RobustDistiller (ICASSP 2023), and Tripathi et al., “Listen like a Teacher” (AAAI 2026), but these studies are absent from the references. Please discuss their relationship to EchoDistill and clarify which aspects of student-trajectory token supervision, task-gated reward shaping, and teacher-referenced optimization provide advances over the established asymmetric-input distillation paradigm. It would also be helpful to explain how the existing Distill-only and Standard GRPO+KL baselines relate to these prior approaches.

  5. Could the authors clarify two numerical inconsistencies? The text accompanying Table 3 reports a 1.46-point advantage over Standard GRPO+KL, whereas 62.94 − 61.50 = 1.44 points. Appendix A.2 also attributes the difference between 62.96% in Table 1 and 62.94% in Table 2 to macro versus pooled averaging, although the three domains reportedly contain equal numbers of examples. Please correct the arithmetic and clarify the per-domain denominators and treatment of invalid outputs.

Reviewer_oJqFOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces EchoDistill, a post-training method to improve the noise robustness of large audio language models. The approach uses a frozen teacher that receives clean audio to supervise a student that receives noisy audio. Training combines knowledge distillation with a GRPO-inspired objective and task-correctness rewards. The method is evaluated on several audio language models and benchmarks, showing improvements particularly under severe additive noise.

Critical strengths
  • Extensive experimental evaluation. The paper evaluates the method on multiple audio language models, benchmarks, and noise conditions, comparing it against several training strategies.

  • Ablation studies. The authors investigate the contribution of different training components, helping to explain where the improvements come from.

Critical weaknesses
  • Limited methodological novelty. This is my main concern with the paper. The proposed method combines several existing techniques, including noisy-to-clean knowledge distillation, on-policy distillation, and GRPO-style training. While the combination is reasonable, the underlying idea is relatively straightforward, and it is unclear what the main methodological contribution is beyond combining these techniques.

  • Modest and inconsistent improvements. Although EchoDistill improves performance in several settings, the gains over existing baselines are generally small and vary across models and noise conditions. The method appears most effective under severe additive noise, while improvements under other distortions are less consistent.

  • Overly complicated presentation. The paper introduces elaborate terminology and mathematical formulations for relatively simple concepts. For example, the proposed task-gated clean-noisy consistency shaping essentially adds a consistency bonus when the generated response is correct. The presentation makes the method harder to understand than necessary, especially when trying to identify what is actually new.

Questions for authors
  • Clarify the novelty. What are the main methodological contributions compared with existing noisy-to-clean and on-policy knowledge distillation methods? A clearer discussion of the differences would help establish the novelty of the approach.

  • Investigate the limited generalization. Why are the improvements much larger for some models and noise conditions than others? A more detailed analysis would help clarify when the method is most useful.

  • Simplify the presentation. I recommend explaining the main ideas more directly and reducing unnecessary terminology, particularly for components that build on established techniques. For example, "on-trajectory semantic correction with privileged clean audio" essentially refers to using a teacher with access to clean audio to guide a student processing noisy audio.

Comments on the paper

5 comments

Thank you for the additional experiments and clarifications.

  1. The additional Step-Audio-2-mini experiment provides evidence that the overall gains can transfer to another backbone. Although it does not yet cover the requested component ablations or clean-performance retention evaluation, it is a useful addition.

  2. The evaluation protocols and reported figures in the new audit appear inconsistent with those in the original results and require further checking and clarification. For example, the reported Step-Audio clean-teacher accuracy of 49.56% and clean–noisy gap of 9.89 points imply an initial noisy accuracy of 39.67%, whereas Table 1 reports 59.36%. Please clarify the differences in evaluation data, decoding settings, and answer extraction, and whether the new audit can directly explain the gains reported in Table 1.

  3. The additional comparisons do not establish a clear improvement over Standard GRPO+KL. The reported gains are +1.44 points on the submitted checkpoint and +0.17 points on the second seed, with both confidence intervals including zero. Consequently, the additional performance value of the proposed formulation over this baseline remains unproven. The new results narrow the claim of the method’s advantage over a strong controlled baseline.

  4. The numerical errors and related minor issues have been addressed through the corrections provided.

At this stage, I will maintain my original score. The additional experiments and corrections are helpful, but the performance advantage over strong controlled baselines remains insufficiently established. I would be open to raising my score if further experimental results provide convincing evidence of this advantage.

Thank you for clarifying the evaluation protocols and the audit figures. I acknowledge the targeted robustness advantage over Standard GRPO+KL demonstrated by ECAC. However, this localized benefit does not resolve my main concern: the performance advantage over strong controlled baselines in the primary evaluation remains insufficiently established. I therefore maintain my original score.

Thank you for your continued engagement. Although the main-table accuracy gap over Standard GRPO+KL is small, EchoDistill demonstrates stronger performance in three specific respects: external-task accuracy on DREAM-TTS, judged relevance of open-ended responses under noise, and preservation of correct answers under evidence-critical acoustic corruption. Its supported advantage is therefore better performance in these robustness settings, rather than a large, uniform increase in aggregate accuracy. Importantly, the external-task and open-ended results extend the evidence beyond ECAC alone. The cost audit also supports the practicality of this contribution.

1. Cross-Benchmark Transfer

We highlight the external-benchmark comparison whose paired interval excludes zero. Accuracy is measured at -10 dB; differences are EchoDistill minus Standard GRPO+KL in percentage points, with paired 95% bootstrap confidence intervals. The complete cross-benchmark results are in Table 4 of the submitted paper.

BenchmarkStandard GRPO+KLEchoDistillDifference [95% CI]
DREAM-TTS60.7566.50+5.75 [2.50, 9.25]

DREAM-TTS demonstrates a direct external-task accuracy benefit, not a localization or agreement effect. The remaining cross-benchmark comparisons are uncertain, so this supports benchmark-dependent transfer rather than a uniform advantage.

2. Open-Ended Responses

For free-form answers on 300 AIR-Bench Chat examples at -10 dB, the clearest improvement is in relevance assessed by an audio-conditioned automatic judge. The complete open-ended results are in Table 15 of the submitted paper.

Metric (%)Standard GRPO+KLEchoDistillDifference [95% CI]
Judged relevance48.0055.67+7.67 [0.67, 14.67]

This result demonstrates a complementary advantage in open-ended audio understanding: EchoDistill produces responses judged more relevant under severe noise, extending the demonstrated benefit beyond answer-option selection. Together with the DREAM-TTS result, it provides evidence of benefits in both external-task accuracy and free-form response relevance.

3. Interpreting ECAC

ECAC asks whether correct answers survive damage to task-relevant acoustic information. On 470 preregistered eligible pairs, it compares matched-energy corruption of model-localized critical windows, irrelevant windows, and the full waveform. The following paired effects are relative to Standard GRPO+KL; positive values favor EchoDistill. The complete ECAC results are in Table 12 of the submitted paper.

ECAC measureEffect (pp)Paired 95% CI
Critical-condition accuracy gain+4.26[1.91, 6.60]
Clean-to-Critical drop reduction+2.98[0.21, 5.74]
Evidence preservation index+3.19[0.85, 5.53]

The first measures correct answers under critical corruption. The second accounts for each method's clean-input performance. The third subtracts the gain under irrelevant-window corruption from the gain under critical-window corruption. Thus, the result goes beyond an unadjusted accuracy difference: EchoDistill better preserves task performance when important acoustic information is disrupted, relative to GRPO+KL. This is the specific robustness interpretation, rather than a claim of universally stronger acoustic grounding than Initial.

4. Training Cost

The audited Qwen seed-43 runs used two RTX 4090 GPUs and matched the recorded example, update, rollout-count, and teacher-scoring-call budgets. We repeat only the key costs below; the complete cost breakdown is provided in our earlier response.

Recorded resourceEchoDistillStandard GRPO+KL
Clean-teacher scoring calls57,58857,588
Training wall time (h)42.01 (+1.16 diagnostics)52.18
Allocated GPU-hours86.33104.35
Peak per-GPU memory (GiB)45.6027.82

Taken together, EchoDistill demonstrates stronger transfer accuracy on DREAM-TTS, better judged relevance in noisy open-ended responses, and greater robustness to evidence-critical corruption than Standard GRPO+KL. The audited Qwen comparison uses the same training rollout and teacher-scoring call budgets, while recording lower training time. This combination demonstrates practical value beyond the modest main-table accuracy gap: stronger performance in these generalization and robustness settings, supported by an implementation that does not require a larger training-query budget.

We hope this combined evidence provides a clearer picture of the method's practical contribution. We would be grateful if you could reconsider your overall assessment in this light, and would welcome your feedback on any remaining concerns so that we can address them directly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.