- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
EchoDistill improves audio-language-model robustness using privileged clean audio during post-training. A noisy-input student generates candidate responses, while a frozen copy of the backbone evaluates the same response prefixes under clean audio. Training combines response-token distillation, correctness-gated consistency shaping, and a teacher-referenced clipped group-relative objective. Only the student is retained at inference. Experiments cover three backbones and three audio domains under severe additive noise, with additional clean-retention, generalization, and audio-intervention analyses. The paper reports a 1.63-percentage-point average advantage over the strongest signal/representation baseline, with substantially heterogeneous gains across backbones.
Critical strengths
-
The evaluation distinguishes correctness from clean–noisy agreement. Controlled noisy-data SFT, standard GRPO+KL, component ablations, and paired clean-retention results provide a stronger assessment than agreement metrics alone. Using only the student at inference is also practically attractive.
-
The additional controls meaningfully qualify the claims. Audio replacement experiments establish dependence on matched evidence, while held-out noises, external benchmarks, non-additive distortions, and uncertainty estimates expose important boundaries. The paper generally avoids treating positive point estimates as uniformly established improvements.
Critical weaknesses
-
The empirical advantage is heterogeneous and sometimes small. Section 4.2 reports gains over the strongest corresponding baseline of only 0.04 percentage points on Step-Audio-2-mini and 0.76 points on MiniCPM-o, compared with 3.43 points on Qwen2.5-Omni. The advantage over standard GRPO+KL is also uncertain in several generalization settings. For example, the independent noise-resampling evaluation in Table 17 yields a pooled improvement of 1.17 points with a 95% confidence interval of [−1.33, 3.67]. These results support improvements in particular settings but leave the breadth and consistency of the advantage less firmly established.
-
The incremental benefit of the additional optimization components needs stronger validation. Table 3 provides useful component ablations and favorable point estimates for the full method, but these comparisons are based on a single training run. The reported evaluation-sample confidence intervals do not capture variability across independently trained checkpoints. It therefore remains unclear whether the additional components provide a reproducible benefit over standard distillation and GRPO+KL under comparable hyperparameter tuning and compute budgets. Repeated training runs and clearer reporting of baseline tuning and training costs would strengthen the justification for the added complexity.
Questions for authors
-
Can the full method and the strongest controlled optimizer be compared across multiple training seeds? The existing confidence intervals address evaluation-sample uncertainty but not checkpoint variability.
-
What training compute, teacher-scoring budget, and hyperparameter-search budget were used for EchoDistill and the controlled baselines? Were checkpoints selected using the same validation criterion?
-
Does the advantage of the teacher-referenced clipped objective persist across reasonable clipping settings and repeated runs, compared with standard GRPO+KL using the same teacher access?
Thank you for the additional experiments and clarifications.
The additional Step-Audio-2-mini experiment provides evidence that the overall gains can transfer to another backbone. Although it does not yet cover the requested component ablations or clean-performance retention evaluation, it is a useful addition.
The evaluation protocols and reported figures in the new audit appear inconsistent with those in the original results and require further checking and clarification. For example, the reported Step-Audio clean-teacher accuracy of 49.56% and clean–noisy gap of 9.89 points imply an initial noisy accuracy of 39.67%, whereas Table 1 reports 59.36%. Please clarify the differences in evaluation data, decoding settings, and answer extraction, and whether the new audit can directly explain the gains reported in Table 1.
The additional comparisons do not establish a clear improvement over Standard GRPO+KL. The reported gains are +1.44 points on the submitted checkpoint and +0.17 points on the second seed, with both confidence intervals including zero. Consequently, the additional performance value of the proposed formulation over this baseline remains unproven. The new results narrow the claim of the method’s advantage over a strong controlled baseline.
The numerical errors and related minor issues have been addressed through the corrections provided.
At this stage, I will maintain my original score. The additional experiments and corrections are helpful, but the performance advantage over strong controlled baselines remains insufficiently established. I would be open to raising my score if further experimental results provide convincing evidence of this advantage.