- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Audio compression can maintain acceptable overall answer accuracy in large audio language models (LALMs) while significantly degrading performance on specific query families. This work proposes a task-aware evaluation framework that uses statistical tests of worst-family excess error to identify hidden performance issues.
Critical strengths
This work focuses on an important challenge for audio language models in real-world applications. It also provides detailed mathematical derivations to support the proposed framework. The task-aware evaluation reveals performance drops in specific query families that may be hidden by overall accuracy. The authors also use independent development and confirmation sets to make compression decisions more reliable. Furthermore, extensive experiments across multiple LALMs, datasets, and compression methods demonstrate the framework's applicability.
Critical weaknesses
The confirmation experiments only evaluate Uniform Retention and EnCodec, without testing the learned compression methods. Including these methods would make the evaluation more complete.
Some methods, such as EnCodec, reduce bitrate but do not reduce audio length or the number of tokens processed by the LALM. The paper should discuss these differences.
The paper includes many detailed mathematical derivations. Some could be moved to the appendix to make room for additional experiments and practical analysis.
Questions for authors
Could you extend the formal confirmation experiments to include learned compression methods?
Final Author Summary: Contributions, Additional Evidence, and Revisions
Dear Reviewers, AC, SAC, and PC,
Thank you for your time and constructive feedback.
Reviewer 3TBA confirmed that the additional experiments and resource comparisons addressed their main concerns and maintained their weak acceptance. We summarize below what the paper contributes, how the rebuttal strengthened its evidence, and what we will incorporate into the revised manuscript.
The contribution and its practical consequence. The paper provides an acceptance criterion and statistical audit for a concrete question: does compression preserve answer accuracy within a specified tolerance for every question family the application requires? Settings are selected on development data and assessed using paired answers from the same model on raw and compressed audio. The audit distinguishes approval, demonstrated harm, and insufficient evidence.
This already changes a deployment decision in the submitted experiments. On 5580 held-out AudioMCQ recordings, 3 kbps EnCodec increased overall error by 2.63 percentage points (pp), with an upper bound of 3.77 pp, satisfying the 5 pp tolerance. However, the upper bounds were 5.49 pp for music and 6.91 pp for sound. The same setting therefore earns overall approval but cannot be approved when music and sound must each meet that tolerance. This is a concrete use of the framework, already reported in Sec. 6 and Appendix C.
Evidence added in response to the reviews
Broader evaluation and learned compression [G9Su, 3TBA]. We extended evaluation to MMAU and MMAU-Pro, including development-selected learned settings and comparisons at fixed retention levels. The observed discrepancy recurred for Kimi and MERaLiON on both benchmarks. At 40% question-conditioned retention on MMAU-Pro, MERaLiON lost 4.57 pp overall but 11.64 pp on speech; Kimi lost 3.9 pp overall but 10.05 pp on speech.
Alternative teachers and transfer [G9Su]. We trained selectors using Qwen2-Audio, Omni, and MERaLiON supervision, holding the training recordings, architecture, and features fixed. Switching from Qwen to Omni supervision at 20% agnostic retention changed Qwen's observed speech accuracy by +6.35 pp and Omni's sound accuracy by -4.43 pp. These comparisons directly examine teacher dependence.
Internal token pruning [4L61]. We tested importance pruning, random pruning, and token merging inside Qwen2.5-Omni-3B. With importance pruning retaining 10% of tokens, music error increased by 17.01 pp, with a lower bound of 5.65 pp, exceeding the 5 pp tolerance. This extends the evaluation to compression inside the model.
Measured resource costs [4L61, 3TBA]. We separately measured bitrate, duration, token count, and time to first answer scores. On matched clips, uniform 20% retention reduced the median time from 104 to 74 ms, whereas 3 kbps EnCodec required 202 ms and preserved input duration and token count. Learned selectors reduced duration and tokens but were slower because of preprocessing overhead. These measurements establish which resources each implementation actually saves.
Application guidance and theoretical clarification [4L61]. We added explicit guidance for choosing protected families and sample size calculations under stated rates of harmful and beneficial answer changes. We also extended the two-bit example to positive error tolerance, quantifying the information cost of encoding before the question is known while identifying the classical theory used.
Reproducibility and manuscript revisions. We corrected the reproducibility claim: the supplement regenerates statistics from saved answers - it does not rerun training or inference. We will clarify the contribution in the Introduction, lead Sec. 6 with the worked deployment decision and expanded comparisons, add family-selection guidance to Algorithm 1, and move extended derivations and sample-size calculations to the appendix. The disclosed experimental and replay limitations will remain explicit.
Remaining assessment. G9Su's review reports no substantive soundness concerns. 4L61 acknowledged that the additions addressed several original concerns and identified novelty and practical usefulness as the remaining reservations. We respectfully ask that these judgments be considered separately from specific unresolved concerns about correctness.
Our case for acceptance is a concrete answer-preservation requirement, a statistically grounded way to test it, and evidence showing that it changes compression decisions across models and methods. The framework's value lies in determining when compression is justified for the intended application, including when the available evidence requires withholding approval.
We respectfully ask the AC to assess the added evidence against the original concerns and identify which substantive concerns remain unresolved.
We hope you find our paper acceptable, in content and form.
Thank you for your great effort and care.