acceptodds
Under review as a conference paper at ICLR 2027

Task-Aware Answer Preservation under Audio Compression for Large Audio Language Models

Abstract

Large audio language models (LALMs) increasingly reason over long audio clips, motivating audio compression to reduce memory use and inference latency. However, audio compression can leave the model's overall answer accuracy acceptable while severely degrading accuracy for particular query families. We introduce a framework for auditing a given audio compression method by measuring the increase in an LALM's answer error relative to uncompressed audio. We formalize an acceptance criterion that limits this increase for every required query family and derive a practical protocol for testing compression settings with statistical confidence. Experiments spanning five multiple-choice audio question-answering benchmarks and four LALMs expose compression-induced damage hidden by overall accuracy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large audio language models, task-aware audio compression, answer preservation, worst-family excess risk

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3TBAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Audio compression can maintain acceptable overall answer accuracy in large audio language models (LALMs) while significantly degrading performance on specific query families. This work proposes a task-aware evaluation framework that uses statistical tests of worst-family excess error to identify hidden performance issues.

Critical strengths

This work focuses on an important challenge for audio language models in real-world applications. It also provides detailed mathematical derivations to support the proposed framework. The task-aware evaluation reveals performance drops in specific query families that may be hidden by overall accuracy. The authors also use independent development and confirmation sets to make compression decisions more reliable. Furthermore, extensive experiments across multiple LALMs, datasets, and compression methods demonstrate the framework's applicability.

Critical weaknesses

The confirmation experiments only evaluate Uniform Retention and EnCodec, without testing the learned compression methods. Including these methods would make the evaluation more complete.

Some methods, such as EnCodec, reduce bitrate but do not reduce audio length or the number of tokens processed by the LALM. The paper should discuss these differences.

The paper includes many detailed mathematical derivations. Some could be moved to the appendix to make room for additional experiments and practical analysis.

Questions for authors

Could you extend the formal confirmation experiments to include learned compression methods?

Reviewer_4L61OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes how to check if audio compression affects specific question families even if average accuracy seems stable. The paper selects a setting on the development data and evaluates it on 5 benchmarks and 4 LALM models. The paper additionally shows that pooled scores may hide large family level losses.

Critical strengths
  • The paper introduces that, for audio compression, the worst family level measure can differ from the dataset average
  • The paper's protocol also locks the decisions before confirmation and says that insufficient evidence does not establish either outcome.
Critical weaknesses
  • The authors present the paper results specialize comparison of experiments ideas and rate distortion ideas. However, the insights/ benefits over existing methods are not clear to me.
  • The paper describes chunk retention compression methods and Encodec, but the paper does not describe in-model token pruning or merging. These additional methods can be included to address the gap in the included experiments. Additionally, the LALM inference speedup cannot be attributed to Encodec's bitrate savings.
  • The results depend on the chosen query family partition, and the protocol does handle unsupported families, but it's not very clear on the sample size needed for a useful decision on such rare families.
  • On reproducibility, I noticed that the authors say the zip files reproduce the empirical results, but the readme says: "This is a clean-room implementation of the submitted paper. Exact numerical identity with the submitted experiments is not claimed." This discrepancy is not clear to me.
  • The paper is very long and dense, making it harder to follow.
Questions for authors
  • Can you test the protocol to a method that prunes low importance audio tokens inside the LALM, and report if the effects differ across query families?
  • How should one select the query families, and how large a family should be? (See weakness)
  • Is there any theoretical result beyond the existing worst group and comparison of experiments ideas?
Reviewer_G9SuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper introduces a task-aware framework used for evaluating whether audio compression actually preserves answer accuracy in large audio language models or not. The framework measures error rates introduced due to compression across different query families in different audio benchmarks, and it also designs a statistical protocol for deciding whether to accept or reject compression settings at a specific error tolerance of 5%. The authors conduct experiments on five audio benchmarks and four large audio language models, and they compare Encodec compression, as well as chunk retention, and learned query selection. The paper shows that average accuracy for benchmarks can usually hide high amounts of degradation in specific question families, and family-aware evaluation does lead to different compression decisions.

Critical strengths
  1. The problem that the paper works on is an important problem in audio compression evaluation, and the paper shows that average accuracy can hide high amounts of degradation in specific query families.

  2. I liked the auditing protocol, which is well motivated. And the framework helps distinguish acceptable compression rates and helps us distinguish them from excessive degradation. For example, on MMSU, when using Encodec at 1.5kbps compression, Qwen2 Audio's error increases by 2.99% on average, but the emotion recognition family's error actually increases by 25%.

Critical weaknesses
  1. One of the major limitations is that the whole experimental protocol is done mostly on Qwen2 Audio, and the comparison across the four large audio language models is done only on 200 AudioMCQ examples. Although Qwen2.5 Omni and MiniCPM are evaluated in other ablations, the full experimental protocol uses only Qwen2 Audio as the main model. The paper does not test other prominent models like Qwen3 Omni as well as Kimi Audio and Step Audio. The paper also does not show these results on prominent benchmarks like MMAU and MMAU Pro. So it is hard to generalize their results across other models and benchmarks.

  2. Both the learned chunk selectors, for training, use the targets generated from Qwen2 Audio's likelihood of the correct answer, and since Qwen2 Audio is the main evaluation model, the learned selection policies may be optimized for Qwen2 Audio. The paper does not show whether the same benefits would hold if the relevance targets used to train the learned chunk generators were generated using another large audio language model like when Qwen3 Omni or Qwen 2.5 Omni or Kimi Audio.

Questions for authors
  1. It would be helpful if the authors could run the complete evaluation protocol on MMAU Pro or MMAU benchmarks as well as on one more model at least. For example, Qwen3 Omni or a newer large audio language model like Kimi Audio or Step Audio.

  2. The authors could show another ablation where they can show how sensitive the learned selectors are to the large audio language model we choose. They could use the supervision from a different large audio language model like Kimi Audio or Qwen 2.5 Omni or Qwen 3 Omni, to train the learned selectors and show how sensitive they are and how well the learned selectors generalize to other models.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Author Summary: Contributions, Additional Evidence, and Revisions

Dear Reviewers, AC, SAC, and PC,

Thank you for your time and constructive feedback.

Reviewer 3TBA confirmed that the additional experiments and resource comparisons addressed their main concerns and maintained their weak acceptance. We summarize below what the paper contributes, how the rebuttal strengthened its evidence, and what we will incorporate into the revised manuscript.

The contribution and its practical consequence. The paper provides an acceptance criterion and statistical audit for a concrete question: does compression preserve answer accuracy within a specified tolerance for every question family the application requires? Settings are selected on development data and assessed using paired answers from the same model on raw and compressed audio. The audit distinguishes approval, demonstrated harm, and insufficient evidence.

This already changes a deployment decision in the submitted experiments. On 5580 held-out AudioMCQ recordings, 3 kbps EnCodec increased overall error by 2.63 percentage points (pp), with an upper bound of 3.77 pp, satisfying the 5 pp tolerance. However, the upper bounds were 5.49 pp for music and 6.91 pp for sound. The same setting therefore earns overall approval but cannot be approved when music and sound must each meet that tolerance. This is a concrete use of the framework, already reported in Sec. 6 and Appendix C.

Evidence added in response to the reviews

  1. Broader evaluation and learned compression [G9Su, 3TBA]. We extended evaluation to MMAU and MMAU-Pro, including development-selected learned settings and comparisons at fixed retention levels. The observed discrepancy recurred for Kimi and MERaLiON on both benchmarks. At 40% question-conditioned retention on MMAU-Pro, MERaLiON lost 4.57 pp overall but 11.64 pp on speech; Kimi lost 3.9 pp overall but 10.05 pp on speech.

  2. Alternative teachers and transfer [G9Su]. We trained selectors using Qwen2-Audio, Omni, and MERaLiON supervision, holding the training recordings, architecture, and features fixed. Switching from Qwen to Omni supervision at 20% agnostic retention changed Qwen's observed speech accuracy by +6.35 pp and Omni's sound accuracy by -4.43 pp. These comparisons directly examine teacher dependence.

  3. Internal token pruning [4L61]. We tested importance pruning, random pruning, and token merging inside Qwen2.5-Omni-3B. With importance pruning retaining 10% of tokens, music error increased by 17.01 pp, with a lower bound of 5.65 pp, exceeding the 5 pp tolerance. This extends the evaluation to compression inside the model.

  4. Measured resource costs [4L61, 3TBA]. We separately measured bitrate, duration, token count, and time to first answer scores. On matched clips, uniform 20% retention reduced the median time from 104 to 74 ms, whereas 3 kbps EnCodec required 202 ms and preserved input duration and token count. Learned selectors reduced duration and tokens but were slower because of preprocessing overhead. These measurements establish which resources each implementation actually saves.

  5. Application guidance and theoretical clarification [4L61]. We added explicit guidance for choosing protected families and sample size calculations under stated rates of harmful and beneficial answer changes. We also extended the two-bit example to positive error tolerance, quantifying the information cost of encoding before the question is known while identifying the classical theory used.

Reproducibility and manuscript revisions. We corrected the reproducibility claim: the supplement regenerates statistics from saved answers - it does not rerun training or inference. We will clarify the contribution in the Introduction, lead Sec. 6 with the worked deployment decision and expanded comparisons, add family-selection guidance to Algorithm 1, and move extended derivations and sample-size calculations to the appendix. The disclosed experimental and replay limitations will remain explicit.

Remaining assessment. G9Su's review reports no substantive soundness concerns. 4L61 acknowledged that the additions addressed several original concerns and identified novelty and practical usefulness as the remaining reservations. We respectfully ask that these judgments be considered separately from specific unresolved concerns about correctness.

Our case for acceptance is a concrete answer-preservation requirement, a statistically grounded way to test it, and evidence showing that it changes compression decisions across models and methods. The framework's value lies in determining when compression is justified for the intended application, including when the available evidence requires withholding approval.

We respectfully ask the AC to assess the added evidence against the original concerns and identify which substantive concerns remain unresolved.

We hope you find our paper acceptable, in content and form.

Thank you for your great effort and care.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.