acceptodds
Under review as a conference paper at ICLR 2027

Audio Tokens as a Budgeted Resource: Marginal-Utility Allocation for Scalable Audio Representations

Abstract

Discrete audio tokens are widely used as a representation interface, yet fixed-depth RVQ tokenizers allocate equal capacity to every frame despite varying refinement value. We introduce UniAdapt, which learns marginal utility of RVQ refinements on a frozen codec and allocates them under exact serialized-bit budgets. A rate-independent causal controller predicts acoustic utility, while an optional semantic head supports speech-only utterance-level allocation; measured acoustic and semantic marginal gains on speech have a correlation of 0.42. For causal allocation, a primal–dual allocator selects prefix-valid depths, while an exact guard constrains each sequence prefix to its matched fixed-depth serialized budget. Under utterance-level allocation, UniAdapt reduces Log-STFT distortion by 1.07–4.39% across speech, music, and environmental audio without larger budgets. Causally, it improves three of four speech rates with zero violations across 800 utterance–rate evaluations and runs faster than real time. A 20-listener utterance-level MUSHRA study shows a significant 3.52-point speech improvement, with no significant differences on music or environmental audio. These results support separating utility prediction from budget enforcement for scalable, budget-conditioned audio representations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Neural audio coding, streaming audio compression, adaptive residual vector quantization, marginal-utility allocation, exact bitrate control, discrete audio tokens

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_j7BEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes UniAdapt, a content-adaptive token allocation framework for frozen RVQ-based audio codecs. Instead of assigning a fixed RVQ depth to each frame, UniAdapt predicts the marginal utility of individual RVQ refinements and dynamically allocates tokens under serialized-bit budget constraints. The framework combines a rate-independent causal utility predictor, a primal–dual allocator, and an exact prefix-feasibility guard. An optional semantic utility head is also introduced for offline speech allocation. The proposed approach is evaluated across multiple audio domains and bitrates, including both offline and causal streaming settings.

Critical strengths
  1. The paper provides comprehensive evaluations across different audio domains, bitrates, downstream tasks, and streaming settings, demonstrating consistent reconstruction improvements in the offline setting.
  2. The proposed exact prefix-budget enforcement is practically meaningful, as it accounts for actual serialization overhead and ensures that causal allocation remains within the matched fixed-depth budget.
Critical weaknesses
  1. The large performance gap between online and offline oracle allocation raises questions about the effectiveness of the proposed causal allocation strategy.
  2. The lack of comparisons with simpler causal allocation baselines makes it difficult to assess the advantages of the proposed method over less complex alternatives.
Questions for authors
  1. How does UniAdapt compare with simpler causal allocation strategies, such as greedy allocation or residual-based heuristics, under the same exact budget constraints?

  2. The performance gap between online and offline oracle allocation is substantial. Could the authors further analyze whether this gap mainly arises from causal information constraints, prefix-budget enforcement, or limitations of the proposed allocation strategy?

Reviewer_mmExOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes an adapter module to control the number of RVQ layers of an audio tokenizer. The number of frames is kept fix, but the number of layers is fixed, which allows to adaptively adjust the number of layers in an RVQ setting. The paper claims that this adapter can be applied on different tokenizer backbones. The paper provides results on speech, audio and music, with one backbone (unistream).

Critical strengths
  • The approach seems sound, and interesting. The idea is clear and well presented.

  • There seems to be good improvements with speech, shown with log-STFT error reduction, and Mushra tests. Though the benefits on environmental audio and music are more limited, as evidenced by the experimental results.

Critical weaknesses
  • The improvements are mainly concentrated on the speech domain. We do not see substantial improvements on audio or music.

  • The experimental setup seems a bit limited. There is only one tokenization pipeline that was tested. It would significantly help to add another tokenizer.

Questions for authors
  • It would have helped to include a qualitative experiment so that we can clearly see how the tokenization budget is allocated. It would help to share a demo page with some examples where we clearly see the benefits on different domains.

  • At times the paper is hard to follow. For instance, I believe figure 1 is too detailed. I would simplify the figure so that it is easier for reader to understand the main idea, and the flow of the pipeline.

Reviewer_XEd1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work proposed UniAdapt, which is an audio codec based on variable-depth residual vector quantization (RVQ) scheme with content-adaptive bit allocator that could support causal or non-causal (utterance-level) allocation mode. In causal mode, the bit allocator guarantees zero violation to the quota of usable bits. The evaluation demonstrates the proposed variable-depth scheme achieved rate reduction (i.e. saving of bits) while preserving the acoustic fidelity in certain cases.

Critical strengths
  • Frame-wise causal bit allocation with zero violation guarantee for variable-depth RVQ.
Critical weaknesses
  • This work extends variable depth RVQ into a frame-wise causal setup with zero violation. However, in its workflow, a non-causal semantic head is included. In order to enjoy the content dependent optimization, the system becomes non-causal. The value of such setup in practical scene is not well-explained. Furthermore, in its implementation, the causal bit allocation even works in a unit of 500ms block (75 frames), which compromised the necessity of being strictly frame-wise causal. Furthermore, most of the system is a composition of existing mechanisms, makes the significance of the proposed method unclear.

  • According to the evaluation, most of the results are showing trade-offs (Table 4, Table 29), marginal improvements (Table 1-2), and sometimes insignificant (Table 3). Also, despite being an integrative work, it lacks the comparison with existing RVQ-based baselines (e.g. UniStream, BAMU, etc.).

  • The manuscript is really hard to follow due to lack of necessary information in its main text. Undefined terms, unexplained symbols (some of them are explained in Appendix, but no reference from the main text), abused use of hyphened vocabularies, etc. It takes huge effort to comprehend and even require guessing to reconstruct the detail of proposed algorithm.

Questions for authors

None

1 comment

I appreciate the authors for their extra experiments (point 2 and 4) and detailed responses. Nevertheless, the main concern on the significance of improvements still remains. Also, although I do appreciate authors' promise regarding the future refinement on the manuscript, the current state of the manuscript is not updated yet nor sufficiently clear in the standard of scientific writing. Therefore, I am inclined toward keeping my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.