acceptodds
Under review as a conference paper at ICLR 2027

Where codecs draw the line: allocation profiles for speech tokenisers

Abstract

Quantising a speech representation is dividing it: a codebook partitions latent space, and every distinction finer than a cell is destroyed. Analyses of speech tokenisers ask what their codes retain, not where their boundaries fall. We charac- terise a partition by the reconstruction error each phonetic category incurs — an allocation profile, comparable where the partitions themselves are not. It separates two things a single test conflates: added levels may narrow the gaps between cate- gories, or change which ones are finely resolved. Across four widely used codecs and one we train to reverse a single architectural choice, depth does the first, and only quantisation in a learnt low-dimensional projection does the second. And what the first level resolves finely is largely a restatement of the signal: energy, duration and frequency account for 82.2% of one codec’s acoustic profile against 16.2% of the distilled stream trained beside it, which allocates along unrelated lines. Depth mostly refines an allocation that architecture has already settled.

Primary area:
interpretability and explainable AI
Keywords:
neural audio codecs, residual vector quantisation, discrete speech representations, speech tokenisers, analysis of representations, phonetic categories, quantisation geometry

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nLDBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents an analytical framework, allocation profiles, for speech tokenisers. It characterizes how speech tokenisers partition acoustic space across phonetic categories, instead of probing what linguistic information tokens retain. Evaluating four widely used codecs and one custom-trained ablation model on TIMIT, it is found that adding RVQ depth universally causes levelling but largely preserves the initial ranking. It concludes that the phonetic organization is primarily settled at the first quantization level and by its architectural geometry, rather than by subsequent cascade depth.

Critical strengths
  1. The analysis is novel. It makes use of codeword-based allocation profiles instead of conventional index-based probing.
  2. The refinement decomposition is also well designed, by separating cascade refinement into amplitude and ranking.
  3. Mimi is used smartly with parallel acoustic and semantic streams for within-model comparison.
Critical weaknesses
  1. The major concern is that the analysis relies exclusively on TIMIT dataset, which a speech corpus of clean and reading style American English. Whether the findings in the paper are still valid for more diversified speech (different accents/languages, noisy conditions, conversational speech) is unknown.
  2. It is mentioned in the Conclusion that “whether it propagates that far, to phone recognition, minimal-pair discrimination or spoken language modelling, is the natural next question, not shown here”. However, without linking the allocation profiles to the performance of downstream tasks such as speech recognition, the contribution is limited.
Questions for authors

The toy RVQ codec is trained with a simple architecture and a small speech dataset. Whether this toy model may exhibit certain behaviors due to under-training and simplified capacity?

1 comment

Thank you for your review and for your positive remarks.

  • TIMIT only. We agree. We have run the analysis on FLEURS in four languages and under babble noise; the corrected version reports both, including the conditions in which the result does not hold.
  • Downstream link. We agree. Our subsequent tests show that the latent profile does not predict per-phone recognition loss, and the corrected version says so.
  • Toy codec. The runs stopped on a plateau of validation loss and reach the reconstruction quality of DAC at the same depth (comparable PESQ and STOI). Its more serious limitation is that it was evaluated on decoded audio rather than in the latent space; the corrected version addresses this.

We will submit a corrected version accordingly.

Reviewer_VEGSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper appears to be investigating how well the quantization boundaries in neural audio codecs line up with phonetic boundaries, but it reads as if it was completely generated by Claude and is nearly incomprehensible.

Critical strengths

None. I am 99% certain this paper is AI-generated.

Critical weaknesses
  • The paper reads as if it is completely AI-generated

  • The paper is nearly incomprehensible. There is no clearly stated problem, motivation, or hypothesis.

Questions for authors

None. I am 99% certain this paper is AI-generated.

1 comment

We take the point on clarity: the corrected version states the problem, the hypothesis and the contributions at the outset. We used AI assistance for writing; this was declared on submission. We take full responsibility for all of its content.

Reviewer_wrpHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper examines how reconstruction error is distributed across phonetic categories as residual vector quantization (RVQ) depth increases. It introduces an “allocation profile” and measures two properties: whether error differences between categories become smaller, and whether the categories change their relative ordering. Experiments on EnCodec, SpeechTokenizer, Mimi, and DAC show that the error distribution becomes more uniform in all four models, while DAC exhibits substantially larger changes in ordering. A controlled codec experiment investigates the role of factorized quantization. The paper also examines acoustic covariates, Mimi’s two streams, alternative phone inventories, and linear probes.

Critical strengths
  1. The distinction between error dispersion and category ordering is clearly formulated and provides a complementary description of RVQ behavior.
  2. The paper includes relevant controls, particularly alternative phone inventories, acoustic covariates, and a controlled factorization experiment.
Critical weaknesses
  1. The proposed analysis has not demonstrated substantive diagnostic value. My main concern is what allocation profiles allow us to diagnose or improve. For compression and transmission, the paper does not show that these statistics explain reconstruction quality or perceptual failures at a fixed bitrate. Matching RVQ depth does not match bitrate. Appendix F also shows that latent reconstruction error can worsen while decoded spectral quality improves, illustrating the limits of this proxy. For downstream modeling, the paper does not establish that levelling or re-ranking predicts phonetic discrimination, token modeling difficulty, or downstream performance. The linear probes do not connect these statistics to a useful outcome. As acknowledged in Section 7, category-wise reconstruction accuracy does not establish whether phonetic contrasts are preserved. Consequently, the analysis remains descriptive. It does not demonstrate a failure that existing evaluations miss, identify a corrective intervention, or provide a validated basis for choosing a codec. This leaves the significance of the central contribution insufficiently established.
  2. The incremental insight and empirical support are limited, with additional implementation concerns. Existing work has already explored information across RVQ layers, semantic supervision, and phonetic discriminability, including SpeechTokenizer and X-Codec. Earlier analyses also examine accumulated decoded representations and their geometry, such as Speech Codec Probing and Codec-SUPERB. The additional insight provided by the proposed statistics therefore needs stronger justification. The model panel comprises four established codec families and does not demonstrate applicability to more recent designs. Evaluation is restricted to TIMIT, a small corpus of read English. Its manual alignments are valuable, but this narrow setting provides limited support for general conclusions. The substantial reduction in DAC’s levelling after removing stop closures further illustrates the dependence on the evaluation inventory. Implementation issues also weaken confidence: Equation (2) specifies squared distance while the main code averages unsquared distance; some bootstrap routines discard repeated-speaker multiplicities; and the supplementary pipeline contains missing training components and incompatible command-line arguments. The consequences for the reported findings remain unresolved.
Questions for authors
  1. Can allocation profiles identify a concrete codec failure or predict an outcome in communication or downstream modeling beyond conventional metrics?
  2. What substantive finding goes beyond existing empirical studies, and what evidence supports its generality beyond the selected codecs and TIMIT?
  3. Which distance definition produced the reported results? Do the conclusions remain unchanged after reconciling the metric, correcting the affected bootstrap routines, and repairing the reproduction pipeline?
1 comment

Thank you for this careful review, and for examining our code.

  • Implementation. All three issues were real. The reported results use the unsquared distance, and Equation (2) is corrected. The bootstrap routines that dropped repeated speakers did not feed any reported interval; they are fixed. The supplementary pipeline was incomplete; it has been rebuilt and tested from a clean installation. The submitted numbers reproduce exactly.
  • Diagnostic value. We agree. The tests we have since run confirm your concern: on decoded audio the ordering result does not hold, and the latent profile does not predict per-phone recognition loss.
  • Prior work and generality. We will cite Shi et al. (2026), position the work more clearly against SpeechTokenizer, X-Codec and Codec-SUPERB, report results beyond TIMIT and against bitrate rather than depth, and qualify the levelling claim in light of its dependence on the inventory.

We will submit a corrected version that takes these points into account.

Comments on the paper

1 comment

We thank the three reviewers. We agree with their main criticisms: the submission did not show what allocation profiles predict, its evidence rested on a single corpus, and its code and supplementary material contained errors. Further analyses we have run since confirm the first point: the effects we report hold in the codecs' latent space but not, in general, in the decoded audio. We will therefore rework the paper thoroughly rather than defend the current version, and we are grateful for reviews that made this clear. Our replies below address each reviewer's points. We will withdraw the submission at the end of the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.