acceptodds
Under review as a conference paper at ICLR 2027

2D Learned Speech Compression with Content-Adaptive Time-Frequency Modeling

Abstract

Neural audio codecs have relied on one-dimensional sequence representations, where speech is transformed into temporally ordered latent representations. However, this formulation does not explicitly model the inherent time-frequency structure of speech, treating frequency as a representation-dependent dimension and limiting natural adaptation across different sampling rates. To address this core limitation, we represent speech as a time-frequency spatial field via a complex short-time Fourier transform, preserving frequency and time as explicit spatial dimensions. Inspired by learned image compression, we then formulate speech coding as a two-dimensional learned compression problem over this spatial field, applying learned image compression models directly to compress the field. This enables a single model to operate on native 16, 24, and 48 kHz speech without sampling-rate-specific branches. We further apply content-adaptive token organization to the time-frequency field for sequence modeling, stabilize entropy skipping, and merge entropy-coded streams into a compact bitstream. Experiments show that CA2D, with only 69M parameters, achieves strong low-bitrate quality while maintaining a unified architecture across sampling rates, a notable achievement given its compact size. This single compact model operates across 16, 24, and 48 kHz and consistently provides competitive or superior objective and subjective quality compared with existing neural codecs: at around 800 bps, CA2D achieves comparable or better quality than 1 kbps-class models with over 100M parameters, such as BigCodec (159M parameters, 1040 bps), while the same model at 48 kHz is comparable to the 6 kbps operating point of FlowDec, a full-band codec. These results clearly demonstrate the effectiveness of treating speech as a two-dimensional spatial field for scalable neural audio compression with native multi-rate operation and low-bitrate perceptual quality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Neural Audio Codec, Learned Compression, Time-Frequency Modeling, Content-Adaptive Modeling, Multi-Sampling-Rate Audio Coding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5dx3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors proposed a 2D learned speech compression with content-adaptive time-frequency modeling. In listening test, the proposed methods outperform BigCodec, MimiCodec, and FlowDec in MUSHRA test. The idea is promising. But the experiment is only on speech and lacks the generalization ability.

Critical strengths
  • The writing of Section 1 and Section 2 are clear.

  • The result of the method is promising from the listening test.

Critical weaknesses
  • The writing of Section 2 is not clear. I can not understand how the system is built from the current version.

  • There is a lack of definition of CA2D.

  • No code is available.

  • 3.1, there is a lack of description how X is compressed to y. What shape is y?

  • 3.1, In Fig. 2, there are channels of 256, 384, and 768. However, in the main text of 3.1, there is a lack of description of how the purple block of Fig. 2 is done. I do not understand where does the 256, 384, and 768 are concatenated, or where do they go.

  • 3.2, what is the full name of CMIC? How Mamba will affect the result? What is the ablation if there are not Mamba block?

  • 3.3, There are several symbols that are not easy to understand. Such as the symbols in line 3 of Section 3.3.

  • 3.3, Is the equation in line 207 necessary? Is this a new contribution, or is it just from some other papers?

  • 3.4 Is this some standard bitstream method?

  • Experiment. For speech, 16kHz, 24kHz, 48kHz do not make big difference. To verify the effectiveness of the method, it is better to evaluate on HiFI music as well. This is very important for a research on Codec.

Questions for authors
  • See Critical weaknesses.
3 comments
AuthorsOpenReview ↗

Making the system explicit: dimensions, symbols, and the bitstream

We thank the reviewer for the detailed comments.

  1. Definition of CA2D and overview of the method CA2D (Content-Adaptive 2D Time-Frequency Codec); "content-adaptive" refers to the token organization being determined by the latent content rather than by a fixed traversal. Sections 2 and 3 have been reorganized: §2 presents the design (moving from 1-D waveform modeling to a 2-D time-frequency field, and then transferring learned image compression to the audio time-frequency domain), while §3 and §4 study feature reordering and the adaptations to audio-specific requirements, including bitstream integration.
  2. The mapping X→yX\to y and the structure of Fig. 2 The input is mapped by an STFT to a 3×F×T3\times F\times T field (log-magnitude, real, imaginary), where FF is set by the FFT size and TT by the hop length. Adjusting the STFT parameters aligns different rates on a common frequency grid: for a 1-s clip, 16 kHz and 24 kHz share the first 256 frequency bins, and 24 kHz has bins 256–383 in addition. y∈RCy×fy×ty,fy=F/16,ty=T/16y\in\mathbb{R}^{C_y\times f_y\times t_y},\qquad f_y=F/16,\quad t_y=T/16 Cy=320C_y=320 for CA2D and 9696 for CA2D-small. The hyper-latent is z∈RCz×fz×tzz\in\mathbb{R}^{C_z\times f_z\times t_z} with fz=fy/4f_z=f_y/4, tz=ty/4t_z=t_y/4; Cz=192C_z=192 for CA2D and 6464 for CA2D-small. (F,T)(F,T) correspond to (H,W)(H,W) in LIC, and the channels to image channels. The numbers 256/384/768 in Fig. 2 are FF for the three rates (nfft=510/766/1534n_{\text{fft}}=510/766/1534, so F=nfft/2+1F=n_{\text{fft}}/2+1); they are not parallel branches but the input frequency resolution of the same encoder, and this choice makes fy=F/16f_y=F/16 integral.
  3. CMIC and Mamba CMIC is Content-Adaptive Mamba for Learned Image Compression (the third reference in the paper). CA2D reuses its codec skeleton; the module's design and effectiveness were validated by the ablations in the original paper, and Mamba itself is not our contribution. Our focus is adapting the LIC framework to the audio time-frequency domain and balancing reconstruction quality, computation, and cross-rate adaptation at extremely low bitrates.
  4. Symbols and equations in §3.3 sn=1(σn≤τσ)s_n=\mathbb{1}(\sigma_n\le\tau_\sigma) sn=1s_n=1 when σn≤τσ\sigma_n\le\tau_\sigma (the residual is not transmitted and the decoder reconstructs from μn\mu_n), and sn=0s_n=0 otherwise; without entropy skipping all sn=0s_n=0. τσ=0.2\tau_\sigma=0.2, given in the appendix. The rendering of this symbol has been fixed. This expression is not a new loss but the gradient of the classical LIC rate loss. We include it to show that masking the likelihood of skipped positions removes supervision from both μn\mu_n and σn\sigma_n (we call this missing likelihood supervision), so that σn\sigma_n stays small as the residual grows, the entropy model underestimates uncertainty, and bitrate spikes appear.
  5. The bitstream method in §3.4 This is not an off-the-shelf method. CompressAI writes several independent arithmetic streams with a 32-bit header each; for images this costs about 0.0009 bpp and is negligible, and video encoders likewise run dozens of independent units in parallel at much higher rates, so no merging is needed there. At 2.55 s, 16 kHz, and 1 kbps, however, the original 11-stream implementation adds about 138 bps, i.e. 13.8% of the stream; merging into a single stream with one 8-bit header reduces this to about 3.14 bps without touching the learned probability model.
  6. Generalization to HiFi music We partly agree. This work focuses on speech; the three rates are chosen to test wideband-to-fullband coverage with one model and to match the sampling setup of speech codecs such as FreqCodec and BigCodec. 48 kHz corresponds to a 24 kHz Nyquist frequency and covers the audible high band, but music differs from speech in signal distribution and spectral structure. A codec for both speech and music is future work and will be listed as a limitation in the revision.
  7. Code and models Code, pretrained weights, and an online demo: https://anonymous.4open.science/r/CA2D-iclr27 https://anonymous.4open.science/api/repo/CA2D-iclr27/file/docs/nojs.html We have completed the demo validation and released the code; we would be grateful if you could take the completeness and reproducibility of the paper into account and raise your score accordingly.

Thanks the comments from the authors. I have read though it and keep my score unchanged.

Thank you for taking the time to review our responses. We have also released the inference code, pretrained weights, and an online demo to improve reproducibility. We would appreciate it if you could reconsider the Soundness score in light of these additions.

Reviewer_qckaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CA2D, a neural speech codec that formulates audio compression as a two-dimensional learned image compression problem over complex STFT representations. The proposed architecture combines content-adaptive Mamba blocks, scalar quantization, and probabilistic entropy coding, enabling a single model to operate across 16, 24, and 48 kHz sampling rates. The paper presents extensive experimental comparisons, subjective listening tests, and ablation studies, demonstrating highly competitive reconstruction performance at extremely low bitrates.

Critical strengths
  • Strong reconstruction performance. CA2D achieves highly competitive reconstruction quality at extremely low bitrates, using a very compact model with 69M parameters.

  • Unified multi-sampling-rate architecture. The ability to support native 16, 24, and 48 kHz audio using the same model parameters is a valuable contribution.

  • Comprehensive experimental evaluation. The paper includes extensive comparisons with existing neural audio codecs, objective reconstruction metrics, subjective listening tests, and ablation studies.

  • Entropy-skipping analysis. The investigation of masked-likelihood instability and the proposed full-likelihood training strategy provide useful insights that may generalize beyond the proposed architecture.

Critical weaknesses
  • Limited methodological novelty. Most architectural components, including content-adaptive token organization, are inherited from existing learned image compression methods. The main contribution is therefore their adaptation to speech compression rather than a fundamentally new compression technique. Nevertheless, demonstrating their effectiveness for neural speech compression is an interesting contribution.

  • Limited evidence supporting the 2D formulation. This is my main concern with the paper. One of its central claims is that preserving time and frequency as explicit spatial dimensions provides substantial advantages over conventional one-dimensional neural audio codecs. However, CA2D differs from existing codecs in several aspects, including architecture, quantization, and entropy coding. In particular, many baselines employ fixed-rate vector quantization, whereas CA2D relies on probabilistic entropy coding. Consequently, it is difficult to determine how much of the reported improvement can be attributed specifically to the proposed 2D representation rather than other architectural or compression choices.

  • Limited ablation of content-adaptive token organization. The reported improvements over fixed-order serialization are relatively modest. It remains unclear whether the learned codebook-based ordering provides meaningful advantages over simpler serialization strategies.

  • Limited subjective evaluation. Although the inclusion of MUSHRA listening tests is appreciated, the evaluation involves only 10 listeners and four audio samples per sampling rate. This is a relatively small evaluation set, particularly when comparing codecs with similar objective performance.

Questions for authors
  • Include a controlled 1D-versus-2D comparison. I recommend introducing an ablation that isolates the contribution of the two-dimensional representation. Comparing CA2D against a one-dimensional variant using similar architectures, quantization strategies, entropy models, and training objectives would provide substantially stronger evidence for the paper's central claim.

  • Clarify the contribution of entropy coding. Since many competing neural audio codecs do not employ entropy coding, it would be valuable to discuss how this difference affects the comparisons. Additionally, the observation that skipping approximately 98% of latent elements reduces bitrate by only about 1% deserves further explanation.

  • Extend the serialization ablation. Comparing content-adaptive token organization against simpler alternatives, such as time-first traversal, at matched bitrates would help establish whether the learned codebook-based approach provides meaningful advantages.

  • Better subjective evaluation. A larger MUSHRA evaluation with more audio samples and listeners, together with confidence intervals or statistical significance tests, would provide stronger evidence for the reported perceptual improvements.

  • Provide audio samples. An anonymous project website with reconstructed audio samples from CA2D and competing codecs across different sampling rates and bitrates would allow readers to directly assess perceptual quality.

3 comments
AuthorsOpenReview ↗

What the 2-D field buys is rate flexibility, not reconstruction gain

We thank the reviewer for the constructive comments.

  1. Novelty We agree that many architectural components are inherited from learned image compression. Our contribution is not a new fundamental compression operator, but a redesign of the key stages of the LIC framework for the properties of audio time-frequency data and for extremely low bitrates: content-adaptive token organization is brought to time-frequency latents, and entropy skipping and bitstream integration reduce entropy-model computation and fixed stream overhead respectively. These are adaptations to distinct practical problems, not a reuse of existing image-compression modules.
  2. Evidence for the 2-D claim Attribution. CA2D differs from the baselines in architecture, quantization, and entropy coding, so we cannot attribute the full gain to the 2-D representation. The paper therefore states its role as providing a basis rather than raising quality: making frequency an explicit spatial axis lets the model exploit structure along the frequency direction directly, whereas cross-rate operation is a system-level result supported jointly by the STFT parameterization, multi-rate joint training, and the variable frequency-axis length. Magnitude. FunCodec likewise reports only a modest gap between time- and frequency-domain variants under matched architecture and parameter count (its Tables 3 and 4). Controlled 1-D vs 2-D. We agree this is the most direct evidence. A fair variant must match architecture, quantization, entropy model, and training objective, and change only the front-end representation, which requires retraining the full model; we will add this ablation in the revision and report the result as it is. Until then, the degradation of 1-D baselines when evaluated natively at other rates (Tables A5/A6) is the closest evidence we have.
  3. Entropy coding, and 98% versus 1% Rate is allocated by prediction difficulty, not by element count. The ideal cost is Rn≈−log⁡2p(y^n∣context)R_n \approx -\log_2 p(\hat y_n \mid \text{context}): the higher the predicted probability, the lower the information content of the symbol. The criterion σn≤τσ\sigma_n \le \tau_\sigma only selects which elements need not be transmitted; it is not the coding cost. 98% is the fraction of skipped elements, not the bitrate reduction. Skipped elements have low latent scale and high conditional probability, i.e. little information to begin with; skipping removes the transmission of a symbol, not its information. Table 1 bears this out: 98.58% skipped gives 704 bps versus 712 bps with everything transmitted, a difference of about 1%.
  4. Token organization The comparison is already in Table 2 (48 kHz, fixed frequency-first versus content-adaptive, both with skipping disabled). We use frequency-first as the main baseline because UniSRCodec and MelCAP already showed that frequency-first traversal suits time-frequency features better; measuring the incremental gain on top of the stronger fixed strategy is the stricter setting. The essential difference: CA2D's order is determined by the similarity of the current input latent, so different inputs produce different serializations. Numerically, content-adaptive ordering reaches 3.900 ViSQOL-A / 0.879 STOI at 724 bps, whereas fixed frequency-first needs 819 bps to reach 3.967 / 0.876 (about 95 bps more for a marginal gain); at 852 bps the adaptive ordering improves to 4.011 / 0.890. The gain is admittedly modest, and the paper reports these numbers as they are; the revision will add time-first and raster baselines at matched bitrates.
  5. Subjective evaluation and audio examples The MUSHRA panel and sample count are limited; we will state this in the revision, enlarge both, and report confidence intervals. The demo covers all sampling rates × bitrates, and code and weights are released: https://anonymous.4open.science/api/repo/CA2D-iclr27/file/docs/nojs.html https://anonymous.4open.science/r/CA2D-iclr27 We have completed the demo validation and released the code; we would be grateful if you could take the completeness and reproducibility of the paper into account and raise your score accordingly.

Thank you for the clarifications. I maintain my initial positive assessment of the paper and my original rating. I acknowledge that adapting existing techniques to a different modality is not necessarily trivial and is still a valuable contribution. I understand the difficulty of retraining the entire system within the short rebuttal window, and I appreciate the authors commitment to include such ablation in the revised version, as well as the additional insights provided. The provided audio samples empirically confirm the good reconstruction performance.

Thank you for your positive assessment and thoughtful feedback. Since we have now released the inference code, pretrained weights, and an online demo to improve reproducibility, would you consider updating the Soundness score to 3?

Reviewer_JzUHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a neural audio compression approach over time-frequency representations instead of waveform representations as in competing approaches such as DAC/SoundStream/EnCodec.

It follows a common 2D synthesis/analysis framework common in image-compression. The same architecture used for image compression is adapted for speech compression. The tokens are re-ordered adaptively based on cross-token similarity for further compression before being encoded by the Mamba layers. Low-entropy bits are skipped via the entropy prior for further compression. The model compares favorably to other 2D codecs, but is on par with strong 1D codecs such according to the objective measures.

Critical strengths
  • Multi-rate behavior merges from the 2D approach, as you don't need to train multiple codecs for different rates.
  • Clever use of token organization to prevent a suboptimal raster scan of the spectrogram (which may have empty or uninteresting patches).
  • Tackles ultra wideband (48khz) audio.
  • Good ablation of the skipping approach.
Critical weaknesses
  • Missing one important baseline + citation of Vocos, and similar comparisons to other Vocos-style codecs such as APCodec, FreqCodec.
  • Limited comparisons to other 2D codecs.
  • Lots of moving pieces with relatively high complexity
  • No audio examples or companion page to listen to the results, making it hard to evaluate.
  • Mostly an application of off-the-shelf image compression to speech.
  • One model per bitrate.
  • More expensive than MDCTCodec.
  • Only evaluated on clean read speech, and not in broader environments.
Questions for authors
  • What are the latency characteristics?
  • The MUSHRA has only a few participants.
  • One note here is that waveform codecs behave the same way - treating a 16khz waveform as if it was 44khz and passing it through DAC/SoundStream yields good reconstruction quality even at 16khz. The 1D convs adapt in a similar way that the 2D convs in the suggested approach do. Add a comparison to using an off-the-shelf 1D codec (e.g. DAC-44kHz) at different sampling rates to check if it also adapts to varying sample rate.
1 comment
AuthorsOpenReview ↗

Audio-specific designs, a 6M variant, and reproducible results

We thank the reviewer for the detailed comments.

  1. Novelty relative to image-compression methods A complex STFT turns speech into a 2-D time-frequency field, so starting from the 2-D modeling paradigm of image compression is a reasonable point of departure. Audio, however, differs from images: the time and frequency axes are asymmetric, and log-magnitude, real, and imaginary parts are coupled rather than independent RGB channels. Prior 2-D audio codecs such as UniSRCodec and MelCAP mostly serialize the latent with a fixed raster scan, which does not exploit the content structure of the field — a fixed traversal is a poor fit for a time-frequency latent. CA2D therefore reorders tokens by similarity ahead of each Mamba block. We also adapted the entropy coding to extremely low bitrates. The original CompressAI implementation writes 11 independent arithmetic streams with a 32-bit header each, which costs about 138 bps for a 2.55-s segment; merging them into one stream with a single 8-bit header reduces this to about 3.14 bps.
  2. Baselines We have added citations and discussion for Vocos and APCodec, and six operating points of FreqCodec on the full 16 kHz test set (Table A4). At 1 kbps, CA2D outperforms FreqCodec on all eight metrics at the same rate, and its quality is close to FreqCodec's 2 kbps operating point at half the bitrate. APCodec has no public pretrained weights, so we discuss it but do not benchmark it. Our comparison targets competitive codecs rather than methods sharing our architecture; ECC is the relevant frequency-domain baseline, and beyond rate-distortion CA2D also offers native multi-rate support with fewer parameters and less memory.
  3. One model per bitrate Two axes should be distinguished: sampling-rate scalability and bitrate scalability. A single CA2D model covers 16/24/48 kHz with shared parameters; separate operating points are still trained per bitrate, which is common in learned image and audio compression (FocalCodec and TAAE likewise). Covering multiple bitrates with one model requires rate conditioning or scalable latents, which we leave to future work.
  4. Complexity CA2D is more expensive than lightweight codecs such as MDCTCodec, which reflects a quality–complexity trade-off: at 48 kHz and 789 bps it reaches MUSHRA 93.39, close to MDCTCodec's 94.53 at 6 kbps, at roughly an order of magnitude lower bitrate. We therefore introduce CA2D-small: 6M parameters (8.7% of CA2D), about 12% of the MACs at 16 kHz, 126 MB of inference memory (about 30%), and 4 autoregressive steps instead of 10.
  5. Latency CA2D is an offline codec: the content-adaptive ordering is computed over the full latent, so the entire utterance must be available. Chunked processing (e.g. 2.55-s pieces) is a plausible way to reduce latency; streaming support is future work.
  6. Low-bitrate reconstruction versus cross-rate generalization At high bitrates the reviewer's observation holds. Upsampling a 16 kHz signal and feeding it to a 44.1 kHz DAC leaves the information to be coded within an 8 kHz band, so the bitrate budget absorbs the mismatch between the model's frequency prior and the input. Opus works the same way: it covers arbitrary rates through mode and bandwidth switching, falling back to narrowband/wideband at low rates, and full-band quality needs more bits. In short, cross-rate capability is bought with bitrate. At 0.5–1 kbps there is no such slack: every bit has to carry the signal itself. The frequency-specific designs of BigCodec (Snake activations) and MagiCodec (Gaussian masks) improve low-rate reconstruction but also tie the model more tightly to the frequency distribution of the training rate; in our experiments both degrade clearly when evaluated natively at other rates — objective scores look acceptable, but the decoded speech is audibly distorted. CA2D instead represents frequency explicitly, so a single shared model supports native 16/24/48 kHz while operating at low bitrate.
  7. Evaluation scope and audio examples We acknowledge that our evaluation focuses on clean speech, and that the MUSHRA test used 10 listeners and 4 samples per rate; we will state both as limitations in the revision. Code, pretrained weights, and an online demo: https://anonymous.4open.science/r/CA2D-iclr27 https://anonymous.4open.science/api/repo/CA2D-iclr27/file/docs/nojs.html We have completed the demo validation and released the code; we would be grateful if you could take the completeness and reproducibility of the paper into account and raise your score accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.