acceptodds
Under review as a conference paper at ICLR 2027

Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners

Abstract

Self-supervised learning has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre-training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context. Autoregressive prediction thereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio's temporal structure makes autoregressive prediction of patch embeddings a natural fit. We introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework in which a causal Transformer predicts each next patch embedding of a log-mel spectrogram from the previous ones, using causal masking and stop-gradient as its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance, scales consistently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
self-supervised learning, audio representation learning, next-embedding prediction

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_ZJevOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This work introduces a new SSL framework for audio representation learning (NAPE). NAPEs pretraining does not use masked spectrogram/image reconstruction or student-teacher distillation but only uses a causal transformer to predict the next patch embedding from previous ones.

Critical strengths

Simple and novel new SSL method in audio. The proposed objective itself is much simpler than previous methods in audio and potentially removes some of the complicated or cumbersome objectives such as student-teacher distillation or reconstruction with extra decoders in pretraining. It only predicts the next patch embedding based on prior ones, an elegant alternative and potentially easy framework. It also seems to achieve competitive performance.

Exhaustive ablations. One of the most important aspects of SSL papers or new methods are exhaustive ablations. This work provides a pretty comprehensive investigation of the different architectural components and beyond.

Critical weaknesses

Incomplete SOTA, overstated positioning and missing baslines. Several recent SOTA methods are missing from Table 6. This includes the AudioMosaic paper as well as the Better Audio Transformer (both from this year's ICML). They would change performance rankings in the main benchmark and question the SOTA claims the authors make in the introduction. The linear probing comparison is also misleading and provides an incomplete picture. The paper references work in related work (which should be in the main paper to better position this work btw!) about more advanced probing methods (e.g., the unmute or the better audio transformer paper) and the change of model rankings without finetuning or the embedding positions for probing. However, no results for more advanced probing methods and/or other models are shown in this work. To make more credible SOTA claims and a better positioning of this model also without finetuning, some probes of the baselines (not offered in all the original papers but easy to do for a better evaluation within the same setup as this model) and/or advanced probing methods would be very important. The baseline results are all taken from previously published papers without the same training conditions.

Efficiency claims and runtime. The paper emphasizes simplicity and scalability compared to others (which is partly true), there is no runtime or computational comparison to other models. NAPE's does not provide real performance gains to other models. For instance, related base models like SSLAM (and other missing SOTA models) are even better than the large NAPE model on AS20k with much less parameters. So it's not clear if NAPE also offers a better training efficieny for scaling and computational cost/performance per parameter.

Audio-level causality. The paper's linearization/scanning strategy does not fully capture the chronological ordering across spectrogram frequencies. So the causlity is not really the audio but only the patch causality. Changes to the transformer architecture itself (to the patches to make full timeßband patches) would be a natural solution and caputre true audio-level causality.

Questions for authors

The questions are the weaknesses that can be addressed. I like the method and the paper and I'm open to lean toward accept for this paper if the weaknesses are properly addressed/answered.

Reviewer_cJYXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the problem of self-supervised learning from audio media. In contrast to the popular approach of reconstruction learning from audio streams, the paper borrows insights from vision learning to train a model based on direct prediction of latent embeddings. By splitting the spectrogram of an audio signal in a systematic manner, their approach learns in the embedding space to predict the next patch. The pre-trained model is then used as the foundation for a classifier that is tuned for a specific downstream task. The empirical testing demonstrates that the proposed method can achieve results that are comparable to the state of the art models on several benchmark audio and speech tasks.

Critical strengths

The paper is the first to use next-patch prediction in the embedding space for audio processing. While the underlying approach is not new in itself, its use for audio signal analysis is novel.

The paper provides detailed empirical evaluation of the proposed approach, including a thorough ablation study to justify the structure of the proposed model.

Critical weaknesses

The underlying approach is derived from similar approaches to self-supervised learning in video. As such, the novelty of the proposed approach is somewhat limited to the design and demonstration of how this approach can be used for audio data. The proposed method has comparable performance to the top SOTA method (SSLAM), while reducing the pre-training cost by a factor of two. Thus, the theoretical contribution is limited, although the practical significance is interesting.

Questions for authors

A key contribution is the empirical testing and performance of the proposed method. In the paper on the main competitor method SSLAM (Alex et al ICLR 2025), they have carried out additional evaluations on polyphonic sound benchmarks. How would does your proposed approach perform on such datasets?

Reviewer_VPAQOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a causal next token prediction based self supervised method for learning audio representations.

Critical strengths

The paper is well written and explain the central idea well. It contains comprehensive experiments and ablation experiments.

Critical weaknesses

My primary concern is with the main claim "a simple causal paradigm" which implies the method is simple and easy to use. However the ablation studies show that the configuration of NAPE needs to be (i) Conv2d embedding layer, (ii) raster/diagonal scanning order, (iii) SimSiam-style predictor, (iv) patch embedding as the prediction target, and (v) negative cosine similarity loss with stop-gradient on the target branch. So NAPE also requires "elaborate pre-training recipes to reach competitive performance" and even in the optimal setting does not achieve SOTA performance for base model. Even the large model does not outperform existing smaller models (e.g. AS-20K).

The paper has limited novelty. The proposed method is very similar to contrastive predictive coding yet the the paper does not discuss the differences between them.

Questions for authors

Instead of comparing GPU hours the paper should report flops, throughput.

In section 3.6, line 459-460 "NAPE predicts the next patch embedding accurately almost everywhere on the grid, with similarity close to the ceiling of 1.0 both on average and per clip" do you think this high similarity might be because the adjacent patches have low-level acoustic regularities instead of model learning semantic information?

Reviewer_atKBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NAPE, a simple SSL framework to learn audio representations. The model receives square audio patches flattened as a 1D sequence for input, which are then processed by a ViT. The core idea is to apply next patch prediction in the latent space by restricting the attention mask to be causal during pre-training. To facilitate representation learning, 4 different scanning orders (audio patch input schedule) were compared. After pre-training, the model is evaluated under both frozen and fine-tuned setups. 3 sizes of NAPE model are trained and evaluated.

The model achieves strong empirical results on multiple benchmarks, including AS-20k, AS-2M, SC and IEMOCAP, matching the prior state-of-the-art. Extensive ablation studies are performed, which provides useful and convincing evidence for multiple design choices, including scanning orders, predictor style and causal masking.

Critical strengths
  1. A simple and effective framework with strong empirical performance. The proposed next-patch prediction objective is simple to follow, easy to implement and achieves competitive performance compared to prior state-of-the-art. Unlike prior methods that usually require maintaining an EMA model or a reconstruction decoder, the architecture of NAPE is simple and straightforward.

  2. The paper provides a comprehensive investigation of the major design choices underlying NAPE. In particular, the ablations on causality, prediction shift, stop-gradient, scanning order, predictor architecture, and prediction target offer useful empirical guidance. These ablations make the empirical contribution more convincing.

Critical weaknesses

W1: The definition of "causality" is ambiguous.

A central motivation of NAPE is that autoregressive prediction naturally aligns with the temporal structure of audio. However, the causal attention mask only enforces causality with respect to the flattened patch sequence, rather than the real physical time axis of the audio signal.

Specifically, under raster and diagonal scanning, the model may attend to patches corresponding to later timestamps when predicting an earlier patch. For example, under raster scanning (see Figure 2), predicting the first patch of the second frequency row (i.e., patch 6) can condition on all patches (i.e., patch 0-5) from the preceding frequency row, including those corresponding to substantially later timestamps. Only time-major scanning preserves chronological ordering across time columns, yet it consistently produces the worst downstream performance in Figure 3.

This observation raises questions about the interpretation that NAPE succeeds by modeling the natural temporal evolution of audio. The results instead appear to support structured sequential patch prediction, which is not necessarily equivalent to temporal forecasting. Although this does not undermine the empirical effectiveness of NAPE, the distinction is important for accurately characterizing its contribution. I would suggest that the authors reframe the claim on causality.

W2: Lack of frozen representation evaluation for other baselines

Although NAPE achieves competitive results under full fine-tuning, the main comparisons in Table 4 rely on results reported in previous studies, which may use different downstream training configurations. More importantly, the linear-probing experiments in Table 5 only compare different NAPE model sizes, without including representative competing SSL methods.

Questions for authors
  1. This is related to the first weakness, where the best configuration of NAPE does not ensure physical time causality, and the strict causal configuration of NAPE performs the worst. Could the authors clarify how these observations align with the motivation that NAPE benefits from the intrinsic temporal evolution of audio? It would be helpful to distinguish sequence-level autoregression from physical-time causality and adjust the framing and interpretation accordingly.

  2. Figure 7 reports cosine similarities close to 1.0 between predicted and target patch embeddings across much of the spectrogram. Given the strong local correlations in audio signals, how much of this performance can be explained by simple spectrotemporal dependencies rather than contextual representation learning?

  3. The ablation study of the prediction target in Table 3 does not show a clear distinction between raw mel and patch embeddings. Does the author have any explanation for this?

  4. Could the authors provide linear-probing comparisons against representative baselines such as Audio-MAE, EAT, or SSLAM under consistent frozen representation settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.