acceptodds
Withdrawn from ICLR 2027

CompAdapt: When Audio Representation Adaptation Helps for New Sound Combinations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Audio models encounter familiar sounds in combinations absent from downstream training. We introduce CompAdapt, a prototype-guided map that adapts a frozen audio representation. Separate recordings define class-mean prototypes; their sums guide a nonlinear residual map while an anchor keeps it close to the original representation. The fitted map preserves representation width, serves separately trained downstream models, and needs only audio at inference. Across four encoders on ESC-50, it improves mean separation of withheld class combinations by 0.16–0.89 dB at a common 60-epoch endpoint. With each separator’s calibration-selected checkpoint, differences from the original representation range from −0.028 to +0.042 dB. An optional low-rank student retains selected full-map performance within ±0.15 dB on all four encoders with 86.8–96.1% fewer coefficients. On natural SONYC-UST recordings, source-fitted maps improve macro average precision by 0.15–0.19 percentage points in three corrected comparisons. Together with alternative adaptors and cross-dataset reuse, these results show when a reusable representation map helps and how its benefit depends on downstream training. Code and numerical artifacts are available anonymously: https://anonymous.4open.science/r/compadapt-iclr2027-artifact-E83F/README.md

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
audio representation learning, compositional generalization, class prototypes, sound separation, sound recognition

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_t3sKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates feature adaptation for unseen audio mixtures and proposes a lightweight residual module (CompAdapt) fitted on frozen speech/audio encoders to align joint mixture representations with combined class-mean prototypes.

The paper itself is hard to read, with no clear goal and little real world applications and its core empirical findings reveal that the method offers almost no benefit in realistic settings. Under proper calibration-based checkpoint selection, performance gains over a baseline identity map completely vanish. Furthermore, the paper's own control experiments reveal that random or constant targets match the proposed gains, demonstrating that any observed boost stems from simple feature shifting rather than compositional semantics. Lastly, the dataset scope is rather limited, only using the 2000-sample ESC-50 and ~20000 sample SONYC-UST-V2 dataset, even though the proposed method seems to be general enough to be used in other domains like music (stem separation). Thus, I recommend a reject.

Critical strengths
  • The paper has a somewhat broad coverage using four audio encoders, two tasks and a very long and extensive appendix.
Critical weaknesses
  • The study uses four audio encoder models, two of which (wav2vec2, HuBERT) have no known audio recognition/classification capability. Why are even HuBERT and wav2vec2 considered as reasonable baselines? These models are trained mainly for ASR, thus should perform bad for audio events, which the paper also shows.
  • Dataset for training and evaluation is very limited. ESC-50 might be a reasonable test set for some purposes, but given the general nature of the paper, authors could also run separation experiments for Music on i.e., musdb18 and gtzan. Overall the experiments seem lack luster due to the small number of datasets as well as samples ( 2000 in ESC-50 ).
  • To be honest, I read the paper multiple times over but still do not understand the goal of the work, i.e., what actual problem needs to be solved here? The proposed scenario of having "a downstream training set may contain dogs, rain and keyboards without containing their joint mixture" is very artificial.
  • Proposed approach is statistically indistinguishable from identity under calibration-selected checkpoints. Also the SONYC gains vanish under sensor composition.
  • The paper's own controls (Appendices F/H) show random, constant, and even rotation/mean-shift targets reproduce the fixed-endpoint gains, which lessens the impact of the paper.
  • Paper is extremely dense and hard to understand with dozens of appendix tables and the novelty over the SSLAM/CARL-style baselines is limited (2-layer residual MLP + anchor).
  • BEATs/EAT were pretrained on AudioSet clips, which contains mixtures of most ESC-50 events, so the encoders already know those mixtures, thus the claim of "unseen combinations" is only artificial.
Questions for authors

I still am missing a motivation for this work, i.e., what kind of problem are the authors solving? In which real-world scenario would we encounter a problem that this work would help solve?

Reviewer_eHRuOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an encoder adaption mechanism that aims to improve generalization on unseen class combinations in polyphonic / overlapping audio. It does so through a student-teacher setup where the teacher and the students share the same frozen encoder, but the teacher averages over time and examples of isolated sources for a class to create a class prototype, and then creates a training target by averaging over three class prototypes in artificial training mixtures of 3 classes. They evaluate this adaptor on two downstream tasks: source separation and sound tagging, focusing on performance on unseen class combinations.

Critical strengths

None

Critical weaknesses
  1. The results do not show a clear improvement for the proposed adaptor. Throughout the results, it seems there is no or minimal improvement over the identity mapping or the BCE mapping baselines. When there are improvements, they are not consistent, and there effect size is small.
  2. The motivation of the paper is unclear. The paper would be stronger if there was a preliminary experiment that demonstrated poor generalization to unseen class combinations, or at least citing prior work that has tested poor generalization to unseen classes in the downstream audio tasks they are concerned with.
  3. The goals of the method are unclear. What are the desiderata? It seems that the method they are proposing would decrease within-class variation. Is that their goal? That could be a desirable goal for a tagging task, but it is unclear if that is a desirable goal for their source separation model. It's also unclear for what other downstream tasks besides tagging that would be desirable for.
  4. The presentation is unclear. There are terms that are introduced that are never used. There are terms that are used that are never defined. The hierarchical structure of the paper is confusing at times.
  5. The analysis is weak. There is no analysis of the embeddings themselves, which limits how we interpret the contribution. Even if the proposed method shows significant performance improvements, the paper lacks analysis that explains how it can be used in practice.

I would recommend that the authors:

  1. Clearly identify the problem that they are trying to solve and what they want their method to do to solve it.
  2. Once those are well defined, establish if the adaptor is achieving its goals before looking at downstream performance
  3. If it isn't clearly better than the baselines on average, do a deeper analysis to find where it clearly is better than those baselines (e.g., higher polyphony scenarios that maybe motivated this work)
  4. Think about the practical concerns of the method (e.g., it requires isolated sources which are hard to come by, and it requires 3-class combinations which grow cubically as the number of classes increases) and do an analysis that investigates those concerns and what would be minimally necessary to achieve results.
  5. Revise the text for clarity
Questions for authors
  1. It seems there is an assumption that these representations are additive. Is that the case? Have they been tested with techniques like Audio-COAT by Chen et al. 2026 (Evaluating compositional structure in audio representations)?
  2. How much embedding error is there in unseen class combos vs seen class combos?
  3. How do unseen classes affect embedding error?
  4. Is the model achieving the intended goals in the embedding space?
  5. How does the model perform in varying amounts of polyphony/overlap?
  6. How many class combinations are necessary for the model to see?
  7. How many isolated recordings are necessary?
  8. How does this scale to larger vocabularies?
  9. How does this compare simply to data augmentation rather than embedding adaptors?
  10. How does matched vs unmatched classes on the same dataset?
  11. How much does matching polyphony matter in training?
  12. How does this potentially less within-class variant encoder perform as the number of training examples in the downstream task decreases? i.e., can it be used for few-shot learning? (yes, a bit tangential, but could be worth investigating to strengthen the paper)
Reviewer_jnmGOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

CompAdapt learns a residual map on top of a frozen audio encoder, using sums of class prototypes as targets and an identity anchor. The map is then frozen while a separator or classifier is trained. A low-rank affine student can replace the full map. The experiments distinguish unseen class combinations from unseen recordings and compare fixed-epoch results with independently selected checkpoints.

Critical strengths

I appreciate the authors’ willingness to show where the method fails. The appendices include strong controls, acknowledge limitations in earlier protocols, and make clear that the selected-checkpoint advantage largely disappears.

Critical weaknesses

I do not see a convincing separation benefit once each system is allowed to select its own checkpoint. Can the author explain more on it?

I'm not convinced yet the paper provide a strong reason to use the map. Three SONYC gains of 0.15–0.19 AP points survive tests across fitted runs, but all three intervals accounting for sensor composition include zero (Appendix D). Direct fitting also beats the student on average for BEATs and EAT by about 0.73 and 0.86 points. Can the author clarify it a bit?

Questions for authors

See the weaknesses part

Can the author explain if this approach could expand to other domain, e.g. vision encoder or other type of audio task?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.