acceptodds
Under review as a conference paper at ICLR 2027

CacheSep: Amortizing Text-Queried Sound Extraction Across Queries in Codec Latents

Abstract

Text-queried target sound extraction models, including recent ones that operate in the compact latent space of a neural audio codec, rerun a query-conditioned separator for every query, even though a recording is often queried repeatedly for different sources. Each request thus repeats work on a mixture that has not changed. We present CacheSep, a codec-domain extractor that runs a query-independent Transformer trunk over the mixture's DAC codec latent once and caches it, then answers each query with a shorter tail conditioned on a frozen CLAP text embedding. In the codec domain, decoding the latent back to audio dominates each query's cost, so we also train a lightweight decoder for the DAC latent, 19.37x faster than DAC's original decoder at comparable reconstruction quality, and share it across all codec-latent systems. We train and evaluate on 1,227 hours of speech, music, environmental, machine, and animal recordings from 25 public datasets, mapped to 89 sound classes. On 8,900 held-out mixtures, every cache split we evaluate stays within 0.11 dB SI-SDRi of the matched codec-domain baseline CodecSep at comparable training cost, while later queries run up to 2.10x faster end to end. Sparse experts in the tail add a small quality gain at a latency cost.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
target sound extraction, repeated queries, efficient inference, neural audio codecs, conditional computation, mixture of experts

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9YHSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CacheSep, a text-queried sound extraction framework that reduces redundant computation across repeated queries. It splits a Transformer separator into a query-independent cached trunk and a query-conditioned tail, and adopts a lightweight WaveNeXt decoder to reduce reconstruction costs. Experiments on 89 sound classes show separation quality comparable to CodecSep, with up to 2.10× speedup for subsequent queries.

Critical strengths
  1. The paper addresses a meaningful efficiency problem in repeated-query sound extraction. The proposed caching mechanism is simple and effective, achieving up to 2.10× speedup with comparable separation quality.

  2. The experiments are reasonably well controlled. The authors retrain CodecSep under matched conditions and provide useful ablations on trunk–tail splits, sparse MoE, and decoder efficiency.

Critical weaknesses
  1. The architectural ablations mainly explore different trunk–tail splits within the same Transformer design. Comparisons with alternative lightweight, cache-aware conditional prediction architectures are missing.

  2. The evaluation reports aggregate separation quality over synthetic mixtures, but does not clearly analyze how the caching trade-off varies with mixture difficulty, such as the number of interfering sources or the relative target-to-interference level. Such analysis would help determine whether different trunk–tail splits remain equally effective under harder separation conditions.

Questions for authors
  1. Could the authors compare CacheSep with a simpler cache-aware architecture, such as a query-independent audio backbone followed by a lightweight conditional prediction head, under comparable training and inference budgets?

  2. How does the SI-SDRi difference between CacheSep and CodecSep vary with mixture difficulty, particularly the number of interfering sources and the target-to-interference ratio? Could the authors provide results for different trunk–tail splits?

  3. Could the authors provide per-category separation results for the fastest 12/4 configuration to examine whether the reduced query-conditioned depth causes larger quality degradation for specific sound categories?

Reviewer_yFEfOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

CacheSep builds on CodecSep to make repeated text-queried sound extraction cheaper. It splits the Transformer separator into a query-independent trunk, computed once for each mixture, and a shorter query-conditioned tail. Later queries reuse the cached mixture representation. The authors also replace DAC's original decoder with a WaveNeXt decoder trained on continuous DAC latents, and explore sparse experts in the tail. The experiments compare several trunk/tail splits and a dense-tail control against CodecSep retrained with the same data, budget, and fast decoder. On synthetic mixtures covering 89 sound classes, the cached variants retain similar separation quality, with later queries running up to 2.10 times faster.

Critical strengths
  1. The repeated-query setting is practical, and the design addresses it directly. Keeping the trunk independent of the query allows the same state to serve queries that arrive later, without knowing the full prompt set in advance. The reported latency measurements make the benefit concrete, including the cost of the first query.

  2. There is care in separating the effects of the different components. The shared decoder, matched retraining, split sweep, and dense-tail control are useful choices. In particular, the dense 8/8 model stays close to CodecSep in quality while giving a reported 1.84 times speedup for later queries. This is fairly direct evidence that caching itself is useful, even without MoE. The paper is also open about the limits of the fixed-prompt evaluation and the gap to AudioSep's absolute quality.

Critical weaknesses
  1. I find the technical contribution fairly incremental. The central change to CodecSep is to defer query conditioning and cache the preceding blocks. The codec representation, masking approach, conditioning mechanism, and main loss are inherited. The fast decoder and sparse expert layers also follow existing methods. These are sensible choices, but I am not yet convinced that the combination establishes a substantial new insight about sound extraction.

    The decoder illustrates this distinction. Its 19.37 times speedup is useful, but Section 3.2 and App. B.2 describe an adaptation of the published WaveNeXt architecture and objective. Most of the reduction in operation count relative to the original DAC-based path comes from this replacement. The authors correctly give the same decoder to CodecSep, so the additional caching gains remain meaningful; my concern is how much new technical knowledge the decoder component adds. More generally, the experiments establish that delayed conditioning works for this backbone, codec, and fixed set of class prompts. They give less evidence about what makes the shared representation sufficient, or whether the result extends to other architectures or richer queries. The extensive reporting is helpful, but the insight still feels quite specific to this model and setup.

  2. The role of MoE needs a more targeted ablation. I understand the motivation: moving blocks into the shared trunk leaves fewer query-conditioned layers, so experts could add capacity to the remaining tail without executing every expert. The dense-versus-sparse comparison at 8/8 is a useful start. However, the dense model already preserves the baseline quality, so this comparison does not show that MoE compensates for a loss caused by delaying conditioning.

    At 8/8, the reported gain is only about 0.08 dB SI-SDRi, while stored separator parameters increase from 18.1M to 64.4M and later-query latency rises from 3.7 to 5.3 ms (Section 5.3). I would like to see whether the benefit of MoE actually grows as the query-conditioned tail gets shorter. Adding an MoE control at 0/16 and a dense control at 12/4 would help separate a general capacity benefit from an interaction with the cache split. Repeating the key comparisons across seeds also matters for a difference this small. As it stands, the evidence for including MoE in the reference configuration is rather limited, especially given the paper's emphasis on efficiency.

Questions for authors
  1. Does the quality-preserving split also hold for a different separator backbone or for queries beyond the fixed canonical class prompts? Either would help assess whether this is a broader finding about where query conditioning is needed. An analysis explaining when early conditioning matters would also be useful; I do not think the paper needs an exhaustive benchmark across all codecs.

  2. Could you compare dense and MoE tails at 0/16 and 12/4, alongside the existing 8/8 pair? Does the benefit of experts become larger when fewer query-conditioned layers remain, or is it similar regardless of the split? Repeated runs of the key pairs would help establish how much weight to put on the small quality gain and whether it justifies the extra latency and stored parameters.

Reviewer_2E5mOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a method to extract multiple sources from an audio mixture based on multiple text queries. It builds upon CodecSep, which, however, extracts a single source based on a single text query.

To reduce the computational costs at inference time, the method avoids conditioning every layer of the transformer masker on the text query, as is done in CodecSep. Rather, query conditioning is applied only to a subset of layers (the later layers). The initial layers are then shared for multiple queries.

The architectural design and training objectives are similar to CodecSep. A caching mechanism is proposed. Also, conditioning is applied in a pre-LayerNorm configuration, instead of a post-LayerNorm configuration. Experiments demonstrate efficiency and performance gains compared with several state-of-the-art baselines, including CodecSep.

Critical strengths

The paper is technically sound and is primarily empirical in nature. The proposed architectural modifications are well-motivated. The experiments are rigorous, and their results substantiate improvements in computational efficiency and performance.

Critical weaknesses

The paper has limited conceptual contributions. The proposed architectural changes are empirically effective but rather straightforward and not fundamentally new or insightful. It makes limited conceptual advances beyond CodecSep, and is very incremental in nature. So I think that the methodological innovation falls short of what is expected at ICLR. The paper would better suit an audio processing venue, where its empirical contributions will be very valuable.

Questions for authors

The paper is well written, and the method is empirically well supported. I have no substantive concerns of technical nature. My primary concern is the limited conceptual novelty, which I believe cannot be adequately addressed through clarification.

Comments on the paper

1 comment
Area_Chair_9QX4OpenReview ↗

Reminder: Discussion Phase Ends November 18 (11:59 PM AoE)

Hi reviewers, a quick reminder that the discussion phase ends on November 18 at 11:59 PM AoE. Please take some time to read the author responses, engage in further discussions if you have any remaining questions or concerns, and update your reviews and scores as appropriate. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.