acceptodds
Under review as a conference paper at ICLR 2027

SemanticAudio: Audio Generation and Editing in Semantic Space

Abstract

In recent years, Text-to-Audio Generation has achieved remarkable progress, offering sound creators powerful tools to transform textual inspirations into vivid audio. However, existing models predominantly operate directly in the acoustic latent space of a Variational Autoencoder (VAE), often leading to suboptimal alignment between generated audio and textual descriptions. In this paper, we introduce SemanticAudio, a novel framework that decouples both audio generation and editing into semantic planning in a high-level semantic space followed by acoustic synthesis. We define this semantic space as a compact representation capturing the global identity and temporal sequence of sound events, distinct from fine-grained acoustic details. SemanticAudio employs a two-stage Flow Matching architecture: the Semantic Planner first generates these compact semantic features to sketch the global semantic layout, and the Acoustic Synthesizer subsequently produces high-fidelity acoustic latents conditioned on this semantic plan. Leveraging this decoupled design, we further introduce a training-free text-guided editing mechanism that enables precise attribute-level modifications on general audio without retraining. Specifically, this is achieved by steering the semantic generation trajectory via the difference of velocity fields derived from source and target text prompts. Extensive experiments demonstrate that SemanticAudio achieves competitive generation quality while providing consistently improved semantic alignment and enabling effective training-free audio editing.

Primary area:
generative models
Keywords:
text-to-audio generation, audio editing, semantic representation, flow matching, semantic planning, multimodal generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LXWVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SemanticAudio proposes a two-stage flow-matching framework that separates text-conditioned semantic planning from acoustic synthesis and enables training-free editing in a compact semantic space. Experiments on audio generation and editing benchmark show competitive generation quality and improved semantic alignment and editing performance.

Critical strengths

The paper proposes a two-stage framework comprising a semantic planner and an acoustic synthesizer. The framework performs audio generation in a high-level semantic space and enables training-free editing within that space. This is a relatively novel approach to audio generation.

Critical weaknesses
  1. As the authors acknowledge in Section 4.5, “The full two-stage system has greater total capacity than that single-module baseline; these results therefore do not isolate the effect of semantic planning from model size.” Further evidence is needed to justify the two-stage design.
  2. The existing comparisons do not sufficiently cover representative methods and relevant technical approaches, such as Make-An-Audio 2 and Stable Audio models.
  3. Among the aesthetic metrics, the main experiments report only CU and PQ, omitting CE and PC. Furthermore, the aesthetic evaluation does not cover the AudioCaps generation or editing tasks.

Reference: [1] Huang, Jiawei, et al. "Make-an-audio 2: Temporal-enhanced text-to-audio generation." arXiv preprint arXiv:2305.18474 (2023). [2] Evans, Zach, et al. "Stable audio 3." arXiv preprint arXiv:2605.17991 (2026).

Questions for authors
  1. The experimental results include “Ours (d=128, REPA),” but the paper does not clearly explain what REPA means or how it is implemented. In addition, Section 4.4.3 presents its results by referring to Table 7 in the appendix, which is not an ideal presentation choice.
  2. For the acoustic reconstruction evaluation, please report the same generation metrics used in the main experiments, such as CLAP, to help assess the model’s potential performance ceiling.
  3. Why does the TTA-Bench evaluation not follow the subjective evaluation protocol of the original benchmark paper? Moreover, the paper states, “Because this subset includes AudioCaps prompts, we do not interpret the aggregate score as a separate test of out-of-distribution generalization.” Given this limitation, what is the intended purpose of this experiment?
  4. Could the authors clarify the following statement: “since it uses 50 Synthesizer steps instead of 25, this comparison does not isolate REPA’s effect”?
Reviewer_NxrNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SemanticAudio, which incorporates semantic planning into audio generation. Inspired by prior work in vision generation that leverages semantic representations to improve generative modeling, the authors propose a two-stage framework comprising a Semantic Planner and an Acoustic Synthesizer. The method achieves competitive performance in text-to-audio generation and shows improvements in training-free audio editing.

Critical strengths
  1. It is interesting to explore semantic planning for audio generation, inspired by similar approaches in the vision domain.
  2. The authors evaluate the method on different audio generation benchmarks and various audio editing tasks.
Critical weaknesses
  1. The methodological novelty seems somewhat limited, as similar approaches using semantic representations for generative modeling have already been explored in the vision domain. Furthermore, previous audio generation methods (e.g., AudioLDM2) have also explored generating semantic representations.
  2. Despite introducing a separate Semantic Planner and using pretrained semantic representations, the method does not show substantial improvements over existing baselines in audio generation. For example, in Table 1, Resonate still achieves better FD and CLAP scores, while TangoFlux achieves a higher MOS. The proposed method performs better in audio editing, but the improvements over existing baselines are relatively small.
Questions for authors

The proposed two-stage architecture uses approximately twice as many parameters as the Base Model. Therefore, it is unclear whether the improvements come from semantic planning or simply from the increased model capacity. Could the authors provide a comparison of # params with baselines?

Reviewer_kXbBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Summary

This paper proposes SemanticAudio, a two-stage Flow Matching framework using two ∼610M\sim 610\text{M}-parameter DiTs (p. 6, L. 292–294) to decouple text-to-audio generation and editing into frame-level semantic planning (s^∈R250×128\hat{s} \in \mathbb{R}^{250 \times 128} from frozen PE-A-Frame-Large) and acoustic synthesis (z∈R250×128z \in \mathbb{R}^{250 \times 128} in frozen DAC-VAE; p. 13, L. 664–675). However, the subjective evaluation shows mediocre audio quality and lacks statistical rigor (p. 6, L. 313–315; p. 7, Table 1), and the empirical results strongly suggest that this 2×2\times-parameter architecture (∼1.22B\sim 1.22\text{B} total) mainly acts as a workaround for the non-smooth latent space and reconstruction bottleneck of DAC-VAE rather than offering fundamental architectural gains.

Critical strengths

Strengths

  • Effective Projection Head Training: Jointly training the 2-layer MLP projector PθP_\theta with the Acoustic Synthesizer before freezing it for the Semantic Planner (p. 4, L. 168–175, 200–204) effectively regularizes the target manifold, outperforming unprojected rawPE (FD=19.1FD = 19.1 vs. 33.833.8, CLAPL=0.381CLAP_L = 0.381 vs. 0.3200.320; p. 7, Table 1).
  • Better Event Ordering: Frame-level semantic planning improves multi-event Temporal-Order Match (TOM=0.86TOM = 0.86) over the single-stage Base Model (0.640.64) in the 30-prompt LLM-judge test (p. 15, Table 5).
Critical weaknesses

Weaknesses

Major Issues

  1. Weak Subjective Listening Results and Lack of Statistical Rigor:

    • Inferior Generation MOS Without Explanation: In Table 1 (p. 7, L. 340), SemanticAudio (d=128d=128) gets a human MOSMOS of 3.723.72, losing to both single-stage baselines: TangoFlux (3.853.85) and pre-trained Resonate (3.783.78). The authors merely call this "close behind" (p. 7, L. 348) without explaining why perceptual quality drops despite using 2×2\times parameters.
    • Missing Alignment MOS and Confidence Intervals: Despite claiming better semantic alignment (p. 1, L. 012–018), the generation listening test (2020 listeners ×\times 3030 samples; p. 6, L. 313–315) only rates overall audio quality and omits text-audio relevance. The editing test (1010 listeners ×\times 2020 samples; p. 6, L. 322–325) conflates edit accuracy, naturalness, and preservation into one score, and neither Table 1 nor Table 3 reports standard deviations, 95%95\% confidence intervals, or significance tests.
  2. Questionable Architectural Significance (Compensating for DAC-VAE Defects):

    • No Dimensionality Compression: The paper repeatedly claims the semantic space is "compact" compared to the "dense, high-dimensional" acoustic space (p. 1, L. 014–016, 047–053). Yet at the optimal d=128d=128 setting, the semantic latent s^∈R250×128\hat{s} \in \mathbb{R}^{250 \times 128} (25 fps25\text{ fps}) and DAC-VAE latent z∈R250×128z \in \mathbb{R}^{250 \times 128} (25 Hz25\text{ Hz}) have the exact same dimensions (p. 13, L. 664–675).
    • Fixing DAC-VAE Smoothness Rather Than Proving Two-Stage Necessity: DAC-VAE is known to have a poorly smoothed latent space (hard for a DiT to model directly) and limited music training data, which explains why the single-stage Base Model on DAC-VAE collapses (FD=25.0,IS=7.11,MOS=3.42FD = 25.0, IS = 7.11, MOS = 3.42; p. 7, Table 1). Conversely, single-stage models on smoother VAEs do not have this issue: Resonate_PT.pth (Mel-VAE without its Flow-GRPO stage; p. 14, L. 728) beats or matches the 1.22B1.22\text{B}-parameter SemanticAudio on every single metric in both Table 1 (FD=16.0FD = 16.0 vs. 19.119.1, IS=11.04IS = 11.04 vs. 9.139.13, CLAPL=0.403CLAP_L = 0.403 vs. 0.3810.381, MOS=3.78MOS = 3.78 vs. 3.723.72) and Table 2 (CLAPM=0.459CLAP_M = 0.459, AES-CU 5.2585.258 vs. 5.1875.187, AES-PQ 5.9425.942 vs. 5.9365.936), while TangoFlux (Stable Audio Open VAE) also achieves higher MOSMOS (3.853.85), ISIS (12.0512.05), and TTA-Bench CLAPMCLAP_M (0.4640.464). This indicates the two-stage design mainly serves as a crutch to smooth out DAC-VAE.

Minor Issues

  1. Unfair Capacity Comparison and Confounded REPA Ablation: SemanticAudio uses 2×2\times parameters and 1.5×1.5\times steps (∼1.22B\sim 1.22\text{B}, 50+2550+25 steps) compared to the Base Model (∼610M\sim 610\text{M}, 5050 steps; p. 6, L. 297–305). Moreover, Ours (d=128, REPA) in Tables 1–2 is not a single-stage DiT with REPA loss, but still a two-stage model using 5050 synthesizer steps instead of 2525 (p. 7, L. 375–377; p. 13, L. 694–699).
  2. Editing Math/Implementation Mismatch and Missing Baseline Scores: Section 3.4 integrates s^1\hat{s}^1 (s_src) backward to t=0t=0 (p. 5, L. 252) even though t=0t=0 is defined as pure Gaussian noise in Section 3.2 (p. 4, L. 186), and Equations (4)–(5) use source text C_src and N_avg averaging whereas actual experiments use null conditioning (force_uncond=True) and N_avg = 1 (p. 6, L. 305; p. 13, L. 692). Also, Table 6 (p. 15, L. 783; 1919 pairs) reports non-target preservation only for SemanticAudio, omitting all baselines from Table 3.
  3. Inconsistent Reconstruction Targets and Steps (Table 7): Appendix D (p. 16, L. 812–826) evaluates Ours against DAC-VAE reconstructed audio (hiding DAC-VAE's own reconstruction loss) while evaluating Semantic Vocoder against original ground-truth audio, and compares mismatched checkpoints (290k290\text{k} steps for d=128d=128, 415k415\text{k} for d=64d=64, 500k500\text{k} for d=32,1024d=32, 1024; p. 16, L. 820).
Questions for authors

Questions for the Authors

  1. Why does SemanticAudio (∼1.22B\sim 1.22\text{B} params) lose to both TangoFlux and unaligned Resonate_PT.pth on human MOSMOS and ISIS in Table 1? Also please provide 95%95\% confidence intervals for Tables 1 and 3
  2. Since s^\hat{s} and zz have the exact same shape (250×128250 \times 128), how can you show that the two-stage design is not simply compensating for DAC-VAE's non-smooth latent space? How does a single-stage Base Model perform when scaled to ∼1.2B\sim 1.2\text{B} parameters, trained with a true single-stage REPA loss against PE-A-Frame, or trained on a smoother VAE?
  3. In Table 7, what are the Mel/STFT losses at a uniform 500k500\text{k} checkpoint evaluated directly against original ground-truth waveforms (including frozen DAC-VAE itself)? What are the baseline preservation scores in Table 6?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.