acceptodds
Under review as a conference paper at ICLR 2027

Spatial Audio Generation and Editing with Executable Plans and Self-Distilling Transfusion

Abstract

Generating and editing multi-source spatial audio requires fine-grained control over source semantics, temporal activity, and 3D trajectories, while preserving unmodified acoustics during edits. Unifying these tasks requires explicit source-level instruction grounding and preservation of multichannel spatial relationships. We present AMBIT, an end-to-end framework for native First-Order Ambisonics (FOA) generation and editing using an executable ScenePlan that binds source descriptions to activity intervals and 3D trajectories. Built on a unified Transfusion backbone, our framework introduces three contributions: (1) autoregressive planning coupled with rectified-flow rendering, using contrastive reference features for instruction grounding and full reference latents for acoustic preservation; (2) an FOA-aware continuous VAE with shared-decoder omnidirectional recovery, asymmetric grouped KL, and spatial covariance matching; and (3) on-policy self-distillation (OPSD) tailored to Transfusion, jointly refining planning and rendering with target-conditioned teacher predictions on student-generated prefixes and sampling states. Target audio is used only for teacher conditioning during refinement, not inference. Evaluations across speech, music, and sound scenes show improved generation content and spatial quality over the compared baselines, alongside better acoustic fidelity and unedited-source preservation during targeted edits.

Primary area:
generative models
Keywords:
spatial audio, diffusion transformers, executable scene reasoning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_TW4WOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AMBIT, a unified framework for First-Order Ambisonics (FOA) generation and editing. The core idea is to first predict an executable ScenePlan, which specifies each source's content, activity interval, and 3D trajectory, and then render FOA audio conditioned on the compiled plan. Planning and rendering share one Transfusion backbone, in which planning is autoregressive and rendering uses rectified flow. The paper further proposes 1) an FOA-aware VAE with auxiliary W-channel reconstruction, grouped KL, and spatial covariance matching, and 2) an on-policy self-distillation (OPSD) stage, in which frozen teachers conditioned on the target plan and target audio supervise the student's planning and rendering on its own rollouts. The model is trained on 1.6M generation examples and 1M editing pairs, most of which are synthetic scenes rendered with Pyroomacoustics. Compared with text-to-audio models plus spatialization, existing audio editors, and SwanWeave, AMBIT shows better content, spatial, and preservation metrics. The paper also includes VAE ablations, post-training comparisons, and a user study.

Critical strengths
  • The overall design is reasonable and well motivated. Using an explicit ScenePlan as the interface between language and FOA rendering makes the content, timing, and trajectory of each source inspectable, and it naturally supports both generation and source-level editing in one model. The comparison with Qwen3.5-4B + DiT, which uses the same renderer, is a useful controlled experiment. The preservation results also show that the editing path keeps unedited sources well.

  • The paper is a substantial and carefully documented system. The FOA-specific VAE objectives, especially spatial covariance matching, are sensible. The evaluation covers content, binaural, scene/object direction, and preservation metrics, plus a user study. The appendix gives enough detail that the work should be reproducible if the code is released.

Critical weaknesses
  • My main concern is that most of the evaluation is in-distribution for AMBIT but not for the baselines. The 8k generation and 5k editing test sets are built with the same pipeline as the training data (Pyroomacoustics rendering, the same room classes, the same five edit operations). Only 50 of the 8k generation examples are real recordings, and they are not reported separately. On the other hand, the baselines are general T2A models, mono/stereo editors evaluated on downmixes, or cascades whose planner is a prompted Qwen3.5-4B. For example, in Table 4, Qwen3.5-4B + DiT uses the same renderer as AMBIT but has almost the same direction errors as the AS pipelines (DOA 1.07 vs. 1.04). This suggests that most of the spatial gain comes from AMBIT's planner having learned the request conventions of the dataset (e.g., the eight compass categories and five time phases in Table 10), rather than from better spatial modeling. I think a planner baseline fine-tuned on the same plans, together with results on real recordings and free-form requests, is needed to support the claims.

  • My second concern is about OPSD, which is presented as one of the main contributions. In Table 6, most of the improvement over the initial model already comes from continued SFT (e.g., CRW 11.95 →\rightarrow 7.52, FAD 0.242 →\rightarrow 0.232). The further gain of joint OPSD over SFT is relatively small. The comparison with ReFL and GRPO is also not quite fair. Their reward is the semantic CLAP score, which the paper itself says does not measure direction, timing, or preservation. The OPSD teachers, by contrast, have access to the GT target plan and the GT target audio, and batch sizes and budgets also differ across recipes. In addition, OPSD is only applied to editing, and the main results in Tables 2–5 seem to be reported without OPSD. More broadly, the paper combines many components (ScenePlan, Transfusion, the VAE, AMBIT CLAP, OPSD), and the main text is very dense, with many key details left to the appendix. This makes it hard to see which part is responsible for which result.

Questions for authors
  • Are the AMBIT results in Tables 2–5 obtained without OPSD? Is OPSD applied to generation at all?

  • How are the test requests written? Are they generated from the plans with the same templates as in training? Could you report the results on the 50 real STARSS23 clips separately?

  • Why is Qwen3.5-4B used as the planner baseline when Qwen3.5-27B reaches 100% executability in Table 13? Was it prompted zero-shot, and did the prompt include the compass and time-phase conventions used by AMBIT?

  • For the Sphere360 pseudo plans, Qwen3-Omni only receives the W channel as audio. How are source directions inferred, and how reliable are these pseudo labels?

  • In Table 1, are DAC, EnCodec, and WavTokenizer applied per channel, and at what bitrates? A comparison with an FOA-specific autoencoder (e.g., the ImmerseDiffusion VAE or the FOA Tokenizer) would be more informative.

4 comments

Thank you for your thorough and constructive review. We answer your questions below, including scene-rewarded RL comparisons, fine-tuned Qwen3.5-4B and Qwen3.5-27B planners, GT-plan controls, fine-tuned generation/editing baselines, a human audit of Sphere360 pseudo-plans, and comparisons with an FOA-specific tokenizer (PhysWave).

Q1: OPSD explanation clarity

We apologize for the unclear presentation. Tables 2–5 report AMBIT before OPSD, whereas Table 6 separately evaluates the additional gains from OPSD. We agree that the final system’s results should be presented alongside the baselines.

And yes, Joint OPSD is applicable to generation as well. Below is the detailed comparison between AMBIT and AMBIT + OPSD on both generation and editing:

For generation:

MetricAMBITAMBIT + OPSDMetricAMBITAMBIT + OPSD
Music — Paired CLAP ↑\uparrow0.460.56Sound — Paired CLAP ↑\uparrow0.410.49
Music — FAD ↓\downarrow0.710.65Sound — FAD ↓\downarrow0.750.83
Music — KL-PANNs ↓\downarrow0.510.45Sound — KL-PANNs ↓\downarrow0.530.43
Music — KL-PaSST ↓\downarrow0.950.88Sound — KL-PaSST ↓\downarrow1.781.67
Music — FD-PaSST ↓\downarrow261.45164.80Sound — FD-PaSST ↓\downarrow235.49161.06
Music — FD-OpenL3 ↓\downarrow44.5925.59Sound — FD-OpenL3 ↓\downarrow49.9932.38

For editing:

MetricAMBITAMBIT + SFTAMBIT + OPSDMetricAMBITAMBIT + SFTAMBIT + OPSD
Paired CLAP ↑\uparrow0.8390.8550.867DOA ↓\downarrow0.3890.3840.331
FD-CLAP ↓\downarrow0.0470.0420.040Scene Az. MAE ↓\downarrow0.3630.3570.301
FAD ↓\downarrow0.2420.2320.211Scene Ang. traj. RMSE ↓\downarrow0.5210.5200.458
FD-PANNs ↓\downarrow6.0125.3985.282wCS ↑\uparrow0.7050.7040.744
KL ↓\downarrow0.4550.4020.351Isolated-source Direction MAE ↓\downarrow0.1710.1690.161
LSD ↓\downarrow1.2681.1101.076Isolated-source Abs. level change ↓\downarrow1.6541.4601.285
GCC ↓\downarrow14.80511.89711.130Object Spherical MAE ↓\downarrow0.2000.2040.165
CRW ↓\downarrow11.9477.5185.854Object Azimuth MAE ↓\downarrow0.1540.1650.120
FSAD ↓\downarrow0.01420.01190.0115Object Angular traj. RMSE ↓\downarrow0.2700.2690.225

W1: Fair comparison for AMBIT and baselines

evaluation is in-distribution for AMBIT but not for the baselines.

To answer your concern in that the baselines were not trained on our data distribution, we fine-tuned all generation and editing baselines on our exact training set and evaluated them on the same test sets. AMBIT still clearly outperforms all fine-tuned baselines:

Fine-tuned Generation Baselines:

DomainSystemPaired CLAP ↑\uparrowFAD ↓\downarrowKL-PANNs ↓\downarrowKL-PaSST ↓\downarrowFD-PaSST ↓\downarrowFD-OpenL3 ↓\downarrow
MusicAudioX-Turbo0.391.020.662.03244.7353.33
MusicAudioX0.401.230.601.25223.3459.14
MusicStable Audio Open0.331.590.761.62269.3056.51
MusicTangoFlux0.422.150.401.24571.2985.06
MusicBEWO0.341.200.691.40285.5759.99
MusicAMBIT0.460.710.510.95261.4544.59
SoundAudioX-Turbo0.381.120.713.01318.4559.51
SoundAudioX0.371.410.582.09216.6676.47
SoundStable Audio Open0.291.260.692.43231.6261.35
SoundTangoFlux0.381.960.411.96440.2072.49
SoundBEWO0.311.360.672.15250.0567.86
SoundAMBIT0.410.750.531.78235.4949.99

Fine-tuned Editing Baselines:

SystemPaired CLAP ↑\uparrowFD-CLAP ↓\downarrowFAD ↓\downarrowFD-PANNs ↓\downarrowKL-PANNs ↓\downarrowLSD ↓\downarrow
Audio-Omni0.660.150.7616.421.001.79
MMEdit0.570.241.0524.671.262.11
SwanWeave0.780.070.368.900.631.47
SmartDJ0.730.110.4413.170.771.53
ZETA0.600.170.9318.561.232.11
AudioEditor-BEWO0.630.160.6215.850.961.76
SDEdit0.470.230.8122.641.592.61
AMBIT0.840.050.246.010.461.27

W2: Fair Comparison with ReFL and GRPO

The comparison with ReFL and GRPO is also not quite fair.

Originally, the RL reward used only the semantic head (LsemL_{\text{sem}}) on the edit instruction because the scene head (LsceneL_{\text{scene}}) was trained on full scene descriptions rather than local edit instructions. The scene head here we add is the frozen scene-level head of AMBIT CLAP, measuring cosine similarity between the generated audio’s scene embedding and a description derived from the ground-truth target ScenePlan, including source content, timing and trajectories. To ensure a strictly fair comparison, we evaluated how ReFL and GRPO perform when rewarded with both heads (Lsem+LsceneL_{\text{sem}} + L_{\text{scene}}):

MetricAMBITSFTReFL (semantic)GRPO (semantic)ReFL (+scene)GRPO (+scene)Joint OPSD
Paired CLAP ↑\uparrow0.8390.8550.8530.8510.8510.8530.867
FD-CLAP ↓\downarrow0.0470.0420.0440.0430.0440.0430.040
FAD ↓\downarrow0.2420.2320.2330.2280.2440.2340.211
FD-PANNs ↓\downarrow6.0125.3985.5055.5535.5475.4095.282
Classifier KL ↓\downarrow0.4550.4020.3910.4170.3980.4060.351
LSD ↓\downarrow1.2681.1101.1461.1491.1321.1381.076
GCC ↓\downarrow14.80511.89712.05312.55310.97412.06711.130
CRW ↓\downarrow11.9477.5187.8458.9236.7188.1905.854
FSAD ↓\downarrow0.01420.01190.01160.01170.01140.01250.0115

Compared to SFT, Joint OPSD further reduces CRW by 22.1% (7.518→5.8547.518 \to 5.854) and Classifier KL by 12.7% (0.402→0.3510.402 \to 0.351). Compared to ReFL (+scene), Joint OPSD reduces FAD by 13.5% and CRW by 12.9% while achieving the best results on 7 of 9 metrics.

W2: Measurement of Direction, Timing, & Preservation

We evaluated direction, timing (activity IoU), and background preservation across all post-training methods:

MetricAMBITSFTReFLGRPOReFL (+scene)GRPO (+scene)Joint OPSD
Isolated-source level change ↓\downarrow (dB)1.6541.4601.9121.9921.9781.9791.285
Isolated-source direction MAE ↓\downarrow (rad)0.1710.1690.1680.1720.1660.1680.161
Overlap W time-frequency NMSE ↓\downarrow0.7090.7670.6120.6990.8010.6120.507
Overlap W spectral error ↓\downarrow0.4420.4360.4340.4380.4630.4320.380
Overlap level change ↓\downarrow (dB)2.8722.4252.8992.8372.6202.7191.859
Object spherical MAE ↓\downarrow (rad)0.2000.2040.1980.2290.1980.1950.165
Object azimuth MAE ↓\downarrow (rad)0.1540.1650.1530.1880.1520.1520.120
Object angular trajectory RMSE ↓\downarrow (rad)0.2700.2690.2640.2980.2660.2590.225
Scene wCS ↑\uparrow0.7050.7040.7220.7090.7120.7120.744
Scene activity IoU ↑\uparrow0.9330.9360.9290.9310.9300.9330.942

W2: Component ablation for OPSD

To support the benefits of the joint design, we ablated:

  • AR OPSD: distills planning only while retaining GT-plan flow supervision for rendering.
  • DiT OPSD: distills rendering only while retaining GT-token supervision for planning.
  • GT-plan rollout: replaces the student plan with the GT plan during rendering rollout.
  • GT clean target: retains AR OPSD and student-plan rollout, but replaces the DiT teacher prediction with the GT clean latent.

Our OPSD component ablation supports the benefits of jointly distilling planning and rendering on student rollouts with teacher guidance:

MetricAR OPSDDiT OPSDGT-plan rolloutGT clean targetJoint OPSD
Paired CLAP ↑\uparrow0.8520.8600.8580.8590.867
FD-CLAP ↓\downarrow0.0420.0410.0410.0410.040
FAD ↓\downarrow0.2370.2230.2200.2140.211
FD-PANNs ↓\downarrow5.4725.3145.2615.1765.282
KL ↓\downarrow0.4130.3850.3790.3650.351
LSD ↓\downarrow1.1091.0911.0911.0831.076
GCC ↓\downarrow11.90311.60612.15311.60011.130
CRW ↓\downarrow8.1936.7667.7176.4705.854
FSAD ↓\downarrow0.01170.01170.01180.01170.0115

Q2: Test Request Construction & Real-World STARSS23 Results

Test requests are constructed from scene annotations using natural-language templates that differ from those used in training. Generation requests describe source count/content, approximate positions or motion, and qualitative onset/offset times; editing requests describe the target modification to the scene.

Below are the results on the 50-clip real STARSS23 subset, the rest synthetic subset on original baselines, and the full public STARSS23 dev-test split:

STARSS23 (50-clip subset):

MetricAMBITAudioX-TurboTangoFlux
Paired CLAP ↑\uparrow0.4210.3780.373
FAD ↓\downarrow2.8963.0043.637
KL-PANNs ↓\downarrow1.7972.4312.361
FD-PaSST ↓\downarrow1035.797709.212766.587
FD-OpenL3 ↓\downarrow204.355156.873141.287
KL-PaSST ↓\downarrow2.0532.1582.216

Rest (Synthetic):

MetricAMBITAudioX-TurboTangoFlux
Paired CLAP ↑\uparrow0.4370.3510.363
FAD ↓\downarrow0.7270.9170.984
KL-PANNs ↓\downarrow0.5140.6470.556
FD-PaSST ↓\downarrow193.474204.769240.061
FD-OpenL3 ↓\downarrow45.41238.46488.458
KL-PaSST ↓\downarrow1.3671.9691.803

Full public STARSS23 dev-test:

MetricAMBITAudioX-TurboTangoFlux
Paired CLAP ↑\uparrow0.4520.3760.394
FAD ↓\downarrow2.5723.0023.511
KL-PANNs ↓\downarrow1.4501.6161.281
FD-PaSST ↓\downarrow915.305846.674880.995
FD-OpenL3 ↓\downarrow142.084176.584173.611
KL-PaSST ↓\downarrow2.0281.9761.672

Q3/W1: Planner issue explanation & Planner comparison

Why is Qwen3.5-4B used as the planner baseline when Qwen3.5-27B reaches 100% executability in Table 13? Was it prompted zero-shot, and did the prompt include the compass and time-phase conventions used by AMBIT?

We initially chose Qwen3.5-4B as the main planner baseline because it is closer in scale to our 0.8B planner while already achieving high executability. It was prompted zero-shot, with the prompt explicitly specifying the ScenePlan schema, the eight compass-to-azimuth mappings, coordinate conventions, onset/offset constraints, and motion rules.

I think a planner baseline fine-tuned on the same plans, together with results on real recordings and free-form requests

To thoroughly address your concerns, we conducted the following additional controlled experiments:

  1. Fine-tuned Qwen3.5-27B planner on our training plans:
SystemGCC ↓\downarrowCRW ↓\downarrowFSAD ↓\downarrowDOA ↓\downarrowScene Az. MAE ↓\downarrowScene Ang. Traj. RMSE ↓\downarrowwCS ↑\uparrowObject Spherical MAE ↓\downarrowObject Az. MAE ↓\downarrowObject Ang. Traj. RMSE ↓\downarrow
Qwen3.5-27B (FT) + DiT12.98710.1170.01940.5160.4070.6210.5530.5450.4220.632
AMBIT12.5928.7530.02040.5110.4030.6300.5570.5310.4150.629
  1. Fine-tuned Qwen3.5-4B planner on our training set:
SystemGCC ↓\downarrowCRW ↓\downarrowFSAD ↓\downarrowDOA ↓\downarrowScene Az. MAE ↓\downarrowScene Ang. Traj. RMSE ↓\downarrowwCS ↑\uparrowObject Spherical MAE ↓\downarrowObject Az. MAE ↓\downarrowObject Ang. Traj. RMSE ↓\downarrow
BEWO + AS (FT)20.47412.9230.02470.6560.5260.7140.4920.5910.6420.725
TangoFlux + AS (FT)21.23511.8720.01970.7250.4510.6620.4760.5780.5600.712
Qwen3.5-4B (FT) + DiT15.4199.9540.02200.6280.4820.6430.5060.5690.5730.658
AMBIT12.5928.7530.02040.5110.4030.6300.5570.5310.4150.629
  1. GT-plan control

Here we provide identical ground-truth plans to all systems to test spatial rendering independently of planning conventions:

MetricBEWO + ASTangoFlux + ASAMBIT DiT
Paired CLAP ↑\uparrow0.37600.42780.5840
FAD ↓\downarrow2.41683.43630.6158
KL-PANNs ↓\downarrow1.76451.63961.1690
FD-PaSST ↓\downarrow253.2301232.2574151.4262
FD-OpenL3 ↓\downarrow44.497662.240627.9837
CRW ↓\downarrow11.305611.38606.9785
Scene DOA ↓\downarrow0.52200.55170.4425
Object azimuth MAE ↓\downarrow0.18880.18470.1637
  1. Free-form requests

We also compare AMBIT (0.8B) against Qwen3.5-4B (FT) + DiT on non-templated, free-form natural-language requests (paraphrased by an external LLM: GPT-6 Astra):

MetricAMBITQwen3.5-4B (FT) + DiTMetricAMBITQwen3.5-4B (FT) + DiT
Paired CLAP ↑\uparrow0.400.48FSAD ↓\downarrow0.020.03
FAD ↓\downarrow0.940.88Scene DOA ↓\downarrow0.971.03
KL-PANNs ↓\downarrow0.710.68Scene Az. MAE ↓\downarrow0.900.93
KL-PaSST ↓\downarrow1.441.13Scene Ang. Traj. RMSE ↓\downarrow1.131.16
FD-PaSST ↓\downarrow228.31180.82wCS ↑\uparrow0.350.44
FD-OpenL3 ↓\downarrow47.1230.75Object Spherical MAE ↓\downarrow0.780.83
GCC ↓\downarrow17.0517.65Object Az. MAE ↓\downarrow0.810.79
CRW ↓\downarrow11.0112.97Object Ang. Traj. RMSE ↓\downarrow1.010.90

On unseen free-form requests, the larger Qwen3.5-4B (FT) planner shows stronger linguistic generalization on semantic/acoustic metrics, which is expected given its larger pretrained language capacity. Nevertheless, our 0.8B AMBIT still outperforms Qwen3.5-4B (FT) + DiT on 7 of 10 spatial metrics—winning all three physical metrics (GCC, CRW, FSAD), all three scene-level angular error metrics (Scene DOA, Az. MAE, and Ang. Traj. RMSE), and Object Spherical MAE. This demonstrates that unified Transfusion training provides good physical and angular spatial coherence, while scaling the AR backbone is a promising direction for further improving free-form linguistic robustness.

Q4: Pseudo Plan clarification

For the Sphere360 pseudo plans, Qwen3-Omni only receives the W channel as audio. How are source directions inferred, and how reliable are these pseudo labels?

We use Qwen3-Omni to generate ScenePlan pseudo-annotations for Sphere360 recordings through a two-stage audiovisual pipeline. Importantly, Qwen3-Omni receives synchronized 360∘360^\circ video alongside the downmixed omnidirectional W-channel audio, first producing an audiovisual caption and then expanding it into a structured ScenePlan. Thus, directional attributes are inferred from visual spatial cues in the 360∘360^\circ video and audiovisual correspondence rather than from the W channel alone.

To evaluate the reliability of these pseudo-plans, we conducted a human-reference audit on 50 Sphere360 clips (5 raters, 10 audiovisual clips each):

MetricResult
Coarse event-category coverage ↑\uparrow44/54 (81.5%)
Mean activity-interval IoU ↑\uparrow0.839
Activity boundaries within 0.5 s ↑\uparrow72%
Coarse azimuth agreement within 22.5∘22.5^\circ ↑\uparrow88%

These human-audit results confirm that our two-stage audiovisual pipeline produces reliable spatial and temporal pseudo-plans for supervision.

Q5: VAE comparison

Yes, DAC, EnCodec, and WavTokenizer independently reconstruct each of the 4 FOA channels before reassembly:

CodecPer ch. (kbps)4 ch. (kbps)Configuration
DAC7.75231.00844.1 kHz; 9×1,0249 \times 1{,}024 codes; hop 512
EnCodec62424 kHz; bandwidth 6
WavTokenizer0.4801.92024 kHz; 40 tokens/s; 4,096 codes

Since the ImmerseDiffusion VAE is not open-source, we additionally compared AMBIT VAE with the open-source FOA tokenizer PhysWave:

Full-source subset:

VAEW SI-SDR ↑\uparrowDirection ↓\downarrow (∘^\circ)Intensity error ↓\downarrowW CLAP ↑\uparrow
AMBIT6.394.960.040.90
PhysWave-32.8423.910.260.69

Single-source subset:

VAEW SI-SDR ↑\uparrowDirection ↓\downarrow (∘^\circ)Intensity error ↓\downarrowW CLAP ↑\uparrow
AMBIT8.263.040.030.92
PhysWave-29.4912.600.170.73

PhysWave's direction error rises from 12.60∘12.60^\circ (single-source) to 23.91∘23.91^\circ (Full-source), while its single-source SI-SDR remains −29.49-29.49 dB (on a 64-clip alignment probe, alignment improves its SI-SDR from −32.02-32.02 to −20.78-20.78 dB, though neither alignment nor source count fully explains the gap). AMBIT VAE substantially outperforms PhysWave in both waveform fidelity and spatial accuracy.

Reviewer_nBeDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AMBIT, a single model for generating and editing multi-source FOA spatial audio. The model first predicts a structured ScenePlan with each source's content, timing, and 3D trajectory, then renders FOA audio with flow matching conditioned on that plan. Both steps share one Transfusion-style backbone. The authors also train their own FOA VAE for spatial consistency, and an on-policy self-distillation step that post-trains the planning and rendering parts together. Experiments on mostly simulated test sets show gains over several generation and editing baselines.

Critical strengths
  1. The problem is interesting and practical, and the author did a lot of work to build the full system, including large-scale data construction, a dedicated VAE, a spatial encoder, the shared planner and renderer, and post-training.
Critical weaknesses
  1. My main concern is whether the comparisons are reliable and fair. All results use the authors' own test sets, which are not public. The editing set is fully simulated. Public options include STARSS23 and BEWO-1M, etc; they are not be used. AMBIT is trained on the same simulation pipeline as the test data (same rooms, assets, and loudness normalization), while the baselines are not. As for Qwen3-Omni, it only receives the W channel, so it cannot know where any source is, and the Qwen planners are not told the coordinate conventions and decode without grammar constraints.

  2. The whole system combines many existing techniques but does not justify why each one is needed, and the ablations don't show it either. For the shared Transfusion planner and renderer, which is the core of the paper, there is no ablation. There is no plan-free variant, no separate-model variant, and no runs without the CLAP features or without the reference latents. For OPSD, the gains over SFT are small and come from single runs; the motivation behind OPSD is not clear, and the teacher-student design seems redundant.

In summary, I think the contributions are not very clear, and the experiments do not clearly support them.

Questions for authors
  1. Could the authors provide the results on the 50 STARSS23 clips separately, and on other public test sets, such as STARSS23 dev-test, the BEWO-1M test set, or some out-of-domain test sets?

  2. Can the authors add more ablations for the main design choices, such as shared Transfusion, without the AMBIT CLAP features, etc.?

  3. Are the Qwen planners told about the coordinate conventions? For editing, could Qwen3-Omni receive the source plan or SELD output instead of only the W channel?

6 comments

Thank you for your valuable comments. We have added separated real-world evaluations, full public STARSS23 dev-test results, fine-tuned baselines on our exact training data, comprehensive component ablations (shared vs. separated backbone, w/o CLAP, w/o reference latent, OPSD ablations), and SELD-enriched Qwen editing controls to answer your concerns below.

W1(Q3): Fair Planner Comparison in Table 4 & Coordinate Convention Issue

AMBIT is trained on the same simulation pipeline as the test data (same rooms, assets, and loudness normalization), while the baselines are not

First, in our original setup, the prompts for Qwen3-Omni and Qwen3.5-4B already explicitly included the full ScenePlan schema, the eight compass directions, and the exact coordinate conventions.

Second, to control for adaptation to our ScenePlan conventions and training distribution, we fine-tuned the Qwen3.5-4B planner and the audio baselines directly on our training set. We also evaluated a GT-plan control in which all compared systems receive the same ground-truth ScenePlan:

Fine-tuned Planner & Baselines:

SystemGCC ↓\downarrowCRW ↓\downarrowFSAD ↓\downarrowDOA ↓\downarrowScene Az. MAE ↓\downarrowScene Ang. Traj. RMSE ↓\downarrowwCS ↑\uparrowObject Spherical MAE ↓\downarrowObject Az. MAE ↓\downarrowObject Ang. Traj. RMSE ↓\downarrow
BEWO + AS (FT)20.47412.9230.02470.6560.5260.7140.4920.5910.6420.725
TangoFlux + AS (FT)21.23511.8720.01970.7250.4510.6620.4760.5780.5600.712
Qwen3.5-4B (FT) + DiT15.4199.9540.02200.6280.4820.6430.5060.5690.5730.658
AMBIT12.5928.7530.02040.5110.4030.6300.5570.5310.4150.629

To further evaluate rendering independently of planning, we provide all compared baselines with GT-plan:

GT-Plan Control:

MetricBEWO + ASTangoFlux + ASAMBIT DiT
Paired CLAP ↑\uparrow0.37600.42780.5840
FAD ↓\downarrow2.41683.43630.6158
KL-PANNs ↓\downarrow1.76451.63961.1690
FD-PaSST ↓\downarrow253.2301232.2574151.4262
FD-OpenL3 ↓\downarrow44.497662.240627.9837
CRW ↓\downarrow11.305611.38606.9785
Scene DOA ↓\downarrow0.52200.55170.4425
Object azimuth MAE ↓\downarrow0.18880.18470.1637

W1: Fair Comparison for baselines

Currently there is no other existing text-FOA pair benchmark. So in order to evaluate the performance of our tasks, we construct our own testsets. In order to verify the generalizability of our model, we run the comparison between AMBIT and other baselines on STARSS23 dev testset in Q1 below. To further ensure a strictly fair comparison under identical training distribution and normalization, we fine-tuned all generation and editing baselines on our exact training set and evaluated them on the same test sets:

For generation:

DomainSystemPaired CLAP ↑\uparrowFAD ↓\downarrowKL-PANNs ↓\downarrowKL-PaSST ↓\downarrowFD-PaSST ↓\downarrowFD-OpenL3 ↓\downarrow
MusicAudioX-Turbo0.391.020.662.03244.7353.33
MusicAudioX0.401.230.601.25223.3459.14
MusicStable Audio Open0.331.590.761.62269.3056.51
MusicTangoFlux0.422.150.401.24571.2985.06
MusicBEWO0.341.200.691.40285.5759.99
MusicAMBIT0.460.710.510.95261.4544.59
SoundAudioX-Turbo0.381.120.713.01318.4559.51
SoundAudioX0.371.410.582.09216.6676.47
SoundStable Audio Open0.291.260.692.43231.6261.35
SoundTangoFlux0.381.960.411.96440.2072.49
SoundBEWO0.311.360.672.15250.0567.86
SoundAMBIT0.410.750.531.78235.4949.99

For editing:

SystemPaired CLAP ↑\uparrowFD-CLAP ↓\downarrowFAD ↓\downarrowFD-PANNs ↓\downarrowKL-PANNs ↓\downarrowLSD ↓\downarrow
Audio-Omni0.660.150.7616.421.001.79
MMEdit0.570.241.0524.671.262.11
SwanWeave0.780.070.368.900.631.47
SmartDJ0.730.110.4413.170.771.53
ZETA0.600.170.9318.561.232.11
AudioEditor-BEWO0.630.160.6215.850.961.76
SDEdit0.470.230.8122.641.592.61
AMBIT0.840.050.246.010.461.27

After fine-tuning all baselines on our training set, AMBIT substantially outperforms them on both generation and editing tasks.

Q1: Separated Real-World vs. Synthetic Results & Public Test Set Evaluation

Below are the separated results on the 50-clip real-world subset (STARSS23) and the synthetic subset on original baselines, followed by our evaluation on the full public STARSS23 dev-test set:

STARSS23 (50-clip subset):

MetricAMBITAudioX-TurboTangoFlux
Paired CLAP ↑\uparrow0.4210.3780.373
FAD ↓\downarrow2.8963.0043.637
KL-PANNs ↓\downarrow1.7972.4312.361
FD-PaSST ↓\downarrow1035.797709.212766.587
FD-OpenL3 ↓\downarrow204.355156.873141.287
KL-PaSST ↓\downarrow2.0532.1582.216

Rest (Synthetic):

MetricAMBITAudioX-TurboTangoFlux
Paired CLAP ↑\uparrow0.4370.3510.363
FAD ↓\downarrow0.7270.9170.984
KL-PANNs ↓\downarrow0.5140.6470.556
FD-PaSST ↓\downarrow193.474204.769240.061
FD-OpenL3 ↓\downarrow45.41238.46488.458
KL-PaSST ↓\downarrow1.3671.9691.803

Public test set — STARSS23 dev-test:

MetricAMBITAudioX-TurboTangoFlux
Paired CLAP ↑\uparrow0.4520.3760.394
FAD ↓\downarrow2.5723.0023.511
KL-PANNs ↓\downarrow1.4501.6161.281
FD-PaSST ↓\downarrow915.305846.674880.995
FD-OpenL3 ↓\downarrow142.084176.584173.611
KL-PaSST ↓\downarrow2.0281.9761.672

AMBIT achieves the highest Paired CLAP and lowest FAD on both the 50-clip real subset and the full STARSS23 dev-test evaluation, and also achieves the lowest FD-OpenL3 on the full dev-test set. These results support AMBIT’s generalization to real-world recordings. Regarding BEWO-1M, its public benchmark is stereo/binaural rather than 4-channel FOA like STARSS23. However, we fine-tuned both BEWO and AudioEditor-BEWO on our 4-channel FOA training set in the tables above.

W2/Q2: More Ablations Ablation on AMBIT CLAP Features and Reference Latents: We ablated removing the CLAP features (w/o CLAP) and the reference FOA latent (w/o reference) during editing:

MetricAMBITw/o CLAPw/o reference
Paired CLAP ↑\uparrow0.8390.7810.625
FAD ↓\downarrow0.2420.2660.826
Classifier KL ↓\downarrow0.4550.6961.194
GCC ↓\downarrow14.80515.18315.307
CRW ↓\downarrow11.94715.01713.298
FSAD ↓\downarrow0.01420.01440.0149

Ablation on Shared vs. Two-Stage Separated Transfusion Model: We also compared our unified shared Transfusion backbone against a two-stage separated model (planner and renderer trained on the exact same training set separately):

Generation:

MetricSharedTwo-stage separated
Paired CLAP ↑\uparrow0.440.35
FAD ↓\downarrow0.731.81
KL ↓\downarrow0.520.71
GCC ↓\downarrow12.5917.94
CRW ↓\downarrow8.7512.85
FSAD ↓\downarrow0.020.03

Editing:

MetricSharedTwo-stage separated
Paired CLAP ↑\uparrow0.840.72
FAD ↓\downarrow0.240.37
KL ↓\downarrow0.461.01
GCC ↓\downarrow14.8126.02
CRW ↓\downarrow11.9561.87
FSAD ↓\downarrow0.010.03

W2: OPSD Motivation and Teacher–Student Design Ablation: Joint OPSD bridges the train–inference exposure bias between teacher-forced training and autoregressive/flow rollout by supervising planning on student-generated prefixes and rendering on states visited under student-generated plans.

To support the benefits of the joint design, we ablated:

  • AR OPSD: distills planning only while retaining GT-plan flow supervision for rendering.
  • DiT OPSD: distills rendering only while retaining GT-token supervision for planning.
  • GT-plan rollout: replaces the student plan with the GT plan during rendering rollout.
  • GT clean target: retains AR OPSD and student-plan rollout, but replaces the DiT teacher prediction with the GT clean latent.
MetricAR OPSDDiT OPSDGT-plan rolloutGT clean targetJoint OPSD
Paired CLAP ↑\uparrow0.8520.8600.8580.8590.867
FD-CLAP ↓\downarrow0.0420.0410.0410.0410.040
FAD ↓\downarrow0.2370.2230.2200.2140.211
FD-PANNs ↓\downarrow5.4725.3145.2615.1765.282
KL ↓\downarrow0.4130.3850.3790.3650.351
LSD ↓\downarrow1.1091.0911.0911.0831.076
GCC ↓\downarrow11.90311.60612.15311.60011.130
CRW ↓\downarrow8.1936.7667.7176.4705.854
FSAD ↓\downarrow0.01170.01170.01180.01170.0115

Joint OPSD outperforms both single-branch variants on all 9 metrics, as well as GT-plan rollout and GT clean target on 8 of 9 metrics, supporting our joint teacher–student distillation design.

W2: Plan-Free Variant To directly isolate the benefit of explicit planning, we trained a plan-free variant of AMBIT (w/o ScenePlan), where the AMBIT DiT renderer is conditioned directly on the natural-language prompt (and reference FOA latents for editing) without predicting an intermediate ScenePlan. As shown below, explicit ScenePlan conditioning improves all reported generation acoustic/semantic metrics and editing spatial metrics:

Generation metricMusic: AMBITMusic: Plan-freeSound: AMBITSound: Plan-freeEditing metricAMBITPlan-free
Paired CLAP ↑\uparrow0.460.400.410.35GCC ↓\downarrow14.8122.11
FAD ↓\downarrow0.711.740.751.81CRW ↓\downarrow11.9537.74
KL-PANNs ↓\downarrow0.510.720.530.75FSAD ↓\downarrow0.010.02
KL-PaSST ↓\downarrow0.951.211.782.19Scene DOA ↓\downarrow0.390.99
FD-PaSST ↓\downarrow261.45312.05235.49295.15wCS ↑\uparrow0.710.45
FD-OpenL3 ↓\downarrow44.5955.7249.9967.76Object azimuth MAE ↓\downarrow0.151.01

Q3: Qwen3-Omni input

Are the Qwen planners told about the coordinate conventions?

In our pipeline, the prompts for Qwen3-Omni and Qwen3.5-4B already explicitly included the full ScenePlan schema, the eight compass directions, and the exact coordinate conventions.

For editing, could Qwen3-Omni receive the source plan or SELD output instead of only the W channel?

For editing, ground-truth source plans are unavailable for unannotated input FOA recordings in practical scenarios. Because Qwen3-Omni only accepts mono/omnidirectional W-channel audio and lacks directional FOA channels, we implemented your suggestion by augmenting Qwen3-Omni with SELD estimated automatically from the 4-channel source FOA:

MetricAMBITW onlyW + SELD
Paired CLAP ↑\uparrow0.840.760.78
FAD ↓\downarrow0.240.290.25
KL-PANNs ↓\downarrow0.460.720.63
Object azimuth MAE ↓\downarrow (rad)0.150.710.50
Preserved direction MAE ↓\downarrow (rad)0.171.420.82
Preserved level change ↓\downarrow (dB)1.6520.9410.90

While adding SELD spatial cues clearly improves Qwen3-Omni over W only, AMBIT still outperforms W + SELD across all semantic, spatial, and background-preservation metrics.

Reviewer_LQpSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AMBIT, a unified Transfusion framework that couples autoregressive (AR) scene planning (SCENEPLAN) with continuous Diffusion Transformer (DiT) rectified-flow rendering for multi-source First-Order Ambisonics (FOA) generation and instruction-guided editing, supported by a spatial-covariance-aware FOA VAE and joint on-policy self-distillation (OPSD). While extending Transfusion-style planning and editing to native four-channel FOA with spatial covariance supervision is a meaningful contribution, the paper suffers from significant flaws in its subjective evaluation protocol, statistical reporting, and experimental/architectural design that currently undermine several of its empirical claims.

Critical strengths
  • Meaningful Unification of Native FOA Generation and Source-Preserving Editing: While native FOA generation (e.g., ImmerseDiffusion, SonicMotion, Diff-SAGe; Page 1, Lines 035-038; Page 2, Lines 105-107) and stereo Transfusion planning/editing (AudioChat; Page 1, Line 043) have been explored separately, unifying multi-source 3D FOA generation and localized editing via executable SCENEPLAN controls and full reference FOA latents (Page 3, Lines 145-151) is well-motivated and preserves unedited background sources well.
  • Physically Grounded FOA Continuous VAE (AMBIT VAE): Supervising inter-channel energy ratios, coherence magnitudes, and stabilized phase via Spatial Covariance Matching (SCM) alongside asymmetric grouped KL and auxiliary W-channel recovery (Page 4, Lines 197-215; Page 18, Lines 944-975) is acoustically principled and well-supported by the cumulative ablations in Table 1 (Page 7, Line 340).
Critical weaknesses
  1. Flawed Subjective Listening Test Protocol (Page 24, Appendix D.4, Lines 1261-1306):

    • Single Rater per Stimulus (N = 1): The authors sampled 400 cases per task, partitioned them into 50 non-overlapping sets of 20 cases, and had 20 student participants rate one set each without reuse ("no set was reused... with one participant rating all three outputs for each case", Page 24, Lines 1266-1272). Consequently, every individual audio case was evaluated by only a single listener, making it impossible to measure inter-rater reliability or disentangle stimulus quality from individual rater bias.
    • Static Binaural Evaluation Only: Despite targeting 360-degree 3D FOA authoring, the listening test only evaluates static binaural downmixes via fixed MIT KEMAR filters without head-tracking (Page 22, Lines 1173-1174; Page 24, Lines 1289-1290), and is relegated entirely to the appendix (mentioned only briefly on Page 7, Line 333).
  2. Statistical Reporting Flaws Across Subjective and Objective Evaluations:

    • No Confidence Intervals, Variance, or Hypothesis Tests: In Table 12 (Page 24, Lines 1273-1286) and the accompanying text (Page 25, Lines 1298-1306), only raw mean MOS scores are reported without standard deviations, 95% confidence intervals, or p-values. Several claimed subjective margins are small (e.g., 3.11 vs. 2.93 in generation quality MOS; 3.64 vs. 3.51 in editing quality MOS on a 5-point scale), making it unclear whether these differences are statistically significant under a 20-subject single-rater-per-item design. Similarly, Tables 1-6 (Pages 7-9) report single-run point estimates without variance or error bars.
  3. Minor Issues (Missing Native FOA Generation Baselines and Synthetic Test Dominance):

    • Although the paper cites native FOA generators such as ImmerseDiffusion, SonicMotion, and Diff-SAGe (Page 1, Lines 035-038; Page 2, Lines 105-107), Table 4 (Page 8, Line 412) only compares against cascaded mono/stereo generators paired with Pyroomacoustics (+ AS).
    • In Appendix A (Page 16, Lines 861-866; Page 17, Table 9), only 50 of the 8,000 generation test clips are real FOA recordings (STARSS23 dev-test), and their scores are pooled directly into the 7,950 synthetic Pyroomacoustics clips; all 5,000 editing test pairs are synthetic.
Questions for authors
  1. Subjective Test Statistics and Rater Design: Can you provide 95% confidence intervals and statistical significance tests (p-values) for Table 12 (Page 24), and justify why each stimulus was rated by only a single listener rather than multiple raters?
  2. Fair RL Post-Training Comparison: Why was the trained scene alignment head L_scene (Page 5, Eq. 5) omitted from the reward of ReFL and GRPO in Appendix E.5 (Page 28, Line 1472)? How do ReFL and GRPO perform in Table 6 when rewarded with L_sem + L_scene?
3 comments

Thank you for your insightful comments and positive recognition of our unified FOA framework and physically grounded VAE. We answer your questions and concerns below.

W1/Q1: Single Rater

Originally, to avoid listener fatigue (20 cases/task ×\times 3 systems per listener), we prioritized scene diversity (400 cases) over multi-rater overlap, using blinded within-case paired ratings to control relative system comparisons. However, as you rightly pointed out, this N=1N = 1 design precludes measuring inter-rater reliability and introduces item-level rater variance.

To rigorously resolve this flaw while keeping the exact same per-listener workload as the original study, we conducted a follow-up multi-rater study with 20 listeners and 80 cases per task (80 generation and 80 editing). Each listener rated 20 generation cases (3 systems each) and 20 editing cases (3 systems each) under overlapping set assignments, ensuring that every case was independently rated by N=5N = 5 different listeners:

Generation (Follow-up study, N=5N = 5 raters/case) — mean [95% CI]:

SystemQuality MOSInstruction adherenceSpatial adherenceSpatial naturalness
BEWO + AS2.86[2.64,3.09]2.92[2.67,3.19]2.80[2.59,3.00]2.98[2.72,3.24]
TangoFlux + AS2.72[2.50,2.95]2.88[2.65,3.11]2.66[2.42,2.90]2.83[2.62,3.06]
AMBIT3.07[2.87,3.27]3.33[3.13,3.55]3.36[3.14,3.57]3.17[2.91,3.44]

Editing (Follow-up study, N=5N = 5 raters/case) — mean [95% CI]:

SystemQuality MOSEdit successPreservation
SwanWeave1.98[1.78,2.18]2.00[1.81,2.20]1.93[1.76,2.10]
Qwen3-Omni + DiT3.63[3.42,3.83]3.17[2.96,3.37]3.36[3.15,3.57]
AMBIT3.75[3.54,3.96]3.55[3.32,3.78]3.59[3.40,3.78]

Generation — pairwise pp-values:

ComparisonQuality MOSInstruction adherenceSpatial adherenceSpatial naturalness
AMBIT vs. BEWO + AS0.12190.0010<0.00010.0858
AMBIT vs. TangoFlux + AS0.00670.0004<0.00010.0133
BEWO + AS vs. TangoFlux + AS0.23950.74150.29770.2511

Editing — pairwise pp-values:

ComparisonQuality MOSEdit successPreservation
AMBIT vs. SwanWeave<0.0001<0.0001<0.0001
AMBIT vs. Qwen3-Omni + DiT0.33300.00250.0516
SwanWeave vs. Qwen3-Omni + DiT<0.0001<0.0001<0.0001

AMBIT achieves the highest mean scores across all evaluated dimensions in the follow-up multi-rater study.

W1: Static Binaural Evaluation Only We used fixed KEMAR binaural rendering to ensure standardized rendering conditions and reproducible stimuli across A/B/C systems: dynamic head-tracking allows different head-rotation trajectories across systems, introducing viewing-angle confounders during source motion. KEMAR HRTFs still preserve ITD/ILD, pinna spectral, and trajectory cues, while front-back/elevation accuracy is directly evaluated in native 4-channel FOA via our 3D spherical metrics (Tables 4–5).

W2/Q1: Subjective Test Statistics

No Confidence Intervals, Variance, or Hypothesis Tests / Subjective Test Statistics

To answer your concern, below are the 95% confidence intervals (CIs) and pairwise statistical significance tests (pp-values) for the original Table 12.

Generation — mean [95% CI]:

SystemQuality MOSInstruction adherenceSpatial adherenceSpatial naturalness
BEWO + AS2.93[2.790,3.072]2.96[2.817,3.087]2.83[2.683,2.985]2.89[2.763,3.025]
TangoFlux + AS2.77[2.647,2.890]2.84[2.712,2.965]2.63[2.475,2.775]2.76[2.600,2.925]
AMBIT3.11[3.005,3.215]3.25[3.130,3.370]3.37[3.243,3.493]3.03[2.857,3.205]

Generation — pairwise pp-values:

MetricAMBIT vs. BEWOAMBIT vs. TangoFluxBEWO vs. TangoFlux
Quality MOS0.0153<0.00010.0078
Instruction adherence<0.0001<0.00010.1155
Spatial adherence<0.0001<0.00010.0081
Spatial naturalness0.15790.01100.0386

Editing — mean [95% CI]:

SystemQuality MOSEdit successPreservation
SwanWeave1.89[1.785,1.992]1.96[1.883,2.040]1.93[1.792,2.072]
Qwen3-Omni + DiT3.51[3.345,3.663]3.26[3.140,3.380]3.44[3.310,3.560]
AMBIT3.64[3.470,3.800]3.57[3.462,3.670]3.71[3.605,3.815]

Editing — pairwise pp-values:

MetricAMBIT vs. SwanWeaveAMBIT vs. QwenSwanWeave vs. Qwen
Quality MOS<0.00010.0707<0.0001
Edit success<0.00010.0002<0.0001
Preservation<0.0001<0.0001<0.0001

Q2: Fair RL Post-Training Comparison:

Originally, the RL reward used only the semantic head (LsemL_{\text{sem}}) on the edit instruction because the scene head (LsceneL_{\text{scene}}) was trained on full scene descriptions rather than local edit instructions. The scene head here we add is the frozen scene-level head of AMBIT CLAP, measuring cosine similarity between the generated audio’s scene embedding and a description derived from the GT target ScenePlan, including source content, timing and trajectories. To ensure a fair comparison, we evaluated how ReFL and GRPO perform when rewarded with both heads (Lsem+LsceneL_{\text{sem}} + L_{\text{scene}}):

MetricAMBITSFTReFL (semantic)GRPO (semantic)ReFL (+scene)GRPO (+scene)Joint OPSD
Paired CLAP ↑\uparrow0.8390.8550.8530.8510.8510.8530.867
FD-CLAP ↓\downarrow0.0470.0420.0440.0430.0440.0430.040
FAD ↓\downarrow0.2420.2320.2330.2280.2440.2340.211
FD-PANNs ↓\downarrow6.0125.3985.5055.5535.5475.4095.282
Classifier KL ↓\downarrow0.4550.4020.3910.4170.3980.4060.351
LSD ↓\downarrow1.2681.1101.1461.1491.1321.1381.076
GCC ↓\downarrow14.80511.89712.05312.55310.97412.06711.130
CRW ↓\downarrow11.9477.5187.8458.9236.7188.1905.854
FSAD ↓\downarrow0.01420.01190.01160.01170.01140.01250.0115

With scene-aware rewards included, Joint OPSD still achieves the best results on 7 of 9 metrics and remains within 1.5% of the best value on the remaining two. Compared with ReFL (+scene), Joint OPSD reduces FAD by 13.5% and CRW by 12.9% while retaining near-best GCC and FSAD, demonstrating broad improvements across both acoustic quality and spatial consistency.

W3(a) Native FOA Baseline Comparison

Initially, we could not evaluate ImmerseDiffusion, SonicMotion, and Diff-SAGe because their code and checkpoints are not publicly available. During the rebuttal period we have found PhysWave, a native FOA generator system. Below is the comparison between AMBIT and fine-tuned PhysWave:

MetricAMBITPhysWaveMetricAMBITPhysWave
Music—Paired CLAP ↑\uparrow0.460.31Sound—Paired CLAP ↑\uparrow0.410.34
Music—FAD ↓\downarrow0.712.15Sound—FAD ↓\downarrow0.751.58
Music—KL-PANNs ↓\downarrow0.510.66Sound—KL-PANNs ↓\downarrow0.530.57
Music—KL-PaSST ↓\downarrow0.951.65Sound—KL-PaSST ↓\downarrow1.782.27
Music—FD-PaSST ↓\downarrow261.45408.46Sound—FD-PaSST ↓\downarrow235.49295.39
Music—FD-OpenL3 ↓\downarrow44.5998.06Sound—FD-OpenL3 ↓\downarrow49.9996.05
MetricAMBITPhysWaveMetricAMBITPhysWave
GCC ↓\downarrow12.5911.33Scene Ang. Traj. RMSE ↓\downarrow0.630.70
CRW ↓\downarrow8.7518.02Scene wCS ↑\uparrow0.560.48
FSAD ↓\downarrow0.020.03Object Spherical MAE ↓\downarrow0.530.47
Scene DOA ↓\downarrow0.510.56Object Az. MAE ↓\downarrow0.420.52
Scene Az. MAE ↓\downarrow0.400.53Object Ang. Traj. RMSE ↓\downarrow0.630.62

As shown above, AMBIT outperforms fine-tuned PhysWave on all 12 acoustic/semantic metrics and 7 of 10 spatial metrics, demonstrating clear advantages in both audio fidelity and spatial rendering.

W3(b) Synthetic Test Dominance Below are the separated results on the 50-clip real-world subset and the synthetic subset on original baselines, along with an evaluation on the public STARSS23 dev-test split:

STARSS23 (50-clip subset):

MetricAMBITAudioX-TurboTangoFlux
Paired CLAP ↑\uparrow0.4210.3780.373
FAD ↓\downarrow2.8963.0043.637
KL-PANNs ↓\downarrow1.7972.4312.361
FD-PaSST ↓\downarrow1035.797709.212766.587
FD-OpenL3 ↓\downarrow204.355156.873141.287
KL-PaSST ↓\downarrow2.0532.1582.216

Rest (Synthetic):

MetricAMBITAudioX-TurboTangoFlux
Paired CLAP ↑\uparrow0.4370.3510.363
FAD ↓\downarrow0.7270.9170.984
KL-PANNs ↓\downarrow0.5140.6470.556
FD-PaSST ↓\downarrow193.474204.769240.061
FD-OpenL3 ↓\downarrow45.41238.46488.458
KL-PaSST ↓\downarrow1.3671.9691.803

Full STARSS23 dev-test:

MetricAMBITAudioX-TurboTangoFlux
Paired CLAP ↑\uparrow0.4520.3760.394
FAD ↓\downarrow2.5723.0023.511
KL-PANNs ↓\downarrow1.4501.6161.281
FD-PaSST ↓\downarrow915.305846.674880.995
FD-OpenL3 ↓\downarrow142.084176.584173.611
KL-PaSST ↓\downarrow2.0281.9761.672

AMBIT achieves the highest Paired CLAP and lowest FAD on both the 50-clip real subset and the full STARSS23 dev-test evaluation, and also achieves the lowest FD-OpenL3 on dev-test set. The results support AMBIT’s generalization to real-world recordings.

Dear Reviewer LQpS,

Thank you again for your thoughtful review. We have supplemented our responses with additional evaluations, statistical reporting for the listening study, and further post-training comparisons, with key additions incorporated into the revised manuscript in red. We hope these updates are helpful in addressing your comments, and we would be happy to clarify any remaining questions.

Thank you very much for your time and for engaging with our submission.

Reviewer_V5CNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a new system for FOA audio generation and editing. It is based on Transfusion, where a shared transformer performs planning (discrete scene description token prediction) and rendering (continuous VAE latent generation). While Transfusion itself has already been explored for audio generation and editing, the authors extend it for FOA audio with extra improvements, which include: (1) structured plan, (2) improved VAE, and (3) leveraging on-policy self-distillation (OPSD) to improve the inference-time performance of the model trained with teacher forcing. The training and comparison are conducted on a custom benchmark, where the proposed model outperforms some recent models.

Critical strengths

This is a system paper with careful engineering. The ablation study demonstrates the effectiveness of the proposed VAE and OPSD compared to their respective standard techniques. The finding that OPSD is favorable over other post-training methods for both autoregressive decoding and diffusion sampling seems valuable, given that many audio tasks (not just ones presented in this paper) involve both discrete (e.g., descriptions and instructions) and continuous (e.g., waveform or VAE latent) features.

Critical weaknesses

I wonder how fair the current benchmark is to the tested baselines. Most seem to be off-the-shelf checkpoints (correct me if I am wrong), and the evaluation setup appears in-domain for the proposed model. Also, some native FOA generators such as ImmerseDiffusion are not compared.

The results also do not separate synthetic and real-world data (STARSS23). I suspect the former is much easier than the latter because of the simplicity and deterministic nature of Pyroomacoustics (image-source method).

While the current ablations are informative, the current study does not fully address the key component-level questions. For example, the benefit of explicit planning does not seem to be evaluated here.

Overall, I find this a borderline submission. I recommend weak reject for now, but I am open to changing my assessment if the concerns regarding the evaluations are at least partly resolved.

Questions for authors

Did the authors reproduce and retrain the baselines (or some of them) using the same training data?

Could the authors report the real-world and synthetic data results separately?

I found Table 4 difficult to interpret, as the "system" column does not separate the planner, source generator, and the spatial renderer of each baseline. The cascaded baselines use the same room acoustics simulator used to create the ground truth (Pyroomacoustics), yet their spatial metrics are much worse. The gap seems to come mainly from the planning of Qwen3.5-4B, which appears not to be fine-tuned (is this correct?). This raises some questions. Could this result suggest that the proposed model performs well largely because it learned the dataset's conventions for mapping natural language to fine-grained spatial parameters? Also, how would the systems compare if all of them were given ground-truth plans?

AudioChat also uses Transfusion, but the current paper does not mention this. Could the authors state this more upfront and clarify the core differences between the proposed system and AudioChat (e.g., stereo vs. FOA, the use of OPSD, and so on)?

5 comments

Thank you for your constructive comments and for being open to raising your assessment. We have conducted new experiments including fine-tuning all baselines on our training set, evaluating on the full public STARSS23 dev-test split, fine-tuning the Qwen planner, and adding GT-plan and architecture ablations. We answer your questions below.

W1/Q1: Fair Comparison between AMBIT and baseline

Did the authors reproduce and retrain the baselines using the same training data? / I wonder how fair the current benchmark is to the tested baselines.

In our initial submission, we evaluated unfine-tuned checkpoints for the baselines.

To ensure a strictly fair in-domain comparison, we fine-tuned all generation and editing baselines on our exact training set and evaluated them on the same test sets. The results are shown below:

For generation:

DomainSystemPaired CLAP ↑\uparrowFAD ↓\downarrowKL-PANNs ↓\downarrowKL-PaSST ↓\downarrowFD-PaSST ↓\downarrowFD-OpenL3 ↓\downarrow
MusicAudioX-Turbo0.391.020.662.03244.7353.33
MusicAudioX0.401.230.601.25223.3459.14
MusicStable Audio Open0.331.590.761.62269.3056.51
MusicTangoFlux0.422.150.401.24571.2985.06
MusicBEWO0.341.200.691.40285.5759.99
MusicAMBIT0.460.710.510.95261.4544.59
SoundAudioX-Turbo0.381.120.713.01318.4559.51
SoundAudioX0.371.410.582.09216.6676.47
SoundStable Audio Open0.291.260.692.43231.6261.35
SoundTangoFlux0.381.960.411.96440.2072.49
SoundBEWO0.311.360.672.15250.0567.86
SoundAMBIT0.410.750.531.78235.4949.99

For editing:

SystemPaired CLAP ↑\uparrowFD-CLAP ↓\downarrowFAD ↓\downarrowFD-PANNs ↓\downarrowKL-PANNs ↓\downarrowLSD ↓\downarrow
Audio-Omni0.660.150.7616.421.001.79
MMEdit0.570.241.0524.671.262.11
SwanWeave0.780.070.368.900.631.47
SmartDJ0.730.110.4413.170.771.53
ZETA0.600.170.9318.561.232.11
AudioEditor-BEWO0.630.160.6215.850.961.76
SDEdit0.470.230.8122.641.592.61
AMBIT0.840.050.246.010.461.27

Even after fine-tuning all baselines on our exact training data, AMBIT consistently outperforms them across both generation and editing tasks.

W1: Native FOA generator Comparison

Initially, we could not evaluate ImmerseDiffusion, SonicMotion, and Diff-SAGe because their code and checkpoints are not publicly available. During the rebuttal period we have found PhysWave, a native FOA generator system. Below is the comparison between AMBIT and fine-tuned PhysWave:

MetricAMBITPhysWaveMetricAMBITPhysWave
Music—Paired CLAP ↑\uparrow0.460.31Sound—Paired CLAP ↑\uparrow0.410.34
Music—FAD ↓\downarrow0.712.15Sound—FAD ↓\downarrow0.751.58
Music—KL-PANNs ↓\downarrow0.510.66Sound—KL-PANNs ↓\downarrow0.530.57
Music—KL-PaSST ↓\downarrow0.951.65Sound—KL-PaSST ↓\downarrow1.782.27
Music—FD-PaSST ↓\downarrow261.45408.46Sound—FD-PaSST ↓\downarrow235.49295.39
Music—FD-OpenL3 ↓\downarrow44.5998.06Sound—FD-OpenL3 ↓\downarrow49.9996.05
MetricAMBITPhysWaveMetricAMBITPhysWave
GCC ↓\downarrow12.5911.33Scene Ang. Traj. RMSE ↓\downarrow0.630.70
CRW ↓\downarrow8.7518.02Scene wCS ↑\uparrow0.560.48
FSAD ↓\downarrow0.020.03Object Spherical MAE ↓\downarrow0.530.47
Scene DOA ↓\downarrow0.510.56Object Az. MAE ↓\downarrow0.420.52
Scene Az. MAE ↓\downarrow0.400.53Object Ang. Traj. RMSE ↓\downarrow0.630.62

As shown above, AMBIT outperforms fine-tuned PhysWave on all 12 acoustic/semantic metrics and 7 of 10 spatial metrics, demonstrating clear advantages in both audio fidelity and spatial rendering.

W2/Q2: Separate synthetic and real-world data

Below are the separated results on the 50-clip real-world subset (STARSS23) and the synthetic subset on original baselines. And to verify the generalizability of our model, we also run an additional evaluation on the full public STARSS23 dev-test split:

STARSS23 (50-clip subset):

MetricAMBITAudioX-TurboTangoFlux
Paired CLAP ↑\uparrow0.4210.3780.373
FAD ↓\downarrow2.8963.0043.637
KL-PANNs ↓\downarrow1.7972.4312.361
FD-PaSST ↓\downarrow1035.797709.212766.587
FD-OpenL3 ↓\downarrow204.355156.873141.287
KL-PaSST ↓\downarrow2.0532.1582.216

Rest (Synthetic):

MetricAMBITAudioX-TurboTangoFlux
Paired CLAP ↑\uparrow0.4370.3510.363
FAD ↓\downarrow0.7270.9170.984
KL-PANNs ↓\downarrow0.5140.6470.556
FD-PaSST ↓\downarrow193.474204.769240.061
FD-OpenL3 ↓\downarrow45.41238.46488.458
KL-PaSST ↓\downarrow1.3671.9691.803

Full public STARSS23 dev-test:

MetricAMBITAudioX-TurboTangoFlux
Paired CLAP ↑\uparrow0.4520.3760.394
FAD ↓\downarrow2.5723.0023.511
KL-PANNs ↓\downarrow1.4501.6161.281
FD-PaSST ↓\downarrow915.305846.674880.995
FD-OpenL3 ↓\downarrow142.084176.584173.611
KL-PaSST ↓\downarrow2.0281.9761.672

AMBIT achieves the highest Paired CLAP and lowest FAD on both the 50-clip real subset and the full STARSS23 dev-test evaluation, and also achieves the lowest FD-OpenL3 on the full dev-test set. These results support AMBIT’s generalization to real-world recordings.

Q3: System interpretability

We explicitly clarify the three components in Table 4 here:

SystemPlannerSource audio generationSpatial rendering
BEWO + ASQwen3.5-4BBEWOPyroomacoustics
TangoFlux + ASQwen3.5-4BTangoFluxPyroomacoustics
Qwen3.5-4B + DiTQwen3.5-4BAMBIT DiTAMBIT DiT
AMBITAMBIT ARAMBIT DiTAMBIT DiT

In BEWO + AS and TangoFlux + AS, BEWO or TangoFlux generates individual source audio clips from Qwen3.5-4B's plan, which are then spatialized and mixed into 4-channel FOA via Pyroomacoustics (+ AS), whereas AMBIT DiT jointly performs multi-source generation and native 4-channel FOA spatial rendering in a single stage.

Q3: Fine-tuned Planner & Baselines

Initially, Qwen3.5-4B was prompted zero-shot with the ScenePlan schema, coordinate conventions, and eight compass directions. To address your concern about learning dataset conventions, we fine-tuned the Qwen3.5-4B planner and the audio baselines on our training set. As shown below, AMBIT still significantly outperforms both the fine-tuned cascades and Qwen3.5-4B (fine-tuned) + DiT:

SystemGCC ↓\downarrowCRW ↓\downarrowFSAD ↓\downarrowDOA ↓\downarrowScene Az. MAE ↓\downarrowScene Ang. Traj. RMSE ↓\downarrowwCS ↑\uparrowObject Spherical MAE ↓\downarrowObject Az. MAE ↓\downarrowObject Ang. Traj. RMSE ↓\downarrow
BEWO + AS (FT)20.47412.9230.02470.6560.5260.7140.4920.5910.6420.725
TangoFlux + AS (FT)21.23511.8720.01970.7250.4510.6620.4760.5780.5600.712
Qwen3.5-4B (FT) + DiT15.4199.9540.02200.6280.4820.6430.5060.5690.5730.658
AMBIT12.5928.7530.02040.5110.4030.6300.5570.5310.4150.629

Q3: GT-Plan Control

We also agree with your suggestion to provide identical GT-plan to all systems to evaluate rendering independently of planning. Under identical GT-plan inputs, AMBIT DiT still outperforms the baselines across all semantic and spatial metrics:

MetricBEWO + ASTangoFlux + ASAMBIT DiT
Paired CLAP ↑\uparrow0.37600.42780.5840
FAD ↓\downarrow2.41683.43630.6158
KL-PANNs ↓\downarrow1.76451.63961.1690
FD-PaSST ↓\downarrow253.2301232.2574151.4262
FD-OpenL3 ↓\downarrow44.497662.240627.9837
CRW ↓\downarrow11.305611.38606.9785
Scene DOA ↓\downarrow0.52200.55170.4425
Object azimuth MAE ↓\downarrow0.18880.18470.1637

W3: Benefit of Explicit Planning To directly isolate the benefit of explicit planning, we trained a plan-free variant of AMBIT (w/o ScenePlan), where the AMBIT DiT renderer is conditioned directly on the natural-language prompt (and reference FOA latents for editing) without predicting an intermediate ScenePlan. As shown below, explicit ScenePlan conditioning consistently improves all reported generation acoustic/semantic metrics and editing spatial metrics:

Generation metricMusic: AMBITMusic: Plan-freeSound: AMBITSound: Plan-freeEditing metricAMBITPlan-free
Paired CLAP ↑\uparrow0.460.400.410.35GCC ↓\downarrow14.8122.11
FAD ↓\downarrow0.711.740.751.81CRW ↓\downarrow11.9537.74
KL-PANNs ↓\downarrow0.510.720.530.75FSAD ↓\downarrow0.010.02
KL-PaSST ↓\downarrow0.951.211.782.19Scene DOA ↓\downarrow0.390.99
FD-PaSST ↓\downarrow261.45312.05235.49295.15wCS ↑\uparrow0.710.45
FD-OpenL3 ↓\downarrow44.5955.7249.9967.76Object azimuth MAE ↓\downarrow0.151.01

Furthermore, to isolate the benefit of our unified Transfusion backbone, we compare our shared backbone against a two-stage separated variant, where AR planner and DiT renderer are trained separately on the same dataset using the same ScenePlan interface:

Generation:

MetricSharedTwo-stage separated
Paired CLAP ↑\uparrow0.440.35
FAD ↓\downarrow0.731.81
KL-PANNs ↓\downarrow0.520.71
GCC ↓\downarrow12.5917.94
CRW ↓\downarrow8.7512.85
FSAD ↓\downarrow0.020.03

Editing:

MetricSharedTwo-stage separated
Paired CLAP ↑\uparrow0.840.72
FAD ↓\downarrow0.240.37
KL-PANNs ↓\downarrow0.461.01
GCC ↓\downarrow14.8126.02
CRW ↓\downarrow11.9561.87
FSAD ↓\downarrow0.010.03

Q4: AudioChat Difference

In our original submission, AudioChat is cited on page 3 (lines 120–121). Both AudioChat and AMBIT build on Transfusion. Here we clarify the three main differences between AudioChat and AMBIT:

  1. Task and Control (Stereo vs. 3D FOA): AudioChat targets mono/stereo audio via text-audio interleaving, whereas AMBIT is a unified FOA system supporting both generation and editing. It targets 4-channel 3D FOA via an explicit ScenePlan (specifying source content, timing, and 3D trajectories) and reference FOA latent conditioning.

  2. Tokenizer (Standard VAE vs. FOA VAE): AudioChat uses a standard audio VAE, whereas AMBIT introduces a 4-channel FOA VAE with Spatial Covariance Matching (SCM), grouped KL, and W-channel recovery to preserve inter-channel energy, coherence, and phase.

  3. Post-Training (Teacher Forcing vs. Joint OPSD): AudioChat relies on standard teacher-forced training. AMBIT introduces Joint OPSD to jointly refine discrete AR planning and continuous DiT rendering on student rollouts.

Comments on the paper

2 comments

Dear Reviewer TW4W,

Thank you for your constructive feedback and helpful suggestions during the rebuttal process. We appreciate your careful consideration of our responses and your positive assessment of our work.

Dear Reviewers,

As the discussion period draws to a close, we wanted to thank you again for your constructive comments and for helping us improve our work. We would be very grateful to hear if you have any further feedback or remaining questions before the deadline.

Thank you very much for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.