- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents AMBIT, a unified framework for First-Order Ambisonics (FOA) generation and editing. The core idea is to first predict an executable ScenePlan, which specifies each source's content, activity interval, and 3D trajectory, and then render FOA audio conditioned on the compiled plan. Planning and rendering share one Transfusion backbone, in which planning is autoregressive and rendering uses rectified flow. The paper further proposes 1) an FOA-aware VAE with auxiliary W-channel reconstruction, grouped KL, and spatial covariance matching, and 2) an on-policy self-distillation (OPSD) stage, in which frozen teachers conditioned on the target plan and target audio supervise the student's planning and rendering on its own rollouts. The model is trained on 1.6M generation examples and 1M editing pairs, most of which are synthetic scenes rendered with Pyroomacoustics. Compared with text-to-audio models plus spatialization, existing audio editors, and SwanWeave, AMBIT shows better content, spatial, and preservation metrics. The paper also includes VAE ablations, post-training comparisons, and a user study.
Critical strengths
-
The overall design is reasonable and well motivated. Using an explicit ScenePlan as the interface between language and FOA rendering makes the content, timing, and trajectory of each source inspectable, and it naturally supports both generation and source-level editing in one model. The comparison with Qwen3.5-4B + DiT, which uses the same renderer, is a useful controlled experiment. The preservation results also show that the editing path keeps unedited sources well.
-
The paper is a substantial and carefully documented system. The FOA-specific VAE objectives, especially spatial covariance matching, are sensible. The evaluation covers content, binaural, scene/object direction, and preservation metrics, plus a user study. The appendix gives enough detail that the work should be reproducible if the code is released.
Critical weaknesses
-
My main concern is that most of the evaluation is in-distribution for AMBIT but not for the baselines. The 8k generation and 5k editing test sets are built with the same pipeline as the training data (Pyroomacoustics rendering, the same room classes, the same five edit operations). Only 50 of the 8k generation examples are real recordings, and they are not reported separately. On the other hand, the baselines are general T2A models, mono/stereo editors evaluated on downmixes, or cascades whose planner is a prompted Qwen3.5-4B. For example, in Table 4, Qwen3.5-4B + DiT uses the same renderer as AMBIT but has almost the same direction errors as the AS pipelines (DOA 1.07 vs. 1.04). This suggests that most of the spatial gain comes from AMBIT's planner having learned the request conventions of the dataset (e.g., the eight compass categories and five time phases in Table 10), rather than from better spatial modeling. I think a planner baseline fine-tuned on the same plans, together with results on real recordings and free-form requests, is needed to support the claims.
-
My second concern is about OPSD, which is presented as one of the main contributions. In Table 6, most of the improvement over the initial model already comes from continued SFT (e.g., CRW 11.95 7.52, FAD 0.242 0.232). The further gain of joint OPSD over SFT is relatively small. The comparison with ReFL and GRPO is also not quite fair. Their reward is the semantic CLAP score, which the paper itself says does not measure direction, timing, or preservation. The OPSD teachers, by contrast, have access to the GT target plan and the GT target audio, and batch sizes and budgets also differ across recipes. In addition, OPSD is only applied to editing, and the main results in Tables 2–5 seem to be reported without OPSD. More broadly, the paper combines many components (ScenePlan, Transfusion, the VAE, AMBIT CLAP, OPSD), and the main text is very dense, with many key details left to the appendix. This makes it hard to see which part is responsible for which result.
Questions for authors
-
Are the AMBIT results in Tables 2–5 obtained without OPSD? Is OPSD applied to generation at all?
-
How are the test requests written? Are they generated from the plans with the same templates as in training? Could you report the results on the 50 real STARSS23 clips separately?
-
Why is Qwen3.5-4B used as the planner baseline when Qwen3.5-27B reaches 100% executability in Table 13? Was it prompted zero-shot, and did the prompt include the compass and time-phase conventions used by AMBIT?
-
For the Sphere360 pseudo plans, Qwen3-Omni only receives the W channel as audio. How are source directions inferred, and how reliable are these pseudo labels?
-
In Table 1, are DAC, EnCodec, and WavTokenizer applied per channel, and at what bitrates? A comparison with an FOA-specific autoencoder (e.g., the ImmerseDiffusion VAE or the FOA Tokenizer) would be more informative.
4 comments
W2: Fair Comparison with ReFL and GRPO
The comparison with ReFL and GRPO is also not quite fair.
Originally, the RL reward used only the semantic head () on the edit instruction because the scene head () was trained on full scene descriptions rather than local edit instructions. The scene head here we add is the frozen scene-level head of AMBIT CLAP, measuring cosine similarity between the generated audio’s scene embedding and a description derived from the ground-truth target ScenePlan, including source content, timing and trajectories. To ensure a strictly fair comparison, we evaluated how ReFL and GRPO perform when rewarded with both heads ():
| Metric | AMBIT | SFT | ReFL (semantic) | GRPO (semantic) | ReFL (+scene) | GRPO (+scene) | Joint OPSD |
|---|---|---|---|---|---|---|---|
| Paired CLAP | 0.839 | 0.855 | 0.853 | 0.851 | 0.851 | 0.853 | 0.867 |
| FD-CLAP | 0.047 | 0.042 | 0.044 | 0.043 | 0.044 | 0.043 | 0.040 |
| FAD | 0.242 | 0.232 | 0.233 | 0.228 | 0.244 | 0.234 | 0.211 |
| FD-PANNs | 6.012 | 5.398 | 5.505 | 5.553 | 5.547 | 5.409 | 5.282 |
| Classifier KL | 0.455 | 0.402 | 0.391 | 0.417 | 0.398 | 0.406 | 0.351 |
| LSD | 1.268 | 1.110 | 1.146 | 1.149 | 1.132 | 1.138 | 1.076 |
| GCC | 14.805 | 11.897 | 12.053 | 12.553 | 10.974 | 12.067 | 11.130 |
| CRW | 11.947 | 7.518 | 7.845 | 8.923 | 6.718 | 8.190 | 5.854 |
| FSAD | 0.0142 | 0.0119 | 0.0116 | 0.0117 | 0.0114 | 0.0125 | 0.0115 |
Compared to SFT, Joint OPSD further reduces CRW by 22.1% () and Classifier KL by 12.7% (). Compared to ReFL (+scene), Joint OPSD reduces FAD by 13.5% and CRW by 12.9% while achieving the best results on 7 of 9 metrics.
W2: Measurement of Direction, Timing, & Preservation
We evaluated direction, timing (activity IoU), and background preservation across all post-training methods:
| Metric | AMBIT | SFT | ReFL | GRPO | ReFL (+scene) | GRPO (+scene) | Joint OPSD |
|---|---|---|---|---|---|---|---|
| Isolated-source level change (dB) | 1.654 | 1.460 | 1.912 | 1.992 | 1.978 | 1.979 | 1.285 |
| Isolated-source direction MAE (rad) | 0.171 | 0.169 | 0.168 | 0.172 | 0.166 | 0.168 | 0.161 |
| Overlap W time-frequency NMSE | 0.709 | 0.767 | 0.612 | 0.699 | 0.801 | 0.612 | 0.507 |
| Overlap W spectral error | 0.442 | 0.436 | 0.434 | 0.438 | 0.463 | 0.432 | 0.380 |
| Overlap level change (dB) | 2.872 | 2.425 | 2.899 | 2.837 | 2.620 | 2.719 | 1.859 |
| Object spherical MAE (rad) | 0.200 | 0.204 | 0.198 | 0.229 | 0.198 | 0.195 | 0.165 |
| Object azimuth MAE (rad) | 0.154 | 0.165 | 0.153 | 0.188 | 0.152 | 0.152 | 0.120 |
| Object angular trajectory RMSE (rad) | 0.270 | 0.269 | 0.264 | 0.298 | 0.266 | 0.259 | 0.225 |
| Scene wCS | 0.705 | 0.704 | 0.722 | 0.709 | 0.712 | 0.712 | 0.744 |
| Scene activity IoU | 0.933 | 0.936 | 0.929 | 0.931 | 0.930 | 0.933 | 0.942 |
W2: Component ablation for OPSD
To support the benefits of the joint design, we ablated:
- AR OPSD: distills planning only while retaining GT-plan flow supervision for rendering.
- DiT OPSD: distills rendering only while retaining GT-token supervision for planning.
- GT-plan rollout: replaces the student plan with the GT plan during rendering rollout.
- GT clean target: retains AR OPSD and student-plan rollout, but replaces the DiT teacher prediction with the GT clean latent.
Our OPSD component ablation supports the benefits of jointly distilling planning and rendering on student rollouts with teacher guidance:
| Metric | AR OPSD | DiT OPSD | GT-plan rollout | GT clean target | Joint OPSD |
|---|---|---|---|---|---|
| Paired CLAP | 0.852 | 0.860 | 0.858 | 0.859 | 0.867 |
| FD-CLAP | 0.042 | 0.041 | 0.041 | 0.041 | 0.040 |
| FAD | 0.237 | 0.223 | 0.220 | 0.214 | 0.211 |
| FD-PANNs | 5.472 | 5.314 | 5.261 | 5.176 | 5.282 |
| KL | 0.413 | 0.385 | 0.379 | 0.365 | 0.351 |
| LSD | 1.109 | 1.091 | 1.091 | 1.083 | 1.076 |
| GCC | 11.903 | 11.606 | 12.153 | 11.600 | 11.130 |
| CRW | 8.193 | 6.766 | 7.717 | 6.470 | 5.854 |
| FSAD | 0.0117 | 0.0117 | 0.0118 | 0.0117 | 0.0115 |
Q2: Test Request Construction & Real-World STARSS23 Results
Test requests are constructed from scene annotations using natural-language templates that differ from those used in training. Generation requests describe source count/content, approximate positions or motion, and qualitative onset/offset times; editing requests describe the target modification to the scene.
Below are the results on the 50-clip real STARSS23 subset, the rest synthetic subset on original baselines, and the full public STARSS23 dev-test split:
STARSS23 (50-clip subset):
| Metric | AMBIT | AudioX-Turbo | TangoFlux |
|---|---|---|---|
| Paired CLAP | 0.421 | 0.378 | 0.373 |
| FAD | 2.896 | 3.004 | 3.637 |
| KL-PANNs | 1.797 | 2.431 | 2.361 |
| FD-PaSST | 1035.797 | 709.212 | 766.587 |
| FD-OpenL3 | 204.355 | 156.873 | 141.287 |
| KL-PaSST | 2.053 | 2.158 | 2.216 |
Rest (Synthetic):
| Metric | AMBIT | AudioX-Turbo | TangoFlux |
|---|---|---|---|
| Paired CLAP | 0.437 | 0.351 | 0.363 |
| FAD | 0.727 | 0.917 | 0.984 |
| KL-PANNs | 0.514 | 0.647 | 0.556 |
| FD-PaSST | 193.474 | 204.769 | 240.061 |
| FD-OpenL3 | 45.412 | 38.464 | 88.458 |
| KL-PaSST | 1.367 | 1.969 | 1.803 |
Full public STARSS23 dev-test:
| Metric | AMBIT | AudioX-Turbo | TangoFlux |
|---|---|---|---|
| Paired CLAP | 0.452 | 0.376 | 0.394 |
| FAD | 2.572 | 3.002 | 3.511 |
| KL-PANNs | 1.450 | 1.616 | 1.281 |
| FD-PaSST | 915.305 | 846.674 | 880.995 |
| FD-OpenL3 | 142.084 | 176.584 | 173.611 |
| KL-PaSST | 2.028 | 1.976 | 1.672 |
Q3/W1: Planner issue explanation & Planner comparison
Why is Qwen3.5-4B used as the planner baseline when Qwen3.5-27B reaches 100% executability in Table 13? Was it prompted zero-shot, and did the prompt include the compass and time-phase conventions used by AMBIT?
We initially chose Qwen3.5-4B as the main planner baseline because it is closer in scale to our 0.8B planner while already achieving high executability. It was prompted zero-shot, with the prompt explicitly specifying the ScenePlan schema, the eight compass-to-azimuth mappings, coordinate conventions, onset/offset constraints, and motion rules.
I think a planner baseline fine-tuned on the same plans, together with results on real recordings and free-form requests
To thoroughly address your concerns, we conducted the following additional controlled experiments:
- Fine-tuned Qwen3.5-27B planner on our training plans:
| System | GCC | CRW | FSAD | DOA | Scene Az. MAE | Scene Ang. Traj. RMSE | wCS | Object Spherical MAE | Object Az. MAE | Object Ang. Traj. RMSE |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.5-27B (FT) + DiT | 12.987 | 10.117 | 0.0194 | 0.516 | 0.407 | 0.621 | 0.553 | 0.545 | 0.422 | 0.632 |
| AMBIT | 12.592 | 8.753 | 0.0204 | 0.511 | 0.403 | 0.630 | 0.557 | 0.531 | 0.415 | 0.629 |
- Fine-tuned Qwen3.5-4B planner on our training set:
| System | GCC | CRW | FSAD | DOA | Scene Az. MAE | Scene Ang. Traj. RMSE | wCS | Object Spherical MAE | Object Az. MAE | Object Ang. Traj. RMSE |
|---|---|---|---|---|---|---|---|---|---|---|
| BEWO + AS (FT) | 20.474 | 12.923 | 0.0247 | 0.656 | 0.526 | 0.714 | 0.492 | 0.591 | 0.642 | 0.725 |
| TangoFlux + AS (FT) | 21.235 | 11.872 | 0.0197 | 0.725 | 0.451 | 0.662 | 0.476 | 0.578 | 0.560 | 0.712 |
| Qwen3.5-4B (FT) + DiT | 15.419 | 9.954 | 0.0220 | 0.628 | 0.482 | 0.643 | 0.506 | 0.569 | 0.573 | 0.658 |
| AMBIT | 12.592 | 8.753 | 0.0204 | 0.511 | 0.403 | 0.630 | 0.557 | 0.531 | 0.415 | 0.629 |
- GT-plan control
Here we provide identical ground-truth plans to all systems to test spatial rendering independently of planning conventions:
| Metric | BEWO + AS | TangoFlux + AS | AMBIT DiT |
|---|---|---|---|
| Paired CLAP | 0.3760 | 0.4278 | 0.5840 |
| FAD | 2.4168 | 3.4363 | 0.6158 |
| KL-PANNs | 1.7645 | 1.6396 | 1.1690 |
| FD-PaSST | 253.2301 | 232.2574 | 151.4262 |
| FD-OpenL3 | 44.4976 | 62.2406 | 27.9837 |
| CRW | 11.3056 | 11.3860 | 6.9785 |
| Scene DOA | 0.5220 | 0.5517 | 0.4425 |
| Object azimuth MAE | 0.1888 | 0.1847 | 0.1637 |
- Free-form requests
We also compare AMBIT (0.8B) against Qwen3.5-4B (FT) + DiT on non-templated, free-form natural-language requests (paraphrased by an external LLM: GPT-6 Astra):
| Metric | AMBIT | Qwen3.5-4B (FT) + DiT | Metric | AMBIT | Qwen3.5-4B (FT) + DiT |
|---|---|---|---|---|---|
| Paired CLAP | 0.40 | 0.48 | FSAD | 0.02 | 0.03 |
| FAD | 0.94 | 0.88 | Scene DOA | 0.97 | 1.03 |
| KL-PANNs | 0.71 | 0.68 | Scene Az. MAE | 0.90 | 0.93 |
| KL-PaSST | 1.44 | 1.13 | Scene Ang. Traj. RMSE | 1.13 | 1.16 |
| FD-PaSST | 228.31 | 180.82 | wCS | 0.35 | 0.44 |
| FD-OpenL3 | 47.12 | 30.75 | Object Spherical MAE | 0.78 | 0.83 |
| GCC | 17.05 | 17.65 | Object Az. MAE | 0.81 | 0.79 |
| CRW | 11.01 | 12.97 | Object Ang. Traj. RMSE | 1.01 | 0.90 |
On unseen free-form requests, the larger Qwen3.5-4B (FT) planner shows stronger linguistic generalization on semantic/acoustic metrics, which is expected given its larger pretrained language capacity. Nevertheless, our 0.8B AMBIT still outperforms Qwen3.5-4B (FT) + DiT on 7 of 10 spatial metrics—winning all three physical metrics (GCC, CRW, FSAD), all three scene-level angular error metrics (Scene DOA, Az. MAE, and Ang. Traj. RMSE), and Object Spherical MAE. This demonstrates that unified Transfusion training provides good physical and angular spatial coherence, while scaling the AR backbone is a promising direction for further improving free-form linguistic robustness.
Q4: Pseudo Plan clarification
For the Sphere360 pseudo plans, Qwen3-Omni only receives the W channel as audio. How are source directions inferred, and how reliable are these pseudo labels?
We use Qwen3-Omni to generate ScenePlan pseudo-annotations for Sphere360 recordings through a two-stage audiovisual pipeline. Importantly, Qwen3-Omni receives synchronized video alongside the downmixed omnidirectional W-channel audio, first producing an audiovisual caption and then expanding it into a structured ScenePlan. Thus, directional attributes are inferred from visual spatial cues in the video and audiovisual correspondence rather than from the W channel alone.
To evaluate the reliability of these pseudo-plans, we conducted a human-reference audit on 50 Sphere360 clips (5 raters, 10 audiovisual clips each):
| Metric | Result |
|---|---|
| Coarse event-category coverage | 44/54 (81.5%) |
| Mean activity-interval IoU | 0.839 |
| Activity boundaries within 0.5 s | 72% |
| Coarse azimuth agreement within | 88% |
These human-audit results confirm that our two-stage audiovisual pipeline produces reliable spatial and temporal pseudo-plans for supervision.
Q5: VAE comparison
Yes, DAC, EnCodec, and WavTokenizer independently reconstruct each of the 4 FOA channels before reassembly:
| Codec | Per ch. (kbps) | 4 ch. (kbps) | Configuration |
|---|---|---|---|
| DAC | 7.752 | 31.008 | 44.1 kHz; codes; hop 512 |
| EnCodec | 6 | 24 | 24 kHz; bandwidth 6 |
| WavTokenizer | 0.480 | 1.920 | 24 kHz; 40 tokens/s; 4,096 codes |
Since the ImmerseDiffusion VAE is not open-source, we additionally compared AMBIT VAE with the open-source FOA tokenizer PhysWave:
Full-source subset:
| VAE | W SI-SDR | Direction () | Intensity error | W CLAP |
|---|---|---|---|---|
| AMBIT | 6.39 | 4.96 | 0.04 | 0.90 |
| PhysWave | -32.84 | 23.91 | 0.26 | 0.69 |
Single-source subset:
| VAE | W SI-SDR | Direction () | Intensity error | W CLAP |
|---|---|---|---|---|
| AMBIT | 8.26 | 3.04 | 0.03 | 0.92 |
| PhysWave | -29.49 | 12.60 | 0.17 | 0.73 |
PhysWave's direction error rises from (single-source) to (Full-source), while its single-source SI-SDR remains dB (on a 64-clip alignment probe, alignment improves its SI-SDR from to dB, though neither alignment nor source count fully explains the gap). AMBIT VAE substantially outperforms PhysWave in both waveform fidelity and spatial accuracy.
Thank you for your thorough and constructive review. We answer your questions below, including scene-rewarded RL comparisons, fine-tuned Qwen3.5-4B and Qwen3.5-27B planners, GT-plan controls, fine-tuned generation/editing baselines, a human audit of Sphere360 pseudo-plans, and comparisons with an FOA-specific tokenizer (PhysWave).
Q1: OPSD explanation clarity
We apologize for the unclear presentation. Tables 2–5 report AMBIT before OPSD, whereas Table 6 separately evaluates the additional gains from OPSD. We agree that the final system’s results should be presented alongside the baselines.
And yes, Joint OPSD is applicable to generation as well. Below is the detailed comparison between AMBIT and AMBIT + OPSD on both generation and editing:
For generation:
For editing:
W1: Fair comparison for AMBIT and baselines
To answer your concern in that the baselines were not trained on our data distribution, we fine-tuned all generation and editing baselines on our exact training set and evaluated them on the same test sets. AMBIT still clearly outperforms all fine-tuned baselines:
Fine-tuned Generation Baselines:
Fine-tuned Editing Baselines: