acceptodds
Under review as a conference paper at ICLR 2027

Unified Spatial Audio Generation via Iterative Expert Routing

Abstract

Spatial audio generation aims to synthesize spatial sound fields that faithfully reflect user intent under diverse conditioning signals. However, existing methods remain limited in their ability to generate accurate, high-fidelity spatial audio from heterogeneous inputs. To address this limitation, we propose UniAmbi, a unified framework for jointly synthesizing environmental audio and speech in first-order Ambisonics (FOA) format from multi-conditions. Our framework integrates three complementary designs to achieve efficient generation with semantic consistency, spatial accuracy, and perceptual fidelity. First, we introduce a Looped DiT backbone that addresses the complexity of multi-conditional FOA generation through iterative computation with shared parameters, increasing effective depth without a proportional increase in parameter count. Second, we introduce a loop aware sliding window Mixture-of-Experts (MoE) module to mitigate the representational limitations of strict parameter sharing. By incorporating expert routing on the current loop iteration and local temporal context, the module enables round-dependent transformations while encouraging consistent expert selection among neighboring tokens. % Finally, we introduce per-loop on-policy self-distillation (OPSD) to supervise predictions at every loop depth along the model’s own generation trajectories, using feedback on spatial accuracy, semantic consistency, and perceptual quality. Finally, we introduce per-loop on-policy self-distillation (OPSD), which converts generation-level feedback into direct supervision for predictions at every loop depth along the model's own generation trajectories. Experimental results show that our model outperforms existing methods on multiple spatial audio generation tasks. Demos can be found at https://uniambi.github.io.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spatial audio, looped transformer, first-order ambisonics, mixture-of-experts

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_3WMcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces UniAmbi, a unified framework for generating first-order Ambisonics audio from text, video, or joint video-text inputs, supporting both environmental sounds and transcript-controlled speech. It combines a four-channel audio VAE, a parameter-sharing looped diffusion transformer, and loop-aware sliding-window mixture-of-experts routing.

Critical strengths

UniAmbi addresses a meaningful problem by supporting text, video, and joint video-text conditioning for both environmental sounds and transcript-controlled speech in native first-order Ambisonics. The four-channel VAE is well motivated: jointly encoding the sound field, with intensity and axis-wise reconstruction objectives, explicitly accounts for cross-channel spatial relationships. The integration of this representation with iterative generation and generation-level feedback is technically coherent.

Critical weaknesses

Parameter sharing reduces model size, but the main paper report neither inference latency nor FLOPs, peak memory, or training cost. Table 3 also omits parameter counts for the different loop configurations, preventing a direct assessment of the quality-versus-resource trade-off. At the same effective depth, the looped models do not consistently outperform the unshared DiT, and additional loops generally worsen several metrics. Consequently, the results establish parameter compactness more clearly than an advantageous computational trade-off.

Removing MoE changes both model capacity and routing, so the resulting degradation does not establish how much improvement comes specifically from expert specialization. Likewise, the “w/o SW” comparison does not separately establish the benefits of loop conditioning and sliding windows relative to the fixed-chunk routing discussed in Section 3.3. Parameter- and compute-matched controls, together with separate ablations of loop conditioning and window overlap, would make the architectural contribution more convincing.

The reported spatial metrics primarily measure agreement in directional energy or intensity, whereas semantic alignment is evaluated separately. These measurements may not reliably distinguish correct source placement from a scene with plausible overall spatial statistics but incorrect assignments of individual sounds to directions. The authors acknowledge this limitation but it directly affects the central claim of controllable spatial generation. Source-aware localization tests, multi-source identity-direction association tests, and a listening study would provide stronger validation.

Section 4.1 describes a substantial training corpus, but does not clearly report evaluation-set sizes, train/test splitting criteria, or source-recording/video-level overlap controls. It is also unclear whether the compared systems receive comparable task-specific training data or adaptation. Given the extensive synthetic spatialization pipeline and multimodal training mixture, these details are necessary to distinguish architectural improvements from data advantages and to assess generalization to genuinely unseen recordings and scenes.

Questions for authors

How exactly are the per-loop OPSD targets constructed from the combined feedback, including the evaluator implementations, reward weights, and reward-gating threshold δ\delta? How sensitive are the reported improvements to these choices?

Under matched training data and inference FLOPs, does loop-aware sliding-window routing still outperform an unshared DiT and a fixed-window MoE? Could the authors also report intermediate-loop quality and reward trajectories to establish whether successive loops actually refine the generated sound field?

Reviewer_AedYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UniAmbi generates first-order Ambisonics sound effects and transcript-controlled speech from text, panoramic video, or both. It combines a joint four-channel VAE, Looped DiT, loop-aware sliding-window MoE, and per-loop OPSD with reward-gated cross-loop distillation. Experiments cover all three conditioning modes and ablate routing, loop configurations, and post-training.

Critical strengths
  1. Unifying these conditioning modes and sound types is useful. Table 1 reports improvements over ViSAGe and OmniAudio on every comparable V2FOA metric, covering audio fidelity and spatial consistency.

  2. In Table 2, the full model improves all eight aggregate metrics over the no-MoE, Final-loop, and Ungated ILSD variants, supporting expert capacity, intermediate supervision, and reward-based teacher selection within this architecture.

Critical weaknesses
  1. The main comparisons are difficult to assess. Section 4.1 does not distinguish quoted from newly evaluated results or establish common test data and metric implementations. Appendix A.3 refers to Table 4 for training and test statistics (p. 15, line 795), but the table lists only training data. Test splits and synthetic/recorded composition remain unspecified.

  2. The post-training contribution is insufficiently documented. Section 3.4 points to appendix details (p. 6, lines 289–291 and 322–323), but Appendices A/B cover data construction and the VAE; no separate supplement is available. Reward definitions and weights, differentiable speech feedback, and OPSD target construction remain unclear. ReFL is a relevant direct reward-backpropagation baseline, but its FOA adaptation and training settings are also unclear, preventing attribution of the gap to OPSD itself.

  3. The parameter–quality trade-off remains unresolved. Table 3's two-loop model has the worst WER: 0.156 versus 0.079 for dense DiT, 0.120 for three loops, and 0.112 for four. It also trails dense DiT on FD, FAD, and wCS while improving four other metrics, but parameter counts are missing. The main model's loop configuration and the settings behind the FD difference between Table 2 (Ours: 402.825) and Table 3 (two loops: 428.280) are unexplained. The routing ablations also leave loop-index conditioning and sliding-window versus chunk-based routing unisolated.

A secondary concern is speech quality: WER is the only speech-specific generation metric in Tables 2/3, while Table 5's PESQ concerns VAE reconstruction. Evidence of generated speech naturalness and perceptual quality would strengthen the paper.

Questions for authors
  1. Please specify the provenance of Table 1's baseline results, per-task test sets and sizes, source-level train/test separation, and synthetic/recorded composition. Do all methods use the same evaluation data and metric implementations?

  2. Please define the rewards and weights, including video alignment, OPSD target construction, and key post-training settings. What role does Figure 2's ASR/WER play, and what differentiable objective or gradient surrogate is used if it contributes to target construction? How is ReFL adapted, and are rewards, loop supervision, distillation, and training budgets matched?

  3. What loop configuration does the 385M model use? Please explain how Tables 2/3 differ in training and evaluation settings and provide parameter counts to assess the quality trade-off. If gated ILSD is used beyond two loops, how is Eq. (4) extended?

  4. As a secondary point, is there evidence of generated speech naturalness or perceptual quality beyond WER?

Reviewer_6EN3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents UniAmbi, a unified framework for generating first-order Ambisonics (FOA) audio, including environmental sounds and transcript-controlled speech, from text, video, or joint video–text conditions. The framework uses a jointly trained four-channel FOA VAE, a parameter-shared Looped DiT, and loop-aware sliding-window mixture-of-experts routing. Post-training combines per-loop on-policy self-distillation with reward-gated intra-loop self-distillation using spatial, semantic, and audio-quality feedback. Training data combine synthetically spatialized recordings, synthetic multi-event scenes, and real-world panoramic video–audio pairs. The reported results show improvements over the selected FOA baselines in V2FOA, while the text-conditioned settings show mixed advantages over non-FOA audio generation baselines. Ablations examine routing components, supervision strategies, and loop configurations.

Critical strengths
  1. The paper addresses a practically relevant task by supporting text, video, and joint video–text conditioning within one FOA generation framework. The ability to generate both environmental sounds and transcript-controlled speech is also useful.

  2. The joint four-channel FOA VAE has a clear spatial audio motivation. The intensity-direction and axis-wise MR-STFT losses explicitly consider spatial relationships, and Table 5 shows improvements over the evaluated reconstruction baselines on most reported metrics.

  3. The V2FOA results are competitive. In Table 1, the proposed model achieves better numerical results than both ViSAGe and OmniAudio on all six comparable metrics, including wCS, CC-5fps, and CC-all, with substantially fewer reported parameters than OmniAudio.

  4. The paper includes ablations of several important components and training choices. The authors also acknowledge negative results and limitations, including the poorer VT2FOA FAD, the mixed effects of sliding-window routing, and the distinction between parameter savings and inference cost.

Critical weaknesses

1.Unclear presentation of the architecture and data flow.

I found it difficult to map the description in Section 3.2 to Figure 2. The text refers to a “Looped DiT denoiser equipped with loop aware sliding window MoE,” while the figure uses “Loop-MoE denoiser,” “Round-aware Loop-MoE,” “Shared DiT block,” and “Generator” without clearly explaining their hierarchy. Key latent variables and the relationship between the depicted generation and training processes are also insufficiently labelled. There are specific sources of confusion: Section 3.3 states that internal loops do not advance sampler time (lines 248–259), but the loop diagram is labelled from x_t to x_{t+1}. Section 3.2 describes temporal alignment through repetition of encoded video tokens (lines 215–220), whereas Figure 2 places “temporal align” before the vision encoder. These ambiguities make the framework unnecessarily difficult to follow.

2.Insufficient evidence connecting the main algorithmic choices to specific spatial audio improvements.

Section 3.1 does provide a spatially motivated VAE design, including joint FOA encoding, intensity-direction consistency, and axis-wise MR-STFT losses. However, the connection between the main routing and post-training mechanisms and particular spatial audio properties is less clearly established. The sliding-window motivation in Section 3.3 (lines 266–272) concerns temporal continuity and variable-duration sound events, which also apply to non-spatial audio. Table 2 reports aggregate improvements but does not isolate whether the mechanism improves cross-channel relationships, event–direction association, or moving-source continuity. In fact, removing the sliding window slightly improves both CC metrics. More targeted analysis would help distinguish improvements in general audio modelling from improvements in spatial modelling.

3.Missing comparison with a two-stage spatial audio generation baseline.

The introduction argues that end-to-end generation reduces coordination difficulties and error propagation relative to generating sounds before spatialization (lines 45–47), but this claim is not directly tested. I would particularly like to see a condition-matched baseline that first generates audio and then applies spatialization or spatial audio rendering. Table 1 provides direct spatial comparisons for V2FOA using ViSAGe and OmniAudio, but the T2FOA and VT2FOA comparisons do not include comparable baseline spatial scores. The authors acknowledge this limitation in Section 4.2 (lines 397–401), but the advantage over a two-stage alternative remains unclear.

4.Missing implementation details limit reproducibility.

Section 3.3 describes top-k expert routing but does not specify key configurations such as the number of experts, the value of k, the window size, or how information within the window is aggregated. Section 3.4 describes the feedback terms conceptually, but does not provide enough detail about their implementation and weights or the settings used for reward-gated distillation. The section refers readers to the appendix for feedback and training details (lines 291 and 323), yet the appendix contains dataset construction and VAE details rather than a complete specification of the proposed post-training procedure. I therefore had to infer several aspects of how the components and training objectives are combined.

5.Unclear test-set composition and real-world generalization.

Section 4.1 and Appendix A describe a mixture of synthetically spatialized audio and real-world video–FOA data. However, the paper does not clearly report the test-set sizes, source-specific splits, or separate results for synthetic and real-world data. Appendix A.3 states that training and test statistics are provided in Table 4 (line 795), but Table 4 is labelled “Statistics of the training data” and does not distinguish training and test subsets. I would like to see performance on a clearly defined, held-out real-world test set, with clarification of how source recordings and videos are separated across splits.

6.Limited evidence of realistic and consistent room acoustics.

In my listening to the demo samples, spatial placement sounded coarse and was mainly apparent through simple left–right differences. In some examples, different sounds within the same scene appeared to occupy different acoustic environments, reducing the perceived realism. Fro example : Text to audio 02 First <Des>a file</Des> <D>at the left side</D>, after that, <Des>a female speaker</Des> saying <S>A little later Clark approached hastily and said:</S> <D>from the back-right direction</D>. The sound effects of insects chirping suggest an outdoor setting, while the human voices clearly imply an indoor environment—this contradiction makes the entire scene feel highly artificial.

A related limitation appears in the FSDK rendering procedure in Appendix A.2 (lines 733–749): a mono RIR is multiplied by fixed FOA directional gains, so the direct sound and reflections share the same directional encoding rather than retaining path-specific arrival directions. The retained response is also limited to 50 ms after onset, subject to a 0.10 s upper bound. This concern applies to the described FSDK procedure, not necessarily to all training data, and does not by itself establish the cause of the demo artefacts. Nevertheless, the reported evaluation does not sufficiently establish realistic reflection structure, reverberation, or acoustic consistency between generated sources.

7.Unclear audio-format handling in the evaluation.

Section 4.1 specifies the feature models used for FD, KL, FAD, and CLAP, but does not explain how four-channel FOA is supplied to these evaluators or compared with non-FOA outputs. It is unclear whether evaluation uses the W channel, a downmix, directional decoding, or channel-wise scores. This processing choice is important for interpreting the comparisons in Table 1 and should be stated explicitly.

8.Presentation and reference errors require correction.

The manuscript contains concrete reference errors. The Luo et al. (2026a) entry on page 11, lines 570–571, has visibly corrupted formatting around “360°” and the subsequent title text in the submitted PDF. The same “On-policy self-distillation in diffusion models” paper, with the same arXiv identifier, also appears twice as Zhou et al. (2026a) and Zhou et al. (2026b) on page 12, lines 641–647. These are minor issues individually, but together with the architectural naming and figure ambiguities, they indicate that the manuscript needs more careful checking.

Questions for authors
  1. Could the authors compare the proposed model with a two-stage spatial audio generation baseline, where audio is first generated and then spatialized or rendered into FOA? The introduction argues that end-to-end generation reduces coordination difficulties and error propagation, but the current experiments do not directly test this claim. Such a comparison would help establish the benefit of the proposed framework over a simpler pipeline.

  2. Could the authors revise Figure 2 to clearly show the module hierarchy, key intermediate variables, and conditioning paths, using consistent names throughout the figure and text? I found it difficult to understand how the “Generator,” “Shared DiT block,” “Round-aware Loop-MoE,” and “Loop-MoE denoiser” relate to one another. Please also clarify why the internal loop diagram is labelled from x_t to x_{t+1}, although Section 3.3 states that internal loops do not advance sampler time.

  3. Why are direct comparisons of spatial audio quality provided only for V2FOA? Although the proposed model has spatial scores in all three settings, the T2FOA and VT2FOA baselines do not have comparable spatial scores. Given the title and the emphasis on unified spatial audio generation, I expected comparative evaluation of spatial quality across all three tasks, rather than comparisons limited to general audio quality and semantic alignment in two of them. Could the authors provide suitable spatial baselines or explain why these comparisons are unavailable?

  4. Could the authors report performance separately on a clearly defined, held-out real-world test set? Section 4.1 and Appendix A describe real-world video–FOA data alongside synthetic data, but the results do not distinguish between these sources. Are the current V2FOA and VT2FOA test sets composed of real-world recordings, synthetically augmented recordings, or a mixture? Please provide the test-set sizes and explain how source recordings and videos are separated between training and testing.

  5. How are FOA outputs processed when calculating FD, KL, FAD, and CLAP? Does the evaluation use the W channel, a downmix, directional decoding, or channel-wise calculations? Please explain how the comparison with non-FOA baselines is made consistent.

Reviewer_AwwaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents UniAmbi, a unified framework for native first-order Ambisonics generation under text, video, and joint video–text conditioning, supporting both environmental sounds and transcript-controlled speech. The framework combines a jointly trained four-channel audio VAE, a parameter-shared Looped DiT with loop-aware sliding-window expert routing, and per-loop on-policy self-distillation with reward-gated cross-loop distillation. The reported results demonstrate improvements over the evaluated video-to-FOA baselines with a compact parameter budget, while results under text-bearing conditions show mixed quality trade-offs.

Critical strengths

1,Useful unified generation setting. Jointly generating spatial sound effects and transcript-controlled speech under multiple conditioning modalities is a meaningful capability. Allowing text to supplement off-screen sounds is particularly relevant to audiovisual content creation.

2,Spatially motivated representation design. Joint four-channel encoding, together with intensity-direction and axis-wise spectral objectives, explicitly addresses dependencies among FOA channels. The reconstruction results in Table 5 support this design.

3,Promising video-to-FOA results. UniAmbi improves all reported comparable metrics over ViSAGe and OmniAudio in Table 1. Compared with OmniAudio, FAD decreases from 10.514 to 7.253 while using substantially fewer reported parameters.

4,Evidence supporting loop-wise post-training. Per-loop supervision and reward gating outperform their respective ablations across all eight aggregate metrics in Table 2, providing empirical support for directly supervising intermediate predictions and filtering cross-loop teachers.

Critical weaknesses
  1. Key methodological details are missing. The sliding-window router is not specified precisely, including context aggregation, window size, expert count, and routing configuration. The paper also omits explicit feedback objectives, reward weights, and important OPSD target-construction and optimization settings. Although Section 3.4 refers readers to the appendix, the provided appendices do not supply these details, substantially limiting reproducibility.

  2. The evaluation protocol and comparison fairness are insufficiently documented. Test-set sizes, source-level train/test separation, and baseline training or adaptation procedures are unclear. Since the framework uses approximately 10,890 hours of constructed data, the contribution of the proposed architecture cannot be cleanly separated from differences in training data. Moreover, T2FOA and VT2FOA lack comparisons against native spatial-audio methods, including relevant methods discussed in the paper.

  3. The distinct contribution of iterative expert routing is not sufficiently isolated. The approach builds on existing looped Transformers, loop-conditioned MoE, chunk routing, OPSD, and ILSD. The current ablations do not directly compare sliding-window routing with chunk routing or isolate loop-aware routing. Table 3 also shows substantial trade-offs: the non-looped DiT achieves better FD, FAD, wCS, and WER than the two-loop configuration, with WER increasing from 0.079 to 0.156. Parameter counts and computational costs for these configurations are needed to assess the claimed trade-off.

  4. Evidence for perceptual quality and fine-grained spatial control remains incomplete. There is no human listening study or dedicated evaluation of whether individual sound events are assigned to their requested directions. The authors acknowledge that semantic and spatial feedback are evaluated separately, which can miss incorrect event–direction associations. Furthermore, VT2FOA FAD is worse than all listed baselines, and the aggregate ablations do not reveal which tasks benefit or regress.

Questions for authors

See weakness

Comments on the paper

1 comment
Area_Chair_Z78sOpenReview ↗

Reminder: Discussion Phase Ends November 18 (11:59 PM AoE)

Hi reviewers, a quick reminder that the discussion phase ends on November 18 at 11:59 PM AoE. Please take some time to read the author responses, engage in further discussions if you have any remaining questions or concerns, and update your reviews and scores as appropriate. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.