acceptodds
Under review as a conference paper at ICLR 2027

SpiderAudio: Unified Binaural Representation for Joint Spatial Speech Understanding, Generation and Editing

Abstract

Recent advances in unified continuous representations for joint monaural speech understanding, generation, and even editing have shown promising results. Extending these representations to spatial speech tasks requires not only modeling the semantic and acoustic information, but also preserving spatial state information. Existing spatial representations, however, are designed for a single capability: some take hand-crafted interaural features as input and do not support waveform synthesis, while others optimize reconstruction fidelity rather than compatibility with a language model. To this end, we propose SpiderTok, a single all-round speech tokenizer that integrates semantic, acoustic, and spatial features in binaural speech for unified spatial understanding, generation and editing. SpiderTok simultaneously encodes 16 kHz binaural speech into a compact 50 Hz unified continuous representation. Its multi-stage training mainly includes acoustic and spatial reconstruction, semantic distillation, and joint refinement of unified representations. Building on this tokenizer, we further develop SpiderAudio, an all-round large speech language model that supports joint binaural speech understanding, generation, and editing within a single framework. In addition, we propose a “ Thinking-Before-Generation" strategy to guide binaural spatial speech editing. Experiments show that SpiderTok reconstructs binaural speech with high fidelity while outperforming binaural baselines on all four spatial metrics, with up to 23% relative gain in angular accuracy, and that the unified representation lets SpiderAudio surpass spatial baselines on understanding across static, moving, and orbiting sources, while also supporting instruction-guided spatial editing.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Binaural Spatial Speech, Joint Speech Understanding and Generation, Speech Editing, Unified Representation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_duyfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper focuses on joint understanding, generation, and editing for binaural speech. It introduces two main components: 1) SpiderTok, a continuous tokenizer trained with interleaved-channel encoding, spatial losses, and staged semantic alignment to pack acoustic, semantic, and spatial info into shared representations; and 2) SpiderAudio, which connects SpiderTok to an LLM with a diffusion audio head. The authors also build a spatially rendered dataset (SpiderData) and use a "Thinking-Before-Generation" step (generating intermediate spatial descriptions) for audio editing. Tested on static, moving, and orbiting sources, the system outperforms binaural baselines on spatial reconstruction and understanding, while handling spatial TTS, editing, and standard speech tasks well.

Critical strengths
  1. Unified modeling for binaural speech tasks. The paper puts spatial understanding, spatial TTS, and spatial editing into a single framework covering static, moving, and orbiting sources. Unlike standard monaural speech models, the shared representation here has to both read and render spatial states, making this a solid system-level extension.

  2. Clear gains in spatial reconstruction and understanding. SpiderTok beats existing binaural baselines on all four spatial metrics, such as raising angular cosine similarity from 0.742 to 0.913. SpiderAudio also improves moving-source direction accuracy from 58.5% to 70.5% over a baseline fine-tuned on the same data. These results support that the proposed representation actually keeps spatial cues intact for downstream tasks.

Critical weaknesses
  1. Incremental novelty over Ming-UniAudio. The system mostly follows Ming-UniAudio's setup (continuous representations, semantic distillation, staged training, LLM generation). Adapting these to spatial audio is useful, but the paper doesn't isolate the benefits of its architectural designs—like interleaved-channel encoding—against simpler stereo adaptations. Without these controls, it reads mainly like an engineering extension of an existing framework to binaural data.

  2. Weak baselines and incomplete evaluation for generation/editing. Tables 3 and 4 compare against non-spatial baselines, which makes it hard to judge how well the spatial generation actually works. A standard cascaded pipeline (mono TTS/editing + HRTF rendering) is completely missing, despite being the most direct competitor. Furthermore, for spatial editing, "success rate" is measured purely by spatial metrics, it is unclear whether the spatial success rates in Table 4 also require content and speaker preservation.

Questions for authors
  1. Interleaved-channel ablation:The overall pipeline is quite similar to Ming-UniAudio. Table 5 shows the effect of different loss functions, but what about the interleaved-channel design itself? Have you compared it with a simpler stereo adaptation (e.g., channel concatenation) under the same training conditions?

  2. Mono + HRTF baseline: Since SpiderData is constructed by applying HRTF rendering to monaural speech, a natural baseline would be a strong monaural TTS or editing model followed by HRTF rendering. Have you compared SpiderAudio with such a pipeline?

  3. Attribute preservation in editing: Section 5.2 states that spatial editing should preserve content and speaker identity, while Appendix D.3.3 explicitly lists ANG Cos and DIS Cos thresholds for success. Do the success rates in Table 4 also require content and speaker identity to be preserved? Similarly, is spatial preservation checked for regular content editing?

3 comments
AuthorsOpenReview ↗

Response to Reviewer duyf (1/2)

First, thank you very much for the time you have devoted to reviewing our work. We believe your constructive comments have been of tremendous value in improving the rigor and quality of our paper. Below, I respond to your comments one by one. Please feel free to ask us any further questions at any time.

W1 & Q1: The main contributions or innovations compared with Ming-UniAudio and the benefits of its architectural designs (e.g., interleaved-channel encoding against simpler stereo adaptations).

Answer: Thanks for your helpful comment. We would like to clarify that: (1) In terms of overall design, SpiderAudio does draw on the design philosophy of Ming-UniAudio [1], a current state-of-the-art unified understanding and generation model. However, Ming-UniAudio mainly targets monaural speech. To enable the unified representation to better model spatial information in binaural speech, we incorporate a substantial number of original architectural designs, such as interleaved-channel encoding, an independent HiFiGAN-based StereoDecoder, and a "Thinking-before-generation" speech editing paradigm, none of which are present in Ming-UniAudio.

(2) More importantly, as explained in Section 3.1.1 and Appendix C.2, designing a comprehensive spatial-information supervision loss consisting of seven different losses for the spatial-information-embedded unified representations is highly challenging and vital, and it ensures that SpiderAudio no longer needs to explicitly rely on additional spatial representation inputs as conventional binaural speech understanding or generation models [2, 3, 4] do.

(3) As you suggested, we have added an ablation study comparing the reconstruction performance of SpiderTok under two schemes: the proposed interleaved-channel encoding and a simple left-right channel concatenation. As shown in Table I, the evaluation demonstrates the advantage of interleaved-channel encoding in modeling interaural information in binaural speech, with spatial state evaluation metrics significantly outperforming those of the simple concatenation method.

Table I: Reconstruction performance comparison of SpiderTok with the proposed interleaved-channel encoding method and the simply concatenating the left and right channels method.

SystemPESQ$\uparrow$STOI$\uparrow$UTMOS$\uparrow$ANG Cos$\uparrow$DIS Cos$\uparrow$IPD Corr$\uparrow$ILD Corr$\uparrow$
Interleaved-channel encoding3.2800.9703.6820.9130.9940.9750.893
Simply concatenating3.1240.9653.5560.8720.9460.9350.878
AuthorsOpenReview ↗

Response to Reviewer duyf (2/2)

W2 & Q2: A standard cascaded pipeline (mono TTS/editing + HRTF rendering) is completely missing, despite being the most direct competitor.

Answer: Thanks for your great suggestion. We would like to clarify that: (1) In our work, spatial TTS mainly consists of two categories: zero-shot spatial TTS and instruct spatial TTS, as illustrated by the two examples in Table 9 of Appendix B.3 of the manuscript. For zero-shot spatial TTS, the task is defined as follows: given a reference spatial binaural speech without any description, the model must imitate the speaker's spatial position, motion state, and timbre to generate spatial binaural speech for a target text. Therefore, for the mono TTS plus HRTF rendering approach, the speaker's spatial position and motion state in the reference binaural speech must be determined in advance. However, no existing model can currently localize the reference binaural speech accurately (not spatial sound effects). Consequently, mono TTS plus HRTF rendering cannot accomplish this task. For instruct spatial TTS, the task is defined as follows: given a target text along with a natural-language-style speaking style description and a spatial state description, the model must generate spatial binaural speech that meets these specifications. This task can indeed be implemented using mono TTS plus the signal processing method. We have added three mono instruct TTS plus spatial binaural speech rendering baselines, with results shown in Table II. The results show that after rendering into spatial binaural speech, all three single-channel instruct TTS models exhibit some degradation in style consistency. Moreover, SpiderAudio achieves relatively good performance with less training data. In addition, since ANG Cos and DIS Cos themselves take the spatial state rendered from spatial binaural speech as ground truth, both metrics are 1 for the binaural speech obtained by single-channel instruct TTS plus spatial binaural speech rendering. However, this cascaded system is more complex, and end-to-end SpiderAudio has already achieved very high spatial accuracy, with ANG Cos and DIS Cos equal to 0.845 and 0.977, respectively.

(2) Regarding spatial editing tasks, as with zero-shot spatial TTS, accurate localization of the reference binaural speech is difficult to obtain. Therefore, the mono TTS plus HRTF rendering approach cannot accomplish the spatial editing task.

Table II: Instruct spatial TTS performance comparison. “*” denotes the original single-channel TTS without the capability for spatial state rendering. “#” means the approach of single-channel instruct TTS plus the spatial binaural speech rendering method. “DSD” means descriptive-style directive metric to measure the ability of the generative model to understand and follow unstructured natural-language-style descriptions.

SystemDSD$\uparrow$ANG Cos$\uparrow$DIS Cos$\uparrow$
Uniaudio2.0*84.2--
Mimo-Audio*91.6--
FireRedAudio*97.6--
Uniaudio2.0+Rendering#80.41.0001.000
Mimo-Audio+Rendering#89.31.0001.000
FireRedAudio+Rendering#95.01.0001.000
SpiderAudio93.30.8450.977

W3 & Q3: It is unclear whether the spatial success rates in Table 4 also require content and speaker preservation.

Answer: Thanks for your good question. We would like to clarify that all editing tasks adopt a strict "joint success rate": an edit is considered successful only when the target specified by the explicit editing instruction is accomplished and the other dimensions without editing instructions are not severely altered. Therefore, in the spatial editing task, the content and the speaker's timbre or prosody information are not allowed to be altered. We will add this clarification at the relevant location in the manuscript.

References

[1] C. Yan, C. Jin, D. Huang, et al. Ming-uniaudio: Speech llm for joint understanding, generation and editing with unified representation. arXiv preprint arXiv:2511.05516, 2025.

[2] Z. Zheng, P. Peng, Z. Ma, et al. Bat: Learning to reason about spatial sounds with large language models, ICML 2024.

[3] S. Biswas, M. Khan, B. Islam. Owl: Geometry-aware spatial reasoning for audio large language models, ICLR 2026.

[4] K. Wilkinghoff, Z. Tan. Dspast: Disentangled representations for spatial audio reasoning with large language models, ICASSP 2026.

Thanks for the response and the extra experiments.

Table I shows that interleaved-channel encoding beats simple concatenation on reconstruction, which answers Q1. Still, the overall novelty over Ming-UniAudio feels limited. The stereo decoder and extra spatial losses look mostly like an engineering adaptation to binaural data.

For W2/Q2, Table II actually shows FireRedAudio + Rendering beating SpiderAudio on every metric (DSD 95.0 vs. 93.3; spatial 1.000 vs. 0.845/0.977), which weakens the case for the end-to-end setup. Also, arguing that mono + HRTF cannot do zero-shot TTS or editing because no model can localize reference speech does not quite hold up. Since SpiderAudio itself does spatial localization, pairing its understanding module (or a standard DOA estimator) with mono TTS/editing and HRTF rendering is a straightforward baseline.

On Q3, "not severely altered" is still vague without concrete criteria (e.g., WER/speaker similarity thresholds or human/LLM checks) for Table 4.

Overall, my main concerns remain, so I will keep my original score.

Reviewer_Pm3iOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

SpiderAudio is a unified model for binaural speech understanding, generation, and editing. Its core component, SpiderTok, encodes speech content, acoustic characteristics, and spatial cues into a shared continuous representation. The system combines this tokenizer with a language model and diffusion-based audio head, and uses a “Thinking-Before-Generation” strategy to describe the spatial state and intended edit before producing audio. Trained on synthetically spatialized speech, it reports improved binaural reconstruction and spatial understanding, while supporting spatial speech synthesis and instruction-guided editing across static, moving, and orbiting sources.

Critical strengths
  1. A unified framework covering several useful spatial speech tasks.
  2. Strong reported reconstruction and spatial understanding results, including comparisons with retrained binaural baselines.
  3. Evaluation includes static, moving, and orbiting sources.
  4. Loss ablations demonstrate the importance of interaural supervision.
Critical weaknesses
  1. Evaluation uses synthetic rendering with the same KEMAR-based pipeline, without demonstrating robustness to unseen HRTFs, real recordings, or reverberant environments.
  2. Distance is simulated through amplitude scaling, but spatial evaluation normalizes loudness. For static sources, this can remove the very cue that encodes distance.
  3. ANG Cos and DIS Cos measure embedding similarity rather than physical angular or distance error. Their thresholds lack perceptual validation.
  4. There is no comparison against a strong monaural generator/editor followed by spatial rendering, leaving the benefit of end-to-end unification unclear.
  5. The paper does not isolate the benefit of a unified representation against separate semantic and spatial representations or independently trained task models.
  6. Author listening checks do not replace an independent study of localization, motion continuity, and preservation of unedited attributes.
Questions for authors
  1. How can DIS Cos validate absolute distance when loudness normalization removes the simulated distance cue?
  2. How does performance change on unseen HRTFs and real binaural recordings?
  3. Does SpiderAudio outperform a monaural generation/editing pipeline followed by HRTF rendering?
  4. What demonstrates that a unified representation improves all three tasks compared with separate representations?
  5. Can you report time-resolved localization errors and independent listening evaluations?
6 comments
AuthorsOpenReview ↗

Response to Reviewer Pm3i (1/5)

First, thank you very much for the time you have devoted to reviewing our work and for your meaningful comments. We provide detailed responses to each suggestion and question below. Please feel free to ask us any further questions at any time.

W1 & Q2: Without demonstrating robustness to unseen HRTFs, real recordings, or reverberant environments.

Answer: Thanks for your very meaningful question—this is also a limitation of our work, as described in Appendix F. We would like to clarify: (1) Simulating spatial audio data that satisfies the perception of most listeners based on a generic HRTF is common practice in this field. For example, several representative studies on spatial audio and audio-visual understanding, including BAT [1], OWL [2], and JAEGER [3], all use simulated data for training and evaluation with the CIPIC-based HRTF function. (2) To better demonstrate the robustness of our method on real recorded binaural speech, we have added evaluation results on the Dev subset of CHiME-6 [4], as shown in Table I. It can be seen that although SpiderAudio is trained only on simulated binaural speech data, it exhibits great robustness to real recordings, since KEMAR, as a general HRTF function, can match the head anatomy of most listeners. (3) Our work mainly considers open-space scenarios, so we use the KEMAR-based HRTF function for simulation without considering reverberant effects. The perceptual effects of our simulated spatial binaural speech can be experienced on the anonymous demo page https://conf123456789.github.io/iclr2027/.

Furthermore, we commit to incorporating more acoustic scene data in future work, including binaural speech data simulated with multiple HRTFs as well as real recorded binaural speech data, to explore self-adaptive spatial binaural speech understanding and generation models.

Table I: Spatial ASR performance comparison on the dev subset in CHiME-6. “*” denotes the Kaldi-based monaural ASR method is only evaluated on the left channel based as described in [4]. “**’’ means the original baseline BAT w/ DSpAST has been retrained on our rendered binaural speech data for a fairer comparison.

SystemKaldi*BAT w/ DSpAST+SFT**SpiderAudio
WER ↓\downarrow41.2112.157.62

W2 & Q1: Distance is simulated through amplitude scaling, but normalizing loudness can remove the very cue that encodes distance in the ANG Cos and DIS Cos calculation.

Answer: Thank you for pointing this out. We would like to clarify that, when evaluating spatial state similarity, we actually computed ANG Cos and DIS Cos following the procedure in [5], where spatial state embeddings are extracted using the Spatial-AST model proposed in [1], and no loudness normalization is performed during this process. However, in Appendix C.3.4, when describing the computation of these two metrics, we referred to [1], in which the spatial state embeddings extracted by Spatial-AST are divided into two cases: with and without loudness normalization. We mistakenly described loudness normalization as part of the actual operation and wrote it into the metric computation process in the manuscript. In fact, our actual evaluation process is consistent with the procedure in [5], avoiding interference from loudness normalization in distance prediction. We apologize for this careless mistake and we will correct this description error.

AuthorsOpenReview ↗

Response to Reviewer Pm3i (2/5)

W3 & W6 & Q5: ANG Cos and DIS Cos measure embedding similarity rather than physical angular or distance error; Lack perceptual validation.

Answer: We thank the reviewer and agree with the point: (1) ANG Cos and DIS Cos are similarities between Spatial-AST’s learned DOA/DIS tokens, not errors in degrees or meters. We used them because they are the protocol established by the previous study on binaural speech generation [5], allowing direct comparison with prior work.

(2) In addition, we measured objective time-resolved physical errors between the generated binaural speech and the HRTF-rendered reference, including time difference of arrival error (ΔTDOA) via the GCC-PHAT method [8], interaural time difference error (ΔITD), interaural level difference error (ΔILD), and distance error (ΔD), as shown in Table II. Specifically, for zero-shot spatial TTS and spatial state editing, generation or editing must be performed end-to-end from the input zero-shot reference binaural audio, so the errors are relatively larger; nevertheless, both satisfy common perceptual thresholds (|ΔTDOA| < 100 µs and |ΔILD| < 3 dB). For the other two settings, where generation or editing follows the spatial information described in the instruction, the errors are all very small. These time-resolved physical errors further demonstrate the high spatial accuracy of the binaural speech generated by SpiderAudio.

(3) We also conducted an independent listening evaluation on spatial binaural generation and editing tasks, comparing SpiderAudio's binaural speech with that from the monaural speech generator/editor plus HRTF rendering, as shown in Table III. We designed a real-user listening and rating questionnaire around four dimensions: spatial localization accuracy, spatial motion accuracy, content accuracy, and speaker timbre/prosodic accuracy. Each dimension is scored on a 0–3 scale, with an average score computed. For each task type, we designed 30 questions (10 each for static, moving, and orbiting states). A total of 52 non-author listeners completed the questionnaire. Since the monaural-plus-HRTF approach cannot accomplish zero-shot spatial TTS and spatial editing (see the answer to W4 & Q3), we consider only instruct spatial TTS and binaural speech rendering tasks. Results show that end-to-end SpiderAudio is no weaker than the cascaded approach on instruct TTS, and performs basically on par with direct HRTF rendering on binaural speech rendering. Overall, SpiderAudio avoids severe performance degradation on tasks the monaural-plus-HRTF approach can handle, while also accomplishing more tasks existing methods cannot—making it an all-capable, convenient unified model for spatial binaural speech understanding, generation, and editing.

Table II: Objective time-resolved physical errors of the spatial binaural speech generated by SpiderAudio for spatial TTS and spatial editing tasks. Note that the sign of the numbers in the table indicates the bias direction of the generated binaural speech relative to the HRTF-rendered one. For example, ΔILD > 0 means that the generated spatial speech is on average biased to the left (i.e., the generated left-ear speech is louder than the HRTF-rendered one), corresponding to the sound source being systematically “placed to the left”.

TaskΔTDOA (µs)ΔITD (µs)ΔILD (dB)ΔD (m)
Zero-shot spatial TTS−30.2 / −50.8 / −39.1−43.2 / +35.6 / −59.4+0.93 / +1.54 / −1.290.69 / 0.67 / 0.63
Instruct spatial TTS-3.2 / −6.8 / −4.1+13.8 / +20.6 / −9.0+0.19 / +0.54 / −0.290.22 / 0.19 / 0.18
Spatial state editing-69.8 / −76.1 / −44.3−56.8 / −60.6 / −52.7+1.70 / +1.66 / +1.480.62 / 0.53 / 0.57
Binaural speech rendering-3.2 / −6.8 / −4.1+13.8 / −20.6 / −9.0+0.19 / +0.54 / −0.290.22 / 0.19 / 0.18

Table III: Independent listening evaluations on instruct spatial TTS and HRTF rendering tasks to compare the binaural speech generated by SpiderAudio with that generated the method of monaural speech generator plus HRTF rendering or the binaural speech rendering.

SystemInstruct spatial TTSBinaural speech rendering
SpiderAudio2.042.73
FireRedAudio+ HRTF Rendering2.23-
Uniaudio 2.0+ HRTF Rendering1.73-
Mimo-Audio+ HRTF Rendering2.01-
Input monaural speech+HRTF Rendering-2.77
AuthorsOpenReview ↗

Response to Reviewer Pm3i (3/5)

W4 & Q3: There is no comparison against a strong monaural generator/editor followed by HRTF rendering.

Answer: Thanks for your very meaningful question. We would like to clarify that: (1) Like traditional monaural TTS, spatial TTS mainly falls into two categories: zero-shot spatial TTS and instruct spatial TTS, as illustrated by the two examples in Table 9 of the manuscript. For zero-shot spatial TTS, the task is defined as follows: given a reference spatial binaural speech without any description, the model must imitate the speaker's spatial position, motion state, and timbre to generate spatial binaural speech for a target text. For the single-channel TTS plus signal processing approach, the speaker's spatial position and motion state in the reference binaural speech must therefore be determined in advance. However, no existing model can currently localize binaural speech accurately (not spatial sound effects). Consequently, monaural TTS plus HRTF rendering cannot accomplish this task, whereas end-to-end SpiderAudio can easily achieve it. For instruct spatial TTS, the task is defined as follows: given a target text along with a natural-language-style speaking style description and a spatial state description, the model must generate spatial binaural speech that meets these specifications. This task can indeed be implemented using monaural TTS plus HRTF rendering. We have added three monaural instruct TTS plus HRTF rendering baselines, with results shown in Table IV. The results show that after rendering into spatial binaural speech, all three monaural instruct TTS models exhibit some degradation in style consistency. Moreover, SpiderAudio achieves relatively good performance with less training data. In addition, since ANG Cos and DIS Cos themselves take the spatial state rendered from spatial binaural speech as ground truth, both metrics are 1 for the binaural speech obtained by monaural TTS plus HRTF rendering. However, this cascaded system is more complex, and end-to-end SpiderAudio has already achieved very high spatial accuracy, with ANG Cos and DIS Cos equal to 0.845 and 0.977, respectively.

(2) Regarding spatial editing tasks, as with zero-shot spatial TTS, accurate localization of the reference binaural speech is difficult to obtain. Therefore, the mono editor plus HRTF rendering approach cannot accomplish the spatial editing task.

Table IV: Instruct spatial TTS performance comparison. “*” denotes the original monaural TTS without the capability for spatial state rendering. “#” means the approach of monaural TTS plus HRTF rendering. “DSD” means descriptive-style directive metric to measure the ability of the generative model to understand and follow unstructured natural-language-style descriptions.

SystemDSD$\uparrow$ANG Cos$\uparrow$DIS Cos$\uparrow$
Uniaudio2.0*84.2--
Mimo-Audio*91.6--
FireRedAudio*97.6--
Uniaudio2.0+Rendering#80.41.0001.000
Mimo-Audio+Rendering#89.31.0001.000
FireRedAudio+Rendering#95.01.0001.000
SpiderAudio93.30.8450.977
Reviewer_JpwPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a codec model that can deal with binaural audio, which can be uniformly applied to audio understanding, generation and editing. It subsequently build the audio language model using the codec, which is able to process the spatial audio properly, including spatial understanding, generation and even editing.

Critical strengths

(1) although the spatial audio is well studied in previous literature, a holistic work that covers codec, understand and generation is a good milestone. (2) The paper also provides sensible experimental strength in its target application; it can even deal with moving and orbiting audio sources.

Critical weaknesses

(1) Everything is synthetic, which lacks realism. Previous chime series resources may include some realistic options for this spatial audio, and I recommend that the authors consider adopting them in some approaches, at least with some adaptation. (2) A important baseline is that, instead of using LLM to deal with spaital TTS generation, can we also use the signal processing method plus single-channel TTS to do so; if not, what is the reason? (3) even with more info (channel info), the model shows a clear gap for its understanding/ASR performance (table 3); it's better to explain or have some study on this.

Questions for authors

(1) as the model is trained for both understanding and generation, it's not clear if this joint training bring benefits for each direction, or would have negative impact on each direction

I would like to increase my score as long as the authors can address my questions properly :)

3 comments
AuthorsOpenReview ↗

Response to Reviewer JpwP (1/3)

We would like to thank you very much for your time and valuable comments, which we believe are of great help in improving the rigor and quality of our work. In response to your questions and suggestions, we provide point-by-point answers below. Please feel free to ask us any further questions at any time.

W1: Everything is synthetic, which lacks realism. Previous Chime series resources may include some realistic options for this spatial audio.

Answer: Thank you for your excellent suggestion. We have identified the real binaural recorded speech data in CHiME-6 (the same audio data as in CHiME-5) and use the Dev subset in CHiME-6 as the real evaluation data for assessing SpiderAudio. Since this dataset contains only binaural speech and the corresponding transcriptions, we compare only on ASR performance with the Kaldi-based method mentioned in [1] and the task-adapted baseline in our manuscript, as shown in the Table I. The evaluation results show that the proposed SpiderAudio significantly outperforms the other baselines on the spatial ASR task, demonstrating its robustness on real spatial binaural recordings.

Table I: Spatial ASR performance comparison on the dev subset in CHiME-6. “*” denotes the Kaldi-based monaural ASR method is only evaluated on the left channel based as described in [1]. “**’’ means the original baseline BAT w/ DSpAST has been retrained on our rendered binaural speech data for a fairer comparison.

SystemKaldi*BAT w/ DSpAST+SFT**SpiderAudio
WER ↓\downarrow41.2112.157.62

W2: Can we also use the signal processing method plus single-channel TTS to do so; if not, what is the reason?

Answer: Thank you for your great question. In our work, spatial TTS mainly consists of two categories: zero-shot spatial TTS and instruct spatial TTS, as illustrated by the two examples listed in Table 9 of Appendix B.3 in the manuscript. (1) For zero-shot spatial TTS, the task is defined as follows: given a reference spatial binaural speech without any description, the model is required to imitate the speaker's spatial position, motion state, and timbre to generate spatial binaural speech for a target text. Therefore, for the approach of single-channel TTS plus the signal processing method, the speaker's spatial position and motion state in the reference spatial binaural speech must be determined in advance. However, no existing model is currently capable of accurate localization of binaural speech (not spatial sound effects). Consequently, the approach of single-channel TTS plus the signal processing method cannot accomplish this task. (2) For instruct spatial TTS, the task is defined as follows: given a target text, along with a natural-language-style speaking style description and a spatial state description, the model is required to generate spatial binaural speech that meets these specifications. Therefore, this task can indeed be implemented using the approach of single-channel TTS plus the signal processing method. We have added three single-channel instruct TTS plus the spatial binaural speech rendering method as comparison baselines, as shown in the results in Table II. The results show that after rendering into spatial binaural speech, all three single-channel instruct TTS models exhibit a certain degree of degradation in style consistency. Moreover, the SpiderAudio model achieves relatively good performance with less training data. In addition, since ANG Cos and DIS Cos themselves take the spatial state rendered from spatial binaural speech as the ground-truth one, the two metrics for the binaural speech obtained by single-channel instruct TTS plus spatial binaural speech rendering are both 1. However, this cascaded system is more complex, and the end-to-end SpiderAudio has already achieved a very high level of accuracy in spatial state with the ANG Cos and DIS Cos equal to 0.845 and 0.977, respectively.

Table II: Instruct spatial TTS performance comparison. “*” denotes the original single-channel TTS without the capability for spatial state rendering. “#” means the approach of single-channel instruct TTS plus the spatial binaural speech rendering method. “DSD” means descriptive-style directive metric to measure the ability of the generative model to understand and follow unstructured natural-language-style descriptions.

SystemDSD$\uparrow$ANG Cos$\uparrow$DIS Cos$\uparrow$
Uniaudio2.0*84.2--
Mimo-Audio*91.6--
FireRedAudio*97.6--
Uniaudio2.0+Rendering#80.41.0001.000
Mimo-Audio+Rendering#89.31.0001.000
FireRedAudio+Rendering#95.01.0001.000
SpiderAudio93.30.8450.977
AuthorsOpenReview ↗

Response to Reviewer JpwP (2/3)

W3: Even with more info (channel info), the model shows a clear gap for its understanding/ASR performance (table 2).

Answer: Thank you for your good question. I would like to clarify: currently, there are not many open-source unified speech LLMs for both generation and understanding, and most of them are released by industry, including Mimo-Audio [2], Ming-UniAudio [3], FireRedAudio [4], and Qwen2.5-Omni [5] mentioned in the manuscript; in academia, the only open-source works are UniAudio 2.0 [6] and Audex [7] before the submission deadline. Industry models, owing to larger-scale and higher-quality training data, including much closed-source data, are inherently not directly comparable with academic work that strictly uses open-source data. Therefore, in terms of ASR capability, SpiderAudio should be fairly compared with UniAudio 2.0 and Audex, which likewise use only open-source data. In addition, to further improve the fairness of the comparison, we retrained the BAT w/ DSpAST model [8] on the same training data as SpiderAudio, and the evaluation results in Table 2 of the manuscript (also Table III here) demonstrate that the ASR capability of the SpiderAudio model lies between that of academic works and industry models. More importantly, we would like to emphasize that, while the ASR capability of SpiderAudio is not weak, it possesses spatial localization capability that current industry models lack.

Table III: Spatial ASR performance comparison. “*” and “#” denote the open-source works from academia and industry, respectively. “**” means the original baseline BAT w/ DSpAST has been retrained on our rendered binaural speech data for a fairer comparison.

SystemASR$\downarrow$
Non-spatial Unified Understanding and Generation LALMs
Audex*3.60 / 3.17 / 3.46
Uniaudio 2.0*3.79 / 3.78 / 4.01
Mimo-Audio#2.61 / 2.19 / 2.72
Ming-UniAudio#1.93 / 1.93 / 1.88
FireRedAudio#1.23 / 0.96 / 1.26
Qwen2.5-Omni#2.04 / 2.13 / 2.05
Binaural Spatial Understanding LALMs
BAT w/ DSpAST +SFT**6.34 / 6.85 / 6.46
SpiderAudio3.11 / 3.18 / 3.16
AuthorsOpenReview ↗

Response to Reviewer JpwP (3/3)

Q1: If this joint training brings benefits for each direction, or would have negative impact on each direction.

Answer: Thank you very much for this insightful suggestion. We provide the evaluation results of the variants trained separately on understanding (V1) and generation (V2) tasks for comparison, as shown in Table IV and Table V here. The experimental results show that, compared with training only on the speech understanding task, the performance of joint training is basically on par (see Table IV), while compared with training only on the speech generation task, joint training generally achieves better generation performance (see Table V). More importantly, joint training on speech understanding and generation tasks serves as a prerequisite for the speech editing task and can better improve the unified model's editing capability as illustrated in [3].

Table IV: Spatial understanding and ASR performance comparison between SpiderAudio and its variant only trained on the understanding data.

SystemDirection$\uparrow$Updown$\uparrow$Distance$\uparrow$Motion$\uparrow$ASR$\downarrow$
SpiderAudio96.5 / 70.5 / 84.092.5 / 65.0 / 93.567.5 / 60.0 / 67.0100.0 / 97.8 / 98.53.11 / 3.18 / 3.16
V196.9 / 70.7 / 83.691.9 / 65.1 / 93.267.8 / 59.9 / 66.899.9 / 98.3 / 98.73.09 / 3.15 / 3.15

Table V: Zero-shot and instruct spatial TTS performance comparison between SpiderAudio and its variant only trained on the generation data.

SystemWER$\downarrow$TimberSIM$\uparrow$ANG Cos$\uparrow$DIS Cos$\uparrow$DSD$\uparrow$ANG Cos$\uparrow$DIS Cos$\uparrow$
SpiderAudio3.47 / 3.57 / 3.520.468 / 0.451 / 0.4600.790 / 0.774 / 0.7890.974 / 0.973 / 0.97493.30.8450.977
V23.78 / 3.92 / 3.820.457 / 0.446 / 0.4490.725 / 0.713 / 0.7220.970 / 0.961 / 0.96491.20.7980.923

References

[1] S. Watanabe, M. Mandel, J. Barker, et al. CHiME-6 challenge: Tackling multispeaker speech recognition for unsegmented recordings. arXiv preprint arXiv:2004.09249, 2020.

[2] Mimo-audio Team. Mimo-audio: Audio language models are few-shot learners. arXiv preprint arXiv:2512.23808, 2025.

[3] C. Yan, C. Jin, D. Huang, et al. Ming-uniaudio: Speech llm for joint understanding, generation and editing with unified representation. arXiv preprint arXiv:2511.05516, 2025.

[4] F. Shen, F. Xie, J. Li, et al. FireRedAudio: A general-purpose audio language model with decoupled continuous representations for understanding and generation. arXiv preprint arXiv:2608.24168, 2026.

[5] J. Xu, Z. Guo, J. He, et al. Qwen2.5-omni technical report. arXiv preprint arXiv: 2503.20215, 2025.

[6] D. Yang, Y. Wang, D. Chong, et al. Uniaudio 2.0: A unified audio language model with text-aligned factorized audio tokenization. arXiv preprint arXiv:2602.04683, 2026.

[7] Z. Kong, S. Lee, J. Kim, et al. Unified Audio Intelligence Without Regressing on Text Intelligence. arXiv preprint arXiv:2607.05196, 2026.

[8] K. Wilkinghoff, Z. Tan. Dspast: Disentangled representations for spatial audio reasoning with large language models, ICASSP 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.