- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper focuses on joint understanding, generation, and editing for binaural speech. It introduces two main components: 1) SpiderTok, a continuous tokenizer trained with interleaved-channel encoding, spatial losses, and staged semantic alignment to pack acoustic, semantic, and spatial info into shared representations; and 2) SpiderAudio, which connects SpiderTok to an LLM with a diffusion audio head. The authors also build a spatially rendered dataset (SpiderData) and use a "Thinking-Before-Generation" step (generating intermediate spatial descriptions) for audio editing. Tested on static, moving, and orbiting sources, the system outperforms binaural baselines on spatial reconstruction and understanding, while handling spatial TTS, editing, and standard speech tasks well.
Critical strengths
-
Unified modeling for binaural speech tasks. The paper puts spatial understanding, spatial TTS, and spatial editing into a single framework covering static, moving, and orbiting sources. Unlike standard monaural speech models, the shared representation here has to both read and render spatial states, making this a solid system-level extension.
-
Clear gains in spatial reconstruction and understanding. SpiderTok beats existing binaural baselines on all four spatial metrics, such as raising angular cosine similarity from 0.742 to 0.913. SpiderAudio also improves moving-source direction accuracy from 58.5% to 70.5% over a baseline fine-tuned on the same data. These results support that the proposed representation actually keeps spatial cues intact for downstream tasks.
Critical weaknesses
-
Incremental novelty over Ming-UniAudio. The system mostly follows Ming-UniAudio's setup (continuous representations, semantic distillation, staged training, LLM generation). Adapting these to spatial audio is useful, but the paper doesn't isolate the benefits of its architectural designs—like interleaved-channel encoding—against simpler stereo adaptations. Without these controls, it reads mainly like an engineering extension of an existing framework to binaural data.
-
Weak baselines and incomplete evaluation for generation/editing. Tables 3 and 4 compare against non-spatial baselines, which makes it hard to judge how well the spatial generation actually works. A standard cascaded pipeline (mono TTS/editing + HRTF rendering) is completely missing, despite being the most direct competitor. Furthermore, for spatial editing, "success rate" is measured purely by spatial metrics, it is unclear whether the spatial success rates in Table 4 also require content and speaker preservation.
Questions for authors
-
Interleaved-channel ablation:The overall pipeline is quite similar to Ming-UniAudio. Table 5 shows the effect of different loss functions, but what about the interleaved-channel design itself? Have you compared it with a simpler stereo adaptation (e.g., channel concatenation) under the same training conditions?
-
Mono + HRTF baseline: Since SpiderData is constructed by applying HRTF rendering to monaural speech, a natural baseline would be a strong monaural TTS or editing model followed by HRTF rendering. Have you compared SpiderAudio with such a pipeline?
-
Attribute preservation in editing: Section 5.2 states that spatial editing should preserve content and speaker identity, while Appendix D.3.3 explicitly lists ANG Cos and DIS Cos thresholds for success. Do the success rates in Table 4 also require content and speaker identity to be preserved? Similarly, is spatial preservation checked for regular content editing?
3 comments
Response to Reviewer duyf (2/2)
W2 & Q2: A standard cascaded pipeline (mono TTS/editing + HRTF rendering) is completely missing, despite being the most direct competitor.
Answer: Thanks for your great suggestion. We would like to clarify that: (1) In our work, spatial TTS mainly consists of two categories: zero-shot spatial TTS and instruct spatial TTS, as illustrated by the two examples in Table 9 of Appendix B.3 of the manuscript. For zero-shot spatial TTS, the task is defined as follows: given a reference spatial binaural speech without any description, the model must imitate the speaker's spatial position, motion state, and timbre to generate spatial binaural speech for a target text. Therefore, for the mono TTS plus HRTF rendering approach, the speaker's spatial position and motion state in the reference binaural speech must be determined in advance. However, no existing model can currently localize the reference binaural speech accurately (not spatial sound effects). Consequently, mono TTS plus HRTF rendering cannot accomplish this task. For instruct spatial TTS, the task is defined as follows: given a target text along with a natural-language-style speaking style description and a spatial state description, the model must generate spatial binaural speech that meets these specifications. This task can indeed be implemented using mono TTS plus the signal processing method. We have added three mono instruct TTS plus spatial binaural speech rendering baselines, with results shown in Table II. The results show that after rendering into spatial binaural speech, all three single-channel instruct TTS models exhibit some degradation in style consistency. Moreover, SpiderAudio achieves relatively good performance with less training data. In addition, since ANG Cos and DIS Cos themselves take the spatial state rendered from spatial binaural speech as ground truth, both metrics are 1 for the binaural speech obtained by single-channel instruct TTS plus spatial binaural speech rendering. However, this cascaded system is more complex, and end-to-end SpiderAudio has already achieved very high spatial accuracy, with ANG Cos and DIS Cos equal to 0.845 and 0.977, respectively.
(2) Regarding spatial editing tasks, as with zero-shot spatial TTS, accurate localization of the reference binaural speech is difficult to obtain. Therefore, the mono TTS plus HRTF rendering approach cannot accomplish the spatial editing task.
Table II: Instruct spatial TTS performance comparison. “*” denotes the original single-channel TTS without the capability for spatial state rendering. “#” means the approach of single-channel instruct TTS plus the spatial binaural speech rendering method. “DSD” means descriptive-style directive metric to measure the ability of the generative model to understand and follow unstructured natural-language-style descriptions.
| System | DSD$\uparrow$ | ANG Cos$\uparrow$ | DIS Cos$\uparrow$ |
|---|---|---|---|
| Uniaudio2.0* | 84.2 | - | - |
| Mimo-Audio* | 91.6 | - | - |
| FireRedAudio* | 97.6 | - | - |
| Uniaudio2.0+Rendering# | 80.4 | 1.000 | 1.000 |
| Mimo-Audio+Rendering# | 89.3 | 1.000 | 1.000 |
| FireRedAudio+Rendering# | 95.0 | 1.000 | 1.000 |
| SpiderAudio | 93.3 | 0.845 | 0.977 |
W3 & Q3: It is unclear whether the spatial success rates in Table 4 also require content and speaker preservation.
Answer: Thanks for your good question. We would like to clarify that all editing tasks adopt a strict "joint success rate": an edit is considered successful only when the target specified by the explicit editing instruction is accomplished and the other dimensions without editing instructions are not severely altered. Therefore, in the spatial editing task, the content and the speaker's timbre or prosody information are not allowed to be altered. We will add this clarification at the relevant location in the manuscript.
References
[1] C. Yan, C. Jin, D. Huang, et al. Ming-uniaudio: Speech llm for joint understanding, generation and editing with unified representation. arXiv preprint arXiv:2511.05516, 2025.
[2] Z. Zheng, P. Peng, Z. Ma, et al. Bat: Learning to reason about spatial sounds with large language models, ICML 2024.
[3] S. Biswas, M. Khan, B. Islam. Owl: Geometry-aware spatial reasoning for audio large language models, ICLR 2026.
[4] K. Wilkinghoff, Z. Tan. Dspast: Disentangled representations for spatial audio reasoning with large language models, ICASSP 2026.
Thanks for the response and the extra experiments.
Table I shows that interleaved-channel encoding beats simple concatenation on reconstruction, which answers Q1. Still, the overall novelty over Ming-UniAudio feels limited. The stereo decoder and extra spatial losses look mostly like an engineering adaptation to binaural data.
For W2/Q2, Table II actually shows FireRedAudio + Rendering beating SpiderAudio on every metric (DSD 95.0 vs. 93.3; spatial 1.000 vs. 0.845/0.977), which weakens the case for the end-to-end setup. Also, arguing that mono + HRTF cannot do zero-shot TTS or editing because no model can localize reference speech does not quite hold up. Since SpiderAudio itself does spatial localization, pairing its understanding module (or a standard DOA estimator) with mono TTS/editing and HRTF rendering is a straightforward baseline.
On Q3, "not severely altered" is still vague without concrete criteria (e.g., WER/speaker similarity thresholds or human/LLM checks) for Table 4.
Overall, my main concerns remain, so I will keep my original score.
Response to Reviewer duyf (1/2)
First, thank you very much for the time you have devoted to reviewing our work. We believe your constructive comments have been of tremendous value in improving the rigor and quality of our paper. Below, I respond to your comments one by one. Please feel free to ask us any further questions at any time.
Answer: Thanks for your helpful comment. We would like to clarify that: (1) In terms of overall design, SpiderAudio does draw on the design philosophy of Ming-UniAudio [1], a current state-of-the-art unified understanding and generation model. However, Ming-UniAudio mainly targets monaural speech. To enable the unified representation to better model spatial information in binaural speech, we incorporate a substantial number of original architectural designs, such as interleaved-channel encoding, an independent HiFiGAN-based StereoDecoder, and a "Thinking-before-generation" speech editing paradigm, none of which are present in Ming-UniAudio.
(2) More importantly, as explained in Section 3.1.1 and Appendix C.2, designing a comprehensive spatial-information supervision loss consisting of seven different losses for the spatial-information-embedded unified representations is highly challenging and vital, and it ensures that SpiderAudio no longer needs to explicitly rely on additional spatial representation inputs as conventional binaural speech understanding or generation models [2, 3, 4] do.
(3) As you suggested, we have added an ablation study comparing the reconstruction performance of SpiderTok under two schemes: the proposed interleaved-channel encoding and a simple left-right channel concatenation. As shown in Table I, the evaluation demonstrates the advantage of interleaved-channel encoding in modeling interaural information in binaural speech, with spatial state evaluation metrics significantly outperforming those of the simple concatenation method.
Table I: Reconstruction performance comparison of SpiderTok with the proposed interleaved-channel encoding method and the simply concatenating the left and right channels method.