SpiderAudio: Unified Binaural Representation for Joint Spatial Speech Understanding, Generation and Editing

Recent advances in unified continuous representations for joint monaural speech understanding, generation, and even editing have shown promising results. Extending these representations to spatial speech tasks requires not only modeling the semantic and acoustic information, but also preserving spatial state information. Existing spatial representations, however, are designed for a single capability: some take hand-crafted interaural features as input and do not support waveform synthesis, while others optimize reconstruction fidelity rather than compatibility with a language model. To this end, we propose SpiderTok, a single all-round speech tokenizer that integrates semantic, acoustic, and spatial features in binaural speech for unified spatial understanding, generation and editing. SpiderTok simultaneously encodes 16 kHz binaural speech into a compact 50 Hz unified continuous representation. Its multi-stage training mainly includes acoustic and spatial reconstruction, semantic distillation, and joint refinement of unified representations. Building on this tokenizer, we further develop SpiderAudio, an all-round large speech language model that supports joint binaural speech understanding, generation, and editing within a single framework. In addition, we propose a “ Thinking-Before-Generation" strategy to guide binaural spatial speech editing. Experiments show that SpiderTok reconstructs binaural speech with high fidelity while outperforming binaural baselines on all four spatial metrics, with up to 23% relative gain in angular accuracy, and that the unified representation lets SpiderAudio surpass spatial baselines on understanding across static, moving, and orbiting sources, while also supporting instruction-guided spatial editing.