- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes StereoBind, which targets trajectory-conditioned joint stereo audio-video generation: the perceived position of a sound should shift as the sound source moves in the video. This task is often overlooked but worth studying. Built on LTX-2.3, StereoBind fuses visual context, absolute position, and relative motion information through Visual Motion Binding (VMB), a Spatial Track Encoder (STE), and Residual Track RoPE (RT-RoPE). The paper also introduces the StereoWorld-29K dataset and StereoWorldBench, giving the community a trainable and evaluable baseline.
Critical strengths
- This paper's motivation is strong. Dynamic Spatial Correspondence is indeed a problem that needs to be addressed, as it directly affects the user experience of audio-video generation results.
- VMB, STE, and RT-RoPE introduce source-aware visual context, absolute spatial conditioning, and relative motion information, respectively. Removing any one of these components degrades multiple spatial metrics, which demonstrates their contribution to performance. Moreover, these components are not arbitrarily stacked but are combined based on well-grounded motivations.
- The human evaluation results confirm the effectiveness of the proposed method and further enhance the credibility of the results. This is especially important for video generation, since automatic metrics cannot fully reflect the quality of generated results.
- The combination of audiovisual filtering, source localization and tracking, geometry-based spatialization, and synthetic scene generation offers a scalable approach to spatial supervision. The dataset and complementary evaluation metrics could be useful for subsequent research.
Critical weaknesses
Major weaknesses:
- The reference stereo audio in SWBench is constructed with the proposed VAS pipeline, and the training data are rendered with the same pipeline. As a result, ILD-W, SMR-Err, AST-Ang, and AST-Cal essentially measure how similar the outputs are to those of the VAS pipeline, rather than whether the spatial perception is correct. Outputs with reasonable spatial perception may thus be overly penalized simply because their rendering style differs. This may undermine the credibility of the evaluation for the key problem addressed in this paper.
- The necessity of the proposed mechanisms is questionable, since the main baselines lack the same trajectory conditioning and spatialization training. The component-wise ablations cannot replace a simple trajectory-conditioned fine-tuning baseline trained with the same data and budget. The paper also does not compare against an approach that uses the same generation backbone and motion control followed by post-hoc spatialization. It therefore remains unclear how much irreplaceable benefit the complex joint modeling provides.
Minor weaknesses:
- Typos and wording errors: "blcok" should be "block", "SOUDING" should be "SOUNDING", "Systhesis" should be "Synthesis", etc.
- The trajectory extraction is underspecified: Appendix B.3 describes smoothing but does not clarify what the representative 2D point denotes or how it is selected from AllTracker outputs. Its relationship to the source region used in SELD-Acc also needs clarification.
Questions for authors
- Could the authors evaluate StereoBind against references not produced by the VAS pipeline, such as real recorded stereo clips or audio rendered with a different spatializer? Does its advantage on ILD-W, SMR-Err, AST-Ang, and AST-Cal persist under such an evaluation?
- Could the authors compare StereoBind with a simple trajectory-conditioned fine-tuning baseline on the same LTX-2.3 backbone, trained with the same data and budget?
- Could the authors compare StereoBind with a cascaded baseline that uses the same backbone and motion control, followed by post-hoc spatialization with the known trajectory? If StereoBind does not clearly outperform it, what benefit does the joint modeling uniquely provide?
- What does the representative 2D point in the motion track denote, and how is it obtained from the AllTracker outputs? How does it relate to the bounding box used in SELD-Acc?
- Is there a commitment to open source?
If most of the above questions are answered, I'll consider raising my score.
3 comments
response to authors
Thank you to the author for the response. After carefully reviewing the additional information provided, most of my questions have been answered, so I’ve decided to keep my rating as is.
Dear Reviewer J1fc:
We sincerely thank the reviewer for the constructive feedback and thoughtful consideration of our rebuttal. We are pleased that our responses have addressed your concerns, and we greatly appreciate your suggestions, which have helped us strengthen the manuscript.
We would like to briefly highlight the key findings regarding your two main concerns: Baselines Trained on StereoWorld-29K and Benefit of Joint Modeling
For Baselines Trained on StereoWorld-29K.
To isolate architectural contributions from the effect of additional training data, we fine-tune baselines on StereoWorld-29K for 8,000 steps. We train a LoRA adapter for MiniMax-H3 Ref2VA using DiffSynth-Studio and LoRA adapters for LTX-2.3/2.5. Ovi is excluded due to the lack of available training code. As shown below, StereoBind consistently outperforms all fine-tuned baselines across the five spatial-audio metrics. These results suggest that its improvements cannot be explained solely by exposure to StereoWorld-29K, and instead provide further evidence for the effectiveness of the proposed architecture.
| Method | ILD-W ↓ | SELD-Acc ↑ | SMR-Err ↓ | AST-Ang ↑ | AST-Cal ↑ |
|---|---|---|---|---|---|
| MiniMax-H3 Ref2VA | 3.257 | 0.603 | 26.14 | 0.246 | 0.157 |
| LTX-2.3 | 3.214 | 0.576 | 34.85 | 0.242 | 0.145 |
| LTX-2.5 | 3.312 | 0.531 | 33.92 | 0.249 | 0.166 |
| StereoBind | 2.754 | 0.757 | 14.86 | 0.465 | 0.314 |
For Benefit of Joint Modeling:
We emphasize that these mechanisms are functionally distinct and operating on different aspects.
VMB and Joint VA Attention. The original Joint VA Attention in LTX-2.3 provides general information exchange between video and audio features, however, does not explicitly establish which visual entity should be associated with a track, which is addressed by VMB.
VMB and STE. The STE determines how the context is accessed and incorporated under the track. The shift and scale in STE modulate the audio queries for VMB retrieval, while the gate controls the strength of the update. Therefore, STE provides the track-adaptive control for VMB, while VMB establishes what visual-motion context should be transferred to audio.
VMB and RT-RoPE. RT-RoPE introduces track-dependent residual rotary phases into audio self-attention, encoding relative spatial displacement between audio tokens, but does not associate the track with a particular visual entity. VMB operates in the cross-modal feature space, associating the motion with the generated visual source and transferring the resulting context to audio.
Joint Benefits of VMB, STE, and RT-RoPE. The advantage of jointly incorporating these mechanisms lies in their roles in modeling dynamic spatial correspondence. VMB establishes which visual source the motion belongs to, STE provides where the source is located through absolute spatial conditioning, and RT-RoPE captures how its position evolves through relative motion modeling. More importantly, these mechanisms jointly constrain the audio generation process at different levels: VMB transfers source-aware visual context to audio, STE adaptively controls its retrieval and integration, while RT-RoPE explicitly models motion-dependent spatial relationships within audio self-attention. This coordinated design enables the audio stream to jointly capture visual source identity, absolute spatial position, and relative motion dynamics.
Additional Analysis of VMB Attention Map in Sec. 5.4.
we have added a detailed analysis of VMB in Section 5.4 of the revised main text, highlighted in red, to clarify how its progressive reference–target–audio pathway supports audiovisual spatial correspondence. Specifically, we visualize attention maps for all three stages—reference aggregation, target association, and audio readout. All maps are extracted at Transformer block 8 and denoising step 8 and averaged across attention heads.
Finally, We sincerely hope that the additional evidence and clarifications have further strengthened the case for our approach and may provide a basis for reconsidering the overall assessment of our work.
If there are any remaining concerns that may affect your assessment, we would be grateful for the opportunity to clarify them further.
Response To Reviewer J1fc:
Thanks for the suggestions. We will add relevant details to the final version.
W1: Concern on SWBench
W1.1: The Renderer Dependence of Spatial Metrics
Please refer to our response to Reviewer QoMj (W1.1) and Reviewer J1fc (W1.2) for details.
W1.2: Additional Renderer-Independent Evaluation
SELD-Acc is renderer-independent, we further report an additional renderer-independent metric, Direction-of-Arrival MAE (DOA-MAE), comparing the azimuth estimated by pretrained Spatial-AST with the target azimuth from the VAS-predicted 3D trajectory. As shown below, StereoBind achieves the best.
The details of the additional metric is highlighted in red at the end of Appendix H.2 in the revised manuscript.
W1.3: Cross-Renderer Evaluation
We test renderer sensitivity by replacing gpuRIR in VAS with HRTF/BRIR renderers and recomputing spatial metrics. StereoBind maintains consistent spatial performance across renderers.
W2: Concern on Necessity of Proposed Mechanism.
W2.1: Baselines trained on StereoWorld-29K.
To isolate architectural contributions from the effect of additional training data, we fine-tune baselines on StereoWorld-29K for 8,000 steps. We train a LoRA adapter for MiniMax-H3 Ref2VA using DiffSynth-Studio and LoRA adapters for LTX-2.3/2.5. Ovi is excluded due to the lack of available training code. As shown below, fine-tuning yields marginal improvements.
W2.2: Comparison with the LTX-2.3 + Motion Track IC-LoRA + VAS cascade
Please refer to our response to Reviewer J1fc (Q1) for details.
W2.3: Benefit of Joint Modeling
Please refer to our response to Reviewer wPGN (W1.3) for details.
W3: Tpyo Error
Thank you for pointing this out. We have corrected our manuscript. The revised are highlighted in red.
W4: Motion-Track Point Extraction and Its Relationship to SELD-Acc Bounding Boxes
First, Grounded-SAM identifies the source and provides its binary mask and bounding box. Then, representative 2D point is initialized at the center of the bounding box and propagated across frames by AllTracker. The coordinates are then resampled and smoothed using Gaussian filtering and robust third-order polynomial fitting. SELD-Acc uses the same bounding box and considers a prediction correct if its audio-location interval overlaps the box's horizontal interval.
Q1: Evaluating Generalization to Real-recorded Binaural Audio.
Given the limited time and resources during rebuttal, collecting a new real-world stereo dataset with professional recording equipment was impractical. We therefore leveraged MRSAudio dataset to develop a training and evaluation pipeline. Specifically, we extracted 5-second speech segments and used GPT-6.1 to generate LTX-2.3 text prompts from their speech content. We converted MRSAudio's 3D sound-source locations into 2D motion tracks. Each stereo recording was downmixed to mono and fed into LTX-2.3 with its prompt and motion track to generate a video. We then replaced the generated audio with the original stereo recording, obtaining 220 stereo audio-video pairs, with 200 for StereoBind fine-tuning and 20 for evaluation.
For evaluation, we downmixed each real-world stereo recording to mono as the audio input to both StereoBind and LTX-2.3, together with the corresponding prompt and motion track. Finally, we evaluated both methods against the original real-world stereo recordings. As shown below, both methods achieve comparable performance under this setting.
Q2: Baselines Trained on StereoWorld-29K.
Please refer to our response to Reviewer QoMj (W2.2) for details.
Q3: Benefit of Joint Modeling
Please refer to our response to Reviewer wPGN (W1.3) for details.
Q4: Motion-Track Point Extraction and Its Relationship to SELD-Acc Bounding Boxes
Please refer to our response to Reviewer J1fc (W4) for details.
Q5: Open Source.
Yes. We commit to releasing the inference and training code of StereoBind, StereoWorld-29K dataset and SWBench.