acceptodds
Under review as a conference paper at ICLR 2027

Hear the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generation

Abstract

Recent joint video-audio generation models have achieved strong semantic correspondence and temporal synchronization. However, applications such as AR/VR and interactive gaming further require stereo audio to provide an immersive sense, which remains largely overlooked. Effective stereo audio requires the perceived sound location to evolve consistently with the motion of its corresponding visual source. We refer to this property as Dynamic Spatial Correspondence and propose StereoBind, a framework that binds visual source motion to stereo sound generation. StereoBind uses motion tracks to coordinate visual motion and stereo audio through three complementary mechanisms. Visual Motion Binding establishes source-aware audiovisual correspondence, the Spatial Track Encoder captures absolute source positions, and Residual Track RoPE models relative motion. For supervision and evaluation, we construct StereoWorld-29K, a large-scale stereo audio-video dataset with paired motion tracks, and StereoWorldBench for measuring audiovisual spatial consistency. Experiments show that StereoBind substantially improves spatial alignment in stereo audio generation over existing models while preserving overall audiovisual quality.

Primary area:
generative models
Keywords:
Audio-Video Generation, Stereo Audio, Audio-Visual Alignment, Motion-Aware Generation, Multimodal Generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_J1fcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes StereoBind, which targets trajectory-conditioned joint stereo audio-video generation: the perceived position of a sound should shift as the sound source moves in the video. This task is often overlooked but worth studying. Built on LTX-2.3, StereoBind fuses visual context, absolute position, and relative motion information through Visual Motion Binding (VMB), a Spatial Track Encoder (STE), and Residual Track RoPE (RT-RoPE). The paper also introduces the StereoWorld-29K dataset and StereoWorldBench, giving the community a trainable and evaluable baseline.

Critical strengths
  1. This paper's motivation is strong. Dynamic Spatial Correspondence is indeed a problem that needs to be addressed, as it directly affects the user experience of audio-video generation results.
  2. VMB, STE, and RT-RoPE introduce source-aware visual context, absolute spatial conditioning, and relative motion information, respectively. Removing any one of these components degrades multiple spatial metrics, which demonstrates their contribution to performance. Moreover, these components are not arbitrarily stacked but are combined based on well-grounded motivations.
  3. The human evaluation results confirm the effectiveness of the proposed method and further enhance the credibility of the results. This is especially important for video generation, since automatic metrics cannot fully reflect the quality of generated results.
  4. The combination of audiovisual filtering, source localization and tracking, geometry-based spatialization, and synthetic scene generation offers a scalable approach to spatial supervision. The dataset and complementary evaluation metrics could be useful for subsequent research.
Critical weaknesses

Major weaknesses:

  1. The reference stereo audio in SWBench is constructed with the proposed VAS pipeline, and the training data are rendered with the same pipeline. As a result, ILD-W, SMR-Err, AST-Ang, and AST-Cal essentially measure how similar the outputs are to those of the VAS pipeline, rather than whether the spatial perception is correct. Outputs with reasonable spatial perception may thus be overly penalized simply because their rendering style differs. This may undermine the credibility of the evaluation for the key problem addressed in this paper.
  2. The necessity of the proposed mechanisms is questionable, since the main baselines lack the same trajectory conditioning and spatialization training. The component-wise ablations cannot replace a simple trajectory-conditioned fine-tuning baseline trained with the same data and budget. The paper also does not compare against an approach that uses the same generation backbone and motion control followed by post-hoc spatialization. It therefore remains unclear how much irreplaceable benefit the complex joint modeling provides.

Minor weaknesses:

  1. Typos and wording errors: "blcok" should be "block", "SOUDING" should be "SOUNDING", "Systhesis" should be "Synthesis", etc.
  2. The trajectory extraction is underspecified: Appendix B.3 describes smoothing but does not clarify what the representative 2D point denotes or how it is selected from AllTracker outputs. Its relationship to the source region used in SELD-Acc also needs clarification.
Questions for authors
  1. Could the authors evaluate StereoBind against references not produced by the VAS pipeline, such as real recorded stereo clips or audio rendered with a different spatializer? Does its advantage on ILD-W, SMR-Err, AST-Ang, and AST-Cal persist under such an evaluation?
  2. Could the authors compare StereoBind with a simple trajectory-conditioned fine-tuning baseline on the same LTX-2.3 backbone, trained with the same data and budget?
  3. Could the authors compare StereoBind with a cascaded baseline that uses the same backbone and motion control, followed by post-hoc spatialization with the known trajectory? If StereoBind does not clearly outperform it, what benefit does the joint modeling uniquely provide?
  4. What does the representative 2D point in the motion track denote, and how is it obtained from the AllTracker outputs? How does it relate to the bounding box used in SELD-Acc?
  5. Is there a commitment to open source?

If most of the above questions are answered, I'll consider raising my score.

3 comments

Response To Reviewer J1fc:

Thanks for the suggestions. We will add relevant details to the final version.

W1: Concern on SWBench

W1.1: The Renderer Dependence of Spatial Metrics

Please refer to our response to Reviewer QoMj (W1.1) and Reviewer J1fc (W1.2) for details.

W1.2: Additional Renderer-Independent Evaluation

SELD-Acc is renderer-independent, we further report an additional renderer-independent metric, Direction-of-Arrival MAE (DOA-MAE), comparing the azimuth estimated by pretrained Spatial-AST with the target azimuth from the VAS-predicted 3D trajectory. As shown below, StereoBind achieves the best.

The details of the additional metric is highlighted in red at the end of Appendix H.2 in the revised manuscript.

MethodsDOA-MAE ↓
Ovi51.96
MiniMax-H349.96
LTX-2.532.82
LTX-2.331.05
StereoBind19.34

W1.3: Cross-Renderer Evaluation

We test renderer sensitivity by replacing gpuRIR in VAS with HRTF/BRIR renderers and recomputing spatial metrics. StereoBind maintains consistent spatial performance across renderers.

RendererILD-W ↓SMR-Err ↓AST-Ang ↑AST-Cal ↑
HRTF2.86416.310.4410.301
BRIR2.91315.940.4520.28
gpuRIR2.75414.860.4650.314

W2: Concern on Necessity of Proposed Mechanism.

W2.1: Baselines trained on StereoWorld-29K.

To isolate architectural contributions from the effect of additional training data, we fine-tune baselines on StereoWorld-29K for 8,000 steps. We train a LoRA adapter for MiniMax-H3 Ref2VA using DiffSynth-Studio and LoRA adapters for LTX-2.3/2.5. Ovi is excluded due to the lack of available training code. As shown below, fine-tuning yields marginal improvements.

MethodILD-W ↓SELD-Acc ↑SMR-Err ↓AST-Ang ↑AST-Cal ↑
MiniMax-H3 Ref2VA3.2570.60326.140.2460.157
LTX-2.33.2140.57634.850.2420.145
LTX-2.53.3120.53133.920.2490.166
StereoBind2.7540.75714.860.4650.314

W2.2: Comparison with the LTX-2.3 + Motion Track IC-LoRA + VAS cascade

Please refer to our response to Reviewer J1fc (Q1) for details.

W2.3: Benefit of Joint Modeling

Please refer to our response to Reviewer wPGN (W1.3) for details.

W3: Tpyo Error

Thank you for pointing this out. We have corrected our manuscript. The revised are highlighted in red.

W4: Motion-Track Point Extraction and Its Relationship to SELD-Acc Bounding Boxes

First, Grounded-SAM identifies the source and provides its binary mask and bounding box. Then, representative 2D point is initialized at the center of the bounding box and propagated across frames by AllTracker. The coordinates are then resampled and smoothed using Gaussian filtering and robust third-order polynomial fitting. SELD-Acc uses the same bounding box and considers a prediction correct if its audio-location interval overlaps the box's horizontal interval.

Q1: Evaluating Generalization to Real-recorded Binaural Audio.

Given the limited time and resources during rebuttal, collecting a new real-world stereo dataset with professional recording equipment was impractical. We therefore leveraged MRSAudio dataset to develop a training and evaluation pipeline. Specifically, we extracted 5-second speech segments and used GPT-6.1 to generate LTX-2.3 text prompts from their speech content. We converted MRSAudio's 3D sound-source locations into 2D motion tracks. Each stereo recording was downmixed to mono and fed into LTX-2.3 with its prompt and motion track to generate a video. We then replaced the generated audio with the original stereo recording, obtaining 220 stereo audio-video pairs, with 200 for StereoBind fine-tuning and 20 for evaluation.

For evaluation, we downmixed each real-world stereo recording to mono as the audio input to both StereoBind and LTX-2.3, together with the corresponding prompt and motion track. Finally, we evaluated both methods against the original real-world stereo recordings. As shown below, both methods achieve comparable performance under this setting.

MethodILD-W ↓SELD-Acc ↑SMR-Err ↓AST-Ang ↑AST-Cal ↑
LTX-2.3+VAS2.8950.72017.290.4220.284
StereoBind2.8480.74918.260.4150.286

Q2: Baselines Trained on StereoWorld-29K.

Please refer to our response to Reviewer QoMj (W2.2) for details.

Q3: Benefit of Joint Modeling

Please refer to our response to Reviewer wPGN (W1.3) for details.

Q4: Motion-Track Point Extraction and Its Relationship to SELD-Acc Bounding Boxes

Please refer to our response to Reviewer J1fc (W4) for details.

Q5: Open Source.

Yes. We commit to releasing the inference and training code of StereoBind, StereoWorld-29K dataset and SWBench.

Reviewer_J1fcOpenReview ↗

response to authors

Thank you to the author for the response. After carefully reviewing the additional information provided, most of my questions have been answered, so I’ve decided to keep my rating as is.

Dear Reviewer J1fc:

We sincerely thank the reviewer for the constructive feedback and thoughtful consideration of our rebuttal. We are pleased that our responses have addressed your concerns, and we greatly appreciate your suggestions, which have helped us strengthen the manuscript.

We would like to briefly highlight the key findings regarding your two main concerns: Baselines Trained on StereoWorld-29K and Benefit of Joint Modeling

For Baselines Trained on StereoWorld-29K.

To isolate architectural contributions from the effect of additional training data, we fine-tune baselines on StereoWorld-29K for 8,000 steps. We train a LoRA adapter for MiniMax-H3 Ref2VA using DiffSynth-Studio and LoRA adapters for LTX-2.3/2.5. Ovi is excluded due to the lack of available training code. As shown below, StereoBind consistently outperforms all fine-tuned baselines across the five spatial-audio metrics. These results suggest that its improvements cannot be explained solely by exposure to StereoWorld-29K, and instead provide further evidence for the effectiveness of the proposed architecture.

MethodILD-W ↓SELD-Acc ↑SMR-Err ↓AST-Ang ↑AST-Cal ↑
MiniMax-H3 Ref2VA3.2570.60326.140.2460.157
LTX-2.33.2140.57634.850.2420.145
LTX-2.53.3120.53133.920.2490.166
StereoBind2.7540.75714.860.4650.314

For Benefit of Joint Modeling:

We emphasize that these mechanisms are functionally distinct and operating on different aspects.

VMB and Joint VA Attention. The original Joint VA Attention in LTX-2.3 provides general information exchange between video and audio features, however, does not explicitly establish which visual entity should be associated with a track, which is addressed by VMB.

VMB and STE. The STE determines how the context is accessed and incorporated under the track. The shift and scale in STE modulate the audio queries for VMB retrieval, while the gate controls the strength of the update. Therefore, STE provides the track-adaptive control for VMB, while VMB establishes what visual-motion context should be transferred to audio.

VMB and RT-RoPE. RT-RoPE introduces track-dependent residual rotary phases into audio self-attention, encoding relative spatial displacement between audio tokens, but does not associate the track with a particular visual entity. VMB operates in the cross-modal feature space, associating the motion with the generated visual source and transferring the resulting context to audio.

Joint Benefits of VMB, STE, and RT-RoPE. The advantage of jointly incorporating these mechanisms lies in their roles in modeling dynamic spatial correspondence. VMB establishes which visual source the motion belongs to, STE provides where the source is located through absolute spatial conditioning, and RT-RoPE captures how its position evolves through relative motion modeling. More importantly, these mechanisms jointly constrain the audio generation process at different levels: VMB transfers source-aware visual context to audio, STE adaptively controls its retrieval and integration, while RT-RoPE explicitly models motion-dependent spatial relationships within audio self-attention. This coordinated design enables the audio stream to jointly capture visual source identity, absolute spatial position, and relative motion dynamics.

Additional Analysis of VMB Attention Map in Sec. 5.4.

we have added a detailed analysis of VMB in Section 5.4 of the revised main text, highlighted in red, to clarify how its progressive reference–target–audio pathway supports audiovisual spatial correspondence. Specifically, we visualize attention maps for all three stages—reference aggregation, target association, and audio readout. All maps are extracted at Transformer block 8 and denoising step 8 and averaged across attention heads.

Finally, We sincerely hope that the additional evidence and clarifications have further strengthened the case for our approach and may provide a basis for reconsidering the overall assessment of our work.

If there are any remaining concerns that may affect your assessment, we would be grateful for the opportunity to clarify them further.

Reviewer_QoMjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies stereo audio in joint video-audio (VA) generation. The authors call the property they target "dynamic spatial correspondence", where the perceived sound location should follow the motion of the visible source. The proposed StereoBind builds on LTX-2.3 with a frozen motion-track IC-LoRA on the video branch. It adds three track-conditioning modules to the audio branch: 1) Visual Motion Binding (VMB), in which learnable tokens aggregate the rendered motion reference and the current video state, and the audio stream reads them out, 2) Spatial Track Encoder (STE) that produces shift/scale/gate modulation from the track, and 3) Residual Track RoPE (RT-RoPE) for adding track-dependent phases to the audio RoPE. For training, the authors build StereoWorld-29K, where the stereo targets are rendered from mono audio along the tracked source trajectory (lifted to 3D with VGGT, rendered with gpuRIR). They also propose a 95-sample benchmark named SWBench. StereoBind outperforms existing VA and V2SA baselines on the proposed spatial metrics, and the paper also includes ablations and a user study.

Critical strengths
  • The paper addresses a meaningful and practically relevant problem. Current joint VA models already output two-channel audio, but the stereo cues are mostly not meaningful. Conditioning stereo generation on the motion track is a natural way to control this, and the SILD/IDS visualizations show the gap of existing models quite clearly.

  • The data pipeline is a useful practical contribution. Real stereo recordings with tracked sound sources are hard to collect at scale, and the VAS pipeline is a reasonable way to obtain paired track/stereo data. If released, StereoWorld-29K and SWBench could be useful for follow-up work

Critical weaknesses
  • My main concern is about the evaluation part. The training targets are rendered by the VAS pipeline, and the evaluation references are also obtained by re-spatializing the generated audio with the same pipeline. So ILD-W, SMR-Err, AST-Ang and AST-Cal mainly measure how close the output is to this renderer. StereoBind is trained to imitate it, while the baselines are not. The only metric that does not depend on the renderer is SELD-Acc, and there the gap to MiniMax-H3 is small. In addition, no real recorded stereo audio is used in training or evaluation, so it is unclear whether the model learns real spatial cues or just the simulator's cues.

  • I also think the experiments miss an important baseline. Since the motion track is given at inference, one could simply generate audio with LTX-2.3 + the track IC-LoRA and then render it with the VAS pipeline along the track. Under the current metrics, this cascade would probably be very strong. Without this comparison, and without a baseline that only fine-tunes on StereoWorld-29K without VMB/STE/RT-RoPE, it is hard to tell what the proposed modules actually bring. Also, SWBench has only 95 samples and all results come from a single run. Some ablation numbers are hard to interpret, for example, w/o VMB gets a SELD-Acc of 0.506, which is even lower than the untuned LTX-2.3 (0.617).

Questions for authors
  • For the VA baselines, is the reference rendered along the prescribed track, or along a track extracted from each method's generated video? Their videos may not follow the given track. In that case, they are penalized for the video motion rather than for the stereo audio.

  • In Eq. (2), the audio query is LN(Xal)LN(X^l_a), but the text says βl\beta_l and γl\gamma_l modulate the audio queries. Eq. (4) also uses the modulated X~al\tilde{X}^l_a as the residual base, which does not seem to "retain the pretrained audio residual pathway". Which formulation is actually implemented?

  • For other questions, please see above Critical Weaknesses.

4 comments

Response To Reviewer QoMj:

Thanks for valuable suggestions. We will incorporate relevant explanation into the final version.

W1: Concern on Evaluation Part.

W1.1: Clarifying the Renderer Dependence of Metrics.

We agree that these metrics measure consistency relative to the VAS renderer. However, our purpose is to isolate stereo quality from differences in audio content, allowing the evaluation to focus on whether the generated audio follows the motion. Additionally, we also includes two metrics that do not compare generated audio against VAS-rendered audio: SELD-Acc and human perceptual evaluation. StereoBind achieves the best results in both metrics, indicating superior perceived stereo quality.

To further substantiate, we provide an additional renderer-independent metric. Please refer to our response to W1.2 of Reviewer J1fc.

W1.2: MiniMax-H3's SELD-Acc

SELD-Acc measures whether the predicted audio-localization interval overlaps with the sound source's bounding-box interval. However, it can overestimate performance when visual motion is limited. This is particularly relevant to MiniMax-H3, whose text-conditioned motion control is insufficient. For example, even with a left-to-right motion prompt, the subject may remain near one side of the frame or exhibit only minor horizontal movements around the center. Consequently, a nearly stationary audio-localization prediction may overlap with the bounding box across multiple frames, yielding a high SELD-Acc despite failing to capture meaningful dynamic spatial correspondence.

To solve, we revise our evaluation. For each prompt and each baseline, we generate 10 samples with different seed. We use Qwen3-VL-30B-A3B-Instruct to assess whether each generated video follows the motion, followed by manual selection. We conduct experiments based on filtered candidates. As shown below, MiniMax-H3's SELD-Acc decreased.

MethodILD-W ↓SELD-Acc ↑SMR-Err ↓AST-Ang ↑AST-Cal ↑
Ovi3.9220.44266.210.1560.021
MiniMax-H33.6410.52929.040.2150.123
LTX-2.33.8740.50441.670.2140.105
LTX-2.53.7710.48639.110.2290.109

W1.3: Generalization to Real-recorded Audio.

Given the limited time and resources during rebuttal, collecting a new real-world stereo dataset with professional recording equipment was impractical. We therefore leveraged MRSAudio [1] dataset to develop a training and evaluation pipeline. Specifically, we extracted 5-second speech segments and used GPT-6.1 to generate LTX-2.3 text prompts from their speech content. We converted MRSAudio's 3D sound-source locations into 2D motion tracks. Each stereo recording was downmixed to mono and fed into LTX-2.3 with its prompt and motion track to generate a video. We then replaced the generated audio with the original stereo recording, obtaining 220 stereo audio-video pairs, with 200 for StereoBind fine-tuning and 20 for evaluation.

For evaluation, we downmixed each real-world stereo recording to mono as the audio input to both StereoBind and LTX-2.3, together with the corresponding prompt and motion track. Finally, we evaluated both methods against the original real-world stereo recordings. As shown below, both methods achieve comparable performance under this setting.

MethodILD-W ↓SELD-Acc ↑SMR-Err ↓AST-Ang ↑AST-Cal ↑
LTX-2.3+VAS2.8950.72017.290.4220.284
StereoBind2.8480.74918.260.4150.286

[1] Guo W, Pan C, Zhu Z, et al. Mrsaudio: A large-scale multimodal recorded spatial audio dataset with refined annotations[J]. Advances in Neural Information Processing Systems, 2026, 38.

W2: Concern on Baselines

W2.1: Comparison with the LTX-2.3 + Motion Track IC-LoRA + VAS

ILD-W, AST-Ang, and AST-Cal require reference stereo audio, for which we use VAS-rendered audio as ground truth. Applying the same VAS pipeline to LTX-2.3 with track IC-LoRA would reproduce the reference-generation process, yielding an upper bound and making comparison with StereoBind circular. This cascade also requires Qwen3-VL, VGGT, and AllTracker, introducing substantial GPU memory and inference-time overhead. More fundamentally, StereoBind aims to learn motion–spatial-audio correspondence within the generative model. Its complementary mechanisms inject spatial information directly into audio representations, enabling joint modeling of audio content and spatial characteristics during generation.

Nevertheless, to address the concern, we construct a dedicated evaluation for this cascade approach. Please refer to our response to W1.3.

W2.2: Baselines Trained on StereoWorld-29K.

To isolate architectural contributions from the effect of additional training data, we fine-tune baselines on StereoWorld-29K for 8,000 steps. We train a LoRA adapter for MiniMax-H3 Ref2VA using DiffSynth-Studio and LoRA adapters for LTX-2.3/2.5. Ovi is excluded due to the lack of available training code. As shown below, fine-tuning yields marginal improvements.

MethodILD-W ↓SELD-Acc ↑SMR-Err ↓AST-Ang ↑AST-Cal ↑
MiniMax-H3 Ref2VA3.2570.60326.140.2460.157
LTX-2.33.2140.57634.850.2420.145
LTX-2.53.3120.53133.920.2490.166
StereoBind2.7540.75714.860.4650.314

W2.3: Expanding StereoWorldBench and Conducting Multi-seed Evaluations.

Please refer to red-highlighted text in section 5.5 in the revised manuscript.

W2.4: The SELD-Acc of the w/o VMB variant.

Without VMB, STE and RT-RoPE still provide absolute-position conditioning and relative-motion cues, preserving spatially structured stereo generation. However, they do not explicitly associate spatial information with the corresponding visual source. Consequently, the model may retain spatial cues but fail to bind them to the correct visual entity, resulting in a pronounced drop in SELD-Acc, which measures audio-visual localization overlap.

Q1: Reference-Track Construction for Evaluation of VA Baselines

For VA baselines, references are rendered along tracks extracted from their generated videos. To avoid penalizing visual-motion deviations, we revised the evaluation protocol, please refer to our response to Reviewer QoMj (W1.2) for further details.

Q2: Audio-Query Modulation and the Residual Pathway

We apologize for the ambiguity. βℓ\beta^\ell and γℓ\gamma^\ell first modulate the audio hidden states to obtain X~aℓ\widetilde{X}_a^\ell, which is used as the input to the VMB audio readout. Thus, Eq. (2) should use LN⁡(X~aℓ)\operatorname{LN}(\widetilde{X}_a^\ell). X~aℓ\widetilde{X}_a^\ell does not replace the original audio hidden state in the main stream. After the VMB, its output is modulated by the gℓg^\ell. Therefore, the residual base in Eq. (4) is indeed the original XaℓX_a^\ell, not X~aℓ\widetilde{X}_a^\ell. We have corrected Eq. (2) and (4), with the revised highlighted in red.

Reviewer_QoMjOpenReview ↗

Official Comment by Reviewer QoMj

I thank the authors for the thoughtful response and for the additional experiments. Most of my concerns have been addressed, so I raised my rating to weak acceptance. However, I still have two follow-up questions:

  • In W1.3, StereoBind and LTX-2.3+VAS perform similarly on real recordings, so besides efficiency, what is the advantage of the proposed modules over the cascade? Could you provide the inference time and memory comparison to support the overhead argument in W2.1?

  • In W2.2, does the fine-tuned LTX-2.3 include the motion-track IC-LoRA, with the same training data and steps as StereoBind? This would help me understand whether the gain comes from the proposed modules or from the track conditioning.

Dear Reviewer QoMj:

We sincerely thank the reviewer for the constructive feedback and thoughtful consideration of our rebuttal. We are pleased that our responses have addressed your concerns, and we greatly appreciate your suggestions, which have helped us strengthen the manuscript.

We also thank you for raising these additional questions. We provide our clarifications below.

Advantage of the proposed modules over the cascade

Our contribution in this setting is to incorporate dynamic spatial correspondence directly into joint audiovisual generation. VMB associates visual motion with the sounding source, while STE and RT-RoPE introduce absolute-position and relative-motion conditioning into the audio branch. Spatial information therefore participates in audio synthesis itself. In contrast, the cascade spatializes the generated waveform through a separate perception-and-rendering pipeline.

To quantify the overhead in W2.1, we profile both approaches on a single NVIDIA H20 GPU. Both generators use the same 8-step distilled checkpoint, ltx-2.3-22b-distilled-1.1.safetensors, and the same spatial upscaler, ltx-2.3-spatial-upscaler-x2-1.1.safetensors. Each sample contains 121 frames, generated at 640×384 in Stage 1 and upscaled to 1280×768 in Stage 2. For VAS, Qwen3-Omni-30B-A3B-Instruct is deployed using vLLM and receives the 640×384 Stage 1 output.

Stage / ComponentLTX-2.3 + IC-LoRA + VAS(s)StereoBind(s)
Stage 1: Generation at 640×38458.4667.87
Stage 2: Upscaling to 1280×76823.8027.21
VAS 1: Source discovery — Qwen3-Omni3.88—
VAS 2: Source grounding — Grounded SAM236.25—
VAS 2: Source verification — Qwen3-Omni3.57—
VAS 3: Source tracking — AllTracker9.36—
VAS 4: Geometry estimation — VGGT19.45—
VAS 4: Spatial audio rendering — gpuRIR1.25—
VAS subtotal73.76—
Total summed stage time156.0295.08

The memory usage comparison is shown below.

Stage / ComponentLTX-2.3 + IC-LoRA + VAS(GB)StereoBind(GB)
Audiovisual generation42.2052.73
VAS 1: Source discovery — Qwen3-Omni83.81—
VAS 2: Source grounding — Grounded SAM221.58—
VAS 2: Source verification — Qwen3-Omni84.21—
VAS 3: Source tracking — AllTracker21.58—
VAS 4: Geometry estimation and spatial audio rendering — VGGT + gpuRIR16.32—
Total memory (pipeline peak)269.7052.73

Compared with the full LTX-2.3 + IC-LoRA + VAS cascade, StereoBind reduces the summed inference time and memory usage. Additionally, StereoBind also incorporates spatial correspondence directly into audiovisual synthesis through VMB, STE, and RT-RoPE. Its inference workflow therefore avoids sound source grounding, tracking, as well as spatial audio estimation and rendering, reducing reliance on auxiliary perception and rendering models.

Implementation Details of Fine-tuned LTX-2.3

Yes, the fine-tuned LTX-2.3 baseline in W2.2 includes the same pretrained Motion Track IC-LoRA as StereoBind. Both the pretrained LTX-2.3 weights and Motion Track IC-LoRA are kept frozen, and only the added LoRA parameters in audio branch of LTX-2.3 are optimized. The Audio LoRA configuration matches StereoBind, with adapters in the audio-branch attention and feed-forward layers. Both models are trained on the same StereoWorld-29K for 8K iterations, using the same training objective, optimizer, and learning-rate schedule. Evaluation also uses the same motion tracks. The key difference is that StereoBind additionally includes VMB, STE, and RT-RoPE.

Under these matched settings, StereoBind outperforms the fine-tuned LTX-2.3 baseline across all five spatial-fidelity metrics. Since both models share the same pretrained Motion Track IC-LoRA, Audio LoRA configuration, training data, and training schedule, these gains support the additional contribution of the proposed spatial modeling through VMB, STE, and RT-RoPE. The improvement therefore goes beyond the benefit of the track conditioning and stereo-data fine-tuning. We will explicitly clarify this baseline configuration and the matched experimental settings in the final manuscript.

Finally, we sincerely hope that the additional evidence and clarifications have further strengthened the case for our approach. If there are any remaining concerns that may affect your assessment, we would be grateful for the opportunity to address them further. Thank you again for your thoughtful comments and constructive suggestions.

Reviewer_wPGNOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses dynamic spatial correspondence in joint video–stereo audio generation: the perceived sound location should evolve consistently with the motion of the visible sound source. The authors introduce StereoBind, built on LTX-2.3, with Visual Motion Binding (VMB), a Spatial Track Encoder (STE), and Residual Track RoPE (RT-RoPE). They also construct StereoWorld-29K by combining real-world audiovisual content with synthetic scenes, and introduce StereoWorldBench for evaluating spatial audiovisual alignment. The reported results demonstrate improvements in stereo spatial fidelity while maintaining audiovisual generation quality.

Critical strengths
  • Dataset and benchmark contributions. StereoWorld-29K and StereoWorldBench address an important need for spatially supervised audiovisual data and evaluation of correspondence between visible source motion and perceived sound location.
  • Scalable and diversity-aware data construction. Combining real-world audiovisual content with controllable synthetic scenes provides a practical approach to scaling the dataset and expanding motion coverage. The Scene Distribution Bank is a useful mechanism for reducing repetition and encouraging diversity in sound sources, scenes, and motion configurations.
  • Detailed methodology. The paper provides detailed descriptions of the data construction pipelines and model training setup. These implementation details support reproducibility and help readers understand how the overall system is constructed.
  • Promising qualitative results. The presented demonstrations make the benefits of spatially aligned stereo audio perceptually apparent. In examples with clear left–right source motion, StereoBind produces a more convincing correspondence between visual motion and perceived sound location than the compared methods.
  • Appropriate overall approach. The proposed framework addresses complementary aspects of the task by incorporating visual source context, explicit spatial conditioning, and relative-motion information into audio generation.
  • Clear presentation. The paper is generally well organized and easy to follow. The terminology, notation, and architectural illustrations help communicate the problem and the overall method.
Critical weaknesses
  • Limited analysis of the intermediate behavior of VMB. The quantitative ablations support the usefulness of VMB as a whole, but provide limited insight into the behavior of its progressive reference–target–audio pathway. In particular, it remains unclear how the intermediate representations associate the prescribed motion with the generated visual source and subsequently inform the audio stream. Given that the backbone already includes audiovisual attention and RT-RoPE also provides motion conditioning, a more concrete explanation of VMB’s intended contribution and its relationship to these mechanisms would strengthen the paper. This is primarily a concern about the interpretation and explanation of the proposed mechanism, rather than its demonstrated overall utility.

  • Line 267: “blcok” should be corrected to “block.”

Questions for authors

Q1

Could the authors clarify the rationale for the reference–target–audio ordering in VMB, and explain how this pathway complements the backbone’s existing audiovisual attention and RT-RoPE?

Q2

  • Could the authors provide a qualitative attention-map analysis illustrating the three VMB stages? For example: a) Reference aggregation: Which spatiotemporal regions of the motion reference does each VMB token aggregate information from? b) Target association: How do the reference-conditioned VMB tokens attend to the sound-producing entity and surrounding scene in the target video? c) Audio readout: Which fused VMB tokens do audio tokens at different times attend to, and how do these patterns relate to the visual and motion information observed in the preceding stages?

Showing these attention maps together for the same sample and transformer block, with the denoising timestep specified, would help readers understand the intended progressive binding behavior. Such an analysis could provide qualitative support for the proposed interpretation without requiring additional architectural comparisons, although attention maps alone would not establish functional specialization.

2 comments

Response To Reviewer wPGN:

Thanks for your valuable suggestions. We will incorporate relevant explanation into the final version for better understanding.

W1: Additional Analysis of VMB

W1.1: Analysis of the Reference–Target–Audio Pathway

The three stages of VMB serve complementary functions. Reference Aggregation conditions the VMB tokens on the motion features which encodes the track as a motion-conditioned anchor. Target Association then incorporating the semantic context of the visual source. Audio Readout allows the audio features to query the fused representation, producing the spatially conditioned audio feature.

Beyond their individual functions, the ordering among these stages are important. First, Audio Readout must follow both visual stages, because the audio stream needs to retrieve a representation that integrates the motion with its corresponding visual context. Second, Reference Aggregation must precede Target Association to enable motion-guided visual retrieval. This is crucial because the reference specifies where and how the source should move but does not identify its visual appearance, whereas the target provides visual context. Reversing the order to Target–Reference–Audio would fundamentally alter the process. If Target Cross-Attention is performed first, the initial learnable VMB tokens lack an explicit criterion for determining which visual context should be associated with the track. Consequently, the attention may potentially producing a diffuse representation that fails to isolate motion-relevant source.

W1.2: Additional Ablation Study on VMB.

To support the analysis above, we conduct ablation experiments that compare the original pathway against reversed Target–Reference–Audio, as well as variants without reference or target. The results are shown below.

VMB VariantILD-W ↓SELD-Acc ↑SMR-Err ↓AST-Ang ↑AST-Cal ↑
Reversed Reference3.5550.63523.770.3720.269
w/o Reference Aggregation3.2150.60221.850.2940.232
w/o Target Association2.9490.65522.650.3180.241
Full VMB (existing)2.7540.75714.860.4650.314

W1.3: Relationship to Joint VA Attention, STE, and RT-RoPE.

We emphasize that these mechanisms are functionally distinct and operating on different aspects.

VMB and Joint VA Attention. The original Joint VA Attention in LTX-2.3 provides general information exchange between video and audio features, however, does not explicitly establish which visual entity should be associated with a track, which is addressed by VMB.

VMB and STE. The STE determines how the context is accessed and incorporated under the track. The shift and scale in STE modulate the audio queries for VMB retrieval, while the gate controls the strength of the update. Therefore, STE provides the track-adaptive control for VMB, while VMB establishes what visual-motion context should be transferred to audio.

VMB and RT-RoPE. RT-RoPE introduces track-dependent residual rotary phases into audio self-attention, encoding relative spatial displacement between audio tokens, but does not associate the track with a particular visual entity. VMB operates in the cross-modal feature space, associating the motion with the generated visual source and transferring the resulting context to audio.

Joint Benefits of VMB, STE, and RT-RoPE. The advantage of jointly incorporating these mechanisms lies in their roles in modeling dynamic spatial correspondence. VMB establishes which visual source the motion belongs to, STE provides where the source is located through absolute spatial conditioning, and RT-RoPE captures how its position evolves through relative motion modeling. More importantly, these mechanisms jointly constrain the audio generation process at different levels: VMB transfers source-aware visual context to audio, STE adaptively controls its retrieval and integration, while RT-RoPE explicitly models motion-dependent spatial relationships within audio self-attention. This coordinated design enables the audio stream to jointly capture visual source identity, absolute spatial position, and relative motion dynamics.

W2: Typo Error.

Thank you for pointing this out. We have corrected "blcok" to "block". The revised are highlighted in red.

Q1: Detailed Analysis of VMB

Please refer to our response to W1.

Q2: Attention-Map Analysis of VMB.

We visualize the attention of the three stages. All three stages are visualized at Transformer block 8 and denoising step 8, with attention maps obtained by averaging all attention heads.

Please refer to the red-highlighted text in Section 5.4, Attention-Map Analysis of VMB, in the revised main text.

Dear Reviewer,

We are writing to follow up on our rebuttal and the additional evidence and clarifications we provided. When you have a chance, we would appreciate your feedback on whether our responses have addressed your concerns. Please let us know if anything requires further clarification.

Thank you for your time and consideration.

Best regards,
The Authors

Comments on the paper

1 comment

We thank all reviewers for their time and thoughtful feedback, which has helped us strengthen the experimental evidence, clarify the proposed mechanisms, and improve the presentation. We appreciate the reviewers' recognition of the following strengths:

  • Motivation and practical relevance. The reviewers recognized dynamic spatial correspondence as a meaningful relevant problem that remains insufficiently addressed in existing joint audiovisual generation models (QoMj, J1fc).

  • Dataset and benchmark contributions. The reviewers appreciated the potential value of StereoWorld-29K and StereoWorldBench for future research. They also recognized the practicality of combining real-world audiovisual content with controllable synthetic scenes to obtain spatial supervision (wPGN, QoMj, J1fc).

  • Motivated architectural design. The reviewers acknowledged the complementary roles of VMB, STE, and RT-RoPE in incorporating source-aware visual context, absolute spatial positions, and relative motion into audio generation (wPGN, J1fc).

  • Qualitative and perceptual evidence. The reviewers found that the demonstrations make the benefits of spatially aligned stereo audio perceptually apparent and highlighted the human evaluation as valuable evidence supporting the method's effectiveness (wPGN, J1fc).

  • Detailed methodology and clear presentation. The reviewers appreciated the organization, architectural illustrations, and implementation details, noting their value for understanding and reproducibility (wPGN).

In response to the reviewers' constructive suggestions, we acknowledge that our work can be further improved and have addressed all concerns as folows:

  1. Evaluation validity and reference construction. We added evaluation using real recorded stereo audio to examine performance beyond VAS-generated references. This addition distinguish agreement with a particular renderer from broader evidence of audiovisual spatial correspondence (QoMj, J1fc).

  2. Matched fine-tuning baselines. We added baselines fine-tuning on StereoWorld-29K, with same training implemetation as StereoBind. StereoBind's improvements under these matched conditions support the additional contribution of VMB, STE, and RT-RoPE beyond shared video-side track conditioning and stereo-data fine-tuning (QoMj, J1fc).

  3. Cascaded baseline and computational costs. We added the LTX-2.3 + Motion Track IC-LoRA + VAS cascade and provided runtime and memory measurements. We clarified the benefits of incorporating spatial conditioning into generation, including fewer auxiliary model dependencies at inference time and less GPU memory overhead. We have bounded our claims where the two approaches achieve comparable quality on real recordings (QoMj, J1fc).

  4. VMB rationale and intermediate behavior. We expanded the explanation of the reference–target–audio ordering and its relationship to the backbone's existing audiovisual attention and RT-RoPE. We added attention-map visualizations of the three VMB stages (wPGN).

  5. Implementation consistency and trajectory definitions. We clarified the implemented audio-query modulation and residual update, and revised the equations and accompanying descriptions for consistency. We also specified how the representative 2D trajectory point is obtained from AllTracker outputs and how it relates to the source region used in SELD-Acc (QoMj, J1fc).

  6. Experimental reliability and ablation interpretation. We supplemented the results with multi-run analysis, expanded the discussion of the ablation results, including the behavior of the variant without VMB (QoMj).

We have uploaded a revised manuscript, with all changes highlighted in red. The main revisions are as follows:

  1. Revised the notation in the equations in Secs. 4.2 and 4.3.
  2. Added attention map visualizations for the three stages of VMB in Sec. 5.4.
  3. Added a multi-seed evaluation on Extended SWBench in Sec. 5.5.
  4. Added an explanation of the principles underlying the Spatial-AST-based DOA-MAE metric at the end of Appendix H.2, clarifying that this metric does not require ground-truth.

We thank the reviewers again for their constructive comments. We hope that the additional evidence and revisions clarify the scope of our contributions and address the concerns raised.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.