acceptodds
Under review as a conference paper at ICLR 2027

Sound Follows Contact: Streaming Audio-Visual Generation with Contact-Aware State Grounding

Abstract

Streaming audio-visual generation requires sound events to align with their corresponding visual causes. In rigid-body scenes, sound timing is strictly governed by physical contact, making it highly sensitive to subtle motion errors. Under autoregressive chunk-wise rollout, these errors accumulate across transitions and manifest as delayed, missing, or spurious audio events. We formalize this failure mode as contact-phase drift, which compounds over long horizons where contact timing degrades substantially faster than perceptual quality. To address this challenge, we present Tora-live, a state-grounded framework for streaming audio-visual generation in contact-dominated rigid-body scenes. Tora-live augments a few-step chunk-causal generator with a compact contact-aware hybrid state that combines dominant-object kinematics with explicit contact-phase variables. This state is maintained online through lightweight state propagation, contact-phase construction, and closed-loop correction from generated video, conditioning both modalities simultaneously through current-state modulation and short-term state-history attention. We further introduce a contact-focused rollout optimization objective that encourages generated audio onsets to align with scheduled contacts while suppressing missed and spurious responses. To evaluate this setting, we build ContactBench, a benchmark for contact-dominated audio-visual generation with contact-centric alignment metrics. Experiments show that Tora-live improves contact timing, audio-visual alignment, and long-horizon robustness over strong streaming baselines while maintaining competitive perceptual quality, with gains that increase consistently with rollout length.

Primary area:
generative models
Keywords:
streaming audio-visual generation, contact-aware state grounding, rigid-body dynamics, long-horizon generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_arHiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary
  • This paper studies streaming joint audio–video generation in rigid-body scenes where physical contact is the primary cause of sound events. The authors identify contact-phase drift, in which the timing of visible contacts and generated sounds becomes increasingly misaligned during autoregressive generation across chunks, and propose a method to mitigate this problem.
  • The proposed method, Tora-live, builds on a four-step, chunk-causal adaptation of LTX-2.3 that generates synchronized one-second audio–video chunks. It introduces a 10-dimensional state comprising the dominant object's position and velocity proxies, in-plane orientation, angular rate, a field representing recent and ongoing contact, and the time to the next scheduled contact. A lightweight dynamics model propagates this state within each chunk, and both modalities are conditioned on it through current-state modulation and attention to recent state history. After a chunk is generated, observations from its video frames are used to correct the state and initialize the next chunk. The method further applies contact-focused rollout optimization (CFRO), using Flow-GRPO to reward alignment between audio onsets and scheduled contacts, together with consistency between generated motion and the predicted state.
  • For evaluation, the authors introduce ContactBench, comprising 240 videos across six rigid-motion regimes. They report comparisons with offline and streaming baselines, generation results up to 30 seconds, component ablations, human evaluation, and an additional evaluation on 47 cases from VABench's physical-interaction subset. The reported results show improved contact alignment and greater robustness to increasing rollout length, while maintaining competitive perceptual quality in the evaluated scenes.
Critical strengths
  • The method combines a contact-aware state representation with feedback from generated video. The contact-phase variables (c,τ)(c,\tau), short-term state history, and closed-loop correction are well aligned with the goal of maintaining sound–motion correspondence during long rollouts. In particular, correcting the state using generated video helps keep subsequent contact schedules consistent with the motion actually produced by the model. The component ablations in Table 3 shows the usefulness of these design choices, providing empirical evidence for their role in mitigating contact-timing drift.
Critical weaknesses
  • ContactBench is insufficiently documented to assess it as a standalone benchmark contribution. Although the paper reports the number of videos, categories, and annotation types, several essential aspects of benchmark construction remain unclear: the sources of the 240 videos, collection and selection criteria, annotation and verification procedures, source-video-level deduplication against the training data, and separation from data used for validation or model selection. These omissions are consequential because ContactBench is presented as a central contribution and provides the primary evidence for the effectiveness of a model trained on a specially curated rigid-motion dataset. Without this information, it is difficult to assess potential data overlap, the independence of the evaluation, and the reproducibility of the results. The authors should document these procedures in sufficient detail to substantiate both the benchmark contribution and the experimental claims.
  • The empirical evaluation is limited in scope and does not adequately establish overall generation quality after contact-focused adaptation. Table 3 shows improvements in synchronization and perceptual-quality metrics on ContactBench after CFRO. However, the external evaluation is restricted to only 47 cases from the physical-interaction subset of VABench. This small evaluation within a closely related domain provides limited independent evidence of robustness and generalization.
In particular, the experiments do not establish whether contact-focused adaptation preserves legitimate sounds unrelated to contact events, or audio and motion diversity across samples generated from the same prompt. Because CFRO penalizes onsets that are not matched to scheduled contacts, it may suppress valid background sounds in scenes containing both contact sounds and other audio sources. KL regularization and the kinematic-consistency reward provide useful constraints, but their presence alone does not demonstrate that these capabilities are retained.
My inspection of the provided demos also raises concerns about overall physical plausibility and audio–visual semantic consistency, despite convincing contact timing in some examples: Figure1_videos/case2: The bouncing motion in the middle of the clip appears unnaturally prolonged and physically implausible., Figure1_videos/case3: The clip is silent, and the die barely rotates even as its visible face markings change, producing an unnatural visual result., Figure1_videos/case4: The sound does not semantically match the depicted event, and the motion of the metal disc appears physically implausible. I also checked other samples from Figure4_vides~Figure7_videos. A broader evaluation of physical plausibility and audio–visual semantic consistency, together with matched comparisons before and after CFRO, is needed to determine whether the gains in contact synchronization preserve other aspects of generation quality and diversity.
Questions for authors
  • ContactBench construction and annotation: Please clarify the sources of the ContactBench videos, the collection and selection procedures, and how separation from training and validation data was ensured at the source-video level. How many annotators verified the contact timestamps and state trajectories, what criteria did they follow, and how were disagreements and uncertainty in event timing handled?

  • Relationship to PAVAS and the cascaded baseline: PAVAS [1] estimates object-level mass and speed, combines these quantities with object-specific features, and conditions an audio generator through modulation. How does Tora-live relate to this approach, and what are its additional technical contributions beyond explicit conditioning on object-level physical quantities? Using the same generated videos, would replacing MMAudio with PAVAS in the cascaded baseline change the conclusions regarding contact alignment and audio quality?

  • Relationship to the FlatSounds metrics: FlatSounds [2] already evaluates contact- and impact-related acoustic events using Timing Error, Hit Coverage, and Perfect Alignment. Could you clarify how the contact-centric metrics in this paper relate to these measures? In particular, how do CTE and MCR differ from Timing Error and Hit Coverage in their definitions, event matching, and applicability, and what additional failure modes does the proposed evaluation protocol capture?

[1] Hyun-Bin, Oh, et al. "PAVAS: Physics-Aware Video-to-Audio Synthesis." CVPR 2026.

[2] Li, Tingle, et al. "Benchmarking Single-Factor Physical Video-to-Audio Generation." ECCV 2026.

4 comments
AuthorsOpenReview ↗

Response to Reviewer arHi (Part 1/2)

[Q1] ContactBench construction, annotation verification, and source-level separation

[A1] We expanded Section 4.1 and Appendix B.2 with ContactBench sources, selection, and annotation: 240 clips from Pexels (88), OpenVid1M (104), and self-recordings (48) across six regimes. Clips with ambiguous contacts or masking speech/music were excluded. To ensure independence from training and validation, we applied video ID/URL filtering, DINOv2 similarity screening above 0.85, and CLAP screening above 0.80. The benchmark was not used for tuning. Three annotators independently verified timestamps and trajectories, merged contacts agreeing within 1–2 frames by their median, corrected masks, and excluded unresolved cases. We plan to release the benchmark subject to source licenses.

[Q2] Relationship to PAVAS and comparison with physics-aware cascaded baselines

[A2] We added PAVAS to Section 2. PAVAS estimates mass and 3D velocity from existing video to condition audio. Tora-live instead targets contact-phase drift in joint causal generation using scale-normalized motion proxies and contact-phase variables, corrected from generated frames and injected into both modalities (Equations 3–6). Table 3(a) shows kinematics alone reduce 20 s CTE from 0.132 to 0.129, while the full state reaches 0.112 and CFRO reaches 0.099.

A direct PAVAS substitution was not feasible as its model weights are not publicly released. Moreover, its offline physical estimation takes 7.16 s per 10 s video, including 4.82 s CUT3R reconstruction. To test physics-guided streaming, we added RealWonder[1] + MMAudio-Large in Table 1 and Figure 4. RealWonder reconstructs a 3D scene and runs physics simulation for streaming video, then MMAudio synthesizes audio independently per chunk. Its 14.3 s upfront reconstruction contributes to 28.4 s for 5 s output. We use RealWonder’s PhysReal metric with its original prompt and scoring protocol, replacing GPT-4o with Gemini-4-Argon for all methods. RealWonder + MMAudio improves PhysReal over LTX-2.3† (0.708 vs. 0.664) but yields 0.091 CTE and 2.31 FAD. Tora-live reaches 0.072 CTE and 1.71 FAD in 8.0 s, supporting joint contact-aware conditioning without heavy reconstruction.

Method at 5 sCTE ↓\downarrowMCR ↓\downarrowSOR ↓\downarrowFGAS ↑\uparrowTC ↑\uparrowFVD ↓\downarrowFAD ↓\downarrowPhysReal ↑\uparrowRuntime ↓\downarrow
LTX-2.3† + MMAudio0.0880.2240.2000.2120.836234.22.240.66410.2 s
RealWonder + MMAudio0.0910.2290.2060.2100.812276.42.310.70828.4 s
LTX-2.3†0.0930.2320.1890.2160.836234.21.830.6647.4 s
Tora-live0.0720.1820.1580.2430.861221.51.710.7268.0 s

[Q3] Relationship to the FlatSounds temporal alignment metrics

[A3] We added FlatSounds (Li et al., 2026c) to Section 2 and evaluated its temporal metrics on generated videos. The protocols differ in three ways. (1) Reference: FlatSounds uses fixed input-video contacts, whereas we measure realized contacts in jointly generated videos. (2) Event matching: FlatSounds uses nearest-onset coverage, while our fixed-tolerance one-to-one matching prevents one onset from covering multiple contacts. (3) Failure coverage: FlatSounds reports Timing Error, Hit Coverage, and Perfect Alignment, while our SOR additionally identifies spurious onsets even when every visible contact is covered. Tora-live leads under both protocols, raising Perfect Alignment from 43.3% to 51.2% over LTX-2.3† at 5 s.

Method at 5 sCTE ↓\downarrowMCR ↓\downarrowSOR ↓\downarrowTiming Error ↓\downarrowHit Coverage ↑\uparrowPerfect Align ↑\uparrow
OmniForcing0.1580.4180.42894.6 ms62.4%24.2%
LTX-2.30.1080.2690.25468.3 ms75.8%38.8%
LTX-2.3†0.0930.2320.18961.2 ms79.1%43.3%
RealWonder + MMAudio0.0910.2290.20659.8 ms79.6%44.2%
Tora-live0.0720.1820.15851.4 ms84.5%51.2%

[1]Wei Liu, et al. "RealWonder: Real-Time Physical Action-Conditioned Video Generation"

AuthorsOpenReview ↗

Response to Reviewer arHi (Part 2/2)

[Q4] Overall generation quality, preservation of non-contact background sounds, diversity after CFRO, and demo inspection

[A4] Table 7 shows that CFRO aligns sounds with realized contacts rather than blindly rewarding scheduled onsets: removing RkinR_{\mathrm{kin}} improves schedule alignment from 0.862 to 0.931 but worsens visual-contact CTE from 0.099 to 0.118. We compare the same state-grounded model before (⋆\starReference) and after CFRO for quality, ambience, and diversity. For ambience-specific analysis, we select 76 clips with sustained natural ambience based on ContactBench annotations.

(1) Perceptual quality, audiovisual semantics, and physical plausibility. Across 240 clips at 20 s, CFRO improves CTE from 0.112 to 0.099, CLIP from 0.276 to 0.278, CLAP from 0.615 to 0.618, FAD from 1.85 to 1.79, and PhysReal from 0.689 to 0.706. The gains also hold on this annotation-defined ambient subset.

Subset at 20 sMethodCTE ↓\downarrowCLIP ↑\uparrowCLAP ↑\uparrowPhysReal ↑\uparrowFAD ↓\downarrow
All 240 clipsLTX-2.3†0.1320.2680.5920.6351.95
⋆\starReference0.1120.2760.6150.6891.85
Tora-live0.0990.2780.6180.7061.79
Ambient 76 clips⋆\starReference0.1160.2730.6080.6811.91
Tora-live0.1030.2740.6110.6981.87

(2) Preservation of legitimate background sounds. On these 76 annotated clips, we examine sustained non-contact ambience such as wind and room tone. We measure non-transient log-RMS after excluding impact-dominated peaks, together with CLAP similarity to ambience-specific captions. After CFRO, background log-RMS changes only from -30.8 to -31.0 dBFS and ambient-caption CLAP from 0.287 to 0.282, indicating little suppression of legitimate ambience.

Ambient Metric on 76 clips at 20 s⋆\starReferenceTora-live
Background log-RMS in dBFS-30.8-31.0
Ambient-caption CLAP ↑\uparrow0.2870.282

(3) Diversity under fixed initial conditions. We generate eight seeds for each of eight prompts, keeping the initial image, prompt, and motion state fixed so the comparison captures stochastic diversity. After CFRO, pairwise LPIPS changes from 0.137 to 0.133 and audio-embedding distance from 0.126 to 0.123, decreases of just 2.9% and 2.4%. Figure 6 also demonstrates controllable outcome diversity: changing only initial velocity produces rebound versus shattering and different degrees of sandcastle collapse with corresponding sounds.

Method at 5 s across 8 seedsVideo Diversity ↑\uparrowAudio Diversity ↑\uparrow
LTX-2.3†0.1560.142
⋆\starReference0.1370.126
Tora-live0.1330.123

(4) Visual fidelity and Figure 1 examples. The cited cases reveal residual limitations in visual consistency, such as changing die-face markings, and motion/acoustic plausibility, such as prolonged bouncing and the metal-disc response. These are challenging for a four-step distilled, chunk-causal backbone optimized for low-latency streaming. CFRO nevertheless improves timing without sacrificing measured visual quality: TC rises from 0.826 to 0.836 and FVD falls from 254.7 to 245.8 in Table 3, consistent with the CLIP, PhysReal, and audio metrics reported above. Figure 5 provides matched pre- and post-CFRO comparisons, and all newly added qualitative video comparisons (Figures 4–7 and 11) have been included in the updated supplementary material.

Thank you for your response. My concerns have been addressed. I'll raise my score to 3.

We are delighted to receive your response and suggestions. Thank you for raising your evaluation and for your support.

Best regards,

The Tora-live Authors

Reviewer_eax6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses sound drifting out of sync with visible contacts during streaming audio-video generation. Tora-live maintains a compact state containing object motion and contact timing, updates it from generated video and uses it to condition both modalities. A rollout optimization objective further encourages synchronized sound events. The authors introduce ContactBench and evaluate through baseline comparisons, ablations, human judgments, and a second benchmark. Results show improved contact timing, especially over longer sequences.

Critical strengths
  • The paper targets a concrete failure that general perceptual metrics can overlook: sound occurring before, after, or without visible contact. Explicitly representing recent contacts and time to the next contact is a sensible way to address this.
  • The comparison with the same-backbone streaming baseline is convincing. At 30 seconds, Tora-live reduces contact timing error and missed contacts, while also improving temporal consistency.
  • The ablations help identify what matters. Kinematic conditioning alone offers limited improvement, while adding contact-phase information and correcting the state from generated video produces stronger results. This supports the specific design beyond simply adding motion features.
  • The rollout-objective analysis is a strong part of the paper. Removing kinematic consistency improves alignment with scheduled timestamps but worsens alignment with visible contacts. This clearly motivates checking whether the generated motion actually supports the rewarded sounds.
Critical weaknesses
  • The contact evaluator needs more direct validation on generated videos. Human preference agreement and the reference-video measurement floor are useful, but neither establishes event-level timestamp accuracy across methods. Confidence intervals and sensitivity to the matching threshold would help assess the reported timing gains.
  • State estimation pipeline leaves important practical questions unanswered. A single initial frame does not directly provide velocity, and the paper does not explain first-chunk initialization or recovery from failed object masks. The projected-area depth proxy may also confuse rotation, occlusion or camera motion with object movement.
  • Proposed metrics mainly evaluate discrete audio onsets. They do not directly measure whether rolling or scraping sounds persist for the correct duration, change appropriately with motion, or stop when contact ends. This leaves the sustained-contact claims less thoroughly tested than the impact results.
  • Table 1 reports 8 s to generate 5 s of output, but does not provide first-chunk latency, steady-state throughput, or a breakdown including tracking and correction. These measurements matter for assessing the practical benefit of the streaming design.
Questions for authors
  • Where do the ContactBench clips come from, and how were duplicates or overlapping source videos excluded from training? Will the benchmark, annotations, and evaluation code be released?
  • How accurately does the final evaluator recover contact and onset timestamps on manually annotated generated videos? Do the main conclusions hold across matching thresholds and with confidence intervals?
  • How are the initial velocity and first chunk initialized? What happens when tracking fails or projected object area changes because of rotation, occlusion, or camera movement?
  • Can you evaluate sound duration and cessation during sustained rolling or scraping, in addition to discrete onset timing?
  • Does the reported runtime include segmentation, decoding, state extraction, and correction? What are the first-chunk latency and sustained throughput over a 30-second rollout?
3 comments
AuthorsOpenReview ↗

Response to Reviewer eax6 (Part 1/2)

[Q1] Where do the ContactBench clips come from, how were duplicates excluded, and will the benchmark be released?

[A1] We added ContactBench construction details to Section 4.1 and Appendix B.2. The 240 clips come from Pexels (88), OpenVid1M (104), and self-recordings (48). We retain trackable rigid-body contacts and exclude masking speech or music while preserving ambience. Source-level separation from the 1.4M training corpus combines video ID/URL exclusion with DINOv2 similarity screening above 0.85 and CLAP screening above 0.80. ContactBench was not used for tuning or checkpoint selection. We plan to release the benchmark upon acceptance.

[Q2] How accurately does the evaluator recover timestamps on generated videos, and do conclusions hold across thresholds and with confidence intervals?

[A2] We added human–automatic validation on 105 generated clips, with 15 clips from each of the seven Table 1 methods except RealWonder + MMAudio, spanning all six categories. Against human-derived clip metrics, the evaluator achieves 0.016 s CTE MAE and Pearson correlations of 0.878–0.912. This complements the reference-video measurement floor in Table 6.

MetricAuto–Human MAE ↓\downarrowClip-level Pearson r↑r \uparrow
CTE0.016 s0.912
MCR0.0420.888
SOR0.0460.878

We added a threshold-sensitivity analysis on all 240 ContactBench clips at 5 s for δmatch∈{0.10,0.20,0.25} s\delta_{\mathrm{match}}\in\{0.10,0.20,0.25\}\,\mathrm{s}, with 95% bootstrap CIs. In paper, we choose 0.20 s to balance contact/onset localization uncertainty against false event associations. A 0.10 s window is more sensitive to jitter, while 0.25 s permits looser matches. Tora-live ranks first in CTE, MCR, and SOR at all three thresholds.

Threshold δmatch\delta_{\text{match}}MethodCTE ↓\downarrowMCR ↓\downarrowSOR ↓\downarrow
0.10 sOmniForcing0.081 ± 0.0050.584 ± 0.0360.591 ± 0.041
LTX-2.30.065 ± 0.0030.392 ± 0.0340.376 ± 0.030
LTX-2.3†0.058 ± 0.0040.356 ± 0.0290.304 ± 0.031
Tora-live0.045 ± 0.0030.271 ± 0.0270.243 ± 0.024
0.20 sOmniForcing0.158 ± 0.0090.418 ± 0.0370.428 ± 0.034
LTX-2.30.108 ± 0.0060.269 ± 0.0260.254 ± 0.029
LTX-2.3†0.093 ± 0.0060.232 ± 0.0280.189 ± 0.023
Tora-live0.072 ± 0.0040.182 ± 0.0220.158 ± 0.023
0.25 sOmniForcing0.182 ± 0.0080.369 ± 0.0320.385 ± 0.037
LTX-2.30.121 ± 0.0090.241 ± 0.0290.228 ± 0.025
LTX-2.3†0.103 ± 0.0060.198 ± 0.0240.165 ± 0.024
Tora-live0.079 ± 0.0050.159 ± 0.0230.137 ± 0.019

[Q3] How are initial velocity and the first chunk initialized, and how does the model handle rotation, occlusion, or camera movement?

[A3] Appendix A.2 details initialization: SAM3 estimates first-frame position and orientation, and Qwen3.5-Omni predicts prompt-conditioned translational and angular velocity priors. Velocity can also be user-specified (Figure 6). Projected-area smoothing and the second-order prior mitigate rotation-induced depth fluctuations. If tracking fails, we set λcorr=0\lambda_{\mathrm{corr}}=0 until observations recover, triggered by segmentation confidence below 0.5 or mask-area jumps above 40%. Chunk-local support updates in Equation 11 accommodate camera motion.

We evaluated 157 Standard and 83 Challenging ContactBench clips. Challenging cases satisfy at least one of three criteria: over 15% projected-area variation, frame-to-frame mask-area jumps above 40%, or median background optical flow above 1.5 px/frame. On these 83 clips, Tora-live reduces 5 s CTE from 0.114 to 0.080 and MCR from 0.264 to 0.203 over LTX-2.3†. Figure 7 and the updated supplementary material further show 30 s rollouts under occlusion and camera motion.

Subset at 5 sMethodCTE ↓\downarrowMCR ↓\downarrowSOR ↓\downarrowFGAS ↑\uparrow
Standard, 157 clipsLTX-2.30.1060.2610.2460.216
LTX-2.3†0.0820.2150.1740.226
Tora-live0.0680.1710.1490.249
Challenging, 83 clipsLTX-2.30.1120.2840.2690.201
LTX-2.3†0.1140.2640.2170.197
Tora-live0.0800.2030.1750.232
AuthorsOpenReview ↗

Response to Reviewer eax6 (Part 2/2)

[Q4] Can you evaluate sound duration and cessation during sustained rolling or scraping?

[A4] We added Cessation Error and Duration Error on 72 rolling and scraping clips at 5 s, measuring mismatches in stopping time and active sound duration. Visual stopping requires SAM3 centroid velocity below 1.0 px/frame for five frames, verified by Qwen3.5-Omni. Acoustic stopping requires a 30 dB drop in the 24 kHz short-time RMS envelope (hop 512) for at least 0.1 s. Tora-live reduces cessation error from 0.284 s to 0.117 s and duration error from 0.326 s to 0.146 s, improvements of 58.8% and 55.2% over LTX-2.3†.

Method on 72 clipsCTE ↓\downarrowMCR ↓\downarrowSOR ↓\downarrowFGAS ↑\uparrowCessation Error ↓\downarrowDuration Error ↓\downarrow
LTX-2.30.1160.2870.2740.1960.269 s0.341 s
LTX-2.3†0.1030.2530.2180.2030.284 s0.326 s
Tora-live0.0890.2200.1990.2290.117 s0.146 s

[Q5] Does reported runtime include all state modules, and what are first-chunk latency and 30 s throughput?

[A5] Table 1 reports end-to-end 480P runtime on one H100, including VAE decoding, SAM3 segmentation, state extraction, correction, and conditioning. Tora-live takes 8.0 s per 5 s output versus 7.4 s for LTX-2.3†. The 0.6 s overhead comprises 0.36 s segmentation, 0.04 s state extraction, 0.02 s correction, and 0.18 s state injection (Section 4.2). We additionally measure 2.08 s time-to-first-chunk and 16.1 FPS steady-state throughput over 30 s with a fixed-window KV cache.

[Ethics Clarification]

[A-Ethics] We added an Ethics Statement. The study uses non-sensitive object videos without personal or biometric data. Participants gave informed consent and received compensation above the local statutory minimum wage. The study was treated as exempt under institutional guidelines.

Dear Reviewer,

Thank you for your time and effort in reviewing our paper. As the discussion phase draws to a close, we would like to know whether our response and revisions have addressed the concerns you raised. We would also welcome any further feedback you may have and would be happy to address any remaining concerns.

Thank you again for your consideration and for your valuable comments.

Best regards, The Tora-live Authors

Reviewer_TcxQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Tora-live generates streaming audio and video for rigid-body contact. For one dominant moving object it keeps a 10-D state: position, velocity, orientation, spin, time since the last contact, and time to the next. Each chunk rolls this state forward, corrects it from the generated frames, and injects it into both video and audio. A rollout reward aligns audio onsets with the scheduled contacts. On ContactBench (240 clips, six contact types), contact timing improves over the same streaming backbone, and the gap grows from 5 s to 30 s.

Critical strengths

The problem is well defined. In long chunk-wise generation, contact sounds become late, missing, or spurious faster than the video itself becomes implausible. Separating this contact-phase drift from perceptual quality is useful, and it is the reason I lean accept.

Critical weaknesses

The evidence stays inside scenes with one dominant moving object. The other body is usually static, such as the floor, a wall, a bell, or a block. I do not see a result with two moving objects, or with one contact setting off the next. For a paper about physical contact, this is a narrow slice, and it is what keeps the score at a weak accept.

Questions for authors
  1. Did you evaluate any scene with more than one moving object, or where contact is passed from one body to another? If so, please point me to it. If not, please say so. This changes how broadly I read the benchmark.
  2. The state only times a contact sound for one rigid body. Can the same idea carry other physical outcomes? For example, when an object falls, can speed, mass, and material predict whether it breaks? Please say whether that is in scope, or whether the method is specific to scheduling one onset.
2 comments
AuthorsOpenReview ↗

Response to Reviewer TcxQ

[Q1] Did you evaluate any scene with more than one moving object, or where contact is passed from one body to another?

[A1] ContactBench focuses on one dominant moving body to attribute contacts to a tracked trajectory. To evaluate generalization beyond this assumption, we added 30 multi-object cases at 5 s in Appendix D.5, with quantitative results in Table 9 and visualizations in Figure 11 (full rollouts in the updated supplementary material). The prompts, generated with Gemini-4-Argon and verified by humans, cover simultaneous multi-body motion, direct two-body collisions, and multi-hop contact chains. To distinguish directly grounded interactions from secondary transfers, we separately measure timing error for contacts involving the tracked primary object (CTEpri\mathrm{CTE}_{\mathrm{pri}}) and contacts occurring only among untracked bodies (CTEother\mathrm{CTE}_{\mathrm{other}}).

Tora-live achieves the best overall alignment, lowering CTEall\mathrm{CTE}_{\mathrm{all}} from 0.118 to 0.097 and CTEpri\mathrm{CTE}_{\mathrm{pri}} from 0.111 to 0.083 relative to LTX-2.3†. Figure 11 illustrates this distinction through simultaneous free fall, billiard collisions, and a Newton's cradle. Co-motion and first-hop two-body contacts remain well aligned, whereas later transfers between untracked bodies can drift. Secondary-contact CTE improves only from 0.134 to 0.131, showing where explicit multi-object relational states would further extend the approach (Appendix E).

MethodCTEall↓\mathrm{CTE}_{\mathrm{all}} \downarrowCTEpri↓\mathrm{CTE}_{\mathrm{pri}} \downarrowCTEother↓\mathrm{CTE}_{\mathrm{other}} \downarrowMCR↓\mathrm{MCR} \downarrowSOR↓\mathrm{SOR} \downarrowFGAS↑\mathrm{FGAS} \uparrow
OmniForcing0.1810.1740.1970.4720.4640.158
LTX-2.30.1250.1230.1290.3120.2980.194
LTX-2.3†0.1180.1110.1340.2860.2470.191
Tora-live0.0970.0830.1310.2410.2030.219

[Q2] Can the same idea carry other physical outcomes beyond scheduling one contact sound?

[A2] Yes. Our 10-D contact-aware state represents evolving motion and contact, rather than merely scheduling acoustic onsets. The state jointly carries kinematics, recent or ongoing contact, and time to the next contact. During rollout, the contact regime and restitution update affect subsequent motion (Equations 3 and 16). Current-state modulation and history attention then inject this evolving state into both audio and video DiT blocks (Equations 5–6). The input image and prompt provide geometry and material cues, while joint training on 1.4M real-world clips links motion conditions to post-contact visual and acoustic outcomes.

We added two controlled initial-velocity comparisons in Section 4.4, Figure 6, and the updated supplementary material, using identical initial frames and prompts. At higher drop speed, an ice block changes from an intact rebound with a dull thud to shattering with sharp cracking transients. A faster ball impact causes greater sandcastle collapse and stronger granular scattering audio. These examples show state-dependent outcomes beyond onset timing, without claiming calibrated fracture thresholds.

Dear Reviewer,

We hope that our detailed responses and new experiments have effectively addressed your comments. Since the discussion window will be closing soon, we look forward to your feedback and remain available to address any follow-up questions. We would greatly appreciate it if our clarifications meet your expectations.

Many thanks once again for your time and thoughtful review of our manuscript.

Sincerely,

The Tora-live Authors

Reviewer_sG3yOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses contact–sound misalignment in streaming audio-video generation. Tora-live predicts short-term motion and contact parameters, uses them to construct a compact state, and conditions both video and audio generation on that state. After each chunk, the state is corrected using the generated video. The method also uses a contact-timing reward and a chunk-end motion consistency reward. Experiments on ContactBench, longer sequences, human preference tests, and the physical-interaction subset of VABench show improved synchronization.

Critical strengths
  1. The shared contact-aware state is a useful way to connect motion and sound. It combines motion information with recent or ongoing contact and the time until the next predicted contact. This gives both video and audio generation an explicit, interpretable reference for contact timing. I find this a meaningful design contribution, with the state-content ablations in Table 3 supporting the value of the contact information.
  2. The online prediction-and-correction loop is well suited to streaming generation. The method predicts a short state sequence, uses it to guide generation, and then corrects it using the generated video before continuing. This allows later chunks to adjust to the motion actually produced, without requiring a complete motion trajectory in advance. The combination of state prediction and visual feedback is a thoughtful part of the method, and the closed-loop ablations support its usefulness.
Critical weaknesses
  1. The compact state representation limits the range of interactions the method can handle. The method focuses on a dominant moving object and does not explicitly represent contact propagation among multiple interacting objects.
  2. State-estimation accuracy remains a limitation. The useful oracle-state analysis in Table 6 shows that replacing the estimated state with a manually verified trajectory improves CTE from 0.099 to 0.078 and MCR from 0.231 to 0.168. This indicates that state estimation remains a bottleneck despite the closed-loop correction. The proposed representation is effective, but its benefits still depend on how accurately motion and contact can be recovered.
Questions for authors
  1. How does the state update handle an object settling on a surface? Could the authors briefly walk through the transition from repeated bouncing to rest, explaining how the REST/SUSTAIN labels relate to the motion updates in Appendix A.2? A simple illustrative example would be helpful.

  2. Is the dominant object fixed throughout a generated sequence? Appendix A.2 describes selecting the object in the first frame, while the closed-loop correction re-localizes it after each chunk. Could the authors clarify whether this always tracks the initially selected object, or whether the selected object can change during generation? This would help clarify the scope of the current implementation.

2 comments
AuthorsOpenReview ↗

Response to Reviewer sG3y

[Q1] How does the state update handle an object settling on a surface?

[A1] The transition from bouncing to rest is handled by the interleaved kinematic rollout, velocity update, and regime classification in Appendix A.2 and Equations 3 and 11–16.

Consider a ball bouncing on a surface. During free flight, the state evolves under the dynamics prior in Equation 3 in the FREE regime. When signed clearance crosses the support boundary with an approaching normal velocity, a contact is scheduled and velocity is updated via Equation 16. Importantly, effective restitution is attenuated as normal approach speed decreases, producing progressively weaker rebounds. As the ball settles near the surface, Equation 14 distinguishes two regimes by normalized velocity. SUSTAIN corresponds to small normal velocity with appreciable tangential motion, whereas REST occurs when both components fall below their thresholds. Thus, an object with appreciable tangential translational motion is classified as SUSTAIN, while an object with both normal and tangential velocities below their respective thresholds enters REST. The backward contact field in Equation 15 captures residual or ongoing contact to condition both audio and video. REST and SUSTAIN are inferred contact regimes rather than explicit commands that zero out velocity. Motion decays through the dynamics prior and contact updates, while the regime labels characterize the evolving contact state and provide contact-phase cues for both audio and video generation.

[Q2] Is the dominant object fixed throughout a generated sequence?

[A2] Yes. The dominant object identity is fixed throughout generation and is not dynamically reassigned. SAM3 segments the dominant moving object in the first frame to initialize position, scale, and orientation. At subsequent chunk boundaries, closed-loop correction re-localizes this same object and fuses its observed kinematics with the predicted state, as detailed in Section 3.2, Equation 4, and Appendix A.2. Under partial occlusion, we set λcorr=0\lambda_{\text{corr}}=0 and retain the propagated state until reliable observations return.

To clarify this scope, we evaluated 30 multi-object scenes at 5 s in Appendix D.5, Table 9, Figure 11, and the updated supplementary material. We separate contacts involving the tracked primary object (CTEpri\mathrm{CTE}_{\mathrm{pri}}) from those among untracked bodies (CTEother\mathrm{CTE}_{\mathrm{other}}). Tora-live lowers CTEall\mathrm{CTE}_{\mathrm{all}} from 0.118 to 0.097 and CTEpri\mathrm{CTE}_{\mathrm{pri}} from 0.111 to 0.083 over LTX-2.3†. Co-motion and first-hop collisions remain well synchronized, whereas contacts among untracked bodies show limited gain, with CTEother\mathrm{CTE}_{\mathrm{other}} moving from 0.134 to 0.131. Its CTEother\mathrm{CTE}_{\mathrm{other}} also remains comparable to the ungrounded LTX-2.3 baseline (0.129), consistent with the limitation of not explicitly modeling interactions among untracked objects. The oracle-state gap in Table 6 (Appendix D.1) further highlights the remaining room for improving visual state estimation, while extending the representation to multi-object relational states remains an important future direction (Appendix E).

MethodCTEall↓\text{CTE}_{\text{all}} \downarrowCTEpri↓\text{CTE}_{\text{pri}} \downarrowCTEother↓\text{CTE}_{\text{other}} \downarrowMCR↓\text{MCR} \downarrowSOR↓\text{SOR} \downarrowFGAS↑\text{FGAS} \uparrow
OmniForcing0.1810.1740.1970.4720.4640.158
LTX-2.30.1250.1230.1290.3120.2980.194
LTX-2.3†0.1180.1110.1340.2860.2470.191
Tora-live0.0970.0830.1310.2410.2030.219

[Ethics Clarification] We added an Ethics Statement to the paper. The annotation and human preference study involved only non-sensitive object videos without personal or biometric data, and qualified for exemption under institutional guidelines. All participants provided informed consent and received fair compensation for their time.

Thank you once again for stepping in to review our paper and for your encouraging feedback.

Dear Reviewer,

We hope our response has addressed your questions. As the discussion phase is coming to a close, we are looking forward to your feedback and would like to know if you have any remaining concerns we can address. We are grateful if you find our revisions satisfactory.

Thank you once again for the time and effort you have dedicated to reviewing our paper.

Best regards!

The Tora-live Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all reviewers for their constructive feedback. We are encouraged that the reviewers found our formulation of contact-phase drift well-defined and practically important (TcxQ, eax6), noted that our contact-aware state and closed-loop visual correction align well with maintaining long-horizon sound–motion correspondence (eax6, arHi), and highlighted our same-backbone comparisons, ablations, and rollout-objective analysis as convincing empirical support (TcxQ, eax6, arHi).

Below, we briefly summarize our responses to shared concerns and list the revisions in the updated PDF and supplementary material.

Shared concerns

  • ContactBench construction and independence (eax6, arHi). Section 4.1 and Appendix B.2 now document the 240 clips collected from OpenVid1M, Pexels, and self-recordings, all verified by three independent annotators. Source-level separation from the 1.4M training set is enforced via video ID and URL filtering alongside DINOv2 and CLAP similarity screening. We plan to release the benchmark upon acceptance.
  • Interaction scope, initialization, and tracking robustness (TcxQ, eax6). Section 4.4 and Figure 6 show that varying initial velocity controls post-contact outcomes such as intact rebound versus shattering, while Appendix D.5, Table 9, and Figure 11 evaluate 30 multi-object scenes covering co-motion, two-body collisions, and multi-hop chains. Appendix A.2 further details first-chunk initialization via Qwen3.5-Omni velocity priors and open-loop fallback under unreliable masks, and Section 4.4 and Figure 7 verify robustness on 83 challenging clips with rotation, occlusion, or camera motion.
  • Evaluator validation and additional metrics (eax6, arHi). On 105 generated clips, our automatic evaluator closely matches human annotations, achieving 0.016 s CTE MAE and Pearson correlations of 0.878–0.912, and Tora-live consistently ranks first across matching thresholds of 0.10 s, 0.20 s, and 0.25 s with 95% confidence intervals. On 72 rolling and scraping clips, Tora-live reduces cessation and duration errors by 58.8% and 55.2% over LTX-2.3†, and also leads under the FlatSounds evaluation protocol.
  • Physics-aware baselines, runtime, and post-CFRO quality (eax6, arHi). Section 4.2 breaks down the 0.6 s state-grounding overhead per 5 s clip, and we additionally measure a 2.08 s first-chunk latency and 16.1 FPS steady-state throughput over 30 s rollouts. We also add the physics-guided streaming baseline RealWonder + MMAudio-Large in Table 1 and Figure 4, and verify that CFRO improves synchronization and physical plausibility with negligible impact on background ambience and multi-seed diversity.

Main revisions in the uploaded PDF (changes in blue) and supplementary material

  • Section 2: Discussion of PAVAS and FlatSounds (arHi).
  • Section 4.1 and Appendix B.2: ContactBench sources, selection criteria, multi-annotator verification, and deduplication against training data (eax6, arHi).
  • Section 4.2, Table 1, and Figure 4: RealWonder + MMAudio-Large baseline and module-level breakdown of the 0.6 s runtime overhead (eax6, arHi).
  • Section 4.4, Figure 6, and Figure 7: Initial-velocity outcome control and evaluation under rotation, occlusion, and camera motion (TcxQ, eax6).
  • Ethics Statement on Page 11: Ethics exemption, participant consent, and compensation details (TcxQ, eax6).
  • Appendix A.2: First-frame velocity initialization and open-loop fallback under tracking disturbances (eax6).
  • Appendix D.5, Table 9, Figure 11, and Appendix E: Evaluation on 30 multi-object scenes separating primary and secondary contacts, together with expanded limitations (TcxQ).
  • Updated Supplementary Material: Full video rollouts for all newly added qualitative results in Figures 4–7 and 11 (TcxQ, eax6, arHi).

Our detailed response to each reviewer is posted under each review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.