- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
- This paper studies streaming joint audio–video generation in rigid-body scenes where physical contact is the primary cause of sound events. The authors identify contact-phase drift, in which the timing of visible contacts and generated sounds becomes increasingly misaligned during autoregressive generation across chunks, and propose a method to mitigate this problem.
- The proposed method, Tora-live, builds on a four-step, chunk-causal adaptation of LTX-2.3 that generates synchronized one-second audio–video chunks. It introduces a 10-dimensional state comprising the dominant object's position and velocity proxies, in-plane orientation, angular rate, a field representing recent and ongoing contact, and the time to the next scheduled contact. A lightweight dynamics model propagates this state within each chunk, and both modalities are conditioned on it through current-state modulation and attention to recent state history. After a chunk is generated, observations from its video frames are used to correct the state and initialize the next chunk. The method further applies contact-focused rollout optimization (CFRO), using Flow-GRPO to reward alignment between audio onsets and scheduled contacts, together with consistency between generated motion and the predicted state.
- For evaluation, the authors introduce ContactBench, comprising 240 videos across six rigid-motion regimes. They report comparisons with offline and streaming baselines, generation results up to 30 seconds, component ablations, human evaluation, and an additional evaluation on 47 cases from VABench's physical-interaction subset. The reported results show improved contact alignment and greater robustness to increasing rollout length, while maintaining competitive perceptual quality in the evaluated scenes.
Critical strengths
- The method combines a contact-aware state representation with feedback from generated video. The contact-phase variables , short-term state history, and closed-loop correction are well aligned with the goal of maintaining sound–motion correspondence during long rollouts. In particular, correcting the state using generated video helps keep subsequent contact schedules consistent with the motion actually produced by the model. The component ablations in Table 3 shows the usefulness of these design choices, providing empirical evidence for their role in mitigating contact-timing drift.
Critical weaknesses
- ContactBench is insufficiently documented to assess it as a standalone benchmark contribution. Although the paper reports the number of videos, categories, and annotation types, several essential aspects of benchmark construction remain unclear: the sources of the 240 videos, collection and selection criteria, annotation and verification procedures, source-video-level deduplication against the training data, and separation from data used for validation or model selection. These omissions are consequential because ContactBench is presented as a central contribution and provides the primary evidence for the effectiveness of a model trained on a specially curated rigid-motion dataset. Without this information, it is difficult to assess potential data overlap, the independence of the evaluation, and the reproducibility of the results. The authors should document these procedures in sufficient detail to substantiate both the benchmark contribution and the experimental claims.
- The empirical evaluation is limited in scope and does not adequately establish overall generation quality after contact-focused adaptation. Table 3 shows improvements in synchronization and perceptual-quality metrics on ContactBench after CFRO. However, the external evaluation is restricted to only 47 cases from the physical-interaction subset of VABench. This small evaluation within a closely related domain provides limited independent evidence of robustness and generalization. In particular, the experiments do not establish whether contact-focused adaptation preserves legitimate sounds unrelated to contact events, or audio and motion diversity across samples generated from the same prompt. Because CFRO penalizes onsets that are not matched to scheduled contacts, it may suppress valid background sounds in scenes containing both contact sounds and other audio sources. KL regularization and the kinematic-consistency reward provide useful constraints, but their presence alone does not demonstrate that these capabilities are retained. My inspection of the provided demos also raises concerns about overall physical plausibility and audio–visual semantic consistency, despite convincing contact timing in some examples: Figure1_videos/case2: The bouncing motion in the middle of the clip appears unnaturally prolonged and physically implausible., Figure1_videos/case3: The clip is silent, and the die barely rotates even as its visible face markings change, producing an unnatural visual result., Figure1_videos/case4: The sound does not semantically match the depicted event, and the motion of the metal disc appears physically implausible. I also checked other samples from Figure4_vides~Figure7_videos. A broader evaluation of physical plausibility and audio–visual semantic consistency, together with matched comparisons before and after CFRO, is needed to determine whether the gains in contact synchronization preserve other aspects of generation quality and diversity.
Questions for authors
-
ContactBench construction and annotation: Please clarify the sources of the ContactBench videos, the collection and selection procedures, and how separation from training and validation data was ensured at the source-video level. How many annotators verified the contact timestamps and state trajectories, what criteria did they follow, and how were disagreements and uncertainty in event timing handled?
-
Relationship to PAVAS and the cascaded baseline: PAVAS [1] estimates object-level mass and speed, combines these quantities with object-specific features, and conditions an audio generator through modulation. How does Tora-live relate to this approach, and what are its additional technical contributions beyond explicit conditioning on object-level physical quantities? Using the same generated videos, would replacing MMAudio with PAVAS in the cascaded baseline change the conclusions regarding contact alignment and audio quality?
-
Relationship to the FlatSounds metrics: FlatSounds [2] already evaluates contact- and impact-related acoustic events using Timing Error, Hit Coverage, and Perfect Alignment. Could you clarify how the contact-centric metrics in this paper relate to these measures? In particular, how do CTE and MCR differ from Timing Error and Hit Coverage in their definitions, event matching, and applicability, and what additional failure modes does the proposed evaluation protocol capture?
[1] Hyun-Bin, Oh, et al. "PAVAS: Physics-Aware Video-to-Audio Synthesis." CVPR 2026.
[2] Li, Tingle, et al. "Benchmarking Single-Factor Physical Video-to-Audio Generation." ECCV 2026.
4 comments
Response to Reviewer arHi (Part 2/2)
[Q4] Overall generation quality, preservation of non-contact background sounds, diversity after CFRO, and demo inspection
[A4] Table 7 shows that CFRO aligns sounds with realized contacts rather than blindly rewarding scheduled onsets: removing improves schedule alignment from 0.862 to 0.931 but worsens visual-contact CTE from 0.099 to 0.118. We compare the same state-grounded model before (Reference) and after CFRO for quality, ambience, and diversity. For ambience-specific analysis, we select 76 clips with sustained natural ambience based on ContactBench annotations.
(1) Perceptual quality, audiovisual semantics, and physical plausibility. Across 240 clips at 20 s, CFRO improves CTE from 0.112 to 0.099, CLIP from 0.276 to 0.278, CLAP from 0.615 to 0.618, FAD from 1.85 to 1.79, and PhysReal from 0.689 to 0.706. The gains also hold on this annotation-defined ambient subset.
| Subset at 20 s | Method | CTE | CLIP | CLAP | PhysReal | FAD |
|---|---|---|---|---|---|---|
| All 240 clips | LTX-2.3† | 0.132 | 0.268 | 0.592 | 0.635 | 1.95 |
| Reference | 0.112 | 0.276 | 0.615 | 0.689 | 1.85 | |
| Tora-live | 0.099 | 0.278 | 0.618 | 0.706 | 1.79 | |
| Ambient 76 clips | Reference | 0.116 | 0.273 | 0.608 | 0.681 | 1.91 |
| Tora-live | 0.103 | 0.274 | 0.611 | 0.698 | 1.87 |
(2) Preservation of legitimate background sounds. On these 76 annotated clips, we examine sustained non-contact ambience such as wind and room tone. We measure non-transient log-RMS after excluding impact-dominated peaks, together with CLAP similarity to ambience-specific captions. After CFRO, background log-RMS changes only from -30.8 to -31.0 dBFS and ambient-caption CLAP from 0.287 to 0.282, indicating little suppression of legitimate ambience.
| Ambient Metric on 76 clips at 20 s | Reference | Tora-live |
|---|---|---|
| Background log-RMS in dBFS | -30.8 | -31.0 |
| Ambient-caption CLAP | 0.287 | 0.282 |
(3) Diversity under fixed initial conditions. We generate eight seeds for each of eight prompts, keeping the initial image, prompt, and motion state fixed so the comparison captures stochastic diversity. After CFRO, pairwise LPIPS changes from 0.137 to 0.133 and audio-embedding distance from 0.126 to 0.123, decreases of just 2.9% and 2.4%. Figure 6 also demonstrates controllable outcome diversity: changing only initial velocity produces rebound versus shattering and different degrees of sandcastle collapse with corresponding sounds.
| Method at 5 s across 8 seeds | Video Diversity | Audio Diversity |
|---|---|---|
| LTX-2.3† | 0.156 | 0.142 |
| Reference | 0.137 | 0.126 |
| Tora-live | 0.133 | 0.123 |
(4) Visual fidelity and Figure 1 examples. The cited cases reveal residual limitations in visual consistency, such as changing die-face markings, and motion/acoustic plausibility, such as prolonged bouncing and the metal-disc response. These are challenging for a four-step distilled, chunk-causal backbone optimized for low-latency streaming. CFRO nevertheless improves timing without sacrificing measured visual quality: TC rises from 0.826 to 0.836 and FVD falls from 254.7 to 245.8 in Table 3, consistent with the CLIP, PhysReal, and audio metrics reported above. Figure 5 provides matched pre- and post-CFRO comparisons, and all newly added qualitative video comparisons (Figures 4–7 and 11) have been included in the updated supplementary material.
Thank you for your response. My concerns have been addressed. I'll raise my score to 3.
We are delighted to receive your response and suggestions. Thank you for raising your evaluation and for your support.
Best regards,
The Tora-live Authors
Response to Reviewer arHi (Part 1/2)
[Q1] ContactBench construction, annotation verification, and source-level separation
[A1] We expanded Section 4.1 and Appendix B.2 with ContactBench sources, selection, and annotation: 240 clips from Pexels (88), OpenVid1M (104), and self-recordings (48) across six regimes. Clips with ambiguous contacts or masking speech/music were excluded. To ensure independence from training and validation, we applied video ID/URL filtering, DINOv2 similarity screening above 0.85, and CLAP screening above 0.80. The benchmark was not used for tuning. Three annotators independently verified timestamps and trajectories, merged contacts agreeing within 1–2 frames by their median, corrected masks, and excluded unresolved cases. We plan to release the benchmark subject to source licenses.
[Q2] Relationship to PAVAS and comparison with physics-aware cascaded baselines
[A2] We added PAVAS to Section 2. PAVAS estimates mass and 3D velocity from existing video to condition audio. Tora-live instead targets contact-phase drift in joint causal generation using scale-normalized motion proxies and contact-phase variables, corrected from generated frames and injected into both modalities (Equations 3–6). Table 3(a) shows kinematics alone reduce 20 s CTE from 0.132 to 0.129, while the full state reaches 0.112 and CFRO reaches 0.099.
A direct PAVAS substitution was not feasible as its model weights are not publicly released. Moreover, its offline physical estimation takes 7.16 s per 10 s video, including 4.82 s CUT3R reconstruction. To test physics-guided streaming, we added RealWonder[1] + MMAudio-Large in Table 1 and Figure 4. RealWonder reconstructs a 3D scene and runs physics simulation for streaming video, then MMAudio synthesizes audio independently per chunk. Its 14.3 s upfront reconstruction contributes to 28.4 s for 5 s output. We use RealWonder’s PhysReal metric with its original prompt and scoring protocol, replacing GPT-4o with Gemini-4-Argon for all methods. RealWonder + MMAudio improves PhysReal over LTX-2.3† (0.708 vs. 0.664) but yields 0.091 CTE and 2.31 FAD. Tora-live reaches 0.072 CTE and 1.71 FAD in 8.0 s, supporting joint contact-aware conditioning without heavy reconstruction.
[Q3] Relationship to the FlatSounds temporal alignment metrics
[A3] We added FlatSounds (Li et al., 2026c) to Section 2 and evaluated its temporal metrics on generated videos. The protocols differ in three ways. (1) Reference: FlatSounds uses fixed input-video contacts, whereas we measure realized contacts in jointly generated videos. (2) Event matching: FlatSounds uses nearest-onset coverage, while our fixed-tolerance one-to-one matching prevents one onset from covering multiple contacts. (3) Failure coverage: FlatSounds reports Timing Error, Hit Coverage, and Perfect Alignment, while our SOR additionally identifies spurious onsets even when every visible contact is covered. Tora-live leads under both protocols, raising Perfect Alignment from 43.3% to 51.2% over LTX-2.3† at 5 s.
[1]Wei Liu, et al. "RealWonder: Real-Time Physical Action-Conditioned Video Generation"