acceptodds
Withdrawn from ICLR 2027

Sonic-AMG: Streaming Human Motion Generation from Spatial Audio

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modeling human reactions to spatial audio requires understanding both what is heard and where it comes from. Prior work generates motion offline from complete acoustic and spatial input sequences. Humans, however, react to sounds without knowing what they will hear next. We present Sonic-AMG, the first autoregressive framework for generating human motion from streaming spatial audio. The model encodes acoustic content and body-relative source geometry as separate tokens. A causal transformer with a flow head generates continuous motion latents, which a causal decoder reconstructs into motion segments. Teacher-forced training uses motion histories and body-relative source locations derived from recorded motion, whereas streaming inference relies on generated motion for both. We introduce *rehearsal training*, which combines full-sequence teacher-forced supervision with supervised prediction on histories constructed through short autoregressive self-rollouts, improving robustness to accumulated prediction errors. To keep spatial conditioning consistent with generated motion, we propose *spatial feedback*, which recomputes body-relative source locations from generated root poses during both rehearsal training and inference. On SAM, Sonic-AMG achieves state-of-the-art FID and R-precision. It further supports real-time online generation of coherent reactions across sound events and source locations. Our code and model will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human Motion Generation, Spatial Audio, Autoregressive Modeling, Causal Generation, Multimodal Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_T6DwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Sonic-AMG, an autoregressive model for generating human reactions as spatial audio arrives. The main model takes binaural audio, a known world-space sound-source position, a reaction-style label, and an initial motion segment. In the SAM evaluation, this segment consists of four ground-truth frames. It encodes acoustic content and body-relative source geometry as separate what/where tokens, then generates four-frame motion blocks with a causal Transformer, a rectified-flow head, and a frozen causal temporal VAE. Rehearsal training combines full-sequence teacher-forced supervision with supervision on short, 16-step self-rollouts. Spatial feedback updates the source geometry from generated root poses during these rollouts and inference. The experiments use the existing SAM dataset. The model achieves FID/R@1 of 6.505/0.957 under the GT-SSL comparison protocol (Table 2) and 8.714/0.909 under the Online protocol (Tables 3 and 4). The paper also includes an audio-derived No-SSL variant and qualitative examples spanning successive sound events.

Critical strengths
  1. The paper identifies a concrete problem in streaming spatial-audio generation and tests its proposed solution directly. Once the body moves, the source's body-relative location changes too, so replacing ground-truth motion histories during training is not enough if the spatial conditions still come from recorded motion. Table 4 compares two rehearsal-trained models under the same Online evaluation protocol: training with spatial feedback improves FID from 25.872 to 8.714 and R@1 from 0.714 to 0.909. This is convincing evidence for updating spatial conditions during training, not just at inference (Section 4.3).

  2. The rehearsal ablations are also useful and reasonably targeted. All variants in Table 3 share the generator architecture, and the Online two-forward baselines also update spatial conditions. Compared with the Online two-forward baseline with the lowest FID, rehearsal improves FID/R@1 from 11.766/0.887 to 8.714/0.909. Rollout Only with History reaches 12.320/0.891, which supports keeping full-sequence supervision alongside the short rollouts. Table 5 gives further support for the separate conditioning tokens and learned acoustic features. The No-SSL extension in Appendix D is useful as well: Table 8 reports 11.917/0.915 without source-location inputs at inference, although it still uses geometric supervision during training and does not use spatial feedback. Together, these results support the proposed streaming model within the SAM setting.

Critical weaknesses

The actual incremental value of the fine-grained audio and the clarity of the task formulation are not fully established. While I agree that the inherent ambiguity of how people react to sound doesn't invalidate conditional motion generation, it raises a critical question: how much meaningful reaction information does the fine-grained audio actually provide once we already supply the reaction-style label c, the sound source position g, and the four initial seed frames (or motion history)? Note that c is an annotated reaction-style label, and while it might not completely dictate every single action, its precise semantics and the constraints it places on reaction selection remain quite vague. Table 5's ablations on feature replacement and fusion show that the representation design works, but they do not isolate the actual incremental benefits of the fine-grained audio over simpler baselines (such as just using event labels combined with style, spatial, and history information). Similarly, the No-SSL variant in Appendix D changes both the spatial condition provider and the semantic encoding at the same time, failing to isolate this issue. Furthermore, since FID only measures how well the motion distribution matches the paired dataset, and R@ measures alignment in the joint embedding space, neither of them directly guarantees that the generated motion is actually "situationally appropriate." Even with metric improvements across 27 scenarios and 3 styles, this evidence is not quite enough to back up the paper's core claim about generating highly meaningful reactions to sound (pp. 4–7, 9, and 21–23).

Questions for authors

Could you provide more details on how the system handles transition boundaries? Specifically, during cross-event root-alignment, how are the motion histories, acoustic/Transformer caches, and decoder states managed? Also, how does the two-pose timing configuration relate to the four-frame pooling used for quality evaluation? Is the postprocessing shown in the comparison videos also applied during quantitative evaluation, does it feed back into the model, or does it require future frames (which would affect online streaming)? If you have end-to-end timing measurements that include the acoustic front-end, or results comparing raw (or identically postprocessed) outputs across multiple event boundaries, please share them. Otherwise, I would suggest toning down the claims around real-time speed, displayed quality, and cross-event robustness to match what has been strictly verified.

7 comments
AuthorsOpenReview ↗

Response to Reviewer T6Dw (1/3)

We thank the reviewer for recognizing the evidence for spatial feedback (Table 4, Online FID from 25.872 to 8.714), the rehearsal ablations, the conditioning representations in Table 5, the No-SSL variant in Appendix D, and the clarity of the paper.

W1. Incremental value of fine-grained audio given c, g and the initial frames.

We thank the reviewer for this insightful question. Clarifying what each input contributes helps in understanding the model, and the analysis it prompted has strengthened the paper.

1. Roles of a, c, g and the initial frames.

  • a (audio) is a streaming, time-varying condition. A new audio token arrives every 133 ms, so the motion can follow the sound as it starts, stops or changes. Unlike a text prompt, it is not a global prefix.
  • c (reaction style) takes only three values, defined in SAM: insensitive (tends to react mildly to sounds), neutral (reacts moderately) and sensitive (tends to react strongly). It describes the person, not the sound, and is unrelated to the event category. We include c because people react to the same sound with different intensity.
  • g is the sound source position.
  • Initial frames. The first four frames are usually a standing pose.

2. The suggested "event-label replacing spatial audio" baseline. We think this baseline may not fit the task well. A real-world deployed system may not have event labels, only the incoming sound. Moreover, an event label is a global condition, like a text prompt, rather than a streaming, temporal one, so it cannot tell the model when a sound starts, stops or changes.

3. New evidence that the model uses the audio itself (See General Response C). We directly tested whether the model uses the audio beyond c, g and the initial frames, in two ways:

  • Analysis of Attention Map in the Transformer Layers on all 292 test clips (new Appendix G): audio tokens receive as much attention as spatial tokens (24.4% vs. 24.6%), and at sound onsets, attention to the current audio rises ten times more than when the same clip is aligned to the onsets of a control clip (7.3% vs. 0.7%).
  • Audio ablation, keeping c, g and the initial frames of each clip unchanged and altering only the audio. Without audio, FID rises from 8.714 to 87.90 and R@1 drops from 0.909 to 0.272, so c, g and the initial frames alone are far from sufficient. With a sound of a different category, FID rises to 22.78 and R@1 drops to 0.202, so the motion follows the sound itself.

As the reviewer notes, the No-SSL variant changes two components at once and was not designed to isolate the audio; the experiments above do so directly. We have added the audio ablation to Appendix G (in blue).


W2. The meaning and constraints of the style label c. SAM defines three reaction styles: insensitive (tends to react mildly to sounds), neutral (reacts moderately) and sensitive (tends to react strongly). The style sets only the general intensity of the reaction, not the specific event. In the W1 experiments, c is fixed, yet the motion changes substantially with the audio.

AuthorsOpenReview ↗

Response to Reviewer T6Dw (2/3)

W4. "FID and R@k do not establish that the reactions are contextually appropriate." Although these metrics have limitations, FID and R-precision are well-studied metrics for generation quality and condition alignment, widely used in motion generation tasks. To assess contextual appropriateness directly, we conducted an evaluation with three multimodal LLM judges.

Setup. We sampled 60 test clips, balanced across the three reaction styles (20 each) and the four source directions (15 each), and covering all 27 sound categories. The first 8 s of the ground truth, of MOSPA and of our raw output are rendered with the same renderer and camera, with the loudspeaker shown as a sphere. The 180 videos are shuffled and anonymised, and each is rated on its own: the judge never knows which method produced a video and never compares methods side by side.

Prompt. All three judges receive the same prompt:

You are an expert evaluator of human motion plausibility in spatial acoustic environments.

You will be shown one video. It depicts a rendered 3D human character responding to a sound. The video includes the corresponding audio, and a sphere indicates the position of the loudspeaker.

Your task is to judge whether the character's behavior is a reasonable, plausible and natural human response to this sound.

Evaluate the video on the following criteria:

  1. Semantic Consistency: Does the character's behavior make sense given the nature and context of the sound?
  2. Spatial Consistency: Are the character's orientation, movement, and reactions appropriate relative to the loudspeaker's position?
  3. Temporal Consistency: Are the timing and progression of the movements coherent with the audible events?
  4. Motion Plausibility: Are the movements physically and behaviorally natural, without noticeable artifacts such as foot sliding, jitter, floating, or unnatural limb movements?

Important considerations:

  • Many different behaviors can be reasonable for the same sound. Judge whether this behavior is reasonable, not whether it matches a specific reaction you expected.
  • Subtle movements, staying mostly still, or delayed reactions can be entirely natural depending on the acoustic context.
  • Do not favor exaggerated reactions, larger movements, or more complex motion merely because they are more visually noticeable.
  • Consider the overall coherence of the motion throughout the entire video, rather than focusing on isolated moments.
  • Ignore rendering quality, character appearance, camera perspective, and visual aesthetics.

Score each criterion on a 1-5 scale, with one sentence of concrete evidence (timestamps, body parts, audible events) each: 1 = clearly unreasonable, 2 = mostly unreasonable, 3 = borderline, 4 = reasonable with minor issues, 5 = fully reasonable and natural.

Then give an overall 1-5 score on the same scale.

JudgeGT (ref.)MOSPAOursp-value (Ours vs. GT)p-value (Ours vs. MOSPA)
Claude Opus 5.53.703.573.750.470.022
Gemini 3.1 Pro3.523.503.550.760.64
GPT-6 Astra4.574.504.620.510.035

Overall score (1-5, higher is better), averaged over the 60 clips. p-value < 0.05 (bold) means a significant difference.

All three judges rate our motion above MOSPA in score, significantly so for Claude and GPT-6 Astra (p-value ≤ 0.035), and rate it as appropriate as the recorded human reactions (p-value ≥ 0.47).

AuthorsOpenReview ↗

Response to Reviewer T6Dw (3/3)

(3/3)

Q1. Handling of event boundaries.

At an event switch, the motion history is kept and the event conditions are refreshed, without retraining:

  • Kept: the motion history.
    • Generated motion is expressed in the current body frame and re-encoded by the frozen VAE.
    • The decoder retains up to 45 latents (including the current one).
    • Previously emitted frames are never changed.
  • Refreshed: the event conditions.
    • The Transformer clears its KV cache and restarts from the last re-encoded latent.
    • The acoustic context resets to the new sound.
    • The new source's body-relative spatial cues are computed from the four most recently decoded poses.

Generation then continues block by block, using only past motion. We have clarified this protocol in Appendix B (in blue).


Q2. "How does the two-pose timing configuration relate to the four-frame pooling used for quality evaluation"

Please see General Response A, which shows one step of the system with the time of each stage, now including the acoustic front-end. In short, there is only one system, and it operates on four-frame blocks throughout, in training, quality evaluation and latency measurement. The phrase "last two poses" in Appendix C came from an outdated timing script, which used an old version of the spatial update. We measured the timings with the same four-frame model evaluated in the main paper, and the P95 step time is 53.7 ms on the RTX A6000 and 125.5 ms on the TITAN RTX. We thank the reviewer for pointing this out.


Q3. Clarification on Postprocessing Usage. All quantitative results use raw outputs without postprocessing. Postprocessing is never fed back to the model and is not part of the streaming system. It is applied offline only to demonstration videos. For visualization, we apply the same postprocessing to both the baselines and our method: a typical motion-cleanup step that smooths the motion and removes foot sliding, as in the publicly released MotionMillion code (https://github.com/VankouF/MotionMillion-Codes, postprocess/remove_sliding). We have stated this in the paper (in blue).


Q4. "End-to-end timing and raw outputs at multi-event boundaries."

  1. Timing. Please see General Response A, which shows the end-to-end computation of one inference step, now including the acoustic front-end (1.4 ms per step on the RTX A6000 and 3.9 ms on the TITAN RTX). The system evaluated for quality has a P95 step time of 53.7 ms on an RTX A6000 and 125.5 ms on a TITAN RTX, below the 133.3 ms real-time budget. We have corrected Appendix C accordingly.
  2. Multi-event boundaries. Please see General Response B. All successive-event results use raw outputs, and after each switch rehearsal remains well above baselines (R@1 0.588 vs. 0.188 and 0.605 vs. 0.258).

We are looking forward to your feedback on the rebuttal. Feel free to ask for any additional clarifications or experiments, and we will do our best to provide them.

Reviewer_BhDHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Sonic-AMG generates motion from streaming spatial audio using a causal motion VAE, an autoregressive Transformer, and a flow head. Separate acoustic/spatial tokens, short self-rollout rehearsal, and generated-root spatial feedback address conditioning and history mismatch.

Critical strengths
  • The paper identifies a meaningful deployment mismatch: body-relative source locations derived from recorded motion are unavailable during online generation.
  • The mechanism ablations are convincing: online rehearsal improves FID for the best two-forward variant, while feedback training improves GT-to-online deployment ( as seen in the tables 3–4).
  • The appendices document causal feature availability, initialization, caching, the No-SSL variant, and component runtime clearly.
Critical weaknesses
  • Rehearsal is not isolated from additional training compute. Table 3 does not present a comparison with GPU times like two-forward or network calls. Appendix E only gives the main model training times, so it is not possible to determine how much of the gain comes from the recursion history itself.
  • The method excludes acoustic preprocessing/encoding and updates spatial conditions from two poses rather than the four-frame pooling used for quality evaluation, resulting in different pipelines between runtime and quality.
  • Missing references:

Li R, Zhao J, Zhang Y, et al. Finedance: A fine-grained choreography dataset for 3d full body dance generation[C]//2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023: 10200-10209.

Zhang J, Kang Z, Liu L, et al. OpenDance: Multimodal Controllable 3D Dance Generation with Large-scale Internet Data[J]. arXiv preprint arXiv:2506.07565, 2025.

Li J, Kang D, Pei W, et al. Audio2gestures: Generating diverse gestures from audio[J]. IEEE Transactions on Visualization and Computer Graphics, 2023, 30(8): 4752-4766.

Questions for authors

What quality and end-to-end P95 latency result from the two-pose and four-frame spatial updates?

How does an adapted MotionStreamer or another causal generator perform with identical online inputs, initialization, and compute, especially across successive events?

2 comments
AuthorsOpenReview ↗

Response to Reviewer BhDH (1/2)

We thank the reviewer for recognizing the deployment mismatch identified in the paper, the targeted mechanism ablations in Tables 3 and 4, and the clear documentation in the appendices.


W1. "The gain of rehearsal is not separated from additional training compute."

Rehearsal and two-forward are trained for the same number of steps (130k), i.e., the same number of parameter updates on the same GPU, and rehearsal improves Online FID from 11.766 to 8.714 (Table 3). The two strategies differ in how each training sample is constructed: rehearsal spends more computation per step building its inputs through short self-rollouts, so that the model is trained on the motion histories and conditions it will face at inference. The per-step costs, measured on one L40S, are:

TrainingComputation per stepGT-SSLOnline
Two-forward2 full-sequence forward passes0.47 s0.49 s
Rehearsal (H = 16)1 full-sequence forward pass + 16 rollout steps1.20 s1.64 s

Rehearsal is therefore 2.5–3.4× more expensive per step. This is a trade-off in training hours only: at inference, both models share the same architecture and the same network calls per step, so their inference cost is identical. We have added this table to Appendix E (in blue).


W2 and Q1. "Acoustic preprocessing is not timed, and timing (two-pose) and quality (four-frame) use different systems."

Same as General Response A:

Sonic-AMG operates on four-frame blocks throughout: training, quality evaluation and latency measurement all use the same configuration. The diagram shows one step with the mean time of each stage (batch size one, 680 steps after warm-up; RTX A6000 / TITAN RTX):

one step = 1 block = 4 frames = 133.3 ms at 30 fps


            audio of block k                    latent z_k <----------------+
                    |                     +----------+----------+           |
                    |                     v                     v           |
                 log-mel             VAE decoder          linear layer      |
                    v                     v                     |           |
               causal CNN      motion block k (output)          |           |
                    v                     v                     |           |
             4-frame pooling         root poses                 |           |
                    |                     v                     |           |
                    |              source location              |           |
                    |                     v                     |           |
                    |              4-frame pooling              |           |
                    |                     v                     |           |
                    |                    MLP                    |           |
                    |                     |                     |           |
             (1.4 / 3.9 ms)        (8.4 / 16.5 ms)              |           |
                    v                     v                     v           |
 [ ...             a_k                   s_k                   m_k ]        |
                    +---------------------+---------------------+           |
                                          v                                 |
                            causal Transformer (KV cache)                   |
                                          v                                 |
                        rectified-flow head (10 Euler steps)                |
                      (Transformer + flow head: 41.3 / 97.7 ms)             |
                                          v                                 |
                                   latent z_{k+1} --- next step ------------+

-----------------------------------------------------------------------------
 whole step:   mean 51.1 / 118.1 ms   |   P95 53.7 / 125.5 ms   (< 133.3 ms)

The phrase "last two poses" in Appendix C came from an outdated timing script, which used an old version of the spatial update; the model evaluated in the main paper always uses four frames. We thank the reviewers for pointing this out. We measured the timings with the same four-frame model evaluated in the main paper, now including the acoustic front-end (diagram above). The P95 step time is 53.7 ms on the RTX A6000 and 125.5 ms on the TITAN RTX. The whole step stays below the 133.3 ms real-time budget on both GPUs. We have corrected Appendix C and report these end-to-end timings (in blue).

AuthorsOpenReview ↗

Response to Reviewer BhDH (2/2)

W3. Missing references. We have cited FineDance, OpenDance and Audio2Gestures and discussed them in the related work (Section 2, in blue).


Q2. "A causal baseline with identical online inputs, initialization and compute, especially across successive events."

  1. Single event. The two-forward rows of Table 3 are exactly this baseline: they implement the training strategy of MotionStreamer in our architecture, with the same inputs, initialization and spatial updates, and same training steps (W1). Rehearsal outperforms all two-forward variants.

  2. Successive events (See General Response B). We evaluated rehearsal and two-forward on 292 streams of three successive events, using raw outputs. After each switch, rehearsal remains well above two-forward in R@1 (0.588 vs. 0.188 and 0.605 vs. 0.258), showing that it is more robust than baselines when the conditions change within a stream.

  3. MOSPA cannot be applied to successive events, since it takes no past motion as input, requires the complete future audio, and is evaluated with ground-truth spatial conditions.


We are looking forward to your feedback on the rebuttal. Feel free to ask for any additional clarifications or experiments, and we will do our best to provide them.

Reviewer_vTDfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to solve the problem of modeling people's actions from spatial audio and propose a new method named Sonic-AMG, which is a autoregressive framework for generating human motion from streaming spatial audio. To alleviate the problem between training and inference, Sonic-AMG further introduced the rehearsal training process and spatial feedback signal. Sonic-AMG is verified on SAM dataset and sota FID and R-precision and support realtime motion generation.

Critical strengths
  1. Reformulate the problem from full sequence to causual is vaulable, not only for inference efficience but also to generalization ability. Sonic-AMG introduces two main techniques for spatial audio to the human motion field. Specifically, Sonic-AMG introduces autoregressive framework with causal transformers to follow the reaction of people and support realtime stream generation. Sonic-AMG further uses rehearsal training process and spatial feedback to alleviate the gap between training and inference.
  2. The overall experiments are quite comprehensive from main results to ablation study and motion visualization can further support the core opinions.
Critical weaknesses
  1. Athough AR framework and rehearsal training is first used in spatial audio to motion task, these two techniques have been used in genral motion field for a long time, such as previous work MotionStreamer, GestureFAR and ReactiveBFM. The main technique contribution seems quite limited.
Questions for authors

Please add more discussions about the main techniques about ar framework and techer forcing; I think these techniques are commonly used in the motion generation field

1 comment
AuthorsOpenReview ↗

Response to Reviewer vTDf

We thank the reviewer for recognizing the value of reformulating the problem as causal generation and the comprehensiveness of our experiments.

On the cited works. MotionStreamer is a central baseline of our paper (Table 3). GestureFAR (arXiv, 18 Sep 2026) and ReactiveBFM (arXiv v1: 29 Jun, v2: 19 Jul, v3: 22 Oct 2026; to appear at CoRL 2026) were preprints at the submission deadline and are therefore contemporaneous work under the ICLR 2027 Reviewer Guidelines. We nevertheless discuss and compare with both below.


W+Q. "The autoregressive framework and rehearsal training have been used for a long time."

We agree that autoregression itself is standard, and we do not claim it. Our contributions lie elsewhere, as summarized below:

MotionStreamerGestureFARReactiveBFMOurs
Status at the ICLR 2027 deadline (25 Sep 2026)Published (ICCV 2025)arXiv preprint (18 Sep 2026)arXiv preprint (v1: 29 Jun 2026)Submitted (25 Sep 2026)
ConditionText (global prefix)Speech (fused by cross-attention)Text + target positionSpatial audio (separate audio and spatial tokens)
Condition depends on the generated body✗✗✓ (at deployment only)✓
TrainingTwo-forward (not recursive)Teacher forcing onlyGT prefixes replaced by rolloutsFull teacher forcing + short rollouts
Addresses the AR gap✓✗✓✓
Addresses the condition gap✗✗✗✓

AR gap: the model is trained on ground-truth motion history but generates from its own history at inference.
Condition gap: the model is trained on conditions computed from recorded motion, but at inference the conditions depend on its own generated motion.

1. We identify, analyze and address the problems of making this task streaming. As the reviewer notes, ours is the first streaming framework for spatial audio to motion. There is no general autoregressive solution that fits all tasks, and this task raises problems that prior frameworks do not face:

  • Two streaming conditions. Both the sound and the spatial cues arrive as streams. We encode them as separate audio and spatial tokens, interleaved with the motion tokens, which outperforms fusing them into a single token (FID 11.445 → 8.714, Table 5).
  • Conditioning the flow-matching head. The head is conditioned on the concatenated transformer outputs at the audio, spatial and motion positions of each step, rather than on a single output.
  • Body-dependent spatial condition. The sound source is fixed in the world, so its position relative to the body changes as the generated body moves. The standard GT-SSL protocol hides this, because it computes the condition from the recorded motion: a model trained this way reaches FID 6.505 under GT-SSL, but 25.872 once the condition comes from its own motion. Spatial feedback during training reduces this to 8.714 and raises R@1 from 0.714 to 0.909 (Table 4).

2. Rehearsal training is first proposed in this paper, and it reduces both the AR gap and the condition gap.

  • Prior strategies reduce at most the AR gap, as summarized in the table above; GestureFAR, trained with teacher forcing only, reduces neither.

  • Our task also has a condition gap. The spatial condition depends on the body, so at inference it drifts with the generated motion. This gap arises whenever streamed conditions depend on the generated output, and to our knowledge no prior training strategy addresses it.

  • Rehearsal reduces both. It keeps full-sequence teacher forcing and adds short self-rollouts, in which the spatial condition is recomputed from the generated motion, so that the model learns from its own motion and from the conditions this motion induces.

  • Rehearsal outperforms the training strategies of the cited works (Table 3). Two-forward is the training strategy of MotionStreamer. ReactiveBFM gradually replaces ground-truth prefixes with self-rollouts; we include a similar variant in Table 3, Rollout Only with History.

    Two-forward (MotionStreamer)Rollout Only with History (similar to ReactiveBFM)Rehearsal (ours)
    FID ↓11.76612.3208.714
    R@1 ↑0.8870.8910.909

We have added this discussion to the related work (Section 2, in blue).


We are looking forward to your feedback on the rebuttal. Feel free to ask for any additional clarifications or experiments, and we will do our best to provide them.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response (3/3)

C. Contribution of fine-grained audio (T6Dw). Given the reaction style c, the source position g and the initial frames, does the model still use the audio itself? We answer with an attention analysis and an audio ablation.

  • Analysis of Attention Map in the Transformer Layers. We recorded the transformer attention on all 292 test clips (new Appendix G, Figs. 10–12 of the revised paper):

    • Audio receives as much attention as the spatial cues. When generating motion, the model assigns 24.4% of its attention to audio tokens and 24.6% to spatial tokens; in the early layers, the two together receive up to 70.5% (Figs. 10, 11).
    • Attention follows the sound in time. In the attention map aligned with the spectrogram, the audio tokens of a sound event receive more attention as soon as the sound starts, and all subsequent motion steps keep attending to them (Fig. 12). At sound onsets, attention to the current audio rises ten times more than when the same clip is aligned to the onsets of a control clip (7.3% vs. 0.7%), and it follows the loudness of the clip's own sound.
  • Audio ablation. We keep c, g and the initial frames of each clip unchanged and alter only the audio (main model, Online protocol).

    cgAudio inputFID ↓R@1 ↑
    unchangedunchangedOriginal Spatial Audio8.7140.909
    unchangedunchangedEmpty Spatial Audio87.900.272
    unchangedunchangedRandom Spatial Audio from a Different Category22.780.202
    • c, g and the initial frames alone are far from sufficient. Without audio, FID rises from 8.714 to 87.90 and R@1 drops from 0.909 to 0.272.
    • The motion follows the sound. With a sound of a different category, FID rises to 22.78 and R@1 drops to 0.202, even below the no-audio case (0.272), although c, g and the initial frames are unchanged.

D. Contextual appropriateness of the reactions (T6Dw). FID and R@1 do not directly measure whether a reaction is contextually appropriate, so we ran a blind evaluation with three multimodal LLM judges. We sampled 60 test clips, balanced across the three reaction styles and the four source directions and covering all 27 sound categories, and rendered the first 8 s of the ground truth, MOSPA and our raw output with the same renderer and camera. The 180 videos are shuffled and anonymised and each is rated on its own, so the judge never knows which method produced a video. Each judge scores semantic, spatial and temporal consistency and motion plausibility, then gives an overall score (1–5). The prompt is given in our response to T6Dw.

JudgeGT (ref.)MOSPAOursp-value (Ours vs. GT)p-value (Ours vs. MOSPA)
Claude Opus 5.53.703.573.750.470.022
Gemini 3.1 Pro3.523.503.550.760.64
GPT-6 Astra4.574.504.620.510.035

Overall score (1–5, higher is better), averaged over the 60 clips. p-value: p < 0.05 (bold) means a significant difference.

All three judges rate our motion above MOSPA in score, significantly so for Claude and GPT-6 Astra (p ≤ 0.035), and rate it as appropriate as the recorded human reactions (p ≥ 0.47).


Revisions to the paper. All changes in the revised paper are highlighted in blue.

  • Section 2 (Related Work): discussion of autoregressive training strategies, the AR gap and the condition gap (response to vTDf).
  • Appendix A: limitations updated with the successive-event evaluation (B).
  • Appendix B: handling of event boundaries, and the successive-event evaluation (B).
  • Appendix C: corrected description of the spatial update, and end-to-end latency (A).
  • Appendix E: training cost (response to BhDH).
  • New Appendix G: attention analysis and audio ablation (C).
  • New Appendix H: Contextual appropriateness of the reactions (D).
  • A statement that postprocessing is applied only to demonstration videos, for both the baselines and our method; all evaluation results use raw outputs.
  • More discussions regarding reaction appropriateness and cross-event robustness.
  • Citations to FineDance, OpenDance, Audio2Gestures and ReactiveBFM.
AuthorsOpenReview ↗

General Response (2/3)

A. End-to-end latency including the acoustic front-end (BhDH, T6Dw).

Sonic-AMG operates on four-frame blocks throughout: training, quality evaluation and latency measurement all use the same configuration. The diagram shows one step with the mean time of each stage (batch size one, 680 steps after warm-up; RTX A6000 / TITAN RTX):

one step = 1 block = 4 frames = 133.3 ms at 30 fps


            audio of block k                    latent z_k <----------------+
                    |                     +----------+----------+           |
                    |                     v                     v           |
                 log-mel             VAE decoder          linear layer      |
                    v                     v                     |           |
               causal CNN      motion block k (output)          |           |
                    v                     v                     |           |
             4-frame pooling         root poses                 |           |
                    |                     v                     |           |
                    |              source location              |           |
                    |                     v                     |           |
                    |              4-frame pooling              |           |
                    |                     v                     |           |
                    |                    MLP                    |           |
                    |                     |                     |           |
             (1.4 / 3.9 ms)        (8.4 / 16.5 ms)              |           |
                    v                     v                     v           |
 [ ...             a_k                   s_k                   m_k ]        |
                    +---------------------+---------------------+           |
                                          v                                 |
                            causal Transformer (KV cache)                   |
                                          v                                 |
                        rectified-flow head (10 Euler steps)                |
                      (Transformer + flow head: 41.3 / 97.7 ms)             |
                                          v                                 |
                                   latent z_{k+1} --- next step ------------+

-----------------------------------------------------------------------------
 whole step:   mean 51.1 / 118.1 ms   |   P95 53.7 / 125.5 ms   (< 133.3 ms)

The phrase "last two poses" in Appendix C came from an outdated timing script, which used an old version of the spatial update; the model evaluated in the main paper always uses four frames. We thank the reviewers for pointing this out. We measured the timings with the same four-frame model evaluated in the main paper, now including the acoustic front-end (diagram above). The P95 step time is 53.7 ms on the RTX A6000 and 125.5 ms on the TITAN RTX. The whole step stays below the 133.3 ms real-time budget on both GPUs. We have corrected Appendix C and report these end-to-end timings (in blue).


B. Successive sound events (BhDH, T6Dw). We thank the reviewers for this question. No dataset of successive sound events with paired motion exists, so the successive-event results in the paper are intended as an application: because Sonic-AMG generates causally from the incoming stream, it can continue across sound events without retraining, even though this lies outside its training distribution. Nevertheless, we evaluated this setting quantitatively as far as the available data allow.

  1. Training data. All models are trained on SAM, in which every clip contains a single sound event.
  2. Constructing the evaluation. We chain three SAM test clips into one stream, so that consecutive events differ in both sound category and source direction. Over 292 streams, each test clip appears exactly once at each of the three positions. Each stream is generated continuously, without postprocessing; only the first event starts from ground-truth initial frames. We then split the generated stream at the event boundaries and evaluate each event against the conditions of its own clip.
  3. Metrics. R@1 measures whether the motion follows the current sound event.
TrainingEvent 1Event 2Event 3
R@1 ↑R@1 ↑R@1 ↑
Two-forward0.7760.1880.258
Rehearsal (ours)0.9090.5880.605
  1. Conclusion. After each switch, rehearsal remains well above two-forward baselines (R@1 0.588 vs. 0.188 and 0.605 vs. 0.258), showing that it is more robust than baselines when the conditions change within a stream.
AuthorsOpenReview ↗

General Response (1/3)

We sincerely thank all reviewers for their time and constructive feedback. We are encouraged that they found that:

  • the reformulation from full-sequence to causal generation is valuable for both inference efficiency and generalization (vTDf);
  • the paper identifies a meaningful training-inference mismatch and tests the proposed solution directly (BhDH, T6Dw);
  • the mechanism ablations are convincing and well targeted: rehearsal improves over the best two-forward variant (BhDH, T6Dw), Table 4 convincingly supports updating spatial conditions during training rather than only at inference (T6Dw), and Table 5 supports the separate conditioning tokens and learned acoustic features (T6Dw);
  • the experiments are comprehensive (vTDf), and the appendices clearly document the causal features, initialization, caching, No-SSL variant and runtime (BhDH, T6Dw);
  • the paper is clear and easy to follow (T6Dw).

In response to the reviewers' questions, we conducted new experiments with the main model. Below, we first address the questions raised by more than one reviewer (A, B), and then report new experiments that we think are relevant to all reviewers' assessment (C, D). All other questions are answered, and the remaining experiments are reported in the individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.