- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Sonic-AMG, an autoregressive model for generating human reactions as spatial audio arrives. The main model takes binaural audio, a known world-space sound-source position, a reaction-style label, and an initial motion segment. In the SAM evaluation, this segment consists of four ground-truth frames. It encodes acoustic content and body-relative source geometry as separate what/where tokens, then generates four-frame motion blocks with a causal Transformer, a rectified-flow head, and a frozen causal temporal VAE. Rehearsal training combines full-sequence teacher-forced supervision with supervision on short, 16-step self-rollouts. Spatial feedback updates the source geometry from generated root poses during these rollouts and inference. The experiments use the existing SAM dataset. The model achieves FID/R@1 of 6.505/0.957 under the GT-SSL comparison protocol (Table 2) and 8.714/0.909 under the Online protocol (Tables 3 and 4). The paper also includes an audio-derived No-SSL variant and qualitative examples spanning successive sound events.
Critical strengths
-
The paper identifies a concrete problem in streaming spatial-audio generation and tests its proposed solution directly. Once the body moves, the source's body-relative location changes too, so replacing ground-truth motion histories during training is not enough if the spatial conditions still come from recorded motion. Table 4 compares two rehearsal-trained models under the same Online evaluation protocol: training with spatial feedback improves FID from 25.872 to 8.714 and R@1 from 0.714 to 0.909. This is convincing evidence for updating spatial conditions during training, not just at inference (Section 4.3).
-
The rehearsal ablations are also useful and reasonably targeted. All variants in Table 3 share the generator architecture, and the Online two-forward baselines also update spatial conditions. Compared with the Online two-forward baseline with the lowest FID, rehearsal improves FID/R@1 from 11.766/0.887 to 8.714/0.909. Rollout Only with History reaches 12.320/0.891, which supports keeping full-sequence supervision alongside the short rollouts. Table 5 gives further support for the separate conditioning tokens and learned acoustic features. The No-SSL extension in Appendix D is useful as well: Table 8 reports 11.917/0.915 without source-location inputs at inference, although it still uses geometric supervision during training and does not use spatial feedback. Together, these results support the proposed streaming model within the SAM setting.
Critical weaknesses
The actual incremental value of the fine-grained audio and the clarity of the task formulation are not fully established. While I agree that the inherent ambiguity of how people react to sound doesn't invalidate conditional motion generation, it raises a critical question: how much meaningful reaction information does the fine-grained audio actually provide once we already supply the reaction-style label c, the sound source position g, and the four initial seed frames (or motion history)? Note that c is an annotated reaction-style label, and while it might not completely dictate every single action, its precise semantics and the constraints it places on reaction selection remain quite vague. Table 5's ablations on feature replacement and fusion show that the representation design works, but they do not isolate the actual incremental benefits of the fine-grained audio over simpler baselines (such as just using event labels combined with style, spatial, and history information). Similarly, the No-SSL variant in Appendix D changes both the spatial condition provider and the semantic encoding at the same time, failing to isolate this issue. Furthermore, since FID only measures how well the motion distribution matches the paired dataset, and R@ measures alignment in the joint embedding space, neither of them directly guarantees that the generated motion is actually "situationally appropriate." Even with metric improvements across 27 scenarios and 3 styles, this evidence is not quite enough to back up the paper's core claim about generating highly meaningful reactions to sound (pp. 4–7, 9, and 21–23).
Questions for authors
Could you provide more details on how the system handles transition boundaries? Specifically, during cross-event root-alignment, how are the motion histories, acoustic/Transformer caches, and decoder states managed? Also, how does the two-pose timing configuration relate to the four-frame pooling used for quality evaluation? Is the postprocessing shown in the comparison videos also applied during quantitative evaluation, does it feed back into the model, or does it require future frames (which would affect online streaming)? If you have end-to-end timing measurements that include the acoustic front-end, or results comparing raw (or identically postprocessed) outputs across multiple event boundaries, please share them. Otherwise, I would suggest toning down the claims around real-time speed, displayed quality, and cross-event robustness to match what has been strictly verified.
7 comments
Response to Reviewer T6Dw (2/3)
W4. "FID and R@k do not establish that the reactions are contextually appropriate." Although these metrics have limitations, FID and R-precision are well-studied metrics for generation quality and condition alignment, widely used in motion generation tasks. To assess contextual appropriateness directly, we conducted an evaluation with three multimodal LLM judges.
Setup. We sampled 60 test clips, balanced across the three reaction styles (20 each) and the four source directions (15 each), and covering all 27 sound categories. The first 8 s of the ground truth, of MOSPA and of our raw output are rendered with the same renderer and camera, with the loudspeaker shown as a sphere. The 180 videos are shuffled and anonymised, and each is rated on its own: the judge never knows which method produced a video and never compares methods side by side.
Prompt. All three judges receive the same prompt:
You are an expert evaluator of human motion plausibility in spatial acoustic environments.
You will be shown one video. It depicts a rendered 3D human character responding to a sound. The video includes the corresponding audio, and a sphere indicates the position of the loudspeaker.
Your task is to judge whether the character's behavior is a reasonable, plausible and natural human response to this sound.
Evaluate the video on the following criteria:
- Semantic Consistency: Does the character's behavior make sense given the nature and context of the sound?
- Spatial Consistency: Are the character's orientation, movement, and reactions appropriate relative to the loudspeaker's position?
- Temporal Consistency: Are the timing and progression of the movements coherent with the audible events?
- Motion Plausibility: Are the movements physically and behaviorally natural, without noticeable artifacts such as foot sliding, jitter, floating, or unnatural limb movements?
Important considerations:
- Many different behaviors can be reasonable for the same sound. Judge whether this behavior is reasonable, not whether it matches a specific reaction you expected.
- Subtle movements, staying mostly still, or delayed reactions can be entirely natural depending on the acoustic context.
- Do not favor exaggerated reactions, larger movements, or more complex motion merely because they are more visually noticeable.
- Consider the overall coherence of the motion throughout the entire video, rather than focusing on isolated moments.
- Ignore rendering quality, character appearance, camera perspective, and visual aesthetics.
Score each criterion on a 1-5 scale, with one sentence of concrete evidence (timestamps, body parts, audible events) each: 1 = clearly unreasonable, 2 = mostly unreasonable, 3 = borderline, 4 = reasonable with minor issues, 5 = fully reasonable and natural.
Then give an overall 1-5 score on the same scale.
| Judge | GT (ref.) | MOSPA | Ours | p-value (Ours vs. GT) | p-value (Ours vs. MOSPA) |
|---|---|---|---|---|---|
| Claude Opus 5.5 | 3.70 | 3.57 | 3.75 | 0.47 | 0.022 |
| Gemini 3.1 Pro | 3.52 | 3.50 | 3.55 | 0.76 | 0.64 |
| GPT-6 Astra | 4.57 | 4.50 | 4.62 | 0.51 | 0.035 |
Overall score (1-5, higher is better), averaged over the 60 clips. p-value < 0.05 (bold) means a significant difference.
All three judges rate our motion above MOSPA in score, significantly so for Claude and GPT-6 Astra (p-value ≤ 0.035), and rate it as appropriate as the recorded human reactions (p-value ≥ 0.47).
Response to Reviewer T6Dw (3/3)
(3/3)
Q1. Handling of event boundaries.
At an event switch, the motion history is kept and the event conditions are refreshed, without retraining:
- Kept: the motion history.
- Generated motion is expressed in the current body frame and re-encoded by the frozen VAE.
- The decoder retains up to 45 latents (including the current one).
- Previously emitted frames are never changed.
- Refreshed: the event conditions.
- The Transformer clears its KV cache and restarts from the last re-encoded latent.
- The acoustic context resets to the new sound.
- The new source's body-relative spatial cues are computed from the four most recently decoded poses.
Generation then continues block by block, using only past motion. We have clarified this protocol in Appendix B (in blue).
Q2. "How does the two-pose timing configuration relate to the four-frame pooling used for quality evaluation"
Please see General Response A, which shows one step of the system with the time of each stage, now including the acoustic front-end. In short, there is only one system, and it operates on four-frame blocks throughout, in training, quality evaluation and latency measurement. The phrase "last two poses" in Appendix C came from an outdated timing script, which used an old version of the spatial update. We measured the timings with the same four-frame model evaluated in the main paper, and the P95 step time is 53.7 ms on the RTX A6000 and 125.5 ms on the TITAN RTX. We thank the reviewer for pointing this out.
Q3. Clarification on Postprocessing Usage.
All quantitative results use raw outputs without postprocessing. Postprocessing is never fed back to the model and is not part of the streaming system. It is applied offline only to demonstration videos. For visualization, we apply the same postprocessing to both the baselines and our method: a typical motion-cleanup step that smooths the motion and removes foot sliding, as in the publicly released MotionMillion code (https://github.com/VankouF/MotionMillion-Codes, postprocess/remove_sliding). We have stated this in the paper (in blue).
Q4. "End-to-end timing and raw outputs at multi-event boundaries."
- Timing. Please see General Response A, which shows the end-to-end computation of one inference step, now including the acoustic front-end (1.4 ms per step on the RTX A6000 and 3.9 ms on the TITAN RTX). The system evaluated for quality has a P95 step time of 53.7 ms on an RTX A6000 and 125.5 ms on a TITAN RTX, below the 133.3 ms real-time budget. We have corrected Appendix C accordingly.
- Multi-event boundaries. Please see General Response B. All successive-event results use raw outputs, and after each switch rehearsal remains well above baselines (R@1 0.588 vs. 0.188 and 0.605 vs. 0.258).
We are looking forward to your feedback on the rebuttal. Feel free to ask for any additional clarifications or experiments, and we will do our best to provide them.
Response to Reviewer T6Dw (1/3)
We thank the reviewer for recognizing the evidence for spatial feedback (Table 4, Online FID from 25.872 to 8.714), the rehearsal ablations, the conditioning representations in Table 5, the No-SSL variant in Appendix D, and the clarity of the paper.
W1. Incremental value of fine-grained audio given c, g and the initial frames.
We thank the reviewer for this insightful question. Clarifying what each input contributes helps in understanding the model, and the analysis it prompted has strengthened the paper.
1. Roles of a, c, g and the initial frames.
2. The suggested "event-label replacing spatial audio" baseline. We think this baseline may not fit the task well. A real-world deployed system may not have event labels, only the incoming sound. Moreover, an event label is a global condition, like a text prompt, rather than a streaming, temporal one, so it cannot tell the model when a sound starts, stops or changes.
3. New evidence that the model uses the audio itself (See General Response C). We directly tested whether the model uses the audio beyond c, g and the initial frames, in two ways:
As the reviewer notes, the No-SSL variant changes two components at once and was not designed to isolate the audio; the experiments above do so directly. We have added the audio ablation to Appendix G (in blue).
W2. The meaning and constraints of the style label c. SAM defines three reaction styles: insensitive (tends to react mildly to sounds), neutral (reacts moderately) and sensitive (tends to react strongly). The style sets only the general intensity of the reaction, not the specific event. In the W1 experiments, c is fixed, yet the motion changes substantially with the audio.