acceptodds
Under review as a conference paper at ICLR 2027

Sound Sparks Motion: Audio and Text Tuning for Video Motion Editing

Abstract

Motion-centric video editing remains difficult for large generative video models, which often respond well to appearance changes but struggle to produce specific, localized actions or state transitions in an existing clip. We introduce Sound Sparks Motion, a training-free framework that enables motion editing in an audio-visual video generation model by tuning its internal multimodal conditioning signals at test time. Rather than modifying model weights, our method tunes two lightweight variables: the text conditioning and the audio latent. We find that tuning the text alone is often insufficient, as audio and motion are highly entangled in the underlying model. By also optimizing the audio latent, we can more effectively steer the desired motion, while the resulting audio can simply be discarded after generation. Since there is no direct way to evaluate temporal alignment between text and motion, we guide the tuning process using a vision–language model that provides feedback indicating whether the intended motion appears in the generated video. This simple supervision yields an effective semantic objective for motion editing, while regularization and perceptual-temporal constraints help preserve content and visual quality. Beyond per-video tuning, we show that the learned latent controls are transferable across videos, suggesting that they capture reusable motion-edit directions rather than overfitting to a single example. Our results highlight multimodal conditioning tuning, particularly through the audio pathway, as a promising direction for motion-aware video editing, and suggest that test-time tuning can serve as a lightweight probing mechanism that helps reveal latent motion controls embedded in the model’s multimodal conditioning.

Primary area:
generative models
Keywords:
video editing, audio-video generative models, video motion editing, generative models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_s2LMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a training-free framework for motion-centric video editing. It optimizes audio latents and residual text conditioning at test time while keeping the audio-video generator frozen. A frozen VLM evaluates the generated video and provides gradients to guide optimization, together with latent, perceptual, and temporal regularization. Experiments on LTX-2 Retake and MiniMax-H3 demonstrate improvements over their prompt-only outputs.

Critical strengths
  1. The observation that the audio-conditioning can help control visual motion is interesting and motivates exploring multimodal conditioning for video editing.
  2. The method is evaluated on two different audio-video backbones, providing evidence of applicability beyond a single architecture.
Critical weaknesses
  1. The technical contribution is limited, and the audio-specific benefit is not sufficiently established. Differentiable feedback from frozen VLMs and truncated backpropagation have been explored in Diffusion-DRF. The main distinction is therefore the choice of audio and text conditioning as optimization variables. However, Supplementary Fig. 15 shows that joint tuning mainly improves naturalness and preservation rather than motion strength, leaving the stronger interpretation of audio as a motion-specific control space insufficiently supported.

  2. The experimental setting does not fully support the claimed scope of video motion editing. The LTX experiments preserve only 1–25 frames of an 89-frame video and regenerate the remaining suffix. This primarily demonstrates prefix-conditioned motion generation, while preservation of unedited dynamics, camera motion, and content appearing later in the source video remains insufficiently tested. The paper also does not clearly establish whether baselines use comparable source-conditioning, making the results difficult to interpret.

  3. The evaluation is too limited to establish broad effectiveness. The main comparison uses only 12 examples. The full LTX set contains 25 examples, including six transfer targets, while H3 is evaluated on a different set. These results provide promising examples but limited evidence of robustness across scenes. Evaluation on a fixed public motion-editing subset, such as that of IVEBench [1], would strengthen the conclusions.

  4. The computational cost limits practical use. Each optimization iteration requires a complete video generation followed by backpropagation. Editing a short clip takes approximately 10 minutes on one H100 for LTX and 20–30 minutes on two H100s for H3, making the method costly and limiting its practical use.

[1] IVEBench: Modern Benchmark Suite for Instruction-Guided Video Editing Assessment. ICLR 2026.

Questions for authors
  1. Under the same cost, how does the method compare with best-of-N, i.e., generating multiple candidates and selecting the best using the same VLM critic?
  2. How sensitive is the editing performance to the location and length of the backpropagation window, particularly the use of only the final two denoising steps for H3?
4 comments

Thank you very much for your precise review and concrete suggestions; both of your questions led to new experiments which we will discuss below.

Q1: Best-of-N at the same cost The controls in the general response already spend our budget on other variables and showed that our method works best. For best-of-N, we rendered prompt-only Retake with N seeds per scenario and kept the sample that the same VLM critic scores highest. Mean yes-probability of held-out VLMs over the 12 scenarios:

MethodQwen3-VLInternVL3
Ours0.730.73
Prompt-only best-of-30.700.70
Ours + best-of-30.810.79
Prompt-only best-of-80.810.88
Ours + best-of-80.950.95

For some motions, no value of N helps: when the prior rarely or never samples the action, more seeds do not help, and no sample makes the rose bloom or the headlights flash periodically.

Best-of-N works when the backbone already produces the action for some seed. Where a seed does produce the action, the critic picks the one with the most motion, which is often abrupt or exaggerated: the turtle's neck collapses, a fruit appears in the monkey's hand, and a fish appears in the tank instead of jumping out.

The two approaches are also complementary rather than competing. Best-of-N selects among samples, while our method changes the conditioning and yields a reusable latent, so they can be combined: applying best-of-N on top of our tuned latents beats all of the other approaches, and at N = 8 both Qwen3-VL and InternVL3 (held-out judges) judge the edit successful in all 12 scenarios. Sample videos are on the supplementary website (https://sound-sparks-motion.supplementary-site.workers.dev/#rb-bon8).

Q2: Sensitivity to the backpropagation window LTX-2 backpropagates through the last 8 of 30 denoising steps. In a sweep over K in {2,4,8,12} on five scenarios, easier edits (a dog yawning, a bird opening its wings) work with any window, while harder ones benefit from a longer window, such as the turtle extending its neck. Memory grows with K, and K=12 already fills an 80 GB H100, so we use K=8. On H3, memory limits us to the last 2 of 16 steps (a 33B model across two GPUs), so we cannot run more than 2 steps with our limited compute, even on a two-H100 setup. However, in K=2 the method still raises task achievement from 46.7% to 68.3% and improves H3 meaningfully.

W1: Technical contribution and the audio-specific benefit We agree with your interpretation of Supp. Fig. 15. The text residual controls what happens (objects and actions), while the audio latent controls how it happens (motion trajectory). Joint optimization therefore improves naturalness, source preservation, and locality rather than simply increasing motion strength. These are important for motion editing, where prompt-only methods often produce abrupt changes, unwanted objects, or drift. The benefit of audio conditioning is further supported by our control experiments (46.3% first-place votes for ours versus at most 15.4% for the controls; see general response) and H3 results (28.9% for audio-only versus 12.2% for text-only).

W2: Prefix conditioning and baseline inputs Retake is the available LTX-2 pipeline for text-driven editing of existing videos. LTX-2 does not provide a general video-to-video editing pipeline. Its IC-LoRA adapters target specific appearance edits and preserve the source structure, making them unsuitable for motion editing. We therefore use Retake for LTX-2. In contrast, MiniMax-H3 supports full-video editing without preserving a prefix. Despite these differences in architecture and editing pipelines, our method improves motion-editing success on both models: from 50% to 83.3% on LTX-2 and from 46.7% to 68.3% on H3. To further support this result, we extended our VLM evaluation using Claude Opus 5.5 to compare Ours+H3 against H3:

Judgement MetricOursTieH3 baseline
Preferred overall68%10%22%
Stronger motion84%6%10%
Edit accomplished74%–46%

W3: Evaluation scope Thank you for the IVEBench pointer. We now report results on all 64 subject-motion items in the general response. The comparison in the paper is backed by human raters and GPT-5.5 Thinking as a VLM judge. For the rebuttal we also added Two more independent judges (mean yes-probability):

MethodQwen3-VL-8BInternVL3-8B
Ours0.730.73
Runway-Aleph0.620.64
LoRA-Edit0.560.51
LTX-2 Retake0.480.56

Also we used Claude Opus 5.5 as a judge, with same prompt as GPT-5.5:

MethodTask successOverall score (0-10)Avg. rank (1 = best)
Ours75%6.032.08
Runway-Aleph58%5.402.79
LoRA-Edit46%5.302.96
LTX-2 Retake46%5.013.12

Gemini-3.1 also ranks ours first in 11 of 12 scenarios.

W4: Computational cost Each iteration costs 1.44 times a forward Retake pass, about ten minutes per edit on LTX-2 and 20 to 30 minutes on H3 across two GPUs. We see the method as test-time control and a probe of the conditioning space, not an interactive editor. However, learned latents can be reused on new videos without re-tuning (Sec. 4.4), which spreads the cost. We will move the runtime table to the main text if needed.

We hope these answers are helpful, and we would be glad to hear your thoughts. Thank you again for your careful review.

Dear Reviewer s2LM,

Thank you again for your concrete suggestions. As the discussion period nears its end, we wanted to highlight the new experiments prompted by your review: the same-critic best-of-N comparison, the backpropagation-window study, and evaluation on all 64 IVEBench subject–motion items. We also clarified what audio conditioning contributes, the source conditioning used by the baselines, and the runtime limitation. If our responses have resolved your concerns, we would be grateful if you could reconsider your score before the discussion period closes; we would also be glad to clarify any remaining points.

Thank you for the responses and additional experiments, which strengthen the evidence for the method’s effectiveness. However, I still find the technical novelty limited beyond existing test-time optimization methods. My concerns about the experimental setting and computational cost are also not fully addressed, leaving doubts about the method’s applicability to practical video editing. I will therefore maintain my original rating.

Thank you again for your thoughtful responses.

Reviewer_qrnhOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents Sound Sparks Motion, a training-free framework for motion-centric video editing using pretrained joint audio-video generation models. The key idea is to leverage the internal audio-conditioning space of a frozen generator as an additional control signal for motion editing. Specifically, the method jointly optimizes an audio-conditioning latent and a residual text-conditioning embedding while keeping the video generator and all other model parameters frozen. The optimization is guided by a vision-language model (e.g., Qwen2.5-VL) that evaluates whether the generated video performs the requested action, together with regularization terms designed to preserve the source content and temporal consistency. The framework is evaluated using LTX-2 Retake and MiniMax-H3. Experiments show improvements over prompt-only generation and several existing video editing methods in motion realization, edit alignment, and source preservation. Ablation studies further demonstrate the benefits of jointly optimizing audio and text conditioning, while qualitative experiments suggest that the optimized representations can transfer motion edits across different videos.

Critical strengths
  • The method substantially improves motion editing performance over existing baselines, achieving a 56.3% human preference win rate and improving task success from 50.0% to 83.3% over LTX-2 Retake.

  • The method operates without updating the pretrained generator and requires optimization of only two conditioning representations. The ablation studies demonstrate that joint audio-text optimization consistently outperforms optimizing either modality independently.

Critical weaknesses
  • The proposed method primarily relies on iterative VLM-guided optimization of audio and text conditioning latents while keeping the pretrained generator frozen. Although effective, the approach resembles repeated refinement of the generated video through test time optimization, rather than introducing a fundamentally new motion control mechanism. The unique contribution of audio conditioning optimization remains insufficiently justified.

  • The method requires approximately 10 minutes of test-time optimization, while most baselines use standard inference. It remains unclear whether the performance gains arise from the proposed audio-text latent optimization or additional computation and iterative VLM feedback.

  • The supplementary videos show convincing audio-motion synchronization in examples involving vocal actions, such as screaming. However, other examples, such as the goldfish, appear to lack meaningful audio cues for the target motion while still producing successful edits. This makes it difficult to assess how much audio conditioning contributes to the observed improvements, beyond text conditioning and VLM-guided optimization.

Questions for authors
  • What distinguishes the proposed approach from conventional iterative refinement through test-time optimization, and what is the unique contribution of audio conditioning optimization to motion control?
  • How can the authors disentangle the benefits of audio-text latent optimization from those of additional test time computation and iterative VLM feedback?
  • For examples without clear audio-related cues, such as the goldfish, how much does audio-latent optimization contribute compared with text-only optimization?
4 comments

Thank you very much for your review, and for finding the gains substantial and the joint audio-text ablation convincing. Your questions go to the heart of the paper.

Q1 and W1: Beyond iterative refinement. What audio contributes?

Our main contribution is using audio conditioning in an audio-video model as a meaningful control for visual motion. The key insight is that what we optimize matters, rather than the optimization algorithm itself. We demonstrate this through the following experiments:

Why audio for motion? To understand why optimizing audio can control motion, we visualized the audio-to-video attention maps of an AV model. As shown in Fig. 2a and Supp. Fig. 9, audio attention is meaningfully related to motion trajectories and end states. This suggests that our results are not simply due to optimization, but rather to the internal properties of AV models, where audio and motion are closely connected.

Is this just more compute or tuning with more parameters? To test whether the improvement comes simply from optimizing more parameters rather than using audio as a meaningful control, we conducted two sets of ablations. First, we optimized the audio branch or the entire model using LoRA to test whether more trainable parameters improve the results. Second, we initialized the audio latents with random values or zeros to test whether audio is just an arbitrary conditioning signal. As shown in Supp. C.9, we evaluated these settings through a human study, keeping the critic, optimization budget, and seed fixed.

MethodCapacity (number of params)Win%Top-3%Borda
Ours4.2M46.382.4271
Random Initialization (seed 42)same as ours15.470.6175
Random Initialization (seed 1)same as ours14.069.1167
LoRA-audio164M13.236.8104
Zero initializationsame as ours9.632.479
LoRA-all428M1.58.820

These results show that more compute or more trainable parameters are not the key. Interestingly, random or zero initialization can sometimes produce motion, but not the intended one. For example, instead of opening the car doors, the model moves the entire car (Supp. C.9). This further suggests that audio and motion are meaningfully connected in AV models, making audio a useful control for motion editing.

PPO vs gradient-based optimization: To further strengthen our point, we also tried an ablation to check whether any kind of iterative refinement works, or the key is gradient-based refinement so that the audio (as a control) can get meaningful feedback and adapt. Supp. G.4. shows that ours achieves much better results than a PPO version of the method.

Why not optimize noise?: A natural alternative is to optimize the initial noise, as in DNO and ReNO. We tested both gradient-free and gradient-based approaches. For gradient-free optimization, we used best-of-N sampling with the same critic (Reviewer s2LM, Q1). The results show that best-of-N alone does not outperform our method, but it is complementary and can further improve our results. For gradient-based optimization, we optimized the initial noise of the Retake segment through all 30 denoising steps, using the same critic while keeping audio and text conditioning fixed. We also constrained the noise norm to keep it close to a Gaussian distribution. We tested four edits with 4–5 learning rates each:

MethodQwen2.5-VLQwen3-VLInternVL3Edit success rate
Ours0.820.990.774 / 4
Noise optimization, learning rate chosen by its own critic0.430.500.441 / 4
Noise optimization, best fixed learning rate0.630.660.701 / 4
LTX-2 Retake0.180.270.260 / 4

Our method succeeds on all four edits, while noise optimization succeeds on only one, regardless of how the learning rate is selected. Ours also achieves higher scores across all three VLM judges, including 0.99 on Qwen3-VL compared to 0.66 for the best fixed learning rate. The main issue with noise optimization is its instability. Since noise controls the entire generated segment, changing it can affect the scene rather than just the intended motion. For example, opening the car door also changes the camera angle and car appearance, the turtle video sometimes collapses to blank frames or cuts away, and the man never pets the dog. The results are also highly sensitive to the learning rate. Higher rates can break the scene, while lower rates produce little or no editing. Although some runs produce good results, such as the dog yawning, they require careful tuning and are not consistent across examples. Noise optimization also costs about 1.6× more than ours, increasing to up to 8× when searching for a suitable learning rate. In contrast, our method optimizes audio latents that the model already uses to condition motion. This provides more stable control while better preserving the source video. The optimized audio latents can also be transferred to new videos (Sec. 4.4). Samples are on the supplementary website (https://sound-sparks-motion.supplementary-site.workers.dev/#rb-noiseopt).

Q2 and W2: Separating the method from extra compute The controls above use the same compute and feedback, so the gap between them is not a compute effect. Another suggestion for extra compute ablation, suggested by Reviewer s2LM (Q1) is to report best-of-N with the same critic. We showed that even best-of-N sampling alone cannot beat our method, and it is in fact complementary to our approach and can improve our results further. We agree that the runtime of about ten minutes per edit is a real limitation for interactive use; we already state it in Sec. 5 and will move the runtime table from appendix to the main text if needed.

Q3 and W3: The goldfish example We do not claim that the motion must have a sound. The audio latent is a temporally structured conditioning that the model uses for motion regardless, and the attention maps (Supp. Fig. 9) show this for the fish too. To answer directly, we tuned audio only and text only on the exact goldfish clip (mean yes-probability):

VLMAudio-only tuningText-only tuning
Optimization critic (Qwen2.5-VL)0.250.18
Qwen2.5-VL0.560.29
Qwen3-VL-8B0.980.75
InternVL3-8B0.660.55

In the modality ablation, joint tuning also gives the cleanest goldfish result (7.5 vs. 7.0 for text-only). So even without a natural sound, audio carries much of the control.

We would be grateful for your thoughts on whether this addresses your concerns. Thank you again for your time and your review.

Dear Reviewer qrnh,

Thank you again for your review. As the discussion period nears its end, we wanted to follow up on the experiments in our response. With the critic and optimization budget held fixed, the audio-latent ablations outperform random initialization and alternatives with more trainable parameters. We also compared against noise optimization and tested audio-only versus text-only tuning on the goldfish example. If our responses have resolved your concerns, we would be grateful if you could reconsider your score before the discussion period closes; we would also be glad to clarify any remaining points.

Thank you for your detailed responses and additional experiments. I appreciate the authors' efforts, which have addressed many of my original concerns.

However, I remain somewhat unconvinced about the technical novelty of the proposed approach. While the additional experiments demonstrate the effectiveness of audio latent optimization, from my perspective, it is still unclear what makes optimizing audio latents a sufficiently distinct technical contribution beyond existing test time tuning methods.

Therefore, I will maintain my original rating. Thank you again for your thoughtful responses and efforts.

Reviewer_NS7jOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Sound Sparks Motion, a training-free approach for motion-centric video editing in pretrained audio-visual generators. Instead of changing model weights, the method tunes only two small conditioning variables at test time: the audio latent (initialized from the source) and a residual on the text-conditioning vector, guided by a frozen VLM (Qwen2.5-VL) that answers whether the requested action is present. With regularization and perceptual-temporal constraints, the approach substantially improves edit success for motion-centric instructions on LTX-2 Retake and MiniMax-H3, and shows limited transferability of the learned latents across related videos.

Critical strengths

1, Technical novelty

a, Using the internal audio pathway of joint audio-video generators as a latent handle for controlling motion is a simple but fresh idea that leverages cross-modal entanglement productively.

b, The VLM-based binary motion supervision defines a pragmatic, differentiable test-time objective for action realization when no general-purpose text–motion alignment model is available.

2, Experimental validation

a, Human studies and VLM-based evaluations both report consistent gains over strong baselines, including a commercial editor and a LoRA-based first-frame editor.

b, The method is instantiated on two distinct backbones (LTX-2 Retake and MiniMax-H3) with consistent improvements, indicating generality.

Critical weaknesses

1, Technical limitations

a, Reliance on a single VLM critic raises vulnerability to “critic hacking” and may bias edits toward the critic’s blind spots; the paper partially mitigates this with regularizers but does not explore multi-verifier setups.

b, The approach implicitly treats audio as a disposable control signal; implications for synchronized audio output or editing when source audio is absent or mismatched are underexplored.

2, Experimental gaps

a, The main comparisons span 12 scenarios for all baselines and 25 for partial evaluations, which is relatively small

b, Objective video quality and temporal coherence metrics (e.g., FVD, VBench temporal stability, optical-flow consistency) are not reported, relying primarily on VLM-based scores and human preferences.

c, Transferability is shown qualitatively on a few examples without a systematic study (e.g., how often does transfer succeed across subjects, scenes, and styles?).

3, Clarity

The exact form/dimension and injection point of the residual text-conditioning variable are under-specified in the main text.

4, Missing related works

Papers already refer to Diffusion-DRF, which has used logit feedback of frozen VLM for differentiable video optimization, so the VLM feedback itself cannot bear the main novelty. In addition, ReNO and DNO, which are not discussed, have studied reward-guided noise optimization in the inference stage; DNO also explicitly handles out-of-distribution reward hacking in optimization.

Questions for authors

1, How robust is the method to the choice of VLM critic? Have you tried a second verifier (e.g., a temporal-reasoning or synchronization signal) in a small multi-critic setup to evaluate susceptibility to critic hacking?

2, Could you quantify transferability (success rate across k new videos per learned latent, with varying subject/scene/style gaps) and provide a simple diagnostic for when a learned motion latent is likely to transfer?

3, Where exactly is the residual text-conditioning injected (global token vs. per-layer cross-attention conditioning)?

4 comments

Thank you very much for your detailed review, and for finding the audio pathway a simple but fresh control. We answer your points below.

Q1 and W1a: Robustness to the critic and critic hacking To address this concern, as requested by the reviewer, we now also scored all outputs with two different VLMs that play no role in optimization:

MethodQwen3-VL-8BInternVL3-8B
Ours0.730.73
Runway-Aleph0.620.64
LoRA-Edit0.560.51
LTX-2 Retake0.480.56

Also we used Claude Opus 5.5 as a judge, with the same prompt as our GPT-5.5 judge:

MethodTask successOverall score (0-10)Avg. rank (1 = best)
Ours75%6.032.08
Runway-Aleph58%5.402.79
LoRA-Edit46%5.302.96
LTX-2 Retake46%5.013.12

Both Qwen3-VL-8B and InternVL3-8B rank ours first (0.73), compared to 0.62 and 0.64 for the closest competitor. Claude Opus 5.5 also ranks ours first, with 75% task success compared to 58% for the closest competitor. Gemini-3.1 ranks ours first in 11 out of 12 scenarios. Importantly, the optimization critic that we used in our optimization (Qwen2.5) does not rank ours first or give it a perfect score. This shows that our method is not simply exploiting its preferences. The control methods discussed in our general response also use the same critic but perform worse. Similar to DNO and ReNO, we use LPIPS, temporal, and latent-norm regularizers (Sec. 3.4) to keep the edits close to the source and reduce the risk of critic hacking.

Following your suggestion, we additionally optimized against two critics with different architectures (Qwen2.5-VL-7B + InternVL3-8B, equal weights, everything else unchanged). Results judged by Qwen3-VL-8B (a held-out verifier):

EditQwen3-VL yes-prob. (Retake / 1 critic / 2 critics)
Turtle extends its neck0.63 / 1.00 / 1.00
Dog yawns0.64 / 1.00 / 1.00
IVEBench "take off"0.98 / 1.00 / 1.00
IVEBench "wing flap"1.00 / 1.00 / 0.95

Using one or two critics produces almost the same results in terms of held-out scores and source preservation. This suggests that our method is not exploiting Qwen2.5-VL, as such edits would be unlikely to perform equally well with a second critic using a different architecture. However, we also observe that the two critics can complement each other. For example, in the turtle case, InternVL3 correctly recognizes the neck extension (0.97), while Qwen2.5-VL assigns a very low score (0.02). Overall, using multiple critics can provide more reliable feedback and help generate higher-quality, more natural motions, especially for challenging edits, at the cost of additional computation.

Q2 and W2c: Quantifying transferability To address your question, we reused 5 learned latents on 33 new target-action videos that vary subject, species and style (real, generated, cartoon and film), with no tuning on the targets and we compare edit success rate:

VLMTransferred latent (ours)Prompt-only Retake
Qwen2.5-VL88%73%
Qwen3-VL94%85%
InternVL391%82%

Overall, transfer improves task success rate meaningfully, without any overhead in computation. The gain is especially largest when prompt-only Retake under-realizes the requested action, particularly for challenging motions such as shouting. When Retake already succeeds, transfer tends to make smaller changes, with broadly comparable visual naturalness rather than a consistent quality gain. (https://sound-sparks-motion.supplementary-site.workers.dev/#rb-transfer).

Q3 and W3: Where the text residual is injected Δv\Delta v is a dense per-token residual on LTX-2's video text context after the Gemma connector (1 x 1024 x 4096). It is added to the positive prompt, which feeds the text cross-attention of every video block, so it acts as per-layer conditioning. The audio-stream text encoding and the negative prompt are unchanged. On H3, the residual is added to the Qwen3-VL prompt embeddings. We will state this in Sec. 3.2 of paper if needed.

W1b: Should tuned audio be necessarily meaningful? Our goal is visual motion editing, where audio latents serve as a temporal control signal. Our method does not guarantee meaningful or synchronized audio, since we optimize for visual motion rather than sound. Optimization in the audio latent space can find solutions that achieve the desired motion without producing meaningful audio. Interestingly, H3 often generates sounds consistent with the edit, such as screams, a cat falling, or laughter, suggesting differences between the latent spaces of H3 and LTX. However, meaningful audio generation is not explicitly optimized.

W2: Evaluation size and objective metrics Besides the 25 LTX-2 and 25 H3 edits, we now evaluate on IVEBench (please see the general response, 2nd paragraph). As requested by the reviewer, we extend the objective metrics on the 12-scenario comparison:

MetricOursRetakeUnedited input
Motion, RAFT flow (px/frame)0.510.480.25
Motion, VBench continuous dynamic degree0.980.940.64
Imaging quality, MUSIQ0.6880.6860.699
VBench subject consistency0.9520.9650.976
VBench temporal flickering0.9870.9880.992
Aligned-frame DINOv2 similarity to the source0.8270.8861.000

Ours produces the highest average optical-flow magnitude, without reducing image quality. VBench's consistency and flicker metrics compare a video with itself. Therefore, they favor videos with less motion, so the unedited source gets the highest scores. For motion editing, these metrics should be considered together with human and VLM evaluations that measure edit success and source preservation. We also use DINOv2 to measure similarity between the source and edited videos. Ours scores 0.827 compared to Retake's 0.886. However, successful motion edits naturally change the source video and can reduce this similarity.

W4: Related work ReNO and DNO optimize the initial noise, while we keep the noise fixed and optimize the conditioning. Our controls show that this choice matters under the same reward. We will add both works and cite them and note that DNO's out-of-distribution regularization parallels our regularizers. We also do not state that using VLM critic is a novelty of ours (and thus, already cite Diffusion-DRF in our paper). Instead, we are using a VLM critic and solving a completely different task by introducing a new control mechanism in AV models. Our contribution is not to show how to optimize, but rather show that what we optimize matters.

We would be grateful for your thoughts, and we are happy to discuss any point further. Thank you again for your careful review.

Dear Reviewer NS7j,

Thank you again for your detailed review. As the discussion period nears its end, we wanted to follow up on our responses. We added independent VLM evaluations and a two-critic test, quantified transfer across 33 new videos, and provided the requested objective metrics and implementation details. If our responses have resolved your concerns, we would be grateful if you could reconsider your score before the discussion period closes; we would also be glad to clarify any remaining points.

Thank you for the detailed responses and additional experiments. They help clarify the method and provide some evidence of effectiveness. The concerns regarding the experimental setting and method novelty only partially addressed. I will therefore maintain my original rating.

Thank you again for your thoughtful responses and efforts.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response for Rebuttal

Thank you very much for your careful reviews and for the time you spent on our paper. Your questions led to several new experiments. There were two shared concerns: whether audio conditioning provides a useful control for visual motion, and whether the evaluation supports the paper’s claims. We address these concerns below, followed by reviewer-specific replies.

Is audio really a good handle for motion? The reviewers raised an important question about the contributions of the audio pathway and VLM feedback to the observed improvements. We have investigated this question from several perspectives throughout the paper, and here we provide the supporting evidence to clarify the role of each component.

The model uses audio to guide motion. In the attention maps (Fig. 2a, Supp. Fig. 9), text attention stays on subjects and objects, while audio attention follows the motion trajectory, including for actions without an obvious sound, such as a turtle extending its neck. The tuned latents show the same split (Supp. Fig. 14): audio latents group by motion, while text residuals group by object.

Optimizing other variables does not work. The controls in Supp. C.9 keep the critic, iterations, learning rate and seed fixed, and change only what is optimized:

VariantTrainableFirst-place votes (humans)InternVL3 yes-prob. (held out)
Ours (source-audio init + text residual)4.2M46.3%0.94
Random audio init (two seeds)4.2M15.4% / 14.0%0.79 / 0.67
Zero audio init4.2M9.6%0.75
LoRA, audio branch164M13.2%0.65
LoRA, whole DiT428M1.5%0.38

It’s insightful that random initialization often produces a motion, but the wrong one: the whole car moves instead of its door opening (please refer to the inline fig in Supp. C.9.). This further suggests the entanglement of audio and motion, while suggesting the fact that where we start tuning should be a related audio to our video.

It generalizes. On MiniMax-H3, a different architecture that edits the full clip, the same recipe raises task achievement from 46.7% to 68.3%, and audio-only tuning beats text-only tuning (28.9% vs. 12.2% win rate). The learned latents also transfer to new videos (please see our reply to Reviewer NS7j, Q2). These results on H3 provide further evidence of a connection between audio and motion, suggesting that audio-based optimization can be effective across different architectures.

Evaluation scale and a public benchmark.

Most motion-editing benchmarks condition on a reference, such as a driving video, a trajectory or key frames; few test edits from an instruction alone. Following Reviewer s2LM's suggestion, we ran our method and prompt-only Retake on all 64 subject-motion items of IVEBench with the official evaluation. Our method targets complex motions that cannot be reliably generated by the backbone using text prompts alone. IVEBench mixes these with two other kinds: edits close to the source's own motion (easy and local motions), where both methods succeed, and edits that need a new entity or effect (i.e. an explosion) , where both fail. For the full actions in between, such as dancing, fighting or standing up, our result is more complete and natural and is preferred by a VLM judge.

Condition, judged by Claude Opus 5.5 VLMNumber of Items
Ours clearly better15
Ours slightly better, not significant6
Both succeed, about equal17
Retake better2
Both fail24
Official IVEBench metricOursRetakeUnedited source
Instruction satisfaction (1-5), all items3.113.001.69
Instruction satisfaction, 31 items with a visible edit4.454.132.29
Items where the judge scores this method higher114-
Compliance (instruction following)0.4030.3870.188
Quality0.8020.8030.769
Fidelity0.7360.7600.760
Total0.6470.6500.572

IVEBench metrics evaluate two aspects: instruction satisfaction and following and visual quality. The quality metrics compare neighbouring frames, so videos with little or no motion can receive higher scores. This is not ideal for motion editing, especially for complex motions where larger changes are expected. We therefore suggest considering all metrics together, with more emphasis on instruction following and satisfaction. The fidelity gap mainly comes from inconsistencies in IVEBench's content-fidelity judgement. For example, it gives the unedited source a score of only 1 out of 5 on 13 items, even when compared with itself. The two methods receive the same fidelity score on 58 of 64 items. Most of the difference comes from just two items where the videos are nearly identical, but ours receives 1 while Retake receives 4–5. As shown in the tables above, the judge never prefers Retake when a visible edit is present. Overall, our method achieves the requested motion better on 21 items and performs worse on only 2, although this comes with a small decrease in quality metrics. Videos of some of these cases are on the supplementary website (https://sound-sparks-motion.supplementary-site.workers.dev/#rebuttal).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.