acceptodds
Withdrawn from ICLR 2027

MotionCoach: Constructing Multimodal Guidance for Video Professional Motion Editing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video editing models achieve strong performance in modifying appearance, backgrounds, styles, and motions, empowered by large-scale training data. Beyond general scenarios, professional motions are significant in performance, sports, and martial arts. This paper focuses on professional motion editing, which involves fine-grained motion technique and temporal structure. Given professional motion names but lacking detailed descriptions, existing models often fail to generate professional motions, such as incorrectly lifting the front wheel under a Stoppie instruction. To address this challenge, we introduce MotionCoach, a trained multimodal agent that uses multi-turn tools and performs self-verification, producing a grounded prompt and a reference clip as reliable and sufficient multimodal guidance to inject external professional motion knowledge into video editing models. To supply training data, we build a data pipeline to construct trajectories. We cold-start the agent on MotionCoach-SFT-10k and apply Group Relative Policy Optimization (GRPO) on MotionCoach-RL-2k with rewards on the motion guidance prompt, the motion outcome video, and the format. For evaluation, we introduce VPMEBench, comprising 314 professional motion editing tasks with fine-grained references and annotations. Extensive experiments show that MotionCoach improves the overall score from 55.1 to 65.6 on Seedance 2.0 and from 51.2 to 64.5 on MiniMax-H3. Learning to construct multimodal guidance enables precise professional motion editing without retraining video editing models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Agent, Vision-Language Models, Video Editing, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FLE1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies professional motion editing, where video editing models given only the name of a professional motion often produce the wrong motion. It proposes MotionCoach, a Qwen3-VL-8B agent that uses search, inspect, and clip tools with self-verification to produce a short reference clip and a detailed grounded prompt, which are fed to a frozen video editor. The agent is trained by SFT on teacher-generated trajectories, then GRPO with prompt, video-outcome, and format rewards. The paper also introduces VPMEBench, 314 human-verified tasks over 25 categories scored by a VLM judge, and reports overall gains from 55.1 to 65.6 on Seedance 2.0 and 51.2 to 64.5 on MiniMax-H3.

Critical strengths
  • The paper proposes MotionCoach, an agent trained with SFT and GRPO (prompt, video-outcome, and format rewards) that retrieves and verifies a reference clip and writes a grounded prompt, so frozen video editors can execute named professional motions.
  • The paper introduces MotionCoach-SFT-10k and RL-2k, teacher-generated trajectories with VLM checks and human spot-checks, and VPMEBench, 314 human-verified tasks over 25 categories with reference clips, expert keyframes, and technique annotations, whose VLM judge is validated against domain experts.
  • The paper achieves the best results on VPMEBench, raising the overall score from 55.1 to 65.6 on Seedance 2.0 and from 51.2 to 64.5 on MiniMax-H3, supported by ablations on training stages, inputs, agent components, and reward weights.
Critical weaknesses
  • Claim–method mismatch. The introduction defines the gap as acquiring and verifying external motion knowledge. The method that fills this gap is a fixed workflow made of a system prompt, three tools over a pre-built index, and a self-check. This workflow exists before any training, because the teacher, Gemini 3.5 Flash, already runs it end to end to produce the training data. The paper’s contribution, however, is to train an 8B policy to imitate those teacher trajectories, which answers a different question, whether a small model can reproduce the teacher, that the introduction never raises. The two sides are left unreconciled by the experiments. On retrieval, no baseline lets an LLM write the same detailed prompt from its own knowledge without search or clip, although on MiniMax-H3 removing the reference clip costs only 0.8 points and on IVEBench the score improves even though the professional-motion corpus cannot match its generic instructions. On training, the untrained agents compared are only Qwen3.5-9B and 27B, while the teacher and the other flagship models already used in the pipeline, GPT-5.5 and Claude Opus 4.7, are not run as agents. If Gemini 3.5 Flash solves the task, the trained policy adds at most an unmeasured cost advantage. If it does not, the SFT trajectories and the teacher prompts used as RL reward targets are unreliable.

  • Human-only scope. The paper claims that video editing models fail at professional motion editing in general, and the title, abstract and contributions keep this general framing. Every piece of evidence, however, is a person performing a dance, sports or martial-arts move. The data preparation keeps only human-centered videos, the taxonomy has only these three domains, and the judge prompt is written for human motion. The restriction is never stated, and no limitation is given. Within this human scope, pose-driven human animation and reference-based video motion editing take exactly a source video plus a reference motion and directly target the motion correctness this paper measures, yet this line of work is neither cited nor compared.

  • Results rest on a self-built benchmark. All main results and ablations are reported on VPMEBench, which the authors construct, score with a judge of their choice, and aggregate with weights of their choice. The only human evaluation checks agreement between this judge and annotators on 100 pairs and does not compare methods. The single external result, on the subject-motion subset of IVEBench, does not add support. It uses one editor, compares only the base model with and without MotionCoach, gives neither the subset size nor the definitions of its ten metrics, and its total gain comes mainly from one metric while two others decrease. Its instructions are generic motions that the professional-motion index cannot match, so the improvement cannot come from retrieved professional knowledge and does not test the paper’s claim.

  • Limited novelty. Retrieval-augmented generation and visual-outcome-trained agents exist and are cited, and the new ingredient, the video reward, adds 0.7 points. The methods used for positioning, RS-Gen, VACE and Kiwi-Edit, are not compared. There is also no lookup baseline without an agent, i.e. exact-match retrieval, fixed clipping and an LLM-written prompt, which would show whether the agent’s decisions add anything beyond the pre-built index.

Questions for authors
  • How does search work? What is the source, size and labeling of the reference index, does it contain the tutorials behind the VPMEBench reference clips, and will it be released?

  • How were the training data collected and filtered? How many trajectories were human spot-checked, by whom, and with what acceptance rate? Who wrote the technique and negative-constraint annotations in VPMEBench?

  • What are the details of the 100-pair human study? Which pairs were sampled, what were the human preference rates, and why are the Table 1 baselines not included?

4 comments

For Reviewer FLE1 (R3)

To W1 (claim-method mismatch):

Adaptive policy and outcome-based learning. MotionCoach learns temporal grounding through frozen-editor feedback, rather than following a fixed workflow: it adaptively selects queries, references, intervals, and refinement actions. Teacher trajectories initialize tool use through SFT; RL optimizes guidance with prompt and edited-video rewards (Sections 3.1–3.3). Table 4 shows further gains from RL beyond SFT.

Added comparisons. The no-retrieval LLM writes a prompt from its own knowledge, without search or clips. Teacher and flagship agents share MotionCoach's tools, 65,536-token context budget, and 10-round limit.

ConfigurationSD2H3
OverallMPPPVQOverallMPPPVQ
No-retrieval LLM prompt51.0022.6184.7175.8554.6713.8592.2097.66
Gemini 3.5 Flash agent (teacher)61.5355.8978.0359.9360.7536.9482.4885.93
GPT-5.5 agent58.9251.9173.4160.9358.8138.2276.2781.48
Claude Opus 4.7 agent64.3955.8979.3068.6364.2443.1582.1787.42
MotionCoach (8B)65.6153.0387.1072.2464.4633.192.298.2

On SD2 / H3, MotionCoach gains 14.61 / 9.79 Overall and 30.42 / 19.25 MP over no retrieval, and 4.08 / 3.71, 6.69 / 5.65, and 1.22 / 0.22 Overall over Gemini, GPT, and Claude. These gains show that editor-feedback training can surpass the teacher's aggregate performance.

Clip ablation. Table 5 removes the editor's reference clip but retains the prompt grounded in the searched video. Most external motion evidence is already encoded in the prompt; the clip supplies supplementary visual guidance.

To W2 (scope and comparisons with motion-control methods):

Why professional motion editing and human-motion scope. IVEBench identifies “more challenging editing types, such as subject motion editing” and notes that “the editing capability of current models is particularly limited” (Sec. 6.2). MiniMax-H3 scores only 48.0/100 Total on this subset (Table 9). On VPMEBench, Seedance 2.0 scores 55.1/100 Overall and 29.6/100 MP (Table 1). We therefore focus on human professional motions in dance, sports, and martial arts, and will clarify this scope and limited non-human coverage.

Motion-control comparisons. Pose or reference-conditioned methods “assume that suitable reference is already available” (Introduction, p. 1, L049–052). MotionCoach takes only a source video and short instruction, then searches for demonstrations, localizes relevant intervals, and verifies motion correctness, visibility, and completeness. It addresses guidance acquisition and verification that those input settings leave to users.

To W3 (benchmark credibility and evaluation robustness):

Evaluation protocol. Sections 3.1 and A.3.2–A.3.4 disclose the rubric, scoring levels, and aggregation weights. Human preferences agree with the automatic judge, with Overall QWK = 0.846 (Table 10); Q3 details the study design. Reward and evaluation use different judge models: Qwen3-VL-30B-A3B computes RL rewards, while Gemini 3.5 Flash performs evaluation and also serves as the teacher.

IVEBench. Both editors are evaluated on the same 64 subject-motion tasks from its short subset. We retain its video-quality (SC, BC, TF, MS, VTSS), instruction-compliance (OSC, PSC, IS), and video-fidelity (SF, CF) metrics. Total averages the three dimension scores, each a weighted mean of normalized applicable metrics (Sec. 4 and Appendix C); we report it on a 100-point scale.

Online search uses each editing instruction, so generic motions are not excluded by a fixed professional-motion index (Q1).

ModelTotalSFCFISOSCVTSSSCTFPSCMSBC
MiniMax-H348.069.51.402.7319.36.490.996.519.198.995.3
MiniMax-H3 + MotionCoach53.895.32.052.6224.56.192.397.519.299.095.2
Seedance 2.049.7292.862.052.1023.744.8893.7497.9119.2199.1795.90
Seedance 2.0 + MotionCoach57.2094.262.482.8124.145.9193.3297.5019.2599.0895.53

MotionCoach raises Total by 5.8 on MiniMax-H3 and 7.48 on Seedance 2.0. H3 gains mainly in SF and OSC; Seedance also improves IS and CF. These aggregate gains support transfer to generic motion instructions across both frozen editors.

To W4 (methodological contribution and novelty):

Our core contribution is a policy that learns temporal grounding for professional motion guidance, with feedback from a frozen video editor shaping how it selects and organizes temporal evidence. MotionCoach also enables Editor-in-the-loop RL with cross editor transfer: the policy is trained with video rewards from MiniMax-H3, but its guidance raises Seedance 2.0 MP from 29.6 to 53.0 (Table 1).

The lookup baseline uses Exact-Match Retrieval → Fixed Clipping → LLM-Written Prompt, with code-controlled execution and the same candidate references, video observations, and inference budget, but no adaptive reference selection or temporal refinement.

ConfigurationSD2H3
MPPPVQOverallMPPPVQOverall
Base editor29.685.877.155.116.284.787.351.2
Fixed retrieval–clipping–captioning31.0677.7854.5547.4531.4785.9436.9044.00
MotionCoach (Ours)53.0387.1072.2465.6133.192.298.264.46

MotionCoach gains 18.16 / 20.46 Overall points on SD2 / H3 over this fixed pipeline. This supports learned temporal decisions beyond reference access and reliable tool execution.

Video reward. With matched training budgets, the table reports means and 95% CIs across three independent training seeds.

ConfigurationOverall [95% CI]MP [95% CI]PP [95% CI]VQ [95% CI]
RL: prompt reward only64.70 [64.03, 65.37]53.56 [51.53, 55.59]84.45 [79.95, 88.94]70.11 [68.78, 71.45]
RL: prompt + video rewards65.28 [64.48, 66.07]53.98 [51.61, 56.36]85.72 [82.16, 89.28]70.47 [66.24, 74.70]

All metric means improve with mixed rewards. The mean Overall gain is +0.58 [95% CI: −0.10, 1.26].

To Q1 (search implementation and index overlap):

MotionCoach searches Bilibili online using the editing instruction, rather than a pre-built index (Introduction, p. 2, L072–075; Ethics Statement, p. 11, L556–557). SFT, RL, and benchmark splits have disjoint target motion names, source videos, and edit prompts (Section 3.1, L195–196).

To Q2 (training-data filtering and benchmark annotations):

Teacher trajectories undergo programmatic validation of formats and tool arguments, VLM checks, and human spot-checks (Section 3.1). VLM checks cover visible and complete motion phases/mechanics and prompt–clip consistency.

Of 14,621 candidates, 12,686 (86.77%) passed programmatic and VLM checks. Five experts in sports, dance, and martial arts spot-checked 500 randomly selected cases, with a 94% pass rate.

For VPMEBench, Gemini 3.5 Flash drafted motion-technique and negative-constraint annotations; five experts reviewed and corrected every item.

To Q3 (human-study design and method comparisons):

Category-stratified sampling. Appendix A.1 (p. 14, L725–734) uses stratified random sampling of 100 Base vs. Base+RL pairs: 50 per editor, with 46 dance, 34 sports, and 20 martial-arts pairs. Each pair shares a source video and instruction and is independently rated by three of eight experienced annotators, with randomized, anonymized A/B order.

Human preferences. Annotators compare Base+RL with the same frozen editor's Base output and assign Good / Same / Bad (G/S/B) judgments for MP, PP, VQ, and Overall. We aggregate the three judgments per pair by their ordinal median (Bad < Same < Good). For each editor and dimension, we count G, S, and B across the 50 pairs and compute the GSB ratio R = (G + S) / (S + B). R > 1 indicates more improvements than regressions relative to Base. For MiniMax-H3, MotionCoach achieves an Overall GSB ratio of 2.10 over Base.

Choice of comparisons. Comparing the same editor with and without MotionCoach measures the incremental value of agent-constructed guidance while controlling for the generation model. Following the reviewer's suggestion, we additionally compare MotionCoach + MiniMax-H3 against Kiwi-Edit and Ditto from Table 1 using 50 video pairs per comparison. MotionCoach achieves Overall GSB ratios of 6.25 and 5.00, respectively, indicating more favorable than unfavorable judgments against both baselines.

We hope the clarification and additional experiments have sufficiently addressed your concerns. If you are satisfied, we kindly invite you to consider adjusting the score accordingly. We will also remain responsive to any further issues during the discussion phase.

Thank you for the rebuttal. It resolves part of my concerns, but several important issues remain.

  1. The paper demonstrates improvements in editing quality and tool-execution success. However, it remains unclear whether training also improves verification, reference selection or temporal localization.

  2. There are still no comparisons with related specialized methods for human motion editing.

  3. I spent considerable time examining the implementation details described in the paper but the search procedure remains insufficiently unclear. In particular, the original paper does not clearly specify the online Bilibili search implementation described in the rebuttal. The clarification leaves important questions about candidate retrieval and processing unanswered. What's more, the authors have not provided any code to help verify these details.

Therefore, I maintain my original score.

Reviewer_FsQQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies professional motion editing from short, specialized action names. MotionCoach constructs a grounded prompt and a verified reference clip through a search–inspect–clip workflow, and trains an 8B policy with SFT followed by GRPO.

Critical strengths

The application is well motivated, and the benchmark provides useful motion-specific references and annotations. The paper presents a systematic effort encompassing data collection, agent training, and benchmark construction and evaluation, resulting in a well-developed system with promising empirical performance. The paper also includes meaningful ablations, cross-editor transfer, and a domain-expert pairwise validation study.

Critical weaknesses
  1. Insufficiently established methodological novelty and distinctive contribution. The paper’s core pipeline—retrieving external references, performing visual verification, constructing fine-grained prompts, and training through teacher-generated trajectories followed by SFT and RL—closely resembles existing work on agentic generation and editing. Temporal clipping and verification of motion-specific technical details are meaningful task-specific designs. However, the paper does not sufficiently clarify how these designs go beyond adapting existing paradigms to a new domain, or what new, transferable insights they provide. Thus, despite the system’s practical value, its distinctive methodological contribution remains unclear.

  2. Lack of evidence for the effectiveness of the core mechanisms. Although the experiments demonstrate improvements from the complete system and include several ablations, the results do not yet clearly disentangle the gains attributable to external knowledge, teacher distillation, and subsequent RL. For example, in Table 2, RL improves over SFT by 3.3 and 1.7 points on Seedance 2.0 and MiniMax-H3, respectively. In Table 7, the best combined-reward setting outperforms the prompt-only reward by just 0.72 points. These results do not imply that the mechanisms are ineffective, but they do not sufficiently establish their consistent, independent incremental value. Strong retrieval-and-prompt-construction baselines with access to the same information and matched inference budgets, together with a continued-SFT baseline on the RL task set and uncertainty estimates for the key differences, would provide more convincing evidence for the value of the core designs.

Questions for authors
  1. Explanation for distinctive contribution. Could you clarify which aspects of MotionCoach go beyond a task-specific adaptation of existing agentic generation/editing approaches, such as GenAgent and WeAgent-MMGenEdit? In particular, what new capability or transferable insight arises from the proposed temporal clipping and motion-specific verification, and which results best support this distinction?

  2. Incremental value of RL and outcome-video rewards. Table 2 reports gains of 3.3/1.7 points from RL beyond SFT, while Table 7 shows a 0.72-point advantage of the best combined reward over prompt-only rewards. Could you provide evidence that these gains are consistent, and clarify what capabilities they improve? A continued-SFT control on the RL task set and uncertainty estimates for the key differences would help distinguish the contribution of RL from additional data and training.

  3. Comparison with a simpler pipeline. The untrained-workflow and policy-scaling comparisons are useful. However, could a strong fixed retrieval–clipping–captioning pipeline achieve similar gains when given the same candidate references, video observations, and inference budget? A comparison with reliable tool execution would help establish whether the benefits come from learned adaptive decisions rather than primarily from richer guidance or improved tool-call reliability.

I would be open to increasing my score if the response clarifies the distinctive contribution and provides convincing evidence that the proposed mechanisms offer consistent gains beyond strong, information- and budget-matched alternatives.

6 comments

For Reviewer FsQQ (R2)

To Q1 / W1 (distinctive contribution and key component):

Distinctive contribution. Our core contribution is a policy that learns temporal grounding for professional motion guidance, with feedback from a frozen video editor shaping how it selects and organizes temporal evidence. Unlike GenAgent and WeAgent-MMGenEdit, which focus on image generation and editing, MotionCoach must additionally reason about temporal phases and their chronological order to guide motion editing. MotionCoach also enables Editor-in-the-loop RL with cross editor transfer: the policy is trained with video rewards from MiniMax-H3, but its guidance raises Seedance 2.0 MP from 29.6 to 53.0 (Table 1).

ConfigurationSD2H3
MPPPVQOverallMPPPVQOverall
Base editor29.685.877.155.116.284.787.351.2
Existing agentic approach (inference only)25.0056.6944.8037.2829.4658.2825.3233.98
Existing agentic approach (trained)24.8878.8466.1248.0526.2979.1251.3744.38
MotionCoach (Ours)53.0387.1072.2465.6133.192.298.264.46

Both baselines use Search → VLM Verification → Prompt Rewriting, with or without task-specific training. MotionCoach gains 17.56 / 20.08 Overall points over the trained variant and 28.33 / 30.48 over inference only on SD2 / H3; MP, PP, and VQ also exceed the trained variant. These gains support our temporal grounding design.

Key components. Static-image search replaces video search; removing temporal evidence retains video search but uses one reference image. Without temporal clipping, interval selection uses uniform sampling; without motion-specific verification, motion-phase/mechanics checks and recovery are disabled.

ConfigurationSD2H3
MPPPVQOverallMPPPVQOverall
MotionCoach (Ours)53.0387.1072.2465.6133.192.298.264.46
w/ static-image search4.8846.9575.6134.517.1447.6294.0541.31
w/o temporal video evidence41.7973.5264.1754.8524.9173.7180.2751.28
w/o temporal clipping38.8586.4469.0257.4221.8385.8886.5154.05
w/o motion-specific verification41.2585.1468.1158.0922.3387.4286.6354.64

Static-image search has reported Overall gaps of 31.10 / 23.15 points. Removing temporal evidence, temporal clipping, and motion-specific verification lowers Overall by 10.76 / 13.18, 8.19 / 10.41, and 7.52 / 9.82 on SD2 / H3.

New capability and insight. Our results suggest mutual improvement between video understanding and generation: VLM-based temporal grounding guides a frozen DiT, while generation feedback helps refine the VLM's temporal understanding and guidance decisions.

To Q2 / W2 (stability of mix reward and continued SFT):

We add uncertainty estimates across three independent training seeds to assess the incremental value of mix rewards. Continued SFT starts from the same SFT checkpoint and uses the RL task set and teacher supervision.

ComparisonΔ SD2 [95% CI]Δ H3 [95% CI]
Continued SFT → RL (same tasks and matched updates)+2.00 [1.27, 2.73]+1.20 [0.95, 1.45]
Prompt-only → mixed-reward RL (3 training seeds each)+0.58 [−0.10, 1.26]+2.50 [1.59, 3.41]

RL outperforms continued SFT on both editors with matched tasks and updates, supporting gains beyond additional data alone. Mixed rewards improve mean Overall by 0.58 / 2.50 on SD2 / H3.

To Q3 (simpler and stronger pipelines):

The fixed pipeline uses Exact-Match Retrieval → Fixed Clipping → LLM-Written Prompt, with code-controlled execution and matched references, video observations, and inference budget. The closed-source agents and MotionCoach share tools, a 65,536-token context limit, and a maximum of 10 inference rounds.

ConfigurationSD2H3
OverallMPPPVQOverallMPPPVQ
Fixed retrieval–clipping–captioning47.4531.0677.7854.5544.0031.4785.9436.90
Gemini 3.5 Flash agent (teacher)61.5355.8978.0359.9360.7536.9482.4885.93
GPT-5.5 agent58.9251.9173.4160.9358.8138.2276.2781.48
Claude Opus 4.7 agent64.3955.8979.3068.6364.2443.1582.1787.42
MotionCoach (8B)65.6153.0387.1072.2464.4633.192.298.2

MotionCoach gains 18.16 / 20.46 Overall points over the fixed pipeline and 4.08 / 3.71, 6.69 / 5.65, and 1.22 / 0.22 over Gemini, GPT, and Claude on SD2 / H3, respectively. PP and VQ are higher on both editors, while some closed-source baselines retain higher MP. The fixed-pipeline comparison supports gains from learned adaptive decisions beyond reference access and reliable tool execution.

We hope the clarification and additional experiments have sufficiently addressed your concerns. If you are satisfied, we kindly invite you to consider adjusting the score accordingly. We will also remain responsive to any further issues during the discussion phase.

Thank you for the substantial additional experiments. These additions strengthen the evidence for the system’s effectiveness and clarify the benefits of task-specific training. However, my principal concern about methodological novelty remains, and I maintain my rating at 2.

  1. Task-specific temporal requirements versus methodological novelty: The need to reason about motion phases and chronological order distinguishes this task from image generation and editing, but this task difference alone does not establish a methodological advance. The current approach still appears primarily to combine retrieval, temporal clipping, verification, prompt construction, and standard policy training. I would appreciate a more precise explanation of what the motion-specific grounding design contributes beyond this task such as targeted visualizations and controlled experiments.

  2. Comparison with the strongest autonomous agent. Although MotionCoach achieves slightly higher Overall scores than Claude Opus 4.7, its MP is lower on both editors. The Overall advantages are only 1.22 and 0.22 points, respectively, and arise from higher PP and VQ rather than higher MP. How should these results be reconciled with the claimed advantage in professional-motion guidance?

Reviewer_Ug97OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies professional motion editing, where a short instruction names an action whose defining mechanics may be unfamiliar to a video editor. MotionCoach trains a multimodal agent to search for demonstrations, inspect and trim a reference, verify the selected clip, and produce a detailed editing prompt. The prompt and clip condition a frozen video editor. The agent is trained with supervised trajectories followed by GRPO using prompt, video-outcome, and format rewards. The paper also introduces VPMEBench, a 314-task benchmark with motion references and annotations. Results show improvements on MiniMax-H3 and Seedance 2.0, supported by component ablations and an expert pairwise validation of the automatic evaluator.

Critical strengths
  1. The method addresses a useful problem and improves both frozen video editors. Seedance's motion professionalism rises from 29.6 to 53.0, supporting transfer beyond the editor used for RL.

  2. VPMEBench provides motion-specific annotations covering technique and temporal structure. Blinded evaluation by experienced annotators offers useful validation of the automatic scores.

Critical weaknesses
  1. Methodological novelty is limited: retrieval, VLM verification, prompt rewriting, and SFT/GRPO follow an established agent recipe. The main additions are task-specific criteria and data, with limited evidence of a distinct technical advance.

  2. The added value of video-outcome rewards is unclear: Overall improves from 64.89 to 65.61 over prompt-only reward, while MP falls from 54.46 to 53.03 (Table 7). Uncertainty and matched-budget comparisons are missing.

Questions for authors
  1. Evidence that could change my assessment: Beyond the task-specific data and rubric, which design is the main methodological contribution? Can its value be isolated against a generic reference-verification procedure with the same task supervision, tools, and comparable inference budget? This differs from removing verification altogether or comparing against an untrained policy, both of which the paper already does.

  2. Evidence that could change my assessment: Can you report uncertainty for the prompt-only versus mixed-reward comparison, ideally under matched training budgets? Please also explain the lower MP under the selected mixed reward and whether reward weights were selected on a separate validation set.

  3. Clarification of the data and generalization contribution: What corpus does search query, who or what produces its normalized motion/category metadata, and can retrieved tutorials overlap with the benchmark's evaluation references? How are aliases and repeated tutorials handled across splits? This affects whether the benchmark demonstrates acquisition of unfamiliar motion knowledge or selection within a curated, task-aligned reference library.

4 comments

For Reviewer Ug97 (R1)

To Q1 / W1 (methodological contribution and comparison):

Core contribution. Different from reference-verification procedure, MotionCoach learns a policy for temporally grounded motion guidance that unifies temporal evidence selection, understanding of temporal motion structure, and multimodal guidance construction. From long demonstration videos, it selects key intervals of the target motion and compiles them into a prompt and reference clip to provide reliable and sufficient guidance for a frozen video editor (Section 3.2; Appendix A.3.1). The policy is further optimized using feedback on the downstream editor’s outputs.

We test these choices by replacing video search with static-image search, video evidence with a single image, or adaptive clipping with uniform sampling.

ConfigurationSD2H3
MPPPVQOverallMPPPVQOverall
MotionCoach (Ours)53.0387.1072.2465.6133.192.298.264.46
w/ static-image search↓ 48.15↓ 40.15↑ 3.37↓ 31.10↓ 25.96↓ 44.58↓ 4.15↓ 23.15
w/o temporal video evidence (replace with a single image)↓ 11.24↓ 13.58↓ 8.07↓ 10.76↓ 8.19↓ 18.49↓ 17.93↓ 13.18
w/o temporal clipping (uniform sampling)↓ 14.18↓ 0.66↓ 3.22↓ 8.19↓ 11.27↓ 6.32↓ 11.69↓ 10.41

These results support temporal grounding through video-based motion dynamics and adaptive interval selection.

Comparison with generic reference-verification agents.

ConfigurationSD2H3
MPPPVQOverallMPPPVQOverall
Generic reference-verification agent (inference only)25.0056.6944.8037.2829.4658.2825.3233.98
Generic reference-verification agent (trained)24.8878.8466.1248.0526.2979.1251.3744.38
MotionCoach (Ours)53.0387.1072.2465.6133.192.298.264.46

Both generic agents follow a workflow of Search → VLM Verification → Prompt Rewriting, with the same tools and inference limits as MotionCoach (see the A–C in C1 of the Common Response). The inference-only variant has no task-specific training; the trained variant receives the same task supervision as MotionCoach. MotionCoach's Overall scores are 17.56 / 20.08 points higher than the trained generic agent's on SD2 / H3, with higher MP, PP, and VQ on both editors. These results support the benefits of temporal grounding.

To Q2 / W2 (reward uncertainty, matched training budget):

Budget and uncertainty. Both prompt-only and mixed-reward RL are trained for 1 epoch on the same task set, using 8 tasks per batch and 8 rollouts per task. Our uncertainty analysis uses an additional validation set and reports means across three independent training seeds with cross-seed 95% confidence intervals.

ConfigurationMP [95% CI]PP [95% CI]VQ [95% CI]Overall [95% CI]
RL: prompt reward only53.56 [51.53, 55.59]84.45 [79.95, 88.94]70.11 [68.78, 71.45]64.70 [64.03, 65.37]
RL: prompt + video rewards53.98 [51.61, 56.36]85.72 [82.16, 89.28]70.47 [66.24, 74.70]65.28 [64.48, 66.07]

Mixed-reward RL improves mean Overall by +0.58 and mean MP by +0.43 over prompt-only RL.

MP decrease. Table 6 evaluates Seedance 2.0, while video rewards use MiniMax-H3. Adding video rewards trades 1.4 MP points for +4.1 PP and +2.0 VQ. This is consistent with the intended role of the video judge: penalizing guidance the editor cannot plausibly realize, such as overly specified limb trajectories. Compared with prompt-only RL, adding the video reward increases MiniMax-H3’s MP by 5.3 points.

Validation set. The 32-sample validation set has no overlap with the 2,219 training samples in source-video–instruction pairs. Additional reward-weight selection on this validation set yields 0.7:0.3, consistent with the paper.

To Q3 (search corpus, metadata and split separation):

Search and metadata. Search queries Bilibili online (Ethics Statement, p. 11, L556–557). We use raw platform metadata; interpreting and organizing it to formulate and refine queries and select relevant demonstrations is part of the agent’s own decision-making capability.

Split separation. Searched tutorials are disjoint from the benchmark's expert-annotated reference tutorials. The SFT, RL, and benchmark partitions have disjoint target motions, source videos, and edit prompts (L195–196). Aliases and bilingual names are grouped by motion identity, and repeated tutorials and their clips by source tutorial, preventing cross-partition reuse.

We hope the clarification and additional experiments have sufficiently addressed your concerns. If you are satisfied, we kindly invite you to consider adjusting the score accordingly. We will also remain responsive to any further issues during the discussion phase.

Dear Reviewer,

As the discussion deadline approaches, we would be deeply grateful if you could let us know whether our updates have addressed your concerns, or if any questions remain unresolved. As the discussion period will close soon, we remain fully available and eager to provide any further clarifications.

Thank you once again for your valuable time and constructive feedback!

Best regards, The Authors

Reviewer_Ug97OpenReview ↗

Reply to rebuttal

I thank the authors for their detailed rebuttal and additional experiments. The response partially addresses my concerns and strengthens the empirical evidence for the method’s effectiveness. However, I remain unconvinced by the claimed methodological novelty.

In particular, the temporal design appears to be a task-specific adaptation involving interval selection, self-verification, and structured guidance construction. While these choices are useful, the response does not sufficiently clarify their technical advance over existing temporal grounding and agentic verification workflows. The additional experiments demonstrate their utility, but do not fully resolve this distinction.

Overall, I still consider the methodological contribution incremental and will therefore maintain my original rating.

Comments on the paper

2 comments

Reviewers: R1 = #Ug97; R2 = #FsQQ; R3 = #FLE1.

We sincerely thank all reviewers, Ug97 (R1), FsQQ (R2), and FLE1 (R3), for their insightful and constructive feedback. The reviewers acknowledge the well motivated problem and practical significance of our work, recognizing MotionCoach as a systematic agentic framework for professional motion editing (R1, R2, R3). They highlight the value of VPMEBench, featuring specialized motion annotations and evaluation validated by domain experts (R1, R2, R3). The reviewers also acknowledge the promising empirical results across different video editors achieved through our two stage training strategy of SFT followed by Agentic RL, supported by extensive experiments and meaningful ablation studies (R1, R2, R3). Furthermore, all reviewers commend the clarity of the presentation and writing (R1, R2, R3). We address all concerns and suggestions with detailed responses and additional evidence in the following sections. The shared concerns are (C1) the distinctive contribution, (C2) matched-budget baselines, (C3) uncertainty of the RL and video-reward gains, and (C4) online search and adaptive temporal localization.

C1. Core Contribution and Impact of Individual Components. Different from generic visual generation agents, MotionCoach learns a policy for temporally grounded motion guidance that unifies temporal evidence selection, understanding of temporal motion structure, and multimodal guidance construction. From long demonstration videos, it selects intervals capturing the target motion’s phases, order, and mechanics, verifies motion correctness and completeness, and organizes the evidence into a prompt and reference clip, providing reliable and sufficient guidance for a frozen video editor (Section 3.2; Appendix A.3.1). The policy is further optimized using feedback on the downstream editor’s outputs.

The table compares MotionCoach with generic reference-verification agents (A/B) and a fixed retrieval–clipping–captioning pipeline (C). D-1–D-5 test video search, temporal video evidence, adaptive clipping, motion-specific verification, and fine-grained temporal descriptions in the prompt. All A–C baselines use MotionCoach's inference budget. MotionCoach's scores are highlighted in bold.

VariantConfigurationSD2H3Addresses
OverallMPPPVQOverallMPPPVQ
0Base editor55.129.685.877.151.216.284.787.3—
AGeneric reference-verification agent (inference only)37.2825.0056.6944.8033.9829.4658.2825.32R1, R2
BGeneric reference-verification agent (trained)48.0524.8878.8466.1244.3826.2979.1251.37R1, R2
CFixed retrieval–clipping–captioning47.4531.0677.7854.5544.0031.4785.9436.90R2, R3
DMotionCoach (Ours)65.6153.0387.1072.2464.4633.192.298.2R1, R2, R3
D-1w/ static-image search34.514.8846.9575.6141.317.1447.6294.05R1, R2
D-2w/o temporal video evidence (replace with a single image)54.8541.7973.5264.1751.2824.9173.7180.27R1, R2
D-3w/o temporal clipping (uniform sampling)57.4238.8586.4469.0254.0521.8385.8886.51R1, R2
D-4w/o motion-specific verification58.0941.2585.1468.1154.6422.3387.4286.63R1, R2
D-5w/o temporal detailed prompt52.3423.2086.9377.8355.5514.0893.6999.24R1, R2

Compare with generic reference-verification agents and a fixed pipeline. For A/B, we draw on general visual-agent designs, including GenAgent and WeAgent-MMGenEdit mentioned by R2. These methods mainly target static images and do not match our video motion-editing setting, so we adapt their design logic to this task. A uses generic reference verification without task-specific training; B adds the same task supervision as MotionCoach. C uses Exact-Match Retrieval → Fixed Clipping → LLM-Written Prompt with the same candidate references, video observations, and inference budget.

MotionCoach exceeds A by 28.33 / 30.48 Overall points and B by 17.56 / 20.08 on SD2 / H3. We find that generic reference-verification agents (A/B) can fail to understand the meaning of specialized motion terminology and accept noisy or mismatched references that mislead the editor. MotionCoach also exceeds C by 18.16 / 20.46 Overall points on SD2 / H3.

Our key design and impact of individual components. D is full MotionCoach. D-1 replaces video search with static-image search; D-2 replaces the retrieved video with one image. D-3 uniformly samples a five-second segment instead of adaptively selecting an interval. D-4 removes motion-specific verification and recovery. D-5 removes fine-grained descriptions of spatiotemporal motion structure from the prompt

Across D-1–D-5, the reported Overall gaps range from 7.52 to 31.10 on SD2 and 8.91 to 23.15 on H3. Despite higher VQ, D-5 loses 29.83 / 19.02 MP and 13.27 / 8.91 Overall, underscoring the need for detailed motion guidance. Together, these results support the need to acquire temporal evidence, select and verify relevant intervals, and translate their motion structure into detailed guidance.

C2. Matched-budget alternatives. We further compare MotionCoach with teacher and flagship agents using the same tools, 65,536-token context budget, 10-round inference limit, and tool-failure retry mechanism. The no-retrieval baseline uses Qwen3-VL-8B to write an editing prompt from only the source video and instruction. Bold marks column maxima, including ties.

ConfigurationSD2H3Addresses
OverallMPPPVQOverallMPPPVQ
No-retrieval prompt (Qwen3-VL-8B)51.0022.6184.7175.8554.6713.8592.2097.66R3
Gemini 3.5 Flash agent (teacher)61.5355.8978.0359.9360.7536.9482.4885.93R2, R3
GPT-5.5 agent58.9251.9173.4160.9358.8138.2276.2781.48R2, R3
Claude Opus 4.7 agent64.3955.8979.3068.6364.2443.1582.1787.42R2, R3
MotionCoach (8B)65.6153.0387.1072.2464.4633.192.298.2R2, R3

MotionCoach (8B) improves Overall over the no-retrieval Qwen3-VL-8B baseline by 14.61 / 9.79 and MP by 30.42 / 19.25 on SD2 / H3. Its Overall gains over Gemini, GPT, and Claude are 4.08 / 3.71, 6.69 / 5.65, and 1.22 / 0.22, respectively. Some baselines retain higher MP or VQ; MotionCoach achieves the highest Overall through a stronger balance across objectives. Under identical tool protocols and fallback mechanisms, including matched retry limits, trained MotionCoach achieves a 99.7% trajectory success rate, compared with 89.49% for Gemini 3.5 Flash, 83.44% for GPT-5.5, and 89.81% for Claude Opus 4.7. This task-specific training helps explain why an 8B policy performs better under identical context, tools, round limits, and retry mechanisms. Even Gemini teacher trajectories require quality control: 13.23% (1,935/14,621) failed programmatic or VLM checks.

C3. Uncertainty, continued SFT, and the video reward. We compare prompt-only and mixed-reward RL across three independent training seeds, evaluating both on a new validation set. Both configurations use the same training tasks, one epoch, 8 tasks per batch, 8 rollouts per task, and identical GRPO update counts. The table reports means and 95% t-based confidence intervals (df = 2).

Reward configurationOverall [95% CI]MP [95% CI]PP [95% CI]VQ [95% CI]Addresses
RL: prompt reward only64.70 [64.03, 65.37]53.56 [51.53, 55.59]84.45 [79.95, 88.94]70.11 [68.78, 71.45]R1, R2, R3
RL: prompt + video rewards65.28 [64.48, 66.07]53.98 [51.61, 56.36]85.72 [82.16, 89.28]70.47 [66.24, 74.70]R1, R2, R3

We also compare RL with continued SFT to assess gains beyond additional teacher supervision. Both start from the same SFT checkpoint and use the same training tasks; continued SFT uses teacher supervision, and RL denotes the full mixed-reward policy. The table below reports Overall differences and 95% CIs for both comparisons.

ComparisonΔ SD2 Overall [95% CI]Δ H3 Overall [95% CI]Addresses
Continued SFT → RL (same tasks)+2.00 [1.27, 2.73]+1.20 [0.95, 1.45]R2
Prompt-only → mixed-reward RL (matched updates and rollouts per task)+0.58 [−0.10, 1.26]+2.50 [1.59, 3.41]R1, R2, R3

RL improves Overall over continued SFT on both editors, supporting gains beyond additional teacher supervision on the same tasks. Adding video feedback improves mean Overall over prompt-only rewards on both editors.

C4. Online search. MotionCoach searches Bilibili online rather than querying a pre-built index. It draws on broadly available videos, not a corpus curated for the evaluation set; the Ethics Statement identifies “instructional demonstrations from Bilibili” as its reference source (p. 11, L556–557). Search and temporal localization involve multi-turn autonomous decisions: the agent adapts queries, selects demonstrations, and refines temporal boundaries based on tool observations (Introduction, p. 2, L070–075).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.