acceptodds
Withdrawn from ICLR 2027

MotionHarness: Agentic Spatiotemporal Coordination for Human Motion Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pretrained motion models provide expressive control over individual motions, yet this control alone does not ensure spatiotemporal coordination among multiple characters and objects. We propose MotionHarness, an agentic framework that jointly plans behavior, trajectories, and timing without task-specific retraining of the motion backbone. Event specifications make the required spatial and temporal relations explicit and guide verification of execution outcomes. History-guided search explores alternative plans and selectively recombines locally successful decisions from prior executions. To adapt transferred decisions to revised execution contexts, the framework invalidates affected evidence, rebinds transferred targets, and performs dependency-aware motion regeneration. This separates reuse of coordination decisions from reuse of their original trajectories and verification outcomes. We also introduce MotionCoordBench, a benchmark of 300 tasks spanning ten coordination families and three difficulty levels, with offline and online generation settings. Evaluation separately measures task satisfaction, motion validity, and computational cost. Evaluation on offline-generation tasks shows that MotionHarness improves both task success and validity-constrained task success over direct generation across three language-model backbones.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Human Motion Generation, Text-to-Motion Generation, Spatiotemporal Coordination, Agentic Planning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fcDZOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an agentic framework for motion generation involving interactions among multiple characters and objects. By incorporating event specifications, history-guided search, and conditional reuse, the framework improves task success rates. In addition, the authors introduce a benchmark consisting of 300 tasks.

Critical strengths

The paper introduces a comprehensive benchmark, MotionCoordBench, consisting of 300 tasks, including 261 offline-generation tasks and 39 online-generation tasks. This benchmark could serve as a valuable resource for evaluating future agentic frameworks for human motion generation and coordination.

Critical weaknesses
  1. Insufficient experiments. The main table only compares against direct generation, Codex and Pi. The ablation study shows that the proposed plan recombination brings only a marginal improvement. Most performance gains seem to come from the overall agentic framework, which is not entirely novel. Therefore, the core contribution is not sufficiently validated by the experiments.

  2. Unclear contributions. Although the paper discusses many related studies, it does not clearly distinguish its contributions from existing agentic motion generation frameworks. The introduction and experiments seem to position MotionHarness as a pioneering agentic framework, without sufficiently clarifying what is fundamentally new compared to prior work.

Questions for authors

Belows are my major comments:

  1. I think the authors should clarify the unique contributions of MotionHarness more carefully. For example, the current title does not clearly highlight what distinguishes MotionHarness from existing work, as agentic spatiotemporal coordination has already been explored. The title could better emphasize the specific motivation or technical contributions, such as conditional reuse. The same concern applies to the first contribution listed at the end of the Introduction, since the paper itself has already discussed several related works addressing similar problems. I suggest that the authors reconsider the title and the presentation of their contributions to better highlight the novelty of this work.
  2. To my understanding, conditional reuse and related mechanisms are the main technical innovations of this paper. However, Table 3 shows that removing plan recombination only reduces ValidTaskSR by 1.9%. In contrast, Table 2 shows a substantial improvement of MotionHarness over direct generation with GPT-6 Astra. This suggests that much of the improvement may come from the overall agentic framework rather than the novel mechanisms proposed in this paper. I think this discrepancy deserves further discussion in Section 5.3.
  3. The paper does not provide sufficient analysis of how Codex searches and refines its solutions. Since Codex may already perform similar forms of historical decision reuse, it is important to investigate whether the proposed mechanism is fundamentally different from what existing agents can achieve.
  4. Comparing Section 3.4 and Appendix B.4, it seems that MotionHarness and Codex/Pi do not have access to the same feedback information. So I have concerns about the fairness of the comparison. Could the authors clarify whether the baselines receive equivalent feedback and, if not, explain why?

Belows are my minor comments:

  1. Since all modules are training-free, I think the proposed framework may not necessarily improve the upper bound of performance, but could improve exploration efficiency. Comparing methods under a limited token budget is a good approach. I suggest that the authors further plot success rate as a function of token budget to better demonstrate the advantages of the proposed method.
Reviewer_x1oTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies LLM-guided motion generation for multi-person collaborative scenes. The proposed framework, named MotionHarness, seems to form a closed creation loop. It allows an LLM to plan high-level events and to delegate low-level execution to an existing motion generator. So as to produce multi-person trajectories. A rule-based validator checks event-level execution success (i.e., pass, fail, or unknown), and the LLM uses these results to revise the motion through reflection, conditional decision transfer, and regeneration.

Critical strengths
  • The high-level concept is interesting, workable, and in principle reliable. A creation loop of this kind can support agentic AI and humanoid robotics, and I believe this direction is promising.
  • The video demos are provided, which give a reviewer a direct view of the method's behavior. I see this as a positive aspect of the submission.
Critical weaknesses

Generally, the writing is high-level and often replaces detail with one-sentence hooks. Several mechanisms appear suddenly and are never connected across sections. For example, Scissor is never named in the main text and is introduced only through ("Scissor predicts a residual over a cubic Hermite bridge", sec. A.3), and keypoint selection is limited to ("Kimodo synthesizes motion conditioned on selected rollout keypoints", sec. 5.1). The overall impression is that the draft is template-generated rather than carefully written, which makes the reported data hard to trust.

My most concern is that, there is a mismatch between the method narrative and the evaluation. Sec. 3.3 presents Kimodo as the fixed motion generator, but sec. 5.1 states that all reported numbers come from a different system ("the reported metrics and costs concern executor-based plan evaluation", sec. 5.1), and the only Kimodo diagnostic produces a failure ("the resulting motion fails formal task evaluation", sec. A.3). The evaluated system is therefore not the system described in sec. 3.3, and I cannot tell from the tables how the neural generator performs.

My second concern is, the quantitative results are weak, especially on the interaction families. Task success is below 50%. Because object motion is only ("prescribed kinematically rather than obtained from physical simulation", sec. 3.1) and contact is only checked after the fact, the coordination claims reduce to geometric predicate satisfaction, and the demos mostly show the easier non-contact tasks. Meanwhile, the visual review is not a reliable evidence source. Scores come from an uncalibrated vision language protocol with no human reference, so the qualitative comparisons in fig. 3 are not convincing. I suggest involving a user study to evaluate the performance.

I'm very confused that, those core mechanisms are not reproducible as specified. The verifier is said to measure ("body occupancy, contact geometry, relative poses, and motion continuity", sec. 3.4) without any definition of how these quantities are computed, decision transfer requires "rechecking its applicability" (sec. 3.5) without defining the predicate, and the paper never explains how a source decision is matched to corresponding events in the parent plan (sec. C.4).

Questions for authors
  • Which generator, the 60 Hz executor or Kimodo, produced each reported number, and why is the neural backbone excluded from quantitative evaluation?
  • How are the control keypoints selected by the vision-language model, including the model, prompt, and selection criterion?
  • How are body occupancy and contact geometry actually computed for the validity checks?
  • What predicate decides that a transferred decision is applicable, and how are source components matched to parent events?
  • Why is Scissor never introduced in the main text, and what is its role in the pipeline?
Reviewer_BPDcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents MotionHarness, an LLM-based agentic framework for coordinated multi-character and human–object motion generation. Its core is a structured search over motion plans, using execution feedback to refine candidates and reuse locally successful decisions. The paper also introduces MotionCoordBench for evaluation. Experiments show improved task completion over direct LLM planning under an executor-based evaluation protocol.

Critical strengths

Overall, the paper is easy to follow. Separating coordination decisions, trajectories, and verification evidence is a useful formulation. The benchmark covers diverse coordination requirements, and the ablations support the value of joint planning and refinement.

Critical weaknesses
  1. The main results do not evaluate the final generated motion. Tables 2–3 evaluate trajectories produced by a procedural gait/IK executor. Kimodo synthesis follows plan refinement (Sec. 5.1; Supp. B.4). It remains unclear whether the final motions preserve the verified contacts, validity, and task constraints, how much Kimodo improves motion quality, and how synthesis failures are handled. Paired evaluation before and after Kimodo is needed to establish both constraint preservation and improvements in motion quality.

  2. The main comparison does not isolate the framework’s benefit from additional inference resources. Table 2 is uncapped. GPT-6 Astra, MotionHarness uses approximately 12× the tokens and 7× the runtime of Direct, which also lacks execution feedback. The budgeted ablations do not resolve this baseline comparison. I think execution feedback should be provided to these baselines.

  3. Although using medium reasoning effort is kind of ok, the results do not establish an advantage over baselines allocating comparable resources to higher reasoning effort or repeated attempts. I think xhigh or even max should be tried. Budget-matched comparisons with these configurations, alongside additional agent baselines such as Claude Code, would strengthen the evidence.

  4. Beyond the static examples in Fig. 3, matched baseline videos are also missing from the supplementary video, limiting assessment of the claimed qualitative advantages.

Questions for authors
  1. What fraction of executor-passing plans remain valid after Kimodo synthesis, and does the improvement persist against baselines matched for budget and feedback?

  2. Does MotionHarness retain its advantage over the feedback-driven single-lineage baseline in Table 3 at higher reasoning effort, when both methods use the same backbone, verifier feedback, and total computational budget?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.