acceptodds
Withdrawn from ICLR 2027

Actpin: Closed-Loop Semantic-Spatial Control of Generated Motion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Sparse spatial control of text-to-motion looks solved: drop an anchor, name the verb, and a guided or inpainted clip meets the point. We measure what that success is made of. When anchors are mined from clips that already perform the verb, a guided controller reads better than text alone (.62 to .92), since the mined anchor narrows generation toward its source performance; yet the pin is met by a snap, the responsible joint accelerating at 148x the clip median, invisible to retrieval. On freely authored anchors the verb collapses (.24-.51 across four families): every optimization these methods run serves space alone, so the point is bought by breaking the verb that no term defends. Actpin closes the loop: on a frozen motion-primitive stream whose motion seeds are the one differentiable input, it seeks per event the seed with legibility an objective measured on the rendered frames and the anchor and travel site both -constraints; across six coupling laws the strict read splits by whether the price remembers (we deploy the closed-form dynamic barrier). The deployed program holds every anchor at 0.7cm and arrival at 0.5cm, reads R@3 .90 / R@1 .71 averaged over three stream draws where the strongest baseline reads .51 / .28, and transfers to a held-out split.

Primary area:
generative models
Keywords:
text-to-motion generation, controllable motion synthesis, spatial and semantic control, closed-loop / test-time optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8RU7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies joint semantic and spatial control of generated human motion. Actpin optimizes latent noise through a frozen autoregressive motion generator, combining a semantic objective with joint-anchor and arrival-site constraints. It compares constraint-coupling strategies and evaluates on authored multi-action itineraries, including spatial accuracy, semantic retrieval, motion continuity, and robustness tests. The central contribution is a closed-loop control formulation together with an evaluation that exposes failures hidden by aggregate motion metrics.

Critical strengths
  1. A consequential control problem and useful diagnostics. Jointly satisfying a spatial target and an identifiable action is more demanding than optimizing either alone. The analysis of snapping, root motion, bone distortion, and transition behavior makes the benchmark useful for understanding practical failures in controllable motion generation.

  2. Substantial examination of the proposed controller. The coupling-law comparisons, redraws, re-authored test itineraries, and reported confidence intervals provide useful evidence beyond selected examples. The discussion of runtime and the dependence on TMR also makes important limitations visible.

Critical weaknesses
  1. The role of retiming in the reported gains needs clarification. Actpin allows temporal adaptation, whereas the fixed-window baselines follow the commanded timing. This is a reasonable design choice, but it makes the source of the performance gains harder to isolate. Please clarify whether precise timing is part of the task requirements and, if feasible, report a timing-controlled comparison to distinguish the benefit of temporal flexibility from that of the proposed controller.

  2. The evidence does not fully establish joint constraint satisfaction and semantic success. Table 14 labels the deployed anchor tolerance as 0.5 cm but reports mean anchor errors of 0.68–0.81 cm; if these use the same distance and evaluation frame as the constraint, the means alone cannot support universal feasibility. Per-event feasibility and failure distributions are needed. In addition, TMR supplies the optimization representation and the semantic evaluation; p.19 explicitly replaces the reach-up prototype with the caption embedding used by the evaluator. The distinction between motion and text branches does not provide an independent semantic check. Together, these issues weaken the claim that the method reliably satisfies both parts of the task, rather than improving the selected proxy.

Questions for authors
  1. Evaluation clarification / existing-output reanalysis: Under a common timing rule—either all methods evaluated at commanded frames or all given equivalent retiming freedom—how much of the reported advantage remains? Please also report realized duration and timing errors.
  2. Constraint clarification: Do the 0.5 cm constraint and the reported anchor metric measure the same quantity? What fraction of events satisfies the tolerance at the final evaluated frame, and what are the error tails and non-convergence rate?
  3. Assessment-changing validation: Is there an independent motion-semantic evaluator or blinded human assessment supporting the TMR gains, particularly for reach-up? If unavailable, the semantic claims should be narrowed accordingly.
Reviewer_pyPTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies text-to-motion generation with spatial constraints: a specified joint must reach a 3D anchor at a specified frame, while the motion span must be recognizable as the specified action. To address the problem that existing methods distort the body or lose the action semantics in order to reach the anchor, the authors propose Actpin, which optimizes the per-event initial noise of a frozen autoregressive motion generator, jointly solving with the anchor as a hard constraint and action recognizability as the optimization objective. On a self-built benchmark, the method maintains anchor accuracy while achieving clearly higher action recognition rates than existing methods.

Critical strengths
  1. The problem is interesting and practically relevant, and the diagnosis is valuable.** Requiring both spatial precision and action semantics at the same event is a real need in motion authoring.

  2. The experiments are careful and detailed.** Beyond anchor error and action recognition, the authors also report metrics such as joint acceleration to check whether the motion is natural, and provide results over multiple sampling runs.

Critical weaknesses
  1. The optimization objective and the evaluation metric share the same source.** The semantic objective in test-time optimization is TMR similarity, while the main metric is TMR retrieval. The algorithm may simply be hacking this metric; its real effectiveness needs further support such as human evaluation.

  2. Only closed-set classes are supported, and evaluation on public benchmarks is missing.** The main results are all on the authors' rule-generated APB; results on other standard benchmarks are needed. In addition, under the current design, the method seems unable to accept free-form text: conditioning comes from a fixed bank of 13 class embeddings. The authors claim that extension is only "a data question", but provide no evidence.

  3. Typo in Sec. 4.5: "Figures 7 and 7". Figure 3 has many unpolished details: formulas exceed their boxes, and the formula style is inconsistent with the main text. Heavy use of coined terminology makes the paper hard to read.

Questions for authors
  1. Since both the optimization objective and the main metric are based on TMR, can you provide a human evaluation or results from an independent evaluator not used during optimization?
  2. Can you report results on other standard public benchmarks beyond APB?
Reviewer_GHxCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses sparse spatial control in text-to-motion generation, where a user asks for a specific joint to hit a point in space at a given frame (e.g., the right foot at a mark for a kick) while the motion still has to look like the requested action. The authors start by arguing that the problem only looks solved. Under the usual evaluation, where target points are taken from real dataset clips, current methods reach the point by snapping the joint or sliding the whole body, and the standard metrics don’t notice. When the target points are placed freely instead, none of the existing controllers reliably does both: they often miss the point or lose the action.

Their method, “Actpin,” builds on a frozen streaming motion generator (DART). For each action, it optimizes the generator’s noise seeds so that TMR recognizes the motion as the requested action, while the joint and the body’s arrival point are kept within a small tolerance of their targets. The authors also release a benchmark of hand-specified multi-action scenarios at three difficulty levels, with a held-out split. They report sub-centimetre accuracy on both targets with no snapping and better action recognition than the baselines at every difficulty level.

Critical strengths
  • A well-supported diagnosis and a benchmark that follows from it. The paper shows that the apparent success of existing sparse spatial controllers under the standard mined-anchor evaluation often comes from breaking the body: the joint snaps onto the target with extreme acceleration, or the whole body slides into place. Retrieval metrics do not register either failure. The anchor-shift experiment is convincing: as the target moves away from its source clip, the breakage grows steadily while retrieval scores remain high (Fig. 2). The proposed metrics, such as joint acceleration at the core frame calibrated against real motion, are useful beyond this method.

    The Action Pinning Benchmark follows directly from this analysis. Its anchors are authored rather than mined, so they carry no information about the action, and its composition tiers make it possible to measure how methods degrade as more actions are combined.

  • Accurate spatial control without breaking the body. Optimizing the noise seeds through a frozen decoder lets Actpin hit both the anchor and the arrival point with subcentimeter accuracy, without the snapping and sliding seen in guidance-based methods. Because only the seeds change, every output is a valid sample from the generator, which keeps the motion physically plausible. The results hold across stream draws and on a re-authored held-out split, and the streaming setup extends to longer itineraries that fixed-window methods cannot attempt.

Critical weaknesses
  • The method is judged by the same model it optimizes:
    The central claim is that Actpin keeps the action recognizable while meeting the spatial target, but the quantitative evidence for this comes entirely from TMR, which is also the model the method optimizes against. The semantic objective is TMR similarity on the generated span, legibility is reported as TMR retrieval on the reach span, and the generator is conditioned in a TMR-aligned embedding space. Optimizing directly against a differentiable judge will raise that judge's score whether or not a person would see the action more clearly, so the results cannot distinguish a motion that genuinely looks more like a kick from one that has been pushed toward what TMR scores as a kick. The paper includes no human study and no action recognizer trained independently of TMR, and since this is its main claim, an independent evaluation is needed before the claim can be accepted.

  • The comparison with the baselines is confounded by timing and does not isolate what the semantic objective contributes:

    • Timing: The baselines must reach each anchor at the specified frame within a fixed time window. Actpin's stream instead lets each reach take as long as it naturally needs and is scored at its own core frame, which is usually later than the specified one. Its action sequences also last much longer than specified. The time allowed per action gets shorter with each tier, and the hardest tier requires a running pace by design, yet only the baselines have to fit their actions into this shrinking time limit. The claim that the baselines get worse as more actions are combined while Actpin's read stays high therefore mixes the effects of composition with the effects of timing.
    • Substrate and conditioning: The anchor-only variant does not optimize for the action at all. Even so, it hits the target at least as accurately as the strongest baseline and reads as the correct action more often than every baseline (Table 1). On R@3, most of Actpin's advantage is therefore already there before the semantic objective is added. This early advantage could come from three things: the streaming generator, the way the generator is told about each action, or the more relaxed timing described above. On the second point, Actpin gives a separate TMR-aligned embedding for each event, while most baselines get one caption for the whole sequence. The paper does not test these factors separately; in particular, it never compares the action embeddings with DART's original text input on the benchmark. Yet the Discussion credits the difference to "whether anything defends the verb while the point is met."
Questions for authors
  • Can you measure how recognizable the actions are using an evaluator independent of TMR, such as a human study or a separately trained action recognizer?
  • Can you report the per-tier results (Table 9) with matched timing, either by holding Actpin to the commanded frames or by giving the baselines Actpin’s durations where their window allows?
  • How do DART† and the anchor-only variant perform on the benchmark when using DART’s original text conditioning instead of the action embeddings?
  • Is the Transition Bond part of the final method, given the note under Table 16 that a transition head was removed?
  • Have you tried adding the TMR objective to a baseline’s own optimization?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.