acceptodds
Under review as a conference paper at ICLR 2027

Codebook-Geometry Correction: Replacing Test-Time Refinement with a Few Analytic Steps in Trajectory-Controlled Text-to-Motion Generation

Abstract

Text-to-motion generators produce fluent, text-aligned motion, but steering a joint along a prescribed path usually costs either a duplicated control backbone or thousands of per-sample optimization steps at inference. We show that a handful of analytic steps can do the same job, provided the correction is measured in the right geometry. Our central contribution is Codebook-Geometry Correction (CodeGeo): the generator’s part codebooks record which changes its frozen decoder was trained to absorb, and so define a metric in which a correction can be measured; CodeGeo moves a generated clip onto the commanded path by the smallest change that metric allows, in at most eight damped steps in place of 1,600 refinement iterations. CodeGeo needs a start that is already close to the targets, which Key/Value Control (KV-Control) supplies: a thin interface (10.5 M trainable parameters) that feeds the trajectory into a frozen masked transformer as extra key/value memory at every attention layer, over Anatomy-Aware Part Tokenization (PartVQ), one codebook per body part unpacked so that every part-frame token is an addressable attention site. CodeGeo has one hyperparameter, the anisotropy of its step metric, and its setting is not free: it follows how far the commanded anchors spread across body parts, with single-part and body-spanning anchor sets preferring opposite settings in every configuration we tested. On pelvis control the method tracks to 0.24 cm at FID 0.067 in 1/21 the per-sample time of the iterative schedule it replaces; on multi-joint control it reaches 0.25 cm at FID 0.072, improving on MaskControl’s published 0.72 cm and 0.083 on both axes.

Primary area:
generative models
Keywords:
motion generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LvYrOpenReview ↗
Rating
1(−1)
Confidence
4(+1)
Informativeness
1
Presentation
1
Soundness
1(−1)
Read review
Paper summary

Text-to-motion generation with trajectory control typically requires additional control networks or costly iterative refinement. This paper proposes Codebook-Geometry Correction (CodeGeo), which leverages part codebooks to define a geometry for minimally correcting generated motions toward prescribed trajectories in at most eight steps, replacing 1,600 refinement iterations. To provide a suitable initialization, Key/Value Control (KV-Control) introduces trajectory-conditioned key/value memory into a frozen masked transformer, using only 10.5M trainable parameters and Anatomy-Aware Part Tokenization (PartVQ). CodeGeo’s metric anisotropy is adapted to the distribution of controlled body parts. Our method achieves 0.24 cm error and 0.067 FID for pelvis control, and 0.25 cm error and 0.072 FID for multi-joint control, outperforming MaskControl’s 0.72 cm error and 0.083 FID.

Critical strengths
  1. Presents solid and credible engineering results, demonstrating effective trajectory control with high accuracy and improved inference efficiency.
  2. Provides a practical solution for efficient motion control while preserving motion quality.
Critical weaknesses
  1. The paper is difficult to follow due to heavy use of coined terminology, crowded figures, and several formatting issues that make verification harder.

  2. The contribution seems primarily an application of standard engineering optimization to motion control, with limited advances in representation learning or generative modeling, so the fit for ICLR is unclear.

  3. The paper does not discuss closely related work on damped least-squares inverse kinematics, latent-space editing (e.g., DragGAN and DragDiffusion), or constrained generation (e.g., DPS and RePaint). T-Concat and KV-Control also resemble existing multi-codebook sequence modeling and attention-based conditioning techniques, making the novelty of these components unclear.

  4. Table 1 bolds the proposed method rather than the best value in each column. In the pelvis block, it underperforms MaskControl in Top-3 (0.802 vs. 0.809) and FID (0.067 vs. 0.061), and its feed-forward variant performs better in skating (0.0425 vs. 0.0477) and latency (0.17 vs. 0.21 s). This presentation may overstate the method’s overall advantages.

Questions for authors

see weaknesses. If my concerns are adequately addressed, I would consider increasing my score; otherwise, I may consider lowering it.


Final justification

Fundamentally, this work incorporates physical constraints into motion representations through an optimization procedure. However, its main contribution appears to be the application of Gauss–Newton iterations to test-time correction, rather than a novel generative modeling methodology. Gauss–Newton itself is a well-established optimization technique, not introduced by this work. I therefore remain concerned that the contribution is not well aligned with ICLR’s generative modeling community and offers limited methodological insights to the broader motion generation and AIGC communities.

The writing remains very poor. In my assessment, its formulaic phrasing, repetition and weak reasoning give it a pronounced AI-like style. It reads as mechanically assembled technical text rather than a clear account for human readers. The rewrite has not resolved these problems or my concerns about the contribution. Moreover, the current version differs so substantially from the original submission that it is almost a complete rewrite. Given the extent of these changes, I would suggest that the authors consider withdrawing and resubmitting the paper.

9 comments
Reviewer_LvYrOpenReview ↗

Clarification Requests to Authors

Thank you for the detailed response and the revised manuscript. However, I still have several concerns.

First, the title is overly long and seems to frame the work within a rather narrow scope. I encourage the authors to carefully reconsider whether the title and the Introduction accurately reflect the actual scope and contributions of the work.

Second, please ensure that all modifications are consistently highlighted in blue. For example, the revised caption of Figure 1 is not highlighted. I cannot verify whether other modifications are also unmarked, which could potentially mislead reviewers and the AC regarding the extent of the revisions.

Third, I would like to point out that the paper, with the Introduction being one example (not just intro.), differs substantially from the version I originally reviewed. However, only a very small portion of the modifications appears to be highlighted in blue. I have retained a local copy of the original manuscript and would appreciate the authors being transparent about the changes between the two versions.

For example, in the initial version, the first paragraph is:

3D human motion generation drives animation, games, and embodied-AI pipelines, where a text prompt is the fastest way to author a moving character. Recent text-conditioned generators Guo et al. (2024); Pinyoanuntapong et al. (2024b); Zhang et al. (2024a); Tevet et al. (2023); Chen et al. (2023) already turn a prompt into diverse, semantically faithful motion. That motion, however, gives the user no handle on placement: even a high-quality clip cannot be pinned to a specified root path, a foot contact on a prop, or a planner’s trajectory—geometric constraints no prompt can express. The problem we address is how to add this precise, extrinsic geometric control on top of such a generator while preserving its generation quality. We study joint trajectory control: constraining user-specified joints—a single end-effector keyframe, a root path, or a multi-joint trajectory—to hit given target positions while the prompt continues to dictate gait, style, and intent.

However, the current version is (without any blue highlights):

Generating 3D human motion from text is useful for animation, games and embodied agents, because a text prompt is a fast way to describe what a character should do. Recent text-to-motion models (Guo et al., 2024; Pinyoanuntapong et al., 2024b; Zhang et al., 2024a; Tevet et al., 2023; Chen et al., 2023) produce diverse motion that matches the prompt. A prompt, however, cannot say where the body should be. It cannot place the root on a given path or put a hand on an object at a given frame. Trajectory control adds this ability. The user specifies target positions for chosen joints at chosen frames, for example a path for the pelvis or keyframes for the wrists, and the generated motion must pass through these positions while it still follows the prompt. We call each target position of one joint at one frame an anchor.

Forth, ... .

I appreciate the authors’ efforts, but I believe these issues warrant further attention.

BTW, I trust that, as stated in the manuscript, the authors have used AI tools only within the declared scope. I hope the authors will carefully adhere to the stated boundaries when using such tools.

Thank you for your prompt response and engagement with our paper.

I am the supervisor and corresponding author of this paper (the previous responses were posted by the first author following discussions among all co-authors). We took your feedback very seriously and carried out a comprehensive revision over the weekend to improve the readability of the paper. I would like to address the additional concerns you raised today:

  1. Because the main text was densely packed with information, we initially chose a long and descriptive title to help understanding. Now the manuscript has been rewritten for better readability, so we agreed to make a more concise and intuitive title. Our proposed new title is: "Codebook-Geometry Correction for Trajectory-Controlled Text-to-Motion Generation". We will check the OpenReview guidelines regarding title modifications and make this change in the camera-ready version.

  2. The blue-highlighted text in the newly uploaded PDF does NOT show general modifications BUT represents newly added parts in response to the reviewers' comments. This is explained in the separate new thread titled "To all reviewers and the AC: revised PDF uploaded, summary of changes." because it should be noticed to all reviewers and chairs. The modifications and polishing are addressed in detail below.

  3. Improving readability could not be achieved by simply editing some paragraphs or sentences. As noted in the first author's previous response ("Over the weekend, all co-authors carefully re-read the entire paper and rewrote it thoroughly to improve clarity and flow"), we polished the entire manuscript. From Friday to Sunday, we restructured paragraphs, expanded overly compressed phrasing into clearer expressions, and utilised AI assistance to polish the text. Because nearly every paragraph has major/minor refinements, highlighting every textual difference would have required highlighting almost the entire paper. We had actually discussed the potential risk of misunderstanding if these edits were not highlighted as you pointed out. However, we concluded that there is no point of highlighting the entire paper, and instead chose to state explicitly in our response that the whole manuscript was rewritten. Additionally, please note that the current uploaded version cannot not be the final paper. Expanding the explanations for clarity inevitably increased the length of the paper. Although we moved some parts including the original Figure 3 to the Appendix, the current draft slightly exceeds the ICLR page limit. We will need to shorten the text again for the final version. Due to the limited time remaining in the rebuttal period, we uploaded this version first so that you could evaluate the updated writing style. As you have the original submission on hand, if you compare the original and updated versions side-by-side, you will see how the narrative flow and readability have been improved. To be completely transparent, aside from the parts highlighted in blue, the revised paper has only reorganised and rewritten content from the original submission without any unmentioned technical additions.

All co-authors agree that your feedback has helped make the paper significantly clearer, and we sincerely appreciate your time and guidance throughout this review process. Although little time remains in the discussion period, we are happy to be fully committed to incorporating any further suggestions into the camera-ready version.

Thank you for the detailed clarification.

However, I would like to reiterate that, according to the ICLR review policy, “a pdfdiff will be applied to compare new changes to the paper against the original submission” and “Area chairs and reviewers reserve the right to ignore changes that are significantly different from the original paper.” The policy also explicitly states that “reviewers and area chairs are not required to look at every revision.”

These guidelines make it clear that reviewers and ACs are entitled to understand the differences between the original submission and the revised manuscript, and are not expected to independently compare two PDF versions to identify and assess the changes themselves.

I therefore strongly encourage the authors to explicitly highlight and explain the modifications made to the manuscript, rather than relying on reviewers to conduct a side-by-side comparison. I understand that improving readability may involve extensive rewriting; however, this does not negate the importance of making the revisions transparent and readily identifiable to the reviewers and AC.

Reviewer_DgfyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a trajectory-controlled text-to-human motion generation method CodeGeo, which replaces a time-cost test-time optimization with at most eight correction steps. A lightweight KV-Control model generates the motion first guided by the input trajectory, and then the motion latent representations are adjusted by the proposed CodeGeo with motion codebooks. Experiments on HumanML3D show the faster inference and more accurate trajectory following comparing the baselines.

Critical strengths
  1. The motivation is clear and reasonable. Previous trajectory controlled motion generation methods use the post optimization to improve the trajectory-controling performance and the post optimization run many iterations and it is time-consuming. This paper directly address the high cost of existing methods
  2. The proposed methods achieve impressive performance in the speed and trajectory following.
  3. The control adaptation module is lightweight with approximately 10.5M parameters, avoiding a full duplicate control network.
Critical weaknesses
  1. The perceptual study is missing in this work. HumanML3D has been extensively benchmarked, and the number with the standard metrics may not reliably reflect the perceived motion quality. A user study would therefore provide valuable complementary evidence.
Questions for authors
  1. I am curious when the conflicting motion trajectories are given, how will the model perform. The correction accepts steps based on reduced anchor error, which does not guarantee natural poses or physically plausible motion.
  2. How runtime and memory scale with the actual number of anchors?
Reviewer_fRk3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses trajectory-controlled text-to-motion generation. PartVQ exposes part-level tokens, KV-Control injects trajectory-conditioned memory into a frozen generator, and CodeGeo corrects continuous codebook embeddings using a small number of Jacobian-based steps. Experiments evaluate tracking, generation quality, runtime, and the choice of codebook metric.

Critical strengths
  1. Substantial same-generator speed and tracking improvements, with useful early-stopping and batched-throughput comparisons.
  2. Controlled metric sweeps and validation results provide practical insight into the choice of correction geometry.
Critical weaknesses
  1. The claim of preserving motion quality needs stronger support. In Table 1, relative to the same-generator iterative refinement, multi-joint CodeGeo worsens FID from 0.053 to 0.072 and foot-skating from 0.0486 to 0.0545; pelvis foot-skating also increases from 0.0444 to 0.0477. These results indicate a trade-off between tracking accuracy, speed, and motion quality that should be discussed explicitly. The visual evidence also has limits: the walks-squats-resumes video shows pronounced torso bending around 3.2–3.75 seconds.
  2. The writing would benefit from careful revision. Sec. 3.4 (p. 5) calls ee "decoded embeddings", leaving its position relative to the decoder unclear, whereas the appendix describes a soft codebook mixture fed into the motion decoder (Algorithm 2, p. 14; p. 16).
Questions for authors
  1. Quality trade-off: How was the reported operating point chosen when weighing the FID and foot-skating regressions against the tracking and latency gains?
  2. Evaluation clarification: Do the all-joint results average separate single-joint evaluations, or pool motions generated under joint-combination control, as Appendix B appears to describe? Please explain the correspondence to MaskControl's quoted Average row, since its Cross setting is reported separately ([Appendix A.5, Table 7]).
  3. Mathematical clarification: Please clarify or revise the single-anchor explanation in Sec. 4.4 and Appendix A.1, distinguishing the observed FID similarity from any claimed metric invariance.
  4. Writing clarification: Please replace "decoded embeddings" in Sec. 3.4 with an explicit definition of ee relative to the motion decoder. State its initialization and the residual map r(e)r(e) in the main text, consistently with the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.