acceptodds
Withdrawn from ICLR 2027

GenTrack: Online Co-Training of Text-to-Motion Generation and Zero-Shot Humanoid Tracking

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Humanoid motion trackers need diverse reference motions for training, but collecting motion data and adapting it to a robot is costly. Pretrained text-to-motion generators can expand this training set by producing new robot motion references. However, motions learned from human data can remain difficult for robots to follow accurately, even after retargeting. These references can train the tracker, whose execution feedback can in turn improve the generator. We introduce GenTrack, a unified online post-training framework that co-adapts robot-native text-to-motion generation and humanoid tracking. We first train a robot-native text-to-motion generator to initialize the motion source. then alternates two updates: generated motions augment public references for tracker training, and tracker rollouts provide execution rewards for refining the generator. A lagged tracker stays fixed within each generator phase and is refreshed between phases. Controlled comparisons with one-way updates and offline replay examine whether generation and tracking benefit from learning together. The results show that online co-training can improve tracking of unseen motions while making generated motions easier to execute. Replay comparisons further suggest that updating the motion source during training offers benefits beyond reusing a fixed generated dataset. These findings support online co-training as a way to expand motion supervision without collecting additional motion data while narrowing the gap between generated references and robot execution.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Embodied AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MXDSOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents GenTrack, an online post-training framework that jointly adapts a text-conditioned, robot-native motion generator and a humanoid motion tracker. Generated references augment public motion data for tracker training, while execution feedback from a lagged tracker supplies rewards that update the generator through FlowGRPO. KL regularization, supervised rehearsal, and distillation limit how far the generator’s motion distribution can shift.

The framework is evaluated in simulated Unitree G1 environments using ProtoMotions and SONIC backbones. Experiments measure tracking on three fixed held-out motion splits, and generation quality using a frozen SONIC executor and a G1-specific text-motion evaluator. Comparisons include reference-only continuation, frozen-generator replay, final-generator replay, generator-only updates, and alternative objectives. Results are most favorable for the SONIC backbone, whereas the ProtoMotions backbone shows trade-offs between generator quality and tracker coverage.

Critical strengths

The evaluation separates generator improvements from tracker improvements and includes meaningful controls. Tracker evaluation uses fixed held-out references, so apparent tracking gains cannot arise solely from the generator producing easier test motions. Generator variants are evaluated with the same frozen executor, prompts, noise, and motion lengths. Within each backbone, tracker comparisons are matched on initialization and environment-transition budgets. Results are reported over five independent online training runs with confidence intervals, and trajectory-error metrics include failures. These choices make the comparisons substantially more informative than evaluating only the final generator-tracker pair.

The SONIC results and retention ablations provide concrete evidence of useful improvements. Under frozen-SONIC evaluation, the SONIC-based GenTrack generator reduces key-body error from 0.410 m to 0.325 m while maintaining retrieval performance and diversity. The tracker also lowers position errors relative to frozen-G0G_0 replay. Table 7 further shows that removing the retention mechanisms can improve execution metrics while degrading retrieval, FID, and diversity. This illustrates the trade-off between executability and content preservation and supports including distribution-preserving objectives.

Critical weaknesses

The benefit of mutual adaptation is inconsistent. In Table 6, generator-only training outperforms the full ProtoMotions loop on several generation metrics. For SONIC, Wild-G1 tracking success is 48.0%, barely above the 47.8% obtained with final-generator replay. A matched sequential baseline (generator refinement followed by tracker training) would better establish the value of online coupling.

Reliance on private data limits reproducibility. Wild-G1-clean supplies approximately 85% of the motions underlying the aggregate tracking errors, but its source and evaluation trajectories are unavailable. Although the evaluator code and weights are provided, the missing evaluation inputs and limited documentation of data curation and deduplication prevent independent verification of the central results.

Questions for authors

Mutual adaptation: Could you compare GenTrack with a sequential control that first refines the generator against a frozen tracker and then post-trains the tracker on the resulting references, with matched generation, scoring, and tracker-training budgets? Please also report confidence intervals for the differences between the two configurations.

Data and reproducibility: Could you provide anonymized provenance, action and duration statistics, curation thresholds, and the duplicate and near-duplicate detection procedure for the internal corpus and Wild-G1-clean? Is there an access mechanism or a set of evaluation inputs that could be released so the private-set results can be reproduced?

Reviewer_eCk4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper jointly finetune a text-to-motion generator and a humanoid motion-tracking policy through an alternating training framework. The generator is finetuned using execution feedback from the tracker to produce more physically executable motions, while the tracker is finetuned on an evolving dataset of generated motions and existing reference motions to improve its generalization to unseen motions. Experiments on the Unitree G1 humanoid using two pretrained tracking policies, ProtoMotions and SONIC, demonstrate improvements in motion executability and zero-shot tracking performance, while the gains vary across the two backbones.

Critical strengths
  • Meaningful and well-motivated problem: The paper addresses the limited availability of diverse motion references and the gap between generated reference motions and their physical execution. The proposed co-finetuning framework offers a reasonable approach to improving both motion generation and humanoid tracking through mutual feedback.

  • Comprehensive experimental evaluation with meaningful baselines and ablations: The paper evaluates both motion generation and tracking using two pretrained tracking backbones, ProtoMotions and SONIC. Importantly, the evaluation separates the two components: Table 1 evaluates tracker generalization on fixed held-out references, while Table 2 evaluates generated-motion executability using a frozen SONIC tracker alongside semantic and distributional quality metrics. The experiments also include meaningful comparisons with reference-only training, frozen-generator replay, final-generator replay, and one-way optimization under matched training budgets. Additional ablation studies in Tables 6 and 7 examine the effects of execution feedback, optimization objectives, and generator regularization, providing useful insights into the trade-offs between motion executability, tracking generalization, and semantic preservation.

Critical weaknesses
  • Limited evidence for the benefits of mutual co-adaptation: The experiments do not consistently demonstrate that jointly updating the generator and tracker outperforms one-way optimization or offline replay. In Table 6(a), generator-only training achieves better generator metrics than full GenTrack with ProtoMotions, while Table 1 shows mixed improvements over replay baselines. Although SONIC results are more favorable, the experiments do not clearly distinguish the benefits of mutual adaptation from those of an evolving training curriculum.

  • Limited generalization of generated-motion executability: The generator's executability is evaluated exclusively using a frozen SONIC tracker. Since the SONIC branch also uses lagged SONIC policies to provide execution rewards, the improvements may reflect compatibility with a specific tracker family rather than general physical executability. Although the ProtoMotions branch provides some cross-tracker evidence, evaluation with additional independent trackers would strengthen the generality of the conclusions.

  • Lack of convergence and stability analysis for alternating optimization: The proposed framework alternates between optimizing two components with interdependent objectives, without providing theoretical convergence guarantees or sufficient empirical evidence of training stability. Although lagged trackers and regularization are introduced, the paper does not demonstrate whether the alternating updates converge or oscillate. This could potentially explain why full co-training underperforms one-way optimization in some experiments. Round-by-round performance curves and sensitivity analyses of the alternating schedule would help clarify this concern.

Questions for authors
  1. Clarification on the benefits of mutual co-adaptation: Table 6(a) shows that generator-only training outperforms full GenTrack on several generator metrics with ProtoMotions, while the improvements in tracker performance are relatively modest. Could the authors clarify what evidence supports the necessity of bidirectional co-adaptation rather than one-way optimization or an evolving training curriculum? Would a matched update-direction ablation for SONIC, or a control that updates both components while keeping the scoring tracker frozen, lead to similar conclusions?

  2. Clarification on the generalization of generated-motion executability: Since the generator's execution quality is evaluated exclusively using a frozen SONIC tracker, to what extent do the reported improvements generalize to other tracking policies? In particular, would evaluating the SONIC-trained generator using an independently trained tracker, such as ProtoMotions, demonstrate similar improvements? This would help distinguish general improvements in motion executability from compatibility with a particular tracker family.

  3. Clarification on the convergence and stability of alternating training: The proposed framework alternates between optimizing two components with interdependent objectives. Could the authors clarify whether the training procedure exhibits empirical convergence or stability across the 12 rounds? In particular, can the authors provide round-by-round generator and tracker performance curves and explain the choice of the alternating schedule? This would help assess whether unstable or conflicting updates contribute to the weaker performance of full co-training compared with one-way optimization in some experiments.

Reviewer_wLk1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an online co-training framework for robot-native text-to-motion generation and zero-shot humanoid motion tracking. Starting from a pretrained generator and tracker, it alternates between augmenting tracker training with generated references and refining the generator through execution-based rewards using FlowGRPO. A lagged tracker provides feedback, while regularization and supervised rehearsal help preserve motion diversity and text alignment. The framework is evaluated in simulation on Unitree G1 with ProtoMotions and SONIC, using three held-out tracking splits and a fixed SONIC executor for generator evaluation. Comparisons with reference-only training, offline replay, and single-component updates show improvements in generated-motion executability and selected tracking metrics, with stronger results on SONIC and trade-offs on ProtoMotions.

Critical strengths
  1. This paper is well written. The motivation is clearly articulated, and the proposed framework and experimental evaluation are presented in a logical and easy-to-follow manner.

  2. The paper evaluates generated-motion executability with a frozen executor and tracker generalization on fixed held-out references. This separation makes the reported component-level improvements easier to interpret and reduces ambiguity arising from evaluating two simultaneously changing models.

Critical weaknesses
  1. While the SONIC results provide positive evidence, generator-only training outperforms the full loop in ProtoMotions on key-body error (0.381 versus 0.399 m), and tracker gains over replay are mixed. The paper acknowledges these trade-offs but provides limited analysis of their cause. Furthermore, final-generator replay does not separate mutual adaptation from curriculum effects. These limitations reduce the generalizable insight into when the added online coupling is beneficial.

  2. Prior work such as PARC already establishes iterative generator–tracker co-training, while RobotMDM studies execution-guided generator refinement and tracker training on generated motions. GenTrack differs in its online execution-reward optimization of robot-native, text-conditioned humanoid motions and its separate evaluation of held-out tracking performance. Nevertheless, its algorithmic contribution appears primarily to be an integration of existing co-training and generator-alignment ideas using FlowGRPO. Given this incremental methodological novelty, the mixed empirical benefits leave the reviewer unconvinced that the paper establishes a sufficiently substantial advance over existing generation–control frameworks.

Questions for authors
  1. What explains the different outcomes across the two backbones, particularly why generator-only training achieves better key-body accuracy than the full loop in ProtoMotions, whereas co-training improves the SONIC results? Do the existing training diagnostics or ablations identify conditions under which updating both components is beneficial? This would help assess the scope and significance of the proposed coupling.

  2. Relative to PARC and RobotMDM, which specific design choice constitutes the main methodological advance, and which existing comparison most directly supports its value? In particular, what evidence distinguishes the benefit of ongoing generator–tracker feedback from that of an evolving reference curriculum, given the limitations of the final-generator replay control? Clarifying this distinction would materially affect my assessment of the paper’s originality and contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.