acceptodds
Withdrawn from ICLR 2027

Humanoid Motion Foundation Model for Multimodal Streaming Closed-loop Interaction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent advances in vision-language-action (VLA) systems have enabled increasingly general-purpose embodied behaviors. However, existing approaches still formulate humanoid interaction as a static conditional generation problem, limiting their ability to support temporally evolving multimodal interaction and execution-aware behavioral adaptation in open-world environments. In this paper, we propose MSC, a multimodal streaming behavioral framework for closed-loop humanoid interaction. Instead of conditioning humanoid behaviors on fixed multimodal conditions, MSC reformulates humanoid control as a streaming conditional flow process under continuously evolving multimodal and execution-aware states. Built upon a structured conditioning representation and a streaming flow matching mechanism, MSC enables continuous interaction and closed-loop behavioral refinement during execution. Experiments show that MSC consistently improves motion quality, multimodal alignment, and robustness across text-, vision-, and audio-conditioned humanoid control tasks. We further demonstrate real-world deployment on Unitree G1 humanoids under interleaved multimodal interactions, validating the practicality of streaming closed-loop humanoid interaction in real-world environments.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Multimodal Reasoning, Vision-Language Model, Vision-Language-Action Model

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_fXTwOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes MSC for humanoid motion generation under temporally evolving text, audio, and visual inputs. It combines structured conditioning on multimodal intent, motion history, and execution feedback with streaming flow matching. The authors evaluate motion quality, streaming interaction, and robustness to execution perturbations, and report deployment on a Unitree G1 humanoid.

Critical strengths
  1. The separation of multimodal intent, motion history, and execution feedback is a sensible design for continuous humanoid interaction, allowing subsequent motion generation to account for both changing instructions and deviations during execution.

  2. The paper integrates text-, audio-, and video-conditioned generation within a shared humanoid motion framework. Its runtime breakdown and distinction between policy updates and low-level tracking provide useful insight into the computational constraints of deployment.

Critical weaknesses
  1. The claimed real-world deployment lacks verifiable evidence. The abstract, introduction, and experiments repeatedly emphasize deployment on a physical humanoid, yet no supplementary videos are available, and the paper contains no anonymous link to video demos. Static figures are insufficient to assess motion quality. This absence of evidence raises serious concerns about the credibility of a central experimental claim and leads me to recommend a clear rejection.
Questions for authors

The authors should try to provide anonymized video demos during rebuttal to support a proper review of their paper.

Reviewer_QD1HOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets the interesting problem of multimodal streaming closed-loop interaction which is of great importance to the community. Rather than conditioning humanoid behaviors on fixed multimodal conditions, the proposed MSC reformulates humanoid control as a streaming conditional flow process under continuously evolving multimodal and execution-aware states. Experimental results show that the proposed algorithm obtains reasonable results based on motion quality, multimodal alignment, and robustness across multi-modal conditions.

Critical strengths
  • The task of multimodal streaming closed-loop interaction is of great importance to the community.

  • The paper presents an quantitative evaluations of the proposed algorithm. Also, the proposed algorithm is validated on Unitree G1with the SONIC whole-body controller.

  • The paper is well presented and the proposed idea is easy to follow.

Critical weaknesses
  • The details of the evaluation dataset should be presented. Is the setup of the evaluation fair to compare with the other baselines?

  • For the comparison baselines, it should include the policy baselines such as π0, GR00T or wholebodyvla.

  • Table 1 provides the comparison on different modalities. How about the results with the composition of different modalities.

  • There are a set of hyper-parameters in the paper, for example, in Fig. 2. How about the setting of these parameters and the influence on the final results?

Questions for authors

Please mainly address the issues raised in the weakness section.

Reviewer_UTi7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MSC, a framework for streaming multimodal humanoid motion generation with closed-loop feedback. It introduces structured conditioning that separates multimodal intent, motion history, and execution feedback, and uses flow matching for continuous motion generation. Experiments evaluate motion quality, streaming interaction, robustness, and real-world deployment on Unitree G1.

Critical strengths
  1. Practical system integration. The paper integrates multimodal motion generation with real-time humanoid execution and demonstrates deployment on a physical G1 robot, showing the feasibility of the proposed framework.
Critical weaknesses
  1. Insufficiently established novelty and problem gap. Existing methods such as TextOp and UniAct already support streaming motion generation, motion-history conditioning, and online instruction updates. The paper does not clearly explain what fundamental limitations remain or how its proposed streaming flow matching differs from standard conditional flow matching with receding-horizon replanning.

  2. Insufficient experimental evidence for the core contributions. The experiments do not adequately isolate the benefits of structured conditioning and high-level closed-loop feedback. In particular, the streaming baselines are evaluated by independently generating and concatenating motion segments, potentially disadvantaging methods that already support continuous generation. Furthermore, the robustness experiments do not distinguish improvements from high-level feedback from those provided by the underlying SONIC tracking controller.

Questions for authors
  1. What is the fundamental methodological difference between MSC and existing streaming methods such as TextOp and UniAct? Specifically, what capabilities cannot be achieved through conventional conditional flow matching with motion-history conditioning and receding-horizon replanning?

  2. Can the authors compare structured conditioning against directly concatenating the same conditioning information under identical architectures and training settings? This would clarify whether the proposed structure provides benefits beyond simply incorporating additional information.

  3. Can the authors isolate the contribution of high-level execution feedback by comparing MSC with and without correction conditioning while keeping the same SONIC controller under identical perturbations? Additionally, why are MSC's MPJPE and S-MPJPE identical in Tables 2 and 3, given that S-MPJPE is defined using single-shot conditioning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.