acceptodds
Under review as a conference paper at ICLR 2027

Behavior Turing Test: Evaluating Human-Likeness of Humanoid Behaviors in Context

Abstract

Recent advances in humanoid robotics have enabled increasingly complex behaviors in real and simulated environments. However, existing evaluation remains largely motion-centric, focusing on task success, tracking accuracy, or motion-level human-likeness, while leaving a key question unexplored: would a human behave this way under the same task and scene? To address this, we introduce the Behavior Turing Test (BTT), which evaluates human-likeness at behavior level under explicit task and scene context. We construct HuCo, a dataset of 1,280 behavior clips spanning real humans, digital humans, real humanoids, and simulated humanoids, each paired with a Task Goal and Scene Description. We introduce Context-SMPL to suppress embodiment-specific appearance cues while preserving motion and scene context, and collect human judgments of contextual human-likeness. Our analysis shows that some humanoid behaviors can pass the Motion Turing Test yet fail BTT once task and scene context are considered, and that a clear human-humanoid gap persists under contextual evaluation. Building on HuCo, we establish HuCoBench for BTT and construct task and scene counterfactual subsets to evaluate context sensitivity. We further introduce CoBE, a Context-conditioned Behavior Evaluator that retrieves context-relevant behavioral evidence and achieves the strongest alignment with human judgments among evaluated methods. We will release dataset and code to support the community.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human Motion, Human Behavior, Humanoid Behavior, Behavior Turing Test, Human-likeness

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_L7myOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces the Behavior Turing Test (BTT), which evaluates the human-likeness of humanoid behavior given an explicit task goal and scene description, rather than motion alone as in the previous work Motion Turing Test (MTT). The authors build a dataset of human-rated clips across real and digital humans and real and simulated humanoids, rendered in a unified SMPL-X representation, and use it to benchmark evaluators, proposing CoBE, a cross-attention evaluator that best matches human ratings.

Critical strengths
  • Human-likeness evaluation is an important problem overall, much needed in the humanoid research community.
  • The work is complete, from data to benchmark to evaluation models, with CoBE beating all baselines by a wide margin.
Critical weaknesses
  • I am mostly concerned about the scalability of the pipeline. Context-SMPL construction relies on manual checks and judgment at several stages, and evaluators are trained and tested only on the filtered dataset, while the practical use case is scoring new, unfiltered data.
  • The analysis does not yet clearly show that BTT captures more than MTT. Importantly, the pass threshold is the real-human mean, with the result that over a third of real-human clips (Fig. 4b) fail BTT themselves. More on this in the questions below.
Questions for authors
  1. Do the real-human clips that fail BTT also fail MTT? Please report pass/fail counts for both tests across all four domains, to clarify whether passing MTT but failing BTT is specific to humanoids.
  2. Could you show counterfactual results for smaller changes, where the score is supposed to stay more or less the same?
  3. GVHMR is a human pose estimation model and could introduce artifacts specific to robot embodiments. What is the reconstruction error for digital (or real) humans vs. simulated humanoids, where ground-truth motion is available?
  4. What fraction of candidate clips was rejected at each stage, per domain? What would you do when new data does not pass the current process?
  5. How do the evaluators perform on data processed without manual correction or rejection, with human ratings on a held-out set?
  6. CoBE remains strong even without its text branch. How does this indicate contextual awareness?
Reviewer_95FLOpenReview ↗
Rating
3(+1)
Confidence
4(+2)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes the Behavior Turing Test (BTT), which rates whether observed behavior resembles how a human would act given an explicit task and scene. HuCo contains 1,280 clips balanced across real humans, digital humans, real humanoids, and simulated humanoids. A Context-SMPL pipeline removes the original actor and re-renders recovered body motion in the scene to reduce appearance cues. Human annotations provide contextual and motion-only ratings; task and scene counterfactuals probe context sensitivity. HuCoBench compares general-purpose VLMs, adapted VLMs, and dedicated scorers. CoBE uses text queries to retrieve relevant frozen video features and regress human scores. On the shared test split, CoBE obtains MAE of 0.459 and Spearman correlation of 0.703, compared with 0.512 correlation for QGVL. The paper also reports annotation reliability, background-only source recognition, and contextual ablations.

Critical strengths

I believe this paper has two core strengths and one experimental bonus point:

It raises a meaningful evaluation question: beyond motion naturalness, behavior should also be evaluated for whether it aligns with the task and the scene. For example, a reaching motion may be very smooth yet fail to actually press the elevator button; judging only by motion naturalness would easily miss such problems. Considering task goals, human–object relations, and the scene together has practical value for evaluating complete robot behavior.

It constructs a reusable human-annotation resource with substantial effort. HuCo contains 1,280 videos, covering real humans, digital humans, real robots, and simulated robots. The authors not only curate the materials but also unify the body representation, preserve the scenes, and collect task descriptions and human ratings. Context-SMPL attempts to reduce the influence of appearance on judgments, and the aggregated human ratings achieve an ICC of 0.887, indicating good label reliability. This is the most substantive contribution of the paper.

The experiments center on whether context is useful, and the automatic scoring results show clear improvement. The authors examine the role of context through experiments such as swapping tasks or scenes, removing text, and removing cross-attention. CoBE achieves a rank correlation of 0.703 with human ratings, higher than QGVL's 0.512, supporting the feasibility of training a dedicated evaluator that leverages context.

Critical weaknesses

What exactly is it measuring—"human-likeness," or "good task performance and contextually appropriate behavior"? A human might make the same mistake of failing to press a button; conversely, a robot whose motions don't look human-like may still complete the task very well. The authors need to demonstrate how much additional information this concept provides beyond "motion naturalness + task success + scene plausibility."

Another immediate question: is this evaluator's score actually useful for selecting or improving robot policies? The current experiments mainly show that it can fit human ratings on its own dataset; its downstream value in practice has not yet been validated. This also means that my earlier weak-accept judgment primarily valued its worth as an evaluation resource.

Questions for authors

Clarification Request: The authors identify a promising angle, but the motivation requires further explanation. Specifically, what does this "human-likeness" score provide beyond existing metrics such as task success rate and motion naturalness? Future Work/Impact: Furthermore, can this metric actually help select or train better robot policies?

Could clips cut from the same original long video appear in both the training and test sets? Could the authors provide results with grouping by original video, as well as results on unseen scenes or robot types?

Have the authors checked reconstruction accuracy for contact positions and relative body–object positions across data sources? In a small-scale study retaining the original video as reference, which low scores genuinely stem from the original behavior and which from processing errors?

1 comment

Thank you for the additional experiments. Two concerns remain:

  1. Additional information beyond existing metrics. The original paper reports motion–BTT correlations of ρ = 0.831–0.957, whereas R1 reports ρ = 0.182 for the motion-only predictor. Are the motion measures and evaluation subsets comparable? Please explain this difference and report the reliability of the new component annotations. Low R² alone does not establish that BTT captures meaningful additional information.

  2. Downstream value beyond task success. Scores increasing with task returns, or distinguishing successful executions from failures, could primarily reflect task performance. Can you quantitatively demonstrate that CoBE selects more human-preferred rollouts among equally successful executions of the same task, using independent human judgments and a simple baseline?

Reviewer_EghyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigated behavior turining test (BTT) to expand humanoid human-likeness assessment from motion realism to CONTEXT-AWARE perspective. The authors proposed HuCo benchmark which pairs with text descriptions of tasks and environments. Context-SMLP is introduced to exclude visual bia to some extent. The paper reveal that contextual behavior scores are lower and stricter than motion realism scores. Mainstream VLMs show weak correlation with human perceptual ratings. The authors further proposed context-conditioned behavior evaluator (CoBE) and demosntrated the effectiveness.

Critical strengths
  1. The motivation is clear and the studied topic is timely. The distinction between "moving like a human" and "behaving like a human under a particular condition" is currently underrepresented in humanoid evaluation.
  2. The proposed dataset, HuCo, is relatively complete. it spans diverse domains, i.e., real/digital humans and real/simulated humanoids, with comprehensive evaluation metrics. The authors done a lot of efforts (e.g., Context-SMPL and quality checks) for improving the dataset, which is valuable.
  3. The benchmarking is also comprehensive. The experiments include diverse models, and also the proposed context-conditioned behavior evaluator (CoBE). The overall results are reasonable.
Critical weaknesses
  1. The validity of contextual human-likeness. This metric currently combines different aspects (motion naturalness, action selection, etc.) into one score. So a low BTT score can arise from behavior that is physically non-human-like / simply unsuccessful / inappropriate for specific tasks. These failure modes have different implications.
  2. Potential shortcut learning from domain gaps. As seen, human contextual ratings differ substantially across the four domains. models might achieve high correlation simply by recognizing domain differences instead of performing contextual reasoning. Contextual and motion-level scores are already highly correlated, and CoBE's video-only ablation ("w/o Text Branch") has already shown good performance. a text-only baseline ("w/o Video Branch") should be added to figure out the effect of text-based shortcuts. Therefore, further investigation on domain correlations are needed to isolate true contextual reasoning.
  3. Human performance ceiling. It is difficult to assess whether CoBE's performance is already approaching the upper bound (or if room for improvement remains), as the metrics and statistic are not directly comparable.
Questions for authors

The term "Behavior Turing Test" may overstate the presented paper. For example, HuCo is a scalar human-rating prediction problem, which differs from the classical notion of a Turing test. I suggest the authors tone down several magic terms in the manuscript.

Comments on the paper

1 comment
Area_Chair_SHQROpenReview ↗

Rebuttal Discussion

Dear Reviewers,

The authors have now submitted their rebuttal and responded to the concerns raised in the reviews. We would greatly appreciate it if you could take some time to review their responses and participate in the discussion, particularly regarding whether your concerns have been adequately addressed.

Thank you very much for your time and valuable contributions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.