- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces the Behavior Turing Test (BTT), which evaluates the human-likeness of humanoid behavior given an explicit task goal and scene description, rather than motion alone as in the previous work Motion Turing Test (MTT). The authors build a dataset of human-rated clips across real and digital humans and real and simulated humanoids, rendered in a unified SMPL-X representation, and use it to benchmark evaluators, proposing CoBE, a cross-attention evaluator that best matches human ratings.
Critical strengths
- Human-likeness evaluation is an important problem overall, much needed in the humanoid research community.
- The work is complete, from data to benchmark to evaluation models, with CoBE beating all baselines by a wide margin.
Critical weaknesses
- I am mostly concerned about the scalability of the pipeline. Context-SMPL construction relies on manual checks and judgment at several stages, and evaluators are trained and tested only on the filtered dataset, while the practical use case is scoring new, unfiltered data.
- The analysis does not yet clearly show that BTT captures more than MTT. Importantly, the pass threshold is the real-human mean, with the result that over a third of real-human clips (Fig. 4b) fail BTT themselves. More on this in the questions below.
Questions for authors
- Do the real-human clips that fail BTT also fail MTT? Please report pass/fail counts for both tests across all four domains, to clarify whether passing MTT but failing BTT is specific to humanoids.
- Could you show counterfactual results for smaller changes, where the score is supposed to stay more or less the same?
- GVHMR is a human pose estimation model and could introduce artifacts specific to robot embodiments. What is the reconstruction error for digital (or real) humans vs. simulated humanoids, where ground-truth motion is available?
- What fraction of candidate clips was rejected at each stage, per domain? What would you do when new data does not pass the current process?
- How do the evaluators perform on data processed without manual correction or rejection, with human ratings on a held-out set?
- CoBE remains strong even without its text branch. How does this indicate contextual awareness?
Thank you for the additional experiments. Two concerns remain:
Additional information beyond existing metrics. The original paper reports motion–BTT correlations of ρ = 0.831–0.957, whereas R1 reports ρ = 0.182 for the motion-only predictor. Are the motion measures and evaluation subsets comparable? Please explain this difference and report the reliability of the new component annotations. Low R² alone does not establish that BTT captures meaningful additional information.
Downstream value beyond task success. Scores increasing with task returns, or distinguishing successful executions from failures, could primarily reflect task performance. Can you quantitatively demonstrate that CoBE selects more human-preferred rollouts among equally successful executions of the same task, using independent human judgments and a simple baseline?