acceptodds
Under review as a conference paper at ICLR 2027

Traceable Human-to-Humanoid Sign Language Benchmarking

Abstract

Sign data collection is costly, and teleoperation scales poorly, motivating reuse of large video corpora. Humanoid signing requires converting video-derived human motion into robot trajectories while preserving linguistic motion cues. Errors from fitting, human-motion repair, retargeting, robot geometry repair, and control are hard to separate from the final trajectory alone. We introduce HumanoidCSL-20K, a dataset and benchmark of 20,648 sentence-level Chinese Sign Language sequences, each with four aligned versions: the source, the repaired human motion, the direct robot reference, and the geometry-repaired robot reference. Observation-supported local human-motion repair, full-robot geometry repair, and cross-representation provenance make each transformation traceable. Paired evaluations measure human-motion continuity and content preservation, robot-reference feasibility, and physical execution. A sign-specific kinematic-reference protocol scores handshape, location, palm orientation, and inter-hand relation over the full planned motion. Full-corpus results show fewer abnormal arm / hand steps and less inter-hand and hand-body penetration after repair. Control experiments separate reference learnability from curriculum effects, while component scores expose remaining execution errors.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Sign Language Robot, Humanoid Retargeting

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_55oWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Sign data collection is costly and teleoperation scales poorly, making the reuse of large video corpora an attractive alternative. However, enabling humanoid signing requires converting video-derived human motion into robot trajectories while preserving linguistic motion cues, and errors introduced by fitting, human-motion repair, retargeting, robot geometry repair, and control are hard to distinguish from the final trajectory alone. To address this, the authors introduce HumanoidCSL 20K, a dataset and benchmark of 20,648 sentence-level Chinese Sign Language sequences, each paired with four aligned versions: the source, the repaired human motion, the direct robot reference, and the geometry-repaired robot reference. Observation-supported local human-motion repair, full-robot geometry repair, and cross-representation provenance make each transformation traceable, while paired evaluations separately assess human-motion continuity and content preservation, robot reference feasibility, and physical execution. In addition, a sign-specific kinematic-reference protocol scores hand shape, location, palm orientation, and inter-hand relation over the full planned motion.

Critical strengths
  1. I think the application perspective of this paper is relatively novel, as sign language is indeed a domain rich in finger and wrist information.
  2. The benchmark effectively demonstrates that the retargeting method proposed in the paper achieves relatively small errors.
  3. The paper is clearly written and easy to read.
Critical weaknesses
  1. As a benchmark, it does not compare the performance of other retargeting methods.
  2. The positioning of this benchmark is rather ambiguous; it is unclear whether it is intended to evaluate imitation policies or to assess the quality of retargeting methods.
  3. Sign language is related to facial expressions, yet this issue has not been taken into consideration.
Questions for authors
  1. Can the benchmark evaluate the performance of other retargeting methods?
  2. How much better is the performance of a policy trained on data processed by the proposed method compared with one trained on data not processed by the proposed method?
  3. How is the issue that sign language also involves facial expressions taken into account?
  4. Can it be adapted to dexterous hands of different sizes?
Reviewer_g5LAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HumanoidCSL-20K, a dataset and benchmark for transferring human sign-language motions to humanoid robots. The dataset contains 20,648 sentence-level Chinese Sign Language sequences with four aligned representations: original human motion, repaired human motion, directly retargeted robot motion, and geometry-repaired robot motion.

The proposed pipeline combines observation-guided local motion repair, collision-aware robot retargeting, and a two-stage reinforcement learning curriculum for whole-body and finger-motion tracking. Experiments demonstrate substantial reductions in abnormal joint movements and self-collisions, along with improved simulated execution performance. The paper further provides stage-wise evaluation protocols to analyze errors introduced during different stages of human-to-humanoid motion transfer.

Critical strengths
  1. A useful and traceable benchmark. The dataset provides over 20K aligned sign-language sequences with intermediate human and robot motion representations. Unlike pipelines that only release final robot trajectories, the proposed benchmark allows researchers to examine errors at individual processing stages. This is a meaningful contribution to reproducibility and systematic evaluation.

  2. Substantial improvements in geometric feasibility. The experimental results demonstrate clear improvements in motion continuity and collision avoidance. In particular, Table 2 shows that penetration-free robot references increase from 1.21% to 88.39% after geometry repair. The controlled comparisons between reference processing and controller training also strengthen the experimental analysis.

Critical weaknesses
  1. The preservation of sign-language semantics is not sufficiently validated. This is my primary concern. The paper emphasizes that handshape, hand location, palm orientation, and inter-hand relationships are essential to linguistic meaning. However, the evaluation mainly measures geometric similarity and trajectory tracking. In Table 1, semantic preservation is evaluated using a frozen sign-language recognizer on only 75 sequences, while the dataset contains over 20K sequences. More importantly, the recognition evaluation is performed on human-motion representations rather than the final robot-executed signing motions. Therefore, the reported results do not establish whether the complete human-to-robot pipeline preserves understandable sign-language content. This is a substantial limitation given the central motivation of the work.

  2. The methodological contribution is relatively incremental. The overall framework integrates existing techniques, including local motion interpolation, constrained geometry optimization, conventional human-to-robot retargeting, and curriculum-based reinforcement learning. Although the integration is carefully engineered, the paper does not sufficiently establish a new algorithmic principle beyond adapting and combining these components. In particular, the two-stage controller largely follows existing reference-tracking methods, while the motion repair procedure relies on predefined thresholds and interpolation. The contribution appears stronger as a dataset and benchmarking system than as a methodological advance for ICLR.

  3. The robot execution results remain limited despite high completion rates. Table 3 reports 100% completion for the proposed controller using repaired references. However, the final ActionScore is only 54.10, and 13.03% of rollout frames still exhibit penetration. These results suggest that completing a trajectory does not necessarily imply accurate or physically consistent sign-language execution. Moreover, quantitative execution experiments cover only 20 motion sequences with three random seeds, while the physical robot demonstrations are qualitative. Consequently, the evidence for robust deployment across the full benchmark remains limited.

Questions for authors
  1. Semantic preservation: Could the authors clarify why recognition BLEU is evaluated on only 75 human-motion sequences? Is there any existing quantitative or qualitative evidence that the final robot references and executed trajectories preserve the original sign-language meaning?

  2. Methodological novelty: What are the key algorithmic contributions beyond integrating existing interpolation, retargeting, constrained optimization, and reinforcement learning methods? Could the authors explain which components are specifically novel and necessary for the proposed benchmark?

  3. Execution quality: Although the repaired references achieve 100% completion, the ActionScore remains 54.10 and the rollout penetration rate is 13.03%. What are the dominant sources of the remaining errors, and how do these errors affect the fidelity of the generated signs?

  4. Evaluation coverage: Could the authors clarify how representative the 20 selected motions are of the full dataset, particularly regarding handshape complexity, bimanual interactions, and difficult signing configurations?

Reviewer_oqVYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces HumanoidCSL-20K, a dataset of 20,648 sentence level Chinese Sign Language sequences built from CSL-Daily videos through SOKE SMPL-X fits. Its organising idea is that every sequence is kept in four time aligned versions, namely the source fit, a locally repaired human motion, a direct robot reference for a Unitree G1 with Inspire hands, and a geometry repaired robot reference, and that each step records what it changed so a downstream failure can be traced to the stage that caused it. Three components do the work. Local repair flags unreliable frames using an image cue and a motion cue, then rebuilds them by interpolating from neighbouring anchors. Geometry repair runs a two stage quadratic program that removes inter hand, hand body and intra hand penetration under joint, speed and finger coupling limits. A per reference tracking controller based on BeyondMimic learns body and wrist motion first and ramps in finger targets afterwards. Evaluation gives one task to each transformation, scoring human repair by abnormal step rate and recognizer BLEU, robot references by penetration rate and depth over 1,519 audited collision pairs, and execution by a sign specific protocol covering handshape, location, palm orientation and inter hand relation. The reported outcomes are that repair cuts abnormal arm steps from 19.34 percent to 0.62 percent and hand steps from 10.65 percent to 0.10 percent without a detectable BLEU change, that geometry repair cuts penetrating frames from 50.18 percent to 0.29 percent and certifies 88.21 percent of sequences, and that the full curriculum reaches ActionScore 54.10 while completing every replay.

Critical strengths
  1. The dataset is real and the scale advance is genuine: I checked this against the nearest published work. SignBot builds a retargeting and interaction framework but releases no aligned corpus of robot signing motion. Khan et al. address the same collision problem on robot sign references, but on nine sequences, with no physical platform and no release. RoBoSTAR retargets 30 sequences to a G1 while focusing on generation quality. Against that, 20,648 sequences in four aligned versions with per stage records is three orders of magnitude larger, and the first release of its kind. The claim holds.

  2. Traceability is delivered, not just promised: Plenty of papers promise modular evaluation and then evaluate end to end anyway. This one does not. Each stage has a defined input, output and evaluation set, and baselines are released at every stage. The authors go further and show that weighting supervision by record score lowers high support finger RMS from 0.066 rad to 0.059 rad, so the records can guide training and not only diagnosis.

Critical weaknesses
  1. The control evaluation covers 20 sequences out of 20,648: That is 0.097 percent of the corpus, and it is my main concern. The paper is framed around a 20K scale resource, but every execution number in Table 3, Table 24 and Figure 10 comes from a fixed sample of 20 motions across three seeds. The sample is handled correctly in the narrow sense. It is fixed in advance, held constant across conditions, and bootstrapped over motions rather than trials. The resulting intervals still describe variation across those 20 motions and nothing more, which the paper itself concedes when it states that the motion is the unit of inference. Small effect sizes make this worse. The curriculum gain is ActionScore 52.58 to 54.10 and JointPass 16.32 to 17.98. Table 25 shows seed 777 reaching 53.51 under single stage training against 54.51 under the full curriculum. A one point gap on 20 motions could plausibly reverse under a different draw. I would want either an evaluation on a few hundred stratified sequences, or control claims limited explicitly to the sampled motions with the corpus level framing taken out of the abstract.

  2. Nothing shows that the robot references keep the meaning of the signs: The content argument runs entirely through a frozen Uni-Sign recognizer on 75 sequences, and that recognizer only ever sees human motion. It never sees a robot reference. So the paper establishes that human repair does not change recognized content and establishes nothing about the geometry repaired robot references. This matters because the repair makes changes that are large next to the paper's own linguistic tolerances. Inter wrist distance is widened by 7.96 cm RMS and the handshape budget permits 20 degrees of change on active finger joints. Appendix D.2 then reports that eye to nose spacing is 5.8 cm, nose to mouth 4.4 cm, and mouth to chin 4.9 cm, and concedes that the 10 cm location tolerance exceeds the spacing between some neighbouring sites. The authors call it a declared operating point rather than a bound derived from anatomy, which is honest, but it means the repair can move a hand across a site boundary while every metric reports a pass. The limitations section notes that comprehension is untested. I think the gap is structural rather than a limitation. A benchmark built for communication currently cannot detect the failure that would most directly defeat it. A small study with fluent signers on the 20 control motions would change my assessment, and so would running the recognizer on rendered robot motion.

  3. The single external baseline is not competitive: AMP reaches ActionScore 2.73 and finishes zero percent of replays, stopping after 10 to 12 frames out of references running up to 481. The authors say openly that a general purpose motion prior is not built for dexterous signing, which is precisely why the claim in Table 3 that the proposed controllers far exceed AMP tells the reader nothing. The comparison worth having is against a tracker of the same class. BeyondMimic without the proposed curriculum is the obvious candidate, the authors already build on it, and running it should be cheap. Without that, the controller contribution rests only on the small internal ablations above.

Questions for authors
  1. Can you show how the 20 control motions compare to the full corpus on sequence length, signing-space extent, and pre-repair penetration rate? If the sample is representative on these dimensions, the execution results carry more weight and partially address Weakness 1.

  2. Does the frozen Uni-Sign recognizer produce usable scores on rendered robot motion? If yes, what is the BLEU gap between repaired human motion and the geometry-repaired robot reference? This is the most direct available test of whether the robot references preserve sign meaning, and a positive result would substantially address Weakness 2.

  3. What does BeyondMimic without the proposed curriculum reach on the same 20 motions and seeds? Without this, the controller contribution cannot be separated from the base tracker, which affects whether Section 5 supports its claims.

1 comment

Thank you for your constructive review. The authors have posted their rebuttal. Could you please review their response to see if you have any remaining questions?

Reviewer_hcMDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces HumanoidCSL-20K, a benchmark containing 20,648 Chinese Sign Language sequences for a humanoid with dexterous hands. It retains four aligned representations: source human fits, repaired human motions, direct robot references, and geometry-repaired robot references. The pipeline combines local motion repair, robot geometry repair, and a tracking curriculum that introduces finger control after body and wrist tracking. Validation includes corpus-wide motion and collision analysis, a frozen-recognizer evaluation of human-motion repair, controlled tracking experiments, provenance-based diagnostics, and qualitative hardware replays.

Critical strengths

The aligned data and repair records provide a useful basis for studying failures across the pipeline. Researchers can inspect how a reference was produced, locate changes to particular body parts and time intervals, and compare individual stages while keeping the others fixed. Appendix D.3 demonstrates a concrete use of these records in interpreting tracking errors. I see this as a worthwhile resource contribution that can support work beyond the authors’ particular repair and control methods.

The experiments explain why motion quality needs to be checked at several stages. Local repair and global smoothing differ in how well they retain recognizable content, even though both improve continuity. The collision analysis also identifies failures missed on the source sampling grid but exposed on the control grid (Section 5.2 and Appendix C.2). These findings offer practical guidance for constructing signing references and give the benchmark value beyond reporting final execution scores.

Critical weaknesses

The content-preservation evidence is limited to human-motion repair. Appendix C.1 compares Raw, Global, and Fused human motions, while the subsequent robot geometry repair also changes relative wrist placement and finger configurations. Appendix C.2 documents the magnitude of these changes, but does not directly assess their effect on signing content. Similarly, self-target replay measures tracking of the modified reference, so improved execution alone cannot establish that the source content is preserved. This limits how the final robot references can be interpreted as signing targets. The paper acknowledges the broader comprehension limitation in Section 6, and the evidence for its core contributions in diagnosis, feasibility assessment, and tracking evaluation remains convincing.

Questions for authors

Content validation: Is there any assessment of signing content before and after robot geometry repair, beyond the deformation statistics and collision examples in Appendices C.2 and E.1? Evidence concerning sequences with larger wrist or finger changes would help establish the scope of content preservation in the final references.

Provenance experiment: In Appendix D.3, is an unweighted result available using the same training motions and evaluation subset as the score-weighted policy? The comparison with discarding low-support motions changes the training data, so a matched comparison would clarify the added value of weighting.

2 comments

Thank you for your constructive review. The authors have posted their rebuttal. Could you please review their response to see if you have any remaining questions?

Thank you for the clarification of my concerns. I have no more questions and remain with my assessment of the rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.