acceptodds
Withdrawn from ICLR 2027

RobotSTAR: Next-Scale Autoregressive Sign Language Translation for Humanoid Robots

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Sign-language interpretation in public communication relies on qualified professional interpreters and can be difficult to scale, motivating robotic signing as a complementary accessibility interface. We present RoBoSTAR, a text-conditioned sign language production (SLP) framework for generating human-centric sign motion that can be retargeted for robotic execution, with speech supported optionally through an external ASR front end. Conventional autoregressive approaches flatten motion into a single full-resolution token sequence, forcing long-range and local dependencies to be modeled at a uniform temporal granularity. RoBoSTAR instead combines part-wise Finite Scalar Quantization with next-scale autoregression, generating motion over progressively finer temporal resolutions while predicting synchronized body and hand tokens in parallel within each step. This coarse-to-fine formulation provides compact long-range context before progressively refining motion details, while self-conditioning and context corruption improve robustness to cross-scale prediction errors. The generated motion is subsequently retargeted for physical humanoid execution. Extensive qualitative and quantitative evaluations are conducted to demonstrate the effectiveness of RoBoSTAR.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotics, Motion generation, SLP

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_baD4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a text generation framework based on the Conditional Design Language for generating human-centered gesture actions, which can be re-directed for robot execution, and optionally implemented with external speech recognition front-end for speech support. The paper has considerable practical value, but there are still some issues that need to be improved.

Critical strengths

The paper focuses on sign-language interpretation on robotic signing. Conventional autoregressive approaches flatten motion into a single full-resolution token sequence, forcing long-range and local dependencies to be modele data uniform temporal granularity.

Critical weaknesses

(1) The main comparison method in this paper is NAR. Could the author add some methods from recent years for comparison as well, such as DARSLP? (2) The author mentioned in the article that the VQ codebooks learned by the existing sign language generation methods may exhibit issues related to imbalance. How does the model proposed by the author solve this problem? (3) The paper proposes an autoregressive model. Could you please explain how this model addresses the problem of error accumulation inherent in autoregressive models? (4) The results presented by SC only and the final model are very similar, especially for the hands and the body. Has the effect of GCC been demonstrated? (5) Suggestions for additional ablation experiments on Dynamic Degree, Magnitude Ratio, and Freeze-tail.

Questions for authors

Please see the weakness section.

Reviewer_LYmfOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RobotSTAR, a text-conditioned sign language production framework whose generated human motion is retargeted to a humanoid robot. The method combines part-wise finite scalar quantization (FSQ) with coarse-to-fine autoregressive generation, predicting synchronized body and bilateral-hand tokens at progressively finer temporal resolutions. Self-conditioning and context corruption address cross-scale exposure bias. Experiments on How2Sign show improved geometric accuracy over the evaluated baselines, supported by ablations, a small user study, and physical robot demonstrations.

Critical strengths
  1. Applying sign language generation to humanoid robots is a practically valuable research direction with potential benefits for accessible communication.
  2. Quantitative comparisons, component ablations, a blinded user study, and physical robot demonstrations collectively support the method’s effectiveness and the feasibility of robotic execution.
  3. The coarse-to-fine temporal hierarchy provides a well-motivated approach to modeling both global motion structure and fine-grained hand articulation.
Critical weaknesses
  1. The paper uses sign language production (SLP), sign language translation (SLT), and sign language generation (SLG) in similar contexts. The title describes robotic SLT, whereas the core task is text-to-sign motion production. These terms should be clearly defined and used consistently.
  2. FSQ and part-wise motion quantization have been explored. The robotic retargeting pipeline also largely follows prior work. Although robotic signing is a valuable application, the paper should more clearly establish the novelty of its coarse-to-fine formulation beyond combining existing components.
  3. Line 224 introduces a length predictor, while Equation (6) includes its loss in the joint training objective. Joint optimization is a reasonable design, but the paper should clarify whether this predictor is a separate module or a head attached to the language model, and how its gradients affect the backbone.
  4. The appendix states that akasheroor/American-Sign-Language-Dataset supplies the lexical retrieval dictionary. Extra datasets are used for training and testing?
  5. Evaluation is restricted to How2Sign. Experiments on other widely used benchmarks, such as CSL-Daily and PHOENIX-2014T, would strengthen the evidence for generalization across sign languages and domains.
  6. Appendix A.7 identifies the evaluator as a Hybrid FSQ-T5 model built on Flan-T5-base. Why do the authors use a custom evaluator rather than an established sign language translation (SLT) model used in prior work? This choice requires justification, particularly regarding evaluator reliability and comparability with previous results.
Questions for authors

See Weaknesses

Reviewer_kXHyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RoboSTAR, a framework for generating sign language motions and transferring them to humanoid robots. It combines part-wise FSQ with next-scale autoregressive generation to produce body and hand motions from coarse to fine temporal resolutions. Self-conditioning and context corruption are used to improve generation robustness. Experiments on How2Sign show improved performance over existing methods.

Critical strengths
  1. This is a meaningful work that explores sign language generation for humanoid robots.
  2. The paper is well-written and easy to follow.
  3. The experimental results are promising, with clear improvements over existing methods.
Critical weaknesses
  1. The main idea of next-scale autoregressive generation is very similar to MoScale for human motion generation. Most other components are also taken from existing methods. I find the overall methodological novelty rather limited.
  2. Even the ground-truth motions achieve only 1.96 BLEU-4 in back-translation. With such a low score, it is difficult to assess the semantic accuracy of generated signs using this metric.
Questions for authors

Please see the weaknesses above.

Reviewer_sxXDOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ROBOSTAR, a text-conditioned sign-motion generator followed by humanoid retargeting. It combines part-wise finite scalar quantization, autoregression across temporal scales, and self-conditioning with context corruption. On How2Sign, the paper reports improved geometric accuracy and competitive back-translation scores, supplemented by motion-dynamics metrics and a blinded study with four fluent signers. Generated motions are demonstrated on a humanoid robot, with additional retargeting and simulated tracking measurements.

Critical strengths
  1. The coarse-to-fine formulation provides a concrete way to model motion at different temporal resolutions while maintaining synchronized body and hand streams. The training strategies explicitly address errors propagated from preceding scales.

  2. The evaluation combines geometric, back-translation, and motion-dynamics measures with blinded signer ratings. Highlighting that favorable geometric errors can coexist with nearly static motion is useful for interpreting sign-generation results.

  3. The hierarchy ablation shares the backbone and tokenizer, while the quantizer comparison matches the part-wise architecture and compression ratio. The paper also clearly distinguishes retargeting error from simulated tracking error and acknowledges the robot's missing non-manual channels.

Critical weaknesses
  1. The key next-scale and FSQ ablations in Tables 2 and 3 report only geometric generation errors, although Table 1 shows that these can favor nearly static predictions. Please include back-translation and motion-dynamics results for these controls to establish that the main design choices improve meaningful signing rather than only aligned pose accuracy.

  2. The geometric scores sum DTW costs over native-length sequences and therefore depend on alignment-path length. Meanwhile, back-translation resamples predictions to the reference duration. Please report generated-length statistics, geometric errors normalized by alignment-path length, and semantic scores without reference-length resampling to distinguish motion-quality improvements from duration effects.

  3. Several baselines exhibit extreme motion collapse: NAR and T2S-GPT have zero Dynamic Degree and Freeze-tail of 1.0. A common joint topology alone does not establish faithful baseline inference. Please document the checkpoints, adaptation, inverse normalization, and duration handling, identify the common test subset, and verify that the collapse persists under the intended inference settings.

  4. Robot-level intelligibility remains insufficiently evaluated. Table 5 measures retargeting and simulated tracking, with a mean wrist-orientation transfer error of 31.3 degrees, while the user study does not specify whether participants viewed physical robot outputs. Please clarify the presentation and assess robot-output comprehension separately, or limit the embodied claims to demonstrated execution feasibility.

Questions for authors
  1. Do the next-scale and FSQ variants retain their advantages in semantic fidelity and motion dynamics when evaluated at their predicted lengths?

  2. Were the user-study stimuli human-motion renderings or physical robot executions, and how much intelligibility is retained after retargeting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.