acceptodds
Under review as a conference paper at ICLR 2027

Generative Action Synthesis: Inferring Robot Actions from Kinematic Demonstrations

Abstract

For dexterous hands, kinematic demonstrations from motion-capture gloves and DexUMI-style exoskeletons are easier and faster to collect than demonstrations collected via robot teleoperation, but they record only observed hand motion, not the corresponding joint commands. This missing information is crucial during contact, when applying forces to objects or the environment requires joint commands to differ from the hand's observed configuration. Robot teleoperation data directly capture this missing information through the residual between commanded joint targets and observed hand states. In this work, we propose Generative Action Synthesis (GAS), which trains a conditional generative model of these residuals from teleoperation data and uses it to synthesize action labels for kinematic-only demonstrations. For each frame, the model uses past and future visual observations together with observed joint states to sample plausible command residuals for relabeling kinematic-only demonstrations. GAS synthesizes useful actions zero-shot across diverse manipulation tasks beyond its teleoperation training distribution, including mug grasping, towel folding, cap twisting, test tube insertion, USB unplugging, and Rubik's cube manipulation. With these synthesized actions, we train policies for mug grasping, test tube insertion, cap twisting, and towel folding that achieve high success rates and outperform both naive state-as-action baselines and constant heuristic offsets. These results suggest that teleoperation data can provide supervision for converting action-free kinematic demonstrations into useful training data for robot policies. More videos are available on https://generativeactionsynthesis.github.io.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Dexterous Manipulation, Learning from Human Demonstration, Robot Learning, Imitation Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ap9JOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the gap between observed hand configurations and the joint-position commands needed to sustain contact. GAS learns command–state residuals from 11,564 teleoperation episodes using conditional flow matching, with RGB and past/future hand-state context. The frozen model relabels retargeted glove demonstrations, which then fine-tune a pretrained arm–hand behavior-cloning policy. On tasks excluded from synthesis-model training, the paper reports a 57.1 percentage-point replay improvement over State-copy across seven tasks and downstream success of 86% versus 32% across five tasks. The contribution is a practical transfer of robot control supervision to independently collected human demonstrations, with evaluation on a single hand/controller configuration.

Critical strengths
  1. A useful formulation of a consequential data-labeling problem. Learning the discrepancy between measured states and commanded targets directly addresses why geometrically plausible hand trajectories can fail during contact. Reusing teleoperation supervision to label new glove demonstrations is practically attractive: it separates acquisition of control knowledge from collection of target-task behavior. The general infer-actions-then-clone framework is established by BCO and VPT; the valuable step here is making it work for contact-sensitive, high-dimensional hand commands across the robot-to-human observation gap.

  2. The central empirical comparison controls the right variables. Replay keeps arm trajectories fixed, while downstream label variants share glove episodes, splits, initialization, architecture, and training settings. Thus, the large improvement from 32% to 86% supports the value of the synthesized hand labels rather than merely additional demonstrations. Seven replay tasks, five policy tasks, and vision/future-context ablations provide complementary evidence. The contact-sensitive grasping and insertion gains are particularly compelling; the much smaller benefit on towel folding also gives a useful boundary to the result. These controlled gains are the main reason for my weak-accept recommendation.

Critical weaknesses
  1. The specific value of generative modeling remains unresolved. Section 3 motivates a distribution over commands by multimodality, but inference averages four generated samples and then averages overlapping chunks. This does not establish that preserving multiple action hypotheses is responsible for the gains; averaging can also blend distinct modes. State-copy and corpus-mean overshoot are useful controls, but neither is a learned conditional predictor with matched observations and capacity. A deterministic residual-regression baseline would separate the benefit of learning context-dependent corrections from that of flow matching. Section 5 candidly acknowledges this gap. It limits the methodological contribution, although it does not negate the demonstrated utility of relabeling.

  2. The transfer and collection-efficiency claims need more auditable data accounting. Appendix C gives six families but not the identities of the 31 source tasks. Separately, Appendix A initializes BC from 3,381 robot episodes including folding and cap removal, without specifying their relation to the evaluation tasks or the GAS corpus. This leaves the scope of “without target-task teleoperation” unclear beyond final fine-tuning, even though the shared initialization preserves the label comparison. For the matched-time comparison, Table 4 explicitly reports recorded demonstration minutes, not the actual elapsed collection budgets. The episode ratios are therefore checkable, but the time matching is not. The 77.1-hour synthesis corpus is an upfront resource whose collection cost should be distinguished from marginal target-task effort. These reporting gaps chiefly constrain the broader generalization and efficiency interpretation.

Questions for authors
  1. Targeted experiment / claim clarification: With the same encoder, temporal context, training data, and overlap averaging, how does a deterministic residual predictor compare on replay success? This is the most informative missing comparison for the generative contribution. If unavailable, please restrict conclusions to learned conditional action synthesis rather than a demonstrated advantage from multimodality.

  2. Clarification of existing data: Please provide the source-task/object inventory and explain overlap among the 11,564-episode GAS corpus, the 3,381-episode BC pretraining corpus, and evaluation tasks. In particular, are the pretrained folding/cap-removal behaviors the evaluated tasks or distinct variants? Please scope the no-target-task-teleoperation claim accordingly; no cross-embodiment experiment is requested.

  3. Clarification of existing collection records: What were the actual wall-clock budgets and number/experience of operators for each interface and task? Please report whether resets and glove calibration were counted, and distinguish the upfront source-corpus cost from per-task collection effort. An equal-episode comparison would help explain the source of the advantage, but is not a prerequisite for a properly documented matched-time claim.

2 comments

We thank the reviewer for the thorough and insightful review, and for the positive assessment.

W1 / Q1: The specific value of generative modeling.

We agree this comparison would strengthen the paper and have added additional experiments. The deterministic predictor shares GAS's encoder, temporal context, training data and overlap averaging; only the flow head is replaced by a regressor head. We evaluate both by real-world open-loop replay of glove demonstrations on three contact-sensitive unseen tasks (10 trials each).

MethodMug-rimMug-topTest tubeTotal
State-copy (zero residual)0/103/100/103/30 (10%)
Deterministic regressor5/109/108/1022/30 (73%)
GAS, without chunk ensembling10/1010/109/1029/30 (97%)
GAS9/1010/108/1027/30 (90%)

GAS matches or exceeds the regressor on every task, with the largest gap on mug-rim grasping, which requires firm grasp to lift the cup. The full table is in the General Response including the absolute-target and no-ensembling variants.

W2 / Q2: Task inventory and overlap among the GAS corpus, the BC pretraining data and the evaluation tasks.

  • GAS training corpus. We listed all 31 tasks and their objects in Appendix C in the updated manuscript. No evaluation task is in the corpus: some objects recur (teacups, a towel, test tubes), but the evaluated skills do not (twisting, cloth folding, in-hand rotation, plug pulling). In the nearest-neighbour analysis from our reply to Reviewer PtuT (Q2) , every evaluation task is farther from the corpus than the median left-out corpus task, in both GAS features and raw hand states. Same-object tasks supply few neighbours (0.9% for towel folding, 1.4% for test-tube insertion). Mug-rim grasping draws 42% of its neighbours from teacup hanging, yet it is the farthest evaluation task from the corpus in GAS features.

  • BC pretraining corpus. The BC initialization is pretrained on a 3,381-episode subset of the GAS corpus under pick and place family. All task variants and the teleoperation baseline share this initialization, so it does not affect the comparisons.

Q3: Collection time audit and same episode experiments. Since data collection spanned multiple days, we therefore conducted separate 15-minute continuous collection trials for each interface and task, including resets, re-randomization, failed attempts, but excluding setup-time overhead. Accounting for setup time would further widen the data-efficiency gap between glove-based collection and teleoperation.

Mug-rimMug-topTowelCapTest tube
Teleop6449502142
Glove1451761034465
Ratio2.27×3.59×2.06×2.10×1.55×

Glove collection achieved 1.55–3.59× higher throughput over equal 15-minute trials. GAS additionally requires a one-time source corpus containing 77.1 hours of recorded teleoperation data, reused across unseen tasks. Thus, our comparison measures marginal per-task collection throughput rather than total end-to-end collection cost. The equal-episode policy comparison is reported in Table 2 of the General Response.

Dear Reviewer ap9J,

Thank you again for your detailed and constructive review, which helped strengthen our submission a lot. As the discussion period draws to a close, we wanted to briefly follow up on the three points you raised:

  • (Q1): we added the replay comparison you identified as an informative experiment, with the same encoder, temporal context, training data and overlap averaging. Generative predictor matches or exceeds the regressor on every task.

  • (Q2): the revised Appendix C lists all 31 corpus tasks and their objects, and we clarified that the BC pretraining set is a subset of the GAS corpus that contains no evaluation task.

  • (Q3): we report a throughput audit that includes resets and failed attempts and add the equal-episode comparison you suggested.

We would be grateful to know whether these address your concerns, and we are happy to clarify anything further before the deadline.

Reviewer_PtuTOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses a practical bottleneck in learning dexterous manipulation from human-centric demonstrations: motion-capture gloves and UMI-style interfaces record hand kinematics but not the joint-position commands required for control, and the two diverge precisely during contact. The authors propose Generative Action Synthesis (GAS): a conditional flow-matching model trained on a multitask teleoperation corpus (11,564 episodes, 31 tasks) to predict the residual between commanded and measured joint positions, conditioned on multi-view RGB and past/future joint-state context. The frozen model is applied zero-shot to relabel glove demonstrations of unseen tasks, and visuomotor BC policies are trained exclusively on the synthesized labels. On a bimanual system with 20-DoF hands, GAS improves open-loop replay success over state-copy labels by 57.1pp across 7 unseen tasks, improves downstream policy success from 32% to 86% across 5 tasks, and — under approximately matched collection time — GAS-labeled glove data yields 2.3–3.6x more demonstrations and higher average policy success than task-specific teleoperation data (86% vs 55%).

Critical strengths

A clearly identified, consequential problem with a well-motivated reframing. The command–state residual during contact is exactly the information that kinematic-only demonstrations lack, and using teleoperation data to learn a general state-to-command mapping elegantly decouples control supervision from task demonstrations. The design choices follow naturally: residual parameterization, offline conditioning on future observations, and a generative model for the one-to-many command mapping during contact. If the approach scales, it meaningfully changes the quality of dexterous data collection.

Careful evaluation design with large, consistent effect sizes. The experiments isolate the contribution of synthesized actions (open-loop replay with identical arm commands; identical glove demos differing only in labels), include task-disjoint zero-shot transfer, ablate visual and future context, and use matched initial configurations with 20 trials per condition, Beta posteriors, and pairwise Fisher tests. The gains (57pp replay, 54pp downstream) are largest precisely on contact-sensitive tasks, as the underlying account predicts.

Critical weaknesses

The headline teleoperation comparison conflates label quality with dataset size. The 86% vs 55% result (Sec. 4.3) compares GAS-labeled glove datasets of 450–1,081 episodes against teleoperation datasets of 150–301 episodes under matched wall-clock time. The paper's central practical claim that glove demonstrations with synthesized labels can substitute for teleoperation is therefore entangled with a 2.3–3.6x data-quantity advantage. A matched-episode-count comparison is the key missing experiment. Alternatively, a more direct comparison would be to evaluate the success rates of policies trained on smartphone-based teleoperation data versus glove-based demonstrations, given the same data collection time budget.

The degree of zero-shot transfer is difficult to assess. The teleoperation corpus spans 31 tasks in 6 families (pick-and-place, hanging, insertion/transfer, pouring, sorting, pressing), and the evaluation tasks (mug grasping, cap twisting, towel folding, test-tube insertion) appear kinematically and contact-wise close to these families; moreover, all experiments share the same hand hardware and controller as the corpus. Since the learned residual encodes controller- and embodiment-specific behavior (position-controller stiffness, actuator dynamics), the current evidence supports within-embodiment, near-distribution transfer rather than the broader "general relationship between observed kinematics and robot actions" suggested in the introduction. Characterizing corpus–evaluation similarity, or adding a genuinely distant task family, would substantially strengthen the claim.

Questions for authors
  • Can you report downstream policy success with glove demonstrations subsampled to the same episode count (or same data collection time budget) as the teleoperation baseline? This would isolate label quality from dataset size in the Sec. 4.3 comparison.

  • How much functional overlap exists between the 31 corpus tasks and the 7 evaluation tasks (e.g., does the corpus include grasping cylindrical containers, lid twisting, cloth folding)? A task-similarity vs. synthesis-quality analysis would clarify the zero-shot claim.

  • Is the learned residual tied to the Wuji hand's low-level position controller? Would the model require retraining if controller gains or the hand changed? Even small-scale evidence would be informative.

  • Does the generative formulation outperform a deterministic regressor with identical conditioning and chunk ensembling? Since inference averages K samples, the added value of flow matching itself is currently unclear.

3 comments

We thank the reviewer for the thorough and insightful review, and for the positive assessment of our problem framing and evaluation design.

W1 + Q1: Matched episode count.

We agree this comparison would strengthen the paper and have conducted additional experiments: BC on glove demonstrations subsampled to the teleoperation episode count, everything else identical (full table in the General Response). At equal count, GAS-labeled glove data reaches 50% success / 0.79 progress vs. 65% / 0.85 for teleoperation, without any robot demonstration of the target task. With the full glove datasets, GAS reaches 87.5% / 0.94. This isolates the two factors as the reviewer PtuT suggested: synthesized labels come close to true commands, and the downstream advantage comes from the larger dataset that glove collection enables. Collection throughput is reported in our reply to Reviewer ap9J (Q3).

Q2: Overlap between the corpus and the evaluation tasks.

The GAS training corpus shares objects with some evaluation tasks, but not the skills we test in the evaluation. We listed all 31 tasks and their objects in Appendix C.

To quantify overlap, we cut every episode into the 20-frame windows GAS conditions on and embed each window in two ways: GAS's own 768-d conditioning vector, and the raw hand-state window (20 frames × 20 joints). For each window we take the Euclidean distance to its 10th-nearest corpus training window, following the nearest-neighbour similarity used in [1], and divide it by the median of that distance over held-out episodes of corpus tasks. A ratio of 1.0 is therefore as close as a new episode of a task GAS was trained on; each row reports the median over a task's windows. As a reference for an unseen task, we score each corpus task's held-out episodes against the other 30 tasks only (leave-one-task-out). The last column names the corpus task that supplies most of a task's nearest neighbours in GAS features.

GAS-feature distancehand-state distancemain source of nearest neighbours (share)
Held-out episodes of seen tasks, median [range] over tasks0.98 [0.64–1.41]1.03 [0.79–1.52]
Held-out corpus task (left out), median [range] over tasks1.10 [0.95–1.51]1.83 [1.28–2.83]
Mug-top grasping1.482.77corn pick-and-place
USB unplugging1.493.71coffee-bean pouring
Rubik's-cube rotation1.552.73test-tube transfer
Towel folding1.603.36headphone hanging
Test-tube insertion1.632.51coffee-bean pouring
Cap twisting1.832.90corn pick-and-place
Mug-rim grasping1.953.46teacup hanging

All seven evaluation tasks lie farther from the corpus than the median held-out corpus task in both representations.

[1] Sun, et al. (2022). Out-of-Distribution Detection with Deep Nearest Neighbors. ICML 2022.

Q3: Dependence on the hand and its low-level controller.

Yes. The residual encodes how this hand’s position controller responds to commands, so a different hand would require retraining GAS on that hand’s teleoperation data. Changing controller gains also changes the required residual: in quasi-static contact, holding torque scales with stiffness × residual, so a stiffer controller needs a smaller residual for the same grip. We therefore expect substantial gain changes to require retraining, or possibly fine-tuning on a small amount of new data. This is by design, since a label must match the controller that executes it. Within a fixed configuration, GAS is robust to cross-station variation: the training data corpus was collected from four stations and evaluation spans two (see our reply to Reviewer 2tDJ, W3)).

Q4: Generative formulation vs. a deterministic regressor with identical conditioning and chunk ensembling.

We added the suggested baseline: GAS with the flow head replaced by an MSE regressor (full results in General Response). In open-loop replay on three contact-sensitive unseen tasks, GAS matches or exceeds the regressor on every task, with the largest gap on mug-rim grasping where firm grasp is needed.

We adopted the flow head as it performed better during development. Those comparisons were not comprehensive, and we listed this as a limitation (Sec. 5) in the submission. The controlled comparison above confirms the choice (details in our reply to Reviewer 2tDJ, W1). Both learned heads recover most of the gain over State-copy, and our contribution is the synthesis pipeline rather than a particular prediction head. A systematic study of architecture and formulation, including when a generative model outperforms deterministic predictors, remains future work (Sec. 5).

We hope these responses address your concerns. We would appreciate it if you would consider them in your final evaluation, and we are glad to discuss further.

Dear Authors,

Thank you for your prompt responses to my questions. Their responses have addressed most of my concerns, and I will maintain my score and support acceptance of the paper.

I find this work highly novel, with an elegant approach and solid experimental evidence. I have two remaining points for consideration:

  1. The method’s reliance on the underlying low-level controller may limit its generalization across controllers and, potentially, across different dexterous hand morphologies. Further investigation of this limitation would be valuable in future work.
  2. Does a similar issue involving end-effector states and actions also arise in manipulation datasets collected with two-finger grippers?

Best,

Thank you for the kind words and for supporting acceptance.

  1. We agree on the dependence on the controller and hand morphology, where we explicitly acknowledge this limitation and leave cross embodiments and controllers for future work (Sec. 5). The proposed general framework, GAS, learns the command–state residual of the hand and controller that produced the teleoperation data. We will expand this point in the limitations (Sec. 5). Promising directions include training one synthesis model on teleoperation data from several hands or controllers, conditioned on their parameters, and adapting to a new setup with a small amount of teleoperation data. Our aim in this work was to show that the contact supervision missing from human kinematic demonstrations can be inferred from existing teleoperation data.

  2. Yes, the same gap arises. A handheld gripper interface such as UMI [1] records the gripper width at contact, not the closing command needed to hold the object. With a single degree of freedom, a fixed closing offset [1] or a close-until-contact rule is usually sufficient (Sec. 2.1). Dexterous hands coordinate many joints over multiple changing contacts, and a single offset is not enough. Our Constant overshoot baseline, the dexterous hand analogue of this heuristic, improves only some tasks and remains well below GAS (Fig. 5). Therefore, we learn the correction from teleoperation data instead.

[1] Chi, et al. (2024). Universal Manipulation Interface: In-the-Wild Robot Teaching Without In-the-Wild Robots. RSS 2024.

If our responses resolved the remaining concerns, we would be grateful if you would consider reflecting this in the final rating. Please let us know if you have any further questions. Thank you again for your valuable feedback and support. We greatly appreciate your time and consideration.

Reviewer_2tDJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Generative Action Synthesis (GAS), a method for generating robot action labels from human kinematic demonstrations. The method trains a conditional flow matching model on robot teleoperation data to predict the residual between commanded joint positions and observed joint states. The trained model is then applied to human glove demonstrations to synthesize robot actions for downstream behavior cloning. Experiments on real robot manipulation tasks demonstrate improvements in action replay and policy success rates compared with directly using observed joint states as actions.

Critical strengths

The paper addresses an important problem in dexterous robot manipulation, particularly the lack of action labels in human kinematic demonstrations.

The idea of learning command residuals from teleoperation data and transferring them to human demonstrations is reasonable and interesting.

The method can generate action labels for previously unseen tasks without requiring task-specific robot teleoperation data.

The experiments include real robot evaluations across seven action replay tasks and five downstream policy learning tasks, demonstrating substantial improvements in manipulation success.

The paper provides ablation studies on visual observations and future context, as well as comparisons with heuristic action labeling and robot teleoperation.

Critical weaknesses

The baseline comparisons are insufficient. The authors do not compare against direct action prediction, deterministic residual prediction, or stronger trajectory-dependent correction methods. Furthermore, the model averages multiple generated samples during inference, making the necessity of generative modeling unclear. The current comparisons do not establish whether either the residual formulation or generative modeling provides advantages over simpler approaches.

The physical interpretation of the predicted residual is insufficiently investigated. The difference between commanded and observed joint positions may result from actuator dynamics, control latency, or physical contact. The proposed method does not explicitly distinguish these factors. Furthermore, the paper does not provide quantitative measurements of contact forces or grasp stability to validate whether the synthesized residuals accurately capture the physical interactions required for manipulation.

The generalizability of the proposed method remains unclear. Although the authors evaluate on unseen tasks, all experiments use the same robot hand and control system. Since command residuals depend on actuator characteristics and controller parameters, it is uncertain whether the learned relationship can transfer to different robot embodiments or controllers.

The comparison with robot teleoperation is not entirely fair. Under matched collection time, glove demonstrations contain substantially more samples than teleoperation demonstrations. The comparison also does not account for the large teleoperation corpus used to pretrain GAS. Consequently, the higher downstream success rate cannot be attributed solely to the proposed action synthesis method. Additional controlled experiments are needed to isolate the effects of action labeling, dataset size, and pretraining.

Questions for authors

The paper lacks comparisons with existing trajectory correction methods, making it unclear whether the proposed generative residual model offers meaningful advantages over simpler approaches.

4 comments

We thank the reviewer for the detailed review and for recognizing the importance of the problem and the strength of the real-robot evaluation.

W1 + Q1: Generative modeling and residual prediction; comparison with trajectory-correction methods.

Please also see the General Response for the full table comparison. We trained the baselines below on the same corpus and split as GAS, and evaluated them by real-world open-loop replay of glove demonstrations on mug-rim grasping, mug-top grasping and test-tube insertion (10 trials each; results in Table 1 of the General Response):

  1. Ridge regression from a 20-frame joint-state window (16 past, current, 4 future) to the residual. It contains the linear inverse-dynamics correction u = s + τ·ṡ as a special case and serves as the trajectory-correction baseline (Q1).
  2. Deterministic regressor: GAS with the flow head replaced by an MSE head (same encoder, context, pooler and chunk ensembling).
  3. Absolute target: the same flow model predicting the command itself instead of the residual.
  4. GAS without chunk ensembling: each predicted chunk is used as it is.

A linear correction is not enough. Linear inverse dynamics succeeds in 60% of replays vs. 90% for GAS, and fails most on mug-rim grasping (2/10), which requires a firm grasp to lift the heavy mug.

Generative vs. deterministic. We adopted the flow head due to empirical stronger performance during development. In the controlled replay comparison above, which directly tests whether synthesized commands accomplish the task, confirms the stronger performance of the flow head. Every generative variant matches or exceeds the regressor on all three tasks. Both learned heads recover most of the gain over State-copy, and our contribution is the synthesis pipeline rather than a particular prediction head design. A systematic study of architecture and formulation, including when a generative model outperforms deterministic predictors remains future work (Sec. 5).

W2: Physical interpretation of the residual and contact quantification.

We intentionally do not decompose the residual. Actuator dynamics, control latency and contact all act on the same robot that executes the labels, so a useful label must reproduce their combined effect. Isolating them would require joint-torque or fingertip force sensing, which our setup does not have. We therefore evaluate synthesized actions functionally: does executing them on the robot produce the physical interaction the task requires?

In the evaluation, replay gains are largest on tasks that require sustained grip: mug-rim grasping ([0%] → [90%]), test-tube insertion ([0%] → [75%]) and USB unplugging ([0%] → [95%]). For deformable towel folding, which requires less grip force, shows no gain (Fig. 5). Grasp stability is measured directly if the recovered data preserve the hand-object interaction described in the dataset.

We will add this discussion and a per-stage failure breakdown to the revision.

W3: Generalization across embodiments and controllers.

We agree the trained model is specific to a robot type, and we do not claim cross-embodiment transfer (Sec. 5). We propose a transferable method for synthesizing action labels that align with the commands required by the executing controller. We argue the method itself is what transfers across settings rather than the synthesized labels. For a new hand, GAS is trained once on that robot’s teleoperation data and then labels glove demonstrations of new tasks without task-specific teleoperation data.

Within one hardware configuration, GAS already handles variation in actuator characteristics. Physical properties of the same type of hand differ in friction, wear and calibration. Our teleoperation corpus pools data from 4 robot stations, and evaluations were run on 2 of them, which shows the synthesized label work for the same type of the robot

W4: Matched episodes comparison.

We agree this comparison would strengthen the paper and add additional experiments (General Response, "Same-episode-count comparison"). With glove data subsampled to the teleoperation episode count, GAS labels reach 50% success vs. 65% for teleoperation, where the advantage of the full glove datasets (87.5%) comes mainly from dataset size rather than per-episode label quality. This gap is expected: teleoperation records ground-truth actions on the target robot and thus serves as an oracle upper bound. The goal of GAS is not to produce labels superior to real robot demonstrations, but synthetic labels to train policies from human kinematic data, which can then be scaled. Regarding the teleoperation corpus used to train GAS: it is a one-time cost per robot, reused for every new task, and contains no evaluation task (our reply to Reviewer ap9J, Q2). We will state this scope explicitly in Sec. 4.3.

We hope these clarifications address your concerns. We would be grateful if you would take them into account in your final assessment, and we are happy to answer any further questions.

Thank you for addressing my comments. I will maintain my positive score.

Dear Reviewer 2tDJ,

Thank you for your positive assessment and for the constructive discussion, which has helped us strengthen the paper. Please feel free to let us know if any concerns remain; we would be happy to address them.

Comments on the paper

1 comment

We thank all reviewers for their thoughtful feedback. Below we address the questions shared by all three reviewers; all other points are answered in the individual replies.

What GAS contributes. Our contribution is the action-synthesis framework rather than the choice of a generative head:

  1. Identifying and learning the missing label. The command–state residual is exactly the information that kinematic-only demonstrations lack, and it matters most during contact. We show that it can be learned from a multi-task teleoperation corpus (11,564 episodes, 31 tasks).
  2. Zero-shot relabeling of human demonstrations. A single frozen model relabels glove demonstrations of unseen tasks, with no target-task teleoperation or robot interaction.
  3. Controlled evidence that the synthesized labels matter. Replay keeps the arm trajectories fixed; policy learning shares glove episodes, splits, initialization and architecture, with only the hand labels differing. Under these controls, GAS improves over State-copy labels by 57.1 pp in replay success across seven unseen tasks and downstream policy success from 32% to 86% across five tasks.

1. Role of the generative formulation (Reviewers 2tDJ W1, PtuT Q4, ap9J W1/Q1)

Why a conditional generative model over a deterministic regressor. We adopted the flow head due to empirical stronger performance. Those comparisons were not comprehensive, and we acknowledge this as a limitation (Sec. 5). The controlled replay comparison in Table 1 confirms the choice that every generative variant matches or exceeds the regressor on all three tasks. Both learned heads recover most of the gain over State-copy, and our contribution is the synthesis pipeline rather than a particular prediction head design. A systematic study of architecture and formulation, including when a generative model outperforms deterministic predictors remains future work (Sec. 5).

Additional baselines. We trained the suggested baselines on the same corpus and split and evaluated them by open-loop replay of glove demonstrations on mug-rim grasping, mug-top grasping and test-tube insertion (Table 1). A linear correction (ridge regression on the joint-state window) recovers only part of the gain over State-copy. Predicting the absolute command works as well as the residual, and we claim no advantage for either parameterization; the gain comes from learned, context-conditioned action synthesis.

Table 1. Open-loop replay success of glove demonstrations (10 trials per task).

MethodMug-rimMug-topTest tubeTotal
State-copy (zero residual)0/103/100/103/30 (10%)
Ridge (linear inverse dynamics)2/107/109/1018/30 (60%)
Deterministic regressor5/109/108/1022/30 (73%)
GAS, absolute target9/1010/109/1028/30 (93%)
GAS, without chunk ensembling10/1010/109/1029/30 (97%)
GAS9/1010/108/1027/30 (90%)

2. Label quality vs. dataset size (Reviewers 2tDJ W4, PtuT W1/Q1, ap9J Q3)

To separate label quality from dataset size, we trained BC on glove demonstrations subsampled to each task's teleoperation episode count (GAS-matched), keeping initialization, architecture and training identical (Table R2). Each policy is evaluated on 10 real-robot trials per task with matched initial configurations (metrics as in Sec. 4.1). Due to time constraints we omit towel folding, where GAS labels don't show much gain over State-copy (Fig. 7).

Table 2. Downstream BC policy success rate and progress ratio.

TaskTeleopGAS-matched (same #eps)GAS (full glove data)
Mug-rim10/10, 1.009/10, 0.9810/10, 1.00
Mug-top4/10, 0.842/10, 0.7610/10, 1.00
Test tube4/10, 0.643/10, 0.588/10, 0.88
Cap8/10, 0.926/10, 0.847/10, 0.88
Average65%, 0.8550%, 0.7987.5%, 0.94

At equal episode count, GAS labels are slightly below teleoperation on every task. This is expected, since teleoperation labels are the robot's true commands while GAS augments human demonstration of these tasks without robot data. With the full glove datasets, GAS outperforms teleoperation, where the advantage comes mainly from the larger number of demonstrations that robot-free glove collection provides.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.