- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses the gap between observed hand configurations and the joint-position commands needed to sustain contact. GAS learns command–state residuals from 11,564 teleoperation episodes using conditional flow matching, with RGB and past/future hand-state context. The frozen model relabels retargeted glove demonstrations, which then fine-tune a pretrained arm–hand behavior-cloning policy. On tasks excluded from synthesis-model training, the paper reports a 57.1 percentage-point replay improvement over State-copy across seven tasks and downstream success of 86% versus 32% across five tasks. The contribution is a practical transfer of robot control supervision to independently collected human demonstrations, with evaluation on a single hand/controller configuration.
Critical strengths
-
A useful formulation of a consequential data-labeling problem. Learning the discrepancy between measured states and commanded targets directly addresses why geometrically plausible hand trajectories can fail during contact. Reusing teleoperation supervision to label new glove demonstrations is practically attractive: it separates acquisition of control knowledge from collection of target-task behavior. The general infer-actions-then-clone framework is established by BCO and VPT; the valuable step here is making it work for contact-sensitive, high-dimensional hand commands across the robot-to-human observation gap.
-
The central empirical comparison controls the right variables. Replay keeps arm trajectories fixed, while downstream label variants share glove episodes, splits, initialization, architecture, and training settings. Thus, the large improvement from 32% to 86% supports the value of the synthesized hand labels rather than merely additional demonstrations. Seven replay tasks, five policy tasks, and vision/future-context ablations provide complementary evidence. The contact-sensitive grasping and insertion gains are particularly compelling; the much smaller benefit on towel folding also gives a useful boundary to the result. These controlled gains are the main reason for my weak-accept recommendation.
Critical weaknesses
-
The specific value of generative modeling remains unresolved. Section 3 motivates a distribution over commands by multimodality, but inference averages four generated samples and then averages overlapping chunks. This does not establish that preserving multiple action hypotheses is responsible for the gains; averaging can also blend distinct modes. State-copy and corpus-mean overshoot are useful controls, but neither is a learned conditional predictor with matched observations and capacity. A deterministic residual-regression baseline would separate the benefit of learning context-dependent corrections from that of flow matching. Section 5 candidly acknowledges this gap. It limits the methodological contribution, although it does not negate the demonstrated utility of relabeling.
-
The transfer and collection-efficiency claims need more auditable data accounting. Appendix C gives six families but not the identities of the 31 source tasks. Separately, Appendix A initializes BC from 3,381 robot episodes including folding and cap removal, without specifying their relation to the evaluation tasks or the GAS corpus. This leaves the scope of “without target-task teleoperation” unclear beyond final fine-tuning, even though the shared initialization preserves the label comparison. For the matched-time comparison, Table 4 explicitly reports recorded demonstration minutes, not the actual elapsed collection budgets. The episode ratios are therefore checkable, but the time matching is not. The 77.1-hour synthesis corpus is an upfront resource whose collection cost should be distinguished from marginal target-task effort. These reporting gaps chiefly constrain the broader generalization and efficiency interpretation.
Questions for authors
-
Targeted experiment / claim clarification: With the same encoder, temporal context, training data, and overlap averaging, how does a deterministic residual predictor compare on replay success? This is the most informative missing comparison for the generative contribution. If unavailable, please restrict conclusions to learned conditional action synthesis rather than a demonstrated advantage from multimodality.
-
Clarification of existing data: Please provide the source-task/object inventory and explain overlap among the 11,564-episode GAS corpus, the 3,381-episode BC pretraining corpus, and evaluation tasks. In particular, are the pretrained folding/cap-removal behaviors the evaluated tasks or distinct variants? Please scope the no-target-task-teleoperation claim accordingly; no cross-embodiment experiment is requested.
-
Clarification of existing collection records: What were the actual wall-clock budgets and number/experience of operators for each interface and task? Please report whether resets and glove calibration were counted, and distinguish the upfront source-corpus cost from per-task collection effort. An equal-episode comparison would help explain the source of the advantage, but is not a prerequisite for a properly documented matched-time claim.
2 comments
Dear Reviewer ap9J,
Thank you again for your detailed and constructive review, which helped strengthen our submission a lot. As the discussion period draws to a close, we wanted to briefly follow up on the three points you raised:
-
(Q1): we added the replay comparison you identified as an informative experiment, with the same encoder, temporal context, training data and overlap averaging. Generative predictor matches or exceeds the regressor on every task.
-
(Q2): the revised Appendix C lists all 31 corpus tasks and their objects, and we clarified that the BC pretraining set is a subset of the GAS corpus that contains no evaluation task.
-
(Q3): we report a throughput audit that includes resets and failed attempts and add the equal-episode comparison you suggested.
We would be grateful to know whether these address your concerns, and we are happy to clarify anything further before the deadline.
We thank the reviewer for the thorough and insightful review, and for the positive assessment.
W1 / Q1: The specific value of generative modeling.
We agree this comparison would strengthen the paper and have added additional experiments. The deterministic predictor shares GAS's encoder, temporal context, training data and overlap averaging; only the flow head is replaced by a regressor head. We evaluate both by real-world open-loop replay of glove demonstrations on three contact-sensitive unseen tasks (10 trials each).
GAS matches or exceeds the regressor on every task, with the largest gap on mug-rim grasping, which requires firm grasp to lift the cup. The full table is in the General Response including the absolute-target and no-ensembling variants.
W2 / Q2: Task inventory and overlap among the GAS corpus, the BC pretraining data and the evaluation tasks.
GAS training corpus. We listed all 31 tasks and their objects in Appendix C in the updated manuscript. No evaluation task is in the corpus: some objects recur (teacups, a towel, test tubes), but the evaluated skills do not (twisting, cloth folding, in-hand rotation, plug pulling). In the nearest-neighbour analysis from our reply to Reviewer PtuT (Q2) , every evaluation task is farther from the corpus than the median left-out corpus task, in both GAS features and raw hand states. Same-object tasks supply few neighbours (0.9% for towel folding, 1.4% for test-tube insertion). Mug-rim grasping draws 42% of its neighbours from teacup hanging, yet it is the farthest evaluation task from the corpus in GAS features.
BC pretraining corpus. The BC initialization is pretrained on a 3,381-episode subset of the GAS corpus under pick and place family. All task variants and the teleoperation baseline share this initialization, so it does not affect the comparisons.
Q3: Collection time audit and same episode experiments. Since data collection spanned multiple days, we therefore conducted separate 15-minute continuous collection trials for each interface and task, including resets, re-randomization, failed attempts, but excluding setup-time overhead. Accounting for setup time would further widen the data-efficiency gap between glove-based collection and teleoperation.
Glove collection achieved 1.55–3.59× higher throughput over equal 15-minute trials. GAS additionally requires a one-time source corpus containing 77.1 hours of recorded teleoperation data, reused across unseen tasks. Thus, our comparison measures marginal per-task collection throughput rather than total end-to-end collection cost. The equal-episode policy comparison is reported in Table 2 of the General Response.