- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes RoboTok, a scalable data engine that retrieves manipulation-relevant human demonstrations from internet videos given a query human demonstration. RoboTok represents videos using actor-centered 3D hand trajectories, uses DTW to define motion similarity, and distills the resulting local neighborhood structure into an efficiently searchable embedding space. The retrieved demonstrations are further shown to provide useful guidance for dexterous robot policy learning in simulation and real-world experiments.
Critical strengths
A compelling formulation of internet video as a queryable source of robot-learning supervision. Rather than assuming a fixed curated human dataset, the paper formulates data acquisition itself as retrieval: a target human demonstration is used to actively identify behaviorally similar demonstrations from a large web-video corpus. Importantly, relevance is defined through manipulation motion rather than visual appearance or semantic labels, providing a conceptually useful interface for exploiting continuously growing human video data.
The proposed method is tightly aligned with the formulation and supported by strong downstream evidence. Each component addresses a clear obstacle in behavior-based retrieval: actor-centered canonicalization reduces viewpoint variation, DTW accommodates differences in execution speed, and the learned embedding amortizes expensive sequence alignment into efficient nearest-neighbor search. The evaluation further connects retrieval fidelity to downstream policy utility, including held-out and cross-dataset retrieval, simulation experiments, and real-world policy evaluation.
Critical weaknesses
The scope of hand-trajectory similarity as a proxy for manipulation relevance is not yet fully established. The formulation operationalizes manipulation similarity entirely through actor-centered 3D hand kinematics. The generally works for coarse-grained tasks, however, similar hand trajectories can correspond to different object interactions or contact modes, while the same manipulation may require different trajectories across object geometries. Moreover, the main retrieval evaluation uses exact DTW over the same hand representation as its ground truth, so it primarily measures how faithfully RoboTok approximates the proposed similarity metric rather than independently validating that metric as the correct notion of manipulation relevance. The downstream results provide important independent evidence, but the main modified tasks were explicitly selected for meaningful 3D hand motion and the policy guidance itself is defined through robot hand-state proximity, making the current evaluation (Table 2) particularly favorable to hand-kinematic relevance.
The dependence on near-static-camera videos is not yet well characterized. The current data engine explicitly removes clips with nontrivial camera translation or zoom before extracting hand trajectories. Unlike hand-visibility or single-actor constraints, which are natural prerequisites for reliably defining the proposed representation, the near-static-camera assumption appears to be a stronger restriction on the video distribution itself. It is therefore unclear how sensitive RoboTok is to moderate egocentric or third-person camera motion, and whether this constraint is potentially a limitation of its application. Clarifying this boundary would help better establish the coverage of the proposed internet-scale retrieval framework.
Questions for authors
Scope of hand-motion-based manipulation similarity. How do the authors view the boundary of actor-centered hand trajectories as a representation of manipulation relevance? In particular, cases where similar hand motions correspond to different object/contact interactions, or where the same manipulation induces different trajectories across object geometries, seem particularly informative. I would be interested in the authors' perspective on when hand kinematics is expected to be sufficient and when object-relative or contact information may become important.
Sensitivity to camera motion. Could the authors clarify how sensitive the trajectory extraction and retrieval pipeline is to camera motion beyond the current static-camera threshold? In particular, I would be interested in whether the filtering criterion is mainly a conservative choice for obtaining reliable reconstructions, or whether moderate camera motion leads to failures in the resulting actor-centered trajectories or retrieval quality. This would help clarify how restrictive the current assumption is in practice.
5 comments
Part 2/3: Independent Relevance Validation and Task/Reward Setup (W1)
W1. Moreover, the main retrieval evaluation uses exact DTW over the same hand representation as its ground truth ... the main modified tasks were explicitly selected for meaningful 3D hand motion and the policy guidance itself is defined through robot hand-state proximity, making the current evaluation (Table 2) particularly favorable to hand-kinematic relevance.
1. Validating manipulation relevance independently of DTW (W1). Regarding DTW as ground truth, we agree that Tab. 2 measures DTW distillation fidelity (Sec. 5.1). To strengthen independent validation of manipulation relevance:
- Expanded multi-query and Exact DTW evaluation (R4): We expanded downstream simulation evaluation to 3 queries per task and added an Exact DTW baseline, where RoboTok reaches within 2.1 points of Exact DTW (49.9% vs. 52.0%). Added to Sec. 5.2 (Tab. 3, pp. 8-9) and App. E.5 (Tab. 19, pp. 27-28), highlighted in blue.
- DTW-to-success correlation (R3): Correlating retrieval DTW cost with simulation success across all six methods shows that lower DTW cost strongly predicts higher task success (Spearman , ). Added as a plotted graph to Sec. 5.2 (Fig. 6, pp. 9-10) and App. E.6 (Tab. 20, p. 28), highlighted in blue.
R4. Simulation success rate (%) across three queries per task.
| Tasks | Split | Base | Random | Flow | HAND | STRAP | RoboTok | Exact DTW |
|---|---|---|---|---|---|---|---|---|
| BottleCap Turning | Seen | 0.1±0.1 | 1.8±0.5 | 1.9±1.0 | 54.3±5.2 | 58.8±4.2 | 75.4±2.3 | 78.9±3.2 |
| BottleCap Turning | Unseen | 0.0±0.0 | 0.7±0.3 | 0.4±0.4 | 26.6±3.9 | 27.1±3.0 | 36.8±3.8 | 40.3±4.3 |
| Faucet Screwing | Seen | 1.0±0.6 | 2.3±1.6 | 1.3±0.6 | 3.9±1.7 | 4.1±1.8 | 42.7±2.7 | 46.6±3.5 |
| Faucet Screwing | Unseen | 1.7±1.7 | 1.0±0.5 | 0.3±0.2 | 1.5±0.7 | 1.8±1.1 | 9.7±2.4 | 10.2±1.7 |
| Lever Sliding | Seen | 0.8±0.3 | 13.9±3.0 | 13.4±3.1 | 22.4±4.0 | 15.8±3.3 | 77.6±3.2 | 81.4±2.7 |
| Lever Sliding | Unseen | 2.1±2.1 | 12.8±4.6 | 13.2±5.3 | 22.7±6.7 | 14.1±3.4 | 57.5±4.5 | 54.4±4.8 |
R3. Retrieval DTW cost@10 vs. mean downstream simulation success (3 queries × 8 seeds across 6 task/split settings). Spearman , between DTW cost and success rate.
| Method | DTW cost@10 (m) ↓ | Cost rank | Mean sim success (%) ↑ | Success rank |
|---|---|---|---|---|
| Exact DTW | 1.041 | 1 | 52.0 | 1 |
| RoboTok | 1.197 | 2 | 49.9 | 2 |
| STRAP | 3.903 | 3 | 20.3 | 4 |
| HAND | 4.478 | 4 | 21.9 | 3 |
| Random | 4.737 | 5 | 5.4 | 5 |
| Flow | 6.708 | 6 | 5.1 | 6 |
2. Task selection and hand-state guidance (W1).
- Unmodified VTDexManip tasks: On all six original VTDexManip tasks (where the hand base is fixed), RoboTok still leads all retrieval methods in 11 of 12 task/split settings (App. E.4, Tab. 15, now Tab. 18).
- Controlled RL and shaping comparison: Guiding RL policies via robot hand-state proximity to retargeted video hand poses alongside an outcome-based task reward follows established practice in learning dexterous manipulation from human video (Mandikal & Grauman, 2021). All evaluated retrieval methods share the exact same retargeting pipeline, hand-state shaping reward, and RL setup, differing only in the retrieved demonstrations, while test-time evaluation is judged strictly by physical object outcomes with no demonstrations.
References:
P. Mandikal & K. Grauman, DexVIP: Learning dexterous grasping with human hand pose priors from video, CoRL 2021.
Part 3/3: Static-Camera Filtering and Camera Motion Sensitivity (W2, Q2)
W2 / Q2. The dependence on near-static-camera videos is not yet well characterized. ... whether the filtering criterion is mainly a conservative choice for obtaining reliable reconstructions, or whether moderate camera motion leads to failures in the resulting actor-centered trajectories or retrieval quality.
We appreciate you raising this point and question. The near-static-camera filter is a conservative choice in our current extraction pipeline rather than a tuned failure boundary or a requirement of RoboTok's encoder and retrieval engine.
- 1. Conservative static-camera threshold and corpus coverage. Our 1-pixel translation and 0.5% zoom cutoffs were chosen as conservative priors for reliable 3D kinematics rather than tuned failure limits. Even with this filter, static-camera instructional video remains abundant on the web, where cooking, crafts, and mechanical work comprise roughly 30%, 10%, and 10% of our corpus.
- 2. Sensitivity to camera motion and relaxing the assumption. Crucially, the RoboTok torso estimator, encoder, and retrieval engine are agnostic to how 3D hand trajectories are obtained. Because our current hand-pose extraction pipeline estimates one static camera-to-torso transform and one median focal length per clip, uncompensated camera translation and zoom mix with camera-frame hand motion and depth scaling. Relaxing this filter requires world-coordinate hand tracking to subtract camera motion before torso canonicalization, which we intend to explore in future work and have added to our future work discussion in Sec. 6 (p. 10), highlighted in blue.
Thank you again for your thoughtful review and positive support of our work, and please let us know if you have any follow-up questions that we can clarify. We hope these clarifications and new evaluations fully address your questions, and we would be grateful if you would consider raising your score to 4 (accept) in light of these results.
We sincerely appreciate the time you spent reviewing our paper, your clear articulation of our formulation and method's strengths, and your insightful questions and comments. We address each point below.
Part 1/3: Scope of Actor-Centered 3D Hand Kinematics (W1, Q1)
Thank you for this thoughtful feedback and insightful question. We agree that hand kinematics is a partial proxy for manipulation relevance. Actor-centered 3D hand trajectories provide a body-relative prior for shared manipulation poses and finger articulation without requiring noisy or unavailable 3D object/contact annotations in web video. We also agree that object-relative and contact/force cues are essential for finer physical distinctions and remain a natural direction for future work.