acceptodds
Under review as a conference paper at ICLR 2027

RoboTok: A Scalable Data Engine for Internet Demonstration Video Retrieval and Dexterous Manipulation Learning

Abstract

Robot learning increasingly depends on broad and diverse demonstrations, yet collecting robot data remains expensive and difficult to scale across the wide range of real-world tasks. To address this bottleneck, we introduce RoboTok, a scalable data engine that uses a query human manipulation video to retrieve manipulation-relevant internet demonstrations for training dexterous robot policies. Specifically, we learn a latent motion space from 3D hand trajectories expressed in estimated actor-centered reference frames. This representation enables manipulation behaviors to be compared across variations in camera viewpoint, scene appearance, and actor occlusions, while remaining compact enough for efficient indexing and retrieval over internet video collections. We evaluate RoboTok against existing robot-data retrieval approaches using retrieval metrics and downstream robot policy performance, showing that RoboTok retrieves more manipulation-relevant demonstrations and improves downstream task success. In real-world robot experiments, RoboTok-guided policies achieve a mean success improvement of 42.2 percentage points over the strongest baseline for each task, establishing hand-pose trajectory-aware retrieval as a scalable way to leverage continuously growing web video for robot learning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot learning, demonstration retrieval, dexterous manipulation, learning from human video, motion representation learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9WEEOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RoboTok, a scalable data engine that retrieves manipulation-relevant human demonstrations from internet videos given a query human demonstration. RoboTok represents videos using actor-centered 3D hand trajectories, uses DTW to define motion similarity, and distills the resulting local neighborhood structure into an efficiently searchable embedding space. The retrieved demonstrations are further shown to provide useful guidance for dexterous robot policy learning in simulation and real-world experiments.

Critical strengths

A compelling formulation of internet video as a queryable source of robot-learning supervision. Rather than assuming a fixed curated human dataset, the paper formulates data acquisition itself as retrieval: a target human demonstration is used to actively identify behaviorally similar demonstrations from a large web-video corpus. Importantly, relevance is defined through manipulation motion rather than visual appearance or semantic labels, providing a conceptually useful interface for exploiting continuously growing human video data.

The proposed method is tightly aligned with the formulation and supported by strong downstream evidence. Each component addresses a clear obstacle in behavior-based retrieval: actor-centered canonicalization reduces viewpoint variation, DTW accommodates differences in execution speed, and the learned embedding amortizes expensive sequence alignment into efficient nearest-neighbor search. The evaluation further connects retrieval fidelity to downstream policy utility, including held-out and cross-dataset retrieval, simulation experiments, and real-world policy evaluation.

Critical weaknesses

The scope of hand-trajectory similarity as a proxy for manipulation relevance is not yet fully established. The formulation operationalizes manipulation similarity entirely through actor-centered 3D hand kinematics. The generally works for coarse-grained tasks, however, similar hand trajectories can correspond to different object interactions or contact modes, while the same manipulation may require different trajectories across object geometries. Moreover, the main retrieval evaluation uses exact DTW over the same hand representation as its ground truth, so it primarily measures how faithfully RoboTok approximates the proposed similarity metric rather than independently validating that metric as the correct notion of manipulation relevance. The downstream results provide important independent evidence, but the main modified tasks were explicitly selected for meaningful 3D hand motion and the policy guidance itself is defined through robot hand-state proximity, making the current evaluation (Table 2) particularly favorable to hand-kinematic relevance.

The dependence on near-static-camera videos is not yet well characterized. The current data engine explicitly removes clips with nontrivial camera translation or zoom before extracting hand trajectories. Unlike hand-visibility or single-actor constraints, which are natural prerequisites for reliably defining the proposed representation, the near-static-camera assumption appears to be a stronger restriction on the video distribution itself. It is therefore unclear how sensitive RoboTok is to moderate egocentric or third-person camera motion, and whether this constraint is potentially a limitation of its application. Clarifying this boundary would help better establish the coverage of the proposed internet-scale retrieval framework.

Questions for authors

Scope of hand-motion-based manipulation similarity. How do the authors view the boundary of actor-centered hand trajectories as a representation of manipulation relevance? In particular, cases where similar hand motions correspond to different object/contact interactions, or where the same manipulation induces different trajectories across object geometries, seem particularly informative. I would be interested in the authors' perspective on when hand kinematics is expected to be sufficient and when object-relative or contact information may become important.

Sensitivity to camera motion. Could the authors clarify how sensitive the trajectory extraction and retrieval pipeline is to camera motion beyond the current static-camera threshold? In particular, I would be interested in whether the filtering criterion is mainly a conservative choice for obtaining reliable reconstructions, or whether moderate camera motion leads to failures in the resulting actor-centered trajectories or retrieval quality. This would help clarify how restrictive the current assumption is in practice.

5 comments

We sincerely appreciate the time you spent reviewing our paper, your clear articulation of our formulation and method's strengths, and your insightful questions and comments. We address each point below.

Part 1/3: Scope of Actor-Centered 3D Hand Kinematics (W1, Q1)

W1 / Q1. The formulation operationalizes manipulation similarity entirely through actor-centered 3D hand kinematics. ... In particular, cases where similar hand motions correspond to different object/contact interactions, or where the same manipulation induces different trajectories across object geometries ... when hand kinematics is expected to be sufficient and when object-relative or contact information may become important.

Thank you for this thoughtful feedback and insightful question. We agree that hand kinematics is a partial proxy for manipulation relevance. Actor-centered 3D hand trajectories provide a body-relative prior for shared manipulation poses and finger articulation without requiring noisy or unavailable 3D object/contact annotations in web video. We also agree that object-relative and contact/force cues are essential for finer physical distinctions and remain a natural direction for future work.

  • When actor-centered hand kinematics suffices: Body-relative 3D hand trajectories capture how a person manipulates objects similarly across tasks and settings without requiring 3D object models or contact annotations in web video (which are often unavailable or noisy to reconstruct). For example, turning different bottle caps or faucet handles shares an enclosing hand pose with finger-rolling motions, and pulling different levers or handles shares a finger-wrapping posture. Paired with outcome-based RL, this hand-pose prior guides the robot hand into appropriate poses while our outcome-based RL discovers fine contacts, enabling transfer without test-time demonstrations to unseen simulated geometries (Tab. 3) and zero-shot to three 3D-printed objects per task in the real world (Fig. 6, now Fig. 7).
  • When object-relative, contact, or force cues are needed: Object/contact interactions are crucial for finer task-level manipulation similarity, particularly when (i) similar hand motions yield different physical interactions without contact or force cues (e.g., gently holding and twisting a deformable paper cup vs. applying high finger torque to unscrew a tight jar lid), or (ii) differences in object size or articulation geometry alter the torso-frame hand path for the same manipulation, causing pure hand kinematics to separate similar behaviors without object-relative cues (e.g., pulling open a narrow cabinet door vs. a wide door along a larger arc, or turning a small crank vs. a large wheel). Extending RoboTok with object-relative and contact cues alongside actor-centered hand trajectories is a natural direction we have added to future work in Sec. 6 (p. 10), highlighted in blue.

Part 2/3: Independent Relevance Validation and Task/Reward Setup (W1)

W1. Moreover, the main retrieval evaluation uses exact DTW over the same hand representation as its ground truth ... the main modified tasks were explicitly selected for meaningful 3D hand motion and the policy guidance itself is defined through robot hand-state proximity, making the current evaluation (Table 2) particularly favorable to hand-kinematic relevance.

1. Validating manipulation relevance independently of DTW (W1). Regarding DTW as ground truth, we agree that Tab. 2 measures DTW distillation fidelity (Sec. 5.1). To strengthen independent validation of manipulation relevance:

  • Expanded multi-query and Exact DTW evaluation (R4): We expanded downstream simulation evaluation to 3 queries per task and added an Exact DTW baseline, where RoboTok reaches within 2.1 points of Exact DTW (49.9% vs. 52.0%). Added to Sec. 5.2 (Tab. 3, pp. 8-9) and App. E.5 (Tab. 19, pp. 27-28), highlighted in blue.
  • DTW-to-success correlation (R3): Correlating retrieval DTW cost with simulation success across all six methods shows that lower DTW cost strongly predicts higher task success (Spearman ρ=−0.94\rho = -0.94, p=0.017p = 0.017). Added as a plotted graph to Sec. 5.2 (Fig. 6, pp. 9-10) and App. E.6 (Tab. 20, p. 28), highlighted in blue.

R4. Simulation success rate (%) across three queries per task.

TasksSplitBaseRandomFlowHANDSTRAPRoboTokExact DTW
BottleCap TurningSeen0.1±0.11.8±0.51.9±1.054.3±5.258.8±4.275.4±2.378.9±3.2
BottleCap TurningUnseen0.0±0.00.7±0.30.4±0.426.6±3.927.1±3.036.8±3.840.3±4.3
Faucet ScrewingSeen1.0±0.62.3±1.61.3±0.63.9±1.74.1±1.842.7±2.746.6±3.5
Faucet ScrewingUnseen1.7±1.71.0±0.50.3±0.21.5±0.71.8±1.19.7±2.410.2±1.7
Lever SlidingSeen0.8±0.313.9±3.013.4±3.122.4±4.015.8±3.377.6±3.281.4±2.7
Lever SlidingUnseen2.1±2.112.8±4.613.2±5.322.7±6.714.1±3.457.5±4.554.4±4.8

R3. Retrieval DTW cost@10 vs. mean downstream simulation success (3 queries × 8 seeds across 6 task/split settings). Spearman ρ=−0.94\rho = -0.94, p=0.017p = 0.017 between DTW cost and success rate.

MethodDTW cost@10 (m) ↓Cost rankMean sim success (%) ↑Success rank
Exact DTW1.041152.01
RoboTok1.197249.92
STRAP3.903320.34
HAND4.478421.93
Random4.73755.45
Flow6.70865.16

2. Task selection and hand-state guidance (W1).

  • Unmodified VTDexManip tasks: On all six original VTDexManip tasks (where the hand base is fixed), RoboTok still leads all retrieval methods in 11 of 12 task/split settings (App. E.4, Tab. 15, now Tab. 18).
  • Controlled RL and shaping comparison: Guiding RL policies via robot hand-state proximity to retargeted video hand poses alongside an outcome-based task reward follows established practice in learning dexterous manipulation from human video (Mandikal & Grauman, 2021). All evaluated retrieval methods share the exact same retargeting pipeline, hand-state shaping reward, and RL setup, differing only in the retrieved demonstrations, while test-time evaluation is judged strictly by physical object outcomes with no demonstrations.

References:

P. Mandikal & K. Grauman, DexVIP: Learning dexterous grasping with human hand pose priors from video, CoRL 2021.

Part 3/3: Static-Camera Filtering and Camera Motion Sensitivity (W2, Q2)

W2 / Q2. The dependence on near-static-camera videos is not yet well characterized. ... whether the filtering criterion is mainly a conservative choice for obtaining reliable reconstructions, or whether moderate camera motion leads to failures in the resulting actor-centered trajectories or retrieval quality.

We appreciate you raising this point and question. The near-static-camera filter is a conservative choice in our current extraction pipeline rather than a tuned failure boundary or a requirement of RoboTok's encoder and retrieval engine.

  • 1. Conservative static-camera threshold and corpus coverage. Our 1-pixel translation and 0.5% zoom cutoffs were chosen as conservative priors for reliable 3D kinematics rather than tuned failure limits. Even with this filter, static-camera instructional video remains abundant on the web, where cooking, crafts, and mechanical work comprise roughly 30%, 10%, and 10% of our corpus.
  • 2. Sensitivity to camera motion and relaxing the assumption. Crucially, the RoboTok torso estimator, encoder, and retrieval engine are agnostic to how 3D hand trajectories are obtained. Because our current hand-pose extraction pipeline estimates one static camera-to-torso transform and one median focal length per clip, uncompensated camera translation and zoom mix with camera-frame hand motion and depth scaling. Relaxing this filter requires world-coordinate hand tracking to subtract camera motion before torso canonicalization, which we intend to explore in future work and have added to our future work discussion in Sec. 6 (p. 10), highlighted in blue.

Thank you again for your thoughtful review and positive support of our work, and please let us know if you have any follow-up questions that we can clarify. We hope these clarifications and new evaluations fully address your questions, and we would be grateful if you would consider raising your score to 4 (accept) in light of these results.

Reviewer_dVGSOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents RoboTok, a scalable data engine for retrieving manipulation-relevant human demonstrations from internet videos for dexterous robot learning. The method first extracts 3D hand trajectories from unconstrained videos using a combination of WiLoR, MoGe-2, and HaWoR. It then estimates an actor-centered torso reference frame from wrist motion alone, allowing hand trajectories to be compared across different camera viewpoints and scene appearances. RoboTok defines manipulation similarity using Dynamic Time Warping (DTW) over the canonicalized 3D hand trajectories.

Critical strengths
  • The paper addresses an important and timely data bottleneck in robot learning. Collecting large-scale dexterous robot demonstrations is expensive, while internet videos provide a potentially much broader source of human manipulation behaviors. The proposed web-video-to-robot-data pipeline is practically relevant and has a clear end-to-end motivation.

  • The representation and retrieval design are well aligned with the target problem. Canonicalizing metric 3D hand trajectories into an actor-centered reference frame is a sensible way to reduce sensitivity to camera viewpoint and scene appearance. Distilling DTW neighborhoods into an indexable motion embedding is also technically reasonable and avoids the high query cost of exhaustive sequence alignment.

  • The experimental evaluation is relatively comprehensive and includes downstream validation. In addition to retrieval metrics, the paper evaluates cross-dataset transfer on AssemblyHands, simulated dexterous manipulation with seen and unseen objects, and real-world deployment on a LEAP hand. The consistent improvements across retrieval, simulation, and real-world experiments make the proposed system more convincing than a retrieval-only study.

Critical weaknesses
  • The retrieval evaluation is largely tied to the authors’ self-defined DTW objective. Exact DTW over the estimated actor-centered hand trajectories is used as the retrieval ground truth, while RoboTok is explicitly trained to distill this DTW ranking. Therefore, the retrieval results primarily demonstrate that RoboTok approximates DTW efficiently, but they do not independently establish that DTW similarity corresponds to task relevance, object interaction, contact behavior, or human-perceived manipulation similarity. Additional validation using human judgments, object/contact information, or correlations with downstream task success would strengthen the central claim.

  • The paper lacks several important ablations and reference baselines. In particular, the authors do not report the downstream performance of demonstrations retrieved by the exact DTW oracle, even though exact DTW is treated as the retrieval upper bound. It is also unclear how much of the gain comes from the torso-frame canonicalization, 3D rather than 2D motion, the trajectory completion procedure, or the set/ranking losses. Comparisons with simpler 3D trajectory encoders, direct trajectory descriptors with approximate nearest-neighbor search, and motion-semantic hybrid retrieval methods would help isolate the contribution of RoboTok itself.

  • The scalability and generalization claims are not fully validated. The internet-video experiments use a random clip-level split, which may allow visually or temporally related clips from the same source video to appear in both training and evaluation. Moreover, the reported retrieval runtime excludes the substantial cost of hand detection, metric depth estimation, trajectory completion, and torso-frame prediction, and the actual experiments use exhaustive vector scanning rather than a measured sublinear index. The downstream evaluation also relies on one manually selected query per task and only three real-world tasks, so broader source-level splits, multiple queries, and more diverse manipulation scenarios would make the conclusions more robust.

Questions for authors

See weakness.

6 comments

Thank you for the careful review, for highlighting the strengths of our formulation, representation, and experimental evaluation, and for providing constructive feedback. We address each point in detail below.

Part 1/4: Manipulation Relevance and Downstream Success (W1)

W1. The retrieval evaluation is largely tied to the authors' self-defined DTW objective. ... they do not independently establish that DTW similarity corresponds to task relevance, object interaction, contact behavior, or human-perceived manipulation similarity. Additional validation using ... correlations with downstream task success would strengthen the central claim.

Thank you for raising this important point. We agree that Tab. 2 measures DTW distillation fidelity rather than semantic manipulation relevance on its own. We therefore validate that DTW similarity tracks manipulation relevance independently via downstream simulation policy success across all six retrieval methods.

1. DTW distillation fidelity vs. semantic annotations.

  • Distillation evaluation: Tab. 2 evaluates how faithfully RoboTok approximates Exact DTW (Sec. 5.1), following the standard protocol for learned trajectory-similarity embeddings (Yao et al., 2019).
  • Why DTW over coarse text labels: For dexterous manipulation, independent semantic annotations or text labels are often too coarse to capture continuous 3D hand poses and finger articulation throughout a trajectory. Often, clips sharing a semantic label can involve very different grasps and wrist motions, while semantically distinct tasks can share similar hand kinematics. This was a core motivation behind this work and the key reason we chose DTW to measure similarity between human demonstrations.

2. Validating manipulation relevance via downstream task success.

  • Correlation with policy success: Following your suggestion and prior manipulation retrieval works (FlowRetrieval, STRAP, and HAND), we correlated retrieval DTW cost with downstream simulation success across all six methods, including our new Exact DTW baseline (R3 and R4 tables below, added to Sec. 5.2, Fig. 6, pp. 9-10, and App. E.6, Tab. 20, p. 28). Lower DTW cost strongly correlates with higher task success (Spearman ρ=−0.94\rho = -0.94, p=0.017p = 0.017).
  • Role of hand kinematics vs. object/contact cues: Because unconstrained web videos lack object-pose or contact annotations, RoboTok uses hand kinematics to guide coarse motion while RL discovers fine contacts on the task reward. Incorporating object-relative and contact-aware cues for finer physical interactions is an important direction that we have added to future work in Sec. 6 (p. 10), highlighted in blue.

R3. Retrieval DTW cost@10 vs. mean downstream simulation success (3 queries × 8 seeds across 6 task/split settings). Spearman ρ=−0.94\rho = -0.94, p=0.017p = 0.017 between DTW cost and success rate.

MethodDTW cost@10 (m) ↓Cost rankMean sim success (%) ↑Success rank
Exact DTW1.041152.01
RoboTok1.197249.92
STRAP3.903320.34
HAND4.478421.93
Random4.73755.45
Flow6.70865.16

References:

D. Yao et al., Computing trajectory similarity in linear time: A generic seed-guided neural metric learning approach, IEEE ICDE 2019.

Part 2/4: Exact DTW Downstream Evaluation and Representation Ablations (W2)

W2. The paper lacks several important ablations and reference baselines. In particular, the authors do not report the downstream performance of demonstrations retrieved by the exact DTW oracle ... It is also unclear how much of the gain comes from the torso-frame canonicalization, 3D rather than 2D motion, the trajectory completion procedure, or the set/ranking losses.

We appreciate this thorough feedback. Per your request, we conducted additional experiments evaluating Exact DTW downstream and ablating every major design choice of RoboTok.

1. Exact DTW downstream evaluation. Following our expanded three-query, 8-seed simulation protocol (R4), Exact DTW achieves the highest overall simulation success (52.0% mean across all six task/split settings), and RoboTok is the closest retrieval method at 49.9% (within 2.1 points of Exact DTW), whereas HAND (21.9%) and STRAP (20.3%) perform less than half as well. Added to Sec. 5.2 (Tab. 3, pp. 8-9) and App. E.3 (Fig. 10, now Fig. 11, and Tab. 14, now Tab. 17, p. 27), highlighted in blue.

R4. Simulation success rate (%) across three queries per task.

TasksSplitBaseRandomFlowHANDSTRAPRoboTokExact DTW
BottleCap TurningSeen0.1±0.11.8±0.51.9±1.054.3±5.258.8±4.275.4±2.378.9±3.2
BottleCap TurningUnseen0.0±0.00.7±0.30.4±0.426.6±3.927.1±3.036.8±3.840.3±4.3
Faucet ScrewingSeen1.0±0.62.3±1.61.3±0.63.9±1.74.1±1.842.7±2.746.6±3.5
Faucet ScrewingUnseen1.7±1.71.0±0.50.3±0.21.5±0.71.8±1.19.7±2.410.2±1.7
Lever SlidingSeen0.8±0.313.9±3.013.4±3.122.4±4.015.8±3.377.6±3.281.4±2.7
Lever SlidingUnseen2.1±2.112.8±4.613.2±5.322.7±6.714.1±3.457.5±4.554.4±4.8

2. Hand trajectory representation ablations. Keeping Exact DTW fixed on the source-video-disjoint split (R1), we ablated torso-frame canonicalization, 3D vs. 2D, and HaWoR infilling:

  • Torso-frame canonicalization and 3D lifting: Camera-centered 3D and 2D image coordinates recover almost none of the actor-centered neighborhoods (mAP@10 of 0.015 and 0.016), showing that 3D lifting alone helps little without torso-frame canonicalization because unaligned trajectories group clips by camera viewpoint rather than body-relative motion. This aligns with downstream RL task success (R4), where 2D camera-space hand-trajectory DTW (HAND) reaches 21.9% versus 49.9% for RoboTok.
  • Trajectory infilling: Removing HaWoR trajectory infilling retains most of the complete structure (mAP@10 0.728).
  • Paper updates: Added to Sec. 5.1 (p. 7) and App. D.5 (Tab. 15, pp. 24-25), highlighted in blue.

R1. Exact DTW retrieval across hand trajectory representations at k=10k=10 (N=5,000N=5,000).

RepresentationmAP@10 ↑nDCG@10 ↑DTW cost@10 (m) ↓
Camera-centered 3D0.0150.0422.801
2D image coordinates0.0160.0443.145
Actor-centered 3D without infilling0.7280.8531.047
Actor-centered 3D (reference)1.0001.0001.009

Part 3/4: Training Objective Ablations, Direct ANN, and Hybrid Retrieval (W2)

W2. It is also unclear how much of the gain comes from ... the set/ranking losses. Comparisons with simpler 3D trajectory encoders, direct trajectory descriptors with approximate nearest-neighbor search, and motion-semantic hybrid retrieval methods would help isolate the contribution of RoboTok itself.

3. Embedding objective and batch-sampling ablations. Keeping the actor-centered 3D input and encoder fixed (R2):

  • Loss comparison: With neighbor-mined batches, top-kk objectives (set + rank, set-only, rank-only, triplet) reach 0.500-0.533 nDCG@10, with set + rank best at every k∈[10,100]k \in [10, 100], whereas the soft contrastive objective drops to 0.397.
  • Batch sampling: Replacing mined batches with random batches drops nDCG@10 by 0.14-0.18 across all losses.
  • Paper updates: Added to Sec. 5.1 (p. 7) and App. D.5 (Tab. 16, pp. 24-25), highlighted in blue.

4. Simpler 3D encoders, direct ANN descriptors, and motion-semantic hybrids.

  • Why direct fixed-vector ANN cannot replace distillation: Because action trajectories vary in duration and execution speed, fixed-vector ANN (such as HNSW) cannot align unwarped trajectory descriptors directly, whereas exact DTW handles speed variation but cannot be indexed in HNSW.
  • Minimal encoder design: RoboTok therefore uses a minimal single cross-attention block (App. B.1) to distill DTW into an HNSW-indexable vector space.
  • Motion-semantic hybrid baselines: HAND already serves as a visual-semantic + motion hybrid baseline (DINOv2 pre-filtering followed by wrist-track DTW), and combining 3D actor-centered motion with object-relative and contact-aware cues is a promising direction that we have highlighted in future work (Sec. 6, p. 10).

R2. Embedding loss and batch-sampling ablation on the source-video-disjoint split (Neval=10,000N_{\mathrm{eval}}=10{,}000). Only the loss or the batch sampler changes between rows. Bold indicates the best among trained models.

LossBatchesnDCG@10 ↑nDCG@20 ↑nDCG@50 ↑nDCG@100 ↑
Set + rank (RoboTok)mined0.5330.5840.6420.676
Set onlymined0.5060.5590.6230.663
Rank onlymined0.5320.5790.6290.657
Tripletmined0.5000.5560.6230.670
Soft contrastivemined0.3970.4540.5310.586
Set + rankrandom0.3700.4260.5030.561
Set onlyrandom0.3630.4180.4940.552
Rank onlyrandom0.3500.4100.4900.549
Exact DTW (reference)-1.0001.0001.0001.000
Reviewer_DPYfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RoboTok, a framework for retrieving manipulation-relevant human demonstrations from internet videos. It represents hand trajectories in an actor-centered 3D coordinate frame and learns compact embeddings that approximate Dynamic Time Warping (DTW) similarity for efficient retrieval. Experiments demonstrate improved retrieval performance and downstream dexterous manipulation success in simulation and real-world settings.

Critical strengths

S1. The paper addresses an important problem in robot learning by proposing a scalable way to retrieve relevant human demonstrations from internet videos. The actor-centered 3D hand representation is intuitive and well motivated.

S2. The paper provides extensive retrieval comparisons and encouraging downstream results in both simulation and real-world experiments, demonstrating the practical potential of the proposed framework.

Critical weaknesses

W1. Insufficient ablations and potentially biased retrieval evaluation.

The retrieval ground truth is defined using the same DTW similarity measure used to train RoboTok, which naturally favors the proposed approach. Furthermore, the paper lacks key ablations to isolate the contributions of actor-centered canonicalization, 3D hand representation, and DTW-based embedding learning. Consequently, the source of the performance improvements remains unclear.

W2. Limited evidence of downstream generalization.

The main downstream evaluation covers only three modified manipulation tasks, each using a single manually selected query demonstration. The real-world evaluation is also limited to three objects and ten trials per object for each task. Additionally, the random clip-level train/test split does not establish whether source videos are disjoint. These limitations raise concerns about the robustness and generalizability of the reported improvements.

Questions for authors
  1. Can the authors provide ablations comparing camera-centered versus actor-centered representations, 2D versus 3D trajectories, and alternative embedding objectives?

  2. Can retrieval quality be evaluated using independently annotated manipulation relevance rather than the DTW metric used during training?

  3. How sensitive are the downstream results to query demonstration selection and training seeds? Are the retrieval train/test splits source-video-disjoint?

5 comments

Thank you for taking the time to review our work, for recognizing its practical potential and intuitive representation, and for your constructive feedback. We address each of your comments and questions below.

Part 1/4: Ablation Studies (W1, Q1)

W1 / Q1. Can the authors provide ablations comparing camera-centered versus actor-centered ... 2D versus 3D ... alternative embedding objectives?

Thank you for suggesting these ablations. Our new experiments show that camera-centered 3D and 2D inputs recover almost none of the actor-centered neighborhoods, and Set + rank with neighbor-mined batches outperforms every tested objective.

1. Hand trajectory representation ablations. Following the internet video retrieval setup in the paper on our source-video-disjoint split, we keep exact DTW alignment fixed and vary only the input hand trajectory representation. Our full Actor-centered 3D representation transforms metric 3D hand poses into the person's torso frame and infills briefly occluded frames with HaWoR. We compare it against Camera-centered 3D hand poses in camera coordinates, 2D image coordinates of hand keypoints on the video frame, and Actor-centered 3D without infilling, which keeps torso-frame 3D poses from directly detected frames only. Against the full actor-centered 3D reference, we measure top-10 neighbor overlap (mAP@10), ranking agreement (nDCG@10), and mean body-relative 3D motion distance in meters (DTW cost@10).

R1. Exact DTW retrieval across hand trajectory representations at k=10k=10 (N=5,000N=5{,}000).

RepresentationmAP@10 ↑nDCG@10 ↑DTW cost@10 (m) ↓
Camera-centered 3D0.0150.0422.801
2D image coordinates0.0160.0443.145
Actor-centered 3D without infilling0.7280.8531.047
Actor-centered 3D (reference)1.0001.0001.009
  • 2D image coordinates (mAP@10 0.016, 3.145 m): Projecting hands onto the image plane discards depth and viewing-axis motion, so distinct 3D actions look alike and retrieved clips are 3.1× farther in body-relative motion (3.15 m vs. 1.01 m).
  • Camera-centered 3D (mAP@10 0.015, 2.801 m): Lifting to 3D preserves depth, but poses remain camera-relative, so the same motion from two angles yields different trajectories (Sec. 4.1). Retrieval then tracks viewpoint rather than body-relative motion, returning clips 2.8× farther (2.80 m vs. 1.01 m).
  • Actor-centered 3D without infilling (mAP@10 0.728, nDCG@10 0.853, 1.047 m): Canonicalized trajectories without infilling already recover most of the neighborhood structure when hands are visible, while HaWoR infilling provides additional robustness in unconstrained web videos where manipulated objects frequently occlude the hands.

Because R1 is scored against the actor-centered reference, it shows how strongly retrieved neighborhoods change rather than manipulation utility on its own. Downstream object-state policy success speaks to utility and is consistent with this gap. The 2D-motion baselines HAND and Flow reach 21.9% and 5.1% success rates vs. 49.9% for RoboTok (R4, in our W2 response).

2. Embedding objective and batch-sampling ablations. Following the encoder training and evaluation setup in the paper on the source-video-disjoint split, we keep the actor-centered 3D input and encoder architecture fixed while varying the loss function and batch sampler. For the loss function, we compare our combined Set + rank objective against Set only, Rank only, pairwise Triplet, and Soft contrastive losses. For batch sampling, we compare neighbor-mined batches against uniform random batches (b=210b=210). We evaluate retrieval quality across neighborhood cutoffs by top-k ranking agreement (nDCG@kk) with exact DTW.

R2. Embedding loss and batch-sampling ablation on the source-video-disjoint split (Neval=10,000N_{\mathrm{eval}}=10{,}000). Only the loss or the batch sampler changes between rows. Bold indicates the best among trained models.

LossBatchesnDCG@10 ↑nDCG@20 ↑nDCG@50 ↑nDCG@100 ↑
Set + rank (RoboTok)mined0.5330.5840.6420.676
Set onlymined0.5060.5590.6230.663
Rank onlymined0.5320.5790.6290.657
Tripletmined0.5000.5560.6230.670
Soft contrastivemined0.3970.4540.5310.586
Set + rankrandom0.3700.4260.5030.561
Set onlyrandom0.3630.4180.4940.552
Rank onlyrandom0.3500.4100.4900.549
Exact DTW (reference)-1.0001.0001.0001.000

Set + rank is best at every cutoff, and neighbor mining matters more than the choice among top-k losses.

Added in blue to Sec. 5.1 (p. 7) and App. D.5 (Tabs. 15-16, pp. 24-25) of the revised manuscript.

Part 2/4: Evaluating Retrieval Relevance Independently of DTW (W1, Q2)

W1 / Q2. The retrieval ground truth is defined using the same DTW similarity measure used to train RoboTok ... Can retrieval quality be evaluated using independently annotated manipulation relevance rather than the DTW metric used during training?

Thank you for raising this point. We agree that Tab. 2 scores methods against exact DTW over actor-centered 3D hand trajectories, which is the teacher signal that RoboTok distills. To clarify, as stated in Sec. 5.1, Tab. 2 evaluates exact-DTW distillation fidelity, which enables neighborhood recovery in O(dlog⁡N)O(d \log N) search time without O(NL2)O(NL^2) query-time alignment (standard protocol for learned trajectory-similarity embeddings, Yao et al., 2019), rather than an independent notion of task relevance. For dexterous manipulation, we found that semantic or text captions on web video are too coarse to capture continuous 3D hand poses and finger articulation throughout a trajectory: clips sharing a semantic label can involve very different hand motions, while semantically distinct tasks can share similar hand kinematics. This motivated our use of DTW over hand kinematics for retrieval.

Therefore, following prior manipulation retrieval works (FlowRetrieval, STRAP, and HAND), we evaluate retrieval quality independently of the training metric through downstream policy success. Following the simulation training and evaluation setup in the paper (3 distinct human query clips × 8 RL seeds per task), we keep the policy training pipeline fixed and change only the method used to retrieve the top-10 examples. Across three manipulation tasks on seen and unseen objects, test-time policy success is measured strictly by physical object state without demonstrations.

R3. Held-out top-10 retrieval DTW cost (N=10,000N=10{,}000 queries) vs. mean downstream simulation policy success (3 queries × 8 seeds across 6 task and object-split settings). Spearman ρ=−0.94\rho = -0.94, p=0.017p = 0.017 between DTW cost and success rate.

MethodDTW cost@10 (m) ↓Cost rankMean sim success (%) ↑Success rank
Exact DTW1.041152.01
RoboTok1.197249.92
STRAP3.903320.34
HAND4.478421.93
Random4.73755.45
Flow6.70865.16

Among the baseline retrieval methods, RoboTok achieves the lowest DTW cost and the highest policy mean success rate at 49.9%, coming within 2.1 percentage points of Exact DTW (52.0%) and more than doubling HAND (21.9%) and STRAP (20.3%). Prior methods fall short because STRAP matches scene appearance instead of 3D hand motion, while HAND and Flow rely on 2D image-plane motion without depth or viewpoint invariance. Across all six methods, lower DTW cost for retrieved examples correlates strongly with higher policy success (Spearman ρ=−0.94\rho = -0.94, p=0.017p = 0.017 between DTW cost and success rate). Downstream policy success thus provides an independent validation of retrieval quality and confirms that RoboTok retrieves manipulation-relevant demonstrations for policy learning. We have added the expanded policy evaluation in blue to Sec. 5.2 (Tab. 3, pp. 8-9), App. E.3 (Fig. 11 and Tab. 17, p. 27), and App. E.5 (Tab. 19, pp. 27-28), and the DTW-to-success correlation analysis to Sec. 5.2 (Fig. 6, pp. 9-10) and App. E.6 (Tab. 20, p. 28).

References:

D. Yao et al., Computing trajectory similarity in linear time: A generic seed-guided neural metric learning approach, IEEE ICDE 2019.

Part 3/4: Downstream Breadth and Query/Seed Sensitivity (W2, Q3)

W2 / Q3. Limited evidence of downstream generalization. The main downstream evaluation covers only three modified manipulation tasks, each using a single manually selected query demonstration. The real-world evaluation is also limited to three objects ... How sensitive are the downstream results to query demonstration selection and training seeds?

Thank you for raising these questions and feedback. Across 270 real-world trials, all six original VTDexManip tasks, and an expanded multi-query simulation evaluation with 8 RL seeds, RoboTok consistently outperforms all retrieval baselines and closely tracks Exact DTW.

1. Downstream evaluation breadth (W2).

  • Real-world breadth: Our real-world evaluation (Fig. 6, now Fig. 7) spans 270 physical trials across three tasks, three object geometries per task, and 10 trials per object per method, where RoboTok outperformed the strongest baseline per task by 42.2 percentage points on average.
  • Simulation task breadth: Beyond the three modified simulation tasks in Tab. 3, Appendix Tab. 15 (now Tab. 18) evaluates all six original VTDexManip tasks (12 task/split settings), where RoboTok leads all retrieval methods in 11 of 12 settings.

2. Sensitivity to query selection and training seeds (W2, Q3).

  • Training seeds: Tab. 3 already trains 8 RL seeds per method (100 rollouts/object/seed).
  • Expanded multi-query and Exact DTW evaluation: Following your suggestion, we expanded the downstream simulation evaluation to three distinct human query clips per task and added an Exact DTW baseline (R4 summary; per-query results are in Tab. 19). RoboTok remains the best retrieval method across all six task/split settings (49.9% mean success), closely behind Exact DTW (52.0%) and more than doubling HAND (21.9%) and STRAP (20.3%). Across individual queries (Tab. 19), RoboTok achieves the highest success among baseline retrieval methods in all query/task/split combinations.
  • Paper updates: Added the 3-query summary with Exact DTW (R4) to Sec. 5.2 (Tab. 3, pp. 8-9) and the per-query results to App. E.5 (Tab. 19, pp. 27-28), highlighted in blue.

R4. Simulation success rate (%) across three queries per task.

TasksSplitBaseRandomFlowHANDSTRAPRoboTokExact DTW
BottleCap TurningSeen0.1±0.11.8±0.51.9±1.054.3±5.258.8±4.275.4±2.378.9±3.2
BottleCap TurningUnseen0.0±0.00.7±0.30.4±0.426.6±3.927.1±3.036.8±3.840.3±4.3
Faucet ScrewingSeen1.0±0.62.3±1.61.3±0.63.9±1.74.1±1.842.7±2.746.6±3.5
Faucet ScrewingUnseen1.7±1.71.0±0.50.3±0.21.5±0.71.8±1.19.7±2.410.2±1.7
Lever SlidingSeen0.8±0.313.9±3.013.4±3.122.4±4.015.8±3.377.6±3.281.4±2.7
Lever SlidingUnseen2.1±2.112.8±4.613.2±5.322.7±6.714.1±3.457.5±4.554.4±4.8

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We sincerely thank all reviewers (DPYf, dVGS, and 9WEE) for their time, thoughtful comments, and constructive suggestions, which have helped us significantly strengthen our paper. We are grateful that all reviewers recognized the core strengths of our work.

  • Addressing an important data bottleneck in robot learning by formulating unconstrained internet video as a queryable source of human demonstrations.
  • Proposing an intuitive and well-motivated actor-centered 3D hand trajectory representation and DTW-distilled motion embedding for scalable nearest-neighbor search.
  • Providing comprehensive evaluations across held-out internet video retrieval, cross-dataset transfer on AssemblyHands, simulated dexterous manipulation, and real-world deployment.

We have uploaded a revised version of our manuscript with all changes highlighted in blue to help reviewers and Area Chairs easily identify the updates. These highlights will be removed in the camera-ready version. Below, we summarize the new experiments and revisions incorporated into the updated manuscript.

  1. Validating manipulation relevance independently of DTW (DPYf, dVGS, 9WEE). Correlated retrieval DTW cost@10 with downstream simulation success across all six methods, confirming that lower DTW cost strongly tracks higher task success with Spearman ρ=−0.94\rho = -0.94, p=0.017p = 0.017 (Sec. 5.2, Fig. 6, and App. E.6, Tab. 20).
  2. Multi-query downstream robustness and Exact DTW upper bound (DPYf, dVGS, 9WEE). Expanded downstream simulation to three human query clips per task (8 RL seeds each) and added an Exact DTW baseline, where RoboTok reaches within 2.1 points of Exact DTW (49.9% vs. 52.0%) and leads all retrieval baselines on every query (Sec. 5.2, Tab. 3, and App. E.3 and E.5).
  3. Isolating representation and training design contributions (DPYf, dVGS). Added ablations over trajectory representations (camera-centered 3D, 2D image coordinates, and without HaWoR infilling) and training designs (five loss objectives under mined vs. random batch sampling) on the video-disjoint split (Sec. 5.1 and App. D.5, Tabs. 15-16).
  4. Preventing source-video leakage in retrieval evaluation (DPYf, dVGS). Rebuilt the 90k/10k Internet Video split to be strictly source-video-disjoint, retrained RoboTok, and re-evaluated all baselines, with all retrieval metrics changing by at most 0.003 (Sec. 5.1, Tab. 2, and App. D.1-D.2).
  5. Offline preprocessing cost and sublinear indexing scalability (dVGS). Added a stage-by-stage offline extraction runtime breakdown and evaluated FAISS HNSW indexing, which matches exhaustive search at k=10k=10 while reducing query latency from 0.29 ms to 0.036 ms (App. B.2, Tabs. 4 and 6).
  6. Scope of hand-kinematic similarity and static-camera filtering (9WEE, dVGS). Clarified the scope of actor-centered hand kinematics and near-static-camera filtering, and added future work directions on extending beyond static-camera footage to dynamic third-person and egocentric videos via camera-motion compensation and incorporating object-relative and contact-aware cues alongside torso-centered trajectories for finer physical interactions (Sec. 6).
  7. Prior-work context for DTW distillation and hand-pose RL guidance (DPYf, dVGS, 9WEE). Added citations to Yao et al. (2019) on evaluating learned trajectory-similarity embeddings against exact DTW and to Mandikal & Grauman (2021) on guiding dexterous RL policies via hand-state proximity alongside task rewards (Sec. 5.1 and Sec. 5.2).

We deeply appreciate the insightful review process, and we welcome any further questions or discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.