- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes improving dexterous VLA post-training by selecting human egocentric data that is semantically and kinematically similar to robot demonstrations. To retrieve relevant data, SiMDex uses language and initial hand pose for recall, wrist and fingertip motion in a shared action space for ranking, and optical flow for qualitative verification. The selected data yields higher average task scores than equally sized randomly sampled human data across three real-world manipulation tasks and multiple robot-demonstration budgets.
Critical strengths
-
The paper addresses a practical problem: identifying which human demonstrations are useful for a target robot task. Its retrieval pipeline combines semantic and motion similarity without changing the policy architecture, providing a practical way to reuse existing egocentric datasets for VLA post-training.
-
The controlled real-robot experiments provide convincing evidence for targeted data selection. With the model, data volume, and training settings matched, SiMDex improves the mean normalized task score from 47.7% to 61.7% and maintains average gains across multiple robot-data budgets, directly supporting the paper’s central claim.
Critical weaknesses
-
The paper reports approximately 1.35 million robot samples and 1.49 million mined human samples, but does not provide per-task anchor counts, the number of samples retrieved for each task, or the overlap among the retrieved subsets. These aggregate counts therefore do not reveal each task’s contribution to the training data, making the task-dependent performance differences harder to interpret. For example, a per-task breakdown would help assess whether differences in the amount and coverage of retrieved data contribute to the weaker results on Drill, alongside the failure modes already discussed.
-
The experimental coverage and ablation study are limited. The evaluation covers three tasks within a single application scenario, while the data-scaling ablation varies only the robot-demonstration budget and keeps the human-data subset fixed. Since human-data selection is the central contribution, varying the amount of retrieved human data, with equally sized random subsets as baselines, would help clarify how much selected data is needed and whether the gains persist across different human-data budgets.
Questions for authors
- Regarding W1, could the authors provide per-task anchor counts, retrieved sample counts, and overlap statistics among the retrieved subsets, and analyze whether differences in data amount or coverage may contribute to the weaker results on Drill?
- Regarding W2, how does performance change when varying the amount of retrieved human data (and comparisons with equally sized random subsets)?
- I am curious how a policy trained with task-specific mined human data performs on unrelated tasks. I understand that achieving strong performance on target tasks is valuable in its own right, but broader generalization remains an important goal. Compared with using an equally sized random human-data subset, do the improvements on the target tasks come at the expense of performance on unrelated tasks?
I would be willing to raise my rating if the authors address my concerns.
5 comments
Response to Reviewer RmSw (R3), part 2/3
Amount of human data.
We agree that equal-size random baselines are the right test of human-data selection, and we made this comparison the main additional experiment in our response to you. The added condition is a fixed random subset of 1.49M samples, the same size as the mined set. To keep it directly comparable:
- Training: at all four robot-data budgets, with the same model, training steps and 1:1 mixture as SiMDex.
- Evaluation: the same rubric and layouts as Table 1 and the scaling study (Figure 4), with 25 real-robot trials per task (5 per budget as in Figure 4, but 10 at 1× as in Table 1), which adds 75 real-robot trials (25 per task × 3 tasks) to those reported in the paper.
With GR-Dexter and SiMDex, this gives three human-data conditions at four budgets:
Normalized progress score (%) per task and robot-data budget; 5 trials per cell, 10 at 1× (two rounds, as in Table 1; Figure 4 and Table 8 show the first round only). Overall is the mean of the three task scores shown (Table 8 prints 40.9 for GR-Dexter at 0.5×, averaged before rounding).
| Human data | Task | 0.25× | 0.5× | 1× | 2× |
|---|---|---|---|---|---|
| Random, fixed 1.49M subset (added) | Drill | 8.8 | 15.5 | 31.1 | 35.5 |
| Flick Wheel | 22.2 | 20.0 | 32.2 | 37.8 | |
| Pick & Place | 44.9 | 54.9 | 50.7 | 71.6 | |
| Overall | 25.3 | 30.1 | 38.0 | 48.3 | |
| Random, full ~32M pool (GR-Dexter) | Drill | 26.6 | 35.5 | 64.5 | 68.8 |
| Flick Wheel | 16.7 | 35.5 | 24.5 | 22.2 | |
| Pick & Place | 45.0 | 51.5 | 54.0 | 61.6 | |
| Overall | 29.4 | 40.8 | 47.7 | 50.9 | |
| SiMDex (1.49M) | Drill | 35.5 | 48.9 | 54.5 | 62.2 |
| Flick Wheel | 30.0 | 48.9 | 47.2 | 32.2 | |
| Pick & Place | 43.2 | 76.7 | 83.3 | 76.7 | |
| Overall | 36.2 | 58.2 | 61.7 | 57.0 |
The 1× column of GR-Dexter and SiMDex is Table 1. Pooled tests below use 60 layout-matched pairs (5 layouts × 3 tasks × 4 budgets), with the two 1× rounds averaged per layout so that every budget counts equally.
Each difference is the mean over matched pairs (20 per task), in score points from unrounded trial scores; p is from the same paired test as in Q1, and a 95% CI entirely above zero means SiMDex is reliably higher.
- Against the equal-size subset. SiMDex is higher at every budget and in 10 of 12 task–budget cells. Pooled: +17.8 (p < 0.0001, 95% CI [+11.8, +24.1]); per task, +27.5 on Drill (p < 0.001), +11.5 on Flick Wheel (p = 0.03) and +14.5 on Pick & Place (p < 0.01).
- Against GR-Dexter. SiMDex is higher at every budget and in 9 of 12 cells. Pooled: +11.1 (p < 0.01, 95% CI [+3.9, +18.2]); per task, +14.9 on Flick Wheel (p < 0.01) and +17.0 on Pick & Place (p = 0.001); on Drill the two are level (Q1).
- Between the random conditions. The larger pool adds +6.8 (p = 0.09, not significant), driven by Drill (Q1). Within this range, selection matters more than a larger random pool.
Smaller mined sets. You also ask how performance changes when the amount of mined data varies.
Our cascade ablation at 1× gives two smaller mined sets: it mines from a subset of the anchors and keeps the top 50 per anchor, and two of its runs include Ranking (Stage II). Its full design and analysis are in our reply to Reviewer CzZk (R1-W2/Q1). These two runs add 30 real-robot trials (15 each: 3 tasks × 5 layouts):
| Mined set | Samples | Overall |
|---|---|---|
| SiMDex (Table 1) | 1.49M | 61.7 |
| Ranking (Stage II), top 50 per anchor | 1.05M | 58.3 |
| Ranking + Re-ranking (Stage III) as filter | 1.03M | 61.9 |
Compared with SiMDex's 1.49M set, the paired differences are −3.3 (p = 0.69) and +0.2 (p = 0.99), so about 30% fewer mined samples score within 3.4 points. These runs also bear on your question of whether the gains persist across human-data budgets: both smaller mined sets score above the larger 1.49M random subset at 1× (38.0), by +20.3 (p = 0.045) and +23.9 (p = 0.017). The revised limitations will restrict our claims to the amounts reported.
Clarification of App. B.3 and D.4. Both sections call the Table 1 baseline a matched random subset; precisely, GR-Dexter draws at random from the full pool during training, matching SiMDex in the number of human samples seen, not in set size. Table 1 is unaffected. The equal-size comparison your summary describes is the added condition above, and there SiMDex's margin is even larger.
We will revise both sections. Because Q2 goes to the core of our claim, the revision will present this equal-size comparison in the main text and give the full per-task, per-budget score table in the appendix.
Response to Reviewer RmSw (R3), part 3/3
Unrelated tasks. Thank you for noting that strong performance on the target tasks is valuable in its own right. We also agree that broader generalization remains an important goal, and that it matters whether selecting data for the target tasks costs performance elsewhere. We evaluated the policies only on the target tasks, so we cannot say whether the target-task gains come at the expense of unrelated tasks.
- What differs between the policies. Policies trained with SiMDex or with random selection start from the same pre-trained model and differ only in which human samples are added during post-training.
- Within the three evaluated tasks. One policy is post-trained for all three tasks with one mined set, so our trials can show whether gains on some tasks cost another; they show no such loss. On Drill, where no EgoDex task involves drilling (Q1), SiMDex shows no measurable loss against GR-Dexter (50.3 vs. 48.9 averaged over the four budgets; 54.5 vs. 64.5 at 1×, p = 0.51), and it stays above the equal-size random subset (Q1).
- What SiMDex is designed for. SiMDex aims to let a robot learn each new skill from fewer teleoperated demonstrations. As in a recommender, the robot's own demonstrations of a skill act as the query, and SiMDex recommends human data for that skill (see our separate comment on the vision behind SiMDex); a new skill, once demonstrated, can receive its own recommendations in the same way.
- In the revision. The limitations will restrict our claims to performance on the three target tasks within the reported scenario and platform, and will list your question as an open one.
In brief.
- Q1, Drill: the new results point to both amount (only Drill gains from a larger random pool) and coverage (EgoDex has no drilling task); future versions of SiMDex will record per-task counts.
- Q2, amount: SiMDex beats an equal-size random subset at every budget (+17.8 pooled, p < 0.0001), and about 30% fewer mined samples score within 3.4 points.
- Q3, unrelated tasks: within the three tasks we see no trade-off; unrelated tasks remain an open question in the limitations.
We thank you again for engaging so closely with our work. The equal-size comparison you asked for (Q2) now gives the clearest evidence for SiMDex, and the revision will present it in the main text. We hope these answers address your concerns, and we would be happy to discuss any of them further.
Response to Reviewer RmSw (R3), part 1/3
Thank you for the constructive review and for your openness to reconsider. We also appreciate your view that SiMDex addresses a practical problem (S1) and that the controlled real-robot experiments support targeted selection (S2).
Please accept our apologies for the late reply: breaking our real-robot trial records down by task and budget for the results below took longer than we had hoped, as our resources during the discussion period were limited. This comment covers Q1. The second covers Q2, for which we added an equal-size random baseline at all four robot-data budgets, and a clarification of App. B.3/D.4; the third covers Q3.
Q1 / W1. Why Drill is weaker: amount and coverage.
Your underlying question is whether the amount and coverage of retrieved data, beyond the failure modes in App. C.3, explain the weaker Drill result. We answer that question directly with real-robot results for three human-data conditions, each evaluated on all three tasks at all four robot-data budgets.
In short: Drill is the only task that gains from a larger random pool, and the Table 1 baseline (GR-Dexter) draws from the full ~32M pool, which points to amount; EgoDex contains no drilling task, which points to coverage. The evidence bears on both factors but cannot separate them for the mined data.
Amount. We compare three human-data conditions, each mixed 1:1 with the same robot data: SiMDex, GR-Dexter, and an added equal-size random subset (full results in Q2).
Per-task normalized progress score (%), averaged over the four robot-data budgets with equal weight; in parentheses, the 1× budget alone.
How the entries are computed.
What the table shows.
Coverage. We designed the three tasks from an industrial assembly scenario, not from EgoDex's task list, so gaps in coverage are expected: none of the 194 EgoDex tasks involves drilling, and clips mined for the drilling stage show a tube, a saw and a hair dryer (App. C.3).
At 1×, trials that reach the drilling stage score alike on it: SiMDex 0.45 over 6 of 10 trials, GR-Dexter 0.42 over 8 of 10 (averaged over all 10 trials: 0.27 and 0.33, Table 1's Drill entries). Since neither data source contains drilling, this stage is similar for both; the deficit arises earlier, at Grasp and Assembly.
Per-task counts. This is a valuable suggestion, as such statistics make task-level results easier to interpret, and we regret that we cannot give them for the reported run.
SiMDex beats both random references on Flick Wheel and Pick & Place (each p < 0.05; Q2) and is level with GR-Dexter on Drill. The revised limitations will discuss amount and coverage for Drill, and the appendix will add the per-task table.