acceptodds
Under review as a conference paper at ICLR 2027

SiMDex: Mining Similar Egocentric Videos for Cross-Embodiment Dexterous Manipulation

Abstract

Recent years have witnessed an explosive trend of scaling ego-centric human videos for robot manipulation, yet it remains unclear which data actually benefits dexterous manipulation. We present SiMDex, a similarity-based data mining framework that casts human data selection for VLA post-training in dexterous manipulation as a recommendation problem. For each robot demonstration, SiMDex runs a three-stage recall–ranking–re-ranking cascade over  32M egocentric human samples: recall and ranking select a task-relevant subset in a morphology-agnostic action space, and re-ranking verifies it on pixels, with no change to VLA architecture or training. Against a strong baseline trained with an equal amount of randomly sampled human data, SiMDex uses only  1.49M mined samples (<5% of the pool) yet improves the overall success rate from 47.7% to 61.7%—showing that selective curation outperforms indiscriminate data mixing.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot manipulation, Dexterous manipulation, Vision-language-action models, Data curation, Egocentric human video

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RmSwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes improving dexterous VLA post-training by selecting human egocentric data that is semantically and kinematically similar to robot demonstrations. To retrieve relevant data, SiMDex uses language and initial hand pose for recall, wrist and fingertip motion in a shared action space for ranking, and optical flow for qualitative verification. The selected data yields higher average task scores than equally sized randomly sampled human data across three real-world manipulation tasks and multiple robot-demonstration budgets.

Critical strengths
  1. The paper addresses a practical problem: identifying which human demonstrations are useful for a target robot task. Its retrieval pipeline combines semantic and motion similarity without changing the policy architecture, providing a practical way to reuse existing egocentric datasets for VLA post-training.

  2. The controlled real-robot experiments provide convincing evidence for targeted data selection. With the model, data volume, and training settings matched, SiMDex improves the mean normalized task score from 47.7% to 61.7% and maintains average gains across multiple robot-data budgets, directly supporting the paper’s central claim.

Critical weaknesses
  1. The paper reports approximately 1.35 million robot samples and 1.49 million mined human samples, but does not provide per-task anchor counts, the number of samples retrieved for each task, or the overlap among the retrieved subsets. These aggregate counts therefore do not reveal each task’s contribution to the training data, making the task-dependent performance differences harder to interpret. For example, a per-task breakdown would help assess whether differences in the amount and coverage of retrieved data contribute to the weaker results on Drill, alongside the failure modes already discussed.

  2. The experimental coverage and ablation study are limited. The evaluation covers three tasks within a single application scenario, while the data-scaling ablation varies only the robot-demonstration budget and keeps the human-data subset fixed. Since human-data selection is the central contribution, varying the amount of retrieved human data, with equally sized random subsets as baselines, would help clarify how much selected data is needed and whether the gains persist across different human-data budgets.

Questions for authors
  1. Regarding W1, could the authors provide per-task anchor counts, retrieved sample counts, and overlap statistics among the retrieved subsets, and analyze whether differences in data amount or coverage may contribute to the weaker results on Drill?
  2. Regarding W2, how does performance change when varying the amount of retrieved human data (and comparisons with equally sized random subsets)?
  3. I am curious how a policy trained with task-specific mined human data performs on unrelated tasks. I understand that achieving strong performance on target tasks is valuable in its own right, but broader generalization remains an important goal. Compared with using an equally sized random human-data subset, do the improvements on the target tasks come at the expense of performance on unrelated tasks?

I would be willing to raise my rating if the authors address my concerns.

5 comments
AuthorsOpenReview ↗

Response to Reviewer RmSw (R3), part 1/3

Thank you for the constructive review and for your openness to reconsider. We also appreciate your view that SiMDex addresses a practical problem (S1) and that the controlled real-robot experiments support targeted selection (S2).

Please accept our apologies for the late reply: breaking our real-robot trial records down by task and budget for the results below took longer than we had hoped, as our resources during the discussion period were limited. This comment covers Q1. The second covers Q2, for which we added an equal-size random baseline at all four robot-data budgets, and a clarification of App. B.3/D.4; the third covers Q3.

Q1 / W1.\color{blue}{\text{Q1 / W1.}} Why Drill is weaker: amount and coverage.

Your underlying question is whether the amount and coverage of retrieved data, beyond the failure modes in App. C.3, explain the weaker Drill result. We answer that question directly with real-robot results for three human-data conditions, each evaluated on all three tasks at all four robot-data budgets.

In short: Drill is the only task that gains from a larger random pool, and the Table 1 baseline (GR-Dexter) draws from the full ~32M pool, which points to amount; EgoDex contains no drilling task, which points to coverage. The evidence bears on both factors but cannot separate them for the mined data.

Amount. We compare three human-data conditions, each mixed 1:1 with the same robot data: SiMDex, GR-Dexter, and an added equal-size random subset (full results in Q2).

Per-task normalized progress score (%), averaged over the four robot-data budgets with equal weight; in parentheses, the 1× budget alone.

Human dataDrillFlick WheelPick & Place
Random, fixed 1.49M subset (added)22.7 (31.1)28.1 (32.2)55.5 (50.7)
Random, full ~32M pool (GR-Dexter)48.9 (64.5)24.7 (24.5)53.0 (54.0)
SiMDex (1.49M)50.3 (54.5)39.6 (47.2)70.0 (83.3)

How the entries are computed.

  • In parentheses: for GR-Dexter and SiMDex, exactly the values in Table 1, where 1× averages two rounds of five trials.
  • Pooled values: the plain mean of the four budget columns in the Q2 table, so every budget counts equally; e.g., SiMDex on Drill: (35.5 + 48.9 + 54.5 + 62.2) / 4 = 50.3.
  • Differences: paired by layout and computed from unrounded scores, so they can differ by 0.1 from the table; p is from a two-sided paired permutation test, and p < 0.05 means a difference this large would rarely arise by chance.

What the table shows.

  • Only Drill gains when random data come from the full pool rather than the fixed subset: +26.1 (paired, p < 0.01), vs. −3.3 and −2.5 elsewhere (both not significant).
  • On Drill, SiMDex is +27.5 over the equal-size subset (p < 0.001) and level with GR-Dexter (+1.4, p = 0.85); at 1× alone, this is Table 1's Drill gap (54.5 vs. 64.5, p = 0.51).

Coverage. We designed the three tasks from an industrial assembly scenario, not from EgoDex's task list, so gaps in coverage are expected: none of the 194 EgoDex tasks involves drilling, and clips mined for the drilling stage show a tube, a saw and a hair dryer (App. C.3).

At 1×, trials that reach the drilling stage score alike on it: SiMDex 0.45 over 6 of 10 trials, GR-Dexter 0.42 over 8 of 10 (averaged over all 10 trials: 0.27 and 0.33, Table 1's Drill entries). Since neither data source contains drilling, this stage is similar for both; the deficit arises earlier, at Grasp and Assembly.

Per-task counts. This is a valuable suggestion, as such statistics make task-level results easier to interpret, and we regret that we cannot give them for the reported run.

  • Why. SiMDex merges the per-anchor lists into one deduplicated training set (Eq. 12), so per-task counts and overlaps would have to be rebuilt from the intermediate retrieval results, which, as intermediate outputs usually are, were cleared once the training set was finalized.
  • Why not re-run. We would have liked to re-run retrieval, but doing so with all anchors over the full ~32M pool was not feasible within the time and compute available during the discussion period; we have therefore answered the underlying question with the downstream results above.
  • Going forward. We will adopt your suggestion in future iterations of SiMDex: the pipeline will record, for every anchor, its task label and the IDs of the samples it retrieves, so that per-task counts and overlaps can be traced for any mined set.

SiMDex beats both random references on Flick Wheel and Pick & Place (each p < 0.05; Q2) and is level with GR-Dexter on Drill. The revised limitations will discuss amount and coverage for Drill, and the appendix will add the per-task table.

AuthorsOpenReview ↗

Response to Reviewer RmSw (R3), part 2/3

Q2 / W2.\color{blue}{\text{Q2 / W2.}} Amount of human data.

We agree that equal-size random baselines are the right test of human-data selection, and we made this comparison the main additional experiment in our response to you. The added condition is a fixed random subset of 1.49M samples, the same size as the mined set. To keep it directly comparable:

  • Training: at all four robot-data budgets, with the same model, training steps and 1:1 mixture as SiMDex.
  • Evaluation: the same rubric and layouts as Table 1 and the scaling study (Figure 4), with 25 real-robot trials per task (5 per budget as in Figure 4, but 10 at 1× as in Table 1), which adds 75 real-robot trials (25 per task × 3 tasks) to those reported in the paper.

With GR-Dexter and SiMDex, this gives three human-data conditions at four budgets:

Normalized progress score (%) per task and robot-data budget; 5 trials per cell, 10 at 1× (two rounds, as in Table 1; Figure 4 and Table 8 show the first round only). Overall is the mean of the three task scores shown (Table 8 prints 40.9 for GR-Dexter at 0.5×, averaged before rounding).

Human dataTask0.25×0.5×1×2×
Random, fixed 1.49M subset (added)Drill8.815.531.135.5
Flick Wheel22.220.032.237.8
Pick & Place44.954.950.771.6
Overall25.330.138.048.3
Random, full ~32M pool (GR-Dexter)Drill26.635.564.568.8
Flick Wheel16.735.524.522.2
Pick & Place45.051.554.061.6
Overall29.440.847.750.9
SiMDex (1.49M)Drill35.548.954.562.2
Flick Wheel30.048.947.232.2
Pick & Place43.276.783.376.7
Overall36.258.261.757.0

The 1× column of GR-Dexter and SiMDex is Table 1. Pooled tests below use 60 layout-matched pairs (5 layouts × 3 tasks × 4 budgets), with the two 1× rounds averaged per layout so that every budget counts equally.

Each difference is the mean over matched pairs (20 per task), in score points from unrounded trial scores; p is from the same paired test as in Q1, and a 95% CI entirely above zero means SiMDex is reliably higher.

  • Against the equal-size subset. SiMDex is higher at every budget and in 10 of 12 task–budget cells. Pooled: +17.8 (p < 0.0001, 95% CI [+11.8, +24.1]); per task, +27.5 on Drill (p < 0.001), +11.5 on Flick Wheel (p = 0.03) and +14.5 on Pick & Place (p < 0.01).
  • Against GR-Dexter. SiMDex is higher at every budget and in 9 of 12 cells. Pooled: +11.1 (p < 0.01, 95% CI [+3.9, +18.2]); per task, +14.9 on Flick Wheel (p < 0.01) and +17.0 on Pick & Place (p = 0.001); on Drill the two are level (Q1).
  • Between the random conditions. The larger pool adds +6.8 (p = 0.09, not significant), driven by Drill (Q1). Within this range, selection matters more than a larger random pool.

Smaller mined sets. You also ask how performance changes when the amount of mined data varies.

Our cascade ablation at 1× gives two smaller mined sets: it mines from a subset of the anchors and keeps the top 50 per anchor, and two of its runs include Ranking (Stage II). Its full design and analysis are in our reply to Reviewer CzZk (R1-W2/Q1). These two runs add 30 real-robot trials (15 each: 3 tasks × 5 layouts):

Mined setSamplesOverall
SiMDex (Table 1)1.49M61.7
Ranking (Stage II), top 50 per anchor1.05M58.3
Ranking + Re-ranking (Stage III) as filter1.03M61.9

Compared with SiMDex's 1.49M set, the paired differences are −3.3 (p = 0.69) and +0.2 (p = 0.99), so about 30% fewer mined samples score within 3.4 points. These runs also bear on your question of whether the gains persist across human-data budgets: both smaller mined sets score above the larger 1.49M random subset at 1× (38.0), by +20.3 (p = 0.045) and +23.9 (p = 0.017). The revised limitations will restrict our claims to the amounts reported.

Clarification of App. B.3 and D.4. Both sections call the Table 1 baseline a matched random subset; precisely, GR-Dexter draws at random from the full pool during training, matching SiMDex in the number of human samples seen, not in set size. Table 1 is unaffected. The equal-size comparison your summary describes is the added condition above, and there SiMDex's margin is even larger.

We will revise both sections. Because Q2 goes to the core of our claim, the revision will present this equal-size comparison in the main text and give the full per-task, per-budget score table in the appendix.

AuthorsOpenReview ↗

Response to Reviewer RmSw (R3), part 3/3

Q3.\color{blue}{\text{Q3.}} Unrelated tasks. Thank you for noting that strong performance on the target tasks is valuable in its own right. We also agree that broader generalization remains an important goal, and that it matters whether selecting data for the target tasks costs performance elsewhere. We evaluated the policies only on the target tasks, so we cannot say whether the target-task gains come at the expense of unrelated tasks.

  • What differs between the policies. Policies trained with SiMDex or with random selection start from the same pre-trained model and differ only in which human samples are added during post-training.
  • Within the three evaluated tasks. One policy is post-trained for all three tasks with one mined set, so our trials can show whether gains on some tasks cost another; they show no such loss. On Drill, where no EgoDex task involves drilling (Q1), SiMDex shows no measurable loss against GR-Dexter (50.3 vs. 48.9 averaged over the four budgets; 54.5 vs. 64.5 at 1×, p = 0.51), and it stays above the equal-size random subset (Q1).
  • What SiMDex is designed for. SiMDex aims to let a robot learn each new skill from fewer teleoperated demonstrations. As in a recommender, the robot's own demonstrations of a skill act as the query, and SiMDex recommends human data for that skill (see our separate comment on the vision behind SiMDex); a new skill, once demonstrated, can receive its own recommendations in the same way.
  • In the revision. The limitations will restrict our claims to performance on the three target tasks within the reported scenario and platform, and will list your question as an open one.

In brief.

  • Q1, Drill: the new results point to both amount (only Drill gains from a larger random pool) and coverage (EgoDex has no drilling task); future versions of SiMDex will record per-task counts.
  • Q2, amount: SiMDex beats an equal-size random subset at every budget (+17.8 pooled, p < 0.0001), and about 30% fewer mined samples score within 3.4 points.
  • Q3, unrelated tasks: within the three tasks we see no trade-off; unrelated tasks remain an open question in the limitations.

We thank you again for engaging so closely with our work. The equal-size comparison you asked for (Q2) now gives the clearest evidence for SiMDex, and the revision will present it in the main text. We hope these answers address your concerns, and we would be happy to discuss any of them further.

Reviewer_2wRCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SiMDex retrieves human demonstrations for dexterous robot-policy post-training using language and hand-pose recall, wrist/fingertip motion ranking, and source deduplication. It selects about 1.49 million samples from a 32 million-sample egocentric pool and mixes them with robot demonstrations using the GR-Dexter training setup. The overall normalized sub-task score rises from 47.7% to 61.7% across three real-world tasks. Optical-flow re-ranking is used only to inspect retrieval quality.

Critical strengths
  1. The main comparison holds architecture, robot data, training procedure, and human-data volume fixed, changing only how the human samples are selected. Against random sampling, SiMDex gains 14.0 percentage points overall, with large improvements on Flick Wheel and Pick & Place (Table 1). Drill decreases from 64.5% to 54.5%, so the benefit is task-dependent, but the two positive results provide useful evidence for targeted human-data selection.

  2. Appendix E specifies scoring, instruction switching, and resets, with execution and scoring performed by an independent team blind to method identity. The scaling study also discloses reuse of the full-budget mining anchors at smaller training budgets. This makes the scope of the data-efficiency results clear.

Critical weaknesses
  1. The experiments establish an advantage over random sampling, but leave the contribution of motion ranking unresolved. There is no downstream policy comparison against language/pose recall using an equally sized subset and the same deduplication rules. Broad task relevance and source diversity could account for part of the improvement. The retrieval visualizations do not show how much Stage II adds to policy performance.

  2. Table 9 interprets ratios of mean partial scores as conditional pass rates. For Twist given Grasp, the calculation is approximately 0.47/0.85 = 0.55. Grasp can receive 0, 0.5, or 1, so its mean is not the fraction of trials entering Twist; Twist itself earns thirds (Table 6; Appendix F.2). The ratio therefore gives neither a conditional completion rate nor mean progress among trials reaching Twist. Drill also allows progression without the trigger press. The claims about improved conditional skill and unchanged drilling do not follow from these ratios, although the composite-score comparison in Table 1 is unaffected. Ten trials per method/task and standard deviations based on two session means further limit the skill-level analysis.

Questions for authors
  1. How much does motion ranking add over language/pose recall when subset size, training budget, and deduplication are held fixed? A downstream comparison with this simpler selector would clarify the contribution of Stage II.

  2. Can Table 9 be recomputed using actual stage-entry counts, reporting conditional completion and partial progress separately? Do the Twist and Drill conclusions survive this correction, including the trigger-press exception?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 2wRC (R2), part 1/2

Thank you for a careful and constructive review. We appreciate your recognition that the main comparison isolates the effect of SiMDex's selection (S1), that the evaluation is documented and blind, and that the scaling study states its scope clearly (S2). Both your points lead to concrete changes: the stage-wise ablation below goes into the main text, and Table 9 is recomputed from stage-entry counts.

We apologize for replying late: assembling the stage-wise ablation below from our real-robot trial records, variant by variant, took more time than we expected, given our limited resources. For your questions, this response adds results from 135 real-robot trials that are not in the submission, in rounds of 15 (3 tasks × 5 layouts):

  • 60 for the stage-wise ablation (4 rounds).
  • 75 for an equal-size random subset (5 rounds: one per budget, two at 1×).

This comment answers W1/Q1; the next adds the alternatives in W1 and answers W2/Q2.

In brief.

  • W1/Q1, motion ranking: Ranking (Stage II) adds +17.7 over Recall (Stage I) alone (p = 0.017); the number of distinct samples, task relevance and source diversity do not account for the gain on their own.
  • W2/Q2, Table 9 (appendix): recomputed from stage-entry counts. The revision restates the Flick Wheel claim as what the counts show: SiMDex completes Grasp (7 vs. 2) and Twist (3 vs. 0) more often; the Drill claim survives, trigger-press exception included.

W1 / Q1.\color{blue}{\text{W1 / Q1.}} Motion ranking vs. language/pose recall.

Your question goes to the core of SiMDex's contribution. As you note, retrieval figures show what Ranking (Stage II) selects, not what it adds to the policy; the ablation below is the downstream comparison with the simpler selector that you describe.

  • Training. At 1×, we post-train the same model on each stage's output, with the same training steps and 1:1 mixture as Table 1.
  • Top 50, not 100. Re-ranking (Stage III) only reorders and verifies the samples it receives: with the top 100 per anchor, as in Table 1, the runs with and without it would train on identical data. The cascade runs therefore keep the top 50 per anchor, Re-ranking its best 50 of the top 100 from Ranking, and mine from a subset of the anchors.
  • Statistics. Paired differences are means over layout pairs of unrounded trial scores; p is from a two-sided paired permutation test, and CIs are 95% bootstrap intervals.
  • Rescoring. All trials are scored by the App. E stage-entry rule you raised for Table 9, which changes the credit of four Flick Wheel trials.

Columns 2–4 are the three conditions in your Q1: the training budget is the Table 1 setting (1× robot data, same training steps), subset size is the top 50 kept per anchor, and deduplication merges the per-anchor lists, keeping each sample once. Recall's lists overlap more across anchors, hence fewer distinct samples.

MethodTraining budgetSubset sizeDeduplicationDistinct samplesTrials per taskDrillFlick WheelPick & PlaceOverall
GR-Dexter (Table 1 baseline)same––drawn from ~32M1064.524.554.047.7
Recall (Stage I) only (language + initial pose)samesamesame0.14M10*2.260.559.140.6
+ Ranking (Stage II) (hand motion)samesamesame1.05M531.158.985.058.3
+ Re-ranking (Stage III), used as a filtersamesamesame1.03M564.537.883.361.9
SiMDex (Table 1)same–same1.49M1054.547.283.361.7

*Two rounds of 5: the first round's Drill score of zero looked surprising to us, so we ran a second round; the result held, and both rounds are reported.

What Ranking adds.

  • Adding Ranking to Recall lifts the overall score from 40.6 to 58.3. Compared layout by layout (15 pairs: 3 tasks × 5 layouts, with Recall's two rounds averaged), the gain is +17.7 (p = 0.017, 95% CI [+5.5, +30.7]), coming from Drill (+28.9) and Pick & Place (+25.9).
  • The full SiMDex of Table 1 is +21.1 above Recall alone (30 pairs matched by round and layout; p = 0.003, 95% CI [+8.6, +33.6]).
AuthorsOpenReview ↗

Response to Reviewer 2wRC (R2), part 2/2

W1.\color{blue}{\text{W1.}} In addition, the alternatives you raise do not account for the gain on their own.

The equal-size random subset is 1.49M samples drawn once from the full pool. Numbers come from the ablation table in our first comment and the table below.

Alternative (W1)How we test itResult
More distinct samplesEqual-size random subset, with more distinct samples than Ranking (1.49M vs. 1.05M)38.0 at 1×, vs. 58.3 for Ranking (+20.3, p = 0.045)
Broad task relevanceRecall only, which selects by instruction and initial-pose similarity40.6, no better than GR-Dexter (47.7; −7.0, p = 0.44)
Source diversityEqual-size random subset, from twice as many episodes as SiMDex (~124,000 vs. ~63,000)Below SiMDex at every budget (+17.8 pooled, p < 0.0001)

Overall progress score (%) by robot-data budget.

Human data (1.49M samples each)EgoDex episodes0.25×0.5×1×2×
Equal-size random subset~124,00025.330.138.048.3
SiMDex~63,00036.258.261.757.0

Conclusion.

  • The number of distinct samples alone is not enough: a random subset with more distinct samples scores far below Ranking.
  • Task relevance alone is not enough: Recall picks task-relevant clips, yet is no better than random.
  • Source diversity alone is not enough: a random subset from twice as many episodes stays below SiMDex at every budget.

What all three controls lack is motion ranking, which again supports the core design of SiMDex: selecting human data by hand motion. Thank you for raising these alternatives; they made our analysis more complete, and this analysis will be added to the revised paper.

W2 / Q2.\color{blue}{\text{W2 / Q2.}} Table 9 (appendix).

Thank you for checking Table 9 against the scoring rules: as you note, the ratio mixes how often a stage is reached with how far trials get inside it. Below, it is recomputed from stage-entry counts, reporting conditional completion and partial progress separately (1×). Per App. E, a stage is entered only after the previous one is complete; in Drill, Assembly also starts once the drill is held stably without the trigger press.

TaskStageGR-Dexter: completed / enteredGR-Dexter: mean progress if enteredSiMDex: completed / enteredSiMDex: mean progress if entered
Flick WheelGrasp2 / 100.607 / 100.85
Flick WheelTwist0 / 20.673 / 70.67
Flick WheelFlick0 / 0–0 / 30.33
DrillGrasp6 / 100.804 / 100.73
DrillAssembly8 / 81.006 / 80.79
DrillDrilling0 / 80.420 / 60.45
  • Flick Wheel: the revision restates the claim as stage completions. Trials reaching Twist make the same progress (0.67 vs. 0.67), and too few GR-Dexter trials reach Twist (two) or Flick (none) to compare completion rates. The gain lies in reaching and finishing stages (SiMDex vs. GR-Dexter): Grasp completed 7 vs. 2, Twist 3 vs. 0 (the trials that reach Flick).
  • Drill: the unchanged-drilling claim survives, trigger-press exception included. Trials reaching drilling progress similarly (0.45 vs. 0.42). Four of SiMDex's eight Assembly entries use the exception, vs. two for GR-Dexter. The deficit arises earlier, at Grasp (4 vs. 6 completed) and Assembly (6/8 vs. 8/8).
  • Sample size. Agreed: at ten trials per method, stage counts are descriptive only. For Table 1's normalized progress score (unaffected, as you note), the revision will also report a paired permutation test over layout-matched trials: +14.0 at 1× (30 pairs: 3 tasks × 5 layouts × 2 rounds; p = 0.022, 95% CI [+2.7, +24.9]).

In short, following your review we will:

  • add the stage-wise ablation, with the numbers of distinct samples and paired statistics, to the main text;
  • replace Table 9, and all text that uses its ratios (App. F.2 included), with stage-entry counts, and revise the Flick Wheel claim as above;
  • report paired tests for the main comparison;
  • describe SiMDex as a two-stage selector (Recall (Stage I) + Ranking (Stage II)): as your summary notes, optical flow serves only to inspect retrieval.

Thank you again for checking our analysis so carefully. Both the stage-wise ablation, laid out by the conditions in your Q1, and the recomputed Table 9 will go into the revised paper. We hope these answers resolve both points, and we would be glad to clarify or discuss anything further.

AuthorsOpenReview ↗

Revised paper addresses both of your questions

Dear Reviewer 2wRC,

We have uploaded the revised PDF. Your Q1 goes to the core of SiMDex, so the revision adds the stage-wise ablation you asked for to the main text rather than the appendix (Sec. 4.5 and Table 2), with subset size, training budget and deduplication held fixed. Adding Ranking to Recall raises the overall success rate by 17.7 points (p = 0.017). For Q2, Table 15 (App. F.2) recomputes the former Table 9 from stage-entry counts. We hope these changes resolve your concerns.

As the author response period comes to a close, we would like to thank you once more for the time and care you have given our paper. Your careful reading and precise questions have made it stronger, and we are truly grateful.

Best regards,

SiMDex Team

Thank you for the detailed response, which addresses most of my previous concerns. I maintain my score.

Reviewer_CzZkOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SiMDex, a retrieval pipeline that selects human egocentric samples from EgoDex to add to VLA post-training on a bimanual dexterous robot. After filtering, EgoDex gives about 32M frame-level samples. For each robot demo frame, the method first recalls candidates using initial fingertip pose and instruction embedding, fused with RRF to a top 10k. It then ranks them by wrist trajectory, translation and rotation speed profiles, and fingertip trajectory, keeping one sample per source episode. A third stage re-ranks with optical flow, but it is used only for analysis and does not change the training set. Human samples are supervised only on the 42 wrist and fingertip dimensions through a loss mask, so the GR-Dexter-style model itself is unchanged. The evaluation covers three real tasks (Drill, Flick Wheel, Pick & Place). Training on ~1.49M mined samples instead of the same number of random ones raises overall success from 47.7% to 61.7%. A robot-data scaling study from 0.25x to 2x shows SiMDex ahead at every budget.

Critical strengths
  1. The main comparison is well controlled. Both methods use the same model, pool, filtering, human-data budget, and seed, so only the choice of samples differs. Trials are run and scored blind by a separate team. Because of this, I believe the central claim as stated: targeted selection beats random selection at equal budget.

  2. The method is simple and drop-in. The only change on the policy side is a loss mask, so it should transfer to other VLAs and human datasets. The gains also show up where human hand data should help most. On Flick Wheel, the conditional pass rate rises from 0.22 to 0.55 at Twist and from 0.00 to 0.21 at Flick, and small-object grasping improves in Pick & Place. The failure analysis in App. C.3 (hair dryer and saw clips retrieved for drilling) is also candid and useful.

Critical weaknesses
  1. The evaluation is small relative to the claims. Each task gets 10 trials, and each scaling point only 5. Round-to-round noise is large. From Tables 1 and 8, SiMDex on Drill goes from 37.7% in round 1 to about 71% in round 2 on the same layouts, a swing bigger than most of the reported gaps. Drill is below the baseline at 1x and 2x, so the overall improvement rests on two tasks and few trials. The Flick-stage gain amounts to roughly one partial success.

  2. The cascade itself is not ablated, and the only baseline is uniform random sampling. Since most of EgoDex has nothing to do with industrial assembly, random sampling is a weak reference. It is unclear how much comes from the ranking stage compared with simple language or task-label filtering, recall alone, or any single motion component. As it stands, the paper shows that some relevance filtering helps more than it shows this particular design is needed. The “three-stage” framing also overstates the method, since Stage III never affects training.

Questions for authors
  1. Can you ablate the cascade? Useful variants would be language-only or task-label filtering, recall-only, and ranking with each component removed. Even one budget would help.

  2. How diverse is the mined set: how many EgoDex episodes and tasks does it draw from?

  3. Can you report a robot-only result? Even though the claim is about selection, it would put both numbers in context.

5 comments
AuthorsOpenReview ↗

Response to Reviewer CzZk (R1), part 1/3

Thank you for a detailed review that focuses on three questions central to our claim: evaluation noise, whether the cascade is needed, and what the headline numbers mean. We also appreciate your recognition of the controlled main comparison (S1), SiMDex's drop-in design and the candid failure analysis in App. C.3 (S2).

We are sorry for the late reply: pairing the trials below by layout and round from our real-robot trial records took longer than we had hoped, with limited resources during the discussion period. For your questions, this response adds results from 225 real-robot trials that are not in the submission, in 15 evaluation rounds of 15 trials each (3 tasks × 5 layouts):

  • 60 for the cascade ablation (W2/Q1): Recall (Stage I) only in two rounds; up to Ranking (Stage II) and Re-ranking (Stage III) as a filter in one round each.
  • 90 for the robot-only references (Q3): one round at each of the four budgets, plus two rounds of a 1× run with the same training steps.
  • 75 for an equal-size random subset, the reference column of the table in Q2: one round at each budget, two at 1× (full results in our reply to Reviewer RmSw (R3-Q2)).

This first comment answers W1; the second answers W2 and Q1; the third answers Q2 and Q3.

In brief.

  • W1, noise: SiMDex leads in both rounds and on layout-matched pairs (+14.0, p = 0.022).
  • W2/Q1, cascade: Recall alone is no better than the random baseline (GR-Dexter); Ranking lifts it (+17.7, p = 0.017). The revision describes SiMDex as a two-stage selector.
  • Q2, diversity: about half the episodes of random, yet nearly as many instructions, and no episode or instruction dominates.
  • Q3, robot-only: SiMDex is above it at every budget (+18.0 on average).

W1.\color{blue}{\text{W1.}} Evaluation size and round-to-round noise.

The round-to-round swing you note is real: on Drill, SiMDex scores 37.7 vs. 71.2 (same checkpoint and layouts; table below). So the claim must hold on layout-matched pairs, which we test below.

Table 1 at 1×, by evaluation round: normalized progress score (%); each entry is the mean of 5 real-robot trials. Table 1 averages all 10 trials before rounding, so SiMDex's Pick & Place is 83.3 (83.325), not 83.4 from the two rounded entries.

MethodRoundDrillFlick WheelPick & PlaceOverall
GR-Dexter (Table 1 baseline)164.516.754.845.3
264.532.353.250.0
SiMDex137.751.186.758.5
271.243.380.064.8
  • Rounds. SiMDex leads overall in both rounds; each method's overall score moves at most 6.3 points between rounds, under half the 14.0-point gap.
  • Paired tests. At 1×: +14.0 over 30 pairs (3 tasks × 5 layouts × 2 rounds; 19 better / 6 worse / 5 ties; permutation p = 0.022; 95% bootstrap CI [+2.7, +24.9]). Scaling points have 5 trials per task, so we pool the four budgets: +11.1 over 60 pairs (3 tasks × 5 layouts × 4 budgets), averaging the two 1× rounds per layout so that each budget counts equally (p < 0.01; CI [+3.9, +18.2]).
  • Two tasks. Agreed. On Flick Wheel and Pick & Place, SiMDex is higher in 16 of 20 pairs (2 tasks × 5 layouts × 2 rounds) and lower in none (+26.1). All six worse pairs are Drill, where SiMDex is −10.0 at 1× (p = 0.51, not significant) and level with the baseline when pooled over the four budgets (50.3 vs. 48.9); we claim no Drill gain.
  • Flick stage. As you note, in Table 1 three SiMDex trials reach Flick, none for GR-Dexter. The Flick Wheel gain comes from completing Grasp (7 vs. 2 of 10) and Twist (3 vs. 0). The revision replaces the Table 9 ratios your review cites with these counts.

The revision will add these tests and counts. Thank you again for raising this point: the round-by-round view and the paired tests make the main result much clearer.

AuthorsOpenReview ↗

Response to Reviewer CzZk (R1), part 2/3

W2 / Q1.\color{blue}{\text{W2 / Q1.}} Ablation of the cascade and the "three-stage" framing.

Thank you for being considerate in noting that even one budget would help: real-robot evaluation, with each trial run and scored blind by an independent team, is indeed time-consuming. As you suggest, the ablation uses one budget, 1×, the setting of Table 1, so its results sit directly beside Table 1.

  • Training. Each variant post-trains on the cascade output up to a given stage, with the same model, training steps and 1:1 mixture as Table 1, so all see the same number of human samples.
  • Trials. 60 real-robot trials in rounds of 15 (3 tasks × 5 layouts): two rounds for Recall (Stage I) only, one for each other variant.
  • Top 50, not 100. Re-ranking (Stage III) only reorders and verifies the samples it receives. With the top 100 per anchor, as in Table 1, it would reorder the same 100 samples that Ranking (Stage II) returns, so the runs with and without it would train on identical data, leaving nothing to compare. Each cascade run therefore keeps the top 50 per anchor: Ranking its own top 50, and Re-ranking its best 50 of the top 100 from Ranking. The cascade runs also mine from a subset of the anchors (Table 1: all anchors), so the numbers of distinct samples differ (second column).
  • Scoring. Differences are paired by layout, from unrounded trial scores.
MethodDistinct samplesTrials per taskDrillFlick WheelPick & PlaceOverall
GR-Dexter (Table 1 baseline)drawn from ~32M1064.524.554.047.7
Recall (Stage I) only (language + initial pose)0.14M10*2.260.559.140.6
+ Ranking (Stage II) (hand motion)1.05M531.158.985.058.3
+ Re-ranking (Stage III), used as a filter1.03M564.537.883.361.9
SiMDex (Table 1)1.49M1054.547.283.361.7

*Two rounds of 5: the first round's Drill score of zero looked surprising to us, so we ran a second round; the result held, and both rounds are reported.

  • Recall only vs. GR-Dexter: 40.6 vs. 47.7 (−7.0, p = 0.44).
  • Adding Ranking: 40.6 → 58.3; paired over 15 pairs (3 tasks × 5 layouts; Recall's two rounds averaged), +17.7 (p = 0.017, 95% CI [+5.5, +30.7]). The gain comes from Drill (+28.9) and Pick & Place (+25.9).
  • SiMDex vs. Recall only: +21.1 (30 round- and layout-matched pairs, as both have two rounds; 22 better / 6 worse / 2 ties, p = 0.003, 95% CI [+8.6, +33.6]).
  • Per task: on Drill, Recall only is near zero (2.2). In none of its ten Drill trials, over both rounds, did it hold the drill stably, and, from the evaluators' feedback in the first round, we recorded that it kept reaching for the flick wheel instead. As no EgoDex task involves drilling, we suspect that the small Recall set leans toward wheel-turning motions.
  • Re-ranking and the "three-stage" framing: we agree and will revise the framing. Used as a filter on the same cascade output, Re-ranking scores 61.9 vs. 58.3 without it (one round each; +3.5, p = 0.73). The abstract, contributions, Fig. 1 and Sec. 3 will describe SiMDex as a two-stage selector (Recall + Ranking), with optical flow used only for verification.

Taken together:

  • Most of EgoDex is unrelated to assembly (W2), as you note, and this is the case SiMDex is built for: it ranks clips by hand motion, where a grasp or a twist from an unrelated scene can still match a step of an assembly task (the mined set spans about 2,100 EgoDex instructions; Q2). As a baseline for selecting human data, random is also harder to beat than it looks: Recall-only data, already filtered by instruction and initial fingertip pose, are no better than GR-Dexter's random draws; only variants with Ranking score above it. We believe SiMDex can make the vast amount of egocentric data a reliable source for robot skill learning.
  • Recall returns a much smaller set. Only 0.14M distinct samples (Ranking: 1.05M) despite the same top-50 quota per anchor: anchors that share instructions and initial poses most likely recall the same clips, so each sample is repeated more often in training. Ranking tells these clips apart by motion, which is why SiMDex needs both stages.
  • What we claim. Recall falls short as a selector, and Ranking closes the gap. Our claim concerns Ranking as a whole, not any single ranking component; language-only and task-label filters are left for future work. The table, the numbers of distinct samples and statistics will go into the main text, and the limitations will state this scope.
AuthorsOpenReview ↗

Response to Reviewer CzZk (R1), part 3/3

Q2.\color{blue}{\text{Q2.}} Diversity of the mined set.

To check whether the mined set collapses onto a few episodes or tasks, the table compares it with a random subset of the same size, trained like SiMDex at all four robot-data budgets (full design and results in our reply to Reviewer RmSw (R3-Q2)):

Each set has 1,493,001 samples; percentages are shares of this total. Instructions are EgoDex's episode-level descriptions of task and scene, finer than its 194 tasks. Scores use 5 trials per task at each budget and 10 at 1× (two rounds, as in Table 1).

SiMDex (1.49M)Equal-size random subset
EgoDex source episodes~63,000~124,000
Largest single episode1,171 samples (0.08%)468 samples (0.03%)
Distinct task instructionsabout 2,100about 2,400
Most frequent task instruction30,046 samples (2.0%)13,779 samples (0.9%)
Overall progress score (%) at 0.25× / 0.5× / 1× / 2×36.2 / 58.2 / 61.7 / 57.025.3 / 30.1 / 38.0 / 48.3

What the table shows:

  • Fewer episodes, similar breadth. The mined set draws on about half as many episodes as random, as expected of SiMDex's task-targeted mining, yet it covers nearly as many task instructions (about 2,100 vs. 2,400).
  • No collapse. No episode or task instruction dominates: the largest episode holds 0.08% of the set and the most frequent task instruction 2.0%. For each anchor, SiMDex keeps at most one frame per source episode, so an episode contributes several samples only when many different anchors pick it.
  • Higher scores. SiMDex scores higher at every budget (pooled against this subset: +17.8, p < 0.0001, with the same 60-pair test as in W1).

Thank you for raising this; we will add this table to the appendix.

Q3.\color{blue}{\text{Q3.}} Robot-only result.

Thank you for suggesting this control: it puts 47.7 and 61.7 in context by comparing them with training on robot data alone. This control adds 90 real-robot trials that are not in the submission, in rounds of 15 (3 tasks × 5 layouts): one round at each of the four budgets, plus two rounds of a 1× run with the same training steps. All runs start from the same pre-trained model and are scored with the same rubric on the same layouts.

Normalized progress score (%) per task and robot-data budget. Each entry is the mean of 5 real-robot trials, or of 10 where two rounds are averaged (GR-Dexter and SiMDex at 1×, as in Table 1). Overall is the mean of the three task scores shown (Table 8 prints 40.9 for GR-Dexter at 0.5×, averaged before rounding).

Training dataTask0.25×0.5×1×2×
Robot only, same passesDrill11.011.042.224.3
Flick Wheel16.740.132.330.0
Pick & Place39.853.256.666.6
Overall22.534.843.740.3
GR-DexterDrill26.635.564.568.8
Flick Wheel16.735.524.522.2
Pick & Place45.051.554.061.6
Overall29.440.847.750.9
SiMDexDrill35.548.954.562.2
Flick Wheel30.048.947.232.2
Pick & Place43.276.783.376.7
Overall36.258.261.757.0
  • Same passes over robot data: robot-only sees the robot data as many times as the 1:1 mixed runs, so the human samples come on top of the same robot exposure. SiMDex is above robot-only at every budget and in all 12 task–budget cells, by +18.0 on average over the four budgets.

At 1× (the Table 1 setting); the same-step run also averages two rounds.

Training data (1×)DrillFlick WheelPick & PlaceOverall
Robot only, same training steps51.253.953.252.8
GR-Dexter64.524.554.047.7
SiMDex54.547.283.361.7
  • Same training steps (Table 1 setting): half of each mixed batch is human data, so this robot-only run sees the robot data twice as often as SiMDex; it reaches 52.8, and SiMDex is still higher, by +8.9.

In short:

  • Where they fall. GR-Dexter (47.7) lies between the two robot-only references (43.7 and 52.8); SiMDex (61.7) is above both.
  • Why. At equal robot exposure, GR-Dexter is above robot-only at every budget; at equal training steps, it falls below robot-only (47.7 vs. 52.8), while SiMDex stays above it, by +8.9. This matches the paper's motivation: which human data are used matters.
  • Why random is the control. As you note, our claim is about selection at a fixed human-data budget.

The revision will add the robot-only references to the main text and the per-task tables to the appendix.

Thank you again for such a careful review. Your questions on noise, the cascade, diversity and robot-only training gave SiMDex a much fuller evaluation, and all of these results will go into the revised paper. We would welcome any further discussion.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response to the AC and all reviewers

We are grateful to the Area Chair for their time and effort in handling our submission, and to Reviewers CzZk (R1), 2wRC (R2) and RmSw (R3) for their careful, constructive reviews.

Strengths noted by the reviewers.

  • The central claim holds. All three find the main comparison of SiMDex with random selection well controlled: only the choice of human samples differs (R1-S1, R2-S1, R3-S2). Reviewer CzZk (R1-S1): "I believe the central claim as stated: targeted selection beats random selection at equal budget."
  • A practical problem and a drop-in method. Reusing egocentric data for VLA post-training is a practical problem (R3-S1), as it can reduce reliance on real-robot data. SiMDex needs no change to the policy architecture (R3-S1); only a loss mask changes, so it should transfer to other VLAs and human datasets (R1-S2).
  • Careful evaluation. Trials are run and scored blind by an independent team, with the scoring documented in App. E (R1-S1, R2-S2); the scaling study states its scope clearly (R2-S2); the failure analysis is candid (R1-S2).

We sincerely apologize for our late responses: our dexterous real-robot trials are scored stage by stage by an independent team (App. E), and with limited manpower and resources, compiling and checking these records per stage, task and budget took longer than we had hoped.

Our responses add results from 225 real-robot trials not in the submission, all on the paper's platform, tasks, rubric and layouts, in 15 rounds of 15 trials (3 tasks × 5 layouts):

  • 75 equal-size random: one round per budget, two at 1×.
  • 90 robot-only: one round per budget, plus two rounds of a 1× run with the same training steps.
  • 60 cascade: Recall (Stage I) only in two rounds, the other two variants in one each.

The individual replies give details. Tests pair trials by layout (pooled: 60 pairs = 3 tasks × 5 layouts × 4 budgets, the two 1× rounds averaged); p < 0.05 means a difference this large would rarely arise by chance, and a 95% CI entirely above zero means SiMDex is reliably higher.

  1. Random references (R1, R2, R3). SiMDex beats GR-Dexter (Table 1 baseline; random draws from the full ~32M pool) by +14.0 at 1× (p = 0.022, 95% CI [+2.7, +24.9]) and +11.1 pooled (p < 0.01), and is higher at every budget than the equal-size random subset (1.49M; +17.8 pooled, p < 0.0001); our reply to Reviewer RmSw (R3-Q2) breaks this down per task and budget.
  2. Drill (R1, R3). Pooled over budgets, SiMDex beats both references on Flick Wheel and Pick & Place (each p < 0.05) and is level with GR-Dexter on Drill. Only Drill gains from the larger random pool, and no EgoDex task involves drilling, which points to both the amount and the coverage of human data (R3-Q1).
  3. Robot-only (R1). SiMDex is above robot-only training at every budget (+18.0 on average over the four budgets). With the same training steps (1×), robot-only scores 52.8, between GR-Dexter (47.7) and SiMDex (61.7).
  4. Cascade ablation at 1× (R1, R2, R3). With the same training budget and the same top 50 per anchor, Recall alone scores 40.6, no better than GR-Dexter, and adding Ranking (Stage II) lifts it to 58.3 (+17.7, p = 0.017). Neither the number of distinct samples, task relevance nor source diversity accounts for this gain on its own (R2-W1), and with about 30% fewer mined samples than SiMDex, the two cascade runs with Ranking score within 3.4 points of it. The revision describes SiMDex as a two-stage selector, with optical flow used for verification.
  5. Diversity of the mined set (R1). It draws on ~63,000 EgoDex episodes and about 2,100 task instructions, and no episode or instruction dominates: the largest episode holds 0.08% of the set and the most frequent instruction 2.0%.
  6. Table 9 in the appendix (R1, R2). Recomputed from stage-entry counts: the Flick Wheel claim will be restated as stage completions (Grasp 7 vs. 2, Twist 3 vs. 0); the unchanged-drilling claim survives.

Clarification. The revision will make App. B.3/D.4 precise: GR-Dexter matches SiMDex in human samples seen, not in set size. Table 1 is unaffected.

Revision. We will upload the revised PDF, with all changes marked in blue, before the revision deadline, and will post a separate summary of the revisions once it is ready.

We sincerely thank all three reviewers again for their valuable suggestions, which have made SiMDex stronger. All results and clarifications from this discussion will be included in the revised paper, and we would be glad to discuss any point further.

AuthorsOpenReview ↗

The motivation and vision behind SiMDex

Dear Area Chair and Reviewers,

Thank you for the time and care you have given our submission. Beyond our point-by-point responses, we would like to share the motivation and vision behind SiMDex.

In our community, egocentric datasets are growing at an astonishing pace: EPIC-KITCHENS [1] (2018) offered 55 hours, Ego4D [2] (2022) 3,670, and Egocentric-100K [3] (2025) 100,405 in a single release. EgoScale [4] (2026) shows why this matters: its dexterous policy, pre-trained on up to 20,854 hours of human video, improves steadily with data scale.

Yet most groups have no way to digest this much data, as their compute and storage fall far short. A complementary question has so far received less attention: which of these data actually help a dexterous robot learn a given skill, and can effectively replace part of its real-robot data?

We believe that reducing how much dexterous robots depend on teleoperated demonstrations is a direction worth exploring. Dexterous teleoperation data are costly, time-consuming to collect and dependent on experienced operators; above all, they are tied to the hardware. Dexterous-hand platforms are scarce, and each records one demonstration at a time, so collection scales only with the number of robots and operators. Egocentric video, by contrast, is far easier to scale, since it grows with the number of people who record it.

When a robot has only a few demonstrations of a skill, they act like a user's history, and egocentric data like a vast catalog: the setting recommender systems are built for. Computing detailed motion similarity between every robot demonstration and tens of millions of human samples would be prohibitively costly, so SiMDex borrows their recall-then-rank design to stay efficient on limited compute: a fast recall first narrows the pool to a short list, and only these candidates are ranked by motion, since the same grasp or twist recurs in kitchens, workshops and play.

In a controlled comparison, the 1.49M samples it selects, only 4.66% of a ~32M-sample pool from EgoDex [5] (829 hours), outperform as many random draws from the full pool (Table 1: 47.7 → 61.7). Our responses to the reviewers add that it stays ahead of an equal-size random subset at every robot-data budget, and that the gain comes from ranking by hand motion, the core design of SiMDex, not from language and initial pose alone.

We hope a humanoid robot with dexterous, human-like hands will one day learn a skill from a handful of teleoperated demonstrations, with the rest recommended from existing human data. We also hope that the real potential of the large-scale egocentric data now available online will be fully unlocked, so that more groups in our community can put it to use.

We see SiMDex as one step toward it: if egocentric datasets come with wrist and fingertip trajectories, any dexterous robot could search them with its own demonstrations, and the hours our community records today could become skills robots learn tomorrow. We thank the reviewers for questions that have made this work clearer. As egocentric data keeps growing, we believe choosing the right data for dexterous robots will matter as much as collecting more, and we hope SiMDex brings more attention to this question.

Best regards,

SiMDex Team


References:

[1] D. Damen et al., "Scaling Egocentric Vision: The EPIC-KITCHENS Dataset", In ECCV 2018.

[2] K. Grauman et al., "Ego4D: Around the World in 3,000 Hours of Egocentric Video", In CVPR 2022.

[3] Build AI, "Egocentric-100K", Hugging Face Datasets, 2025.

[4] R. Zheng et al., "EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data", arXiv:2602.16710, 2026.

[5] R. Hoque et al., "EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video", In ICLR 2026.

AuthorsOpenReview ↗

Summary of revisions

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair for overseeing the review of our paper, and all three reviewers for the time and care they have given to it. Their thoughtful questions and suggestions shaped this revision, and the paper is clearer and more rigorous because of them.

We have uploaded the revised PDF. New and rewritten text is marked in blue, while deletions and most minor wording edits are not highlighted. The main text remains within 10 pages. R1, R2 and R3 refer to Reviewers CzZk, 2wRC and RmSw.

Main text

  • Two selection stages (R1, R2). The abstract, contributions, Fig. 1 and Sec. 3 describe SiMDex as two selection stages, Recall and Ranking, with optical flow (Re-ranking) used for verification.
  • Equal-size random subset (R3). We add GR-Dexter*, a fixed random subset of the same size as the mined set. It appears in Table 1 and Sec. 4.3 at 1×, and in Sec. 4.4 and Fig. 4 at all four robot-data budgets.
  • Paired tests (R1, R2). Sec. 4.3 reports the paired permutation test for Table 1 (+14.0, p = 0.022, 95% CI [+2.7, +24.9]). Sec. 4.4 reports the pooled test against GR-Dexter* (+17.8, p < 0.0001).
  • Stage-wise ablation (R1, R2, R3). The new Sec. 4.5 and Table 2 give the ablation, with the numbers of distinct samples and paired statistics.
  • Robot-only references (R1). Sec. 4.3 states the comparison. Owing to the page limit, the full results are in App. F.1.
  • Stage-entry counts (R1, R2). The per-task discussion in Sec. 4.3 now rests on stage-entry counts, which replace the conditional pass rates of the former Table 9.
  • Limitations (R1, R3). We now restrict our claims to the tasks, platform and amounts of human data reported in the paper.

Appendix

  • App. B.3 and D.4: the baseline description is corrected. GR-Dexter matches SiMDex in the number of human samples seen, not in set size.
  • App. C.1 (Table 5): diversity of the mined set.
  • App. E.2 and E.5: the recorded step values (0.33 and 0.67) and the paired-test procedure.
  • App. F.1 (Tables 11–14): per-task results of the equal-size subset at every budget, with pooled tests, and of the two robot-only references (same robot-data passes at every budget, same training steps at 1×).
  • App. F.2 (Table 15): stage-entry counts, replacing the former Table 9, with the Flick Wheel claim restated.
  • New App. G (Table 16): details of the stage-wise ablation, including the three conditions in R2's Q1, the other explanations raised by R2, and the smaller mined sets discussed with R3.

Table numbers. The new tables shift the numbering: the former Tables 2–8 are now Tables 3, 4, 6, 7, 8, 9 and 10, and the former Table 9 is replaced by Table 15. Figure numbers are unchanged. In Table 10 (the former Table 8) and Sec. 4.4, GR-Dexter's overall success rate at 0.5× now reads 40.8. It is the mean of the printed task rates, as noted in our replies.

We particularly appreciated how concrete the reviews were. The requests for paired statistics and robot-only references, the call to separate the contribution of each selection stage, and the suggestion of an equal-size random baseline each led to a new experiment or analysis that now strengthens the paper.

Best regards,

SiMDex Team

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.