acceptodds
Withdrawn from ICLR 2027

Evo-DexBench: Benchmarking Generalization across Related Tasks for Dexterous Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

19 views

Abstract

General-purpose dexterous manipulation requires policies to transfer knowledge from related tasks to new tasks with only a few target demonstrations. Existing benchmarks often provide too many target demonstrations. They also leave the differences between source and target tasks uncontrolled. As a result, they cannot clearly distinguish knowledge transfer from task memorization. Our key insight is the Core-Gen principle. Core tasks are related tasks that share a functional interaction and provide source knowledge, while a Gen task is a novel target within the same functional family that has only a few demonstrations. This setup tests whether a policy can reuse functional interaction knowledge.Based on this principle, we propose Evo-DexBench, which controls both source-task relatedness and target-task exposure.It comprises 30 tasks in six functional families and a VR teleoperation pipeline. The resulting dataset covers three everyday scenes per task. It contains 1,248 multimodal demonstrations collected with simulated Shadow Hands equipped with 123 taxels. Evaluating representative policies, we find that the best achieves only 38.5% mean success, and that functionally matched Core data tends to yield larger gains than mismatched data. Results show that existing methods still struggle to extract and reuse functional interaction knowledge. Evo-DexBench provides an effective evaluation of the generalization ability of general-purpose dexterous manipulation policies across related tasks.

Primary area:
datasets and benchmarks
Keywords:
Robotics, Dexterous Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_feqmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Evo-DexBench, a simulation benchmark for few-shot transfer across related dexterous manipulation tasks. It organizes 30 tasks into six functional families, each containing four Core tasks with 50 demonstrations each and one Gen task with eight demonstrations. The benchmark includes single-arm and bimanual configurations, three scene layouts per task, and distributed tactile observations, with 1,248 demonstrations collected through VR teleoperation. The paper compares five policies on 15 tasks and conducts additional experiments on adaptation strategies, source-task relatedness, source-set composition, and tactile sensing. The results suggest that joint Core–Gen training and functionally matched source data can improve target performance, while the tactile-conditioned control pathway provides task-dependent benefits.

Critical strengths
  1. The benchmark targets a practically relevant transfer setting. Organizing related source tasks around shared functional interactions and limiting target demonstrations makes the evaluation question concrete: whether source experience improves learning a sparsely demonstrated target task. This setting is relevant to reducing demonstration-collection costs for new dexterous manipulation tasks.
  2. The environment and dataset provide a useful integrated research resource. The benchmark includes 30 tasks across six functional families, single-arm and bimanual configurations, three layouts per task, and 1,248 demonstrations. The unified observation/action interfaces and documented task criteria could facilitate reproducible experiments across different manipulation scenarios.
  3. The experiments provide useful initial diagnostic evidence. Beyond reporting baseline success rates, the paper examines adaptation strategies, source-task relatedness, source-set composition, and tactile feedback. These comparisons offer concrete observations about how training-data selection and sensing affect performance. Although their coverage remains limited, they provide starting points for understanding policy behavior and designing subsequent experiments.
Critical weaknesses
  1. The incremental novelty of the Core–Gen evaluation principle needs clearer justification. Related-task adaptation already appears in prior benchmarks. Bi-DexHands ML4 trains on three related throwing/catching tasks and adapts to a fourth, explicitly motivating the selection through shared skills. LIBERO constructs task suites for controlled knowledge transfer. These precedents use different learning settings—primarily meta-RL and lifelong learning, respectively—and do not constitute identical benchmarks. Evo-DexBench contributes a specific combination of functional-family organization, limited target demonstrations, and tactile-enabled arm–hand environments. However, the manuscript does not sufficiently establish which additional transfer distinctions this organization enables, limiting the justification for presenting Core–Gen as the central conceptual contribution. (Bi-DexHands, LIBERO)
  2. The policy comparison is incomplete for the stated transfer objective. The baseline suite reasonably covers conventional visuomotor learning, pretrained VLAs, and tactile-conditioned control. However, their benefits from source experience are not evaluated consistently. DP-C achieves the highest reported overall mean and 68% success on DishwasherKnob, yet corresponding Gen-only and mismatched-source results are not reported in the source-relatedness study. The manuscript therefore does not quantify the transfer gain for this strong baseline. Additionally, the adaptation-strategy study evaluates only π₀.₅ on two targets. These omissions limit conclusions about how different policy families exploit source knowledge.
  3. The central transfer experiments cover a restricted portion of the benchmark. The main source-relatedness matrix evaluates three single-arm Gen tasks, and cross-model replication is restricted to DishwasherKnob. Although bimanual tasks are included in the standard baseline and tactile evaluations, an analogous source-relatedness study for bimanual targets is not reported. Each functional family also uses one fixed Gen task without reported results for alternative target assignments. Consequently, the evidence that functional-family membership reliably predicts transfer across different interaction regimes and target choices remains limited.
  4. The source-breadth results have a limited interpretation. Table 17 explicitly varies source-set breadth and composition. With 50 available demonstrations per Core task, the two-task and four-task conditions also differ in the amount of available source data: 100 versus 200 demonstrations. This is a useful comparison of source collections, but it does not isolate the effect of task diversity under a fixed source-demonstration budget. I regard this as a secondary limitation of the interpretation of the breadth analysis.
Questions for authors
  1. What distinctive transfer property does Core–Gen evaluate beyond the related-task adaptation and controlled-transfer settings discussed above? A concrete example would help clarify the incremental contribution.
  2. Can the authors provide Gen-only and mismatched-source results for DP-C, particularly on DishwasherKnob, to quantify its benefit from matched Core data?
  3. How were the Gen tasks selected? Are results available for representative bimanual targets or alternative target assignments that support the same source-relatedness pattern?
  4. Are data-volume-matched results available for the two-Core versus four-Core comparison? Such results would help distinguish the effects of source-task diversity and demonstration quantity.
  5. Could the authors clarify the relationship between task success and StageScore? Appendix A states that the final stage is equivalent to task success. Under Eq. (1), this appears to imply StageScore ≥ SR, yet Table 3 reports exceptions, such as 41% SR versus 35.2 StageScore for SmolVLA on LampButton. Are terminal success and ordered stage completion evaluated using different predicates? Please reconcile the definitions and reported interpretation.
  6. Could the authors reconcile the descriptions of the No-Touch variant? Section 4.5 states that the corresponding model branch is removed, whereas Appendix C.3 states that the tactile encoder is retained and receives zero inputs. Please clarify which component is removed and align the descriptions.
Reviewer_LidyOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces EVO-DexBench, which covers three everyday scenes per task, specifically 1,248 multimodal demonstrations collected with simulated Shadow Hands equipped with 123 taxels. Five representative robot learning policies are tested. And the authors conduct ablations over few‑shot adaptation strategies, source‑task functional relatedness, and tactile input.

Critical strengths

The benchmark setting is novel, where Core‑Gen setup explicitly isolates functional interaction transfer from simple memorization, which is a critical gap for general‑purpose dexterous manipulation research.

The simulation environment supports single‑arm and bimanual Shadow Hand manipulation, which also provides provides high‑fidelity 123‑taxel per‑hand tactile observations, comprehensive multi‑camera visual inputs, and systematic evaluation perturbations including camera, lighting, physical dynamics, and spatial object offsets.

Several competative baselines are benchmarked, namely VLA models (π₀.₅, SmolVLA), diffusion policies (DP‑C, DP‑T), and tactile‑aware KineDex.

Critical weaknesses

There are still existing weaknesses.

Only simulation is considered. Specifically, All evaluations are restricted to the simulated Franka Panda + Shadow Hand setup. There is no cross‑embodiment transfer study, and no sim‑to‑real validation. As such, it is difficult to identify whether the evaluation conclusion can be transferred to real-robot.

Experiments are not sufficient. three out of six functional families are fully evaluated in the main experiments. Access‑and‑transfer, Structured configuration, Assembly are defined in the benchmark but lack systematic main‑paper experimental analysis.

Detail studies on the number of Gen demonstration are missing. The paper fixes Gen tasks to exactly 8 demonstrations. Different number of demonstrations are not evaluated. Thus how transfer performance evolves as the amount of target‑task supervision changes are not being presented.

Questions for authors

Check the detailed information in the weaknessses part.

Reviewer_4cm3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces a simulation benchmark for studying transfer from functionally related Core tasks to Gen tasks with limited target demonstrations. It combines single-arm and bimanual dexterous manipulation with visual, proprioceptive, and simulated tactile observations collected through VR teleoperation. The evaluation examines adaptation strategies, source-task relatedness, source-data composition, and tactile feedback.

Critical strengths
  1. This work addresses a relevant problem in dexterous learning by combining bimanual manipulation, tactile sensing, and Core–Gen task families. This organization gives the benchmark a distinctive focus on reusing related interaction experience under limited target supervision.
  2. The experimental design compares Gen-only training, sequential adaptation, and joint training, together with matched and mismatched Core sources. Each task uses a shared evaluation schedule with 100 episodes: 20 nominal and 20 each for camera, lighting, physical, and spatial variations. These choices provide a reasonable framework for examining source-data use and policy sensitivity.
  3. The implementation and evaluation protocols are described in useful detail, including state/action conventions, source-balanced sampling, tactile processing, policy adaptations, and reset schedules. These details make the experimental construction reasonably easy to follow.
Critical weaknesses
  1. The StageScore definition is inconsistent with the reported results. The stated equivalence between final-stage completion and task success implies StageScore ≥ SR, yet LampButton reports 15% SR and 12.6 StageScore for DP-T, with similar inconsistencies for KineDex and SmolVLA. The actual computation of this metric is therefore unclear.
  2. The source-family comparisons aim to establish transfer of functional knowledge. However, selecting matched or mismatched sources also changes object structure, appearance, and motion patterns. Better performance with matched sources could therefore reflect perceptual or geometric similarity rather than reuse of the intended functional knowledge.
  3. Equal epoch counts do not establish comparable training budgets across Gen-only, sequential, and mixed-data training. Optimizer-step counts and Gen-sample exposure are not reported, so the comparisons do not separate the benefit of retaining Core data from differences in optimization budget.
  4. Experimental coverage is incomplete. The main policy comparison covers 15 of 30 tasks; adaptation strategies are compared on two Gen tasks, and source-relatedness and breadth studies use three single-arm Gen tasks. Comparable controlled bimanual transfer studies are absent. The reported 38.5% SR averages 12 Core and three Gen tasks; the corresponding Gen-task mean is 27.7%.
  5. Given the substantial sim-to-real gap in dexterous contact and tactile sensing, simulation-only results leave the real-world relevance of the task-family and transfer design unverified. A representative real-robot study following the same Core–Gen principle is needed, even if it is reported separately from the reproducible benchmark.
Questions for authors
  1. Can you clarify how StageScore is computed in the implementation, including how stage completion is accumulated and related to terminal success? Why does it fall below SR in the reported LampButton results?
  2. Can the source-family comparisons separate functional relatedness from visual and geometric similarity, including functionally matched but visually distinct sources and visually similar but functionally mismatched sources?
  3. Can you report optimizer steps and Gen-sample exposure for each adaptation strategy, and compare the key settings under matched budgets?
  4. Can controlled transfer comparisons cover bimanual Gen tasks and the families omitted from the main evaluation? Can Core and Gen aggregate results also be reported separately?
  5. Can you add a representative real-robot Core–Gen experiment with bimanual dexterous hands and tactile feedback, retaining related source tasks and limited target demonstrations? This would help establish whether the benchmark design reflects transfer difficulties encountered on real hardware.
  6. Can you clarify how the pretrained π₀.₅ checkpoint is adapted to the 62-D state and 60-D action interface, including how its original 32-D action projections are expanded to 64 dimensions and loaded? Please also further clarify the initialization of the expanded SmolVLA projection parameters.
  7. Can Gen-only and Core+Gen training be compared with and without the complete tactile pathway, particularly on bimanual Gen tasks? The current ablation measures execution gains but does not establish whether touch improves transfer from Core tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.