- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Evo-DexBench, a simulation benchmark for few-shot transfer across related dexterous manipulation tasks. It organizes 30 tasks into six functional families, each containing four Core tasks with 50 demonstrations each and one Gen task with eight demonstrations. The benchmark includes single-arm and bimanual configurations, three scene layouts per task, and distributed tactile observations, with 1,248 demonstrations collected through VR teleoperation. The paper compares five policies on 15 tasks and conducts additional experiments on adaptation strategies, source-task relatedness, source-set composition, and tactile sensing. The results suggest that joint Core–Gen training and functionally matched source data can improve target performance, while the tactile-conditioned control pathway provides task-dependent benefits.
Critical strengths
- The benchmark targets a practically relevant transfer setting. Organizing related source tasks around shared functional interactions and limiting target demonstrations makes the evaluation question concrete: whether source experience improves learning a sparsely demonstrated target task. This setting is relevant to reducing demonstration-collection costs for new dexterous manipulation tasks.
- The environment and dataset provide a useful integrated research resource. The benchmark includes 30 tasks across six functional families, single-arm and bimanual configurations, three layouts per task, and 1,248 demonstrations. The unified observation/action interfaces and documented task criteria could facilitate reproducible experiments across different manipulation scenarios.
- The experiments provide useful initial diagnostic evidence. Beyond reporting baseline success rates, the paper examines adaptation strategies, source-task relatedness, source-set composition, and tactile feedback. These comparisons offer concrete observations about how training-data selection and sensing affect performance. Although their coverage remains limited, they provide starting points for understanding policy behavior and designing subsequent experiments.
Critical weaknesses
- The incremental novelty of the Core–Gen evaluation principle needs clearer justification. Related-task adaptation already appears in prior benchmarks. Bi-DexHands ML4 trains on three related throwing/catching tasks and adapts to a fourth, explicitly motivating the selection through shared skills. LIBERO constructs task suites for controlled knowledge transfer. These precedents use different learning settings—primarily meta-RL and lifelong learning, respectively—and do not constitute identical benchmarks. Evo-DexBench contributes a specific combination of functional-family organization, limited target demonstrations, and tactile-enabled arm–hand environments. However, the manuscript does not sufficiently establish which additional transfer distinctions this organization enables, limiting the justification for presenting Core–Gen as the central conceptual contribution. (Bi-DexHands, LIBERO)
- The policy comparison is incomplete for the stated transfer objective. The baseline suite reasonably covers conventional visuomotor learning, pretrained VLAs, and tactile-conditioned control. However, their benefits from source experience are not evaluated consistently. DP-C achieves the highest reported overall mean and 68% success on DishwasherKnob, yet corresponding Gen-only and mismatched-source results are not reported in the source-relatedness study. The manuscript therefore does not quantify the transfer gain for this strong baseline. Additionally, the adaptation-strategy study evaluates only π₀.₅ on two targets. These omissions limit conclusions about how different policy families exploit source knowledge.
- The central transfer experiments cover a restricted portion of the benchmark. The main source-relatedness matrix evaluates three single-arm Gen tasks, and cross-model replication is restricted to DishwasherKnob. Although bimanual tasks are included in the standard baseline and tactile evaluations, an analogous source-relatedness study for bimanual targets is not reported. Each functional family also uses one fixed Gen task without reported results for alternative target assignments. Consequently, the evidence that functional-family membership reliably predicts transfer across different interaction regimes and target choices remains limited.
- The source-breadth results have a limited interpretation. Table 17 explicitly varies source-set breadth and composition. With 50 available demonstrations per Core task, the two-task and four-task conditions also differ in the amount of available source data: 100 versus 200 demonstrations. This is a useful comparison of source collections, but it does not isolate the effect of task diversity under a fixed source-demonstration budget. I regard this as a secondary limitation of the interpretation of the breadth analysis.
Questions for authors
- What distinctive transfer property does Core–Gen evaluate beyond the related-task adaptation and controlled-transfer settings discussed above? A concrete example would help clarify the incremental contribution.
- Can the authors provide Gen-only and mismatched-source results for DP-C, particularly on DishwasherKnob, to quantify its benefit from matched Core data?
- How were the Gen tasks selected? Are results available for representative bimanual targets or alternative target assignments that support the same source-relatedness pattern?
- Are data-volume-matched results available for the two-Core versus four-Core comparison? Such results would help distinguish the effects of source-task diversity and demonstration quantity.
- Could the authors clarify the relationship between task success and StageScore? Appendix A states that the final stage is equivalent to task success. Under Eq. (1), this appears to imply StageScore ≥ SR, yet Table 3 reports exceptions, such as 41% SR versus 35.2 StageScore for SmolVLA on LampButton. Are terminal success and ordered stage completion evaluated using different predicates? Please reconcile the definitions and reported interpretation.
- Could the authors reconcile the descriptions of the No-Touch variant? Section 4.5 states that the corresponding model branch is removed, whereas Appendix C.3 states that the tactile encoder is retained and receives zero inputs. Please clarify which component is removed and align the descriptions.