acceptodds
Under review as a conference paper at ICLR 2027

CARAT: Static Compatibility and Dynamic Readiness for Adaptive Human-to-Robot Dexterous Manipulation Transfer

Abstract

Transferring human demonstrations offers a promising solution to the scarcity of embodiment-specific robot data in dexterous manipulation learning. However, existing human-to-robot transfer methods leave two complementary sources of variation underexplored: the reusability of human interaction priors differs across task-hand pairs, while the learner’s competence varies across trajectory phases and evolves throughout training. To exploit these variations, we propose CARAT, a unified framework that leverages Static Compatibility and Dynamic Readiness to adaptively facilitate human-to-robot manipulation transfer. CARAT formulates a Static Compatibility metric to calibrate task-hand-specific interaction constraints, balancing human prior preservation with alternative contact discovery. It also employs a Dynamic Readiness scheduler to track phase-wise policy competence online, adaptively shifting training emphasis toward the current learning frontier. Across seven manipulation tasks and four heterogeneous dexterous hands, CARAT outperforms baselines under a unified evaluation protocol, demonstrating the vital role of exploiting these variations in human-to-robot transfer.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Dexterous Manipulation, Human-to-Robot Transfer, Reinforcement Learning, Simulation-based Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KqtcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies human-to-robot transfer in dexterous manipulation, building on ManipTrans pipeline. It argues that two sources of variation are underexploited: prior reusability depends on the task–hand pair, and learner competence varies across trajectory phases. For the first problem, Static Compatibility combines hand-crafted morphology and interaction discrepancy measures into a scalar that sets the tolerance radius of an object-centric soft fingertip constraint. For the second problem, Dynamic Readiness estimates phase-wise look-ahead success online and redirects a fraction of episode resets to the identified learning frontier phase.

Experiments use seven OakInkV2 tasks spanning single-hand and bimanual manipulation, transferred to four heterogeneous dexterous hands (Inspire, Allegro, Shadow, and Artimano) in Isaac Gym, and evaluate frozen policies under a deterministic protocol with fixed start frames and contact-consistency checks. Compared against ManipTrans and an adapted DexMachina, the full method achieves the best average learning efficiency and final strict success rate, and ablations indicate that each component contributes on its own.

Critical strengths

Good problem decopmosition. The paper frames human-to-robot transfer along two intuitive axes: how much of the human interaction prior a given embodiment can reuse, and where along the trajectory the learner should concentrate training. The two components are cleanly separated and plug into a shared ManipTrans backbone, so each can be ablated in a controlled way.

Light-weight, drop-in design. Both components add no network branches, no extra policy inputs, and no additional simulation. The look-ahead success label is computed counterfactually from already-recorded tracking errors, and the soft interaction constraint is a single additive reward term. As a result, the method can be plugged into existing reference-guided transfer pipelines with minimal engineering overhead.

Critical weaknesses

Static Compatibility shifts tuning rather than removing it, and it remains unclear whether it identifies the optimal tolerance for each pair. The score is intended to replace per-pair tuning of the interaction tolerance, but computing it relies on a fair number of manually set constants, including morphology weights whose values are not reported. In addition, the fusion coefficient is selected based on experimental results, and the bound that the score is mapped to (Sec. 3.4.2) appear to be taken directly from the endpoints of the fixed-tolerance values tried in Sec. 4.2 (Fig. 5), rather than derived in a principled way. The calibration pairs used for this selection also overlap with the evaluation pairs. And there's no direct evidence that the score tracks the best tolerance for each pair other than comparison with baselines. Given the relatively small margin over a single tuned fixed tolerance, such evidence would help clarify how much signal the score provides.

The source and robustness of the empirical gains could be clarified. Table 2 compares Dynamic Readiness only against the original ManipTrans curriculum. However, the full curriculum comparison (Table 9) shows that simply training without any curriculum performs close to Dynamic Readiness and better than the ManipTrans curriculum. This suggests that part of the reported improvement may come from removing that curriculum rather than from the readiness-guided resets. Including the no-curriculum and fixed-schedule baselines in Table 2 would give readers a clearer picture. In addition, the per-pair results in Fig. 7 suggest that a large share of the overall improvement comes from the Rearrange task, while changes on many other pairs are modest or occasionally negative. More rigorous statistical significance measures would help support the reported improvements. Since the gains are not consistent across all tasks, it would also be helpful for the authors to explain why the method helps on some tasks but not others.

Questions for authors

Generalization to unseen tasks and hands. We understand that only the fusion coefficient is calibrated as a single global value, and that per-pair tolerances are then computed from the compatibility score. The method’s advantage over directly tuning the tolerance therefore depends on whether this coefficient transfers to unseen tasks and hands, yet the calibration pairs overlap with the evaluation pairs. Could the authors report results on held-out tasks and hands, compared against a per-pair tuned tolerance?

Choice of constants. The weights of the three morphology terms and the normalization constant in the mobility term do not appear to be reported. The remaining constants in the compatibility score are given in Appendix A but without justification. Could the authors report the missing values, explain how these constants were chosen, and discuss how sensitive the results are to them?

Source of the gains. Since training without any curriculum already outperforms the ManipTrans curriculum (Table 9), does adding readiness-guided resets on top of the no-curriculum setting still bring a clear improvement? In addition, could the authors explain why the gains are concentrated on Rearrange while many other pairs show little or no improvement?

7 comments

Thank you for recognizing our problem decomposition and lightweight design. We address your concerns below.


1. Generalization and Comparison with Per-pair Tuning (W1 & Q1)

  • Held-Out Generalization: We added a separate analysis of the remaining 16 pairs (28 total − 12 calibration), covering three additional tasks and four new task–hand combinations. All four hand types were represented during calibration. | Method | S-AUC${1000}$ (%) | S-SR${E1000}$ (%) | | :--- | :---: | :---: | | Fixed-25mm | 36.5 | 42.7 | | SC-SIC | 38.0 | 43.0 |

    • On these pairs, SC-SIC improves over Fixed-25mm by 1.5 pp in S-AUC$_{1000}$ and 0.3 pp in final success. This supports using Static Compatibility to guide tolerance selection.
  • Tracking Good Per-Pair Tolerances:

    • On the 12 calibrated pairs: Comparing Fig. 5 with Table 8 shows that SC-SIC directly tracks pair-wise tolerance preferences across both hands and tasks. Where Fig. 5 favors a tight 10 mm radius (Artimano on Cap the Pen and Rearrange), SC-SIC predicts a tight 16 mm tolerance. Where Fig. 5 favors a loose 40 mm radius (Allegro on Rearrange and Place Inside), SC-SIC predicts a loose 33 mm tolerance.
    • On the 16 held-out pairs: To verify that the calibrated α=0.7\alpha = 0.7 generalizes to unseen task-hand pairs and selects a suitable ρ\rho without costly per-pair grid search, we are completing the full [10, 25, 40] mm sweep on all 16 held-out pairs (please see our follow-up comment below for the updated results).

2. Missing Constants, Parameter Choices, and Sensitivity (W1 & Q2)

  • We apologize for omitting these details. Appendix A.3.2 now reports (ωsize,ωL,ωC)=(0.5,1,1)(\omega_{\mathrm{size}},\omega_L,\omega_C)=(0.5,1,1) and Cref=6C_{\mathrm{ref}}=6. The latter provides a reference scale for six-dimensional relative-pose capability. Unit weights apply to the normalized geometry and mobility terms, while the smaller size weight avoids premature saturation.

  • The 10–40 mm interval is an engineering choice. 10 mm defines a local, centimeter-scale interaction tolerance, while 40 mm is consistent with the thumb-tip tracking tolerance in our evaluation; 25 mm is the midpoint.

  • Because evaluating the training-performance effects of all constants would require substantial additional RL runs, our sensitivity analysis only examines α\alpha, which is the key parameter and central fusion coefficient controlling how task-specific interaction mismatch modifies morphology-based compatibility. For α=0.5/0.7/0.9\alpha=0.5/0.7/0.9, Table 7 reports calibration regrets of 2.021/1.825/2.149 pp, suggesting limited sensitivity around the selected value.


3. Curriculum Contribution and Variation Across Tasks (W2 & Q3)

  • Curriculum Comparison: We moved the full comparison into Table 2. | Curriculum Variant | Average (S-AUC / S-SR) | | :--- | :---: | | ManipTrans Baseline | 36.3 / 41.0 | | No-Curriculum | 37.6 / 44.3 | | Fixed Schedule (F-TTS) | 36.9 / 42.1 | | DR-RMS | 38.2 / 44.7 |

    • DR-RMS improves S-AUC/final success by 1.3/2.6 pp over F-TTS and 0.6/0.4 pp over No-Curriculum. To directly address your question, we are also evaluating No-Curriculum + readiness-guided resets. We will update this response once the results are available (please see follow-up comment).
  • Task-Dependent Gains: CARAT improves task-average S-SR E1000 across all 7/7 tasks and improves or ties on 20/28 pairs, including +15.3 pp on Alcohol Lamp/Inspire, +9.2/+8.0 pp on Cap the Pen/Artimano & Inspire, and +9.1 pp on Place Inside/Allegro. It also raises full-trajectory S-SR$_{t=0}$ from 16.4% to 23.2% and achieves strong statistical significance across all 28 pairs (N=84N=84 runs, Wilcoxon p = 4.82e-6 / 0.0001 and paired t-test p = 3.07e-5 / 0.0003 for S-AUC1000 /S-SR E1000). Excluding Rearrange, CARAT still gains +2.58/+1.96 pp. The learning curves and reference diagnostics reveal three relevant patterns:

    • Late-Training Differences on Rearrange: From epoch 700 to 1000, ManipTrans success falls from 35.64% to 24.80%, while CARAT rises from 48.93% to 52.80%. Both CARAT's continued improvement and the baseline's decline contribute to the large final gap.
    • Limited room for higher final success: ManipTrans already reaches 94.0%/97.1% on Brush Whiteboard and 82.8%/81.0% on Place Inside with Shadow/Artimano. CARAT nevertheless improves Brush Whiteboard/Artimano S-AUC by 10.46 pp, showing faster learning despite little endpoint headroom.
    • Severe Initial Reference Failures: When constructing the initial reference τr0\tau_r^0 (Section 3.3), the frozen imitator already suffers severe full-sequence fingertip RMSE against the IK target on certain Allegro pairs, such as 554.04 mm on Brush Whiteboard and 1882.41 mm on Unscrew. Consequently, ManipTrans and CARAT reach 0% and 9.64% final success, as severe initial reference failures bound downstream adaptation gains.

Update on Additional Experiment: No-Curriculum + Readiness-Guided Resets (W2 & Q3)

  • Following up on Question 3, we evaluated No-Curriculum + readiness-guided resets under the exact same strict evaluation protocol (q=0.7q = 0.7 fixed from epoch 0, with all other settings unchanged). The results are summarized below:
Task (Matched Pairs)PairsNo-Curriculum (S-AUC / S-SR$_{E1000}$, %)No-Curriculum + Readiness Resets (S-AUC / S-SRE1000_{E1000}, %)DR-RMS (S-AUC / S-SR_{E1000}\, %)
Rearrange439.48 / 44.6340.84 / 43.3638.66 / 39.78
Brush Whiteboard440.33 / 48.0539.07 / 46.5842.25 / 51.73
Cap the Pen430.05 / 36.3626.46 / 30.5332.01 / 37.27
Place Inside457.37 / 59.2458.43 / 60.5557.60 / 58.95
Put off Alcohol Lamp446.21 / 56.0541.84 / 53.1647.27 / 60.06
Uncap the Jar431.76 / 45.8734.73 / 47.7232.58 / 46.48
Unscrew417.93 / 19.3717.72 / 17.1216.86 / 18.52
All Completed Pairs2837.59 / 44.2237.01 / 42.7238.18 / 44.68
  • Across these 28 pairs, while adding readiness-guided resets directly on top of No-Curriculum improves tasks with gentler initial contact transitions (Rearrange, Uncap the Jar), enforcing q=0.7q = 0.7 from epoch 0 may trigger frequent premature terminations on contact-sensitive tasks (Cap the Pen, Put off Alcohol Lamp). Pairing readiness-guided resets with the progressive threshold schedule (DR-RMS: 38.18% / 44.68%) works complementarily, preventing premature early-stage failures while concentrating resets on the evolving frontier to achieve the strongest overall performance.

Update on Held-Out 16 Pairs: Comparison Against Per-Pair Tuned Tolerance (W1 & Q1)

  • Following up on Question 1, we are evaluating the full ρ∈{10,25,40}\rho \in \{10, 25, 40\} mm grid across all 16 held-out task-hand pairs (3 seeds each) to verify whether SC-SIC identifies an appropriate tolerance ρ\rho for unseen pairs without per-pair tuning.

  • Cost vs. Performance: Per-pair tuning across [10, 25, 40] mm requires 3x training runs per pair (9 seeded runs) and selects the best radius post hoc, whereas SC-SIC predicts ρ\rho offline before training and requires only 1x training run per pair (3 seeded runs).

  • The results on the 16 held-out pairs are summarized below: | Tolerance Strategy (16 Held-Out Pairs, seed=42/43/44) | S-AUC${1000}$ (%) | S-SR${E1000}$ (%) | | :--- | :---: | :---: | | Fixed Tolerance (ρ=10 mm\rho = 10\text{ mm}) | 34.66 | 38.68 | | Fixed Tolerance (ρ=25 mm\rho = 25\text{ mm}) | 36.49 | 42.72 | | Fixed Tolerance (ρ=40 mm\rho = 40\text{ mm}) | 37.77 | 42.53 | | SC-SIC (α=0.7\alpha = 0.7) | 38.04 | 42.74 |

Reviewer_89NAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses human-to-robot dexterous manipulation transfer via reinforcement learning with reference trajectory guidance, building upon the ManipTrans framework. The authors observe that the reusability of human interaction priors varies across task-hand combinations, and policy competence evolves dynamically across different trajectory phases during training. To address these variations, the authors propose CARAT: (1) Static Compatibility (SC), an offline metric that combines intrinsic morphology discrepancy (scale, geodesic skeletal geometry, kinematic path mobility) and task-specific interaction discrepancy (landmark pose, contact locations, contact normals) to calibrate the tolerance radius ρ of a Soft Interaction Constraint (SIC) reward term—tightening constraints for high-compatibility pairs and relaxing them for low-compatibility pairs to discover embodiment-specific alternative contacts; and (2) Dynamic Readiness (DR), which partitions demonstrations into four phases, tracks phase-wise competence online using look-ahead success labels and Wilson lower bounds, identifies the current "learning frontier", and uses Reset-Mixture Scheduling (RMS) to bias training episode resets toward that frontier. The framework is evaluated in Isaac Gym simulation across seven manipulation tasks from OakInkV2 and four heterogeneous dexterous hands (Shadow Hand, Allegro Hand, Inspire Hand, and Artimano), totaling 28 task-hand pairs, using strict metrics (S-AUC1000 and S-SRE1000). Both components are shown to improve performance over the uncalibrated ManipTrans baseline.

Critical strengths
  1. Broad Multi-Hand Empirical Testbed in Simulation: The authors evaluate CARAT across 28 distinct task-hand configurations (7 OakInkV2 tasks spanning single- and bi-manual manipulation crossed with 4 heterogeneous robotic hands: Shadow, Allegro, Inspire, and Artimano). The paper provides detailed multi-seed training curves and strict, standardized metrics (S-AUC1000 and S-SRE1000), offering a rich diagnostic testbed for understanding how kinematic hand differences impact reference-guided policy learning.

  2. Intuitive Decomposition of Transfer Dynamics: Decomposing reference-guided dexterous RL into embodiment-dependent interaction constraints (Static Compatibility) and phase-dependent exploration scheduling (Dynamic Readiness) presents a coherent, structured taxonomy for addressing transfer bottlenecks.

Critical weaknesses
  1. Complete Absence of Physical Robot Experiments and Sim-to-Real Analysis: Dexterous manipulation is an inherently contact-rich domain where real-world deployment is severely challenged by unmodeled contact compliance, dynamic friction, actuator backlash, and tactile sensing latency. CARAT is evaluated exclusively within the idealized rigid-body simulator Isaac Gym. The paper provides zero real-robot hardware verification and zero Sim-to-Real transfer experiments. Tuning millimeter-level contact tolerance thresholds (e.g., ρ = 10 mm vs. 25 mm) in simulation without validating their feasibility against physical calibration errors and sensor noise leaves the real-world utility of the method completely unproven in the embodied AI context.

  2. Marginal Empirical Gains Over Simple Fixed Baselines and Inferior Single-Task Performance: A rigorous examination of Table 1 demonstrates that the quantitative improvements of SC-SIC over a simple, globally fixed tolerance baseline (Fixed-ρ = 25 mm) are statistically marginal:

  • Across all 28 pairs, SC-SIC achieves an average S-AUC1000 of 38.2% compared to 36.8% for Fixed-ρ = 25 mm—an improvement of only 1.4 percentage points, while S-SRE1000 improves by merely 0.6 percentage points (42.4% vs. 41.8%).
  • On individual tasks, SC-SIC is actually worse than the fixed baseline or original ManipTrans: on Cap the Pen, Fixed-ρ achieves 35.2 ± 1.8% while SC-SIC drops to 33.4 ± 4.9%; on Put off Alcohol Lamp, the original ManipTrans baseline achieves 44.9 ± 2.0% while SC-SIC degrades to 43.1 ± 2.0%. Given the heavy computational and engineering complexity of calculating offline kinematic metrics, the practical benefit of SC-SIC is largely indistinguishable from random noise.
  1. Artificial Evaluation Protocol Masking Full Long-Horizon Failures: As explicitly acknowledged by the authors in Section 5 (Limitations): "our strict protocol evaluates continuation from sampled reference phases, so success does not imply reliable execution from task onset for every task-hand pair." The reported metrics evaluate the policy's ability to complete short trajectory suffixes starting from pre-initialized reference states rather than executing full rollouts from initial object configurations. This substantially overestimates the policy's true autonomous execution capabilities.

  2. Heavy Reliance on Handcrafted Heuristic Weighting: The Static Compatibility formulation relies on multiple manually tuned hyperparameters: weighting coefficients (ω_size, ω_L, ω_C in Eq. 1), predefined landmark correspondences Π, and empirical mobility descriptors. Similarly, Dynamic Readiness enforces an arbitrary, coarse division into B = 4 uniform segments. The framework lacks end-to-end adaptivity and requires extensive manual re-engineering for new embodiments or object geometries.

Questions for authors
  1. Sim-to-Real Feasibility: Have the authors attempted to deploy the learned policies onto a physical dexterous hand (such as the Inspire or Shadow hand)? How robust are the learned millimeter-scale contact thresholds against real-world camera tracking drift, joint latency, and dynamic contact slippage?

  2. Lack of Statistical Superiority Over Fixed-Tolerance Baselines: In Table 1, SC-SIC improves average S-AUC1000 by only 1.4 percentage points over Fixed-ρ = 25 mm, while performing worse on Cap the Pen (33.4% vs. 35.2%) and Put off Alcohol Lamp (43.1% vs. 44.9%). Can the authors provide formal hypothesis testing (e.g., paired t-tests or Wilcoxon signed-rank tests) demonstrating that SC-SIC yields statistically significant gains over a properly tuned constant tolerance?

  3. True End-to-End Task Success Rates: What is the policy's success rate when evaluated strictly from task onset (t=0) to completion, rather than continuing from mid-trajectory reference phase resets as evaluated in the paper?

3 comments

Thank you for your constructive comments and insightful questions. We address each of your concerns and questions point by point.


1. Sim-to-Real Feasibility and Role of Millimeter-Scale Tolerance ρ\rho (W1 & Q1)

  • New Physical Robot Validation (Section 4.5 & Supplementary Video): Following your suggestion and the open-loop deployment protocol of ManipTrans (CVPR 2025), we conducted real-world experiments using a Realman robotic arm equipped with a physical 6-DoF Inspire Hand on two representative tasks, Brush Whiteboard and Rearrange. The trajectories generated by CARAT successfully complete both wiping and object rearrangement on real hardware, confirming their physical feasibility.

  • Tolerance ρ Shapes Macroscopic Interaction Patterns Rather Than Runtime Sensor Thresholds: We clarify that ρ in Section 3.4.2 (Eq. 4 and 5) is an offline reward-shaping tolerance radius used during simulation training rather than an online sensor threshold at deployment. Once training is completed, LSIC,t\mathcal{L}_{\mathrm{SIC},t} and ρ\rho are disabled, the trained policy's inference inputs do not include ρ\rho, and deployment does not threshold live camera measurements against ρ\rho. Thus, ρ\rho only guides which macroscopic contact strategy the policy discovers during offline exploration, rather than acting as a runtime sensor threshold.

  • Analysis of Camera Drift, Joint Latency, and Slippage in Downstream Sim-to-Real:

    • Focus on Scalable Simulation Data Generation: Consistent with Section 1 and recent transfer studies (ManipTrans CVPR 2025; DexMachina ICML 2026; BiDexHD AAAI 2026), our primary objective is to rapidly translate human demonstrations into physically executable robot interaction data in simulation.
    • Relation to Closed-Loop Sim-to-Real: Following ManipTrans, our policy generates trajectories from privileged simulation states (such as exact 6D object poses) rather than live camera or tactile streams. Sensor-based closed-loop Sim-to-Real deployment is beyond our scope and may involve system identification, domain randomization, and teacher-student distillation. Our hardware tests demonstrate execution feasibility, but robustness to camera drift, joint latency, and slippage has not been systematically evaluated.

2. Statistical Significance and Why SC-SIC Outperforms Fixed-ρ\rho (W2 & Q2)

  • Following your suggestion, we performed one-sided paired t-tests and non-parametric Wilcoxon signed-rank tests across all 28 task-hand pairs (N=84 seeded runs). | Comparison | Δ\Delta S-AUC${1000}$ (pp) | Δ\Delta S-SR${E1000}$ (pp) | Paired tt-test (pp-value) (AUC/SR) | Wilcoxon Signed-Rank (pp-value) (AUC/SR) | | :--- | :---: | :---: | :---: | :---: | | SC-SIC vs. ManipTrans | +1.9 | +1.4 | 0.0165 / 0.2024 | 0.0021 / 0.4073 | | SC-SIC vs. Fixed-25mm| +1.4 | +0.6 | 0.0592 / 0.3628 | 0.0223 / 0.0852 | | CARAT vs. ManipTrans| +4.3 | +5.7 | 3.07e-05 / 0.0003 | 4.82e-06 / 0.0001 | | CARAT vs. Fixed-25mm | +3.8 | +4.9 | 0.0006 / 0.0013 | 5.62e-05 / 0.0011 |

  • Standalone SC-SIC achieves statistically significant gains in learning efficiency (S-AUC1000\text{S-AUC}_{1000}, Wilcoxon p=0.0223p = 0.0223) over Fixed-25mm and outperforms it on 6 of 7 tasks. On complex bimanual tasks like Put off Alcohol Lamp and Cap the Pen, applying spatial constraints under an unadapted reset schedule may mildly restrict late-stage exploration. Combining SC-SIC with frontier resets (DR-RMS) in CARAT may help alleviate this bottleneck and yields overall gains in both aggregate metrics (see our follow-up comment below for a detailed task-level analysis).


3. True End-to-End Task Success Rates from Task Onset t=0t=0 (W3 & Q3)

  • We have added the evaluation of complete task execution initialized strictly from task onset (t=0t=0) across all 28 task-hand pairs in Table 3 (Section 4.4) of the revised manuscript: | Method | S-AUC${1000}$ (%) | S-SR${E1000}$ (%) | S-SR$_{t=0}$ (%) | | :--- | :---: | :---: | :---: | | DexMachina$^*$ | 28.2 | 33.6 | 13.7 | | ManipTrans | 36.3 | 41.0 | 16.4 | | SC-SIC | 38.2 | 42.4 | 22.7 | | DR-RMS | 38.2 | 44.7 | 21.2 | | CARAT | 40.6 | 46.7 | 23.2 |

  • Executing full trajectories from t=0t=0, standalone SC-SIC and DR-RMS clearly outperform ManipTrans, while CARAT boosts S-SRt=0\text{S-SR}_{t=0} from 16.4%16.4\% to 23.2%23.2\%.


4. Handcrafted Parameter Design and Adaptability to New Embodiments/Objects (W4)

  • While CARAT uses rule-based descriptors and a coarse B=4B=4 partition, all weights (ωsize,ωL,ωC)=(0.5,1.0,1.0)(\omega_{\text{size}}, \omega_L, \omega_C) = (0.5, 1.0, 1.0) and α=0.7\alpha = 0.7 are fixed globally. Extending CARAT to new objects or tasks requires near-zero manual re-engineering since contact geometry and temporal quartiles are extracted automatically. Onboarding a new hand requires only its URDF and the canonical landmarks required by ManipTrans retargeting (Section 3.3).

Further Discussion on Task-Level Performance and Component Synergy (Addendum to W2 & Q2)

To further address your question regarding the task-level behavior of SC-SIC and its margin over Fixed-ρ = 25 mm, we provide a detailed breakdown based on our experimental results:

1. Consistent Advantage of Standalone SC-SIC Over Fixed Tolerance

  • Even in isolation, SC-SIC outperforms Fixed-ρ = 25 mm in learning efficiency (S-AUC$_{1000}$) on 6 out of 7 tasks (Table 1). On the two tasks you highlighted, SC-SIC still demonstrates advantages:

    • On Cap the Pen, although mid-training variance slightly lowers S-AUC1000, SC-SIC achieves a higher final success rate (S-SR$_{E1000}$) than Fixed-ρ = 25 mm (36.9% vs. 36.5%, Table 1).
    • On Put off Alcohol Lamp, Fixed-ρ = 25 mm suffers a severe performance drop compared to the ManipTrans baseline (39.6% vs. 44.9%), whereas SC-SIC largely avoids this degradation (43.1% vs. 39.6%, a +3.5 pp gain over fixed tolerance).
  • Moreover, under true end-to-end execution from task onset (t=0t = 0, Table 3), standalone SC-SIC boosts S-SR$_{t=0}$ from 16.4% (ManipTrans) to 22.7% (+6.3 pp), as calibrated fingertip constraints alleviate cumulative contact drift over full rollouts.

2. Why the 28-Pair Average Margin Over Fixed-ρ = 25 mm Appears Modest

  • Fixed-ρ = 25 mm is not an arbitrary constant, but the post-hoc optimal fixed radius selected via a multi-seed RL sweep across 12 task-hand pairs (Section 4.2 and Appendix D, Table 7). In contrast, SC-SIC predicts ρ\rho offline before training without per-pair reward tuning.

  • As shown in Table 8 (Appendix D), because 25 mm lies near the center of the [10, 40] mm range, roughly a third of the task-hand pairs with moderate compatibility map to radii near 21-29 mm. On these pairs, SC-SIC and Fixed-ρ = 25 mm behave similarly, which dilutes the 28-pair average difference. The primary value of SC-SIC lies in automatically tightening constraints for high-compatibility anthropometric pairs while relaxing them for low-compatibility hands.

3. Why Standalone SC-SIC Dips on Put off Alcohol Lamp and How DR-RMS Resolves It

  • We agree that on complex bimanual tasks like Put off Alcohol Lamp, adding spatial constraints alone slightly lowers S-AUC$_{1000}$ relative to the unconstrained baseline (43.1% vs. 44.9%). When the reset distribution remains unadapted, enforcing fingertip penalties during delicate bimanual transitions may mildly restrict late-stage exploration.

  • However, this phenomenon validates our motivation in Section 1:

    • Spatial interaction calibration (SC-SIC) and temporal training allocation (DR-RMS) are complementary and should be jointly modeled. Once DR-RMS concentrates episode resets at the current learning frontier, the policy receives sufficient training exposure to master the constrained interaction phase.
    • Consequently, full CARAT boosts S-SR$_{E1000}$ on Put off Alcohol Lamp / Inspire from 66.9% to 82.2% (+15.3 pp) and on Cap the Pen / Artimano from 70.7% to 79.9% (+9.2 pp) (Figure 7), achieving the highest overall performance across all metrics (Table 3).
Area_Chair_9DiAOpenReview ↗

Reminder to respond to the authors’ rebuttal

With one day left in the rebuttal period, please read the authors’ response and leave a comment. It would help the authors to know which of your concerns they have addressed and which questions remain open, so they can make the most of the time left.

Thanks for your time. Your constructive feedback helps the authors improve their work and strengthens the ICLR community.

Best regards,

Area Chair

Reviewer_grkPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CARAT, a framework for human-to-robot dexterous manipulation transfer that introduces two mechanisms: Static Compatibility (SC) and Dynamic Readiness (DR). SC estimates task-hand-specific discrepancies in morphology and interaction patterns to adapt the tolerance of an imitation reward. DR estimates the policy's competence across trajectory phases and adaptively allocates training resets toward the current learning frontier. Experiments were conducted across 7 tasks and 4 robotic hands.

Critical strengths
  • The paper addresses two relevant challenges in human-to-robot transfer: differences in embodiment compatibility and varying learning difficulty across trajectory phases.
  • SC and DR operate on different aspects of policy learning (reward shaping and training allocation), and their combination is interesting.
  • The experiments cover 28 task-hand combinations, with ablation studies showing that both components contribute to the overall performance improvement.
Critical weaknesses
  • The framework already employs IK-based retargeting to construct robot reference trajectories. Why do we need a complicated hand-crafted compatibility score instead of simply measuring the residual error after IK? The proposed morphology and interaction discrepancies may contain redundant information already captured by retargeting errors. An ablation comparing the proposed metric against simpler IK-error-based alternatives would be important to justify its necessity.
  • The proposed frontier-guided reset scheduling is closely related to existing competence-based curriculum learning and adaptive initialization methods. The paper would benefit from clearer comparisons with these approaches.
  • All experiments are conducted in simulation, without real-world validation. Moreover, the evaluation measures task completion from sampled intermediate reference states rather than consistently from the beginning of each task, which limits conclusions about full-task execution.
Questions for authors
  • Since the framework already uses optimization-based kinematic retargeting to construct robot reference trajectories, why is an additional hand-crafted compatibility metric necessary?
  • How does the proposed learning-frontier scheduler fundamentally differ from existing competence-based curriculum learning or adaptive reset strategies?
  • Since evaluation episodes start from sampled reference phases, how well does CARAT perform when executing complete manipulation tasks from the beginning, without intermediate resets?
2 comments

Thank you for your constructive comments and valuable suggestions. We address each of your questions point by point.


1. Necessity of Static Compatibility Beyond IK Residual Error (W1 & Q1)

Residual errors from kinematic retargeting alone cannot reflect whether a robot hand can dynamically sustain human-like contacts under physical interaction. We have clarified this motivation in Section 3.4 of the revised manuscript (blue part).

  • Physical Rationale: During offline IK retargeting, Shadow, Artimano, and Inspire may all exhibit similarly small landmark pose errors. However, during dynamic policy learning, high degrees of freedom and fine articulation allow Shadow and Artimano to make subtle in-hand adjustments to preserve human-like contact patterns while stabilizing the object. In contrast, the Inspire hand has only 6 independent actuators coupled across 12 joints. Even when its pose error is small, its limited degrees of freedom may prevent it from maintaining the exact human contact pattern while keeping the object dynamically stable, thus requiring a relaxed tolerance to find alternative contacts. This capability gap depends on intrinsic hand mobility, scale, and local contact states, which pure IK positional residuals cannot capture.

  • Empirical Ablation: We added an ablation study in Appendix D (Table 7, blue part) on the 12-pair subset from Fig. 5, comparing Static Compatibility against a variant that calibrates ρ\rho from post-IK positional residuals only: | Metric Used to Calibrate Tolerance ρ\rho | Projected Pairs (10/25/40 mm) | Online AUC Regret (pp) ↓\downarrow | | :--- | :---: | :---: | | IK Residual Error Only | 9 / 3 / 0 | 3.056 | | Static Compatibility (DSD_S) | 4 / 5 / 3 | 1.825 |

    • Because IK residuals are often small even for underactuated hands, IK Residual Error Only overly assigns 9 of the 12 pairs to the tightest 10 mm tolerance, resulting in a larger performance gap (3.056 pp regret) relative to each pair's best fixed tolerance. Incorporating morphology and contact discrepancies in DSD_S better distributes the tolerances (4 / 5 / 3) and reduces calibration regret to 1.825 pp.

2. Differences Between Frontier-Guided Reset Scheduling (DR-RMS) and Competence-Based Curricula (W2 & Q2)

We fully agree that DR-RMS is conceptually an adaptive, competence-based curriculum. As discussed in Sections 2.2 and 3.5, it differs from existing curriculum and adaptive reset methods at two levels:

  • Comparison with Direct Transfer Curricula: Existing human-to-robot transfer methods in our domain, such as ManipTrans (CVPR 2025) and DexMachina (ICML 2026), do not explicitly use phase-wise competence estimates to redistribute resets along the trajectory.
  • Mechanistic Differences from Adaptive Reset Methods: While related tracking works cited in Section 2.2 (DemoBot, ICRA 2026; ConTrack, ECCV 2026) also adapt mid-trajectory resets, DR-RMS differs in both competence estimation and reset allocation:
    • Look-Ahead Competence: DemoBot and ConTrack guide resets using stage success or continuation statistics under current training conditions. In our training setup, tracking thresholds progressively tighten, so success under early relaxed thresholds can overestimate readiness for the upcoming stricter criterion. DR-RMS addresses this via a zero-rollout look-ahead check (Section 3.5.1) that tests whether recorded trajectories already satisfy the upcoming tighter threshold.
    • Local Frontier Mixture: Rather than advancing through hard stage gates (DemoBot) or restricting resets to policy-reachable states (ConTrack), DR-RMS uses a Wilson lower confidence bound (Sections 3.5.2–3.5.3) to locate the sharpest competence drop between an unmastered phase and a competent successor phase, injecting 20% resets at this frontier while retaining 80% baseline coverage across the full trajectory.

3. Complete Task Execution from t=0t=0 and Real-World Validation (W3 & Q3)

  • Full-Trajectory Evaluation (t=0t=0): We added the evaluation of complete task execution initialized strictly from task onset (t=0t=0) across all 28 task-hand pairs in Table 3 (Section 4.4). | Method | S-AUC${1000}$ (%)| S-SR${E1000}$ (%)| S-SR$_{t=0}$ (%)| | :--- | :---: | :---: | :---: | | DexMachina$^*$ | 28.2 | 33.6 | 13.7 | | ManipTrans | 36.3 | 41.0 | 16.4 | | SC-SIC | 38.2 | 42.4 | 22.7 | | DR-RMS | 38.2 | 44.7 | 21.2 | | CARAT | 40.6 | 46.7 | 23.2 |

  • Real-World Hardware Validation: Following the open-loop deployment protocol of ManipTrans (CVPR 2025), we conducted physical tests using a single robotic arm and an Inspire Hand on two representative single-hand tasks (Brush Whiteboard and Rearrange). As shown in Section 4.5 of the revised manuscript and the updated video in the Supplementary Material, CARAT successfully executes these transferred trajectories in the real world.

Area_Chair_9DiAOpenReview ↗

Reminder to respond to the authors’ rebuttal

With one day left in the rebuttal period, please read the authors’ response and leave a comment. It would help the authors to know which of your concerns they have addressed and which questions remain open, so they can make the most of the time left.

Thanks for your time. Your constructive feedback helps the authors improve their work and strengthens the ICLR community.

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.