acceptodds
Under review as a conference paper at ICLR 2027

Human-to-Robot Transfer via Bridging Representation and Action Space in VLAs

Abstract

Pretrained Vision-Language-Action (VLA) policies provide a strong starting point for deployment, but adapting them to a target robot and environment still requires costly embodiment-specific demonstrations. Human demonstrations are far easier to collect, yet directly leveraging them is challenging due to the substantial human-robot embodiment gap, particularly when the underlying VLA has been pretrained primarily on robot data. We introduce **H2RB** (**H**uman-**T**o-**R**obot **B**ridge), a human-robot co-training framework for adapting pretrained VLA policies from unpaired human and robot demonstrations. H2RB addresses the embodiment mismatch at both the representation and action levels: Hand geometry-anchored alignment brings human and robot observations into a compatible task representation, while a decoupled-decoder action tokenizer shares action latents but embodiment-specific action generation. This design allows human demonstrations to provide useful behavioral supervision while retaining the rich visual, semantic, and control priors of the pretrained VLA. Across real-world human-to-robot transfer tasks, H2RB substantially improves object transfer performance from 8.0 to 51.8 and enables skill transfer to score 62.7, in settings where co-train baseline methods totally fail to transfer human-demonstrated skills. The benefit also persists under object-placement shifts, improving out-of-distribution transfer from 6.0 to 45.3. H2RB suggests a way for human demonstrations to serve as a scalable supervision source for adapting pretrained robotic policies, substantially reducing the dependence on costly target-robot data collection.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Human-to-robot transfer, Vision-language-action models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3aEbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents H2RB, a framework for adapting a robot-pretrained VLA from unpaired human and robot demonstrations. The method has two main components: (1) hand geometry-anchored alignment, which grounds human and robot observations through a shared hand-geometry target without paired data, and (2) a decoupled-decoder latent action tokenizer, which shares a latent action-prediction space while preserving embodiment-specific decoders for human and robot motion dynamics. The authors evaluate on a real OpenArm bimanual platform (using one arm/hand) on object transfer (Box/Ball/Towel pick-and-place introduced via human demonstrations) and skill transfer (air-fryer Open/Close absent from robot source data). H2RB substantially improves over shared-action baselines, e.g., object transfer average 51.8 vs. 27.2 for Ego-Pi-shared action, and skill transfer 62.7 vs. 0.0. Additional experiments study ID/OOD placement shift, heterogeneous mid-training, instruction conditioning, latent-gap analysis, and execution speed.

Critical strengths

Real-world non-paired human-to-robot transfer with a meaningful empirical gap. The paper demonstrates actual robot rollouts on object and skill transfer, and the gains over shared-action baselines are large, especially for skill transfer (0.0 → 62.7). This is consequential because it supports the central claim that sharing a latent action space while preserving embodiment-specific decoding can transfer human-demonstrated task semantics without paired human-robot data.

The method decomposition is conceptually sensible and partially supported by analysis. The separation between hand-geometry anchoring for perception and embodiment-specific decoding for action realization is a reasonable way to address the human-robot gap. The latent-gap reduction from 3.81 to 1.53 and the tokenizer/regression ablation are useful evidence that the learned latent space becomes more compatible across embodiments.

Critical weaknesses

The empirical evidence is too narrow and statistically weak for the strength of the claims. All real-robot results appear to come from a single robot platform, one arm/hand configuration, a small set of objects and skills, one independently trained policy per condition, and limited rollout counts without reported variance, confidence intervals, or significance tests. This is consequential because the paper claims general human-to-robot transfer and scaling to heterogeneous data, but the evidence does not establish robustness across seeds, tasks, embodiments, or evaluation conditions.

Baseline fairness and component attribution are not fully established. The main comparisons are mainly against HaWoR-shared action and Ego-Pi-shared action; stronger cross-embodiment or human-video baselines are not compared. The method combines several components—geometry anchoring, latent tokenizer, cross-domain regression, decoupled decoders, and instruction conditioning—and the ablation does not fully isolate their contributions across both object and skill transfer. In addition, H2RB’s object-transfer in-domain Doll PnP score is 49.0 versus 55.0 for the in-domain-only baseline, so the claim that in-domain robot performance is “preserved” needs clarification.

Questions for authors

In Table 1, H2RB obtains 49.0 on in-domain Doll PnP while the in-domain-only baseline obtains 55.0. Is this difference within noise, and how do you justify the claim that in-domain robot performance is preserved?

Were the HaWoR-shared-action and Ego-Pi-shared-action baselines tuned with the same backbone, data, compute, and hyperparameter search budget? Please clarify any baseline-specific tuning.

Can you provide a component-wise ablation isolating hand-geometry alignment, latent tokenizer, cross-domain regression, decoupled decoders, and instruction renewal on both object and skill transfer?

Why is the human decoder used at inference for the main transfer results? Is this required for deployment, and how sensitive are results to decoder selection?

4 comments

[W1] The limited statistical reliability, task diversity, and embodiment coverage leave the robustness and scalability of H2RB insufficiently established.

H2RB already demonstrates full-task transfer, out-of-distribution generalization, and the benefits of heterogeneous mid-training in the original evaluation (Tables 3, 12; Sec. 3.3). We further strengthen this evidence through independent training seeds, additional target tasks, and a distinct robot embodiment.

[W1-a] Statistical reliability. The original evaluation already demonstrates improvements in strict task completion over Ego-Pi on both object (12.2% vs. 2.2%) and skill transfer (30.0% vs. 0.0%; Table 12).

To further assess reliability, we extend policy fine-tuning to independent training seeds while keeping the tokenizer fixed. We report seed-wise performance and task- and placement-stratified bootstrap confidence intervals, distinguishing rollout-level uncertainty from training-seed variability. Across all three seeds, H2RB consistently outperforms the originally reported Ego-Pi baseline in strict task completion, providing further evidence that the transfer gains are not specific to the original fine-tuning seed. We have added the seed-wise results and bootstrap confidence intervals to Appendix Table 15.

Training seedObject weighted scoreObject strict successSkill weighted scoreSkill strict success
42 (original)51.812.2% (11/90)62.730.0% (9/30)
4349.011.1% (10/90)60.033.3% (10/30)
4455.820.0% (18/90)65.340.0% (12/30)
Mean ± SD52.2 ± 3.414.4 ± 4.8%62.7 ± 2.734.4 ± 5.1%
95% Bootstrap CI[49.9, 54.5][11.9%, 17.8%][57.6, 67.0][27.8%, 41.1%]

[W1-b] Evaluation diversity and generalization. We already demonstrated generalization to unseen object placements, improving out-of-distribution transfer from 21.3 to 45.3 over Ego-Pi (Table 3). We next extend evaluation beyond the original Box/Ball/Towel PnP and Open/Close tasks to Coffee PnP and Pour Coffee, while also testing changes in workspace geometry. H2RB consistently outperforms the evaluated baselines across these additional settings. We have added the new task and table-height comparisons to Appendix Tables 16 and 17, with the data collection setup in Appendix Figure 9.

MethodTable-height shiftCoffee PnPPour Coffee
Naive co-training10.018.00.0
HaWoR13.027.00.0
Ego-Pi31.051.00.0
H2RB45.088.054.0

Weighted task-progress scores (0-100). The table-height shift (79 -> 89 cm) changes camera-to-workspace geometry and reachability.

We additionally evaluate visual background variations by replacing the original tabletop covering with a checkered tablecloth. H2RB achieves 47.3 compared with 51.8 under nominal conditions, showing a 4.5-point decrease under this appearance shift. Together, these additional evaluations broaden the evidence for H2RB beyond the original tasks and evaluation conditions, demonstrating effective transfer to new manipulation behaviors and resilience to the tested geometric and visual appearance changes. We have added the quantitative background shift results to Appendix Table 17(b) and the qualitative evaluation sequences to Appendix Figure 10.


[W1-c] Heterogeneous scaling across embodiments. We already demonstrated that heterogeneous human-robot mid-training improves object and skill transfer on OpenArm (Sec. 3.3). We further confirm that these gains extend to full-task completion, with strict success increasing from 12.2% to 21.1% for object transfer and from 30.0% to 56.7% for skill transfer. We have added the MT completion results to Appendix Table 28 and progress results to Appendix Table 12.

OpenArm evaluationH2RBH2RB + MT
Object weighted score51.856.9
Skill weighted score62.784.7
Object strict success12.2%21.1%
Skill strict success30.0%56.7%

We additionally evaluate RB-Y1, a distinct bimanual robot with a 58-dimensional action interface. Human-robot MT improves parcel-handling success from 64% to 72%, reaching 78% with geometry anchoring.

Parcel handlingBinary success
Robot-only heterogeneous training64%
Human-robot MT72%
Human-robot MT + Geometry78%

50 parcel-orientation trials per condition.

Unlike the OpenArm task transfer, the RB-Y1 experiment evaluates heterogeneous mid-training followed by robot-only fine-tuning, without the shared action tokenizer. This complementary result indicates that human-derived representations can also benefit adaptation to a different robot embodiment, beyond the original OpenArm setting. We have added the RB-Y1 evaluation result to Appendix Table 29 and Figure 12.

[W2-a, Q2] The fairness of shared-action baselines and the lack of broader cross-embodiment comparisons make the transfer improvements difficult to attribute.

Our shared-action comparisons already use the same robot-pretrained VLA backbone, human-robot training data, and 30k fine-tuning updates, reflecting that H2RB's gains are not attributable to differences in these conditions (Sec. 3.1; Appendix B.1). We also use the same hyperparameter-search budget across these methods. Importantly, H2RB itself relies on HaWoR-based hand reconstruction and Ego-Pi-style action retargeting, making the quality of these baseline components equally critical to our method.

  • Broader baseline comparisons. Ego-Pi directly addresses human-to-robot adaptation of pretrained VLAs. To further broaden the evaluation, we additionally compare against HAT [1] and DexWild [2], retaining their original architectures, training recipes, and training scales while matching the human-robot data combinations and multi-task evaluation protocol. (Appendix Table 13)
MethodObject transferSkill transfer
HaWoR-shared action10.70.0
Ego-Pi-shared action27.20.0
HAT-style adaptation15.60.0
DexWild-style adaptation0.00.0
H2RB51.862.7

Weighted task-progress scores (0-100). HAT/DexWild-style comparisons adapt their training principles.

Neither additional baseline surpasses Ego-Pi under our transfer protocol, while H2RB substantially outperforms all evaluated methods. These comparisons provide broader evidence for H2RB's transfer effectiveness beyond the original shared-action baselines.

[1] Qiu, Ri-Zhao, et al. "Humanoid policy~ human policy." arXiv preprint arXiv:2503.13441 (2025).
[2] Tao, Tony, et al. "Dexwild: Dexterous human interactions for in-the-wild robot policies." arXiv preprint arXiv:2505.07813 (2025).


[W2-b, Q3] The individual contributions of geometry alignment, latent action modeling, decoder architecture, and instruction conditioning are not sufficiently isolated across object and skill transfer.

We already demonstrated that geometry anchoring improves object transfer from 27.2 to 34.7, the latent tokenizer enables skill transfer at 49.7, and cross-domain regularization further improves it to 62.7 (Table 4a and Appendix Table 18). To further isolate their individual contributions, we additionally evaluate matched geometry-loss and decoder-architecture controls on both object and skill transfer. Instruction renewal is evaluated separately as an optional extension (Sec. 3.4). The staged tokenizer comparison captures the combined effect of latent-action modeling and decoupled decoding, whereas the controlled decoder comparisons isolate the architectural contribution.

Staged component ablation.

ConfigurationObject transferSkill transfer
Shared physical action27.20.0
Geometry-anchored alignment34.70.0
Latent tokenizer, without latent regularization43.349.7
Cross-domain latent regularization (Full H2RB)51.862.7

Controlled component comparisons. (Appendix Table 19)

ComponentConfigurationObjectSkill
GeometryH2RB w/o geometry loss49.757.7
GeometryH2RB w/ geometry loss51.862.7
DecoderShared33.83.3
DecoderSource-conditioned38.911.7
DecoderIndependent (H2RB)51.862.7
InstructionH2RB51.862.7
InstructionH2RB + Instruction renewal49.483.3

Object/skill values are weighted task-progress scores (0-100). Decoder comparisons use matched tokenizer and VLA training budgets.

[W2-c,Q1] The lower in-domain performance relative to robot-only fine-tuning raises concerns about whether H2RB preserves existing robot capabilities.

H2RB substantially mitigates in-domain performance degradation while acquiring new human-demonstrated behaviors, with only a 6-point decrease on Doll PnP relative to robot-only fine-tuning, compared with a 25-point decrease for Ego-Pi (Table 1). We clarify that capability preservation refers to retaining existing robot performance during human-to-robot adaptation.

While a performance gap remains on Doll PnP, H2RB substantially reduces the degradation observed with other human-robot adaptation baselines. On Box PnP, it also maintains performance comparable to robot-only fine-tuning (81.0 vs. 80.5), supporting its ability to retain existing robot capabilities while learning new target behaviors.

MethodObject: Doll PnP (In-domain)Skill: Box PnP (In-domain)
Robot-only FT55.080.5
Naive co-training18.016.5
HaWoR10.519.5
Ego-Pi30.070.5
H2RB49.081.0

Object/skill values are weighted task-progress scores (0-100).


[Q4] The necessity of human decoding at inference and its impact on object and skill transfer remain unclear.

We thank the reviewer for this question. We use the human decoder at inference because it better preserves the target motion patterns learned from human demonstrations. Both decoders produce robot-compatible wrist-hand actions, so human decoding is an empirical choice. The original object transfer decoder comparison in Appendix Table 21 shows higher performance with human decoding (51.8 vs. 32.7). We further evaluate skill transfer in Appendix Table 23, where the human decoder achieves 62.7 compared with 0.0 for the robot decoder. These results show that transfer performance is sensitive to decoder selection, supporting the use of the human decoder for human-demonstrated target tasks.

Inference decoderObject Avg.Skill Avg.
Robot decoder32.70.0
Human decoder51.862.7

Weighted task-progress scores (0-100).

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer 3aEb,

 

We greatly appreciate your careful review and constructive feedback on our manuscript.

As the discussion period approaches its end, we wanted to follow up and ask whether you have any additional questions or concerns. We would be glad to provide further clarification.

 

Best regards,

Authors

Reviewer_PNRxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

H2RB adapts a robot-pretrained VLA using unpaired human and robot demonstrations. Human motions are reconstructed and retargeted into a robot-compatible wrist–hand action space. A shared action encoder and source-specific decoders provide latent-action targets, while an auxiliary hand-geometry prediction objective adapts the policy’s conditioning representations. At deployment, robot observations condition the latent policy, and human-demonstrated target behaviors are executed through the human-source decoder. Experiments evaluate transfer to new objects and opening/closing skills without target-task robot demonstrations during adaptation.

Critical strengths

The paper demonstrates meaningful transfer on a real robot. Compared with the shared wrist–hand action baseline, full-task success improves from 2.2% to 12.2% for object transfer and from 0% to 30.0% for skill transfer. Although absolute reliability remains limited, these results show progress beyond partial execution alone. The setting is practically relevant: target behaviors are supplied through human demonstrations, without requiring paired human–robot trajectories or target-task robot demonstrations during adaptation.

Critical weaknesses
  1. The necessity and interpretation of the source-specific decoders remain unresolved.

    The tokenizer ablation introduces latent prediction, action-autoencoder training, temporal reconstruction, and separate decoders together. It therefore does not isolate the contribution of decoder separation. A matched single-decoder VAE, or a shared decoder conditioned on source identity, is an important missing control.

    More fundamentally, source and skill coverage are confounded in the skill-transfer experiment: robot demonstrations contain pick-and-place, whereas human demonstrations contain opening and closing. Both decoders ultimately produce the same robot-compatible action representation. Consequently, the human decoder’s advantage could reflect its target-skill trajectory prior rather than a separation of embodiment-specific execution dynamics.

    The inference-decoder swap is informative, but only establishes that the human branch is preferable within the trained two-decoder system. It does not establish that separate decoders are necessary or identify which factor—source, skill, or motion style—accounts for their benefit.

  2. The claimed representation alignment lacks direct sample-level validation.

    The paper presents PCA visualizations and a reduction in the aggregate human–robot latent gap. However, bringing marginal distributions closer does not establish that nearby human and robot representations correspond to similar hand configurations, interaction states, or task stages. The centroid regularization itself does not enforce such correspondence.

    I could not find cross-domain nearest-neighbor examples, retrieval accuracy, or measurements relating feature similarity to geometric or behavioral similarity. The PCA arrows represent adapter-induced changes for the same observation, rather than correspondences between human and robot examples. Likewise, predicting current hand geometry may improve visual localization without necessarily establishing shared task semantics.

    These distinctions matter because representation alignment is a central contribution. The downstream improvements demonstrate that the overall recipe is useful, but leave its proposed explanation underdetermined.

Questions for authors
  1. Controlled evidence: How does the method compare with a single-decoder latent tokenizer under the same geometry supervision, latent regularization, and tokenizer-training budget? Would a source-conditioned shared decoder retain the gains?
  2. Mechanism clarification: Can decoder specialization be evaluated with some overlapping skill coverage across human and robot sources? This would help distinguish embodiment-related benefits from target-skill priors.
  3. Representation validation: Can you show human–robot nearest neighbors in the learned feature space and assess whether they match in hand geometry, interaction state, or task stage? Quantitative correspondence evidence would materially strengthen the alignment claim.
5 comments

Dear Reviewer PNRx,

We greatly appreciate the time and effort you have devoted to evaluating our manuscript. Your insightful comments have helped us improve the clarity and quality of our work. We provide point-by-point responses below, with all revisions highlighted in blue in the updated manuscript.


[W1-a, Q1] The tokenizer ablation does not isolate the contribution of independent decoders from latent-action modeling.

We thank the reviewer for highlighting this distinction. While the original ablation evaluates the tokenizer as a whole, decoder separation is a distinct design choice whose contribution can be evaluated independently. To examine this, we additionally compare shared, source-conditioned, and independent decoders while keeping latent-action modeling and training conditions fixed.

Under these controlled conditions, independent decoding improves both object and skill transfer over shared and source-conditioned decoding. These additional results isolate the contribution of decoder architecture within the latent-action modeling framework, complementing the original tokenizer ablation. We report this result in Table 19.

Decoder architectureObject transferSkill transfer
Shared (D(z)D(z))33.83.3
Source-conditioned (D(z,s)D(z,s))38.911.7
Independent (DHD_H, DRD_R)51.862.7

Weighted task-progress scores (0-100). All variants retain the same 64-dimensional action encoder architecture, latent regularization, geometry supervision, reconstruction objectives, training data, and budgets (100k tokenizer and 30k VLA updates). The source-conditioned variant incorporates a learned source embedding into a shared decoder.


[W1-b, Q2] The benefit of source-specific decoders may arise from differences in target-skill coverage rather than embodiment-dependent action mappings.

Our reported decoder-swap experiment already shows that the human decoder outperforms the robot decoder within the shared PnP skill category (51.8 vs. 32.7, Table 21), suggesting that its advantage is not restricted to the Open/Close setting. This comparison uses the same trained latent policy and changes only the inference decoder.

Inference decoderBox PnPBall PnPTowel PnPAvg.
Robot decoder37.016.045.032.7
Human decoder53.049.053.351.8

Weighted task-progress scores (0-100) with the same trained latent policy.

To more directly control for differences in skill coverage, we additionally train shared and independent decoders on all five tasks (Box/Ball/Towel PnP, Open, and Close) from both sources. Both variants use identical tokenizer objectives and training budgets (100k tokenizer and 30k VLA updates). Importantly, robot demonstrations of the target Open/Close skills are used only to match skill coverage during tokenizer training, while VLA fine-tuning retains the original transfer protocol without target-skill robot supervision. As shown in Table 22, under matched tokenizer-stage skill coverage, independent decoding substantially outperforms shared decoding, while the human branch remains preferable under the same trained policy. These results demonstrate that independent decoding improves transfer even when task coverage is matched across sources during tokenizer training.

Decoder configurationWeighted scoreStrict success
Shared decoder (retrained)0.00.0% (0/30)
Split-Robot decoder25.010.0% (3/30)
Split-Human decoder56.336.7% (11/30)

[W2, Q3] The representation alignment analysis lacks direct sample-level evidence of correspondence in hand geometry, interaction state, and task stage.

We thank the reviewer for highlighting the need for sample-level evidence of cross-embodiment alignment. To provide more fine-grained evidence, we additionally analyze human-robot latent distances within behaviorally matched groups, categorized by task stage (approach, carry, retreat), interaction state (open, closing, holding, opening), and hand geometry (wrist-local fingertip configurations). These groups are defined directly from action trajectories without requiring paired human-robot demonstrations.

As shown in Table 20, cross-domain latent separation decreases consistently across all evaluated groups, with reductions of 27-56%. These results provide additional quantitative evidence that the alignment extends to behaviorally matched subsets, complementing the aggregate latent-gap analysis in the original manuscript.

CriterionGroupw/o reg.w/ reg.Reduction [95% CI]
Task stageApproach4.612.42−48% [46, 49]
Task stageCarry5.813.07−47% [45, 48]
Task stageRetreat4.292.12−51% [48, 53]
Interaction stateOpen3.191.78−44% [41, 46]
Interaction stateClosing5.823.45−41% [39, 42]
Interaction stateHolding6.223.61−42% [40, 43]
Interaction stateOpening5.482.83−49% [47, 49]
Hand geometryOpen, thumb abducted4.372.76−37% [35, 39]
Hand geometryOpen, thumb partially opposed5.142.25−56% [54, 57]
Hand geometryOpen, thumb opposed (pre-grasp)4.452.80−37% [35, 39]
Hand geometrySlightly flexed5.033.16−37% [34, 39]
Hand geometryHalf closed6.094.42−27% [26, 29]
AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer PNRx,

 

Thank you again for your time and valuable comments on our work.

With the discussion period coming to an end, we would like to kindly check whether there are any remaining questions or concerns. Please feel free to let us know if further clarification would be helpful.

 

Best regards,

Authors

Reviewer_YR5pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes H2RB (Human-To-Robot Bridge), a framework for transferring manipulation skills from unpaired human and robot demonstrations to pretrained vision-language-action (VLA) models. The key motivation is that existing approaches primarily address the action-space discrepancy between human and robot embodiments, while the perceptual representation gap remains insufficiently resolved.

H2RB introduces two main components: (1) hand geometry-anchored alignment, which uses wrist and fingertip geometry to align visual conditioning representations across embodiments, and (2) a decoupled-decoder latent action tokenizer, which learns a shared latent action space while maintaining separate human and robot reconstruction decoders.

The authors evaluate the method on real-world object transfer and skill transfer tasks using an OpenArm robot. H2RB improves weighted task scores from 8.0 to 51.8 for object transfer and from 0.0 to 62.7 for skill transfer compared with the standard co-training baseline. Additional experiments demonstrate improvements under object-placement distribution shifts and through heterogeneous human-robot mid-training.

Critical strengths
  1. A practically relevant problem with substantial real-world improvements.

The paper addresses an important challenge in robotics: adapting pretrained robot policies using human demonstrations without requiring robot demonstrations for each new target task. The improvements over standard human-robot co-training and shared-action baselines are substantial, particularly for skill transfer, where the co-training baselines fail to acquire the human-demonstrated opening and closing behaviors. The real-world experiments provide encouraging evidence that human demonstrations can contribute useful task knowledge to robot-pretrained VLAs.

  1. A clear decomposition of the cross-embodiment gap with useful ablation evidence.

The paper explicitly addresses both perceptual and action-level discrepancies, rather than relying solely on a shared action representation. The ablations provide evidence for the contribution of each component. In particular, Table 4 shows that the latent tokenizer enables skill transfer, while cross-domain latent regularization further improves performance. The experiments under distribution shift and with heterogeneous mid-training also strengthen the empirical evaluation.

Critical weaknesses
  1. Limited absolute task success and experimental generalizability.

Although the relative improvements are substantial, the main results are reported using weighted task-progress scores rather than full-task success rates. According to Table 12 in the appendix, H2RB achieves only 12.2% full-task success for object transfer and 30.0% for skill transfer. This suggests that reliable task completion remains challenging despite the improvements in intermediate task progress.

Moreover, the controlled experiments are conducted on a single robot platform, with a relatively small collection of target objects and skills. Human and robot demonstrations are also collected in the same environment with approximately matched camera viewpoints. It therefore remains unclear how well the approach generalizes to more substantial embodiment, viewpoint, or environmental differences.

The experimental evaluation also lacks statistical uncertainty estimates across independently trained policies. Appendix B.4 indicates that each condition uses one trained policy. Given the limited number of evaluation rollouts, additional evidence of robustness would strengthen the conclusions.

  1. The novelty and precise role of embodiment-specific action decoding could be better established.

The proposed framework combines geometry-based auxiliary supervision with a shared latent encoder and embodiment-specific decoders. These are reasonable design choices, but several related approaches already explore shared latent representations and embodiment-specific decoding. Consequently, the main novelty appears to lie primarily in the integration of these components for human-to-robot VLA adaptation.

Furthermore, the relationship between embodiment-specific decoding and robot execution knowledge remains somewhat unclear. The paper uses the human reconstruction decoder for robot deployment, and Table 14 shows that this decoder substantially outperforms the robot decoder (51.8 versus 32.7). Although the human decoder outputs robot-compatible actions, this raises questions about how the claimed preservation of robot-specific execution knowledge is achieved and why the human decoder is consistently preferable.

Finally, the tokenizer requires a separate 100k-step training stage, while policy fine-tuning is performed for 30k steps. Although the authors match the fine-tuning budgets, a comparison accounting for total computational resources would help establish the efficiency of the proposed approach.

Questions for authors

Q1. Full-task success and evaluation reliability.

Could the authors provide full-task success rates for each method and target task, rather than primarily reporting weighted task-progress scores? The relatively low full-completion rates in Table 12 make this distinction important. Additionally, could the authors provide uncertainty estimates or results across multiple independently trained policies?

Q2. Role of the human and robot action decoders.

The method uses the human decoder for robot deployment, even though the framework emphasizes preserving embodiment-specific action realization. Could the authors clarify why the human decoder significantly outperforms the robot decoder and how robot-specific execution knowledge is retained in this setting? An analysis of the generated latent actions and resulting robot trajectories under the two decoder choices would be helpful.

Q3. Training budget and generalization.

How does H2RB compare with the strongest shared-action baseline under a matched total computational budget, including tokenizer training? Additionally, how sensitive is the method to errors in human hand reconstruction, changes in camera viewpoint, or larger differences between human and robot hand embodiments?

4 comments

Dear Reviewer YR5p,

We sincerely thank you for your careful review and constructive feedback on our manuscript. We address each of your comments below. All corresponding revisions in the updated manuscript are highlighted in blue.


[W1-a, Q1] The reported weighted scores leave concerns about full-task completion and statistical reliability across independent policies.

Our evaluation distinguishes the transfer of human-demonstrated behaviors from reliable end-to-end task execution. Accordingly, our weighted scores capture observable manipulation milestones, including grasping and object movement, rather than only full completion (Appendix B.2). Full-task success additionally requires reliable execution throughout the entire manipulation sequence, which may also depend on the underlying robot policy's control capabilities. We therefore report both metrics to distinguish behavioral transfer from end-to-end execution reliability.

We report full-task success rates in the original submission (Table 12), where H2RB improves strict completion over Ego-Pi from 2.2% to 12.2% on object transfer and from 0.0% to 30.0% on skill transfer. We additionally report strict success rates after heterogeneous mid-training, whose improvements were already demonstrated through weighted scores in Sec. 3.3, showing further gains to 21.1% and 56.7%, respectively. We have added the strict completion results to Appendix Table 28 and the cumulative task progress results to Appendix Table 12.

MethodObject totalSkill total
Ego-Pi shared action2.2% (2/90)0.0% (0/30)
H2RB12.2% (11/90)30.0% (9/30)
H2RB + MT21.1% (19/90)56.7% (17/30)

[W1-b] The limited task and robot diversity leaves generalization across embodiments, viewpoints, and environments insufficiently established.

We would like to clarify that our evaluation setup considers generalization beyond nominal task conditions with heterogeneous multi-embodiment training, which trains VLAs with different embodiments and in-the-wild human videos that have diverse view points and environments. In this setup, we demonstrated that H2RB consistently improves the performance, suggesting that effectiveness of H2RB extends beyond VLAs trained on a single embodiment with a small set of tasks.

Our original heterogeneous mid-training results already demonstrate that H2RB can effectively leverage human-robot data spanning diverse embodiments, viewpoints, and environments (Sec. 3.3). We agree that broader evaluations would further strengthen the evidence for generalization. We therefore conduct additional experiments on unseen manipulation tasks and a distinct robot platform. To further address this, we conduct additional experiments as follows:

  • Additional human-demonstrated tasks. We first increase the task diversity by evaluating H2RB on two additional human-demonstrated tasks (Coffee PnP and Pour Coffee). As shown in the table below, H2RB substantially outperforms all shared-action baselines, particularly in Pour Coffee, on which all baselines fail to make meaningful progress. We have added these additional task results to Appendix Table 16.
MethodCoffee PnPPour Coffee
Naive co-training18.00.0
HaWoR shared action27.00.0
Ego-Pi shared action51.00.0
H2RB88.054.0
  • Generalization to a distinct robot embodiment. We further evaluate heterogeneous training on RB-Y1, a distinct bimanual robot with different camera configurations and deployment environments. Across 50 parcel-handling trials, human-robot MT improves binary success from 64% (robot-only MT) to 72%, reaching 78% with geometry-anchored alignment. Unlike the OpenArm target-task transfer setting, RB-Y1 uses robot-only downstream fine-tuning without the shared action tokenizer. These results further demonstrate the utility of human-derived representations across robot embodiments and environments. We have added the RB-Y1 evaluation to Appendix Table 29 and the parcel handling sequence to Appendix Figure 12.

[W1-c] Reporting statistical reliability and robustness across training seeds.

Our original evaluation already showed substantial improvements in strict task completion over Ego-Pi (12.2% vs. 2.2% for object transfer and 30.0% vs. 0.0% for skill transfer), using a fixed evaluation protocol with all scheduled rollouts included (Table 12; Appendix B.4). To further strengthen this evidence, we additionally report rollout-level bootstrap confidence intervals and repeat VLA fine-tuning with two new seeds (43 and 44, Table 15), keeping the tokenizer and evaluation protocol fixed. Across all three seeds, strict success ranges from 11.1% to 20.0% for object transfer and 30.0% to 40.0% for skill transfer. These results provide further evidence that the observed transfer gains are not specific to the original fine-tuning seed.

Training seedObject weighted scoreObject strict successSkill weighted scoreSkill strict success
42 (reported)51.812.2% (11/90)62.730.0% (9/30)
4349.011.1% (10/90)60.033.3% (10/30)
4455.820.0% (18/90)65.340.0% (12/30)
Mean ± SD52.2 ± 3.414.4 ± 4.8%62.7 ± 2.734.4 ± 5.1%
95% Bootstrap CI[49.9, 54.5][11.9%, 17.8%][57.6, 67.0][27.8%, 41.1%]

[W2-a] Novelty and contribution of H2RB.

We clarify that our main contribution is enabling robot-pretrained VLAs to acquire human-demonstrated target behaviors from unpaired human-robot demonstrations by jointly addressing perception and action gaps. While shared latent representations and separate decoders have been studied individually, sharing physical action coordinates alone does not resolve mismatches in visual conditioning and source-dependent action realization. H2RB addresses these challenges by combining geometry-anchored conditioning with shared latent-action prediction and source-specific decoding. This integration substantially improves object transfer from 27.2 to 51.8 and enables skill transfer from 0.0 to 62.7 over Ego-Pi shared action (Tables 1-2).


[W2-b, Q2] Role of embodiment-specific action decoding: how human decoder achieves preserving robot-specific execution knowledge and why human decoder is preferable

Robot-control priors are retained through the robot-pretrained VLA and continued robot-domain supervision, while the human decoder reconstructs human-demonstrated target motions in a robot-compatible action space. The latent policy learns from both human and robot demonstrations, while both decoders reconstruct the same robot-compatible wrist-hand actions for deployment. The human decoder is preferable because it more faithfully reconstructs motion patterns from human demonstrations, which provide the target behaviors absent from robot training.

We further examine this distinction by decoding identical ground-truth and policy-predicted latents through both branches, as reported in Appendix Table 24. For identical policy-predicted latents, we measure the mean wrist-trajectory difference between the two decoders separately before and after hand closure.

EvaluationHuman decoderRobot decoder
Human GT latent trajectory accuracy90.9%77.9%
Robot GT latent trajectory accuracy76.7%90.0%
Robot target transfer score51.832.7
Policy predicted latent comparisonBefore hand closureAfter hand closure
Mean wrist trajectory difference between decoders1.38 mm16.24 mm

These results show that each decoder more faithfully reconstructs motions from its corresponding source, while their generated wrist trajectories diverge substantially after hand closure. This supports source-specific motion reconstruction as the mechanism underlying decoder selection, without requiring separate robot execution policies.

[W2-c, Q3] Efficiency of the proposed approach

To account for the additional tokenizer-training cost, we evaluate the strongest shared-action baseline under a matched total computational budget. The original comparisons already use the same pretrained backbone, adaptation data, geometry supervision, and 30k VLA fine-tuning updates. Specifically, we extend the shared-action baseline to 35k updates using the same initialization and a cosine learning-rate schedule (Table 14). As shown below, H2RB retains substantial gains in both object and skill transfer under comparable total compute.

MethodTotal GPU-h (hours)Object transferSkill transfer
Shared action + Geometry (30k)31.634.70.0
Shared action + Geometry (35k)36.938.90.0
H2RB (Tokenizer 100k + FT 30k)36.351.862.7

Scores are weighted task-progress scores (0-100). H2RB's total includes tokenizer training (4.0 GPU-h), latent caching (0.3-0.7 GPU-h), and VLA fine-tuning (31.6 GPU-h). Tokenizer cost is estimated from a matched-recipe A100 run. The tokenizer is frozen after training and can be reused across subsequent policy fine-tuning runs without retraining.


[Q4] Robustness to human hand reconstruction errors, camera viewpoint shifts, and substantial human-robot embodiment differences.

We thank the reviewer for raising these robustness concerns. Our original results demonstrate transfer using reconstructed human motions and a robot-compatible hand representation, providing evidence of transfer across different hand kinematics. H2RB already demonstrates human-to-robot transfer using reconstructed human motions across different hand kinematics, while the submitted evaluation shows substantial gains under out-of-distribution object placements (45.3 vs. 21.3 for Ego-Pi, Table 3).

  • Hand reconstruction and embodiment differences. H2RB does not require identical human and robot hand configurations. Human motions are reconstructed and retargeted into a robot-compatible wrist-hand representation, while geometry anchoring provides shared physical supervision despite differences in degrees of freedom and joint structures.

  • Camera viewpoint variations. To further assess robustness to geometric differences, we additionally evaluate a 10 cm table-height mismatch between robot (79 cm) and human (89 cm) demonstrations, altering the camera-to-workspace geometry. H2RB achieves 45.0, outperforming Ego-Pi (31.0) and standard co-training (10.0). We have added the table height evaluation to Appendix Table 17(a) and the corresponding robot and human data collection setups to Appendix Figure 9.

  • Workspace background variations. We additionally evaluate a visual background shift using a checkered tablecloth, where H2RB retains a weighted score of 47.3 compared with 51.8 under nominal conditions. We have added the quantitative background shift comparison to Appendix Table 17(b) and the corresponding evaluation sequences to Appendix Figure 10.

MethodBoxBallTowelAvg.
Base co-train9.09.012.010.0
+ HaWoR Shared action12.015.012.013.0
+ Ego-pi Shared action30.027.036.031.0
+ H2RB45.036.054.045.0

Weighted task-progress scores (0-100) under the shifted table-height setting.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer YR5p,

 

We sincerely appreciate your time and thoughtful feedback on our manuscript.

As the discussion period draws to a close, we would like to kindly ask if you have any remaining questions or concerns. We would be happy to address them.

 

Best regards,

Authors

Comments on the paper

2 comments

Dear Reviewers,

As the discussion period is coming to a close, please take a moment to review the authors’ responses and revised manuscript. Kindly assess whether your concerns have been adequately addressed or if any questions remain, and update your scores accordingly, if necessary.

Thank you for your time and thoughtful evaluations.

Best regards, AC

AuthorsOpenReview ↗

General Response

Dear Reviewers and Area Chair,

 

We sincerely appreciate your valuable time and thoughtful feedback on our manuscript.

As the reviewers highlighted, our work addresses a practically important challenge in robot learning: enabling robot-pretrained VLAs to acquire new manipulation behaviors from unpaired human demonstrations (Reviewers YR5p, PNRx, 3aEb). We are encouraged that the reviewers recognized the meaningful real-world transfer improvements over shared-action baselines (Reviewers YR5p, PNRx, 3aEb), as well as the clear decomposition of perception and action gaps and useful supporting ablation evidence (Reviewers YR5p, 3aEb).

 

In our rebuttal, we have carefully addressed the reviewers’ comments and suggestions through detailed responses and additional analyses. Accordingly, we have enhanced the manuscript by incorporating the following additional analyses, experiments, and discussions:

  • Reliability and generalization. Additional training seeds, strict-success evaluations, unseen manipulation tasks, environmental shifts, and RB-Y1 heterogeneous-training experiments provide broader evidence for transfer beyond the original evaluation setting (YR5p [W1, Q1, Q4]; 3aEb [W1]; Tables 15-17, 28-29).

  • Necessity and roles of the proposed components. Controlled decoder ablations, matched human–robot skill coverage, decoder-swap analyses, and behavior-conditioned latent comparisons provide stronger evidence for the proposed architecture and clarify its underlying mechanisms (PNRx [W1, W2, Q1-Q3]; YR5p [W2, Q2]; 3aEb [W2, Q3, Q4]; Tables 18-24).

  • Fairness and capability retention. Additional baselines and total-compute-matched comparisons confirm that the transfer improvements persist under more comprehensive comparisons, while clarifying how H2RB mitigates degradation of existing robot capabilities (YR5p [W2, Q3]; 3aEb [W2, Q1-Q2]; Tables 1, 13-14).

 

Detailed responses are provided in the corresponding reviewer discussions, and all revisions are highlighted in blue\textcolor{blue}{\text{blue}} in the updated manuscript.

We sincerely appreciate your time and consideration.

 

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.