acceptodds
Withdrawn from ICLR 2027

Frame-Level Temporal Alignment for Human-to-Robot Visual Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Transferring visual representations pretrained on human videos to robot manipulation requires learning reliable correspondences between human and robot demonstrations. However, paired demonstrations can differ in execution rate and in the proportion of non-key frames that do not directly reflect task progress. Frames at the same relative timestamp may therefore represent different task stages, making correspondence learning failed. To address these issues, we propose Frame-Level Temporal Alignment (FLTA), a framework that uses two temporal priors to adapt visual encoders pretrained on human videos for robot manipulation. It learns shared task-progress representations without frame-level correspondence annotations, allowing frames at different relative temporal positions to match. A global progress prior combines normalized temporal positions with visual similarity to construct a soft correspondence target. A local temporal order prior penalizes backward transitions while allowing stays and varying forward rates to accommodate execution-rate differences. With ResNet-50 and ViT encoders, our method achieves relative improvements of 46.93% and 65.96%, respectively, over the best baselines in average simulation success rates and achieves higher task success rates on real-world manipulation tasks. These results also suggest that effective human-robot adaptation depends less on the number of parameters updated than on which parameters are selected. Our project page is available at https://rtx5090ultra.github.io/FLTA-Project-Page/.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Human-to-Robot Transfer, Temporal Alignment, Visual Representation Learning, Robotic Manipulation, Self-Supervised Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_sQpCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FLTA adapts human-video-pretrained visual encoders using task-level paired human and robot demonstrations without frame-level correspondence annotations. It combines visual similarity, a normalized temporal-position prior, and a local ordering regularizer to learn soft cross-domain correspondences, supported by frame discrimination. Only normalization-layer affine parameters are updated. Evaluation covers multiple encoder initializations, unseen simulation tasks, real-robot manipulation, and task-progress retrieval. Presentation

Critical strengths

1.The problem is clearly defined and supported by dataset statistics. The paper quantifies differences in execution speed and pre-action redundancy between human and robot videos, providing an empirical basis for frame-level alignment.

2.The adaptation procedure is simple and parameter-efficient. It requires no frame-level annotations and updates only normalization-layer affine parameters. Its modest parameter and computational requirements increase its practical value.

3.The evaluation provides complementary evidence. Multiple encoder initializations, real-robot experiments, task-progress retrieval, and alignment-loss ablations support effectiveness. The matched-adapter comparison with HRAlign also provides evidence for the contribution of the alignment objective.

Critical weaknesses

1.The matching mechanism remains in tension with the redundancy problem. The global prior favors similar normalized temporal positions, while Sinkhorn normalization approximately enforces balanced matching. Substantially different proportions of irrelevant frames may therefore force unrelated frames into correspondence. Their matching quality is not directly analyzed.

2.Cross-task gains are concentrated. In Table 9, lid opening contributes substantially to Level-2 improvements, while several tasks remain near zero success. Large relative improvements in average success do not alone establish broad cross-task generalization.

3.Uncertainty estimates do not capture training randomness. The appendix states that standard deviations come from three evaluations of the same checkpoint rather than independent training runs. These estimates therefore do not establish robustness to adaptation and policy-training randomness.

4.The contribution of parameter selection is not fully isolated. The paper compares HRAlign and FLTA with matched adapters, and compares adapter versus normalization-affine adaptation within FLTA. However, HRAlign with the same normalization-affine adaptation is missing, leaving the effects of the objective, parameter selection, and their interaction partially unresolved.

Questions for authors
  1. Method clarification: How does balanced Sinkhorn matching avoid incorrect correspondences when redundant-frame proportions differ substantially? Have the assigned weights for these frames been examined?

  2. Validation: What are the Level-2 and overall improvements after excluding lid opening? This would clarify the breadth of the gains.

  3. Robustness clarification: Are independent training seeds or more extensive downstream training results available? Does the advantage over HRAlign persist?

Reviewer_pfxzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Frame-Level Temporal Alignment (FLTA) to adapt pretrained visual representations for robotic manipulation, addressing differences in execution speed and the proportion of non-key frames between human and robot demonstrations. Using task-paired human and robot videos, the method combines global task progress and local temporal relationships to learn cross-domain frame correspondences without frame-level correspondence annotations.

The experiments cover generalization to unseen tasks on the AGNOSTOS benchmark, real-world robotic manipulation, and cross-domain task-stage retrieval. The paper also compares different pretrained encoders, method components, and adaptation strategies through comparative and ablation studies to evaluate effectiveness and parameter efficiency.

Critical strengths
  1. The paper has a compelling motivation: using human data to support robot learning. Its design addresses the mismatch between human and robot actions and enables representation transfer without frame-level annotations.

  2. The proposed method is supported by experiments and outperforms the baselines.

Critical weaknesses
  1. Although the method does not require frame-level annotations, I have concerns about the criterion used to match human and robot actions. Is visual-feature similarity a suitable basis for establishing these correspondences? If the goal is to learn better visual representations, relying on the visual representations themselves to construct the loss raises questions about the reliability of the learning signal.

  2. The evaluated tasks are relatively simple, consisting mainly of short-horizon pick-and-place tasks. Can the method also be evaluated on long-horizon tasks?

  3. Recent foundation models have strong multimodal understanding capabilities and can post-process human and robot videos, including identifying key frames and transferring knowledge. Earlier work on learning from human demonstrations has also shown that joint pretraining on large amounts of human and robot data can yield strong representations. What advantages does this method offer over recent work?

Questions for authors
  1. Please provide additional experiments with quantitative and qualitative evaluations to demonstrate that the proposed method can reliably match key frames.

  2. Please evaluate the method on more challenging tasks.

  3. Please compare against recent work on learning from human demonstrations, such as human-to-robot, EgoScale, and the Astra model, which uses a single human demonstration to directly guide robot execution.

Reviewer_LmmQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper adapts human-video-pretrained visual encoders (R3M-RN50, D4R-ViT, DecisionNCE-CLIP, AcTOL-CLIP) to robot manipulation using ~64K paired human–robot videos from RH20T, without frame-level labels. Eight frames are sampled per video. A soft correspondence target is built by Sinkhorn normalisation of visual similarity plus a global progress prior D on normalised temporal position (Eq. 1); the predicted matching matrix is trained toward it by cross-entropy, with a local order prior Q penalising backward transitions (Eq. 2–5) and an in-domain InfoNCE term. Only normalisation-affine parameters are updated. The frozen adapted encoder is evaluated with a reduced RVT-2 on the 23 unseen AGNOSTOS tasks, with ACT on four real WidowX tasks, and by human-to-robot task-progress retrieval. The paper reports +66% (D4R-ViT vs HRP) and +48% (R3M-RN50 vs HRAlign) relative gains in average zero-shot success, higher real-world success than Frozen/HRP, and ablations supporting D, Q and InfoNCE.

Critical strengths
  1. The comparison protocol is clean and the direction of the effect is consistent: the same ordering (Frozen ≤ HRP/HRAlign < Ours) appears for all four encoder initialisations (Table 1), on the real robot (Table 2) and in the retrieval test (Table 3), and the component ablations (Table 4) are complete. Adaptation takes ~13 GPU-hours on one GPU, so the work is easy to reproduce.
  2. Moving human-to-robot encoder adaptation from video-level to frame-level alignment is a natural and well-motivated step beyond HRAlign.
Critical weaknesses
  1. The reported metrics overstate what the results show. The relative gains in the abstract (+65.96%, +46.93%) are computed against baselines whose average success is 10–12%, and they are carried by a handful of tasks. In Table 9, 11 of the 23 unseen tasks are at or below 5% for every method and USB is at 93–100% for every method. Counting tasks, FLTA with D4R-ViT beats HRP on 7 tasks, loses on 9 and ties on 7; with R3M-RN50 it beats HRAlign on 8, loses on 5 and ties on 10. The entire Level-2 gain comes from "Take Lid off Saucepan" (0→85.33, 0→84.00, 5.33→69.33, 2.67→94.67 across the four encoders); without this task the Level-2 averages are 12.15 vs 13.48 (HRP), 12.30 vs 14.21 (HRAlign), 10.22 vs 9.93 and 13.63 vs 13.33, i.e. not better than the baselines. The standard deviations in all tables are across three evaluation runs of the same checkpoint (Appendix B.1); each setting is a single adaptation and a single policy training run, so a 0→85 flip on one task cannot be separated from training noise. The abstract also quotes 46.93% (vs Frozen R3M) while Sec. 3.1 quotes 48.27% (vs HRAlign) for the same number, which hides that the reproduced HRAlign is below the frozen encoder (12.12 vs 12.23); the AGNOSTOS paper itself evaluates HRAlign's released R3M checkpoint at 9.7 (Frozen) → 12.2 (HRAlign), so the main baseline appears to be inactive in this setup. Please report per-task results in the main text, at least three training seeds, and the gap to the best fixed baseline per task.

  2. Related work omits the human-to-robot paradigms the paper should be positioned against. (a) Two established ways of using human data for robot manipulation are absent: co-training human and robot data (EgoMimic, ICRA 2025; Humanoid Policy ~ Human Policy, CoRL 2025; DexWild, RSS 2025; EgoBridge, NeurIPS 2025; SimHum, CoRL 2026) and data editing that renders human videos as robot videos (Phantom, CoRL 2025; Masquerade, ICRA 2026). These lines use the same kind of paired or human data to close the human–robot gap at the data or policy level, so the paper needs to say why representation-level alignment is preferable, and Masquerade produces a pretrained encoder that could be compared under the same frozen-encoder protocol. (b) HARP-VLA is mischaracterised: it already performs frame-level temporal alignment on RH20T pairs via DTW on hand keypoints, whereas the paper says only that it "highlights execution-rate differences and redundant frames" in its limitations.

  3. The narrative contradicts the method and the paper's own data. (a) The motivation says frames at the same relative time show different stages, yet the prior D = |t−s|/(|T|−1) with ρ/ε = 10 down-weights a one-index offset by 0.24 and a two-index offset by 0.057, i.e. it enforces same-relative-position matching; that removing D hurts (Table 4) suggests the assumption the paper argues against largely holds. The headline statistic (robot execution 2.12× longer, Fig. 6) is also irrelevant after uniform resampling to 8 frames, which removes any global rate difference. (b) Appendix A.1 reports 2.6% leading non-key frames in robot videos, but the authors' own annotations (Table 10) show 18–66% (mean ≈ 41%, similar to the human 44%), so the asymmetry motivating D is absent from the paper's own data. (c) The adaptation data overlap the "unseen" tasks: RH20T contains "Cover the pot with the lid" (task 0222, annotated in Table 10) and "Close the microwave door" (0078), the two tasks with the largest gains, and the paper does not say whether overlapping tasks were removed.

Questions for authors
  1. Are all simulation numbers from one adaptation run and one policy run per setting? Please report ≥3 training seeds with per-task results; does the Lid improvement persist?
  2. Was HRAlign's released R3M checkpoint used? Why is it below Frozen here when the AGNOSTOS paper measures the released checkpoint at 9.7 → 12.2? Please also report HRAlign for D4R-ViT and on the real robot.
  3. Which RH20T tasks were used, and were tasks overlapping AGNOSTOS test tasks (0222 lid, 0078 microwave, 0010 phone, 0011 plant, 0128 USB, 0112 charger) removed? Please report an adaptation run with them excluded.
  4. How do you reconcile the 2.6% robot non-key-frame statistic (A.1) with the 18–66% leading segments in Table 10, and what statistic shows that stage proportions (not global rate) differ after uniform resampling?
  5. Given ρ/ε = 10, how is D consistent with "allowing frames at different relative positions to match"? Please report the distribution of |t − argmax_s P_ts| on held-out pairs.
  6. Please add a control that updates the same normalisation-affine parameters with LInfoNCE only (no cross-domain alignment); "w/o D & Q" still trains cross-domain correspondences.
  7. Discuss the co-training and data-editing lines of human-to-robot learning and position the method against them; state 46.93% and 48.27% against their respective baselines; mark the R3M HRAlign row as reproduced; describe HARP-VLA accurately.
Reviewer_SAkqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Frame-Level Temporal Alignment (FLTA), which adapts visual encoders pretrained on human videos to robotic manipulation using paired human-robot demonstrations. FLTA constructs soft frame correspondences through Sinkhorn normalization based on visual similarity and normalized temporal distance. It further introduces a local temporal regularizer that penalizes backward matching and an intra-domain InfoNCE loss for frame discrimination. Experiments in simulation and real-world settings demonstrate the effectiveness of the proposed approach.

Critical strengths

Valuable Problem. Transferring visual representations from large-scale human videos to robotic manipulation is an important problem with substantial practical value. Efficient Adaptation. FLTA updates only the affine parameters of normalization layers, resulting in relatively low training costs. The method also demonstrates improvements across ViT-, ResNet-, and CLIP-based initializations.

Critical weaknesses
  1. Table 9 shows an unusually large improvement on the Level-2 Lid task, where success rates increase from nearly zero to approximately 70%-95% across different encoders. Comparable improvements are not observed on most other tasks. This raises concerns about whether the reported aggregate gains reflect general improvements or task-specific effects.
  2. The real-world experiments use ACT as the only downstream policy. It remains unclear whether FLTA can provide similar benefits when integrated with more recent robot-policy architectures, such as the π0 and π0.5 families.
  3. The reported standard deviations are computed from three evaluations of the same trained policy checkpoint. They therefore do not capture randomness in representation adaptation or downstream policy training. Experiments with independently trained policies and different random seeds are needed to establish statistical reliability.
Questions for authors
  1. Why is D4R-ViT compared only with HRP in Table 1, whereas the other encoder initializations are compared with HRAlign? Similarly, why is HRAlign excluded from the real-world evaluation in Table 2? Please clarify the baseline-selection criteria and provide comparisons under matched data and adaptation settings.
  2. How many annotators participated in labeling task-progress stages for the retrieval evaluation? How were annotation consistency, scientific validity, and disagreements assessed or adjudicated?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.