acceptodds
Under review as a conference paper at ICLR 2027

Learning Physically Grounded Latent Actions from Human and Robot Videos

Abstract

Latent-action pretraining provides a scalable way to learn robot policies from actionless videos, but existing objectives are driven primarily by visual change and therefore provide only indirect supervision of physical interaction. Tactile and force measurements can provide direct physical signals, but they are available for only a limited subset of existing human and robot videos. Our key insight is to decouple the acquisition of physical supervision from its transfer to latent-action learning: physical supervision derived from a limited corpus of instrumented interactions can be propagated to broader collections of human and robot videos without paired sensor measurements. Building on this insight, we propose PHYSLA, a Physically grounded Latent-Action framework that learns a shared, horizon-structured representation from heterogeneous physical measurements, including pressure distributions, tactile marker displacements, and joint torques, and infers these representations from causal RGB observations. During vision-language-action pretraining, the inferred physical targets and future visual prediction jointly supervise the same latent actions. Importantly, the transferred targets capture interaction-relevant latent structure rather than calibrated sensor readings, allowing physical supervision to scale beyond the instrumented corpus while requiring no tactile or torque sensing at deployment. PHYSLA achieves 98.2% average success on LIBERO, 74.0% and 64.7% on SimplerEnv’s Google Robot and WidowX settings, respectively, and 65.5% across five real-world G1 tasks. These results support the effectiveness of transferring measurement-derived physical supervision to latent-action learning from human and robot videos

Primary area:
applications to robotics, autonomy, planning
Keywords:
latent action models, robot learning, human video pretraining, visuo-tactile learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_XwZTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

PHYSLA learns shared physical representations from pressure, tactile measurements, and joint torques, then transfers them through RGB prediction to human and robot videos without sensor annotations. It combines future visual prediction and physical-target prediction to learn latent actions, and uses robot action data for policy learning. The method is evaluated on LIBERO, SimplerEnv, and real AgiBot G1 tasks, supporting the value of physically-grounded latent action.

Critical strengths
  1. The approach to extending physical supervision is valuable. By separating the acquisition and use of physical supervision, RGB prediction transfers information from limited sensor data to broader video collections. This addresses the practical imbalance between scarce physical measurements and abundant ordinary videos.
  2. The integration of heterogeneous physical information offers novelty. Pressure, tactile deformation, and joint torques are organized into shared targets with temporal structure and connected to latent-action learning from human and robot videos, providing a common interface for different sensor data sources.
  3. Joint supervision supports both learning and practical deployment. Visual and physical prediction jointly supervise latent actions, without requiring additional sensors or target generation modules at deployment. The physical-attribute probes in Table 1 and downstream results provide initial support for this design.
Critical weaknesses
  1. The sources of performance gains are not sufficiently separated. Table 10 shows that Full retains the physical loss during adaptation, whereas w/o physical does not. Their 1.5-point difference in Table 4 therefore combines pretraining and adaptation effects and cannot be attributed to physical pretraining alone. G1 and SimplerEnv also lack a w/o-physical comparison under matched pretraining conditions, leaving the sources of their larger gains unclear.
  2. The specific value of the physical targets and their temporal structure remains unclear. Table 1 includes a pressure-supervised baseline, where Table 4 only changes physical-loss routing and does not compare control performance when simple physical statistics replace the learned targets. The temporal perturbations in Appendix B.3 evaluate reconstruction in a fixed model, without establishing a downstream advantage of temporal grouping over a single window-level representation.
  3. The reliability of physical targets in new video domains is not directly established. The physical probes in Tables 1 and 9 cover EgoTouch, RH20T, and VLA-touch, whereas most policy-pretraining videos come from EgoDex, SSV2, and DROID. Section 4.3 acknowledges that RGB recovers physical information only partially, but whether this recovery holds in the latter three sources is not directly tested.
Questions for authors
  1. How much do physical pretraining and adaptation each contribute? Could the authors compare Full and w/o physical on G1 or SimplerEnv with matched pretraining data, initialization, training budget, and adaptation losses?
  2. How does target quality across domains affect latent-action learning? What evidence supports that RGB-inferred targets retain useful physical interaction information on EgoDex, SSV2, and DROID? How sensitive is downstream learning to target prediction errors?
4 comments
AuthorsOpenReview ↗

Response to Reviewer XwZT (Part 1 / 4)

We sincerely thank the reviewer for the thoughtful and constructive feedback, and for recognizing the value of transferring heterogeneous physical information to ordinary videos and latent-action learning. We greatly appreciate the suggestions on identifying the sources of improvement and evaluating the quality and design of the physical targets.

We have revised the manuscript and appendix to separate pretraining and fine-tuning effects, evaluate target-domain reliability and error sensitivity, and analyze learned physical targets and their temporal structure. Key additions and substantive revisions are highlighted in blue for easy reference. Our point-by-point responses are provided below.

Q1: How much do physical pretraining and adaptation each contribute? Could the authors compare Full and w/o physical on G1 or SimplerEnv with matched pretraining data, initialization, training budget, and adaptation losses?

A1: We thank the reviewer for identifying this attribution issue. We agree that the original comparison combines physical supervision during pretraining and fine-tuning. We therefore report separate stage-specific comparisons and extend the matched evaluation beyond LIBERO. Separating the two stages. We compare w/o physical, Physical pretrain only, and Full PhysLA. The first two differ in physical pretraining while using the same fine-tuning objective. The last two start from the same physically pretrained checkpoint and differ only in physical supervision during fine-tuning. Training data, base initialization, policy architecture, optimization budgets, and evaluation protocols are matched. The action-learning and V-JEPA-based world-prediction objectives remain enabled in all conditions.

LIBERO success rates (%).

ConditionPhysical pretrainingPhysical fine-tuningSpatialObjectGoalLongAvg.
w/o physicalOffOff96.27 ± 0.1299.33 ± 0.1296.67 ± 0.3194.53 ± 0.3196.70 ± 0.00
Physical pretrain onlyOnOff97.87 ± 0.3199.53 ± 0.1297.93 ± 0.3196.07 ± 0.1297.85 ± 0.15
Full PhysLAOnOn98.27 ± 0.1299.53 ± 0.1298.47 ± 0.1296.60 ± 0.2098.22 ± 0.13

Results are mean ± sample standard deviation over three fine-tuning seeds, with the pretrained checkpoint fixed for each method. Avg. gives equal weight to the four suites within each run. Its standard deviation is computed across the three run-level averages. Across three fine-tuning seeds, physical pretraining increases average success from 96.70% to 97.85%, and enabling physical supervision during fine-tuning further increases it to 98.22%. These correspond to gains of 1.15 and 0.37 percentage points, respectively. Full PhysLA improves over w/o physical by 1.52 percentage points, supporting the contribution of physical supervision at both stages. The zero standard deviation of the baseline average reflects offsetting changes across suites. The per-suite scores vary.

SimplerEnv success rates (%). We corrected the SimplerEnv success rates reported in the original submission by recalculating them from episode success counts. The corrected Full PhysLA averages are 74.1% on Google Robot and 64.6% on WidowX. The corresponding task-level results, ablation comparisons, and discussion have been updated in the revised manuscript.

Google Robot uses SimplerEnv’s visual-matching setting. Google Robot.

ConditionPhysical pretrainingPhysical fine-tuningPickMoveDrawerPlaceAvg.
w/o physicalOffOff85.072.566.760.271.1
Physical pretrain onlyOnOff88.776.369.062.074.0
Full PhysLAOnOn88.776.368.563.074.1

WidowX.

ConditionPhysical pretrainingPhysical fine-tuningSpoonCarrotBlockEggplantAvg.
w/o physicalOffOff79.270.812.575.059.4
Physical pretrain onlyOnOff83.375.020.879.264.6
Full PhysLAOnOn79.275.025.079.264.6

Each robot's average weights its four task groups equally. Physical pretraining raises Google Robot success from 71.1% to 74.0% and WidowX success from 59.4% to 64.6%, with higher scores in all four task groups for each robot. Physical pretraining accounts for most of the observed gains. Physical fine-tuning maintains comparable average performance on both robots, with task-dependent changes.

AuthorsOpenReview ↗

Response to Reviewer XwZT (Part 2 / 4)

G1: Full PhysLA versus w/o physical. We additionally compare Full PhysLA with w/o physical using matched pretraining data, base initialization, five-task demonstrations, fine-tuning budgets, and evaluation conditions. The w/o physical variant removes the physical objective during both pretraining and fine-tuning while retaining the action-learning and world-prediction objectives.

MethodPickFlowerDiskCupPlugAvg.
w/o physical90.076.756.743.33.354.0
Full PhysLA87.593.373.360.013.365.5

Full PhysLA raises the five-task average from 54.0% to 65.5%, an 11.5-percentage-point improvement over w/o physical. Success increases on Flower, Disk, Cup, and Plug, while Pick decreases from 90.0% to 87.5%. This comparison assesses the contribution of physical supervision in the complete method on the real robot. The average weights the five tasks equally. Each model is evaluated on 40 Pick trials and 30 trials for each remaining task.

AuthorsOpenReview ↗

Response to Reviewer XwZT (Part 3 / 4)

Q2: How does target quality across domains affect latent-action learning? What evidence supports that RGB-inferred targets retain useful physical interaction information on EgoDex, SSV2, and DROID? How sensitive is downstream learning to target prediction errors?

A2: Target-domain validation. Both models use the same 200 annotated videos: 66 DROID, 67 EgoDex, and 67 SSV2. The visual reference uses pooled frozen DINOv2 features. Probes are fitted on source training data, selected on source validation data, and evaluated on target annotations. Contact scores use the middle group. Event scores use the maximum across three groups. AUROC/AP are computed once over all valid windows per domain and task, without averaging over groups or videos.

DomainRepresentationContact AUROC / APEstablishment AUROC / APRelease AUROC / AP
DROIDPhysical RGB teacher0.83 / 0.780.72 / 0.680.69 / 0.73
DROIDFrozen DINOv20.51 / 0.580.49 / 0.330.42 / 0.25
EgoDexPhysical RGB teacher0.82 / 0.950.86 / 0.690.83 / 0.73
EgoDexFrozen DINOv20.71 / 0.770.53 / 0.570.59 / 0.74
SSV2Physical RGB teacher0.89 / 0.920.86 / 0.890.73 / 0.81
SSV2Frozen DINOv20.67 / 0.590.41 / 0.520.57 / 0.44

The physical teacher improves AUROC in all nine comparisons and AP in eight. Source and target contact labels refer to the final history frame; establishment and release labels cover the same eight-frame history window. An event is positive only if its annotated transition interval is wholly inside that window. Negatives require explicit whole-clip absence. Out-of-window, boundary-overlapping, or uncertain events are excluded. This evaluates observed-history contact and transitions, not future-target accuracy or pressure/force magnitudes (Appendix B.5).

Learning value. Complementing A1's stage-specific ablations, we compare physical targets with Visual-only and w/o physical. Physical and visual conditions share the policy architecture, prediction head, target size (3 × 128), loss coefficient, policy training data, base initialization, and policy update budget. Both use the same eight-frame RGB windows. Visual targets use pooled frozen DINOv2 features with training-only PCA projection and whitening. All conditions retain action learning and V-JEPA world prediction, with additional physical and visual supervision disabled during fine-tuning.

ConditionPretraining targetSpatialObjectGoalLongAvg.
w/o physicalNone96.27 ± 0.1299.33 ± 0.1296.67 ± 0.3194.53 ± 0.3196.70 ± 0.00
Visual-only supervisionVisual96.07 ± 0.1299.40 ± 0.2096.07 ± 0.1294.93 ± 0.2396.62 ± 0.13
Physical pretrain onlyPhysical97.87 ± 0.3199.53 ± 0.1297.93 ± 0.3196.07 ± 0.1297.85 ± 0.15

Results use the same three-seed protocol as A1. Physical pretraining improves success by 1.15 points over w/o physical and 1.23 over Visual-only. Visual-only performs similarly to w/o physical (96.62% vs. 96.70%). Under these policy settings, physical pretraining outperforms the tested DINOv2 target.

Error sensitivity (Appendix D.4, Table 26). From the same physically pretrained checkpoint, we fine-tune with original RGB-inferred, noisy, or mismatched targets. All runs share 30k updates, batch size 256, prediction head, loss weight, and evaluation episodes. Zero-mean Gaussian noise is added in the original target space, scaled by 0.1 times each dimension's training-set standard deviation and fixed per sample across training visits. For mismatch, 25% of windows receive another same-source window's target. Robot targets come from a different episode. Temporal groups remain ordered. Rollouts use identical clean conditions.

Fine-tuning conditionTarget treatmentGoogle RobotWidowX
Physical pretrain onlyPhysical fine-tuning disabled74.064.6
Full PhysLAOriginal RGB-inferred targets74.164.6
Target noiseAdditive noise, 0.1 × training standard deviation69.759.4
Target mismatch25% mismatched target windows70.660.4

WidowX uses 24 trials per task; scores average four unrounded task success rates. Compared with Full, noise reduces success by 4.4 points on Google Robot and 5.2 on WidowX. Mismatch reduces success by 3.5 and 4.2 points, respectively. Both also underperform disabling physical fine-tuning. This test measures fine-tuning sensitivity with a fixed pretrained checkpoint. It does not quantify natural errors or their propagation through pretraining.

AuthorsOpenReview ↗

Response to Reviewer XwZT (Part 4 / 4)

Additional concern: Learned physical targets versus simple statistics.

We compare fixed statistics with learned targets in a separate controlled training experiment, using the same measured windows, RGB backbone, training budget, and attribute-readout protocol. Both RGB predictors are trained for 30,000 updates. Both comparisons share the data, evaluation protocol, and grouped RGB predictor (Appendix C.4, Table 18). Relative to the single-source study and RGB-teacher evaluation, these attribute experiments largely reuse the training data but retrain the models and use different attribute-evaluation protocols and test samples.

AttributeMetricRGB predictor of statisticsRGB predictor of learned targets
Contact stateAUROC ↑0.680.94
Pressure magnitudeR² ↑0.480.64
Signed pressure changeR² ↑0.060.29
Pressure-change magnitudeR² ↑0.120.43
Optical deformation change magnitudeR² ↑0.050.31
Six-channel proxy magnitudeR² ↑0.010.36

Learned targets improve RGB-based decoding across all six evaluated physical attributes. Statistical targets contain 32 summaries per temporal group (96 dimensions in total): pressure means, contact-area fractions and changes, per-channel proxy means, standard deviations and changes, and optical-displacement summaries. They preserve the same three temporal groups as the learned targets. The two designs use 96-dimensional statistical targets and 384-dimensional learned targets. The comparison therefore includes both representation learning and output dimensionality. The source-specific normalization is shared. For all three sources, teachers, RGB predictors, and probe weights are fitted on the training split. Checkpoints and probe regularization are selected on the validation split. Final scores are computed on the held-out test split with all settings frozen. The RH20T attributes describe dimensionless force/torque proxies (Appendix A).

We additionally evaluate the three target designs on downstream LIBERO control (Appendix C.4, Table 19). Each condition uses its corresponding target during both pretraining and fine-tuning. The policy conditions share training data, base initialization, pretraining and fine-tuning budgets, and evaluation protocol. We report one run per condition using the same fine-tuning seed. Full PhysLA uses the run reported in Table 7 of the paper and included in A1’s three-seed summary.

LIBERO target-design comparison success rates (%).

MethodSpatialObjectGoalLongAvg.
Statistics-based targets95.899.295.895.296.5
Single-window learned targets97.499.297.895.697.5
Full PhysLA (temporally grouped learned targets)98.299.698.496.698.2

Full PhysLA achieves 98.2% average success, compared with 96.5% for statistics-based targets and 97.5% for single-window learned targets. The 1.7-percentage-point improvement over statistics-based targets provides direct evidence for the control value of the learned-target design, complementing the attribute-decoding analysis.

Additional concern: Temporal grouping versus a single window-level representation. We train separate sensor teachers with three ordered temporal groups (3 × 128) or one whole-window representation (1 × 384), followed by their corresponding RGB predictors. Both use the same input frames, time span, data splits, and update budgets. They use the same attribute-evaluation protocol as the statistics-versus-learned comparison above. Total output dimensionality is 384 in both cases. Both representations are flattened to 384 dimensions, and probes predict the same attributes for the same three time segments, using identical label and score-aggregation rules.

AttributeMetricSingle-window RGBGrouped RGB
Contact stateAUROC ↑0.750.94
Pressure magnitudeR² ↑0.510.64
Signed pressure changeR² ↑0.010.29
Pressure-change magnitudeR² ↑0.200.43
Optical deformation change magnitudeR² ↑0.030.31
Six-channel proxy magnitudeR² ↑−0.080.36

The grouped RGB predictor achieves higher scores across the six reported attributes. The shared LIBERO comparison above (Appendix C.4, Table 19) shows that Full PhysLA improves average success from 97.5% to 98.2% over single-window learned targets, with higher scores in all four suites. Together, these results support the value of temporal grouping for physical-attribute decoding and downstream control.

Reviewer_nddkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The submission proposes PhysLA, which adds physical-interaction supervision to latent-action pretraining for VLA policies. A masked-reconstruct. encoder maps 3x unpaired sensor sources (egotouch glove pressure, VLA-touch marker displacements, RH20T joint torqus) to three ordered embeddings that summarize the next second of measurements. A second model, built on frozen DINOv2, learns to predict these embeddings from the eight preceding RGB frames. This predictor then pseudo-labels human videos (EgoDex, SSV2, EgoTouch) and robot videos (droid). On top of the VLA-JEPA architecture (qwen3vl-2b, vJEPA-2 future-feature prediction, flow-match action decoder), an auxiliary head is trained so that the latent-action tokens predict these pseudo-labels. All physical modules are discarded at deployment.

The submission reports ~75 and ~65% on real2sim-eqse SimplerEnv sim sub-evals, ~65% across five AgiBot G1 tasks, and standard high numbers on (the easy) libero suites. Linear probes indicate that both the RGB predictor and the policy’s latent-action tokens encode more pressure and contact information than DINOv2 features or a variant trained without the physical loss. In the reading of the reviewer, the method amounts to an additional pseudo-labelled auxiliary regression target on VLA-JEPA.

Critical strengths

The underlying reserach question is relevant and well motivated. Latent actions learned from visual change carry contact and force only indirectly, while sensorized data is scarce. Using a small sensorized corpus as a teacher for large unsensorized video, with no sensing needed at deployment, is a practical way to address this. Combining pressure, tactile and torque data in one target space is an ambitious step beyond single-sensor tactile representation learning.

The paper includes a controlled variant. The “w/o physical” model shares architecture, data, initialization and budget with the full model and differs only in the physical loss. Together with the routing ablation and the seen/unseen probes, this gives a cleaner comparison than most latent-action papers offer. The probes are informative in their own right: contact timing transfers from video to unseen tasks (AUROC around 0.9), while pressure magnitude and change transfer far less (R² 0.15 to 0.53). The appendix documents splits, counts and selection rules in detail.

Critical weaknesses

W1. The paper does not show that the physical signal is what improves the policy.

--The individual pieces are known. Predicting pressure from RGB (PressureVision++, EgoPressure, TouchAnything), shared tactile embeddings across sensors (Sparsh, T3, AnyTouch), and auxiliary VLA targets from a frozen teacher that is dropped at test time (Spatial Forcing, DreamVLA, TA-VLA, villa-X) all exist. The contribution is the combination, so the paper needs to show that it is specifically the touch/force content that helps.

-- The controlled evidence for this is thin. The only like-for-like comparison is "w/o physical" on LIBERO, 96.7 to 98.2, with one seed and on a near-saturated benchmark. Nothing comparable is reported on SimplerEnv or the real robot.

-- The obvious alternative explanation is also not ruled out. The targets come from a DINOv2-based model run on the same video, so any extra prediction target of this kind might help equally. A control that keeps everything fixed but swaps in non-physical targets of the same size (e.g. from an untrained predictor, from future DINOv2 or V-JEPA features, or targets shuffled across clips) would settle this.

--The pseudo-labels are never checked where they are actually used. Transfer quality is measured only on held-out data from the three sensor datasets, not on SSV2 or DROID, which make up most of the supervised video and look quite different. The target space itself is loosely defined too: three unpaired sensors share one space with no indication of which sensor a target stands for, and a "horizon" covers a different length of time for each source.

-- Two analyses weaken the support further. Table 1(b) partly measures what the policy was trained to do, since its tokens are trained to predict these same targets. In Table 10, the robot-only and human-only variants also differ in whether the physical loss stays on during fine-tuning.

W2. The baseline comparisons are incomplete and in places misreported.

-- SimplerEnv (Table 12) is taken from VLA-JEPA but leaves out VLA-JEPA's own "w/o human videos" row, which scores 78.4 on Google Robot, above the submission's 74.0. It also reports a villa-X ablation in place of villa-X's main result, and a GR00T N1 number (~1%) that its source has since replaced. Several stronger methods trained on the same real data are missing (e.g. X-VLA, InternVLA-M1, MemoryVLA, CogACT). X-VLA (80.4 Google / 95.8 WidowX), InternVLA-M1 (80.7 / 71.7), MemoryVLA (77.7 / 71.9), EO-1 (76.5 / 72.7), CogACT (74.8 Google), and an open pi0 reproduction (68.4 WidowX).

-- The LIBERO rows (Table 2) are copied from VLA-JEPA as well, along with its attribution errors. The LAPA numbers are UniVLA's re-implementation (the LAPA paper does not report LIBERO), the pi0 numbers come from the openpi repository rather than the cited paper, the rows mix different training and evaluation protocols, and several methods at 98%+ are omitted. LIBERO is also largely saturated (LIBERO-PRO, LIBERO-Plus); VLA-JEPA already reports LIBERO-Plus, so the submission could too.

-- For the real robot, the number of demonstrations per task is not given, test conditions vary only object pose and position, and the supplementary shows a single rollout per task, so the scope of generalization is unclear. VLA-JEPA starts from its own released weights, so the gap is not a controlled comparison, and no individual task difference is significant at 30 trials.


On exposition: None of the minor points below would change the assessment on their own, but together they make the paper hard to follow. Much of the text relies on dense compound phrasing and near-synonyms where a plain sentence would do, and as a result the positioning relative to prior touch-prediction and latent-action work does not come through clearly. For a contribution like this, intentional exposition matters: defining terms when they are introduced, a figure that explains the mechanism, a clear path from the gap to the method to the evidence, and results that are interpreted for the reader. The reviewer would encourage a careful, author-driven revision with a new reader in mind, so that the community can follow and build on the work.

Terminology. The abstract and introduction rely on terms that are never defined, such as "instrumented interactions", "instrumented corpus", "transferred targets", "interaction-relevant latent structure", "causal RGB history" and "horizon-structured". Horizon-structured, for instance, only becomes clear in Sec. 3.2 (the next second split into three chunks). Defining these where they first appear, or replacing them with plain descriptions, would help a lot.

"Physical supervision". To most readers in robot learning this suggests action labels (end-effector, joint or force targets), whereas here it refers to predicted sensor embeddings. Relatedly, joint torque is listed as something that actionless video lacks, though torque is arguably itself an action quantity. A different name, or an early definition, would avoid this confusion.

Framing in the introduction. The first two paragraphs seem to mix two different kinds of pairing: videos paired with actions (the usual latent-action setting, actionless pretraining followed by labelled fine-tuning) and videos paired with extra sensor readings. The gap the paper addresses is the second, but the text reads as the first. It would also help to state in the method section that DROID action labels are used during pretraining; currently this is mentioned only in passing in the experimental setup.

Figures. In Figure 1 the dashed arrows are not explained, the sensor icons reappear in the latent-action stage (suggesting sensors are present there), and some of the bar-like glyphs do not seem to carry meaning. Figure 2 introduces symbols (f, f_prefix, f-hat) that do not appear in the text and does not show which components remain at deployment, which is the main question a reader has.

Notation. The same concept often goes by several names ("horizons", regions, groups, horizon groups; and many variants of "targets"), and a number of symbols are introduced once and not used again. The index k means different things in Eqs. 3 and 4, and Eq. 4 standardizes the targets while Eq. 5 does not, without explanation. Small things: L286-287 repeats a sentence, and the SimplerEnv citation renders as "(?)".

Evaluation details. Contact events are used in Table 1 but only defined in the appendix (as thresholded glove readings). The test split does not appear in Table 5 and has to be pieced together from B.1 and table captions; the "seen" test set also appears to include tasks with no training recordings. Table 1(b) is not discussed in the text, and the results paragraphs mostly restate numbers (to four decimals) rather than explain what they mean.

Questions for authors

Questions for the authors

  1. Please report a control that keeps the auxiliary head, target size and loss weight fixed but replaces the physical targets with non-physical ones (e.g. an untrained predictor, future DINOv2 or V-JEPA features, or targets shuffled across clips). Does the gain over "w/o physical" remain, across at least three seeds?

  2. Please report "w/o physical" vs. full PhysLA on SimplerEnv and on the real robot, not only LIBERO.

  3. Please report results on LIBERO-Plus and LIBERO-PRO (for PhysLA, "w/o physical", and VLA-JEPA), so that the gain can be assessed under perturbations rather than on the saturated standard suites.

  4. Please update Tables 2 and 12 with the omitted rows (VLA-JEPA w/o human videos, villa-X main result, and stronger recent methods trained on the same data), and note per row whether training was per-suite or joint, how many trials per task, and where the number comes from.

  5. How accurate are the pseudo-labels on DROID, EgoDex and SSV2? A small labelled check, or at least where these clips fall relative to the three sensor sources in the target space, would help. Does a teacher trained on a single sensor (pressure only, tactile only, torque only) do as well as the combined one?

  6. For the real robot: how many demonstrations per task were used, which test conditions (objects, poses, scenes) are outside them, and what are the action space and control rate of the deployed policy?

7 comments
AuthorsOpenReview ↗

Response to Reviewer nddk (Part 1 / 6)

We sincerely thank the reviewer for the thoughtful and constructive feedback, and for recognizing the value of transferring physical information from sensorized interactions to ordinary videos without requiring sensors at deployment. We greatly appreciate the reviewer’s valuable suggestions, which have helped us improve the quality of our paper.

We have revised the manuscript and appendix to clarify the method and evaluation protocols, strengthen the experimental comparisons, and provide additional analyses of the physical supervision. Key additions and substantive revisions are highlighted in blue for easy reference. Our point-by-point responses are provided below.

Q1: Please report a control that keeps the auxiliary head, target size and loss weight fixed but replaces the physical targets with non-physical ones (e.g. an untrained predictor, future DINOv2 or V-JEPA features, or targets shuffled across clips). Does the gain over "w/o physical" remain, across at least three seeds?

A1: We thank the reviewer for suggesting this control. With the same fine-tuning objective, physical pretraining reaches 97.85%, compared with 96.62% for the DINOv2 visual target and 96.70% for w/o physical, across three fine-tuning seeds with a fixed pretrained checkpoint for each method. Thus, the tested visual auxiliary target does not reproduce the physical-target gain. We also separate pretraining and fine-tuning contributions below. Comparison setting. The physical and visual pretraining conditions share the policy architecture, prediction head, loss coefficient, training data, initialization, and update budget. The visual control uses frozen DINOv2 features from the same eight-frame RGB history windows used by the physical RGB teacher, without sensor fine-tuning. After spatial pooling and temporal grouping, the visual features are projected to 128 dimensions using PCA and whitened with statistics fitted exclusively on training data, without sensor labels, producing 3 × 128 targets. All four conditions retain the original V-JEPA-based world-prediction objective. The first three conditions use the same fine-tuning objective, with both additional physical and visual supervision disabled. This compares the pretraining target choices under a common fine-tuning objective. PhysLA additionally enables physical supervision during fine-tuning from the same physically pretrained checkpoint used by Physical pretrain only, keeping the other fine-tuning settings fixed. Both target types are used only as training supervision. The comparison matches policy training and RGB history; target-generator architecture and training are not separately controlled.

LIBERO success rates (%). The table reports all four suites, with equal weight per suite in Avg.

ConditionPretraining targetFine-tuning physicalSpatialObjectGoalLongAvg.
w/o physicalNoneOff96.27 ± 0.1299.33 ± 0.1296.67 ± 0.3194.53 ± 0.3196.70 ± 0.00
Visual-only supervisionVisualOff96.07 ± 0.1299.40 ± 0.2096.07 ± 0.1294.93 ± 0.2396.62 ± 0.13
Physical pretrain onlyPhysicalOff97.87 ± 0.3199.53 ± 0.1297.93 ± 0.3196.07 ± 0.1297.85 ± 0.15
Full PhysLAPhysicalOn98.27 ± 0.1299.53 ± 0.1298.47 ± 0.1296.60 ± 0.2098.22 ± 0.13

Results are mean ± sample standard deviation over these three fine-tuning runs. Avg. gives equal weight to the four suites within each run. Its standard deviation is computed across the three run-level averages.

Physical pretraining reaches 97.85 ± 0.15%, improving average success by 1.15 percentage points over w/o physical and 1.23 points over the visual-only control. The visual-only control performs similarly to w/o physical (96.62% vs. 96.70%). With the shared V-JEPA objective retained, physical pretraining outperforms the tested DINOv2 control. Enabling physical supervision during fine-tuning further raises average success from 97.85% to 98.22%. Full PhysLA improves over w/o physical by 1.52 percentage points. The zero standard deviation of the baseline average reflects offsetting changes across suites. The per-suite scores vary. The settings and three-seed results are reported in Appendix C.1 and D.2 (Table 23).

AuthorsOpenReview ↗

Response to Reviewer nddk (Part 2 / 6)

Q2: Please report "w/o physical" vs. full PhysLA on SimplerEnv and on the real robot, not only LIBERO.

A2: We thank the reviewer for requesting matched comparisons beyond LIBERO. We report the comparison across all eight SimplerEnv task groups and the five G1 tasks. SimplerEnv includes w/o physical, physical pretraining only, and full PhysLA to separate the two stages. The G1 comparison evaluates Full PhysLA against w/o physical under matched training and evaluation conditions. SimplerEnv. We corrected the SimplerEnv success rates reported in the original submission by recalculating them from episode success counts. The corrected Full PhysLA averages are 74.1% on Google Robot and 64.6% on WidowX. The corresponding task-level results, ablation comparisons, and discussion have been updated in the revised manuscript.

We compare three conditions: physical supervision disabled in both stages, enabled only during pretraining, and enabled in both stages. Google Robot is evaluated using SimplerEnv’s visual-matching setting. All conditions share the same training data, initialization rules, optimization budgets, and evaluation protocol. On/Off indicates whether physical supervision is enabled at the corresponding stage.

Google Robot success rates (%).

ConditionPretrainingFine-tuningPickMoveDrawerPlaceAvg.
w/o physicalOffOff85.072.566.760.271.1
PhysLA (pretraining only)OnOff88.776.369.062.074.0
Full PhysLAOnOn88.776.368.563.074.1

WidowX success rates (%).

ConditionPretrainingFine-tuningSpoonCarrotBlockEggplantAvg.
w/o physicalOffOff79.270.812.575.059.4
PhysLA (pretraining only)OnOff83.375.020.879.264.6
Full PhysLAOnOn79.275.025.079.264.6

Each embodiment's average weights its four task groups equally. Physical pretraining raises Google Robot success from 71.1% to 74.0% and WidowX success from 59.4% to 64.6%, with higher scores in all four task groups for each robot. Full PhysLA achieves 74.1% and 64.6%, respectively. Physical pretraining accounts for most of the observed gains. Physical fine-tuning maintains comparable average performance on both robots, with task-dependent changes.

G1. We compare Full PhysLA with w/o physical on all five tasks. Full PhysLA uses physical supervision during both pretraining and fine-tuning, whereas w/o physical disables it in both stages. The two conditions share pretraining data, base initialization, demonstrations, optimization budgets, observations, and evaluation conditions.

MethodPickFlowerDiskCupPlugAvg.
w/o physical90.076.756.743.33.354.0
Full PhysLA87.593.373.360.013.365.5

The average weights the five tasks equally. Full PhysLA raises the five-task average from 54.0% to 65.5%, an 11.5 percentage point improvement over w/o physical. Success increases on Flower, Disk, Cup, and Plug, while Pick decreases from 90.0% to 87.5%. This comparison assesses the contribution of physical supervision in the complete method on the real robot. The simulation ablations separate its effects during pretraining and fine-tuning. Each model is evaluated on 40 Pick trials and 30 trials for each remaining task.

AuthorsOpenReview ↗

Response to Reviewer nddk (Part 3 / 6)

Q3: Please report results on LIBERO-Plus and LIBERO-PRO (for PhysLA, "w/o physical", and VLA-JEPA), so that the gain can be assessed under perturbations rather than on the saturated standard suites.

A3: We thank the reviewer for this suggestion and acknowledge that our original evaluation did not sufficiently assess policy robustness under perturbations. We have therefore extended the evaluation to LIBERO-Plus and LIBERO-PRO, comparing full PhysLA and w/o physical under matched settings and including VLA-JEPA as a reference. PhysLA uses physical supervision during both pretraining and fine-tuning, while w/o physical disables it in both stages. This comparison evaluates the robustness of the complete method. The contributions of the two stages are examined separately in A1–A2.

LIBERO-Plus success rates (%).

PolicyCameraRobot init.LanguageLightingBackgroundNoiseLayoutAvg.
VLA-JEPA63.367.185.495.693.666.385.179.5
w/o physical65.266.385.694.994.264.986.779.7
Full PhysLA69.975.288.996.295.870.588.683.6

LIBERO-PRO success rates (%).

PolicyObjectSwapLanguageTaskEnvironmentAvg.
VLA-JEPA81.94.097.222.141.549.3
w/o physical81.54.796.923.939.849.4
Full PhysLA84.77.297.028.846.252.8

The columns follow the seven LIBERO-Plus perturbation categories and the five categories in the evaluated original LIBERO-PRO version. Avg. weights the perturbation categories equally. VLA-JEPA results on LIBERO-Plus are quoted from the original paper (Table 3). Its LIBERO-PRO results are our evaluation of the released policy under the same settings as Full PhysLA and w/o physical. On LIBERO-Plus, Full PhysLA achieves 83.6%, exceeding w/o physical (79.7%) by 3.9 percentage points and the published VLA-JEPA result (79.5%) by 4.1 points. Full PhysLA scores higher than both comparisons in all seven perturbation categories. Relative to w/o physical, the largest gains occur under robot initialization changes (+8.9 points), noise (+5.6), and camera changes (+4.7). On LIBERO-PRO, Full PhysLA achieves 52.8%, compared with 49.4% for w/o physical and 49.3% for VLA-JEPA. The gain over w/o physical is 3.4 percentage points, with higher scores in all five categories. The largest gains occur under environment changes (+6.4 points) and task changes (+4.9). Together, the two benchmarks support improved robustness under the evaluated perturbations. Absolute success on Swap remains low (7.2%), indicating that this category remains challenging.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Response to Reviewer 8nYj (Part 1 / 3)

We sincerely thank the reviewer for the thoughtful and constructive feedback, and for recognizing the value of transferring sensor-derived physical information to ordinary human and robot videos without requiring sensors at deployment. We greatly appreciate the suggestions on separating the sources of improvement and strengthening the evidence for the inferred targets.

We have revised the manuscript and appendix to separate physical supervision during pretraining and fine-tuning, add matched visual and real-robot comparisons, and evaluate the physical targets on the video domains where they are used. Key additions and substantive revisions are highlighted in blue for easy reference. Our point-by-point responses are provided below.

Q1: Can the authors separate physical supervision in pretraining from physical supervision during LIBERO adaptation, and provide a G1 comparison against the same PhysLA pretraining setup with only the physical objective removed?

A1: We thank the reviewer for identifying this attribution issue. We agree that the original Full versus w/o physical comparison combines the effects of pretraining and fine-tuning. We therefore separate these contributions and add a matched G1 comparison. LIBERO: separating the two stages. We compare three conditions with physical supervision disabled in both stages, enabled only during pretraining, and enabled in both stages. The conditions share the training data, base initialization, policy architecture, and optimization budgets. All retain the action-learning and V-JEPA-based world-prediction objectives. The first two use the same fine-tuning objective. The last two start from the same physically pretrained checkpoint and differ only in whether physical supervision remains enabled during fine-tuning.

ConditionPhysical pretrainingPhysical fine-tuningSpatialObjectGoalLongAvg.
w/o physicalOffOff96.27 ± 0.1299.33 ± 0.1296.67 ± 0.3194.53 ± 0.3196.70 ± 0.00
Physical pretrain onlyOnOff97.87 ± 0.3199.53 ± 0.1297.93 ± 0.3196.07 ± 0.1297.85 ± 0.15
Full PhysLAOnOn98.27 ± 0.1299.53 ± 0.1298.47 ± 0.1296.60 ± 0.2098.22 ± 0.13

Results are mean ± sample standard deviation over three fine-tuning seeds, with the pretrained checkpoint fixed for each method. Avg. gives equal weight to the four suites within each run. Its standard deviation is computed across the three run-level averages. Across three fine-tuning seeds, physical pretraining increases average success from 96.70% to 97.85%, and enabling physical supervision during fine-tuning further increases it to 98.22%. These correspond to gains of 1.15 and 0.37 percentage points, respectively. Full PhysLA improves over w/o physical by 1.52 percentage points, supporting the contribution of physical supervision at both stages. The zero standard deviation of the baseline average reflects offsetting changes across suites. The per-suite scores vary.

AuthorsOpenReview ↗

Response to Reviewer 8nYj (Part 2 / 3)

G1: Full PhysLA versus w/o physical. We compare Full PhysLA with a matched variant that removes the physical objective during both pretraining and fine-tuning while retaining the action-learning and world-prediction objectives. The two conditions share pretraining data, base initialization, five-task demonstrations, the 20,000-update fine-tuning budget, and evaluation conditions.

MethodPickFlowerDiskCupPlugAvg.
w/o physical90.076.756.743.33.354.0
Full PhysLA87.593.373.360.013.365.5

The average weights the five tasks equally. Full PhysLA raises the five-task average from 54.0% to 65.5%, an 11.5-percentage-point improvement over w/o physical. Success increases on Flower, Disk, Cup, and Plug, while Pick decreases from 90.0% to 87.5%. This comparison assesses the contribution of physical supervision in the complete method on the real robot. The simulation ablations separate its effects during pretraining and fine-tuning. Each model is evaluated on 40 Pick trials and 30 trials for each remaining task. Both human and robot video streams are retained in all routing variants. Human-only versus w/o physical and Full versus Robot-only isolate the effect of applying physical supervision to human videos, with the fine-tuning objective held fixed within each pair.

The same stage comparison on SimplerEnv (Tables 4 and 21) gives 71.1% → 74.0% → 74.1% on Google Robot and 59.4% → 64.6% → 64.6% on WidowX for w/o physical, Physical pretrain only, and Full, respectively. With the fine-tuning objective held fixed, physical pretraining improves both embodiments; continuing physical supervision during fine-tuning has little effect on their average success.

We also corrected the SimplerEnv success rates reported in the original submission by recalculating them from episode success counts. The corrected Full PhysLA averages are 74.1% on Google Robot and 64.6% on WidowX. The corresponding task-level results, ablation comparisons, and discussion have been updated in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 8nYj (Part 3 / 3)

Q2: How reliable are the inferred physical targets on the uninstrumented pretraining domains? Does a capacity- and budget-matched auxiliary target derived only from visual features give comparable downstream gains?

A2: We thank the reviewer. We evaluate contact information on the target domains and the learning value of physical supervision through a visual control with matched policy training. Target-domain validation. Both models use the same 200 manually annotated videos: 66 DROID, 67 EgoDex, and 67 SSV2. We evaluate contact state, establishment, and release with AUROC and average precision (AP). The visual reference uses pooled features from frozen DINOv2. Linear probes are fitted on source training data and selected on source validation data. Target annotations are reserved for evaluation. Window scores use the middle temporal group for contact and the maximum across three groups for establishment/release. Metrics are computed once over all valid windows per domain and task, without averaging over groups or videos.

DomainRepresentationContact AUROC / APEstablishment AUROC / APRelease AUROC / AP
DROIDPhysical RGB teacher0.83 / 0.780.72 / 0.680.69 / 0.73
DROIDFrozen DINOv20.51 / 0.580.49 / 0.330.42 / 0.25
EgoDexPhysical RGB teacher0.82 / 0.950.86 / 0.690.83 / 0.73
EgoDexFrozen DINOv20.71 / 0.770.53 / 0.570.59 / 0.74
SSV2Physical RGB teacher0.89 / 0.920.86 / 0.890.73 / 0.81
SSV2Frozen DINOv20.67 / 0.590.41 / 0.520.57 / 0.44

The physical teacher improves AUROC in all nine comparisons and AP in eight, supporting visible-interaction transfer to the pretraining domains. Source-probe and target-domain contact labels refer to the last frame of the eight-frame history. Establishment and release labels are defined over the same eight-frame history window. An event is positive only if its annotated transition interval is wholly inside that window. Negatives require explicit whole-clip absence. Out-of-window, boundary-overlapping, or uncertain events are excluded. This evaluates observed-history contact and transitions, not future-target accuracy or pressure/force magnitudes (Appendix B.5).

DINOv2 auxiliary-target control. Physical and visual conditions share the policy architecture, prediction head, target dimensions, loss coefficient, policy pretraining data, base initialization, and policy update budget. Frozen DINOv2 features from the same eight-frame RGB windows undergo spatial pooling and temporal grouping, followed by PCA projection to 128 dimensions and whitening, producing 3 × 128 targets. PCA and whitening statistics are fitted exclusively on training data, without sensor labels. All three use the same fine-tuning objective, retaining action learning and V-JEPA-based world prediction, with both additional physical and visual supervision disabled. This comparison controls policy training and the RGB history used for target generation. It does not separately control the target generators’ architecture and training procedure.

ConditionPretraining targetSpatialObjectGoalLongAvg.
w/o physicalNone96.27 ± 0.1299.33 ± 0.1296.67 ± 0.3194.53 ± 0.3196.70 ± 0.00
Visual-only supervisionVisual96.07 ± 0.1299.40 ± 0.2096.07 ± 0.1294.93 ± 0.2396.62 ± 0.13
Physical pretrain onlyPhysical97.87 ± 0.3199.53 ± 0.1297.93 ± 0.3196.07 ± 0.1297.85 ± 0.15

Results use the same three-seed protocol as A1. Physical pretraining improves success by 1.15 points over w/o physical and 1.23 over Visual-only. Visual-only remains similar to w/o physical (96.62% vs. 96.70%). Under these policy settings, physical pretraining outperforms the tested DINOv2 target. Temporal context. The policy receives current observations in all conditions. The physical and visual targets use the same eight-frame windows at identical timestamps for offline supervision. This comparison controls the temporal information available for target generation while keeping policy observations fixed. RGB ambiguity. Similar RGB histories can precede different actions and contact forces. Our RGB teacher estimates physical information predictable from the observed history, providing sensor-informed supervision without uniquely determining the subsequent interaction. Our method does not require exact recovery of the physical quantities associated with every interaction. The target-domain evaluations and matched policy ablations assess the information retained by these estimates and their value for downstream control.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.