- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
PHYSLA learns shared physical representations from pressure, tactile measurements, and joint torques, then transfers them through RGB prediction to human and robot videos without sensor annotations. It combines future visual prediction and physical-target prediction to learn latent actions, and uses robot action data for policy learning. The method is evaluated on LIBERO, SimplerEnv, and real AgiBot G1 tasks, supporting the value of physically-grounded latent action.
Critical strengths
- The approach to extending physical supervision is valuable. By separating the acquisition and use of physical supervision, RGB prediction transfers information from limited sensor data to broader video collections. This addresses the practical imbalance between scarce physical measurements and abundant ordinary videos.
- The integration of heterogeneous physical information offers novelty. Pressure, tactile deformation, and joint torques are organized into shared targets with temporal structure and connected to latent-action learning from human and robot videos, providing a common interface for different sensor data sources.
- Joint supervision supports both learning and practical deployment. Visual and physical prediction jointly supervise latent actions, without requiring additional sensors or target generation modules at deployment. The physical-attribute probes in Table 1 and downstream results provide initial support for this design.
Critical weaknesses
- The sources of performance gains are not sufficiently separated. Table 10 shows that Full retains the physical loss during adaptation, whereas w/o physical does not. Their 1.5-point difference in Table 4 therefore combines pretraining and adaptation effects and cannot be attributed to physical pretraining alone. G1 and SimplerEnv also lack a w/o-physical comparison under matched pretraining conditions, leaving the sources of their larger gains unclear.
- The specific value of the physical targets and their temporal structure remains unclear. Table 1 includes a pressure-supervised baseline, where Table 4 only changes physical-loss routing and does not compare control performance when simple physical statistics replace the learned targets. The temporal perturbations in Appendix B.3 evaluate reconstruction in a fixed model, without establishing a downstream advantage of temporal grouping over a single window-level representation.
- The reliability of physical targets in new video domains is not directly established. The physical probes in Tables 1 and 9 cover EgoTouch, RH20T, and VLA-touch, whereas most policy-pretraining videos come from EgoDex, SSV2, and DROID. Section 4.3 acknowledges that RGB recovers physical information only partially, but whether this recovery holds in the latter three sources is not directly tested.
Questions for authors
- How much do physical pretraining and adaptation each contribute? Could the authors compare Full and w/o physical on G1 or SimplerEnv with matched pretraining data, initialization, training budget, and adaptation losses?
- How does target quality across domains affect latent-action learning? What evidence supports that RGB-inferred targets retain useful physical interaction information on EgoDex, SSV2, and DROID? How sensitive is downstream learning to target prediction errors?
4 comments
Response to Reviewer XwZT (Part 2 / 4)
G1: Full PhysLA versus w/o physical. We additionally compare Full PhysLA with w/o physical using matched pretraining data, base initialization, five-task demonstrations, fine-tuning budgets, and evaluation conditions. The w/o physical variant removes the physical objective during both pretraining and fine-tuning while retaining the action-learning and world-prediction objectives.
| Method | Pick | Flower | Disk | Cup | Plug | Avg. |
|---|---|---|---|---|---|---|
| w/o physical | 90.0 | 76.7 | 56.7 | 43.3 | 3.3 | 54.0 |
| Full PhysLA | 87.5 | 93.3 | 73.3 | 60.0 | 13.3 | 65.5 |
Full PhysLA raises the five-task average from 54.0% to 65.5%, an 11.5-percentage-point improvement over w/o physical. Success increases on Flower, Disk, Cup, and Plug, while Pick decreases from 90.0% to 87.5%. This comparison assesses the contribution of physical supervision in the complete method on the real robot. The average weights the five tasks equally. Each model is evaluated on 40 Pick trials and 30 trials for each remaining task.
Response to Reviewer XwZT (Part 3 / 4)
Q2: How does target quality across domains affect latent-action learning? What evidence supports that RGB-inferred targets retain useful physical interaction information on EgoDex, SSV2, and DROID? How sensitive is downstream learning to target prediction errors?
A2: Target-domain validation. Both models use the same 200 annotated videos: 66 DROID, 67 EgoDex, and 67 SSV2. The visual reference uses pooled frozen DINOv2 features. Probes are fitted on source training data, selected on source validation data, and evaluated on target annotations. Contact scores use the middle group. Event scores use the maximum across three groups. AUROC/AP are computed once over all valid windows per domain and task, without averaging over groups or videos.
| Domain | Representation | Contact AUROC / AP | Establishment AUROC / AP | Release AUROC / AP |
|---|---|---|---|---|
| DROID | Physical RGB teacher | 0.83 / 0.78 | 0.72 / 0.68 | 0.69 / 0.73 |
| DROID | Frozen DINOv2 | 0.51 / 0.58 | 0.49 / 0.33 | 0.42 / 0.25 |
| EgoDex | Physical RGB teacher | 0.82 / 0.95 | 0.86 / 0.69 | 0.83 / 0.73 |
| EgoDex | Frozen DINOv2 | 0.71 / 0.77 | 0.53 / 0.57 | 0.59 / 0.74 |
| SSV2 | Physical RGB teacher | 0.89 / 0.92 | 0.86 / 0.89 | 0.73 / 0.81 |
| SSV2 | Frozen DINOv2 | 0.67 / 0.59 | 0.41 / 0.52 | 0.57 / 0.44 |
The physical teacher improves AUROC in all nine comparisons and AP in eight. Source and target contact labels refer to the final history frame; establishment and release labels cover the same eight-frame history window. An event is positive only if its annotated transition interval is wholly inside that window. Negatives require explicit whole-clip absence. Out-of-window, boundary-overlapping, or uncertain events are excluded. This evaluates observed-history contact and transitions, not future-target accuracy or pressure/force magnitudes (Appendix B.5).
Learning value. Complementing A1's stage-specific ablations, we compare physical targets with Visual-only and w/o physical. Physical and visual conditions share the policy architecture, prediction head, target size (3 × 128), loss coefficient, policy training data, base initialization, and policy update budget. Both use the same eight-frame RGB windows. Visual targets use pooled frozen DINOv2 features with training-only PCA projection and whitening. All conditions retain action learning and V-JEPA world prediction, with additional physical and visual supervision disabled during fine-tuning.
| Condition | Pretraining target | Spatial | Object | Goal | Long | Avg. |
|---|---|---|---|---|---|---|
| w/o physical | None | 96.27 ± 0.12 | 99.33 ± 0.12 | 96.67 ± 0.31 | 94.53 ± 0.31 | 96.70 ± 0.00 |
| Visual-only supervision | Visual | 96.07 ± 0.12 | 99.40 ± 0.20 | 96.07 ± 0.12 | 94.93 ± 0.23 | 96.62 ± 0.13 |
| Physical pretrain only | Physical | 97.87 ± 0.31 | 99.53 ± 0.12 | 97.93 ± 0.31 | 96.07 ± 0.12 | 97.85 ± 0.15 |
Results use the same three-seed protocol as A1. Physical pretraining improves success by 1.15 points over w/o physical and 1.23 over Visual-only. Visual-only performs similarly to w/o physical (96.62% vs. 96.70%). Under these policy settings, physical pretraining outperforms the tested DINOv2 target.
Error sensitivity (Appendix D.4, Table 26). From the same physically pretrained checkpoint, we fine-tune with original RGB-inferred, noisy, or mismatched targets. All runs share 30k updates, batch size 256, prediction head, loss weight, and evaluation episodes. Zero-mean Gaussian noise is added in the original target space, scaled by 0.1 times each dimension's training-set standard deviation and fixed per sample across training visits. For mismatch, 25% of windows receive another same-source window's target. Robot targets come from a different episode. Temporal groups remain ordered. Rollouts use identical clean conditions.
| Fine-tuning condition | Target treatment | Google Robot | WidowX |
|---|---|---|---|
| Physical pretrain only | Physical fine-tuning disabled | 74.0 | 64.6 |
| Full PhysLA | Original RGB-inferred targets | 74.1 | 64.6 |
| Target noise | Additive noise, 0.1 × training standard deviation | 69.7 | 59.4 |
| Target mismatch | 25% mismatched target windows | 70.6 | 60.4 |
WidowX uses 24 trials per task; scores average four unrounded task success rates. Compared with Full, noise reduces success by 4.4 points on Google Robot and 5.2 on WidowX. Mismatch reduces success by 3.5 and 4.2 points, respectively. Both also underperform disabling physical fine-tuning. This test measures fine-tuning sensitivity with a fixed pretrained checkpoint. It does not quantify natural errors or their propagation through pretraining.
Response to Reviewer XwZT (Part 4 / 4)
Additional concern: Learned physical targets versus simple statistics.
We compare fixed statistics with learned targets in a separate controlled training experiment, using the same measured windows, RGB backbone, training budget, and attribute-readout protocol. Both RGB predictors are trained for 30,000 updates. Both comparisons share the data, evaluation protocol, and grouped RGB predictor (Appendix C.4, Table 18). Relative to the single-source study and RGB-teacher evaluation, these attribute experiments largely reuse the training data but retrain the models and use different attribute-evaluation protocols and test samples.
| Attribute | Metric | RGB predictor of statistics | RGB predictor of learned targets |
|---|---|---|---|
| Contact state | AUROC ↑ | 0.68 | 0.94 |
| Pressure magnitude | R² ↑ | 0.48 | 0.64 |
| Signed pressure change | R² ↑ | 0.06 | 0.29 |
| Pressure-change magnitude | R² ↑ | 0.12 | 0.43 |
| Optical deformation change magnitude | R² ↑ | 0.05 | 0.31 |
| Six-channel proxy magnitude | R² ↑ | 0.01 | 0.36 |
Learned targets improve RGB-based decoding across all six evaluated physical attributes. Statistical targets contain 32 summaries per temporal group (96 dimensions in total): pressure means, contact-area fractions and changes, per-channel proxy means, standard deviations and changes, and optical-displacement summaries. They preserve the same three temporal groups as the learned targets. The two designs use 96-dimensional statistical targets and 384-dimensional learned targets. The comparison therefore includes both representation learning and output dimensionality. The source-specific normalization is shared. For all three sources, teachers, RGB predictors, and probe weights are fitted on the training split. Checkpoints and probe regularization are selected on the validation split. Final scores are computed on the held-out test split with all settings frozen. The RH20T attributes describe dimensionless force/torque proxies (Appendix A).
We additionally evaluate the three target designs on downstream LIBERO control (Appendix C.4, Table 19). Each condition uses its corresponding target during both pretraining and fine-tuning. The policy conditions share training data, base initialization, pretraining and fine-tuning budgets, and evaluation protocol. We report one run per condition using the same fine-tuning seed. Full PhysLA uses the run reported in Table 7 of the paper and included in A1’s three-seed summary.
LIBERO target-design comparison success rates (%).
| Method | Spatial | Object | Goal | Long | Avg. |
|---|---|---|---|---|---|
| Statistics-based targets | 95.8 | 99.2 | 95.8 | 95.2 | 96.5 |
| Single-window learned targets | 97.4 | 99.2 | 97.8 | 95.6 | 97.5 |
| Full PhysLA (temporally grouped learned targets) | 98.2 | 99.6 | 98.4 | 96.6 | 98.2 |
Full PhysLA achieves 98.2% average success, compared with 96.5% for statistics-based targets and 97.5% for single-window learned targets. The 1.7-percentage-point improvement over statistics-based targets provides direct evidence for the control value of the learned-target design, complementing the attribute-decoding analysis.
Additional concern: Temporal grouping versus a single window-level representation. We train separate sensor teachers with three ordered temporal groups (3 × 128) or one whole-window representation (1 × 384), followed by their corresponding RGB predictors. Both use the same input frames, time span, data splits, and update budgets. They use the same attribute-evaluation protocol as the statistics-versus-learned comparison above. Total output dimensionality is 384 in both cases. Both representations are flattened to 384 dimensions, and probes predict the same attributes for the same three time segments, using identical label and score-aggregation rules.
| Attribute | Metric | Single-window RGB | Grouped RGB |
|---|---|---|---|
| Contact state | AUROC ↑ | 0.75 | 0.94 |
| Pressure magnitude | R² ↑ | 0.51 | 0.64 |
| Signed pressure change | R² ↑ | 0.01 | 0.29 |
| Pressure-change magnitude | R² ↑ | 0.20 | 0.43 |
| Optical deformation change magnitude | R² ↑ | 0.03 | 0.31 |
| Six-channel proxy magnitude | R² ↑ | −0.08 | 0.36 |
The grouped RGB predictor achieves higher scores across the six reported attributes. The shared LIBERO comparison above (Appendix C.4, Table 19) shows that Full PhysLA improves average success from 97.5% to 98.2% over single-window learned targets, with higher scores in all four suites. Together, these results support the value of temporal grouping for physical-attribute decoding and downstream control.
Response to Reviewer XwZT (Part 1 / 4)
We sincerely thank the reviewer for the thoughtful and constructive feedback, and for recognizing the value of transferring heterogeneous physical information to ordinary videos and latent-action learning. We greatly appreciate the suggestions on identifying the sources of improvement and evaluating the quality and design of the physical targets.
We have revised the manuscript and appendix to separate pretraining and fine-tuning effects, evaluate target-domain reliability and error sensitivity, and analyze learned physical targets and their temporal structure. Key additions and substantive revisions are highlighted in blue for easy reference. Our point-by-point responses are provided below.
Q1: How much do physical pretraining and adaptation each contribute? Could the authors compare Full and w/o physical on G1 or SimplerEnv with matched pretraining data, initialization, training budget, and adaptation losses?
A1: We thank the reviewer for identifying this attribution issue. We agree that the original comparison combines physical supervision during pretraining and fine-tuning. We therefore report separate stage-specific comparisons and extend the matched evaluation beyond LIBERO. Separating the two stages. We compare w/o physical, Physical pretrain only, and Full PhysLA. The first two differ in physical pretraining while using the same fine-tuning objective. The last two start from the same physically pretrained checkpoint and differ only in physical supervision during fine-tuning. Training data, base initialization, policy architecture, optimization budgets, and evaluation protocols are matched. The action-learning and V-JEPA-based world-prediction objectives remain enabled in all conditions.
LIBERO success rates (%).
Results are mean ± sample standard deviation over three fine-tuning seeds, with the pretrained checkpoint fixed for each method. Avg. gives equal weight to the four suites within each run. Its standard deviation is computed across the three run-level averages. Across three fine-tuning seeds, physical pretraining increases average success from 96.70% to 97.85%, and enabling physical supervision during fine-tuning further increases it to 98.22%. These correspond to gains of 1.15 and 0.37 percentage points, respectively. Full PhysLA improves over w/o physical by 1.52 percentage points, supporting the contribution of physical supervision at both stages. The zero standard deviation of the baseline average reflects offsetting changes across suites. The per-suite scores vary.
SimplerEnv success rates (%). We corrected the SimplerEnv success rates reported in the original submission by recalculating them from episode success counts. The corrected Full PhysLA averages are 74.1% on Google Robot and 64.6% on WidowX. The corresponding task-level results, ablation comparisons, and discussion have been updated in the revised manuscript.
Google Robot uses SimplerEnv’s visual-matching setting. Google Robot.
WidowX.
Each robot's average weights its four task groups equally. Physical pretraining raises Google Robot success from 71.1% to 74.0% and WidowX success from 59.4% to 64.6%, with higher scores in all four task groups for each robot. Physical pretraining accounts for most of the observed gains. Physical fine-tuning maintains comparable average performance on both robots, with task-dependent changes.