acceptodds
Under review as a conference paper at ICLR 2027

PHI: A Hierarchical World Action Model with Selective Prediction and Force Feedback for Humanoid Bimanual Manipulation

Abstract

Humanoid bimanual dexterous manipulation requires coordinated dual-arm and finger actions under evolving contact interactions. Existing world action models (WAMs) offer a promising way to model such future dynamics, but primarily optimize future representations for prediction, leaving their utility for downstream action generation implicit. We introduce PHI (Predictive Hierarchical Intelligence), a hierarchical world action model for humanoid bimanual manipulation that integrates selective future prediction with force feedback. Rather than reconstructing or preserving the complete future observation, PHI selectively learns compact future visual representations that retain information useful for action generation. A hierarchical predictive pathway first extracts task-relevant future representations from privileged future observations during training and then learns to infer them from the current interaction context at deployment. To complement visual prediction in contact-rich interactions, PHI further incorporates a temporal force encoder that captures recent force evolution and provides physical feedback about contact establishment, loading, and interaction changes. The predicted future representation and force feedback are jointly integrated to guide action generation. Across five real-world humanoid bimanual manipulation tasks, PHI achieves an aggregate success rate of 82.0%, outperforming the strongest baseline by 23.3 percentage points and up to a 30.0 percentage points gain on unseen book instances. These results demonstrate PHI's effectiveness through selective future prediction and temporal force feedback.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Humanoid Robot, World Action Model, Bimanual Manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jts4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PHI, a framework for humanoid bimanual manipulation that combines task-oriented future representation prediction with temporal force feedback. Its three-stage training procedure first learns future representations from privileged future observations using action supervision, then trains the WAM to predict these representations from the current context, and finally jointly optimizes future prediction and action generation. A temporal force encoder processes recent force measurements to provide additional contact information to a flow-matching action expert. Experiments on five real-world Unitree G1 tasks report an average success rate of 82.0%, compared with 58.7% for the strongest baseline, along with evaluations on unseen objects and visual distractors.

Critical strengths
  1. The method is well motivated. Learning future representations through action supervision offers a reasonable way to focus on information useful for control.

  2. Combining future visual predictions with force history is sensible for contact-rich manipulation.

  3. The paper evaluates the method on real-world humanoid manipulation tasks and includes tests with unseen objects and visual distractors. The demo is convincing.

Critical weaknesses
  1. The architecture allows the action expert to rely mainly on the current context and ignore the future branch. The training objectives do not rule out this shortcut. Specifically, (Z^+) could become redundant, nearly constant, or predictable but unused by the action expert. It is unclear what information (Z^+) contains. For the “w/o Future Learning Module” variant in Table 3, does removing this module reduce the total number of parameters?

  2. The fairness and coverage of the baseline comparisons are unclear. Are the baselines fine-tuned from pretrained checkpoints or trained from scratch on the collected data? Do the baselines receive force inputs? Could the authors compare against additional relevant methods, such as [1] and [2]?

[1] Zheng, Ruijie, et al. "Flare: Robot learning with implicit world modeling." arXiv preprint arXiv:2505.15659 (2025). [2] https://huggingface.co/nvidia/GR00T-N1.5-3B

Questions for authors
  1. What happens if the force encoding is also fed directly into the WAM?
  2. The authors can describe this cascaded structure as hierarchical. Many VLAs, including the baseline π₀, already use a vision-language backbone followed by an action expert. PHI mainly adds future latent predictions supervised by future observations and passes them to the action expert. Why do the authors need the emphasize it?
  3. Despite feature alignment, the paper doesn't demonstrate or evaluate the ability of the "WAM" to predict the future. How is the term "world action model" justified?
8 comments
AuthorsOpenReview ↗

Response to Reviewer jts4 (1/2)

We thank the reviewer for recognizing the motivation, the combination of future visual prediction and force feedback, and the real-robot evaluation. We clarify the role of the future representation, the baseline comparisons, and PHI's hierarchical design below.

Q1: The action expert could ignore the fture branch, and Z+Z^+ could be redundant or nearly constant. What information does it contain? Does removing the Future Learning Module reduce the parameter count?

We use Z+Z^+ as a compact representation of future visual interaction, shaped to retain information useful for action generation; it is not intended to reconstruct every detail of a future frame. Appendix A.5 reports statistics over 20 Open Book trajectories.

RepresentationAcross-sample varianceEffective rankPC1 variance ratioMean prediction MSE
Predicted future latent0.0278864.604148.21%0.069287
Target future latent0.0612035.323246.49%—

The predicted latent retains 45.6% of the target variance and multiple effective dimensions, which is inconsistent with a constant representation in these samples. With the action predictor fixed, shuffling latent tokens increases joint MSE by 23.18%; replacing each latent with its global mean or zeros increases it by about 802%.

ConditionJoint MSE (rad$^2$)Change from normal
Normal predicted latent0.00047742—
Within-latent token shuffle0.00058808+23.18%
Per-latent global-mean replacement0.00430706+802.16%
Zero replacement0.00430749+802.25%

The component ablation provides complementary behavioral evidence: removing the Future Learning Module reduces Place Book Straight success from 76.7% to 50.0%. The Future Learning Module contains approximately 0.6B parameters. Removing it reduces the model from 4.7B to about 4.1B. In the real-robot ablation in Table 3, including this module improves Place Book Straight success from 50.0% (15/30) to 76.7% (23/30), with the other PHI components retained.

Q2: Are the baselines fine-tuned from pretrained checkpoints or trained from scratch? Do they receive force inputs? Could the authors compare with FLARE and GR00T?

The evaluated backbones use publicly released pretrained parameters and are fine-tuned on our collected demonstrations; PHI's newly introduced modules use random initialization. All policies receive current force measurements through the robot-state input. PHI additionally encodes recent force history. Appendix A.7 reports model sizes, training resources and steps, total batch sizes, RTC availability, and inference settings: π0\pi_0 and π0.5\pi_{0.5} use 500k steps/GPU with total batch 32, VLA-JEPA uses 400k steps/GPU with total batch 96, Fast-WAM uses 400k steps/GPU with total batch 32, and PHI uses 500k steps/GPU with Stage I/II/III total batches 32/32/48. PHI's single-GPU setting uses batch size 6. The methods use the same robot platform, task definitions, action space, low-level controller, and evaluation initial-condition sequence, with 30 real-robot trials per task.

We added VLA-JEPA and Fast-WAM to the same-protocol comparison because they are more directly aligned with PHI's future-representation and WAM design. PHI reaches 82.0% average success, compared with 45.3% for Fast-WAM and 1.3% for the evaluated VLA-JEPA configuration. The full per-task results are in Table 2.

MethodWalk to TablePlace BottleOpen BookPlace CubePlace Book StraightAverage
π0.5\pi_{0.5}100.0 (30/30)33.3 (10/30)46.7 (14/30)36.7 (11/30)43.3 (13/30)52.0
π0\pi_0100.0 (30/30)66.7 (20/30)50.0 (15/30)40.0 (12/30)36.7 (11/30)58.7
VLA-JEPA6.7 (2/30)0.0 (0/30)0.0 (0/30)0.0 (0/30)0.0 (0/30)1.3
Fast-WAM100.0 (30/30)30.0 (9/30)56.7 (17/30)10.0 (3/30)30.0 (9/30)45.3
PHI (Ours)100.0 (30/30)76.7 (23/30)86.7 (26/30)70.0 (21/30)76.7 (23/30)82.0

We did not include FLARE as a quantitative baseline because no public implementation or checkpoint was available to us for a same-protocol reproduction; NVIDIA's project page lists its GitHub release as forthcoming. GR00T-N1.5 is described by NVIDIA as a general-purpose humanoid foundation model, rather than a future-predictive WAM (project page), so it is less directly matched to PHI's central question. We therefore report same-protocol results for VLA-JEPA and Fast-WAM, the closer alternatives for this comparison.

AuthorsOpenReview ↗

Response to Reviewer jts4 (2/2)

Q3: What happens if force encoding is also fed directly into the WAM?

Feeding the encoded force history into the WAM could make its future predictions more sensitive to contact evolution and potentially provide a modest improvement on contact-rich tasks. For example, recent force changes could help distinguish stable contact from emerging slip when predicting subsequent interaction. We would expect the additional benefit to be incremental, since the WAM already receives current force through the robot state and the action expert directly uses the encoded force history. This is an architectural expectation; we have not evaluated this additional conditioning route.

Our choice balances this potential benefit against the additional training and inference computation needed to integrate force-history conditioning into the large WAM backbone. PHI gives the two pathways complementary roles: the WAM predicts future visual interaction, while the action expert combines that prediction with recent force evolution to generate contact-sensitive actions. Routing force history directly to the action expert preserves its contribution to control without requiring that information to pass through the large backbone. We therefore favor this design for real-robot deployment, where task performance, computational efficiency, and timely action generation must be considered together.

Q4: Why emphasize a hierarchical structure when many VLAs already use a vision-language backbone followed by an action expert?

As discussed in the Introduction (Section 1 and Figure 1), we distinguish three architectural paradigms for connecting future modeling with action generation in WAMs: cascaded approaches, joint autoregressive approaches, and joint diffusion approaches. Our emphasis on a hierarchical structure explains PHI's architectural position within this WAM landscape and how it connects future prediction to control.

In cascaded WAMs, future visual observations are first reconstructed, and representations for action generation are then derived from them. Joint autoregressive and joint diffusion WAMs instead model future and action representations together. PHI follows a hierarchical cascaded design, with an explicit predicted future representation serving as an intermediate input to the downstream action expert. Its key distinction within this design is what the upper pathway predicts: a compact future visual latent shaped by downstream action supervision, rather than a reconstructed future image or video. The Exploit–Predict–Align procedure first learns this action-relevant target from privileged future observations, then learns to predict it from the current context, and finally trains action generation with the predicted representation.

This organization combines an explicit prediction-to-action cascade with selective latent prediction. The upper pathway anticipates task-relevant future visual interaction, while the lower action expert combines that prediction with current state and temporal force feedback to produce actions. Thus, “hierarchical” communicates an architectural choice among WAM designs: PHI separates future representation prediction from action generation while connecting them through an action-shaped latent interface. This distinction is why we emphasize the term in the Introduction.

Q5: How is the term “world action model” justified if the paper does not show future prediction?

PHI predicts future visual information in latent space rather than generating future video pixels. During training, privileged future observations provide the target representation. At deployment, the WAM predicts that representation from the current interaction context, and the action expert uses it together with force feedback. Appendix A.5 evaluates this prediction over 20 Open Book trajectories, reporting a mean latent prediction MSE of 0.069287 and cross-sample statistics for predicted and target latents. Thus, “world action model” refers to PHI's explicit prediction of a future interaction representation for action generation; it does not imply a pixel-level simulator or full future-video reconstruction.

We appreciate the reviewer's questions, which help clarify the role of PHI's future pathway, force input, and hierarchical design.

  1. I noticed that the other two reviewers also highlighted the lack of comparisons with alternative designs for the future prediction module. Comparisons with additional baselines are helpful. However, a more convincing evaluation would be an ablation study comparing the proposed method with other variants, as suggested by reviewer VwGx, while keeping all other design choices fixed. It's not clear to me why the authors choose not to conduct the ablation experiments specified by reviewer VwGx.

  2. The experiment result in the comment shows that the features have variance and randomly perturbing it causes the performance. It's still not clear what information (Z^+) contains. Can it be used to generate (partial) future states or images?

  3. "All policies receive current force measurements through the robot-state input. PHI additionally encodes recent force history." Does it mean the baselines do not take in recent force history as input?

  4. Not critical, but the neuroscience framing in Figure 1 is redundant and decorative.

Reviewer_xruaOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a new method to address humanoid bimanual dexterous manipulation. The authors argue that shaping the future representation by downstream action supervision is better than reconstruction-based objectives. They propose PHI, which uses privileged future observations during training to build a compact future latent. The action loss decides what this latent keeps. A second stage then trains the model to predict the same latent from the current observation alone. A temporal force encoder is used to capture recent force evolution and physical feedback about contact information. Experiments on five real-robot tasks are reported across increasing contact complexity, together with generalization to visual distractors and unseen objects.

Critical strengths
  1. The motivation about a future representation should be shaped by its utility for action generation is sound.

  2. The proposed three-stages training procedure is reasonable.

  3. The real-robot evaluation is non-trivial in scale. The shared initial-condition sequence makes the comparison paired rather than independently randomized.

Critical weaknesses
  1. The main claim in the paper is shaping the future representation by downstream action supervision is better than reconstruction-based objectives. However, there are only two non-WAM VLA baselines included in the experiments. The proposed method is not evaluated against any of the WAM-based baselines.

  2. I don't see any analysis or measurement about the selectivity in the paper. Nothing quantifies what the future latent retains and what discards. The only relevant ablation removes this module entirely, so its only compares to the absence of a future representation rather than a non-selective one.

  3. The generalization experiments are well constructed. However PHI and the baseline Pi_0 degrade identically under visual distractors, and on unseen books PHI's absolute drop exceeds both baselines.

Questions for authors

I am not sure if the reported compactness of the future latent implies the selectivity or a partial latent collapse. Could you provide any further analysis to answer this?

4 comments
AuthorsOpenReview ↗

Response to Reviewer xrua(1/2)

We thank the reviewer for recognizing the motivation, the three-stage training design, and the scale of our real-robot evaluation. We have expanded the WAM comparison and added representation analyses to address the questions about selectivity, latent collapse, and the contribution of future prediction. We clarify the generalization results below.

Q1: The paper lacks comparisons with WAM-based baselines, despite its claims about future representation learning.

We added VLA-JEPA and Fast-WAM to Table 2, providing comparisons with future-representation prediction and future-video modeling, respectively. Each method was evaluated on the five-task Unitree G1 suite with 30 trials per task and 150 trials in total. The policies share the demonstration data, robot platform, task definitions, action space, low-level controller, and evaluation initial-condition sequence.

MethodWalk to TablePlace BottleOpen BookPlace CubePlace Book StraightAverage
π0.5\pi_{0.5}100.0 (30/30)33.3 (10/30)46.7 (14/30)36.7 (11/30)43.3 (13/30)52.0
π0\pi_0100.0 (30/30)66.7 (20/30)50.0 (15/30)40.0 (12/30)36.7 (11/30)58.7
VLA-JEPA6.7 (2/30)0.0 (0/30)0.0 (0/30)0.0 (0/30)0.0 (0/30)1.3
Fast-WAM100.0 (30/30)30.0 (9/30)56.7 (17/30)10.0 (3/30)30.0 (9/30)45.3
PHI (Ours)100.0 (30/30)76.7 (23/30)86.7 (26/30)70.0 (21/30)76.7 (23/30)82.0

PHI achieves the highest average success among the evaluated policies at 82.0%, exceeding the strongest baseline, π0\pi_0 (58.7%), by 23.3 percentage points and Fast-WAM (45.3%) by 36.7 percentage points. It outperforms Fast-WAM on all four contact-rich tasks and matches its 100.0% success on Walk to Table. The evaluated VLA-JEPA configuration achieves 1.3% success (2/150). Appendix A.7 reports model sizes, training resources and steps, total batch sizes, RTC settings, and action-chunk lengths, documenting the configurations used for the comparison. These additions extend the evaluation to relevant WAM alternatives and demonstrate PHI's strong performance on contact-rich humanoid manipulation.

Q2: The paper does not analyze what the future latent selects.

PHI makes task-dependent selection explicit in how its future target is learned. In Stage I, the future visual encoder follows an EMA update, while the action objective trains the future-feature resampler to retain information useful for action generation. Stage II learns to predict this action-shaped target from the current context, and Stage III trains the action expert to use the predicted representation. This procedure connects the content of the future latent directly to the manipulation objective.

We added a qualitative attention comparison across PHI, VLA-JEPA, and Fast-WAM in Appendix A.7. In the displayed Open Book example, Fast-WAM's attention is broadly distributed across the image. VLA-JEPA attends to fewer regions but still distributes attention across the scene, while PHI concentrates on the interacting hands, arms, and book. These regions are directly involved in grasping, supporting, and opening the book. The comparison illustrates PHI's emphasis on task-relevant visual interaction, complementing the quantitative evidence of representation structure and downstream use in Q4.

AuthorsOpenReview ↗

Response to Reviewer xrua(2/2)

Q3: PHI and π0\pi_0 degrade similarly under visual distractors, and PHI's absolute drop is larger on unseen books.

We thank the reviewer for highlighting an ambiguity in our original wording. Our intended claim concerns PHI's ability to transfer learned manipulation behavior to new visual and physical conditions while maintaining a substantial advantage over the evaluated baselines.

The generalization tests challenge PHI in two complementary ways: irrelevant objects introduce visual distraction, while unseen books vary in appearance, thickness, and compliance, changing the contact conditions required for manipulation. PHI achieves the highest success among the evaluated policies across these settings, maintaining a 40-percentage-point advantage under visual clutter and gains of 30.0 and 23.3 percentage points on the two unseen-book tasks. In particular, it doubles the strongest baseline's success rate on unseen-book Place Book Straight. These results demonstrate that PHI's manipulation capability extends beyond the training objects and original scene layout, supporting practical generalization across both visual variation and changes in physical interaction.

Q4: Does compactness imply partial collapse, and what further evidence shows that the action policy uses the latent?

We added two complementary quantitative analyses in Appendix A.5 to examine variation across samples and the latent's contribution to action prediction. First, we analyzed predicted and target latents over 20 Open Book trajectories:

RepresentationAcross-sample varianceEffective rank (maximum 19)PC1 variance ratioMean prediction MSE
Predicted latents0.0278864.604148.21%0.069287
Target latents0.0612035.323246.49%—

The predicted latents retain 45.6% of target variance and an effective rank of 4.6041, compared with 5.3232 for the target. Their PC1 variance ratios are also similar (48.21% and 46.49%). These measurements show that the predicted representation preserves multidimensional variation across the analyzed trajectories.

Second, we held the downstream action predictor fixed and perturbed its predicted latent on 20 seen-book Open Book samples:

ConditionJoint MSE (rad$^2$)Change from normalJoint MAE (rad)
Normal predicted latent0.00047742—0.014045
Within-latent token shuffle0.00058808+23.18%0.015572
Per-latent global-mean replacement0.00430706+802.16%0.048197
Zero replacement0.00430749+802.25%0.048196

Token shuffling increases joint action MSE by 23.18%, demonstrating sensitivity to the latent's token organization. Replacing each latent with its scalar mean or zeros increases joint MSE by approximately 802%, showing that its feature values carry information used by the action predictor.

The real-robot component ablation provides complementary behavioral evidence: including the future-learning module improves Place Book Straight success from 50.0% to 76.7%, a gain of 26.7 percentage points, with the other PHI components retained. Together, the cross-sample statistics, latent interventions, attention visualization, and behavioral ablation support the same conclusion: PHI learns a structured future representation that varies across trajectories and contributes to action generation. Combined with the expanded WAM comparison and the generalization results, these analyses strengthen the evidence for PHI's approach to contact-rich humanoid manipulation.

I thank the authors for the substantial response. Two of my concerns are resolved or partly resolved: the generalization claim has been appropriately narrowed, and Fast-WAM is a meaningful addition to the baselines. The intervention analysis in Q4 is also a real contribution and I appreciate it.

However, my central concern stands. The paper's stated contribution is that shaping the future representation by downstream action supervision is better than reconstruction-based objectives, and selectivity remains unmeasured (the added analyses establish that the latent is used and is not fully collapsed, which are different claims) Since this is the paper's central claim, I am keeping my score.

AuthorsOpenReview ↗

Respose to Reviewer xrua

Thank you for recognizing the additional comparisons and intervention analysis. We would like to clarify the distinction between the task-oriented learning objective and the resampling operation described as “selective” in the paper.

Our main contribution is task-oriented future representation learning: the future target is learned through its usefulness for downstream action generation. “Selective” refers to the learned feature aggregation and compression performed by the Perceiver resampler. Specifically, the resampler takes features from five future frames, with shape 5×512×20485\times512\times2048, and aggregates them into 512×2048512\times2048 latent features. Its learned queries attend across the input features to construct a compact representation, reducing the token count by a factor of five while retaining the 2048-dimensional feature channels. This is the architectural operation intended by the term.

The task orientation comes from how this resampler is trained. In Stage I, the downstream action loss updates the resampler, so the future target is optimized to support manipulation. Stage II learns to predict this target from the current context, and Stage III trains action generation using the predicted representation. The paper therefore contributes a procedure for constructing, predicting, and using an action-shaped future target. The Perceiver-style compression provides the representation structure; action supervision determines how that structure is learned for control.

To examine where the models focus their visual attention, we added a comparison of PHI, VLA-JEPA, and Fast-WAM in Appendix A.7. The information selected within a high-dimensional latent is difficult to visualize directly, while the attention maps show which image regions the models emphasize. In the displayed Open Book example, Fast-WAM’s attention extends broadly across the image, and VLA-JEPA’s attention remains distributed across several scene regions. PHI concentrates more strongly on the interacting hands, arms, and book—the regions directly involved in carrying out the task. This comparison provides qualitative evidence of more focused, task-relevant spatial attention in PHI. It also illustrates the motivation for our design: an action-shaped future representation can emphasize the visual interaction needed for manipulation, while full-frame reconstruction objectives encourage preserving information throughout the image.

We have also added the decoding experiment in Appendix A.8 to show more concretely what these compressed representations contain. With PHI fixed, separately trained decoders recover future hand and arm motion, the book’s position, and the spatial relationship between the hands and the book from both target and predicted latents. These results complement the attention and intervention analyses: the attention maps show where PHI focuses, the reconstructions reveal recoverable future interaction information, and the interventions show that the action predictor uses the learned representation.

Together, these analyses support our central contribution from three perspectives: task-focused visual attention, meaningful future interaction content, and its use in action generation. PHI combines learned compression of future visual features with an action-based training criterion, and the real-robot evaluations demonstrate the effectiveness of the resulting task-oriented future-prediction framework.

PHI’s advantage also extends to new visual and physical conditions. Under visual clutter, it achieves 70.0% success, compared with 30.0% for the strongest evaluated baseline. On unseen books, PHI reaches 60.0% on Place Book Straight and 56.7% on Open Book, compared with 30.0% and 33.3% for the strongest baselines, respectively. These books differ from the training objects in appearance, thickness, and compliance, requiring the policy to handle changes in both visual input and physical interaction. PHI doubles the strongest baseline’s success rate on unseen-book Place Book Straight and remains ahead on Open Book. These results show that its learned manipulation behavior transfers beyond the training objects and original scene layout, demonstrating the practical value of the framework under visual distraction and unfamiliar contact conditions.

Please let us know if you have any further questions about the method or the additional results. We would be happy to provide further clarification.

Reviewer_VwGxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PHI, a hierarchical world action model for humanoid bimanual manipulation. Instead of reconstructing future observations, PHI uses privileged future frames during training to learn a compact future latent shaped by the action loss, then trains the model to predict this latent from the current context through a three stage Exploit, Predict, and Align curriculum. A temporal force encoder over recent fingertip forces provides additional contact information to the flow matching action expert. On five Unitree G1 tasks, PHI reaches 82.0% average success, outperforming pi0 and pi0.5, and also improves under visual clutter and on unseen books.

Critical strengths
  1. The paper is well motivated. It clearly lays out the problem background and the key challenges of humanoid bimanual manipulation under evolving contact. The central question raised in the introduction is also well posed: what a future representation should preserve when its sole purpose is to support action generation.

  2. The full evaluation is on a real 29 DoF humanoid with dexterous hands and loco-manipulation. The task suite is organized by contact level, and the same initial condition sequence is used across methods (Sec. 4.1).

  3. The force featurization in Sec. 3.3 is practical and sensibly designed. Log scaling, a contact mask, and tangential to normal ratios give the policy an explicit slip signal. The ablation gives a plausible qualitative account of how failures change without it, with too little contact or too much pressing.

  4. The generalization experiments on unseen books (Fig. 5) vary both appearance and physical properties such as thickness and compliance.

Critical weaknesses
  1. The central claim, that selective, action shaped prediction outperforms reconstruction based prediction, is never tested directly. The only relevant ablation in Table 3 removes the entire future module. This shows that future conditioning is helpful, but it does not show that action shaped targets are superior to conventional ones. It is better to comare with three variants to support the claim: a target produced by a frozen encoder that receives no action gradient, as in VLA-JEPA style latent prediction; a reconstruction or feature prediction target; and a model trained without Stage I.

  2. The baselines are weak, and the gains are confounded. The paper compares only against pi0 and pi0.5. There is no WAM baseline (VLA-JEPA, Fast-WAM, GR-2 style), no force-aware policy (ForceVLA, FoAR, RDP), and no plain diffusion policy. The paper does not specify what the WAM backbone is or how it is initialized, and it does not say whether the baselines received the same training budget, RTC, and force channels. The pi0.5 result of 10% on Place Bottle, against 66.7% for pi0, also suggests the baselines may not be tuned well.

  3. The alignment analysis is weak evidence. Fig. 6 and Fig. 10 report cosine similarity of 0.969 for a single exported frame. This says little about whether prediction quality relates to success, or how it behaves out of distribution. A dataset level analysis, broken down by success and failure or by seen and unseen objects, would be more informative.

  4. The neuroscience framing, which draws on the cerebellum and the motor cortex, is largely decorative. It is not connected to any design choice that the experiments actually test.

  5. The paper also has several minor issues. Eq. 8 reuses the flow time symbol as the loss weight, which creates notational ambiguity. There are typographical errors such as “persentage” and “Alinement”. Sec. 3.2 describes the resampler output as “task oriented” before any action supervision has been introduced. The force sampling rate, and therefore the temporal span of the 10 step force history, is not reported.

Questions for authors
  1. If the privileged target is replaced by an embedding of the ground truth future action chunk and Stages II and III are kept the same, how does performance compare?

  2. How do the results compare when the Stage I target is a frozen encoder with no action gradient, or a reconstruction based target? This is the ablation that would actually support the “selective” claim.

  3. What is the WAM backbone and its pretraining? Were pi0 and pi0.5 finetuned with the same steps, batch size, RTC, and access to the force channels in the 49 dimensional state?

  4. Do the effector specific heads and loss weights alone account for part of the gain over pi0?

  5. Is predicted latent error correlated with task failure, especially on unseen books?

6 comments
AuthorsOpenReview ↗

Response to Reviewer VwGx (1/4)

We thank the reviewer for the thoughtful comments and for recognizing the motivation, real-humanoid evaluation, force-feature design, and generalization experiments.

Q1: The core claim requires further support, and comparisons with other WAM baselines are needed.

We would first like to clarify our central claim. PHI explores task-driven compression of future visual information for robot control. Reconstruction-based objectives can preserve rich visual information; our question is whether compact features shaped directly by the action objective can provide an effective predictive signal for manipulation. PHI therefore learns what future information to retain through downstream action supervision, rather than asking the policy to preserve the complete future observation. This design also addresses practical requirements of real-robot deployment, including computational cost and smooth action generation.

We added VLA-JEPA and Fast-WAM to the real-robot comparison in Table 2. Each method is evaluated over 30 trials per task; the average covers 150 trials.

Success rates (%) on the real-robot task suite.

MethodWalk to TablePlace BottleOpen BookPlace CubePlace Book StraightAverage
π0.5\pi_{0.5}100.0 (30/30)33.3 (10/30)46.7 (14/30)36.7 (11/30)43.3 (13/30)52.0
π0\pi_0100.0 (30/30)66.7 (20/30)50.0 (15/30)40.0 (12/30)36.7 (11/30)58.7
VLA-JEPA6.7 (2/30)0.0 (0/30)0.0 (0/30)0.0 (0/30)0.0 (0/30)1.3
Fast-WAM100.0 (30/30)30.0 (9/30)56.7 (17/30)10.0 (3/30)30.0 (9/30)45.3
PHI (Ours)100.0 (30/30)76.7 (23/30)86.7 (26/30)70.0 (21/30)76.7 (23/30)82.0

PHI achieves 82.0% average success, improving on the strongest baseline, π0\pi_0, by 23.3 percentage points, and on Fast-WAM by 36.7 percentage points. Together with the within-PHI future-pathway ablation and the latent perturbation analysis in Q5, these results show that the predictive pathway contributes to action generation and that PHI performs strongly against relevant WAM alternatives on this real-robot suite. The WAM comparisons are system-level evaluations: a frozen-encoder or reconstruction-target swap would instead isolate a narrower target-construction choice, not replace the need for an end-to-end WAM comparison.

We trained and tuned VLA-JEPA repeatedly; its performance remained low on this task suite despite convergence of its training loss. Its smaller model capacity may contribute to the difficulty of learning these humanoid manipulation tasks. For Fast-WAM, we additionally retrained and fine-tuned the Place Cube policy after observing poor performance. In our trials, approaching the cube from above caused hand occlusion and made grasp alignment difficult, especially for Fast-WAM. Its slower inference also interacted with action smoothing: stronger smoothing delayed grasp execution, so we selected a smoothing parameter of 5, which worked best in our tests. These observations explain the practical execution difficulties behind the reported results. Model sizes and inference settings are summarized under Q7.

Q2: Please clarify initialization, evaluation fairness, and the low π0.5\pi_{0.5} result on Place Bottle.

PHI's PaliGemma backbone is initialized from the public π0.5\pi_{0.5} checkpoint. The Perceiver, temporal force encoder, and other newly introduced modules are randomly initialized. All methods use the same demonstration data, robot platform, task definitions, action space, low-level controller, and initial-condition sequence for evaluation, as described in Section 4.1.

We trained each model to loss convergence and performed repeated training and tuning for methods with poor initial results, including VLA-JEPA. We also tuned the available execution settings, including RTC where supported and linear smoothing, rather than simply deploying every policy with a single untuned configuration.

For π0.5\pi_{0.5} on Place Bottle, we observed hand oscillations near the bottle that could knock it over. Retraining did not fully resolve this behavior. Increasing the smoothing parameter from 10 to 15 reduced some oscillation, although stronger smoothing also slowed execution. The new 30-trial evaluation after this task-specific tuning yielded 33.3% (10/30), replacing the original 10.0% (3/30). The updated result and the corresponding 52.0% overall success rate are included in Table 2 above.

The relative ordering of π0\pi_0 and π0.5\pi_{0.5} is task-dependent. Our suite contains a small set of manipulation objectives, for which the broader generalization capabilities of π0.5\pi_{0.5} need not translate into higher success. Although PHI is initialized from π0.5\pi_{0.5}, we compare its overall improvement against the stronger evaluated baseline, π0\pi_0.

AuthorsOpenReview ↗

Response to Reviewer VwGx (2/4)

Q3: The single-example alignment visualization is insufficient. How does latent error relate to success on seen and unseen books?

We expanded the analysis beyond the single-example visualization in Figures 6 and 10. The additional results cover seen/unseen books, cross-sample latent statistics, and a qualitative comparison of the three WAMs.

1. Latent MSE and success rates on seen and unseen books.

For each task, we use 30 trials with the seen book and 10 trials with each unseen book. The results are included in Appendix A.6.

Place Book Straight.

BookTrialsLatent MSESuccess rate (%)
Seen book300.06107776.7
Unseen: Deep Learning100.10720060.0
Unseen: Federated Learning100.09510960.0
Unseen: Linear Algebra Done Right100.09343860.0

Open Book (Open Book Cover).

BookTrialsLatent MSESuccess rate (%)
Seen book300.06900086.7
Unseen: Deep Learning100.05800050.0
Unseen: Federated Learning100.06100080.0
Unseen: Linear Algebra Done Right100.04700040.0

For Place Book Straight, unseen books have a higher mean MSE (0.0986 vs. 0.0611) and lower success (60.0% vs. 76.7%). For Open Book, unseen books instead have a lower mean MSE (0.0553 vs. 0.0690) alongside lower success (56.7% vs. 86.7%). These book-level results show that the relationship between latent MSE and success depends on the task, rather than following a universal monotonic trend.

One possible explanation is that MSE measures agreement with the target, whereas successful control also depends on the task-relevant information captured by that target and its interaction with historical and force information. Thus, the operational role of the predicted latent should be assessed together with its effect on downstream action generation, as examined in Q5.

2. Cross-sample latent statistics.

We additionally analyzed predicted and target latents from 20 Open Book trajectories; the statistics are reported in Appendix A.5.

RepresentationAcross-sample varianceEffective rank (maximum: 19)PC1 variance ratioMean prediction MSE
Predicted latents0.0278864.604148.21%0.069287
Target latents0.0612035.323246.49%—

The predicted latents retain 45.6% of the target variance, multiple effective dimensions, and a similar PC1 variance ratio. These measurements show that the predicted representation varies across samples and retains a multidimensional structure, providing evidence against collapse to a constant representation in the analyzed data.

3. Attention visualization.

The three-WAM attention comparison in Appendix A.7 shows that, in the displayed Open Book example, PHI emphasizes the interacting hands, arms, and book, whereas VLA-JEPA and Fast-WAM exhibit more spatially diffuse attention. This provides a qualitative view of the task-relevant regions emphasized by PHI, complementing the statistics and action-perturbation results. The figure is provided in the paper rather than embedded in this response.

Q4: How does the neuroscience motivation inform the design, and how are the presentation issues addressed?

The motivation corresponds to two concrete design choices: task-dependent selection of future information is implemented by action supervision of the future resampler in Sections 3.2 and 3.4, while effector-wise organization motivates the separate action projectors described in Section 3.1. The future-pathway ablation in Table 3 tests the utility of the former, and Q8 explains the optimization role of the latter. The neuroscience connection serves as motivation for these design choices.

The notation and implementation details are clarified as follows: Eq. (8) uses λfuture=0.1\lambda_{\mathrm{future}}=0.1 as the future-loss weight, leaving τ\tau for flow time; Section 3.2 introduces a compact future target and Section 3.4 explains how action supervision makes it task-oriented; “persentage” and “Alinement” are corrected to “percentage” and “Alignment.” All recorded modalities, including force measurements, are sampled at 30 Hz. The 10-sample force history spans 0.30 seconds between its first and last timestamps.

AuthorsOpenReview ↗

Response to Reviewer VwGx (3/4)

Q5: What if the privileged target is replaced by an embedding of the ground-truth future action chunk, keeping Stages II and III unchanged?

An action-chunk embedding is a different target from the one studied by PHI. It asks whether a policy can compress and predict the demonstrated action sequence, whereas PHI asks whether future visual interaction information, shaped by its usefulness to action generation, improves manipulation. Since the action expert is already trained to predict that action sequence, an action-derived target would change the modeling question rather than serve as a direct substitute for the future visual target. We therefore focus the real-robot comparison on WAMs that model future representations or video, and use the PHI ablation and latent intervention to examine the contribution of its predictive pathway.

To examine how the learned visual latent contributes to action generation, we performed the following perturbation experiment on 20 seen-book Open Book samples, reported in Appendix A.5.

ConditionJoint MSE (rad²)MSE change from normalJoint MAE (rad)
Normal0.00047742—0.014045
Within-latent token shuffle0.00058808+23.18%0.015572
Per-latent global-mean replacement0.00430706+802.16%0.048197
Zero replacement0.00430749+802.25%0.048196

Token shuffling increases joint MSE by 23.18%, showing sensitivity to token organization. Replacing each latent with its scalar mean or zeros increases joint MSE by approximately 802%, while joint MAE rises from 0.0140 to approximately 0.0482 rad. The nearly identical effects of mean and zero replacement indicate that the scalar mean preserves little of the signal used by the policy. The downstream action predictor therefore uses the feature values and internal structure of the predicted latent.

Q6: What happens when the Stage I target is produced by a frozen encoder with no action gradient, or by a reconstruction-based objective?

The future visual encoder in PHI already receives no direct action gradients: it follows the online encoder through EMA. The action loss instead trains the future resampler that constructs the target. A frozen encoder on its own is not a complete WAM baseline; a meaningful comparison must also specify how the future target is predicted and how the action policy uses it. We therefore added two complete WAM alternatives under the same real-robot task protocol: VLA-JEPA, which predicts future representations, and Fast-WAM, which uses future-video modeling as training supervision. Their results in Table 2 compare the resulting systems, while the future-pathway ablation and Q5's latent intervention test whether PHI's predictive pathway contributes within the same policy:

Success rates (%) on the real-robot task suite.

MethodWalk to TablePlace BottleOpen BookPlace CubePlace Book StraightAverage
π0.5\pi_{0.5}100.0 (30/30)33.3 (10/30)46.7 (14/30)36.7 (11/30)43.3 (13/30)52.0
π0\pi_0100.0 (30/30)66.7 (20/30)50.0 (15/30)40.0 (12/30)36.7 (11/30)58.7
VLA-JEPA6.7 (2/30)0.0 (0/30)0.0 (0/30)0.0 (0/30)0.0 (0/30)1.3
Fast-WAM100.0 (30/30)30.0 (9/30)56.7 (17/30)10.0 (3/30)30.0 (9/30)45.3
PHI (Ours)100.0 (30/30)76.7 (23/30)86.7 (26/30)70.0 (21/30)76.7 (23/30)82.0

PHI improves average success over Fast-WAM by 36.7 percentage points. Importantly, the result is not explained solely by the temporal force encoder: on Place Book Straight, PHI without that encoder still achieves 63.3%, compared with 30.0% for Fast-WAM, a 33.3-point advantage. The relevant results from Tables 2 and 3 are summarized below.

Method / variantPlace Book Straight success rate (%)
Fast-WAM30.0 (9/30)
PHI without temporal force encoder63.3 (19/30)
PHI without future-learning module50.0 (15/30)
PHI76.7 (23/30)

The temporal-force ablation retains current force measurements in the robot state. Within PHI, retaining the future pathway improves success from 50.0% to 76.7% with the other components retained. Together, the WAM comparisons and internal ablations support PHI's system-level effectiveness and the contribution of its task-driven future-prediction pathway on these manipulation tasks.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to the Reviewers

We thank all three reviewers for their thoughtful comments. A common concern across the reviews is whether PHI’s performance has been adequately compared with other World Action Models (WAMs), and what evidence supports the distinction between PHI and these methods. To address this shared concern, we added real-robot evaluations of VLA-JEPA and Fast-WAM under the same task protocol as the existing baselines, together with a comparison of model configurations and a new attention visualization across the three WAMs. We have updated the paper, and all revised and newly added material is highlighted in blue.

Same-protocol WAM evaluation. Table 2 reports success rates over five Unitree G1 tasks, with 30 trials per task. All evaluated policies are initialized from publicly released pretrained parameters and fine-tuned on our collected data. They receive current force measurements through the robot-state input; PHI additionally encodes recent force history. The policies use the same robot platform, task definitions, action space, low-level controller, and evaluation initial-condition sequence.

MethodWalk to TablePlace BottleOpen BookPlace CubePlace Book StraightAverage
(\pi_{0.5})100.0 (30/30)33.3 (10/30)46.7 (14/30)36.7 (11/30)43.3 (13/30)52.0
(\pi_0)100.0 (30/30)66.7 (20/30)50.0 (15/30)40.0 (12/30)36.7 (11/30)58.7
VLA-JEPA6.7 (2/30)0.0 (0/30)0.0 (0/30)0.0 (0/30)0.0 (0/30)1.3
Fast-WAM100.0 (30/30)30.0 (9/30)56.7 (17/30)10.0 (3/30)30.0 (9/30)45.3
PHI (Ours)100.0 (30/30)76.7 (23/30)86.7 (26/30)70.0 (21/30)76.7 (23/30)82.0

PHI achieves 82.0% average success, compared with 58.7% for the strongest baseline, (\pi_0), 45.3% for Fast-WAM, and 1.3% for VLA-JEPA. We trained and tuned VLA-JEPA over multiple runs; even when its training loss converged, its measured real-robot performance remained low. We report the observed result rather than treating the method as an unevaluated or failed run.

For transparency, Appendix A.7 reports model size, training steps, inference latency, RTC availability, and smoothing settings:

MethodParametersTraining stepsInference latencyRTCLinear smoothingAverage success
(\pi_0)4.0B500k/GPU100.99 msYes1058.7%
(\pi_{0.5})4.1B500k/GPU114.93 msYes1052.0%
VLA-JEPA2.8B400k/GPU39.20 msNo101.3%
Fast-WAM12.4B (5.68B LLM)400k/GPU183.63 msNo545.3%
PHI4.7B500k/GPU159.48 msYes1082.0%

These results provide a direct comparison of success rates and deployment configurations under the reported evaluation protocol.

New qualitative comparison of the WAM representations. We added an attention visualization for the Open Book Cover task comparing Fast-WAM, VLA-JEPA, and PHI. In the displayed example, Fast-WAM’s attention is the most spatially diffuse and extends across much of the image. VLA-JEPA attends to fewer regions than Fast-WAM, but its attention remains distributed across the scene. PHI’s attention is more concentrated on the task-relevant hands, arms, and book. This visualization offers a qualitative comparison of which regions the three models emphasize in their predicted latent spaces.

Quantitative analysis of PHI’s predicted latent. Appendix A.5 analyzes predicted and target latents over 20 Open Book trajectories. The predicted latents have an across-sample variance of 0.027886, an effective rank of 4.6041, a PC1 variance ratio of 48.21%, and a mean prediction MSE of 0.069287. The corresponding target values are 0.061203, 5.3232, and 46.49% for variance, effective rank, and PC1 variance ratio. Thus, the predicted latents retain 45.6% of target variance and multiple effective dimensions.

We also perturbed the predicted latent while holding the action predictor fixed. Shuffling tokens within each latent increases joint action MSE by 23.18%; replacing each latent with its scalar mean or with zeros increases it by approximately 802%. These results complement the attention visualization: the comparison shows where the WAMs attend in the example, while the perturbation analysis shows that PHI’s predicted latent affects downstream action prediction.

Together, the added same-protocol WAM evaluations and representation comparisons address the shared concern about the scope of the baseline comparison. PHI achieves the highest average success among the evaluated methods, while the new attention visualization and latent analyses provide complementary evidence about how its future representation differs and contributes to action generation.

Dear Reviewers,

Thank you again for your reviews of this submission. The authors have now posted their responses, and the author–reviewer discussion period ends on November 18, 2026 (AoE).

Please read the authors' rebuttal and let them know whether it addresses your concerns. A short reply is enough, for example:

  • which of your concerns are now resolved;
  • which remain open, and what evidence or clarification would resolve them;
  • whether you plan to update your rating.

If the rebuttal changes your assessment, please update your review and rating before the discussion period ends. If you have already engaged, thank you, and no further action is needed unless the authors reply again.

Your participation is important for a fair and well-informed decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.