- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PhysWord is a reasoning-before-acting VLA built as a mixture of transformers: a Cosmos-Predict2.5 temporal branch and a Qwen3-VL-2B branch for reasoning and action. Before denoising actions, it autoregressively predicts "Physical Words", which are tokens for 16 atomic verbs with visual and tactile variants (pick_v, pick_t). Each word's hidden state gets two losses on top of : cross-entropy against a verb label annotated by GPT, and alignment to a pooled future visual or tactile latent. Words are predicted every steps over . After pretraining on about 7.4k hours of data, the model reaches 90.3% on UniVTAC and 99.6% on LIBERO-Long, and beats π0.5 and FTP-1 on four real-robot tasks with tactile sensing.
Critical strengths
- Tying a discrete verb to a continuous future latent through a single shared hidden state is clean, and it works across embodiments, with or without tactile input.
- The real-world evaluation is substantial: bimanual grippers and 22-DoF tactile dexterous hands, tests on unseen configurations, and a compelling extended-horizon result (70% vs. 0% on repeated Pick A4 Paper).
- Engineering detail and reproducibility are good: runtime optimisation to 10.6 Hz, attention masks, data filtering, and an annotation accuracy check.
Critical weaknesses
-
The novelty is incremental, and key prior work is missing. Each ingredient already exists:
- Predicting atomic skills or subtasks: AtomicVLA, π0.5, RT-H.
- Aligning hidden states to future latents: FLARE, VLA-JEPA, LaRA-VLA, LaST0.
- Predicting future tactile latents: DreamTacVLA, UniTacVLA.
- A video-model branch inside an MoT: Cosmos Policy, Motus.
The real delta is putting cross-entropy and cosine losses on the same token. FLARE, VLA-JEPA, DreamTacVLA and UniTacVLA aren't cited, and AtomicVLA is cited but neither discussed nor compared against.
-
The headline SOTA is confounded. The gains could come from the Cosmos video branch, 7.4k hours of pretraining (including 2k hours of in-house human video), the FTP-1-initialised T3 encoder, higher input resolution than π0.5 (224px), or GPT-annotated labels that no baseline receives. Removing Cosmos alone costs 3.9 points (PhysWord-Lite, 86.4%). The ablations lack two controls:
- the same backbone and data with no reasoning tokens at all;
- Physical Words without pretraining.
-
Possible benchmark contamination. PhysWord pretrains on OpenNeoData (5,000+ h). N0-VTLA's data pipeline explicitly converts UniVTAC-simulator episodes into that corpus. If UniVTAC task episodes are in pretraining, the "50 demos per task" protocol is misleading for PhysWord (and N0-VTLA) relative to FTP-1, T-Rex, InternVLA-A1, etc. The claim of "no advantage in pretraining data volume" does not address overlap.
-
The 16-word vocabulary is unjustified. No vocabulary-size ablation, no per-word frequency in the 7k h corpus, and no long-tail analysis. Common verbs (pour, wipe, fold, cut, twist, tear, screw) are absent; the paper's own Express Packing and Screw Driving tasks must be mapped to approximate labels. Word-prediction accuracy (90.5%) and all representation analyses cover only the four most frequent words. The "extensible vocabulary" claim is untested. With 2k h of diverse human-hand data, Move/Approach likely act as catch-all bins, and the reported 41% intra-class distance reduction may partly reflect collapsing physically distinct motions into those bins.
Questions for authors
-
How does predicting a verb class differ from subtask planning, and why is less information better? A subtask such as "pick up the bottle and place it on the shelf" carries the verb, the object, the target, and their relation; a Physical Word keeps only the verb. If the object and spatial content are meant to live in the latent , show that it does (e.g., probe for object identity/position), and show that a subtask token with the same latent alignment does not match PhysWord. Otherwise the vocabulary is a strictly less informative subtask predictor, and the gain must be attributed entirely to the alignment loss.
-
Why do approach/move get tactile variants when no contact exists? What do their tactile targets encode?
-
LIBERO-Long is saturated. Can you add a non-saturated vision benchmark (RoboCasa, LIBERO-Plus/Pro, RoboTwin2.0, RoboDojo)?
12 comments
Responses to Weaknesses 1 (part 2)
[W1]-2. Architectural choices.
We adopt MoT and a pretrained video model as established components supporting joint training and temporal perception. These architectural choices are not our main contribution. Our contribution lies in using pretrained action semantics to organize future physical information into structured reasoning states for action generation.
Here we aim to show even without such architecture (e.g., Cosmos), Physical Word reasoning and future visual/tactile latent prediction remain intact when the Cosmos branch is removed. We have evaluated this variant on UniVTAC. As shown in Table 7 of revised version, this variant (named as PhysWord-Lite) achieves 86.4%, resulting only a 3.9% decrease compared to PhysWord. This shows that though the temporal branch can strengthen perception, Physical Words alone retain their effectiveness even without this additional support.
We further provide such comparisons in real-world tasks below. Following the long-horizon protocol in Appendix C.3.1, we compare full PhysWord with two ablations on extended Pick A4 Paper and Express Packing:
- w/o Cosmos, which removes the video branch while retaining complete Physical Words
- w/o intermediate reasoning, which removes all intermediate word/latent reasoning tokens and their supervision while retaining Cosmos.
All other settings follow Table 15, and we include the existing latent-only variant for reference. As shown in the table, removing the entire Physical Word reasoning stage causes a substantially larger performance drop than removing the Cosmos video branch on both real-world tasks. The latent-only variant also underperforms full PhysWord, further supporting the value of semantic anchoring within the reasoning stage. Together, these results identify Physical Word reasoning as a key contributor to long-horizon execution, with Cosmos providing complementary temporal perception. We report the detailed settings and results of these real-world ablations in Appendix C.3.3.
| Variant | Extended Pick A4 Paper | Extended Express Packing |
|---|---|---|
| Full PhysWord | 70.0% | 66.7% |
| Latent-only reasoning(Table 15) | 36.7% | 50.0% |
| w/o Cosmos | 53.3% | 50.0% |
| w/o intermediate reasoning | 33.3% | 36.7% |
[W1]-3. Missing references and methodological distinctions.
We apologize for the missing references and add FLARE, VLA-JEPA, DreamTacVLA, and UniTacVLA to Related work section in the revised main paper. FLARE aligns projected future-token features within a diffusion policy with future observation embeddings. VLA-JEPA learns continuous latent action representations by conditioning a latent world model to predict future states. DreamTacVLA combines hierarchical visuotactile alignment with action-conditioned future tactile-latent prediction for action refinement. UniTacVLA jointly learns unified tactile representations through language-based tactile reasoning and coarse-to-fine future tactile prediction. In contrast, our contribution is a new semantically anchored latent-reasoning paradigm, in which pretrained action semantics guide continuous physical representation learning through joint semantic and future-latent supervision. Beyond PhysWord, this formulation offers a transferable design principle for other latent-reasoning methods.
[W1]-4. Additional comparison with AtomicVLA.
Since AtomicVLA does not use tactile inputs, we compare it with PhysWord on LIBERO-Long and LIBERO-Plus (Long) under vision-only settings. We evaluate our LIBERO-Long checkpoint and AtomicVLA’s publicly released checkpoint on the LIBERO-Long subset of LIBERO-Plus, using the official evaluation protocol. PhysWord outperforms AtomicVLA in both evaluations.
| Method | LIBERO-Long | LIBERO-Plus (Long) |
|---|---|---|
| AtomicVLA | 96.2% | 78.8% |
| PhysWord(vision-only) | 99.6% | 80.7% |
AtomicVLA uses atomic skill abstractions to guide expert routing, while PhysWord anchors future visual predictions in action semantics within its intermediate reasoning states. This provides action generation with context representing both the intended skill and its anticipated physical evolution. The results support the usefulness of this predictive reasoning context for long-horizon manipulation under both standard and perturbed conditions. We include these comparisons in Table 1, the main paper’s simulation results table.
Finally, we add a comparison with TacDyn-WAM, a recent tactile world-action model, on UniVTAC and report the results in Table 1. TacDyn-WAM achieves a mean success rate of 81.5%, which is 8.8 percentage points lower than PhysWord’s 90.3%.
Responses to Weaknesses 2 (part 1)
[W2] Controlling Architecture, Pretraining, and Perception
Thank you for identifying these additional ablation studies. We add five controlled comparisons to distinguish the contribution of Physical Words from the backbone architecture, additional pretraining, tactile encoder initialization, and input resolution.
[W2]-1. Matched direct-action baseline.
Simulation tasks. We evaluate w/o intermediate reasoning on Put Bottle and Lift Can, the two UniVTAC tasks used in Figure 3. This variant removes all intermediate word/latent reasoning tokens during both training and inference and is trained without their corresponding classification and latent-alignment losses. By removing all reasoning tokens, this aims to show whether the benefit comes from the designed physical word reasoning. All other settings remain identical to those used in the ablation study. The policy therefore generates actions directly from the perceptual context and instruction.
This ablation achieves 69% and 63% success on Put Bottle and Lift Can, respectively, compared with 93.0% and 95.0% for full PhysWord, corresponding to decreases of 24 and 32 percentage points. These controlled comparisons support the effectiveness of Physical Word reasoning and its supervision in improving manipulation performance under matched architectural and training conditions.
| Method | Put Bottle | Lift Can |
|---|---|---|
| w/o intermediate reasoning | 69% | 63% |
| PhysWord | 93% | 95% |
| Improvement | 24% | 32% |
Real-world long-horizon tasks. We have evaluated the removal of all intermediate reasoning tokens on real-world long-horizon tasks in W1-Part 2. This leads to a substantial performance drop, from 70.0% to 33.3% on Extended Pick A4 Paper and from 66.7% to 36.7% on Extended Express Packing. These results are consistent with our simulator experiments, further demonstrating the importance of semantic anchoring in the intermediate reasoning process. We incorporate these simulation and real-world controls into Figure 3 (left) and Appendix C.3.3, respectively.
[W2]-2. Controlling access to GPT-annotated action labels.
This aims to show whether the gain comes from providing annotated action semantics or the learned semantic knowledge. Therefore, we further append a GPT-generated verb sequence to the original instruction (instruction + GPT-generated verb variant) on Put Bottle and Lift Can. Specifically, at each policy step, GPT provides a proposed sequence of action verbs for the upcoming action chunk based on the current observations and task instruction.
For example, the augmented instruction is: “[original instruction] + [The next actions are: Approach, Approach,…].”
This ablation appends externally supplied verbs to the task instruction as semantic guidance while retaining visual and tactile latent predictions and their alignment losses. The policy generates actions conditioned on the augmented instruction, perceptual context, and predicted latents. It does not generate intermediate action-word tokens or use the corresponding word generation loss.
| Method | Put Bottle | Lift Can |
|---|---|---|
| Instruction+verb | 81% | 84% |
| PhysWord | 93% | 95% |
PhysWord outperforms instruction + verb conditioning by 12 and 11 percentage points, respectively. These results suggest that coupling action semantics with future physical prediction within intermediate reasoning states helps the model learn semantically structured motion representations more effectively than simply conditioning on GPT-generated action words. We include these comparisons in Appendix C.2.
[W2]-3. Physical Words without additional pretraining.
This aims to show whether the benefit comes from pretraining or our design. We retain the same original pretrained Cosmos, Qwen, and T3 initialization but skip Stage 1 on the approximately 7,400 hours of manipulation data, directly training the complete PhysWord model on downstream demonstrations. The downstream SFT and evaluation protocols remain unchanged.
| Method | UniVTAC mean success |
|---|---|
| Previous SOTA N0-VTLA | 83.1% |
| PhysWord without additional robot pretraining | 88.3% |
| Full PhysWord | 90.3% |
Even without additional Stage 1 pretraining, PhysWord achieves 88.3% on UniVTAC, retaining strong manipulation performance after downstream SFT. This result is consistent with our design: action-word embeddings already inherit semantic knowledge from large-scale VLM pretraining, providing shared semantic anchors that guide the learning of future physical latents from downstream demonstrations. These findings demonstrate the effectiveness of PhysWord even without the additional large-scale manipulation pretraining stage.
Responses to Weaknesses 1 (part 1)
Dear Reviewer fNh6
We sincerely thank you for recognizing the clean design of tying a discrete verb to a continuous future latent through a single shared hidden state, our substantial real-world evaluation and compelling extended-horizon result, and our good engineering detail and reproducibility. We respond to your valuable comments point by point below. We have updated the manuscript, with the corresponding revisions marked in blue.
[W1] Novelty and Comparison with Prior Work
Thank you for your detailed comments. We clarify our contribution, provide supporting evidence, and address the missing discussions and comparisons below. The corresponding revisions in the main paper are marked in blue.
[W1]-1. Distinction from subtask planning and latent reasoning.
Our core contribution is a structured latent-reasoning paradigm that uses pretrained action semantics to guide physical representation learning. Language-based subtask planning (AtomicVLA, pi0.5, RT-H) struggles to express fine-grained motion and contact dynamics, while future-latent alignment (FLARE, VLA_JEPA, LaRA-VLA, LaST0) or future tactile latent (DreamTacVLA, UniTacVLA)alone lacks explicit structure for organizing and reusing physical knowledge across contexts. To our knowledge, PhysWord is the first robotic reasoning framework designed to synergistically unlock the potential of both semantic reasoning and latent alignment, going beyond either approach in isolation or their naive combination. Simply concatenating subtask descriptions with separately visual and tactile latent tokens does not explicitly couple semantic and physical supervision within the same hidden states. In contrast, we use pretrained action-word semantics and the VLM’s native autoregressive generation process to jointly organize future visual and tactile latent predictions.
Specifically, each action word’s contextual hidden state receives both semantic classification and corresponding future-latent supervision: pretrained semantics guide the organization of physical representations, while sensory targets ground them in context-specific dynamics. This coupling turns pretrained language knowledge into a structural prior for physical learning, rather than simply combining language and latent outputs. The resulting semantically structured, physically predictive reasoning states provide context for action generation.
Our ablations support this semantic-guidance mechanism from three complementary perspectives: