acceptodds
Under review as a conference paper at ICLR 2027

Physical Words: A Vocabulary for Structuring Latent Reasoning in Robot Manipulation

Abstract

Multimodal reasoning provides robotic foundation models with predictive context needed for reliable action generation. However, language-based reasoning captures task semantics but struggles to express fine-grained physical dynamics, while latent reasoning models future state evolution but offers limited structure for organizing these dynamics into reusable reasoning primitives. To bridge this gap, we introduce PhysWord, a reasoning-before-acting foundation model built on Physical Words, which use atomic action verbs as shared semantic anchors for continuous representations of physical dynamics. Specifically, PhysWord predicts upcoming atomic action words and directly aligns their contextual hidden states with latent targets derived from the corresponding future observations. Building on these primitives, we construct a temporally structured PhysWord space with visual and optional tactile variants of each action word. Autoregressively predicting Physical Words across future timesteps captures transitions between execution stages, providing structured multimodal context for action generation. PhysWord achieves state-of-the-art success rates of 90.3% on visuotactile UniVTAC and 99.6% on vision-based LIBERO-Long, and outperforms competing methods on real-world tasks using bimanual grippers and dexterous hands. Further evaluations demonstrate that our structured latent reasoning framework generalizes to unseen scenarios and enables long-horizon manipulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Manipulation+Robot Foundation Model+Latent Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fNh6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PhysWord is a reasoning-before-acting VLA built as a mixture of transformers: a Cosmos-Predict2.5 temporal branch and a Qwen3-VL-2B branch for reasoning and action. Before denoising actions, it autoregressively predicts "Physical Words", which are tokens for 16 atomic verbs with visual and tactile variants (pick_v, pick_t). Each word's hidden state gets two losses on top of Lvideo+Lact\mathcal{L}{\text{video}}+\mathcal{L}{\text{act}}: cross-entropy against a verb label annotated by GPT, and 1−cos⁡(z,g)1-\cos(z,g) alignment to a pooled future visual or tactile latent. Words are predicted every s=4s{=}4 steps over H=16H{=}16. After pretraining on about 7.4k hours of data, the model reaches 90.3% on UniVTAC and 99.6% on LIBERO-Long, and beats π0.5 and FTP-1 on four real-robot tasks with tactile sensing.

Critical strengths
  • Tying a discrete verb to a continuous future latent through a single shared hidden state is clean, and it works across embodiments, with or without tactile input.
  • The real-world evaluation is substantial: bimanual grippers and 22-DoF tactile dexterous hands, tests on unseen configurations, and a compelling extended-horizon result (70% vs. 0% on repeated Pick A4 Paper).
  • Engineering detail and reproducibility are good: runtime optimisation to 10.6 Hz, attention masks, data filtering, and an annotation accuracy check.
Critical weaknesses
  1. The novelty is incremental, and key prior work is missing. Each ingredient already exists:

    • Predicting atomic skills or subtasks: AtomicVLA, π0.5, RT-H.
    • Aligning hidden states to future latents: FLARE, VLA-JEPA, LaRA-VLA, LaST0.
    • Predicting future tactile latents: DreamTacVLA, UniTacVLA.
    • A video-model branch inside an MoT: Cosmos Policy, Motus.

    The real delta is putting cross-entropy and cosine losses on the same token. FLARE, VLA-JEPA, DreamTacVLA and UniTacVLA aren't cited, and AtomicVLA is cited but neither discussed nor compared against.

  2. The headline SOTA is confounded. The gains could come from the Cosmos video branch, 7.4k hours of pretraining (including 2k hours of in-house human video), the FTP-1-initialised T3 encoder, higher input resolution than π0.5 (224px), or GPT-annotated labels that no baseline receives. Removing Cosmos alone costs 3.9 points (PhysWord-Lite, 86.4%). The ablations lack two controls:

    • the same backbone and data with no reasoning tokens at all;
    • Physical Words without pretraining.
  3. Possible benchmark contamination. PhysWord pretrains on OpenNeoData (5,000+ h). N0-VTLA's data pipeline explicitly converts UniVTAC-simulator episodes into that corpus. If UniVTAC task episodes are in pretraining, the "50 demos per task" protocol is misleading for PhysWord (and N0-VTLA) relative to FTP-1, T-Rex, InternVLA-A1, etc. The claim of "no advantage in pretraining data volume" does not address overlap.

  4. The 16-word vocabulary is unjustified. No vocabulary-size ablation, no per-word frequency in the 7k h corpus, and no long-tail analysis. Common verbs (pour, wipe, fold, cut, twist, tear, screw) are absent; the paper's own Express Packing and Screw Driving tasks must be mapped to approximate labels. Word-prediction accuracy (90.5%) and all representation analyses cover only the four most frequent words. The "extensible vocabulary" claim is untested. With 2k h of diverse human-hand data, Move/Approach likely act as catch-all bins, and the reported 41% intra-class distance reduction may partly reflect collapsing physically distinct motions into those bins.

Questions for authors
  1. How does predicting a verb class differ from subtask planning, and why is less information better? A subtask such as "pick up the bottle and place it on the shelf" carries the verb, the object, the target, and their relation; a Physical Word keeps only the verb. If the object and spatial content are meant to live in the latent zz, show that it does (e.g., probe zz for object identity/position), and show that a subtask token with the same latent alignment does not match PhysWord. Otherwise the vocabulary is a strictly less informative subtask predictor, and the gain must be attributed entirely to the alignment loss.

  2. Why do approach/move get tactile variants when no contact exists? What do their tactile targets encode?

  3. LIBERO-Long is saturated. Can you add a non-saturated vision benchmark (RoboCasa, LIBERO-Plus/Pro, RoboTwin2.0, RoboDojo)?

12 comments
AuthorsOpenReview ↗

Responses to Weaknesses 1 (part 1)

Dear Reviewer fNh6

We sincerely thank you for recognizing the clean design of tying a discrete verb to a continuous future latent through a single shared hidden state, our substantial real-world evaluation and compelling extended-horizon result, and our good engineering detail and reproducibility. We respond to your valuable comments point by point below. We have updated the manuscript, with the corresponding revisions marked in blue.

[W1] Novelty and Comparison with Prior Work

Thank you for your detailed comments. We clarify our contribution, provide supporting evidence, and address the missing discussions and comparisons below. The corresponding revisions in the main paper are marked in blue.

[W1]-1. Distinction from subtask planning and latent reasoning.

Our core contribution is a structured latent-reasoning paradigm that uses pretrained action semantics to guide physical representation learning. Language-based subtask planning (AtomicVLA, pi0.5, RT-H) struggles to express fine-grained motion and contact dynamics, while future-latent alignment (FLARE, VLA_JEPA, LaRA-VLA, LaST0) or future tactile latent (DreamTacVLA, UniTacVLA)alone lacks explicit structure for organizing and reusing physical knowledge across contexts. To our knowledge, PhysWord is the first robotic reasoning framework designed to synergistically unlock the potential of both semantic reasoning and latent alignment, going beyond either approach in isolation or their naive combination. Simply concatenating subtask descriptions with separately visual and tactile latent tokens does not explicitly couple semantic and physical supervision within the same hidden states. In contrast, we use pretrained action-word semantics and the VLM’s native autoregressive generation process to jointly organize future visual and tactile latent predictions.

Specifically, each action word’s contextual hidden state receives both semantic classification and corresponding future-latent supervision: pretrained semantics guide the organization of physical representations, while sensory targets ground them in context-specific dynamics. This coupling turns pretrained language knowledge into a structural prior for physical learning, rather than simply combining language and latent outputs. The resulting semantically structured, physically predictive reasoning states provide context for action generation.

Our ablations support this semantic-guidance mechanism from three complementary perspectives:

  • Semantic supervision improves physical latent reasoning. In Figure 3 (left), joint word–visual supervision achieves 99.6% on LIBERO-Long, versus 97.0% with visual-latent-only and 97.4% with word-only supervision. The benefit extends to tactile representations: joint word–tactile supervision improves Put Bottle success from 73% to 88%.
  • How semantics and dynamics are coupled matters. As reported in Section 4.2.1 (Group 3) and summarized in the table below, pretrained VLM word embeddings consistently outperform random initialization across both reasoning orders and tasks. These results suggest that semantic knowledge acquired through large-scale VLM pretraining provides a useful prior for organizing continuous physical latents. By jointly supervising the same contextual hidden states, PhysWord transfers this pretrained semantic knowledge into dynamic prediction, supporting latent reasoning.
Reasoning orderWord-embedding initializationPut BottleLift Can
WT → WOPretrained VLM embeddings90%90%
WT → WORandom initialization84%86%
WO → WTPretrained VLM embeddings93%95%
WO → WTRandom initialization89%90%
  • Attention provides complementary qualitative evidence. Figure 4(b) shows that joint supervision directs attention more consistently toward manipulated objects and task-relevant interaction regions, supporting improved task grounding.
AuthorsOpenReview ↗

Responses to Weaknesses 1 (part 2)

[W1]-2. Architectural choices.

We adopt MoT and a pretrained video model as established components supporting joint training and temporal perception. These architectural choices are not our main contribution. Our contribution lies in using pretrained action semantics to organize future physical information into structured reasoning states for action generation.

Here we aim to show even without such architecture (e.g., Cosmos), Physical Word reasoning and future visual/tactile latent prediction remain intact when the Cosmos branch is removed. We have evaluated this variant on UniVTAC. As shown in Table 7 of revised version, this variant (named as PhysWord-Lite) achieves 86.4%, resulting only a 3.9% decrease compared to PhysWord. This shows that though the temporal branch can strengthen perception, Physical Words alone retain their effectiveness even without this additional support.

We further provide such comparisons in real-world tasks below. Following the long-horizon protocol in Appendix C.3.1, we compare full PhysWord with two ablations on extended Pick A4 Paper and Express Packing:

  • w/o Cosmos, which removes the video branch while retaining complete Physical Words
  • w/o intermediate reasoning, which removes all intermediate word/latent reasoning tokens and their supervision while retaining Cosmos.

All other settings follow Table 15, and we include the existing latent-only variant for reference. As shown in the table, removing the entire Physical Word reasoning stage causes a substantially larger performance drop than removing the Cosmos video branch on both real-world tasks. The latent-only variant also underperforms full PhysWord, further supporting the value of semantic anchoring within the reasoning stage. Together, these results identify Physical Word reasoning as a key contributor to long-horizon execution, with Cosmos providing complementary temporal perception. We report the detailed settings and results of these real-world ablations in Appendix C.3.3.

VariantExtended Pick A4 PaperExtended Express Packing
Full PhysWord70.0%66.7%
Latent-only reasoning(Table 15)36.7%50.0%
w/o Cosmos53.3%50.0%
w/o intermediate reasoning33.3%36.7%

[W1]-3. Missing references and methodological distinctions.

We apologize for the missing references and add FLARE, VLA-JEPA, DreamTacVLA, and UniTacVLA to Related work section in the revised main paper. FLARE aligns projected future-token features within a diffusion policy with future observation embeddings. VLA-JEPA learns continuous latent action representations by conditioning a latent world model to predict future states. DreamTacVLA combines hierarchical visuotactile alignment with action-conditioned future tactile-latent prediction for action refinement. UniTacVLA jointly learns unified tactile representations through language-based tactile reasoning and coarse-to-fine future tactile prediction. In contrast, our contribution is a new semantically anchored latent-reasoning paradigm, in which pretrained action semantics guide continuous physical representation learning through joint semantic and future-latent supervision. Beyond PhysWord, this formulation offers a transferable design principle for other latent-reasoning methods.

[W1]-4. Additional comparison with AtomicVLA.

Since AtomicVLA does not use tactile inputs, we compare it with PhysWord on LIBERO-Long and LIBERO-Plus (Long) under vision-only settings. We evaluate our LIBERO-Long checkpoint and AtomicVLA’s publicly released checkpoint on the LIBERO-Long subset of LIBERO-Plus, using the official evaluation protocol. PhysWord outperforms AtomicVLA in both evaluations.

MethodLIBERO-LongLIBERO-Plus (Long)
AtomicVLA96.2%78.8%
PhysWord(vision-only)99.6%80.7%

AtomicVLA uses atomic skill abstractions to guide expert routing, while PhysWord anchors future visual predictions in action semantics within its intermediate reasoning states. This provides action generation with context representing both the intended skill and its anticipated physical evolution. The results support the usefulness of this predictive reasoning context for long-horizon manipulation under both standard and perturbed conditions. We include these comparisons in Table 1, the main paper’s simulation results table.

Finally, we add a comparison with TacDyn-WAM, a recent tactile world-action model, on UniVTAC and report the results in Table 1. TacDyn-WAM achieves a mean success rate of 81.5%, which is 8.8 percentage points lower than PhysWord’s 90.3%.

AuthorsOpenReview ↗

Responses to Weaknesses 2 (part 1)

[W2] Controlling Architecture, Pretraining, and Perception

Thank you for identifying these additional ablation studies. We add five controlled comparisons to distinguish the contribution of Physical Words from the backbone architecture, additional pretraining, tactile encoder initialization, and input resolution.

[W2]-1. Matched direct-action baseline.

Simulation tasks. We evaluate w/o intermediate reasoning on Put Bottle and Lift Can, the two UniVTAC tasks used in Figure 3. This variant removes all intermediate word/latent reasoning tokens during both training and inference and is trained without their corresponding classification and latent-alignment losses. By removing all reasoning tokens, this aims to show whether the benefit comes from the designed physical word reasoning. All other settings remain identical to those used in the ablation study. The policy therefore generates actions directly from the perceptual context and instruction.

This ablation achieves 69% and 63% success on Put Bottle and Lift Can, respectively, compared with 93.0% and 95.0% for full PhysWord, corresponding to decreases of 24 and 32 percentage points. These controlled comparisons support the effectiveness of Physical Word reasoning and its supervision in improving manipulation performance under matched architectural and training conditions.

MethodPut BottleLift Can
w/o intermediate reasoning69%63%
PhysWord93%95%
Improvement24%32%

Real-world long-horizon tasks. We have evaluated the removal of all intermediate reasoning tokens on real-world long-horizon tasks in W1-Part 2. This leads to a substantial performance drop, from 70.0% to 33.3% on Extended Pick A4 Paper and from 66.7% to 36.7% on Extended Express Packing. These results are consistent with our simulator experiments, further demonstrating the importance of semantic anchoring in the intermediate reasoning process. We incorporate these simulation and real-world controls into Figure 3 (left) and Appendix C.3.3, respectively.

[W2]-2. Controlling access to GPT-annotated action labels.

This aims to show whether the gain comes from providing annotated action semantics or the learned semantic knowledge. Therefore, we further append a GPT-generated verb sequence to the original instruction (instruction + GPT-generated verb variant) on Put Bottle and Lift Can. Specifically, at each policy step, GPT provides a proposed sequence of action verbs for the upcoming action chunk based on the current observations and task instruction.

For example, the augmented instruction is: “[original instruction] + [The next actions are: Approach, Approach,…].”

This ablation appends externally supplied verbs to the task instruction as semantic guidance while retaining visual and tactile latent predictions and their alignment losses. The policy generates actions conditioned on the augmented instruction, perceptual context, and predicted latents. It does not generate intermediate action-word tokens or use the corresponding word generation loss.

MethodPut BottleLift Can
Instruction+verb81%84%
PhysWord93%95%

PhysWord outperforms instruction + verb conditioning by 12 and 11 percentage points, respectively. These results suggest that coupling action semantics with future physical prediction within intermediate reasoning states helps the model learn semantically structured motion representations more effectively than simply conditioning on GPT-generated action words. We include these comparisons in Appendix C.2.

[W2]-3. Physical Words without additional pretraining.

This aims to show whether the benefit comes from pretraining or our design. We retain the same original pretrained Cosmos, Qwen, and T3 initialization but skip Stage 1 on the approximately 7,400 hours of manipulation data, directly training the complete PhysWord model on downstream demonstrations. The downstream SFT and evaluation protocols remain unchanged.

MethodUniVTAC mean success
Previous SOTA N0-VTLA83.1%
PhysWord without additional robot pretraining88.3%
Full PhysWord90.3%

Even without additional Stage 1 pretraining, PhysWord achieves 88.3% on UniVTAC, retaining strong manipulation performance after downstream SFT. This result is consistent with our design: action-word embeddings already inherit semantic knowledge from large-scale VLM pretraining, providing shared semantic anchors that guide the learning of future physical latents from downstream demonstrations. These findings demonstrate the effectiveness of PhysWord even without the additional large-scale manipulation pretraining stage.

Reviewer_obh3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PhysWord, a reasoning-before-acting foundation model, which uses atomic action verbs as shared semantic anchors for continuous representations of physical dynamics. PhysWord predicts upcoming atomic action primitives and aligns their contextual hidden states with latent targets derived from the corresponding future observations. These predicted primitive tokens provide a structured reasoning context for action generation.

Critical strengths
  • The paper proposes a novel method combining latent and semantic supervision for action primitive representation, which provides a structured reasoning context for robot manipulation.

  • The experiments on UniVTAC and Libero-Long show strong performance gains, and ablations support that latent and semantic supervision complement each other for structured reasoning.

Critical weaknesses
  • The ablation studies support the usefulness of joint supervision but do not isolate the overall benefit of the intermediate reasoning stage. A matched policy generating actions without intermediate word or latent reasoning tokens would show the contribution of action primitives beyond the training data and architecture.
Questions for authors

Please refer to the Weaknesses section above.

6 comments
AuthorsOpenReview ↗

Responses to Weaknesses 1.

Dear Reviewer obh3

We sincerely thank you for recognizing the novelty of combining semantic and latent supervision, the structured reasoning context provided by Physical Words, and our strong benchmark performance. We respond to your valuable comments point by point below. We have updated the manuscript, with the corresponding revisions marked in blue.

[W1] A Matched Policy without Intermediate Reasoning

Following your suggestion, we provide more ablation study in both simulation and real-world long-horizon experiments.

1. Simulation experiments: extending Figure 3.

Figure 3 (left) already includes word-only (W), visual-latent-only (O), and tactile-latent-only (T) variants. On Put Bottle and Lift Can, these variants achieve 77%/67%, 77%/83%, and 73%/65%, respectively, compared with 93%/95% for full PhysWord, corresponding to decreases of 12–30 percentage points. These results support the benefit of combining action semantics with visual and tactile predictions within Physical Word reasoning. Following your suggestion, we further remove the intermediate reasoning stage entirely to evaluate its overall contribution under matched conditions.

New experimental setting. We evaluate w/o intermediate reasoning on Put Bottle and Lift Can, the two UniVTAC tasks used in Figure 3. This variant removes all intermediate word and latent reasoning tokens and their corresponding CE and latent-alignment losses during training and inference. All other settings remain identical to those used in the ablation study. The policy therefore generates actions directly from the perceptual context and instruction.

MethodPut BottleLift Can
w/o intermediate reasoning69%63%
PhysWord93%95%
Improvement24 pp32 pp

PhysWord improves success by 24 and 32 percentage points, respectively. These gains support the contribution of intermediate Physical Word reasoning. We incorporate this ablation into Figure 3 (left) and describe its setting and results in Section 4.2.1.

2. Real-world long-horizon experiments.

In Table 15, we already evaluate a latent-only reasoning variant that removes atomic-word CE supervision while retaining visual/tactile latent reasoning and the Cosmos branch, feeding predicted latents back instead of word embeddings. This variant achieves 36.7% and 50.0% success on extended Pick A4 Paper and Express Packing, respectively, compared with 70.0% and 66.7% for full PhysWord. Following your suggestion, we further evaluate w/o intermediate reasoning on these two extended tasks, removing all intermediate word/latent reasoning tokens and their associated losses. Both comparisons use the same original task demonstrations and matched training and evaluation protocols.

MethodExtended Pick A4 PaperExtended Express Packing
w/o intermediate reasoning33.3%36.7%
PhysWord70.0%66.7%
Improvement36.7 pp30.0 pp

The improvements of 36.7 and 30 percentage points further support the benefit of intermediate reasoning for continuous execution across repeated and revisited stages. We report these additional results in Table 17 and describe the detailed control settings in Appendix C.3.3. Together, these additional experiments directly compare reasoning-before-acting with direct action generation under matched conditions, complementing our existing ablations on the supervision used within the reasoning stage.

AuthorsOpenReview ↗

Responses to Reviewer obh3 (summary of the changes).

Dear Reviewer obh3

Thank you again for your time and constructive feedback. As the discussion period draws to a close, we would be grateful to hear whether the additional experiments resolve your concern or whether further clarification would be helpful. We would be happy to respond promptly. If you find that the new evidence satisfactorily addresses your concern and strengthens the paper, we would greatly appreciate your consideration of an updated rating. Your recognition of these improvements would mean a great deal to us.

Following your suggestion, we have added matched direct-action baselines in both simulation and real-world long-horizon experiments. The revised manuscript includes these results, with changes marked in blue. For your convenience, we summarize the responses below:

[W1] A Matched Policy without Intermediate Reasoning

  • Simulation experiments. We add w/o intermediate reasoning on Put Bottle and Lift Can, removing all intermediate word/latent reasoning tokens and their associated losses while retaining the remaining ablation settings. This baseline achieves 69%/63%, compared with 93%/95% for full PhysWord, corresponding to gains of 24/32 percentage points. The comparison complements the existing word-only and latent-only variants by evaluating the overall contribution of intermediate reasoning and its supervision (Figure 3, left, Section 4.2.1).

  • Real-world long-horizon experiments. We add the same direct-action control on extended Pick A4 Paper and Express Packing, using the original task demonstrations and matched evaluation protocols. Full PhysWord achieves 70.0%/66.7%, compared with 33.3%/36.7% without intermediate reasoning. These gains of 36.7/30.0 percentage points support its contribution to execution across repeated and revisited stages. Detailed settings and results are provided in Appendix C.3.3, Table 17.

Thank you again for suggesting this important control and for your continued consideration.

The Authors

AuthorsOpenReview ↗

Discussion Closing Soon, We Would Appreciate Your Feedback

Dear Reviewer obh3,

Thank you again for your constructive feedback. As the discussion period draws to a close, we would greatly appreciate hearing your thoughts on our responses and revisions. We hope the matched direct-action baselines added in simulation and real-world experiments have addressed your concern. A summary of the changes is provided in our previous comment, with detailed explanations and results in our earlier rebuttal response.

If any questions remain or new concerns arise, please let us know. we would be happy to respond promptly. If your concern has been satisfactorily resolved, we would be grateful if you could consider revising your rating. Thank you for your time and continued consideration.

The Authors

Reviewer_rm31OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PhysWord, which uses atomic action words to structure predictions of future visual and tactile information for robot manipulation. The model combines semantic supervision with future sensory representations to guide action generation. Experiments on UniVTAC, LIBERO-Long, and real-world manipulation tasks show strong performance compared with existing baselines. The authors also evaluate generalization to unseen settings and longer task sequences.

Critical strengths
  1. The idea of using action words to structure visual and tactile predictions is interesting. The ablations show that semantic supervision and future sensory prediction can complement each other.

  2. The experimental results are strong. The paper includes simulation benchmarks, real-world experiments on different robot platforms, and evaluations under unseen configurations.

Critical weaknesses
  1. The evidence for reusable action primitives is still limited. The long-horizon experiments mainly repeat previously learned stages, rather than testing new compositions of skills.

  2. The predicted tactile feedback results are interesting, but it is unclear how much the predicted tactile information actually contributes. A comparison with simpler alternatives, such as fixed tactile features, would strengthen the claim.

Questions for authors
  1. Can the authors further isolate the contribution of semantic action words from other architectural choices, particularly in the long-horizon experiments?

  2. How does predicted tactile feedback compare with simpler alternatives, such as constant or mean tactile representations?

5 comments
AuthorsOpenReview ↗

Responses to Weakness 1, Weakness 2, and Question 2.

Dear Reviewer rm31

We sincerely thank you for finding the idea of using action words to structure visual and tactile predictions interesting, recognizing that semantic supervision and future sensory prediction can complement each other, and acknowledging our strong experimental results. We respond to your valuable comments point by point below. We have updated the manuscript, with the corresponding revisions marked in blue.

[W1] Reusability of Learned Action Primitives

Thank you for this constructive comment. We clarify the scope of reusability and provide additional evaluations below.

1. Scope and existing evidence.

By “reusable reasoning primitives,” we refer to reusing learned action semantics across scenes, objects, and execution stages, including repeated execution and stage revisiting. Our existing long-horizon experiments evaluate this scope, they do not establish generalization to novel compositions of learned skills. In Appendix C.3.1 Table 15, PhysWord achieves 70.0% and 66.7% success on extended Pick A4 Paper and Express Packing, respectively, compared with 36.7% and 50.0% for latent-only reasoning. These results support the contribution of semantic anchoring to reuse learned behaviors at stage transitions beyond the training trajectories.

2. Reusability and Compositions of Skills.

Following your suggestion, we evaluate compositional generalization of learned skills through a three-stage manipulation sequence. Specifically, within a single episode, the robot sequentially picks and places an A4 sheet, Logitech Zone Vibe headphones, and a tennis ball, where the headphones and tennis ball are unseen during training. Although all three stages share the same Pick/Place semantics, these objects differ substantially in geometry and deformability, requiring distinct dexterous hand grasping configurations and contact interactions. This setting tests new compositions of learned skills, requiring the model to sequentially reuse and adapt shared atomic skills to unseen objects with different physical interaction requirements, rather than merely reproducing previously observed skill sequences. We compare PhysWord with π0.5 under identical evaluation conditions, without additional SFT for either method. The final column reports the cumulative success rate of completing the entire three-stage sequence.

MethodStage 1: A4 paperStage 2: HeadphonesStage 3: Tennis ball
π0.5\pi_{0.5}53.3%20.0%0%
PhysWord (Ours)93.3%76.7%43.3%

3. Clarification in the manuscript.

We add the sequential manipulation evaluation to Appendix C.3.2 and summarize its findings in Section 4.3. We also qualify “reusable reasoning primitives” in the main text to specify reuse across scenes, objects, and task stages. Visualizations of the experimental setups are provided in Appendix Figure 6.

[W2 & Q2] Contribution of Predicted Tactile Feedback

Thank you for this valuable suggestion. We conduct two additional tactile ablation studies on Put Bottle and Lift Can, the two UniVTAC tasks used in our ablation experiments:

  • Fixed Tactile Features (F1): We randomly select a contact frame from a sampled training episode and extract its tactile representation using T3 and our original feature aggregation procedure. This fixed representation is then used across all frames and episodes.
  • Task-Mean Tactile Features (F2): For each task, we randomly sample three training trajectories and construct a mean tactile image by averaging the values at the same pixel coordinates across all frames. We then extract features from this mean image using T3 and the original feature aggregation procedure. The resulting task-specific representation is used across all frames and episodes.

For each ablation, we evaluate two settings: (1) inference-only replacement, where the model is trained with measured tactile inputs but uses F1 (Row 2) or F2 (Row 4) during inference; and (2) training-and-inference replacement, where the measured tactile inputs are replaced with F1 (Row 3) or F2 (Row 5) during both training and inference. We compare all four variants against our proposed approach (Row 1), which uses measured tactile inputs during training and predicted tactile latent during inference

RowTrainingtactileinputEvaluationtactileinputPutBottleLiftCan
1MeasuredPredicted(PhysWord-feedback)92%94%
2MeasuredFixed82%87%
3FixedFixed85%88%
4MeasuredTaskmean82%88%
5TaskmeanTaskmean87%90%

These results demonstrate that our dynamically predicted tactile feedback improves manipulation performance over fixed and task-mean tactile representations, highlighting the value of context-dependent tactile information for action generation. We add these controlled comparisons to Appendix C.4.2 and summarize their findings in the main Experiments Section 4.1.

AuthorsOpenReview ↗

Responses to Question 1.

[Q1] Isolating the Contribution of Semantic Action Words

We clarify the existing control and provide additional ablations to distinguish the contributions of semantic action words, temporal perception, and intermediate reasoning.

1. Existing long-horizon comparison with latent-only reasoning.

Appendix C.3.1, Table 15 compares PhysWord with a variant that retains visual/tactile latent reasoning and the Cosmos branch, but removes atomic-word CE supervision and feeds predicted latents back instead of word embeddings. PhysWord achieves 70.0% and 66.7% success on extended Pick A4 Paper and Express Packing, respectively, compared with 36.7% and 50.0% for latent-only reasoning. These improvements of 33.3 and 16.7 percentage points support the benefit of the semantic-word formulation beyond future-latent supervision alone.

2. Additional ablations on real-world long-horizon tasks.

Following your suggestion, we further evaluate two variants:

  • w/o Cosmos: Removes the temporal perception branch while retaining complete Physical Words. Comparing to Row 4 (w/o intermediate reasoning), this drops a relative small gap, showing the importance of semantic action words
  • w/o intermediate reasoning: Building upon the ablation in Table 15 that removes only the atomic-word CE supervision, we further remove all intermediate word and latent reasoning tokens while retaining the original perceptual inputs and action generation process. Success rates on extended Pick A4 Paper and Express Packing decrease further to 33.3% and 36.7%, respectively. Together with Table 15, these results support the effectiveness of the complete Physical Word reasoning framework for long-horizon manipulation.
RowVariantExtended Pick A4 PaperExtended Express Packing
1Full PhysWord70.0%66.7%
2Latent-only reasoning (Table 15)36.7%50.0%
3w/o Cosmos53.3%50.0%
4w/o intermediate reasoning33.3%36.7%

3. Additional ablations on simulation tasks.

We additionally evaluate w/o intermediate reasoning on Put Bottle and Lift Can, the two UniVTAC tasks used in ablation study (Figure 3). This variant removes all intermediate word/latent reasoning tokens and their associated losses, directly generating actions from the perceptual context and instruction. All other settings remain identical to those used in the ablation study. Full PhysWord improves success by 24 and 32 percentage points, respectively, supporting the contribution of intermediate Physical Word reasoning. We incorporate the simulation control into Figure 3 (left) and the additional long-horizon results into Appendix C.3.3

MethodPut BottleLift Can
w/o intermediate reasoning69%63%
PhysWord93%95%
Improvement24 pp32 pp
AuthorsOpenReview ↗

Responses to Reviewer rm31 (summary of the changes).

Dear Reviewer rm31

Thank you again for your time and constructive feedback. As the discussion period draws to a close, we would be grateful to hear whether our responses and revisions have addressed your concerns or whether any questions remain. We would be happy to provide further clarification promptly. If your concerns have been satisfactorily addressed, we would greatly appreciate your consideration of an updated assessment and rating. Your recognition of these improvements would mean a great deal to us.

We have responded to each weakness and question and revised the manuscript accordingly, with changes marked in blue. For your convenience, we summarize the main response below:

[W1] Reusability of learned action primitives. We qualify reusability as applying learned action semantics across scenes, objects, and execution stages, without claiming that the original experiments establish generalization to arbitrary new skill compositions. We add a continuous paper–headphones–tennis-ball sequence involving unseen objects and different manipulation configurations, without additional SFT. PhysWord completes the sequence at 43.3%, outperforming π0.5. These results and setup visualizations are presented in Appendix C.3.2 and summarized in Section 4.3.

[W2 & Q2] Contribution of predicted tactile feedback. Following your meaningful suggestion, we add fixed and task-mean tactile controls under both inference-only replacement and replacement during training and inference. Predicted feedback achieves 92%/94% on Put Bottle/Lift Can, outperforming all static alternatives, including those trained with consistent static inputs. These comparisons support the value of context-dependent tactile predictions beyond a fixed contact representation or task-level average (Appendix C.4.2, Section 4.1).

[Q1] Isolating the Contribution of Semantic Action Words.

  • Simulation experiments. We add the w/o intermediate reasoning control on Put Bottle and Lift Can (ablation study tasks). Removing all intermediate word/latent reasoning tokens and their associated losses reduces success by 24 and 32 percentage points, respectively, supporting the contribution of entire Physical Word reasoning and its supervision under matched conditions (Figure 3, left).
  • Real-world experiments. We complement the existing latent-only comparison with w/o Cosmos and w/o intermediate reasoning controls on extended Pick A4 Paper and Express Packing. Removing intermediate reasoning reduces success by 36.7 and 30.0 percentage points, respectively, compared with 16.7 points on each task when removing Cosmos. Together with the latent-only comparison, these results support the contribution of Physical Word reasoning beyond the temporal perception branch (Appendix C.3.3).

Thank you again for helping us strengthen the paper and for your continued consideration.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.