acceptodds
Desk rejected at ICLR 2027

Interaction Latents as Actions for Contact-Rich Manipulation

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Contact-rich manipulation requires robots to coordinate not only where to move, but also how contact and force should evolve throughout an interaction. Yet most robot policies represent actions primarily as motion trajectories, using tactile signals as inputs or auxiliary prediction targets without explicitly encoding contact dynamics in the action representation. We propose ILA (Interaction Latents as Actions), which uses decodable interaction latents that jointly capture the evolution of motion, contact, and force application as policy action units, providing a shared representation for action generation, evaluation, and offline improvement. We learn these latents through visually conditioned joint action-tactile reconstruction with a time-conditioned implicit decoder. Using them as flow-matching generation targets supports post-training of pretrained vision-language-action models (VLAs) and world action models (WAMs), as well as training flow-matching policies from scratch. A shared decoder maps generated latents to executable actions. For offline improvement, we learn a tactile-conditioned expert reference model from interaction latents encoded from expert demonstrations. The resulting expert-consistency rewards guide weighted learning from successful rollouts, while observation-compatible expert references guide correction of low-reward predictions from failed rollouts. Across three policy families, ILA improves success rates averaged over four real-world tasks by 21.0–36.5 percentage points relative to tactile-conditioned direct-action baselines. Ablations highlight the benefit of encoding contact evolution in action targets beyond tactile conditioning, while the same latent space supports expert-guided offline improvement.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Visuo-tactile Policy, Contact-Rich Manipulation, Embodied AI

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wdfqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In the paper, the authors propose Interaction Latents as Actions for combining motion, contact, and forces and using those latents as generation targets for different robot policy types. They evalute the proposed method on real-world contact rich task and show that their method is superior to the proposed baselines.

Critical strengths

Relevant tasks are addressed using a proposed, intuitive approach that results in its good perfomance.

Critical weaknesses

It is is not clear how much novel is the proposed method, as the new algorithmic additions are common in field as the authors state in the introduction.

Evaluation using methods that consider the same training data is lacking. Comparison to baselines that are outside the related families should be added for complete picture of the performance of the method.

Questions for authors
  1. Could you explain in more detail how does the proposed work differ from the existing works? Can you clearly state what is novel in this work?
  2. Have you considered other methods that learn from the the multiple data that does not fall into the 3 family groups? For example, KineDex (Zhang et al., 2025) and ForceMimic (Liu et al., 2025) are mentioned in Section 4.1.
5 comments
AuthorsOpenReview ↗

Response to wdfq: Methodological Novelty and Baseline Comparisons (1/2)

W1 / Q1: Novelty and distinctions from prior work

Weakness W1: It is not clear how much novel is the proposed method, as the new algorithmic additions are common in field as the authors state in the introduction.

Question Q1: Could you explain in more detail how does the proposed work differ from the existing works? Can you clearly state what is novel in this work?

Response: Thank you for encouraging us to clarify the contribution. Our central idea is to represent the joint evolution of motion and contact as a decodable policy action, connecting action generation, expert-consistency evaluation, and offline policy improvement within the same interaction space. Sections 2–3 (pp. 3–6) clarify this formulation, and additional experiments examine its contribution.

(1) Incorporating contact evolution into the policy's action representation. Prior methods have explored tactile feedback, force–motion learning, and joint prediction of actions and tactile signals [1–5]. ILA jointly encodes temporally aligned action and tactile sequences under visual context. The resulting interaction latent becomes the generation target of the policy's action branch, and the same latent is decoded into actions, tactile deformation, and resultant-force proxies. Contact thus directly shapes the action space that the policy learns to generate. Under matched backbones, inputs, capacity, and training budgets, ILA outperforms action-only latents, separately learned action/tactile latents, and action latents with auxiliary tactile prediction. Relative to auxiliary tactile prediction, ILA improves SR by 15 and 17 percentage points on peeling and celery cutting, respectively, while reducing execution force-proxy error (main-text Table 4, p. 10). These controlled comparisons directly support the benefit of joint encoding.

(2) Using the same interaction action for evaluation and offline correction. A tactile-conditioned expert reference model evaluates the consistency of predicted interactions with the expert distribution in this generation space, providing chunk-level rewards for weighted learning from successful rollouts. For low-quality interactions in failed rollouts, observation-compatible and directionally consistent expert references provide correction targets under the current observation. Action generation, evaluation, and correction therefore operate on a common motion–contact representation. Using the same rollout corpus and a fixed latent space, our update achieves higher SR and lower execution force-proxy error than IQL [6] and RECAP [7] on all four tasks (main-text Table 1, p. 7). In the peeling ablation, adding failure correction increases SR from 80% to 85% and reduces the error from 2.89 to 0.51, supporting the contribution of reference-guided correction (main-text Table 3, p. 9).

(3) Applicability across policy architectures and transfer to new tasks. The interaction representation serves as a generation target for FM, VLA, and WAM policies. In WAM, we retain the original video branch and its generation target while replacing the action branch's target with interaction latents. Shared and task-specific representations yield comparable downstream policy performance (Appendix Table 12, p. 34). For vase wiping and carrot peeling, excluded from representation pretraining, we train new FM-ILA policies using 20 demonstrations and 16,000 steps per task, keeping the shared encoder frozen. With each policy checkpoint fixed and no additional force feedback, adapting only the decoder using ten demonstrations per task increases SR from 80%/40% to 100%/65%, respectively, and reduces execution force-proxy error (main-text Table 2, p. 9). Together, these results support the applicability of interaction representations across policy architectures and their transfer to unseen tasks with a frozen encoder.

AuthorsOpenReview ↗

Response to wdfq: Methodological Novelty and Baseline Comparisons (2/2)

W2 / Q2: Direct comparisons with existing methods

Weakness W2: Evaluation using methods that consider the same training data is lacking. Comparison to baselines that are outside the related families should be added for complete picture of the performance of the method.

Question Q2: Have you considered other methods that learn from the multiple data that does not fall into the 3 family groups? For example, KineDex (Zhang et al., 2025) and ForceMimic (Liu et al., 2025) are mentioned in Section 4.1.

Response: We have added RDP [3], ForceMimic [2], KineDex [1], and Tactile-WAM [5], providing five existing-method comparisons alongside DreamTac [4]. We also add WAM-tac*, which jointly generates actions and tactile sequences within FastWAM, with DreamTac reported separately.

On the original four tasks, all methods use the same 150 expert demonstrations per task and evaluation protocol, with 20 real-world trials per method per task. FM, KineDex, ForceMimic, and RDP train for 50,000 steps; VLA is post-trained for 50,000 steps; WAM, DreamTac, and Tactile-WAM are post-trained for 40,000 steps. ILA's encoder/decoder is jointly pretrained on the four-task dataset for 240,000 steps and frozen during downstream training (Appendix A.1, pp. 17–18).

Among the five existing methods in Table 1, RDP has the highest average SR (71.25%). FM-ILA exceeds it on all four tasks, averaging 80.0% (+8.75 percentage points). WAM-ILA matches or exceeds the best SR of these methods on every task, averaging 73.75%. FM-ILA, VLA-ILA, and WAM-ILA outperform their direct joint action–tactile prediction variants in SR on every task. Against WAM-tac*, WAM-ILA gains 25/60/17/40 percentage points on peeling/flipping/cutting/wiping. These results support jointly encoding motion and contact as decodable policy action targets.

Table 1. Existing-method comparisons. Each cell reports SR (%) / FextF_{\mathrm{ext}} L1 (mean ± standard deviation). SR is scored per cut for cutting and per trial otherwise. From manuscript Table 1 (p. 7).

MethodCucumber PeelingEgg FlippingCelery CuttingPlate Wiping
RDP65% / 2.21 ± 0.4080% / 1.53 ± 0.7245% / 6.80 ± 2.4495% / 3.65 ± 0.58
ForceMimic30% / 2.43 ± 0.6745% / 1.89 ± 0.8020% / 7.76 ± 1.0580% / 2.80 ± 0.70
KineDex20% / 2.33 ± 0.6270% / 2.05 ± 0.7415% / 8.15 ± 1.7075% / 3.08 ± 0.60
DreamTac20% / 2.63 ± 0.6825% / 2.13 ± 0.8318% / 8.54 ± 2.2370% / 2.92 ± 0.65
Tactile-WAM30% / 2.29 ± 0.6625% / 1.78 ± 0.8825% / 8.03 ± 1.6885% / 1.41 ± 0.61

Simulation comparisons. We also compare FastWAM-ILA, FastWAM-tac, DreamTac, Tactile-WAM, and RDP on six UniVTAC tasks [8]. All methods use the same settings and 50 demonstrations per task, with 100 evaluation trials per method per task.

Table 2. UniVTAC success rates (%, ↑). Average is the unweighted six-task mean; column maxima are bolded. From Appendix Table 9 (p. 30), where ILA denotes FastWAM-ILA.

MethodInsert HoleInsert TubeLift CanPull-out KeyPut BottleInsert HDMIAverage
RDP26711822462033.83
DreamTac3660235526834.67
Tactile-WAM33821328412537.00
FastWAM-tac60991676311749.83
FastWAM-ILA72943486401657.00

Table 2 shows that FastWAM-ILA achieves 57.00% average SR, exceeding FastWAM-tac by 7.17 percentage points and outperforming the other three methods. Together, these comparisons support interaction latents across diverse contact operations (Appendix B.8, p. 30).

References

[1] Di Zhang et al. KineDex: Learning Tactile-Informed Visuomotor Policies via Kinesthetic Teaching for Dexterous Manipulation. arXiv:2505.01974, 2025.

[2] Wenhai Liu et al. ForceMimic: Force-Centric Imitation Learning with Force-Motion Capture System for Contact-Rich Manipulation. ICRA, 2025.

[3] Han Xue et al. Reactive Diffusion Policy: Slow-Fast Visual-Tactile Policy Learning for Contact-Rich Manipulation. RSS, 2025.

[4] Yunfan Lou et al. Dream-Tac: A Unified Tactile World Action Model for Contact-Rich Robot Manipulation. arXiv:2606.08737, 2026.

[5] Siyu Wu et al. Tactile-WAM: Touch-Aware World Action Model with Tactile Asymmetric Attention. arXiv:2606.26663, 2026.

[6] Ilya Kostrikov et al. Offline Reinforcement Learning with Implicit Q-Learning. arXiv:2110.06169, 2021.

[7] Physical Intelligence et al. π0.6∗\pi^{*}_{0.6}: a VLA That Learns From Experience. arXiv:2511.14759, 2025.

[8] Baijun Chen et al. UniVTAC: A Unified Simulation Platform for Visuo-Tactile Manipulation Data Generation, Learning, and Benchmarking. arXiv:2602.10093, 2026.

AuthorsOpenReview ↗

Follow-up on Revisions and Additional Experiments

Dear Reviewer wdfq,

Thank you again for your thoughtful review. In response to your concerns, we have clarified ILA’s contribution and its distinction from existing methods, added controlled ablations, and expanded comparisons with prior approaches, including KineDex and ForceMimic, on real-robot and simulation tasks (Tables 1 and 4, and Appendix B.8).

We have submitted the updated manuscript and revised supplementary material, incorporating the corresponding clarifications and additional experimental results. Manuscript revisions are highlighted in blue.

We would be grateful if you could review our updated responses and manuscript and let us know whether these additions address your concerns. We would be happy to provide any further clarification.

We sincerely appreciate your time and consideration.

Best regards,

The authors

Reviewer_fJWVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ILA, a latent action space learned through joint reconstruction of motion and tactile deformation. Policies predict interaction latents, and a frozen time-conditioned decoder converts them into executable actions. The same space supports expert-consistency scoring and offline improvement through successful-rollout reweighting and reference-guided failure correction. Experiments cover four real-robot tasks and three policy families: flow matching, π0.5, and FastWAM. The central contribution is to make contact evolution part of the learned action representation, with evidence that this helps beyond providing tactile observations alone.

Critical strengths
  1. Joint action–tactile reconstruction is a well-motivated way to incorporate contact evolution into the action space.
  2. Gains across three policy families and four real-robot tasks are supported by ablations that retain tactile policy inputs while removing tactile representation learning.
  3. The same latent space enables offline improvement in task success and agreement with expert deformation profiles.
Critical weaknesses
  1. Limited responsiveness to new tactile feedback. Appendix A.1 states that all 128 predicted actions are executed at 30 Hz before synchronous replanning, giving approximately 4.27 seconds of execution between policy updates, before accounting for inference latency. The time-conditioned decoder produces commands from the existing latent; it does not incorporate new tactile measurements during that interval. Many contact-rich tasks require rapid responses to unexpected contact changes or slip, so the reported results do not establish suitability for this important regime. The baselines share the same horizon, but the comparison remains confined to this relatively slow feedback schedule. End-to-end inference latency is also not reported.
  2. The dependence on tool grasp is not resolved. In peeling, cutting, and flipping, the sensors on the tool-holding gripper measure contact between the fingers and the tool, while the task-critical interaction occurs between the tool and the object. Fingertip deformation may therefore change with grasp position or clamping strength even when the task interaction remains effective. It is unclear whether the learned representation and tactile-conditioned expert-consistency score distinguish changes in interaction quality from changes in how the tool is held. This matters because the score reweights successful rollouts and selects references for failure correction: an effective interaction with a different grasp might receive a lower score. A discussion of this dependence would help clarify the scope of the representation.
Questions for authors
  1. The paper includes compatibility checks and retrieval examples. How do the authors assess whether a retrieved expert continuation is appropriate for correcting the robot's current failure? For example, a failed peeling attempt may require re-establishing contact before continuing the expert motion.
  2. Appendix C shows that one encoder and decoder can reconstruct all four tasks. Is there evidence that this shared representation also preserves the downstream policy gains obtained with task-specific representations?
5 comments
AuthorsOpenReview ↗

Response to fJWV: Feedback, Interaction Evaluation, and Shared Representations (1/3)

W1: Tactile feedback and end-to-end latency

Weakness W1: Limited responsiveness to new tactile feedback. Appendix A.1 states that all 128 predicted actions are executed at 30 Hz before synchronous replanning, giving approximately 4.27 seconds of execution between policy updates, before accounting for inference latency. The time-conditioned decoder produces commands from the existing latent; it does not incorporate new tactile measurements during that interval. Many contact-rich tasks require rapid responses to unexpected contact changes or slip, so the reported results do not establish suitability for this important regime. The baselines share the same horizon, but the comparison remains confined to this relatively slow feedback schedule. End-to-end inference latency is also not reported.

Response: Thank you. We measured latency and compared original 128-step execution, 32-step synchronous replanning, and 128-step execution with 30 Hz admittance feedback for FM-ILA and VLA-ILA.

Inference latency. FM, VLA, and WAM use flow matching, π0.5\pi_{0.5}, and FastWAM, respectively. Table 1 reports latency on one RTX 4090D (batch size 1, BF16, 4 denoising steps). Model time spans policy input to decoded actions; end-to-end time spans observation acquisition to robot receipt of commands.

Table 1. Inference latency (Appendix Table 6, p. 26).

PolicyModel prediction time (ms)End-to-end latency (ms)
FM-ILA87.8101.5
VLA-ILA240.6254.2
WAM-ILA642.3654.9

For each policy, settings share the checkpoint, observations, inference configuration, 30 Hz command rate, evaluation protocol, and 128-action prediction. The 32-step setting executes the first 32 actions, pauses, acquires new observations, and replans synchronously. The open-loop motion window falls from 4.27 s to 1.07 s. The complete interval is Δtupdate≈Hexec/30+Lblock\Delta t_{\mathrm{update}}\approx H_{\mathrm{exec}}/30+L_{\mathrm{block}}. From measured latency, update intervals are approximately 1.17 s for FM-ILA and 1.32 s for VLA-ILA; fresh observations therefore affect actions more frequently.

Shorter execution windows. Table 2 shows lower force-proxy error in 6/8 comparisons and average SR increases from 80.0% to 80.5% for FM-ILA and 66.75% to 68.75% for VLA-ILA. Error standard deviations increase in 6/8 comparisons, consistent with observed startup transients and motion discontinuities from repeated stops. Egg-flipping error increases for both policies, and VLA SR falls from 85% to 80%. Thus, frequent updates improve force-profile matching on most tasks, while interruptions affect task performance.

30 Hz tactile feedback. Decoded 3D force proxies provide a time-varying reference. Reference–observation error passes through an admittance controller [1] to an end-effector increment, then through the Jacobian to joint corrections:

\Delta\mathbf x(t)=\mathcal A(\mathbf e_{0:t}),\quad J(\mathbf q_{\mathrm{obs}}(t))\Delta\mathbf q(t)\approx\Delta\mathbf x(t).$$ The final command is $\mathbf a_{\mathrm{cmd}}(t)=\mathbf a_{\mathrm{pred}}(t)+0.75\,\Delta\mathbf a(t)$. Arm and gripper corrections update at 30 Hz without rerunning the policy, retaining 128-step prediction/execution and synchronous policy scheduling. Compared with 32-step execution, 30 Hz feedback lowers all eight error means and standard deviations while maintaining or improving SR. Against original execution, average SR reaches 85.25%/73.75% for FM/VLA, with lower error in 7/8 comparisons. Thus, interaction latents support continuous execution with actions and aligned contact references for local feedback (Appendix B.6, pp. 25–28). **Table 2. Execution length and tactile feedback** (Appendix Table 7, p. 28). Cells show **SR (%) ↑ / $F_{\mathrm{ext}}$ L1 (mean ± standard deviation) ↓**. | Policy | Execution setting | Cucumber Peeling | Egg Flipping | Celery Cutting | Plate Wiping | | :---: | :---: | :---: | :---: | :---: | :---: | | FM-ILA | 128 (original) | 75 / 2.04 ± 0.37 | 95 / 0.56 ± 0.64 | 50 / 7.52 ± 0.78 | 100 / 3.44 ± 0.35 | | FM-ILA | 32 (sync.) | 75 / 1.88 ± 0.36 | 95 / 0.62 ± 0.71 | 52 / 7.24 ± 0.91 | 100 / 2.53 ± 0.52 | | FM-ILA | 128 + 30 Hz feedback | 90 / 1.31 ± 0.32 | 95 / 0.52 ± 0.51 | 56 / 6.46 ± 0.70 | 100 / 1.22 ± 0.47 | | VLA-ILA | 128 (original) | 50 / 2.28 ± 0.41 | 85 / 0.38 ± 0.54 | 37 / 7.22 ± 1.17 | 95 / 2.51 ± 0.38 | | VLA-ILA | 32 (sync.) | 55 / 1.94 ± 0.61 | 80 / 0.46 ± 0.67 | 40 / 7.04 ± 1.08 | 100 / 2.26 ± 0.50 | | VLA-ILA | 128 + 30 Hz feedback | 60 / 1.49 ± 0.48 | 90 / 0.43 ± 0.55 | 45 / 6.67 ± 0.82 | 100 / 1.10 ± 0.32 | [1] Arvid Q. L. Keemink et al. *Admittance control for physical human–robot interaction*. IJRR, 2018.
AuthorsOpenReview ↗

Response to fJWV: Feedback, Interaction Evaluation, and Shared Representations (2/3)

W2: Dependence on tool grasp position and grip strength

Weakness W2: The dependence on tool grasp is not resolved. In peeling, cutting, and flipping, the sensors on the tool-holding gripper measure contact between the fingers and the tool, while the task-critical interaction occurs between the tool and the object. Fingertip deformation may therefore change with grasp position or clamping strength even when the task interaction remains effective. It is unclear whether the learned representation and tactile-conditioned expert-consistency score distinguish changes in interaction quality from changes in how the tool is held. This matters because the score reweights successful rollouts and selects references for failure correction: an effective interaction with a different grasp might receive a lower score. A discussion of this dependence would help clarify the scope of the representation.

Response: We have added peeling and wiping tests under varied grasping and tool conditions. Figure 13 in the manuscript (p. 35) presents successful trajectories under the following conditions:

  • Peeling: the original peeler grasped at its middle with normal grip strength (P1); an upper grasp with normal strength (P2); a middle grasp with stronger (P4) or lighter (P5) grip strength; and a new peeler grasped at its middle (P6).
  • Wiping: a regular sponge held horizontally with normal grip strength (W1), stronger grip strength (W2), or lighter grip strength (W3); the sponge held vertically with normal strength (W4); and a harder foam material held horizontally (W5).

The encoder shared across the four tasks remains frozen and is not fine-tuned on these new data. We adapt only a lightweight task-specific scorer and evaluate it on test trajectories. Scores are mapped to [0, 1] through a sigmoid, and all displayed trajectories are excluded from the corresponding scorer's training and validation sets.

The test balanced accuracies are 97.2% for peeling and 98.8% for wiping.

Figure 13 (p. 35) aligns top-view and right-wrist observations with score curves, starting from the tool-grasping stage; red dashed lines mark the reference contact onset. Scores and their rise times vary during grasping and contact establishment, but all successful conditions reach high scores during sustained peeling or wiping. This pattern persists when changing the peeler or its grasp position, as well as the sponge orientation or material. The light-grip peeling case reaches a high score after a later rise, supporting consistent evaluation of different effective interaction patterns.

Figure 14 (p. 35) compares success and failure under four matched conditions: a middle grasp on the original peeler with normal, stronger, or lighter grip strength, and a horizontal grasp on the regular sponge with normal grip strength. Tool, grasp position, and grip strength are matched within each pair. Successful trajectories receive high scores during manipulation, while the corresponding failures remain at low scores. In particular, the stronger-grip failure still receives a low score, whereas the lighter-grip success rises to a high score as effective manipulation develops. These pairs show that the scores track interaction quality across different grip strengths. Together, the two figures demonstrate consistent scoring of successful interactions across conditions and separation of success from failure within the same condition.

These results show that the frozen interaction representation retains task-outcome information that a lightweight scorer can read out even when contact conditions change (Appendix C.5, pp. 34–36).

AuthorsOpenReview ↗

Response to fJWV: Feedback, Interaction Evaluation, and Shared Representations (3/3)

Q1: Suitability of retrieved references for correcting the current failure

Question Q1: The paper includes compatibility checks and retrieval examples. How do the authors assess whether a retrieved expert continuation is appropriate for correcting the robot's current failure? For example, a failed peeling attempt may require re-establishing contact before continuing the expert motion.

Response: We assess reference suitability at two levels: compatibility with the current interaction state and whether the expert continuation contains the contact process missing from the failure. During retrieval, observation similarity selects expert windows compatible with the current state, while directional consistency is checked across candidate continuations. A closer observation match places the offline correction target closer to the selected expert latent (Section 3.3, pp. 5–6). Figure 7 in the manuscript (p. 25) illustrates observation-compatible pairs across all four tasks.

For insufficient-contact failures, we further examine tool center point (TCP) motion, tactile deformation, and contact-load evolution together in an offline paired analysis. We assess whether the expert continuation contains motion accompanied by contact establishment or sustained loading while the corresponding failure remains at low load. We analyzed 50 failed peeling trajectories and selected low-load intervals with sufficient prediction-window coverage, yielding 20 comparable intervals from 12 trajectories. In all 20 intervals, the selected expert references exhibit sustained contact-load proxies above those of the recorded failures, providing the loading targets missing from the failed interactions.

Figure 8 (p. 26) illustrates two such pairs. In the upper example, the reference exhibits more sustained loading and a different vertical motion profile. In the lower example, the reference transitions from low load to sustained loading, accompanied by changes in TCP height and increased tactile deformation, while the failure remains at low load. Camera observations show the failed trajectory; the curves and tactile maps compare the evolution of separate failure and selected expert windows. These paired observations show that references selected through observation compatibility and directional consistency contain the establishment and maintenance of contact loading, providing local correction targets that couple motion with the contact missing from the failure (Appendix B.4, p. 24).

Q2: Whether the shared representation preserves downstream policy gains

Question Q2: Appendix C shows that one encoder and decoder can reconstruct all four tasks. Is there evidence that this shared representation also preserves the downstream policy gains obtained with task-specific representations?

Response: Main experiments use one shared encoder/decoder jointly pretrained across four tasks for 240,000 steps, then frozen during task-specific policy training. Each task uses 150 demonstrations, with 20 real-world trials per method.

We have added a direct comparison between shared and task-specific representations (Table 3), matching downstream data, policy inputs, latent dimensions, decoder type, and evaluation protocol. Both FM variants are trained from scratch for 50,000 steps, and both VLA variants are post-trained for 50,000 steps (Appendix A.1, pp. 17–18). Average SR across the four tasks is 80.0% versus 79.5% for shared versus task-specific FM-ILA, and 66.75% versus 67.5% for VLA-ILA. Per-task SR differences are no larger than 5 percentage points, and the shared representation yields lower execution force-proxy error in 6/8 comparisons. Thus, a single frozen shared encoder/decoder supports policy learning across all four tasks while preserving downstream performance comparable to task-specific representations.

Table 3. Downstream policy comparison of shared and task-specific representations. These results also appear in Appendix C.4, Table 12 (p. 34). Downstream policies are trained separately for each task, with the encoder and decoder frozen. Each cell reports SR (%) / FextF_{\mathrm{ext}} L1 (mean ± standard deviation). Cutting SR is per cut; other SR is per trial.

PolicyRepresentation trainingCucumber PeelingEgg FlippingCelery CuttingPlate Wiping
FM-ILAShared across four tasks75 / 2.04 ± 0.3795 / 0.56 ± 0.6450 / 7.52 ± 0.78100 / 3.44 ± 0.35
FM-ILATask-specific75 / 2.16 ± 0.42100 / 0.60 ± 0.4548 / 7.96 ± 0.7195 / 3.23 ± 0.57
VLA-ILAShared across four tasks50 / 2.28 ± 0.4185 / 0.38 ± 0.5437 / 7.22 ± 1.1795 / 2.51 ± 0.38
VLA-ILATask-specific45 / 2.46 ± 0.4890 / 0.44 ± 0.4040 / 7.06 ± 0.9795 / 2.70 ± 0.68
Reviewer_cPYqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ILA learns a visually conditioned latent representation that jointly reconstructs robot actions, tactile deformation, and aggregate deformation. Policies generate these latents, which a frozen decoder converts into executable actions. The same representation supports expert-consistency scoring and reference-guided offline policy improvement. Experiments cover four real-world contact-rich tasks across flow-matching, VLA, and world-action-model policies.

Critical strengths
  1. The paper investigates a meaningful question: whether contact information should shape action targets rather than serve only as policy input.

  2. Results show consistent gains across three policy families. Comparisons include both tactile-conditioned action prediction and joint action–tactile prediction.

  3. Removing tactile representation learning while retaining tactile policy inputs substantially reduces performance, providing useful evidence for the proposed supervision.

Critical weaknesses
  1. The source of improvement is unclear. ILA changes the action representation, pretraining procedure, and visual features together. More controlled comparisons are needed to establish the benefit of joint action–tactile encoding.

  2. Retaining both modalities does not establish physical coordination. Reconstruction and probing show that the latent contains action and tactile information, but not that decoded actions reliably produce the predicted contact.

  3. Cross-task reuse is not demonstrated. Main experiments train task-specific encoders and decoders. The shared-model study evaluates reconstruction rather than policy performance or transfer.

  4. Experiments cover only four tasks on one robot platform, with 20 rollouts per method per task. There is no broader simulation benchmark or systematic evaluation across objects and contact conditions, leaving the method’s generality and robustness unclear.

Questions for authors
  1. Under matched inputs, capacity, and training budgets, how does ILA compare with separately learned action and tactile latents, or an action-only latent with auxiliary tactile prediction?

  2. Can a shared encoder and decoder support policy learning across all four tasks? Can they remain frozen when adapting to a new task or contact condition?

  3. How closely do decoded tactile predictions match actual contact during execution, particularly under object-position or material changes? Does the advantage persist with shorter execution chunks?

9 comments
AuthorsOpenReview ↗

Response to cPYq: Joint Encoding, Contact Consistency, and Representation Transfer (1/4)

W1 / Q1: Isolating joint encoding

Weakness W1: The source of improvement is unclear. ILA changes the action representation, pretraining procedure, and visual features together. More controlled comparisons are needed to establish the benefit of joint action–tactile encoding.

Question Q1: Under matched inputs, capacity, and training budgets, how does ILA compare with separately learned action and tactile latents, or an action-only latent with auxiliary tactile prediction?

Response: Thank you for suggesting more controlled comparisons to clarify the contribution of joint action–tactile encoding. We have added comparisons on cucumber peeling and celery cutting using the same VLA (π0.5\pi_{0.5}) backbone as in the original ablations. The existing Action-only latent baseline encodes and reconstructs only actions during representation pretraining, removing tactile inputs and contact-reconstruction supervision while retaining visual context and downstream tactile conditioning. We add two further baselines:

  • Separate action/tactile latents: Action and tactile autoencoders are pretrained independently with visual context from the same sources, without exchanging action and tactile information. The policy jointly generates both latents, which are decoded into actions and tactile fields by their respective decoders.
  • Action latent + auxiliary tactile prediction: After pretraining and freezing an action autoencoder, the policy performs flow-matching denoising only on action latents and predicts future tactile fields through an auxiliary head attached to shared policy features. The tactile loss updates the policy and auxiliary head.

All four variants use the same four-task representation-pretraining data, 150 downstream demonstrations per task, and policy observations, with 20 real-world trials per method and task. They retain the same visual features and S&D tokens, with matched latent sequence lengths and total dimensions, decoder types, model capacity, and training budgets for both stages. Representation training uses 240,000 steps (Appendix A.1, pp. 17–18). During policy training, the autoencoders remain frozen, and all policies are post-trained for 50,000 steps from the same π0.5\pi_{0.5} weights, using the same prediction horizon and execution protocol.

Table 1. Controlled comparisons of joint interaction encoding. VLA (π0.5\pi_{0.5}) results: SR and execution force-proxy error, FextF_{\mathrm{ext}} L1 (mean ± SD). Cutting SR is per cut; other SR is per trial. Corresponds to manuscript Table 4 (p. 10).

SettingPeeling SR ↑Peeling FextF_{\mathrm{ext}} L1 ↓Cutting SR ↑Cutting FextF_{\mathrm{ext}} L1 ↓
Full ILA50%2.28 ± 0.4137%7.22 ± 1.17
Action-only latent20%2.64 ± 0.3810%8.04 ± 1.90
Separate action/tactile latents15%2.78 ± 0.538%9.07 ± 2.35
Action latent + auxiliary tactile prediction35%2.40 ± 0.4620%7.90 ± 2.19

As shown in Table 1, Full ILA achieves the highest SR and lowest execution force-proxy error on both tasks. It improves SR by 35 and 29 percentage points over separate action/tactile latents, and by 15 and 17 percentage points over action latents with auxiliary tactile prediction. Auxiliary tactile prediction improves upon Action-only latent but remains below Full ILA, showing that incorporating touch into the joint action representation provides benefits beyond auxiliary supervision.

The separate-latent baseline achieves 15% and 8% SR, close to VLA-tac*, which directly generates actions and tactile fields jointly (10% and 10%; Table 2), and below Action-only latent. These results support jointly encoding motion and contact during representation learning to provide a coordinated generation target for the policy.

Table 2. Direct action generation and interaction-latent generation. Matched VLA backbone, task data, and post-training budget. VLA-tac predicts actions; VLA-tac* jointly predicts actions and tactile fields. From manuscript Table 1 (p. 7).

MethodPeeling SR ↑Peeling FextF_{\mathrm{ext}} L1 ↓Cutting SR ↑Cutting FextF_{\mathrm{ext}} L1 ↓
VLA-tac15%2.58 ± 0.436%7.82 ± 1.56
VLA-tac*10%2.82 ± 0.4710%9.24 ± 2.47
VLA-ILA50%2.28 ± 0.4137%7.22 ± 1.17
AuthorsOpenReview ↗

Response to cPYq: Joint Encoding, Contact Consistency, and Representation Transfer (2/4)

W2 / Q3: Predicted versus executed contact

Weakness W2: Retaining both modalities does not establish physical coordination. Reconstruction and probing show that the latent contains action and tactile information, but not that decoded actions reliably produce the predicted contact.

Question Q3: How closely do decoded tactile predictions match actual contact during execution, particularly under object-position or material changes? Does the advantage persist with shorter execution chunks?

Response: We have added experiments examining predicted–executed contact correspondence, an unseen object position, and shorter execution windows.

(1) Correspondence between predicted tactile signals and executed contact. Appendix B.5 (pp. 24–25) and Figure 9 (p. 27) compare contact predictions decoded by FM-ILA with tactile signals recorded while executing the corresponding actions. Since the right hand holds the tool in both peeling and cutting, we show its two fingertips. For each sensor, the contact-load proxy is the magnitude of the spatially summed XY deformation, ∥∑i(dx,i,dy,i)∥2\|\sum_i(d_{x,i},d_{y,i})\|_2. Predictions and measurements use the same native sensor scale.

At the original position, repeated load increases, decreases, and local peaks show temporal correspondence, indicating that the decoded contact evolution reflects the physical execution. Physical replay of demonstration actions also produces contact amplitudes that differ from the demonstrations (Appendix Figure 10, p. 29), showing that hardware execution factors, including arm compliance and motion-tracking error, also affect contact reproduction.

(2) Consistency at a completely unseen position. The shifted-position examples in Figure 9 retain temporal correspondence for some contact events, with larger deviations in local peaks and load amplitudes. We further compare one plate-wiping rollout at the original position with one at a completely unseen position raised by 5 cm, both without additional force feedback. After aligning predictions and measurements by execution time, we normalize each right-fingertip resultant-force proxy, compute its magnitude, average the two fingertips, and calculate the Pearson correlation. The correlations are 0.905 and 0.827 at the original and new positions, respectively. At the new position, the mean predicted and measured amplitudes are 0.598 and 0.394, giving a ratio of 1.52. These results show that predicted contact continues to capture the temporal evolution of actual contact at a height outside the training range.

(3) Performance with shorter execution chunks. For each policy, both execution settings predict 128 steps and use the same checkpoint, observations, inference configuration, and 30 Hz action-command rate. The 32-step variant executes only the first 32 predicted actions, then pauses motion, acquires a new observation, and replans synchronously. This reduces the continuous open-loop motion window from 4.27 s to 1.07 s.

Table 3 shows that 32-step execution reduces FextF_{\mathrm{ext}} L1 in 6/8 ILA comparisons. Average SR increases from 80.0% to 80.5% for FM-ILA and from 66.75% to 68.75% for VLA-ILA. Under the same 32-step execution setting, FM-ILA achieves higher SR and lower force-proxy error than FM-tac on all four tasks, with an average SR of 80.5% versus 55.5%, a 25.0-percentage-point improvement. Thus, ILA retains its advantage over direct action generation with a shorter execution window. However, shorter synchronous execution also introduces more stops and restarts. The observed startup transients and motion discontinuities are consistent with the larger error standard deviations in 6/8 comparisons and the weaker gains on egg flipping.

Table 3. Effect of shorter execution windows on policy performance. Entries: SR (%) / FextF_{\mathrm{ext}} L1 (mean ± SD). All variants predict 128 steps; sync. denotes synchronous blocking replanning. ILA results appear in manuscript Table 7 (p. 28); FM-tac provides the matched 32-step baseline here.

PolicyExecution settingCucumber PeelingEgg FlippingCelery CuttingPlate Wiping
FM-tac32 (sync.)55 / 2.54 ± 0.6280 / 1.21 ± 0.8022 / 9.34 ± 1.1365 / 5.10 ± 0.49
FM-ILA128 (original)75 / 2.04 ± 0.3795 / 0.56 ± 0.6450 / 7.52 ± 0.78100 / 3.44 ± 0.35
FM-ILA32 (sync.)75 / 1.88 ± 0.3695 / 0.62 ± 0.7152 / 7.24 ± 0.91100 / 2.53 ± 0.52
VLA-ILA128 (original)50 / 2.28 ± 0.4185 / 0.38 ± 0.5437 / 7.22 ± 1.1795 / 2.51 ± 0.38
VLA-ILA32 (sync.)55 / 1.94 ± 0.6180 / 0.46 ± 0.6740 / 7.04 ± 1.08100 / 2.26 ± 0.50
AuthorsOpenReview ↗

Response to cPYq: Joint Encoding, Contact Consistency, and Representation Transfer (3/4)

W3 / Q2: Shared representations and transfer

Weakness W3: Cross-task reuse is not demonstrated. Main experiments train task-specific encoders and decoders. The shared-model study evaluates reconstruction rather than policy performance or transfer.

Question Q2: Can a shared encoder and decoder support policy learning across all four tasks? Can they remain frozen when adapting to a new task or contact condition?

Response: All ILA policies in the original main results use the same encoder and decoder jointly pretrained on all four tasks, frozen during task-specific policy training. Thus, main-text Table 1 (p. 7) already reports downstream performance with shared representations.

(1) Downstream performance of the shared representation. Table 4 compares four-task shared and task-specific representations. Both settings match downstream data, policy inputs, latent dimensions, decoder types, and training budgets: 50,000 steps of training from scratch for FM and 50,000 steps of post-training for VLA. Average SR is 80.0% versus 79.5% for FM-ILA and 66.75% versus 67.5% for VLA-ILA, respectively. Every task-level SR difference is within 5 percentage points, and the shared representation achieves lower execution force-proxy error in 6/8 comparisons. These results show that one frozen shared encoder and decoder can support policy learning across all four tasks while preserving downstream performance comparable to task-specific representations.

Table 4. Downstream policies using shared and task-specific representations. Each task cell reports SR (%, ↑) / FextF_{\mathrm{ext}} L1 (mean ± standard deviation, ↓). Both modules remain frozen. Results correspond to manuscript Table 12 (p. 34).

PolicyRepresentation trainingCucumber PeelingEgg FlippingCelery CuttingPlate Wiping
FM-ILAFour-task shared75 / 2.04 ± 0.3795 / 0.56 ± 0.6450 / 7.52 ± 0.78100 / 3.44 ± 0.35
FM-ILATask-specific75 / 2.16 ± 0.42100 / 0.60 ± 0.4548 / 7.96 ± 0.7195 / 3.23 ± 0.57
VLA-ILAFour-task shared50 / 2.28 ± 0.4185 / 0.38 ± 0.5437 / 7.22 ± 1.1795 / 2.51 ± 0.38
VLA-ILATask-specific45 / 2.46 ± 0.4890 / 0.44 ± 0.4040 / 7.06 ± 0.9795 / 2.70 ± 0.68

Reconstruction on the original held-out source-task sets further shows that the shared model reduces action error on all four tasks and force-proxy error on three tasks (manuscript Table 11, p. 34).

(2) Transfer to unseen tasks. We evaluate vase wiping and carrot peeling, neither of which was included in representation pretraining. We compare Original decoder, which reuses both the original encoder and decoder without adaptation, with Decoder FT, which freezes the same encoder and fine-tunes only the decoder using 10 demonstrations per task. Reconstruction is evaluated on identical held-out trajectories.

To evaluate downstream reuse, we train an FM-ILA policy from scratch for each new task, using 20 demonstrations and 16,000 training steps per task (Appendix A.1, p. 18). The shared encoder remains frozen, and the policy learns to generate its latent targets. At evaluation, we fix the policy checkpoint, observations, and execution settings for each task, without additional force feedback, and switch only between the original decoder and the decoder adapted with 10 demonstrations.

Table 5. New-task reconstruction and execution with a frozen encoder. Within each task, the policy is fixed and only the decoder changes. Reconstruction errors are ×10⁻³; execution error is FextF_{\mathrm{ext}} L1 (mean ± SD). From manuscript Table 2 (p. 9).

TaskDecoderAction L1Tactile L1SRExecution error
Vase wipingOriginal27.261.7380%2.18 ± 0.56
Vase wipingFT7.401.36100%1.45 ± 0.34
Carrot peelingOriginal15.794.2840%2.62 ± 0.60
Carrot peelingFT6.614.0565%2.25 ± 0.42

As shown in Table 5, the newly trained policies already achieve 80% and 40% SR on vase wiping and carrot peeling using the original shared encoder and decoder. Adapting only the decoder increases SR to 100% and 65%, while reducing execution force-proxy error by 33.5% and 14.1%, respectively. Main-text Figure 5 (p. 9) shows the complete carrot-peeling (top) and vase-wiping (bottom) rollouts, including tool acquisition, contact establishment, task execution, and completion. Decoder adaptation lowers action and tactile reconstruction errors and improves both task completion and agreement with expert contact loads, supporting the shared interaction representation as a reusable action space for new-task policy learning.

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback. Following their suggestions, we have updated the manuscript with additional baselines and controlled ablations, expanded simulation and transfer experiments, and further analyses of contact consistency, feedback, and robustness. We have also clarified the training and evaluation protocols and statistical reporting. The updated multimedia supplement includes few-shot carrot-peeling and vase-wiping demonstrations with a frozen encoder and a fine-tuned decoder, along with zero-shot transfer to real eggs. We highlighted revisions in the manuscript in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.