- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
TCR-ARM assigns tactile information to two complementary interfaces within a Qwen3.5-based VLA. A shared DINOv3 encoder processes tactile histories and produces compact tokens for persistent VLM context and dense spatial features for local refinement. Wrist visual tokens retrieve dense tactile features through gated cross-attention before conditioning the action expert. Route-aware tactile pretraining (RAT) samples no-tactile, context-only, and full-tactile routes to train contextual processing independently of refinement. Training additionally uses contact classification and future visual-latent prediction supervised by frozen V-JEPA features. The main contribution is this coordinated two-interface architecture and route-training strategy. Following 1,500 hours of pretraining, the model reports 70.0% average success across six physical tasks.
Critical strengths
The distinction between persistent contact context and selective local refinement is well motivated and implemented through complementary spatial representations.
Real-world ablations examine both routes, gating, spatial granularity, and alternative dropout strategies; RAT improves average success from 59.2% to 70.0%.
This work compares various baselines under different task settings, which deserves great soundness.
Critical weaknesses
The main comparison confounds architecture with 1,500 hours of additional tactile pretraining. Matching downstream demonstrations does not isolate the proposed method, and pretraining–evaluation task overlap is insufficiently documented.
Evidence for “context reasoning” and selective refinement relies substantially on qualitative predictions and selected gate values. Aggregate behavioral probes and an ablation of future-latent supervision are missing.
Each condition uses only 20 trials per task without reported training-seed variability. RAT’s five-point advantage over independent route dropout therefore has uncertain reliability; generalization tests cover only appearance changes in pen capping.
Questions for authors
-
How much of the improvement remains when competing architectures receive the same 1,500-hour pretraining corpus and comparable compute? Please also clarify task and object overlap between pretraining and downstream evaluation.
-
Does RAT still outperform independent route dropout when context and refinement activation probabilities are matched to RAT’s respective frequencies of two-thirds and one-third?
-
Can you isolate the contribution of future-latent supervision and provide aggregate gate statistics or controlled interventions demonstrating the intended distinction between persistent context and selective refinement?
3 comments
Response to Reviewer MpLm (2/2): Predictive context, refinement, and transfer
Continuing our response to Reviewer MpLm.
Q3/W2: Predictive supervision and the two tactile roles
Following your suggestion, we added quantitative tests of future-latent supervision and aggregate gate behavior, complemented by controlled route and gating comparisons.
Future-latent supervision. The no-future-prediction variant disables the loss and auxiliary prediction head while retaining future-task tokens. Other training settings are unchanged. A separate variant removes only affine tactile modulation of the predictor, retaining tactile tokens in the VLM prefix.
Four-task results (Section 4.7, manuscript Tables 4 and 5)
| Configuration | Success, mean ± SD (%) | 95% CI (%) | Future MSE ↓ |
|---|---|---|---|
| Without future prediction loss/head | 50.8 ± 1.44 | [47.25, 54.42] | Not applicable |
| Without tactile modulation | 56.7 ± 0.72 | [54.87, 58.46] | 0.534 |
| Full TCR-ARM | 63.3 ± 1.44 | [59.75, 66.92] | 0.421 |
Success uses three post-training runs with 20 trials per task and run; confidence intervals follow Q2 in Part 1. MSE averages offline evaluation on 20 trajectories per task.
These results connect predictive supervision to task performance. Tactile context reasoning denotes predictive inference in the VLM representation space supporting action generation. The future objective supervises the VLM tactile and future-task hidden states during training (Sections 3.2 and 3.4).
Persistent context and selective refinement. Our shared encoder supplies compact tokens to the VLM and dense features retrieved by wrist visual tokens through cross-attention. A wrist-conditioned gate weights the dense updates independently of compact context availability.
Within each group below, variants share the backbone, pretraining initialization, data, training steps, and effective batch size, and none uses RAT. The context comparison disables refinement. The refinement comparisons retain context (Section 4.3, manuscript Table 2).
| Configuration | Mean ± SD (%) |
|---|---|
| Context route only | |
| Contact-gated context | 43.1 ± 0.48 |
| Persistent context | 46.9 ± 0.48 |
| Both routes, with persistent context | |
| Fixed refinement gate, 0.5 | 42.2 ± 0.48 |
| Contact-based refinement gate | 52.8 ± 0.96 |
| Learned refinement gate | 59.4 ± 1.27 |
Persistent context outperforms contact-gated context, while learned refinement outperforms fixed or contact-based weighting. This supports different availability and modulation rules for the two interfaces. Q2 separately tests how RAT coordinates their training.
Aggregate gate statistics. Appendix G, Figure 12 (left) reports frame-level distributions from 60 episodes of the full RAT model, ten per task.
| Contact condition | Approximate gate distribution |
|---|---|
| No contact | Cluster near 0.1 |
| Static contact | Median near 0.35 |
| Dynamic contact | Median near 0.7 |
Contact labels use tactile-image differences above 5% relative to a static reference. Static contact is stable over time; dynamic contact changes or fluctuates. The learned gate is conditioned on wrist visual features rather than set by this threshold. Figures 7 and 12 (right) additionally compare RAT and full-tactile-only models over complete rollouts. Compact context remains available independently of the refinement gate.
W3: Generalization beyond pen appearance changes
We expanded evaluation to paper-cup placement without task-specific adaptation and transfer to an unseen tactile sensor.
Additional transfer success (%)
| Evaluation | TCR-ARM | Baseline results |
|---|---|---|
| Paper-cup placement | 40.0 | π0.5: 10.0; Dream-Tac: 5.0; N0-VTLA: 5.0 |
| Unseen GelSight sensor, four-task average | 26.3 | Dream-Tac: 11.3; N0-VTLA: 8.8 |
Paper-cup placement requires putting the cup into a box without damaging it, with no task-specific demonstrations or fine-tuning (Section 4.6; Appendix D). For sensor transfer, trained policies are directly evaluated with GelSight, unseen during policy training, without sensor-specific fine-tuning, over 20 trials per task (Appendix E, Table 15). These evaluations extend the evidence to a different task and sensor. Absolute success under the sensor change remains low across methods, with TCR-ARM performing best on all four tasks.
Follow-up to Reviewer MpLm
Dear Reviewer MpLm,
Thank you again for your constructive feedback. We would like to briefly follow up and hear your thoughts on our responses and the revised manuscript.
Could you let us know whether any concerns remain? We would be happy to provide further clarification before the discussion closes.
Response to Reviewer MpLm (1/2): Pretraining and matched route sampling
Dear Reviewer MpLm,
Thank you for your careful review and constructive questions about pretraining, route sampling, and behavioral evidence. We address Q1–Q3 in order, including the corresponding weaknesses, followed by the expanded generalization tests.
Q1/W1: Pretraining data, comparison conditions, and overlap
We appreciate the value of a comparison with matched upstream data and compute. Given our computational constraints, we were unable to retrain the baseline architectures on the full 1,500-hour corpus under that setting. The reported baselines use their public pretrained checkpoints followed by post-training on the same 100 demonstrations per target task.
To address your concern about the source of the gains, the revision provides two complementary comparisons. First, we added TCR-ARM without our tactile pretraining corpus. It starts from pretrained QWen3.5-2B weights and applies RAT during post-training on the 100 demonstrations per task (Section 4.1).
Six-task success (manuscript Table 1)
The advantage persists without this additional corpus. Second, the common-backbone comparisons under Q2 and Q3 examine route training and fusion choices with shared training data and controlled settings. They provide evidence about the design beyond the original system-level comparison.
Task and object overlap. The six downstream tasks and their post-training demonstrations are excluded from our pretraining corpus (Section 3.4; Appendix A.1). Object categories overlap, and physical object instances are not guaranteed to be disjoint. The separation is at the task and demonstration level.
Q2/W3: RAT with matched activation probabilities and repeated runs
Yes, RAT retains an advantage when the routes' activation probabilities are matched. Independent dropout probabilities of (1/3,2/3) for context/refinement yield activation frequencies of (2/3,1/3), matching RAT.
The matched comparison uses the same architecture, pretraining initialization, data, losses, training steps, effective batch size, and post-training settings. Only the joint route-sampling distribution changes (Section 4.4, manuscript Table 3).
Six-task route-sampling results
RAT uniformly samples masks from {(0,0), (1,0), (1,1)} to train a specific dependency: contextual touch supports control without dense refinement and remains available whenever refinement is enabled. The matched independent control has the same activation frequencies but permits refinement without context. The gain supports this coordination through the joint sampling distribution.
For key experiments, we repeated post-training with three seeds, evaluating each policy over 20 trials per task. Six-task comparisons therefore include 360 trials per configuration. Means and sample SDs are across the three task-averaged run results. Appendix C.3, Tables 11–14 provide the seed counts.
The two-sided 95% Student-t intervals use unrounded run-level rates: mean ± 4.30265 × SD / √3 (2 degrees of freedom). Each configuration's pretrained checkpoint was fixed, so these intervals describe downstream repeatability under the tested initializations. Pretraining was not repeated. Percentage-point gains use displayed means.
Part 2 addresses predictive supervision, controlled route/gate comparisons, aggregate gate behavior, and generalization.