- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TAD introduces a mechanism for adapting predictive tactile knowledge to robot control. A teacher first learns to predict DINOv3 features of future tactile changes from current observations, then supervises intermediate predictive tokens in a π0.5-based policy. A teacher-initialized assistant subsequently adapts through action supervision while remaining anchored to the frozen teacher. Its EMA supplies evolving distillation targets, while its online features provide residual corrections to the action-denoising field. Asymmetric attention lets action tokens read predictive tokens while preventing reverse information leakage. The core contribution is the combination of preserved predictive supervision and action-driven adaptation. The deployed policy retains the assistant, achieving reported average success rates of 90.0% in simulation and 87.5% on physical tasks.
Critical strengths
The teacher-anchored assistant addresses a meaningful tension between preserving predictive representations and adapting them for control. The asymmetric attention design carefully prevents action-input leakage.
Evaluation spans eight simulation and six physical tasks, with consistent improvements and substantive teacher, assistant, and future-target ablations.
Critical weaknesses
Removing the assistant simultaneously removes adaptive targets, auxiliary supervision, and residual corrections. The experiments therefore cannot establish whether adaptive distillation itself drives the improvement.
I am confused about the archi, especially about the main figure. I strongly recommend the authors to draw a more clear overview, it cannot highlight your special designs with so many redundant information.
Deployment retains the assistant and its frozen visual-language context, but latency and memory costs are unreported. Main success-rate comparisons also lack uncertainty across independently trained models.
Questions for authors
-
Can you independently ablate EMA targets, teacher anchoring, and assistant residual corrections while retaining comparable model capacity and training budgets? Which component explains the gain over direct frozen-teacher distillation?
-
Can prediction and action-correction diagnostics be repeated on episodes withheld from every training stage? The current teacher holdout inherits representation exposure, while correction diagnostics use training demonstrations.
-
What are the end-to-end inference latency, peak memory, and total training cost relative to N0-VTLA, including the deployed assistant and its frozen visual-language context?
Again, I strongly recommend the authors to draw a more clear overview, if the authors could give a clearer model illustration, I will consider raising my score.
4 comments
Q1. Which components explain the gain over direct frozen-teacher distillation?
We changed one Stage II mechanism at a time, retaining the assistant architecture, auxiliary action supervision, and policy-update budget in all three ablations.
- EMA target. We replaced the student’s target from the exponential moving average (EMA) of the assistant with the frozen teacher’s prediction. The online assistant, its anchor, and its action residual remained active.
- Teacher anchoring. We retained the assistant’s initialization from the teacher and set only the anchoring loss weight to zero.
- Assistant action residual. We set the assistant’s residual correction to zero during both Stage II training and evaluation. The student’s Spatial Readout remained active.
Table E1. UniVTAC success rates (%), with 100 evaluation episodes per task and policy.
| Task | Direct teacher distillation, no assistant | Fixed teacher target, assistant retained | Full TAD | No teacher anchor | No assistant residual |
|---|---|---|---|---|---|
| Lift Bottle | 99 | 99 | 100 | 98 | 100 |
| Pull-out Key | 90 | 95 | 99 | 76 | 96 |
| Lift Can | 80 | 91 | 95 | 86 | 96 |
| Put Bottle | 83 | 85 | 90 | 84 | 92 |
| Insert Hole | 95 | 99 | 98 | 84 | 87 |
| Insert HDMI | 32 | 35 | 44 | 25 | 34 |
| Insert Tube | 99 | 98 | 95 | 94 | 96 |
| Grasp Classify | 100 | 100 | 99 | 100 | 99 |
| Mean | 84.75 | 87.75 | 90.00 | 80.88 | 87.50 |
Where the gain comes from. Full TAD exceeded direct frozen-teacher distillation by 5.25 percentage points (84.75% to 90.00%). With the teacher target fixed, retaining the assistant and its action branch raised success to 87.75% (+3.00 points). Switching the student to the adaptive EMA target then raised success to 90.00% (+2.25 points). Among the single-component ablations, removing the teacher anchor produced the largest drop (90.00% to 80.88%). Removing the residual reduced success to 87.50%, including a drop from 44% to 34% on Insert HDMI. These last two comparisons measure each component’s effect with the others active, so their drops cannot be added to partition the overall gain.
Q2. Do the tactile-prediction and action-correction results hold on episodes withheld from every training stage?
Yes. We repeated both diagnostics on complete episodes excluded from every training stage, including tactile representation adaptation and policy initialization.
1. Future tactile prediction. Given current observations, we scored the frozen teacher and the trained online assistant against features of the observed tactile change from the current frame to the frame ten steps later. We report cosine distance averaged equally across the eight UniVTAC tasks (lower is better).
| Frames evaluated | Valid frames | Frozen teacher | Online assistant | Training-target mean baseline |
|---|---|---|---|---|
| All valid frames | 13,622 | 0.0669 | 0.0752 | 0.1731 |
| Subset with measurable tactile change | 12,742 | 0.0646 | 0.0933 | 0.1465 |
The teacher had the lowest error on both frame sets. The assistant also outperformed the baseline that always predicts the mean training target, including on frames with measurable change. This supports retention of future-touch information after action training.
2. Assistant action correction. At each checkpoint, we evaluated ten held-out Insert HDMI demonstrations. Within each pair, we fixed the observations, noisy actions, flow times, and target velocities, and toggled only the assistant’s residual correction.
| Checkpoint (updates) | Residual off (mean MSE) | Residual on (mean MSE) | Relative reduction |
|---|---|---|---|
| 2,000 | 0.01574 | 0.00856 | 45.6% |
| 5,000 | 0.01251 | 0.00365 | 70.8% |
Enabling the residual reduced offline action-field error, measured by normalized flow-matching MSE, on unseen HDMI demonstrations at both checkpoints.
Q3. What are the end-to-end inference latency, peak GPU memory, and total training cost of TAD?
For the fully deployed TAD policy, end-to-end inference took 623.2 ms per action chunk, and peak GPU memory usage was 16.66 GB. Both measurements include the online assistant and its frozen vision–language and tactile context. Training across all five stages used 72.74 H100 GPU-hours in total.
Clarifying the architecture in Figure 2
Thank you for pointing out that the original figure made the architecture difficult to follow. We redrew Figure 2 and omitted lower-level implementation details to highlight how the policy uses current touch and learns to anticipate changes in contact.