acceptodds
Under review as a conference paper at ICLR 2027

TacBootstrap: A Tactile-Informed Copilot for Contact-Rich Bimanual Manipulation from UMI Demonstrations

Abstract

Contact-rich bimanual demonstration collection is difficult when visual teleoperation cannot resolve local contact. We present TacBootstrap, a tactile-informed, phase-gated copilot trained exclusively on Universal Manipulation Interface (UMI) demonstrations. Both UMI and robot grippers use fingertip Daimon vision-based tactile sensors; their shear and depth channels provide the policy’s tactile inputs. The operator controls non-contact motion through virtual reality (VR), then manually transfers authority near contact while VR actions pause. A Slow-Fast encoder combines wrist-image context with three frames of tactile and robot-state observations. Current-state retrieval provides a demonstrated trajectory anchor, and an anchor-based flow head generates bimanual local-pose and gripper action chunks for closed-loop execution. Robot recordings are used only for evaluation. On paper-cup separation, cap tightening, and network-cable insertion, TacBootstrap achieves 77.7% task-macro success versus 55.4% for robot VR, with higher success in every task; direct UMI achieves 100%. Task-level attempts per success decrease from 1.81 to 1.29. Recorded durations are also lower, although differing recording boundaries limit efficiency comparisons. These system-level results support leveraging portable visual-tactile demonstrations for robot-side contact assistance.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot learning, tactile representation learning, multimodal learning, bimanual manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_5LHAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces TacBootstrap, a tactile-informed contact copilot trained exclusively on portable Universal Manipulation Interface (UMI) demonstrations for contact-rich bimanual manipulation. The operator controls non-contact motion through VR and manually initiates a hard handoff near contact. VR commands then pause while the learned policy controls the bounded contact phase. The method combines wrist-image context with three frames of tactile and state observations through a Slow-Fast encoder, retrieves a reference action chunk from UMI demonstrations using the current task, TCP, and gripper state, and uses a conditional flow model to predict residual corrections around that reference. The resulting bimanual local-pose and gripper action chunks are executed in a receding-horizon loop.

The evaluation compares robot VR, TacBootstrap, and direct UMI execution on paper-cup separation, cap tightening, and network-cable insertion. The reported task-macro success rate increases from 55.4% for robot VR to 77.7% for TacBootstrap, while the macro average of task-level attempts per success decreases from 1.81 to 1.29. Direct UMI serves as a portable human-execution reference rather than an autonomous robot baseline. The paper also reports lower recorded mean durations, but explicitly acknowledges that recording boundaries differ across conditions, preventing an equivalent interpretation as improved end-to-end collection efficiency.

Critical strengths
  1. The problem is practically relevant, and the division of authority between the human and the policy is well defined. The paper investigates whether portable visual-tactile demonstrations can support robot-side contact assistance without robot-data policy training. Rather than attempting full-task autonomy, the interface preserves human control over task selection, approach, and handoff timing, while assigning the bounded contact phase to the learned policy. This establishes a clear intended boundary for evaluating transfer from portable demonstrations to robot-side assistance. 
  2. The three real-world tasks provide directionally consistent preliminary evidence, and the paper appropriately qualifies the scope of its comparisons. In Table 2, TacBootstrap improves the success-rate point estimate over robot VR on all three tasks, by 20.0, 21.8, and 25.0 percentage points, respectively. The paper also correctly distinguishes direct UMI execution from an autonomous robot baseline and acknowledges that the interface comparison does not isolate individual policy components.
Critical weaknesses
  1. It is unclear which implementation produced the reported robot results. The main text and Figure 2 describe a three-frame feed-forward encoder without persistent recurrent state, current-state trajectory retrieval, and local actions represented using Euler-angle rotation coordinates. However, Appendix C explicitly states that the audited local prototype is a different variant using a temporal GRU, a global medoid anchor, and rotation-vector actions. It also states that a version-matched checkpoint and robot-run manifest are not included in the present evidence package. 
  2. The reported performance improvement lacks basic trial-level statistics. Section 5.4 and Appendix E state that task-level attempt denominators for TacBootstrap and UMI are unavailable, as is variability information for TacBootstrap durations. Without the number of successes and total attempts for each task and condition, readers cannot verify the statistical basis of the reported success rates or assess the precision and reliability of the differences. The issue is not the absence of a particular significance test; it is the absence of the basic counts needed to interpret the primary result. Rounded percentages cannot substitute for those counts, and a 100% success rate with an unknown sample size does not establish reliable performance.
Questions for authors
  1. Implementation clarification: Which implementation produced the robot results in Table 2?

  2. Data verification: Can you provide the number of successful attempts S and total attempts N for every task and condition, together with the corresponding per-attempt outcomes?

  3. Targeted validation: Are results available for a matched no-tactile variant and a retrieved-reference-only variant without the learned residual?

Reviewer_qscSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes TacBootstrap, a phase-gated "contact copilot" for bimanual demonstration collection. A human operator controls free-space motion via VR and manually hands off authority near contact, after which a policy trained only on portable UMI demonstrations (with Daimon fingertip tactile sensors on both the UMI interface and the robot grippers) executes the contact phase. The policy combines a Slow-Fast encoder (DINOv2 wrist images plus three frames of tactile/state input), current-state retrieval of a UMI trajectory anchor, and an anchor-conditioned flow-matching residual head. Experiments on three contact-rich tasks report macro-average success of 77.7% vs. 55.4% for pure VR teleoperation.

Critical strengths

The evaluation hygiene is unusually careful for a system of this type: success labels are physical task states; handoff failures, safety stops, and operator aborts are counted as failures rather than filtered; and the reported Wilson intervals are correctly computed. The motivation is also clear and practically relevant — contact-phase failures are a genuine bottleneck in teleoperation data collection, and the paper scopes its claims accordingly. However, these strengths concern reporting quality rather than the novelty or verifiability of the contribution itself, so they do not change my recommendation.

Critical weaknesses
  1. Overlap with prior work that the paper neither cites nor differentiates. DexTeleop-0 (arXiv:2606.23431, June 2026, i.e., before this submission) introduces a tactile-driven shared-autonomy layer on top of standard VR tracking, in which coarse human intent remains in charge while a real-time loop applies tactile-based contact corrections. This appears to substantially overlap with the phase-gated, tactile-informed copilot formulation that constitutes this paper's primary contribution, since the individual components (DINOv2 encoding, retrieval anchors, flow-matching residuals) are established modules. Relatedly, the appendix hardware photographs appear to show fiducial-like markers on the UMI gripper, suggesting demonstration poses come from marker-based tracking as in the standard UMI pipeline; it is unclear what is methodologically new on the collection side. This makes the novelty claim difficult to sustain.

  2. The evidence chain is broken in ways that are hard to repair post-hoc. Appendix C states that the audited local prototype is a different architecture (temporal GRU, global medoid anchor, rotation-vector actions) from the reported method, and no version-matched checkpoint or robot-run manifest is provided, so I cannot confirm that the Section 4 architecture produced the Table 2 results. Attempt denominators for the copilot and UMI conditions are unavailable (Appendix E), so no confidence interval or significance test exists for the headline 55.4% -> 77.7% comparison, and the VR intervals are wide enough to cover the copilot point estimates. Combined with the complete absence of component ablations (Appendix D) and unverified cross-device tactile calibration (Section 4.1), the observed improvement cannot be attributed to the tactile modality or to any specific design choice.

Questions for authors
  1. [Clarification] How does TacBootstrap differ from DexTeleop-0 (arXiv:2606.23431), which also keeps coarse human (VR) intent in charge while a tactile-driven loop handles contact corrections? Why is it not cited or compared?
  2. [Clarification] What are the exact attempt counts (S/N) per task for the copilot and UMI conditions, and why are they unavailable from your own trial logs?
  3. [Clarification] Which architecture generated the Table 2 robot results — the three-frame method of Section 4 or the GRU/medoid prototype of Appendix C? Can you provide a version-matched checkpoint and robot-run manifest?
  4. [Future Work] Component ablations (no tactile, 1-frame vs 3-frame, task-level anchor, deterministic head), a comparison against DexTeleop-0 and a learned baseline trained on the same UMI data, paired cross-device tactile calibration, and a multi-operator study.
Reviewer_3awdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

TacBootstrap proposes a visual–tactile copilot trained entirely on portable UMI demonstrations to assist bimanual robot teleoperation: the human controls the approach through VR, then manually hands over contact manipulation to the policy. Its main contributions are this control-sharing workflow and a policy combining visual context, recent tactile and state observations, retrieved demonstration trajectories, and flow-based residual corrections. Experiments on contact-rich manipulation tasks report improved success over VR teleoperation. However, validation is limited by narrow operator coverage, missing trial counts, and absent component ablations, etc. Robot recordings are used only for evaluation, with no downstream policy training demonstrated.

Critical strengths
  • The paper addresses the practical difficulty of contact-rich teleoperation through learned visual–tactile assistance, with reported improvements across the evaluated tasks providing preliminary evidence of its usefulness.
Critical weaknesses
  • The paper is generally difficult to follow. The descriptions of demonstration collection, policy training, and robot deployment are fragmented. Furthermore, visual documentation of the robot and task setups is also limited, with only basic hardware photos in the appendix. These presentation issues make the human–policy interaction, experimental conditions, and precise contributions difficult to understand and assess.
  • The comparison against VR teleoperation evaluates the complete assistance system, but absent component ablations and alternative learned-policy baselines leave the benefits of tactile input, retrieval, and flow decoding unestablished. Missing copilot trial counts further limit confidence in the reported improvements. Consequently, the experiments provide insufficient support for the proposed policy design.
Questions for authors

Which architecture produced the reported robot results, given that Appendix C describes a prototype different from the main-text method?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.