- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
In bimanual imitation learning, demonstrations often follow a fixed timing schedule between both arms and this can lead to spurious cross-arm coupling during learning.
This paper tackles this issue by proposing Constraint-preserving Temporal Recombination (CTR), a training time data intervention that teaches the policy when both arms can act independently, and when both should coordinate. It proceeds in 3 steps:
-
For each arm, segment each demonstration into phases, and identify the true task constraints between arms using VLM assistance + human review.
-
Recombine the timing of both arms within each demonstration while preserving those task constraints.
-
During policy training, apply an action loss mask to mask out the artificial waiting intervals created by temporal recombination.
CTR is validated on 4 simulated and 4 real robot bimanual tasks (in different experiments).
Critical strengths
-
The paper tackles a well defined and practical failure mode in bimanual imitation learning. The proposed method addresses this at training time by recombining existing demonstrations, hence doesn't need additional data collection (but requires good VLM annotations/visual compositing).
-
In the experiments, CTR achieves strong empirical performance on both simulated and real robot tasks.
Critical weaknesses
-
Experimental coverage: the paper considers 3 policy families (diffusion policy, ACT and ) and 8 tasks in total, but all the main robustness tests only evaluate on 1-2 tasks. Table 4 covers all models and tasks but only tests the normal starting state against a single baseline (
Serial). -
CTR combined two components, temporal recombination and
IdleMask. However the ablation removingIdleMask(Table 2) is only tested from the normal starting state (where even standard baselines already achieve high success), and hasn't been evaluated on the main robustness experiments.
Questions for authors
For the main points, please refer to the section on weaknesses.
Suggestions
It would be good to improve the paper's writing clarity.
It would also be good to improve the paper's clarity around how the visual compositing and loss masking work in edge cases.
5 comments
Response to Reviewer jyBi (2/2)
W2. The IdleMask ablation is only from the normal starting state.
TL;DR: IdleMask does not drive the gains, and we have removed it from the method.
We ran the IdleMask ablation on all four simulated tasks at the standard start (a) and repeated the progress-state evaluation (Table 1) with CTR trained without IdleMask (b).
(a) IdleMask ablation at the standard start, four tasks (3 × 100 episodes; only 3 × 50 for Put Object Cabinet; also in General Response, item 1). The with-minus-without differences are +3.7, −10.0, +4.0 and +2.0 points, none significant after Holm correction, mean −0.1. The 14-point gap on Scan Object in Table 2 of the submission came from single 50-trial evaluations.
(b) Progress-state evaluation (Table 1) with CTR trained without IdleMask.
Table R2. Put Object Cabinet (sim), , mean SR over 3 × 50 episodes per starting state; same scenes and policy seeds for all rows. C00: nominal start; Cmid: both arms at mid-progress; L-lead, R-lead: one arm ahead of the other; CL, CR: one arm at its ready or completion state, the other at the start.
| Data | C00 | Cmid | L-lead | R-lead | CL | CR | Six-state mean | Worst |
|---|---|---|---|---|---|---|---|---|
Serial | 34.0 | 92.0 | 70.0 | 35.3 | 60.0 | 42.7 | 55.7 | 34.0 |
Mixed | 58.7 | 98.7 | 62.7 | 50.0 | 47.3 | 82.0 | 66.6 | 47.3 |
CTR w/o IdleMask | 62.0 | 99.3 | 72.0 | 66.7 | 51.3 | 96.7 | 74.7 | 51.3 |
The conclusion of Table 1 holds without IdleMask: CTR is 8.1 points above Mixed on the six-state mean and 19.0 above Serial. Since Mixed is itself a restricted form of CTR, built by our own recombination pipeline, the revision replaces the claim that CTR raises worst-state success from 56% to 76% (abstract, introduction) with a comparison to the original, unretimed data (worst state 0% → 51.3% in simulation and 38.7% → 74.7% on the real robot; General Response, item 2). Since IdleMask has no significant effect at the standard start (a) and the Table 1 gains hold without it (b), we have removed it from the method: Sec. 4 presents CTR as temporal recombination trained with the standard action loss, and the IdleMask experiments appear only in the appendix (App. F.6–F.7).
Suggestion 1. Writing clarity.
TL;DR: We have revised the presentation: Sec. 3 uses plain English in arguments with a clear storyline, Sec. 2 highlights how CTR differs from prior work, Sec. 4 is more concise and explicit, and terms are consistent throughout the paper.
- Sec. 2. Its paragraph on DexMimicGen and TriManPolicy is rewritten to state each method's goal and single schedule per demonstration, and why CTR varies it (see General Response, item 4).
- Sec. 3. Its storyline is revised to flow smoothly, and the core arguments are made in plain English instead of equations: a demonstration records one schedule, so the policy can time one arm by the other's progress (spurious cross-arm coupling) and fail when the arms drift apart (new Sec. 3.2). An arm should depend on the other only under task-required constraints, which hold in every admissible schedule (Sec. 3.3). More such demonstrations cannot tell these constraints from incidental timing, so the schedules must be varied (Sec. 3.4).
- Sec. 4. It is more concise and explicit.
Suggestion 2. Edge cases of visual compositing and loss masking.
TL;DR: App. F.5, F.6 and G describe them. The main cases are below.
- Visual compositing. A recombined frame shows the left arm at one recorded moment and the right arm at another, a pair that may never have been recorded together. Only the main camera image is rebuilt: each arm is cut out of its own source frame and pasted onto the scene background. An object moves with the arm that carries it from pickup to release, and its original spot is filled in so it never appears twice. Where the two arms overlap in the image, depth decides which is in front when available. Every new pairing is checked for collisions with the robot model and task-specific gates, and a schedule is rejected when clearance cannot be certified. Of 363 real sources, 11 (3.0%) failed these checks (General Response, item 3) and the reviewer excluded four (1.1%) (new App. F.2).
- Loss masking.
IdleMaskis no longer part of the method (General Response, item 1). In the appendix experiments it masks only an arm's own action targets during waits that recombination inserted. The other arm's targets, required holding, and holding after completion are never masked.
We thank the reviewer for the writing suggestions, which made the paper easier to read. We hope the added analyses address the reviewer's concerns.
Follow-up on our response
Dear Reviewer jyBi,
Thank you again for your careful review and suggestions. We have posted our responses above and uploaded the revised PDF, with changes in blue. In brief:
- Experimental coverage (W1). The progress-state tests use because DP and ACT rarely succeed even from the nominal state, and the action-level probe covers all three policy families. We also tested a second simulated task, Sort Blocks with a slipped grasp: the untouched arm finishes its own blocks in 91/100 scenes with CTR and 46/100 with
Concurrent. - IdleMask (W2). With CTR trained without
IdleMask, the gains of Table 1 hold (8.1 points overMixedon the six-state mean), andIdleMaskhas no significant effect at the standard start on any of the four simulated tasks. We have therefore removed it from the method (Sec. 4, App. F.6–F.7). - Writing clarity and edge cases. Sec. 2–4 are revised for clarity, and App. F.5, F.6 and G describe the edge cases of visual compositing and loss masking.
Could you let us know whether these responses address your concerns? If anything remains unclear, we would be glad to provide further details during the remaining discussion period.
Best regards,
The Authors
Response to Reviewer jyBi (1/2)
We thank the reviewer for the careful reading and the concrete suggestions. We respond in the order of the review.
W1. The robustness tests cover only π0.5 on one or two tasks; Table 4 compares only against
Serial.TL;DR: The progress-state tests use π0.5 because DP and ACT rarely succeed even from the nominal state; the action-level probe covers all three policy families, and we tested a second simulated task.
The progress-state evaluations cover π0.5 only, for a practical reason: the test needs a policy that often succeeds from the nominal state, otherwise all six starting states fail alike. On Put Object Cabinet, DP and ACT trained with CTR reach 4% and 24% from the nominal state (Table 3), against 62.0% for π0.5 (re-evaluated; 66% in the submitted Table 4).
Evidence across the three policy families comes instead from the action-level probe on Deliver Water (Fig. 4; App. E.4), which measures how much one arm's predicted actions change when only the other arm's state is varied. For ACT, DP and π0.5 alike, the
Concurrent-trained policy is more sensitive to the other arm in the independent phases than in the synchronized phase, and the CTR-trained policy reverses this (selectivity gap, Sserve minus the mean of the two independent phases: +2.26 vs. −5.71 for ACT, +1.60 vs. −3.22 for DP, +1.11 vs. −1.22 for π0.5, computed from Table 10).What we ran during the discussion period: we repeated the progress-state evaluation on Put Object Cabinet three times on shared scenes and seeds (Table R2 below; they replace the simulated rows of Table 1 in the revision), and tested a second simulated task, Sort Blocks, with a different perturbation. There we open one gripper right after its first lift, as in a slipped grasp, and leave the other arm untouched; the untouched arm then finishes its own blocks in 91/100 scenes with CTR but 46/100 with
Concurrent(details in the NBh4 thread, W0). We also tested the official co-trained π0.5 checkpoint of RoboTwin 2.0: on Put Object Cabinet it succeeds in 38/47 episodes when the left arm fetches the object, but in only 1/53 when the two arms' subtasks are swapped (NBh4 thread, W0).The C00 column of Table 1 compares CTR with the other training sets at the standard start (sim: 62.0% vs. 29.3–58.7%; real: 95.3% vs. 78.0–88.7%). Table 3 (Table 4 in the submission) compares against
SerialbecauseSerialis the narrow-support control closest to how demonstrations are collected, one arm's subtask and then the other's, while sharing CTR's visual-synthesis pipeline, so the comparison isolates schedule support. Its simulated π0.5 rows were re-evaluated with 3 × 100 episodes (3 × 50 for Put Object Cabinet) on shared scenes and seeds (General Response, item 2). Sec. 5 (Table 1) states that the progress-state evaluations use π0.5 on Put Object Cabinet (sim) and Sort Letters (real).