acceptodds
Under review as a conference paper at ICLR 2027

Learn the Task, Not the Schedule: Reducing Spurious Cross-Arm Coupling in Bimanual Imitation Learning

Abstract

A bimanual demonstration specifies both what the robot should do and one particular schedule for when each arm should act. Standard imitation learning does not distinguish task-required cross-arm constraints from incidental scheduling choices, and may therefore exploit opposite-arm progress as a shortcut cue, creating spurious cross-arm coupling even where relative timing is irrelevant for the current task phase. We introduce Constraint-preserving Temporal Recombination (CTR), a training-time data intervention that preserves recorded arm-local sensorimotor content while retiming the corresponding segments to realize admissible schedules that preserve task-required cross-arm constraints. We use CTR to train Diffusion Policy, Action Chunking with Transformers, and π₀.₅, and evaluate them on four simulated and four real-robot tasks. Across six real-robot bimanual progress-state initializations, CTR raises worst-state success from 56% for the strongest non-CTR schedule mixture to 76%. With π₀.₅ on an Independent task, a 4 s pause transfers 2.56 s of delay to the opposite arm under the Concurrent schedule distribution, but only 0.25 s with CTR. On real robots, CTR violates task constraints in only 0–4% of trials, versus 36–98% for narrow-schedule baselines.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Bimanual Manipulation, Imitation Learning, Data Augmentation, Spurious Correlations

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jyBiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

In bimanual imitation learning, demonstrations often follow a fixed timing schedule between both arms and this can lead to spurious cross-arm coupling during learning.

This paper tackles this issue by proposing Constraint-preserving Temporal Recombination (CTR), a training time data intervention that teaches the policy when both arms can act independently, and when both should coordinate. It proceeds in 3 steps:

  1. For each arm, segment each demonstration into phases, and identify the true task constraints between arms using VLM assistance + human review.

  2. Recombine the timing of both arms within each demonstration while preserving those task constraints.

  3. During policy training, apply an action loss mask to mask out the artificial waiting intervals created by temporal recombination.

CTR is validated on 4 simulated and 4 real robot bimanual tasks (in different experiments).

Critical strengths
  1. The paper tackles a well defined and practical failure mode in bimanual imitation learning. The proposed method addresses this at training time by recombining existing demonstrations, hence doesn't need additional data collection (but requires good VLM annotations/visual compositing).

  2. In the experiments, CTR achieves strong empirical performance on both simulated and real robot tasks.

Critical weaknesses
  1. Experimental coverage: the paper considers 3 policy families (diffusion policy, ACT and π0.5\pi_{0.5}) and 8 tasks in total, but all the main robustness tests only evaluate π0.5\pi_{0.5} on 1-2 tasks. Table 4 covers all models and tasks but only tests the normal starting state against a single baseline (Serial).

  2. CTR combined two components, temporal recombination and IdleMask. However the ablation removing IdleMask (Table 2) is only tested from the normal starting state (where even standard baselines already achieve high success), and hasn't been evaluated on the main robustness experiments.

Questions for authors

For the main points, please refer to the section on weaknesses.

Suggestions

It would be good to improve the paper's writing clarity.

It would also be good to improve the paper's clarity around how the visual compositing and loss masking work in edge cases.

5 comments
AuthorsOpenReview ↗

Response to Reviewer jyBi (1/2)

We thank the reviewer for the careful reading and the concrete suggestions. We respond in the order of the review.

W1. The robustness tests cover only π0.5\pi_{0.5} on one or two tasks; Table 4 compares only against Serial.

TL;DR: The progress-state tests use π0.5\pi_{0.5} because DP and ACT rarely succeed even from the nominal state; the action-level probe covers all three policy families, and we tested a second simulated task.

The progress-state evaluations cover π0.5\pi_{0.5} only, for a practical reason: the test needs a policy that often succeeds from the nominal state, otherwise all six starting states fail alike. On Put Object Cabinet, DP and ACT trained with CTR reach 4% and 24% from the nominal state (Table 3), against 62.0% for π0.5\pi_{0.5} (re-evaluated; 66% in the submitted Table 4).

Evidence across the three policy families comes instead from the action-level probe on Deliver Water (Fig. 4; App. E.4), which measures how much one arm's predicted actions change when only the other arm's state is varied. For ACT, DP and π0.5\pi_{0.5} alike, the Concurrent-trained policy is more sensitive to the other arm in the independent phases than in the synchronized phase, and the CTR-trained policy reverses this (selectivity gap, SserveS_{\rm serve} minus the mean of the two independent phases: +2.26 vs. −5.71 for ACT, +1.60 vs. −3.22 for DP, +1.11 vs. −1.22 for π0.5\pi_{0.5}, computed from Table 10).

What we ran during the discussion period: we repeated the progress-state evaluation on Put Object Cabinet three times on shared scenes and seeds (Table R2 below; they replace the simulated rows of Table 1 in the revision), and tested a second simulated task, Sort Blocks, with a different perturbation. There we open one gripper right after its first lift, as in a slipped grasp, and leave the other arm untouched; the untouched arm then finishes its own blocks in 91/100 scenes with CTR but 46/100 with Concurrent (details in the NBh4 thread, W0). We also tested the official co-trained π0.5\pi_{0.5} checkpoint of RoboTwin 2.0: on Put Object Cabinet it succeeds in 38/47 episodes when the left arm fetches the object, but in only 1/53 when the two arms' subtasks are swapped (NBh4 thread, W0).

The C00 column of Table 1 compares CTR with the other training sets at the standard start (sim: 62.0% vs. 29.3–58.7%; real: 95.3% vs. 78.0–88.7%). Table 3 (Table 4 in the submission) compares against Serial because Serial is the narrow-support control closest to how demonstrations are collected, one arm's subtask and then the other's, while sharing CTR's visual-synthesis pipeline, so the comparison isolates schedule support. Its simulated π0.5\pi_{0.5} rows were re-evaluated with 3 × 100 episodes (3 × 50 for Put Object Cabinet) on shared scenes and seeds (General Response, item 2). Sec. 5 (Table 1) states that the progress-state evaluations use π0.5\pi_{0.5} on Put Object Cabinet (sim) and Sort Letters (real).

AuthorsOpenReview ↗

Response to Reviewer jyBi (2/2)

W2. The IdleMask ablation is only from the normal starting state.

TL;DR: IdleMask does not drive the gains, and we have removed it from the method.

We ran the IdleMask ablation on all four simulated tasks at the standard start (a) and repeated the progress-state evaluation (Table 1) with CTR trained without IdleMask (b).

(a) IdleMask ablation at the standard start, four tasks (3 × 100 episodes; only 3 × 50 for Put Object Cabinet; also in General Response, item 1). The with-minus-without differences are +3.7, −10.0, +4.0 and +2.0 points, none significant after Holm correction, mean −0.1. The 14-point gap on Scan Object in Table 2 of the submission came from single 50-trial evaluations.

(b) Progress-state evaluation (Table 1) with CTR trained without IdleMask.

Table R2. Put Object Cabinet (sim), π0.5\pi_{0.5}, mean SR over 3 × 50 episodes per starting state; same scenes and policy seeds for all rows. C00: nominal start; Cmid: both arms at mid-progress; L-lead, R-lead: one arm ahead of the other; CL, CR: one arm at its ready or completion state, the other at the start.

DataC00CmidL-leadR-leadCLCRSix-state meanWorst
Serial34.092.070.035.360.042.755.734.0
Mixed58.798.762.750.047.382.066.647.3
CTR w/o IdleMask62.099.372.066.751.396.774.751.3

The conclusion of Table 1 holds without IdleMask: CTR is 8.1 points above Mixed on the six-state mean and 19.0 above Serial. Since Mixed is itself a restricted form of CTR, built by our own recombination pipeline, the revision replaces the claim that CTR raises worst-state success from 56% to 76% (abstract, introduction) with a comparison to the original, unretimed L→RL \rightarrow R data (worst state 0% → 51.3% in simulation and 38.7% → 74.7% on the real robot; General Response, item 2). Since IdleMask has no significant effect at the standard start (a) and the Table 1 gains hold without it (b), we have removed it from the method: Sec. 4 presents CTR as temporal recombination trained with the standard action loss, and the IdleMask experiments appear only in the appendix (App. F.6–F.7).

Suggestion 1. Writing clarity.

TL;DR: We have revised the presentation: Sec. 3 uses plain English in arguments with a clear storyline, Sec. 2 highlights how CTR differs from prior work, Sec. 4 is more concise and explicit, and terms are consistent throughout the paper.

  • Sec. 2. Its paragraph on DexMimicGen and TriManPolicy is rewritten to state each method's goal and single schedule per demonstration, and why CTR varies it (see General Response, item 4).
  • Sec. 3. Its storyline is revised to flow smoothly, and the core arguments are made in plain English instead of equations: a demonstration records one schedule, so the policy can time one arm by the other's progress (spurious cross-arm coupling) and fail when the arms drift apart (new Sec. 3.2). An arm should depend on the other only under task-required constraints, which hold in every admissible schedule (Sec. 3.3). More such demonstrations cannot tell these constraints from incidental timing, so the schedules must be varied (Sec. 3.4).
  • Sec. 4. It is more concise and explicit.

Suggestion 2. Edge cases of visual compositing and loss masking.

TL;DR: App. F.5, F.6 and G describe them. The main cases are below.

  • Visual compositing. A recombined frame shows the left arm at one recorded moment and the right arm at another, a pair that may never have been recorded together. Only the main camera image is rebuilt: each arm is cut out of its own source frame and pasted onto the scene background. An object moves with the arm that carries it from pickup to release, and its original spot is filled in so it never appears twice. Where the two arms overlap in the image, depth decides which is in front when available. Every new pairing is checked for collisions with the robot model and task-specific gates, and a schedule is rejected when clearance cannot be certified. Of 363 real sources, 11 (3.0%) failed these checks (General Response, item 3) and the reviewer excluded four (1.1%) (new App. F.2).
  • Loss masking. IdleMask is no longer part of the method (General Response, item 1). In the appendix experiments it masks only an arm's own action targets during waits that recombination inserted. The other arm's targets, required holding, and holding after completion are never masked.

We thank the reviewer for the writing suggestions, which made the paper easier to read. We hope the added analyses address the reviewer's concerns.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer jyBi,

Thank you again for your careful review and suggestions. We have posted our responses above and uploaded the revised PDF, with changes in blue. In brief:

  • Experimental coverage (W1). The progress-state tests use π0.5\pi_{0.5} because DP and ACT rarely succeed even from the nominal state, and the action-level probe covers all three policy families. We also tested a second simulated task, Sort Blocks with a slipped grasp: the untouched arm finishes its own blocks in 91/100 scenes with CTR and 46/100 with Concurrent.
  • IdleMask (W2). With CTR trained without IdleMask, the gains of Table 1 hold (8.1 points over Mixed on the six-state mean), and IdleMask has no significant effect at the standard start on any of the four simulated tasks. We have therefore removed it from the method (Sec. 4, App. F.6–F.7).
  • Writing clarity and edge cases. Sec. 2–4 are revised for clarity, and App. F.5, F.6 and G describe the edge cases of visual compositing and loss masking.

Could you let us know whether these responses address your concerns? If anything remains unclear, we would be glad to provide further details during the remaining discussion period.

Best regards,

The Authors

Reviewer_16PhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies spurious cross-arm coupling in bimanual imitation learning, where policies learn incidental temporal correlations between arm motions rather than task-required dependencies. The authors identify narrow demonstration schedule support as a source of this problem and formalize selective cross-arm dependence as the desired behavior.

They propose Constraint-preserving Temporal Recombination (CTR), a training-time augmentation method that segments demonstrations into arm-local motions, identifies task-required temporal constraints, and recombines the recorded motions under alternative admissible schedules. The method preserves arm-local sensorimotor content, synthesizes joint observations, and masks artificially introduced waiting actions during training.

The approach is evaluated with Diffusion Policy, ACT, and π0.5 across four simulated and four real-robot tasks. The experiments demonstrate improvements in robustness to asymmetric arm progress, reduced delay propagation between independent arms, and preservation of task-required coordination.

Critical strengths

Well-motivated problem formulation and an architecture-agnostic solution. The distinction between incidental demonstration timing and task-required coordination is important. The identifiability argument provides a useful explanation for why imitation learning can develop spurious cross-arm dependencies. Improving training distributions rather than modifying policy architectures is practically appealing, and evaluation across DP, ACT, and π0.5 strengthens the contribution.

Convincing intervention-based evaluation. The asymmetric progress-state experiments, externally imposed pauses, and task-constraint interventions provide meaningful evidence beyond standard task-success metrics. In particular, CTR raises worst-state success on the real Sort Letters task from 56% (the strongest non-CTR baseline, Mixed) to 76%, and reduces the delay transferred by a 4 s opposite-arm pause from 2.56 s (Concurrent) and 5.12 s (L→R) to 0.25 s, which is compelling.

Critical weaknesses

Novelty relative to prior work and attribution of the gains. Constraint-aware temporal recombination of per-arm segments overlaps with DexMimicGen (Jiang et al., 2025b), which already generates asynchronous per-arm executions under synchronization and ordering constraints. The distinction CTR draws, sampling diverse admissible schedules from recorded data rather than generating new demonstrations or selecting one efficient schedule, is reasonable, and the Serial, Concurrent and Mixed baselines, built with the same constraint-preserving pipeline, show that broader schedule support helps. However, the paper does not test whether the extracted constraints themselves matter, for example by comparing against retiming without them. In addition, Table 1 confounds schedule breadth with IdleMask, which only CTR uses (Sec. 5.1); the IdleMask ablation (Table 2) reports default-condition success on two tasks only, not the progress-state stress tests where CTR's largest gains appear.

Dependence on task-specific constraint extraction and limited statistical analysis. The approach relies heavily on the correctness of automatically generated segmentation and constraint-extraction pipelines. The paper lacks quantitative evaluation of annotation accuracy, human review requirements, and robustness to incorrect or missing constraints (acknowledged in Sec. 6 but not measured). These issues are central to the method's validity and scalability. In addition, all success rates come from 50 trials, with no reported training seeds or confidence intervals outside Fig. 5; in Table 4, 7 of the 19 reported improvements are +2 points, i.e., a single trial. The held-out progress-state evaluation is limited to one region and one task. Without seed variability, it is hard to tell which of the smaller gains are real.

Questions for authors
  1. In Table 1, CTR differs from Mixed in both schedule breadth and IdleMask. Can the authors report the progress-state results for CTR without IdleMask, and for retiming without the extracted constraints, to isolate the contribution of constraint-preserving schedule diversity?

  2. How accurate are the task-specific segment boundaries and cross-arm constraints produced by the generated processing pipeline? What level of human intervention is required, what fraction of candidate schedules are rejected by validity checks, and how sensitive is downstream performance to incorrect or missing constraints?

  3. Can the authors report variability across independently trained seeds for representative tasks, particularly where the gains are small? Additionally, does held-out progress-state generalization persist across multiple excluded regions and coordination-intensive tasks, rather than the single region evaluated on Sort Letters?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 16Ph (1/3)

We thank the reviewer for the precise and constructive review. We answer each point below.

W1 / Q1. Isolating schedule breadth from IdleMask and from the extracted constraints.

We thank the reviewer for this sharp observation: the submitted experiments did not separate schedule breadth from IdleMask clearly enough, and did not test whether the extracted constraints matter. Following the reviewer's suggestion, we ran both experiments.

TL;DR: The gains of the six-progress-state evaluation (Table 1) hold with CTR trained without IdleMask, and IdleMask has no significant effect at the standard start on the four simulated tasks, so we have removed it from the method.

(a) CTR without IdleMask on the progress states. As described in General Response, item 2, we repeated the Put Object Cabinet progress-state evaluation (Table 1) with a CTR model trained without IdleMask. Neither Mixed nor this CTR model uses IdleMask, so the difference between them is schedule breadth alone.

DataC00CmidL-leadR-leadCLCRSix-state meanWorst state
Serial34.092.070.035.360.042.755.734.0
Mixed58.798.762.750.047.382.066.647.3
CTR w/o IdleMask62.099.372.066.751.396.774.751.3

CTR exceeds Mixed by 8.1 points on the six-state mean and Serial by 19.0. At the standard start on all four simulated tasks, IdleMask changes CTR's success by −0.1 points on average, so we have removed it from the method (General Response, item 1).

(b) Retiming without the extracted constraints. The reviewer is right that the submitted comparisons do not measure the value of the constraints: Serial, Concurrent, Mixed and CTR are all built by the same constraint-preserving pipeline, so they isolate schedule breadth. We therefore trained π0.5\pi_{0.5} on Put Object Cabinet (sim) data retimed from the same 50 source demonstrations as CTR, with one difference: the extracted constraints were removed. 82% (41/50) of the recombined demonstrations then violate the task's constraint.

Training data (Put Object Cabinet, sim, retimed from the same 50 source demonstrations)Constraint-violating demonstrations (object inserted before the drawer is open)Success (3 × 50 episodes, shared scenes and seeds)
CTR0/50 (0%)62.0% (93/150)
CTR without the extracted constraints41/50 (82%)14.0% (21/150)

The constraints are therefore necessary: without them, recombination teaches the policy schedules that violate the task.

W1. Overlap with DexMimicGen.

We thank the reviewer for the careful comparison with DexMimicGen and for finding our distinction reasonable. Our discussion of it in the submission was not detailed enough, and the revision expands it.

TL;DR: Unlike prior work, we study how the narrow schedule coverage of demonstrations affects policy robustness. Both DexMimicGen and TriManPolicy give each demonstration one schedule, and our experiments show that one schedule, however efficient, leaves the policy coupled.

Instead of aiming at more or faster data, CTR targets a different failure mode: when demonstrations cover few schedules, the opposite arm's progress predicts each arm's actions even where the task does not require it. The full comparison with DexMimicGen and TriManPolicy, including our results against a single efficient schedule, is in General Response, item 4. The revised Sec. 2 discusses both methods in its last paragraph.

AuthorsOpenReview ↗

Response to Reviewer 16Ph (2/3)

W2 / Q2. Accuracy of boundaries and constraints, human intervention, rejected schedules, sensitivity.

We thank the reviewer for pointing out that our evaluation of the pipeline was not quantitative enough; this suggestion led to the new measurements of accuracy and human effort below.

TL;DR: The constraints were correct on every task where they can be checked, constraint-critical boundaries are off by 0.04–0.10 s on average, and 1.1% of real demonstrations needed a human correction; sensitivity to partially wrong constraints was not measured.

We measured accuracy and human intervention; the full numbers are in item 3 of the General Response and the new App. F.2. On all four simulated tasks and the two real tasks whose constraints are ordering relations, the pipelines' cross-arm constraints were correct: precision and recall 1.00 against the expert planner in simulation, and all 60 ordering relations asserted on the real robot hold in human labels. Boundary offsets are 0.04–0.09 s in simulation and 0.09–0.33 s on the real robot, mostly a consistent event-definition offset (onsets taken from the first joint motion precede visible motion by about 0.2 s; releases are marked on confirmed gripper reopening, 0.15–0.45 s after the visible drop). At constraint-critical boundaries the offset is 0.06–0.10 s. Of the 620 labeled real boundaries, six differ by more than 1 s, none of them constraint-critical, and where a protected window is narrower than the human-labeled one the shortfall is at most 0.17 s.

Human intervention. After a pipeline is frozen, a person acts only on flagged sources and gate failures, and each action is logged. Of the 363 real source demonstrations, 4 (1.1%) received a correction, 4 (1.1%) were excluded by the reviewer, and 11 (3.0%) were rejected by the validity gates: 9 because the cabinet moved during the demonstration, 1 for which no schedule could be certified at 5 cm clearance, and 1 that failed the rendered-appearance audit. Our logs count rejections per source demonstration, not per candidate schedule.

Sensitivity to wrong constraints. The run without constraints (Q1(b)) covers the extreme case of missing constraints. Sensitivity to partially wrong constraints was not measured and is stated as a limitation in Sec. 6.

AuthorsOpenReview ↗

Response to Reviewer 16Ph (3/3)

Response to Reviewer 16Ph (3/3)

W2 / Q3. Evaluation variability, small gains, and held-out generalization.

We are grateful to the reviewer for raising the statistical analysis and the scope of the held-out test. Our original evaluation was not thorough enough on either point, and the reviewer's questions led us to re-run the evaluations and to extend the held-out experiment.

TL;DR: The reviewer is right: at the standard start CTR improves one task substantially, is indistinguishable from Serial on two, and is lower on Sort Blocks. CTR targets timing robustness, and our main result, CTR's gain over the six progress states, still holds. Held-out generalization holds in four new excluded regions on Put Object Cabinet.

As described in General Response, item 2, we re-evaluated Table 3 (Table 4 in the submission) three times on shared scenes and seeds (simulation, π0.5\pi_{0.5}, CTR vs. Serial, 100 scenes; Put Object Cabinet on the 50 scenes of the progress-state test; CTR trained without IdleMask, as in the revised method):

TaskSerialCTRΔ (CTR − Serial) [95% CI]pp (Holm)
Sort Blocks95.089.3−5.7 [−10.0, −1.7]0.038
Place Dual Shoes39.346.3+7.0 [−1.3, +15.3]0.24
Scan Object63.361.7−1.7 [−8.7, +5.3]0.71
Put Object Cabinet34.062.0+28.0 [+17.3, +38.7]<10−4<10^{-4}

The seven +2-point entries the reviewer counts are DP, ACT and real-robot cells, which remain single 50-trial evaluations; a one-trial difference there is not evidence of a gain.

CTR targets timing robustness, not standard-start success: its goal is to make policies robust to timing variation without giving up performance from the standard start (General Response, item 2). The revision therefore removes the claim that CTR achieves higher standard-start success “in 19 of 24 model–task comparisons” (introduction, Sec. 5) and reports these results per task (Table 3).

Training seeds. Because of limited compute, we could not complete retraining with multiple training seeds during the discussion period; the intervals above reflect evaluation variability of single training runs.

Held-out progress states. We repeated the held-out test on Put Object Cabinet (π0.5\pi_{0.5}, simulation) with four excluded regions instead of one. Here each arm's progress is measured in time, which differs from the semantic stages used in Table 1. Written as left × right progress, the regions are [0.60,0.70]×[0.30,0.40][0.60,0.70]\times[0.30,0.40], [0.30,0.40]×[0.60,0.70][0.30,0.40]\times[0.60,0.70], [0.75,0.85]×[0.15,0.25][0.75,0.85]\times[0.15,0.25] and [0.15,0.25]×[0.75,0.85][0.15,0.25]\times[0.75,0.85]. From the same 50 source demonstrations as the CTR dataset, we retimed two new CTR datasets, hole-S without the first two regions and hole-L without the last two, and trained CTR-hole-S and CTR-hole-L on them with the same recipe as CTR-full. Every model starts from the center of each region, with 3 × 50 trials per region, paired across models on the same scenes and seeds.

Excluded region (left × right)CTR-hole-SCTR-hole-LCTR-fullMixed
[0.60,0.70]×[0.30,0.40][0.60,0.70]\times[0.30,0.40]90.7–84.735.3
[0.30,0.40]×[0.60,0.70][0.30,0.40]\times[0.60,0.70]92.7–96.082.7
[0.75,0.85]×[0.15,0.25][0.75,0.85]\times[0.15,0.25]–64.758.750.0
[0.15,0.25]×[0.75,0.85][0.15,0.25]\times[0.75,0.85]–94.097.378.7

In each excluded region, the CTR-hole model trained without it is not significantly worse than CTR-full and is significantly better than Mixed, whose data do not cover these regions either (p≤0.006p \le 0.006).

CTR therefore learns each arm's own progress rather than memorizing specific combinations of the two: removing a region of progress combinations from training leaves success from that region unchanged, while Mixed, which never covers these regions, is clearly worse in all four.

We sincerely thank the reviewer for the many constructive suggestions. Acting on them, we ran several new experiments, which let us state our position relative to prior work more clearly and made the paper's contributions sharper and more precise. We look forward to any further discussion.

Reviewer_NBh4OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method to decouple the behavior of bi-manual robots when they are performing independent parts of a task. They find that public datasets for bi-manual demonstrations very rarely demonstrate multiple ways of performing the task, and address this by using a semi-automated system to separate the two arm’s schedules. They then demonstrate that this method increases performance when used to train popular control policies, compared to several baseline dataset variants, across both real and simulated tasks.

Critical strengths

The amount of information about experiments, availability of code, and general efforts at reproducibility made in this paper are good. We were able to find most information we needed about the author’s experimental setup either in the main paper or in the appendix. Additionally, the quantity of both sim and real tasks for SR, and testing across 3 different control policies gives confidence in both the reliability and generality of this method.

Critical weaknesses

Throughout this paper, we waited for a clear justification for why spurious cross-arm coupling is problematic. The answers are alluded to in many locations (The dataset audit in the introduction, Table 8's timing data, Table 4's SR's), but the costs of having a control policy exhibit spurious cross-arm coupling are not stated clearly. Table 1 comes closest to justifying CTR, but it's not made clear how often these initial states are met in normal use. A clear statement with a simple table to back it up would make this paper much more convincing.

Other issues include:

  1. TriManPolicy is cited but has strong similarities which aren't addressed. It should be compared to more specifically. We think this can help motivate the paper better, since that paper is about enabling the use of user demonstrations for tri-manual arms, whereas this paper is more generally about contending with spurious cross-arm coupling.
  2. The definition of h_t is also unclear, and is unused in equation 4 where it seems applicable. Similarly, x_i and p_i are defined semantically but used in equations. This section is difficult to parse, and would be strong if it remained a purely semantic argument or was defined more rigorously.
  3. Only 50 trials were used, even for simulation, and CTR results were only marginally different from baselines. Without Std. Dev. reported, statistical significance is uncertain.
Questions for authors
  1. Why does CTR increase SR so much from the nominal starting position? (In Put Object Cabinet)
  2. What innovations does your method provide over TriManPolicy, and how does it improve over the other baselines you list in your related work?
  3. How does the video-processing pipeline get generated, and what is the nature of the human review? The details of this are unclear. How reliable is this process compared to human labeling?
5 comments
AuthorsOpenReview ↗

Response to Reviewer NBh4 (1/3)

We thank the reviewer for the careful reading and for recognizing our reproducibility effort and the breadth of the evaluation. We respond to each point below.

W0. Why is spurious coupling a problem, and how often do these states occur in normal use?

TL;DR: Spurious cross-arm coupling turns one arm's trouble into the other arm's failure, and in normal use one arm is often slower, retries a grasp or is briefly blocked. CTR removes most of this cost.

Why is spurious coupling a problem? We agree with the reviewer: the paper did not organize the evidence for why spurious coupling is a problem well enough. Once the arms leave the demonstrated schedule, policies trained on narrow schedules stall or act wrongly, even though each arm on its own is in a state it has seen in training. The revision says so in a new paragraph of Sec. 3.2. The table below, added to the revision as Table 16 (App. H), lists the three costs and the experiment in the paper that shows each:

Cost of spurious couplingFailure modeExampleExperimental evidence
Progress-state brittlenessThe policy fails when the two arms are at combinations of progress not seen in the demonstrated schedule, even though each arm alone is in a familiar state.Put Object Cabinet. The right arm regrasps and falls behind the left arm. Both arms then stall.Sec. 5.2, From action dependence to progress-state brittleness (Table 1, Table 11). SR of rollouts that reach this state from the standard initial state: 4% → 68% (Serial → CTR)
Task-constraint violationThe policy reads task progress off the other arm's state rather than the task itself, so it proceeds before a required condition is met.Put Object Cabinet (real). The drawer is closed just before the left arm inserts the object. The left arm still moves to insert it.Sec. 5.3, Behavior under task-required constraints (Table 2). Violation rate: 98% → 0% (L→R → CTR)
Cross-arm delay propagationA delay of one arm stalls the other arm, even when their subtasks are independent.Sort Letters. One arm is held still for 4 s. The other arm, sorting an unrelated letter, waits too.Sec. 5.4, Unseen execution delays (Fig. 5). Delay of the other arm: 5.12 s → 0.25 s (L→R → CTR)

Throughout this response, CTR is compared with policies trained on four baseline datasets, all built from the same recordings:

  • L→RL \rightarrow R: the original demonstrations (the left arm's subtask first, then the right arm's);
  • Serial: one arm finishes its subtask before the other starts, in both orders;
  • Concurrent: both arms start together;
  • Mixed: the union of the three above.

How common is spurious coupling?

We sincerely apologize for the confusion: our description was not precise enough, and we have rephrased it in the revision. The reviewer's question also led us to new measurements of how often these states arise, which make the motivation of the paper clearer.

In public datasets and released models. Narrow schedules are the norm in public data: in our survey of four public bimanual datasets (716 tasks; Sec. 1 and appendix), 96.6% of the reschedulable tasks are demonstrated under a single schedule, including every such task in RoboTwin 2.0. The official RoboTwin 2.0 demonstrations of our four simulated tasks are an example: Put Object Cabinet always follows one order (the object arm grasps first, then the other arm opens the drawer), Sort Blocks is serial, and Scan Object and Place Dual Shoes are concurrent. To show that the problem is not specific to our training setup, we tested the co-trained π0.5\pi_{0.5} checkpoint released by RoboTwin 2.0, trained by its authors on these official demonstrations. On Put Object Cabinet its success collapses once the right arm leads instead of the left:

Official RoboTwin 2.0 π0.5\pi_{0.5}, Put Object Cabinet (100 episodes)EpisodesSuccesses
Left arm leads4738 (81%)
Right arm leads531 (2%)

How often do these states occur in normal use? The progress states of Table 1 are not special initial states. They are intermediate states that a rollout passes through after starting from the standard initial state; Table 1 starts the policy directly in each of them only to test every state under controlled conditions. The revision states this in Sec. 5.2 and App. B to remove the ambiguity. Ordinary rollouts reach them on their own, as two new analyses, (a) and (b) in our next comment, show.

AuthorsOpenReview ↗

Response to Reviewer NBh4 (2/3)

W0 (continued).

(a) Put Object Cabinet (sim): states reached from the nominal start. We ran each model 3 × 50 times from the ordinary nominal start (C00) and recorded which Table 1 states each rollout passes through. Serial reaches every off-nominal state:

States that the policies pass through on their own after starting from the nominal start (Put Object Cabinet, sim, 150 rollouts per model):

State (Table 1)What it isHow Serial reaches itSerial: rollouts that reach itSerial: successes among themCTR: rollouts that reach itCTR: successes among them
L-leadleft arm aheadright-arm regrasp24/1501/2469/15047/69
CLleft arm done, right arm at startright-arm regrasp80/15045/800/150–
Cmidboth arms halfwayright arm grasps first32/1505/3245/15026/45
R-leadright arm aheadright arm grasps first31/1505/3127/15017/27
CRright arm ready, left arm at startright arm grasps first31/1505/313/1500/3

A right-arm regrasp (the gripper reopening and closing at the object) occurs in 40 of 150 Serial rollouts and a right-first grasp in 33 (Mixed: 13 and 2; CTR: 12 and 25). Most Serial rollouts that reach Cmid, R-lead or CR reach all three (30 of 33), since a right-first grasp carries them through these states in turn. CTR passing through Cmid, L-lead or R-lead succeeds in 58–68%, close to its overall rate. This is consistent with CTR's nominal-start success of 62.0% against 34.0% for Serial (Q1 below).

(b) Sort Blocks (sim): when one arm slips, does the other arm keep working? We open one gripper right after its first lift, as in a slipped grasp, and leave the other arm untouched (2 × 50 paired scenes). Unperturbed, each arm finishes its own two blocks in at least 48 of 50 scenes for both Concurrent and CTR. With the slip it still finishes in 91/100 with CTR but only 46/100 with Concurrent; task success is 34% vs. 6%. Without any intervention the Concurrent policy keeps the arms in lockstep (first lifts 2 s or more apart in 2/100 rollouts, vs. 45/100 for CTR); when its arms do drift 5 s or more apart at the second block, it succeeds in only 5/15 of those rollouts, against 28/31 for CTR. The cost appears when one arm has to redo a step, which is exactly when the arms reach combinations the demonstrated schedule never showed.

Supplementary videos: why spurious coupling is a problem, and how these states arise from the standard initial state. The real-robot videos in the supplementary material (CTR_Real_Robot_Experiments) show both:

  • Slide 3, Put Object Cabinet. In the original demonstrations the drawer is always open before the object is placed, so the policy trained on them does not learn that an open drawer is a prerequisite: when we hold the drawer closed, it tries to release the object in front of the closed drawer. CTR, having seen many admissible schedules, waits while the drawer is closed.
  • Slide 7, Sort Letters from CR. The right arm's subtask is already done, yet the Concurrent policy keeps moving the right arm along with the left; with CTR the finished arm stays still.
  • Slide 9, Sort Letters. Starting from the standard initial state, a 3 s pause of the left arm, like a slow grasp, puts both Concurrent and CTR into R-lead: the arms' progress drifts apart during an ordinary rollout.

W1 / Q2. Relation to TriManPolicy and to the other methods in Sec. 2.

We thank the reviewer for pointing out that our discussion of related work was not thorough enough. This question led to a new comparison against a single efficient schedule and to a clearer positioning of CTR relative to prior work.

TL;DR: TriManPolicy retimes each demonstration to one fastest schedule, which is still narrow support and leaves the policy coupled; CTR trains on many admissible schedules and changes only the data, not the architecture.

The full comparison with TriManPolicy and DexMimicGen, including our results against a single efficient schedule, is in General Response, item 4. The revised Sec. 2 discusses both methods in its last paragraph.

The other methods in Sec. 2 structure cross-arm dependence inside the model (InterACT, BUDS, the Decoupled Interaction Framework, SkillVLA) or recombine state partitions for counterfactual augmentation (CoDA, CAIAC, RoCoDA). CTR changes only the timing structure of the training data and leaves the architecture untouched, which is why the same intervention applies to DP, ACT and π0.5\pi_{0.5}; our experiments hold the architecture fixed and vary only the schedule support. The first paragraph of Sec. 2 states this position.

AuthorsOpenReview ↗

Response to Reviewer NBh4 (3/3)

W2. Notation in Section 3.

TL;DR: Sec. 3 is now a plain-language argument with one equation left.

We take the first option the reviewer suggests. Segment, schedule, local state, progress and shared conditions are now defined in words, and so are the invariance condition and the identifiability argument. The only equation left is the training objective, with hth_t defined explicitly as the last KK observations plus the instruction.

W3. Only 50 trials, no standard deviations, marginal differences.

We are grateful to the reviewer for pressing on statistical reliability; our original analysis was not thorough enough, and this point led us to re-run the evaluations and to state our claims more precisely.

TL;DR: We agree. Under repeated evaluation our main result, CTR's gain on the six-state mean, remains significant; we slightly revise one minor claim, higher standard-start success "in 19 of 24 model–task comparisons."

As detailed in General Response, item 2, which also gives the re-evaluated Tables 1 and 3, we re-ran every simulated π0.5\pi_{0.5} evaluation three times, with all models tested on the same scenes, and now report 95% confidence intervals and paired significance tests. On the six-state mean, CTR exceeds Mixed by 8.1 points and Serial by 19.0. Since Mixed is itself a restricted form of CTR, built by our own recombination pipeline, the revision states the worst-state result against the original L→RL \rightarrow R data (0% → 51.3% in simulation, 38.7% → 74.7% on the real robot), and reports standard-start results per task in place of the "19 of 24" claim (General Response, item 2).

Q1. Why does CTR raise nominal-start success so much on Put Object Cabinet?

We thank the reviewer for this question: answering it led to a new analysis of the states that rollouts reach from the standard start, and it sharpened how we position the paper.

TL;DR: A nominal start does not keep a rollout on the demonstrated schedule; Serial tends to stall once it leaves that schedule, whereas CTR has seen these arm combinations in training.

We analyzed the 3 × 50 nominal-start rollouts of each model:

SerialMixedCTR
Nominal-start success34.0%58.7%62.0%
Failed rollouts that end in a stall (action budget used up; none terminated by a drop or collision)allallall
Neither arm completes its subtask21%5%5%
Prerequisite completed, second subtask not45%37%33%
Success in rollouts that pass through an off-nominal state4–16%–58–68%
Nothing completed when the test starts at R-lead / CR63% / 55%–29% / 1%

The off-nominal states of Table 1 arise on their own: with arm progress estimated from the executed actions (a regrasp is a gripper reopening and closing at the object), Serial passes through each of the five off-nominal states in 16–53% of its nominal-start rollouts. (State definitions and how Serial reaches each state: (1/3), W0.) The success rates in the table are for L-lead, R-lead, Cmid and CR (Serial; 56% for CL) and for Cmid, L-lead and R-lead (CTR, close to its overall rate).

So Serial's nominal-start failures look like its asymmetric-start failures, reached by drift: when one arm takes longer, the policy enters a combination of arm progress its data never contained, and it waits. CTR has seen these combinations during training. The Sort Blocks test in W0 probes the same mechanism with an external perturbation. The revision adds this analysis as App. E.8 (Tables 11–12).

Q3. How is the pipeline generated, what does the human review do, and how reliable is it compared with human labeling?

We thank the reviewer for noting that the paper did not discuss this in enough depth; the suggestion is highly constructive.

TL;DR: The pipeline is accurate (0.06–0.10 s offset at constraint-critical real-robot boundaries; all 60 asserted ordering relations hold), and 1.1% of real demonstrations needed a human correction.

Item 3 of the General Response describes the procedure and the measurements (new App. F.2). In brief: a coding agent writes a program per task, a person reviews rendered sample videos for two to five rounds, and the frozen program then processes all demonstrations; a person acts only on flagged sources. Against 620 hand-labeled real-robot boundaries, the mean offset is 0.09–0.33 s, and 0.06–0.10 s at constraint-critical boundaries; all 60 asserted ordering relations hold in the human labels; of the six boundaries that disagree by more than 1 s, five are pipeline errors and one is ambiguous, none a human labeling error, and none constraint-critical. Of the 363 real source demonstrations, 1.1% needed a human correction. Hand-labeling the same boundaries takes a median of 26–64 s per demonstration, so the pipeline replaces per-demonstration labeling with a few review rounds per task.

We hope the clarifications and the new results address the reviewer's concerns.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response to All Reviewers (1/4)

We sincerely thank all reviewers for their thoughtful and constructive reviews. Reviewer 16Ph finds the problem formulation "well-motivated," calls the distinction between incidental demonstration timing and task-required coordination "important," finds improving training distributions instead of policy architectures "practically appealing," and judges the distinction CTR draws from prior work to be "reasonable." Reviewer jyBi notes that the paper tackles a "well defined and practical failure mode" that CTR addresses without "additional data collection." Reviewer 16Ph also finds the intervention-based evaluation "convincing," and Reviewer NBh4 finds our "efforts at reproducibility … good" and appreciates that the breadth of tasks and policies "gives confidence in both the reliability and generality of this method."

Below we address the concerns shared by several reviewers. During the discussion period we separated the contributions of schedule breadth, IdleMask and the extracted constraints: we re-ran the six-progress-state evaluation (Table 1) with CTR trained without IdleMask, ablated IdleMask from the standard initial state on all four simulated tasks, and trained a policy on data recombined without the extracted constraints. The gains come from schedule breadth, the constraints are necessary, and IdleMask is not, so we removed it from the method (item 1). We also re-evaluated the simulated π0.5\pi_{0.5} experiments three times on shared scenes and seeds (item 2), measured the accuracy of CTR's recombination pipelines (item 3), and tested the official co-trained RoboTwin 2.0 π0.5\pi_{0.5} checkpoint (NBh4 thread). We also repeated the held-out progress-state experiment on Put Object Cabinet with four excluded regions (16Ph thread).

1. Isolating schedule breadth from IdleMask and from the extracted constraints (16Ph, jyBi)

TL;DR: The gains come from schedule breadth, and the extracted constraints are necessary: without them, success drops from 62.0% to 14.0%. The gains of Table 1 hold with CTR trained without IdleMask, and IdleMask has no significant effect at the standard start, so we have removed it from the method. IdleMask itself is a minor design and removing it only makes our main ideas stand out more clearly.

Table 1 of the paper confounded schedule breadth with IdleMask and did not test the extracted constraints. We therefore repeated it with a CTR model trained without IdleMask (a), ran the IdleMask ablation on all four simulated tasks (b), and trained a policy on data recombined without the extracted constraints (c).

(a) Table 1 re-evaluated with CTR trained without IdleMask. Put Object Cabinet (sim), π0.5\pi_{0.5}, six progress states, 3 × 50 episodes per state on shared scenes and seeds. None of these models uses IdleMask.

Training dataMean SR over six statesCTR minus this model
Serial55.7+19.0
Mixed66.6+8.1
CTR w/o IdleMask74.7—

These means are close to Table 1 of the submission (Serial 54.7, Mixed 66.0, CTR 75.3), so the conclusion of that table holds without IdleMask. Per-state results are in the 16Ph and jyBi threads.

(b) IdleMask ablation at the standard start. Four simulated tasks, π0.5\pi_{0.5}, 3 × 100 episodes per cell (3 × 50 for Put Object Cabinet), on shared scenes and seeds; the two models of each pair share the seed and training recipe.

Taskw/o IdleMaskw/ IdleMaskΔ (w/ − w/o)
Sort Blocks89.393.0+3.7
Place Dual Shoes46.336.3−10.0
Scan Object61.765.7+4.0
Put Object Cabinet62.064.0+2.0
Mean−0.1

No task differs significantly after Holm correction over the four tasks, and the 14-point gap on Scan Object in Table 2 of the submission came from single 50-trial evaluations. Taken together, (a) and (b) show that the gains of Table 1 do not depend on IdleMask and that IdleMask has no significant effect at the standard start. We have therefore removed IdleMask from the method entirely. CTR is temporal recombination trained with the standard action loss now. Note that our main arguments will still hold without IdleMask, since IdleMask is not among our listed contributions and is not a critical design in CTR.

AuthorsOpenReview ↗

General Response to All Reviewers (2/4)

1. (continued)

(c) Recombination without the extracted constraints. We trained π0.5\pi_{0.5} on Put Object Cabinet (sim) data retimed from the same 50 source demonstrations as CTR, with one difference: the extracted constraints were removed. 82% (41/50) of the recombined demonstrations then violate the task's constraint.

Training data (Put Object Cabinet, sim, retimed from the same 50 source demonstrations)Constraint-violating demonstrations (object inserted before the drawer is open)Success (3 × 50 episodes, shared scenes and seeds)
CTR0/50 (0%)62.0% (93/150)
CTR without the extracted constraints41/50 (82%)14.0% (21/150)

The constraints are therefore necessary: without them, recombination teaches the policy schedules that violate the task. For consistency with the paper, this comparison uses the same evaluation protocol (scenes and policy seeds) and the same CTR checkpoint as Table 1.

2. Statistical reliability (NBh4, 16Ph)

TL;DR: Under repeated evaluation our main result, CTR's gain on the six-state mean, remains significant. We slightly revise one minor claim, higher standard-start success "in 19 of 24 model–task comparisons."

We agree that a single evaluation of 50 trials cannot resolve differences of a few points. We therefore re-evaluated π0.5\pi_{0.5} three times per cell: each progress state of Table 1 (simulation and real robot) on 50 scenes, and the simulated standard-start results of Table 3 (Table 4 in the submission) on 100 scenes per task (50 for Put Object Cabinet). In simulation, the runs differ only in the policy's sampling seed, and all models share scenes and seeds, so they are compared scene by scene. 95% CIs resample scenes with all three runs; pp-values come from paired permutation tests with Holm correction.

Table 1 of the paper, re-evaluated (success rate, %, 3 × 50 episodes per state):

DataC00CmidL-leadR-leadCLCRSix-state meanWorst state
Put Object Cabinet (sim)
L→RL \rightarrow R43.360.078.00.770.70.042.1 [38.3, 45.8]0.0
Serial34.092.070.035.360.042.755.7 [50.8, 60.4]34.0
Concurrent29.398.70.00.70.00.021.4 [20.2, 22.8]0.0
Mixed58.798.762.750.047.382.066.6 [62.3, 70.9]47.3
CTR62.099.372.066.751.396.774.7 [71.0, 78.2]51.3
Sort Letters (real)
L→RL \rightarrow R86.778.038.774.796.092.777.8 [75.3, 80.1]38.7
Serial86.70.723.356.792.091.358.4 [56.2, 60.7]0.7
Concurrent88.787.357.356.047.325.360.3 [57.4, 63.2]25.3
Mixed78.088.759.353.390.076.074.2 [71.6, 76.9]53.3
CTR95.397.374.793.398.095.392.3 [90.7, 94.0]74.7

Constraint violations drop with CTR from 98% to 0% on Put Object Cabinet (vs. L→RL \rightarrow R) and from 36% to 4% on Organize Hardware (vs. Concurrent).

Standard start (Table 3, simulated π0.5\pi_{0.5}, 3 × 100 episodes; Put Object Cabinet 3 × 50; CTR trained without IdleMask, as in the revised method):

TaskSerialCTRΔ [95% CI]
Sort Blocks95.089.3−5.7 [−10.0, −1.7]
Place Dual Shoes39.346.3+7.0 [−1.3, +15.3]
Scan Object63.361.7−1.7 [−8.7, +5.3]
Put Object Cabinet34.062.0+28.0 [+17.3, +38.7]

Which claims hold and which we revise:

  • Holds: CTR's gain across progress states. Under repeated evaluation, CTR raises the six-state mean from 42.1% with the original, unretimed L→RL \rightarrow R data to 74.7% (Put Object Cabinet, sim) and from 77.8% to 92.3% (Sort Letters, real), and the worst state from 0% to 51.3% and from 38.7% to 74.7%. Its six-state mean also exceeds Serial, Concurrent and Mixed on both tasks. Since Mixed is not an external baseline but a restricted form of CTR that already broadens schedule coverage, we now state this gain against L→RL \rightarrow R (abstract, introduction).
  • Removed: higher standard-start success "in 19 of 24 model–task comparisons" (introduction, Sec. 5). CTR targets robustness to timing variation without giving up standard-start performance. Per task (Table 3), it is clearly higher than Serial on Put Object Cabinet, indistinguishable on Place Dual Shoes and Scan Object, and 5.7 points lower on Sort Blocks; the revision reports these per-task results instead.
AuthorsOpenReview ↗

General Response to All Reviewers (3/4)

3. Pipeline accuracy and human effort (NBh4, 16Ph, jyBi)

TL;DR: The pipelines are accurate (constraints correct wherever they can be checked; constraint-critical boundaries off by 0.04–0.10 s on average) and need little human effort (1.1% of real demonstrations corrected).

How a pipeline is made. For each task, a person first supplies a task contract: the language goal, the arm assignment and the required cross-arm dependencies (e.g., “the drawer is open before insertion”). A coding agent (GPT-6 Astra) then writes a task-specific processing program and renders sample recombined videos; a person reviews the samples and the agent revises, for two to five rounds. The program is then frozen and runs on every demonstration without further model queries or edits. A person acts only on sources that the program flags or that fail a validity gate, and every such action is logged.

How accurate it is (new App. F.2). Simulation compares blind-regenerated pipelines with the expert planner's logs; real-robot boundaries are compared with 620 hand labels on 80 demonstrations. Errors are absolute offsets in seconds; "≤ 0.2 / 0.5 s" is the fraction of boundaries within that tolerance. Constraint-critical boundaries are those that define a cross-arm constraint (e.g., the drawer opening before insertion); "–" means the task has none: Sort Blocks and Sort Letters are independent, and Organize Hardware's exclusion is enforced by a 5 cm clearance check on recorded states, not by boundaries.

Task (relation)BoundariesMean / P90 error (s)≤ 0.2 / 0.5 sConstraint-critical error (s)Constraints
SimSort Blocks (independent)4000.07 / 0.141.00 / 1.00–P/R 1.00/1.00
SimPlace Dual Shoes (exclusion)4000.09 / 0.161.00 / 1.000.09P/R 1.00/1.00
SimScan Object (coordination)2500.04 / 0.081.00 / 1.000.04P/R 1.00/1.00
SimPut Object Cabinet (prerequisite)2500.08 / 0.160.98 / 0.980.08P/R 1.00/1.00
RealSort Letters (independent)800.09 / 0.200.94 / 1.00––
RealOrganize Hardware (exclusion)2000.33 / 0.670.36 / 0.84––
RealDeliver Water (coordination)2000.17 / 0.230.88 / 0.950.0640/40 hold
RealPut Object Cabinet (prerequisite)1400.19 / 0.440.70 / 0.940.1020/20 hold

Most of the real-robot offset is a consistent difference in event definition (for example, arm onsets taken from the first joint motion precede visible motion by about 0.2 s). Six boundaries (1.0%) differ by more than 1 s; none is constraint-critical.

Human effort on the four real-robot datasets (counts are source demonstrations):

TaskSourcesGate-rejectedReviewer-excludedAuto-retriedHuman-correctedManual labeling (s/demo)
Sort Letters140000026
Organize Hardware61242064
Deliver Water75000053
Put Object Cabinet879019448

In total, 4 of 363 sources (1.1%) were corrected by a person and 4 (1.1%) excluded by the reviewer; 11 (3.0%) were rejected automatically by validity gates. Two of the four tasks needed no per-demonstration human action.

AuthorsOpenReview ↗

General Response to All Reviewers (4/4)

4. Positioning relative to DexMimicGen and TriManPolicy (NBh4)

TL;DR: Unlike prior work, we study how the narrow schedule coverage of demonstrations affects policy robustness. Both methods give each demonstration one schedule, and our experiments show that one schedule, however efficient, leaves the policy coupled.

All three methods retime per-arm segments under cross-arm constraints; they differ in what is varied and why:

DexMimicGenTriManPolicyCTR
GoalMore demonstrations (new object poses)Remove waits imposed by the teleoperation interfaceRobustness to incidental timing
Training dataNew demonstrations generated in simulationRecorded segments, retimedRecorded segments, retimed
Schedules per demonstrationOne per generated demonstrationOne (fastest)Many admissible
Needs a simulatorYesNoNo

DexMimicGen generates new demonstrations for new object poses, with asynchronous per-arm execution under synchronization and ordering constraints. TriManPolicy retimes recorded segments into one fastest schedule to remove waits imposed by the teleoperation interface. Both aim at more or faster data, and each demonstration still follows one schedule. CTR targets a different failure: when demonstrations cover few schedules, the opposite arm's progress predicts each arm's actions even where the task does not require it. CTR therefore keeps the recorded arm-local content fixed and samples many admissible schedules per demonstration. Reviewer 16Ph judges this distinction, "sampling diverse admissible schedules from recorded data rather than generating new demonstrations or selecting one efficient schedule," to be reasonable.

The key difference is that one schedule, however efficient, is still narrow support: under any fixed schedule each arm's progress is predictable from the other arm's state (Sec. 3.3; Sec. 3.4 in the revision), so the data cannot separate a coupled policy from a decoupled one.

Our experiments test whether one efficient schedule is enough. The Concurrent baseline is the closest analogue of a TriManPolicy-style single efficient schedule (makespan-minimal in independent phases), built from the same recordings by the same pipeline as CTR (π0.5\pi_{0.5}, success rate in %):

EvaluationConcurrentCTR
Put Object Cabinet (sim), standard start (item 2)29.362.0
Put Object Cabinet (sim), six-state mean (item 2)21.474.7
Put Object Cabinet (sim), worst state0.051.3
Sort Letters (real), standard start88.795.3
Sort Letters (real), six-state mean60.392.3
Sort Letters (real), worst state25.374.7
Sort Letters (real), delay of the other arm after a 4 s pause+2.56 s+0.25 s

In simulation Concurrent falls behind CTR even from the standard start. On the real robot it comes close to CTR from the standard start but fails once the two arms' progress departs from the recorded pairing, and a pause of one arm still stalls the other, independent arm. Selecting a better single schedule therefore does not remove the coupling; broadening the schedule support does. CTR is complementary to DexMimicGen and can be applied to its generated data. The comparison with the other methods of Sec. 2 is in our response to Reviewer NBh4 (W1 / Q2).

Revision (the revised PDF has been uploaded; changes are in blue; submitted Table 4 is now Table 3)

  • Abstract, Sec. 1: gains stated against the original data (six-state mean, Table 1); the “19 of 24” claim removed.
  • Sec. 2, Constraint-aware generation (NBh4, 16Ph): DexMimicGen and TriManPolicy; Concurrent as the closest analogue.
  • Sec. 3 (NBh4, jyBi): plain-language argument with one equation (the BC objective, hth_t defined); new paragraph Why spurious coupling matters (Sec. 3.2; Table 16, App. H).
  • Sec. 4 (16Ph, jyBi): IdleMask removed; CTR uses the standard action loss.
  • Sec. 5.2–5.3, Tables 1 and 3 (NBh4, 16Ph): simulated π0.5\pi_{0.5} re-evaluated three times on shared scenes and seeds, with 95% CIs; Sort Letters progress states re-run 3 × 50; standard-start results per task.
  • Sec. 5.5 (new): ablation studies, including the constraint ablation and the accuracy of the extracted constraints.
  • Sec. 6 (16Ph): pipeline errors and sensitivity to wrong constraints.
  • App. A.1 (new, NBh4): released RoboTwin 2.0 π0.5\pi_{0.5} checkpoint (Table 7).
  • App. E.2 (new): repeated evaluation and statistical tests.
  • App. E.8 (new, NBh4 Q1): progress states reached from the standard start (Tables 11–12).
  • App. F.2 (new, all): pipeline generation, accuracy and human intervention (Tables 13–14).
  • App. F.6–F.7 (16Ph, jyBi): optional IdleMask and its ablation (Table 15).
  • Notation: L→RL \rightarrow R is now Original, and C00 is now (L0, R0).

We thank the reviewers again; their comments have substantially improved the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.