acceptodds
Under review as a conference paper at ICLR 2027

First-Order Control Using Optimal Transport Supervision for Imitation from Observations

Abstract

Imitation learning from observations must recover behavior from state-only demonstrations without known temporal correspondence. Differentiable simulators provide pathwise policy gradients, but differentiating through long, contact-rich trajectories can make optimization unstable. We introduce FOCUS, which combines local optimal-transport alignment with short-horizon differentiable actor–critic learning. FOCUS matches agent rollout windows to multiple sampled expert segments using a soft best-of- objective that emphasizes locally compatible matches. The resulting transport-based costs provide differentiable imitation rewards, while a learned critic bootstraps value beyond the rollout horizon. FOCUS requires neither temporally aligned trajectories nor expert-aligned initialization or teacher forcing. Across contact-rich locomotion tasks in DFlex, including a musculoskeletal humanoid controlled by 152 muscle activations, FOCUS improves sample efficiency and training stability over adversarial imitation-from-observation methods and prior differentiable imitation baselines.

Primary area:
reinforcement learning
Keywords:
Imitation Learning from Observations, Differentiable Physics Simulation, Optimal Transport, First-Order Policy Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LMQHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FOCUS, a method for learning from state-only demonstrations in a differentiable simulator without temporal alignment. It uses optimal transport to match short agent rollouts against several sampled expert segments, weighting the matches according to their costs. The resulting imitation rewards are used to train a short-horizon differentiable actor–critic, with value bootstrapping beyond the rollout horizon. Training does not require expert-aligned initialization or teacher forcing. The authors evaluate the method on four DFlex locomotion tasks and report improved performance over the compared baselines, particularly on the muscle-actuated SNU Humanoid.

Critical strengths
  1. Strong results on SNU Humanoid. On this task with 152 muscle control inputs, the OT variants reach approximately 0.9 expert-normalized return at the final interaction budget, well above the compared baselines. Learning this behavior from state-only demonstrations without expert-aligned initialization or teacher forcing is a meaningful result.
  2. A useful ablation of OT matching. FOCUS-L2 and FOCUS-OT-L2 use the same backbone, reward transformation, normalization, and best-of-K mechanism. FOCUS-OT-L2 achieves higher mean returns on all four tasks at the final interaction budget, providing evidence that OT improves performance beyond the other components of the framework.
Critical weaknesses
  1. The benefit of the best-of-K mechanism is not clearly established. Much of the framework combines existing OT matching and short-horizon differentiable actor–critic methods, so the contribution of multi-candidate alignment is important to assessing its novelty. Figure 13 shows similar aggregate performance for FOCUS-OT-Cos across K = 1–8, with strong performance already at K = 1. FOCUS-OT-L2 improves up to K = 8, but the aggregate results do not show which tasks benefit or by how much. The phase-recovery diagnostic supports the intuition behind the mechanism, but tests hard selection on expert windows rather than the soft-weighted objective on learner rollouts. Per-task comparisons would clarify when multiple candidates offer a meaningful advantage over K = 1.
  2. The evaluation provides limited evidence of imitation fidelity. The reported results primarily measure task returns. I could not find qualitative expert–learner rollout comparisons or quantitative measures of behavioral similarity in the supplied materials. High task return supports task performance, but does not establish how closely the learned behavior matches the demonstrations. This matters for local segment matching, which does not explicitly enforce long-horizon consistency. Expert–learner motion comparisons, particularly on SNU Humanoid, would help assess gait similarity and temporal coherence; quantitative similarity measures would further strengthen this assessment.
Questions for authors
  1. Figure 13 shows little difference between K = 1 and K = 8 for FOCUS-OT-Cos. Could you provide per-task results with confidence intervals to clarify when best-of-K helps?
  2. Could you provide expert–learner rollout comparisons, especially for SNU Humanoid, and behavioral similarity measures beyond task return? These would help assess imitation fidelity and long-horizon coherence.
  3. Could you clarify which code and settings produced the reported results? The supplied code detaches the transport plan, uses different Sinkhorn settings from Appendix B.5, and adds a negative-cost term to the logarithmic reward when critic_reward_shapping=True. These differ from Section 4.3 and Eq. 12.
2 comments

We thank the reviewer for a careful and constructive review. Each question is answered below with new experiments. Numbers are expert-normalized return at the plotted interaction budget, mean with 95 % bootstrap interval, 6 seeds each.

Q3 — Code and settings

The reported results were produced by the released code, with the settings listed below and seeds 100, 110, 120, 1000, 1100, 1200. The reviewer's three observations are accurate descriptions of that code, and the text will be aligned with it:

  1. Fixed coupling. The transport plan is treated as a constant in the actor update: gradients flow through the costs, not through the Sinkhorn iterations. Section 4.3 and the last paragraph of Appendix B.5 say otherwise and will be corrected, and Algorithm 1 will state it explicitly. A new ablation on FOCUS-OT-L2 supports keeping the plan fixed:

    FOCUS-OT-L2HopperAntHumanoidSNU Humanoid
    fixed coupling (Table 9)0.988 [0.981, 0.994]0.818 [0.647, 0.984]0.816 [0.784, 0.847]0.912 [0.902, 0.920]
    gradient through Sinkhorn0.977 [0.961, 0.987]0.195 [0.093, 0.311]0.141 [0.060, 0.225]0.301 [0.217, 0.412]

    Differentiating through the Sinkhorn iterations is much worse on three of four tasks. A likely reason is that the fixed-coupling gradient is the gradient of the entropic OT value, while backpropagating through 60 truncated iterations adds a term for how the plan shifts, which is sensitive at small ε and is then carried through the 32-step rollout. We did not tune the through-plan variant separately. A note on the fixed-coupling row: it is the FOCUS-OT-L2 row of Table 9. The paper's runs were stopped on wall-clock time, so Table 9 reads some entries before the full budget (on Humanoid at about 13M steps instead of 15M). Re-run to the full budget, FOCUS-OT-L2 reaches 0.770 on Ant, 0.673 on Humanoid and 0.817 on SNU Humanoid (reply to Reviewer Q9cS); the Humanoid value is lower because its return declines somewhat late in training. Either set of values is far above the through-plan variant. The revised Table 9 will report the full-budget values and state the step at which every entry is read.

  2. Sinkhorn settings. All runs use ε = 0.1 and 60 iterations, the same as the diagnostic in Appendix E.4. Appendix B.5 will be corrected.

  3. Reward transform. With the exponential transform, used by FOCUS-L2 and FOCUS-OT-L2, the reward is exactly Eq. 12. With the logarithmic transform, used by FOCUS-OT-Cos, the implementation adds the −L term the reviewer noticed, which keeps a useful gradient when the cost is large. Eq. 12 and the Table 11 labels will state this.

For completeness: the L2 variants compute the cost in a fixed random 64-dimensional feature projection of the normalized observations, which Appendix B.5 will document. The supplementary configuration files hold defaults; each reported run adds per-variant command-line overrides (ground cost, reward transform, feature projection, network sizes, demonstrations). We will add the resolved configuration and launch command for every reported number to the supplementary material.

Q1 — Per-task best-of-K

As requested, per task and at a consistent interaction budget. The K = 8 rows are the full-budget runs also reported in the reply to Reviewer Q9cS:

FOCUS-OT-L2

KHopperAntHumanoidSNU Humanoid
10.962 [0.932, 0.981]0.705 [0.490, 0.886]0.702 [0.515, 0.880]0.795 [0.663, 0.878]
20.982 [0.976, 0.988]0.701 [0.475, 0.910]0.834 [0.796, 0.870]0.624 [0.457, 0.791]
40.977 [0.967, 0.985]0.804 [0.618, 0.982]0.736 [0.611, 0.823]0.812 [0.607, 0.945]
80.988 [0.984, 0.991]0.770 [0.656, 0.882]0.673 [0.515, 0.816]0.817 [0.747, 0.879]
160.990 [0.984, 0.994]0.985 [0.980, 0.990]0.650 [0.355, 0.896]0.791 [0.602, 0.927]

FOCUS-OT-Cos

KHopperAntHumanoidSNU Humanoid
10.981 [0.975, 0.986]0.905 [0.896, 0.914]0.921 [0.871, 0.959]0.885 [0.871, 0.900]
20.979 [0.973, 0.984]0.901 [0.893, 0.909]0.910 [0.838, 0.958]0.862 [0.822, 0.902]
40.973 [0.954, 0.985]0.910 [0.901, 0.918]0.972 [0.962, 0.981]0.894 [0.882, 0.905]
80.984 [0.980, 0.990]0.910 [0.900, 0.917]0.912 [0.814, 0.980]0.891 [0.827, 0.935]
160.981 [0.980, 0.982]0.911 [0.905, 0.917]0.832 [0.552, 0.979]0.872 [0.831, 0.910]

The reviewer's reading is right. Multiple candidates help FOCUS-OT-L2 on Ant, clearly so at K = 16 (0.705 → 0.985, non-overlapping intervals; at K = 8 the intervals still overlap), while on SNU Humanoid the differences across K are within the intervals. On Humanoid its mean peaks at K = 2 and is lower for larger K. FOCUS-OT-Cos is strong at every K, including K = 1.

Q1 (continued)

This is what we would expect. OT already aligns agent and expert steps within a window, so the candidate pool matters less. Best-of-K guards against a badly matched segment, which matters more when the within-window alignment is weaker. To test this we ran the variant with no alignment, FOCUS-L2, with a single candidate:

FOCUS-L2 (no alignment)HopperAntHumanoidSNU Humanoid
K = 10.924 [0.906, 0.947]0.348 [0.273, 0.416]0.496 [0.419, 0.568]0.276 [0.174, 0.375]
K = 80.965 [0.951, 0.979]0.513 [0.271, 0.755]0.583 [0.413, 0.759]0.598 [0.388, 0.796]

Without OT, eight candidates are better than one on all four tasks, and the intervals separate on Hopper and SNU Humanoid; on SNU Humanoid the return more than doubles. The motion improves as well: on the behavioural metric of Q2, the median precision gap goes from 3.05 to 1.36 on Hopper and from 23.9 to 3.27 on Ant (lower is better). Subsequence soft-DTW shows the same direction on all four tasks (reply to Reviewer Q9cS). Best-of-K is also nearly free: under 0.7% of epoch time at K = 8 (Table 12). We will present it as a safeguard against poor matches, most useful when the alignment within a window is weak, and put these per-task tables in the paper.

Q2 — Expert–learner comparison and behavioural similarity

Video. Side-by-side expert and learner clips for all four tasks: anonymous video page. Each clip is one full, uncut episode of one policy per method; the tables carry the aggregate comparison.

Behavioural metric. We measure soft-DTW distance between 32-step windows of the learner's rollouts and of the 8 demonstrations it was trained on, in both directions. Precision gap: how far the learner's motion is from the demonstrations. Coverage gap: how far the demonstrations are from the learner's motion, that is, how much of them the learner fails to reproduce. Each is divided by the same distance measured from other expert demonstrations to those 8, so 1.0 means the learner is as close to its demonstrations as other expert data are. Both are distances, so lower is better; the table gives the median over six seeds.

Precision gap / coverage gapHopperAntHumanoidSNU Humanoid
FOCUS-OT-Cos1.12 / 1.700.91 / 0.980.73 / 0.931.47 / 2.10
FOCUS-OT-L20.87 / 1.371.80 / 0.937.26 / 7.176.77 / 3.31
FOCUS-L2 (no alignment)1.36 / 1.843.27 / 4.119.94 / 10.985.08 / 9.16
  • FOCUS-OT-Cos reproduces the expert's motion on Ant and Humanoid, and partly on SNU Humanoid. On Ant all seeds are below 1.0 on precision and at it on coverage (median 0.91 / 0.98). On Humanoid the median is below 1.0 on both (0.73 / 0.93), with four of six seeds there. On SNU Humanoid three of six seeds are below 1.0 on precision (median 1.47). On Hopper both OT variants are close to 1.0 on precision.
  • High return is not the same as fidelity. Without alignment the motion is further from the demonstrations than FOCUS-OT-Cos on every task, even when the return is high: the two best FOCUS-L2 runs on Humanoid reach 0.90 and 0.84 of expert return with precision 2.75 and 3.33. Against FOCUS-OT-L2 the picture is mixed: FOCUS-L2 is further on Hopper, Ant and Humanoid, but closer in precision on SNU Humanoid (5.08 against 6.77), where neither is near 1.0. On this measure it is the cosine configuration, not OT alignment alone, that closes the gap.
  • FOCUS-OT-L2 is faithful on the seeds that learn the expert's gait, and beyond Hopper those are at most half. Five of six seeds on Hopper, three of six on Ant (0.62–0.70) and two of six on Humanoid (0.72, 0.99) are at or below 1.0 on precision; the other seeds, including those that converge to a lower-return gait, are far from it. On SNU Humanoid no FOCUS-OT-L2 seed reaches 1.0 (best 1.57).
  • The coverage gap is at the expert level on Ant and Humanoid for FOCUS-OT-Cos (0.98 and 0.93): the policy reproduces the full range of its demonstrations. On Hopper and SNU Humanoid it is higher (1.4 to 1.7 for the OT variants, and 2.10 for FOCUS-OT-Cos on SNU Humanoid): there the policy settles on a consistent gait that is close to the expert's but does not span every variation in the demonstrations. Local matching does not enforce this, as the reviewer notes, and we will report it.

The metric uses 32-step windows, the length used in training, so it measures local fidelity rather than long-horizon coherence; for the latter the evidence is the full-episode videos. We will say this in the paper.

We hope the new experiments, the corrections and the video address the reviewer's questions, and we would be grateful if the reviewer would reconsider the score in light of them.

Reviewer_zDN4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

FOCUS proposes an imitation-from-observation method that combines local optimal-transport matching of unaligned expert trajectory segments with short-horizon differentiable actor-critic learning. The central idea is: simulator derivatives provide substantially more informative local policy gradients than black-box RL, while soft best-of-K matching avoids requiring temporal alignment, expert actions, or teacher forcing. The experiments on four DFlex locomotion systems, including the challenging 152-actuator SNU Humanoid, provide evidence of sample-efficient learning, but the empirical story is less clean than the headline suggests because performance depends substantially on the underlying first-order RL backbone and a video of the experiments is not included. Overall, this is a technically interesting contribution, although its dependence on a known differentiable simulator, validation only on periodic locomotion, and lack of an isolated comparison establishing exactly how much improvement comes from simulator derivatives limit the generality of the conclusions. Comparison to or mention of any other sampling-based works that rely on differentiable simulators is also lacking (e.g., CrazyFlow or this paper:learning Loco-Manipulation From SMPC Demonstrations With Sparse Offline-to-Online RL).

Critical strengths

Well-motivated combination of ingredients. Local OT addresses temporal misalignment while short differentiable rollouts avoid the unstable gradients associated with backpropagating through long contact-rich trajectories; critic bootstrapping gives longer-horizon credit.

The SNU Humanoid with 152 muscle activations is a genuinely difficult test case, and the FOCUS OT variants substantially outperform zeroth-order approaches there. Results are reported in both interaction and wall-clock budgets, which is particularly valuable for differentiable simulation methods.

Useful robustness and ablation analysis. The paper studies number of demonstrations, critic use, best-of-K sampling, and different first-order backbones and shows that the method can work from a single expert trajectory.

Critical weaknesses

No comparison to other sampling-based (RL or MPC) methods that rely on a differentiable simulator and use some form of demonstrations.

No video of the experiment.

Significant dependence on the RL backbone. With SHAC, FOCUS can collapse on Humanoid, whereas SAPO is much more stable; moreover, SHAC versus SAPO differs in objectives, critic design, activations, optimizers, variance parameterization, and clipping. This makes FOCUS appear somewhat less algorithmically robust than the main results suggest.

Narrow validation domain and a structural alignment limitation. All experiments are periodic locomotion in DFlex. Local best-of-K matching does not guarantee global ordering or coverage and may therefore fail on manipulation or non-periodic multi-stage tasks.

Questions for authors

I am mostly worried about not fully connecting the work to many existing recent approaches that leverage differentiable simulators. And, secondly, no comparison to the SOTA regarding humanoids, which is imitation+retargeting+RL in simulation+sim2real (see e.g. egoHTR). Why does this not solve your problem? Where are the major benefits of this method?

2 comments

We thank the reviewer for the detailed and constructive review. Numbers are expert-normalized return at the plotted budget, mean with 95 % bootstrap interval, 6 seeds.

1. Video

Side-by-side expert and learner clips for all four tasks: https://anonymous.4open.science/w/iclr-44492-videos/, including the 152-muscle SNU Humanoid with muscles coloured by activation. Each clip is one full episode at that task's reported budget, in real time, from one policy trained with the reported configuration, with that policy's return given in the caption.

2. Isolating the contribution of simulator derivatives

We ran the isolated comparison the review asks for, on FOCUS-OT-L2 with the paper's six seeds. Both variants share the SAPO actor-critic, the imitation reward, the networks, the horizon, the demonstrations, the budget and the seeds. Only the actor's gradient estimator differs: the pathwise gradient through the simulator, or a score-function estimate of the same objective with the same critic as baseline.

HopperAntHumanoidSNU Humanoid
pathwise (FOCUS)0.988 [0.981, 0.994]0.818 [0.647, 0.984]0.816 [0.784, 0.847]0.912 [0.902, 0.920]
score-function0.010 [−0.055, 0.082]0.095 [0.071, 0.112]0.010 [0.008, 0.012]0.002 [0.002, 0.003]

With everything else fixed, the score-function estimator does not learn this objective at this budget. It keeps the hyperparameters of the pathwise variant and was not tuned separately, and it fails even on Hopper, where a tuned PPO on the same reward reaches 0.851 [0.693, 0.933] (Table 13; edited: the 0.922 given earlier was the median not the mean). It should therefore be read as an isolated control, not as the best a zeroth-order method can do.

A note on the pathwise row: it is the FOCUS-OT-L2 row of Table 9. The paper's runs were stopped on wall-clock time, so Table 9 reads some entries before the full budget (on Humanoid at about 13M steps instead of 15M). Re-run to the full budget, FOCUS-OT-L2 reaches 0.770 on Ant, 0.673 on Humanoid and 0.817 on SNU Humanoid (reply to Reviewer Q9cS); the Humanoid value is lower because its return declines somewhat late in training. Either set of values is far above the score-function row, so the comparison is unchanged.

Table 13 is the complementary comparison, against tuned zeroth-order algorithms (PPO, SAC). As the paper notes, it compares complete training procedures and does not isolate the estimator; we will make that explicit in its caption. The two comparisons agree on the humanoids, where the tuned zeroth-order variants stay at 0.25 or below on Humanoid and 0.12 or below on SNU Humanoid.

3. Dependence on the backbone

FOCUS is an imitation objective that plugs into a first-order actor-critic, so, like any imitation method built on an RL optimiser, it inherits that optimiser's stability. SAPO was introduced to stabilise SHAC-style training (Xing et al., 2025), and the differences the reviewer lists are the ones documented there and summarised in Appendix E.1. All main results use SAPO and the SHAC results are an ablation; we will say in the main text that SAPO is the recommended backbone. We agree that FOCUS is not invariant to the backbone, as Appendix E.1 states, and the main text will say so next to this recommendation.

4. Methods that use a differentiable simulator with demonstrations

The paper compares against ILD (Chen et al., 2023), the closest prior method that learns from demonstrations through a differentiable simulator (Table 9), and against sampling-based imitation methods that use the same demonstrations without simulator derivatives (GAIfO, OPOLO, PWIL, OOPS and others). The comparison in §2 adds the isolated version of that contrast. Requiring a known differentiable simulator is a limitation we state in Section 6; it is shared with SHAC, SAPO and ILD, and such simulators are becoming more widely available.

We will add the following to the related work. Crazyflow (arXiv:2606.01478) is a GPU-accelerated differentiable drone simulator that supports both analytic-gradient policy learning, the paradigm FOCUS belongs to, and sampling-based methods; it is infrastructure rather than an imitation method. Learning Loco-Manipulation From SMPC Demonstrations (arXiv:2608.12063) learns from sparse task rewards with demonstrations generated by sampling-based MPC; it differs from our setting in using task rewards and demonstrations with actions, and, as far as we can tell, does not rely on simulator derivatives.

5. Why not imitation + retargeting + RL + sim2real (e.g. EgoHTR)?

Those pipelines, including the policies trained on the EgoHTR dataset (arXiv:2607.13472), track a retargeted, time-indexed reference: the learner knows which reference frame it should be at and is typically initialised on reference states. FOCUS targets the complementary case in which that information is not available or not used: states only, no time alignment, no expert actions, no initialisation from expert states. In our benchmark the demonstrations are simulator rollouts, so a tracking method could be applied to them; we use them to study the unaligned setting, not to claim that tracking would fail there. When an aligned reference exists, tracking is the right tool. When it does not, tracking cannot be applied without first solving the alignment, and that is the step FOCUS removes. The benefit of FOCUS is in that regime: it needs no aligned reference and no expert actions, and its first-order optimisation learns a 152-muscle system at a budget where zeroth-order optimisation of the same objective does not (§2). We will make this boundary explicit in the paper.

6. Validation domain; ordering and coverage

We agree that local matching gives no global ordering or coverage guarantee, and we will state it plainly. Two new data points:

  • A non-periodic task. On DFlex CartPole Swing-Up with randomised initial pole angle (recover, then balance; 4M steps, 6 seeds), FOCUS reaches expert level in task return (mean ± standard deviation over seeds; higher is better): −314 ± 27 (OT-L2) and −319 ± 32 (OT-Cos) against −317 ± 11 for the expert. Local matching does not break down on this two-phase task. The other objectives we tried also solve it, so the task is probably too easy to separate them.
  • Measured coverage. With a two-sided behavioural metric (reply to Reviewer LMQH), FOCUS-OT-Cos reproduces its demonstrations as fully as other expert data do on Ant and Humanoid (coverage gap 0.98 and 0.93; lower is better and 1.0 is the expert level). On SNU Humanoid the gap is 2.10, about twice the expert level: the policy settles on a consistent gait that is close to the expert's but does not span every variation in the demonstrations. Local matching does not enforce this, which is the limitation the reviewer describes.

Manipulation and multi-stage tasks remain future work.

We hope the video, the isolated gradient-estimator comparison and the clarified scope address the reviewer's concerns, and we would be grateful if the reviewer would reconsider the score in light of them.

Reviewer_Q9cSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes an optimal transport variant for imitation learning that aligns expert and learner trajectories. The authors evaluate their method on four periodic locomotion tasks in DFlex: Hopper, Ant, Humanoid, and the 152-muscle SNU Humanoid, and compare it against adversarial ILfO methods (GAIfO, MAAD, SAMfO/DACfO, OPOLO), Wasserstein-based methods (PWIL, OOPS, LWAIL), and the first-order ILD baseline.

Critical strengths
  • The submission provides clear and sufficiently detailed information to ensure reproducibility. The methodology, implementation details, and source code are well documented and presented.
Critical weaknesses
  • The paper mainly compares with OT-based methods. Although DTW-based approaches are discussed in the related work, the experiments do not include soft-DTW-based methods, such as ORCA, TemporalOT, or other soft-DTW variants.
  • GDTW has been evaluated on imitation learning tasks before, but the paper does not compare with GDTW-IL and other closely related methods, such as ORCA and Abyaneh et al., are also missing in evaluation section.
  • All four tasks involve periodic locomotion, where the main differences are phase shifts and speed variations, but the paper does not evaluate non-periodic or order-sensitive tasks.
  • The proposed best-of-K softmin is similar to a sampled version of subsequence soft-DTW. Therefore the contribution appears to be mainly incremental and engineering-based.
  • The transport plans in Fig. 3 and Fig. 9 mostly show diagonal or shifted diagonal patterns, which can be naturally handled by DTW. It is unclear whether OT provides additional benefits for these tasks.
  • Some transport plans in Fig. 3(c) and Fig. 9(f)(h) show fragmented patterns, potentially caused by the uniform marginal constraints of OT, which that Open-end DTW could potentially avoid this issue.
Questions for authors
  • Please consider add DTW-based baseline/methods and compare to proposed method.
  • Please justify what advantage of OT-based method and the necessity of OT-based.
  • Could author justify or provide additional experiment on non-periodic or order-sensitive imitation learning tasks?
4 comments

We thank the reviewer for concrete suggestions; two of them led directly to new results. Numbers are expert-normalized return at the plotted budget, mean with 95 % bootstrap interval, 6 seeds.

1. DTW-based objectives and the role of OT

We implemented full soft-DTW (the whole agent window aligned to the whole expert segment), subsequence soft-DTW (aligned to the best-matching stretch of the segment, open at both ends) and a state-based port of ORCA as drop-in replacements for the Sinkhorn coupling, with everything else identical: backbone, horizon, K = 8 candidate segments, demonstrations, budget and seeds. Unless marked otherwise, the soft-DTW and ORCA rows use the same L2 ground cost and reward transform as FOCUS-OT-L2. We also ran full soft-DTW in the configuration of FOCUS-OT-Cos (cosine cost, logarithmic reward, same features), so that the two alignments can be compared in both configurations. The control with no alignment is FOCUS-L2 (index-wise matching of the same segments). All rows are new runs, each taken to the task's full interaction budget.

The FOCUS rows therefore differ from Table 9: the paper's runs were stopped on wall-clock time, and Table 9 reads each method at the last step reached by its shortest seed, which on Humanoid is about 13M steps rather than 15M. On Humanoid the return of FOCUS-OT-L2 also declines somewhat late in training, which is why its full-budget value (0.673) is below the Table 9 value (0.816). Re-running puts every objective at the same budget. The revised Table 9 will report the full-budget values for the FOCUS variants and state the step at which every entry is read.

ObjectiveHopperAntHumanoidSNU Humanoid
FOCUS-OT-Cos0.984 [0.980, 0.990]0.910 [0.900, 0.917]0.912 [0.814, 0.980]0.891 [0.827, 0.935]
Full soft-DTW, cosine configuration0.977 [0.950, 0.996]0.893 [0.887, 0.896]0.714 [0.677, 0.752]0.828 [0.777, 0.869]
FOCUS-OT-L20.988 [0.984, 0.991]0.770 [0.656, 0.882]0.673 [0.515, 0.816]0.817 [0.747, 0.879]
Full soft-DTW, L2 configuration1.002 [0.998, 1.005]0.816 [0.581, 0.954]0.606 [0.336, 0.835]0.733 [0.555, 0.897]
no alignment (FOCUS-L2, L2 configuration)0.965 [0.951, 0.979]0.513 [0.271, 0.755]0.583 [0.413, 0.759]0.598 [0.388, 0.796]
subsequence soft-DTW, K = 80.998 [0.996, 1.000]0.346 [0.123, 0.632]0.538 [0.404, 0.669]0.487 [0.197, 0.750]
subsequence soft-DTW, K = 10.879 [0.652, 0.998]0.221 [0.157, 0.287]0.451 [0.234, 0.660]0.417 [0.237, 0.617]
ORCA, state-based port0.655 [0.339, 0.926]0.206 [0.151, 0.252]0.545 [0.368, 0.742]0.474 [0.352, 0.616]

Every cell is the mean of each run's final 128-episode evaluation at the task's interaction budget, expert-normalized, with a 95% bootstrap interval.

  • OT with the cosine cost has the highest mean on Ant and on both humanoids (0.910, 0.912 and 0.891), and the only one whose interval lies above the no-alignment control on all three. On Hopper every objective except the ORCA port and subsequence soft-DTW at K = 1 is at expert level.
  • In the cosine configuration, OT is ahead of full soft-DTW: on Ant (0.910 against 0.893) and Humanoid (0.912 against 0.714) with separated intervals, and in mean on SNU Humanoid (0.891 against 0.828, intervals overlap); on Hopper the two are level. The cosine configuration also raises the mean of soft-DTW itself on Ant and both humanoids.
  • In the L2 configuration, full soft-DTW works inside FOCUS and is level with OT-L2: slightly ahead on Hopper and Ant, slightly behind on the two humanoids, with overlapping intervals on every task but Hopper, where both are at expert level. For the L2 cost, a monotone alignment is a viable alternative to OT, as the reviewer expected.
  • Subsequence soft-DTW and the ORCA port are lower. Both are clearly below OT-L2 on Ant, and on Ant and the two humanoids their means are below the no-alignment control (intervals overlap, except for the ORCA port on Ant, which is clearly below). The ORCA row is a state-based port of the objective rather than the published video system, and we did not tune it as carefully as our own.

1 (continued). Is OT necessary, and methods we did not run

So the answer to "is OT necessary" depends on the configuration. In the L2 configuration it is not: OT and full soft-DTW are level. In the cosine configuration, which gives the higher mean for both alignments on Ant and both humanoids, OT is the better alignment: ahead on Ant and Humanoid with separated intervals, and in mean on SNU Humanoid. Soft-DTW remains a strong alternative: the framework (local matching optimised through the simulator, best-of-K, critic) works with either alignment, and both benefit from the cosine configuration. The two configurations differ in more than the ground cost: the reward transform differs too (Section 5.2, Table 11), and the L2 cost is computed in a projected feature space (reply to Reviewer LMQH), so we attribute the gain to the configuration and not to the cost alone. We will add this table and both soft-DTW variants to the paper.

These are controlled substitutions inside our pipeline, not reproductions of published systems: the ORCA row is our port, and we did not run TemporalOT or GDTW-IL. GDTW-IL (Cohen et al., 2021b) differentiates a trajectory-level Gromov-DTW loss through known dynamics; its Gromov formulation is designed for incomparable spaces, whereas here learner and expert share a state space, so the full soft-DTW row is its closest counterpart inside our pipeline. TemporalOT is an OT reward for model-free RL with temporally contextualized costs and a temporal mask, and we did not port it. Both are discussed in Appendix A and will be discussed alongside the new table.

For Abyaneh et al., we take the reference to be Contractive Dynamical Imitation Policies for Efficient Out-of-Sample Recovery (ICLR 2025). It learns a contractive dynamical system from state-only demonstrations, with a soft-DTW loss or MSE loss between its rollouts and the expert trajectories. Those rollouts are integrations of the learned system itself, started from the expert's initial states; the system plans states, and a separate low-level controller turns them into robot commands. FOCUS instead learns the controller itself, torques or muscle activations, by differentiating through contact-rich simulated dynamics, without expert-aligned initialisation. Matching whole trajectories is workable when the rollout is a smooth learned system; through contact-rich dynamics, differentiating a whole rollout is what makes optimisation unstable, which is why FOCUS matches short windows and uses a critic beyond them. The full soft-DTW row above uses the same kind of loss inside our pipeline, applied to short windows, and we will cite and discuss the paper.

2. Near-diagonal plans

The plans in Figs. 3 and 9 are snapshots taken during training. They are near-diagonal once the policy's motion is close to the expert's. Earlier, when rollouts are far from the expert's support, the plans are diffuse (Fig. 3a) and there is no monotone correspondence to recover yet. The diagonal structure is therefore an outcome of learning, and the objective also has to guide the policy before it appears. OT assumes no ordering at that stage, while DTW imposes one. With the L2 cost both get there: §1 shows comparable final returns for full soft-DTW and OT-L2. In the cosine configuration OT ends higher on Ant and Humanoid. We agree that a monotone alignment can handle these plans; in our runs it is not always as effective.

The two objectives do differ in what they constrain. DTW also fixes the order of the matched states. OT matches the set of states visited within a window and leaves their order to the dynamics and the critic. With best-of-K candidate segments, both can be applied without phase alignment (§1).

3. Best-of-K and sampled subsequence soft-DTW

We tested this construction directly (subsequence rows in §1). It behaves like OT best-of-K on Hopper but is lower on Ant (0.346 against 0.770) and SNU Humanoid (0.487 against 0.817), so the two are not interchangeable in practice. We agree that best-of-K is a simple mechanism. It is one component of the framework, a safeguard against poorly matched segments, and it is nearly free: on SNU Humanoid the matching step takes 0.67% of epoch time at K = 1 and 0.69% at K = 8 (Table 12). At K = 8 it does not hurt the OT variants (no interval falls below the one at K = 1 on any task; on Ant, FOCUS-OT-L2 improves clearly at K = 16), and it improves the mean of the variant without OT on all four tasks (reply to Reviewer LMQH). The contribution is the combination: alignment-free local matching optimised through the simulator, which learns a 152-muscle humanoid from states only, where zeroth-order optimisation of the same objective does not (reply to Reviewer zDN4).

4. Fragmented plans and uniform marginals

This suggestion led to a variant that can improve the method. We implemented unbalanced OT for FOCUS-OT-L2, with KL-relaxed marginals of strength ρ (ρ → ∞ recovers the balanced plan); 6 seeds per setting, final 128-episode evaluation at 15M steps. The balanced row is the FOCUS-OT-L2 row of §1.

FOCUS-OT-L2SNU HumanoidHumanoid
balanced0.817 [0.747, 0.879]0.673 [0.515, 0.816]
ρ = 1000.845 [0.677, 0.944]0.840 [0.724, 0.932]
ρ = 100.952 [0.932, 0.972]0.536 [0.260, 0.807]
ρ = 10.027 [−0.023, 0.081]0.102 [0.029, 0.222]

A moderate relaxation gives the best mean on both tasks, but at a different ρ for each. On SNU Humanoid the improvement at ρ = 10 is clear and all six seeds succeed; on Humanoid the intervals overlap at ρ = 100, and ρ = 10 lowers the mean. Too strong a relaxation removes the learning signal, so ρ is a task-dependent hyperparameter to tune. We will add the unbalanced variant and this ablation to the paper.

On the suggestion of open-end DTW as a way to avoid the fragmentation: the closest variant we ran is the subsequence soft-DTW of §1 (open at both ends), which had lower means than OT on Ant and on both humanoids.

5. Baselines

To clarify, the evaluation covers four adversarial methods (GAIfO, MAAD, SAMfO/DACfO, OPOLO), the first-order ILD, and three Wasserstein methods (PWIL, OOPS, LWAIL). The missing family was ordered alignment, which §1 now adds. We will group the baselines by family in the main table.

6. Non-periodic tasks

We added DFlex CartPole Swing-Up with randomised initial pole angle (recover, then balance; 4M steps, 6 seeds). FOCUS reaches expert level in task return (mean ± standard deviation over seeds; higher is better): −314 ± 27 (OT-L2) and −319 ± 32 (OT-Cos) against −317 ± 11 for the expert, as do soft-DTW (−313) and a state-based port of the ORCA objective (−327). Local matching does not break down on this non-periodic task; since all objectives solve it, the task is probably too easy to separate them. Harder order-sensitive tasks are future work, and we will state the scope of the evaluation clearly.

The DTW-based comparisons requested in the review are now run inside our pipeline, and the suggestion to relax the OT marginals led to a useful variant. To make the paper easier to read, the revision will describe the objective exactly as implemented and group the baselines and alignment variants by family; we are glad to clarify any specific passage. We would be grateful if the reviewer would reconsider the score in light of these results.

Thanks for the author's response and additional experiments, which addressed some of my concerns. However, some concerns (the necessity of using OT and the non-periodic task) remain. Therefore, I will keep my score as is.

Comments on the paper

1 comment

Dear Reviewers,

I'm writing to gently remind you that the rebuttal discussion phase deadline is approaching. Please take a moment to read the authors' rebuttal, and feel free to respond if any concerns remain that you'd like them to address. Thank you very much for your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.