- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes FOCUS, a method for learning from state-only demonstrations in a differentiable simulator without temporal alignment. It uses optimal transport to match short agent rollouts against several sampled expert segments, weighting the matches according to their costs. The resulting imitation rewards are used to train a short-horizon differentiable actor–critic, with value bootstrapping beyond the rollout horizon. Training does not require expert-aligned initialization or teacher forcing. The authors evaluate the method on four DFlex locomotion tasks and report improved performance over the compared baselines, particularly on the muscle-actuated SNU Humanoid.
Critical strengths
- Strong results on SNU Humanoid. On this task with 152 muscle control inputs, the OT variants reach approximately 0.9 expert-normalized return at the final interaction budget, well above the compared baselines. Learning this behavior from state-only demonstrations without expert-aligned initialization or teacher forcing is a meaningful result.
- A useful ablation of OT matching. FOCUS-L2 and FOCUS-OT-L2 use the same backbone, reward transformation, normalization, and best-of-K mechanism. FOCUS-OT-L2 achieves higher mean returns on all four tasks at the final interaction budget, providing evidence that OT improves performance beyond the other components of the framework.
Critical weaknesses
- The benefit of the best-of-K mechanism is not clearly established. Much of the framework combines existing OT matching and short-horizon differentiable actor–critic methods, so the contribution of multi-candidate alignment is important to assessing its novelty. Figure 13 shows similar aggregate performance for FOCUS-OT-Cos across K = 1–8, with strong performance already at K = 1. FOCUS-OT-L2 improves up to K = 8, but the aggregate results do not show which tasks benefit or by how much. The phase-recovery diagnostic supports the intuition behind the mechanism, but tests hard selection on expert windows rather than the soft-weighted objective on learner rollouts. Per-task comparisons would clarify when multiple candidates offer a meaningful advantage over K = 1.
- The evaluation provides limited evidence of imitation fidelity. The reported results primarily measure task returns. I could not find qualitative expert–learner rollout comparisons or quantitative measures of behavioral similarity in the supplied materials. High task return supports task performance, but does not establish how closely the learned behavior matches the demonstrations. This matters for local segment matching, which does not explicitly enforce long-horizon consistency. Expert–learner motion comparisons, particularly on SNU Humanoid, would help assess gait similarity and temporal coherence; quantitative similarity measures would further strengthen this assessment.
Questions for authors
- Figure 13 shows little difference between K = 1 and K = 8 for FOCUS-OT-Cos. Could you provide per-task results with confidence intervals to clarify when best-of-K helps?
- Could you provide expert–learner rollout comparisons, especially for SNU Humanoid, and behavioral similarity measures beyond task return? These would help assess imitation fidelity and long-horizon coherence.
- Could you clarify which code and settings produced the reported results? The supplied code detaches the transport plan, uses different Sinkhorn settings from Appendix B.5, and adds a negative-cost term to the logarithmic reward when critic_reward_shapping=True. These differ from Section 4.3 and Eq. 12.
2 comments
Q1 (continued)
This is what we would expect. OT already aligns agent and expert steps within a window, so the candidate pool matters less. Best-of-K guards against a badly matched segment, which matters more when the within-window alignment is weaker. To test this we ran the variant with no alignment, FOCUS-L2, with a single candidate:
| FOCUS-L2 (no alignment) | Hopper | Ant | Humanoid | SNU Humanoid |
|---|---|---|---|---|
| K = 1 | 0.924 [0.906, 0.947] | 0.348 [0.273, 0.416] | 0.496 [0.419, 0.568] | 0.276 [0.174, 0.375] |
| K = 8 | 0.965 [0.951, 0.979] | 0.513 [0.271, 0.755] | 0.583 [0.413, 0.759] | 0.598 [0.388, 0.796] |
Without OT, eight candidates are better than one on all four tasks, and the intervals separate on Hopper and SNU Humanoid; on SNU Humanoid the return more than doubles. The motion improves as well: on the behavioural metric of Q2, the median precision gap goes from 3.05 to 1.36 on Hopper and from 23.9 to 3.27 on Ant (lower is better). Subsequence soft-DTW shows the same direction on all four tasks (reply to Reviewer Q9cS). Best-of-K is also nearly free: under 0.7% of epoch time at K = 8 (Table 12). We will present it as a safeguard against poor matches, most useful when the alignment within a window is weak, and put these per-task tables in the paper.
Q2 — Expert–learner comparison and behavioural similarity
Video. Side-by-side expert and learner clips for all four tasks: anonymous video page. Each clip is one full, uncut episode of one policy per method; the tables carry the aggregate comparison.
Behavioural metric. We measure soft-DTW distance between 32-step windows of the learner's rollouts and of the 8 demonstrations it was trained on, in both directions. Precision gap: how far the learner's motion is from the demonstrations. Coverage gap: how far the demonstrations are from the learner's motion, that is, how much of them the learner fails to reproduce. Each is divided by the same distance measured from other expert demonstrations to those 8, so 1.0 means the learner is as close to its demonstrations as other expert data are. Both are distances, so lower is better; the table gives the median over six seeds.
| Precision gap / coverage gap | Hopper | Ant | Humanoid | SNU Humanoid |
|---|---|---|---|---|
| FOCUS-OT-Cos | 1.12 / 1.70 | 0.91 / 0.98 | 0.73 / 0.93 | 1.47 / 2.10 |
| FOCUS-OT-L2 | 0.87 / 1.37 | 1.80 / 0.93 | 7.26 / 7.17 | 6.77 / 3.31 |
| FOCUS-L2 (no alignment) | 1.36 / 1.84 | 3.27 / 4.11 | 9.94 / 10.98 | 5.08 / 9.16 |
- FOCUS-OT-Cos reproduces the expert's motion on Ant and Humanoid, and partly on SNU Humanoid. On Ant all seeds are below 1.0 on precision and at it on coverage (median 0.91 / 0.98). On Humanoid the median is below 1.0 on both (0.73 / 0.93), with four of six seeds there. On SNU Humanoid three of six seeds are below 1.0 on precision (median 1.47). On Hopper both OT variants are close to 1.0 on precision.
- High return is not the same as fidelity. Without alignment the motion is further from the demonstrations than FOCUS-OT-Cos on every task, even when the return is high: the two best FOCUS-L2 runs on Humanoid reach 0.90 and 0.84 of expert return with precision 2.75 and 3.33. Against FOCUS-OT-L2 the picture is mixed: FOCUS-L2 is further on Hopper, Ant and Humanoid, but closer in precision on SNU Humanoid (5.08 against 6.77), where neither is near 1.0. On this measure it is the cosine configuration, not OT alignment alone, that closes the gap.
- FOCUS-OT-L2 is faithful on the seeds that learn the expert's gait, and beyond Hopper those are at most half. Five of six seeds on Hopper, three of six on Ant (0.62–0.70) and two of six on Humanoid (0.72, 0.99) are at or below 1.0 on precision; the other seeds, including those that converge to a lower-return gait, are far from it. On SNU Humanoid no FOCUS-OT-L2 seed reaches 1.0 (best 1.57).
- The coverage gap is at the expert level on Ant and Humanoid for FOCUS-OT-Cos (0.98 and 0.93): the policy reproduces the full range of its demonstrations. On Hopper and SNU Humanoid it is higher (1.4 to 1.7 for the OT variants, and 2.10 for FOCUS-OT-Cos on SNU Humanoid): there the policy settles on a consistent gait that is close to the expert's but does not span every variation in the demonstrations. Local matching does not enforce this, as the reviewer notes, and we will report it.
The metric uses 32-step windows, the length used in training, so it measures local fidelity rather than long-horizon coherence; for the latter the evidence is the full-episode videos. We will say this in the paper.
We hope the new experiments, the corrections and the video address the reviewer's questions, and we would be grateful if the reviewer would reconsider the score in light of them.
We thank the reviewer for a careful and constructive review. Each question is answered below with new experiments. Numbers are expert-normalized return at the plotted interaction budget, mean with 95 % bootstrap interval, 6 seeds each.
Q3 — Code and settings
The reported results were produced by the released code, with the settings listed below and seeds 100, 110, 120, 1000, 1100, 1200. The reviewer's three observations are accurate descriptions of that code, and the text will be aligned with it:
Fixed coupling. The transport plan is treated as a constant in the actor update: gradients flow through the costs, not through the Sinkhorn iterations. Section 4.3 and the last paragraph of Appendix B.5 say otherwise and will be corrected, and Algorithm 1 will state it explicitly. A new ablation on FOCUS-OT-L2 supports keeping the plan fixed:
Differentiating through the Sinkhorn iterations is much worse on three of four tasks. A likely reason is that the fixed-coupling gradient is the gradient of the entropic OT value, while backpropagating through 60 truncated iterations adds a term for how the plan shifts, which is sensitive at small ε and is then carried through the 32-step rollout. We did not tune the through-plan variant separately. A note on the fixed-coupling row: it is the FOCUS-OT-L2 row of Table 9. The paper's runs were stopped on wall-clock time, so Table 9 reads some entries before the full budget (on Humanoid at about 13M steps instead of 15M). Re-run to the full budget, FOCUS-OT-L2 reaches 0.770 on Ant, 0.673 on Humanoid and 0.817 on SNU Humanoid (reply to Reviewer Q9cS); the Humanoid value is lower because its return declines somewhat late in training. Either set of values is far above the through-plan variant. The revised Table 9 will report the full-budget values and state the step at which every entry is read.
Sinkhorn settings. All runs use ε = 0.1 and 60 iterations, the same as the diagnostic in Appendix E.4. Appendix B.5 will be corrected.
Reward transform. With the exponential transform, used by FOCUS-L2 and FOCUS-OT-L2, the reward is exactly Eq. 12. With the logarithmic transform, used by FOCUS-OT-Cos, the implementation adds the −L term the reviewer noticed, which keeps a useful gradient when the cost is large. Eq. 12 and the Table 11 labels will state this.
For completeness: the L2 variants compute the cost in a fixed random 64-dimensional feature projection of the normalized observations, which Appendix B.5 will document. The supplementary configuration files hold defaults; each reported run adds per-variant command-line overrides (ground cost, reward transform, feature projection, network sizes, demonstrations). We will add the resolved configuration and launch command for every reported number to the supplementary material.
Q1 — Per-task best-of-K
As requested, per task and at a consistent interaction budget. The K = 8 rows are the full-budget runs also reported in the reply to Reviewer Q9cS:
FOCUS-OT-L2
FOCUS-OT-Cos
The reviewer's reading is right. Multiple candidates help FOCUS-OT-L2 on Ant, clearly so at K = 16 (0.705 → 0.985, non-overlapping intervals; at K = 8 the intervals still overlap), while on SNU Humanoid the differences across K are within the intervals. On Humanoid its mean peaks at K = 2 and is lower for larger K. FOCUS-OT-Cos is strong at every K, including K = 1.