- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This study focuses on trajectory stitching within the context of goal-conditioned behavior cloning for policy generalization. The proposed TD-Aug uses a flow-based geometric horizon world model to generate additional goals. The model predicts future states under a mixture policy, which can switch between data gathering behaviors at each state. This allows for joining behaviors across separate trajectories. The proposed MCHC training objective adapts prior TD horizon consistency (TD-HC) combining observed future state targets with TD bootstrapping. The empirical results show substantial gains over BC on navigation and manipulation tasks that require stitching for attaining success.
Critical strengths
-
I find the idea of using GHM to relabel imagined goals neat. The trained policy can act without requiring the world model at test time.
-
The empirical gains over BC appear to be consistent across several settings. The MC-Aug control is useful and suggests that the gains arise from the stitched futures, not simply from more synthetic goals.
Critical weaknesses
-
It is not clear if the method learns the future state distribution under the mixture policy as claimed. Sec 3.4 replaces sampling from with next action from the same recorded trajectory (line 220-222). But this conditional distribution need not equal which conditions only on the current state. The authors should clarify the assumptions under which this substitution makes sense.
-
The proposed MCHC objective modifies TD-HC but does not compare against it. Some controls also use different prediction horizons making it difficult to compare the different obnectives. Results use 3 seeds reporting std and no intervals.
While Theorem 1 supports an idealized version of the proposed method, it does not settle these practical considerations.
Questions for authors
-
Why is using recorded next actions for learning the mixture policy distribution justified?
-
How does your approach compare against the original TD-HC objective under the same prediction horizon?
-
Which policy was used to generate the reference trajectories in Fig. 3?
-
Theorem 1 needs clear positioning in relation to prior work. It appears closely related to the reweighting analysis in in Eysenbach et al. (Neurips 2022), Imitating past successes can be very suboptimal. See their Eq. 4 and Proposition 3.2. This paper should be cited. Could the authors clarify what their Theorem 1 adds and position it in relation to the above ref?
The empirical results are promising but the conceptual advance in relation to prior work (Ghughare et al, TD flows and TD-HC) needs a more clear explanation. Eysenbach et al (2022) ref is a notable omission. The authors should give a clearer justification of the sampling procedure and a direct comparison with TD-HC during the discussion phase.
5 comments
Nov 13th, Response 2/3
W2 Comparison TD-HC objective
On the TD-HC objective, Appendix E.5, we conduct a study using the TD-HC objective instead of MCHC with TD-Aug. We incorporate the existing training procedure of TD-HC where we utilize randomly sampling discounts for each sample in the batch, conditioning the GHM on discounts, and also using a mixture of 0.25 proportion of the batch using horizon consistency (HC) loss and 0.75 proportion using the TD^2 loss as utilized in the prior work to reduce compounding error from bootstrapping from a model’s own samples. We then consider a general form, which we call TD-HC(), where for this experiment, we utilize 0.25 HC, proportion our loss, and then 1 - 0.25 - of TD^2. With this setup, we compare TD-HC, MCHC, and also intermediate combinations of their objectives as shown in Table 13:
| Pointmaze | GCFBC | +TD-HC (p=0) | +TD-HC (p=0.25) | +TD-HC (p=0.5) | +TD-Aug (p=1.0) |
|---|---|---|---|---|---|
medium-stitch | 55 ± 10 | 71 ± 3 | 96 ± 0 | 98 ± 0 | 99 ± 0 |
large-stitch | 26 ± 6 | 8 ± 1 | 41 ± 4 | 77 ± 7 | 97 ± 3 |
giant-stitch | 0 ± 0 | 0 ± 0 | 1 ± 0 | 6 ± 2 | 54 ± 17 |
| Average | 27 | 27 | 46 | 60 | 83 |
Where .
The right-most column reports our prior TD-Aug results, while the other columns utilize TD-HC with a varying proportion of the MCHC loss. Here, all models are reported at the same discount (0.998), corresponding to effective horizon 500. We can see that as we increase the proportion of MCHC, we substantially increase the average performance of the downstream policy (27% with TD-HC and 83% with MCHC).
In Appendix E.5, we also plot proportion vs EMD/succcess rate on pointmaze large in Figure 11, where additional MCHC loss proportion decreases distance to held-out evaluation data. We also display samples of TD-HC models in Figure 12, showing improved quality with higher MCHC proportion.
We also highlight that all models are evaluated at the same horizon on manipulation (100), while for the main comparison in pointmaze, we utilize a smaller horizon for baselines, as their EMD grows at larger horizons (500) as shown in Figure 3.
On “Results use 3 seeds”, this is correct, in our main experiments, we perform 3 training seeds, average over the last 3 checkpoints (following the standard in OGBench), and evaluate for 5 tasks * 50 episodes per task, i.e. each mean in table 1,2 is over 2250 evaluation episodes, while the std is performed over seeds (750 episodes per seed).
Nov 13th, Response 3/3
Questions
Q1 (recorded action justification): In Appendix E.3, in pointmaze environments, we find that bootstrapping with sampled actions from an approximate prior hurts performance versus using the original dataset action ’:
| Dataset | Dataset action | Learned prior |
|---|---|---|
pointmaze-large-stitch | 97 ± 3 | 64 ± 4 |
pointmaze-giant-stitch | 54 ± 14 | 7 ± 4 |
| Average | 76 | 35 |
In contrast, in manipulation environments, across both single-action models, and action-chunking, we find better performance in both cases using a learned prior:
| Dataset | h=1 (Dataset action) | h=1 (Learned prior) | h=10 (Dataset action) | h=10 (Learned prior) |
|---|---|---|---|---|
cube-double-stitch | 22 ± 2 | 51 ± 5 | 62 ± 5 | 84 ± 0 |
cube-triple-stitch | 21 ± 1 | 51 ± 2 | 25 ± 1 | 31 ± 2 |
puzzle-4x4-stitch | 1 ± 0 | 31 ± 4 | 29 ± 4 | 40 ± 2 |
| Average | 15 | 44 | 39 | 52 |
One of the reasons that the learned prior may harm performance in point, take next-action case, where is not informed by the previous behavior, as the state contains position, and no velocity information. If the sampled action moves right, while the previous action moves left, this leads to discontinuous behaviors. Additionally, in a maze, many movements are valid from each position. In contrast, on manipulation, the learned action prior can produce actions that are more consistent with the previous action, as the state itself can identify more distinct behavior (being close to a cube may imply grasping or retracting), and contains information like joint velocity, which implies prior motion.
Finally, even though we utilize future actions to bootstrap from in pointmaze, we highlight that empirically, as shown in Figure 7, MCHC, obtains similar samples to our held-out empirical samples (left column) which are an approximate reference for the true .
Q2 TD-HC at the same prediction horizon: we hope that our response to W2, which compares these methods at the same prediction horizon answers your question.
Q3 reference trajectories for Figure 3: We use the original dataset collecting policy for point in OGBench, which is a simple controller that guides the point to the next maze cell specified by a high-level planner. To collect the evaluation data, from an initial maze location, and initial action (e.g. up), we find candidate goal states, which are those that when taking the initial action (e.g. up), lie on the shortest path to the goal. Then, we conduct many rollouts from this initial state-action (e.g. center-up) with these different candidate goals. Then, we have a large pool of trajectories that share an initial-state action, and we can re-sample future states geometrically, yielding the distributions shown in Figure 7, in the directed empirical column. Thus, this empirical distribution tells us which states an oracle goal-reaching policy is likely to visit after taking a specific state-action.
Q4 Relation of Theorem 1 to prior work: We appreciate your suggestion of prior work Eysenbach et al. (Neurips 2022), which we now discuss both in Section 5.2 and Appendix C. Due to this connection, we believe it is more appropriate to rename Theorem 1 to Proposition 1. We clarify that the main purpose of this (now Proposition), is not a novel result on the relationship between policy-improvement and goal-conditioned BC, but to justify that our method TD-Aug performs policy improvement, which has not been previously realizable when we have stitching/compositional generalization gaps on offline data as discussed in Section 5.1. This, we believe our work helps a) resolves the gap highlighted in Ghughare et al., through approximating the successor measure of the mixture policy, and b) contributes a new objective, MCHC which obtains a strong approximation of the successor measure.
Thank you for your feedback. We hope this addresses your main concerns, and look forward to further discussion.
Nov 13th, Response 1/3
Thank you for your review and highlighting that our “empirical results show substantial gains over BC on navigation and manipulation tasks”, and that “policy can act without requiring the world model at test time”.
We address the weaknesses you discuss below:
W1 action bootstrap
In Appendix E.3, in pointmaze environments, we find that bootstrapping with sampled actions from an approximate prior a+∼βθ(⋅∣s+) hurts performance versus using the original dataset action a+’:
pointmaze-large-stitchpointmaze-giant-stitchIn contrast, in manipulation environments, across both single-action models, and action-chunking, we find better performance in both cases using a learned prior:
cube-double-stitchcube-triple-stitchpuzzle-4x4-stitchOne of the reasons that the learned prior may harm performance in point, take next-action case, where a’∼β(⋅∣s’) is not informed by the previous behavior, as the state contains position, and no velocity information. If the sampled action moves right, while the previous action a moves left, this leads to discontinuous behaviors. Additionally, in a maze, many movements are valid from each position. In contrast, on manipulation, the learned action prior can produce actions that are more consistent with the previous action, as the state itself can identify more distinct behavior (being close to a cube may imply grasping or retracting), and contains information like joint velocity, which implies prior motion.
We hope this clarifies “ the assumptions under which this substitution” is made.
Finally, even though we utilize future actions to bootstrap from in pointmaze, we highlight that empirically, as shown in Figure 7, MCHC, obtains similar samples to our held-out empirical samples (left column) which are an approximate reference for the true Mγβ.