acceptodds
Under review as a conference paper at ICLR 2027

Imagining Goals with Geometric Horizon Models for Policy Generalization

Abstract

Training from offline data has allowed for substantial progress in domains such as robotics, leading to general-purpose policies that can be easily applied zero-shot or efficiently finetuned for downstream tasks. However, these policies can still have poor generalization, both due to the choice of modeling objective and from learning from a static dataset. In this work, we focus on the challenging task of zero-shot goal generalization, where a policy is evaluated on unseen tasks that require combining its existing knowledge (compositional generalization). An avenue for improving a policy's generalization is by generating new experience with world models; however, such generation has proven difficult for longer horizons. Thus, to alleviate this issue, we propose TD-Aug, which samples from a geometric horizon model and allows for directly imagining novel outcomes that can be achieved by composing existing knowledge. We demonstrate that training on these future outcomes as goals for goal-conditioned BC policies significantly improves generalization in stitching-based OGBench tasks.

Primary area:
reinforcement learning
Keywords:
behavior cloning, offline rl, geometric horizon model, successor measure, temporal difference flows, data augmentation, combinatorial generalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_qMRgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This study focuses on trajectory stitching within the context of goal-conditioned behavior cloning for policy generalization. The proposed TD-Aug uses a flow-based geometric horizon world model to generate additional goals. The model predicts future states under a mixture policy, which can switch between data gathering behaviors at each state. This allows for joining behaviors across separate trajectories. The proposed MCHC training objective adapts prior TD horizon consistency (TD-HC) combining observed future state targets with TD bootstrapping. The empirical results show substantial gains over BC on navigation and manipulation tasks that require stitching for attaining success.

Critical strengths
  1. I find the idea of using GHM to relabel imagined goals neat. The trained policy can act without requiring the world model at test time.

  2. The empirical gains over BC appear to be consistent across several settings. The MC-Aug control is useful and suggests that the gains arise from the stitched futures, not simply from more synthetic goals.

Critical weaknesses
  1. It is not clear if the method learns the future state distribution under the mixture policy as claimed. Sec 3.4 replaces sampling from β(⋅∣s′)\beta(\cdot|s') with next action from the same recorded trajectory (line 220-222). But this conditional distribution need not equal β(a′∣s′)\beta(a'|s') which conditions only on the current state. The authors should clarify the assumptions under which this substitution makes sense.

  2. The proposed MCHC objective modifies TD-HC but does not compare against it. Some controls also use different prediction horizons making it difficult to compare the different obnectives. Results use 3 seeds reporting std and no intervals.

While Theorem 1 supports an idealized version of the proposed method, it does not settle these practical considerations.

Questions for authors
  1. Why is using recorded next actions for learning the mixture policy distribution justified?

  2. How does your approach compare against the original TD-HC objective under the same prediction horizon?

  3. Which policy was used to generate the reference trajectories in Fig. 3?

  4. Theorem 1 needs clear positioning in relation to prior work. It appears closely related to the reweighting analysis in in Eysenbach et al. (Neurips 2022), Imitating past successes can be very suboptimal. See their Eq. 4 and Proposition 3.2. This paper should be cited. Could the authors clarify what their Theorem 1 adds and position it in relation to the above ref?

The empirical results are promising but the conceptual advance in relation to prior work (Ghughare et al, TD flows and TD-HC) needs a more clear explanation. Eysenbach et al (2022) ref is a notable omission. The authors should give a clearer justification of the sampling procedure and a direct comparison with TD-HC during the discussion phase.

5 comments
AuthorsOpenReview ↗

Nov 13th, Response 1/3

Thank you for your review and highlighting that our “empirical results show substantial gains over BC on navigation and manipulation tasks”, and that “policy can act without requiring the world model at test time”.

We address the weaknesses you discuss below:

W1 action bootstrap

In Appendix E.3, in pointmaze environments, we find that bootstrapping with sampled actions from an approximate prior a+∼βθ(⋅∣s+)a_+ \sim \beta_\theta(\cdot | s_+) hurts performance versus using the original dataset action a+a_+’:

DatasetDataset actionLearned prior
pointmaze-large-stitch97 ± 364 ± 4
pointmaze-giant-stitch54 ± 147 ± 4
Average7635

In contrast, in manipulation environments, across both single-action models, and action-chunking, we find better performance in both cases using a learned prior:

Dataseth=1 (Dataset action)h=1 (Learned prior)h=10 (Dataset action)h=10 (Learned prior)
cube-double-stitch22 ± 251 ± 562 ± 584 ± 0
cube-triple-stitch21 ± 151 ± 225 ± 131 ± 2
puzzle-4x4-stitch1 ± 031 ± 429 ± 440 ± 2
Average15443952

One of the reasons that the learned prior may harm performance in point, take next-action case, where a’∼β(⋅∣s’)a’ \sim \beta(\cdot | s’) is not informed by the previous behavior, as the state contains position, and no velocity information. If the sampled action moves right, while the previous action aa moves left, this leads to discontinuous behaviors. Additionally, in a maze, many movements are valid from each position. In contrast, on manipulation, the learned action prior can produce actions that are more consistent with the previous action, as the state itself can identify more distinct behavior (being close to a cube may imply grasping or retracting), and contains information like joint velocity, which implies prior motion.

We hope this clarifies “ the assumptions under which this substitution” is made.

Finally, even though we utilize future actions to bootstrap from in pointmaze, we highlight that empirically, as shown in Figure 7, MCHC, obtains similar samples to our held-out empirical samples (left column) which are an approximate reference for the true MγβM^\beta_\gamma.

AuthorsOpenReview ↗

Nov 13th, Response 2/3

W2 Comparison TD-HC objective

On the TD-HC objective, Appendix E.5, we conduct a study using the TD-HC objective instead of MCHC with TD-Aug. We incorporate the existing training procedure of TD-HC where we utilize randomly sampling discounts for each sample in the batch, conditioning the GHM on discounts, and also using a mixture of 0.25 proportion of the batch using horizon consistency (HC) loss and 0.75 proportion using the TD^2 loss as utilized in the prior work to reduce compounding error from bootstrapping from a model’s own samples. We then consider a general form, which we call TD-HC(pMCHCp_\text{MCHC}), where for this experiment, we utilize 0.25 HC, proportion pMCHCp_\text{MCHC} our loss, and then 1 - 0.25 - pMCHCp_\text{MCHC} of TD^2. With this setup, we compare TD-HC, MCHC, and also intermediate combinations of their objectives as shown in Table 13:

PointmazeGCFBC+TD-HC (p=0)+TD-HC (p=0.25)+TD-HC (p=0.5)+TD-Aug (p=1.0)
medium-stitch55 ± 1071 ± 396 ± 098 ± 099 ± 0
large-stitch26 ± 68 ± 141 ± 477 ± 797 ± 3
giant-stitch0 ± 00 ± 01 ± 06 ± 254 ± 17
Average2727466083

Where p=pMCHCp = p_{\text{MCHC}}.

The right-most column reports our prior TD-Aug results, while the other columns utilize TD-HC with a varying proportion of the MCHC loss. Here, all models are reported at the same discount (0.998), corresponding to effective horizon 500. We can see that as we increase the proportion of MCHC, we substantially increase the average performance of the downstream policy (27% with TD-HC and 83% with MCHC).

In Appendix E.5, we also plot proportion vs EMD/succcess rate on pointmaze large in Figure 11, where additional MCHC loss proportion decreases distance to held-out evaluation data. We also display samples of TD-HC models in Figure 12, showing improved quality with higher MCHC proportion.

We also highlight that all models are evaluated at the same horizon on manipulation (100), while for the main comparison in pointmaze, we utilize a smaller horizon for baselines, as their EMD grows at larger horizons (500) as shown in Figure 3.

On “Results use 3 seeds”, this is correct, in our main experiments, we perform 3 training seeds, average over the last 3 checkpoints (following the standard in OGBench), and evaluate for 5 tasks * 50 episodes per task, i.e. each mean in table 1,2 is over 2250 evaluation episodes, while the std is performed over seeds (750 episodes per seed).

AuthorsOpenReview ↗

Nov 13th, Response 3/3

Questions

Q1 (recorded action justification): In Appendix E.3, in pointmaze environments, we find that bootstrapping with sampled actions from an approximate prior a+∼βθ(⋅∣s+)a_+ \sim \beta_\theta(\cdot | s_+) hurts performance versus using the original dataset action a+a_+’:

DatasetDataset actionLearned prior
pointmaze-large-stitch97 ± 364 ± 4
pointmaze-giant-stitch54 ± 147 ± 4
Average7635

In contrast, in manipulation environments, across both single-action models, and action-chunking, we find better performance in both cases using a learned prior:

Dataseth=1 (Dataset action)h=1 (Learned prior)h=10 (Dataset action)h=10 (Learned prior)
cube-double-stitch22 ± 251 ± 562 ± 584 ± 0
cube-triple-stitch21 ± 151 ± 225 ± 131 ± 2
puzzle-4x4-stitch1 ± 031 ± 429 ± 440 ± 2
Average15443952

One of the reasons that the learned prior may harm performance in point, take next-action case, where a’∼β(⋅∣s’)a’ \sim \beta(\cdot | s’) is not informed by the previous behavior, as the state contains position, and no velocity information. If the sampled action moves right, while the previous action aa moves left, this leads to discontinuous behaviors. Additionally, in a maze, many movements are valid from each position. In contrast, on manipulation, the learned action prior can produce actions that are more consistent with the previous action, as the state itself can identify more distinct behavior (being close to a cube may imply grasping or retracting), and contains information like joint velocity, which implies prior motion.

Finally, even though we utilize future actions to bootstrap from in pointmaze, we highlight that empirically, as shown in Figure 7, MCHC, obtains similar samples to our held-out empirical samples (left column) which are an approximate reference for the true MγβM^\beta_\gamma.

Q2 TD-HC at the same prediction horizon: we hope that our response to W2, which compares these methods at the same prediction horizon answers your question.

Q3 reference trajectories for Figure 3: We use the original dataset collecting policy for point in OGBench, which is a simple controller that guides the point to the next maze cell specified by a high-level planner. To collect the evaluation data, from an initial maze location, and initial action (e.g. up), we find candidate goal states, which are those that when taking the initial action (e.g. up), lie on the shortest path to the goal. Then, we conduct many rollouts from this initial state-action (e.g. center-up) with these different candidate goals. Then, we have a large pool of trajectories that share an initial-state action, and we can re-sample future states geometrically, yielding the distributions shown in Figure 7, in the directed empirical column. Thus, this empirical distribution tells us which states an oracle goal-reaching policy is likely to visit after taking a specific state-action.

Q4 Relation of Theorem 1 to prior work: We appreciate your suggestion of prior work Eysenbach et al. (Neurips 2022), which we now discuss both in Section 5.2 and Appendix C. Due to this connection, we believe it is more appropriate to rename Theorem 1 to Proposition 1. We clarify that the main purpose of this (now Proposition), is not a novel result on the relationship between policy-improvement and goal-conditioned BC, but to justify that our method TD-Aug performs policy improvement, which has not been previously realizable when we have stitching/compositional generalization gaps on offline data as discussed in Section 5.1. This, we believe our work helps a) resolves the gap highlighted in Ghughare et al., through approximating the successor measure of the mixture policy, and b) contributes a new objective, MCHC which obtains a strong approximation of the successor measure.

Thank you for your feedback. We hope this addresses your main concerns, and look forward to further discussion.

Reviewer_xRfDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Building on the goal-augmentation-and-BC framework of Ghugare et al. (2024), this paper proposes TD-Aug, a new GHM-based goal-augmentation method for improving performance on compositional tasks requiring trajectory stitching. It introduces the MCHC objective, combining Monte Carlo supervision with temporal-difference bootstrapping to improve long-horizon goal generation, and trains BC policies on a mixture of real and generated goals. Experiments on OGBench demonstrate substantial improvements over the corresponding BC baselines across state-based navigation, visual navigation, and manipulation stitching tasks.

Critical strengths
  1. Effective GHM-based goal augmentation. Building on the existing goal-augmentation-and-BC framework, TD-Aug uses GHMs to generate goals supporting cross-trajectory stitching. Its MCHC objective combines Monte Carlo supervision with TD bootstrapping, improving long-horizon generation over the evaluated MC and TD² baselines.

  2. Solid empirical evaluation. Experiments across state-based navigation, visual navigation, and manipulation show substantial improvements over the corresponding BC baselines on stitching tasks. The evaluation covers both generation quality and downstream policy performance, with several goal-augmentation and offline RL baselines, providing convincing evidence of the method’s effectiveness in the studied settings.

Critical weaknesses

Although MCHC is directly derived from the existing TD-HC objective, the experiments compare it against MC and TD² without evaluating TD-HC within the same goal-augmentation pipeline. A direct comparison would help establish whether the proposed modification offers meaningful improvements in performance, stability, or computational efficiency over its closest predecessor, TD-HC. Without this comparison, the incremental benefit of MCHC over the existing TD-HC objective remains insufficiently established.

Questions for authors

For the variants that bootstrap using the next action from the same dataset trajectory (Section 3.4), what justifies treating this action as a sample from the Markovian mixture policy β(⋅∣s′)\beta(\cdot\mid s')? When trajectories are collected under fixed goals or contexts, the paired next action follows pD(a′∣s,a,s′)p_D(a'\mid s, a, s'), which need not equal β(a′∣s′)\beta(a'\mid s'). Could the authors clarify whether this potential discrepancy affects the interpretation of the learned model as an approximation of MγβM_\gamma^\beta?

4 comments
AuthorsOpenReview ↗

Nov 13th, Response 1/1

Thank you for reviewing and highlighting our “effective goal augmentation [method]” and that ”OGBench demonstrate[s] substantial improvements”. We address each of your weaknesses below:

W1: Comparison TD-HC objective

In Appendix E.5, we conduct a study using the TD-HC objective instead of MCHC with TD-Aug. We incorporate the existing training procedure of TD-HC where we utilize randomly sampling discounts for each sample in the batch, conditioning the GHM on discounts, and also using a mixture of 0.25 proportion of the batch using horizon consistency (HC) loss and 0.75 proportion using the TD^2 loss as utilized in the prior work to reduce compounding error from bootstrapping from a model’s own samples. We then consider a general form, which we call TD-HC(pMCHCp_\text{MCHC}), where for this experiment, we utilize 0.25 HC, proportion pMCHCp_\text{MCHC} our loss, and then 1 - 0.25 - pMCHCp_\text{MCHC} of TD^2. With this setup, we compare TD-HC, MCHC, and also intermediate combinations of their objectives as shown in Table 13:

PointmazeGCFBC+TD-HC (p=0)+TD-HC (p=0.25)+TD-HC (p=0.5)+TD-Aug (p=1.0)
medium-stitch55 ± 1071 ± 396 ± 098 ± 099 ± 0
large-stitch26 ± 68 ± 141 ± 477 ± 797 ± 3
giant-stitch0 ± 00 ± 01 ± 06 ± 254 ± 17
Average2727466083

Where p=pMCHCp = p_{\text{MCHC}}.

The right-most column reports our prior TD-Aug results, while the other columns utilize TD-HC with a varying proportion of the MCHC loss. We can see that as we increase the proportion of MCHC, we substantially increase the average performance of the downstream policy (27% with TD-HC and 83% with MCHC).

In Appendix E.5, we also plot proportion vs EMD/succcess rate on pointmaze large in Figure 11, where additional MCHC loss proportion decreases distance to held-out evaluation data. We also display samples of TD-HC models in Figure 12, showing improved quality with higher MCHC proportion.

On your question, Q1: action bootstrap: In Appendix E.3, in pointmaze environments, we find that bootstrapping with sampled actions from an approximate prior a+∼βθ(⋅∣s+)a_+ \sim \beta_\theta(\cdot | s_+) hurts performance versus using the original dataset action a+a_+’:

DatasetDataset actionLearned prior
pointmaze-large-stitch97 ± 364 ± 4
pointmaze-giant-stitch54 ± 147 ± 4
Average7635

In contrast, in manipulation environments, across both single-action models, and action-chunking, we find better performance in both cases using a learned prior:

Dataseth=1 (Dataset action)h=1 (Learned prior)h=10 (Dataset action)h=10 (Learned prior)
cube-double-stitch22 ± 251 ± 562 ± 584 ± 0
cube-triple-stitch21 ± 151 ± 225 ± 131 ± 2
puzzle-4x4-stitch1 ± 031 ± 429 ± 440 ± 2
Average15443952

One of the reasons that the learned prior may harm performance in point, take next-action case, where a’∼β(⋅∣s’)a’ \sim \beta(\cdot | s’) is not informed by the previous behavior, as the state contains position, and no velocity information. If the sampled action moves right, while the previous action aa moves left, this leads to discontinuous behaviors. Additionally, in a maze, many movements are valid from each position. In contrast, on manipulation, the learned action prior can produce actions that are more consistent with the previous action, as the state itself can identify more distinct behavior (being close to a cube may imply grasping or retracting), and contains information like joint velocity, which implies prior motion. Finally, even though we utilize future actions to bootstrap from in pointmaze, we highlight that empirically, as shown in Figure 7, MCHC, obtains similar samples to our held-out empirical samples (left column) which are an approximate reference for the true MγβM^\beta_\gamma.

Thank you for your feedback. We hope this addresses your main concerns, and look forward to further discussion.

Reviewer_xRfDOpenReview ↗

Rebuttal Response

Thank you for the detailed response and additional experiments. The TD-HC comparison addresses W1. Q1 remains partially addressed: the ablations support the empirical choice of dataset actions, but do not clarify the potential discrepancy between the distribution learned by bootstrapping with same-trajectory actions and the successor measure MγβM_\gamma^\beta of the defined Markovian mixture policy. I maintain my weak accept recommendation.

AuthorsOpenReview ↗

Nov 18 Response

Thank you for your continued engagement, we have added an additional empirical experiment aimed at clarifying the discrepancy between the approximation using the learned dataset action, and that of the Markovian mixture policy, which is contained in Appendix E.8 (to maintain table numbering). Where we discuss that:

In prior sections, we compute EMD using a directed dataset, capturing how well a GHM captures action-directed behavior. Here, we collect an additional exploratory reference dataset on pointmaze-larg}, where after taking an initial action, runs the data collecting controller until 10001000 steps, repeatedly towards random adjacent maze cell centers. This aims more closely resemble the behavior of the mixture policy, and the true MβM^\beta, than the directed dataset. We display samples comparing these two empirical distributions, as well as samples the original MCHC using dataset actions, and MCHC with a learned prior in Figure 13. We also list numerical results across different start positions, in Table 17, shown below. We can see that using the learned prior in the MCHC loss leads to being closer in distribution to the exploratory samples, which evenly spread from the initial state. However, utilizing the next-action in the dataset leads to better directed EMD, and higher downstream policy success rate. In summary, we believe that MCHC, with next-action bootstrapping, trades off some deviation from β\beta's behavior with committing to similar behavior multiple steps, such as moving right several steps. This can be more useful for inferring how an action helps reach goals, as reflected in policy success rate.

Table 17:

pointmaze-largeMCHC (dataset action)MCHC (learned prior)
Success (%) ↑97 ± 364 ± 4
Directed EMD ↓9.09 ± 0.5610.16 ± 0.12
Explore EMD ↓5.16 ± 0.254.17 ± 0.09

We appreciate the feedback and hope that this helps further address your question.

Thank you for the additional experiment and clarification. The response acknowledges the discrepancy raised in Q1 and provides additional empirical evidence, together with a plausible explanation for why the biased variant can be useful for goal augmentation. I maintain my weak accept recommendation.

Reviewer_fTeGOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses zero-shot compositional generalization for goal-conditioned behavior cloning on OGBench stitching tasks. It uses a geometric horizon model (GHM) — a generative model of the successor measure of the data-collecting mixture policy β\beta — as a goal augmenter: draw a future state s+∼Mβ(⋅∣s,a)s^+\sim M^\beta(\cdot\mid s,a) and train the BC policy to reach it, mixing imagined and real goals with proportion α=0.5\alpha=0.5 (Eqs. 11–12). The technical contribution is a new GHM objective, MCHC-β (Eq. 9), which takes the horizon-consistent TD-Flow loss of Farebrother et al. (2026) and replaces the on-policy bootstrap with a Monte-Carlo sample s+=st+k+1s^+=s_{t+k+1}, k∼geom(1−ω)k\sim\mathrm{geom}(1-\omega), so that the model mixes dense ground-truth targets with TD targets that can stitch across trajectories while remaining trainable at a large discount γ\gamma. An appendix proof relates the augmentation at α=1\alpha=1 with exact MβM^\beta samples to value-weighted one-step improvement over β\beta, since E_{s⁺∼d^β}[Q^β(s,a,s⁺)/d^β(s⁺)·log π] = (1−γ)⁻¹ E_{s⁺∼M^β}[log π].

It is evaluated on pointmaze-stitch, visual-antmaze-stitch and three manipulation tasks. Reported gains are 31 → 79 averaged over four pointmaze tasks (best external baseline 58), with pointmaze-giant-stitch going 0 → 54 ± 17 where every other method scores 0 except the clustering heuristic at 3 ± 6; 29 → 56 on visual-antmaze; and 13 → 52 on manipulation, above the 44 obtained from the full play dataset. Figure 3 reports earth-mover distance between generated and empirical samples, with MCHC-β lowest at every horizon tested.

Critical strengths

S1. A task on which every baseline fails is solved, and the diagnostic that predicted it is in the paper. On pointmaze-giant-stitch, Table 1 gives 0 for GCFBC, TD-Aug2, MC-Aug, GCIQL, CRL and HIQL and 3 ± 6 for the clustering heuristic, while TD-Aug reaches 54 ± 17. Figure 3 is the corresponding diagnostic and the strongest evidence in the paper: TD²-β starts at EMD 7.2 at horizon 100 and reaches 12.1 ± 1.8 at horizon 500, the one-step models go 8.8 → 12.0 and 4.6 → 10.3, and MCHC-β is lowest at every horizon (4.4 → 6.4). The objective was built to fix exactly that decay, and the gain shows up on the task that needs long-horizon stitching. This is the finding I would cite from the paper.

S2. The manipulation section reports the right ceiling, and the augmentation beats it. Table 3 puts the play dataset next to the stitch dataset, so a reader can see what performance looks like when long-horizon data exists at all: GCFBC scores 44 on play and 13 on stitch, and TD-Aug on stitch reaches 52. Training on deliberately restricted short-trajectory data with imagined goals outperforms training on the full dataset, on average. Appendix E extends the same generated goals to CRL's critic (4 → 36), which makes the case that this is a reusable augmentation rather than a single-pipeline trick.

Critical weaknesses

W1. The advantage over a geometric reachability heuristic is measured only where that heuristic is structurally weakest. The title claims "policy generalization" and the conclusion that TD-based GHMs "allow for enhanced generalization for behavior cloning", both unrestricted; the abstract's closing "in stitching-based OGBench tasks" says where the demonstration ran, not what it covers. Every experiment is in an OGBench maze, where two nearby states are often separated by a wall — exactly where a proximity heuristic breaks; there is no non-stitch task, no other OGBench domain, no real data. In the open continuous setting the conclusion points at, being close usually does mean being reachable, so a clustering stitcher could match TD-Aug. The one wall-free domain, manipulation, is precisely where Cluster-Aug, MC-Aug and Dyn-Aug were not run (Table 3); running them there is two-sided evidence — if one matches 52 the mechanism claim narrows to mazes, and if all fall far short, that is the paper's strongest result. This needs a re-run, not a sentence.

W2. "Outperforms prior offline GCRL methods" (§6.1) is not checkable as presented: the comparison rests on a dataset change the paper never quantifies, and in manipulation on a different action interface.

§6.1 re-collected the OGBench stitch datasets and says "on new datasets, baselines perform similarly, or slightly better than before" — with no numbers. The four external columns carry mean ± std, consistent with re-training, but the paper does not say so; if any is taken from OGBench on the original datasets, that column and the authors' rows are on different protocols and the claimed 21-point margin (79 vs QRL 58) does not hold — and giant-stitch's 0 → 54 ± 17 is exactly where QRL already reaches 54 ± 9. Decisively, the comparability of the two protocols is asserted and never measured. Separately, Table 3 states the chunking used for GCFBC (h=10h=10) but not for GCIQL and HIQL; with stitch scores of 29 and 2, consistent with agents that cannot emit a chunk, the 52 / 29 / 2 ordering mixes method with interface. Both halves need experiments, not clarification: baseline scores on both dataset versions, and GCIQL/HIQL re-run under the same chunking interface.

Additional concerns.

  • The visual number is the weakest of the three. 29% → 56% averages two tasks where the base AE-GCFBC (29) is below the standard visual BC baseline GCBC (45) and the result (56) ties CRL and sits 13 below HIQL (69). §6.2 states this correctly; the abstract does not. Applying TD-Aug to GCBC is possible already — Appendix E decodes generated latents to pixels for CRL — and would settle it.
  • A hand-designed heuristic beats two of the three learned generators. Cluster-Aug — k-means with no learned model — averages 50 on pointmaze, above Dyn-Aug (46) and TD-Aug2 (45). Only MCHC-β clears it (79).
  • RidR_{\mathrm{id}} is defined and never reported. Eq. 10 introduces in-distribution success and no table contains it, although Theorem 1 makes the method one-step improvement over β\beta and therefore predicts a trade-off.
  • "Significantly" has no test. §6.1 and the abstract use the word, but no paired test or confidence interval appears; the reported spread is across 3 seeds only.
  • Averages hide per-task regressions. On cube-triple, TD-Aug (31 ± 2) is below both the unaugmented play baseline (35 ± 5) and TD-Aug2 (36 ± 5), so "closes the gap" (52 vs 44) holds only on the three-task average.
Questions for authors

My rating turns on W1, W2 and Q1–Q3. Q4–Q8 are narrower; Q4 is a single missing column.

  1. How are evaluation goals chosen? Eq. 10 defines RgenR_{\mathrm{gen}} as an expectation over Mβ(⋅∣s)M^\beta(\cdot\mid s), while the augmentation trains on samples from M^β\hat{M}^\beta. Are the evaluation goals in Tables 1–3 OGBench's predefined tasks, how many per task, and are they shared across methods? Since training goals come from M^β\hat{M}^\beta, please also state how close the two sets are — otherwise the gains mix "this supervision was missing" with "these goals were effectively seen".

  2. Baseline provenance. Were GCIQL, QRL, CRL and HIQL retrained on the re-collected stitch datasets, or are those numbers from OGBench on the original datasets? Please report at least GCFBC and one external baseline on both versions of medium-stitch and giant-stitch, so the effect of the re-collection is visible.

  3. The visual base. Appendix E already decodes generated latents to pixels for CRL. Can you apply TD-Aug to GCBC (45) the same way, rather than to AE-GCFBC (29)? If the augmented GCBC also reaches 56, the gain belongs to the augmentation; if it stays near 45, it belongs to the architecture. Please also state in §6.2 that 56 remains below HIQL's 69 and level with CRL's 56.

  4. RidR_{\mathrm{id}}. Please report in-distribution success next to RgenR_{\mathrm{gen}} for GCFBC and TD-Aug in all three domains. How large is the trade-off that Theorem 1 implies?

  5. Figure 3. What is the EMD between two independent held-out empirical sets of the same size, and how many points are used per estimate? Relatedly, MC's error grows about as fast as MCHC's (+56% vs +45% from H=100H=100 to H=500H=500) and is already worse at H=100H=100, where no stitching is needed — suggesting a better short-horizon fit rather than a better stitching mechanism. Can you plot policy success against EMD across methods and horizons?

  6. Truncation in the MC target. With ω=0.99\omega=0.99 the mean jump is 99 environment steps, but §6.1 describes trajectory length in cells ("travel up to 4 maze cells") and never gives the conversion, so it is unclear what fraction of the MC targets fall beyond a trajectory end. What is the mean trajectory length in steps, and when kk exceeds it is s+s^+ clamped, resampled, or dropped? This sets the size and sign of the bias that §4 describes only as growing with ω\omega.

  7. α\alpha and Theorem 1. Theorem 1 is stated at α=1\alpha=1 with exact samples, while every experiment uses α=0.5\alpha=0.5 (Table 5). Why not run the regime the theory describes? Does the ordering TD-Aug > TD-Aug2 > MC-Aug hold at other values of α\alpha?

  8. The clustering heuristic. §1 says the study demonstrates "that TD methods are crucial for generalization", yet a k-means rule with no learned model (Cluster-Aug, 50) beats two of the three learned generators. What does that imply for the claim?

4 comments
AuthorsOpenReview ↗

Nov 13th, Response 1/2

Thank you for your detailed review and highlighting that our method helps with “long-horizon stitching” and is a “reusable augmentation” strategy. We will address your main concerns:

W1: Baselines in Manipulation, “Cluster-Aug, MC-Aug and Dyn-Aug were not run (Table 3)”

We run these additional baselines in the manipulation setting, matching the action-chunking setup of GCFBC and TD-Aug:

DatasetDataGCFBC+Cluster-Aug+Dyn-Aug+MC-Aug+TD-Aug2+TD-AugGCIQLHIQL
cube-doublestitch20 ± 028 ± 281 ± 221 ± 178 ± 284 ± 063 ± 16 ± 2
cube-triplestitch20 ± 011 ± 228 ± 220 ± 036 ± 531 ± 28 ± 20 ± 0
puzzle-4x4stitch0 ± 014 ± 18 ± 10 ± 018 ± 040 ± 216 ± 50 ± 0
Averagestitch131739144452292

We can see that TD-Aug still improves over each of these baselines. We can see that +Cluster-Aug struggles here, as it is more challenging to meaningfully group states in higher-dimensional state spaces, motivating the use of generative models. Similarly, MC-Aug does not stitch across trajectories, obtaining almost identical performance to base GCFBC.

W2: GCRL evaluation: change from re-collected datasets, evaluation protocol, action-chunking

First, we reiterate that all offline RL baselines are re-run on our collected datasets, using the same protocol as our methods. Additionally, we show the performance change of offline RL baselines from this process in Table 8 in Appendix E.1:

pointmazeGCIQL (ours)GCIQL (original)QRL (ours)QRL (original)CRL (ours)CRL (original)HIQL (ours)HIQL (original)
medium-stitch74 ± 121 ± 988 ± 580 ± 124 ± 20 ± 171 ± 874 ± 6
large-stitch40 ± 131 ± 289 ± 1584 ± 150 ± 00 ± 018 ± 413 ± 6
giant-stitch0 ± 00 ± 054 ± 950 ± 80 ± 00 ± 00 ± 00 ± 0
teleport-stitch27 ± 125 ± 32 ± 49 ± 510 ± 54 ± 330 ± 334 ± 4
Average35195856413030

We can see the performance on average is improved for GCIQL, slightly better for QRL, and CRL, and on average the same for HIQL on our datasets. Thus, we can see that our evaluation protocol is fair for these methods.

On chunking for offline RL baselines in Table 3, we reiterate our main focus and comparison in this work is on improving behavior cloning. However, in Appendix E.4 we study using action-chunking with critics for a more direct comparison. To do this, we utilize Q-chunking (QC) (Li et al., 2026), which is a method specifically tailored to use action-chunking, thus is a strong baseline. As we describe in the Appendix, Q-chunking learns action-chunk critic Q, as well as a behavior policy proposal βθ(at:t+h∣st)\beta_\theta(a_{t:t+h} | s_t), and at evaluation-time, uses best-of-N, sampling N actions from the prior, and executes the best scored by Q, which we label QC (β\beta). Then, we construct alternative proposals instead of βθ\beta_\theta, where we utilize GCFBC policy, or a policy trained with TD-Aug. Finally, we also construct a mixed proposal utilizing 50% samples from the prior and 50% samples from the TD-Aug policy, displaying results in Table 12:

Dataset (stitch)QC (β)QC (GCFBC)QC (TD-Aug)QC (mixed)
cube-double65 ± 218 ± 182 ± 382 ± 1
cube-triple41 ± 417 ± 137 ± 147 ± 3
puzzle-4x499 ± 00 ± 098 ± 2100 ± 0
Average69127276

First, we can see that the GC policy trained with TD-Aug provides a slightly better proposal on average, and mixing proposals of TD-Aug with βθ\beta_\theta leads to the strongest performance. A way we interpret this, is our method provides good pretraining, both a strong zero-shot policy, and good prior for RL improvement, illustrated here with best-of-N.

AuthorsOpenReview ↗

Nov 13th, Response 2/2

We also answer several of your questions:

Q1,Q4: For our main evaluation, we follow the standard OGBench procedure which uses out-of-distribution goals, picking 5 tasks, and 50 randomized episodes per task. This leads to 250 evaluation episodes for each method in each environment, and then averaged over 3 checkpoints and 3 seeds. We illustrate the difference between training and evaluation in several places. In Figure 2 and Figure 6, where we contrast the horizon of the shorter training tasks (at most 4 maze cells) and long-horizon evaluation tasks, spanning across the maze.

Then, in new results, in Appendix E.2, including Table 9 we report the difference between ID atomic training tasks (pushing one button) and long-horizon tasks requiring many atomic tasks (such as solving a puzzle):

DatasetGCFBC (atomic, ID)GCFBC (stitch, OOD)+TD-Aug (atomic, ID)+TD-Aug (stitch, OOD)
cube-double97 ± 120 ± 099 ± 084 ± 1
cube-triple94 ± 220 ± 098 ± 032 ± 1
puzzle-4x4100 ± 00 ± 099 ± 042 ± 0
Average97139953

We can see that GCFBC and TD-Aug obtain similar ID results, but TD-Aug obtains substantially higher OOD generalization.

This is also illustrated in a new visualization, where in Figure 10 in pointmaze-{large,giant}, where we conduct an additional evaluation, where training tasks ( <= 4 maze cells), GCFBC performs well, but struggles as the evaluation horizon increases, while TD-Aug retains good performs at long-horizon goals.

Q3: In visual maze, our experiment design already isolates the improvement specifically from the augmentation (TD-Aug) instead as +TD-Aug uses the same architecture AE-GCFBC (in the first column). We specifically design this setup (Figure 4), to avoid reconstruction, which we view as a more scalable path to handling higher-dimensional observations.

Q5: We compute the EMD between two sets of size 2048 each, and then average over 8 initial maze regions. We would also push back on that “H=100, where no stitching is needed”. In Figure 7, we show held-out, empirical samples we use to compute EMD. We can see that even in the first row (discount=0.99, corresponding to expected H = 100), that the model needs to stitch across trajectories as shown in the black samples. We can see that the blue samples from MC, fall short of the necessary horizon & do not stitch, while MCHC performs better, and stitches, at only H=100, which is also reflected in the EMD.

Q7: In Appendix E.6 we add a comparison over different augmentation proportions, α∈{0.25,0.5,0.75}\alpha \in \{0.25, 0.5, 0.75\}, and we find that overall TD-Aug performs well across these values.

Q8: Clustering is a stronger baseline in a lower-dimensional environment like point, but struggles in higher-dimensional manipulation environments as compared to generative methods which we show in the updated Table 3.

Li, Q., Zhou, Z. P., & Levine, S. (2026). Reinforcement learning with action chunking. Advances in Neural Information Processing Systems, 38, 55518-55553.

Thank you for your feedback. We hope this addresses your main concerns, and look forward to future discussion.

Reviewer_fTeGOpenReview ↗

Post-rebuttal assessment: both critical weaknesses resolved

Post-rebuttal assessment. Both critical weaknesses were answered with the experiments they asked for, and both answers went against me. What changed is below, then what remains, then the scores.

Resolved, and recorded in my updated review.

  • W1 (the augmentation is measured only where a geometric reachability heuristic is structurally weakest). Withdrawn. You ran all three baselines in manipulation under the same action-chunking setup, and none comes close: Cluster-Aug 17, MC-Aug 14, Dyn-Aug 39 against TD-Aug 52 on the three-task average — 34 points for the clustering heuristic I expected to be competitive. My hypothesis that a proximity rule would match you in a wall-free domain is refuted.
  • W2, provenance half. Withdrawn. Table 8 gives both versions of every baseline (GCIQL 35 vs 19, QRL 58 vs 56, CRL 4 vs 1, HIQL 30 vs 30), and the four external columns of Table 1 are exactly the "ours" columns, so no column mixes protocols. The pointmaze margin holds, and it is 23 points against the original-dataset numbers rather than smaller — my worry ran the other way.
  • W2, interface half. Resolved, with a better control than the one I proposed. QC(TD-Aug) 72 and QC(mixed) 76 against QC(β) 69 is a real result: the augmented policy is a stronger proposal for a chunked RL agent than that agent's own prior.
  • Q4 (in-distribution success). Table 9 answers it: 97 → 99 in-distribution against 13 → 53 OOD. The trade-off Theorem 1 might have implied does not appear.
  • Q1, Q2, Q7, Q8. Answered — the protocol (5 tasks × 50 episodes, 3 checkpoints and 3 seeds), the provenance, the α sweep, and the clustering heuristic's weakness in higher-dimensional state spaces.
  • One of my own inferences. Withdrawn. I claimed MC's shortfall already at H = 100 reflects only a better short-horizon fit. Figure 7 shows that at γ = 0.99 the directed empirical targets already run outside a single ≤4-cell trajectory, so my premise was wrong.

Remaining. Three items, none of which needs a re-run.

  1. The truncation rule for the MC target (Q6, still unanswered). TD-Aug is trained at ω = 0.998, a mean jump of 499 environment steps, from trajectories that travel up to 4 maze cells. When k runs past the end of the trajectory, is s⁺ clamped to the terminal state, resampled, or dropped — and what is the mean trajectory length in steps? This is not bookkeeping: clamping and dropping both shorten the effective horizon of L_MC, which is the quantity Figure 3 attributes the MC shortfall to, and the blue samples in Figure 7 falling short is consistent with either explanation.
  2. Tables 3 and 9 disagree on two TD-Aug cells (cube-triple 31 ± 2 against 32 ± 1; puzzle-4x4 40 ± 2 against 42 ± 0; the average changes from 52 to 53). Please reconcile, and say which is the reported result.
  3. Framing on manipulation. With QC(β) = 69 now in E.4, §6.3's 52 reads as a best-in-class-BC result rather than a closed gap. §6.1's GCRL claim is correctly scoped to pointmaze, so this is wording, not a contradiction — and stating it alongside QC(TD-Aug) = 72 and QC(mixed) = 76 would make the section stronger. Two smaller ones: the EMD noise floor for Figure 3 is still not given, and §6.2 does not yet say that 56 is level with CRL and 13 below HIQL.

On Q3 I accept the fixed-architecture argument; GCBC + TD-Aug would still be worth one sentence saying it was not run.

Scores. Soundness 2, Presentation 2, Informativeness 2, Confidence 3, and Rating 4 (clear acceptance), up from 3. The two weaknesses that determined my rating were both answered with the experiments they asked for. What keeps Soundness at 2 is Q6 alone; once the truncation rule and the Tables 3/9 reconciliation are in the revision, I have no substantive concern left.

AuthorsOpenReview ↗

Nov 18 Response

We appreciate your feedback, and we are happy to clarify some final questions.

Q9: truncation and dataset statistics: During geometric sampling, kk is truncated to the terminal state in the trajectory for all methods, matching the prior baselines in OGBench. We have also added statistics for all datasets in our paper to Table 15, for example, on pointmaze:

datasetEpisodesMean episode length
pointmaze-medium-stitch12,18081.1
pointmaze-large-stitch11,51585.9
pointmaze-giant-stitch11,59985.2

Q10: “Tables 3 and 9 disagree”: For all results in the main paper (Table 1,2,3), we report results averaged from the last three model checkpoints. For example, when training a policy for 1 million steps, this is average over checkpoints 800k,900k, 1 mil, which is the standard OGBench evaluation protocol. However, Appendix E.2 on ID vs OOD evaluation, for Table 9, we list all results in terms of a single checkpoint (over model three seeds), as indicated in the caption.

Thank you for engaging during this period to help refine our paper.

Reviewer_4deLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets zero-shot compositional generalization of goal-conditioned BC policies trained on offline data. The core idea: train a geometric horizon model (GHM), parameterized as a Temporal-Difference Flow, to approximate the successor measure of the behavior mixture policy, then use samples from this GHM as augmented goals for goal-conditioned flow-BC. Because the mixture-policy successor measure stitches across trajectories (unlike the per-trajectory hindsight goals that standard BC sees), the augmented goals cover the out-of-distribution compositional task distribution

Critical strengths

Well-posed problem and clean conceptual story. The framing that standard BC only covers RidR_{id} (per-policy successor measure), while generated goals from MβM^β cover RgenR_gen is crisp. This identifies why hindsight goal sampling fails to stitch, and motivates the method directly. Figure 2 communicates this well to some extent.

Critical weaknesses

●Non-standard datasets undercut baseline comparability. The authors re-collect the stitch datasets (terminating on sub-goal) because the originals "idle at the goal." This is defensible, but it means the headline numbers are not directly comparable to published OGBench results, and the offline-GCRL baselines (QRL, HIQL, CRL) may be disadvantaged by the re-collection and by using the same hyperparameters across maze sizes. A clearer statement of whether baselines were re-tuned on the new datasets is needed.

●Novelty is a well-executed combination rather than a new primitive. GHMs/TD-Flows (Farebrother 2025), horizon consistency (Farebrother 2026), and goal augmentation for stitching (Ghugare 2024) all pre-exist; the new piece is substituting dataset MC samples for the learned local-horizon GHM in TDHC, plus applying it as goal augmentation. This is useful, but the paper should calibrate its framing accordingly.

Questions for authors

●Sensitivity to α (the mix of generated vs. ground-truth goals)? 0.5 is used throughout. Is there a sweep, and does the optimum shift with horizon?

●How is RgenR_{gen} operationalized at evaluation,i.e., how are the compositional (s, sgs_g) test pairs drawn in OGBench, and are they guaranteed reachable under β (coverage) but not under any single βjβ_j?

●For manipulation you bootstrap from a learned action-chunk prior. How much does this prior quality influence the result?

3 comments
AuthorsOpenReview ↗

Nov 13th, Response 1/2

Thank you for your review and highlighting that our paper has a “well-posed problem and clean conceptual story”.

We address your concerns below:

W1 Datasets and comparisons:

First, we reiterate that all offline RL baselines are re-run on our collected datasets, using the same protocol as our methods. Additionally, we show the performance change of offline RL baselines from this process in Table 8 in Appendix E.1:

pointmazeGCIQL (ours)GCIQL (original)QRL (ours)QRL (original)CRL (ours)CRL (original)HIQL (ours)HIQL (original)
medium-stitch74 ± 121 ± 988 ± 580 ± 124 ± 20 ± 171 ± 874 ± 6
large-stitch40 ± 131 ± 289 ± 1584 ± 150 ± 00 ± 018 ± 413 ± 6
giant-stitch0 ± 00 ± 054 ± 950 ± 80 ± 00 ± 00 ± 00 ± 0
teleport-stitch27 ± 125 ± 32 ± 49 ± 510 ± 54 ± 330 ± 334 ± 4
Average35195856413030

We can see the performance on average is improved for GCIQL, slightly better for QRL, and CRL, and on average the same for HIQL on our datasets. Thus, we can see that our evaluation protocol is fair for these methods.

W2 novelty of approach: We highlight a significant distinction in the area studied in prior work. For example, “GHMs/TD-Flows (Farebrother et al. 2025), horizon consistency (Farebrother et al. 2026)” utilize GHMs both in the off-policy setting, and perform potentially expensive, test-time planning. In contrast, we demonstrate that we can use GHMs at training time (different), for goal augmentation (different), in the on-mixture policy setting (different), and improve goal-conditioned policy pretraining, not test-time planning (different). Ghugare et al. 2024, highlights the open challenge of compositional generalization/stitching, but does not figure out how to fully solve the problem, which we make progress towards in this work through both the conceptual connection to approximate the successor measure, and how to realize this idea through our proposed MCHC objective.

We also add comparison to the original TD-HC objective in TD-Aug, in Appendix E.5. We incorporate the existing training procedure of TD-HC where we utilize randomly sampling discounts for each sample in the batch, conditioning the GHM on discounts, and also using a mixture of 0.25 proportion of the batch using horizon consistency (HC) loss and 0.75 proportion using the TD^2 loss as utilized in the prior work to reduce compounding error from bootstrapping from a model’s own samples. We then consider a general form, which we call TD-HC(pMCHCp_\text{MCHC}), where for this experiment, we utilize 0.25 HC, proportion pMCHCp_\text{MCHC} our loss, and then 1 - 0.25 - pMCHCp_\text{MCHC} of TD^2. With this setup, we compare TD-HC, MCHC, and also intermediate combinations of their objectives as shown in Table 13:

PointmazeGCFBC+TD-HC (p=0)+TD-HC (p=0.25)+TD-HC (p=0.5)+TD-Aug (p=1.0)
medium-stitch55 ± 1071 ± 396 ± 098 ± 099 ± 0
large-stitch26 ± 68 ± 141 ± 477 ± 797 ± 3
giant-stitch0 ± 00 ± 01 ± 06 ± 254 ± 17
Average2727466083

Where p=pMCHCp = p_{\text{MCHC}}.

The right-most column reports our prior TD-Aug results, while the other columns utilize TD-HC with a varying proportion of the MCHC loss. Here, all models are reported at the same discount (0.998), corresponding to effective horizon 500. We can see that as we increase the proportion of MCHC, we substantially increase the average performance of the downstream policy (27% with TD-HC and 83% with MCHC).

In Appendix E.5, we also plot proportion vs EMD/success rate on pointmaze large in Figure 11, where additional MCHC loss proportion decreases distance to held-out evaluation data. We also display samples of TD-HC models in Figure 12, showing improved quality with higher MCHC proportion.

AuthorsOpenReview ↗

Nov 13th, Response 2/2

Questions:

We also answer your questions below.

Q1 sensitivity to α\alpha:

In Appendix E.6 we add a comparison over different augmentation proportions, α∈{0.25,0.5,0.75}\alpha \in \{0.25, 0.5, 0.75\}, and we find that overall TD-Aug performs well across these values. We maintained α=0.5\alpha=0.5 in our main experiments to show that a single hyperparameter works well across many settings.

Q2 generalization/evaluation distribution:

We illustrate the difference between training and evaluation in several places. In Figure 2 and Figure 6, where we contrast the horizon of the shorter training tasks (at most 4 maze cells) and long-horizon evaluation tasks, spanning across the maze.

Then, in new results, in Appendix E.2, including Table 9 we report the difference between ID atomic training tasks (pushing one button) and long-horizon tasks requiring many atomic tasks (such as solving a puzzle):

DatasetGCFBC (atomic, ID)GCFBC (stitch, OOD)+TD-Aug (atomic, ID)+TD-Aug (stitch, OOD)
cube-double97 ± 120 ± 099 ± 084 ± 1
cube-triple94 ± 220 ± 098 ± 032 ± 1
puzzle-4x4100 ± 00 ± 099 ± 042 ± 0
Average97139953

We can see that GCFBC and TD-Aug obtain similar ID results, but TD-Aug obtains substantially higher OOD generalization.

This is also illustrated in a new visualization, where in Figure 10 in pointmaze-{large,giant}, where we conduct an additional evaluation, where training tasks ( <= 4 maze cells), GCFBC performs well, but struggles as the evaluation horizon increases, while TD-Aug retains good performs at long-horizon goals.

Q3 manipulation prior:

In manipulation environments, across both single-action models, and action-chunking, we find better performance in both cases using a learned prior:

Dataseth=1 (Dataset action)h=1 (Learned prior)h=10 (Dataset action)h=10 (Learned prior)
cube-double-stitch22 ± 251 ± 562 ± 584 ± 0
cube-triple-stitch21 ± 151 ± 225 ± 131 ± 2
puzzle-4x4-stitch1 ± 031 ± 429 ± 440 ± 2
Average15443952

Thank you for your review. We hope this answered your concerns and look forward to future discussion.

Dear reviewer 4deL, We appreciate your feedback, which has helped us improve our work. Please let us know if our responses address weaknesses and questions. If so, would you consider to increase your score of our paper? We are also happy to answer any additional concerns. Thank you.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Nov 13th, General Response

We thank all the reviewers for their thorough feedback on our work, proposing TD-Aug, where we train geometric horizon models to image the consequence of composing existing behavior, allowing us to augment the goals of policies trained with behavior cloning.

We appreciate that reviewers found that our work considers a “well-posed problem and clean conceptual story” (4deL), that “our empirical results show substantial improvements/gains over BC” (qMRg,xRfD), especially on “task[s] that needs long-horizon stitching” (fTeG).

We have individually responded to each reviewer, and also updated our paper, with changes reflected in dark blue. This includes extensive ablations in Appendix E. We highlight the main points we have addressed in responses to two or more reviewers below:

Comparison to TD-HC (xRfD, qMRg, 4deL). In Appendix E.5, we compare MCHC in TD-Aug with the original TD-HC objective, where we also construct intermediate comparisons labeled TD-HC(pMCHCp_\text{MCHC}), utilizing a proportion of the MCHC loss. We find that as we increase this proportion, we improve downstream policy, from 27% with TD-HC to 83% on average with MCHC on pointmaze tasks. We also show how MCHC affects EMD in Figure 11, and provide comparison of samples in Figure 12. This demonstrates the importance of our proposed MCHC objective, over the prior TD-HC objective.

Action Bootstrap with Dataset Action or Learned Prior (qMRg, xRfD, 4deL). In Appendix E.3 and Appendix E.8, we conduct ablations on both pointmaze (Table 10, Table 16, Figure 13) and manipulation environments (Table 11). We find that bootstrapping with dataset actions performs better in pointmaze, whereas a learned prior performs better in manipulation.

Offline RL baselines on re-collected vs original datasets (fTeG, 4deL). We clarify that all baselines are re-run on our datasets, and use the same evaluation protocol as our method, showing a comparison in Appendix E.1 in Table 8, showing that performance is similar or improved on our datasets.

In-distribution versus out-of-distribution evaluation (fTeG, 4deL). In Appendix E.2, we conduct additional evaluation studying ID vs OOD. In manipulation (Table 9), we find that GCFBC (97%) and TD-Aug (99%) have similar success ID, but TD-Aug (53% substantially improves performance OOD from GCFBC (13%), and T Furthermore, on pointmaze (Figure 10), we conduct a more granular evaluation, showing that as the task horizon increases, requiring OOD generalization, TD-Aug obtains substantially better performance.

We reply to individual reviewers, and conduct additional ablations for individual reviewers in the below responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.