acceptodds
Under review as a conference paper at ICLR 2027

GTRL: Grounding Divide-and-Conquer Value Learning with Temporal Differences

Abstract

In offline goal-conditioned reinforcement learning (GCRL), divide-and-conquer scales to long horizons by joining two shorter segments at a subgoal. However, under stochastic dynamics, the base case of this rule values the luckiest trajectories through the data. The subgoal must also lie on a shared trajectory, so a state-goal pair that no trajectory connects gets no value update at all. To address both, we present **Grounded Transitive RL (GTRL)**, an offline GCRL value learning algorithm that grounds the divide-and-conquer update with a one-step TD target. Over a single step, TD is correct, as its target averages over the successors and needs no subgoal. GTRL adds this target to the composition rather than replacing it, so every pair receives an update, and the composition still carries the long horizon. GTRL also corrects the bias from hindsight relabeling by reweighting each goal against how reachable it was from other successors. We evaluate our algorithm on nineteen OGBench tasks spanning stochastic, deterministic, and stitching environments, where it achieves the highest average success rate.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Offline Goal-Conditioned RL

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_Qak1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes GTRL to improve value learning in offline goal-conditioned RL. It adds one-step TD targets, random goals from other trajectories, and hindsight weighting to TRL. The theory analyzes the overestimation and coverage problems and gives convergence results and value bounds under idealized tabular assumptions. GTRL achieves the highest average success rates across three task groups covering 19 OGBench tasks.

Critical strengths
  1. The paper identifies two problems: stochastic transitions may cause value overestimation, and in-trajectory goal sampling misses cross-trajectory state–goal pairs.
  2. The method performs well across different tasks, and the ablations support the main components.
Critical weaknesses
  1. Equation 8 takes the larger of the two targets. This may still cause overestimation, even when the TD target is unbiased, because smaller targets can be pushed up.
  2. For decomposable goals, optimistic regression may favor lucky outcomes as well as good subgoals. The paper does not clearly explain how these effects are separated.
Questions for authors
  1. Goal reachability differs from how often a goal is sampled. Why can reachability estimates be used to correct the goal-sampling bias?
  2. How does using only the TD target compare with using both targets under the same settings? This would help show the benefit of combining them.
5 comments
AuthorsOpenReview ↗

Response to reviewer Qak1 (Part 1/2)

We sincerely thank reviewer Qak1 for the constructive review. We address each concern and question below:

Tabular test: W1, W2 and Q1 are about the true values. So, we built a small tabular test where we know them. It is a grid maze with 137 states and the teleporter similar to Example 1 (page 5, line 242-248), which sends the agent next to the goal or to a far corner with probability 1/2 each. We also use the same maze without the teleporter, which is deterministic.

For data, we use 400 random walk trajectories of 100 steps, over 3 seeds, with γ=0.95\gamma=0.95. We solve each variant's loss exactly. So, every error comes from the update rule itself. With random walk data, every variant sits below V∗V^\ast on average. So, what matters is how often a variant goes above V∗V^\ast.

We call a pair (s,g)(s,g) lucky when a lucky outcome can shorten its best route, that is VDC>V∗V_{\rm DC}>V^\ast. The teleport maze has 3,877 such pairs out of 18,632.

Table 1. Tabular test. The error is max⁡aQA(s,a,g)−V∗(s,g)\max_a Q_A(s,a,g)-V^\ast(s,g), averaged over 3 seeds. The teleport columns use the lucky pairs, and the deterministic column uses all pairs.

VariantTeleport, lucky pairs above V∗V^\astTeleport, average errorDeterministic, average error
GTRL8.9%−0.23−0.26
Without the max0.2%−0.39−0.42
κ=0.5\kappa=0.5 on decomposable goals2.6%−0.28−0.30
TD only2.8%−0.30−0.31

W1. Can the max in Eq. 8 overestimate?

W1. Equation 8 takes the larger ... smaller targets can be pushed up.

Under deterministic dynamics, Prop. 3 shows it cannot. Under stochastic dynamics, it adds no new route for luck, its effect is bounded and small, and removing it is worse.

  • Deterministic. In the tabular setting, GTRL never goes above VDCV_{\rm DC} (Prop. 3), and here VDC=V∗V_{\rm DC}=V^\ast (Prop. 1(iii)). So, the larger target can only lift a value that is too low. In the deterministic maze, no pair goes above V∗V^\ast.
  • No new route. With correct values, a composition through a subgoal cannot overshoot, since V∗(s,g)≥V∗(s,w)V∗(w,g)V^\ast(s,g)\ge V^\ast(s,w)V^\ast(w,g) holds under stochastic dynamics too. So, luck enters only through the one-step pieces (Eq. 6 is Eq. 5 at offset one), which are TRL's own base cases (Prop. 1).
  • Bounded and small. No entry in any run goes above VDCV_{\rm DC}. GTRL puts 8.9% of the lucky pairs above V∗V^\ast, which is 1.8% of all pairs, and it is the closest to V∗V^\ast of all variants in Table 1, in both mazes.

Removing the max is worse. From Table 1, values drop far below V∗V^\ast (−0.39 against −0.23), and a third of all entries fall below the behavior value QβQ^\beta, which the max guarantees as a floor (Prop. 3). In deep RL, removing the max, or replacing it with an average or two separate losses, costs 6 to 62 points (Table 3, part 2).


W2. Separating luck from good subgoals

W2. For decomposable goals, optimistic regression ... how these effects are separated.

GTRL separates them by the kind of goal.

  • Counterfactual goals, half of every batch, have no subgoal. So, all of their spread comes from the transition, and κ=0.5\kappa=0.5 averages it out.
  • Decomposable goals use κ=0.7\kappa=0.7 to pick good subgoals. Under deterministic dynamics, there is no luck, and κ=0.7\kappa=0.7 is closer to V∗V^\ast than κ=0.5\kappa=0.5 (Table 1: −0.26 against −0.30), without ever going above it.
  • Under stochastic dynamics, the hindsight weight hh moves each outcome back toward its true frequency p(s′∣s,a)p(s'\mid s, a). Prop. 3 caps what is left at VDCV_{\rm DC}.

Separating the two fully into one decomposable goal would need several successors of the same (s,a)(s,a), which continuous offline data does not have.

AuthorsOpenReview ↗

Response to reviewer Qak1 (Part 2/2)

Q1. Why reachability corrects the sampling bias

Q1. Goal reachability differs from how often a goal is sampled ... correct the goal-sampling bias?

The two differ only by a factor that depends on the goal alone. The hindsight weight is a ratio at a fixed goal, so that factor cancels.

GTRL draws a trajectory goal at offset k∼Geom(1−γ)k\sim\mathrm{Geom}(1-\gamma) with the critic's γ\gamma. So, given s′s', the goal is drawn from μ(g∣s′)\mu(g\mid s'), the discounted occupancy of the behavior policy. This is the sampling frequency. To be at gg, the agent first has to reach gg, and everything after starts from gg. So, μ(g∣s)=Vβ(s,g)μ(g∣g)\mu(g\mid s)=V^\beta(s,g)\mu(g\mid g) ([1] Blier et al., 2021). The first factor is reachability, and the second is the same for every start state. In Eq. 9, μ(g∣g)\mu(g\mid g) cancels, and the frequency ratio becomes a reachability ratio exactly. In our tabular test of the maze of Table 1, this identity holds for every pair.

In practice, GTRL reads V from its critic, as our Limitations state (page 9). It still corrects in the right direction (Table 2). The exact weight cuts the lucky pairs above V∗V^\ast from 10.7% to 6.3%, and ours cuts it to 8.9%, about half of that gain with no extra network. At the teleporter, it moves the value from 0.73 to 0.70, toward V∗=0.66V^\ast=0.66.

Table 2. Teleport maze. V(T,G)V(T,G) is the value from the teleporter to the goal, where V∗=0.66V^\ast=0.66.

WeightLucky pairs above V∗V^\astV(T,G)V(T,G)
No weight10.7%0.73
GTRL, hh from the critic8.9%0.70
Exact hh from VβV^\beta6.3%0.68

Ref. [1] Blier, Tallec and Ollivier. Learning successor states and goal-dependent values: A mathematical viewpoint. arXiv:2101.07123, 2021.


Q2. Both targets beat TD alone

Q2. How does using only the TD target ... benefit of combining them.

We ran this ablation and the results are presented below.

  • TD only uses yTDy_{\rm TD} on every goal and keeps everything else of GTRL. Table 3 also includes the no max ablation of Fig. 4.
  • Average of both: On decomposable goals, the target is 12yDC+12yTD\tfrac12 y_{\rm DC}+\tfrac12 y_{\rm TD} in place of the max.
  • Two losses: On decomposable goals, the critic is trained with two separate loss terms at half weight each, one fitting yDCy_{\rm DC} at κ=0.7\kappa=0.7 and one fitting yTDy_{\rm TD} at κ=0.5\kappa=0.5, with no max.

GTRL beats TD on all of the tasks, by ~6.8 points on average (Table 3).

In Table 1, TD only also sits further below V∗V^\ast in both mazes (−0.30 and −0.31 against −0.23 and −0.26). The composition picks the better route through a subgoal, while TD with the logged next action mostly follows the data's own policy.

It matters how we combine the two targets. The average uses 12yDC+12yTD\tfrac12y_{\rm DC}+\tfrac12y_{\rm TD}, and two losses fits yDCy_{\rm DC} at κ=0.7\kappa=0.7 and yTDy_{\rm TD} at κ=0.5\kappa=0.5 as separate terms. Both use the TD target. Yet they drop as low as the composition alone, because a low composition target pulls them down. The max keeps the TD target as a floor and lets the composition lift it.

Table 3. Success rate (%), 4 seeds.

TaskComposition only (no max)TD onlyAverage of bothTwo lossesGTRL (max)
pointmaze-large-navigate20.0 ± 8.067.4 ± 5.622.1 ± 1.817.6 ± 6.579.1 ± 6.5
antmaze-large-navigate35.3 ± 5.283.2 ± 2.632.2 ± 1.826.2 ± 3.886.5 ± 1.5
humanoidmaze-large-navigate10.3 ± 1.535.6 ± 7.014.1 ± 2.514.1 ± 2.741.2 ± 9.2
antmaze-large-stitch15.7 ± 1.523.7 ± 2.814.0 ± 1.415.9 ± 1.030.7 ± 4.0
pointmaze-teleport-navigate13.6 ± 4.528.8 ± 5.512.6 ± 4.511.3 ± 0.237.0 ± 7.2
antmaze-teleport-stitch25.4 ± 5.131.1 ± 5.329.9 ± 2.425.3 ± 3.136.1 ± 2.5

We hope these results answer your concerns. If you have any further queries, we would love to answer them.

AuthorsOpenReview ↗

Thank you for your reviews

Dear Reviewer Qak1,

We sincerely thank you for your efforts in reviewing our work and for your great support. We have addressed your core concerns in the comments above. We would also like to let you know that the manuscript has been updated, with all revisions marked in blue.

If you feel our revisions have addressed your concerns, we would be truly grateful if you would kindly consider raising your score.

Best regards,

Authors of #42690

Reviewer_nhLCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies stochastic overestimation and missing cross-trajectory updates in transitive offline goal-conditioned reinforcement learning. It proposes GTRL, which combines transitive composition with one-step TD targets, introduces goals sampled across the dataset, and applies critic-derived hindsight weights. The paper provides tabular analyses and evaluates the method on nineteen OGBench tasks covering stochastic transitions, trajectory stitching, and deterministic control.

Critical strengths
  1. Propositions 1 and 2 explain why the tabular transitive update can favor lucky stochastic outcomes and fail to directly supervise pairs outside a shared trajectory. These observations motivate a simple combination of TD learning and transitive composition that addresses an important offline GCRL problem.

  2. The nineteen-task evaluation includes settings specifically designed to test stochasticity and stitching, with four seeds and standard deviations reported. GTRL improves average success over TRL from 32.5% to 42.2% on teleport tasks and from 9.8% to 35.2% on stitch tasks, while also improving deterministic-task performance. These results support the method’s practical potential.

Critical weaknesses

Equation (8) takes a maximum before averaging stochastic targets, which can retain optimism even with correct input values. Equation (9) derives weights from behavior-policy occupancy but substitutes a critic trained with maximum and expectile targets, without establishing that it estimates the required behavior-policy values. Proposition 3 only brackets the fixed point between the behavior value and the optimistic graph value; it does not establish removal of the diagnosed bias.

Questions for authors
  1. How does Equation (8) avoid optimism introduced by maximizing sampled targets before taking their expectation? Can the authors demonstrate value calibration on a small stochastic MDP with known optimal values?

  2. Under what assumptions does Equation (9) remain a valid importance ratio when the occupancy belongs to the behavior policy but the critic is trained using maximum and expectile targets? Is the implemented correction exact or heuristic?

2 comments
AuthorsOpenReview ↗

Response to reviewer nhLC (Part 1/2)

We sincerely thank reviewer nhLC for the constructive review. We address each concern and question below:

Tabular test. As the reviewer asked for calibration on a small stochastic MDP with known optimal values, we built one. It is a grid maze with 137 states and a teleporter like Example 1 (page 5, lines 243 to 247), which sends the agent next to the goal or to a far corner with probability 1/2 each. We also use a slip version, where every move goes the intended way with probability 0.8 and to each side with probability 0.1.

For data, we use 400 random walk trajectories of 100 steps, over 3 seeds, with γ=0.95\gamma=0.95. We solve each variant's loss exactly. So, every error comes from the update rule itself. With random walk data, every variant sits below V∗V^\ast on average. So, what matters is how often a variant goes above V∗V^\ast.

We call a pair (s,g)(s,g) lucky when a lucky outcome can shorten its best route, that is VDC>V∗V_{\rm DC}>V^\ast. The teleport maze has 3,877 such pairs out of 18,632, and in the slip maze every pair is lucky.

Table 1. Tabular test on the lucky pairs. The error is max⁡aQA(s,a,g)−V∗(s,g)\max_a Q_A(s,a,g)-V^\ast(s,g), averaged over 3 seeds.

VariantTeleport, above V∗V^\astTeleport, errorSlip, above V∗V^\astSlip, error
GTRL8.9%−0.238.2%−0.18
Max after averaging over s′s' and a′a' (true pp)1.7%−0.283.1%−0.22
No max0.2%−0.390.1%−0.37
No hindsight weight10.7%−0.2410.0%−0.19
Exact hh from VβV^\beta6.3%−0.255.6%−0.20

W1 and Q1. The max before the average

W1. Equation (8) takes a maximum before averaging stochastic targets, which can retain optimism even with correct input values.

Q1. How does Equation (8) avoid optimism ... small stochastic MDP with known optimal values?

The max can pass a lucky outcome. However, it adds no new route for luck, its effect is bounded and small, and removing it is worse.

  • No new route. With correct values, a composition through a subgoal cannot overshoot, since V∗(s,g)≥V∗(s,w)V∗(w,g)V^\ast(s,g)\ge V^\ast(s,w)V^\ast(w,g) holds under stochastic dynamics too ([1] Myers et al., 2024), however ww is picked. So, luck enters only through the one step pieces, yTDy_{\rm TD} and the base case γ\gamma (Eq. 6 is Eq. 5 at offset one). These are TRL's own base cases, the fault Prop. 1 diagnoses.
  • Counterfactual goals. Counterfactual goals make up half of every batch. Their only target is yTDy_{\rm TD}, with no max, and κ=0.5\kappa=0.5 averages it over outcomes.
  • Bounded. In the tabular setting, GTRL never goes above VDCV_{\rm DC} (Prop. 3), and no entry does in either maze. Under deterministic dynamics, VDC=V∗V_{\rm DC}=V^\ast (Prop. 1(iii)). In a deterministic copy of the maze, no pair goes above V∗V^\ast.
  • Small. GTRL puts 8.9% of the lucky teleport pairs above V∗V^\ast, which is 1.8% of all pairs, and it sits below V∗V^\ast on average. Taking the max after averaging, which needs the true pp, overshoots less, as you expect, but sits further below V∗V^\ast in both mazes.

Removing the max is worse. Table 1 shows that values drop far below V∗V^\ast (−0.39 against −0.23), and in deep RL it costs 11 to 59 points (Fig. 4). The largest drops are on deterministic tasks, where there is no luck to exploit. Two losses, or an average in place of the max, also lose 6 to 62 points (our reply to Xjcq, Table 3). So, the max mainly fixes the composition's underestimation, and its cost under stochastic dynamics is bounded.

Ref. [1] Myers et al., Learning temporal distances: Contrastive successor features can provide a metric structure for decision-making. ICML, 2024.

AuthorsOpenReview ↗

Response to reviewer nhLC (Part 2/2)

W2 and Q2. The weight uses the critic

W2. Equation (9) derives weights from behavior-policy occupancy but substitutes a critic ...

Q2. Under what assumptions does Equation (9) remain a valid importance ratio ... Is the implemented correction exact or heuristic?

The ratio of Eq. 9 is exact for the behavior value VβV^\beta. Our implementation is a plug-in estimate. It is exact when the critic matches VβV^\beta up to a factor that depends only on gg, since that factor cancels in the ratio. This does not hold in general, as our Limitations state (page 9).

Two parts are exact regardless. Goals are drawn at offset k∼Geom(1−γ)k\sim\mathrm{Geom}(1-\gamma) with the critic's γ\gamma, so the sampling side is right by construction. Counterfactual goals are picked without seeing the successor, so their hindsight weight h=1h=1 is exact, on half of every batch.

The plug-in still corrects in the right direction (Table 1). From table 1, the exact weight hh cuts the teleport overshoot from 10.7% to 6.3%. Our GTRL cuts it to 8.9%, about half of that gain with no extra network. The slip maze agrees (10.0%, 5.6% and 8.2%). The logged action in the denominator costs nothing, since the state value does worse (10.4%). The clip is active on under 0.4% of decomposable goals on average in the last 20% of training, and removing it with the normalization changes almost nothing (9.0% against 8.9%). (Kindly see, response to reviewer Xjcq: Table 1: Tabular Test)


W3. What Prop. 3 shows

W3. Proposition 3 only brackets the fixed point between the behavior value and the optimistic graph value; it does not establish removal of the diagnosed bias.

Prop. 3 gives a floor that TRL lacks and a ceiling, and Table 1 measures the rest.

  • Floor. Every pair is trained and floored at QβQ^\beta. TRL leaves the pairs off its trajectories untrained (Prop. 2), about 50% of all entries in the teleport maze. TRL also falls below QβQ^\beta on 3,368 of the entries it trains. GTRL never does.
  • Ceiling. GTRL never goes above VDCV_{\rm DC}, which equals V∗V^\ast under deterministic dynamics. There, it cannot overestimate at all.
  • Stochastic dynamics. The bias we diagnose is that the transitive rule converges to VDCV_{\rm DC}, which is above V∗V^\ast on every lucky pair (Prop. 1). GTRL is above V∗V^\ast on 8.9% of them. So, GTRL reduces this bias and bounds the rest, but does not fully remove it. We will state this precisely.

We hope these results answer your concerns. If you have any further queries, we would love to answer them.

Reviewer_XjcqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new and improved divide-and-conquer value learning algorithm for offline goal-conditioned RL. The paper builds off of the transitive RL approach, a recent paper that uses divide and conquer learning for value updates in the offline goal conditioned RL setting, and includes algorithmic modifications towards resolving two weak points of TRL: TRL's necessity for the environment dynamics to be deterministic, and TRL's requirement that subgoals sampled for the divide and conquer value updates must come from the same trajectory as the start and goal state samples, a choice that limits the amount of cross-trajectory stitching that in theory can take place. The authors proposed algorithm, Grounded TRL (GTRL), does the following. They first observe that 1-step TD learning, if instantiated correctly, can learn to reach goals even if the environment dynamics are stochastic. The authors find that in the divide and conquer update used in TRL, the divide and conquer update itself is fine and does not limit the algorithm to work in only deterministic settings. It is instead the base case algorithm of TRL that enforces this deterministic assumption. So towards removing this assumption, GTRL replaces TRL's base case algorithm with 1-step TD. Then, the authors outline one more issue that still exists before the determinism assumption can be removed: the sampling of goals via hindsight, as is common in off-policy/offline GCRL, in the way that TRL does it, can upsample transitions that lead to a bias if the environment dynamics are stochastic. To resolve this issue, the authors propose using importance sampling, and find a way to compute importance weights using the value functions the authors already train as part of their method. They evaluate GTRL on a number of OGBench offline goal conditioned RL tasks, and find that their method achieves overall the best performance compared to a number of baselines, and they find that particularly for stochastic tasks, including the importance weight term is critical for the best performance.

Critical strengths

(1) The authors make substantial effort at tackling important limitations with divide and conquer value learning. In fact they arguably tackle all of the main limitations of the prior work, TRL. Obtaining a divide and conquer offline GCRL algorithm that can work in stochastic dynamic settings and can stitch well across trajectories is an important problem, and the authors make strides in this direction. It also appears to me that their approach is novel.

(2) The experimental results the authors display are favorable for their method, which is a strength. The authors also run an ablation on dropping the importance weighting, which if done results in worse performance on stochastic tasks (showing that their approach helps).

Critical weaknesses

Overall I have three broad categories of weaknesses (the first and third more major/problematic than the second):

(1) Several algorithmic aspects of the paper seem to not be fully correct. Because of this, I don't think the authors have convincingly shown that their proposed algorithm fully addresses the problems that TRL has under stochastic dynamics.

(2) The paper clarity is not the best. To fully understand the method I had to make several close reads. It would be great if the paper could be restructured/written such that the method is explained better and more clearly. In a large number of places there is also AI-sounding text, which by itself isn't bad, but often it is this text that is difficult to understand.

(3) The authors are missing experiments on the very long horizon tasks from the main baseline, TRL. They evaluate on OGBench tasks, which is good, but as divide-and-conquer learning is most useful for very long horizon tasks, it would be very interesting/necessary to see how GTRL performs on the long horizon tasks from TRL (like puzzle-4x5/4x6, humanoidmaze-giant). These were arguably the most important experiments in TRL, and I think it's a significant omission that GTRL doesn't evaluate on them.

Below I've expanded on the technical/algorithmic qualms I have, as well as experiments I think are missing:

(1) Hindsight importance weighting. This term is meant to resolve the bias in TRL with stochastic dyamics, but I don't think it fully correctly does this.

The importance weighting derivation in equation 9 uses the discounted occupancy of the behavior policy that collected the data. However, the authors approximate this using their learned GTRL Q function, which is not necessarily the Q function of the behavior policy.

Another issue: the denominator in Equation 9 is derived using V^\beta(s',g), but the implementation uses Q(s',a',g).

Finally, the authors clip the importance weights to [0.5, 2] and normalize them to have mean 1. This introduces bias, and isn't really ablated it seems, and especially since the goal of the authors is to remove bias in stochastic settings, this seems questionable.

(2) The taking of the max between the DC (divide and conquer) and TD targets in Equation 8 seems problematic. It seems incorrect to take this max under stochastic dynamics.

Even if the DC target is a valid lower bound on the true Q value, the TD target is a sample of a stochastic transition, not the expected TD target. Taking the maximum of the DC target and this sampled TD target will systematically favor lucky stochastic outcomes. So the update can still overestimate the true Q value even if the critic used to construct the targets is perfectly accurate. This seems particularly problematic because correcting exactly this kind of stochastic overestimation is one of the main motivations of the paper.

It seems like the authors tried to do something IQL like to get around the stochasticity problem, which is good, but it doesn't yet appear fully correct to me. In IQL V and Q are trained, with Q regressing the mean (which is the main trick that gets this to work in stochastic dynamics). In GTRL however the authors do not train V and Q separately, only using Q. For the TD update a tau of 0.5 is used, which is mean regression (which is clever), but then in equation 8 it seems like this benefit is completely removed because then the authors do something wrong under stochastic (namely equation 8). It would also I think help for the clarity of the paper to expand on this 0.5 tau trick (i.e., mean regression) and the relationship to IQL.

The authors do ablate the maximum in Figure 4, and show that removing it substantially hurts performance, which is useful (but again also bad because something that is wrong under stochasticity is needed for good performance) However, the ablation replaces the maximum with just the DC target. This simultaneously removes the maximum and removes TD supervision on these goals. So perhaps there are more principled (and maybe also empirically better) ways of combining targets, like using separate DC and TD losses, or a weighted combination. I think it's very important the authors try this out.

(3) Why not sample subgoals from different trajectories? The authors argue that when a goal is sampled from a different trajectory, there is no subgoal to route through, and therefore they cannot use the DC update. However this isn't true, as you can just sample subgoals from other trajectories as well. TRL found that this was worse for them, so I can see merit in the authors approach, but I think having an ablation of GTRL where subgoals from different trajectories are sampled would be good.

A more major concern: on the stitch tasks, most start-goal pairs are on different trajectories, so only TD updates are used (not divide and conquer updates). And the reason that only TD updates are used is because the authors claim that you can't sample subgoals from different trajectories (which from the prev paragraph isn't true). But this is problematic because then the value learning algorthm is just TD: there's no exponential convergence of value learning that you get with divide and conquer. So at least on this stitching heavy task, without running an experiment with subgoals sampled across trajectories, the GTRL algorithm isn't being demonstrated to be an improvement on TRL (since no divide and conquer is being used at all).

(4) A totally different way to remove the stochasticity bias and precludes the need for the importance weighting is simply sampling all goals randomly, rather than using hindsight goals and importance weighting. The authors could use one-step TD for this baseline, or combine it with random subgoal sampling as discussed above. This feels like an important baseline.

(5) Missing long-horizon experiments. As mentioned earlier, I think this is one of the most significant weaknesses.

(6) Slighty oddity: the standard-task baseline numbers appear to match those reported in the original TRL paper exactly, including the standard deviations. This is not inherently a problem, but GTRL says it evaluates using 50 episodes per goal, whereas TRL used 15. How is possible then that the numbers match exactly?

Questions for authors

My questions are all covered in the weaknesses section. W.r.t. my rating, I'll be inclined to increase it if the major weaknesses are resolved.

5 comments
AuthorsOpenReview ↗

Response to reviewer Xjcq (Part 1/2)

We sincerely thank reviewer Xjcq for the detailed and constructive review. We address each concern below:

Tabular test (for 1 and 2). We built a grid maze with 137 states and a teleporter like Example 1 (page 5), which sends the agent next to the goal or to a far corner with probability 1/2 each. So the true values are known.

For data, we use 400 random walk trajectories of 100 steps, over 3 seeds, with γ=0.95\gamma=0.95. We solve each variant's loss exactly. So, every error comes from the update rule itself. With random walk data, every variant sits below V∗V^\ast on average. So, what matters is how often a variant goes above V∗V^\ast.

We call a pair (s,g)(s,g) lucky when a lucky outcome can shorten its best route, that is VDC>V∗V_{\rm DC}>V^\ast. The teleport maze has 3,877 such pairs out of 18,632. On all other pairs, no variant ever goes above V∗V^\ast.

Table 1. Tabular test Lucky pairs, mean of 3 seeds. The error is max⁡aQA(s,a,g)−V∗(s,g)\max_aQ_A(s,a,g)-V^\ast(s,g).

VariantPairs above V∗V^\astAverage error
GTRL8.9%−0.23
No hindsight weight10.7%−0.24
Exact h from VβV^\beta6.3%−0.25
Denominator averaged over next actions10.4%−0.25
No clip, no normalization9.0%−0.26
Max after averaging over s′s' and a′a' (true pp)1.7%−0.28
No max0.2%−0.39
Two losses0.3%−0.38
Average of both targets0.4%−0.36

(1) Hindsight weighting

The importance weighting derivation ... clip the importance weights ...

Eq. 9 is exact for VβV^\beta. Only our estimate of it is approximate, and we measured each of your three points.

  • Critic in place of QβQ^\beta. The exact weight cuts the lucky pairs above V∗V^\ast from 10.7% to 6.3%. Ours, read from the critic, cuts it to 8.9% and lowers it on every seed, with no extra network. The gap is in our Limitations (page 9).
  • Q(s′,a′,g)Q(s',a',g) in place of Vβ(s′,g)V^\beta(s',g). The critic's state value, averaged over next actions, is worse on every seed (10.4%).
  • Clip and normalization. In the paper's runs, the clip is active on under 0.4% of decomposable goals on average in the last 20% of training, on every task. The normalization keeps the relative weights. Removing both gives 9.0% against 8.9%. In deep RL, the raw weight is within ~2 points of GTRL on four teleport tasks (Table 2).

Table 2. Raw weight vs GTRL Success rate (%), 4 seeds.

TaskRaw weightGTRL
pointmaze-teleport-navigate36.7±7.537.0±7.2
antmaze-teleport-navigate47.1±2.948.5±2.3
pointmaze-teleport-stitch46.4±3.047.3±2.4
antmaze-teleport-stitch31.3±3.936.1±2.5

(2) The max in Eq. 8

Taking the maximum of the DC target and this sampled TD target will systematically favor lucky stochastic outcomes ...

The max can pass a lucky outcome. However, it adds no new route due to luck, its effect is bounded and small. Unfortunately, your suggested alternatives do worse.

  • No new route. Eq. 6 is Eq. 5 at subgoal offset one. So, TRL's own max over subgoals (Eq. 3) already contains this target, through its base case (Prop. 1). GTRL keeps it, and adds counterfactual goals. These goals are half of every batch, which have no max and average over outcomes at κ=0.5\kappa=0.5.
  • Bounded. Composing correct values cannot overshoot, since V∗(s,g)≥V∗(s,w)V∗(w,g)V^\ast(s,g)\ge V^\ast(s,w)V^\ast(w,g) under stochastic dynamics. In the tabular setting, GTRL never goes above VDCV_{\rm DC} (Prop. 3), and VDC=V∗V_{\rm DC}=V^\ast under deterministic dynamics (Prop. 1(iii)).
  • Small. GTRL is above V∗V^\ast on 8.9% of the lucky pairs, which is 1.8% of all pairs. Its average error is the closest to zero of all targets in Table 1, including the max after averaging with the true pp.

Two suggestions do worse. Two losses fits yDCy_{\rm DC} at κ=0.7\kappa=0.7 and yTDy_{\rm TD} at κ=0.5\kappa=0.5 as separate terms. The average uses 12yDC+12yTD\tfrac12y_{\rm DC}+\tfrac12y_{\rm TD}. Both keep the TD target, yet both are 6 to 62 points below GTRL on all six tasks, close to the no-max ablation (Table 3, part 2). In Table 1, they rarely overshoot, but they sit far below V∗V^\ast. The max is also needed where there is no luck at all. On deterministic tasks, removing it drops pointmaze-large-navigate from 79.1 to 20.0 and antmaze-large-navigate from 86.5 to 35.3. So, the max mainly fixes the composition's underestimation.

On IQL. The mean regression is kept on every goal with no subgoal. IQL fits QQ with a mean loss and puts its optimism in VV. GTRL has one network, so on counterfactual goals it fits γQˉ(s′,a′,g)\gamma\bar Q(s',a',g) at κ=0.5\kappa=0.5, a mean over outcomes and logged next actions. On decomposable goals, κ=0.7\kappa=0.7 and the max pick the best subgoal, and are also optimistic over outcomes, which is the bounded overshoot above.

AuthorsOpenReview ↗

Response to reviewer Xjcq (Part 2/2)

(3) Cross-trajectory subgoals and stitch tasks

Why not sample subgoals from different trajectories? ... on the stitch tasks ... only TD updates are used ...

We ran it, and same trajectory subgoals do better. On counterfactual goals, we draw ww at random from the dataset and use Eq. 8. It is 10 to 32 points below GTRL on six tasks (Table 3). TRL also presents similar findings with random subgoals in their paper. With a random ww, both legs are mostly pairs that no trajectory connects, so the product has little data behind it.

Table 3. Success rate (%), 4 seeds.

TaskNo maxTwo lossesAverageCross trajectoryGTRL
pointmaze-large-navigate20.0±8.017.6±6.522.1±1.858.5±1.879.1±6.5
antmaze-large-navigate35.3±5.226.2±3.832.2±1.855.3±6.186.5±1.5
humanoidmaze-large-navigate10.3±1.514.1±2.714.1±2.59.5±2.041.2±9.2
antmaze-large-stitch15.7±1.515.9±1.014.0±1.417.0±1.230.7±4.0
pointmaze-teleport-navigate13.6±4.511.3±0.212.6±4.527.2±3.237.0±7.2
antmaze-teleport-stitch25.4±5.125.3±3.129.9±2.435.8±2.536.1±2.5

Stitch tasks do use divide and conquer. Half of every stitch batch is decomposable (logged fraction 0.50 on all five tasks), and those goals get the composition. On antmaze-large-stitch, removing it (TD only) costs 7 points (23.7 against 30.7), and removing counterfactual goals costs 24.5 (6.2 against 30.7). So, GTRL needs both. GTRL averages 35.2% on the stitch tasks against 9.8% for TRL (paper, Table 1). Pairs across trajectories still get only the one step target, as our Limitations state.


(4) Sampling all goals at random

A totally different way to remove the stochasticity bias ... is simply sampling all goals randomly ...

This removes the reward. In GTRL, the reward sits in the base cases, g=sg=s and g=s′g=s'. A goal drawn at random from the whole dataset almost never hits them. So, the critic has almost nothing to propagate.

We ran it on six tasks (Table 4). Success falls by a large margin on the deterministic, teleport, and stitch tasks. So, hindsight goals are needed, and the hindsight weight is what makes them safe to use under stochastic dynamics.

Table 4. Success rate (%) over 4 seeds.

TaskAll goals randomGTRL
antmaze-large-navigate40±387±1
humanoidmaze-medium-navigate9±275±4
antsoccer-arena-stitch2±038±2
humanoidmaze-medium-stitch6±270±3
antmaze-teleport-stitch26±236±2
pointmaze-teleport-stitch39±147±2

(5) Long horizon tasks

... it would be very interesting/necessary to see how GTRL performs on the long horizon tasks from TRL ...

We ran GTRL on TRL's three long horizon tasks with the 1B datasets, for four seeds and matching the exact TRL's experiment settings. GTRL is higher on all three.

Table 5. Success rate (%). Four seeds.

TaskTRLGTRL
humanoidmaze-giant79±285±2
puzzle-4x597±1100±0
puzzle-4x651±568±6

(6) Baseline numbers match TRL

... GTRL says it evaluates using 50 episodes per goal, whereas TRL used 15. How is possible then that the numbers match exactly?

Thank you for catching this. We ran all of our experiments matching the exact experimental settings of TRL. It was a typing mistake on our part. Appendix B.5 wrongly says every agent used 50 episodes. We used 15 episodes for eval.


Write-up Clarity. We apologize for this. We will rewrite Section 4 in plain language in the revision. Again, we apologize for any hardship that the reviewer may have faced while reading our manuscript.

We hope these results answer your concerns. If you have any further queries, we would love to answer them.

AuthorsOpenReview ↗

Thank you for your reviews

Dear Reviewer Xjcq,

We sincerely thank you for your efforts in reviewing our work and for your great support. We have addressed your core concerns in the comments above. We would also like to let you know that the manuscript has been updated, with all revisions marked in blue.

If you feel our revisions have addressed your concerns, we would be truly grateful if you would kindly consider raising your score.

Best regards,

Authors of #42690

Reviewer_mc86OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the limitations of Transitive RL (TRL), a divide-and-conquer approach to offline goal-conditioned reinforcement learning. The authors show that TRL can overestimate values under stochastic dynamics by treating observed transitions as deterministic base cases, and identify a coverage limitation arising from its in-trajectory goal sampling. Based on these observations, they propose Grounded Transitive RL (GTRL), which extends TRL with counterfactual goal sampling, a modified target combining divide-and-conquer and one-step TD updates, and hindsight weighting. Experiments on OGBench datasets demonstrate that GTRL consistently improves upon TRL on average across deterministic, stochastic, and stitching environments.

Critical strengths

[S1] Clear motivation and theoretical analysis.

The limitations of TRL investigated in this paper are well-motivated. The authors provide a clear theoretical analysis and an intuitive example illustrating why these limitations matter, particularly under stochastic dynamics. Moreover, the proposed components of GTRL are directly motivated by the identified limitations, making the overall approach coherent and easy to follow.

[S2] Strong and consistent empirical results.

The experimental evaluation includes a comprehensive comparison with a diverse set of baseline methods across deterministic and stochastic environments, as well as challenging stitching tasks. The results demonstrate that GTRL consistently achieves strong performance and substantially improves upon TRL across the three categories of environments.

Critical weaknesses

[W1] Limited ablation studies.

The ablation studies do not sufficiently disentangle the contributions of the proposed components. Although Section 5.3 presents ablations of the main components, some experiments change multiple aspects of the algorithm simultaneously, making it difficult to identify the source of the improvements.

For example, the ablation of counterfactual goals in Figure 3 removes not only random goal sampling but also the one-step TD target for in-trajectory goals. Consequently, the observed performance difference cannot be attributed solely to the broader goal coverage provided by counterfactual sampling.

Furthermore, Figure 3 reports results only on navigate datasets, whereas the coverage limitation identified in Proposition 2 should be particularly relevant to stitching datasets. Evaluating this ablation on stitching tasks would provide more direct evidence for the proposed motivation.

[W2] Limited theoretical justification for the modified target.

The target proposed in Equation (8), which takes the maximum of the divide-and-conquer and one-step TD targets, appears to be a reasonable but somewhat heuristic design choice.

Proposition 3 establishes that the fixed point of GTRL lies between the behavior-policy value QβQ^\beta and the divide-and-conquer value VDCV_{\mathrm{DC}}. However, this result does not establish a direct relationship between the learned value QAQ_A and the optimal value Q∗Q^*, nor does it guarantee that GTRL reduces overestimation relative to TRL.

Therefore, although the proposed target works well empirically, its theoretical connection to the stated objective of correcting value overestimation remains somewhat limited.

Questions for authors

[Q1] Could the authors provide a more systematic ablation of counterfactual goal sampling and TD grounding?

In particular, it would be useful to compare the full GTRL objective against a variant that retains the modified target in Equation (8) but samples only in-trajectory goals. This would help isolate the contribution of counterfactual goal sampling from that of adding one-step TD targets.

Additionally, could the authors report the counterfactual goal ablation on stitching tasks? Since these tasks are explicitly designed to require trajectory stitching, they seem particularly suitable for evaluating the coverage limitation discussed in Proposition 2.

[Q2] Why does the one-step TD target use the logged next action rather than an action selected by the current policy?

In Equation (6), the TD target uses Q(s′,a′,g)Q(s',a',g), where a′a' is the logged action from the dataset. This corresponds to a Bellman expectation backup under the behavior policy, rather than a Bellman optimality backup.

While using logged actions is a reasonable choice in offline RL to avoid out-of-distribution action maximization, it would be helpful to clarify how this choice relates to the objective of learning the optimal Q-function. Have the authors considered using the current policy or an implicit action maximization method for the TD target?

[Q3] Could the authors provide more direct empirical evidence that GTRL reduces value overestimation?

Since correcting the optimistic bias of TRL under stochastic dynamics is one of the central motivations, it would be informative to compare the value estimates learned by GTRL and TRL against ground-truth optimal values in a simple stochastic environment. If computing ground-truth optimal values is challenging, comparing the value estimates learned by GTRL and TRL, possibly alongside an independently learned estimate of QβQ^\beta as a reference, could also provide useful insights.

[Q4] Interaction with architectural inductive biases.

Prior studies [1–4] have emphasized the importance of architectural inductive biases for learning goal-conditioned value functions, especially in long-horizon tasks. Since GTRL primarily modifies the learning objective, it would be interesting to investigate whether its improvements extend to structured value-function architectures, such as metric- or quasimetric-based parameterizations. Are the benefits of GTRL complementary to those of these architectural inductive biases? Furthermore, to what extent can the GTRL objective itself overcome the limitations of less structured value-function architectures?

References

[1] Wang et al. "Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning." ICML 2023.

[2] Liu et al. "Metric Residual Network for Sample Efficient Goal-Conditioned Reinforcement Learning." AAAI 2023.

[3] Myers et al. "Offline Goal-Conditioned Reinforcement Learning with Quasimetric Representations." NeurIPS 2025.

[4] Kang et al. "Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning." ICML 2026.

5 comments
AuthorsOpenReview ↗

Response to reviewer mc86 (Part 1/2)

We sincerely thank reviewer mc86 for the constructive review. We address each concern and question below:

W1 and Q1. Counterfactual goals with Eq. 8 held fixed

Q1. ... compare the full GTRL objective against a variant that retains the modified target in Equation (8) but samples only in-trajectory goals ... on stitching tasks?

We ran exactly this variant on 11 tasks. It keeps Eq. 8, the expectiles and hh, and samples only in-trajectory goals (prand=0p_{\rm rand}=0).

Table 1. Success rate (%), 4 seeds.

TaskEq. 8, trajectory goals onlyGTRL
pointmaze-large-navigate14.0±3.579.1±6.5
antmaze-large-navigate39.5±2.886.5±1.5
humanoidmaze-medium-navigate39.2±2.975.4±3.7
antmaze-large-stitch6.2±0.830.7±4.0
humanoidmaze-medium-stitch26.7±3.470.4±2.9
humanoidmaze-large-stitch2.8±0.518.0±2.4
antsoccer-arena-stitch0.3±0.237.6±2.0
pointmaze-teleport-navigate17.5±2.037.0±7.2
antmaze-teleport-navigate26.0±2.148.5±2.3
antmaze-teleport-stitch35.2±2.136.1±2.5
pointmaze-teleport-stitch47.1±1.147.3±2.4

With Eq. 8 fixed, counterfactual goals add 36 to 65 points on navigate tasks and 15 to 44 points on stitch tasks. This is the coverage fault of Prop. 2, measured directly, including on the stitch tasks you asked for. On teleport navigate, they add about 20 points, and on teleport stitch, GTRL is slightly better. We will add this table in the appendix of the paper.


W2. Eq. 8 and the link to Q∗Q^\ast

W2. The target proposed in Equation (8) ... appears to be a reasonable but somewhat heuristic design choice ...

The max is not an extra heuristic. The TD target yTDy_{\rm TD} is the composition through the subgoal s′s' (Eq. 6 is Eq. 5 at offset one). So, Eq. 8 is the max over subgoals of Eq. 3, with s′s' as one more candidate. Prop. 3 does link QAQ_A to the optimal value.

Under deterministic dynamics, VDC=V∗V_{\rm DC}=V^\ast (Prop. 1(iii)), so in the tabular setting Qβ(s,a,g)≤QA(s,a,g)≤V∗(s,g)Q^\beta(s,a,g)\le Q_A(s,a,g)\le V^\ast(s,g). The max can only lift a value that is too low.

Under stochastic dynamics, the cap is VDCV_{\rm DC}, and Q3 measures how close GTRL gets to V∗V^\ast. The floor matters too. TRL has none, and in our tabular test (Q3, random walk) it falls below QβQ^\beta on 3,368 of the entries it trains. GTRL never does.


Q2. Why the logged next action?

Q2. Why does the one-step TD target use the logged next action rather than an action selected by the current policy? ...

The logged action keeps the target in sample, so the target never queries the critic at actions the data lacks. GTRL takes its optimism from κ=0.7\kappa=0.7 and the max over subgoals, and from policy extraction (Eq. 4). We tested both of your alternatives. The policy's action π(s′)\pi(s') is lower on five of six tasks, by 3 to 35 points (Table 2). An implicit max, κ=0.7\kappa=0.7 on the TD target as well, loses 9 to 40 points on all four tasks (Appendix E).

Table 2. Success rate (%), 4 seeds.

TaskPolicy action π(s′)\pi(s')GTRL, logged a′a'
pointmaze-large-navigate65.2±4.379.1±6.5
antmaze-large-navigate51.2±7.186.5±1.5
humanoidmaze-large-navigate8.9±3.541.2±9.2
antmaze-large-stitch16.0±0.530.7±4.0
pointmaze-teleport-navigate40.2±7.737.0±7.2
antmaze-teleport-stitch32.9±3.436.1±2.5
AuthorsOpenReview ↗

Response to reviewer mc86 (Part 2/2)

Q3. Direct evidence on overestimation

Q3. Could the authors provide more direct empirical evidence that GTRL reduces value overestimation? ...

We built a grid maze with 137 states and a teleporter like Example 1, so V∗V^\ast is known. We use two kinds of data, each 400 trajectories of 100 steps over 3 seeds. One is a random walk. The other is a noisy expert that heads to random goals and acts at random with probability 0.2, like OGBench's navigate data. We solve the GTRL and TRL losses exactly, so every error comes from the update rule.

We call a pair (s,g)(s,g) lucky when a lucky outcome can shorten its best route, that is VDC>V∗V_{\rm DC}>V^\ast.

Table 3. Teleport maze, mean of 3 seeds. Errors are on the lucky pairs. Entries with no target are out of all 74,528.

DataMethodAverage absolute errorPairs above V∗V^\astOvershoot when aboveEntries with no target
Random walkGTRL0.248.9%0.080
Random walkTRL0.370.3%0.1036,703
Noisy expertGTRL0.0824.8%0.08635
Noisy expertTRL0.217.2%0.089,887

GTRL is much closer to V∗V^\ast on both kinds of data, and it leaves far fewer entries without a target, as Prop. 2 predicts. When they go above V∗V^\ast, GTRL overshoots by no more than TRL. GTRL crosses V∗V^\ast more often since it sits close to V∗V^\ast, while TRL sits far below it. On all other pairs, neither goes above V∗V^\ast. A maze where every move slips sideways with probability 0.2 gives the same pattern. So, GTRL removes much of TRL's error.


Q4. Architectural inductive biases

Q4. ... Are the benefits of GTRL complementary to those of these architectural inductive biases? ...

The objective alone already goes far. With a plain MLP, GTRL beats the quasimetric QRL on 18 of 19 tasks. GTRL changes only the critic's targets, so any critic with values in (0,1)(0,1) can use it.

We tried a quasimetric critic, Q(s,a,g)=exp⁡(−d(ϕ(s,a),ψ(g)))Q(s,a,g)=\exp(-d(\phi(s,a),\psi(g))), with the MRN distance (your [2]) and the action invariance term of TMD (your [3]). We kept every GTRL hyperparameter as tuned for the MLP.

Table 4. Success rate (%) over 4 seeds. QRL is from our Tables 1 and 2.

TaskGTRL, MRN criticQRLGTRL, MLP critic
antmaze-large-navigate25.0 ± 1.567 ± 886.5 ± 1.5
antmaze-large-stitch10.3 ± 1.518 ± 230.7 ± 4.0
humanoidmaze-medium-stitch8.5 ± 0.518 ± 270.4 ± 2.9

A direct swap hurts a lot. So, the two are not complementary out of the box, and we think the objective, hyperparameters, and the architecture would need to be tuned together.

On your second question: the objective carries most of the gain on its own. With a plain MLP, GTRL beats the quasimetric QRL on 18 of 19 tasks. The exception is pointmaze-large-stitch.


We hope these results answer your concerns. If you have any further queries, we would love to answer them.

AuthorsOpenReview ↗

Thank you for your reviews

Dear Reviewer mc86,

We sincerely thank you for your efforts in reviewing our work and for your great support. We have addressed your core concerns in the comments above. We would also like to let you know that the manuscript has been updated, with all revisions marked in blue.

If you feel our revisions have addressed your concerns, we would be truly grateful if you would kindly consider raising your score.

Best regards,

Authors of #42690

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response from authors of #42690

We thank the AC and reviewers Qak1, nhLC, Xjcq, and mc86 for their careful and constructive reviews. We answered every question in the individual replies. We ran every requested experiment and added the results to the revised PDF. All new material is in blue colour. Page numbers below refer to the revised PDF.


Core findings

  1. GTRL's values are calibrated (asked by all four reviewers). We built a 137-state stochastic maze where the true values are known, and solved each loss exactly. GTRL is the closest to V∗V^\ast, with an average absolute error of 0.24 against 0.37 for TRL. It goes above V∗V^\ast on only 1.8% of all pairs. It never goes above VDCV_{\rm DC}, and under deterministic dynamics it never goes above V∗V^\ast, as Prop. 3 states. TRL leaves about half of all entries without a target. GTRL trains every one.
  2. Each design choice is needed. Removing the max of Eq. 8, or replacing it with an average or two separate losses, costs 6 to 62 points. With Eq. 8 kept, removing counterfactual goals costs 36 to 65 points on deterministic tasks and 15 to 44 points on stitch tasks.
  3. GTRL scales to long horizons. On OGBench's three large-scale long-horizon tasks with 1B datasets, under TRL's exact settings, GTRL beats TRL and every other SOTA on all three tasks (85 vs 79, 100 vs 97, 68 vs 51).

What each reviewer asked, and where it is in the paper

ConcernReviewersWhat we foundWhere
Can the max in Eq. 8 overestimate?Qak1, nhLC, Xjcq, mc86Bounded by VDCV_{\rm DC}, small, and closest to V∗V^\astApp. F (p. 23), App. G (pp. 24–26)
Is the hindsight weight exact?Qak1, nhLC, XjcqExact for VβV^\beta. Read from the critic, it gets about 40% of the exact weight's gain with no extra network. The clip is active on under 0.4% of goalsApp. G.3, G.5 (pp. 25–26), H.3 (p. 28)
What does Prop. 3 show?nhLC, mc86A floor that TRL lacks, and a ceiling. We checked both on every entryApp. A.3 (p. 16), G.4 (p. 26)
TD only, average, two lossesQak1, XjcqAll are below GTRL on all six tasksApp. H.2 (p. 27)
Trajectory goals only, and all goals randommc86, XjcqBoth lose up to 66 points. GTRL needs both kinds of goalsApp. H.1 (p. 27)
Cross-trajectory subgoalsXjcqBelow GTRL on all six tasksApp. H.2 (p. 27)
Long-horizon tasksXjcqGTRL beats TRL and SOTA on all threeTable 1, §5.1 (p. 8)
Policy action in the TD targetmc86Below GTRL on five of six tasksApp. H.4 (p. 28)
Quasimetric criticmc86A direct swap loses, so it needs joint tuning. With a plain MLP, GTRL beats QRL on 18 of 19 tasksApp. H.5 (pp. 28–29)

We believe the revision answers the main concerns. Your feedback has made the paper clearer and stronger, and we are grateful for it.

Thank you again for your time.


Best regards, Authors of #42690

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.