- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes GTRL to improve value learning in offline goal-conditioned RL. It adds one-step TD targets, random goals from other trajectories, and hindsight weighting to TRL. The theory analyzes the overestimation and coverage problems and gives convergence results and value bounds under idealized tabular assumptions. GTRL achieves the highest average success rates across three task groups covering 19 OGBench tasks.
Critical strengths
- The paper identifies two problems: stochastic transitions may cause value overestimation, and in-trajectory goal sampling misses cross-trajectory state–goal pairs.
- The method performs well across different tasks, and the ablations support the main components.
Critical weaknesses
- Equation 8 takes the larger of the two targets. This may still cause overestimation, even when the TD target is unbiased, because smaller targets can be pushed up.
- For decomposable goals, optimistic regression may favor lucky outcomes as well as good subgoals. The paper does not clearly explain how these effects are separated.
Questions for authors
- Goal reachability differs from how often a goal is sampled. Why can reachability estimates be used to correct the goal-sampling bias?
- How does using only the TD target compare with using both targets under the same settings? This would help show the benefit of combining them.
5 comments
Response to reviewer Qak1 (Part 2/2)
Q1. Why reachability corrects the sampling bias
Q1. Goal reachability differs from how often a goal is sampled ... correct the goal-sampling bias?
The two differ only by a factor that depends on the goal alone. The hindsight weight is a ratio at a fixed goal, so that factor cancels.
GTRL draws a trajectory goal at offset with the critic's . So, given , the goal is drawn from , the discounted occupancy of the behavior policy. This is the sampling frequency.
To be at , the agent first has to reach , and everything after starts from . So, ([1] Blier et al., 2021). The first factor is reachability, and the second is the same for every start state. In Eq. 9, cancels, and the frequency ratio becomes a reachability ratio exactly. In our tabular test of the maze of Table 1, this identity holds for every pair.
In practice, GTRL reads V from its critic, as our Limitations state (page 9). It still corrects in the right direction (Table 2). The exact weight cuts the lucky pairs above from 10.7% to 6.3%, and ours cuts it to 8.9%, about half of that gain with no extra network. At the teleporter, it moves the value from 0.73 to 0.70, toward .
Table 2. Teleport maze. is the value from the teleporter to the goal, where .
| Weight | Lucky pairs above | |
|---|---|---|
| No weight | 10.7% | 0.73 |
| GTRL, from the critic | 8.9% | 0.70 |
| Exact from | 6.3% | 0.68 |
Ref. [1] Blier, Tallec and Ollivier. Learning successor states and goal-dependent values: A mathematical viewpoint. arXiv:2101.07123, 2021.
Q2. Both targets beat TD alone
Q2. How does using only the TD target ... benefit of combining them.
We ran this ablation and the results are presented below.
- TD only uses on every goal and keeps everything else of GTRL. Table 3 also includes the no max ablation of Fig. 4.
- Average of both: On decomposable goals, the target is in place of the max.
- Two losses: On decomposable goals, the critic is trained with two separate loss terms at half weight each, one fitting at and one fitting at , with no max.
GTRL beats TD on all of the tasks, by ~6.8 points on average (Table 3).
In Table 1, TD only also sits further below in both mazes (−0.30 and −0.31 against −0.23 and −0.26). The composition picks the better route through a subgoal, while TD with the logged next action mostly follows the data's own policy.
It matters how we combine the two targets. The average uses , and two losses fits at and at as separate terms. Both use the TD target. Yet they drop as low as the composition alone, because a low composition target pulls them down. The max keeps the TD target as a floor and lets the composition lift it.
Table 3. Success rate (%), 4 seeds.
| Task | Composition only (no max) | TD only | Average of both | Two losses | GTRL (max) |
|---|---|---|---|---|---|
| pointmaze-large-navigate | 20.0 ± 8.0 | 67.4 ± 5.6 | 22.1 ± 1.8 | 17.6 ± 6.5 | 79.1 ± 6.5 |
| antmaze-large-navigate | 35.3 ± 5.2 | 83.2 ± 2.6 | 32.2 ± 1.8 | 26.2 ± 3.8 | 86.5 ± 1.5 |
| humanoidmaze-large-navigate | 10.3 ± 1.5 | 35.6 ± 7.0 | 14.1 ± 2.5 | 14.1 ± 2.7 | 41.2 ± 9.2 |
| antmaze-large-stitch | 15.7 ± 1.5 | 23.7 ± 2.8 | 14.0 ± 1.4 | 15.9 ± 1.0 | 30.7 ± 4.0 |
| pointmaze-teleport-navigate | 13.6 ± 4.5 | 28.8 ± 5.5 | 12.6 ± 4.5 | 11.3 ± 0.2 | 37.0 ± 7.2 |
| antmaze-teleport-stitch | 25.4 ± 5.1 | 31.1 ± 5.3 | 29.9 ± 2.4 | 25.3 ± 3.1 | 36.1 ± 2.5 |
We hope these results answer your concerns. If you have any further queries, we would love to answer them.
Thank you for your reviews
Dear Reviewer Qak1,
We sincerely thank you for your efforts in reviewing our work and for your great support. We have addressed your core concerns in the comments above. We would also like to let you know that the manuscript has been updated, with all revisions marked in blue.
If you feel our revisions have addressed your concerns, we would be truly grateful if you would kindly consider raising your score.
Best regards,
Authors of #42690
Response to reviewer Qak1 (Part 1/2)
We sincerely thank reviewer
Qak1for the constructive review. We address each concern and question below:Tabular test:W1, W2 and Q1 are about the true values. So, we built a small tabular test where we know them. It is a grid maze with 137 states and the teleporter similar to Example 1 (page 5, line 242-248), which sends the agent next to the goal or to a far corner with probability 1/2 each. We also use the same maze without the teleporter, which is deterministic.For data, we use 400 random walk trajectories of 100 steps, over 3 seeds, with γ=0.95. We solve each variant's loss exactly. So, every error comes from the update rule itself. With random walk data, every variant sits below V∗ on average. So, what matters is how often a variant goes above V∗.
We call a pair (s,g) lucky when a lucky outcome can shorten its best route, that is VDC>V∗. The teleport maze has 3,877 such pairs out of 18,632.
Table 1. Tabular test. The error is maxaQA(s,a,g)−V∗(s,g), averaged over 3 seeds. The teleport columns use the lucky pairs, and the deterministic column uses all pairs.
W1. Can the max in Eq. 8 overestimate?
Under deterministic dynamics,
Prop. 3 shows it cannot. Under stochastic dynamics,it adds no new route for luck, its effect is bounded and small, and removing it is worse.larger target can only lift a value that is too low. In the deterministic maze, no pair goes above V∗.which are TRL's own base cases(Prop. 1).1.8% of all pairs, and it is the closest to V∗ of all variants in Table 1, in both mazes.Removing the max is worse. From Table 1, values drop far below V∗ (−0.39 against −0.23), and a third of all entries fall below the behavior value Qβ, which the
max guarantees as a floor(Prop. 3). In deep RL, removing the max, or replacing it with an average or two separate losses, costs 6 to 62 points (Table 3, part 2).W2. Separating luck from good subgoals
GTRL separates them by the kind of goal.Prop. 3 caps what is left atVDC.Separating the two fully into one decomposable goal would need several successors of the same (s,a), which continuous offline data does not have.