- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Latent Future Goal Representations (LFG) for offline goal-conditioned reinforcement learning. It learns a goal space from set and temporal relations, building on HGA, and reuses its bilinear containment relation as the critic readout. An action-conditioned predictor maps a state-action pair into this space, and the critic scores a desired goal against the predicted future representation. The predictor is trained on discounted hindsight targets without bootstrapping, while the goal space receives no critic gradient.
The central claim is that sharing the containment relation allows a critic trained on singleton goals to generalize to union goals. The paper evaluates this through union-goal experiments before and after actor-only refinement, single-goal OGBench results, representation-loss ablations, and qualitative probes.
Critical strengths
-
The paper identifies a concrete limitation of separately learned goal representations. A downstream critic does not necessarily preserve the structure encoded by its input representation. Reusing the learned relation is a coherent way to connect representation learning and control. The separation of goal-space supervision from critic gradients also makes the proposed mechanism relatively easy to identify.
-
The shared-versus-dense comparison provides useful evidence for the design. Both variants use the same goal-space construction, hindsight targets, and training schedule. On union goals, their average success rates are 59% versus 31% before refinement and 74% versus 35% afterward. These are substantial differences and support the usefulness of the shared readout within the proposed framework, although they do not establish its advantage over explicit composition of singleton critics.
-
The paper reports both positive results and meaningful limitations. LFG achieves the highest pixel average in Table 2 and performs well on several navigation tasks. The authors also acknowledge weak multi-object performance, reduced single-goal success after union-only refinement, and the fact that visual cube-single does not cleanly isolate union-goal generalization because the dense baseline already struggles with singleton goals.
Critical weaknesses
- (main concern) The experiments do not establish what the union embedding adds over explicit singleton composition.
A natural baseline is to construct an action score from the singleton critics, such as
and use it to select or optimize actions. This is not generally an exact union-goal value function, but it is a relevant alternative for the two-goal tasks studied here. No such baseline is included.
The existing comparisons show that the shared readout handles union representations better than the selected downstream critics. They do not show whether representing the union is more useful than querying its members separately. Using LFG’s own singleton critic for this comparison would be particularly informative because it would reduce differences in underlying control quality.
The HGA comparison has a related limitation. Omitting abstract hindsight relabeling is appropriate for testing transfer under matched singleton-only critic training, but it does not establish an advantage over HGA with its published training procedure. The paper should distinguish these two comparisons and provide a full HGA result if it intends to claim a broader methodological advantage.
- The connection between containment and disjunctive goal achievement needs clarification.
Section 2.1 defines (E(x,y)) as scoring whether the set represented by (x) is contained in the set represented by (y). For a union goal, however, the evaluation criterion is to reach either member. Under the stated set interpretation, evaluating (E(A\cup B,F)) asks whether both members are contained in the predicted future set (F), whereas successful control requires reaching only one.
This does not demonstrate that the learned score cannot work empirically. It does mean that reusing the containment relation alone does not explain why it should rank actions appropriately for an OR objective. The paper needs either a clearer semantic justification or a direct diagnostic showing how union scores relate to singleton scores and successful action choices. The reported control results are encouraging, but they do not resolve this conceptual gap.
- The refinement experiments do not fully isolate the source of improvement.
The frozen-critic protocol is useful, but refinement also includes behavior cloning on actions that lead toward one member of each training union, as Section 3 acknowledges. Thus, improvement after refinement is not by itself evidence that the critic supplies the relevant union-goal signal. The dense comparison helps, but a BC-only refinement control would more directly establish the contribution of the shared critic.
The less-used-goal ratio increases from 11% to 17%, compared with 5% and 3% for the refined baselines. This supports increased use of both members, but does not establish that the agent selects the more appropriate member from each start. Equal visitation is not required by the task; a more informative analysis would relate goal selection to distance or reachability from the initial state.
There is also a practical trade-off: refinement lowers average singleton success from 93% to 87%, including 92% to 72% on antsoccer. This deserves explicit treatment when assessing the benefit of adapting the actor to abstract goals.
- The main-text ablations support a narrower conclusion than the paper’s broad framing suggests.
Table 3 shows that set supervision matters within LFG, particularly on visual cube-single, where removing it reduces success from 72% to 5%. However, the temporal-only comparator is LFG’s own successor objective. This supports the benefit of adding set losses to that objective, rather than a general advantage over established temporal representation-learning objectives.
A controlled comparison against a temporal-contrastive objective should be summarized in the main text if it is central to the abstract’s claim. More directly relevant to the main contribution, the paper should report how removing set or successor supervision affects union-goal transfer. Singleton performance alone does not identify which representation properties enable the claimed generalization.
Questions for authors
-
How does LFG compare with explicit composition using its own singleton critic and with a standard singleton-based agent? Please specify the action-selection procedure and any additional refinement budget.
-
What justifies interpreting the containment score of a union in a predicted future as an action value for reaching either member? Can union and singleton scores be compared directly in cases where an action favors only one goal?
-
How much of the actor-refinement gain remains relative to BC-only refinement, and how does HGA perform with its published abstract hindsight relabeling?
-
Can the main text include a controlled temporal-objective comparison and representation-loss ablations evaluated on union goals?
3 comments
Source of the refinement gain (W3, Q3)
We agree that refinement alone does not show that the critic contributes. Policy extraction with DDPG+BC also relies on behaviour cloning, so we added a BC-only refinement (Table 12, Figures 25 to 30). For LFG (linear), BC-only refinement accounts for part of the gain on antmaze-large, where union success rises from 49% to 64%, against 90% with the critic. On antsoccer it comes close to refinement with the critic (33% versus 40%). On visual cube it lowers union success from 39% to 19%, against 70% with the critic. For LFG (logistic), BC-only refinement lowers union success on antsoccer and visual cube and changes it little on antmaze-large (65% to 69%), while refinement with the critic raises it in every environment. The largest gains therefore need the critic. With the linear readout, the union value ranks actions mainly like the nearer member (Table 10), and after refinement the split between members becomes more even (Figures 25 and 27, rollouts in Figures 31 to 34). The lower single-goal success after refinement is a real trade-off. We see it as a capacity limit of the policy, which now has to handle single and abstract goals at the same time.
Temporal-contrastive comparison (W4, Q4)
Appendix B.2.2 (Table 7) already has a controlled comparison with a temporal-contrastive objective. It replaces the relational losses by the TRA alignment loss and keeps goal encoder, , critic and actor unchanged. Relational supervision matches TRA on antmaze-large (82% both) and is clearly ahead on antsoccer (38% versus 12%), cube-single (68% versus 29%) and visual cube-single (72% versus 0%), with smaller gaps on visual antmaze-medium and cube-double. We agree that this belongs next to Table 3 and will move a compact version of Table 7 into Section 4.
Representation ablations on union goals (W4, Q4)
We evaluated the set and successor ablations of both variants on union and reward goals (Appendix C.3, Table 13). The successor loss is itself a containment loss: a trajectory window is encoded, and the relation decides whether the continuation lies in the set predicted by SUCC. In Set, the OR operator is therefore still trained, but only as the input of SUCC. Under the TRA objective, OR is trained only on single states, never on sets of several states, so union goals are untrained there. On union and reward goals, the ablations show the following:
-
Removing the set losses breaks union and reward goals on visual cube for both variants (refined LFG (linear): 11% instead of 70% union success, 28% instead of 77% reward success), together with single-goal control (72% to 5%, Table 3).
-
On the navigation tasks, the successor loss suffices, and Set stays close to or above the full model after refinement.
-
For LFG (linear), removing the successor loss keeps reward success close to the full model on antmaze-large and visual cube, lowers it on antsoccer (61% instead of 70%) and matches or exceeds it on unions (74% instead of 70% on visual cube). For LFG (logistic), it lowers union success after refinement in every environment, for example on visual cube (58% instead of 68%).
The set losses are therefore necessary on visual cube and are not needed for union and reward goals on the navigation tasks.
Planned changes to the paper
-
Section 2: We will introduce both readouts and explain how each scores a union goal, with reference to the union--member diagnostics in Appendix C.1.
-
Section 4: We will report pick best, the BC-only control and HGA refined with its abstract hindsight relabeling together with the union results. We will also move a compact version of the TRA comparison (Table 7) next to Table 3 and summarize the set and successor ablations on union and reward goals (Table 13).
-
Discussion: We will treat the drop in single-goal success after refinement as an explicit trade-off, which is most visible on antsoccer.
I thank the authors for addressing my questions and concerns. My main concerns have been resolved, so I raise my score accordingly.
We thank the reviewer for pointing out the mismatch between containment of a union and reaching one of its members, which led us to add a second variant of LFG with a linear readout. All new results are in Appendix C (in blue). At the end of this response, we list how we will integrate them into the main text.
Explicit composition and HGA (W1, Q1, Q3)
We added a comparison with an agent that selects a single member of the union goal set with LFG's own critic ("pick best", Table 12 and Figures 28 to 30). For each member, the pretrained actor proposes an action and the critic scores it for that member. The best member is selected once and kept for the whole episode. No additional training is needed. Pick best has the highest average union success of every model in every environment. The refined LFG (linear) actor, which only sees the union goal, comes close on antmaze-large (90% versus 93%) and visual cube (70% versus 78%). On antsoccer, pick best is clearly ahead (74% versus 40%). On the reward-defined sets, the refined union actor is on par on antmaze-large (95% versus 96%) and antsoccer (70% versus 69%) and ahead on visual cube (77% versus 61%).
Pick best is a strong reference when the members are known. The union goal enables an efficient representation of the whole set as a single latent vector. The gripper rewards in visual cube show where this helps (Figure 37). Each member is a full state and therefore also specifies a cube position that is irrelevant for the reward. In the data, gripper and cube positions are highly correlated, so most members of the union goal place the cube inside the region together with the gripper (87%). The refined agent carries the cube along, and the gripper reaches the region in 85% of the episodes (mean over three seeds, Figure 30). Pick best mostly selects a member with the cube outside the region (66%), so the gripper has to reach the region without the cube, and it succeeds in 59% of the episodes. Since actor and critic read the union goal directly, abstract goals could also be provided by a system that composes goals in the shared goal space without listing members. We leave this for future work.
We also refined HGA with its published abstract hindsight relabeling ("HGA, abstract goals" in Table 12). On union goals it stays below HGA refined with our union refinement on antmaze-large (35% versus 53%) and antsoccer (25% versus 28%), and both stay low on visual cube. We think this is due to our protocol, in which only the actor is refined and the critic stays fixed. HGA normally trains representation, actor and critic jointly, so its critic has to learn about abstract goals itself, while LFG shares the relation between critic and goal space.
Refinement fine-tunes only the policy for 50k additional steps on the frozen representation and critic, compared with 1M pretraining steps for state-based and 0.5M for pixel-based tasks. The policy outputs actions directly, without any additional action selection.
Containment and disjunctive goals (W2, Q2)
The mismatch between containment of a union and reaching one of its members comes from the sigmoid of the logistic readout. A union goal lies near the mean of its members, so its logit is roughly the mean of the member logits, and the sigmoid turns this average into a soft AND. The linear readout drops the sigmoid and fits the bilinear score directly to the containment labels and the discounted critic targets. The value of a union is then approximately the mean of the member values, which acts as a soft OR. Because the targets are discounted, the nearer member has the higher value and pulls the actor towards it. We report both variants, LFG (logistic) and LFG (linear). The logistic readout performs better on long-horizon single goals (antmaze-giant, antsoccer, humanoidmaze-medium, Table 11). The linear readout gives unions the intended OR semantics, improves cube-double and scene, and after refinement reaches higher union and reward success on antmaze-large, similar success on visual cube and lower success on antsoccer (Table 12).
At fixed state-action pairs (Appendix C.1), the union value of LFG (linear) follows the member mean (Table 8), lies closer to the nearer member on antmaze-large and visual cube (Table 9) and ranks actions like the nearer member (Table 10). The logistic readout weighs both members alike, and a dense critic shows little relation between union and member values.