acceptodds
Under review as a conference paper at ICLR 2027

Latent Future Goal Representations

Abstract

Goals in goal-conditioned reinforcement learning are usually single observations, yet abstract tasks are more naturally stated as sets of outcomes, for example reaching either of two states. Such abstract goals can be represented in a latent goal space, but critic and policy have no direct way to use this structure. Typically, they must learn again from value targets what those representations mean, so that an agent trained on single goals, for example, does not generalize to more abstract set-valued goals. We introduce Latent Future Goal Representations (LFG), which learns a shared goal space without critic gradients and trains an action-conditioned predictor to map state-action pairs directly into it. Each prediction represents a latent goal region, and a learned bilinear containment relation scores whether a desired goal belongs to that region. These containment scores serve directly as the critic for policy learning. The predictor is trained on observed future goals, with targets discounted by the time until their arrival. In experiments we find that our shared representation enables the critic to inherit the learned goal space structure, allowing generalization to abstract goals. On OGBench with single observation goals, LFG achieves competitive performance across navigation and manipulation with both state and image observations. Ablations show that relational supervision improves control over temporal supervision alone, while the learned containment relation provides an effective critic readout.

Primary area:
reinforcement learning
Keywords:
goal-conditioned RL, goal representation learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mEKFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Latent Future Goal Representations (LFG) for offline goal-conditioned reinforcement learning. It learns a goal space from set and temporal relations, building on HGA, and reuses its bilinear containment relation as the critic readout. An action-conditioned predictor maps a state-action pair into this space, and the critic scores a desired goal against the predicted future representation. The predictor is trained on discounted hindsight targets without bootstrapping, while the goal space receives no critic gradient.

The central claim is that sharing the containment relation allows a critic trained on singleton goals to generalize to union goals. The paper evaluates this through union-goal experiments before and after actor-only refinement, single-goal OGBench results, representation-loss ablations, and qualitative probes.

Critical strengths
  1. The paper identifies a concrete limitation of separately learned goal representations. A downstream critic does not necessarily preserve the structure encoded by its input representation. Reusing the learned relation is a coherent way to connect representation learning and control. The separation of goal-space supervision from critic gradients also makes the proposed mechanism relatively easy to identify.

  2. The shared-versus-dense comparison provides useful evidence for the design. Both variants use the same goal-space construction, hindsight targets, and training schedule. On union goals, their average success rates are 59% versus 31% before refinement and 74% versus 35% afterward. These are substantial differences and support the usefulness of the shared readout within the proposed framework, although they do not establish its advantage over explicit composition of singleton critics.

  3. The paper reports both positive results and meaningful limitations. LFG achieves the highest pixel average in Table 2 and performs well on several navigation tasks. The authors also acknowledge weak multi-object performance, reduced single-goal success after union-only refinement, and the fact that visual cube-single does not cleanly isolate union-goal generalization because the dense baseline already struggles with singleton goals.

Critical weaknesses
  1. (main concern) The experiments do not establish what the union embedding adds over explicit singleton composition.

A natural baseline is to construct an action score from the singleton critics, such as

Qcomp(s,a)=max⁡(Q(s,a,gA),Q(s,a,gB))Q_{\mathrm{comp}}(s,a)=\max{(Q(s,a,g_A),Q(s,a,g_B))}

and use it to select or optimize actions. This is not generally an exact union-goal value function, but it is a relevant alternative for the two-goal tasks studied here. No such baseline is included.

The existing comparisons show that the shared readout handles union representations better than the selected downstream critics. They do not show whether representing the union is more useful than querying its members separately. Using LFG’s own singleton critic for this comparison would be particularly informative because it would reduce differences in underlying control quality.

The HGA comparison has a related limitation. Omitting abstract hindsight relabeling is appropriate for testing transfer under matched singleton-only critic training, but it does not establish an advantage over HGA with its published training procedure. The paper should distinguish these two comparisons and provide a full HGA result if it intends to claim a broader methodological advantage.

  1. The connection between containment and disjunctive goal achievement needs clarification.

Section 2.1 defines (E(x,y)) as scoring whether the set represented by (x) is contained in the set represented by (y). For a union goal, however, the evaluation criterion is to reach either member. Under the stated set interpretation, evaluating (E(A\cup B,F)) asks whether both members are contained in the predicted future set (F), whereas successful control requires reaching only one.

This does not demonstrate that the learned score cannot work empirically. It does mean that reusing the containment relation alone does not explain why it should rank actions appropriately for an OR objective. The paper needs either a clearer semantic justification or a direct diagnostic showing how union scores relate to singleton scores and successful action choices. The reported control results are encouraging, but they do not resolve this conceptual gap.

  1. The refinement experiments do not fully isolate the source of improvement.

The frozen-critic protocol is useful, but refinement also includes behavior cloning on actions that lead toward one member of each training union, as Section 3 acknowledges. Thus, improvement after refinement is not by itself evidence that the critic supplies the relevant union-goal signal. The dense comparison helps, but a BC-only refinement control would more directly establish the contribution of the shared critic.

The less-used-goal ratio increases from 11% to 17%, compared with 5% and 3% for the refined baselines. This supports increased use of both members, but does not establish that the agent selects the more appropriate member from each start. Equal visitation is not required by the task; a more informative analysis would relate goal selection to distance or reachability from the initial state.

There is also a practical trade-off: refinement lowers average singleton success from 93% to 87%, including 92% to 72% on antsoccer. This deserves explicit treatment when assessing the benefit of adapting the actor to abstract goals.

  1. The main-text ablations support a narrower conclusion than the paper’s broad framing suggests.

Table 3 shows that set supervision matters within LFG, particularly on visual cube-single, where removing it reduces success from 72% to 5%. However, the temporal-only comparator is LFG’s own successor objective. This supports the benefit of adding set losses to that objective, rather than a general advantage over established temporal representation-learning objectives.

A controlled comparison against a temporal-contrastive objective should be summarized in the main text if it is central to the abstract’s claim. More directly relevant to the main contribution, the paper should report how removing set or successor supervision affects union-goal transfer. Singleton performance alone does not identify which representation properties enable the claimed generalization.

Questions for authors
  1. How does LFG compare with explicit composition using its own singleton critic and with a standard singleton-based agent? Please specify the action-selection procedure and any additional refinement budget.

  2. What justifies interpreting the containment score of a union in a predicted future as an action value for reaching either member? Can union and singleton scores be compared directly in cases where an action favors only one goal?

  3. How much of the actor-refinement gain remains relative to BC-only refinement, and how does HGA perform with its published abstract hindsight relabeling?

  4. Can the main text include a controlled temporal-objective comparison and representation-loss ablations evaluated on union goals?

3 comments

We thank the reviewer for pointing out the mismatch between containment of a union and reaching one of its members, which led us to add a second variant of LFG with a linear readout. All new results are in Appendix C (in blue). At the end of this response, we list how we will integrate them into the main text.

Explicit composition and HGA (W1, Q1, Q3)

We added a comparison with an agent that selects a single member of the union goal set with LFG's own critic ("pick best", Table 12 and Figures 28 to 30). For each member, the pretrained actor proposes an action and the critic scores it for that member. The best member is selected once and kept for the whole episode. No additional training is needed. Pick best has the highest average union success of every model in every environment. The refined LFG (linear) actor, which only sees the union goal, comes close on antmaze-large (90% versus 93%) and visual cube (70% versus 78%). On antsoccer, pick best is clearly ahead (74% versus 40%). On the reward-defined sets, the refined union actor is on par on antmaze-large (95% versus 96%) and antsoccer (70% versus 69%) and ahead on visual cube (77% versus 61%).

Pick best is a strong reference when the members are known. The union goal enables an efficient representation of the whole set as a single latent vector. The gripper rewards in visual cube show where this helps (Figure 37). Each member is a full state and therefore also specifies a cube position that is irrelevant for the reward. In the data, gripper and cube positions are highly correlated, so most members of the union goal place the cube inside the region together with the gripper (87%). The refined agent carries the cube along, and the gripper reaches the region in 85% of the episodes (mean over three seeds, Figure 30). Pick best mostly selects a member with the cube outside the region (66%), so the gripper has to reach the region without the cube, and it succeeds in 59% of the episodes. Since actor and critic read the union goal directly, abstract goals could also be provided by a system that composes goals in the shared goal space without listing members. We leave this for future work.

We also refined HGA with its published abstract hindsight relabeling ("HGA, abstract goals" in Table 12). On union goals it stays below HGA refined with our union refinement on antmaze-large (35% versus 53%) and antsoccer (25% versus 28%), and both stay low on visual cube. We think this is due to our protocol, in which only the actor is refined and the critic stays fixed. HGA normally trains representation, actor and critic jointly, so its critic has to learn about abstract goals itself, while LFG shares the relation between critic and goal space.

Refinement fine-tunes only the policy for 50k additional steps on the frozen representation and critic, compared with 1M pretraining steps for state-based and 0.5M for pixel-based tasks. The policy outputs actions directly, without any additional action selection.

Containment and disjunctive goals (W2, Q2)

The mismatch between containment of a union and reaching one of its members comes from the sigmoid of the logistic readout. A union goal lies near the mean of its members, so its logit is roughly the mean of the member logits, and the sigmoid turns this average into a soft AND. The linear readout drops the sigmoid and fits the bilinear score directly to the containment labels and the discounted critic targets. The value of a union is then approximately the mean of the member values, which acts as a soft OR. Because the targets are discounted, the nearer member has the higher value and pulls the actor towards it. We report both variants, LFG (logistic) and LFG (linear). The logistic readout performs better on long-horizon single goals (antmaze-giant, antsoccer, humanoidmaze-medium, Table 11). The linear readout gives unions the intended OR semantics, improves cube-double and scene, and after refinement reaches higher union and reward success on antmaze-large, similar success on visual cube and lower success on antsoccer (Table 12).

At fixed state-action pairs (Appendix C.1), the union value of LFG (linear) follows the member mean (Table 8), lies closer to the nearer member on antmaze-large and visual cube (Table 9) and ranks actions like the nearer member (Table 10). The logistic readout weighs both members alike, and a dense critic shows little relation between union and member values.

Source of the refinement gain (W3, Q3)

We agree that refinement alone does not show that the critic contributes. Policy extraction with DDPG+BC also relies on behaviour cloning, so we added a BC-only refinement (Table 12, Figures 25 to 30). For LFG (linear), BC-only refinement accounts for part of the gain on antmaze-large, where union success rises from 49% to 64%, against 90% with the critic. On antsoccer it comes close to refinement with the critic (33% versus 40%). On visual cube it lowers union success from 39% to 19%, against 70% with the critic. For LFG (logistic), BC-only refinement lowers union success on antsoccer and visual cube and changes it little on antmaze-large (65% to 69%), while refinement with the critic raises it in every environment. The largest gains therefore need the critic. With the linear readout, the union value ranks actions mainly like the nearer member (Table 10), and after refinement the split between members becomes more even (Figures 25 and 27, rollouts in Figures 31 to 34). The lower single-goal success after refinement is a real trade-off. We see it as a capacity limit of the policy, which now has to handle single and abstract goals at the same time.

Temporal-contrastive comparison (W4, Q4)

Appendix B.2.2 (Table 7) already has a controlled comparison with a temporal-contrastive objective. It replaces the relational losses by the TRA alignment loss and keeps goal encoder, WW, critic and actor unchanged. Relational supervision matches TRA on antmaze-large (82% both) and is clearly ahead on antsoccer (38% versus 12%), cube-single (68% versus 29%) and visual cube-single (72% versus 0%), with smaller gaps on visual antmaze-medium and cube-double. We agree that this belongs next to Table 3 and will move a compact version of Table 7 into Section 4.

Representation ablations on union goals (W4, Q4)

We evaluated the set and successor ablations of both variants on union and reward goals (Appendix C.3, Table 13). The successor loss is itself a containment loss: a trajectory window is encoded, and the relation decides whether the continuation lies in the set predicted by SUCC. In −-Set, the OR operator is therefore still trained, but only as the input of SUCC. Under the TRA objective, OR is trained only on single states, never on sets of several states, so union goals are untrained there. On union and reward goals, the ablations show the following:

  • Removing the set losses breaks union and reward goals on visual cube for both variants (refined LFG (linear): 11% instead of 70% union success, 28% instead of 77% reward success), together with single-goal control (72% to 5%, Table 3).

  • On the navigation tasks, the successor loss suffices, and −-Set stays close to or above the full model after refinement.

  • For LFG (linear), removing the successor loss keeps reward success close to the full model on antmaze-large and visual cube, lowers it on antsoccer (61% instead of 70%) and matches or exceeds it on unions (74% instead of 70% on visual cube). For LFG (logistic), it lowers union success after refinement in every environment, for example on visual cube (58% instead of 68%).

The set losses are therefore necessary on visual cube and are not needed for union and reward goals on the navigation tasks.

Planned changes to the paper

  • Section 2: We will introduce both readouts and explain how each scores a union goal, with reference to the union--member diagnostics in Appendix C.1.

  • Section 4: We will report pick best, the BC-only control and HGA refined with its abstract hindsight relabeling together with the union results. We will also move a compact version of the TRA comparison (Table 7) next to Table 3 and summarize the set and successor ablations on union and reward goals (Table 13).

  • Discussion: We will treat the drop in single-goal success after refinement as an explicit trade-off, which is most visible on antsoccer.

I thank the authors for addressing my questions and concerns. My main concerns have been resolved, so I raise my score accordingly.

Reviewer_JF75OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In offline GCRL, existing methods struggle to generalize to unseen abstract goals because, even when the goal representation learns the structure of abstract goals, the downstream critic still needs to reinterpret this structure. The authors propose LFG, which represents single-observation goals and abstract goals in the same latent goal space and allows the critic to directly use the containment relation learned in this space, reducing the need for a separate reinterpretation process. Furthermore, critic training is decoupled from the goal representation to prevent it from modifying the learned set structure. The authors evaluate single-goal performance and generalization to unseen two-goal unions in the navigation and manipulation environments of OGBench. After training all models on single goals, they freeze the critic and refine only the actor for union goals. LFG (shared) achieves greater improvements than LFG (dense) and HGA+CRL while maintaining competitive single-goal performance.

Critical strengths
  1. Problem Formulation and Core Idea: The authors clearly identify the problem that, even when the goal representation has already learned the abstract set structure, the downstream critic still needs to relearn this structure through value targets. Their proposed solution of directly using the learned relations in the goal space as the critic is intuitive and clearly distinguishes their approach from existing HGA-based methods. In particular, the key novelty of the paper appears to lie in its design, which enables the critic to score abstract goals even when these goals are not directly encountered during critic training.
  2. Design of Goal Representation and Critic: The proposed method represents single goals, unions, and predicted futures in a shared latent space, where all are compared using the same containment relation. The method is designed to use this relation directly as the critic. Rather than following a two-stage process of first learning a goal representation and then interpreting it, the proposed approach directly uses the learned relation as the critic during goal representation learning. This design establishes a clear connection between the problem identified by the authors and their proposed solution.
  3. Comparative Experiments on the Core Design: LFG (shared) and LFG (dense) use the same goal features but differ only in their critic readout designs. This provides a strong experimental comparison for evaluating whether directly sharing the learned relation is more effective than training a separate dense critic. Indeed, the shared readout achieves higher single-goal and abstract-goal performance than the dense critic across multiple tasks. Furthermore, in the abstract-goal experiments, the critic is frozen while only the actor is refined for union goals. This setup directly evaluates whether a critic with a shared representation can provide useful supervision for previously unseen union goals. These comparative experiments support the effectiveness of the shared containment relation.
  4. Competitive Performance on Single-Observation Goals: Although LFG aims to generalize to abstract goals, it also demonstrates competitive performance against existing baselines across various navigation and manipulation environments on the standard single-observation goal tasks in OGBench. This supports the practical applicability of the proposed structured goal space and shared critic in conventional GCRL settings.
Critical weaknesses
  1. Limited Evaluation on Abstract Goals: Using two goals, A and B, as an example of an abstract goal and making a binary decision about which goal to focus on may be useful for experimental design, but the scope is too narrow to serve as sufficient evidence of generalization. In particular, HGA, the most directly related prior work, does not restrict abstract goals to simple two-goal binary unions but instead investigates novel-task generalization through observation sequences and hierarchical abstractions at various levels. Although this paper presents an abstract goal formulation that includes singletons, regions, subsets, unions, and other structures, and even trains various relational structures such as Membership, Nesting, Union, and Successor in its method, it is questionable that the experimental evaluation is limited to two-goal unions. Additional experiments involving N-goal unions, region goals, nested goals, unions of regions, and other more complex goal structures appear necessary to demonstrate generalization to abstract goals more convincingly.
  2. Lack of Direct Analysis of Critic Scores for Abstract Goals: The core contribution of this paper is not simply that the goal representation accurately captures abstract goals, but rather that directly using this information in the critic eliminates the need for a separate embedding-and-interpretation process. Figure 5 and Appendices B.4 and B.5 demonstrate that the spatial extent of the critic arises from the action-conditioned future representation rather than local similarity in the goal encoder. Furthermore, Figures 14-17 in Appendix B.7 show that the goal representation appropriately captures union goals by containing both constituent goals in the goal space. However, these results provide only indirect evidence and do not directly verify the paper's central claim regarding how the shared critic evaluates unseen abstract goals. To validate this claim more directly, the authors should analyze how the critic scores individual single goals and their union goal while keeping the state-action pair (s,a) fixed.
  3. Insufficient Explanation of the Overall Methodology and Core Mechanisms: The proposed method consists of several modules, including the goal encoder, OR/SUCC operators, containment relation, action-conditioned future predictor, and policy. However, it is difficult to obtain a comprehensive understanding of the roles of these modules, how they interact, and which losses update which parameters. In particular, although both SUCC and the action-conditioned future predictor are related to future states, their actual roles differ substantially, and this distinction needs to be explained more clearly. Furthermore, although the caption and main text explain that the actions differ for each goal in the probe results shown in Figure 5, this is difficult to recognize intuitively from the figure itself. Adding an end-to-end architecture figure that clearly illustrates the overall network structure, connections between modules, gradient flow, and frozen components, along with clearer figure annotations, would substantially improve the clarity and understanding of the proposed method.
Questions for authors
  1. In HGA, abstract hindsight relabeling is one of the mechanisms that enables the policy and critic to effectively utilize abstract goals, and I understand the decision to remove it for a fair comparison. However, it would also be interesting to see a performance comparison with the original HGA, including abstract hindsight relabeling.
  2. In Table 2, LFG's single-goal performance varies considerably across tasks. Although the authors partially attribute this variation to insufficient spatial precision in multi-object tasks and object manipulation, it would be useful to understand what other factors contribute to these task-dependent performance differences. Given that most hyperparameters are configured similarly across tasks, further analysis seems necessary to determine whether these differences arise primarily from hyperparameter tuning, the structure of the goal space, or the characteristics of individual tasks.
3 comments

We thank the reviewer for the detailed review and the constructive suggestions. All new results are in Appendix C (in blue). At the end of this response, we list how we will integrate them into the main text.

More complex abstract goals (W1)

Following HGA, we extended the evaluation on antmaze-large, antsoccer and visual cube to unions of up to four goals and to reward-defined goal sets on the agent and on the object, such as halves, bands, walls, junctions, dead ends, a ring and the four corners (Figures 22 to 24). A reward goal is given as the union of all rewarded validation states.

We also added a second variant of LFG with a linear readout, LFG (linear), and report it next to the logistic readout of the submitted version, LFG (logistic). With the sigmoid of the logistic readout, a union scores as if every member had to lie in the predicted future. The linear readout fits the bilinear score directly to the containment labels and the discounted targets, so the value of a union is approximately the mean of the member values, which matches union semantics. With the linear readout, refinement raises union success from 49% to 90% on antmaze-large and from 39% to 70% on visual cube, and reward success improves as well. With a dense critic on the same goal space, union success after refinement stays at or below 29% for the linear readout and 26% for the logistic one (Table 12). Per-set results are in Figures 25 to 30 and rollouts in Figures 31 to 36.

Critic scores of unions and members (W2)

As suggested, we compared the critic scores of union goals and their members at fixed (s,a)(s,a) on the evaluation unions, for LFG (linear), LFG (logistic) and the dense ablation (Appendix C.1, Tables 8 to 10):

  • LFG (linear): the member mean explains most of the variance of the union value (R2R^2 up to 0.88, Table 8), and the union value lies closer to the nearer member on antmaze-large and visual cube (Table 9). Over actions, the union ranks like its nearer member (Table 10).

  • LFG (logistic): the union follows the mean member logit. In probability space it falls far below the nearer member, with negative R2R^2 against the member mean on the navigation tasks when one member is nearer. Over actions, it weighs both members alike.

  • Dense critic: no consistent relation between union and member values.

Roles of the modules (W3)

We added an architecture overview (Figure 38) that shows which loss trains which module. Representation, critic and actor are trained at the same time, and stop-gradients determine which parameters each loss updates:

  • The set losses (membership, nesting, identity, union consistency) and the successor loss train the encoder, the OR and SUCC operators and the relation matrix WW.

  • The critic loss trains only the future predictor ψ\psi. Goals and relation enter without gradient.

  • The actor loss trains only the policy, which receives the goals as fixed inputs.

  • Union refinement afterwards trains only the actor, on union goals from the frozen OR operator.

SUCC and ψ\psi both concern the future but have different roles. SUCC is a goal-space operator that maps a goal to the goal of its continuation and is only used to train the representation. ψ\psi predicts the future of a state-action pair and is the action-dependent part of the critic.

HGA with abstract hindsight relabeling (Q1)

We added a refinement of HGA that uses its abstract hindsight relabeling instead of our union refinement ("HGA, abstract goals" in Table 12 and Figures 25 to 30). On union goals it stays below HGA with our refinement on antmaze-large (35% versus 53%) and antsoccer (25% versus 28%), and both stay low on visual cube (9% and 8%). On the visual cube reward goals it keeps the success of the pretrained actor (50% versus 49%), while our refinement lowers it to 29%. With either refinement, HGA stays clearly below the shared LFG models on union goals. We think this is due to our protocol, which freezes the critic and fine-tunes only the policy, while HGA normally trains representation, actor and critic jointly. HGA's single-goal results on the state mazes (Table 2) suggest that its pretraining itself worked.

Task-dependent single-goal performance (Q2)

The causes differ between tasks. In the rollouts we inspected, most failures come down to the resolution of the representation or the horizon of critic training:

  • Antsoccer: many failures happen in the last steps, when the ant has pushed the ball in the right direction but narrowly misses the goal. This points to limited resolution.

  • Long-horizon navigation: nearby goals are reached reliably and success decreases with distance. For distant goals the value differences between actions are small, so the critic gives little guidance.

  • Manipulation: cube positions are not resolved precisely enough for the required accuracy.

Policy extraction plays a role too, since CRL, which also uses DDPG+BC, shows similar weaknesses on the same tasks. The linear readout shifts the balance (Table 11). It clearly improves cube-double and scene and is weaker on long-horizon locomotion, where the small values of distant goals are fitted less precisely by squared error. We will discuss this as a limitation.

Planned changes to the paper

  • Section 2: We will move the architecture overview (Figure 38) into the main text and add a short paragraph on which loss trains which module. It will also explain the different roles of SUCC, which only trains the representation, and ψ\psi, the action-conditioned part of the critic. Section 2 will further introduce both readouts and refer to the direct analysis of union and member scores (Tables 8 to 10).

  • Section 4: We will summarize the unions of more than two goals and the reward-defined goal sets on agent and object. The per-set results stay in the appendix.

  • Discussion: We will report HGA refined with its abstract hindsight relabeling and discuss the task-dependent single-goal performance, including the trade-off between the two readouts.

  • Figure 5: We will update the figure to make clear that the action changes with each goal.

I thank the authors for addressing my questions and conducting the additional experiments I requested. My concerns have been resolved, and I will accordingly raise my score.

Reviewer_1hytOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Latent Future Goal Representations (LFG), a framework for offline goal-conditioned reinforcement learning that aims to generalize from individual goals to more abstract, set-valued goals. The main idea is to connect the learned goal representation directly to the critic, rather than treating it as an input feature for a separately trained value function. LFG learns a latent goal space with a containment relation, where individual observations, regions, and unions of goals can be represented in the same space. An action-conditioned predictor then maps state-action pairs into this latent space to represent their possible future outcomes. The containment score between a desired goal and the predicted future is used directly as the critic for policy optimization. The authors evaluate LFG on OGBench across state-based and image-based navigation and manipulation tasks. The main experiments focus on union goals, where reaching either of two targets is considered successful. The results show that LFG generalizes better to these goals than baselines using separately trained critics, particularly after refining the actor while keeping the critic frozen. The paper also reports competitive performance on standard single-goal tasks and provides ablations on the representation objectives and critic architecture.

Critical strengths

S1. A meaningful connection between structured goal representations and critic learning.

I find the central idea of the paper interesting. Existing methods can learn structured goal representations, but a separately trained critic may not automatically preserve the relationships encoded in those representations.

Rather than training another network to interpret the goal features, LFG predicts future outcomes into the same latent space and uses the learned containment relation directly for value estimation. This is a reasonable way to transfer the structure learned by the representation model into the policy learning process.

The approach is also relatively simple, and it does not require additional value targets for abstract goals. I see this as the main technical contribution of the paper.

** S2. The main experimental comparison provides useful evidence for the shared critic design.**

One strength of the experimental evaluation is the comparison between LFG (shared) and LFG (dense), which use the same goal representation but different critic readouts.

As shown in Table 1 and Figure 4, LFG (shared) improves its average union-goal success rate from 59% to 74% after actor refinement, whereas LFG (dense) improves only from 31% to 35%. This is a meaningful difference, particularly because both methods use the same underlying goal representation.

The representation ablations are also informative. For example, removing the set-based objectives substantially reduces performance on cube-single, particularly in the visual setting (72% to 5%, Table 3). These results support the importance of relational supervision rather than only temporal information.

Critical weaknesses

W1. The relationship between set containment and the success condition of union goals needs further justification.

My main technical concern is the interpretation of the containment-based critic.

In Equation (1), the learned relation (E(x,y)) is defined to represent whether the set associated with (x) is contained in the set associated with (y). Equation (2) then uses this relation to evaluate whether a desired goal is contained in the predicted future of a state-action pair.

However, the abstract-goal experiments define a union goal as successful when the agent reaches either of two targets. From a set-theoretic perspective, reaching either target corresponds to having a reachable outcome in the union of the two goal sets. This is not generally equivalent to requiring the entire union to be contained in the predicted future.

For example, if an action can lead to goal A but not goal B, it may still be a good action for the union goal A or B. Under a strict containment interpretation, however, the union would not be contained in the predicted future.

I understand that the learned relation is a soft score rather than an exact logical operator, and the empirical results indicate that it can provide useful guidance. Nevertheless, I think the paper should explain more clearly why this particular containment direction is appropriate for disjunctive goal satisfaction.

A direct analysis comparing the critic scores for A, B, and their union, especially in situations where only one goal is reachable, would help address this concern.

W2. The abstract-goal evaluation does not sufficiently establish the advantage over simpler goal composition strategies, and generalization remains limited.

The abstract-goal experiments focus exclusively on unions of two goals across four selected environments. While this is a reasonable starting point, I am not convinced that these experiments fully establish the need for the proposed representation-based composition.

In particular, when the two constituent goals are available, a relatively simple alternative is to evaluate the individual goals separately and select which one to pursue using their estimated values. Another possibility is to combine individual goal values using a maximum operator for a disjunctive objective.

Such approaches would not require learning a latent union representation and seem especially relevant to the two-goal settings considered in this paper. I would have liked to see at least one such baseline to better understand the additional benefit of LFG.

The results also suggest some limitations in more complicated environments. In Table 2, LFG achieves only 3% success on state-based cube-double and 17% on scene, compared with 60% and 72% for DGR, respectively. Although the authors acknowledge the limited spatial precision of the learned representation, these results raise questions about whether the approach can handle more complex abstract goals involving multiple objects.

Finally, while actor refinement improves union-goal performance, the resulting policy still shows a preference for one member of the goal pair. The less-used goal ratio reaches only 17% on average after refinement (Figure 4). This is an improvement over the baselines, but it suggests that the policy does not yet treat the two alternatives equally flexibly.

I think the main claim would be stronger with a simple compositional baseline and a broader evaluation of set-valued goals beyond pairs of spatial targets.

Questions for authors

Q1. Could the authors clarify the intended semantics of the containment-based critic for union goals? Specifically, if a predicted future contains goal A but not goal B, why should the containment score of the union A or B provide an appropriate value estimate for a task that only requires reaching one of them?

Q2. Have the authors considered a baseline that composes separately estimated single-goal values, for example by selecting the higher-valued goal or using a maximum over the two goal-conditioned Q-values? Since the evaluation unions are constructed from known constituent goals, I think this comparison would help clarify what the learned union representation contributes beyond simple goal selection.

Q3. The abstract-goal results are limited to unions of two goals. How does the method behave when the union contains three or more goals, or when the goal regions have different sizes or distances? Does the learned containment relation preserve the intended behavior as the number of goal alternatives increases?

Q4. The performance on cube-double and scene is substantially lower than that of some existing methods. Could the authors provide further analysis of whether these failures originate primarily from the goal encoder, the learned containment relation, or the future predictor? For example, would stronger within-trajectory negative sampling improve the precision of object-level goal representations?

4 comments

We thank the reviewer for the careful reading and for pointing out the gap between containment and disjunctive success, which led us to add a second variant of LFG with a linear readout. All new results are in Appendix C (in blue). At the end of this response, we list how we will integrate them into the main text.

Union semantics and the linear readout (W1, Q1)

The reviewer is right that requiring the whole union to lie in the predicted future is not the same as reaching one member. In the submitted LFG, relation and critic apply a sigmoid to the bilinear score, E(g,g′)=σ(g⊤Wg′)E(\mathbf{g},\mathbf{g}')=\sigma(\mathbf{g}^\top W\mathbf{g}') and Q(s,a,g)=σ(g⊤Wψ(s,a))Q(s,a,\mathbf{g})=\sigma(\mathbf{g}^\top W\psi(s,a)), and are trained with BCE. A union G=OR({gi}i)\mathbf{G}=\mathrm{OR}(\lbrace \mathbf{g}_i\rbrace _i) lies close to the mean of its members, so its logit is roughly the mean of the member logits. After the sigmoid, a single unreachable member pulls the union far below its best member, so the union behaves more like an AND. The actor is trained through the logits, which is probably why this readout still works reasonably well in practice.

The linear readout drops the sigmoid. It fits the bilinear score by squared error to the containment labels (1 or 0) and to the discounted targets γd−1\gamma^{d-1}. Then Q(s,a,G)≈1n∑iQ(s,a,gi)Q(s,a,\mathbf{G})\approx\frac1n\sum_i Q(s,a,\mathbf{g}_i). The value is high as soon as one member is reachable, unreachable members only dilute it, and because of discounting the nearer member contributes most, so the union acts as a soft OR. We report both variants, LFG (logistic) and LFG (linear). The logistic readout performs better on long-horizon single goals (antmaze-giant, antsoccer, humanoidmaze-medium, Table 11). The linear readout gives unions the intended OR semantics, improves cube-double and scene, and after refinement reaches higher union and reward success on antmaze-large, similar success on visual cube and lower success on antsoccer (Table 12).

As suggested, we compared critic scores of unions and members at fixed (s,a)(s,a) (Appendix C.1):

  • LFG (linear): the member mean predicts the union value well (R2R^2 0.72 to 0.88 when one member is nearer, Table 8). The union value lies closer to the nearer member on antmaze-large and visual cube (Table 9), and the union ranks actions like the nearer member (Table 10).

  • LFG (logistic): the union follows the mean member logit. As a probability it falls far below the nearer member on the navigation tasks.

  • Dense critic on the same goal space: little such relation, so the composition comes from the shared bilinear readout.

Broader abstract goals (W2)

We added unions of three and four goals and reward-defined goal sets on antmaze-large, antsoccer and visual cube (Figures 22 to 24). In the maze the reward sets include halves, bands, walls, junctions and dead ends. On visual cube we put the reward either on the cube or on the gripper, to test whether the model separates object and agent. A reward goal is given as the union of all rewarded validation states. For each model we evaluate the pretrained actor and the actor after union refinement, in which only the actor is fine-tuned on random unions while critic and goal space stay frozen. Table 12 summarizes the results:

  • After refinement, LFG (linear) reaches a member in 90% of the union episodes on antmaze-large and in 70% on visual cube. It has the best reward success on visual cube (77%).

  • Per set (Figures 25 to 30), the four-corner reward sets are the hardest on antsoccer and visual cube, and the two-passage and dead-end sets on antmaze-large. Refinement helps most where the agent has to choose between distant parts of the maze or table.

Composing single-goal values (Q2)

We added a comparison with an agent that selects a single member of the union goal set ("pick best", Table 12, Figures 28 to 30). For each member, the pretrained actor proposes an action and the model's own critic scores it. The agent then pursues the best member for the whole episode. Pick best needs the list of members and is a strong reference. It has the highest average union success of every model in every environment (Table 12). Refined LFG (linear) comes close on antmaze-large (90% versus 93%) and visual cube (70% versus 78%). On antsoccer, pick best is clearly ahead (74% versus 40%). On the reward-defined sets, the refined union actor is on par on antmaze-large (95% versus 96%) and antsoccer (70% versus 69%) and ahead on visual cube (77% versus 61%).

Figure 37 shows why. With the reward on the gripper, each member is a full state and also fixes a cube position that is irrelevant for the reward. In the data, gripper and cube mostly move together, so most members also have the cube in the region (87%). The refined agent therefore carries the cube along, and the gripper reaches the region in 85% of the episodes (mean over three seeds, Figure 30). Pick best chooses a member with the cube outside the region in 66% of the episodes. The gripper then has to reach the region without the cube, and pick best succeeds in 59%. Since actor and critic take the union as a single vector, such goals could later also come from a system that composes goals directly in the goal space. We leave this for future work.

More members and reward goals (Q3)

Unions of three and four corners and the reward sets are now part of the evaluation (Figures 22 to 24). With the linear readout, refinement improves both union and reward success (Table 12). The refined agent heads for different members depending on its start, so the split between members becomes more balanced (Figures 25 and 27). Rollouts are in Figures 31 to 36. For example, with the reward on the cube, the refined actor moves the cube into all four corners.

Cube-double and scene (Q4)

We think the main causes are the contrastive training signal and the training horizons, which matter because episode lengths and goal distances vary a lot across OGBench. On cube-double, the representation encodes whatever separates trajectories, and the position of a randomly placed cube is often enough for that. CRL, which is also contrastive, has low success on cube-double as well (10%, Table 2). Negatives from the same trajectory may help here. The linear readout already improves cube-double (18% versus 3%) and scene (43% versus 17%, Table 11). It is weaker on long-horizon locomotion (antmaze-giant, antsoccer, humanoidmaze-medium, visual antmaze-large), where distant goals have small values that the squared-error fit resolves less precisely than the logit-space fit. Visual scene also drops slightly. We will discuss this trade-off as a limitation.

Planned changes to the paper

  • Section 2: We will present the logistic readout of the submitted version (sigmoid, BCE) and the linear readout (squared error) as two variants of LFG. A new paragraph will explain how each readout scores a union goal and why the linear one acts as a soft OR, with reference to the union--member analysis (Tables 8 to 10).

  • Section 4: We will extend the abstract-goal results with a summary of the unions of three and four goals, the reward-defined goal sets and the comparison with pick best. The per-set results stay in the appendix.

  • Discussion: We will compare the two readouts, including the gain on cube-double and scene and the loss on long-horizon locomotion, and discuss this trade-off as a limitation.

Thank you for the substantial revision. The new linear readout and the union-versus-member analyses address my question about how the critic scores disjunctive goals. I also appreciate the expanded evaluation on unions of three and four goals and on reward-defined goal sets.

My remaining concern is whether the linear readout reliably ranks actions when reaching any one member is sufficient. The union score approximately averages the member values, so low-value alternatives can dilute the contribution of a reachable member. In Table 1, pick best achieves higher union-goal success across all four tasks, with a particularly large gap on antsoccer (74% versus 40% for refined LFG with the linear readout). I recognize that pick best has access to the individual members and that refined LFG performs better on reward-defined goals in visual cube-single (77% versus 61%). That result shows a benefit of the latent representation in that setting.

The revision strengthens the paper, but the reliability of action selection for disjunctive goals remains a central concern for me. I therefore retain my rating of 2 (Weak rejection).

Thank you for the follow-up.

Averaging changes the value, while action selection depends on the ranking. Table 11 (Appendix B.8) tests this directly: under LFG (linear), the union ranks actions like the nearer member on all four tasks (Spearman correlation with the nearer / farther member: .59 / .37, .52 / .26, .81 / .13, .64 / .51).

The antsoccer gap is not specific to averaging. LFG (logistic), which does not average the members, shows the same gap to pick best on antsoccer (52 vs. 82). Antsoccer is also the task where the linear readout is already weaker on single goals (23 vs. 38, Table 3), so this weakness is already present without unions. On the other three tasks, refined LFG (linear) is within 2 to 8 points of pick best, which is given the list of members.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal summary

We thank the reviewers again for their feedback. The revised paper is uploaded and includes the results and changes from the discussion, marked in blue. Since the rebuttal Appendix C has been merged into the paper, the figure and table numbers in our earlier comments no longer match. Below we summarise the changes and where to find them.

Union semantics and a second readout (1hyt W1/Q1, JF75 W2, mEKF W2/Q2)

Reviewers asked why containment of a whole union in the predicted future should value reaching just one of its members. To address this, we added a second variant, LFG (linear), which removes the sigmoid and fits relation and critic with a squared error. A union goal then scores roughly like the mean of its members and acts as a soft OR. With the original logistic readout, now called LFG (logistic), it behaves more like an AND. Tables 9 to 11 (Appendix B.8) show this directly by comparing union and member values at fixed (s,a)(s,a). Both variants are reported throughout the paper, introduced in Section 2 and compared in Section 6.

Broader abstract goals (1hyt W2/Q3, JF75 W1)

To test whether the transfer holds for more general abstract goals, we added unions of up to four distinct goals as well as goal sets defined by a reward on the agent or the object, on all four tasks (Section 3, Figures 15 to 18). After refinement, union success is 74% for LFG (linear) and 67% for LFG (logistic) on average, and about a third for the dense critics and HGA + CRL (Section 4, Table 1, Figure 5). To measure whether the agent uses all members, we replaced the less-used-goal ratio by the member balance, the normalised entropy of the member reached first (Figure 5, right). For both LFG variants, refinement raises success on abstract goals and makes the agent reach the members more evenly. Results for each goal set are in Figures 19 to 26 and rollouts in Appendix B.9.

Composing single-goal values (1hyt Q2, mEKF W1/Q1)

As suggested, we added a baseline that selects among the members using single-goal values. This baseline, referred to as pick best, scores each member with the model's own critic and pursues the best one (Section 4, Controls; Table 1). It requires access to the individual members, whereas our agent receives the union as a single latent goal in the same space as single goals. The union success of pick best is higher than that of the refined actor of the same model on every task. On reward goals, refined LFG (linear) is on par with it on the navigation tasks and better on visual cube-single. Figure 31 shows why for a gripper reward: pick best commits to a member that also fixes an irrelevant cube position, while the union goal abstracts it away.

Contribution of the critic to refinement (JF75 Q1, mEKF W1/W3/Q3)

Since refinement also contains behaviour cloning, we added BC-only refinement to isolate the effect of the critic. On union goals, BC-only refinement stays below refinement with the critic for both readouts and lowers union success on the pixel-based tasks. HGA + CRL refined with its own abstract hindsight relabeling does not improve over our refinement and stays below both LFG variants (Section 4, Controls; Table 1). Refining on unions lowers success on the members as single goals, from 86% to 77% for LFG (linear) and from 89% to 86% for LFG (logistic), mostly on antsoccer. We see this as a trade-off, since the policy now has to handle abstract goals in addition to single goals (Section 6).

Contribution of the representation objectives (mEKF W4/Q4)

Table 3 compares the set and successor ablations of both readouts with the temporal-contrastive TRA objective (details in Appendix B.2.2). A temporal objective alone is not sufficient: TRA matches LFG (logistic) on antmaze-large (82%) but falls far behind on antsoccer (12% vs. 38%) and visual cube-single (0% vs. 72%). Table 6 extends the ablations of LFG (logistic) to eight tasks. Table 8 (Appendix B.2.3) evaluates the same ablations on union and reward goals. Without the set losses, both variants fail on visual cube-single (refined LFG (linear) drops from 70% to 11% union success). On the navigation tasks, removing the set losses keeps success close to or above the full model.

Clarifications (JF75 W3)

To clarify which objectives train which modules, we added Figure 3 (Section 2.2), and the text now explains the different roles of SUCC and ψ\psi. The caption of Figure 6 now states that the action changes with the goal.

Task-dependent single-goal performance (1hyt Q4, JF75 Q2)

In the rollouts we inspected, antsoccer often fails when the ball narrowly misses the goal, long-horizon navigation fails mostly for distant goals with small value differences between actions, and cube-double lacks spatial precision (Section 6). The linear readout improves cube-double (18% vs. 3%) and scene (43% vs. 17%) and is weaker on long-horizon locomotion (Table 2). Section 6 discusses this trade-off.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.