acceptodds
Under review as a conference paper at ICLR 2027

Dual Advantage Fields

Abstract

Offline goal-conditioned reinforcement learning requires both long-horizon reachability estimates and local action comparisons. Dual goal representations provide value fields that capture global goal reachability, but they do not directly specify which action should be preferred at a given state. We propose Dual Advantage Fields (DAF), a policy-extraction method that turns a bilinear dual value model into a local advantage signal. Under bilinear dual parameterization, the goal embedding is the gradient of the value field with respect to the state representation, which directly specifies which action should be preferred. DAF introduces an action-effect model that predicts the discounted direction of change in state representation space induced by actions and scores them by the similarity between this change and the goal direction. In the realizable case, this score equals the goal-conditioned Bellman advantage, yielding a standard local policy-improvement guarantee. On OGBench locomotion, manipulation, and puzzle tasks, DAF improves aggregate RLiable metrics and performs strongly in settings where locally correct actions differ from direct movement toward the final goal.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Offline Reinforcement Learning, Goal-Conditioned Reinforcement Learning, Deep Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_v22XOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduces Dual Advantage Fields (DAF), a policy extraction method for offline goal-conditioned RL. The core insight rests on the bilinear dual value parameterization, observing that the gradient with respect to the state representation is identically the goal embedding. DAF learns an auxiliary action‑effect model and scores candidate actions by computing their alignment with the goal direction using an inner product, bypassing the need for training a separate action‑value critic. Experiments on 20 OGBench locomotion, manipulation, and puzzle tasks demonstrate notable empirical gains over prior baselines.

Critical strengths
  1. Exploiting the gradient identity of bilinear value representations to construct an advantage-like score is mathematically neat and conceptually elegant.

  2. The authors evaluate across a diverse set of 20 OGBench tasks (locomotion, manipulation, puzzles) spanning navigate, stitch, play, and noisy datasets, reporting rigorous statistic results.

Critical weaknesses
  1. The title "Dual Advantage Fields" suggests a general reinforcement learning paradigm. However, the formulation is strictly couched within offline goal-conditioned RL using expectile regression. Could this framework be applied to online RL? If not, the title and claims should be scoped appropriately.

  2. The overall algorithm is an incremental improvement of existing works: bilinear dual embeddings (Park et al., 2025c), model-based local advantages (Dayan & Singh, 1995), and Actor-Free Updates (AFU). More critically, the ablation study in Table 6 indicates that removing the AFU coupling causes a catastrophic drop in performance across multiple tasks. This strongly suggests that the stability and empirical improvements are predominantly driven by the asymmetric Bellman critic loss rather than the proposed Dual Advantage formulation itself.

  3. Assumption 1 (Exact Realizability) is excessively strong: assuming exact bilinear factorizability across continuous state-goal spaces is unrealistic, particularly in offline RL where coverage is limited.

  4. Figure 2 is overly conceptual and fails to clarify what makes the advantage "local". It is difficult to extract actionable intuition from the scattered points and arrows. The authors lack qualitative rollouts or trajectory visualizations of failure modes (e.g., instances where the predicted action effect misaligns).

  5. Appendix G notes that the experimental campaign logged 17,359 runs and consumed 407 days on NVIDIA H100. But lacks an empirical comparison of training time, GPU memory footprint, and inference latency against standard baselines (e.g., HIQL, GCIQL, GCIVL).

Questions for authors

Please refer to "Critical Weaknesses".

3 comments

The title "Dual Advantage Fields" suggests a general reinforcement learning paradigm. However, the formulation is strictly couched within offline goal-conditioned RL using expectile regression. Could this framework be applied to online RL? If not, the title and claims should be scoped appropriately.

We thank the reviewer for this remark on the title. We respectfully disagree. We think Dual Advantage Fields is the right title.

Dual Goal Representations (Park et al., 2025) is a respected paper in offline goal-conditioned RL, and its title names the representation that paper studies. Our title names the dual advantage field in the same way of elaborating further the notion of combining the expressivity of these representations towards theoretically guaranteed reliable training of advantages and value functionals (based on these representations) so with long- and short-term performance benefits. The experiments are the offline setting. The submission does not contain an online training procedure.

The overall algorithm is an incremental improvement of existing works: bilinear dual embeddings (Park et al., 2025c), model-based local advantages (Dayan & Singh, 1995), and Actor-Free Updates (AFU). More critically, the ablation study in Table 6 indicates that removing the AFU coupling causes a catastrophic drop in performance across multiple tasks. This strongly suggests that the stability and empirical improvements are predominantly driven by the asymmetric Bellman critic loss rather than the proposed Dual Advantage formulation itself.

We thank the reviewer for this productive question. We would like to separate AFU from the dual advantage field.

We added AFU to HIQL, and to HIQL with dual representations, in the same manner as AFU is used inside DAF.

TaskHIQL+AFUHIQL dual + AFUHIQLDAFNo AFU
humanoidmaze-medium-navigate0.24±0.080.51±0.020.91±0.010.93±0.030.18±0.03
humanoidmaze-large-navigate0.05±0.020.35±0.040.45±0.040.66±0.030.02±0.01
humanoidmaze-medium-stitch0.38±0.270.25±0.030.86±0.030.90±0.040.47±0.06
humanoidmaze-large-stitch0.03±0.010.13±0.010.32±0.040.48±0.060.06±0.02
antmaze-teleport-navigate0.36±0.040.49±0.050.46±0.030.51±0.080.33±0.05
antmaze-medium-navigate0.26±0.190.92±0.030.96±0.010.98±0.010.78±0.05
antmaze-teleport-stitch0.37±0.040.19±0.040.38±0.020.50±0.050.19±0.04
antmaze-medium-stitch0.76±0.050.63±0.060.96±0.020.97±0.020.42±0.05
cube-double-play0.03±0.010.05±0.010.13±0.010.41±0.040.36±0.07
cube-triple-play0.01±0.010.04±0.010.05±0.020.17±0.030.02±0.01
cube-quadruple-play0.00±0.000.00±0.000.00±0.000.03±0.010.00±0.00
cube-double-noisy0.04±0.010.09±0.020.03±0.010.33±0.050.26±0.05
cube-triple-noisy0.01±0.010.04±0.010.04±0.010.23±0.010.01±0.01
cube-quadruple-noisy0.00±0.000.00±0.000.00±0.000.02±0.010.00±0.00
scene-play0.17±0.060.14±0.020.55±0.090.81±0.040.49±0.05
scene-noisy0.11±0.010.13±0.030.27±0.020.43±0.030.29±0.03
puzzle-3x3-play0.08±0.020.07±0.020.17±0.030.74±0.040.10±0.01
puzzle-4x4-play0.05±0.020.08±0.030.17±0.030.40±0.050.18±0.03
puzzle-3x3-noisy0.63±0.170.17±0.020.70±0.100.98±0.040.15±0.03
puzzle-4x4-noisy0.01±0.000.05±0.010.31±0.070.47±0.030.07±0.01

HIQL+AFU mostly lowers HIQL. HIQL with dual representations and AFU also stays below DAF. AFU is useful as an extension of dual advantage fields, which are the main contribution. DAF is the piece that can sit on a flat backbone or on a hierarchical one, and that is how local action scoring and long-horizon structure are used together.

Assumption 1 (Exact Realizability) is excessively strong: assuming exact bilinear factorizability across continuous state-goal spaces is unrealistic, particularly in offline RL where coverage is limited.

We thank the reviewer for this observation. Indeed, Assumption 1 in Appendix F.1, which considers finite-dimensional bilinear realizability of the true goal-conditioned value function over arbitrary continuous state-goal spaces, is too strong, particularly in the offline RL setting where the data distribution may not sufficiently cover all state-action-goal combinations.

Below we show how to relax Lemma 3, so that the approximation is upper bounded by the value approximation and the action-effect approximation.

From the definition, D^(s,a,g)=r(s,a,g)+u^(s,a)⊤ϕ^(g)\hat D(s,a,g)=r(s,a,g)+\hat u(s,a)^\top\hat\phi(g). The true Bellman advantage is Aπ(s,a,g)=r(s,a,g)+γEs′[Vπ(s′,g)]−Vπ(s,g)A^\pi(s,a,g)=r(s,a,g)+\gamma\mathbb{E}_{s'}[V^\pi(s',g)]-V^\pi(s,g).

Consider the difference ΔV(s,g)=V^(s,g)−Vπ(s,g)\Delta_V(s,g)=\hat V(s,g)-V^\pi(s,g), and assume sup⁡s,g∣V^(s,g)−Vπ(s,g)∣≤ϵV\sup_{s,g}|\hat V(s,g)-V^\pi(s,g)|\leq\epsilon_V. Define the true conditional feature displacement relative to the learned state embedding, uˉ(s,a)=Es′∼P(⋅∣s,a)[γψ^(s′)−ψ^(s)]\bar u(s,a)=\mathbb{E}_{s'\sim P(\cdot\mid s,a)}[\gamma\hat\psi(s')-\hat\psi(s)].

Since V^=ψ^⊤ϕ^\hat V=\hat\psi^\top\hat\phi, subtraction gives the exact decomposition D^−Aπ=(u^−uˉ)⊤ϕ^(g)+γEs′[ΔV(s′,g)]−ΔV(s,g)\hat D-A^\pi=(\hat u-\bar u)^\top\hat\phi(g)+\gamma\mathbb{E}_{s'}[\Delta_V(s',g)]-\Delta_V(s,g).

By Cauchy-Schwarz and the triangle inequality, with Bϕ:=sup⁡g∈G∥ϕ^(g)∥2B_{\phi} := \sup_{g \in \mathcal{G}} \lVert \hat{\phi}(g) \rVert_{2} and ϵu:=sup⁡s,a∥u^(s,a)−uˉ(s,a)∥2\epsilon_{u} := \sup_{s,a} \lVert \hat{u}(s,a) - \bar{u}(s,a) \rVert_{2},

∣D^−Aπ∣≤∥u^−uˉ∥2∥ϕ^(g)∥2+γEs′∣ΔV(s′,g)∣+∣ΔV(s,g)∣≤Bϕϵu+(1+γ)ϵV.\lvert \hat{D} - A^{\pi} \rvert \leq \lVert \hat{u} - \bar{u} \rVert_{2} \lVert \hat{\phi}(g) \rVert_{2} + \gamma \mathbb{E}_{s'} \lvert \Delta_{V}(s', g) \rvert + \lvert \Delta_{V}(s, g) \rvert \leq B_{\phi} \epsilon_{u} + (1 + \gamma) \epsilon_{V}.

Therefore ∥D^−Aπ∥∞≤(1+γ)ϵV+Bϕϵu=:δ\lVert \hat{D} - A^{\pi} \rVert_{\infty} \leq (1 + \gamma) \epsilon_{V} + B_{\phi} \epsilon_{u} =: \delta, where (1+γ)ϵV(1 + \gamma) \epsilon_{V} is the value approximation and BϕϵuB_{\phi} \epsilon_{u} is the action-effect estimation.

In practice, DAF does not assume exact knowledge of either component. The value representation is learned from offline Bellman-based objectives, while the action-effect model is trained on observed feature displacements using Equation (12).

Our OGBench experiments, including the stitch and noisy datasets, provide empirical evidence of DAF's effectiveness with imperfect offline learning conditions. These results do not require exact bilinear realizability, but we do not interpret them as establishing robustness under arbitrary coverage.

We will clarify the role of Assumption 1 in Appendix F.1 and explicitly distinguish the exact realizability result from the approximate-error analysis and its coverage limitations.

Figure 2 is overly conceptual and fails to clarify what makes the advantage "local". It is difficult to extract actionable intuition from the scattered points and arrows. The authors lack qualitative rollouts or trajectory visualizations of failure modes (e.g., instances where the predicted action effect misaligns).

Figure 2 showcases the transition from the global view (left plot), where each state contains features (ideal case) corresponding to the distances to all states, making the extraction of an optimal global policy possible. Such a featurization is not scalable when the number of states grows. Local policy guidance relies on the representations of nearby states. That reduces the computational burden of the global view and provides gradients by means of ∇ψV(s,g)\nabla_\psi V(s,g) for each local state.

Appendix G notes that the experimental campaign logged 17,359 runs and consumed 407 days on NVIDIA H100. But lacks an empirical comparison of training time, GPU memory footprint, and inference latency against standard baselines (e.g., HIQL, GCIQL, GCIVL).

We benchmarked each method in its own process on one NVIDIA H100 with the dimensions of humanoidmaze-medium-navigate. Observations have dimension 69, actions 21, and the update batch contains 1,024 samples. The main comparison uses three hidden layers of width 512. These timings exclude dataset reading.

Each update includes the forward pass, loss and gradient update. Timing ends when a result returns to the CPU. We discard 10 warm-up updates and time the next 50. For actions, we discard 20 warm-up decisions and time the next 200 at temperature zero, using one state and goal. Hierarchical decisions include both actors. Preallocation is disabled. The reported memory peak is taken over both phases, including the batch of 1,024.

MethodParams (M)Update (ms)Action (ms)Peak (GiB)
DAF12.95.060.591.51
HIQL4.12.220.530.35
OTA6.33.210.551.04
CRL3.91.680.400.22
CRL Dual9.43.670.551.37
GCIQL3.71.630.410.22
GCIVL3.01.180.420.20
GCIVL Dual8.73.800.551.15
MQE3.949.760.381.49

Parameter counts above count tree entries, including repeated target weights.

Reviewer_LSXFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Dual Advantage Fields (DAF), a policy-extraction method for offline goal-conditioned reinforcement learning. The authors exploit bilinear parameterizations of the goal-conditioned value function to derive a local advantage signal. This signal is then used for advantage-weighted policy extraction, both in flat and hierarchical architectures. Experiments across 20 state-based OGBench tasks, demonstrate competitive performance and improvements in aggregate metrics over existing baselines.

Critical strengths

The paper shows good experimental results compared to the selected baselines across many different tasks of the OGBench benchmark.

Critical weaknesses

Major:

  1. My main concern is the constant reference to the dual representation in Park et al. 2025. The proposed algorithm has little to do with the dual representation proposed in the aforementioned paper and a lot to do with the bilinear parameterization of the goal-conditioned value function, which, however was not introduced in Park et al. 2025. See for instance the contrastive RL paper [1].

  2. Novelty is another major concern. Overall, the algorithm follows classical TD-learning but with a bilinear value parameterization. The actor-free updates are particular important for performance, as illustrated in Table 6, but they find only minor space in the main paper.

Minor:

  1. The first sentence in the introduction refer to offline GCRL, not GCRL.
  2. In the preliminaries, lines 112-113, the authors claim that quasimetric representations work well in manipulation. This sentence needs some clarification as Quasimetric RL performs very poorly in the manipulation suite of OGBench.
  3. The notation used in Figure 1 and for ∇ψV(s,g)\nabla_{\psi} V(s, g) needs clarifications.

Bibliography

[1] Eysenbach B, Zhang T, Levine S, Salakhutdinov RR. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems. 2022 Dec 6;35:35603-20.

Questions for authors
  1. Does DAF rely on any specific property of dual goal representations beyond the bilinear parameterization of the value function?

  2. The ablation study shows that actor-free coupling is critical to performance. Can the authors better disentangle its contribution from that of the proposed advantage formulation?

  3. The bilinear value function is invariant under invertible changes of basis of the latent representations, whereas the individual embeddings and their geometric interpretation are not. How does this ambiguity affect the interpretation of the goal embedding as a direction toward the goal?

2 comments

My main concern is the constant reference to the dual representation in Park et al. 2025. The proposed algorithm has little to do with the dual representation proposed in the aforementioned paper and a lot to do with the bilinear parameterization of the goal-conditioned value function, which, however, was not introduced in Park et al. 2025. See for instance the contrastive RL paper [1].

Bibliography

[1] Eysenbach B, Zhang T, Levine S, Salakhutdinov RR. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems. 2022 Dec 6;35:35603–20.

We thank the reviewer for this productive remark. We would like to explain the citation to Park et al. Bilinear parameterizations are common in reinforcement learning, and they are not new. We refer to Park et al. [1] for a more specific reason. They interpret a goal through its reachability from other states, they give theoretical guarantees for that view, and they study it in offline goal-conditioned RL. That setting still has limitations, and those limitations are what our method is intended to cover. The bilinear form itself follows Hong et al. [2] and, in a contrastive form, Eysenbach et al. [3].

Bibliography

[1] Park S, Mann D, Levine S. Dual Goal Representations. International Conference on Learning Representations. 2026. arXiv:2510.06714.

[2] Yang G, Hong ZW, Agrawal P. Bi-linear Value Networks for Multi-goal Reinforcement Learning. International Conference on Learning Representations. 2022.

[3] Eysenbach B, Zhang T, Levine S, Salakhutdinov RR. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems. 2022;35:35603–35620.

Novelty is another major concern. Overall, the algorithm follows classical TD-learning but with a bilinear value parameterization. The actor-free updates are particular important for performance, as illustrated in Table 6, but they find only minor space in the main paper.

We thank the reviewer for this concern. We would like to pitch the contribution as follows.

The modification is deliberately efficient and minimalistic. It stands on related work that is already known to be useful, and it puts our theoretically guaranteed modules together into a method for short-term and long-term behavior. The empirical benefit is on the offline data from OGBench.

AFU belongs to the practical implementation. The results we have already reported show its cost, and they show that this benefit is specific to DAF. The same addition does not help the other baselines.

The first sentence in the introduction refer to offline GCRL, not GCRL.

We will add this qualification in the first sentence of the introduction in the updated version.

In the preliminaries, lines 112–113, the authors claim that quasimetric representations work well in manipulation. This sentence needs some clarification as Quasimetric RL performs very poorly in the manipulation suite of OGBench.

We thank the reviewer for this remark. We would like to make the sentence more explicit in the camera-ready version. Quasimetric representations work well in manipulation, as stated by the authors of that related work.

The notation used in Figure 1 and for ∇ψV(s,g)\nabla_\psi V(s,g) needs clarifications.

We thank the reviewer for asking for this clarification. We would like to state the intuition behind Figure 1.

The figure is about the reachability of a goal in representation space. Under the bilinear value, that direction is the goal embedding, and it is equal to the gradient of the value with respect to the state embedding, ∇ψV(s,g)=ϕ(g)\nabla_\psi V(s,g)=\phi(g).

Does DAF rely on any specific property of dual goal representations beyond the bilinear parameterization of the value function?

No. The score requires a bilinear value V(s,g)=ψ(s)⊤ϕ(g)V(s,g)=\psi(s)^\top\phi(g) and an estimate of how each action changes ψ(s)\psi(s). Pairing that estimated change with ϕ(g)\phi(g) gives the predicted change in value for the chosen goal. Park et al.'s reachability interpretation motivates our choice of representation, but the calculation requires no additional property of their construction.

The ablation study shows that actor-free coupling is critical to performance. Can the authors better disentangle its contribution from that of the proposed advantage formulation?

We thank the reviewer for this productive question. We would like to separate AFU from the dual advantage field.

We added AFU to HIQL, and to HIQL with dual representations, in the same manner as AFU is used inside DAF.

TaskHIQL+AFUHIQL dual + AFUHIQLDAFNo AFU
humanoidmaze-medium-navigate0.24±0.080.51±0.020.91±0.010.93±0.030.18±0.03
humanoidmaze-large-navigate0.05±0.020.35±0.040.45±0.040.66±0.030.02±0.01
humanoidmaze-medium-stitch0.38±0.270.25±0.030.86±0.030.90±0.040.47±0.06
humanoidmaze-large-stitch0.03±0.010.13±0.010.32±0.040.48±0.060.06±0.02
antmaze-teleport-navigate0.36±0.040.49±0.050.46±0.030.51±0.080.33±0.05
antmaze-medium-navigate0.26±0.190.92±0.030.96±0.010.98±0.010.78±0.05
antmaze-teleport-stitch0.37±0.040.19±0.040.38±0.020.50±0.050.19±0.04
antmaze-medium-stitch0.76±0.050.63±0.060.96±0.020.97±0.020.42±0.05
cube-double-play0.03±0.010.05±0.010.13±0.010.41±0.040.36±0.07
cube-triple-play0.01±0.010.04±0.010.05±0.020.17±0.030.02±0.01
cube-quadruple-play0.00±0.000.00±0.000.00±0.000.03±0.010.00±0.00
cube-double-noisy0.04±0.010.09±0.020.03±0.010.33±0.050.26±0.05
cube-triple-noisy0.01±0.010.04±0.010.04±0.010.23±0.010.01±0.01
cube-quadruple-noisy0.00±0.000.00±0.000.00±0.000.02±0.010.00±0.00
scene-play0.17±0.060.14±0.020.55±0.090.81±0.040.49±0.05
scene-noisy0.11±0.010.13±0.030.27±0.020.43±0.030.29±0.03
puzzle-3x3-play0.08±0.020.07±0.020.17±0.030.74±0.040.10±0.01
puzzle-4x4-play0.05±0.020.08±0.030.17±0.030.40±0.050.18±0.03
puzzle-3x3-noisy0.63±0.170.17±0.020.70±0.100.98±0.040.15±0.03
puzzle-4x4-noisy0.01±0.000.05±0.010.31±0.070.47±0.030.07±0.01

HIQL+AFU mostly lowers HIQL. HIQL with dual representations and AFU also stays below DAF. AFU is useful as an extension of dual advantage fields, which are the main contribution. DAF is the piece that can sit on a flat backbone or on a hierarchical one, and that is how local action scoring and long-horizon structure are used together.

The bilinear value function is invariant under invertible changes of basis of the latent representations, whereas the individual embeddings and their geometric interpretation are not. How does this ambiguity affect the interpretation of the goal embedding as a direction toward the goal?

Under a change of coordinates ψ′=Mψ\psi'=M\psi, the action effect becomes u′=Muu'=Mu and the goal embedding becomes ϕ′=M−⊤ϕ\phi'=M^{-\top}\phi. Their dot product is exactly invariant in arithmetic without rounding. The Euclidean direction drawn for ϕ\phi depends on the chosen coordinates.

We checked this calculation on three DAF networks, each with latent dimension 256, observation dimension 32 and action dimension 8. The table gives the largest absolute changes after each invertible transformation.

DrawMax score changeMax value changeMean goal cosine
10.001190.00082−0.040
20.001240.00090−0.034
30.001200.00074−0.027

The transformed goal embeddings have almost zero difference according to the results above.

Reviewer_fWbiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Proposed: Dual-Advantage Field, a policy extraction method that turns a bilinear dual value model into a local advantage signal, namely:

  • Recognizing V is linear wrt psi, can consider the goal embedding for directional guidance
  • This means an advantage can be computed wrt how the evolution on psi aligns with the embedded goal vector
  • Then a score can be derived using the approximation that u(s,a) \approx sg(gamma * phi(s’) - phi(s))
  • This score is used for extraction to do advantage-weighted behavior cloning
Critical strengths

Strengths:

  • Simple, nice idea that we can replace goal-conditioned Q with one dot product
  • First two figures clearly illustrate the proposed idea, adding intuition and understanding
  • Strong improvements when the local choices matter: cube, scene, puzzle-noisy (with proper RLiable metrics)
  • The appendix honestly reports that the DAF is just flat extraction (Lemma 7)
Critical weaknesses

Weakness:


  • As mentioned by the authors
    • This is reliable only when we have data for “good” actions which improve method, and without, the method yields incorrect rankings
      • How is this known from the outset?
    • Moreover, under significant noise, the same problem arises, limiting the method to lab-like data/rollouts
      • Given the method is proposed for offline RL, practical usage would likely assume some noise in the data
  • When considered, these things confound the idealistic figures which basically assume u(s,a) is accurate
  • It seems that many of the gains compared to the baselines might simply be attributed to the use of HIQL+AFU, no? Yet this baseline (using these without DAF) seems to be missing, making it difficult to conclude that advantages are due to the DAF contribution
  • The dual-rep + hierarchy baseline looks broken (0.06 vs. HIQL 0.91)
  • The ablations show that No-AFU costs the most, leaving benefit due to the dual score alone unclear
  • Moreover, alpha is tuned for the reported seeds and protocol (please add held-out selection)
  • Antmaze-large is absent from the main results but reported later, which feels a bit fishy
  • Low-level conditioning c is never defined, if I am not mistaken
  • The theory is somewhat decorative, props 1. and 4. Hold by construction
Questions for authors

Listed in weaknesses

3 comments

As mentioned by the authors. This is reliable only when we have data for "good" actions which improve the method, and without, the method yields incorrect rankings. How is this known from the outset?

We appreciate the reviewer on this notion. We train DAF on different kinds of datasets (e.g. navigate and explore). The difference is in the quality (coverage) of the state-action space. The advantage estimate relies on the convergence of the learned Q-function (by means of expectile regularization). During training, the Q-function estimation becomes closer to the optimal one, and DAF leverages this in order to introduce guidance for the learning representation and policy. If the Q-function produces irrelevant estimations, then DAF would incorrectly align state-goal representations.

When considered, these things confound the idealistic figures which basically assume u(s,a)u(s,a) is accurate.

We thank the reviewer for this productive remark. We would like to separate the ideal picture from the continuous case.

The figures are the intuitive case. That case is the ideal one, appropriate for a toy example or a discrete problem. For simulation tasks with continuous inputs, the appendix extends it by amortization. The learned action effect is the conditional expectation required by the theory, and the regression fits the usual offline batch pipeline without changing it.

Empirically, the method also performs well on stochastic and suboptimal data. antmaze-large and antmaze-teleport have the same size and differ by the stochastic teleporters.

DatasetMethodantmaze-largeantmaze-teleport
navigateDAF0.93±0.020.51±0.08
HIQL0.95±0.010.46±0.03
OTA0.94±0.010.53±0.03
stitchDAF0.56±0.040.50±0.05
HIQL0.65±0.040.38±0.02
OTA0.87±0.030.38±0.01

It seems that many of the gains compared to the baselines might simply be attributed to the use of HIQL+AFU, no? Yet this baseline (using these without DAF) seems to be missing, making it difficult to conclude that advantages are due to the DAF contribution.

We thank the reviewer for this productive question. We would like to separate AFU from the dual advantage field.

We added AFU to HIQL, and to HIQL with dual representations, in the same manner as AFU is used inside DAF.

TaskHIQL+AFUHIQL dual + AFUHIQLDAFNo AFU
humanoidmaze-medium-navigate0.24±0.080.51±0.020.91±0.010.93±0.030.18±0.03
humanoidmaze-large-navigate0.05±0.020.35±0.040.45±0.040.66±0.030.02±0.01
humanoidmaze-medium-stitch0.38±0.270.25±0.030.86±0.030.90±0.040.47±0.06
humanoidmaze-large-stitch0.03±0.010.13±0.010.32±0.040.48±0.060.06±0.02
antmaze-teleport-navigate0.36±0.040.49±0.050.46±0.030.51±0.080.33±0.05
antmaze-medium-navigate0.26±0.190.92±0.030.96±0.010.98±0.010.78±0.05
antmaze-teleport-stitch0.37±0.040.19±0.040.38±0.020.50±0.050.19±0.04
antmaze-medium-stitch0.76±0.050.63±0.060.96±0.020.97±0.020.42±0.05
cube-double-play0.03±0.010.05±0.010.13±0.010.41±0.040.36±0.07
cube-triple-play0.01±0.010.04±0.010.05±0.020.17±0.030.02±0.01
cube-quadruple-play0.00±0.000.00±0.000.00±0.000.03±0.010.00±0.00
cube-double-noisy0.04±0.010.09±0.020.03±0.010.33±0.050.26±0.05
cube-triple-noisy0.01±0.010.04±0.010.04±0.010.23±0.010.01±0.01
cube-quadruple-noisy0.00±0.000.00±0.000.00±0.000.02±0.010.00±0.00
scene-play0.17±0.060.14±0.020.55±0.090.81±0.040.49±0.05
scene-noisy0.11±0.010.13±0.030.27±0.020.43±0.030.29±0.03
puzzle-3x3-play0.08±0.020.07±0.020.17±0.030.74±0.040.10±0.01
puzzle-4x4-play0.05±0.020.08±0.030.17±0.030.40±0.050.18±0.03
puzzle-3x3-noisy0.63±0.170.17±0.020.70±0.100.98±0.040.15±0.03
puzzle-4x4-noisy0.01±0.000.05±0.010.31±0.070.47±0.030.07±0.01

HIQL+AFU mostly lowers HIQL. HIQL with dual representations and AFU also stays below DAF. AFU is useful as an extension of dual advantage fields, which are the main contribution. DAF is the piece that can sit on a flat backbone or on a hierarchical one, and that is how local action scoring and long-horizon structure are used together.

The dual-rep + hierarchy baseline looks broken (0.060.06 vs. HIQL 0.910.91).

We thank the reviewer for this observation. We would like to explain what that low number is doing in the paper.

A main point of the work is that putting dual goal representations on a hierarchical backbone, and then extracting a policy in the usual way, leads to a downgrade. The dual representation and the hierarchy are both useful, but not as a plain combination. Dual representations carry the goal through reachability. The hierarchy helps with long-horizon structure in the dataset. DAF is what lets us use both, so that short-term local action choice and long-term behavior improve together.

The ablations show that No-AFU costs the most, leaving benefit due to the dual score alone unclear.

We thank the reviewer for this remark. The table in the previous reply already gives a clearer separation of what AFU contributes. AFU helps when it extends DAF. It does not, by itself, account for the dual advantage field.

Moreover, α\alpha is tuned for the reported seeds and protocol (please add held-out selection).

We thank the reviewer for this notion. We would like to clarify how α\alpha was chosen.

We do not tune α\alpha on the individual seeds that we report. In offline goal-conditioned RL it is standard to run a small grid over this temperature and then report the best configuration. The selected value is then kept fixed. As requested, we also train new seeds with those fixed values, which checks that the link from α\alpha to success rate is not tied to the seeds used in the grid.

Taskα\alphaNew seeds 5–7Published seeds 0–4
scene-play100.80±0.030.81±0.04
cube-double-noisy30.31±0.050.33±0.05
puzzle-4x4-noisy0.10.49±0.040.47±0.03

Antmaze-large is absent from the main results but reported later, which feels a bit fishy.

Appendix D.1 reports the two large ant mazes as deterministic controls for the teleport mazes. Table 2 reuses those DAF runs. DAF trails HIQL and OTA on both deterministic tasks.

DatasetMethodantmaze-largeantmaze-teleport
navigateDAF0.93±0.020.51±0.08
HIQL0.95±0.010.46±0.03
OTA0.94±0.010.53±0.03
stitchDAF0.56±0.040.50±0.05
HIQL0.65±0.040.38±0.02
OTA0.87±0.030.38±0.01

Low-level conditioning cc is never defined, if I am not mistaken.

We thank the reviewer for this remark. We will correct the definition in the updated version.

The symbol cc is the conditioning input of the low-level policy. It is there so that one notation covers two conventions among the baselines. Some methods take the goal itself as input. Others take a representation of the goal. In our writing, cc stands for whichever of these the actor receives.

The theory is somewhat decorative, props 1. and 4. hold by construction.

Proposition 1 differentiates the bilinear value. Proposition 4 applies classical policy improvement when the score including reward equals the true advantage and its expectation under the new policy is nonnegative. Proposition 6 motivates the learned action effect by showing how one random successor can reverse an action ranking. Table 6 tests that choice against using a single transition. Lemma 7 gives a case where DAF and ordinary value differences choose the same action.

The authors have addressed my concerns -- thank you for considering them authors.

Comments on the paper

1 comment

We thank the reviewers for the time and effort they put into this paper!

We have answered the questions in the comments. The same changes are in the revised PDF, if that is easier to read. Most of the requests concerned the appendix, and that is where we added the new material. The main text gained the wording that was missing there and updated references to theoretical sections in appendix.

In the revision:

  • The first sentence of the introduction now says that the setting is offline goal-conditioned RL.
  • The low-level policy is conditioned on cc, where cc is a goal, a goal representation, or an intermediate goal. This is stated in the policy-extraction section and in Algorithm 1.
  • To separate the actor-free coupling from the dual advantage, we added the coupling to HIQL and to HIQL with dual representations (Appendix A). Those runs stay below DAF. The coupling helps as part of DAF, and it does not by itself account for the DAF scores.
  • The theory appendix now gives the approximate case as well as the exact one. When the value or the action effect is only approximate, the gap is bounded. The local-policy-improvement paragraph points to both proofs.
  • The same appendix records that the score is unchanged by an invertible change of basis, with a numerical check.
  • Appendix G now also compares update time, action time, and memory.

We hope this makes the paper easier to check, and we are glad to clarify anything further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.