- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors introduces Dual Advantage Fields (DAF), a policy extraction method for offline goal-conditioned RL. The core insight rests on the bilinear dual value parameterization, observing that the gradient with respect to the state representation is identically the goal embedding. DAF learns an auxiliary action‑effect model and scores candidate actions by computing their alignment with the goal direction using an inner product, bypassing the need for training a separate action‑value critic. Experiments on 20 OGBench locomotion, manipulation, and puzzle tasks demonstrate notable empirical gains over prior baselines.
Critical strengths
-
Exploiting the gradient identity of bilinear value representations to construct an advantage-like score is mathematically neat and conceptually elegant.
-
The authors evaluate across a diverse set of 20 OGBench tasks (locomotion, manipulation, puzzles) spanning navigate, stitch, play, and noisy datasets, reporting rigorous statistic results.
Critical weaknesses
-
The title "Dual Advantage Fields" suggests a general reinforcement learning paradigm. However, the formulation is strictly couched within offline goal-conditioned RL using expectile regression. Could this framework be applied to online RL? If not, the title and claims should be scoped appropriately.
-
The overall algorithm is an incremental improvement of existing works: bilinear dual embeddings (Park et al., 2025c), model-based local advantages (Dayan & Singh, 1995), and Actor-Free Updates (AFU). More critically, the ablation study in Table 6 indicates that removing the AFU coupling causes a catastrophic drop in performance across multiple tasks. This strongly suggests that the stability and empirical improvements are predominantly driven by the asymmetric Bellman critic loss rather than the proposed Dual Advantage formulation itself.
-
Assumption 1 (Exact Realizability) is excessively strong: assuming exact bilinear factorizability across continuous state-goal spaces is unrealistic, particularly in offline RL where coverage is limited.
-
Figure 2 is overly conceptual and fails to clarify what makes the advantage "local". It is difficult to extract actionable intuition from the scattered points and arrows. The authors lack qualitative rollouts or trajectory visualizations of failure modes (e.g., instances where the predicted action effect misaligns).
-
Appendix G notes that the experimental campaign logged 17,359 runs and consumed 407 days on NVIDIA H100. But lacks an empirical comparison of training time, GPU memory footprint, and inference latency against standard baselines (e.g., HIQL, GCIQL, GCIVL).
Questions for authors
Please refer to "Critical Weaknesses".
3 comments
Assumption 1 (Exact Realizability) is excessively strong: assuming exact bilinear factorizability across continuous state-goal spaces is unrealistic, particularly in offline RL where coverage is limited.
We thank the reviewer for this observation. Indeed, Assumption 1 in Appendix F.1, which considers finite-dimensional bilinear realizability of the true goal-conditioned value function over arbitrary continuous state-goal spaces, is too strong, particularly in the offline RL setting where the data distribution may not sufficiently cover all state-action-goal combinations.
Below we show how to relax Lemma 3, so that the approximation is upper bounded by the value approximation and the action-effect approximation.
From the definition, . The true Bellman advantage is .
Consider the difference , and assume . Define the true conditional feature displacement relative to the learned state embedding, .
Since , subtraction gives the exact decomposition .
By Cauchy-Schwarz and the triangle inequality, with and ,
Therefore , where is the value approximation and is the action-effect estimation.
In practice, DAF does not assume exact knowledge of either component. The value representation is learned from offline Bellman-based objectives, while the action-effect model is trained on observed feature displacements using Equation (12).
Our OGBench experiments, including the stitch and noisy datasets, provide empirical evidence of DAF's effectiveness with imperfect offline learning conditions. These results do not require exact bilinear realizability, but we do not interpret them as establishing robustness under arbitrary coverage.
We will clarify the role of Assumption 1 in Appendix F.1 and explicitly distinguish the exact realizability result from the approximate-error analysis and its coverage limitations.
Figure 2 is overly conceptual and fails to clarify what makes the advantage "local". It is difficult to extract actionable intuition from the scattered points and arrows. The authors lack qualitative rollouts or trajectory visualizations of failure modes (e.g., instances where the predicted action effect misaligns).
Figure 2 showcases the transition from the global view (left plot), where each state contains features (ideal case) corresponding to the distances to all states, making the extraction of an optimal global policy possible. Such a featurization is not scalable when the number of states grows. Local policy guidance relies on the representations of nearby states. That reduces the computational burden of the global view and provides gradients by means of for each local state.
Appendix G notes that the experimental campaign logged 17,359 runs and consumed 407 days on NVIDIA H100. But lacks an empirical comparison of training time, GPU memory footprint, and inference latency against standard baselines (e.g., HIQL, GCIQL, GCIVL).
We benchmarked each method in its own process on one NVIDIA H100 with the dimensions of humanoidmaze-medium-navigate. Observations have dimension 69, actions 21, and the update batch contains 1,024 samples. The main comparison uses three hidden layers of width 512. These timings exclude dataset reading.
Each update includes the forward pass, loss and gradient update. Timing ends when a result returns to the CPU. We discard 10 warm-up updates and time the next 50. For actions, we discard 20 warm-up decisions and time the next 200 at temperature zero, using one state and goal. Hierarchical decisions include both actors. Preallocation is disabled. The reported memory peak is taken over both phases, including the batch of 1,024.
| Method | Params (M) | Update (ms) | Action (ms) | Peak (GiB) |
|---|---|---|---|---|
| DAF | 12.9 | 5.06 | 0.59 | 1.51 |
| HIQL | 4.1 | 2.22 | 0.53 | 0.35 |
| OTA | 6.3 | 3.21 | 0.55 | 1.04 |
| CRL | 3.9 | 1.68 | 0.40 | 0.22 |
| CRL Dual | 9.4 | 3.67 | 0.55 | 1.37 |
| GCIQL | 3.7 | 1.63 | 0.41 | 0.22 |
| GCIVL | 3.0 | 1.18 | 0.42 | 0.20 |
| GCIVL Dual | 8.7 | 3.80 | 0.55 | 1.15 |
| MQE | 3.94 | 9.76 | 0.38 | 1.49 |
Parameter counts above count tree entries, including repeated target weights.
We thank the reviewer for this remark on the title. We respectfully disagree. We think Dual Advantage Fields is the right title.
Dual Goal Representations (Park et al., 2025) is a respected paper in offline goal-conditioned RL, and its title names the representation that paper studies. Our title names the dual advantage field in the same way of elaborating further the notion of combining the expressivity of these representations towards theoretically guaranteed reliable training of advantages and value functionals (based on these representations) so with long- and short-term performance benefits. The experiments are the offline setting. The submission does not contain an online training procedure.
We thank the reviewer for this productive question. We would like to separate AFU from the dual advantage field.
We added AFU to HIQL, and to HIQL with dual representations, in the same manner as AFU is used inside DAF.
humanoidmaze-medium-navigatehumanoidmaze-large-navigatehumanoidmaze-medium-stitchhumanoidmaze-large-stitchantmaze-teleport-navigateantmaze-medium-navigateantmaze-teleport-stitchantmaze-medium-stitchcube-double-playcube-triple-playcube-quadruple-playcube-double-noisycube-triple-noisycube-quadruple-noisyscene-playscene-noisypuzzle-3x3-playpuzzle-4x4-playpuzzle-3x3-noisypuzzle-4x4-noisyHIQL+AFU mostly lowers HIQL. HIQL with dual representations and AFU also stays below DAF. AFU is useful as an extension of dual advantage fields, which are the main contribution. DAF is the piece that can sit on a flat backbone or on a hierarchical one, and that is how local action scoring and long-horizon structure are used together.