- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes an algorithm named Normalizing Flow subgoal policies with Triangle-slack Reweighting (NFTR) for offline Goal-conditioned Reinforcement Learning (GCRL). NTFR is built on top of HIQL and replaces the high-level policy with a normalizing flow and corrects the value function with a triangle slack score.
The paper provides a theoretical analysis of NTFR and experiments performed on the OGBench benchmark.
Critical strengths
-
The paper identifies two important problems that are often overlooked in the majority of the offline GCRL literature: optimistic bias in value function updates that follow Temporal-Difference Learning, and averaging modes of Gaussian policies.
-
The authors associate a theoretical analysis to their algorithmic contributions.
Critical weaknesses
-
Experiments, in particular in the stochastic setting, do not justify the authors' claims. Other algorithms such as EikHiQRL in [1], show comparable performance in the stochastic setting without explicitly addressing any of the issues highlighted in this paper.
-
The novelty is incremental. I want to stress that an incremental novelty would be acceptable, in my view, if backed up by strong empirical results.
[1] Giammarino, Vittorio, and Ahmed Hussain Qureshi. "Goal reaching with eikonal-constrained hierarchical quasimetric reinforcement learning." International Conference on Learning Representations. Vol. 2026. 2026.
Questions for authors
-
Any comments on the weaknesses highlighted above?
-
The theoretical analysis seems over-idealized for the tested setting. I was wondering if there is any insight we might get from the theory that applies to the experiments.
5 comments
Response to Reviewer 4K1X (2/3): novelty and the empirical case (W2)
You write that incremental novelty would be acceptable if backed by strong empirical results. We answer both halves: what is new, and what the empirical evidence supports across matched ablations, a fixed-coefficient reference, and the Eik-HiQRL comparison.
What is new. We agree that each component exists: flows inside HIQL (NF-HIQL, now cited as prior work), MRN quasimetric heads, and the composition d(s,w) + d(w,g) (SoRB, LEAP and RIS use it to choose or plan subgoals; QRL, MRN and Transitive RL impose the inequality on value learning).
- Why HIQL needs a multi-modal high level, and where. NF-HIQL motivates flows by expressivity. Its KL and PAC bounds hold for any class with bounded log-likelihood, the Gaussian included, so they cannot say when a flow should win. Thm. 3.1 gives that separation for HIQL's own AWR objective: the optimal Gaussian mean is a convex combination of tilted modal centres and leaks irreducible mass between them. Reviewer NGdT describes it as a genuine explanation rather than a post-hoc rationalization, and Reviewer Q4Fu notes that it turns the intuition into a concrete statement about HIQL's objective.
- How the triangle residual enters. We neither change nor relax the triangle inequality; the MRN head satisfies it exactly. We use its non-negative residual Δ = d(s,w) + d(w,g) − d(s,g) at offline extraction, leave the value untouched, and multiply the AWR weight by exp(−κΔ). Because Δ ≥ 0 holds by architecture, the correction can only remove weight and never rewards a shortcut. When the same distance enters as a progress reward instead, which also rewards a teleport shortcut, the four-task mean is 44.0, no better than value only (44.5); the one-sided residual gives 47.9 (Fig. 4). The 3.9-point difference is comparable to seed variation, so we treat it as indicative.
The empirical case.
- The flow alone (κ = 0). In the matched ablation, replacing the Gaussian adds 8.9 to 37.1 points on the five tasks of Table 4. On manipulation, κ = 0 already exceeds the published HIQL numbers on all four tasks (cube-single-play 22.2 vs 15, cube-single-noisy 55.2 vs 41, scene-play 51.7 vs 38, scene-noisy 27.1 vs 25).
- One fixed coefficient (κ = 1). NFTR stays above HIQL on 10 of the 13 tasks of Table 15.
- Against Eik-HiQRL. Using the NFTR results already reported in the main locomotion table, we find higher sample means on both teleport-navigate tasks; the two stitch differences are small relative to the reported standard deviations (part 1/3).
- Where the slack adds. κ = 0 leaves every other parameter bit-identical, so κ > 0 vs κ = 0 isolates the slack. It raises cube-single-play 22.2 → 47.1, cube-single-noisy 55.2 → 67.4 and antmaze-teleport-explore 14.2 → 26.1. These maxima were selected on the reporting seeds and should be read as selected results.
Where the evidence is weak, and why. The pm-tel-nav slack gain uses a coefficient selected on the reporting seeds. On deterministic stitching the penalty hurts (antmaze-medium-stitch 81.0 → 31.0 as κ grows), as the R_κ term of Thm. 3.8 predicts when optimal subgoals carry learned slack. On HumanoidMaze and long-horizon AntMaze stitching, NFTR is below HIQL; there, planning depth and the physical realizability of sampled modes, not mode averaging, appear to be the bottleneck. NFTR is also below HIQL on puzzle-3x3-noisy, for a reason we have not identified. These results stay in the paper.
What the revision adds. App. A cites NF-HIQL as prior work and the classical triangle-based methods. App. F.12 adds the GMM mode-versus-sample test; Sec. 3.2 and App. F.13 the exact account of which lucky transitions a contrastive slack can detect; App. F.14 the theory-to-experiment table and the same-protocol Eik-HiQRL comparison. The anonymized code is released.
Response to Reviewer 4K1X (3/3): what the theory says about the experiments (Q2)
We agree that parts of the analysis are idealized, and the revision separates which parts (App. F.14).
- Thm. 3.1 matches the implementation. HIQL's high-level Gaussian uses a constant standard deviation (σ = 1), the fixed-σ case of the theorem. Weight clipping makes the AWR weights bounded. A teleporter sends one action sequence to three rooms (Fig. 5), which produces the disjoint waypoint clusters of Eq. 6. The idealized step is exact minimization for each (s, g); a network shares parameters across pairs and can only approach μ*.
- Prop. 3.3(1)–(3) hold exactly for the MRN head, by architecture. Part (4) concerns d*, which the learned head only approximates; App. F.13 shows where that approximation breaks.
- Thm. 3.8 and Prop. 3.9 are population-level statements under concentrability. We use them for qualitative predictions, such as the sign of the κ effect, not for numbers.
New Table 22 pairs each result with a measurement:
| Formal result | Prediction | Observation |
|---|---|---|
| Thm. 3.1 | A multi-modal class helps most where modes are well separated in φ-space | B0 → B1: +37.1 pp on pm-tel-nav (state = x-y position) vs +8.9 to +10.6 on AntMaze teleport |
| Thm. 3.1 | One deterministic subgoal per (s, g) loses the alternatives | GMM returning its mode: 12.8, the Gaussian level (11.7); GMM sampling: 34.4 |
| Remark 3.2 | The deficit belongs to the class, not to the density | Flow, flow-matching, diffusion and mixture heads all beat the Gaussian |
| Thm. 3.8, term R_κ | Bias grows with κ where optimal subgoals carry non-zero slack | Deterministic stitching degrades as κ grows: antmaze-medium-stitch 81.0 → 31.0 |
| Prop. 3.3(4) | Only a one-sided detector under stochastic dynamics | Replay AUROC 0.56–0.68 over all transitions, and 0.78 and 0.81 on nearby subgoals in the two navigate datasets; partial discrimination, not a guarantee. The missed class is identified exactly in App. F.13 |
| Cor. 3.5 + clipping | No effect on samples whose weight is clipped | In the MDP, portal mass is unchanged for κ ≤ 2 when both waypoints saturate the clip |
Two rows explain results rather than restate them.
- Row 1. In PointMaze the state is the x-y position, so the three teleport landing rooms are far apart in φ-space and the Gaussian mean lands between them. In AntMaze the state also contains joint angles and velocities; our reading is that these dilute the separation of the clusters in φ-space, so the mean loses less. The size of the flow's gain follows this ordering.
- Row 4. Thm. 3.8 separates what the penalty can gain (down-weighting off-path subgoals) from what it costs (R_κ, which grows with the slack the head assigns to optimal subgoals). Deterministic stitching has no lucky transitions to filter, so only the cost remains, and success falls as κ grows.
Summary. We agree that our original stochastic-setting claim outran the evidence. The revised paper claims less and supports each claim at the level of mechanism: why the Gaussian fails and when, which component fixes it, how NFTR compares with Eik-HiQRL under one protocol, and what the slack can and cannot detect. We would be glad to clarify any remaining point during the discussion.
Response to Reviewer 4K1X (1/3): the stochastic setting and Eik-HiQRL (W1, Q1)
Thank you for the critical review. We read W1 as two linked points. First, our stochastic-setting experiments did not support the claims we made. Second, Eik-HiQRL reaches comparable teleport numbers without targeting optimistic bias or mode averaging, so these two failures may not be what limits performance. We agree with both points as stated. Performance alone cannot identify a mechanism. We therefore (A) compared with Eik-HiQRL under one protocol, (B) tested each mechanism directly, and (C) narrowed every claim the tests do not support. All revised text is in blue in the uploaded PDF.
(A) Eik-HiQRL under the OGBench evaluation protocol (Sec. 4.2, App. F.14, Table 23). We ran the public Eik-HiQRL code ourselves over 8 seeds and report the average of the last three evaluations, matching the reporting rule used for NFTR. The NFTR row reproduces the eight-seed results in the revised paper's main locomotion table:
What this comparison does and does not show. Eik-HiQRL improves the high-level value. Its analysis assumes deterministic dynamics (its Assumption 4.2), and its authors note that the method still applies to teleport. We read its results as you do: a better value geometry is another route to robustness under stochastic transitions. They do not show that mode averaging is absent. Eik-HiQRL still extracts a unimodal high-level policy by AWR, so Thm. 3.1 applies to it, and the two changes act on different components and can compose. For navigation, it also learns its high-level quasimetric on the agent's x-y coordinates, whereas NFTR and HIQL use the full state.
(B) Direct tests of the two mechanisms.
(C) Revised claims. The abstract no longer says "performs well on stochastic tasks". It states that the flow carries most of the gain and that the slack detects detours in a learned reachability geometry, not luck itself.