acceptodds
Under review as a conference paper at ICLR 2027

NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RL

Abstract

Hierarchical Implicit Q-Learning (HIQL), an offline goal-conditioned RL method, selects subgoals by value-function advantages alone. This rule has two coupled failure modes. Optimistic bias treats lucky stochastic outcomes as skillful choices, and mode collapse reduces a multi-modal subgoal distribution to a single Gaussian mean that often falls in unreachable regions. We propose NFTR (Normalizing Flow subgoal policies with Triangle-slack Reweighting). A conditional Normalizing Flow replaces the Gaussian policy. A closed-form mode-averaging result identifies the Gaussian limitation, while conditional flows support multi-modal weighted maximum likelihood and direct sampling. A triangle slack score, computed from a jointly trained MRN distance head whose triangle inequality is guaranteed architecturally, corrects the AWR weight multiplicatively and measures a detour inside the learned geometry without requiring exact distance recovery. Triangle-slack vanishes on geodesics in deterministic MDPs and remains a conservative upper bound on composability violation under stochastic dynamics. The RWDR objective preserves AWR's population-level monotonic improvement and admits a three-term suboptimality decomposition. On OGBench the flow carries the larger share of the gain, while the full co-trained configuration adds task-dependent gains. The combined method avoids Gaussian mode averaging and performs well on stochastic tasks.

Primary area:
reinforcement learning
Keywords:
Offline Reinforcement Learning, Goal-Conditioned Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4K1XOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an algorithm named Normalizing Flow subgoal policies with Triangle-slack Reweighting (NFTR) for offline Goal-conditioned Reinforcement Learning (GCRL). NTFR is built on top of HIQL and replaces the high-level policy with a normalizing flow and corrects the value function with a triangle slack score.

The paper provides a theoretical analysis of NTFR and experiments performed on the OGBench benchmark.

Critical strengths
  1. The paper identifies two important problems that are often overlooked in the majority of the offline GCRL literature: optimistic bias in value function updates that follow Temporal-Difference Learning, and averaging modes of Gaussian policies.

  2. The authors associate a theoretical analysis to their algorithmic contributions.

Critical weaknesses
  1. Experiments, in particular in the stochastic setting, do not justify the authors' claims. Other algorithms such as EikHiQRL in [1], show comparable performance in the stochastic setting without explicitly addressing any of the issues highlighted in this paper.

  2. The novelty is incremental. I want to stress that an incremental novelty would be acceptable, in my view, if backed up by strong empirical results.

[1] Giammarino, Vittorio, and Ahmed Hussain Qureshi. "Goal reaching with eikonal-constrained hierarchical quasimetric reinforcement learning." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors
  1. Any comments on the weaknesses highlighted above?

  2. The theoretical analysis seems over-idealized for the tested setting. I was wondering if there is any insight we might get from the theory that applies to the experiments.

5 comments

Response to Reviewer 4K1X (1/3): the stochastic setting and Eik-HiQRL (W1, Q1)

Thank you for the critical review. We read W1 as two linked points. First, our stochastic-setting experiments did not support the claims we made. Second, Eik-HiQRL reaches comparable teleport numbers without targeting optimistic bias or mode averaging, so these two failures may not be what limits performance. We agree with both points as stated. Performance alone cannot identify a mechanism. We therefore (A) compared with Eik-HiQRL under one protocol, (B) tested each mechanism directly, and (C) narrowed every claim the tests do not support. All revised text is in blue in the uploaded PDF.

(A) Eik-HiQRL under the OGBench evaluation protocol (Sec. 4.2, App. F.14, Table 23). We ran the public Eik-HiQRL code ourselves over 8 seeds and report the average of the last three evaluations, matching the reporting rule used for NFTR. The NFTR row reproduces the eight-seed results in the revised paper's main locomotion table:

pm-tel-navpm-tel-stitchant-tel-navant-tel-stitch
Eik-HiQRL, our runs (last-3)46.0 ± 3.642.0 ± 6.048.6 ± 2.443.3 ± 3.9
NFTR (main locomotion table; last-3)53.8 ± 5.941.7 ± 4.552.0 ± 0.644.0 ± 5.5
  • Under the same last-three evaluation rule, NFTR has higher sample means on both navigate tasks (+7.8 and +3.4 pp). The stitch differences (−0.3 and +0.7 pp) are small relative to the reported standard deviations. These summaries do not establish per-task significance.
  • Our reading of why the gap appears on navigate: navigate data contain long trajectories through the teleporters, so one (s, g) pair sees several landing rooms. This is where Thm. 3.1 predicts a single-mean policy to fail. Stitch data consist of short segments, so fewer pairs see several outcomes.

What this comparison does and does not show. Eik-HiQRL improves the high-level value. Its analysis assumes deterministic dynamics (its Assumption 4.2), and its authors note that the method still applies to teleport. We read its results as you do: a better value geometry is another route to robustness under stochastic transitions. They do not show that mode averaging is absent. Eik-HiQRL still extracts a unimodal high-level policy by AWR, so Thm. 3.1 applies to it, and the two changes act on different components and can compose. For navigation, it also learns its high-level quasimetric on the agent's x-y coordinates, whereas NFTR and HIQL use the full state.

(B) Direct tests of the two mechanisms.

  1. Mode averaging: supported. A Gaussian-mixture high level that samples recovers 61–79% of the Gaussian-to-flow gap. The same mixture, returning its dominant mode, falls to the Gaussian level on pm-tel-nav (12.8 vs 11.7) on every paired seed (App. F.12).
  2. Optimistic bias: real, but our fix is narrower than we claimed. In a controlled MDP, the value-only weight gives a lucky portal waypoint all of the AWR mass even when walking is optimal (App. F.13). The slack catches such waypoints only when ordinary behaviour also reaches the landing region. Why: a contrastive distance estimates a density ratio p(w|s)/p(w); when the landing is reached only through the lucky event, both terms scale with its probability, which cancels.

(C) Revised claims. The abstract no longer says "performs well on stochastic tasks". It states that the flow carries most of the gain and that the slack detects detours in a learned reachability geometry, not luck itself.

Response to Reviewer 4K1X (2/3): novelty and the empirical case (W2)

You write that incremental novelty would be acceptable if backed by strong empirical results. We answer both halves: what is new, and what the empirical evidence supports across matched ablations, a fixed-coefficient reference, and the Eik-HiQRL comparison.

What is new. We agree that each component exists: flows inside HIQL (NF-HIQL, now cited as prior work), MRN quasimetric heads, and the composition d(s,w) + d(w,g) (SoRB, LEAP and RIS use it to choose or plan subgoals; QRL, MRN and Transitive RL impose the inequality on value learning).

  1. Why HIQL needs a multi-modal high level, and where. NF-HIQL motivates flows by expressivity. Its KL and PAC bounds hold for any class with bounded log-likelihood, the Gaussian included, so they cannot say when a flow should win. Thm. 3.1 gives that separation for HIQL's own AWR objective: the optimal Gaussian mean is a convex combination of tilted modal centres and leaks irreducible mass between them. Reviewer NGdT describes it as a genuine explanation rather than a post-hoc rationalization, and Reviewer Q4Fu notes that it turns the intuition into a concrete statement about HIQL's objective.
  2. How the triangle residual enters. We neither change nor relax the triangle inequality; the MRN head satisfies it exactly. We use its non-negative residual Δ = d(s,w) + d(w,g) − d(s,g) at offline extraction, leave the value untouched, and multiply the AWR weight by exp(−κΔ). Because Δ ≥ 0 holds by architecture, the correction can only remove weight and never rewards a shortcut. When the same distance enters as a progress reward instead, which also rewards a teleport shortcut, the four-task mean is 44.0, no better than value only (44.5); the one-sided residual gives 47.9 (Fig. 4). The 3.9-point difference is comparable to seed variation, so we treat it as indicative.

The empirical case.

  • The flow alone (κ = 0). In the matched ablation, replacing the Gaussian adds 8.9 to 37.1 points on the five tasks of Table 4. On manipulation, κ = 0 already exceeds the published HIQL numbers on all four tasks (cube-single-play 22.2 vs 15, cube-single-noisy 55.2 vs 41, scene-play 51.7 vs 38, scene-noisy 27.1 vs 25).
  • One fixed coefficient (κ = 1). NFTR stays above HIQL on 10 of the 13 tasks of Table 15.
  • Against Eik-HiQRL. Using the NFTR results already reported in the main locomotion table, we find higher sample means on both teleport-navigate tasks; the two stitch differences are small relative to the reported standard deviations (part 1/3).
  • Where the slack adds. κ = 0 leaves every other parameter bit-identical, so κ > 0 vs κ = 0 isolates the slack. It raises cube-single-play 22.2 → 47.1, cube-single-noisy 55.2 → 67.4 and antmaze-teleport-explore 14.2 → 26.1. These maxima were selected on the reporting seeds and should be read as selected results.

Where the evidence is weak, and why. The pm-tel-nav slack gain uses a coefficient selected on the reporting seeds. On deterministic stitching the penalty hurts (antmaze-medium-stitch 81.0 → 31.0 as κ grows), as the R_κ term of Thm. 3.8 predicts when optimal subgoals carry learned slack. On HumanoidMaze and long-horizon AntMaze stitching, NFTR is below HIQL; there, planning depth and the physical realizability of sampled modes, not mode averaging, appear to be the bottleneck. NFTR is also below HIQL on puzzle-3x3-noisy, for a reason we have not identified. These results stay in the paper.

What the revision adds. App. A cites NF-HIQL as prior work and the classical triangle-based methods. App. F.12 adds the GMM mode-versus-sample test; Sec. 3.2 and App. F.13 the exact account of which lucky transitions a contrastive slack can detect; App. F.14 the theory-to-experiment table and the same-protocol Eik-HiQRL comparison. The anonymized code is released.

Response to Reviewer 4K1X (3/3): what the theory says about the experiments (Q2)

We agree that parts of the analysis are idealized, and the revision separates which parts (App. F.14).

  • Thm. 3.1 matches the implementation. HIQL's high-level Gaussian uses a constant standard deviation (σ = 1), the fixed-σ case of the theorem. Weight clipping makes the AWR weights bounded. A teleporter sends one action sequence to three rooms (Fig. 5), which produces the disjoint waypoint clusters of Eq. 6. The idealized step is exact minimization for each (s, g); a network shares parameters across pairs and can only approach μ*.
  • Prop. 3.3(1)–(3) hold exactly for the MRN head, by architecture. Part (4) concerns d*, which the learned head only approximates; App. F.13 shows where that approximation breaks.
  • Thm. 3.8 and Prop. 3.9 are population-level statements under concentrability. We use them for qualitative predictions, such as the sign of the κ effect, not for numbers.

New Table 22 pairs each result with a measurement:

Formal resultPredictionObservation
Thm. 3.1A multi-modal class helps most where modes are well separated in φ-spaceB0 → B1: +37.1 pp on pm-tel-nav (state = x-y position) vs +8.9 to +10.6 on AntMaze teleport
Thm. 3.1One deterministic subgoal per (s, g) loses the alternativesGMM returning its mode: 12.8, the Gaussian level (11.7); GMM sampling: 34.4
Remark 3.2The deficit belongs to the class, not to the densityFlow, flow-matching, diffusion and mixture heads all beat the Gaussian
Thm. 3.8, term R_κBias grows with κ where optimal subgoals carry non-zero slackDeterministic stitching degrades as κ grows: antmaze-medium-stitch 81.0 → 31.0
Prop. 3.3(4)Only a one-sided detector under stochastic dynamicsReplay AUROC 0.56–0.68 over all transitions, and 0.78 and 0.81 on nearby subgoals in the two navigate datasets; partial discrimination, not a guarantee. The missed class is identified exactly in App. F.13
Cor. 3.5 + clippingNo effect on samples whose weight is clippedIn the MDP, portal mass is unchanged for κ ≤ 2 when both waypoints saturate the clip

Two rows explain results rather than restate them.

  • Row 1. In PointMaze the state is the x-y position, so the three teleport landing rooms are far apart in φ-space and the Gaussian mean lands between them. In AntMaze the state also contains joint angles and velocities; our reading is that these dilute the separation of the clusters in φ-space, so the mean loses less. The size of the flow's gain follows this ordering.
  • Row 4. Thm. 3.8 separates what the penalty can gain (down-weighting off-path subgoals) from what it costs (R_κ, which grows with the slack the head assigns to optimal subgoals). Deterministic stitching has no lucky transitions to filter, so only the cost remains, and success falls as κ grows.

Summary. We agree that our original stochastic-setting claim outran the evidence. The revised paper claims less and supports each claim at the level of mechanism: why the Gaussian fails and when, which component fixes it, how NFTR compares with Eik-HiQRL under one protocol, and what the slack can and cannot detect. We would be glad to clarify any remaining point during the discussion.

Reviewer_Q4FuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors proposed NFTR for GCRL, to mitigate two issues in the classical HIQL algorithm: optimistic bias and mode collapse. With the former, lucky transitions in stochastic environments receive high weights. With the latter, a high level policy may produce subgoals that is the average of multi plausible subgoals while actually unreachable.

NFTR addresses these issues with two key ingredients: first, it replace the Gaussian policy with a conditional NF model, which avoids the collapse issue. It also modifies the training of AWR by adding a triangle slack weight.

Experiments on OGBench locomotion and manipulation tasks show that NFTR has gains over HIQL and other baselines, especially on teleport and stitching tasks. However, ablations show that the improvement comes more from the flow policy side rather than the triangular inequality side. The authors also reports regressions on humanoid maze, some long-horizon stitching tasks and puzzle-3x3-noisy.

Critical strengths

The authors clearly show the failure mode of Gaussian policy through well-designed factorial ablation. They turn the intuition into a concrete statement for HIQL's training objective. The theorem 3.1 shows that the AWR optimal gaussian mean is a convex combination of the AWR tilted mode centers. When the modes are separated by an unreachable region, that mean falls inside the region and leaves irreducible mass there for any practical SD.

Critical weaknesses
  1. Using NF for HIQL is not new. 2602.11142 has done so, though the specific architecture choice is not exactly the same.
  2. Triangle slack adds little to the overall performance gain and this trick itself is not new neither since 2023. In table 4, A0 improves over B1 by only 0.8–5.0, mostly within one SD. According to F.8, A0 uses κ=0\kappa=0 on pm-tel-stitch and pm-med-stitch. With κ=0\kappa=0 and stop-gradient on dθd_\theta, the slack didn't enter the policy training, so A0 should be actually equivalent to B1 in these cases.
Questions for authors
  1. A more intuitive solution for the classical HIQL could be a gaussian mixture policy, besides the NF model. For example, 2606.10025 does so. Other earlier works like 2104.09122 also took a conceptually similar approach. I wonder if the authors can add gaussian mixture as another baseline for comparisons.
  2. NF-HIQL already replaces Gaussian policies with RealNVP flows on OGBench, and it appeared well before the submission deadline. Could the authors clarify how NFTR is positioned relative to NF-HIQL beyond the high level only placement?
  3. The authors do not provide code. I wonder if they can share an anonymized repo during the discussion period so that the results can be verified
7 comments

Response to Reviewer Q4Fu (1/3): NF-HIQL and positioning (W1, Q2)

Thank you for a precise review. We agree with W1, and with the κ = 0 observation in W2. Both led to changes and to new experiments. All changes are in blue in the revised PDF.

W1 / Q2. NF-HIQL is prior work. You are right. NF-HIQL (Garg et al., 2026) inserted RealNVP policies into HIQL at both levels well before our deadline, and calling it concurrent was wrong. Sec. 1 and App. A now describe it as prior work, and we no longer present the flow class as our contribution. Beyond placement, NFTR differs in three ways.

  1. Why a multi-modal class is needed, derived from HIQL's own objective. NF-HIQL motivates flows by expressivity and data efficiency. Its theory has two parts: a KL bound between the behaviour policy and a RealNVP policy (KL ≤ B + log M), and a PAC bound for advantage-weighted MLE. The PAC bound needs only a bounded log-likelihood and finite Rademacher complexity, which a Gaussian class also satisfies, and the KL bound concerns closeness to the data. So neither result says when a flow should beat a Gaussian. Thm. 3.1 is that separation. It shows that the AWR-optimal Gaussian mean is a convex combination of AWR-tilted modal centres and leaks irreducible mass into the region between them; Remark 3.2 shows that this holds under weighted least squares too. The diagnosis makes a falsifiable prediction, which the new GMM experiment tests (part 3/3): any deterministic subgoal per (s, g) should fail, and any sampled multi-modal head should recover much of the gap. Both hold.

  2. Where the separation matters. The theorem predicts gains where one action sequence leads to several disjoint waypoint clusters, which is what stochastic transitions create (a teleporter sends the same actions to three rooms). NF-HIQL evaluates five deterministic tasks (antmaze-medium-navigate, two antsoccer tasks, cube-single-play, scene-play). We ran its public implementation on the stochastic teleport suite (Table 10), where the Gaussian-to-flow gap is largest.

  3. Where the flow belongs, measured inside one codebase.

    • A second, low-level flow costs 2.3–8.3 pp (Table 3).
    • Our value-only high-level arm B1 differs from NF-HIQL mainly in keeping HIQL's Gaussian low level. Against the published two-level NF-HIQL (Table 10), B1 scores 48.8 vs 33.6 (pm-tel-nav), 39.3 vs 35.8 (pm-tel-stitch), 49.8 vs 43.4 (ant-tel-nav), and 40.2 vs 41.5 (ant-tel-stitch).
    • Why the low level does not need a flow (our reading): given a k-step subgoal, the action distribution is close to unimodal, so a flow adds variance there without adding a needed mode.
  4. What the slack is, and what it can detect. RWDR reweights the candidate set, which NF-HIQL does not do. The revision adds a precise account of its scope (App. F.13): with a contrastive head, the slack detects detours in a learned reachability geometry. It flags a lucky transition only when ordinary behaviour also reaches the landing region, because otherwise the luck probability cancels in the density ratio the head estimates.

Response to Reviewer Q4Fu (2/3): the slack's contribution (W2)

κ = 0 makes A0 identical to B1. Correct. The distance head has its own parameters, and with κ = 0 its slack is multiplied by zero. We verified numerically that the high-level, value, low-level and encoder parameters are then bit-identical whether or not the head is trained; the check is in the code release. The check runs on CPU with a fixed seed; GPU training is not, in general, bit-reproducible, so separately trained runs still differ. The A0 − B1 gaps on pm-tel-stitch and pm-med-stitch (2.4 and 0.8 pp) are therefore run-to-run variation. Table 4 now marks these entries as replicates (‡).

The pm-tel-nav result is selected on the reporting seeds. The κ=2 value of 53.8 ± 5.9 is the highest of five settings in the sweep (Table 15), compared with 48.2 ± 3.6 at κ=0. Because κ was selected using the same eight seeds, we do not treat the 5.6 pp difference as an independent estimate of the slack's effect.

What remains for the slack. Because κ = 0 leaves everything else bit-identical, Table 15 is a clean slack-only ablation.

  • Selected gains: cube-single-play 22.2 → 47.1, cube-single-noisy 55.2 → 67.4, antmaze-teleport-explore 14.2 → 26.1. These maxima were selected on the reporting seeds and are presented as such.
  • Losses: on deterministic stitching the penalty hurts (antmaze-medium-stitch 81.0 → 31.0 as κ grows). Why: this is the bias term R_κ(d_θ) of Thm. 3.8. Where optimal subgoals carry non-zero learned slack, the penalty moves the target away from them.
  • With one fixed κ = 1 (no per-task selection), NFTR stays above HIQL on 10 of 13 tasks.

The slack's effect on final success varies by task. We now present it as a one-sided correction with a characterized scope, not as a general performance booster.

"Not new since 2023." We agree that the triangle quantity is classical. d(s,w) + d(w,g) selects or plans through subgoals in SoRB (2019), LEAP (2019) and RIS (2021), and the triangle inequality structures value learning in QRL, MRN and Transitive RL. These are now cited (App. A). Our claim is narrower: the non-negative residual of an architecturally quasimetric head, used as a one-sided multiplicative correction of offline AWR weights, together with an account of which stochastic outcomes it can and cannot detect. If you had a specific 2023 paper in mind, we would be grateful for the pointer.

Response to Reviewer Q4Fu (3/3): Gaussian-mixture baseline (Q1) and code (Q3)

Q1. GMM baseline (new App. F.12). We replaced the NF head by a diagonal K-component GMM, trained by the same weighted maximum likelihood, with the same weights and hierarchy (1M steps; 8 seeds per task on pm-tel-nav, ant-tel-nav, and ant-tel-stitch). For context: 2606.10025 (GHOST, RSS 2026) predicts sub-goals with per-patch mixtures in imitation learning, and 2104.09122 (PMOE) uses mixture policies in online, flat RL. Both are now cited.

pm-tel-nav, value-only weightExecutionSuccess (%)
Gaussian (B0)mean11.7 ± 0.4
GMM, K=4dominant mode12.8 ± 2.7
GMM, K=2 / 4 / 8sample32.2 ± 4.7 / 34.4 ± 4.2 / 36.2 ± 5.8
NF (B1)sample48.8 ± 5.4
TaskB0GMM (κ=0)B1 (NF)Gap recoveredGMM + RWDR
ant-tel-nav39.246.5 ± 2.949.869%47.0 ± 2.1 (κ=1)
ant-tel-stitch31.338.4 ± 4.040.279%36.3 ± 2.9 (κ=1)
pm-tel-nav11.734.4 ± 4.248.861%35.0 ± 1.5 (κ=2)

What the numbers show, and why:

  • Sampling, not the mixture alone, removes the Gaussian deficit. Returning the dominant mode drops the GMM to the Gaussian level on every paired seed (difference −21.7, 95% CI [−28.0, −15.4]). Why: Thm. 3.1 concerns the averaged mean, but its consequence is broader. Any deterministic subgoal per (s, g) commits the agent to one cluster. If that cluster is a teleport route that sometimes fails, the agent picks it again at every replanning step; a sampled subgoal also reaches the alternatives. This supports Remark 3.2: the deficit belongs to the class of single-point choices, not to the Gaussian density.
  • More components do not close the gap to the flow. K = 8 is 1.8 pp higher than K = 4 (CI [−3.0, 6.7]), and both stay 12–17 pp below the flow.
  • RWDR does not change success on the GMM backbone (macro change −0.3 pp, CI [−2.9, 2.3]). Why: this is consistent with the mechanism in App. F.13. On teleport navigation, the slack seldom separates lucky from reliable waypoints at the advantages that survive weight clipping. We therefore do not claim that RWDR improves success independently of the policy class.

Q3. Code. https://anonymous.4open.science/r/anon_nftr-FEF2 . It reproduces every NFTR number, including the κ sweep. It also includes the script that checks the κ = 0 equivalence. While preparing it we found and fixed four places where the text did not match the code (General Response 2/2); no reported number changes.

Thank you again. Your W2 observation led directly to the κ=0 equivalence check and a more careful reading of the coefficient sweep. If these changes address your concerns, we would appreciate it if you would consider revising your score.

Reviewer_NGdTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

NFTR addresses HIQL's mode-collapse and optimistic-bias failures by replacing the unimodal Gaussian high-level subgoal policy with a conditional Normalizing Flow and reweighing AWR using a triangle-slack penalty from an MRN-based quasi metric.

Critical strengths

TIn general I like the paper because authors are giving a genuine explanation rather than a post-hoc rationalizations. The main point is Theorem 3.1 that gives a closed-form explanation of why HIQL's unimodal Gaussian high-level policy fail. And Remark 3.2 shows the diagnosis doesn't depend on the density choice. The triangle-slack idea is elegant. It treats the triangle inequality as a prior on the AWR weightt, and the architectural guarantee that Δ ≥ 0 means the correction can only remove weight and without leading a reward shortcut. Also the empirical evaluation that is self-critical, which is a strengths rather than weaknesses.

Critical weaknesses

The slack (s,w,g) = d(s,w) + d(w,g) − d(s,g) is non-negative by construction, so it can only downweight subgoals that require a longer path. And in my intuition the motivating failure is the lucky transition (a teleporter or noise realization) that makes a bad subgoal look good. The empirical support for the mechanism is the claim that it helps on teleport tasks, which is exactly the regime where a geometric correction would be expected to work. So to justify their method, authors considered the easy case and leaves the case that is off-path but can create a geometrically consistent shortcuts. Empirical tests would strengths the paper.

Questions for authors

See weaknesses above.

2 comments

Response to Reviewer NGdT (1/2)

Thank you for the careful reading, and for naming the exact case our evidence did not cover. You were right on both counts. Δ ≥ 0 can only remove weight, and teleporters are the regime where a geometric correction is expected to work. We built the case you described and tested it directly. The result confirms your concern and also explains it, which sharpens what the paper claims. All changes are in blue in the revised PDF (Sec. 3.2, App. F.13, Limitations).

The experiment (new App. F.13). A controlled MDP isolates a geometrically consistent lucky shortcut:

  • From the start, a reliable route walks to the goal in L_A + 1 steps.
  • A portal succeeds with probability p and lands two steps from the goal, on the shortest line to it; on failure it drops the agent into a dead end.
  • We train V and the MRN head with exactly the paper's losses, and compare the slack of the lucky waypoint with that of the reliable one (L_A ∈ {10, 40, 80}, p ∈ {0.1, …, 1}, 8 seeds).
  • Besides the learned head, we compute two exact references from the known chain: the behaviour occupancy distance −log M(s,w)/ρ(w), which is what a contrastive loss estimates, and the first-passage distance −log E[γ^T].

Result 1: under the occupancy distance, luck cancels exactly (Table 19).

  • When the landing is reachable only through the portal, the occupancy slack difference is identical for every p. It equals −m·(L_A − 3) to three decimals (−0.070, −0.372, −0.774), so it measures only the steps the shortcut saves.
  • Why: occupancy distances are density ratios against the stationary distribution. Every visit to the landing state passes through the portal, so its occupancy from the start and its stationary mass are both proportional to p, and p drops out of the ratio. A lucky shortcut of this kind is therefore invisible to any slack built on a perfectly trained contrastive distance. This is a property of the target, not an optimization failure.

Result 2: a first-passage distance keeps the luck premium.

  • Its slack difference falls from 1.87 at p = 0.1 to 0.10 at p = 1 (L_A = 10), close to log(1/p) minus the geometric saving, and it changes sign once the saving outweighs the premium (e.g. L_A = 80, p ≥ 0.75).
  • Why: E[γ^T] multiplies the success probability into the distance of the lucky leg, and nothing in the composition divides it out again.

Result 3: when ordinary behaviour also reaches the landing, the occupancy slack separates the cases.

  • With a second path that walks into the landing state, the occupancy difference becomes positive at low p (0.218 at p = 0.1 vs 0.121 at p = 1, L_A = 10).
  • Why: walking arrivals now dominate the stationary mass of the landing, so the cancellation breaks and the unreliable leg lengthens d(s, w).
  • This is the OGBench situation. The teleport-out rooms are also traversed on foot in the navigate and stitch datasets. Consistently, our replay test (Fig. 5) ranks non-repeatable transitions above repeatable ones with AUROC 0.56–0.68 over all transitions, and 0.78 and 0.81 on nearby subgoals in the two navigate datasets. This is partial discrimination on these datasets, not a general guarantee.

Response to Reviewer NGdT (2/2)

Result 4: the learned head adds another, unwanted bias (Table 20).

  • The learned slack difference is negative in 29 of 30 settings, and its magnitude grows with route length (−0.6 at L_A = 10 vs −2.6 at L_A = 40, p = 0.1). The head penalizes the long reliable route more than the lucky one, even where the exact occupancy distance separates them.
  • Why (our reading): two effects scale with route length. The occupancy target already favours the short route by m per saved step. In addition, 30% of our contrastive positives are random states, which bounds the learned distance, so a long chain cannot keep additive distances in the embedding and its intermediate waypoints pick up spurious slack.

Result 5: two mechanisms decide whether the slack changes the policy (Table 21).

  • Clipping. With a long reliable route, the lucky waypoint's advantage saturates the clip of 100, and the portal receives 100% of the AWR mass for every κ. This reproduces HIQL's optimistic failure, and the slack cannot undo it.
  • Sign. Where the learned difference is negative, a larger κ moves weight toward the gamble (from 27.5% to 99.2% at p = 0.25 with a walkable landing).

What this means for the paper. We now describe the slack as a one-sided detector of detours in the learned reachability geometry, not as a detector of luck. With a contrastive head, it can flag a lucky jump only when the landing region is also reached without luck, as in the OGBench teleport mazes. It misses lucky transitions whose landing is reachable only through the stochastic event, which is the case you raised. This matches the one-sided statement of Prop. 3.3(4), and we now say it explicitly. The analysis also points to a constructive fix: a first-passage distance (as targeted by TMD) would encode the luck premium log(1/p) directly.

We also checked whether the slack improves success where we had claimed it. With κ = 0 the policy is bit-identical to the value-only arm, so κ = 0 vs κ > 0 isolates the slack. The κ sweep (Table 15) shows selected gains on cube-single-play (22.2 → 47.1) and antmaze-teleport-explore (14.2 → 26.1), alongside regressions on deterministic stitching. These maxima were selected on the reporting seeds.

Revisions (blue in the PDF).

  • Sec. 3.2: new paragraph "What the slack can and cannot detect", stating the density-ratio cancellation and its consequence.
  • Sec. 3.2 after Cor. 3.5: the clipping limitation.
  • App. F.13: the controlled MDP with its three tables; App. F.5: why the OGBench teleport case is detectable.
  • Limitations: the missed class, the clipping effect, and first-passage distances as future work.
  • Abstract and Sec. 4.2: removed claims that the MRN "reflects typical reachability rather than lucky outcomes".

We are grateful for this question. It turned an implicit assumption into a stated, tested scope, which we think makes the slack easier to use correctly. If the revision addresses your concern, we would appreciate it if you would consider it in your final assessment.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (2/2)

General Response (2/2): corrections and revised claims

Two further mechanisms limit the slack (App. F.13, Table 21).

  • Clipping. Weights are clipped at 100 before normalization. For samples whose advantage already saturates the clip, the penalty has no effect. In the MDP with a long reliable route, the lucky waypoint receives 100% of the AWR mass for every κ; this reproduces HIQL's optimistic failure, and the slack cannot reverse it.
  • Sign. Our learned head adds slack to the long reliable route rather than to the lucky one (negative difference in 29 of 30 settings). Two causes scale with route length: the occupancy target already favours short routes, and part of the contrastive positives are random states, which bounds the distance and breaks additivity along long chains. With such a sign, a larger κ moves weight toward the gamble.

Code–text corrections found while preparing the release. No reported number changes.

  1. The one-step distance hinge (Eq. 20, Alg. 1) was written in the wrong direction. The code enforces d(s,g) ≥ m + d(s′,g), which propagates the worse observed successor.
  2. The contrastive positives are the HIQL value goals (current 0.2, geometric future 0.5, random 0.3), and the negatives are an in-batch shift. The text had described trajectory-based sampling.
  3. App. C.3 wrongly said the NF targets are not length-normalized. They are, by the last layer of φ, so the inference normalization is consistent with training.
  4. Tables 7–9 now list the network sizes in the released code.
  5. Eik-HiQRL was cited with the wrong reference; it now cites Giammarino & Qureshi (ICLR 2026).
  6. NF-HIQL (ICRA 2026) is prior work, not concurrent. The introduction and a new paragraph in App. A state what NFTR adds: NF-HIQL's KL and PAC bounds hold for any class with bounded log-likelihood, so they do not separate a flow from a Gaussian, and its evaluation covers deterministic tasks only; Thm. 3.1 is the separation, and it predicts the gains on stochastic, multi-modal tasks.
  7. OGBench noisy datasets contain noisier behaviour data, not observation noise; corrected.

What we now claim, and what we no longer claim.

  • Kept: Thm. 3.1 and the flow as the main contribution; high-level-only placement; the one-sided slack with its guarantees (Prop. 3.3, Cor. 3.5, Thm. 3.8).
  • Narrowed: the slack's effect on success is task-dependent. The large gains on cube-single-play (22.2 → 47.1) and antmaze-teleport-explore (14.2 → 26.1) are selected on the reporting seeds, and we say so.
  • New: a characterization of which lucky transitions a contrastive slack can detect, with a constructive implication. A first-passage distance would expose the luck premium that a contrastive head cancels.
  • New: Eik-HiQRL run under the OGBench protocol (8 seeds, last three evaluations) reaches 46.0 / 42.0 / 48.6 / 43.3 on the four teleport tasks. Table 23 reuses the NFTR results from the main locomotion table; under the same last-three evaluation rule, NFTR has higher sample means on both navigate tasks, while the stitch differences are small relative to the reported standard deviations.
  • Removed: "performs well on stochastic tasks" (abstract) and the statement that the MRN "reflects typical reachability rather than lucky outcomes" (Sec. 4.2).

We believe the revised paper is more accurate and more useful, because it states where each component helps and why.

AuthorsOpenReview ↗

General Response (1/2)

General Response (1/2): what the new evidence shows

We thank all three reviewers. The reviews pushed us to run the experiments that test our mechanisms directly, and some results go against our original wording. We report them as they are. A revised PDF is uploaded. Blue text in the revised PDF marks content added or revised during rebuttal. Citation links keep the template colour (orange); in tables, bold orange still marks the best entry, and changed entries carry a ‡. The main text is now 10 pages; new material is in App. F.12–F.14.

Code. https://anonymous.4open.science/r/anon_nftr-FEF2 reproduces every NFTR number, including the κ sweep, and contains a script that checks the κ=0 equivalence below.

Finding 1. The flow carries the gain, and sampling is what matters. We added a Gaussian-mixture (GMM) high level under the identical hierarchy and weights (App. F.12; 8 seeds per task on pm-tel-nav, ant-tel-nav, and ant-tel-stitch).

  • With sampling, a 4-component GMM recovers 61–79% of the Gaussian-to-flow gap (pm-tel-nav 11.7 → 34.4, flow 48.8; ant-tel-nav 39.2 → 46.5, flow 49.8).
  • The same GMM that always returns its dominant mode scores 12.8 on pm-tel-nav, the Gaussian level (11.7), and drops on every paired seed.
  • Why: Thm. 3.1 says the Gaussian mean averages the modes. The mode experiment shows the broader cause: any deterministic subgoal per (s, g), mean or mode, commits the agent to one cluster. A sampled subgoal keeps the alternatives.

Finding 2. The slack's effect depends on the task and coefficient.

  • With κ=0 the policy, value and encoder parameters are bit-identical to the value-only arm B1 (the distance head has separate parameters and its slack is multiplied by zero). So κ=0 vs κ>0 is the only clean slack ablation (Reviewer Q4Fu was right about A0 ≡ B1 at κ=0).
  • The κ sweep (Table 15) includes both gains and regressions. On pm-tel-nav, the selected κ=2 result is 53.8 ± 5.9 versus 48.2 ± 3.6 at κ=0; κ was chosen using the same eight reporting seeds, so this is a selected comparison.

Finding 3. The slack detects detours in a learned geometry, not luck itself. We built a controlled MDP in which a portal succeeds with probability p and lands on the shortest line to the goal (App. F.13). Exact computation shows:

  • Under the behaviour occupancy distance (what a contrastive loss estimates), the slack difference between the lucky and the reliable waypoint does not depend on p. Why: occupancy distances are density ratios against the stationary distribution. When the landing is reached only through the portal, both its occupancy from the start and its stationary mass scale with p, so p cancels.
  • Under a first-passage distance −log E[γ^T], the difference keeps the luck premium (1.87 at p=0.1, 0.10 at p=1).
  • When ordinary behaviour also reaches the landing, the cancellation breaks and the occupancy slack separates the two. The OGBench teleport-out rooms are of this kind, which is consistent with our replay test (Fig. 5): AUROC 0.56–0.68 over all transitions on the four teleport datasets, and 0.78 and 0.81 on nearby subgoals in the two navigate datasets. This is partial discrimination on these datasets, not a general guarantee.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.