acceptodds
Under review as a conference paper at ICLR 2027

Quasimetric Goal Representation for Offline Goal-Conditioned RL

Abstract

Goal-conditioned reinforcement learning (GCRL) constitutes a foundational framework for decision-making in a broad class of goal-reaching tasks. A central challenge in GCRL is to construct a structured goal representation that is both sufficient for optimal decision-making and invariant to exogenous noise. However, designing such a representation remains highly nontrivial and largely underexplored. In this work, we identify that the optimal temporal distance between states and goals is inherently asymmetric, endowing a natural quasimetric structure. Based on this observation, we show that any quasimetric embedding induces a goal representation that automatically satisfies sufficiency and noise invariance, providing a simple and general recipe for representation design. As a concrete instantiation, we introduce a weighted quasimetric that yields a practical and efficient quasimetric goal representation. Empirically, our method consistently improves offline goal-reaching performance across most state- and pixel-based tasks in the OGBench suite, outperforming both raw-goal baselines and prior goal representation methods by a large margin. Our code is available at https://anonymous.4open.science/r/Q-Rep-316F.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Goal-Conditioned RL

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_7VE5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Q-Rep by learning an geometrically optimal quasimetric representation function of the states for GCRL. By parametrizing the quasimetric function as asymmetric functional, the method is simple in implementation and requires only modification to learning value function. Experiments on the OGBench standard benchmark showcases that method obtains comparable performance to existing methods.

Critical strengths

It is easy to follow the text in the paper. The math notation is the same as in the literature, making it easy to read the paper and theorems. The results on the OGBench showcases that method could outperform existing algorithms.

Critical weaknesses
  1. The "Conclusion" section completely misses the limitations of the approach and feels like it is not finished. Basically, it is just 2 sentence summary of the abstract, which completely ignores obtained results, insights made by the paper and, importantly, not discusses limitations.

  2. The paper completely ignores vast literature in the same spirit of learning optimal representations in GCRL (see e.g [1, 2, 3]). Those papers claim learning theoretically optimal GCRL representations, so it is hard to understand whether claims on optimality of proposed Q-Rep are indeed novel, since there are no comparisons with existing literature.

  3. It is not clear what is the novelty in the Section "3.1 MAIN IDEA". From my understanding, all of the introduced theorems are just reiteration of Park. et al "Dual Goal Representations" with more math rigor. It is not clear for me what is the exact contribution?

  4. There are no ablation studies.

[1] Bill Chunyuan Zheng et al. : "Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learning"

[2] H.Viswanath et al.: "Mollified Value Learning"

[3] V. Giammarino et al.: "Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning"

Questions for authors

Unfortunately, I could not download and inspect code since the page showcases "This content can’t be shown. The repository is expired.". It would be better if author's could fix this.

Questions:

  1. Is it correct that the main contribution is in mathematical foundation of quasimetric embeddings? Inspecting Algorithm 1, the only modification is parametrization of VFV_F, where F is a learned assymetric functional from Section 3.1 by equation 4?

  2. What are the limitations of the Q-Rep? What are the examples of the e.g environments, where quasimetric representations are not optimal?

  3. Is OGBench is the only established benchmark in the GCRL literature? From my experience, OGBench is very synthetic and does not generally showcases weaknesses of the algorithms.

  4. Could the authors clarify what the Exogenous Block CMP (Ex-BCMP) framework contributes to the theory of Q-Rep specifically? Ex-BCMP is borrowed from Efroni et al. (2021) and was already used by Park et al. (2025) to establish noise invariance for Dual Goal Representations.

  5. Is there any experiment that tests the Ex-BCMP condition directly? Sec. 5.4 notes that Gaussian goal noise does not, so one example would be goals rendered with different distractors that share the same latent state.

Overall, my main concern is that it is not clear what is the exact novelty and insight the paper introduces. I would consider increasing the score if the authors can answer the questions.

Reviewer_kKWNOpenReview ↗
Rating
2
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Quasimetric Goal Representation (Q-Rep) for offline goal-conditioned RL. It connects optimal temporal distance to quasimetric geometry and establishes sufficiency and noise invariance under an exact distance-preserving embedding. The practical method uses a weighted asymmetric ℓ1\ell_1 distance, learns the encoder through a GCIQL value objective, and conditions a downstream GCIVL policy on the goal embedding. Experiments cover 17 state- and pixel-based OGBench tasks, with a symmetric ablation and additional value-approximation and goal-noise studies.

Critical strengths

1. Simple and well-motivated inductive bias: separate costs for positive and negative latent displacements allow the representation to model directional differences in goal reaching using inexpensive element-wise operations. The resulting goal embedding can be incorporated into an existing offline GCRL pipeline without changing the downstream policy formulation.

2. Strong empirical results with a relevant ablation: Q-Rep achieves the highest or tied-highest mean score on 13/17 tasks, with state-based and pixel-based averages of 52 and 51 versus 45 and 42 for Dual (Table 1). Within Q-Rep, asymmetric weighting raises the state-based average from 45 to 52 and improves the mean on 9/12 tasks (Table 2). Results over five seeds provide meaningful support for the proposed design.

Critical weaknesses

1. The connection between the exact-embedding theory and the proposed distance family is insufficiently established (main concern). The sufficiency and noise-invariance results assume exact preservation of the optimal distance profile, but the weighted asymmetric ℓ1\ell_1 parameterization has a structural restriction: dF(s,g)−dF(g,s)=(α−β)⊤(ϕ(g)−ϕ(s))d_F(s,g)-d_F(g,s)=(\alpha-\beta)^\top(\phi(g)-\phi(s)). Its directional asymmetry is therefore a potential difference with zero sum around every cycle, excluding even a unit-cost directed three-state cycle. The paper does not characterize this restriction or explain its consequences for representation learning. The formal treatment also needs clarification: unreachable goals require extended-valued distances, while latent-equivalent observations in the Ex-BCMP induce a pseudoquasimetric on the observation space. These issues do not invalidate the conditional results, but limit their justification of the practical design.

2. The experiments do not establish why this particular quasimetric parameterization is preferable. Table 2 supports asymmetry within Q-Rep, but neither Table 1 nor Figure 2 compares against alternative quasimetric architectures such as IQE or MRN under the same learning pipeline. Figure 2 measures agreement with a learned value reference, and Figure 3 tests Gaussian goal noise against raw goals; neither resolves this design question. The task-level results also warrant analysis: Dual substantially outperforms Q-Rep on state-based cube-single (89 vs. 64) and cube-double (60 vs. 35), despite Q-Rep winning on their visual counterparts. These mixed results do not contradict the theory, but make a matched architecture comparison and an explanation of the failure cases important. Since Table 1 imports baseline results, training and hyperparameter-selection parity should also be documented.

Questions for authors
  1. How does the zero-circulation restriction of Equation (4) affect the intended class of goal-reaching problems? Can the authors characterize its representational limitations and clarify the treatment of unreachable and latent-equivalent goals?

  2. How does Q-Rep compare with IQE or MRN under matched representation-learning and downstream policy-training settings? Are the budgets and hyperparameter-selection procedures matched to those underlying the reported Dual results?

  3. What explains the substantial regressions on state-based cube tasks and the reversal on their visual counterparts? A controlled analysis would help identify when the proposed geometry is beneficial.

Reviewer_u2yeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Q-Rep, a goal representation method for offline goal-conditioned RL that exploits the asymmetry of optimal temporal distances. It builds on the observation that the optimal temporal distance d⋆(s,g)≜logγV⋆(s,g)d^\star(s,g) \triangleq log_\gamma V^\star(s,g) is a quasimetric rather than a symmetric metric. Q-Rep maps states into a latent space equipped with a learned weighted asymmetric ℓ1\ell_1 norm, trained by distilling a separately-learned value function into this quasimetric geometry. The policy is trained on the resulting goal encoding with GCIVL. Under an exact embedding, the paper argues this representation is sufficient for optimal goal-reaching and, in the Ex-BCMP setting, invariant to exogenous goal noise.

Empirically, Q-Rep matches or outperforms the baselines on 13 of 17 OGBench tasks, with the largest gains on pixel-based tasks, and degrades more gracefully under goal-observation noise than a raw-goal baseline.

Critical strengths

Well-motivated idea. The motivation is intuitive and easy to grasp: goal-reaching dynamics are often asymmetric, so symmetric metrics fail to capture the geometry that matters for control. Framing the optimal temporal distance as a quasimetric, and encoding this via an asymmetric representation, seems a compelling design choice.

Clarity of presentation. The paper introduces key ideas in a clear sequence and makes the relationship between the value function and temporal distance explicit. The method is presented as a clean two-phase procedure (representation learning, then policy learning), which make it easy to follow.

Isolation of the asymmetry effect. Table 2, which keeps everything identical except the αi=βi\alpha_i = \beta_i constraint, is a reasonable test to isolate whether asymmetry itself drives the gains.

Critical weaknesses

Limited novelty. The theoretical findings are largely established in prior work, like the quasimetric structure of the goal-reaching value function in [1], or the sufficiency and Ex-BCMP noise-invariance of a temporal-distance-profile representation were introduced in [2, 3]. Notably, a very relevant and related quasimetric offline-GCRL method, TMD [3], is not even cited, despite overlapping with some of the claimed contributions. To my understanding, what is genuinely new reduces mainly to the weighted asymmetric ℓ1\ell_1 parameterization and the quasimetric-embedding framing.

Unclear attribution of novelties in Section 3. It is hard to tell from the main text which of the definitions and Theorems 3.1-3.3 are a new contribution vs. a restatement of established concepts and results. Section 3 presents the three theorems as a novel conceptual trio without explicitly highlighting their overlap with prior work. What makes it even more difficult to assess novelty and impact based on the main text is that the discussion of related work is deferred to Appendix E, even though the main text is roughly half a page below the page limit.

No comparison against TMD. The paper compares against representation baselines (e.g., VIB, VIP, TRA, BYOL-γ\gamma, Dual-Goal) but not against TMD [3]. This is critical, since TMD is precisely the method that would help judging how much Q-Rep adds over existing quasimetric GCRL methods.

Experimental protocol relative to prior work is somewhat unclear. The paper "adopt[s] baseline results from [2]". But it does not made it explicit whether Q-Rep was evaluated on the identical OGBench configurations (same seeds, normalization, evaluation protocol) as the baselines.

Reproducibility is unclear. The anonymized repository link is expired, so I could not check the implementation or the exact experimental protocol.

Theory-practice gap. Theorems 3.2-3.3 require exact quasimetric embeddings, which the practical method does not achieve as it distills an approximate value function into the ℓ1\ell_1 geometry. The paper does not analyze how approximation error might affect the theoretical guarantees, so the theory does not directly explain or justify the empirical gains.

Minor points that did not change my overall recommendation:

  • Some figures appear to be low resolution and may not be vector graphics.
  • Inconsistency in the specification of a CMP (definition in Section 2 vs. Theorem 3.2).
  • Inconsistencies in the notation of ℓ1\ell_1 norm.
  • There are line breaks within inline equations (e.g., in Line 652).
  • I recommend pointing to the appendix proofs directly after each theorem is stated, rather than only after the third one.

Currently, these concerns outweigh the strengths of the paper. However, I am open to revising my assessment based on the authors' clarifications and any changes to the manuscript during the rebuttal phase.


Questions for authors
  • Which results in Section 3 are genuinely new? Can the authors explicitly describe what Q-Rep adds beyond prior works [1,2]?
  • How does Q-Rep compare against TMD [3]? More specifically, where does the paper's approach differ from the theory and method established in [3]?
  • Was Q-Rep evaluated on the identical OGBench configurations as the baselines? If not, could the authors rerun the baselines under identical conditions to ensure a fair comparison?
  • Theorems 3.2-3.3 hold only for exact quasimetric embeddings, which the practical method never achieves. Can the authors comment on how distance-approximation error degrades sufficiency or noise-invariance? Or can the authors provide any analysis, even empirical, for quantifying this gap? This would help to connect the theory to the reported empirical gains.
Reviewer_3S6hOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose Q‑Rep, a quasimetric‑based goal‑conditioned reinforcement learning (GCRL) algorithm. This approach uses a weighted asymmetric L1 norm to enable asymmetry while preserving the triangle inequality. This parametrization provides useful inductive biases for goal‑conditioned reinforcement learning by implicitly enabling compositionality. In addition, the authors show that Q‑Rep not only uses representations as a parameterization for improving value functions with the inductive bias, but also enables better control with a learned policy. The authors make theoretical claims about the expressivity and robustness of this parameterization. They claim improvements over past GCRL methods on OGBench.

Critical strengths
  • The paper connects the representation learning implicitly required by quasi‑metric value‑function parameterizations to policy learning. This is a key advance over prior work that focuses either on learning representations that are good for control [2] or on learning representations that have a nice metric structure [1]. Conceptually, this is a very appealing idea. When it comes to scaling RL agents and exploiting the specific inductive biases of value functions, the cost of maintaining separate networks and potentially having to repeat learning across the actor and critic becomes significant. The learned goal representation is reused by the geometric value model and the downstream policy and value networks, potentially reducing duplicated representation learning even though the networks are not fully shared. This interleaved training raises potential concerns about instability, interference, and forgetting. The paper provides compelling early results that this kind of representation reuse is possible with the strong inductive biases of quasi‑metrics.
  • Compelling comparisons against alternative parameterizations in Figure 2.
  • The paper provides an interesting goal-noise robustness analysis. They also provide evaluations on visual tasks, which are traditionally challenging for these kinds of metric geometry approaches.

[1] Wang et al., 2022. "Improved Representation of Asymmetrical Distances with Interval Quasimetric Embeddings." Workshop on Symmetry and Geometry in Neural Representations (NeurReps).

[2] Myers et al., 2025. "Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Following." Conference on Neural Information Processing Systems.

Critical weaknesses
  • There are missing comparisons against very related prior quasimetric learning methods [1, 2, 3]. TMD, in particular, seems to use a similar policy-plus-representation architecture, and the published tables show stronger results on the shared OGBench datasets (see tables).
  • The paper in its current form overstates contribution:
    • There are key missing evaluations of prior works that show better performance in the baseline study.
    • The novelty of the theoretical results is overstated. Theorem 3.1, regarding the connection between quasimetric distances and hitting times, seems to be a restatement of CMD's Theorem 3.4 [4], proved in Appendix E, with the hitting-time connection established in Appendix B.

[1] Myers et al., 2025. "Offline Goal-conditioned Reinforcement Learning with Quasimetric Representations." NeurIPS. [2] Zheng et al., 2026. "Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learning." ICLR. [3] Liu et al., 2023. "Metric Residual Network for Sample Efficient Goal-Conditioned Reinforcement Learning." AAAI. [4] Myers et al., 2024. "Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making." ICML.

Modality / taskQ-RepDualBest listed baselineDifference vs Dual
State / cube-single-play64±164\pm189±389\pm3VIB: 90±390\pm3-25
State / cube-double-play35±335\pm360±460\pm4Dual: 60±460\pm4-25
State / scene-play72±572\pm572±672\pm6Dual: 72±672\pm60
State / puzzle-3x3-play9±29\pm25±15\pm1VIB: 14±314\pm3+4
State / puzzle-4x4-play29±329\pm323±323\pm3Dual: 23±323\pm3+6
Pixel / visual-cube-single-play65±365\pm358±558\pm5Dual: 58±558\pm5+7
Pixel / visual-cube-double-play13±213\pm29±19\pm1Dual: 9±19\pm1+4
Pixel / visual-scene-play34±434\pm426±526\pm5Dual: 26±526\pm5+8
TMD datasetTMD reported resultQ-Rep claimed result
humanoidmaze-medium-navigate64.6±1.164.6\pm1.148±248\pm2
humanoidmaze-medium-stitch68.5±1.768.5\pm1.7—
humanoidmaze-large-stitch23.0±1.523.0\pm1.5—
humanoidmaze-giant-navigate9.2±1.19.2\pm1.1—
humanoidmaze-giant-stitch6.3±0.66.3\pm0.6—
pointmaze-teleport-stitch29.3±2.229.3\pm2.2—
antmaze-medium-navigate93.6±1.093.6\pm1.088±488\pm4
antmaze-large-navigate81.5±1.781.5\pm1.761±461\pm4
antmaze-large-stitch37.3±2.737.3\pm2.7—
antmaze-teleport-explore49.6±1.549.6\pm1.5—
antmaze-giant-stitch2.7±0.62.7\pm0.6—
scene-noisy19.6±1.719.6\pm1.7—
visual-antmaze-teleport-stitch38.5±1.538.5\pm1.5—
visual-antmaze-large-stitch26.6±2.826.6\pm2.8—
visual-antmaze-giant-navigate40.1±2.640.1\pm2.6—
visual-cube-triple-noisy17.7±0.717.7\pm0.7—
Questions for authors
  • Could you clarify the universality framing for Theorems 3.2 and 3.3? How does this differ from the expressivity results in IQE [1] and MRN [2]?
    • How does this universality work when you consider distances that form a cycle? It seems that x→y→z→xx \to y \to z \to x will always have the same total distance as the reverse, which is not true for general environments without complete reversibility.
  • What do the error bars represent? Which results are statistically significant?
  • Could you compare against and discuss Temporal Metric Distillation (TMD) [3]?
    • Also, MQE [4] seems relevant.
  • Although there are issues with the presentation and experimental results of the work currently, I would be willing to raise my score if the comparisons and clarification of contributions against prior work are addressed.
  • Could you independently ablate the various components of your approach? In particular, you could imagine doing the policy learning only with the representations, but not the value learning, and vice versa.

[1] Wang et al., 2022. "Improved Representation of Asymmetrical Distances with Interval Quasimetric Embeddings." Workshop on Symmetry and Geometry in Neural Representations (NeurReps). [2] Liu et al., 2023. "Metric Residual Network for Sample Efficient Goal-Conditioned Reinforcement Learning." AAAI. [3] Myers et al., 2025. "Offline Goal-conditioned Reinforcement Learning with Quasimetric Representations." NeurIPS. [4] Zheng et al., 2026. "Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learning." ICLR.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.