acceptodds
Under review as a conference paper at ICLR 2027

Continual Reinforcement Learning with Quasimetric Structure Evolving across Tasks

Abstract

The goal of continual reinforcement learning is to integrate accumulated experience, rapidly learn new tasks, and minimize forgetting. In real-world scenarios, transition dynamics and reward functions often vary substantially across tasks, rewards are frequently sparse, and abrupt changes occur at task boundaries, making it difficult to learn a task-agnostic policy that generalizes across tasks. Inspired by compositional recombination in human cognition, we propose CONQUEST, a framework that learns a cross-task quasimetric representation to capture asymmetric and compositional knowledge-transfer relationships during continual learning. Quasimetric structures naturally characterize the reachability between states and goals, enabling agents to identify transferable knowledge without relying on task-specific rewards and transition dynamics. We introduce the cross-task compositional successor distance that connects task-specific representations through shared latent waypoints, and further learn a unified quasimetric space to capture reusable structures across tasks. CONQUEST enables effective task-agnostic policy extraction, facilitating the reuse of prior experience while mitigating forgetting as the task sequence expands. Our approach outperforms existing baselines on continuous-control tasks under both dense- and sparse-reward settings, as well as on image-based continual reinforcement learning benchmarks.

Primary area:
reinforcement learning
Keywords:
continual reinforcement learning, quasimetric, representation learning, knowledge transfer

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9EW4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes CONQUEST, a continual reinforcement learning framework that models knowledge transfer across tasks through a shared quasimetric representation. The key idea is to introduce cross-task compositional successor distance, where task-specific successor structures are connected through shared latent waypoints, and to learn this structure using task-aware contrastive learning and cross-task multistep quasimetric backups. A task-agnostic policy is then extracted from the learned representation to facilitate knowledge reuse and forward transfer.

Critical strengths
  1. The central idea is clear and reasonably memorable. Modeling cross-task transfer as compositional reachability through shared latent waypoints provides a coherent alternative to indiscriminate replay or parameter regularization. The asymmetric nature of the quasimetric is also well aligned with directional transfer between tasks.

  2. The experiments provide useful mechanistic evidence in addition to aggregate performance. Figure 3 shows that quasimetric-selected waypoint pairs lead to substantially higher cross-task path success than randomly selected compatible pairs, while Figure 4 demonstrates strong zero-shot transfer. The contrastive-learning ablation also shows a meaningful performance degradation when this component is removed.

Critical weaknesses
  1. Limited methodological distinction. CONQUEST combines established quasimetric representations, contrastive learning, replay, and policy distillation, while its cross-task distance composes task-local distances through intermediate waypoints. The paper does not sufficiently establish what new learning principle emerges from this combination. In particular, the claimed importance of asymmetric compositional geometry is not isolated through comparisons with simpler similarity-based transfer, symmetric distances, or matched consolidation without cross-task backups. Consequently, the performance improvements do not clearly establish the originality or necessity of the proposed geometric formulation.

  2. Unsupported cross-task reachability. The formulation allows tasks to have different transition dynamics, yet treats nearby latent waypoints as bridges for composing their trajectories. A quasimetric triangle inequality establishes a relationship between learned distances, but does not guarantee that the resulting path is executable under either task’s dynamics. The stitching experiment restores a source waypoint’s complete simulator state into the target environment before continuing. This demonstrates useful initialization after an externally imposed environment switch, but does not establish the stronger interpretation of dynamically valid cross-task trajectory composition.

  3. Unresolved task ambiguity. The framework assumes shared state and action spaces with potentially different dynamics and rewards, while its representations and extracted policies omit task identity. If tasks overlap in observable states, the same state and goal can require different actions under different dynamics. Goal conditioning alone does not resolve this ambiguity, and the dense-reward policy receives neither a goal nor task context. The paper needs explicit compatibility or observability assumptions to justify its task-agnostic formulation, particularly given its emphasis on substantial differences across tasks.

  4. Unverified selective transfer. The motivation emphasizes identifying useful experience and avoiding irrelevant transfer, but the adaptation objective randomly samples previously successful goals and regularizes the new policy toward their teacher actions. It does not explicitly assess their relevance to the current task. The empirical evidence also remains mixed: MetaWorld forward transfer is 0.015 ± 0.141, while Fetch’s 0.372 ± 0.140 is close to fine-tuning’s 0.367 ± 0.045. Controlled experiments with incompatible source tasks are needed to establish that the method selectively transfers beneficial knowledge and avoids negative transfer.

  5. Ineffective visual presentation. Figure 1 conveys little beyond three task regions connected through shared waypoints. It does not explain how the geometry is learned, evolves, or enables executable transfer, and could be replaced by a brief textual description. Meanwhile, the legends and annotations in Figures 2 and 4 are too small to read at normal viewing scale. These presentation choices weaken communication of both the central idea and the empirical evidence, particularly when interpreting cross-task relationships is essential to the paper’s contribution.

Questions for authors

See the weaknesses.

1 comment

We thank the reviewer for recognizing the compositional reachability perspective and the mechanistic evidence supporting CONQUEST. We have added experiments and clarified the scope of our claims.

1. Methodological distinction (CW1).

Our contribution is not quasimetric learning or replay in isolation, but learning and consolidating directional cross-task goal-reaching structures throughout continual RL.

Unlike methods that primarily preserve parameters or clone accumulated behaviors, CONQUEST combines task-aware backward contrastive learning with cross-task multistep backups to obtain a shared quasimetric representation, from which a task-agnostic policy is extracted.

We have added a symmetric-representation comparison (Fig. 13 and Fig.3) and matched diagonal-only MQE-backup experiments (Fig. 14). Together with the component ablations (Figs. 18–19), these results demonstrate the benefit of asymmetric geometry and broader state–goal backups.

2. Cross-task reachability and waypoint selection (CW2).

Our Fetch experiment evaluates cross-task trajectory stitching through waypoint-based continuation. We restore the selected source waypoint's simulator state in the target environment and execute the learned policy toward the target goal, directly measuring whether the selected bridge enables successful continuation.

In Fig. 3, quasimetric-based selection improves success over random compatible pairs by 12.1 and 10.0 percentage points across the two task orders, and over symmetric selection by 8.3 and 5.8 points.

These results demonstrate that directional quasimetric geometry helps identify more effective bridging waypoints for cross-task trajectory stitching, improving the likelihood of successful target-task continuation. While this does not establish seamless physical transitions across different dynamics, it provides empirical evidence for the waypoint-selection mechanism underlying cross-task trajectory composition.

3. Task ambiguity (CW3).

Our goal is a task-agnostic policy that does not require task identity at evaluation. Task-aware contrastive learning preserves distinguishable task-local successor relationships within the shared representation.

We acknowledge that exact task-agnostic control cannot be guaranteed when identical observations require incompatible actions. We have therefore introduced Assumption 1 and additional policy-regularity conditions in Theorem 2 to clarify the scope of our analysis.

4. Transfer effectiveness (CW4).

CONQUEST transfers knowledge through a quasimetric-derived teacher policy conditioned on historical successful goals, rather than explicitly filtering source tasks using a relevance threshold.

We further evaluate transfer under the challenging Sweep-to-Push setting (Fig. 15)[1] . In Fetch, historical-goal conditioning improves teacher coverage and achieves a higher reported Slide success rate than current-goal conditioning and fine-tuning (Fig. 16).

These findings support effective knowledge reuse and demonstrate that CONQUEST can mitigate negative transfer to some extent in the evaluated settings.

5. Presentation (CW5).

We have added a complete framework overview (Fig. 5) illustrating online adaptation, cross-task quasimetric consolidation, and task-agnostic policy extraction. We will also improve the main-paper figures and their readability.

We appreciate the reviewer's suggestions and believe the revised experiments and clarifications better establish the contribution and limitations of CONQUEST.

[1] Ahn et al., 2025. Prevalence of Negative Transfer in Continual Reinforcement Learning: Analyses and a Simple Baseline. ICLR 2025.

Reviewer_Khk5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

CONQUEST proposes a shared quasimetric representation that captures directed relationships between states, action and goals across a sequence of reinforcement learnig tasks. It aims to connect experience from different tasks through shared latent waypoints, retain that knowledge, and extract a combined policy to guide learning on subsequent tasks. Its claimed contribution is extending quasimetric learning to continual knowledge retention and transfer, supported by theoretical analyses and reported performance gains on Meta-World, Fetch, and MinAtar.

Critical strengths
  1. The research question and representation are well motivated, relevant, and potentially useful. Directed goal reachability and composition are natural concepts for selectively reusing RL experience. Bringing these ideas into a continual task stream through quasimetric learning is a plausible novel contribution.

  2. The empircal results cover multiple benchmarks. The paper includes dense rewards, sparse goal conditioned control tasks and image-based discrete-action tasks. Table 1 reports Meta-World average performance of 0.654 versus 0.625 for PackNet and 0.586 for FAME. Fetch average performance is 0.586 versus 0.467 for fine-tuning and 0.336 for FAME. CONQUEST also reports higher returns than baselines on all three MinAtar games. This breadth is a substantial strength and supports the usefulness of the evaluated system.

  3. The paper includes informative analyses beyond just aggregated benchmark scores. Removing the contrastive loss reduces Meta-World performance from 0.654 to 0.575 (Figure 9). The waypoint-coverage analysis controls for source and target effects, and the Fetch waypoint-selection experiment reports improvements of 15.8 and 11.2 percentage points with confidence intervals. These analyses provide evidence about useful learned structure in CONQUEST.

Critical weaknesses

Critical Weaknesses

  1. Equation 12 constructs a target using a source waypoint, a destination-task waypoint, their connecting distance and the destination goal. In Meta-World, the sampler in the code implementation isntead obtains a transition, immediate state and future goal within one episode, and the loss uses a single-intermediate-state backup. Mixing different tasks within a minibatch does not itself instantiate the two-waypoint target described in the paper. MinAtar likewise samples within episodes and uses a single intermediate state. In Fetch its default is a matched-sample backup, but an optional diag_backup <1 setting also backs up toward other goals in the batch, potentially including other tasks. That option still does not explicitly construct compatible two-waypoint pairs or the bridge-distance term of Equation 12.

  2. Algorithm 1 lists Equations 9 and 12 but does not explicitly state what happens in the first task, as neither of these equations are available for the firs taks. The paper does not specify how this is addressed in the text. The Meta-World code uses a local-learning fallback and also uses an action-invariance term, while MinAtar includes additional terms and uses all in-batch contrastive negatives without the task-aware masking described in Equation 9. These adaptations may be justified, but they mean the benchmark implementations differ beyond action-space handling. A mapping of objectives to each implementation and its coefficients is needed to accurately interpret the empirical evidence shown in the paper.

  3. The contrastive ablation is useful, but there is not matched ordinary-MQE + replay comparison or explicit waypoint-backup ablation with retained experience, policy extraction, next-task transfer and update budgest held fixed. Existing multistep quasimetric learning already studies trajectory stitching (e.g., Scaling Goal-conditioned Reinforcement Learning with Multistep Quasimetric Distances, ICLR 2026). The benchmark gains support the complete system more directly than they establish the added benefit of the proposed cross-task quasimetric representation.

  4. The theory supports a narrower conclusion than the narrative in the paper suggests. A short connection in representation space does not necessarily establish that behavior transfers under different environment dynamics. Two similar-looking situations can require different actions (e.g., adversarial tasks). The proofs introduce compatability and policy-regularity assumptions. These need to connected clearly to the actual method. This weakens the theoretical explanation.

Secondary Weaknesses 4. Figures 5-7 and 9 state ten seeds, whereas the README.md in the code explicitly describes four-seeds in the launch commands. Table 1 should define how the uncertainty is calculated and metric aggregation clearly. MinAtar forgetting is 1.045 for CONQUEST versus 1.007 for FAME, while Fetch forward transfer is close to fine-tuning (0.372 versus 0.367). Whether these results are statistically significant needs to be established.

Questions for authors

Clarification requests

  1. Which exact code revision and objective produced each reported table? Please clarify how Equation 12 is implemented in the code, explain its cross-task pair sampler, and specify the first-task/local-learning objectives and the additional action-invariance loss.

  2. Under what dynamics-compatbility and regularity assumptions do the theorems hold? How does Theorem 2 change with the behavior-cloning term and finite policy-optimization error?

  3. Does the gain remain against ordrinary MQE on the same retained replay, with identical policy extraction, transfer regularization, memory size and update budget? What changes when the cross-task backup is removed?

Suggestions for future work 4. Define the uncertainty estimate and metric aggregation used in the main results.

3 comments

We thank the reviewer for the detailed assessment. We have clarified the relationship between the proposed geometric formulation and its implementation and added controlled ablations.

1. Cross-task backup implementation (CW1, Q1).

CONQUEST maps experiences from different tasks into a shared quasimetric space. During consolidation, mixed-task minibatches enable cross-task state–goal pairing and learning through multistep backups.

We clarify that Eq. (10) describes conceptual waypoint-based composition, while the practical Eq. (12) uses the direct target

dθˉ(wi,gj)−ki′log⁡γ.d_{\bar\theta}(w_i,g_j)-k_i'\log\gamma.

The implementation therefore does not explicitly construct two-waypoint pairs during every update. Instead, cross-task reachability estimates are learned through repeated all-pairs backups over retained experiences. We have revised Eq. (12) and Algorithm 1 accordingly.

2. First-task learning and benchmark-specific objectives (CW2).

During the first task, the same quasimetric learning framework operates on task-local experiences, without requiring previous-task replay. Cross-task pairs become available as experiences accumulate.

For MinAtar, we use in-batch contrastive learning suited to its image-based observations and limited task diversity. The action-invariance regularizer and auxiliary objectives follow established methods [1] [2], rather than being new components of CONQUEST.

3. Matched MQE control and ablations (CW3, Q3).

The requested matched retained-replay MQE-backup comparison is included in Fig. 14. Specifically, ωdiag=1\omega_{\mathrm{diag}}=1 uses standard diagonal multistep MQE backups, while ωdiag=0\omega_{\mathrm{diag}}=0 enables all-pairs backups, including cross-task pairings. Both settings retain the same experience and remaining learning components.

The all-pairs variant consistently performs better across two Fetch and two Meta-World sequences. We also provide a contrastive-learning ablation (Figs. 17–18), a symmetric-representation control (Fig. 13 and Fig. 3), and a backup ablation (Fig. 19). These analyses support the complementary contributions of discriminative task-local representations and cross-sample quasimetric learning.

Unlike prior within-task MQE trajectory composition, CONQUEST investigates directional knowledge reuse across sequentially encountered tasks.

4. Dynamics compatibility and policy reuse (CW4, Q2).

Our objective is to learn a task-agnostic policy without exposing task identity during evaluation. Task-aware contrastive learning helps distinguish task-local successor structures, while the shared quasimetric supports knowledge integration.

We acknowledge that this does not guarantee optimal behavior for identical observations requiring incompatible actions. Assumption 1 clarifies successor compatibility, and Theorem 2 now explicitly incorporates policy regularity and the behavioral-cloning residual.

5. Statistical reporting.

We distinguish training seeds from evaluation seeds: the reported launch configuration uses four training seeds, while evaluation uses ten seeds. Table 1 reports mean ± standard deviation across 10 evaluation seeds, with four training seeds.

Fetch: Although FT is comparable to fine-tuning (0.372 vs. 0.367), CONQUEST achieves higher average performance (0.586 vs. 0.467). Fig. 16 further supports the effectiveness of historical-goal guidance in improving policy coverage and transfer.

MinAtar: CONQUEST achieves stronger FT (0.277 vs. 0.184) and average performance (17.231 vs. 13.471) than FAME, despite slightly higher forgetting (1.045 vs. 1.007). Stronger task acquisition may also lead to greater measured forgetting. Thus, these metrics should be considered jointly rather than in isolation.

We thank the reviewer for helping us improve the reproducibility and interpretation of the method.

[1]Myers et al., 2024. Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making. ICML 2024.

[2]Zheng et al., 2026. Scaling Goal-Conditioned Reinforcement Learning with Multistep Quasimetric Distances. ICLR 2026.

Thank you for the revisions and the new Fetch ablation. The disctinction between the conceptual two-waypoint backup and the direct target in Equation 12 clarifies my concern about explicit waypoint matching.

The updated code at https://anonymous.4open.science/r/continual-rl-anonymous-74C8/ appears to better match the formulation in the paper, however, the README suggests that Meta-World uses the within-episode approach. Could you confirm the backup setting used for each benchmark in Table 1 and for each condition in Figure 19, and clarify whether the reported cross-task loss include same-task pairs?

We thank the reviewer for the follow-up and the opportunity to clarify the backup configurations. The README does not explicitly specify the diag_backup settings, which we clarify below.

In the current manuscript, for the goal-conditioned Fetch benchmark, we use ωdiag=0\omega_{\mathrm{diag}}=0, enabling all-pairs backups to enhance compositional generalization across tasks and goals. For the dense-reward benchmarks Meta-World and MinAtar, we use ωdiag=1\omega_{\mathrm{diag}}=1, corresponding to diagonal-only multistep backups, while jointly optimizing the shared quasimetric representation using replay data from all previously encountered tasks. In these dense-reward settings, task-aware contrastive learning helps distinguish task-local successor structures and mitigate interference across tasks with heterogeneous dynamics. Shared latent waypoint relationships can emerge in the learned representation space after training, Fig. 2 further shows that directed waypoint coverage is positively correlated with cross-task transfer in Meta-World, supporting the presence of transferable relationships across tasks.

We additionally evaluated ωdiag=0\omega_{\mathrm{diag}}=0 on dense-reward tasks while keeping all other settings unchanged. Meta-World average performance improved from 0.654 ± 0.015 to 0.669 ± 0.027 across both task sequences. For MinAtar, the average task performance over three randomly selected sequences improved from 11.365 ± 1.898 to 11.750 ± 1.657. These supplementary results suggest that all-pairs backups can also provide additional benefits in dense-reward settings.

All conditions in Fig. 19 use ωdiag=0\omega_{\mathrm{diag}}=0. The all-pairs backup includes diagonal pairs, within-task off-diagonal pairs, and cross-task pairs.

Reviewer_tqjdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

CONQUEST maintains a shared quasimetric representation across a sequence of RL tasks, trained on retained experience with a task-aware contrastive loss and a cross-task multistep backup through "shared latent waypoints". The extracted policy serves as a KL teacher for new tasks. Experiments on Meta-World, Fetch and MinAtar report higher average performance, and two theorems bound cross-task distances and policy similarity.

Critical strengths
  1. An interesting consolidation perspective. Consolidating a shared reachability geometry rather than parameters or a single policy fits sparse-reward continual RL well.

  2. Consistent average-performance gains. CONQUEST achieves the best average performance on three quite different benchmarks (Table 1), and the waypoint-coverage analysis (Fig. 2) is a reasonable attempt to link the geometry to transfer.

Critical weaknesses
  1. The theory does not support the method in the stated setting. Tasks share states but differ in dynamics, and the encoders take no task input, yet Corollary 1 bounds a task-indexed dθmd^m_\theta while Theorem 1 applies this bound to the shared dθd_\theta (Eqs. 48–49). Eq. (10) →\to (11) also requires two-sided proximity: Definition 1 only gives dθ(wi,wj)≤ϵd_\theta(w_i, w_j) \le \epsilon, which does not bound d(wj,g)−d(wi,g)d(w_j, g) - d(w_i, g) for a directed distance. Theorem 2 likewise assumes a two-sided bound ∣cx(a)−cy(a)∣≤Ld dθ(x,y)|c_x(a) - c_y(a)| \le L_d \, d_\theta(x, y) in terms of a one-directional distance (Eq. 76), and its proof omits the BC term of Eq. (13).

  2. The source of the gains is unclear. The paper states that no explicit cross-task Bellman targets are needed (p. 4, l. 198–200), which is hard to reconcile with Eqs. (10)–(12), and the released code, with its default settings, uses within-episode or diagonal backups. Without a "same retained data, no quasimetric" control, the gains cannot be attributed to the geometry. Table 1 also gives mixed evidence for the retention claim on p. 7 (l. 366–367): on Meta-World, the parameter-regularization baseline EWC forgets less than CONQUEST (0.001 vs. 0.007), and on MinAtar CONQUEST forgets more than fine-tuning (1.045 vs. 0.948).

Questions for authors
  1. Is dθmd^m_\theta in Corollary 1 the shared network, and what additional conditions make Eqs. (11) and (76) valid?
  2. Where is Eq. (12) implemented in the code used for the results, and could the authors add a no-quasimetric control with matched retained data and update budget?
  3. Which policy is evaluated in Table 1, what does ±\pm denote, and could the authors report per-order results with significance tests? Suggestion: an overview figure of the per-task training loop (online learning with the KL teacher, experience retention, quasimetric consolidation, policy extraction) would make the method much easier to follow, and would also clarify where the cross-task waypoint pairs of Eq. (12) are constructed.

If a no-quasimetric control shows a clear gain for CONQUEST and the explicit cross-task backup is shown to be used and to matter, I would be willing to raise my score.

1 comment

We thank the reviewer for the constructive comments and recognition of CONQUEST's shared reachability-geometry perspective. We have revised the theoretical analysis and added controlled experiments.

1. Theoretical validity and directional consistency (CW1, Q1).

CONQUEST learns a single task-agnostic quasimetric shared across all tasks. The task-indexed notation dθmd_\theta^m in Corollary 1 denotes its restriction to task mm's state–goal distribution, rather than a separate network.

Our analysis first characterizes task-local successor consistency and then derives cross-task path relaxation through shared latent waypoints. Task-aware backward contrastive learning distinguishes successor structures induced by heterogeneous dynamics, mitigating interference in the shared representation. We additionally introduce Assumption 1 to clarify the required successor compatibility conditions.

We have revised Eq. (11) using the directional quasimetric triangle inequality, without requiring reverse proximity. We have also explicitly introduced local regularity assumptions and incorporated the behavioral-cloning residual into Theorem 2:

\leq L_\pi\epsilon+\Delta_\pi.$$ These revisions clarify the conditions under which our theoretical guarantees apply. **2. Implementation and source of improvement (CW2, Q2).** We have clarified that cross-task learning does not require collecting trajectories that physically transition between task environments. In the implementation, mixed-task minibatches enable cross-task state–goal backups when $\omega_{\mathrm{diag}}<1$. The revised Eq. (12) uses the implemented direct-distance target, while Eqs. (10)–(11) characterize its conceptual compositional geometry. We have added a symmetric-representation comparison **(Fig. 13 and Fig.3)**, a matched retained-replay diagonal-backup comparison **(Fig. 14)**, and component ablations **(Fig. 19)**. The diagonal-only setting $\omega_{\mathrm{diag}}=1$ serves as a matched ordinary MQE-backup control with contrastive learning, demonstrating the benefit of all-pairs updates while keeping the remaining components unchanged. **3. Evaluation protocol (Q3).** The reported uncertainty represents standard deviations across evaluation seeds. For FAME, the meta policy evaluates average performance and forgetting, while the fast policy evaluates forward transfer. Similarly, CONQUEST uses its extracted teacher for average performance and forgetting and its student for forward transfer. We acknowledge that forgetting should be interpreted jointly with task acquisition and forward transfer. For example, EWC's low Meta-World forgetting partly reflects limited initial task acquisition, while CONQUEST achieves substantially stronger average performance. On MinAtar, CONQUEST achieves higher forward transfer and average performance despite slightly higher forgetting. We hope these clarifications and controlled experiments address the reviewer's concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.