- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes CONQUEST, a continual reinforcement learning framework that models knowledge transfer across tasks through a shared quasimetric representation. The key idea is to introduce cross-task compositional successor distance, where task-specific successor structures are connected through shared latent waypoints, and to learn this structure using task-aware contrastive learning and cross-task multistep quasimetric backups. A task-agnostic policy is then extracted from the learned representation to facilitate knowledge reuse and forward transfer.
Critical strengths
-
The central idea is clear and reasonably memorable. Modeling cross-task transfer as compositional reachability through shared latent waypoints provides a coherent alternative to indiscriminate replay or parameter regularization. The asymmetric nature of the quasimetric is also well aligned with directional transfer between tasks.
-
The experiments provide useful mechanistic evidence in addition to aggregate performance. Figure 3 shows that quasimetric-selected waypoint pairs lead to substantially higher cross-task path success than randomly selected compatible pairs, while Figure 4 demonstrates strong zero-shot transfer. The contrastive-learning ablation also shows a meaningful performance degradation when this component is removed.
Critical weaknesses
-
Limited methodological distinction. CONQUEST combines established quasimetric representations, contrastive learning, replay, and policy distillation, while its cross-task distance composes task-local distances through intermediate waypoints. The paper does not sufficiently establish what new learning principle emerges from this combination. In particular, the claimed importance of asymmetric compositional geometry is not isolated through comparisons with simpler similarity-based transfer, symmetric distances, or matched consolidation without cross-task backups. Consequently, the performance improvements do not clearly establish the originality or necessity of the proposed geometric formulation.
-
Unsupported cross-task reachability. The formulation allows tasks to have different transition dynamics, yet treats nearby latent waypoints as bridges for composing their trajectories. A quasimetric triangle inequality establishes a relationship between learned distances, but does not guarantee that the resulting path is executable under either task’s dynamics. The stitching experiment restores a source waypoint’s complete simulator state into the target environment before continuing. This demonstrates useful initialization after an externally imposed environment switch, but does not establish the stronger interpretation of dynamically valid cross-task trajectory composition.
-
Unresolved task ambiguity. The framework assumes shared state and action spaces with potentially different dynamics and rewards, while its representations and extracted policies omit task identity. If tasks overlap in observable states, the same state and goal can require different actions under different dynamics. Goal conditioning alone does not resolve this ambiguity, and the dense-reward policy receives neither a goal nor task context. The paper needs explicit compatibility or observability assumptions to justify its task-agnostic formulation, particularly given its emphasis on substantial differences across tasks.
-
Unverified selective transfer. The motivation emphasizes identifying useful experience and avoiding irrelevant transfer, but the adaptation objective randomly samples previously successful goals and regularizes the new policy toward their teacher actions. It does not explicitly assess their relevance to the current task. The empirical evidence also remains mixed: MetaWorld forward transfer is 0.015 ± 0.141, while Fetch’s 0.372 ± 0.140 is close to fine-tuning’s 0.367 ± 0.045. Controlled experiments with incompatible source tasks are needed to establish that the method selectively transfers beneficial knowledge and avoids negative transfer.
-
Ineffective visual presentation. Figure 1 conveys little beyond three task regions connected through shared waypoints. It does not explain how the geometry is learned, evolves, or enables executable transfer, and could be replaced by a brief textual description. Meanwhile, the legends and annotations in Figures 2 and 4 are too small to read at normal viewing scale. These presentation choices weaken communication of both the central idea and the empirical evidence, particularly when interpreting cross-task relationships is essential to the paper’s contribution.
Questions for authors
See the weaknesses.
We thank the reviewer for recognizing the compositional reachability perspective and the mechanistic evidence supporting CONQUEST. We have added experiments and clarified the scope of our claims.
1. Methodological distinction (CW1).
Our contribution is not quasimetric learning or replay in isolation, but learning and consolidating directional cross-task goal-reaching structures throughout continual RL.
Unlike methods that primarily preserve parameters or clone accumulated behaviors, CONQUEST combines task-aware backward contrastive learning with cross-task multistep backups to obtain a shared quasimetric representation, from which a task-agnostic policy is extracted.
We have added a symmetric-representation comparison (Fig. 13 and Fig.3) and matched diagonal-only MQE-backup experiments (Fig. 14). Together with the component ablations (Figs. 18–19), these results demonstrate the benefit of asymmetric geometry and broader state–goal backups.
2. Cross-task reachability and waypoint selection (CW2).
Our Fetch experiment evaluates cross-task trajectory stitching through waypoint-based continuation. We restore the selected source waypoint's simulator state in the target environment and execute the learned policy toward the target goal, directly measuring whether the selected bridge enables successful continuation.
In Fig. 3, quasimetric-based selection improves success over random compatible pairs by 12.1 and 10.0 percentage points across the two task orders, and over symmetric selection by 8.3 and 5.8 points.
These results demonstrate that directional quasimetric geometry helps identify more effective bridging waypoints for cross-task trajectory stitching, improving the likelihood of successful target-task continuation. While this does not establish seamless physical transitions across different dynamics, it provides empirical evidence for the waypoint-selection mechanism underlying cross-task trajectory composition.
3. Task ambiguity (CW3).
Our goal is a task-agnostic policy that does not require task identity at evaluation. Task-aware contrastive learning preserves distinguishable task-local successor relationships within the shared representation.
We acknowledge that exact task-agnostic control cannot be guaranteed when identical observations require incompatible actions. We have therefore introduced Assumption 1 and additional policy-regularity conditions in Theorem 2 to clarify the scope of our analysis.
4. Transfer effectiveness (CW4).
CONQUEST transfers knowledge through a quasimetric-derived teacher policy conditioned on historical successful goals, rather than explicitly filtering source tasks using a relevance threshold.
We further evaluate transfer under the challenging Sweep-to-Push setting (Fig. 15)[1] . In Fetch, historical-goal conditioning improves teacher coverage and achieves a higher reported Slide success rate than current-goal conditioning and fine-tuning (Fig. 16).
These findings support effective knowledge reuse and demonstrate that CONQUEST can mitigate negative transfer to some extent in the evaluated settings.
5. Presentation (CW5).
We have added a complete framework overview (Fig. 5) illustrating online adaptation, cross-task quasimetric consolidation, and task-agnostic policy extraction. We will also improve the main-paper figures and their readability.
We appreciate the reviewer's suggestions and believe the revised experiments and clarifications better establish the contribution and limitations of CONQUEST.
[1] Ahn et al., 2025. Prevalence of Negative Transfer in Continual Reinforcement Learning: Analyses and a Simple Baseline. ICLR 2025.