acceptodds
Under review as a conference paper at ICLR 2027

Cross-Trajectory Value Composition for Offline Goal-Conditioned Reinforcement Learning

Abstract

Offline goal-conditioned reinforcement learning (GCRL) learns goal-reaching policies from fixed datasets. In long-horizon tasks, the data may contain useful parts of a solution across different trajectories without containing a complete path from the start to the goal. Standard temporal-difference learning propagates value information through local one-step backups, so supervision from a distant goal must pass through many updates, making long-horizon values slow and difficult to learn. Existing structured methods address this limitation but often require specialized value models or compose only segments from the same trajectory. We propose Cross-Trajectory Value Composition (CVC), which samples intermediate waypoints across trajectories and uses a discounted triangle relation to combine two shorter goal-reaching problems. This divide-and-conquer approach propagates value information over long distances without imposing a specialized structure on the value function. We incorporate CVC into a hierarchical offline GCRL framework and evaluate it on 24 long-horizon OGBench datasets. CVC achieves the best or tied-best performance on 20 tasks, learns substantially faster than prior methods, and performs especially well on stitching and noisy datasets. These results show that cross-trajectory composition can efficiently learn long-horizon behaviors from fragmented offline experience.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, goal-conditioned reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_7827OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Cross-Trajectory Value Composition (CVC), an auxiliary value-learning objective for offline goal-conditioned RL that composes two goal-reaching value estimates via a waypoint sampled independently from the dataset (not restricted to the same trajectory). The method uses a discounted “triangle” relation to form composed targets, filters candidate waypoints when each subproblem appears strictly easier than the direct problem, and stabilizes training with target-clipped expectile regression and a demonstrated-return lower bound for the one-step TD objective. Integrated into HIQL’s hierarchical framework, CVC achieves strong results on 24 state-based OGBench datasets, particularly on long-horizon stitching and exploration settings.

Critical strengths
  1. Cross-trajectory composition is a practical way to propagate long-horizon value in offline goal-conditioned RL. For the sparse reachability reward, the discounted operator correctly combines two path returns, and the method uses an ordinary neural value function rather than a quasimetric architecture. The same-trajectory ablation supports the central design choice: on pointmaze-giant-stitch, success falls from 98% to 20% when the waypoint must lie on the same trajectory.

  2. The empirical gains are large where stitching matters. Across 24 OGBench datasets, CVC reaches an average success rate of 88.6%, compared with 81.7% for LAVL, 72.8% for OTA, and 45.2% for HIQL. The clearest improvements are on giant-stitch and explore, and on manipulation: cube-double 63% versus 22% for LAVL, and Scene 84% versus 35% for OTA.

Critical weaknesses
  1. The gains are not isolated to the proposed composition loss. Appendix B also changes the value parameterization, uses relative latent subgoals, normalizes the high-level advantage, and adds embedding noise, while the objective further mixes in target clipping and a demonstrated-return lower bound. Figure 4 covers only about five tasks. The waypoint filter MM and the negative residual bound c△−c^-_\triangle are never removed on their own, and the return lower bound barely changes four of the five tested datasets. It is therefore unclear how much of Table 1 comes from cross-trajectory composition itself.

  2. The comparison overstates the result. Training length is chosen separately for every method and dataset from preliminary runs scored on the evaluation tasks, and the reported score averages checkpoints at 80%, 90%, and 100% of that length. CVC is often stopped at 50k--300k steps, while HIQL often runs to 1M. TRL and quasimetric methods, which the paper itself treats as the closest prior work, are absent from the experiments. The in-trajectory ablation is not a TRL baseline.

Questions for authors
  1. Please compare with TRL and at least one quasimetric method such as QRL or CMD, using the authors' official implementations where possible. The in-trajectory variant keeps CVC's filter, clipped expectile loss, and network, so it does not stand in for TRL. Please also list the baseline hyperparameters and confirm that their tuning budget matches CVC's.

  2. Please report success at fixed budgets, such as 500k and 1M steps, using the final checkpoint. How many seeds were used in the preliminary runs that selected the training length, and were those runs scored on the test tasks? Several numerical wins are within one standard deviation; please state which of the ``20 best or tied-best'' results remain under a fixed budget.

  3. On a single architecture, please compare one-step GCIVL, in-trajectory composition, and full CVC, and ablate MM and the negative residual bound c△−c^-_\triangle separately. With the default c△−=0c^-_\triangle=0, composition supplies no downward updates. Does this optimistic update inflate values for disconnected state--goal pairs, and do the learned values approximately obey the composition inequality?

  4. What fraction of sampled (s,w,g)(s,w,g) triples pass the filter, and do the retained waypoints lie near the start, near the goal, or on a genuine midpoint? Two easier subproblems can still form a detour. The demonstrated-return bound matters on only one of five datasets; please say when each stabilizer is necessary.

  5. The composition identity assumes deterministic dynamics, and γ=0.999\gamma=0.999 puts returns on a scale of hundreds while c△−c^-_\triangle is only 00, −2-2, or −5-5. How should these bounds be read in return units, and what happens if the transition is stochastic or the waypoint lies in another connected component?

  6. cube-triple remains at 30%. Are the failures mainly from value composition, subgoal selection, or low-level control?

1 comment
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for the careful reading and useful comments.

W2, Q1: TRL and quasimetric comparisons

We evaluated official TRL [1] with distillation and task-family hyperparameters using full-state goals. An oracle goal provides task features: for AntMaze, target (x,y)(x,y); full-state goals also include joint positions and velocities [2]. TRL accepts both. Table E.1 reports:

TaskTRLCVC
pointmaze-large-navigate34 ± 799 ± 1
pointmaze-giant-stitch0 ± 098 ± 1
antmaze-large-explore0 ± 098 ± 2
humanoidmaze-giant-stitch0 ± 190 ± 2
cube-single-noisy26 ± 299 ± 1
scene-noisy6 ± 384 ± 4

PointMaze Large Navigate uses the same state and oracle coordinates; TRL reaches 34%, versus 33% in [1].

LAVL outperforms QRL [3] on all 22 datasets in its comparison [4]. Table E.2 adds published QRL, CMD, and TMD results [6]; TMD reimplemented CMD on OGBench, while CMD originally evaluated D4RL and synthetic navigation [5]. With a quasimetric, CVC loss was zero and performance unchanged (Appendix E.2).

W2, Q2: Training-length selection

We select training lengths using preliminary runs with three random seeds per method and dataset, then fix them for final runs. Section 5.1 and Appendix C.3 describe the shared procedure; Appendix C.1–C.2 gives hyperparameters. The benchmark and composition comparison cover 24 datasets; other analyses use selected tasks.

W1, Q3: Composition and reliability

Figure 4 reports mean success across 24 tasks: 73% without composition, 78% with in-trajectory composition, and 89% with CVC. No composition sets λ△=0\lambda_\triangle=0 while retaining other CVC components and settings. In-trajectory composition draws all three states from one trajectory.

We separately ablated mask MM and composition clipping while keeping the demonstrated-return bound and TD clipping. No mask keeps the composition bounds; no clipping keeps the mask and sets c△−=−∞c^-_\triangle=-\infty. The table reports Figure 4’s five leftmost panels:

TaskCVCNo maskNo composition clipping
pointmaze-giant-stitch98 ± 199 ± 116 ± 10
antmaze-large-explore98 ± 292 ± 530 ± 22
humanoidmaze-giant-stitch90 ± 292 ± 188 ± 1
cube-single-noisy99 ± 199 ± 131 ± 5
scene-noisy84 ± 480 ± 63 ± 2

Removing composition clipping lowers success on four tasks; mask effects are task dependent. Figure 4 also ablates the demonstrated-return bound, which benefits Scene most.

We blocked a bottleneck in pointmaze-large-navigate and split trajectories (Appendix D.1). The true unreachable value is −1000; CVC predicts −556.70 ± 15.97 for unreachable pairs and −94.59 ± 0.58 for connected pairs. Policies succeed on 100% of tested connected tasks and 0% of disconnected tasks. Figure D.1 shows trajectories; Figure D.2 shows values decreasing after adding the wall and continuing training with normalization fixed.

In the modified maze, the violation rate is 2% and mean positive residual 0.029. Figure D.4 compares the original maze: 4% / 0.066 with CVC versus 11% / 0.711 without CVC. A violation occurs when the composed value exceeds the direct value by more than 0.001.

Appendix D.5 gives a one-step bound. If an unreachable subproblem is estimated as B+δB+\delta, where B=−1/(1−γ)B=-1/(1-\gamma), its composed target satisfies T−B=δp2≤δT-B=\delta p_2\leq\delta, with p2∈[0,1]p_2\in[0,1]. An optimistic subproblem estimate can still raise a more pessimistic direct prediction.

W1, Q4 and Q5: Waypoints and clipping

For connected pairs, the mask rejects sampled waypoints in the other region, but 45% of retained waypoints form cell-graph detours. Their measured residuals are negative and contribute zero CVC loss at c△−=0c^-_\triangle=0 (Appendix D.3, Figure D.3). At γ=0.999\gamma=0.999, values lie in [−1000,0][-1000,0]; bounds of −2 or −5 limit downward target deviations. Figure 4 separately tests removing the composition-residual bound.

Q6: Cube Triple

cube-triple-noisy requires generalizing to unseen combinations of three block positions and coordinating multiple pick-and-place operations [2]. Section 6 discusses entity-centric goal decomposition as a possible direction.

References

[1] Park et al., Transitive RL, ICLR 2026; code.

[2] Park et al., OGBench, ICLR 2025.

[3] Wang et al., Quasimetric Learning, ICML 2023.

[4] Kang et al., LAVL, ICML 2026.

[5] Myers et al., Learning Temporal Distances, ICML 2024.

[6] Myers et al., TMD, NeurIPS 2025.

Reviewer_Du3hOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes an algorithm called Cross-Trajectory Value Composition (CVC) that samples intermediate waypoints across trajectories to facilitate trajectory stitching. CVC adopts a triangle-relation based cross-trajectory TD targets and introduce multiple clipping techniques to stabilise learning.

Critical strengths
  1. The authors came up with a novel triangle-relation based TD target that can be used as cross-trajectory objectives.
  2. CVC outperforms existing hierarchical offline RL algorithms by a noticeable margin on various offline RL datasets.
Critical weaknesses
  1. The algorithm assumes that the environmental dynamics are deterministic, which might night be true in some cases.

  2. The ablation for dropping (s,w,g)(s, w, g) tuples when one of the subproblems have lower target value than the direct problem is missing.

Questions for authors
  1. In the last sentence of the first page, it is unclear why "[t]hese approaches motivate [the] complementary question." Overall, the introduction section, particularly the part that gives an overview of CVC, requires some effort to understand.
  2. It would be interesting to see how CVC will perform when some goals are unreachable from certain states.
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for recognizing the value-composition idea and its empirical performance. We have added the missing waypoint-filter ablation in Figure 4, clarified the introduction, and reported a disconnected-goal study in Appendix D.

W1 — Deterministic dynamics

The return-composition identity assumes deterministic dynamics, as stated in Section 4.1. Extending the analysis to stochastic dynamics is a direction for future work.

W2 — Missing ablation of the tuple filter

We have evaluated a variant that removes the tuple filter, implemented as the waypoint mask M(s,w,g)M(s,w,g), and retains every sampled tuple while keeping composition-residual clipping, the demonstrated-return TD bound, and TD target clipping. The table reports the numerical results shown in the five leftmost panels of Figure 4:

TaskFull CVCWithout waypoint mask
pointmaze-giant-stitch98 ± 199 ± 1
antmaze-large-explore98 ± 292 ± 5
humanoidmaze-giant-stitch90 ± 292 ± 1
cube-single-noisy99 ± 199 ± 1
scene-noisy84 ± 480 ± 6

The filter's effect is task dependent: removing it lowers the AntMaze and Scene means, increases the HumanoidMaze mean, and leaves PointMaze and Cube near ceiling. Section 5.4 discusses these results alongside the separate composition-residual clipping ablation.

Q1 — Introduction clarity

We have revised the introduction to explain how independently sampled waypoints broaden composition beyond observed in-trajectory connections, and how filtering and residual clipping control unreliable compositions. The performance benchmark and Figure 4's all-task composition comparison cover 24 datasets; the stabilization, composition-weight, and unreachable-goal analyses use selected tasks.

Q2 — Unreachable goals

We have added a wall at a bottleneck in pointmaze-large-navigate [1], creating two physically disconnected regions. We removed blocked states and transitions and split the original trajectories at the cuts. Appendix D.1 describes the setup, and Figure D.1 shows actual policy trajectories over value heatmaps for three start–goal pairs.

CVC predicts −556.70 ± 15.97 for unreachable pairs, compared with their true infinite-horizon value of −1000 and connected reference estimates of −94.59 ± 0.58. Policies achieve 100% success on each tested connected task and 0% on each disconnected task. Across 4,096 sampled pairs of each type per wall-trained critic, the minimum reachable value is −312 and the maximum sampled unreachable value is −417. In the original maze without the added wall, the minimum value over 100,000 sampled pairs is −308. The sampled reachable and unreachable values remain separated despite the overestimation.

Figure D.2 shows the adaptation after introducing the wall. We have pretrained CVC on the original maze and continued training on the cut trajectories, keeping observation normalization fixed. Values progressively decrease in the newly disconnected region while remaining high in the goal's region.

Appendix D.3 shows that the original mask rejects waypoints in the other region for the two connected endpoint pairs, while clipping suppresses the negative residuals of retained detours.

References

[1] S. Park, K. Frans, B. Eysenbach, and S. Levine. OGBench: Benchmarking Offline Goal-Conditioned RL. ICLR, 2025.

My main concerns have been resolved. I will raise my score to 4. I do recommend the authors to discuss the newly added ablations in greater detail. For instance, it appears "composition-residual clipping" refers to ℓclipτΔ\ell_{\rm clip}^{\tau_\Delta} in the CVC loss, but the connection is never explicitly stated.

Thank you for your positive assessment and for the helpful suggestion. We have expanded the ablation discussion in Section 5.4 to make the role of composition-residual clipping explicit.

Reviewer_HLLKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes Cross-Trajectory Value Composition (CVC) for long-horizon offline goal-conditioned RL.

The main idea is to estimate the value of reaching a goal by combining the values of two shorter problems through an intermediate waypoint. Unlike previous methods that select waypoints from the same trajectory, CVC samples the start state, waypoint, and goal independently from the dataset. This allows value information to be shared across different trajectories.

Since arbitrary waypoints may lead to poor routes, the authors introduce a value-based filter to select useful compositions. They also use target clipping and demonstrated trajectory returns to stabilize value learning. The method is integrated into HIQL and evaluated on 24 OGBench datasets, showing strong performance, especially on stitching and manipulation tasks.

Critical strengths
  1. A simple and useful approach to learning from different trajectories. In offline RL, useful parts of a solution may appear in different trajectories, even when no complete solution is demonstrated. CVC provides a straightforward way to combine value information through intermediate states. It can also be added to existing value-learning methods without requiring a special network architecture. This makes the approach practical and potentially useful beyond HIQL.

  2. Strong experimental results. CVC achieves the highest average success rate on 24 OGBench datasets (88.6%, compared with 81.7% for LAVL). The improvements are particularly strong on stitching and manipulation tasks. The ablation comparing cross-trajectory and in-trajectory waypoint sampling also shows substantial improvements, providing evidence that using waypoints from different trajectories is beneficial.

Critical weaknesses
  1. The advantage over existing quasimetric and value-composition methods is unclear. The main idea of CVC is based on a discounted triangle inequality. After transforming the goal-reaching values into distances, this is equivalent to the triangle inequality used in quasimetric learning. This raises a natural question: if we use a quasimetric network that already satisfies the triangle inequality, do we still need the CVC loss? The loss may help an unconstrained value network learn this property, but it is unclear whether it provides additional benefits when the property is already guaranteed by the network. The paper also discusses TRL, which similarly uses intermediate states to compose values, but does not compare against it directly. Although the authors include an in-trajectory sampling ablation, this is not equivalent to evaluating the full TRL method. These missing comparisons make it difficult to understand how much CVC improves upon closely related approaches.

  2. Cross-trajectory composition may produce incorrect targets when the learned values are inaccurate. The proposed triangle inequality provides a valid lower bound when the values of the two shorter problems are correct. However, CVC uses learned value estimates, which may be inaccurate, especially for state-goal pairs that are not well supported by the offline dataset. For example, the method may select a waypoint that appears easy to reach according to the learned value function, even though it is actually unreachable. If both shorter problems have overestimated values, combining them can produce an overly optimistic target for the original problem. The authors introduce a waypoint filter, but it only checks whether the two shorter problems have higher estimated values than the original problem. It does not verify whether the waypoint is reachable or whether the composed value is accurate. Although the experiments show strong results, the paper does not sufficiently investigate this issue. In particular, it is unclear how well the method works when the dataset has poor connectivity or contains unreachable state-goal pairs.

Questions for authors
  1. Relationship to quasimetric learning: If the value function already satisfies the triangle inequality through a quasimetric architecture, does the CVC loss still improve performance? Could the authors compare quasimetric learning with and without CVC? This would help clarify whether the proposed loss provides benefits beyond enforcing the triangle inequality.

  2. Comparison with existing methods: Could the authors provide a direct comparison with TRL under the same experimental setting? Also, Figure 4 only tests positive composition weights. What happens when the CVC weight is set to zero while keeping the same TD stabilization, network architecture, and policy training? This would help isolate the contribution of the composition loss itself.

  3. Reliability of cross-trajectory composition: How does CVC handle inaccurate value estimates for state-goal pairs that are poorly covered by the dataset? In particular, could the authors evaluate the method in environments with unreachable goals or disconnected regions? This would help determine whether incorrect compositions can introduce and propagate value-estimation errors.

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for the comments. We added the direct TRL comparison in Appendix E and the disconnected-goal study in Appendix D.

W1, Q1: Quasimetric learning

For a quasimetric model, the CVC loss remained zero and adding CVC did not change performance, consistent with its triangle-inequality constraint [3] (Appendix E.2). With our value network on pointmaze-large-navigate, Figure D.4 shows a 4% violation rate and mean positive residual 0.066 with CVC, versus 11% and 0.711 without CVC.

W1, Q2: TRL and composition control

We evaluated official TRL [1] with its distillation branch, networks, sampling, and task-family hyperparameters, using full-state goals. Table E.1 reports:

TaskTRLCVC
pointmaze-large-navigate34 ± 799 ± 1
pointmaze-giant-stitch0 ± 098 ± 1
antmaze-large-explore0 ± 098 ± 2
humanoidmaze-giant-stitch0 ± 190 ± 2
cube-single-noisy26 ± 299 ± 1
scene-noisy6 ± 384 ± 4

TRL reports OGBench results with oracle goals [1]: hand-specified task-relevant features. For example, an AntMaze oracle goal contains target (x,y)(x,y), whereas a full-state goal also includes joint configuration and velocities [2]. TRL's architecture also accepts full-state goals. On PointMaze Large Navigate, where the two goal representations use the same coordinates, our TRL reproduction reaches 34%, close to the published 33% [1].

Figure 4 reports mean success across all 24 tasks: 73% without composition, 78% with in-trajectory composition, and 89% with CVC. The no-composition variant sets λ△=0\lambda_\triangle=0 while retaining the other CVC components and training settings. In-trajectory composition samples all three states from the same trajectory.

W2, Q3: Unreliable compositions and unreachable goals

We added a wall to pointmaze-large-navigate, removed blocked states and transitions, and split trajectories at the wall. Figure D.1 shows policy trajectories for unreachable goals. At γ=0.999\gamma=0.999, the true unreachable value is −1000; CVC predicts −556.70 ± 15.97 for unreachable pairs and −94.59 ± 0.58 for connected pairs. Policies succeed on 100% of tested connected tasks and 0% of disconnected tasks. Over 4,096 sampled pairs of each type per wall-trained critic, the minimum reachable value is −312 and maximum unreachable value is −417. In the original maze, the minimum over 100,000 sampled pairs is −308.

After pretraining on the original maze, we continued training with the wall while keeping observation normalization fixed. Figure D.2 shows values falling in the newly disconnected region while remaining high in the goal's region.

Figure D.3 shows that the mask rejects sampled waypoints in the other region for connected pairs, but 45% of retained waypoints form cell-graph detours. Their measured residuals are negative and contribute zero CVC loss at c△−=0c^-_\triangle=0.

Appendix D.5 bounds one-step composed-target error: if an unreachable subproblem has value estimate B+δB+\delta, where B=−1/(1−γ)B=-1/(1-\gamma), then T−B=δp2≤δT-B=\delta p_2\leq\delta for the other subproblem's normalized value p2∈[0,1]p_2\in[0,1]. This target can still raise a more pessimistic direct prediction.

References

[1] Park et al., Transitive RL, ICLR 2026; code.

[2] Park et al., OGBench, ICLR 2025; code.

[3] Wang et al., Quasimetric Learning, ICML 2023.

Thank you for the detailed rebuttal and additional experiments. The direct TRL comparison, controlled composition ablations, and disconnected-maze experiments largely address my original concerns.

One remaining question is why CVC substantially outperforms existing quasimetric methods, given that both rely on the same triangle relation. Is the advantage mainly due to a more flexible value parameterization, the learning objective, or selective value propagation through clipping? A brief discussion would help clarify the contribution.

Overall, I appreciate the additional evidence and consider my main concerns largely addressed.

Thank you for your thoughtful question. We believe both the value architecture and the learning objective may contribute to the difference.

LAVL [2] provides a controlled comparison of value architectures within the same hierarchical framework. IQE [3] and MRN [4] are specialized quasimetric architectures designed to satisfy the triangle inequality. For pointmaze-giant-stitch-v0, Table 11 reports 91% success for LAN, compared with 39% for IQE and 20% for MRN. The variants use the same IVL-based value objective and continuity-regularization weight. This suggests that the value architecture can substantially affect performance.

LAVL’s objective ablations in Figure 2 and Section 4.2 also suggest that QRL’s learning objective affects performance [1, 2]. Together, these findings indicate that both architecture and objective can influence the difference between methods.

References

[1] Wang et al., Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning, ICML 2023.

[2] Kang et al., Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning, ICML 2026.

[3] Wang and Isola, Improved Representation of Asymmetrical Distances with Interval Quasimetric Embeddings, 2022.

[4] Liu et al., Metric Residual Networks for Sample Efficient Goal-Conditioned Reinforcement Learning, AAAI 2023.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.