- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Cross-Trajectory Value Composition (CVC), an auxiliary value-learning objective for offline goal-conditioned RL that composes two goal-reaching value estimates via a waypoint sampled independently from the dataset (not restricted to the same trajectory). The method uses a discounted “triangle” relation to form composed targets, filters candidate waypoints when each subproblem appears strictly easier than the direct problem, and stabilizes training with target-clipped expectile regression and a demonstrated-return lower bound for the one-step TD objective. Integrated into HIQL’s hierarchical framework, CVC achieves strong results on 24 state-based OGBench datasets, particularly on long-horizon stitching and exploration settings.
Critical strengths
-
Cross-trajectory composition is a practical way to propagate long-horizon value in offline goal-conditioned RL. For the sparse reachability reward, the discounted operator correctly combines two path returns, and the method uses an ordinary neural value function rather than a quasimetric architecture. The same-trajectory ablation supports the central design choice: on pointmaze-giant-stitch, success falls from 98% to 20% when the waypoint must lie on the same trajectory.
-
The empirical gains are large where stitching matters. Across 24 OGBench datasets, CVC reaches an average success rate of 88.6%, compared with 81.7% for LAVL, 72.8% for OTA, and 45.2% for HIQL. The clearest improvements are on giant-stitch and explore, and on manipulation: cube-double 63% versus 22% for LAVL, and Scene 84% versus 35% for OTA.
Critical weaknesses
-
The gains are not isolated to the proposed composition loss. Appendix B also changes the value parameterization, uses relative latent subgoals, normalizes the high-level advantage, and adds embedding noise, while the objective further mixes in target clipping and a demonstrated-return lower bound. Figure 4 covers only about five tasks. The waypoint filter and the negative residual bound are never removed on their own, and the return lower bound barely changes four of the five tested datasets. It is therefore unclear how much of Table 1 comes from cross-trajectory composition itself.
-
The comparison overstates the result. Training length is chosen separately for every method and dataset from preliminary runs scored on the evaluation tasks, and the reported score averages checkpoints at 80%, 90%, and 100% of that length. CVC is often stopped at 50k--300k steps, while HIQL often runs to 1M. TRL and quasimetric methods, which the paper itself treats as the closest prior work, are absent from the experiments. The in-trajectory ablation is not a TRL baseline.
Questions for authors
-
Please compare with TRL and at least one quasimetric method such as QRL or CMD, using the authors' official implementations where possible. The in-trajectory variant keeps CVC's filter, clipped expectile loss, and network, so it does not stand in for TRL. Please also list the baseline hyperparameters and confirm that their tuning budget matches CVC's.
-
Please report success at fixed budgets, such as 500k and 1M steps, using the final checkpoint. How many seeds were used in the preliminary runs that selected the training length, and were those runs scored on the test tasks? Several numerical wins are within one standard deviation; please state which of the ``20 best or tied-best'' results remain under a fixed budget.
-
On a single architecture, please compare one-step GCIVL, in-trajectory composition, and full CVC, and ablate and the negative residual bound separately. With the default , composition supplies no downward updates. Does this optimistic update inflate values for disconnected state--goal pairs, and do the learned values approximately obey the composition inequality?
-
What fraction of sampled triples pass the filter, and do the retained waypoints lie near the start, near the goal, or on a genuine midpoint? Two easier subproblems can still form a detour. The demonstrated-return bound matters on only one of five datasets; please say when each stabilizer is necessary.
-
The composition identity assumes deterministic dynamics, and puts returns on a scale of hundreds while is only , , or . How should these bounds be read in return units, and what happens if the transition is stochastic or the waypoint lies in another connected component?
-
cube-triple remains at 30%. Are the failures mainly from value composition, subgoal selection, or low-level control?
Rebuttal by Authors
We thank the reviewer for the careful reading and useful comments.
W2, Q1: TRL and quasimetric comparisons
We evaluated official TRL [1] with distillation and task-family hyperparameters using full-state goals. An oracle goal provides task features: for AntMaze, target (x,y); full-state goals also include joint positions and velocities [2]. TRL accepts both. Table E.1 reports:
pointmaze-large-navigatepointmaze-giant-stitchantmaze-large-explorehumanoidmaze-giant-stitchcube-single-noisyscene-noisyPointMaze Large Navigate uses the same state and oracle coordinates; TRL reaches 34%, versus 33% in [1].
LAVL outperforms QRL [3] on all 22 datasets in its comparison [4]. Table E.2 adds published QRL, CMD, and TMD results [6]; TMD reimplemented CMD on OGBench, while CMD originally evaluated D4RL and synthetic navigation [5]. With a quasimetric, CVC loss was zero and performance unchanged (Appendix E.2).
W2, Q2: Training-length selection
We select training lengths using preliminary runs with three random seeds per method and dataset, then fix them for final runs. Section 5.1 and Appendix C.3 describe the shared procedure; Appendix C.1–C.2 gives hyperparameters. The benchmark and composition comparison cover 24 datasets; other analyses use selected tasks.
W1, Q3: Composition and reliability
Figure 4 reports mean success across 24 tasks: 73% without composition, 78% with in-trajectory composition, and 89% with CVC. No composition sets λ△=0 while retaining other CVC components and settings. In-trajectory composition draws all three states from one trajectory.
We separately ablated mask M and composition clipping while keeping the demonstrated-return bound and TD clipping. No mask keeps the composition bounds; no clipping keeps the mask and sets c△−=−∞. The table reports Figure 4’s five leftmost panels:
pointmaze-giant-stitchantmaze-large-explorehumanoidmaze-giant-stitchcube-single-noisyscene-noisyRemoving composition clipping lowers success on four tasks; mask effects are task dependent. Figure 4 also ablates the demonstrated-return bound, which benefits Scene most.
We blocked a bottleneck in
pointmaze-large-navigateand split trajectories (Appendix D.1). The true unreachable value is −1000; CVC predicts −556.70 ± 15.97 for unreachable pairs and −94.59 ± 0.58 for connected pairs. Policies succeed on 100% of tested connected tasks and 0% of disconnected tasks. Figure D.1 shows trajectories; Figure D.2 shows values decreasing after adding the wall and continuing training with normalization fixed.In the modified maze, the violation rate is 2% and mean positive residual 0.029. Figure D.4 compares the original maze: 4% / 0.066 with CVC versus 11% / 0.711 without CVC. A violation occurs when the composed value exceeds the direct value by more than 0.001.
Appendix D.5 gives a one-step bound. If an unreachable subproblem is estimated as B+δ, where B=−1/(1−γ), its composed target satisfies T−B=δp2≤δ, with p2∈[0,1]. An optimistic subproblem estimate can still raise a more pessimistic direct prediction.
W1, Q4 and Q5: Waypoints and clipping
For connected pairs, the mask rejects sampled waypoints in the other region, but 45% of retained waypoints form cell-graph detours. Their measured residuals are negative and contribute zero CVC loss at c△−=0 (Appendix D.3, Figure D.3). At γ=0.999, values lie in [−1000,0]; bounds of −2 or −5 limit downward target deviations. Figure 4 separately tests removing the composition-residual bound.
Q6: Cube Triple
cube-triple-noisyrequires generalizing to unseen combinations of three block positions and coordinating multiple pick-and-place operations [2]. Section 6 discusses entity-centric goal decomposition as a possible direction.References
[1] Park et al., Transitive RL, ICLR 2026; code.
[2] Park et al., OGBench, ICLR 2025.
[3] Wang et al., Quasimetric Learning, ICML 2023.
[4] Kang et al., LAVL, ICML 2026.
[5] Myers et al., Learning Temporal Distances, ICML 2024.
[6] Myers et al., TMD, NeurIPS 2025.