- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces DCRL (Divide-and-Conquer RL) to mitigate the accumulation of value estimation errors in long-horizon offline goal-conditioned reinforcement learning. DCRL recursively decomposes trajectory segments into a balanced binary tree and learns values in a child-to-parent (or leaf-to-root) order, ensuring that long-range estimates are grounded as much as possible in previously learned shorter-range estimates. For demonstrated trajectories, DCRL employs a selection-free exact factorization method using fixed midpoints to learn route-specific behavior values, thereby avoiding the optimistic error amplification associated with max-based intermediate-state selection. Since this objective only recovers values for demonstrated routes, the authors further utilize multistep value propagation and globally relabeled goals, enabling a shared value function to compose shorter paths across trajectories and approximate optimal values. Experiments cover challenging long-horizon OGBench tasks, standard state/pixel OGBench tasks, and CALVIN, alongside analyses of propagation horizons, task horizons, and alternative divide-and-conquer strategies. Overall results demonstrate that DCRL outperforms existing flat GCRL methods in most long-horizon settings and surpasses the tested hierarchical baselines in terms of average performance.
Critical strengths
- The core method is simple and grounded in a clear structural rationale. DCRL reformulates long-horizon value learning as a dependency-ordering problem, organizing the progression from short- to long-horizon value learning via an explicit recursive tree and child-to-parent scheduling, rather than merely modifying the loss function or sampling distribution. The ablation studies regarding reverse-order execution and the removal of recursion provide direct empirical support for this core mechanism.
- The empirical validation presented in the main text is comprehensive and directly addresses the central hypothesis. The paper evaluates the method against a wide range of offline GCRL baselines, spanning long-horizon tasks, standard OGBench tasks, state- and pixel-based observations, and the CALVIN benchmark, while further investigating factors such as propagation/task horizons, error accumulation, and various divide-and-conquer variants. The experiments not only demonstrate final benchmark performance but also systematically analyze the roles of recursion, ordering, propagation, and update rules.
Critical weaknesses
- The theoretical consistency between the two training objectives for the shared critic is not yet fully established. The exact factorization of L_{dc} targets the value of demonstrated-route behavior, whereas L_{prop} trains the same critic to learn the optimal value across trajectories. When the fixed midpoint does not lie on the shortest path, the value semantics associated with the two objectives differ. Although Table 3 indicates that they are empirically complementary, the main text does not articulate the compatibility conditions or a fixed-point interpretation for their joint optimization; consequently, it is currently difficult to determine whether this complementarity holds generally.
- The applicability of the method is constrained by the assumption of deterministic dynamics. The exact factorization in DCRL relies on deterministic goal-reaching dynamics; the authors have explicitly noted in the "Limitations" section that this property generally does not hold under stochastic dynamics. While this limitation does not affect the experimental conclusions of this paper, it renders the method less general than the standard offline GCRL setting.
Questions for authors
- Regarding the joint optimization of the shared critic: Did the authors observe significant conflict between L_{dc} and L_{prop} during training, such as inconsistent gradient directions or a scenario where the improvement of one objective leads to the deterioration of the other? Please provide details if results are available regarding separate critics, disabling one of the losses, or employing different loss-balancing strategies. This information would help determine whether these two distinct value semantics are stably compatible in practice.
- Definition of OGBench evaluation scores. Table 1 in the main text uses a task "score", while Figure 7 uses "Aggregated Score (%)", yet the text does not explicitly define these metrics or the aggregation procedure. Please clarify whether these metrics correspond to the goal-reaching success rate and explain how the aggregation is performed across start-goal tasks and environments.
7 comments
A1. Compatibility of the two objectives
The exact factorization of targets the value of demonstrated-route behavior, whereas trains the same critic to learn the optimal value across trajectories. ... the main text does not articulate the compatibility conditions or a fixed-point interpretation for their joint optimization.
We thank the reviewer for this question. Under deterministic dynamics, the value of a recorded route is a lower bound on the optimal value, , with equality when the route is a shortest path. This gives the following interpretation:
- Compatible case. When recorded routes are shortest paths, the two targets coincide and is a fixed point of the joint objective.
- Suboptimal routes. pulls the shared critic toward , while pulls it toward . For a shared critic, the joint minimizer on such pairs lies between the two targets, so the fixed point is between and , at a position set by the loss weight and by how often each pair is sampled.
Our empirical measurements agree with this picture. We built pointmaze-large datasets with the OGBench navigate procedure, in which the agent turns back to a recently visited cell with probability per step. This lengthens the recorded routes at fixed coverage, from a median of 1.18$\times$ the shortest path at to 2.03$\times$ at . On state-goal pairs whose recorded route is longer than the shortest path, DCRL's learned distances lie between the two targets (0.71–0.78$\times$ the recorded route and 1.67–1.86$\times$ the shortest path). The remaining pull toward the recorded route does not hurt DCRL in practice. TD-n is DCRL without , so it has no pull toward the recorded route, yet DCRL outperforms it by 1.7–2.2$\times$ at every level of suboptimality (e.g., 62.8 vs. 28.4% at , , 3 seeds). We have added this fixed-point discussion and the suboptimality sweep to App. H.4 (Table 7).
In short, the two objectives are compatible. They coincide on shortest paths, and on suboptimal routes they settle between the two targets while DCRL still outperforms TD-n at every level of suboptimality.
A2. Stochastic dynamics
The applicability of the method is constrained by the assumption of deterministic dynamics. ... it renders the method less general than the standard offline GCRL setting.
We agree. Under stochastic dynamics, the exact factorization generally fails because the expectation of a product need not factorize. This assumption is shared by the quasimetric GCRL methods discussed in Sec. 2, in particular the "implicit" methods that, like DCRL, enforce the triangle inequality through value backups, such as TDP (Jurgenson et al., 2020), COE (Piękos et al., 2023) and TRL (Park et al., 2026). DCRL could potentially be extended to stochastic dynamics by replacing the exact factorization with the stochastic triangle inequality of Myers et al. (2024), which we leave for future work.
In short, this is a limitation shared by triangle-inequality methods, and the stochastic triangle inequality offers a possible extension.
We thank the reviewer for the positive assessment and the concrete questions. We address them with new analyses and experiments, and have uploaded a revised PDF, to which all table and figure numbers below refer: