- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+2)
Read review
Paper summary
This paper proposes an extension of Jumpy World Planning with occupancy models to the online estimator and proposes an estimator for the terminal component of the occupancy density under a geometric switching policy. The results indicate that the terminal estimator reduces variance and produces significantly better performance in some cases.
Critical strengths
The major strength is that the paper rigorously shows that using a conditional density estimator for the terminal component reduces variance and leads to better performance than using a count-based density estimator.
Critical weaknesses
The paper seems like a direct extension of Jumpy World Models (Farebrother et. al., 2026) to online RL with new a terminal component density estimator and a different planning mechanism (random shooting vs iCEM planner). The proposed method to learn this estimator does not seem to significantly outperform the MC version in 5/9 environments (Figs 4 and 5).
Questions for authors
1 and 2. The density estimator's labels seem to come from MC sampling. I'm guessing that the labels are 0 or close to 0 far away from goal. The density estimator would still produce very small values in such a scenario. Why not use a different density estimator such as CRL/Hutchinson Trace Estimator since you're using a flow-matching model? Following up on this, isn't it technically possible to use density estimators for all components rather than just the last one? Do you have experiments that test this out?
-
Could you clarify how many future states are sampled when running the MC version in the experiments section?
-
Minor comment: "4.2 Q2: DOES OGPC EXTEND TO HIGH-DIMENSIONAL CONTROL?". Please clarify that this refer to a high-dimensional action space.
3 comments
Isn’t it technically possible to use density estimators for all components rather than just the last one? Do you have experiments that test this out?
Thank you for raising this point. Following your question, we implemented a variant in which every component of the score uses a learned conditional probability (we call this variant “all-component”). Here’s the comparison between the terminal-only OGPC-TC runs against the “all-component” variant on Ant Big Maze and Pusher Easy tasks:
| Task | Metric | All-component - OGPC-TC |
|---|---|---|
| Ant Big Maze | Time at Goal | -39.0 [-101.4, +23.4] |
| Success Rate | -4.3% [-13.5, +4.9] | |
| Pusher Easy | Time at Goal | -54.0 [-129.5, +21.5] |
| Success Rate | -1.0% [-3.6, +1.5] |
We found that estimating the conditional probability at every hop performed worse than OGPC-TC on average across 10 seeds in either environment. We believe this is consistent with the role of terminal conditioning. Its main benefit is to mitigate the all-zero candidate scores of MC scoring (Figure 6), and a learned probability at the last hop already addresses this issue.
Implementation note: The only additional component that this variant requires is a second head for the intermediate discount , because the terminal head is trained at the terminal discount . We trained it using the same architecture, loss and update schedule as our terminal head. We then replaced the sampled reward indicators at the first three hops with the second head's predictions. We ran 10 seeds for 50M transitions, with all other components (GHM, planner, evaluation protocol) unchanged.
Just a small clarification on terminology: we would like to clarify that our terminal head is not a density estimator. It predicts a conditional probability mass that the GHM’s terminal distribution assigns to the goal region.
Q3. Could you clarify how many future states are sampled when running the MC version in the experiments section?
OGPC-MC scores every candidate with simulations (Table 12, Appendix E.3.3). Each simulation samples one future state per subgoal, so with subgoals, OGPC-MC draws future states per candidate. We note that OGPC-TC uses the same as OGPC-MC. We agree that this should be easier to find from the main text. We will state next to in the Methods paragraph of Section 4.
(Minor comment) “4.2. Q2. DOES OGPC EXTEND TO HIGH-DIMENSIONAL CONTROL?” Please clarify that this refers to a high-dimensional action space.
Yes, it refers to the action space, but we also note that the state space of the Humanoid tasks is significantly larger as well. The Humanoid tasks have 17 action dimensions, compared to 5 to 8 for the other 6 tasks. Humanoid tasks have state dimensions of 268, compared to 29 in the Ant Maze tasks. The exact details are provided in Table 8 of Appendix E.1. We will specify this in Section 4.2.
References
[1] Bortkiewicz et al. Accelerating Goal-Conditioned RL Algorithms and Research. 2025.
[2] Farebrother et al. Compositional planning with jumpy world models. 2026.
[3] Grathwohl et al. FFJORD: Free-form continuous dynamics for scalable reversible generative models. 2019.
[4] Thakoor et al. Generalised policy improvement with geometric policy composition. 2022.
I thank the authors for their detailed response. While I still maintain that the paper is an incremental contribution over Jumpy World Models (Farebrother et. al., 2026), the method does work and the paper is thorough. I therefore increase my score.
We thank the reviewer for their constructive feedback and for taking the time to review our work. We would like to clarify how OGPC extends prior work and where terminal conditioning provides additional benefits.
W1. The paper seems like a direct extension of Jumpy World Models (Farebrother et. al., 2026) to online RLRelationship to Jumpy World Models: Our work indeed builds on Farebrother et al., 2026 [2], which itself builds on GPC [4]. However, we would like to remark that Farebrother et al. explicitly identify “jointly learning policies and predictive models” as a promising future direction in their Discussion (Section 5). The online setting, where the policy and its GHM are trained concurrently from scratch, was left open by their work.
W2. The proposed method to learn this estimator does not seem to significantly outperform the MC version in 5/9 environments (Figs 4 and 5)We would like to add some context to the significance of the result: OGPC-TC demonstrates most significant improvements over OGPC-MC on Ant Big Maze, Ant Hardest Maze, Humanoid U-Maze, and Humanoid Big Maze. They are among the most challenging tasks in JaxGCRL [1]. On final Time at Goal (Tables 19 and 20, Appendix F.1), TC has 95% confidence intervals that lie entirely above those of MC on six out of nine tasks. Moreover, TC demonstrates faster learning than MC on Humanoid. We believe that these results show benefits in both performance on challenging tasks and sample efficiency.
The density estimator's labels seem to come from MC sampling. I'm guessing that the labels are 0 or close to 0 far away from goal. The density estimator would still produce very small values in such a scenario.Thank you for raising this point. You are correct that the terminal head's targets are Monte Carlo estimates, and that these targets can be zero when goal occupancy is rare. In our setting, small probabilities are appropriate for unlikely outcomes. The benefit is to preserve meaningful differences between candidate plans that would otherwise receive identical zero scores from finite-budget MC scoring.
Q1. Why not use a different density estimator such as CRL / Hutchinson Trace Estimator since you’re using a flow-matching model?Thank you for suggesting these alternatives. We use a terminal conditional head because plan scoring requires repeated estimates of the conditional probability mass assigned to the task’s goal region,
q(s,a,gK,g⋆,γ)=mγπgK(Xg⋆∣s,a)
and the head provides this quantity directly at low computational cost during planning.
Hutchinson Trace Estimator is impractical in our online setting: A Hutchinson-based approach could evaluate the flow model’s log-density through the instantaneous change of variables formula [3]. However, this requires integrating the flow together with Jacobian-trace estimates, and the resulting state density would still need to be integrated over the goal region, including marginalization over state coordinates that do not define success. At our planner's settings (48 candidates, 64 MC samples per hop, 5 Euler steps), an importance-sampled flow-likelihood estimate of the goal region mass with 64 integration points requires 640 GHM evaluations for the terminal term of every simulation, compared to 5 for the Monte Carlo estimator and none for the terminal head. Instead, the terminal head amortizes the cost of this goal-region probability estimation across training updates and makes it directly available during planning.
The existing CRL critic is not directly interchangeable as the terminal conditional head: We acknowledge a contrastive estimator is also a possible alternative. However, the existing CRL critic is not directly interchangeable with our head as the contrastive score represents a density ratio rather than the finite goal-region occupancy. Moreover, the CRL critic takes a single goal as input, whereas our TC head evaluates the policy commanded to gK at a task goal g⋆. A suitable contrastive estimator could be developed, and we believe that it would constitute an interesting extension by itself. However, this falls outside the scope of this paper.
The exact terminal-conditioning result is independent of how the conditional probability is computed, and the experiments establish the usefulness of a simple learned approximation compared with literal MC scoring. We do not claim to establish superiority over flow-likelihood or contrastive alternatives. We will clarify this rationale for the head in Section 3.2.