- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The method learns a world model from offline data. Its simulations are used to evaluate different frozen GCRL policies and then select one that seems most promising in a certain situation and let it act for a short period of time in the environment. Thereby, the approach learns a single shared goal-conditioned value function, whose design is chosen per task family, to evaluate frozen policies that were trained with different objectives and whose own value estimates are not comparable. Experiments are conducted on 18 datasets and several ablations are presented which indicate that switching between an ensemble of expert policies is possible and usually beneficial.
Critical strengths
-
The method treats expert policies as black boxes which enables to plug in or change new/other policies very easily. Additionally, this method is evaluated on quite a number of environments and ablations to identify the usefulness of arbitration.
-
WMPA beats the per-episode hindsight oracle showing that combining policies within one episode helps.
Critical weaknesses
-
It's unclear if the improvements come from arbitration or from the extra value function and world model. Table 8 shows that swapping the value head costs a lot performance. An MPC test with like CEM with sufficient budget (not like in Table 6) and only using the value function seems needed. Here, the planning horizon should be aligned to the time interval of that arbitration uses. In Table 6, the weak MPC baseline already improves performance noticeably. Now, it is important to see if the value function and world model alone are enough without considering any policy prior / expert policy / arbitration at all.
-
Choosing over fixed policies by simulating them and executing the best is classical (Chang et al., 2004 | Bertsekas & Castañon, 1999), which the paper cites and acknowledges. So, the extensions are the learned world model and value function, so the contribution is mostly empirical and might lack novelty. This makes it important to show that the improvements come from arbitration itself (see first negative point).
-
WMPA is a viable approach, when the given policies are not perfect. Why are stronger GCRL baselines not included like Ahn et al., 2025, Kang et al. (2026)? WMPA's metric value head literally follows Kang et al. (2026).
-
The experiment section presents the main results (Table 1) but excludes lots of ablation tables (Tables 4-9) and only describes experiments and results via long text which is hard to follow. The ablation descriptions could be shorter, then shift some related work to the appendix and present some additional figures in the main text. The authors may also find a better visualization, but I have currently no suggestion.
My negatives are moderately hard, so addressing these will probably raise my score.
References:
- Kang, Hyungkyu, Byeongchan Kim, and Min-hwan Oh. "Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning." arXiv preprint arXiv:2605.25740 (2026).
- Ahn, H., Choi, H., Han, J., and Moon, T. Option-aware temporally abstracted value for offline goal-conditioned reinforcement learning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Questions for authors
- Given the action-level MPC ablation. Do you expect that arbitration and action-level MPC combined can perform even better? So, first select the currently best policy with arbitration (for the short time interval) and then perform action-level MPC using this current best policy but only for the small time interval. Then repeat. In that way, you might leverage the world model more for your method.
4 comments
Official Reply to Reviewer vMzS [2/2]
W2 (continued). Three results show that this replacement is not straightforward and that the gains come from arbitration:
- The natural substitute for comparable values, namely scoring each policy with its own value function, fails even when the value functions estimate the same quantity (GCIQL, GCIVL, and HIQL). Own value scoring underperforms the best of the three policies by 4 percentage points, whereas a single shared value function outperforms that policy by 11 points and is significantly better than own value scoring on 14 of the 18 datasets (Table 10).
- In most decisions, the learned model ranks the policies as the true simulator does. Rolling out the branches of WMPA in the true simulator instead of the learned model adds only 2 points on average (Table 6). We added a decision-by-decision comparison (App. C.4, Table 8): on the manipulation datasets, the score gap between the policy chosen by the model and the best policy according to the simulator is at most 1% of the value scale in 85.1–92.6% of decisions.
- The CEM planner we added (see W1 above; App. C.2) optimizes actions directly against the same model and value function and fails where arbitration succeeds, even with far larger budgets.
In short, WMPA makes the classical rule work offline by learning both of its missing pieces from the dataset.
W3. Stronger GCRL learners.
"Why are stronger GCRL baselines not included like Ahn et al., 2025, Kang et al. (2026)?"
We have added both methods to the bank (App. E.2, Table 15).
- LAVL (Kang et al., 2026): we trained it with the official implementation on all 18 datasets with 3 seeds each. We used the published hyperparameters where available (6 datasets) and otherwise those of a related dataset (App. E.2). On its own, LAVL achieves a macro-average of 29.4% (88.7% and 91.5% on the two maze datasets). When added to the bank (M = 7, 21 calls per step), it raises the macro-average of WMPA from 57.7% to 58.9%. The gains are significant on pointmaze (+1.3), scene-play (+6.7), and scene-noisy (+7.2), and no dataset shows a significant decrease. Compared with the best single policy of the extended bank (selected in hindsight), WMPA is significantly better on 11 of the 18 datasets.
- OTA (Ahn et al., 2025): a published setting exists only for the two maze datasets, so we trained it there with 3 seeds. On its own, OTA reaches 80.3% on pointmaze and 89.5% on antmaze. With both LAVL and OTA in the bank (M = 8, 24 calls per step), WMPA reaches 80.8% on pointmaze and 92.4% on antmaze, compared with 79.5% and 90.7% with the original six policies.
Regarding the metric head: yes, it follows the parameterization of LAVL (Sec. 3.2, App. F) and is trained only on the dataset. It is separate from the LAVL policy, which is now a member of the bank.
In short, stronger learners are useful bank members.
W4. Presentation.
"The ablation descriptions could be shorter, then shift some related work to the appendix and present some additional figures in the main text."
We have added Table 2 to Sec. 4. For every control experiment, this table lists the datasets covered by that experiment, the mean difference from WMPA, and the number of datasets on which WMPA is significantly better or worse. We have also shortened the surrounding ablation text, moved most of the related work to App. J, and redrawn Figure 2 to show both training and test time. The appendix adds five tables and four figures (Tables 7, 8, 14, 15, 18; Figs. 4–7).
Q. Arbitration followed by action-level MPC.
"Do you expect that arbitration and action-level MPC combined can perform even better?"
We have added this two-stage planner (App. C.3, Table 7b). WMPA selects a policy for each interval, and within the interval, MPC around the selected policy (six candidates, each imagined for k steps) replans at every step, using 6–11 times as many model calls as WMPA. On the four manipulation datasets, the planner changes the average success rate by +0.7 percentage points with σ = 0.05 and by −20.9 points with σ = 0.2. Even when σ is chosen for each dataset in hindsight, the change is at most +1.3 points. On pointmaze, the planner adds 0.5 and 0.8 points with σ = 0.05 and σ = 0.2, respectively. Replaying the decisions in the simulator suggests why this refinement adds little. When choosing among perturbations of the first action, the model selects the best candidate according to the simulator in 38–67% of decisions, whereas when choosing among policies, it selects the best one in 65–76% of decisions. Choosing among policies is therefore the more reliable use of the model.
We hope these additions address your concerns; if they do, we would be grateful if you would consider raising your score. We would be glad to clarify anything further.
Thanks for your answers. I think the authors properly engaged with my concerns and put a lot of effort in additional experiments.
I raise my score to 3 and soundness to 2.
We added the experiments and clarifications you requested to the paper (changes in blue).
Cool. I think you had a busy week, so some things might not be 100% polished. I read most of the blue text and looked at my requested experiments. If accepted, I encourage the authors to continue to work on the presentation of the experiments section for the camera-ready version. And thereby fill up all ten pages of the main text. While I give presentation 3, I feel like the paper has become more chaotic (incl. Appendix).
An unconstrained optimizer exploits the errors of the learned model and value function, [...] In short, the world model and value function become useful for control through the bank [...]
I think, this is a quite important insight and strong motivation for arbitration. Inaccurate/exploitable models are a general WM issue. Maybe this deserves a bigger portion in the experiments ...
Best regards.
Thank you for reading the revision carefully and for raising your score.
Presentation. We agree that the revision was somewhat rushed due to the limited rebuttal time. We will continue improving the presentation for the camera-ready version.
Model exploitation. We agree that this is a key motivation for arbitration. We will better highlight how search can exploit errors in the world model and value function, and why arbitration helps address this issue.
Thank you again for the constructive discussion, which has improved the paper.
Official Reply to Reviewer vMzS [1/2]
Thank you for the actionable review. We added the experiments and clarifications you requested to the paper (changes in blue).
W1. Arbitration, or the value function and world model alone?
We have added this planner (App. C.2). It performs model-predictive control with the cross-entropy method (CEM) in the same world model as WMPA and maximizes the same rollout score (Eq. 11) over action sequences, without any policy prior, bank, or arbitration. The planning horizon equals the arbitration interval k of each family. Each replanning step runs four CEM iterations, each with 512 sampled action sequences, and refits the sampling distribution to the 64 best sequences. Following iCEM, the planner reuses elite sequences and warm-starts from the previous plan. It replans either at every step or once every k steps, at a cost of 6,144–61,440 model calls per environment step, which is 341–3,413 times the 18 calls used by WMPA. On the four manipulation datasets in Table 7b, CEM achieves a success rate of 0.1–12.9%, which is 51–84 percentage points below WMPA and also below the best single policy. On pointmaze, where k = 1, we also evaluated a horizon of 10. CEM reaches 80.0% with horizon 1 and 78.8% with horizon 10, compared with 79.5% for WMPA.
Replaying the CEM decisions from the same states in the true simulator and scoring them with the same value head explains the poor performance of CEM. In the model, the CEM plan outscores every policy in the bank in 97–100% of decisions. However, its predicted score exceeds its simulated score by 5.2–19.7 value units, compared with a gap of −4.7 to +4.8 for the policy chosen by WMPA. Even in the simulator, the CEM plan scores 1.4–2.0 value units higher than the policy chosen by WMPA, yet CEM reaches the goal in 0–16% of these episodes, compared with 52–90% for WMPA. An unconstrained optimizer exploits the errors of the learned model and value function, and a score over k steps is not a sufficient objective on its own. Restricting the candidates to the behaviors of the policies in the bank avoids both problems.
We agree: the world model and the shared value head, applied around a single policy, raise the macro-average from 44.4% (Best) to 52.7%. We have added stronger searches around the best policy, all using the same world model, value head, and interval as WMPA (App. C.1, Table 7, Fig. 6). Under the same budget as WMPA, trajectory sampling with six candidates reaches 41.7–52.1%, depending on the noise scale, compared with 57.7% for WMPA. With larger budgets on four manipulation datasets, the best-performing search stays at 68.7–69.5% from 135 to 720 model calls per step, compared with 73.0% for WMPA with 18 calls per step. A search matches the performance of WMPA only when given a noise scale chosen for each dataset in hindsight and 20 times as many model calls. Therefore, the bank of policies is important to make world model and the value head most useful for control.
Each family uses the value head whose inductive bias matches the type of progress toward the goal in that family (Sec. 3.2). We have also added a procedure that selects the head for each dataset using only the training data, without any online interaction (App. C.5, Table 11). On logged trajectories, the procedure checks which head assigns a higher value to the state from which the goal is reached sooner. On every dataset, the head selected in this way performs at least as well in evaluation as the other head. In particular, the procedure selects the direct head on cube-triple-play, where this head outperforms the head we used in our main experiments (8.5% vs. 0.7%).
In short, the world model and value function become useful for control through the bank: without policies, CEM fails, and a search around one policy needs hindsight tuning and 20 times the model calls to match WMPA.
W2. Novelty.
We agree that the selection rule is classical, and we have clarified this in the contributions (Sec. 1). The rule is inherited from policy switching and parallel rollout (Chang et al., 2004; Bertsekas & Castañon, 1999). It requires a simulator and values that are comparable across policies, neither of which is available in the offline setting. WMPA replaces both with components learned from the dataset, and we study the conditions under which this replacement succeeds.