acceptodds
Under review as a conference paper at ICLR 2027

World-Model Policy Arbiter for Goal-Conditioned Reinforcement Learning

28 views

Abstract

Offline goal-conditioned reinforcement learning (GCRL) has produced a diverse set of goal-reaching algorithms, yet no single algorithm performs best across environments, goals, and even different phases of the same task. Rather than deploying only the best-performing policy, we ask whether a set of frozen goal-conditioned policies can be used collectively as a portfolio, deciding at every state which policy should act. Choosing a policy at each state is not straightforward. The policies’ own value functions cannot be compared directly: they may use different scales, and some policies have no value function. We need to judge each policy by the states it is likely to reach, even though we can execute only one policy at a time. We also need to avoid switching so often that control becomes unstable. To address these challenges, we introduce World-Model Policy Arbiter (WMPA), a test-time framework that, given a set of frozen policies as input, rolls out each frozen policy in a learned state-space world model, evaluates the imagined futures with a shared goal-conditioned value function, and executes the highest-scoring policy for a short commitment interval before the next round of arbitration (policy selection). WMPA assumes access to a bank of frozen goal-conditioned policies and requires neither policy retraining nor privileged task-specific knowledge. Under the official OGBench evaluation protocol on 18 state-based datasets spanning maze navigation as well as cube, scene, and puzzle manipulation, WMPA improves the macro-average success rate from the 44% achieved by the best policy selected per dataset to 58%, with statistically significant gains on 12 datasets. These gains include +33 percentage points on cube-double-play and +36 percentage points on scene-play.

Primary area:
reinforcement learning
Keywords:
goal-conditioned reinforcement learning, offline reinforcement learning, world model, policy selection, test-time planning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vMzSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The method learns a world model from offline data. Its simulations are used to evaluate different frozen GCRL policies and then select one that seems most promising in a certain situation and let it act for a short period of time in the environment. Thereby, the approach learns a single shared goal-conditioned value function, whose design is chosen per task family, to evaluate frozen policies that were trained with different objectives and whose own value estimates are not comparable. Experiments are conducted on 18 datasets and several ablations are presented which indicate that switching between an ensemble of expert policies is possible and usually beneficial.

Critical strengths
  • The method treats expert policies as black boxes which enables to plug in or change new/other policies very easily. Additionally, this method is evaluated on quite a number of environments and ablations to identify the usefulness of arbitration.

  • WMPA beats the per-episode hindsight oracle showing that combining policies within one episode helps.

Critical weaknesses
  • It's unclear if the improvements come from arbitration or from the extra value function and world model. Table 8 shows that swapping the value head costs a lot performance. An MPC test with like CEM with sufficient budget (not like in Table 6) and only using the value function seems needed. Here, the planning horizon should be aligned to the time interval of that arbitration uses. In Table 6, the weak MPC baseline already improves performance noticeably. Now, it is important to see if the value function and world model alone are enough without considering any policy prior / expert policy / arbitration at all.

  • Choosing over fixed policies by simulating them and executing the best is classical (Chang et al., 2004 | Bertsekas & Castañon, 1999), which the paper cites and acknowledges. So, the extensions are the learned world model and value function, so the contribution is mostly empirical and might lack novelty. This makes it important to show that the improvements come from arbitration itself (see first negative point).

  • WMPA is a viable approach, when the given policies are not perfect. Why are stronger GCRL baselines not included like Ahn et al., 2025, Kang et al. (2026)? WMPA's metric value head literally follows Kang et al. (2026).

  • The experiment section presents the main results (Table 1) but excludes lots of ablation tables (Tables 4-9) and only describes experiments and results via long text which is hard to follow. The ablation descriptions could be shorter, then shift some related work to the appendix and present some additional figures in the main text. The authors may also find a better visualization, but I have currently no suggestion.

My negatives are moderately hard, so addressing these will probably raise my score.

References:

  • Kang, Hyungkyu, Byeongchan Kim, and Min-hwan Oh. "Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning." arXiv preprint arXiv:2605.25740 (2026).
  • Ahn, H., Choi, H., Han, J., and Moon, T. Option-aware temporally abstracted value for offline goal-conditioned reinforcement learning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Questions for authors
  • Given the action-level MPC ablation. Do you expect that arbitration and action-level MPC combined can perform even better? So, first select the currently best policy with arbitration (for the short time interval) and then perform action-level MPC using this current best policy but only for the small time interval. Then repeat. In that way, you might leverage the world model more for your method.
4 comments
AuthorsOpenReview ↗

Official Reply to Reviewer vMzS [1/2]

Thank you for the actionable review. We added the experiments and clarifications you requested to the paper (changes in blue).

W1. Arbitration, or the value function and world model alone?

"An MPC test with like CEM with sufficient budget (not like in Table 6) and only using the value function seems needed. Here, the planning horizon should be aligned to the time interval of that arbitration uses."

We have added this planner (App. C.2). It performs model-predictive control with the cross-entropy method (CEM) in the same world model as WMPA and maximizes the same rollout score (Eq. 11) over action sequences, without any policy prior, bank, or arbitration. The planning horizon equals the arbitration interval k of each family. Each replanning step runs four CEM iterations, each with 512 sampled action sequences, and refits the sampling distribution to the 64 best sequences. Following iCEM, the planner reuses elite sequences and warm-starts from the previous plan. It replans either at every step or once every k steps, at a cost of 6,144–61,440 model calls per environment step, which is 341–3,413 times the 18 calls used by WMPA. On the four manipulation datasets in Table 7b, CEM achieves a success rate of 0.1–12.9%, which is 51–84 percentage points below WMPA and also below the best single policy. On pointmaze, where k = 1, we also evaluated a horizon of 10. CEM reaches 80.0% with horizon 1 and 78.8% with horizon 10, compared with 79.5% for WMPA.

Replaying the CEM decisions from the same states in the true simulator and scoring them with the same value head explains the poor performance of CEM. In the model, the CEM plan outscores every policy in the bank in 97–100% of decisions. However, its predicted score exceeds its simulated score by 5.2–19.7 value units, compared with a gap of −4.7 to +4.8 for the policy chosen by WMPA. Even in the simulator, the CEM plan scores 1.4–2.0 value units higher than the policy chosen by WMPA, yet CEM reaches the goal in 0–16% of these episodes, compared with 52–90% for WMPA. An unconstrained optimizer exploits the errors of the learned model and value function, and a score over k steps is not a sufficient objective on its own. Restricting the candidates to the behaviors of the policies in the bank avoids both problems.

"In Table 6, the weak MPC baseline already improves performance noticeably."

We agree: the world model and the shared value head, applied around a single policy, raise the macro-average from 44.4% (Best) to 52.7%. We have added stronger searches around the best policy, all using the same world model, value head, and interval as WMPA (App. C.1, Table 7, Fig. 6). Under the same budget as WMPA, trajectory sampling with six candidates reaches 41.7–52.1%, depending on the noise scale, compared with 57.7% for WMPA. With larger budgets on four manipulation datasets, the best-performing search stays at 68.7–69.5% from 135 to 720 model calls per step, compared with 73.0% for WMPA with 18 calls per step. A search matches the performance of WMPA only when given a noise scale chosen for each dataset in hindsight and 20 times as many model calls. Therefore, the bank of policies is important to make world model and the value head most useful for control.

"Table 8 shows that swapping the value head costs a lot performance."

Each family uses the value head whose inductive bias matches the type of progress toward the goal in that family (Sec. 3.2). We have also added a procedure that selects the head for each dataset using only the training data, without any online interaction (App. C.5, Table 11). On logged trajectories, the procedure checks which head assigns a higher value to the state from which the goal is reached sooner. On every dataset, the head selected in this way performs at least as well in evaluation as the other head. In particular, the procedure selects the direct head on cube-triple-play, where this head outperforms the head we used in our main experiments (8.5% vs. 0.7%).

In short, the world model and value function become useful for control through the bank: without policies, CEM fails, and a search around one policy needs hindsight tuning and 20 times the model calls to match WMPA.

W2. Novelty.

"the extensions are the learned world model and value function, so the contribution is mostly empirical and might lack novelty"

We agree that the selection rule is classical, and we have clarified this in the contributions (Sec. 1). The rule is inherited from policy switching and parallel rollout (Chang et al., 2004; Bertsekas & Castañon, 1999). It requires a simulator and values that are comparable across policies, neither of which is available in the offline setting. WMPA replaces both with components learned from the dataset, and we study the conditions under which this replacement succeeds.

AuthorsOpenReview ↗

Official Reply to Reviewer vMzS [2/2]

W2 (continued). Three results show that this replacement is not straightforward and that the gains come from arbitration:

  1. The natural substitute for comparable values, namely scoring each policy with its own value function, fails even when the value functions estimate the same quantity (GCIQL, GCIVL, and HIQL). Own value scoring underperforms the best of the three policies by 4 percentage points, whereas a single shared value function outperforms that policy by 11 points and is significantly better than own value scoring on 14 of the 18 datasets (Table 10).
  2. In most decisions, the learned model ranks the policies as the true simulator does. Rolling out the branches of WMPA in the true simulator instead of the learned model adds only 2 points on average (Table 6). We added a decision-by-decision comparison (App. C.4, Table 8): on the manipulation datasets, the score gap between the policy chosen by the model and the best policy according to the simulator is at most 1% of the value scale in 85.1–92.6% of decisions.
  3. The CEM planner we added (see W1 above; App. C.2) optimizes actions directly against the same model and value function and fails where arbitration succeeds, even with far larger budgets.

In short, WMPA makes the classical rule work offline by learning both of its missing pieces from the dataset.

W3. Stronger GCRL learners.

"Why are stronger GCRL baselines not included like Ahn et al., 2025, Kang et al. (2026)?"

We have added both methods to the bank (App. E.2, Table 15).

  • LAVL (Kang et al., 2026): we trained it with the official implementation on all 18 datasets with 3 seeds each. We used the published hyperparameters where available (6 datasets) and otherwise those of a related dataset (App. E.2). On its own, LAVL achieves a macro-average of 29.4% (88.7% and 91.5% on the two maze datasets). When added to the bank (M = 7, 21 calls per step), it raises the macro-average of WMPA from 57.7% to 58.9%. The gains are significant on pointmaze (+1.3), scene-play (+6.7), and scene-noisy (+7.2), and no dataset shows a significant decrease. Compared with the best single policy of the extended bank (selected in hindsight), WMPA is significantly better on 11 of the 18 datasets.
  • OTA (Ahn et al., 2025): a published setting exists only for the two maze datasets, so we trained it there with 3 seeds. On its own, OTA reaches 80.3% on pointmaze and 89.5% on antmaze. With both LAVL and OTA in the bank (M = 8, 24 calls per step), WMPA reaches 80.8% on pointmaze and 92.4% on antmaze, compared with 79.5% and 90.7% with the original six policies.

Regarding the metric head: yes, it follows the parameterization of LAVL (Sec. 3.2, App. F) and is trained only on the dataset. It is separate from the LAVL policy, which is now a member of the bank.

In short, stronger learners are useful bank members.

W4. Presentation.

"The ablation descriptions could be shorter, then shift some related work to the appendix and present some additional figures in the main text."

We have added Table 2 to Sec. 4. For every control experiment, this table lists the datasets covered by that experiment, the mean difference from WMPA, and the number of datasets on which WMPA is significantly better or worse. We have also shortened the surrounding ablation text, moved most of the related work to App. J, and redrawn Figure 2 to show both training and test time. The appendix adds five tables and four figures (Tables 7, 8, 14, 15, 18; Figs. 4–7).

Q. Arbitration followed by action-level MPC.

"Do you expect that arbitration and action-level MPC combined can perform even better?"

We have added this two-stage planner (App. C.3, Table 7b). WMPA selects a policy for each interval, and within the interval, MPC around the selected policy (six candidates, each imagined for k steps) replans at every step, using 6–11 times as many model calls as WMPA. On the four manipulation datasets, the planner changes the average success rate by +0.7 percentage points with σ = 0.05 and by −20.9 points with σ = 0.2. Even when σ is chosen for each dataset in hindsight, the change is at most +1.3 points. On pointmaze, the planner adds 0.5 and 0.8 points with σ = 0.05 and σ = 0.2, respectively. Replaying the decisions in the simulator suggests why this refinement adds little. When choosing among perturbations of the first action, the model selects the best candidate according to the simulator in 38–67% of decisions, whereas when choosing among policies, it selects the best one in 65–76% of decisions. Choosing among policies is therefore the more reliable use of the model.

We hope these additions address your concerns; if they do, we would be grateful if you would consider raising your score. We would be glad to clarify anything further.

Thanks for your answers. I think the authors properly engaged with my concerns and put a lot of effort in additional experiments.

I raise my score to 3 and soundness to 2.

We added the experiments and clarifications you requested to the paper (changes in blue).

Cool. I think you had a busy week, so some things might not be 100% polished. I read most of the blue text and looked at my requested experiments. If accepted, I encourage the authors to continue to work on the presentation of the experiments section for the camera-ready version. And thereby fill up all ten pages of the main text. While I give presentation 3, I feel like the paper has become more chaotic (incl. Appendix).

An unconstrained optimizer exploits the errors of the learned model and value function, [...] In short, the world model and value function become useful for control through the bank [...]

I think, this is a quite important insight and strong motivation for arbitration. Inaccurate/exploitable models are a general WM issue. Maybe this deserves a bigger portion in the experiments ...

Best regards.

Thank you for reading the revision carefully and for raising your score.

Presentation. We agree that the revision was somewhat rushed due to the limited rebuttal time. We will continue improving the presentation for the camera-ready version.

Model exploitation. We agree that this is a key motivation for arbitration. We will better highlight how search can exploit errors in the world model and value function, and why arbitration helps address this issue.

Thank you again for the constructive discussion, which has improved the paper.

Reviewer_pXvrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose WMPA, a method for choosing among several frozen goal-conditioned RL policies. It uses a learned world model to predict where each policy may lead, scores those predicted states with a shared value function, and runs the selected policy for a short period before choosing again. On 18 OGBench datasets, average success rises from 44% for the best fixed policy per dataset to 58%.

Critical strengths
  1. Clear writing. The problem is easy to understand, the paper is well organized, and Figure 1 gives a helpful example of policy switching.

  2. A useful idea. Reusing several frozen policies without retraining them is practical. Different policies may be good at different parts of a task.

  3. Strong results. The paper tests a range of tasks and includes comparisons with fixed policies, random switching, and model-based alternatives.

Critical weaknesses
  1. The offline RL motivation is not clear enough. The main method chooses policies at test time and could also be used with policies trained online. What specific difficulty of offline RL does it solve? Does it help with limited data coverage or distribution shift, or is it mainly a general policy-switching method? The paper should make this distinction clearer.

  2. Figure 2 is too simple to explain the whole method. The writing is clear, but the figure does not show how offline data, world-model training, value learning, imagined rollouts, and policy switching connect. The individual losses and selection rule are already in the text; a more complete overview would make their roles easier to follow.

  3. The experimental analysis does not explain enough about why the method works. Figure 1 and Figure 3 are useful, but most results are still numerical tables. I would like to see PointMaze trajectories, value heatmaps, scores of candidate policies over time, and the states where policies are switched.

  4. World-model errors may lead to wrong policy choices. Some frozen policies may visit states that are poorly covered by the offline dataset. The learned model may then give a high score to the wrong policy.

Questions for authors
  1. What is the main offline RL challenge addressed by WMPA? Could the same approach be tested with online-trained policies?

  2. Could Figure 2 show what is trained, what stays frozen, and how the full training and test-time steps fit together?

  3. Could you show actual trajectories, policy scores, and switch points in PointMaze and a manipulation task, including failure cases?

  4. How often does the learned world model select the same policy as the true simulator? Does this agreement decrease in states poorly covered by offline data?

  5. Since one-step selection already performs well on several tasks, what additional benefit does multi-step prediction provide? Could the authors compare it with a simpler learned policy selector and extend the K,C analysis to more task.

3 comments
AuthorsOpenReview ↗

Official Reply to Reviewer pXvr [1/2]

Thank you for your review and suggestions. We have updated our paper (changes in blue) with new experiments, figures, and clarifications which, we hope, made it better.

W1/Q1. Which difficulty of offline RL does WMPA address?

"What specific difficulty of offline RL does it solve? Does it help with limited data coverage or distribution shift, or is it mainly a general policy-switching method?"

The core difficulty lies in choosing among policies without interacting with the environment; we have clarified this point in the introduction. In the offline setting, policies cannot be compared by executing them, so the standard practice is to rank the candidates once and deploy a single policy. WMPA requires no online data: it compares the policies in imagination, using a world model and a value function trained on the same dataset as the policies, and it makes a new choice at every arbitration point. Offline GCRL is also a setting in which a bank of policies requires no additional data: different offline algorithms extract different behaviors from the same dataset, and no single algorithm dominates in Table 1. Moreover, on 7 datasets, WMPA significantly outperforms the hindsight oracle that selects the best policy for each episode (Fig. 3). The arbitration mechanism itself is general and does not require offline training (Sec. 2). Dataset coverage influences WMPA through the learned model; our response to W4/Q4 below quantifies how coverage affects the choices made with the model.

"Could the same approach be tested with online-trained policies?"

Yes. WMPA treats the policies only as black boxes that map a state and a goal to an action (Sec. 2), so policies trained online can be added to the bank without modification, provided that they use the same observation, action, and goal spaces. To show this, we added a stronger learner, LAVL (Kang et al., 2026), to the bank as a seventh policy: WMPA raises its macro-average from 57.7% to 58.9%, with significant gains on 3 datasets and no significant decrease (App. E.2).

In short, WMPA chooses among policies without interacting with the environment, and any policy with the same interface can join the bank.

W2/Q2. Figure 2.

"Could Figure 2 show what is trained, what stays frozen, and how the full training and test-time steps fit together?"

We have redrawn Figure 2 to show two stages and clarify this in the caption. Training: the dynamics ensemble, together with its value head, is trained on the dataset. The policies are an input to WMPA; in our experiments, the six policies are trained independently on the same dataset, which the figure shows with a dashed line. All components are then frozen, as indicated on each of them in the figure. Test time: at each arbitration point, every frozen policy is rolled out for k steps in the model and scored by the shared value head (mean over the ensemble members and maximum over the horizon; Eq. 11). The highest-scoring policy then acts for c steps before the next arbitration point.

W3/Q3. Trajectories, scores over time and switch points.

"Could you show actual trajectories, policy scores, and switch points in PointMaze and a manipulation task, including failure cases?"

We have added these figures (App. B). Figures 4 and 5 show pointmaze trajectories over the value map of WMPA and the scores of all policies at every step, for the first failed and the first successful episode on the two tasks on which WMPA sometimes fails (bank seed 0). In the successful episodes, WMPA switches among policies that all move toward the goal, and their scores rise together. Both failures are stalls: either the top choice alternates between two policies whose scores stay within a narrow range, or the agent is stuck in a corner, where its state and the scores stop changing. For manipulation, Figure 1 shows a cube-double-play episode annotated with the policy chosen at every arbitration point and the phase of the task (HIQL moves the empty gripper to the next cube, and GCIQL grasps and places it). Table 17 reports the number of switches per episode and the entropy of policy usage on every dataset.

These stalls motivate a simple addition, which we have now included (App. B). When the agent has barely moved over the last 10 steps, the policies executed during that window are excluded from the candidate set of WMPA for the next 100 steps; this rule requires no additional model calls. On pointmaze, this stall guard raises the success rate of WMPA from 79.5% to 98.8%, and both previously failed episodes succeed (Fig. 5, middle column).

In short, Figures 1, 4 and 5 show the requested trajectories, scores and switch points, and the stall guard motivated by the pointmaze failures raises success there from 79.5% to 98.8%.

AuthorsOpenReview ↗

Official Reply to Reviewer pXvr [2/2]

W4/Q4. Does the learned model choose what the true simulator would choose?

"How often does the learned world model select the same policy as the true simulator? Does this agreement decrease in states poorly covered by offline data?"

The original submission already tested this at the level of whole episodes (now Table 6). Rolling out the branches of WMPA in the true simulator instead of the learned model changes the macro-average by +2 percentage points; the change is significant only on the two scene datasets, and no dataset shows a significant decrease. We have now added a decision-by-decision comparison (App. C.4, Table 8). From every decision state in the episodes generated by WMPA (6 datasets, 750 episodes each), every policy in the bank is branched in the true simulator for the same k steps and scored with the same value head. The comparison therefore isolates the ranking errors caused by the learned model. A decision is counted as a disagreement only when the best policy according to the simulator outscores the policy chosen by the model by more than 1% of the value scale of the dataset. On the four manipulation datasets, the model agrees with the simulator in 85.1–92.6% of decisions, compared with 33.0–59.6% for a uniformly random choice; on the two maze datasets, it agrees in all decisions.

Agreement is highest where the dynamics ensemble is confident and the decision state is close to the training data. We sorted the decisions into quartiles by how much the three ensemble members disagree on the chosen rollout. Agreement is 91–99% in the quartile with the least disagreement and 66–78% in the quartile with the most. When the decisions are instead sorted by the distance from the decision state to the nearest of 200,000 randomly sampled training states, agreement is 87–96% in the nearest quartile and 79–91% in the farthest. Ensemble disagreement and the distance from the decision state to the offline data therefore can serve as indication when the ranking produced by the model is most reliable.

In short, we added a decision-by-decision comparison with the true simulator (App. C.4, Table 8): the learned model chooses as the simulator does in 85–93% of manipulation decisions and in all maze decisions, and replacing it with the simulator adds only 2 points on average, so the learned model is accurate enough for arbitration.

Q5. What does prediction over several steps add, and how does WMPA compare with a simpler learned selector?

"Since one-step selection already performs well on several tasks, what additional benefit does multi-step prediction provide? Could the authors compare it with a simpler learned policy selector and extend the K,C analysis to more task."

  • Q-select (Table 9) is the model-free variant of our selector: it uses the twin Q-functions of the GCIQL member of the bank to rank the action proposed by each policy at the current state, and it makes a new choice at every step. It is a strong baseline that significantly outperforms Best on 10 of the 18 datasets. We have revised Sec. 4.2 to present it as a strong option for applications with tighter compute budgets. Both selectors support our main finding that choosing among pretrained policies at test time outperforms the best single policy.
  • Predicting several steps ahead provides two benefits (Table 9). First, it enables the use of the metric value head, which scores states rather than actions and is the better scorer on the maze, cube, and scene families: on average over all datasets, WMPA outperforms Q-select by 12 percentage points. Second, it evaluates the steps after the first one. When both selectors commit to their choice for c = k steps, WMPA with the direct value head, given by the V of the same GCIQL agent, outperforms Q-select by 7 points on average; the gain is significant on 11 of the 18 datasets, and WMPA is never significantly worse.
  • We have extended the (k, c) analysis to all 18 datasets (Table 4, with k ∈ {1, 5, 10} and c ≤ k). WMPA is robust to the choice of k and c: the macro-average of every setting lies between 55.5% and 58.1%, and choosing the best setting for each dataset in hindsight adds only 1.9 points. Selecting a policy at every step with k = 1 achieves 55.5%, compared with 57.7% for the reported settings.

In short, predicting several steps ahead lets WMPA use the stronger metric value head and evaluate the steps after the first, and WMPA is robust to the choice of k and c.

We hope these additions address your concerns; if they do, we would be grateful if you would consider raising your score. We would be glad to clarify anything further.

Dear Reviewer pXvr,

As the discussion period ends soon, we would like to follow up on our response. We have clarified the offline RL motivation, revised Figure 2, and added the requested trajectory visualizations, model reliability analysis, and comparisons with simpler policy selection methods across all 18 datasets.

We would appreciate hearing whether these additions address your concerns, or if anything remains unclear. If your concerns have been resolved, we would be grateful if you would consider updating your score.

Thank you again for your time and feedback.

Reviewer_1ecAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes World-Model Policy Arbiter (WMPA), a method for selecting among frozen goal-conditioned policies during execution. WMPA rolls out each candidate policy in a learned world model, evaluates the predicted states using a shared goal-conditioned value function, and executes the highest-scoring policy for a fixed interval before reassessing. The approach aims to combine useful behaviors from independently trained policies within a single episode. Across 18 state-based OGBench datasets, the paper reports an higher average success rate, compared with the best fixed policy selected separately for each dataset.

Critical strengths

The problem and proposed mechanism are clearly connected. Comparing heterogeneous policies through a shared evaluation of their predicted consequences addresses the incompatibility of their native value estimates. Reusing frozen policies without modifying their parameters is an interesting setting.

Critical weaknesses

W1. Performance depends strongly on task-family-specific scorer choices.

Table 8 shows large differences between value heads while keeping the policy bank and world model fixed (particularly on antmaze-large, puzzle-3x3). Thus, the practical effectiveness of the method depends substantially on choosing an appropriate scorer, beyond the arbitration mechanism itself.

The authors state that the task-family choices were made before evaluation and provide an offline ranking diagnostic in Table 10. However, this diagnostic is a post hoc analysis of four datasets. It does not yet establish a reliable procedure for selecting the scorer prospectively in a new environment. This limits the predictability of the method’s performance and leaves an important design decision.

W2. The advantage of a heterogeneous policy bank over additional test-time search with a single policy is not sufficiently isolated.

Table 6 is a meaningful control: it applies action search to the best single policy using the same world model and value function, obtaining approximately 53% average success versus 58% for WMPA. Moreover, this baseline replans at every environment step and receives more model calls than WMPA on the manipulation tasks. The concern is therefore not that the baseline was given insufficient computational resources.

Rather, the search procedure is narrow. It considers the original action and five Gaussian perturbations at a fixed noise scale. Only the first action differs between candidates; the remaining rollout follows the same deterministic policy. This establishes an advantage over one form of local action refinement, but does not establish how WMPA compares with generating multiple complete trajectories from a single policy or increasing its sampling budget.

Consequently, the experiments do not fully separate the value of heterogeneous policy behaviors from the more general benefit of evaluating multiple candidates with a suitable scorer. A controlled comparison that keeps the world model, scorer, and inference budget fixed while changing the candidate-generation procedure would clarify the contribution of policy diversity.

Questions for authors
  1. Prospective scorer selection: How can the task-family rationale and the diagnostic in Table 10 be turned into a reliable procedure for choosing the value head before evaluating a new environment? Can you assess the stability and resulting performance of this selection procedure without using evaluation returns to choose the head?

  2. Performance relative to inference cost: Could you extend the reported runtime comparison to include Q-select with per-step reselection and the action-search baseline from Table 6? A success-rate versus wall-clock comparison, distinguishing the modest puzzle gains from the larger Cube/Scene gains, would help identify when world-model arbitration provides a practical advantage.

  3. Single-policy trajectory sampling: On one or two representative manipulation tasks, could you compare WMPA with multiple stochastic rollouts from a single policy while holding the world model and scorer fixed? Matching the model-call budget and reporting actual latency would help determine whether heterogeneous policy behaviors offer an advantage beyond additional test-time exploration.

4 comments
AuthorsOpenReview ↗

Official Reply to Reviewer 1ecA [1/2]

Thank you for the careful reading and the precise suggestions. They were helpful to improve our paper and we have uploaded a revised pdf (changes in blue).

W1/Q1. Choosing the value head before evaluating a new environment.

"the practical effectiveness of the method depends substantially on choosing an appropriate scorer" … "How can the task-family rationale and the diagnostic in Table 10 be turned into a reliable procedure for choosing the value head before evaluating a new environment?"

The rationale fixes one head per task family by the kind of progress (Sec. 3.2): a latent distance (metric head) where progress is geometric, as in maze, cube and scene, and a value without geometric structure (direct head) where it is combinatorial, as in puzzle. This coarse choice is nearly optimal: choosing the better head for each dataset in hindsight would change only cube-triple-play and raise the macro by 0.4 points (57.7 → 58.1, Table 9).

We have turned the diagnostic into a procedure that uses only the training data and runs no policy (App. C.5, Table 11). On logged trajectories, it checks which head gives the higher value to the state from which the goal is reached sooner.

Applied unchanged to all 18 datasets, it selects a head that performs at least as well in evaluation as the other one on every dataset. On cube-triple-play, it selects the direct head, which outperforms the head we used in our main experiments (8.5% vs. 0.7%).

In short, using only the training data, our procedure selects the better head on all 18 datasets, and on cube-triple-play it corrects the family heuristic (8.5% vs. 0.7%).

W2/Q3. A bank of policies against search around one policy.

"A controlled comparison that keeps the world model, scorer, and inference budget fixed while changing the candidate-generation procedure would clarify the contribution of policy diversity."

We have added this comparison on all 18 datasets (App. C.1, Table 7, Fig. 6). Each search keeps WMPA's world model, value head, scoring rule (Eq. 11) and interval, and starts from the best policy of each bank seed, as Table 6 does; only the candidates change.

  • Trajectory sampling (your Q3): the candidates consist of one rollout that follows the mean action of the policy and five rollouts with Gaussian action noise of scale σ added at every imagined step. The noise sequence of the highest-scoring candidate is then replayed for c = k steps. With six candidates, trajectory sampling uses the same budget as WMPA, namely 18 model calls per environment step. Across σ ∈ {0.05, 0.1, 0.2, 0.5}, it achieves a macro-average of 41.7–52.1%, compared with 57.7% for WMPA, and WMPA is significantly better on 7–10 datasets (Table 7a). When σ is chosen for each dataset on the test episodes (that is, in hindsight), trajectory sampling reaches 56.6%, which is still below the 57.7% achieved by WMPA.
  • Larger budgets (Fig. 6, Table 7b): on four representative manipulation datasets, larger searches do not improve performance. With a fixed noise scale, a budget about five times larger adds at most 4.4 points; the best search, MPC that perturbs the first action (σ = 0.05), stays at 68.7–69.5% with 6, 12 or 32 candidates (135 to 720 model calls per step), whereas WMPA reaches 73.0% with 18 calls per step. A search matches WMPA only when given a setting chosen for each dataset in hindsight and 360 calls per step, 20 times the budget of WMPA.
  • Seed bank (App. E.1, Table 14): we also added a bank of six training seeds of a single algorithm, namely the best algorithm for each bank seed. With this bank, WMPA reaches 56.5%, compared with 57.7% with the six-algorithm bank and 44.4% for Best. The six heterogeneous algorithm bank is significantly better on 4 datasets and the seed bank on 3, with no significant difference on the remaining 11. Fig. 7 (App. E.1) shows the arbiter switching among policies from different seeds within a cube-double-play episode. We agree that this is a strong baseline with comparable performance. However, it requires the best policy for each dataset to be selected in hindsight, whereas the six-algorithm bank does not.
AuthorsOpenReview ↗

Official Reply to Reviewer 1ecA [2/2]

W2/Q3 (continued).

"the experiments do not fully separate the value of heterogeneous policy behaviors from the more general benefit of evaluating multiple candidates with a suitable scorer"

Together these results separate the two. At the same model budget, choosing among complete policies is clearly ahead of generating candidates around one policy, and the search needs hindsight tuning and 20 times the calls to catch up, so the candidates matter and not only the scorer. The policies can differ in algorithm or in seed: WMPA works with both kinds of bank, and the bank of six algorithms reaches the same level without first identifying the best algorithm, a step the seed bank requires. We have replaced the sentence of Sec. 4.1 about "locally refining the actions of a single best policy" with these comparisons, and Sec. 4.3 now states that the members need not come from different algorithms.

In short, choosing among complete policies outperforms searching around the best policy at the same budget, and larger searches do not close the gap without hindsight tuning.

Q2. Success against wall-clock time.

"Could you extend the reported runtime comparison to include Q-select with per-step reselection and the action-search baseline from Table 6?"

We have added Table 18 (App. H) with the success rate and the time per environment step of WMPA, Q-select with c = 1, the MPC baseline from Table 6 and trajectory sampling on all 18 datasets, measured on the same CPU node. WMPA outperforms all three (macro-average 57.7%, compared with 45.2%, 52.7% and 52.0%) while taking only 2 ms per step on average. Figure 6 compares success with model calls on four manipulation datasets: WMPA is above Q-select and above every search that uses one setting for all datasets.

"distinguishing the modest puzzle gains from the larger Cube/Scene gains"

On the cube and scene families, WMPA outperforms Q-select, the model-free variant of our selector, by 11 and 26 percentage points at a cost of at most 1.0 ms more per step. On the puzzle family, where both selectors rank the policies with the same GCIQL agent, Q-select comes within 2 points of WMPA at a similar cost (1.1–2.1 ms per step for Q-select vs. 1.2–1.9 ms for WMPA). We have revised Sec. 4.2 to present Q-select as a strong option for applications with tighter compute budgets. Both selectors support our main finding that choosing among pretrained policies at test time outperforms the best single policy.

In short, WMPA has the highest success of the four methods in Table 18, at 2 ms per environment step on average.

“2: Generally understandable, but requires substantial effort in places.”

Presentation. We have added Table 2 to Sec. 4, which lists every control with the datasets it covers, its mean difference to WMPA and the number of datasets on which WMPA is significantly better or worse, and we have shortened the ablation text around it.

We hope these additions address your concerns; if they do, we would be grateful if you would consider raising your score. We would be glad to clarify anything further.

Dear Reviewer 1ecA,

As the discussion period ends soon, we would like to follow up on our response. We have added experiments on scorer selection using only training data, stronger single-policy search baselines with matched and larger compute budgets, and runtime comparisons across all 18 datasets.

We would appreciate hearing whether these results address your concerns, or if anything remains unclear. If your concerns have been resolved, we would be grateful if you would consider updating your score.

Thank you again for your time and feedback.

Reviewer_1ecAOpenReview ↗

Comment to author

I think the additional experiments address many of my questions, such as comparison with sampling multiple trajectories from a single policy, performance relative to inference cost. However, I still have the following concerns.

  1. Since different seeds of the same algorithm achieve similar performance, it is hard to say see using different algorithms is necessary.
  2. I still consider the dependence on a suitable scorer and its availability a critical issue. Particularly, it remains unclear whether the selected scorer actually identifies better policies. If future states are generated by the true simulator instead of the world model but evaluated with the same scorer, I interpret this experiment as checking how world-model errors affect policy selection. If the scorer itself evaluates policies incorrectly, agreement between the two methods does not guarantee that the selected policy is better.

For these reasons, I will maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.