acceptodds
Withdrawn from ICLR 2027

Online World-Model-Boosted Planning for Multi-Agent Quadruped Soccer

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Learning coordinated behaviors in multi-agent systems remains challenging due to sparse rewards, strong inter-agent coupling, and large joint action spaces, which make model-free exploration highly sample-inefficient. We study this problem in multi-quadruped robot soccer, which is a representative and commonly-used benchmark. Specifically, each robot selects among pretrained repositioning, dribbling, and kicking skills together with continuous command parameters while competing against mobile opponents through self-play. We propose an online model-based multi-agent reinforcement learning framework in which model predictive control (MPC) method serves as a continually improving training-time teacher for decentralized high-level policies. A skill-level world model is learned online from self-play transitions and used by an MPC to optimize hybrid skill selection and continuous skill parameters. To compensate for the limited planning horizon, a learned centralized value function provides long-term guidance during action evaluation. The resulting MPC decisions are subsequently distilled into decentralized high-level policies for efficient execution. This creates a closed-loop learning process in which evolving policies generate increasingly relevant data for world-model learning, while the improving model provides progressively stronger planning targets for policy optimization. Experiments against model-free Multi-Agent Proximal Policy Optimization (MAPPO) and ablations show that the proposed method improves environment sample efficiency and overall policy performance. The project website is available at https://anonymous.4open.science/w/skills2strategy.

Primary area:
applications to robotics, autonomy, planning
Keywords:
MARL, world models, MPC, quadruped soccer

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_rjrPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies high-level strategy learning in quadruped robot soccer. It uses pretrained, frozen locomotion, dribbling, and kicking skills, predicts skill-level state transitions online with a probabilistic ensemble world model, and then uses a hybrid-action CEM-MPC with terminal value and uncertainty penalty to generate a teacher distribution. This distribution serves as an auxiliary distillation target for MAPPO; environment interaction is still generated by the current policy, and deployment requires neither the world model nor the planner.

The overall framework is reasonable, but the method innovation and experimental persuasion of the current manuscript are seriously insufficient, and the main components are mostly combinations of existing ideas

Critical strengths
  1. The training/deployment split is sound, since the centralized planner only provides supervision while deployment uses decentralized policies and no online planning.

  2. Some design choices have preliminary support, including continuous-vs-discrete MAPPO comparisons and ablations over terminal value and uncertainty penalty.

Critical weaknesses
  1. The method largely combines existing components, and it would help to clarify what new problem the setting raises or what substantive new mechanism is proposed. 
  2. The large win rate over discrete MAPPO may mainly reflect the benefit of continuous commands, since continuous MAPPO already reaches 92.43% against the same baseline. 
  3. The ablations could be strengthened, as Table 3 and Table 2 report only prediction error and rollout time without corresponding policy performance. 
  4. Some key training and evaluation details are missing, including model architecture, ensemble size, CEM settings, warm-up, distillation weight, opponent-pool construction, and draw handling. 
  5. No end-to-end training time or compute-matched comparison is reported, so a training-efficiency advantage is not yet established. 
  6. The experiments are confined to a fixed 2-vs-2 simulation with no real-robot deployment, and the sparse-reward motivation sits uneasily with the extensive shaping rewards in Appendix Table 5.
Questions for authors
  1. Could the authors clarify what specific new algorithmic mechanism this paper adds beyond existing value-guided MPC and planner-distillation frameworks?

  2. Would it be possible to check whether the prediction-error improvements in Table 3 actually translate into better final win rates and learning curves?

  3. Does the advantage over continuous MAPPO still hold under matched total training time or compute budget?

  4. How is the fixed evaluation opponent pool built, and could win/draw/loss ratios and unseen-opponent results be reported?

  5. Given that the work targets real quadruped soccer, could the authors comment on the prospects and challenges of deploying the learned policies on physical robots?

1 comment

Thank you for the constructive feedback. References below refer to the revised manuscript.

Q1. Methodological contribution: Our contribution is a training framework for joint hybrid-action planning and decentralized learning in competitive self-play, rather than a new CEM or distillation primitive. The teacher jointly evaluates teammates' skill choices and continuous commands under opponent responses, while simulator-generated actions and returns preserve MAPPO training. The joint-versus-independent teacher control isolates the benefit of joint search: their decentralized students achieve 80.0%80.0\% versus 72.0%72.0\% at 40M transitions (Table 6). Sections 1--3 clarify this scope.

Q2. Prediction quality versus policy performance: Table 6 reports win rates at 10M/20M/40M transitions: Frozen 48.0/60.0/68.0%48.0/60.0/68.0\%, Online-only 44.0/57.0/64.0%44.0/57.0/64.0\%, and Replay 60.0/76.7/80.0%60.0/76.7/80.0\%. Table 7 additionally links replay to better candidate ranking (ρ=0.82\rho=0.82 versus 0.61/0.540.61/0.54) and lower selection regret (0.080.08 versus 0.19/0.240.19/0.24). Recent/historical errors are 10.2/26.010.2/26.0,cm for Online-only versus 10.4/12.810.4/12.8,cm for Replay (Appendix C.1), indicating poorer historical-state prediction with recent-only training; this is not a direct measurement of forgetting.

Q3. Matched-time or compute-budget advantage: We do not claim a uniform compute-efficiency advantage. Appendix C.3 accounts for approximately 24.224.2 versus 8.08.0 hours at 40M transitions for Full and MAPPO. Under the accounting-based budgets in Table 11, their fixed-pool win rates are 80.0%80.0\% versus 78.0%78.0\% at 24.224.2 hours, and 65.3%65.3\% versus 70.0%70.0\% at 88 hours. These accounting-based comparisons do not establish a profiler-validated wall-time advantage; the stronger evidence is improved interaction efficiency.

Q4. Opponent pool, draws, and unseen opponents: The shared fixed pool comprises MAPPO checkpoints at 8/16/24/32/40M from seeds 100 and 101. Unseen opponents use seeds 200 and 201 and are excluded from training, replay, and selection (Appendix C.2). At 40M, unseen-opponent W/D/L is 75/8/17%75/8/17\% for Full versus 65/8/27%65/8/27\% for MAPPO (Table 11); Table 15 provides per-seed counts.

We distinguish all-match W/(W+D+L)W/(W+D+L) from decisive-match W/(W+L)W/(W+L) rates. The additional head-to-head evaluation gives W/D/L =510/143/347=510/143/347 against MAPPO: 51.0/14.3/34.7%51.0/14.3/34.7\% over all matches and a seed-mean decisive rate of 59.50±1.28%59.50\pm1.28\% (sample SD; Table 9). This is an independent evaluation, not a reconstruction of Table 1.

Q5. Physical deployment and scope: Our evaluation is limited to 2-vs-2 simulation. Deployment requires only decentralized actors and frozen low-level skills, without online planning. As future work, we aim to validate the policies on physical robots using motion capture for initial state estimation while preserving each actor's local observation inputs.

Reviewer_yho2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies simulated 2v2 quadruped soccer using pretrained repositioning, dribbling, and kicking skills with continuous command parameters. It learns a probabilistic ensemble world model conditioned on the global state and both teams' joint actions. A hybrid CEM planner evaluates candidate actions using predicted rewards, ensemble disagreement, a centralized terminal-value critic, and responses from a frozen historical opponent. The resulting first-step action distributions provide auxiliary KL supervision for decentralized MAPPO actors.

Environment actions and PPO training targets remain based on simulator experience, and deployment requires only the actors and pretrained skills. Experiments report faster early learning and a 59.47% +/- 0.92% win rate against continuous-action MAPPO, excluding draws. Additional experiments examine prediction errors and ablate terminal-value guidance and uncertainty penalization.

Critical strengths
  1. The training-time planning architecture is conceptually sensible. Modeling interactions at the skill timescale avoids directly predicting joint-level quadruped dynamics, while conditioning on both teams' actions separates physical prediction from opponent action generation. Distilling planning targets into decentralized actors also provides a practical way to use centralized planning during training without requiring it at deployment.

  2. The comparison with continuous-action MAPPO is relevant and provides encouraging evidence for the complete training system. Table 1 reports results over ten seeds with 100 matches per seed, while Figure 4 suggests faster early learning. Figure 5 also provides useful evidence that terminal-value guidance and uncertainty penalization affect performance within the evaluated budget. These are substantive positive results, although their interpretation requires the controls and reporting discussed below.

Critical weaknesses
  1. The planner and training procedure are insufficiently specified to assess important aspects of correctness. The world model explicitly predicts episode termination in Eqs. (3)-(6), but the planning objective in Eq. (13) contains neither termination masking nor an explicitly stated stopping rule. As written, it accumulates rewards for a fixed horizon and bootstraps from the endpoint even if an imagined episode has already terminated. The manuscript needs to explain how predicted terminations affect subsequent rollout rewards and terminal-value bootstrapping. This is consequential in a task involving goals and accidental terminations; it may be an omitted implementation detail, but the current formulation does not resolve it.

Critic training is also unclear. Algorithm 1 explicitly updates the critic during warm-up, whereas the subsequent branch explicitly describes only the actor update. Line 13 ambiguously mentions the critic in the world-model update, although Eq. (6) does not involve it. The critic's post-warm-up loss, targets, training data, and update schedule should be specified. The manuscript additionally omits essential configuration details, including the actual high-level decision interval, default planning horizon, ensemble size, CEM budget, replay mixture, and distillation schedule. These omissions prevent reliable reconstruction of the central algorithm.

  1. The experiments do not sufficiently isolate the incremental contribution of online world-model planning. Continuous-action MAPPO is an appropriate starting point, but the teacher receives an additional task-specific feasibility prior: Section 3.2 restricts robots far from the ball to repositioning, explicitly only during teacher sampling. Distillation can therefore teach actors to avoid invalid skill requests independently of the world model's ability to predict interactions. A control giving MAPPO the same feasibility prior, together with a teacher-mask ablation, is needed to distinguish these effects.

Table 3 does not close this attribution gap: it reports prediction errors and ensemble disagreement, rather than final policy performance for frozen and recent-only model variants. Lower position error does not establish improved planning or stronger decentralized policies, and lower disagreement alone does not establish calibrated uncertainty. Moreover, the recent-only model has higher position error than the frozen model, so the claim that recent updates are necessary is stronger than the presented evidence supports.

This matters because the principal components - ensemble dynamics models, sampling-based planning, terminal-value bootstrapping, and policy distillation - are established techniques. The paper needs stronger controlled evidence to establish the value of their particular integration. Finally, the head-to-head results omit draws and do not define whether the reported variability is a standard deviation, standard error, or confidence interval. The reported advantage is promising, but its magnitude and reliability remain incompletely characterized.

Questions for authors
  1. Planner and critic implementation. How are predicted episode terminations used in Eq. (13)? Please specify the rollout stopping or survival-masking rule and when terminal-value bootstrapping is disabled. How is the critic trained after warm-up, and what exactly does Algorithm 1, line 13 mean? Please distinguish any notation omissions from differences in the implemented algorithm.

  2. Controls for the claimed contribution. Can you report a continuous-action MAPPO control with the same skill-feasibility prior, an ablation removing the teacher-only sampling restriction, and task-level learning curves/final performance for the Frozen and recent-only models in Table 3? These comparisons would materially affect whether the gains can be attributed to the proposed online modeling and planning mechanism.

  3. Evaluation statistics. Please provide per-seed W/D/L counts, the exact win-rate denominator, and the meaning of the reported variability and shaded regions. Report both all-match outcomes and decisive-match conditional win rates. How was the fixed evaluation opponent pool constructed, and was it shared across the main comparison and ablations?

  4. Baseline equivalence and reproducibility. Which observations, recurrent architectures, rewards, pretrained skills, opponent-population procedures, and training budgets are shared across methods? For the Su et al. adaptation, which parts of its self-play and policy design were retained or changed? Please also provide the complete training/planning configuration and end-to-end training cost; Table 2's milliseconds per unspecified batch do not characterize the total computational overhead.

1 comment

Thank you for the constructive feedback. References below refer to the revised manuscript.

Q1. Planner termination and critic training: Equation (13) now specifies b0=1b_0=1 and bh+1=bh(1−d^t+h)b_{h+1}=b_h(1-\hat d_{t+h}), with d^=1[pd≥0.5]\hat d=\mathbf{1}[p_d\geq0.5]. The terminating transition retains its reward; subsequent terms vanish, and bH=0b_H=0 disables terminal-value bootstrapping. The critic is updated after every rollout, including after warm-up, using simulator-based GAE targets R^t=A^t+Vψold(st)\hat R_t=\hat A_t+V_{\psi_{\rm old}}(s_t) and the normalized loss in Eq. (21), for ten epochs. Algorithm 1 separates critic updates from world-model fitting; imagined transitions do not supply critic targets. Appendix E specifies recurrent-state handling and value normalization. These mechanisms were used in the original experiments; the revision corrects omissions in their description.

Q2. Feasibility-prior and model-update controls: At matched 40M transitions, Full reaches 80.0%80.0\%, versus 70.0%70.0\% for continuous MAPPO, 72.0%72.0\% for MAPPO with the same feasibility prior, and 77.0%77.0\% without the teacher mask (Table 6). Thus, the prior contributes to performance but does not explain the full mean improvement. These are descriptive differences across ten seeds, not significance claims based on mean gaps alone.

Table 6 reports win rates at 10M/20M/40M transitions: Frozen 48.0/60.0/68.0%48.0/60.0/68.0\%, Online-only 44.0/57.0/64.0%44.0/57.0/64.0\%, and Replay 60.0/76.7/80.0%60.0/76.7/80.0\%. Table 7 additionally links replay to better candidate ranking (ρ=0.82\rho=0.82 versus 0.61/0.540.61/0.54) and lower selection regret (0.080.08 versus 0.19/0.240.19/0.24). Recent/historical errors are 10.2/26.010.2/26.0,cm for Online-only versus 10.4/12.810.4/12.8,cm for Replay (Appendix C.1), indicating poorer historical-state prediction with recent-only training; this is not a direct measurement of forgetting. Ensemble disagreement is used as a planning regularizer, not interpreted as calibrated uncertainty.

Q3. Evaluation statistics and opponent pool: We distinguish all-match W/(W+D+L)W/(W+D+L) from decisive-match W/(W+L)W/(W+L) rates. The additional head-to-head evaluation gives W/D/L =510/143/347=510/143/347 against MAPPO: 51.0/14.3/34.7%51.0/14.3/34.7\% over all matches and a seed-mean decisive rate of 59.50±1.28%59.50\pm1.28\% (sample SD; Table 9). This is an independent evaluation, not a reconstruction of Table 1.

The shared fixed pool comprises MAPPO checkpoints at 8/16/24/32/40M from seeds 100 and 101. Unseen opponents use seeds 200 and 201 and are excluded from training, replay, and selection (Appendix C.2). At 40M, unseen-opponent W/D/L is 75/8/17%75/8/17\% for Full versus 65/8/27%65/8/27\% for MAPPO (Table 11); Table 15 provides per-seed counts.

Q4. Baseline equivalence, configuration, and cost: Appendices C.1--C.3 specify shared observations, recurrent architectures, rewards, frozen skills, self-play procedures, seeds, and the 40M control budget. The Su et al.\ adaptation retains recurrent MAPPO and historical self-play but uses eight fixed directions and fixed command magnitudes. Table 10 specifies the 0.10.1,s decision interval, H=4H=4, five models, CEM with 256 candidates/32 elites/four iterations, 50:50 replay, and distillation schedule. Table 13's 41.941.9,ms is one model evaluation of 256 candidates over five ensemble members from one state, not total training time.

Please note that we do not claim a uniform compute-efficiency advantage. Appendix C.3 accounts for approximately 24.224.2 versus 8.08.0 hours at 40M transitions for Full and MAPPO. Under the accounting-based budgets in Table 11, their fixed-pool win rates are 80.0%80.0\% versus 78.0%78.0\% at 24.224.2 hours, and 65.3%65.3\% versus 70.0%70.0\% at 88 hours. These accounting-based comparisons do not establish a profiler-validated wall-time advantage; the stronger evidence is improved interaction efficiency.

Reviewer_ZjctOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors propose an online world-model-based reinforcement learning framework for multi-agent quadruped soccer. the proposed approach combines online world-model learning, hybrid CEM-MPC, terminal-value guidance, and uncertainty penalization for joint action planning. The resulting MPC decisions are distilled into decentralized MAPPO policies through an auxiliary KL objective. The experimental results show the effectiveness of improving early-stage sample efficiency and a 59.47% win rate against continuous-action MAPPO in head-to-head matches.

Critical strengths
  1. Multi-agent quadruped soccer involves complex physical interactions, team coordination, and hybrid action spaces, making it a meaningful and challenging setting for reinforcement learning.

  2. The integration of online world-model learning, joint MPC planning, and policy distillation provides a coherent training framework that leverages planning-based supervision while retaining decentralized execution.

  3. The proposed method improves early-stage learning efficiency and achieves better final performance than MAPPO. The comparisons of world-model update strategies and the ablations on terminal-value guidance and uncertainty penalization also provide evidence supporting several key design choices.

Critical weaknesses
  1. World-model ensembles, CEM, terminal-value guidance, uncertainty penalization, and policy distillation have all been studied in prior work. The paper primarily integrates these techniques in the context of multi-agent soccer, but it does not sufficiently demonstrate a distinctive algorithmic mechanism or new methodological insight arising from this combination.

  2. The main baselines are MAPPO and its discrete-action variant, whereas the proposed method additionally benefits from world-model learning, MPC planning, and auxiliary supervision. Although the results demonstrate the effectiveness of the complete framework, they do not establish whether its planning and distillation design outperforms simpler model-based or planner-guided alternatives. Furthermore, Table 1 shows that continuous-action MAPPO already achieves a 92.43% win rate against discrete MAPPO, compared with 93.72% for the proposed method. This suggests that much of the advantage over the discrete baseline may be attributable to the action-space formulation rather than the proposed planning framework.

  3. Table 3 shows that Online + Historical Replay achieves lower prediction error than Frozen and Online-only models, supporting the value of historical replay. However, the comparison focuses solely on prediction quality and does not investigate how these update strategies affect final policy win rates or sample efficiency. Lower prediction error does not necessarily translate into better planning performance. What matters is whether improved predictions lead to more accurate candidate-action evaluation and more effective policy supervision. Moreover, the Online-only model performs worse than the Frozen model, an observation that deserves further analysis.

  4. The paper emphasizes that joint MPC can improve team coordination, but the evaluation relies primarily on overall win rates and lacks a direct comparison with independent per-agent MPC. Moreover, the centralized MPC teacher optimizes joint actions using global information, while the distilled decentralized policies act based on local observations. The paper does not sufficiently investigate how much of the coordination benefit is preserved through this distillation process.

Questions for authors
  1. What is the primary methodological novelty of the proposed framework relative to existing value-guided MPC and planner-distillation methods?

  2. Could the authors include simpler model-based MAPPO or planner-guided MAPPO baselines to demonstrate the advantages of the proposed planning and distillation framework over alternative approaches?

  3. Could the authors report policy win rates or learning curves for Frozen, Online-only, and Online + Historical Replay world models? Additionally, how do the authors explain the worse prediction performance of Online-only compared with Frozen?

  4. Could the authors compare joint MPC with independent per-agent MPC, or provide quantitative coordination metrics, to further demonstrate the contribution of joint planning and its distillation to team coordination?

2 comments

Thank you for the constructive feedback. References below refer to the revised manuscript.

Q1. Methodological contribution: Our contribution is a training framework for joint hybrid-action planning and decentralized learning in competitive self-play, rather than a new CEM or distillation primitive. The teacher jointly evaluates teammates' skill choices and continuous commands under opponent responses, while simulator-generated actions and returns preserve MAPPO training. The joint-versus-independent teacher control isolates the benefit of joint search: their decentralized students achieve 80.0%80.0\% versus 72.0%72.0\% at 40M transitions (Table 6). Sections 1--3 clarify this scope.

Q2. Simpler baselines and action-space effects: Table 6 adds a one-step teacher with the full model-evaluation budget reallocated to candidates, reward-only MPC, and independent MPC. At 40M transitions, their students achieve 73.0%73.0\%, 55.0%55.0\%, and 72.0%72.0\%, versus 80.0%80.0\% for Full. We agree that superiority over discrete MAPPO largely reflects continuous commands; attribution therefore rests on continuous-action controls, not the discrete-baseline gap.

Q3. World-model updates and downstream performance: Table 6 reports win rates at 10M/20M/40M transitions: Frozen 48.0/60.0/68.0%48.0/60.0/68.0\%, Online-only 44.0/57.0/64.0%44.0/57.0/64.0\%, and Replay 60.0/76.7/80.0%60.0/76.7/80.0\%. Table 7 additionally links replay to better candidate ranking (ρ=0.82\rho=0.82 versus 0.61/0.540.61/0.54) and lower selection regret (0.080.08 versus 0.19/0.240.19/0.24). Recent/historical errors are 10.2/26.010.2/26.0,cm for Online-only versus 10.4/12.810.4/12.8,cm for Replay (Appendix C.1), indicating poorer historical-state prediction with recent-only training; this is not a direct measurement of forgetting.

Q4. Joint planning and coordination transfer: The joint- and independent-teacher students use identical local observations and matched teacher model-evaluation budgets, achieving 80.0%80.0\% versus 72.0%72.0\% (Table 6). Table 8 directly measures coordination: joint planning reduces student obstruction from 13.0%13.0\% to 7.0%7.0\% and increases recovery from 49.0%49.0\% to 61.0%61.0\%. The joint teacher itself achieves 6.0%/64.0%6.0\%/64.0\%. Thus, students preserve benefits on these two metrics; we do not claim exact recovery of privileged joint behavior.

Thanks for the response. My concerns about the main weaknesses remain. I will keep the score unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.