- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies high-level strategy learning in quadruped robot soccer. It uses pretrained, frozen locomotion, dribbling, and kicking skills, predicts skill-level state transitions online with a probabilistic ensemble world model, and then uses a hybrid-action CEM-MPC with terminal value and uncertainty penalty to generate a teacher distribution. This distribution serves as an auxiliary distillation target for MAPPO; environment interaction is still generated by the current policy, and deployment requires neither the world model nor the planner.
The overall framework is reasonable, but the method innovation and experimental persuasion of the current manuscript are seriously insufficient, and the main components are mostly combinations of existing ideas
Critical strengths
-
The training/deployment split is sound, since the centralized planner only provides supervision while deployment uses decentralized policies and no online planning.
-
Some design choices have preliminary support, including continuous-vs-discrete MAPPO comparisons and ablations over terminal value and uncertainty penalty.
Critical weaknesses
- The method largely combines existing components, and it would help to clarify what new problem the setting raises or what substantive new mechanism is proposed.
- The large win rate over discrete MAPPO may mainly reflect the benefit of continuous commands, since continuous MAPPO already reaches 92.43% against the same baseline.
- The ablations could be strengthened, as Table 3 and Table 2 report only prediction error and rollout time without corresponding policy performance.
- Some key training and evaluation details are missing, including model architecture, ensemble size, CEM settings, warm-up, distillation weight, opponent-pool construction, and draw handling.
- No end-to-end training time or compute-matched comparison is reported, so a training-efficiency advantage is not yet established.
- The experiments are confined to a fixed 2-vs-2 simulation with no real-robot deployment, and the sparse-reward motivation sits uneasily with the extensive shaping rewards in Appendix Table 5.
Questions for authors
-
Could the authors clarify what specific new algorithmic mechanism this paper adds beyond existing value-guided MPC and planner-distillation frameworks?
-
Would it be possible to check whether the prediction-error improvements in Table 3 actually translate into better final win rates and learning curves?
-
Does the advantage over continuous MAPPO still hold under matched total training time or compute budget?
-
How is the fixed evaluation opponent pool built, and could win/draw/loss ratios and unseen-opponent results be reported?
-
Given that the work targets real quadruped soccer, could the authors comment on the prospects and challenges of deploying the learned policies on physical robots?
Thank you for the constructive feedback. References below refer to the revised manuscript.
Q1. Methodological contribution: Our contribution is a training framework for joint hybrid-action planning and decentralized learning in competitive self-play, rather than a new CEM or distillation primitive. The teacher jointly evaluates teammates' skill choices and continuous commands under opponent responses, while simulator-generated actions and returns preserve MAPPO training. The joint-versus-independent teacher control isolates the benefit of joint search: their decentralized students achieve 80.0% versus 72.0% at 40M transitions (Table 6). Sections 1--3 clarify this scope.
Q2. Prediction quality versus policy performance: Table 6 reports win rates at 10M/20M/40M transitions: Frozen 48.0/60.0/68.0%, Online-only 44.0/57.0/64.0%, and Replay 60.0/76.7/80.0%. Table 7 additionally links replay to better candidate ranking (ρ=0.82 versus 0.61/0.54) and lower selection regret (0.08 versus 0.19/0.24). Recent/historical errors are 10.2/26.0,cm for Online-only versus 10.4/12.8,cm for Replay (Appendix C.1), indicating poorer historical-state prediction with recent-only training; this is not a direct measurement of forgetting.
Q3. Matched-time or compute-budget advantage: We do not claim a uniform compute-efficiency advantage. Appendix C.3 accounts for approximately 24.2 versus 8.0 hours at 40M transitions for Full and MAPPO. Under the accounting-based budgets in Table 11, their fixed-pool win rates are 80.0% versus 78.0% at 24.2 hours, and 65.3% versus 70.0% at 8 hours. These accounting-based comparisons do not establish a profiler-validated wall-time advantage; the stronger evidence is improved interaction efficiency.
Q4. Opponent pool, draws, and unseen opponents: The shared fixed pool comprises MAPPO checkpoints at 8/16/24/32/40M from seeds 100 and 101. Unseen opponents use seeds 200 and 201 and are excluded from training, replay, and selection (Appendix C.2). At 40M, unseen-opponent W/D/L is 75/8/17% for Full versus 65/8/27% for MAPPO (Table 11); Table 15 provides per-seed counts.
We distinguish all-match W/(W+D+L) from decisive-match W/(W+L) rates. The additional head-to-head evaluation gives W/D/L =510/143/347 against MAPPO: 51.0/14.3/34.7% over all matches and a seed-mean decisive rate of 59.50±1.28% (sample SD; Table 9). This is an independent evaluation, not a reconstruction of Table 1.
Q5. Physical deployment and scope: Our evaluation is limited to 2-vs-2 simulation. Deployment requires only decentralized actors and frozen low-level skills, without online planning. As future work, we aim to validate the policies on physical robots using motion capture for initial state estimation while preserving each actor's local observation inputs.