- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a MicroRTS agent consisting of 1) a strategist that picks four-slot command tuples, with a bandit-inspired training strategy; and 2) a command-conditioned PPO executor, whose training strategy is trying to achieve a success rate of 80% for every command. Against a flat PPO baseline with the same architecture, budget, curriculum and league, it wins significantly more often against 3 baselines, but is indistinguishable against 9 others. Intervention tests show that only two of the four command slots change the behavior of the executor, and a fixed command tuple per opponent is as strong as the strategist. The authors conclude that the gain comes from their command-conditioned executor training rather than from strategy switching.
Critical strengths
- The authors are honest about the negative results found by their analysis.
- Against some opponents, the approach works very well. Win rates rise from 0.55 to 0.97 against obiBotKenobi and from 0.01 to 0.34 against RAISocketAI. Also, changing commands can have significant results: Using HEAVY instead of LIGHT turns 8 wins against coacAI into 8 losses. This means that the command interface is somehow useful, even if the strategist is not.
Critical weaknesses
- It's not quite clear what makes the approach stronger than flat PPO, because they have many more differences than just the command conditioning. There are hand-written success criteria with deadlines and thresholds, hand-written command preconditions and fallbacks, hand-written shaping rewards for each command, hand-written context summaries and evidence summaries/keys, and opponent identities seem to be given.
- Additionally, all evaluation games were on the same map with the agent as the same player (and with only one training run). As the authors show in Appendix H, player strength depends a lot on the choice of map, so using more maps from training as well as held-out ones (like in the microRTS competition) would be much more informative.
- Several components that are presented as contributions are not shown to work as intended. The full system is claied to "win more often than a flat PPO baseline against three of the four strongest opponents", but according to Table 10, the same is true of the "Static" baseline. Therefore, the strategist doesn't seem to be needed. The strategist is also only adapting against coacAI, and has many hand-written rules (see above) that aren't analyzed or ablated individually. There is no test of the adaptive Lagrangian weights vs. fixed weights. Two of four command slots don't have any effect. The presented "main lesson of the study" is not tested.
Questions for authors
- Can you add more training runs (seeds), both for the proposed agent and for the baseline?
- Can you add a baseline that also gets the shaped rewards, for example for LIGHT?
- Can you add a training run with fixed multipliers, to check if the "constrained" part of the title is needed?
- Can you add results on other maps (except against agent_sota), and/or with the agent as player 1?
- Can you report how many distinct trajectories the significant cells of Table 3 contain? The Fisher test assumes independent games.
2 comments
Thank you for the response. The additional seeds, the seat-1 evaluation and the trajectory check resolve my questions 1, 4 (seat) and 5: on map A, the advantage over flat PPO against the three strong opponents is reproducible. I also accept the clarification on opponent identities.
However, the new results narrow the contribution considerably. The shaped baseline shows that the gains against agent_sota and obiBotKenobi come from training under five fixed orders. The production head is still not ablated. The claim that schedules and the bandit are needed against RAISocketAI rests on one seed, and winrates against this opponent vary a lot between checkpoints of the same run (response 3), so I don't think the claim is established. On unseen maps the head-to-head advantage disappears, RAISocketAI is a tie, and Flat is ahead against the middle tier. You also report that adaptive multipliers are not shown to be necessary, that the deployed strategist is not needed, and that the remedy presented as the main lesson costs win rate.
What remains is that training under a fixed Light build order with command shaping beats flat PPO on the main training map, plus the diagnostic finding on success rates. This is a different paper from the one submitted: the title, abstract and contribution list would change, and the residual design is new material that I cannot evaluate from a summary. I therefore keep my rating, and I think the work would be much stronger as a resubmission built around the diagnostic result and the new design.
We appreciate the questions and answer them with a new run or analysis below.
W1: What makes it stronger than flat PPO. The two arms share the game reward, the network (Flat receives a constant command), the interaction budget, the curriculum, the league and the outcome critic. Everything else is listed below, with the run that removes it (final checkpoints, one seed per ablation):
The shaped baseline trains only under the orders the bandit issues during evaluation (STABILIZE, then GROW from tick 400; LIGHT; DEFENSE; ECON_ONLY). Only these two tuples were installed in its training plans, so no precondition repair ever occurred. It beats every Flat seed against agent_sota (.84 vs .03-.61) and obiBotKenobi (.98 vs .04-.55), beats Flat seed 1 head-to-head (90/100), and is significantly below a Bandit seed only against RAISocketAI (.03 vs .34-.58). Against agent_sota and obiBotKenobi the gain therefore comes from conditioned training under these five orders; the generated schedules, context summaries, evidence keys, opponent identities, preconditions and the bandit are not needed for it. Against RAISocketAI the generated schedules and the bandit are needed. We cannot say which part of the training under the five orders matters: their criteria and shaping were not varied one at a time. We will add the table to the paper.
W1: Opponent identities. Identity is only a training label for the payoff table. Decisions use a belief computed from scouting evidence. Only the Static baseline uses identity.
W2: One map, one seat, one run. Seeds: Q1. Player 1: Q4. Maps: as Appendix H suggests, the gains move between opponents on the other training maps. On D the Bandit leads against TMA (.87 vs .13) but not against RAISocketAI (.12 vs .37) and head-to-head (35/98); on I it leads against RAISocketAI (.15 vs .04) and head-to-head (93/100) but trails against TMA (.46 vs .76). Flat leads against the middle tier on both. Unseen maps: see common response 4. All maps are 16x16.
W3: The strategist is not needed. Agreed. Static, one fixed tuple per opponent, keeps the Bandit's gain over Flat in Table 3 (seed 1): RAISocketAI .47 vs .01, agent_sota .90 vs .61, obiBotKenobi 1.00 vs .55, head-to-head 99/100. Each of these four is a LIGHT tuple. During training the bandit does matter: with one fixed schedule the late RAISocketAI gain disappears (response 2). We will remove the gain from the strategist during evaluation from the claimed contributions and present the bandit as the training mechanism that found the orders.
Q1 seeds. Common response 1. Q2 shaped rewards. Common response 2. Q3 fixed multipliers. Common response 3. In the residual design (common response 5), freezing the multipliers lowers posture influence from .41--.45 to .27, so adaptation matters there for control. In the submitted system it affects win rate mainly against TMA and RAISocketAI (one fixed seed).
Q4 maps and player 1. Maps: common response 4. Player 1: we added a seat-1 evaluator (its default mode reproduces our player-0 games exactly; the mirroring was verified in-engine) and reran Table 3's final pair as player 1. With seats swapped, the Bandit still beats Flat in all three seeds (95, 95, 98 of 100; 97, 99, 97 as player 0), and against the 12 opponents (seed 1) the three significant cells replicate (RAISocketAI .31 vs .00, agent_sota .89 vs .43, obiBotKenobi .98 vs .49). Both agents lose some strength as player 1 (Bandit vs mayari .98 to .80, vs droplet .96 to .85, where Flat is now ahead; Flat vs TMA 1.00 to .87), consistent with environment asymmetry: the same flat policy wins .68 against itself from the player 0 seat.
Q5 distinct trajectories. In every significant cell of Table 3 all 100 games are distinct, for Flat, Bandit and Static (distinct = different sequence of unit creations and deaths with tick, type and position, or different length or result)