acceptodds
Under review as a conference paper at ICLR 2027

Constrained Command-Conditioned Reinforcement Learning with Bandit Strategy Selection in Real-Time Strategy Games

Abstract

Deep reinforcement learning agents reach strong performance in real-time strategy games but can be brittle against opponents outside their training distribution. Separating strategic command selection from learned unit control allows different strategies to be selected for different opponents while reusing the same execution policy. This requires an executor that can follow different commands and measurable criteria for assessing whether it does so. We introduce a constrained command-conditioned Proximal Policy Optimization (PPO) policy, the executor, for MicroRTS, a real-time strategy environment. Discrete commands specify strategic objectives and behavioral requirements for economy, army composition, military posture, and worker policy over multiple environment steps; the executor determines the unit-level actions used to fulfill them. A Thompson-sampling bandit acts as the strategist, selecting command tuples from an estimate of the opponent's strategy built from in-game observations rather than opponent identity. In a controlled comparison with a flat PPO baseline trained with the same architecture, budget, curriculum and self-play league, the strategist–executor system wins significantly more often against three of the four strongest opponents on a training map, including the two strongest held-out ones (0.55 → 0.97 and 0.01 → 0.34), with no significant difference against the others.

Primary area:
reinforcement learning
Keywords:
Constrained reinforcement learning, Command-conditioned policies, Hierarchical reinforcement learning, Bandit strategy selection, Opponent adaptation, Real-time strategy games

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_31PQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a MicroRTS agent consisting of 1) a strategist that picks four-slot command tuples, with a bandit-inspired training strategy; and 2) a command-conditioned PPO executor, whose training strategy is trying to achieve a success rate of 80% for every command. Against a flat PPO baseline with the same architecture, budget, curriculum and league, it wins significantly more often against 3 baselines, but is indistinguishable against 9 others. Intervention tests show that only two of the four command slots change the behavior of the executor, and a fixed command tuple per opponent is as strong as the strategist. The authors conclude that the gain comes from their command-conditioned executor training rather than from strategy switching.

Critical strengths
  • The authors are honest about the negative results found by their analysis.
  • Against some opponents, the approach works very well. Win rates rise from 0.55 to 0.97 against obiBotKenobi and from 0.01 to 0.34 against RAISocketAI. Also, changing commands can have significant results: Using HEAVY instead of LIGHT turns 8 wins against coacAI into 8 losses. This means that the command interface is somehow useful, even if the strategist is not.
Critical weaknesses
  • It's not quite clear what makes the approach stronger than flat PPO, because they have many more differences than just the command conditioning. There are hand-written success criteria with deadlines and thresholds, hand-written command preconditions and fallbacks, hand-written shaping rewards for each command, hand-written context summaries and evidence summaries/keys, and opponent identities seem to be given.
  • Additionally, all evaluation games were on the same map with the agent as the same player (and with only one training run). As the authors show in Appendix H, player strength depends a lot on the choice of map, so using more maps from training as well as held-out ones (like in the microRTS competition) would be much more informative.
  • Several components that are presented as contributions are not shown to work as intended. The full system is claied to "win more often than a flat PPO baseline against three of the four strongest opponents", but according to Table 10, the same is true of the "Static" baseline. Therefore, the strategist doesn't seem to be needed. The strategist is also only adapting against coacAI, and has many hand-written rules (see above) that aren't analyzed or ablated individually. There is no test of the adaptive Lagrangian weights vs. fixed weights. Two of four command slots don't have any effect. The presented "main lesson of the study" is not tested.
Questions for authors
  • Can you add more training runs (seeds), both for the proposed agent and for the baseline?
  • Can you add a baseline that also gets the shaped rewards, for example for LIGHT?
  • Can you add a training run with fixed multipliers, to check if the "constrained" part of the title is needed?
  • Can you add results on other maps (except against agent_sota), and/or with the agent as player 1?
  • Can you report how many distinct trajectories the significant cells of Table 3 contain? The Fisher test assumes independent games.
2 comments

We appreciate the questions and answer them with a new run or analysis below.

W1: What makes it stronger than flat PPO. The two arms share the game reward, the network (Flat receives a constant command), the interaction budget, the curriculum, the league and the outcome critic. Everything else is listed below, with the run that removes it (final checkpoints, one seed per ablation):

ComponentFlatShaped baselineBanditResult when removed
Success criteria (deadlines, one 0.8 target) and per-command shaping rewardsno5 orders16 commandsnot ablated separately
Adaptive multipliers and guardnoyesyesfixed at 0.3: lower only against TMA and RAISocketAI (response 3)
Production head and auxiliary lossnoyesyesnot ablated (eZZW Q1)
Preconditions and fallbacksnopresent, never triggeredyesnot needed (shaped baseline)
Generated schedules; bandit with context and evidence keys; opponent identity as a training labelnonoyesshaped baseline: lower only against RAISocketAI (response 2)

The shaped baseline trains only under the orders the bandit issues during evaluation (STABILIZE, then GROW from tick 400; LIGHT; DEFENSE; ECON_ONLY). Only these two tuples were installed in its training plans, so no precondition repair ever occurred. It beats every Flat seed against agent_sota (.84 vs .03-.61) and obiBotKenobi (.98 vs .04-.55), beats Flat seed 1 head-to-head (90/100), and is significantly below a Bandit seed only against RAISocketAI (.03 vs .34-.58). Against agent_sota and obiBotKenobi the gain therefore comes from conditioned training under these five orders; the generated schedules, context summaries, evidence keys, opponent identities, preconditions and the bandit are not needed for it. Against RAISocketAI the generated schedules and the bandit are needed. We cannot say which part of the training under the five orders matters: their criteria and shaping were not varied one at a time. We will add the table to the paper.

W1: Opponent identities. Identity is only a training label for the payoff table. Decisions use a belief computed from scouting evidence. Only the Static baseline uses identity.

W2: One map, one seat, one run. Seeds: Q1. Player 1: Q4. Maps: as Appendix H suggests, the gains move between opponents on the other training maps. On D the Bandit leads against TMA (.87 vs .13) but not against RAISocketAI (.12 vs .37) and head-to-head (35/98); on I it leads against RAISocketAI (.15 vs .04) and head-to-head (93/100) but trails against TMA (.46 vs .76). Flat leads against the middle tier on both. Unseen maps: see common response 4. All maps are 16x16.

W3: The strategist is not needed. Agreed. Static, one fixed tuple per opponent, keeps the Bandit's gain over Flat in Table 3 (seed 1): RAISocketAI .47 vs .01, agent_sota .90 vs .61, obiBotKenobi 1.00 vs .55, head-to-head 99/100. Each of these four is a LIGHT tuple. During training the bandit does matter: with one fixed schedule the late RAISocketAI gain disappears (response 2). We will remove the gain from the strategist during evaluation from the claimed contributions and present the bandit as the training mechanism that found the orders.

Q1 seeds. Common response 1. Q2 shaped rewards. Common response 2. Q3 fixed multipliers. Common response 3. In the residual design (common response 5), freezing the multipliers lowers posture influence from .41--.45 to .27, so adaptation matters there for control. In the submitted system it affects win rate mainly against TMA and RAISocketAI (one fixed seed).

Q4 maps and player 1. Maps: common response 4. Player 1: we added a seat-1 evaluator (its default mode reproduces our player-0 games exactly; the mirroring was verified in-engine) and reran Table 3's final pair as player 1. With seats swapped, the Bandit still beats Flat in all three seeds (95, 95, 98 of 100; 97, 99, 97 as player 0), and against the 12 opponents (seed 1) the three significant cells replicate (RAISocketAI .31 vs .00, agent_sota .89 vs .43, obiBotKenobi .98 vs .49). Both agents lose some strength as player 1 (Bandit vs mayari .98 to .80, vs droplet .96 to .85, where Flat is now ahead; Flat vs TMA 1.00 to .87), consistent with environment asymmetry: the same flat policy wins .68 against itself from the player 0 seat.

Q5 distinct trajectories. In every significant cell of Table 3 all 100 games are distinct, for Flat, Bandit and Static (distinct = different sequence of unit creations and deaths with tick, type and position, or different length or result)

Thank you for the response. The additional seeds, the seat-1 evaluation and the trajectory check resolve my questions 1, 4 (seat) and 5: on map A, the advantage over flat PPO against the three strong opponents is reproducible. I also accept the clarification on opponent identities.

However, the new results narrow the contribution considerably. The shaped baseline shows that the gains against agent_sota and obiBotKenobi come from training under five fixed orders. The production head is still not ablated. The claim that schedules and the bandit are needed against RAISocketAI rests on one seed, and winrates against this opponent vary a lot between checkpoints of the same run (response 3), so I don't think the claim is established. On unseen maps the head-to-head advantage disappears, RAISocketAI is a tie, and Flat is ahead against the middle tier. You also report that adaptive multipliers are not shown to be necessary, that the deployed strategist is not needed, and that the remedy presented as the main lesson costs win rate.

What remains is that training under a fixed Light build order with command shaping beats flat PPO on the main training map, plus the diagnostic finding on success rates. This is a different paper from the one submitted: the title, abstract and contribution list would change, and the residual design is new material that I cannot evaluate from a summary. I therefore keep my rating, and I think the work would be much stronger as a resubmission built around the diagnostic result and the new design.

Reviewer_eZZWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose a strategist–executor framework for real-time strategy games that separates strategic command selection from learned unit control. Specifically, the executor is a GridNet PPO policy conditioned via FiLM on a four-slot command tuple. Each command has an explicit, state-measurable success criterion with a deadline and success/fail/timeout outcomes. The executor is trained as a constrained RL problem. It maximizes game reward subject to a per-command success-rate constraint, implemented by combining a normalized game advantage with normalized slot-adherence advantages weighted by per-command Lagrange multipliers. The multipliers are adapted from smoothed terminal success rates, with a heuristic guard for stalled constraints. A Thompson-sampling bandit acts as the strategist. It keeps a payoff table indexed by opponent class, game context and tuple, uses a slot-additive log-odds prior to share evidence across tuples, and weights the classes with a scouting-based belief over opponent classes built from in-game observations. The framework is evaluated in MicroRTS against a flat PPO baseline with the same architecture, budget, curriculum and self-play league. The results show significantly higher win rates against three of the four strongest opponents, including two held-out ones, and no significant difference against the remaining opponents.

Critical strengths
  • S1: Interpretable, steerable strategic control of RL agents in RTS games with robustness to unseen opponents is a well-motivated problem.
  • S2: Flat PPO shares the architecture, interaction budget, curriculum, maps and self-play league, thereby limiting the usual confounds of RL comparisons.
  • S3: Strong and diverse opponent pool with calibrated difficulty
  • S4: Careful statistics within the single run
Critical weaknesses
  • W1: All significance tests are over games from a single frozen agent per method, where the variance of training seeds could be often large and not considered.
  • W2: The proposed remedy is not implemented.
  • W3: Heavy reliance on hand-tuned heuristics.
  • W4: No empirical comparison with grounded RL and option-based RL baselines.
Questions for authors
  • Q1: It would be great to have an ablation study on the production head and auxiliary cross-entropy loss.
  • Q2: Per W1, it is better to report the results over multiple training seeds per arm.
2 comments

W1/Q2 seeds. See common response 1.

W2 remedy not implemented. See common response 5. We tried the proposed remedy; it improved posture control but cost win rate. The different residual design gives posture influence .41-.50 against .02 for the paper's executor.

W3 heuristics. Two of them are now ablated. Fixed multipliers do as well as adaptive ones on most opponents, but not against TMA and RAISocketAI (see common response 3), and one fixed schedule does as well as the generated schedules plus the bandit except against RAISocketAI (see common response 2). We will add a table listing every hand-set component and whether it was ablated. Please let us know if you would like to see other ablation studies or if these were not the hand-tuned heuristics you were referring to.

W4 Grounded RL and option baselines. Grounded RL bounds a behaviour-cloning loss on human demonstrations, which do not exist for MicroRTS. Option and FeUdal methods learn latent goals with no verifiable criterion, so they cannot be given the same commands. We did not run them and the fixed-multiplier run (common response 3) is the closest we have to a fixed intrinsic-reward objective.

Q1 production head and auxiliary loss. We did not run an ablation study for submitted system in the paper: the shaped-reward baseline (response 2) includes both, so their contribution to win rate is not separated from the order shaping. The residual design (see common response 5) drops the production override, its fixed logit offsets and the auxiliary loss, and composition orders still control production. On fixed recorded states, a HEAVY order raises the heavy share of barracks production from .06 (no order) to .70-.73 (two seeds), and all six composition orders end training with success rates of .85-.98 (three seeds; on the two further base policies HEAVY reaches only .02 and .46, the other five .85-1.00). So the head is not needed for composition control in the new residual design and would likely not have been needed in the submitted system from the paper

Thanks for the response. As most of the concerns are not fully addressed, I will keep my original rating.

Reviewer_P24FOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a hierarchical-RL approach to real-time strategy games, where a strategist selects high-level commands consisting of four slots and an executor performs the command by satisfying the success rate constraint of the command. By designing a tabular (discrete and fixed) command space consisting of 16 commands, the framework achieves explainability and verifiability. In microRTS, the resulting models achieved higher win rates against the SOTA benchmark opponents than flat PPO.

Critical strengths

The paper achieves empirical success in a real-time strategy game against a suite of SOTA benchmark opponents. Because the framework forces the executor to practice diverse commands, the predefined, fixed axes of commands can prevent the learned policy from collapsing into a single dominant strategy during self-play.

Critical weaknesses

The critical strengths of the framework depend heavily on the domain engineering of the command space. Compared to FeUdal Networks from 9 years ago, the benefit of defining a discrete, fixed set of commands resides in explainable, verifiable strategy design and explicit diversity of the self-play pool. On the other hand, a critical shortcoming is the scalability of the framework with a fixed discrete space. Hence, the intellectual findings of this work have limited applicability across the broader AI research community.

Questions for authors
  • Is your proposed framework scalable beyond a single RTS game?
  • You only tested against a fixed set of SOTA benchmark opponents that can easily be exploited, and only compared the win rates against them with flat PPO. Could you provide direct head-to-head matches against the trained flat PPO models and other hierarchical RL methods, such as FeUdal Networks, as unseen opponents?
  • Although the Lagrange multipliers are updated systematically, what critically determines the multipliers is the threshold τc\tau_c. Do you have any alternatives to finding the thresholds beyond extensive hyperparameter tuning and heuristics?
2 comments

Thank you for your review and for the questions on scalability and head-to-head play.

Scalability beyond one RTS game. We have only tested MicroRTS and will mention this in the paper. What the method needs from a domain is a discrete command vocabulary whose success can be measured from state. The criteria are domain engineering; the constrained objective and the command contract (preconditions, achievability checks) are not game-specific. We agree this limits generality compared with learned goal spaces, and will state it as a limitation.

Head-to-head against flat PPO. This was already in the last row of Table 3 (97/100). The additional runs now also shows it replicates over seeds (97, 99, 97 of 100 shown in common response 1) but not on unseen maps (.46 pooled, shown in common response 4).

FeUdal Networks as opponents. We did not train one: there is no FeUdal implementation for the GridNet action space and one would need its own tuning. The learned opponents we do have are agent_sota and RAISocketAI.

Thresholds. All commands share one target (0.8), so nothing is tuned per command. A calibration suite checks that every criterion is achievable by a scripted reference. Removing the adaptation entirely (fixed multipliers, common response 3) changes little against most opponents and costs win rate mainly against TMA and RAISocketAI.

Thank you for your response. However, your responses have deepened my concerns regarding the value of your work rather than resolving them.

  1. Because most components of the framework are tightly coupled to the specific game of MicroRTS, showing the value beyond a single game, preferably beyond RTS games, is essential. As it stands, the empirical effort reads less like a validation of a generalizable RL framework, but more like extensive domain engineering tailored for MicroRTS game.

  2. The failure to generalize to unseen map heightens my concern regarding out-of-distribution generalization and immediately contradicts with your claim of adaptability. Given the extensive domain knowledge injected into your method, comparison to only flat PPO would not be sufficient.

  3. Since hierarchical RL is not a new invention, it is necessary to evaluate against some hierarchical RL baseline.

  4. A shared target threshold resolved my concern on the hyper-parameter tuning and heuristics. However your new ablation results on fixed multipliers raises a bigger question about the role of dual adaptation phase if it is doing any meaningful role at all, which intensifies my concern that hand-crafted domain engineering are doing all the heavy-lifting.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common response

We thank the reviewers. We ran the experiments you asked for; several of them slightly change our claims and we report them as they came out.

1. Training seeds (eZZW W1/Q2, 31PQ Q1). Two more from-zero seeds per arm. Final checkpoints, Bandit / Flat:

Opponentseed 1seed 2seed 3
RAISocketAI.34 / .01.42 / .01.58 / .00
agent_sota.93 / .61.94 / .50.87 / .03
obiBotKenobi.97 / .55.99 / .42.97 / .04
Head-to-head (Bandit wins)97/10099/10097/100

All twelve cells are significant in every seed (Holm-adjusted p < 1e-6). On the other nine opponents Flat is never significantly ahead.

2. Shaped-reward baseline (31PQ Q2). We trained the conditioned recipe on one fixed schedule, the orders the deployed bandit issues (LIGHT composition), with no generated schedules and no bandit. At 70% of training it beat every Flat seed (RAISocketAI .26, agent_sota .92, obiBotKenobi .92; head-to-head 82/100). At the final checkpoint it still beats every Flat seed on agent_sota (.84) and obiBotKenobi (.98) (head-to-head 90/100), but against RAISocketAI it falls to .03, below every Bandit seed (.42-.58, p < 1e-10). The fixed orders explain the gain on two of the three opponents. The late RAISocketAI gain needs the generated schedules and bandit. This was run with one seed and this run also has the production head and auxiliary loss.

3. Fixed multipliers (31PQ Q3). With all multipliers fixed at 0.3 (one seed), the executor matches the adaptive one on most opponents (final checkpoint: agent_sota .95 vs .87-.94, obiBotKenobi 1.00 vs .97-.99). It is behind against TMA at both checkpoints we evaluated (70% of training: .47 vs .66-1.00; final: .83 vs .95-1.00, significant for two seeds each time). Against RAISocketAI the ordering reverses: at 70% the fixed run was far ahead (.48 vs .06-.08), at the final checkpoint it is behind (.34 vs .42-.58, significant only against seed 3). Per-command success rates are similar. With one fixed seed, adaptation helps consistently only against TMA, but this does not show that adaptive multipliers are needed.

4. Other maps (31PQ Q4). Table 3's two agents on five unseen 16x16 maps, pooled. Against the strong opponents the Bandit still beats Flat against obiBotKenobi (.45 vs .31, p < 1e-4) and now TMA (.32 vs .26, p = .02). It ties RAISocketAI (.22 vs .26), loses ground against mayari (.85 vs .92), and its head-to-head edge does not transfer (.46 win rate). Against the middle-tier bots it trails Flat by 9–22 points, as on the training maps D and I. Most of these losses happen in the opening worker fight, which Flat wins by counterattacking with its workers; the Bandit's opening order keeps workers on the economy. We will add the per-map tables and correct the claim that the system matches flat PPO against weaker opponents; that holds on map A only.

5. The proposed remedy and the two dead slots (eZZW W2, 31PQ). We tried the remedy proposed in the paper (success rates estimated from paired interventions). It improved posture control, but only at a cost of 8-15 points of win rate, and it left the worker slot unchanged. A different design works better: a frozen game-only base policy plus a learned order residual, with adherence credited per unit. Measured on one pre-registered instrument (share of army time spent advancing under OFFENSE minus under DEFENSE, 360 games), posture influence is .02 [.00, .05] for the paper's executor and .41-.50 for the new design across three seeds and two further base policies. Although this improves the control of the executor, we do not consider the executor fully controllable yet: the worker slot is still weak and we have no strategist that turns this control into wins

Revision. We will report all seeds, restate the contribution: the measured gain comes from command-conditioned training. At deployment one fixed LIGHT tuple on the same executor keeps it (Table 3, Static), so the deployed strategist is not needed; the generated schedules and the bandit during training matter only against RAISocketAI, and multiplier adaptation is not shown to be necessary. We will add the map tables, and finally we will replace the remedy paragraph with a section on the residual design .

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.