acceptodds
Under review as a conference paper at ICLR 2027

VLNAPO: Gradient-Guided Harness Engineering with Key-Token Supervision for Full-Game VLNA Policy Optimization in OpenRA

Abstract

Full-game control in real-time strategy (RTS) games requires agents to transform partial, multimodal observations into long-horizon strategic decisions while satisfying rapidly changing game-state constraints. The central challenge is that strategic actions are combinatorial: an action that is desirable in isolation may be infeasible because of resource limits, technology prerequisites, unit availability or production-queue conflicts. Thus, an agent must jointly determine what should be done from visual, linguistic and numerical context and generate command sequences that are both strategically effective and executable. We propose VLNA, a structured multimodal policy for full-game control in OpenRA. VLNA fuses tactical renderings, language context, and numerical game states to autoregressively generate structured macro-actions, which a deterministic compiler translates into native commands. A cascaded conditional action space uses prefix-conditioned masks to restrict each action while dynamically updating resource and production-queue budgets. Through training gradients, Harness Engineering combines rule-generated local hard labels with the PPO objective, embedding infrastructure and production rules without replacing policy actions during interaction. Across five seeds with 100 evaluation games each on the singles map, VLNA achieved a win rate of 77%±2.00%, rising to 90%±1.55% with persistent Harness supervision. The cascaded action space achieved a joint validity rate of 99.96%. These findings show that structured multimodal reasoning, prefix-conditioned action constraints, and gradient-guided Harness Engineering can jointly improve executability and strategic performance in full-game RTS agents.

Primary area:
reinforcement learning
Keywords:
Real-Time Strategy game, Reinforcement Learning, Large Language Model, Proximal Policy Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RtcpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VLNAPO for full-game RTS control in OpenRA. It combines a vision-language-numerical policy for macro-action generation, prefix-conditioned masks for action validity, and rule-based key-token supervision during PPO training. Experiments show that the proposed action constraints achieve 99.96% joint validity, while Harness supervision improves the win rate from 77% to 90%. The paper also includes ablation studies and evaluations on three unseen maps.

Critical strengths

The paper demonstrates strong performance in full-game OpenRA control, with a 90% win rate compared to 77% without Harness supervision. The results are evaluated across five random seeds, providing reasonable evidence of the system's effectiveness in the studied setting.

Critical weaknesses
  1. The proposed framework relies heavily on OpenRA-specific action representations, feasibility constraints, and handcrafted rules. While the results demonstrate an effective RTS agent, it is less clear what generalizable principles or methodological insights emerge from these designs. The broader scientific contribution beyond this particular control setting needs to be better established. The Introduction identifies two gaps concerning invalid action generation and the transfer of external rule guidance into learned policies. However, it does not adequately explain why these limitations are particularly important or why existing approaches are insufficient. In particular, the need to remove external rule systems at inference, rather than simply retaining them, is not well justified.

  2. Although the paper distinguishes Harness Engineering from teacher-based distillation, its core training mechanism remains PPO with auxiliary cross-entropy supervision on rule-generated labels. The paper does not establish whether its selective, online supervision offers meaningful advantages over simpler rule-based imitation or existing rule-guided learning methods. This makes its methodological novelty difficult to assess. Section 3.4 motivates Harness Engineering through sparse rewards, yet Appendix A.4 already introduces event-based rewards for construction, production, and economic development. While direct action supervision may provide complementary guidance, the paper does not clearly explain what learning difficulties remain unresolved by reward shaping or why Harness is needed to address them.

Questions for authors
  1. Which parts of the proposed method are generalizable beyond OpenRA, and what supports this claim?
  2. What is the advantage of learning rule guidance into the policy rather than retaining the rule engine at inference?
  3. What does the selective, online supervision in Harness Engineering offer over simpler rule-guided imitation learning methods?
  4. Given the existing event-based reward shaping, what additional learning difficulties does Harness supervision address?
  5. How does Harness compare with a no-Harness baseline on unseen maps, and do its benefits hold against different opponent strategies?
Reviewer_DQmxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents VLNA for full-game OpenRA control, combining visual, linguistic, and numerical observations with an autoregressive macro-action decoder. Prefix-conditioned masks enforce action feasibility, while rule-generated supervision on selected action slots augments PPO during training. The approach achieves a reported 90% win rate on the training map and is additionally evaluated on three unseen maps.

Critical strengths
  • The system addresses a concrete difficulty in RTS control: dependencies among actions competing for resources and production capacity. The improvement from semantic-prefix masking to the full cascade—48% to 77% win rate in Table 5—supports the practical value of resource- and queue-aware constraints.

  • Training-time rule supervision provides a meaningful improvement, raising win rate from 77% to 90% while reducing timeouts from 13% to 2%. Evaluation across five training seeds and additional unseen maps strengthens the empirical evidence. The supervision engine is removed at inference, although feasibility masks and the compiler remain active.

Critical weaknesses
  • The methodological advance over existing auxiliary imitation or distillation approaches is insufficiently established. The central objective combines PPO with cross-entropy on rule-generated labels from policy-visited states. Selective supervision and feasible target construction are useful design choices, but the paper does not clearly establish their advantage over simpler ways of incorporating the same rules, such as supervised pretraining followed by PPO.

  • Several ablations do not fully isolate the claimed effects. Table 2 evaluates soft targets only under triggered annealing, leaving their performance under the best persistent schedule unknown. Moreover, Appendix B.2 states that full-scope supervision assigns a canonical action to all slots when a rule fires; its poor performance may therefore reflect unreliable labels on otherwise unconstrained slots, rather than supervision scope alone. The soft-target construction and removal criterion also need clearer specification.

Questions for authors

How does the method compare with supervised pretraining followed by PPO, using the same rule source and a matched interaction budget?

How are soft targets and the capability-triggered removal schedule defined? Does persistent soft-target supervision close the gap to persistent hard-label supervision?

In Table 5, do the unconstrained and state-only variants retain slot-specific vocabulary restrictions? Please clarify whether their failure reflects cross-slot token errors or genuinely infeasible action combinations.

Reviewer_LnPEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors developed a structured multimodal policy for full-game control for real-time strategy (RTS) games in OpenRA environment. The main contribution is end-to-end training method that learns rule enforcement as a part of the learned policy. Additional contributions are a method of learning rule-derived guidance labels using PPO referred as "gradient-guided harness engineering" and prefix-conditioned validity updates.

Critical strengths

Strengths:

  • End-to-end training training enables joint learning of the semantic representation and the action labels.
  • The paper provides experimental ablations on using different variants of VLM model plus harness design ablations
Critical weaknesses

Weaknesses:

  • Joint learning of semantic representations and control actions generation is a valuable contribution, however the complexity of the pipeline make it challenging to pinpoint the isolated effects of each specific proposed components
  • The term "gradient-based harness engineering" used in the paper sets an expectation of gradient-based learning of the harness itself while the harness used in the paper is rather a rule-based engine used to shape policy's gradient updates. This mismatch suggests some improving clarity of the paper in this aspect.
  • Applying hard-level cross-entropy to the critical actions produced by the harness is an elaborate design choice driven by the application which is experimentally justified. However, in itself, it is rather bringing a specific technique to the specific context than a novel idea brought to the community.
Questions for authors

Q1: The results in Table 2 show that persistent harness with hard cross-entropy give much higher win rate than soft CE/KL. What is your intuition behind this drop? In particular what is the reason soft target results are worse than the results with no harness?

Q2: as of the main contribution of the paper is joint learning of the semantic representation and the executable actions policy, it would be interesting to see how much loss in win rate would separating the two components bring, i.e. learning the policy with the semantic VLM frozen.

Q3: Could you elaborate on contribution (2) "prefix-conditioned validity"? What exactly does the paper propose apart applying the feasible set mask to the logits?

Q4: As the approach considers partially observable sequential decision problem, did you consider using a POMDP-based formulation, e.g. POMDP RNN-based approaches for inferring the belief state?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.