acceptodds
Under review as a conference paper at ICLR 2027

SLICR: Strategy Logic-Guided Multi-Agent Reinforcement Learning

Abstract

Specification-guided reinforcement learning (RL) translates formal specifications of desired behavior of agents into shaped rewards. This paper introduces SLICR, a significant paradigm shift that advances the state of the art to specification-guided RL in adversarial settings under partial observability. We employ Strategy Logic (SL), which allows explicit quantification over agents' strategies and uses the linear temporal logic (LTL) to specify the agents' objectives and constraints. This enables reasoning not only about how agents should act to satisfy a specification, but also about how they should respond to the strategies and behaviors of other agents. We evaluate SLICR on three popular benchmarks: Predator-Prey, Google Research Football, and our home-grown extension of SMAC-HARD, demonstrating its effectiveness, scalability, and ability to adapt to diverse adversarial strategies.

Primary area:
reinforcement learning
Keywords:
Specification-Guided RL, Strategic Logic, Formal Methods, Multi-agent Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5XJVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SLICR, a specification-guided MARL framework for partially observable adversarial environments using Strategy Logic (SL) to express dependencies between agents’ strategies and LTL to specify temporal objectives. SLICR learns best-response policies conditioned on inferred opponent-strategy embeddings, uses robustness-based rewards derived from the specification, and includes a shielding mechanism to filter experiences generated by irrational opponent strategies. The method is evaluated on three benchmarks, where it generally outperforms the LIAM opponent-modeling baseline in both seen and unseen opponent settings.

Critical strengths

Integrating SL into MARL is an interesting problem formulation. SL extends the LTL specifications commonly used in specification-guided MARL with explicit quantification over agents’ strategies, so the specification can describe how agents should respond to the strategies of other agents, which provides a novel way to formally express adversarial interactions.

Critical weaknesses
  1. The problem statement only considers SL sentences with at most one quantifier alternation, which restricts the expressiveness that motivates the use of SL. The paper justifies this restriction by claiming that most adversarial RL problems can be expressed in this fragment, but provides no examples or references to support this claim. In addition, Theorem 5.3 assumes that the latent embedding is information-preserving and that the action-level response functions can represent every well-typed response function. The authors acknowledge that the first assumption may be hard to achieve under partial observability, but the second is also a strong condition that is unlikely to hold in practice.
  2. Although the authors evaluate SLICR on several environments, its scalability does not appear to be evaluated along two important dimensions. First, it is unclear how the method scales with the complexity of the specification, since even in a simple scenario such as Voyage, the inner LTL formula may become complex or deeply nested. Second, it is unclear how the method scales with the number of agents, since each additional agent introduces a new strategy quantifier in the SL sentence and may further increase the complexity of the inner LTL formula.
Questions for authors
  1. In Eq. (2), why is the objective defined as the probability that the robustness value converges to ρmax\rho_{max}? Since ρ>0\rho > 0 already implies that the specification is satisfied, it is unclear why convergence to ρmax\rho_{max} is required. What is the effect of this choice compared with requiring only ρ>0\rho > 0?
  2. Why are irrational strategies restricted to observation-independent uniform-random policies? In adversarial settings, irrational or suboptimal opponents could still exhibit structured, history-dependent, or nonstationary behavior. How would the shielding mechanism generalize to such strategies?
  3. The authors mention that all existentially quantified agents receive the same reward. How is credit assignment handled in this case? The suggested decomposition ψ=⋀iψi\psi=\bigwedge_i\psi_i may not be possible for general nested specifications with coupled objectives. How would SLICR handle such cases?
  4. How is strategy identity defined when constructing ΠTrain\Pi^{Train} and ΠTest\Pi^{Test}? Since a strategy is formally a mapping from finite histories to action distributions, what criterion is used to determine that two separately trained policies constitute different strategies? This is especially important for interpreting the Seen and Unseen settings.
4 comments
AuthorsOpenReview ↗

Reply to Reviewer 5XJV (1st)

We thank the reviewer for their comments. We respond to the critical weaknesses in this comment (1st) and the questions in (2nd) comment.

[W1] The problem statement only considers SL sentences ... references to support this claim.

$\forall^* \exists^* $ and $\exists^* \forall^* $ fragments capture important strategic objectives in two-player and two-team adversarial MARL: learning responses that depend on opponent strategies and learning a single controlled strategy profile that works against all opponent strategy profiles, respectively. These objectives are relevant to many adversarial MARL benchmarks, including:

  • Markov soccer (Littman, ICML 1994)
  • Kuhn poker (Kuhn, 1950)
  • Predator–Prey
  • Google Research Football
  • StarCraft II

In addition, the existence of a Nash equilibrium can also be expressed in SL using at most one quantifier alternation (Beutner, AAMAS 2024). While this is well-established in various areas of computer science, we also revise the discussion in Section 4 in (red) to clarify our motivation.

[W1] In addition, Theorem 5.3 assumes ... but the second is also a strong condition that is unlikely to hold in practice.

The assumption that action-level response functions can represent every well-typed response function ensures that the action-level formulation preserves the optimum of the strategy-level objective in Eq.3. This allows Theorem 5.3 to establish the connection to Lemma 5.2. However, in practice, SLICR directly optimizes the action-level objective in Eq.4. Thus, representability is not required for training or executing SLICR and is assumed to establish the theoretical guarantees. This is why our limitations discussion emphasizes the information-preserving embedding assumption rather than representability. We also add more text in limitation paragraph in Section 7 to clarify these points.

[W2] Although the authors evaluate SLICR on several environments, its scalability does not appear to be ... increase the complexity of the inner LTL formula.

We want to clarify that our evaluation includes variation in both the complexity of the inner-LTL formula and the number of quantified agents.

-- Number of Agents. Regarding agent scalability, our evaluation also varies the number of agents and corresponding strategy quantifiers, from 4 in Predator–Prey to 6 in the SMAC-HARD maps 3m and 3s_vs_3z, and 10 in 2s3z. Thus, our experiments include specifications involving multiple existentially and universally quantified agents. For the rebuttal, we have additionally included decentralized (using IPPO) SMAC-HARD results in the revised Fig. 3 to further evaluate SLICR in this setting. We also want to clarify that scalability with the number of agents is a general challenge in multi-agent RL. Nevertheless, to the best of our knowledge, ours is the only work on agent modeling against populations of adversaries evaluated on such a diverse set of complex case studies, including three Google Research Football scenarios (one added for the rebuttal) and three SMAC-HARD maps.

-- Complexity of the inner-LTL Formula. The complexity of the inner-LTL formula has no or little impact on the structure of SLICR’s learning framework. SLICR translates the inner-LTL formula into a robustness function using quantitative semantics, and its outputs serve as reward signals (computing min and max in sets can be done in linear time). Furthermore, in Appendix E, our specifications range from the relatively simple φpp\varphi_{\mathrm{pp}} in Appendix E.1 to the more complex φSMAC\varphi_{\mathrm{SMAC}} in Appendix E.3, which includes nested temporal operators.

AuthorsOpenReview ↗

Reply to Reviewer 5XJV (2nd)

[Q1]

While ρ>0\rho>0 guarantees for satisfaction satisfaction, maximizing Pr⁡(ρ>0)\Pr(\rho>0) gives equal credit to all satisfying trajectories, regardless of their robustness values. We chose convergence to ρmax⁡\rho_{\max} as a stronger target in the favor of maximizing robust satisfaction, and this is a deliberate optimization choice. However, in practice, SLICR maximizes the expected cumulative discounted robustness in Eq.4, using robustness values as reward signals to guide learning. Hence, convergence target in Eq.2 is not enforced during training. It serves as a theoretical intermediate step connecting the problem statement in Section 4 to the objective in Eq.4.

[Q2]

In our work, we specifically define an irrational strategy as an observation-independent strategy that selects actions uniformly at random. In our terminology, an opponent with structured behavior can be treated as rational even if its strategy is suboptimal, since its behavioral patterns can still be learned by the agent model. The broader operational distinction between rational and irrational behavior depends on the chosen definition, and our analysis and experiments use the definition stated in the paper.

The shielding mechanism is a classifier trained on labeled interaction histories. To accommodate a different definition of irrational behavior, we can collect representative interaction histories, assign labels according to that definition, and retrain the classifier. The same shielding procedure would then use its predictions to determine which experiences contribute to training. Thus, the classifier can be adapted to alternative definitions through its training data and labels.

[Q3]

We first clarify that we do not claim to solve the credit-assignment problem in this work. The decomposition is mentioned because the POSG original formulation permits different reward signals for different agents. When agent-specific objectives and constraints are available, they can be expressed as ⋀i∈Q∃ψi\bigwedge_{i \in Q^\exists}\psi_i, with each agent receiving a reward derived from its corresponding subformula ψi\psi_i. Each ψi\psi_i may itself contain nested temporal operators. For a specification with coupled objectives that does not admit a meaningful agent-specific decomposition, SLICR evaluates the full inner-LTL formula and provides its robustness value as a shared reward to all existentially quantified agents, as we did in our experiments.

[Q4]

We first clarify that, in the Seen setting, training and testing use the same opponent strategy set. In our experiments, opponent identity is associated with a particular pretrained strategy instance or opponent-team configuration. In the Unseen setting, we construct it using held-out strategies or configurations, varying the MARL training algorithms and behaviors. For example, in Google Research Football, the training opponents are trained using IPPO, while the Unseen test opponents are trained using MAPPO. We also include zonal defending in the test set. Similarly, in the SMAC-HARD maps 3m and 3s_vs_3z, we use VDN and QMIX opponents for training and QTRAN opponents for Unseen testing. Tables 3–5 in appendix provide the complete splits. The purpose of the Unseen setting is to evaluate how SLICR performs against opponent strategies or team configurations that it did not encounter during training.

If our responses address all your concerns, we kindly ask you to consider raising your score.

I thank the authors for their response and for the additional experiments. However, my concerns are not fully addressed.

  1. W2 (scalability). The authors point to agent counts from 4 (PP) to 10 (2s3z) and to specifications ranging from simple (Appendix E.1) to complex (Appendix E.3). However, the two grow together: each additional agent adds a strategy quantifier and can enlarge the inner-LTL formula. In the experiments, agent count varies only across SMAC-HARD maps, which also differ in unit types and opponents, and the specifications in Appendices E.1–E.3 come from different environments, so neither factor is varied in a controlled way. My concern is not the cost of computing robustness but how learning is affected as the number of agents and formula complexity increase. A controlled study within the same environment would address this, e.g., increasing the number of agents with the corresponding formula, and increasing the nesting depth of the inner-LTL formula for a fixed number of agents.
  2. Q2. My question was how the shield generalizes to irrational strategies beyond uniform-random ones, such as structured, history-dependent, or nonstationary behavior, which can occur at test time. The response suggests collecting new histories, relabeling them, and retraining the classifier. This concedes that the shield does not generalize to unseen irrational behavior: it must be retrained for each new type rather than adapting to it.
  3. Q3. My concern is whether a shared reward can handle deeper nested LTL formulas, where a single robustness value does not indicate which part of the specification each agent affects. The response only states that the robustness value is shared among all agents, as in the experiments, without addressing this.
  4. Q4. The response identifies a strategy by its pretrained instance, configuration, or training algorithm. However, the paper defines a strategy as a mapping from histories to action distributions (Sec. 3.1), so whether the Unseen setting satisfies ΠTrain∩ΠTest=∅\Pi^{\text{Train}} \cap \Pi^{\text{Test}} = \emptyset depends on behavior, not on how the opponents were trained. Since opponents trained with different algorithms may exhibit similar behavior, the response does not provide a criterion consistent with the paper’s definition, and the Unseen results may not support what the authors report.

Therefore, I maintain my score.

AuthorsOpenReview ↗

Reply to Reviewer 5XJV (3rd)

We are grateful to the reviewer for considering our rebuttal. Also, we are delighted that we have addressed their concerns regarding their first weakness and first question. Below, we address their remaining concerns.

W2

We will discuss scalability with respect to the complexity of the inner-LTL formula and the number of agents (i.e., the number of quantifiers), below:

  • Complexity of the inner-LTL formula: We want to clarify that the complexity of the inner-LTL formula does not change SLICR’s model architecture, number of trainable parameters, or underlying learning. Inner-LTL formulas are evaluated through a robustness function, whose outputs serve as reward signals for the agents. Thus, changing the formula is similar to changing the reward function: it guides the agents toward the new specification, while the learning pipeline, model architecture, and parameter count remain unchanged.

    We also want to clarify that the inner-LTL formulas presented in Appendix E were designed to address the benchmark objectives described in ln402–418. Making these formulas more complex may change the objectives they express. In that case, comparing task-performance metrics would not isolate the effect of formula complexity, because the agents would also be pursuing different objectives. This is similar to changing the reward function while evaluating scalability: the comparison would conflate changes in specification complexity with changes in the learning objective.

  • Number of Agents: First, we want to clarify that the scenarios used in our experiments are standard provided by the benchmarks. We follow their configurations, as is common MARL papers. Moreover, increasing the number of controlled agents while keeping the opponents fixed can make the task easier by giving the controlled team a numerical advantage. Such a comparison would conflate scalability with changes in task difficulty. To the best of our knowledge, SLICR is the first work, even within the broader scope of population-based adversarial MARL, to consider multiple agents on the controlled side.

Q2

We want to clarify that, in this work, irrational behavior is explicitly defined as observation-independent, uniform-random behavior, and the shield is trained according to this definition. How can we expect it to classify other behaviors as irrational at test time when those behaviors fall outside the definition of irrationality adopted in the first place?

In our previous response, by generalizable we meant that the shielding procedure can be adapted to different definitions of irrational strategies through the corresponding training data and labels. We were referring to the adaptability of the procedure, rather than claiming that the trained classifier automatically recognizes unseen forms of irrational behavior beyond its training definition.

Q3

We first want to clarify that solving credit assignment is beyond the scope of this work. Our focus is spec-guided adversarial MARL.

We can express agent-specific objectives through a decomposition of the inner-LTL formula as ⋀i∈Q∃ψi\bigwedge_{i \in Q^\exists}\psi_i, where each ψi\psi_i may itself is deeply nested formula. This allows us to define separate robustness-based rewards for each agent, consistent with the POSG formulation allowing different rewards functions for different agents.

Your concern is valid, but we never claim to solve credit assignment in our paper.

Q4

We want to clarify that we do not claim that ΠTrain\Pi^{\text{Train}} and ΠTest\Pi^{\text{Test}} exhibit qualitatively different behaviors in Unseen setting. Our claim:ΠTrain∩ΠTest=∅\Pi^{\text{Train}}\cap\Pi^{\text{Test}}=\emptyset, meaning that we use different strategies for testing in the Unseen setting. Under the paper’s definition, two strategies are different if they produce different action distributions for at least one common history. Similar behavior does not imply identical strategies. Our experiments also show differences in performance between the Seen and Unseen settings.

For example, in 2s3z (see Tab.5), we used VDN for training and QMIX for testing. The VDN strategies achieved win rate of 93%, while the QMIX strategies achieved 96% against built-in AI. Additionally, we used built-in AI at difficulty level 8 for training and 10 for Unseen testing. Also theoretically, VDN sums individual Q-values, while QMIX uses a learned monotonic mixing function to preserve consistency between individual and joint greedy action selection.

Furthermore, even changing the seed may produce different results, which motivates conducting multiple runs and reporting measures such as the 25–75th percentiles or std errors. This can be seen in well-known cooperative CTDE MARL papers such as: VDN, QMIX, QTRAN, and QPLEX (Wang et al., ICLR'21).

We hope we have clarified the remaining points, and we would be happy to address any further concerns.

Reviewer_EH1AOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SLICR combines temporal-logic reward guidance with opponent modeling under partial observability. Strategy Logic specifies dependencies between agents' strategies. Recurrent A2C and PPO agents use formula robustness as rewards and a local-history embedding trained to reconstruct and predict opponents' behavior. A classifier filters experience involving uniformly random opponents. Experiments on Predator–Prey, Google Research Football, and SMAC-HARD report improvements over LIAM against seen and held-out opponents. Conditional theoretical results connect the representation and reward objective to specification satisfaction.

Critical strengths
  1. The practical gains are substantial across meaningful opponent variation. For example, on unseen opponents in 3s_vs_3z, SLICR reaches roughly 90% win rate while LIAM rarely wins.
  2. The component analysis makes the empirical contribution more credible. The shield ablation in Appendix F.3 and reward-guidance ablations in Figures 5 and 11–13 examine the proposed mechanisms separately. The authors also report mixed results, including weaker unseen R2S performance with temporal guidance. These findings help identify where the additions are useful.
Critical weaknesses
  1. The implemented controller does not establish the claimed strategic dependency. Section 4 lets a response depend on opponents' actual strategies. Section 5.3 instead supplies an embedding computed from the controller's own history. Two opponents producing identical histories before a decisive action cannot receive different responses, even when different responses are needed. Appendix D.2 acknowledges this information barrier, while Theorem 5.3 assumes lossless recovery of the strategy context. The gains in Table 1 could therefore reflect the added opponent-modeling architecture, rather than the claimed distinction between the two quantifier orders. The paper needs an explicit information assumption or a narrower explanation of what the Strategy Logic formulation contributes.
  2. The reward to satisfaction connection is assumed rather than established for the evaluated tasks. Lemmas 5.1–5.2 require the successive objectives to rank response functions consistently, but Appendix B does not establish this for the implemented rewards. More early contacts do not imply a higher probability of final contact. Thus collision counts do not establish improvement in full-formula satisfaction. This leaves the central synthesis claim unsupported, despite useful task-performance gains.
Questions for authors
  1. Does the deployed agent receive any information about the opponent strategy context beyond its local history? If not, what distinguishes the realizable strategy classes of the two quantifier orders, beyond the auxiliary modeling architecture and parameter count?
  2. What property of the implemented robustness functions establishes the ordering assumptions in Lemmas 5.1–5.2? In particular, is the intended Predator–Prey objective final-episode formula satisfaction or accumulated contact frequency, and how are these reconciled in the claimed guarantee?
7 comments
AuthorsOpenReview ↗

Reply to Reviewer EH1A

We thank the reviewer for their comments.

[W1] The implemented controller ... strategy context.

We first clarify that the reviewer identifies precisely the challenge discussed in the Challenges and Overview of SLICR paragraph in Sec.5. Our formulation considers a partially observable stochastic game, where agents cannot directly access other agents’ strategies. SLICR addresses this challenge by learning a representation of the opponent strategies from the agent’s local observation. Thm.5.3 explicitly conditions its guarantee on an information-preserving latent embedding, and in limitation paragraph we acknowledge that such an embedding may be hard to obtain under partial observability. Also, It is standard and acceptable to establish theoretical guarantees under explicitly stated assumptions.

[W1] The gains in Tab.1...contributes. [Q1] Does the deployed ...count?

We want to clarify that the reviewer’s point is correct: the distinction between the SL fragments manifests itself through different architectures in our practical implementation. We strongly believe that this is not a weakness of our work. Rather, we view SLICR’s procedure for translating different quantifier orders into different learning pipelines as a strength. This happens because the quantifier order determines the permitted strategy dependencies, and these dependencies guide the information inputs and components used in the learning pipeline, as explained below from both theoretical and practical perspectives.

Theory. SLICR connects the satisfaction-probability objective in Eq.1 to the action-level optimization objective in Eq.4. The two quantifier orders do not require different underlying strategy classes; their distinction concerns how controlled strategies (∃\exists-agents) may depend on opponent strategies (∀\forall-agents). Under $\exists^* \forall^* $, Pre=∅\mathbf{Pre}=\emptyset, so the strategy context is empty and each response function selects a strategy independently of the opponent strategy context. The resulting strategy still uses the agent’s local history. Under ∀∗∃∗\forall^*\exists^*, the response functions may select different controlled strategy depending on opponent strategies.

Practice. To implement these dependencies when optimizing Eq.4, SLICR introduces an opponent-strategy embedding under $\forall^* \exists^* $, while the $\exists^* \forall^* $ pipeline has no opponent-strategy context input because there is no such dependency. Each deployed agent in the $\forall^* \exists^* $ setting computes the embedding from its own local history. This embedding augments the architecture with a learned representation of opponent behavior, allowing the response function to condition its actions on this representation. It may expand the search space representable by the chosen architecture, but the resulting strategies remain functions of the agent’s local history. A related observation appears in previous work, HyPOLE (Rafieioskouei et al. ICML'26), where the ∀∃\forall\exists formulation expands the policy search space relative to $\forall^* $ by allowing dependencies on preceding universally quantified traces, with improved empirical performance. That work considers cooperative MARL using HyperLTL, whereas SLICR considers adversarial MARL using SL. This is how the formal distinction guides the construction of different learning pipelines and architectures. Tab.1 compares these implementations, including their architectural differences; it does not isolate the effect of quantifier order from the benefit of opponent modeling.

[W2] The reward to ... task-performance gains. [Q2] What property ... guarantee?

We want to clarify that our experiments aim to learn action-level best-response functions that address the maximization problem in Eq.4. This objective is theoretically connected to Eq.1 in our problem statement through Thm.5.3 and Lem.5.1–5.2, under their stated assumptions. These assumptions apart from the information-preserving latent embedding assumption, provide the conditions connecting optimization of Eq.4 to optimization of Eq.1, and they are not prerequisites for using SLICR in practice.

In Eq.4, we use the robustness values of trace prefixes to guide learning as reward signals. Specifically, in Predator–Prey, we express repeated contact using an always eventually condition, which, under our finite-trace interpretation, requires contact at the final step of each evaluated prefix. Evaluating this condition throughout the episode therefore rewards repeated contacts. If an action-level response function achieves maximum robustness at every prefix almost surely, then, under the correspondence established in Thm.5.3, it induces a strategy-level response function that does the same for the complete trace, since the complete trace is itself the final prefix.

If our responses address all your concerns, we kindly ask you to consider raising your score.

Thank you for clarifying that the quantifier orders guide different learning architectures and that the theoretical guarantees are conditional. This addresses part of my first concern, but the connection between cumulative prefix rewards and full-trace satisfaction remains unresolved. Achieving maximum robustness at every prefix would cover the final trace, but maximizing expected cumulative discounted robustness does not establish that stronger condition, since rewards at different times may trade off. As this connection is central to the paper’s formal contribution, I maintain my original rating.

AuthorsOpenReview ↗

Reply to Reviewer EH1A (2nd)

We thank the reviewer for considering our rebuttal and are delighted that some of their concerns have been addressed. We are also highly confident that our theoretical contribution is sound. We explain the remaining point in details below.

Our discussion of maximum robustness at every prefix was intended to illustrate a sufficient condition connecting prefix robustness to full-trace satisfaction, using the Predator–Prey example. It was not intended to suggest that maximizing expected cumulative discounted robustness necessarily establishes this condition. If a response function achieves maximum robustness for the full inner-LTL formula at every evaluated prefix almost surely, then it also achieves maximum robustness on the complete trace, since the complete trace is the final evaluated prefix. This conditional observation is independent of discounting.

The optimality connection in Lemma 5.2 is conditional on its explicit ordering assumption. It does not rely on the claim that maximizing cumulative discounted robustness necessarily maximizes robustness at every prefix. Using robustness values as rewards does not automatically establish this ordering, which is why the lemma explicitly assumes it.

We further clarify that our experiments aim to maximize the action-level objective in Eq.4. The ordering assumptions, together with the representation assumptions in Theorem 5.3, are used to establish the conditional theoretical connection to the satisfaction-probability objective in Eq.1. Note that the ordering assumptions are not prerequisites for training or executing SLICR, but they are required for invoking these guarantees.

If this response addresses your remaining concerns, we kindly ask you to consider raising your score. We believe SLICR offers a very novel contribution to the MARL community through its theoretical framework and extensive experiments, and we hope it will have the opportunity to appear at ICLR 2027.

Reviewer_NxNNOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses specification-guided multi-agent reinforcement learning (RL) toward adversarial and partial observable settings. It proposes SLICR, which uses strategy logic to quantify over agent strategies and linear temporal logic to specify an agent’s objectives and constraints. The approach is evaluated in three benchmark domains and compared with LIAM as its sole baseline.

Critical strengths

The paper addresses an important topic that is becoming increasingly relevant to the field of MARL.

Critical weaknesses
  1. Presentation: For non-expert readers, this paper is very difficult to read due to many formulas (the first formulas already appear in the introduction). I went through the paper several times but still cannot conclude reliably on its soundness.

  2. Evaluation: Despite existing prior work in the field of adversarial/robust MARL [1,2,3] and stochastic observability in MARL [4,5], the paper only compares with a single baseline LIAM. To demonstrate the effectiveness of SLICR against irrational opponents, it would be useful to evaluate against approaches that strictly assume rational opponents by design to show how they fail.

Literature

[1] Bukharin et al., "Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms", NeurIPS 2023

[2] Li et al., "Byzantine Robust Cooperative Multi-Agent Reinforcement Learning as a Bayesian Game", ICLR 2024

[3] Li et al., "Robust Multi-Agent Reinforcement Learning via Minimax Deep Deterministic Policy Gradient", AAAI 2019

[4] Lyu et al., "A Deeper Understanding of State-Based Critics in Multi-Agent Reinforcement Learning", AAAI 2022

[5] Phan et al., "Attention-Based Recurrence for Multi-Agent Reinforcement Learning under Stochastic Partial Observability", ICML 2023

Questions for authors

None

1 comment
AuthorsOpenReview ↗

Reply To Reviewer NxNN

We thank the reviewer for their comments.

Presentation

Please note that our work lies at the intersection of AI and formal methods, and we use Strategy Logic as a rigorous mathematical framework to guide agent behavior in adversarial MARL. Mathematical formalization is essential to precisely define our framework and establish its theoretical foundations. We acknowledge that the paper is mathematically dense, which are inherent to the formal nature of our work. It is also important to note that mathematically dense, specification-guided RL papers are not uncommon at top AI conferences, as demonstrated by prior work such as HyPOLE (Rafieioskouei et al., ICML 2026) and HypRL (Hsu et al., NeurIPS 2025).

Although the introduction includes two formulas, the natural language description of the formulas are also included in lines 49 and 52 to help the reader. We also use the Voyage scenario as a running example to provide concrete illustration of the POSG in Section 3 (lines 125–132) and explain the problem statement in detail in Section 4 (lines 229–249). In fact, literally everywhere in the paper, all the math is coupled with an English description.

The reviewer also express concerns about the soundness of our approach. We are wondering if the reviewers has found any bug in the formal setting of the paper or any of the proofs. We will be happy to address any specific part of the paper, if the reviewer points us to any concrete issues with soundness.


Evaluation

-- Scope of SLICR: We first clarify that our work focuses on specification-guided RL, and to the best of our knowledge SLICR is the first work to combine Strategy Logic with adversarial MARL. In the $\forall^* \exists^* $ fragment of Strategic Logic, existentially quantified strategies (for ∃\exists-agent) may depend on the preceding universally quantified strategies (for ∀\forall-agent). Because agents cannot directly access these strategies, they must infer them from their own action–observation histories. This motivates our comparison of SLICR (in the $\forall^* \exists^* $ setting) with LIAM (an agent-modeling method for partially observable environments). Unfortunately, the reviewer's comment contains a factual error: Our comparison is not limited to irrational strategies. In Fig. 3, we compare against LIAM using rational and competitive opponent strategies.

-- References [1,3] fall in a different RL category compared to SLICR. Broadly speaking, works on adversarial MARL can be grouped into two directions: (1) robustness against worst-case adversaries, and (2) learning adaptive responses to a population of opponent strategies. SLICR addresses category (2), while references [1,3] focus on category (1), which study robustness against worst-case perturbations or adversarial actions through adversarial regularization and minimax optimization, respectively. On the other hand, SLICR considers a population of opponent strategies and learns responses that maximize specification satisfaction across this population.

-- Reference [2] studies a cooperative team with potentially compromised teammates. Its Bayesian Adversarial Robust Dec-POMDP models uncertainty about which teammates are adversarial and learns policies that balance cooperation with robustness against compromised agents. This differs fundamentally from our problem formulation, where agents learn responses to a population of opponent strategies guided by a formal specification.

-- References [4,5] study cooperative MARL under partial observability using Dec-POMDPs, whereas SLICR formulate adversarial interactions using a partially observable stochastic game (POSG). These are not even comparable.

In a nutshell, references [1-5] cannot serve well as comparable and meaningful baselines here.


Since we believe that the reviewer does not identify a specific flaw in our formalization or a concrete soundness issue, we kindly ask you to consider raising your score if our responses address all your concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Clarification, Summarization of our Rebuttal, and List of Main Revisions

In this comment, we first clarify several points that we believe address many of the concerns raised in the reviews, summarize our rebuttal for the reviewers’ convenience, and list the revisions.

Clarification

C1. Contribution. In this paper, we introduce SLICR, the first specification-guided RL framework that uses Strategy Logic to reason about strategy synthesis in adversarial multi-agent RL.

  • Theoretical contribution: We formulate the problem of maximizing the satisfaction probability of an SL specification in Eq.1, then connects it, under the stated assumptions, to the optimization problem in Eq.4, which can be addressed using RL.

  • Practical contribution: We implement and evaluate our approach on three benchmarks: Predator–Prey, Google Research Football, and StarCraft II SMAC-HARD, through comparisons with agent-modeling baselines and a range of ablation studies. To the best of our knowledge, ours is the first work to evaluate such a diverse set of case studies, including three Google Research Football scenarios and three SMAC-HARD maps, in both agent modeling against populations of adversaries and specification-guided MARL.

C2. Assumptions in the Theoretical Framework. Apart from the information-preserving latent embedding assumption, the remaining assumptions are not prerequisites for executing SLICR in practice, whose practical objective is to learn action-level best-response functions that maximize Eq.4. These assumptions support the theoretical guarantees connecting this optimization objective to the maximization of satisfaction probability in Eq.1 of Sec.4.

Summarization of our Rebuttal

Reviewer NxNN

  • Reviewer's main concern is that our paper is not easy to follow due to its many formulas, although mathematically dense, specification-guided RL papers are not uncommon at top AI conferences, as demonstrated by HyPOLE(ICML'26) and HypRL(NeurIPS'25). We accompany our mathematical formalization with natural-language explanations and the Voyage running example.

  • Regarding soundness, we ask the reviewer to identify any concrete errors in the formalization or proofs so that we can address them directly.

Reviewer EH1A

  • Added Opponent Modeling Architecture rather than quantifier Formation. The reviewer is exactly right that the distinction between the ∀∃\forall\exists and ∃∀\exists\forall fragments manifests in different architectures and learning pipelines. We strongly believe that translating this formal distinction into practical RL pipelines is a strength of SLICR.

  • Ordering assumptions in Lem5.1 and 5.2. As clarified in [C2], our practical objective is to learn action-level best-response functions that maximize Eq.4. The ordering assumptions are used to establish the theoretical guarantees connecting this objective to satisfaction probability.

Reviewer 5XJV

  • Justification of SL Fragment Restriction. In [W1], the reviewer asks us to justify the two SL fragments. We clarify that $\forall^* \exists^* $ expresses responses that depend on opponent strategies, while $\exists^* \forall^* $ expresses a single strategy that works against all opponents. These objectives apply to benchmarks with two players or two teams, including Predator and Prey, Google Research Football, StarCraft II, and Kuhn poker.

  • Assumptions of Thm.5.3 in Practice: As clarified in [C2] of this post, apart from the information-preserving latent embedding assumption, the remaining assumptions are not prerequisites for training or executing SLICR in practice.

  • Scalability of Complexity of inner-LTL Formula. Inner-LTL complexity has little direct impact on SLICR. In addition, Appendix E includes specifications ranging from the simple φpp\varphi_{\mathrm{pp}} to φSMAC\varphi_{\mathrm{SMAC}}, which contains nested temporal operators.

  • Scalability on Number of Agents. Scalability with the number of agents is a general limitation in MARL. Nevertheless, our case studies involve 4 to 10 agents, which, to the best of our knowledge, exceeds the agent counts evaluated in other opponent modeling or adversarial specification-guided MARL work.

List of Main Revisions (Changes in Red)

  1. Experiments Added. We added a Google Research Football scenario with four defenders between the goalkeeper and the attacker (Fig.3c). To demonstrate that SLICR supports decentralized learning of action-level response functions, we also added an IPPO setting in SMAC-HARD, the only benchmark with multiple controlled agents (Fig.3g–i).
  2. Justification of SL Fragment Restriction. We revised the beginning of Section 4 to further clarify this restriction.
  3. Our Practical Objectives in SLICR. We added a clarification at the beginning of Section 6 stating that our practical objective is to optimize Eq.4.
  4. Limitations. We clarified the practical implications of the assumptions and their role in establishing SLICR’s theoretical guarantees.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.