- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces SLICR, a specification-guided MARL framework for partially observable adversarial environments using Strategy Logic (SL) to express dependencies between agents’ strategies and LTL to specify temporal objectives. SLICR learns best-response policies conditioned on inferred opponent-strategy embeddings, uses robustness-based rewards derived from the specification, and includes a shielding mechanism to filter experiences generated by irrational opponent strategies. The method is evaluated on three benchmarks, where it generally outperforms the LIAM opponent-modeling baseline in both seen and unseen opponent settings.
Critical strengths
Integrating SL into MARL is an interesting problem formulation. SL extends the LTL specifications commonly used in specification-guided MARL with explicit quantification over agents’ strategies, so the specification can describe how agents should respond to the strategies of other agents, which provides a novel way to formally express adversarial interactions.
Critical weaknesses
- The problem statement only considers SL sentences with at most one quantifier alternation, which restricts the expressiveness that motivates the use of SL. The paper justifies this restriction by claiming that most adversarial RL problems can be expressed in this fragment, but provides no examples or references to support this claim. In addition, Theorem 5.3 assumes that the latent embedding is information-preserving and that the action-level response functions can represent every well-typed response function. The authors acknowledge that the first assumption may be hard to achieve under partial observability, but the second is also a strong condition that is unlikely to hold in practice.
- Although the authors evaluate SLICR on several environments, its scalability does not appear to be evaluated along two important dimensions. First, it is unclear how the method scales with the complexity of the specification, since even in a simple scenario such as Voyage, the inner LTL formula may become complex or deeply nested. Second, it is unclear how the method scales with the number of agents, since each additional agent introduces a new strategy quantifier in the SL sentence and may further increase the complexity of the inner LTL formula.
Questions for authors
- In Eq. (2), why is the objective defined as the probability that the robustness value converges to ? Since already implies that the specification is satisfied, it is unclear why convergence to is required. What is the effect of this choice compared with requiring only ?
- Why are irrational strategies restricted to observation-independent uniform-random policies? In adversarial settings, irrational or suboptimal opponents could still exhibit structured, history-dependent, or nonstationary behavior. How would the shielding mechanism generalize to such strategies?
- The authors mention that all existentially quantified agents receive the same reward. How is credit assignment handled in this case? The suggested decomposition may not be possible for general nested specifications with coupled objectives. How would SLICR handle such cases?
- How is strategy identity defined when constructing and ? Since a strategy is formally a mapping from finite histories to action distributions, what criterion is used to determine that two separately trained policies constitute different strategies? This is especially important for interpreting the Seen and Unseen settings.
4 comments
Reply to Reviewer 5XJV (2nd)
[Q1]
While guarantees for satisfaction satisfaction, maximizing gives equal credit to all satisfying trajectories, regardless of their robustness values. We chose convergence to as a stronger target in the favor of maximizing robust satisfaction, and this is a deliberate optimization choice. However, in practice, SLICR maximizes the expected cumulative discounted robustness in Eq.4, using robustness values as reward signals to guide learning. Hence, convergence target in Eq.2 is not enforced during training. It serves as a theoretical intermediate step connecting the problem statement in Section 4 to the objective in Eq.4.
[Q2]
In our work, we specifically define an irrational strategy as an observation-independent strategy that selects actions uniformly at random. In our terminology, an opponent with structured behavior can be treated as rational even if its strategy is suboptimal, since its behavioral patterns can still be learned by the agent model. The broader operational distinction between rational and irrational behavior depends on the chosen definition, and our analysis and experiments use the definition stated in the paper.
The shielding mechanism is a classifier trained on labeled interaction histories. To accommodate a different definition of irrational behavior, we can collect representative interaction histories, assign labels according to that definition, and retrain the classifier. The same shielding procedure would then use its predictions to determine which experiences contribute to training. Thus, the classifier can be adapted to alternative definitions through its training data and labels.
[Q3]
We first clarify that we do not claim to solve the credit-assignment problem in this work. The decomposition is mentioned because the POSG original formulation permits different reward signals for different agents. When agent-specific objectives and constraints are available, they can be expressed as , with each agent receiving a reward derived from its corresponding subformula . Each may itself contain nested temporal operators. For a specification with coupled objectives that does not admit a meaningful agent-specific decomposition, SLICR evaluates the full inner-LTL formula and provides its robustness value as a shared reward to all existentially quantified agents, as we did in our experiments.
[Q4]
We first clarify that, in the Seen setting, training and testing use the same opponent strategy set. In our experiments, opponent identity is associated with a particular pretrained strategy instance or opponent-team configuration. In the Unseen setting, we construct it using held-out strategies or configurations, varying the MARL training algorithms and behaviors. For example, in Google Research Football, the training opponents are trained using IPPO, while the Unseen test opponents are trained using MAPPO. We also include zonal defending in the test set. Similarly, in the SMAC-HARD maps 3m and 3s_vs_3z, we use VDN and QMIX opponents for training and QTRAN opponents for Unseen testing. Tables 3–5 in appendix provide the complete splits. The purpose of the Unseen setting is to evaluate how SLICR performs against opponent strategies or team configurations that it did not encounter during training.
If our responses address all your concerns, we kindly ask you to consider raising your score.
I thank the authors for their response and for the additional experiments. However, my concerns are not fully addressed.
- W2 (scalability). The authors point to agent counts from 4 (PP) to 10 (2s3z) and to specifications ranging from simple (Appendix E.1) to complex (Appendix E.3). However, the two grow together: each additional agent adds a strategy quantifier and can enlarge the inner-LTL formula. In the experiments, agent count varies only across SMAC-HARD maps, which also differ in unit types and opponents, and the specifications in Appendices E.1–E.3 come from different environments, so neither factor is varied in a controlled way. My concern is not the cost of computing robustness but how learning is affected as the number of agents and formula complexity increase. A controlled study within the same environment would address this, e.g., increasing the number of agents with the corresponding formula, and increasing the nesting depth of the inner-LTL formula for a fixed number of agents.
- Q2. My question was how the shield generalizes to irrational strategies beyond uniform-random ones, such as structured, history-dependent, or nonstationary behavior, which can occur at test time. The response suggests collecting new histories, relabeling them, and retraining the classifier. This concedes that the shield does not generalize to unseen irrational behavior: it must be retrained for each new type rather than adapting to it.
- Q3. My concern is whether a shared reward can handle deeper nested LTL formulas, where a single robustness value does not indicate which part of the specification each agent affects. The response only states that the robustness value is shared among all agents, as in the experiments, without addressing this.
- Q4. The response identifies a strategy by its pretrained instance, configuration, or training algorithm. However, the paper defines a strategy as a mapping from histories to action distributions (Sec. 3.1), so whether the Unseen setting satisfies depends on behavior, not on how the opponents were trained. Since opponents trained with different algorithms may exhibit similar behavior, the response does not provide a criterion consistent with the paper’s definition, and the Unseen results may not support what the authors report.
Therefore, I maintain my score.
Reply to Reviewer 5XJV (3rd)
We are grateful to the reviewer for considering our rebuttal. Also, we are delighted that we have addressed their concerns regarding their first weakness and first question. Below, we address their remaining concerns.
W2
We will discuss scalability with respect to the complexity of the inner-LTL formula and the number of agents (i.e., the number of quantifiers), below:
-
Complexity of the inner-LTL formula: We want to clarify that the complexity of the inner-LTL formula does not change SLICR’s model architecture, number of trainable parameters, or underlying learning. Inner-LTL formulas are evaluated through a robustness function, whose outputs serve as reward signals for the agents. Thus, changing the formula is similar to changing the reward function: it guides the agents toward the new specification, while the learning pipeline, model architecture, and parameter count remain unchanged.
We also want to clarify that the inner-LTL formulas presented in Appendix E were designed to address the benchmark objectives described in ln402–418. Making these formulas more complex may change the objectives they express. In that case, comparing task-performance metrics would not isolate the effect of formula complexity, because the agents would also be pursuing different objectives. This is similar to changing the reward function while evaluating scalability: the comparison would conflate changes in specification complexity with changes in the learning objective.
-
Number of Agents: First, we want to clarify that the scenarios used in our experiments are standard provided by the benchmarks. We follow their configurations, as is common MARL papers. Moreover, increasing the number of controlled agents while keeping the opponents fixed can make the task easier by giving the controlled team a numerical advantage. Such a comparison would conflate scalability with changes in task difficulty. To the best of our knowledge, SLICR is the first work, even within the broader scope of population-based adversarial MARL, to consider multiple agents on the controlled side.
Q2
We want to clarify that, in this work, irrational behavior is explicitly defined as observation-independent, uniform-random behavior, and the shield is trained according to this definition. How can we expect it to classify other behaviors as irrational at test time when those behaviors fall outside the definition of irrationality adopted in the first place?
In our previous response, by generalizable we meant that the shielding procedure can be adapted to different definitions of irrational strategies through the corresponding training data and labels. We were referring to the adaptability of the procedure, rather than claiming that the trained classifier automatically recognizes unseen forms of irrational behavior beyond its training definition.
Q3
We first want to clarify that solving credit assignment is beyond the scope of this work. Our focus is spec-guided adversarial MARL.
We can express agent-specific objectives through a decomposition of the inner-LTL formula as , where each may itself is deeply nested formula. This allows us to define separate robustness-based rewards for each agent, consistent with the POSG formulation allowing different rewards functions for different agents.
Your concern is valid, but we never claim to solve credit assignment in our paper.
Q4
We want to clarify that we do not claim that and exhibit qualitatively different behaviors in Unseen setting. Our claim:, meaning that we use different strategies for testing in the Unseen setting. Under the paper’s definition, two strategies are different if they produce different action distributions for at least one common history. Similar behavior does not imply identical strategies. Our experiments also show differences in performance between the Seen and Unseen settings.
For example, in 2s3z (see Tab.5), we used VDN for training and QMIX for testing. The VDN strategies achieved win rate of 93%, while the QMIX strategies achieved 96% against built-in AI. Additionally, we used built-in AI at difficulty level 8 for training and 10 for Unseen testing. Also theoretically, VDN sums individual Q-values, while QMIX uses a learned monotonic mixing function to preserve consistency between individual and joint greedy action selection.
Furthermore, even changing the seed may produce different results, which motivates conducting multiple runs and reporting measures such as the 25–75th percentiles or std errors. This can be seen in well-known cooperative CTDE MARL papers such as: VDN, QMIX, QTRAN, and QPLEX (Wang et al., ICLR'21).
Reply to Reviewer 5XJV (1st)
We thank the reviewer for their comments. We respond to the critical weaknesses in this comment (1st) and the questions in (2nd) comment.
$\forall^* \exists^* $ and $\exists^* \forall^* $ fragments capture important strategic objectives in two-player and two-team adversarial MARL: learning responses that depend on opponent strategies and learning a single controlled strategy profile that works against all opponent strategy profiles, respectively. These objectives are relevant to many adversarial MARL benchmarks, including:
In addition, the existence of a Nash equilibrium can also be expressed in SL using at most one quantifier alternation (Beutner, AAMAS 2024). While this is well-established in various areas of computer science, we also revise the discussion in Section 4 in (red) to clarify our motivation.
The assumption that action-level response functions can represent every well-typed response function ensures that the action-level formulation preserves the optimum of the strategy-level objective in Eq.3. This allows Theorem 5.3 to establish the connection to Lemma 5.2. However, in practice, SLICR directly optimizes the action-level objective in Eq.4. Thus, representability is not required for training or executing SLICR and is assumed to establish the theoretical guarantees. This is why our limitations discussion emphasizes the information-preserving embedding assumption rather than representability. We also add more text in limitation paragraph in Section 7 to clarify these points.
We want to clarify that our evaluation includes variation in both the complexity of the inner-LTL formula and the number of quantified agents.
-- Number of Agents. Regarding agent scalability, our evaluation also varies the number of agents and corresponding strategy quantifiers, from 4 in Predator–Prey to 6 in the SMAC-HARD maps 3m and 3s_vs_3z, and 10 in 2s3z. Thus, our experiments include specifications involving multiple existentially and universally quantified agents. For the rebuttal, we have additionally included decentralized (using IPPO) SMAC-HARD results in the revised Fig. 3 to further evaluate SLICR in this setting. We also want to clarify that scalability with the number of agents is a general challenge in multi-agent RL. Nevertheless, to the best of our knowledge, ours is the only work on agent modeling against populations of adversaries evaluated on such a diverse set of complex case studies, including three Google Research Football scenarios (one added for the rebuttal) and three SMAC-HARD maps.
-- Complexity of the inner-LTL Formula. The complexity of the inner-LTL formula has no or little impact on the structure of SLICR’s learning framework. SLICR translates the inner-LTL formula into a robustness function using quantitative semantics, and its outputs serve as reward signals (computing min and max in sets can be done in linear time). Furthermore, in Appendix E, our specifications range from the relatively simple φpp in Appendix E.1 to the more complex φSMAC in Appendix E.3, which includes nested temporal operators.