acceptodds
Under review as a conference paper at ICLR 2027

When Nature Whispers to the Opponent: Information Leakage in Robust Markov Games

Abstract

Robust Markov games contain two adverse entities: a strategic opponent and nature, which governs uncertain transitions. Fusing them into a single adversary can change the information structure: if nature observes a hidden mode, the fused opponent may condition on information unavailable to the modeled opponent. We formalize the intended counterfactual by relaxing only the opponent's information while preserving the ambiguity family, transition dynamics, timing, and one complete nature plan. A four-value hierarchy separates this information effect from stronger relaxations that reoptimize continuation plans across public actions or private histories. For each fused profile, we construct a legal public-history shadow with the same nature plan; the KL divergence from the fused law to its shadow equals the cumulative conditional mutual information carried by opponent actions, yielding information-budget bounds and a zero-leakage exactness criterion. Finite-game, learning, pursuit-evasion, and cyber-defense experiments validate the distinction. In cyber defense, fusion raises a policy's worst-case safety cost from to , reversing safe policy selection at threshold . For learned attackers, rollout estimates of leakage and public-shadow values match enumeration, while leakage regularization yields a value-leakage tradeoff.

Primary area:
reinforcement learning
Keywords:
robust Markov games, robust reinforcement learning, stochastic games, asymmetric information, information structure, information leakage, safe reinforcement learning, mutual information

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5MYoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies information leakage in robust Markov games when the strategic opponent and nature are combined into a single adversary. The authors introduce a four-value hierarchy to distinguish information relaxation from nature-plan reselection. They then construct a public-history shadow strategy and use KL divergence and conditional mutual information to bound the value gap caused by information leakage. Experiments on finite games, cyber defense, pursuit–evasion, and learned attackers illustrate the theoretical results and show that information leakage can affect robust safety evaluation and policy selection.

Critical strengths
  1. The presentation of this paper is complete. The topic of this work is interesting and useful.
  2. The authors provide a set of solid mathematical proofs.
  3. The experiments efficiently support the proposed method.
Critical weaknesses
  1. The four values ​​(V00V_{00}, V10V_{10}, V11V_{11}, V∞V_{\infty}), KL divergence, and conditional mutual information in Section 4 are not closely related to the preceding content. This transition feels too abrupt, making it somewhat difficult to follow along while reading.
  2. The explanations for Figures 2 to 6 in the main text are not sufficient. Although the author seems to want to convey a lot of information, I couldn't fully understand them.
  3. In Equation 3, what is the mathematical definition for IPI_P?
  4. In Figure 4, what are the mathematical definitions of mean fusion gap and positive-gap frequency?
  5. Although the author provides definitions for the four values ​​(V00V_{00}, V10V_{10}, V11V_{11}, V∞V_{\infty}) in the table, they are still relatively abstract to understand.
Questions for authors

Please review my weaknesses.

Reviewer_pxvQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose fusing the more knowledgeable “nature” with the strategic opponent in a Robust Markov Game. This changes the information structure, as if nature observes a hidden mode, the fused adversary can condition on this new information a typical strategic opponent would not have. The authors develop a theoretical framework to study this fused adversary. In it, they prove a connection between the amount of hidden information used by the informed attacker and how much it can exceed that of the uninformed adversary. Experiments support this conclusion, where a fused adversary in a cyber defense setting raises the policy’s worst-case safety cost compared to its uninformed counterpart.

Critical strengths
  1. Good theoretical insight. The authors identify three ways to make an adversary more powerful and prove that each additional level of freedom increases the adversary’s worst case value.

  2. The authors further prove that the difference between the fused adversary and original has an information-theoretic bound. Theory is well-explained and well-posed using information theory and robust control literature.

Critical weaknesses

The main cyber-defense experiment is a simplified model and not necessarily realistic.

It is unclear whether the theoretical bounds support practical safety decisions in larger, existing games.

Please see questions for authors

Questions for authors

1.Could the authors expand on the practical scope and scalability of this framework? Specifically:Are there additional realistic or benchmark domain settings where nature and the strategic opponent naturally 'fuse' or share hidden state information?

2.Does the proposed theoretical framework extend to large-scale robust Markov games where exact enumeration is computationally infeasible?

  1. I’d like to understand the practical implications of the proposed bound. Within the cyber-defense setting, is the bound tight enough to resolve possible safety decisions? What would this look like in the context of this running example?

  2. Could the authors clarify how Theorem 2 advances prior work. Do existing causal information-project results already establish a similar guarantee under similar assumptions? I would like to better understand if the novelty is the formulation of the robust-game comparison or if there were new mathematical techniques employed.

Reviewer_fSSBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates impact of treating a strategic oppoennt and nature as a single adversary in Markov games. The main takeaway is that fusion makes nature's private information available to the opponent. The value change due to this private information is bounded by how much private information the opponent's actions leak, measured against a public shadow opponent that keeps nature's plan fixed.

Critical strengths

Information structure of the adversary is a relevant problem for robust MDPs and robust safe RL. Paper's central argument that "alignment alone does not imply a shared information set" is well motivated, and stating it in robust MDP terminology makes known results accessible to that community.

Mostly, the theory development appears sound and complete.

Critical weaknesses

(Novelty) Several results in this paper have close counterparts in games with incomplete information and robust optimzation. I am unable to see clearly what is new. For instance, how is the Bayes-plausible split in Sect. 4.2 different from the splitting lemma [1] and the Bayes-plausibility condition [2]? For Markov chain games with lack of information on one side, [3] distinguishes "using the information" from "revealing a relevant information". How is the V00 to V10 step, and the leakage notion, different from this distinction? How is the budgeted value in Eq. 17 different from entropy-constrained games and persuasion [4]?

[1] R. J. Aumann and M. Maschler, 1995, Repeated Games with Incomplete Information, MIT Press. [2] E. Kamenica and M. Gentzkow, 2011, Bayesian persuasion, American Economic Review 101(6). [3] J. Renault, 2006, The value of Markov chain games with lack of information on one side, Mathematics of Operations Research 31(3). [4] A. Neyman and D. Okada, 2000, Repeated games with bounded entropy, Games and Economic Behavior 30(2).

Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Awaiting Your Feedback on Our Rebuttal

Dear Reviewer fSSB,

Thank you again for your review. As the discussion period is drawing to a close, we would appreciate it if you could take a look at our rebuttal. We would be happy to clarify any points or address any remaining concerns while the discussion is still open.

Best regards,

The Authors

Comments on the paper

1 comment
Area_Chair_WKRXOpenReview ↗

Reviewer–Author Discussion Period Ending Soon

Dear Reviewers,

With the discussion period coming to a close, I encourage you to revisit the authors’ responses and let us know if you have any further comments.

Thank you for your careful reviews and engagement in the discussion!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.