acceptodds
Under review as a conference paper at ICLR 2027

Exploitability-Guided Distillation of Interpretable Strategies

Abstract

Counterfactual regret minimization computes strong strategies for imperfect-information games, but as large tables of mixed strategies that people can neither audit nor execute. Deploying them in human-facing settings means distilling them into small decision trees, and the standard objective for that distillation is a distributional distance to the teacher. We show this objective is misaligned with strategic quality: at the same size, a leaf fit by imitation can be far more exploitable than one fit by exploitability. We propose exploitability-guided distillation. The tree is grown best-first under an exploitability split score, and leaf strategies are optimized directly against per-player best responses, using the separability of NashConv into independent per-player objectives; a single scalarized objective then traces the safety-fidelity frontier. On Leduc poker, best-response leaf fitting is 2.3x to 4.3x less exploitable than imitation fitting across budgets of 4 to 32 leaves, and at aligning the split score with the leaf objective cuts exploitability by 36 to 43 percent; the gain carries over to a five-rank variant. Our contributions are a bound that identifies the weights imitation misassigns to leaves, a distillation pipeline whose structure search and leaf solve share the exploitability objective, and an exact full-game evaluation showing that the leaf objective, with the teacher reduced to a uniform profile, returns the same Leduc structures at .

Primary area:
reinforcement learning
Keywords:
Imperfect Information Games, Policy Distillation, Interpretable Strategy, Multi-Agent Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6h6nOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to compress large strategies computed by counterfactual regret minimization (CFR) into small, interpretable decision trees for imperfect information games. The authors show that conventional imitation based distillation can overlook strategically important information sets, where even small distributional errors may lead to substantial exploitability. To address this issue, they propose exploitability guided distillation, which directly minimizes the payoff achievable by a best-responding opponent instead of solely imitating the teacher's action distributions. The proposed method combines best response based leaf optimization with a tree growing procedure that aligns split scoring with the final optimization objective. The paper also provides a theoretical bound explaining the mismatch between imitation loss and exploitability.

Critical strengths

The paper addresses an important problem in strategy distillation for imperfect information games. When a complex strategy is compressed into a small decision tree, minimizing imitation loss may overlook information sets that occur infrequently but have substantial strategic consequences. As a result, a student strategy that closely imitates a strong teacher can still be highly exploitable. I find this observation meaningful, as it highlights a limitation of conventional imitation-based distillation.

In addition, the paper presents a well structured investigation, connecting the research motivation, theoretical analysis, algorithm design, and experimental evaluation. The authors provide a concrete counterexample and a theoretical bound to explain the limitations of imitation, then develop an exploitability guided approach for both tree construction and leaf optimization. The experiments include controlled comparisons and ablation studies that help identify the contributions of different components. Overall, I find the research framework coherent and the experimental analysis relatively thorough.

Critical weaknesses

Limited evaluation of performance when both players adopt the proposed strategies.

The proposed method directly minimizes NashConv, and its experimental evaluation primarily relies on the same metric. While NashConv is a valid measure of worst case exploitability, I am concerned about how well this evaluation reflects actual gameplay, particularly when both players adopt strategies produced by the proposed method. In this setting, neither player necessarily follows a best response to the other, so the worst case exploitation measured by NashConv may rarely be realized and thus other policies would perform better when its opponent use this policy. Consequently, lower exploitability does not necessarily imply higher expected returns when both players use the distilled strategies.}

Questions for authors

The proposed method optimizes NashConv, which measures exploitability against unrestricted best responses. However, I am interested in how the resulting strategies perform when both players initially adopt the proposed distilled strategies and one player subsequently deviates. For example, Table 3 suggests that the distilled strategy follows relatively predictable folding rules, which might be exploitable through simple adjustments in bluffing frequency.

Could the authors evaluate the actual expected returns under mutual deployment and under deviations restricted to similarly small decision trees? This would help clarify whether the resulting strategies remain robust against simple, practically executable deviations, rather than only characterizing their worst-case exploitability.

1 comment

We thank the reviewer for the careful reading of our manuscript.

Weakness

Limited evaluation of performance when both players adopt the proposed strategies.

To evaluate performance beyond best-response opponents, we now report the exact expected payoff in chips per hand (averaged over both seats and 8 seeds) of every arm against the CFR+ teacher and against a weaker CFR+ opponent (1000 iterations), on Leduc at B=8B=8:

armNashConvvs teachervs weak CFR+
aligned best-found0.867−0.155−0.153
KL best-found1.266−0.298−0.298
expl.-split best-found1.579−0.402−0.402
grouped CFR+4.157−0.633−0.631
KL imitation4.461−0.564−0.564
expl.-split imitation3.954−0.638−0.638

Against a strong opponent that is not best-responding, the aligned tree loses the least, and the three best-found arms lose less than the three imitation-level arms. Payoffs against the teacher and against the weaker opponent agree within 0.003 chips per hand, so the ordering does not depend on the opponent's strength. The picture holds across budgets: at B=16B=16 the aligned tree (−0.154) is within 0.001 of the best arm, and at B=32B=32 (−0.140) it is the best of all six (Appendix A). Deviations from mutual deployment are answered under the question below.

Question

The proposed method optimizes NashConv, which measures exploitability against unrestricted best responses. However, I am interested in how the resulting strategies perform when both players initially adopt the proposed distilled strategies and one player subsequently deviates.

Starting from mutual deployment of the student profile on Leduc (B=8B=8), we measure one player's exact gain (chips per hand, medians over the 16 seed-seat cells) at four deviation levels: refitting one leaf of its own tree, refitting all leaves on the fixed structure, regrowing its own BB-leaf tree against the fixed opponent, and the unrestricted best response (BR):

armone leafall leavesregrown BB-leaf treeunrestricted BR
aligned best-found0.0000.0000.1330.433
KL best-found0.2050.2050.3790.533
expl.-split best-found0.7570.7570.7570.859
grouped CFR+0.0000.1081.3912.149

Within the fixed tree, the aligned profile is a Nash equilibrium of the restricted game: no one-leaf or all-leaf refit gains anything. An opponent who regrows its own eight-leaf tree captures 31 percent of the unrestricted gain, the smallest share of the six arms. At B=16B=16 the one-leaf gain is 0.023 and the regrown-tree gain 0.156 of 0.404 (39 percent), again the smallest share (Appendix A, Table 7).

For example, Table 3 suggests that the distilled strategy follows relatively predictable folding rules, which might be exploitable through simple adjustments in bluffing frequency.

We probed exactly this against the fixed tree of Table 3 (Appendix A). An opponent who opens with a bet with probability bb instead of checking loses value linearly in bb: the tree gains 0.30 (opponent first to act) or 0.33 (opponent second) chips per hand at b=1b=1, because the tree re-raises with the top card and the opponent folds at the raise cap. The full-game best response never opens with a bet; it bets in the second round after a check, which is the 0.433 gain NashConv already counts. The folding rule is exploitable, NashConv prices that exploit, and bluffing more is not the way to it.

Could the authors evaluate the actual expected returns under mutual deployment and under deviations restricted to similarly small decision trees? This would help clarify whether the resulting strategies remain robust against simple, practically executable deviations, rather than only characterizing their worst-case exploitability.

Yes, both are now reported: payoffs under mutual deployment against fixed opponents in the table under the weakness above (Section 5, Table 5, and Appendix A), and deviations restricted to trees of the same leaf budget in the table above (Table 5, columns "one leaf" and "regrown tree").


We believe the revised manuscript and the new tables address the points raised, and we will answer any further question during the discussion period.

Reviewer_WgewOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates compressing strategies for two-player zero-sum imperfect-information games into decision trees with a fixed leaf budget. It explains why imitation objectives can be misaligned with full-game exploitability and proposes optimizing both leaf strategies and candidate splits using an exploitability-based objective. A KL regularizer controls fidelity to the teacher. The experiments separate leaf-fitting and structure-search effects and include teacher-free runs, compute-matched controls, alternative imitation weightings, additional games, and numerical sensitivity analyses.

Critical strengths
  1. The empirical controls provide a convincing analysis of objective mismatch. Comparisons on identical structures isolate the contribution of leaf fitting. Screening counterfactuals, deterministic controls, teacher removal, and compute-matched KL restarts further support the conclusion that aligning split selection with the final objective can improve the resulting strategy.

  2. The evaluation and interpretable outputs are concrete and well documented. Strategies are evaluated through full-game best responses, with independent OpenSpiel checks. The paper supplies executable rules and verifies their evaluation after rounding. The supplement provides experiment drivers, environment requirements, and commands for reproducing the principal comparisons.

Critical weaknesses
  1. The methodological novelty is relatively focused. The paper explicitly relates its objective to prior work on optimizing restricted strategies against full-game best responses. Its clearest additional contribution is aligning greedy tree growth with leaf optimization, supported by careful empirical controls. This is useful, but the contribution is stronger as a systematic study of objective alignment than as a substantially new game-solving principle. This limits the breadth of the methodological advance without undermining the reported comparisons.

  2. The complete structure-search procedure remains expensive relative to the evaluated game sizes. Appendix H reports approximately 74 minutes of single-core growth for eight leaves on Leduc and 4.5–10 hours for sixteen leaves. A dedicated single-core run on the 780-information-set five-rank variant takes approximately 4.6 hours to reach eight leaves. The appendix explains why inexpensive candidate screening can lose the benefit, but that finding also exposes a practical limitation: the cheaper variant can sacrifice the gain motivating aligned search. Liar’s Dice strengthens the leaf-fitting evidence, while the complete aligned-search evaluation remains concentrated in the poker family.

  3. The headline fidelity improvement depends substantially on its numerical measurement convention. Appendix L appropriately provides clipping sensitivity results: the recovered fraction of the attainable fidelity gap decreases from 80–85% at a 10⁻¹² floor to 31–52% at 10⁻⁴. The regularized endpoint remains comparatively stable, so this does not invalidate the exploitability–fidelity trade-off. It does limit the interpretation of the headline percentage, however: much of the relative improvement concerns extremely small action probabilities, and the percentage alone does not establish a similarly large improvement in behavioral fidelity.

  4. The supplement facilitates rerunning experiments more than independently auditing the implementation and published aggregation. The README supplies explicit drivers, dependencies, and commands, including the compute-matched control. However, the numerical core is distributed only as Linux x86-64 binaries, and the existing result files and offline aggregation, audit, and plotting scripts are excluded. Inspecting implementation details or tracing the published summaries therefore requires substantial independent reconstruction or expensive reruns. This is a practical auditability limitation of an otherwise carefully documented empirical study.

Questions for authors

See weaknesses.

2 comments

We thank the reviewer for the detailed review and for naming the empirical controls as a strength of the work. We answer the four weaknesses in order.

W1

The methodological novelty is relatively focused. The paper explicitly relates its objective to prior work on optimizing restricted strategies against full-game best responses. Its clearest additional contribution is aligning greedy tree growth with leaf optimization, supported by careful empirical controls. This is useful, but the contribution is stronger as a systematic study of objective alignment than as a substantially new game-solving principle. This limits the breadth of the methodological advance without undermining the reported comparisons.

The contribution list of Section 1 claims precisely this scope. The objective is the full-game best-response objective of Johanson et al. (2012), cited in Section 2. What is new is three things: Proposition 1 and its counterexample, which show why imitation misallocates error; the separability lemma, which makes the per-player leaf problem exact; and carrying the objective into structure search, where the prefilter counterfactual and the compute-matched control locate the gain in the ranking of candidate splits. The revision keeps that claim and states each result in one sentence in Section 1.

W2

The complete structure-search procedure remains expensive relative to the evaluated game sizes. Appendix H reports approximately 74 minutes of single-core growth for eight leaves on Leduc and 4.5–10 hours for sixteen leaves. A dedicated single-core run on the 780-information-set five-rank variant takes approximately 4.6 hours to reach eight leaves.

Total cost equals the number of candidate evaluations times one candidate solve; on Leduc, growing to B=16B=16 scores 3.3 times as many candidates per seed as B=8B=8 (926 against 283), which accounts for the 4.5–10 hour figure. The revision adds a main-text paragraph "Cost and scope of the aligned search" with Table 4, and Appendix H, with two reductions:

  1. Solving each distinct partition once. A card test and its complement induce the same partition; this removes 31–36% of candidate solves on Leduc (B=8B=8 and B=16B=16) and 19–24% on the five-rank variant and Liar's Dice, without raising NashConv.
  2. Exact parallel scoring. The chosen split is an argmin over deterministic scores, so at 1, 4, 16 and 64 processes the search returns the same tree, the same NashConv and the same evaluation counts:
game (one seed, B=8B=8)CPU seconds of workwall seconds on 64 processes
Leduc61288
Leduc(5,2)8,502791

The five-rank search to eight leaves completes in 13 minutes of wall time on one 64-core node with a bit-identical result.

The appendix explains why inexpensive candidate screening can lose the benefit, but that finding also exposes a practical limitation: the cheaper variant can sacrifice the gain motivating aligned search.

The trade is now quantified under one protocol on the five-rank variant at B=16B=16 (Table 4 in the main text, Appendix H). We report objective evaluations rather than seconds because evaluation counts do not depend on the machine:

armIQM NashConvevaluations per seed
K=8K=8 prefilter0.575128k
K=64K=64 prefilter0.5311.15M
all candidates0.5332.19M

Fine-scoring every candidate costs 17 times the evaluations of the K=8K=8 prefilter for a NashConv improvement of 0.042; a K=64K=64 prefilter reaches the all-candidate value with half its evaluations. On Liar's Dice (Table 4) the K=8K=8 prefilter already reaches the all-candidate value 0.1250 at B=24B=24. The contribution of the paper is to show that the gain is in the ranking of candidates and to give the practitioner the price of recovering it; the user chooses a point on this curve.

Liar's Dice strengthens the leaf-fitting evidence, while the complete aligned-search evaluation remains concentrated in the poker family.

We ran the complete aligned search on the paper's Liar's Dice game, 8 seeds, grown to B=24B=24 and evaluated with the full best-found leaf solve (exact NashConv, IQM over 8 seeds):

armIQM NashConv at B=24B=24per-seed range
lifted best-found (strongest baseline on this game)0.14590.0625–0.2688
aligned, K=8K=8 prefilter0.1250spread 3×10−53 \times 10^{-5}
aligned, all candidates0.1250spread 7×10−67 \times 10^{-6}

The aligned score lowers the IQM by 14 percent and removes the seed variance of the baseline, and the prefiltered and all-candidate arms coincide on this game. These rows are now in the main text (Table 4), with the protocol in Appendix K.

W3

The headline fidelity improvement depends substantially on its numerical measurement convention. Appendix L appropriately provides clipping sensitivity results: the recovered fraction of the attainable fidelity gap decreases from 80–85% at a 10⁻¹² floor to 31–52% at 10⁻⁴. The regularized endpoint remains comparatively stable, so this does not invalidate the exploitability–fidelity trade-off. It does limit the interpretation of the headline percentage, however: much of the relative improvement concerns extremely small action probabilities, and the percentage alone does not establish a similarly large improvement in behavioral fidelity.

The revision replaces the headline percentage in Sections 1 and 5 and in the conclusion with clip-free quantities evaluated on the same Leduc sweep (11 values of η\eta, 4 budgets, 8 seeds). At every budget both the reach-weighted total variation (TV) to the teacher and the Jensen-Shannon divergence (JS) fall with η\eta:

budgetTV (η=0\eta=0)TV (η=1000\eta=1000)JS (η=0\eta=0)JS (η=1000\eta=1000)
B=4B=40.510.420.310.16
B=8B=80.480.410.280.15
B=16B=160.460.380.260.14
B=32B=320.400.330.220.12

The L1L_1 distance on actions the teacher plays with probability at least 0.01 falls likewise (0.70 to 0.56 at B=4B=4, 0.58 to 0.45 at B=32B=32), and the aligned structures show the same TV decrease (0.47/0.46/0.41 to 0.39/0.38/0.30 for B=8/16/32B=8/16/32). The first step η=0.01\eta=0.01 is now described for what it is: it removes 77–81% of the student's mass on actions the teacher plays with probability below 10−1210^{-12}, for a NashConv rise of at most 0.016. The table above is Table 6 of the main text; the captured-share percentage remains only in Appendix L next to its clip sensitivity.

W4

The supplement facilitates rerunning experiments more than independently auditing the implementation and published aggregation. The README supplies explicit drivers, dependencies, and commands, including the compute-matched control. However, the numerical core is distributed only as Linux x86-64 binaries, and the existing result files and offline aggregation, audit, and plotting scripts are excluded. Inspecting implementation details or tracing the published summaries therefore requires substantial independent reconstruction or expensive reruns. This is a practical auditability limitation of an otherwise carefully documented empirical study.

The revised supplementary material now includes what is needed to trace the published summaries without rerunning anything:

  • Result files: every per-run and merged result file (JSON and CSV) behind the tables and figures of the paper and of this rebuttal.
  • Offline scripts: the aggregation, audit, and plotting scripts that turn those files into the reported tables and figures, with a note describing the layout.

The numerical core remains distributed as compiled modules during the review phase, and we commit to releasing the complete source code upon acceptance, including the game definitions, CFR+, exact best response, and the distillation search. The revised Reproducibility statement states both points. Every NashConv in the paper is an exact full-game best-response evaluation, and our evaluator is cross-checked against OpenSpiel to a relative error of 10−1310^{-13} (Appendix E), so the reported values can be verified independently of our implementation.


We believe the revised manuscript and the new tables address the points raised, and we will answer any further question during the discussion period.

Reviewer_k5kROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The writing of this paper is heavily AI generated to the point that I cannot read this paper. This is unfortunate, because there is an interesting idea inside of this paper, namely, that distributional distances are not necessarily good estimators of exploitability. I recognize that what I am saying is rude, but so is asking a reviewer to spend their time reviewing something this unpolished and which should have received many rounds of editing before it was submitted.

Here is my default set of responses to authors of papers like these so they can write better papers in the future:

  • It is not normal, in clear writing, to fill a sentence with the exact values of numbers instead of prose.
  • Individual paragraphs are filled with low-level details, making it challenging to understand what idea is present. If the authors rewrite the prose to clearly list the ideas, and move numeric details to their appropriate spots instead of littered randomly throughout the paper, it may become possible to review it.
Critical strengths

Read summary.

Critical weaknesses

Read summary.

Questions for authors

N/A

1 comment

Thank you for taking the time to comment on the presentation and readability of our submission.

It is not normal, in clear writing, to fill a sentence with the exact values of numbers instead of prose.

You are right about the number density of our text, and The revision corrects it (changes marked in red). The numbers moved into tables now.

Individual paragraphs are filled with low-level details, making it challenging to understand what idea is present. If the authors rewrite the prose to clearly list the ideas, and move numeric details to their appropriate spots instead of littered randomly throughout the paper, it may become possible to review it.

Every paragraph of Section 5 now opens with its idea in plain prose, and low-level details are in the appendices. The method section no longer quotes experimental values, and the one-sentence Appendix A of the submission was removed. The general response lists every change with its location.

The writing of this paper is heavily AI generated to the point that I cannot read this paper.

The AI Use Statement of the submission already discloses that generative AI tools assisted with language polishing and with code. The idea, the algorithm, the structure of the paper, and the content of every paragraph are the authors' own. The density of numbers was a defect of style that we have now fixed. We do not think the submitted text was unreadable to the point of preventing a technical review: the other two reviewers evaluated the manuscript in detail, including its appendices.

With the presentation issues resolved, we respectfully ask you to read the revised Sections 1 and 5 and to reconsider your assessment and score in light of the paper's technical content. We welcome engagement on the substance of the work and will address any further concern during the discussion period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of changes in the revised PDF

We thank the three reviewers for their constructive feedback. Their comments share three themes: policy quality beyond worst-case NashConv, the cost of the complete aligned structure search, and readability and auditability. We revised the paper to address all three. All changes in the revised PDF are marked in red.

Main text

  • Section 5 rewritten: every paragraph opens with its idea and numbers live in tables. Numerals per 100 words of prose fall from 10.3 to 3.4 in Section 5, from 3.9 to 1.1 in Section 1, from 3.9 to 2.2 in the abstract, and from 8.2 to 4.5 in the conclusion; Section 4 no longer quotes experimental values.
  • New paragraph "Cost and scope of the aligned search" with Table 4: exact parallel scoring, the prefilter trade on the five-rank variant at B=16B=16, and the complete aligned search on Liar's Dice.
  • New paragraph "Beyond the worst case" with Table 5 (Leduc, B=8B=8): payoffs against the teacher and a weaker CFR+ opponent, and restricted-deviation gains.
  • The fidelity headline of Sections 1 and 5 and of the conclusion now uses clip-free distances, reported in the new Table 6.

Appendix

  • New Appendix A: the B=16B=16 play table and the bluffing probe. It replaces the one-sentence Appendix A of the submission, so Appendices B to L keep their letters.
  • Appendix E: the tree-overlap ranges that left Section 5. Appendix I: the candidate-rank detail that left Section 5.
  • Appendix H: what drives the search cost, deduplication of equivalent candidates, exact parallel scoring, and the prefilter trade.
  • Appendix K: protocol of the complete aligned search on Liar's Dice.

Supplementary material and code release

  • The supplementary archive keeps the submitted package and adds every result file (per-run JSON, merged CSV) behind the tables of the paper and of this rebuttal, together with the aggregation, audit, and plotting scripts.
  • The Reproducibility statement commits to releasing the complete source code, including the numerical core, upon acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.