acceptodds
Under review as a conference paper at ICLR 2027

Sharp Norm Conditions for Logit Fixed Points

Abstract

Entropy regularization is often used in multi-agent learning to reduce instability caused by agents influencing one another. A natural intuition is that learning should be easier to stabilize when a change in one agent's behavior has only a limited effect on the others. In this paper, we show that the size of a payoff change alone can be misleading: even a large change may leave an agent's choices unchanged. What matters is whether the change alters the relative payoffs of different actions. Building on this observation, we derive a new sufficient condition for stability of logit responses that accounts only for changes that affect choice, and specifies how much regularization is enough. Under this condition, responses converge to the same fixed point from every initial state. A two-action example establishes the sharpness of the condition's universal constant: immediately beyond the bound, the system can have several distinct long-term outcomes. Experiments show that bounds based on the full payoff matrix can count changes that leave choices unchanged, requiring unnecessarily strong regularization or failing to guarantee convergence in systems that do converge. The new condition avoids this problem. We believe these findings help explain stability in multi-agent learning and guide the choice of regularization strength.

Primary area:
learning theory
Keywords:
Logit response, Entropy regularization, Multi-agent learning, Quantal response equilibrium, Natural policy gradient.

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_dehWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates when interactive agents achieve quilibruim when updating their choices with softmax. The core discovery is that, the stability depends on how the changes in choices affect relative payoffs. This is because, common increase/decrease payoffs will not change agent choices. The paper uses a projection matrix WtanW_tan to retrieve effective interactions. They prove that when β∣∣Wtan∣∣2<2\beta ||W_{tan}||_2<2, same equilibrium will be achieved regardless of starting point. They further give an example to show that the constant 2 can't be loosen under some conditions.

Critical strengths
  1. The way they develop to remove commmon payoffs across agents with a projection operator and catch the real element controling equilibrium is clean and elegant.
  2. The two-action example is useful. It proves that the constant 2 connot be imporved in a universal condition based only on this projected norm. This is a clean and clear boundary for the main theorem.
Critical weaknesses
  1. I hesitate if the ammount of theoretical novelty is enough. The core proof combines existing softmax sensitiviy bounds, projections, standard contraction mapping theory. Their primary contribution concentrates on refining the convergence criterion.
  2. Limited experiments. They only use synthetic systems, while natural policy gradient guranttes assume exact payoff evaluation. Therefore, i don't know how much benefit the results can provide in realistic noisy multi-agent learning settings.
Questions for authors
  1. Clarify (best list in bullet point) the theoretical contributions, like specifically what key steps/derivations are core contributions of the work.
  2. Try to test natural policy gradient with sampled payoff estimates. Specificailly, compare convergence speed and final error (across like batch sizes, regularization strenghs) to see if the developed theory still work when payoffs are noisy.
2 comments
AuthorsOpenReview ↗

Response to Reviewer dehW

We sincerely appreciate your insightful and professional comments. We are delighted by your recognition of our core idea. To fully address your concerns, we provide a more detailed discussion and experiments below.

1. The core contributions

The key derivations are:

  1. From payoff changes to choice sensitivity. We project feasible probability changes and relative payoff changes on the two sides of the response derivative. This yields β∥Wtan∥2<2\beta\lVert W_{\mathrm{tan}}\rVert_2<2 and quantifies projection's improvement while holding the known softmax constant fixed. For the LLM tool example, the measured quantity is how another agent changes the preference between tools (Section 3.1).
  2. From a sufficient condition to an optimal universal constant. The two-action construction produces multiple fixed points beyond the norm boundary. The added two-player coordination game proves that the coefficient 1/21/2 in the finite-game NPG condition τ>LT/2\tau>L_T/2 cannot be reduced (Appendix E.1).
  3. From response sensitivity to learning with estimated rewards. Centering log probabilities turns the multiplicative NPG update into an additive recurrence. Contraction and reward-estimation noise then enter the same error bound, giving admissible step sizes, geometric decay of initial error, and a sampling-error term (Section 3.4; Appendix E.2).

Two further results identify useful structure: the Hadamard construction keeps the Euclidean contraction bound at 1/21/2 while the specified block bound grows as m/2\sqrt m/2 (Appendix B.3); symmetry gives an additional uniqueness condition through the entropy-regularized potential (Section 3.3).

2. Learning from sampled rewards

The new analysis gives convergence and error guarantees when exact rewards are replaced by unbiased estimates with bounded conditional variance. Initial error decays geometrically; with a constant step size, the remaining mean-square error bound decreases as 1/M1/M; diminishing steps give convergence. These results include policy error and QRE gap (Appendix E.2).

For the tool-selection example, an agent may observe only the reward of the tool it uses. Appendix E.3 gives an estimator for this setting when joint actions and sampling probabilities are observed. The experiments also estimate the regularization coefficient from these observations:

CheckResult
Sampling-error predictionPolicy-error slopes −1.007-1.007 to −0.963-0.963 at τ=LT\tau=L_T, consistent with 1/M1/M.
Projected norm estimated from exploration observations4.24% mean absolute relative error; every chosen coefficient exceeds the true sufficient bound in our tests.
Learning with that estimate at M=256M=256, relative to the known full-matrix choiceNash-gap difference −0.00179±0.00195-0.00179\pm0.00195; executed-reward difference 0.00045±0.000380.00045\pm0.00038.

Exploration counts toward the same total learning budget. The Nash-gap interval includes zero; the reward interval supports an improvement. See Appendices F.2 and F.4.

3. Batch size, regularization, speed, and final error

The added comparisons identify how batch size and regularization affect both remaining error and learning speed. QRE gap measures the gain from a unilateral policy change in the regularized game. Let τP=1.05LT/2\tau_P=1.05L_T/2 and τF=1.05∥W∥2/2\tau_F=1.05\lVert W\rVert_2/2.

ComparisonParameter ruleResult
Batch size M:1→256M:1\to256τ=τP\tau=\tau_P, ατ=0.5\alpha\tau=0.5Mean QRE gap: 0.0364 → 0.00018.
Regularization τ:τP→LT\tau:\tau_P\to L_TM=1M=1, ατ=0.5\alpha\tau=0.5Mean QRE gap: 0.0364 → 0.0217. Increasing regularization also reduces the step size under this rule.
Weaker versus stronger regularization with exact rewardsτP\tau_P versus τF\tau_F, common α=0.5/τF\alpha=0.5/\tau_FBoth converge; the weaker choice requires more iterations in every nonbinary test game.

At equal observation budgets, larger batches leave fewer policy updates. The measurements show the resulting change from faster early progress to lower later error:

Joint observationsQRE gap, M=64M=64QRE gap, M=256M=256
1,2800.000670.00098
2,5600.000730.00028

Appendix F.2 and Table 4 give these comparisons.

We also clarify the theoretical contributions and present additional experiments in the common response, with the aim of addressing the core weakness you identified. In response to your request for a list of contributions in bullet points, the direct response above presents the technical steps, and our response to Reviewer 92Xf compares the relevant literature to show what our work adds. We have incorporated these discussions into the revised PDF as fully as possible.

Dear Reviewer dehW,

I hope this message finds you well. As the discussion period is nearing its end, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper!

Reviewer_a65QOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when logit response dynamics converge to a unique equilibrium. It derives a convergence condition that ignores payoff changes that do not affect action choices. It also extends the analysis to related learning dynamics and illustrates the results with synthetic experiments.

Critical strengths

The paper clearly states its assumptions and convergence conditions. It also provides detailed proofs and synthetic examples to illustrate the results. This paper is outside my main area of expertise, so my confidence in assessing its theoretical correctness and novelty is limited.

Critical weaknesses

The experiments cover a limited set of synthetic examples. In particular, the convergence result for independent natural policy gradient is not directly evaluated experimentally. This paper is outside my main area of expertise, so my confidence in assessing its weakness is limited.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Response to Reviewer a65Q

We greatly appreciate Reviewer a65Q's positive feedback. Your overall assessment helps us understand how the broader ML/DL community perceives our work and has helped us improve its presentation. We hope the following response addresses your main concern:

1. Direct tests of the NPG convergence result

Direct NPG tests confirm convergence even with regularization only 5% above our sufficient bound. We run the exact update in Theorem 3 from different initial policies:

CheckObserved result
Convergence at τ=1.05LT/2\tau=1.05L_T/2Every tested initialization reaches residual below 10−810^{-8} for 20 consecutive updates.
Agreement between initializationsA common endpoint within each game, with Euclidean distances below 2.23×10−152.23\times10^{-15}.
Binary coordination below the sufficient boundDifferent initializations reach two distinct stable endpoints.

This directly tests both convergence close to the guaranteed boundary and the possibility of different limiting policies below it. See Section 4.2 and Appendix F.2.

We also present the theoretical contributions and additional experiments in the common response, with the aim of addressing the main weakness you identified. We have incorporated these discussions into the revised PDF as fully as possible and welcome your consideration of the additions. In particular, we would appreciate your view on whether the revision makes our contributions easier for readers to understand. We look forward to discussing this with you!

Dear Reviewer a65Q,

I hope this message finds you well. As the discussion period is nearing its end, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper!

Reviewer_92XfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies sufficient conditions for convergence of simple dynamics to a unique fixed point in games with a finite set of actions. These conditions consist of bounds on a measure of the projection of the payoffs (or derivatives of related maps) on the hyperplane containing the simplex. Different cases are considered.

Critical strengths

The initial question posed by the authors, that is How much regularization is sufficient to make the agents converge to a unique logit equilibrium? is interesting and the conditions provided, given the full knowledge of the games, are easy to check. The meaning of these bounds is also quite intuitive and they can be translated in regularization terms in the specific settings of the paper. It seems that the bounds are tight in the sense that surpassing one gives rise to a bifurcation, at least in very simple cases: this seems an interesting direction to explore.

Critical weaknesses

The setting considered is a classical one, and quite simple. It is difficult to assess the novelty of such results for me. The basic observation from which the paper's results originate is hardly novel: considering only tangential components of the dynamics, for instance, is how one usually assesses flow incompressibility. As regularization criteria, the bound provided by the theorems in the paper are quite easy and amenable to be used, but I am not sure that they are not equivalent to other results in the literature, maybe obtained in more complex settings and that could be translated here. There is no deep discussion about this. The authors make extensive use of a recent result on the Lipschitz constant of the softmax function, ans this seems to be the key point in proving those bounds.

As a final point, the presentation is very compressed and sometimes difficult to follow, which is the mark of an extensive AI use. As I have written in other reviews, that is not a problem per-se but, in this case, it makes the paper a bit difficult to read in its more technical parts.

Questions for authors

Can you clarify why and how the bounds that you presented improve or are related to similar results in the literature?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 92Xf

We greatly appreciate the reviewer's detailed and constructive feedback. We are pleased that you consider our approach an interesting direction worth exploring. We believe our response has addressed most of your concerns and we respectfully ask if you're comfortable to re-consider the rating in light of our efforts and dedication. Below, we provide detailed responses to all the comments:

1. What the analysis adds

We have clarified the relationship between our work and prior theory. Existing results supply softmax sensitivity and continuous-time convergence tools; our quantitative conclusions concern the projected norm, its optimal universal constant, and discrete NPG with exact or sampled rewards. The relationships are:

Existing workEstablished result used hereWhat our analysis establishes
Nair [1], Theorem 1 and Section 4.3Sharp 1/21/2 softmax Lipschitz constant; convergence of sequential double-softmax updates in zero-sum matrix games.Holding this constant fixed, projection can improve the sufficient bound arbitrarily; the NPG analysis covers simultaneous updates in general finite games.
Cianfanelli–Como [2], Theorem 2 and Equation (22)Continuous logit dynamics converge at sufficiently high noise; the derivative argument already uses action-payoff differences.An explicit Euclidean projected-norm condition, discrete response-iteration guarantees, and a matching example proving optimality of its universal constant.
Gao–Pavel [3], Theorem 1 and Remark 7Passivity proves convergence of continuous score dynamics and already accounts for feasible policy changes.Their framework combined with the sharp softmax constant recovers our corresponding continuous condition. Our discrete NPG results additionally specify step sizes, geometric rates, and sampling-error bounds.
Sun et al. [4], Theorem 3.3Convergence of the same independent NPG update under τ>2S\tau>2S, where S=∑iniS=\sum_i n_i is the total number of actions across players.For the same reward range [0,1][0,1], our sufficient condition is τ>14S2−∑ini2\tau>\frac14\sqrt{S^2-\sum_i n_i^2}, with 0<ατ≤10<\alpha\tau\le1.

For continuous scores, using the sharp softmax sensitivity in Gao–Pavel's passivity argument gives 2τ>μ2\tau>\mu, where μ\mu bounds the growth of payoff differences along policy changes. Taking μ=LT\mu=L_T recovers our condition. For discrete NPG, the ratio of Sun et al.'s sufficient lower bound to ours is 8S/S2−∑ini2>88S/\sqrt{S^2-\sum_i n_i^2}>8. Appendix E.2 also bounds the same QRE gap used in their analysis. Appendix E.4 gives the comparison of NPG and score dynamics; Appendix B.4 treats population logit dynamics.

[1] Nair, P. (2026). Softmax is 1/2-Lipschitz: A tight bound across all ℓp\ell_p norms. TMLR.

[2] Cianfanelli, L., & Como, G. (2025). On the stability of the logit dynamics in population games. IEEE Transactions on Automatic Control, 70(9), 5910-5925.

[3] Gao, B., & Pavel, L. (2021). On passivity, reinforcement learning, and higher order learning in multiagent finite games. IEEE Transactions on Automatic Control, 66(1), 121-136.

[4] Sun, Y., Liu, T., Kumar, P. R., & Shahrampour, S. (2024). Linear convergence of independent natural policy gradient in games with entropy regularization. IEEE Control Systems Letters, 8, 1217-1222.

2. Making the argument easier to follow

We have revised the presentation in many parts of the PDF to make the arguments easier to follow. AI did assist with the writing. When we asked AI to check individual derivations separately, it also suggested changes to the exposition. As authors working closely on the problem, we generally found these suggestions understandable and appropriate, and adopted some of them. However, we overlooked the experience of readers who lacked that context. We sincerely thank the reviewer for drawing our attention to this issue. We are now reviewing the paper for a more coherent progression of ideas, for example by explaining changes in models and mathematical objects before using them. We have also reviewed the narrative throughout and revised wording that could cause confusion. Time constraints have prevented us from polishing the presentation as carefully as we would like. We commit to continuing this review to make the paper more accessible to readers across the community.

We also clarify the theoretical contributions and present additional experiments in the common response. We hope our responses resolve your concerns. If there are further concerns or comments that we have overlooked, we would greatly appreciate your feedback to help us improve the work. We are happy to see from your comments that you have an overall positive view of our work, and we respectfully ask if you'd be comfortable to adjust your ratings, based on all the new experiments and justification to your questions. Thank you once again for your time and effort. We look forward to incorporating these insightful discussions into the final version.

Dear Reviewer 92Xf,

We would like to follow up on our responses to your comments. We would appreciate your feedback on whether these responses address your concerns and would be happy to clarify any remaining questions. Thank you for your time.

Reviewer_92XfOpenReview ↗

Reply to authors

I thank the authors for the honest and thorough response. I am sorry for answering late, these are busy days for me! You clarified many of my doubts but some of my concerns still remain. I will carefully think about it and discuss with the other reviewers, if needed, but for now I won't change my score.

Thank you very much for your kind reply! We understand that some fundamental contributions and novelty judgments can be difficult to identify solely through discussion with the authors. However, if there are any problems, or if you would like to discuss them with us or need us to clarify anything, please just let us know, and we will respond as soon as possible. Thank you again for your valuable time and effort!

Reviewer_k5jvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the convergence and stability of entropy-regularized learning in multi-agent systems. It proposes a tighter convergence condition by focusing on payoff differences that affect agents' decisions, rather than overall payoff changes. The authors establish theoretical convergence guarantees, extend the analysis to several learning settings, and provide synthetic experiments to support their findings.

Critical strengths
  1. Clear motivation and theoretical contributions.

    The paper addresses a meaningful stability problem in multi-agent learning. The proposed condition has an intuitive interpretation and provides stronger convergence guarantees than conventional bounds.

  2. Well-organized analysis.

    The paper provides convergence guarantees, discusses the tightness of its condition, and extends the results to multiple settings. The presentation is generally clear and easy to follow.

Critical weaknesses
  1. Limited distinction from existing work.

    The analysis builds on an existing tighter bound for the softmax function. Although the proposed projection is useful, the paper does not sufficiently isolate its independent contribution from improvements already established in prior work.

  2. Limited experimental validation.

    Most experiments use simple synthetic settings designed to illustrate the theoretical results. The paper lacks direct evaluation of independent natural policy gradient (NPG), which is one of its main learning-related applications.

Questions for authors
  1. Could the authors clarify how much of the improvement comes specifically from the proposed projection, rather than the previously established tighter softmax bound?

  2. Could the authors provide experiments with independent NPG to demonstrate whether the improved convergence condition offers practical benefits, particularly in terms of stability and learning performance?

2 comments
AuthorsOpenReview ↗

Response to Reviewer k5jv

We greatly appreciate Reviewer k5jv's positive and insightful feedback. Your overall assessment means a great deal to us as researchers and encourages us to continue refining and advancing this line of research. We hope the following responses address your main concerns:

1. The gain from projection

Projection can improve the sufficient regularization bound by an arbitrarily large factor for an unchanged response map. Both conditions use the established softmax constant 1/21/2:

ConditionSufficient regularization, τ=1/β\tau=1/\beta
Full payoff matrixτ>∥W∥2/2\tau>\lVert W\rVert_2/2
Projected payoff matrixτ>∥Wtan∥2/2\tau>\lVert W_{\mathrm{tan}}\rVert_2/2

The additional improvement from projection is therefore the factor ∥W∥2/∥Wtan∥2\lVert W\rVert_2/\lVert W_{\mathrm{tan}}\rVert_2. Adding the same payoff to every action, with that payoff depending on the other agents' policies, can make the full-matrix norm arbitrarily large while leaving every choice probability and the projected norm unchanged.

Figure 1 provides a concrete example: the full-matrix bound on response sensitivity increases from 0.8 to 10.42, while the projected bound remains 0.8 and the actual updates remain exactly the same. Thus, even with the optimal softmax constant already in use, projection independently extends the convergence guarantee to systems that the full-matrix condition cannot certify. Appendix C.1 gives the construction and verification.

2. Direct NPG experiments and practical benefits

The projected choice preserves convergence near the sufficient bound and improves mean Nash gap by 4.0% at the same step size and observation budget. The direct comparison is:

MetricFull-matrix choiceProjected choice
Mean Nash gap ↓0.06910.0663
Mean reward ↑0.43720.4380

The paired gap difference has 95% interval [-0.0043,-0.0012], and the reward difference has interval [0.00017,0.00154]. C2 of common response also includes validation-tuned NPG to show how guaranteed choices compare with parameter search. See Table 2 and Appendix F.3.

We also clarify the theoretical contributions and present additional experiments in the common response. We hope our responses resolve your concerns. If there are further concerns or comments that we have overlooked, we would greatly appreciate your feedback to help us improve the work. We have incorporated these discussions into the revised PDF as fully as possible and welcome your consideration of the additions.

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper!

Comments on the paper

5 comments
AuthorsOpenReview ↗

Common response ( Part 1 / 2 )

We sincerely thank all reviewers for their thoughtful and constructive feedback. We are encouraged by their recognition of the key contributions and strengths of our work. We also thank the Area Chair for organizing the review and discussion of our paper. We appreciate everyone's time and effort in evaluating this work.

We have carefully addressed each comment from the reviewers and believe we have resolved most of their concerns. The revised manuscript incorporates the suggested experiments, additional discussions, and related updates to strengthen the work. New and revised material is highlighted in yellow to make the changes easy to locate. Below, we describe the core contributions, experimental updates, and accompanying discussions in the revision, which address the concerns shared by most reviewers.

C1. Theoretical contribution

For a broad class of multi-agent feedback systems, we establish more precise, explicitly computable stability conditions, prove that they have an optimal universal constant, and demonstrate how they improve convergence guarantees for different learning dynamics.

To illustrate the systems we study, consider a group of AI agents, each adjusting its probabilities of choosing different actions in response to the behavior of the others. As these interactions continue, the agents may keep changing their policies without settling into stable decision-making behavior. Naturally, we would like their choices to stabilize. Entropy regularization is often introduced to ensure convergence in multi-agent learning. However, stronger entropy regularization makes agents choose actions more randomly and can weaken their preference for higher rewards. Previous research has established various convergence guarantees by analyzing payoff sensitivity and interactions between agents. We further investigate how precise these conditions can be and how to measure the interactions that determine stability.

Consider an agent with two actions: action A earns 1 point, and action B earns 3 points. The agent therefore prefers action B. Now suppose the other agents change their behavior, increasing both rewards by 100 points to 101 and 103. Under the logit response, the agent's choice probabilities remain unchanged because the payoff difference is still 2. Yet existing convergence conditions based on the full payoff matrix count this shared 100-point change. The analysis can consequently regard the interaction as very strong and require particularly strong regularization to guarantee stability. Our analysis removes shared payoff changes that do not affect choice and measures how policy changes alter agents' relative preferences between actions. Building on existing softmax sensitivity results and convergence theory, we establish the following conclusions:

  • A unified, computable convergence condition. Projected payoff sensitivity gives an explicit condition ensuring that logit responses converge from every initial state to a unique equilibrium. Here, an equilibrium means that the entire policy profile remains unchanged when each agent responds to the others' current policies. For exactly the same response system, we prove that removing irrelevant payoff changes can lower the sufficient regularization threshold by an arbitrarily large factor relative to the full-matrix condition.
  • An optimal universal constant. We construct a two-action system showing that the constant cannot be relaxed in a sufficient condition based only on this projected norm and valid for all corresponding systems: any relaxation would include instances with multiple equilibria. A two-player coordination game also proves optimality of the constant in the corresponding condition for independent natural policy gradient (NPG).
  • Further guarantees for learning dynamics. The analysis applies to multiple populations, nonlinear payoffs, and different forms of learning dynamics. We illustrate its application through a representative example. For simultaneous NPG, we obtain a lower sufficient regularization threshold, explicit step-size ranges and convergence rates, and learning-error bounds for sampled payoff estimates. For the same NPG update and reward range, even our action-count-based sufficient lower bound is more than eight times smaller than the existing bound.

These results show that accurately measuring the interactions that affect agents' decisions can substantially improve existing stability guarantees. We further identify the best constant in this universal criterion and demonstrate how it provides directly usable convergence analysis for different learning and decision-making systems.

AuthorsOpenReview ↗

Common response ( Part 2 / 2 )

C2. Direct learning experiments and practical parameter choices

We added experiments to address the reviewers' shared concerns. They directly test the learning theorem and show how the sharper condition changes parameter selection. The following results use independent NPG in coordination, constant-sum, and general-sum games. Here LTL_T measures how policy changes alter relative action rewards, and MM is the batch size.

QuestionResultEvidence
Does NPG converge close to the new sufficient bound?With regularization only 5% above the bound, all tested initializations converge to a common endpoint within each game.Section 4.2; Appendix F.2
Does the sampling theory predict the observed error?At τ=LT\tau=L_T, policy-error slopes against batch size range from −1.007-1.007 to −0.963-0.963, closely matching the predicted 1/M1/M scaling.Appendix F.2
Can the coefficient be chosen from observed outcomes?With exploration observations, the projected-norm estimate has 4.24% mean absolute relative error; every selected coefficient satisfies the true sufficient condition in our tests.Appendix F.4

To assess reward performance, we compare parameter choices based on the projected and full-matrix conditions. The Nash gap measures the gain available from changing one player's policy alone; a smaller value means less incentive to deviate.

Regularization choiceMean Nash gap ↓Mean reward ↑
Full-matrix condition0.06910.4372
Projected condition0.06630.4380
Validation-tuned NPG0.01860.6956

Table 2 and Figure 3 report final performance and learning progress; Appendix F.3 gives the protocol.

Dear Reviewers,

During the rebuttal period, we have conducted additional experiments and refined parts of the theoretical analysis and presentation of the paper.

As the discussion period is coming to an end, we would greatly appreciate hearing whether these updates have addressed your concerns and improved your overall assessment of the paper.

Thank you all for your time, effort, and thoughtful feedback!

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.