- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates when interactive agents achieve quilibruim when updating their choices with softmax. The core discovery is that, the stability depends on how the changes in choices affect relative payoffs. This is because, common increase/decrease payoffs will not change agent choices. The paper uses a projection matrix to retrieve effective interactions. They prove that when , same equilibrium will be achieved regardless of starting point. They further give an example to show that the constant 2 can't be loosen under some conditions.
Critical strengths
- The way they develop to remove commmon payoffs across agents with a projection operator and catch the real element controling equilibrium is clean and elegant.
- The two-action example is useful. It proves that the constant 2 connot be imporved in a universal condition based only on this projected norm. This is a clean and clear boundary for the main theorem.
Critical weaknesses
- I hesitate if the ammount of theoretical novelty is enough. The core proof combines existing softmax sensitiviy bounds, projections, standard contraction mapping theory. Their primary contribution concentrates on refining the convergence criterion.
- Limited experiments. They only use synthetic systems, while natural policy gradient guranttes assume exact payoff evaluation. Therefore, i don't know how much benefit the results can provide in realistic noisy multi-agent learning settings.
Questions for authors
- Clarify (best list in bullet point) the theoretical contributions, like specifically what key steps/derivations are core contributions of the work.
- Try to test natural policy gradient with sampled payoff estimates. Specificailly, compare convergence speed and final error (across like batch sizes, regularization strenghs) to see if the developed theory still work when payoffs are noisy.
2 comments
Dear Reviewer dehW,
I hope this message finds you well. As the discussion period is nearing its end, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.
Thank you for your time and effort in reviewing our paper!
Response to Reviewer dehW
We sincerely appreciate your insightful and professional comments. We are delighted by your recognition of our core idea. To fully address your concerns, we provide a more detailed discussion and experiments below.
1. The core contributions
The key derivations are:
Two further results identify useful structure: the Hadamard construction keeps the Euclidean contraction bound at 1/2 while the specified block bound grows as m/2 (Appendix B.3); symmetry gives an additional uniqueness condition through the entropy-regularized potential (Section 3.3).
2. Learning from sampled rewards
The new analysis gives convergence and error guarantees when exact rewards are replaced by unbiased estimates with bounded conditional variance. Initial error decays geometrically; with a constant step size, the remaining mean-square error bound decreases as 1/M; diminishing steps give convergence. These results include policy error and QRE gap (Appendix E.2).
For the tool-selection example, an agent may observe only the reward of the tool it uses. Appendix E.3 gives an estimator for this setting when joint actions and sampling probabilities are observed. The experiments also estimate the regularization coefficient from these observations:
Exploration counts toward the same total learning budget. The Nash-gap interval includes zero; the reward interval supports an improvement. See Appendices F.2 and F.4.
3. Batch size, regularization, speed, and final error
The added comparisons identify how batch size and regularization affect both remaining error and learning speed. QRE gap measures the gain from a unilateral policy change in the regularized game. Let τP=1.05LT/2 and τF=1.05∥W∥2/2.
At equal observation budgets, larger batches leave fewer policy updates. The measurements show the resulting change from faster early progress to lower later error:
Appendix F.2 and Table 4 give these comparisons.
We also clarify the theoretical contributions and present additional experiments in the common response, with the aim of addressing the core weakness you identified. In response to your request for a list of contributions in bullet points, the direct response above presents the technical steps, and our response to Reviewer 92Xf compares the relevant literature to show what our work adds. We have incorporated these discussions into the revised PDF as fully as possible.