- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies scenarios of agents that compete in pairwise competition (e.g. think of participants in an on-line chess site; you want to work out their strength at chess based on pairwise game results). We're in a setting of win-lose-draw competition. The paper studies Nash averaging, a rule for computing ratings of players that is intended to be resistant to the operation of duplicating a player (which ought not to affect the relative rating of the other players). Since Nash averaging can in fact affect these relative ratings, the paper studies the effects of such duplication, and conditions under which it doesn't change things.
Critical strengths
Highlights a problem with a technique that's been well-studied and used in practice. Studies the impact of the problem on real world data.
On para 2, it looks plausible that copying an agent looks like it shouldn't change any rating, since it's like duplicating a strategy in a game, which ought not to change the game's value. It's pointed-out that the property fails. Paragraph 3 gives a nice example of the property failing; the choice of "max entropy" when there are multiple solutions, is bad.
Critical weaknesses
The abstract is hard to read, I did not get much out of it. Para 1 of the paper is much better.
Some unclear/vague wording "Agent 4's own results are untouched". line 046 "even distribution", I'd prefer "uniform distribution". But I get the idea.
Questions for authors
Question: Given that the max-entropy rule has been previously used/studied, the paper considers the question of conditions under which it works. To me there is raised the bigger question of what, if any, alternatives to the max-entropy rule we ought to be using (which is touched-on in para in middle of p.3).
Suggestions: On p.2 it might be helpful to make the discussion: We have a symmetric 0-sum game whose value is zero, hence every agent (ie pure strategy) that gets played, achieved payoff zero, and any agent that doesn't get played (and is inessential) would get a negative payoff if they were played.
Line 080 "no step inside the equilibrium set gains more from this tilt", usage of "step" and "tilt" is unclear to me. They should be explained better, especially "tilt". Line 087 "As the copy becomes exact" -- I'm guessing I'm supposed to be thinking of a process whereby an inexact copy converges towards an exact one, is that right?
p.2, top, you should say what p and q denote, although it's not hard to figure out.