- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper discusses learning a low-dimensional population representation rather than directly modeling the full state-action distribution. Overall, the paper is clearly written and makes a certain theoretical contribution.
Critical strengths
This paper discusses learning a low-dimensional population representation rather than directly modeling the full state-action distribution. Overall, the paper is clearly written and makes a certain theoretical contribution.
Critical weaknesses
-
Despite its theoretical contribution, I doubt the practical means really work far better than alternative representation learning methods. Meanwhile, the environment seems to require only that the learner estimate rewards and cannot fully support the paper's statements.
-
The authors are suggested to clarify their theoretical contribution. The current methodology and theoretical proof follows largely what has been done in the era of offline RL. Is there any relaxed assumption or different insight?
Questions for authors
Clarify the contributions from both theoretical uniqueness and methodological comparisons.
3 comments
Thanks for your comment. I will maintain the score due to the remaining concern on novelty.
Details
Can the reviewer provide more details about their concerns? We will try to alleviate them.
Rebuttal
Scope of contributions
The theoretical contributions of this paper are providing a language to study low-dimensional representation for large population multi-agent systems and a statistical guarantee for offline equilibrium learning with an unknown low-dimensional representation. We also perform experiments to provide evidence for the claim that learning the population representation can improve both prediction and the equilibrium quality of the resulting policies. This is shown using a simple experiment for two reasons: first, a one-step experiment cleanly isolates the representation learning mechanism and is easy to understand, and second, there is no computationally efficient algorithm in the paper for complex multi-step settings.
Experiment baselines
Section 5 already compares the learned representation with a single-agent model, a monolithic raw-mean model, and a full-population-law model, under approximately matched parameter counts. We also include two oracle baselines. Figure 3 reports both prediction error and the exact true Nash gaps of the resulting policies.
The experiments support the claim that learning the population representation can improve prediction and equilibrium quality in the tested setting. The guarantees for multi-step problems with unknown transitions are theoretical results under the stated assumptions. We don’t claim to provide either a computationally efficient algorithm for this problem or empirical implementation of the main algorithm in the paper. This is very clearly mentioned in the paper; see, for example, the beginning of section 5 where we discuss experiments or the conclusion.
Novelty
It is common in theory to build on top of other works. That, by itself, cannot be a weakness of a paper. We are very clear that we build on Zhang et al.’s technique of minimizing an estimated equilibrium gap based on optimistic deviation values and pessimistic policy values. However, this isn’t the extent of our theoretical analysis. Our setting introduces two additional statistical issues relative to Zhang et al.: first, the population is observed through finite snapshots, and the empirical representation computed from each snapshot depends on the candidate feature map; second, planning under a candidate model uses that model’s own induced population flow. Also, unlike Zhang et al., we don’t impose Bellman-completeness conditions on a value function class.