- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes an actor-critic algorithm for networked MDPs. In this setting, the transition and reward dynamics factorise over a graph, and each agent only uses information from its -hop neighbourhood. The main contribution is the critic, which uses a truncated approximation of the state-action values. The authors propose a TD-style algorithm to learn these truncated values and bound the approximation errors this introduces. The method is evaluated on a ring MDP and a wireless network environment.
Critical strengths
-
The introduction is well written. It places the work clearly within the existing literature and states its scope and contributions precisely.
-
The theoretical analysis is thorough. It quantifies the main sources of error introduced by the proposed value approximation.
Critical weaknesses
-
I found Section 3.3 hard to follow. Algorithm 1 depends on a second, fairly long algorithm that is only given in the Appendix. A clearer description of the auxiliary algorithm in the main text would help. I also could not find where the parameters are defined in this section.
-
The numerical results section could be strengthened. The experimental setup is not described in enough detail to clarify the comparisons and metrics used, while the discussion mostly restates what the figures already show. It would be more helpful to offer an interpretation of why the methods behave as they do. The comparison includes only one external baseline (Zhang), and this baseline is not described in the main paper from what I could find. For the wireless network environment, I could not find the performance of the RL algorithm itself in the main paper. Only the critic input dimensions are reported, and these reflect the problem's topology rather than its dynamics. I understand that the paper's contributions are mainly theoretical, and I do not think the experiments need to be extensive. Still, a more complete numerical section would make the paper considerably stronger.
Questions for authors
-
How is the Zhang baseline defined? From the figure, its performance looks comparable to that of the proposed method, appearing to trade variance for bias.
-
Why are the local configuration encodings zero-padded to a common length? Each agent has its own weights for its local critic so it seems the encodings could have different lengths across agents.
-
Since -hop configurations are not-Markovian, shouldn't each agent's policy depend on the history of its local configurations up to the current step, rather than only on the current configuration ? This is a common approach in partially observable MDPs, and it would be helpful to know whether the authors considered it.
-
As the paper shows, networked MDP methods are most useful in large environments. Such environments are often homogeneous, with all agents sharing the same state and action spaces and the same transition dynamics. The environments in Section 5 appear to belong to this category. Could the method proposed be further improved by exploit this structure, for example by sharing parameters across agents?
Author-Reviewer Discussion Period is Ending
Dear Reviewers,
Thank you for your service in the review process. Now that we are approaching the deadline of the author-reviewer discussion period. Please take time to read authors' responses and let them know if it addresses your concerns. Thank you.
Best,
Area Chair