- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
I do not have sufficient expertise to assess this submission reliably and have alerted the AC. None of my current scores should be taken into account when deciding whether to accept this paper.
Critical strengths
I do not have sufficient expertise to assess this submission reliably and have alerted the AC. None of my current scores should be taken into account when deciding whether to accept this paper.
Critical weaknesses
I do not have sufficient expertise to assess this submission reliably and have alerted the AC. None of my current scores should be taken into account when deciding whether to accept this paper.
Questions for authors
I do not have sufficient expertise to assess this submission reliably and have alerted the AC. None of my current scores should be taken into account when deciding whether to accept this paper.
Response to Reviewer suPZ (1/2)
We sincerely thank the reviewer for the careful reading, and for recognizing the clear empirical benefit of DG-PG and our validation of Assumptions 4.2 and 4.3. Beyond cloud scheduling and power dispatch, we have also applied DG-PG to voltage control in distribution networks and fleet rebalancing in robotaxi operations, where references built from the analytical models that operators already use likewise make DG-PG effective.
The reviewer's main concern is the derivation of Equation 10. Below we give the full derivation from the policy-gradient formulation and Equation 9, and state the condition under which it holds. We also answer the question on Figure 2.
We thank the reviewer for this careful reading. Equation 10 holds for the guidance direction that DG-PG uses, and we give the full derivation below.
Equations 9 and 10 read
\qquad \nabla_{\theta^i}\mathcal{G} = \mathbb{E}_{s_t \sim \nu^{\boldsymbol{\pi}},\\, \boldsymbol{a}_t \sim \boldsymbol{\pi}}\\!\left[\langle \boldsymbol{x}_{t+1}-\tilde{\boldsymbol{x}}_{t+1}, \boldsymbol{z}_t^i\rangle\\,\nabla_{\theta^i}\log\pi^i(a_t^i\mid o_t^i)\right].$$ The coefficient $\langle \boldsymbol{x}_{t+1}-\tilde{\boldsymbol{x}}_{t+1}, \boldsymbol{z}_t^i\rangle$ differs from $d$ only by terms that do not depend on agent $i$'s action. Fix $s_t$ and the other agents' actions, and let $\boldsymbol{x}_{t+1}^{-i} \triangleq \boldsymbol{x}_{t+1}-\boldsymbol{z}_t^i$ be the next system state without agent $i$'s contribution. Then $$\begin{aligned} d(\boldsymbol{x}_{t+1},\tilde{\boldsymbol{x}}_{t+1}) - d(\boldsymbol{x}_{t+1}^{-i},\tilde{\boldsymbol{x}}_{t+1}) &= \frac{1}{2}\\|\boldsymbol{x}_{t+1}-\tilde{\boldsymbol{x}}_{t+1}\\|^2 - \frac{1}{2}\\|\boldsymbol{x}_{t+1}-\boldsymbol{z}_t^i-\tilde{\boldsymbol{x}}_{t+1}\\|^2 \\\\ &= \langle \boldsymbol{x}_{t+1}-\tilde{\boldsymbol{x}}_{t+1}, \boldsymbol{z}_t^i\rangle - \frac{1}{2}\\|\boldsymbol{z}_t^i\\|^2 . \end{aligned}$$ Since agents sample their actions independently given $s_t$, the term $d(\boldsymbol{x}_{t+1}^{-i},\tilde{\boldsymbol{x}}_{t+1})$ does not depend on $a_t^i$, and neither does $\\|\boldsymbol{z}_t^i\\|^2$ when agent $i$'s contribution has the same magnitude under every action. The score-function identity then gives $$\begin{aligned} \nabla_{\theta^i}\\,\mathbb{E}\\!\left[d(\boldsymbol{x}_{t+1},\tilde{\boldsymbol{x}}_{t+1}) \mid s_t\right] &= \mathbb{E}\\!\left[d(\boldsymbol{x}_{t+1},\tilde{\boldsymbol{x}}_{t+1})\\,\nabla_{\theta^i}\log\pi^i(a_t^i\mid o_t^i) \mid s_t\right] \\\\ &= \mathbb{E}\\!\left[\Big(d(\boldsymbol{x}_{t+1}^{-i},\tilde{\boldsymbol{x}}_{t+1}) + \langle \boldsymbol{x}_{t+1}-\tilde{\boldsymbol{x}}_{t+1}, \boldsymbol{z}_t^i\rangle - \tfrac{1}{2}\\|\boldsymbol{z}_t^i\\|^2\Big)\nabla_{\theta^i}\log\pi^i(a_t^i\mid o_t^i) \mid s_t\right] \\\\ &= \mathbb{E}\\!\left[\langle \boldsymbol{x}_{t+1}-\tilde{\boldsymbol{x}}_{t+1}, \boldsymbol{z}_t^i\rangle\\,\nabla_{\theta^i}\log\pi^i(a_t^i\mid o_t^i) \mid s_t\right], \end{aligned}$$ where the last equality holds since $d(\boldsymbol{x}_{t+1}^{-i},\tilde{\boldsymbol{x}}_{t+1})$ and $\tfrac{1}{2}\\|\boldsymbol{z}_t^i\\|^2$ do not depend on $a_t^i$, and thus have zero expectation when multiplied by $\nabla_{\theta^i}\log\pi^i(a_t^i\mid o_t^i)$. By definition, $\mathcal{G} = \mathbb{E}_{s_t \sim \nu^{\boldsymbol{\pi}}}\big[\mathbb{E}[d(\boldsymbol{x}_{t+1},\tilde{\boldsymbol{x}}_{t+1}) \mid s_t]\big]$. With $\nu^{\boldsymbol{\pi}}$ held fixed, its gradient is the average of the per-step gradient above, $$\nabla_{\theta^i}\mathcal{G} = \mathbb{E}_{s_t \sim \nu^{\boldsymbol{\pi}}}\big[\nabla_{\theta^i}\mathbb{E}[d(\boldsymbol{x}_{t+1},\tilde{\boldsymbol{x}}_{t+1}) \mid s_t]\big],$$ which is Equation 10, the direction for which Theorems 5.1–5.3 are stated. In the original submission, Equation 10 and the DG-PG update already compute this per-step direction. To make this explicit, we have added the derivation and its conditions to Section 4 and the appendix.