- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies multiagent planning in which communication capability changes as agents move; the problem is modeled using the existing SDec-POMDP framework. The authors first propose SDecMCTS, an online MCTS over the shared belief: while agents remain in contact, it searches over joint actions; each time contact is lost, the decentralized window is treated as an option and handed to a Dec-POMDP solver, and search then continues from the belief at which communication resumes. Building on this, the authors propose the AlphaZero-style SDecZero, which guides search with a policy head and a value head and additionally trains a communication head. The experiments show that SDecMCTS roughly matches the quality of the offline planner approximate RS-SDA* on 17 planning benchmarks and takes less time on the 15-node DARPA caves. On three 25-node real-world caves, SDecZero is far more sample-efficient than MAPPO, QMIX, MAZero, and other methods.
Critical strengths
The problem is real and understudied, and the idea of folding out-of-contact windows into the same search process as options is simple and highly reusable.
Critical weaknesses
-
SDecZero itself uses the true model and performs 1,000 simulations per decision, and its return already reaches about 40–50 before training begins, yet the evaluation-time head ablation lacks the configuration that uses no learned head at all. Therefore, how much improvement the networks actually bring, and the significant scaling the authors claim, are not verified.
-
SDecMCTS is only run on problems with at most 15 nodes, and SDecZero is only run on the 25-node caves; the two are never compared on the same problem.
Questions for authors
-
Can results with more than two agents be reported?
-
Can pseudocode be provided, and why is assigning zero value at window truncation conservative?
-
Can the learned communication head be evaluated in an environment where communication is actually stochastic?
5 comments
Thank you for the detailed response and the additional experiments. The most useful part for me is the table that runs SDecMCTS and SDecZero on the same three 25-node maps with online times; the n = 3 results also help. The PDF I can download from the forum is still identical to the submitted version, so I go by the text of the response. Four points remain open for me.
-
W1 and W2 (what the networks add). Your table gives the number I asked for: search without learned heads returns 58.45 / 44.12 / 48.35, against 73.7 / 62.0 / 61.8 with them. Two things should be stated with the table. To me you write "the same hyperparameter configuration as SDecZero", while to rf7P the same table is "across hyperparameter configurations". Is it one configuration or several? And is ± over the 3 seeds a standard error or a standard deviation? On Chamber-25 (44.12 ± 2.85 against 62.0 ± 7.5), how firm the gap is depends on which. Your table also shows SDecMCTS completing 25-node episodes in 56.6 to 97.3 s, so I would narrow "enable significant scaling" in the contributions list to the gain in return and the lower online time.
-
Q1 (more than two agents). The n = 3 table answers this for SDecMCTS on graphs of up to 15 nodes. The claim "faster than any other considered baseline" holds on three of the five rows. PF-MAA* is faster on Tunnels-10 at horizon 6 (21 against 23.6) and horizon 10 (36 against 53.9). The response reports no n = 3 result for SDecZero. Unless the revision adds one, the paper should say that SDecZero is evaluated for two agents only.
-
Q2 (value at truncation). Please state which rule produced the SDecMCTS numbers in Table 1 of the submission and in the two tables in your reply to me, and reconcile two descriptions: to me you write that the probability mass reaching the depth cap receives the centralized upper bound, and to 2AHM that a capped window "chains into another optimistically priced by" V_dec.
-
Q3 (stochastic communication). The modified environment is the right test, but the response states no numbers. Table 4 of the submission lists the head as used in planning as "communication predicted where the head exceeds 0.5". A thresholded prediction is deterministic, so if the same threshold is used here, I do not see how the head can be used unchanged when contact is random. Please state whether search samples contact from the predicted probability, and give the error of the predicted contact probability against the true one and the return gap to planning with the true F.
Thank you again. I hope these points are useful for the revision.
Response to XPLG's remaining questions (hyperparameter clarification, scoping contributions, clarifying estimator, updated manuscript)
We appreciate the follow-up questions. The feedback has substantially helped improve the paper!
You will find that the manuscript is now updated on Open Review. This remains an interim product, as we continue to facilitate additional experiments and ablations in support of reviewer 2AHM. But this should at least enable review of the updated Figure 6, Figure 7, and pseudocode in the technical appendix.
1. One configuration. The SDecMCTS hyperparameter configuration in Figure 6 matches that of SDecZero (simulation count, window depth, visits before decentralized solve, and decentralized solver recursion) for a fair comparison, inasmuch as these two can be compared directly. Tables 5, 6, and 7 outline these hyperparameters clearly. The "across hyperparameter configurations" in response to reviewer rf7P is confusing and has been dropped. The match is clean. Standard error. All tables and figures report SE other than the in-simulation drone experiments in Table 3. We agree and modify the contribution claim accordingly.
2. Concur. We clarify, however, that the contribution is to be scoped against the semi-decentralized baseline. SDecMCTS is faster than RS-SDA* on all semi-decentralized rows and substantially outperforms it on one. For additional context: PF-MAA* and o-SARSA$^{seq}$ are solving the fully decentralized version of the problem to provide a sort of lower bound (note the dec. in the column header). RS-SDA* and SDecMCTS are solving the semi-decentralized version of the problem (note the semi-dec. in the column header). That is correct; we have thus far run the n = 3 experiments for only SDecMCTS. although there is no technical or structural blocker to doing so for SDecZero as well.
3. The two descriptions are the same rule; there is no discrepancy. names the estimator for a belief that is still out of contact. It does not mean a fully decentralized (pessimistic) value. In the window solve (Algorithm 1, line 41), probability mass that reaches the depth cap is priced by at its belief, and search then continues in a further window (line 39). SDecMCTS fills there with the centralized upper bound, which is optimistic, and SDecZero with a learned semi-decentralized value.
Which rule produced which numbers: Table 1 of the original submission priced that mass at zero, as Section 4 of that version stated. Table 1 of the revised manuscript and both tables in our reply, use the centralized bound: exact on the tabular, Search & Return and 15-node rows, on Search & Rescue, and a 3-step lookahead on the 25-node maps (reference Table 5).
4. Please reference the updated Figure 7 now in the uploaded manuscript. Yes, search samples contact from predicted probability in the stochastic communication experiment. The head remains a Bernoulli model and outputs a probability. Its architecture and inputs are unchanged, and it is fit by the plain likelihood to the contact outcomes observed in execution. The 0.5 threshold in the settings table is how planning reads the head on the benchmarks where contact is deterministic, so that is 0 or 1. Where contact is random, the planning model takes to be the predicted probability. Each simulation samples the contact outcome from it, beliefs are conditioned on that outcome, and the window solver weights communication by it. The reviewer's reasoning about thresholding is correct, and we measure it below.
Setting: ExtCross9 Search & Return (), with contact probability on the 29 line-of-sight node pairs and 0 on the other 16; 12 rounds of 40 episodes, 1,000 simulations per decision.
Error of the predicted contact probability. Mean squared error against the true probability over all 45 node pairs: 0.317 before training, 0.016 after 160 episodes, 0.010 at the end (mean absolute error 0.05). By hop distance, predicted against true: 1.00 / 1.00, 0.61 / 0.70, 0.46 / 0.49, 0.34 / 0.34, 0.39 / 0.24, and below 0.001 against 0 on pairs out of line of sight.
Return, mean ± SE over 480 episodes.
| Planning model | Return |
|---|---|
| True | 80.6 ± 2.0 |
| Learned , predicted probability | 83.0 ± 1.8 |
| Learned , thresholded at 0.5 | 68.1 ± 2.3 |
The learned row includes the episodes played while the head was still being fit. Unsurprisingly, thresholding the same head loses 12.5 against the true .
Response to reviewer XPLG: matched SDecMCTS on 25-node labyrinths, three agent experiments, stochastic communication learning
We appreciate reviewer XPLG for their review of our paper and for highlighting the underlying problem’s importance and utility of the options-like approach. We address the identified weaknesses and questions:
Weaknesses
1. Evaluation-time head ablation. Agreed. We run the suggested ablation without learned heads, on the 25-node benchmarks, below in response to point two. These results have been added to Figure 6.
2. SDecMCTS and SDecZero direct comparison. We now do so; we run SDecMCTS on all 25-node benchmarks with the same hyperparameter configuration as SDecZero for matched comparison across components and include online computation times.
** updated to include learned component breakdown (learned priors vs values)
We further discussed why SDecMCTS struggles with these larger benchmarks, in our response to reviewer rf7p.
Questions
SDecMCTS remains tractable - and faster than any other considered baseline - for n = 3 in problems with large state and observation spaces, while also returning a near optimal semi-decentralized result.
Other assumptions about communication dynamics and agent replanning naturally lead to different results, but we leave extensive discussion of these assumptions to future work.
Agreed. We now include algorithm pseudocode for SDeMCTS and SDecZero in the technical appendix. The original intent of zero value decentralized window truncation was to advantage policies that result in returned connectivity when beyond the depth cap. However, we later deemed it more principled and performant to assign it the centralized upper bound value in SDecMCTS and the learned semi-decentralized value in SDecZero. The paper text has been modified accordingly.
It is a good suggestion, and doing so captures the representative power of the underlying SDec-POMDP. We evaluate and now depict in the updated Figure 7 a modified version of S&R ExtCross9 in which connectivity degrades probabilistically as range between agents increases. No changes are required to the existing learned communication head, which is a Bernoulli model trained by observed contact outcomes. The learned communication head easily captures probabilistic communication in this environment, and the solver quickly converges on the optimal policy.