- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a Planner-Aware Consequence Feedback method, which allows a multi-agent dispatcher in dynamic indoor rescue scenarios to incorporate the set of candidate routes returned by the planner into target selection before making the final decision. Meanwhile, the paper also introduces a new benchmark for the corresponding scenario. Theoretical analysis and experimental results are provided to validate the performance advantages of the proposed method.
Critical strengths
- The motivation of this paper is clear. The method first obtains candidate-route information for different targets, compares the targets using the route information, and then makes the final target decision. This addresses a practical and intuitive limitation of conventional methods.
- This work also constructs a benchmark and dataset for the corresponding scenario, which effectively and accurately match the problem setting studied in the paper and enable evaluation in this specific scenario.
- The modular design of the proposed method allows it to be flexibly integrated with other methods.
Critical weaknesses
- The authors make several assumptions that may not be sufficiently realistic, such as assuming the exact locations of the targets and access to the global hazard state. These assumptions limit the applicability of the proposed method to some extent.
- From the experimental results, the performance improvement brought by the proposed method is relatively limited. In some cases, especially at the Medium level, there is even a considerable performance gap compared with conventional methods.
- In the comparison experiments, the authors mainly compare against conventional methods and the two base methods, MAPDP and MAPT. Comparisons with other dynamic multi-agent rescue methods are missing, which makes it difficult for reviewers to accurately assess the effectiveness of the proposed method. Minor Comments
- The proposed method seems to depend on several specific hyperparameters, such as the number of candidate routes and the setting in the Per-route Scoring block where is mapped to a bounded score with . If these values are fixed and not tunable, the authors should provide corresponding explanations. Otherwise, the authors should explain why these values are chosen and provide parameter sensitivity experiments.
- The current manuscript lacks a complete and detailed ablation study. PACF is not an indivisible module. Therefore, adding an ablation study section to analyze the role of different components in more detail would help readers better understand the effectiveness of each component in the proposed method.
- The main motivation of PACF is to utilize several candidate routes. Intuitively, different numbers of candidate routes should lead to different performance gains. Therefore, the authors should report and analyze the performance and computational cost under different numbers of candidate routes.
Questions for authors
- In Table 1, adding PACF to MAPDP leads to a performance decrease at the Medium level, while the improvement at the Hard level is also very limited. Moreover, at the Medium level, there is still a performance gap compared with conventional methods such as Nearest and Hu. I appreciate that the authors report and discuss the experimental results honestly and accurately, but the current results are still somewhat confusing. Could the authors explain this phenomenon? Is it related to the benchmark setting?
- The authors assume that the exact locations of the targets are known. Have the authors considered a more realistic setting where only approximate target locations are known, or where the targets can move within a small region around their known locations? If PACF remains effective under such a more realistic setting, it would help improve the applicability of the proposed method.
3 comments
Retrained sensitivity and component controls
M1/M3: Candidate counts, score bounds, and computational cost
These are tunable design choices. Executable cap limits retained route candidates; limits routes visible to feedback; bounds learned scores. Defaults are , , and raw candidate budget six. They bound representation size and correction magnitude; they are not claimed optimal.
We completed separate full MAPT retraining runs for all six variants below, each with seed 1, 64 epochs/8,192 episodes and Validation-only selection. The default is the matching seed-1 checkpoint, not the five-seed mean. All are evaluated on the same 600 scenarios (60 buildings × ten initializations), with difficulty weights 12:18:24.
| k | K | q_max | Rescue (%) | Difference (pp) | 95% building CI |
|---|---|---|---|---|---|
| 3 | 3 | 4 | 96.06 | 0.00 | Reference |
| 3 | 1 | 4 | 96.49 | +0.43 | [0.03, 0.82] |
| 3 | 2 | 4 | 96.91 | +0.84 | [0.24, 1.48] |
| 3 | 3 | 2 | 96.47 | +0.41 | [0.04, 0.77] |
| 3 | 3 | 8 | 96.69 | +0.62 | [0.06, 1.23] |
| 1 | 1 | 4 | 97.13 | +1.06 | [0.44, 1.74] |
| 2 | 2 | 4 | 95.69 | -0.37 | [-0.91, 0.11] |
Differences are computed before rounding. These are retraining comparisons, not inference-time substitutions. The CIs use 10,000 paired building resamples in six difficulty-by-generator strata, retain all initializations, and condition on a single fitted seed per setting. They do not establish rankings across future training seeds and are unadjusted for multiple comparisons. Truncated outcomes remain included.
Neither more candidates nor larger bounds yield monotonic gains. The model has the highest observed mean, so several retained routes are not necessary for the best result here. With one candidate, PACF still feeds route information into target choice. We retain the original default instead of reselecting it on Test. The candidate cap is an upper bound: fewer routes may be feasible.
| Actual cap k (all retained routes visible) | L40 median / p95 (ms) |
|---|---|
| 1 | 25.51 / 30.74 |
| 2 | 25.55 / 31.61 |
| 3 | 25.76 / 32.06 |
Separately, at fixed k=3, median/p95 values are K=1: 26.00/32.40; K=2: 26.27/32.64; q_max=2: 25.45/31.86; q_max=8: 25.54/32.12 ms, versus 25.37/31.76 for the matched default. These small differences do not establish a speed ranking. Timing includes candidate construction and neural decisions, using 24 Train-context labels (20 states), three warmups and ten repeats, synchronized L40 inference and normal caches. Loading and environment advancement are excluded; host-wide exclusivity was not established. Raw construction/first-hop searches persist at smaller caps, so retained-count reductions do not yield proportional savings.
M2: Input ablations and information/capacity controls
These two groups answer different questions. All use common MAPT training seeds 1–3, the same budget, Validation selection, and 600 scenarios. ± denotes sample SD across training-seed aggregate rates. Conditional building CIs describe fitted ensembles, not future retraining.
| Input condition | Rescue (%) | Difference from Full (pp) | 95% CI |
|---|---|---|---|
| Full PACF | 96.64 ± 0.66 | 0.00 | Reference |
| Relative statistics, no route tokens | 95.27 ± 1.38 | -1.36 | [-1.97, -0.81] |
| First-hop + statistics | 96.55 ± 0.40 | -0.09 | [-0.40, 0.20] |
| No observed hazard trends | 96.16 ± 1.09 | -0.48 | [-0.69, -0.27] |
Removing tokens or trends lowers the mean. The first-hop CI includes zero, so complete routes are neither established as necessary nor proved equivalent to first-hop information.
| Integration/capacity control | Rescue (%) | Control − PACF (pp) | 95% CI |
|---|---|---|---|
| Base | 94.86 ± 1.08 | -1.77 | [-2.42, -1.11] |
| Route-free capacity control | 93.63 ± 2.41 | -3.01 | [-3.47, -2.55] |
| Ordinary route head | 97.02 ± 0.34 | +0.39 | [0.10, 0.69] |
| Fixed minimum-cost bias | 96.74 ± 0.22 | +0.10 | [-0.17, 0.39] |
Active parameters are PACF 594,170, Ordinary 594,298, capacity control 594,493, versus Base 223,677. The capacity control uses Base inputs only; parameter matching does not equate function classes. Ordinary mean-pools the same route representations before standard target scoring, with independently trained encoders and random initialization; PACF scores before aggregation and initializes feedback at zero. Fixed feedback retrains with a Train-normalized minimum-cost bias, coefficient one, without a learned scorer.
These results support planner-derived information across several designs. They do not establish a unique benefit of PACF aggregation: Ordinary is higher, and fixed feedback is competitive. This is a component study, not a full factorial analysis.
Environment assumptions and location robustness
Comment
W1: Scope of the environment assumptions
We agree that exact target locations and centrally available hazard observations limit operational applicability. DMRIE is a controlled simulation platform for dispatch after a coordinator receives observations, not a validated deployment system. It does not model how reliable sensing or communication is achieved. Victim health is hidden from both actor and critic, and finite hazard trends need not form a sufficient belief state. The formulation is centralized partially observable control with variable-duration decision epochs, not a fully observed MDP or decentralized learning.
Known maps/targets also occur in assignment-and-planning studies, but that precedent does not prove realism. We therefore tested two specific relaxations of the location assumption, while retaining reliable communication and current hazard observations.
Q2: Approximate locations and local target movement
Frozen policies were evaluated on the first initialization of each of the 60 buildings, six policy families, three training seeds, and five conditions: 5,400 episodes. This is a 60-scene subset, not all 600 main scenarios. The 1,080 unperturbed episodes exactly reproduce their archived outcomes. Rescue rates use the same within-building/difficulty aggregation and difficulty weights 12:18:24; ± is training-seed sample SD.
For location errors, every 30 s a waiting victim is reported in a neighboring same-floor room within 10 m with probability 0.10 or 0.30. Physical positions do not change. Positive inspection confirms the room; negative inspection clears the false report until the next refresh without revealing the true room. Observations, masks, and commitments use these reports consistently. Victim identity/lifecycle remain centrally available, so this is not unrestricted search for unknown victims.
For movement, eligible waiting victims attempt a move every 60 or 30 s within the initial room and its one-hop same-floor neighborhood, within 10 m. Moves require passable spaces, exclude stairs/safe areas, and cannot occur during service, carrying, or rescuer co-location. Current positions remain exactly observed. Thus movement and uncertain reports are separate interventions, not combined uncertainty and motion.
Perturbation draws are keyed by scene, victim, and time independently of hazard randomness. All methods share the exogenous rule, though actions can change realized movement eligibility. All outcomes are retained.
| Condition | MAPDP Base | MAPDP PACF | MAPT Base | MAPT PACF | MAPT Ordinary | MAPT Fixed |
|---|---|---|---|---|---|---|
| Unperturbed | 95.22 ± 1.04 | 97.53 ± 1.08 | 95.93 ± 0.79 | 97.41 ± 0.81 | 98.06 ± 0.56 | 97.07 ± 0.75 |
| Location error 10% | 95.49 ± 0.54 | 97.50 ± 0.33 | 95.62 ± 0.89 | 97.59 ± 0.81 | 97.81 ± 1.02 | 97.07 ± 0.14 |
| Location error 30% | 95.52 ± 0.43 | 96.20 ± 1.28 | 92.25 ± 1.19 | 96.76 ± 0.09 | 96.36 ± 0.78 | 95.83 ± 0.32 |
| Motion every 60 s | 95.00 ± 0.91 | 96.82 ± 1.23 | 94.26 ± 1.45 | 97.35 ± 0.77 | 98.24 ± 0.33 | 97.50 ± 0.73 |
| Motion every 30 s | 95.19 ± 0.56 | 96.91 ± 1.65 | 93.02 ± 1.17 | 97.07 ± 0.75 | 97.59 ± 1.20 | 97.01 ± 0.05 |
All rates are percentages. Paired differences use unrounded results:
| Perturbation | MAPDP gain (pp) | 95% building CI | MAPT gain (pp) | 95% building CI |
|---|---|---|---|---|
| Location error 10% | +2.01 | [0.52, 3.64] | +1.98 | [0.74, 3.18] |
| Location error 30% | +0.68 | [-0.52, 1.91] | +4.51 | [2.69, 6.39] |
| Motion every 60 s | +1.82 | [0.71, 3.02] | +3.09 | [1.57, 4.57] |
| Motion every 30 s | +1.73 | [0.34, 3.27] | +4.04 | [2.04, 6.08] |
MAPT gains remain positive with conditional CIs above zero in all four perturbations. At 30% location error, its Base rate falls from 95.93% to 92.25%, while PACF falls from 97.41% to 96.76%. The larger gap reflects less degradation, not improvement caused by errors. MAPDP also has positive mean gains, but its 0.68 pp location-error gain has a CI containing zero. Ordinary remains competitive and often higher, so robustness is not unique to PACF aggregation.
The CIs condition on the fitted three-seed ensembles and sampled first initializations. These tests address the specified local disturbances, not uncertain global hazards, communication failures, unknown victim identities, unrestricted human movement, or real-world deployment. The revised claims retain those boundaries.
Performance, conventional and external baselines
Thank you for recognizing the motivation and modular design. We address all three weaknesses, three minor comments, and two questions in three comments: performance/baselines, sensitivity/ablations, and assumptions/robustness. W, M, and Q follow their order in your review.
W2: Gains and conventional methods
We expanded evaluation to 60 buildings disjoint from Train/Validation, 20 per difficulty, with ten initializations each (600 scenarios). Rescue counts completed evacuation, averaged within buildings and difficulty, then weighted 12:18:24. All methods share risk-weighted candidate planning, minimum-cost route execution, legal masks, commitments, and service rules. Neural methods use five training seeds, stochastic allocators five search seeds, and deterministic rules one rollout per scene.
Five-seed sample SDs for MAPDP Base/PACF are 1.47/0.58 pp; for MAPT 0.85/0.50 pp. Paired gains are positive in all five seeds per backbone. Mean gains are +2.39 pp (MAPDP, 95% building CI [1.96, 2.85]) and +1.98 pp (MAPT, [1.52, 2.46]). CIs condition on fitted ensembles, not future retraining; differences use unrounded values.
We also completed all SA/GA budgets rather than selecting one:
All 18,000 episodes, including truncations, remain included. The cost oracle matches execution, but the coverage-first nominal chain-cost surrogate does not predict future hazards or victim health. More search does not monotonically improve terminal rescue. Equal objective counts do not imply equal wall-clock budgets.
Q1: The original Medium decline and small Hard gain
We retain the original three-seed counterexample: MAPDP + PACF scored 95.66% at Medium versus 100% for Nearest/Hungarian. Extending the original Test to five seeds still gives MAPDP a -1.15 pp Medium difference, so a single unfavorable seed does not explain it.
Current Medium results are near saturation, limiting absolute gains. This does not explain why the original PACF model was worse. Few original buildings, training across mixed difficulties, and seed variation are plausible contributors, not verified causes. Both layouts and initializations changed, so the sign change cannot be assigned to either alone. Hard gains also vary by backbone and protocol. We claim scenario-dependent improvements over matched Base models, not uniform superiority.
W3: External assignment-and-planning comparator
We added RMCA-adapted from Chen et al., Integrated Task Assignment and Path Planning for Capacitated Multi-Agent Pickup and Delivery. We checked the author implementation, then implemented a separate rescue adaptation: regret-based request insertion and six LNS rounds, destroying 25% of requests per round. Requests are waiting room-level victim batches; insertion costs use the common planner and current service/queue estimates. The method respects commitments and shares rescue/carrying/handoff execution, without hidden health or future hazards.
DMRIE imposes no additional hard carrying capacity and does not reproduce the original collision-avoidance guarantees. This is an adapted external allocator, not an unchanged reproduction or native fire-rescue algorithm.
On 3,000 episodes, RMCA obtains 96.06 ± 0.25% (five search seeds). PACF-minus-RMCA is +0.33 pp for MAPDP (building CI [-0.48, 1.19]) and +0.65 pp for MAPT ([-0.23, 1.57]). Both include zero; no advantage over RMCA is established. Search and training seeds are not paired.
CPU median/p95 decision latency is 19.00/356.05 ms for RMCA versus 33.69/112.51 for SA-600 and 32.48/112.93 for GA-600, measured serially on fixed Train contexts (240 samples/method). Host-wide exclusivity was not established; this is not equal-time rescue quality or a GPU throughput ranking. Other stochastic-hazard rescue frameworks were reviewed but not all implemented, so external-method coverage remains partial.