- Rating
- 3(+1)
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes CONS-GNN for planner feasibility prediction in multi-UAV task planning. The formulation of conservative screening based on three planner outcomes is reasonable, and the experimental evaluation includes multiple GNN baselines, ablation studies, and scalability analyses across different UAV team sizes. However, the proposed method primarily combines existing techniques, including heterogeneous GNNs, auxiliary multi-task supervision, and threshold-based selective classification, resulting in limited algorithmic novelty.
Critical strengths
-
Paper is generally well written and clearly organized, with the methodology and experimental setup presented in an understandable manner.
-
It achieves promising experimental results and outperforms existing methods on several evaluation metrics, although some concerns remain regarding the interpretation and validity of these results.
Critical weaknesses
-
The core components of CONS-GNN include three parts: a heterogeneous UAV–task–region graph, relation-aware edge-conditioned message passing with an auxiliary difficulty prediction objective, and a validation-selected threshold for conservative screening. However, these components are largely based on established techniques. Relation-aware message passing, heterogeneous graph representations, and auxiliary multi-task learning are well-studied approaches in graph learning, while validation-based rejection thresholds are standard in selective classification. Section 4 mainly describes how these components are integrated, without introducing a clearly distinguishable new learning mechanism.
Although Table 2 shows that CONS-GNN outperforms GAT, MPNN, R-GCN, HGT, and Adapted GRN, the improvements over stronger baselines are relatively modest. For example, CONS-GNN achieves an accuracy of 0.9517 compared with 0.9462 for Adapted GRN, while its PR-AUC of 0.9831 is only slightly higher than the approximately 0.981 achieved by several existing GNN models. Its planner-call savings exceed those of Adapted GRN by 3.71 percentage points.
The main contribution appears to be an effective task-specific heterogeneous graph representation and selective screening pipeline, rather than a substantial methodological advance in GNN learning. The authors should clarify which aspects of the proposed algorithm cannot be readily reproduced using existing relation-aware GNN architectures and multi-task classification frameworks.
-
One of the paper's central claims is that explicitly distinguishing Unresolved from Infeasible substantially reduces false rejections compared with binary outcome modeling. Table 4 reports that three-way CONS-GNN achieves 41.36% planner-call savings with an observed false-rejection rate (FRR) of 0.0241%, whereas the binary-collapse variant achieves 52.96% savings with an FRR of 0.2414%. Although the three-way model has an approximately tenfold lower observed FRR, the two models operate at substantially different rejection levels: the binary variant rejects 11.60 percentage points more instances.
This comparison does not establish that three-way modeling provides a better safety–efficiency trade-off. A more conservative operating threshold generally reduces false rejections at the cost of rejecting fewer instances. The reported difference may therefore partly reflect the selected operating points rather than an inherent advantage of the three-class formulation.
Furthermore, both models achieve observed test FRRs below the validation constraint of 0.5%. As the authors acknowledge in Section 5.4, these results do not establish superiority at matched test risk.
A more convincing evaluation would compare planner-call savings at the same FRR levels or provide complete FRR–savings curves for both formulations. Without such comparisons, the current experiments demonstrate different operating trade-offs but do not conclusively support the claimed advantage of three-way outcome modeling.
-
This paper reports that CONS-GNN avoids 41.36% of planner calls while falsely rejecting only 3 of 12,449 planner-feasible instances across three test sets, corresponding to an observed FRR of 0.0241%. This is a promising result under the evaluated conditions.
However, this low false-rejection rate is achieved through empirical threshold selection on the validation set. Specifically, the authors jointly select the model checkpoint and rejection threshold to maximize planner-call savings while satisfying a validation FRR constraint of 0.5%. Such a procedure does not guarantee that the same risk level will be maintained on unseen or shifted task distributions.
And all primary labels are generated using a fixed EECBS–3D-SIPP planning pipeline with a restrictive default budget of 0.03 seconds. In Section 5.4, the authors increase this budget by factors of two and five and find that 0.80% and 1.57% of the evaluated instance labels change, respectively. However, this analysis only examines label stability and does not evaluate the screening gate's false-rejection behavior under the modified planning budgets.
This limitation is important because CONS-GNN predicts the outcome of a particular planner-budget pair rather than intrinsic task feasibility. Increasing the planning budget may turn some previously Unresolved instances into Feasible ones, potentially changing the false-rejection risk associated with the previously selected threshold.
So, the current experiments support the effectiveness of CONS-GNN under a fixed planner, computational budget, and synthetic test distribution, but do not establish that its conservative behavior remains reliable across different planning budgets.
Questions for authors
See weakness
4 comments
Response to 1eAr (2/2): Weaknesses 2–3 — Gate risk and planning budgets
Part 2/2: Weaknesses 2–3. The numbering below follows your original review.
Weakness 2. Three-way versus binary screening at the same FRR
We agree that the approximately tenfold FRR difference at the original operating points is not sufficient to establish a better trade-off: the three-way gate also rejects fewer requests. We have added a comparison at the same observed false-rejection counts in Supplementary Section H.5 and Table S13.
| False rejections | FRR (%) | Three-way p_I savings (%) | Binary savings (%) |
|---|---|---|---|
| 3 | 0.0241 | 41.36 | 39.92 |
| 12 | 0.0964 | 46.52 | 46.03 |
| 30 | 0.2410 | 50.38 | 52.96 |
| 62 | 0.4980 | 54.40 | 56.12 |
At 3 and 12 errors, the three-way gate saves 1.44 and 0.49 percentage points more calls, corresponding to 432 and 147 additional avoided calls among 30,000 requests. At 30 and 62 errors, binary collapse has higher savings. FRR here uses pooled counts with 12,449 feasible instances; 30/12,449 gives 0.2410%, whereas the original 0.2414% is a run mean.
We also compare p_I with 1 − p_F = p_I + p_U from the same three-way model. At the two low-FRR points, the latter saves 38.87% and 45.23%, below the 41.36% and 46.52% obtained with p_I alone. Figure S8 presents all three scores. Its lines connect the evaluated points and do not imply separately evaluated intermediate thresholds.
We have revised Contribution 3, Section 5.4, and the Conclusion to limit the advantage to the two reported low-FRR points. These results are from post hoc test-set threshold scans and serve as diagnostics; deployment evaluations still use validation-selected frozen thresholds. They support a benefit in the evaluated low-FRR region, with the preferred trade-off depending on the tolerated cost of false rejection.
Weakness 3. False rejections when the planning budget changes
We agree that label stability alone does not measure the original gate's risk under a larger budget. We have added a frozen-gate evaluation in Supplementary Section H.6 and Table S14, and summarized the false-rejection results in Section 5.4. On 3,000 instances, we keep the model weights and threshold selected under the 0.03 s budget fixed, then rerun the reference pipeline with budgets of 0.06 and 0.15 s.
| Budget (s) | Feasible instances | CONS-GNN false rejections | CONS-GNN FRR (%) | Binary false rejections | Binary FRR (%) |
|---|---|---|---|---|---|
| 0.03 | 1,245 | 0 | 0.0000 | 3 | 0.2410 |
| 0.06 | 1,262 | 1 | 0.0792 | 6 | 0.4754 |
| 0.15 | 1,278 | 2 | 0.1565 | 11 | 0.8607 |
CONS-GNN rejects one newly feasible instance at 0.06 s and two at 0.15 s. These instances become solvable under the larger budget but are still rejected by the original threshold, directly measuring the effect raised in the review.
Because the inputs, models, and thresholds are fixed, the rejection decisions do not change: call savings remain 41.37% for CONS-GNN and 52.97% for binary collapse. What changes is the reference outcome, including the feasible denominator and false-rejection count. Binary collapse also saves more calls, so the FRR difference alone is not a comparison at matched risk.
The validation FRR constraint is empirical; it does not provide a formal guarantee that the same FRR will hold under a shifted data distribution. Our conclusion is restricted to the tested budgets: at two and five times the original time budget, CONS-GNN's observed FRRs are 0.0792% and 0.1565%, with savings of 41.37%. As stated in Section 6, changes in planner, search limits, or task distribution require renewed threshold validation.
Thank you for the additional experiments and clarifications. I appreciate the authors' efforts to address my concerns.
The response for W1 has helped me better understand the practical contribution of this work, particularly the reduction in planner calls. This partially alleviates my original concern. However, from the perspective of further improving the paper, I would still encourage the authors to clarify this contribution more explicitly in the paper's narrative. In particular, it would be helpful to articulate how the proposed components jointly contribute to reducing planner calls and what the central contribution of this domain-specific formulation is.
For W2,the additional results are largely consistent with my original concern. From the perspective of planner-call savings, the three-way does not demonstrate a clear overall advantage over the binary alternative, as its advantage is mainly observed in the low-FRR region, while the binary formulation performs better at higher FRR levels. Nevertheless, I still consider this additional analysis valuable.
For W3, I consider my original concern largely addressed.
I am willing to raise my rating from 2 to 3 in light of the additional experiments and clarifications. However, I would like to emphasize that my overall assessment remains closer to Borderline rather than Weak Accept. As reflected in the responses to W1 and W2, the proposed components have not yet demonstrated clear and substantial performance advantages individually, and the work, in its current form, is better characterized as a domain-specific framework integrating existing techniques than as a significant methodological advance. While I acknowledge its practical value, I believe these limitations should still be taken into consideration in the overall assessment.
Thank you for carefully reviewing the additional experiments and updating your assessment. We agree with your suggestions on the contribution and interpretation of the results and have revised the paper accordingly.
The Introduction and Section 4 overview now clarify the contribution as a task-specific formulation and screening procedure for multi-UAV planner-call allocation, built on established learning techniques. The graph encodes assignment compatibility, task competition, and airspace constraints; relation-aware message passing combines this information, while difficulty supervision is used only during training. The three outcome labels distinguish feasible, definitively negative, and unresolved results of the fixed pipeline under its budget. At deployment, only requests whose infeasibility score exceeds the validation-selected threshold are rejected; all others are forwarded to the planner.
Section 5.2 more clearly separates the practical benefit of the complete pipeline from its modest gains over tuned baselines. Section 5.3 and Figure S5 also clarify that the ablation variants differ in operating points and observed FRRs, so the original savings gaps cannot be interpreted as independent component contributions.
Regarding W2, we agree that the results do not establish a clear overall advantage for three-way screening. Section 5.4 and the Conclusion now explicitly state that, in the post hoc test-set diagnostics, three-way screening saves more calls at the two evaluated low-FRR points, whereas binary collapse performs better at the two higher-FRR points.
We appreciate your suggestions for presenting the work's practical value and limitations more precisely.
Response to 1eAr (1/2): Weakness 1 — Contribution and baselines
Part 1/2: Weakness 1. The numbering below follows your original review.
Thank you for the careful assessment of the contribution and evaluation. We have clarified the method's scope, added a closer tuned baseline, and extended the gate evaluation to address the operating-point and budget concerns.
Part 2 addresses Weaknesses 2–3 on matched-FRR comparisons and the frozen gate under larger planning budgets.
Weakness 1. Contribution and comparison with existing relation-aware GNNs
We agree that heterogeneous graphs, relation-aware message passing, auxiliary supervision, and validation-based rejection are established techniques. Our contribution is their task-specific formulation for allocating multi-UAV planner calls. Section 6 now states this scope explicitly.
The graph represents UAV–task assignment compatibility, temporal and spatial competition between tasks, task–region usage, and regional connectivity. Its target is the outcome of a fixed EECBS–3D-SIPP pipeline under a stated computation budget: Feasible, Infeasible, or Unresolved. This separates failure to reach a decision within budget from a definitive negative result of that procedure. At deployment, requests are rejected only when their predicted Infeasible probability exceeds a validation-selected threshold; the reference planner handles the rest.
These principles can be implemented within existing relation-aware GNN and multi-task frameworks. We therefore position the contribution as the task-specific representation, three-outcome formulation, and conservative screening procedure, rather than a new general-purpose GNN learning rule.
To test whether the chosen configuration brings a practical benefit, we added a closer relation-aware baseline. It retains the node types, edge attributes, type-wise pooling, three-way output, and auxiliary difficulty supervision, but uses within-relation mean aggregation. CONS-GNN instead computes message gates from source-node, destination-node, and edge attributes. We also searched 12 hyperparameter configurations for this baseline and Adapted GRN (Supplementary Section H.7).
Thresholds were selected on validation data and frozen before testing. Supplementary Table S15 reports:
With the same three observed test errors, CONS-GNN avoids 498 and 306 additional calls relative to the two tuned baselines. These are comparisons of the complete tuned configurations, so we do not attribute the entire difference to the message gate alone.
Section 5.2 now reports this comparison alongside the online cost: mean latency falls from 18.84 ms/request without screening to 15.31 ms/request with CONS-GNN, including graph construction and inference. This 3.53 ms, or 18.74%, reduction supports the practical value of the screening procedure beyond its modest gains in classification metrics.