acceptodds
Withdrawn from ICLR 2027

PA-MAPPO: Phase-Aware Policy Optimization for Cooperative Multi-UAV Pursuit under Partial Observability

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Cooperative multi-UAV pursuit offers a flexible means of intercepting unauthorized UAVs in urban low-altitude airspace. However, limited fields of view and building occlusions require the UAV team to search without current target information, coordinate interception after detection, and reacquire the evading UAV after visual contact is lost. Existing multi-agent reinforcement learning methods often rely on uniform objectives, reactive target following, and a single team-level value estimate, limiting phase-aware decision making and differentiated credit assignment. We propose Phase-Aware Multi-Agent Proximal Policy Optimization (PA-MAPPO), a centralized-training and decentralized-execution framework organized around search, pursuit, and reacquisition. To anticipate evader motion, PA-MAPPO introduces an attention-based LSTM encoder–decoder to predict a multi-step evader trajectory, providing interception guidance during visual contact and after temporary occlusion. To coordinate complementary approaches, we introduce an interception-oriented task representation that encodes relative-motion geometry and interception feasibility. To improve credit assignment, an individual–team dual-critic architecture separately learns agent-specific residual returns and the mean team return. A phase-dependent reward further aligns the learning signals with spatial coverage during search, coordinated encirclement during pursuit, and directed motion toward the last observed target position during reacquisition. Comparisons with representative multi-agent reinforcement learning baselines demonstrate improved capture reliability, mission efficiency, and visual-contact recovery. Ablation studies further verify the contributions of trajectory prediction, task representation, and dual-critic learning.

Primary area:
reinforcement learning
Keywords:
PA-MAPPO, cooperative pursuit, individual-team dual-critic learning

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_YveKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PA-MAPPO, a phase-aware multi-agent RL framework for cooperative UAV pursuit, around search, and reacquisition under occlusion and limited field of view. It combines an attention-based evader trajectory predictor, an interception-oriented task representation, a dual-critic design for finer credit assignment, and phase-dependent rewards. Experiments show improved capture reliability, mission efficiency, and visual-contact recovery over representative MARL baselines.

Critical strengths
  1. This paper is well-written and easy to follow.

  2. The problem is clearly defined, and the overall methodology is well-designed and targeted.

Critical weaknesses
  1. The baselines seem outdated.

  2. The experimental results are too thin. The paper only includes basic metrics and visualization results. More analysis on the design rationale of the components and methods, as well as ablation studies, should be presented.

  3. The trajectory prediction mechanism during target loss needs clarification. The paper predicts the target’s trajectory over a five-step horizon using observations up to the last team-visible time τt\tau_t, and continues to supply predicted features while the target remains invisible. However, it is unclear how these predictions are updated once the target loss exceeds the prediction horizon.

Questions for authors
  1. Could you provide some more recent baselines?

  2. Could the authors provide more experiments and analysis to better validate the method's superiority? For example, some comprehensive ablation experiments involving target-motion prediction, interception geometry, differentiated value estimation, and phase-dependent rewards within a CTDE framework. Existing experiments are insufficient to demonstrate that its specific design has unique advantages over other reasonable alternatives.

  3. PA-MAPPO is built on LSTM. Are there newer Transformer-based frameworks? What are their respective advantages and disadvantages?

Reviewer_D2k9OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PA-MAPPO for cooperative multi-UAV pursuit under partial observability. The problem is studied in a 2-D urban environment where buildings obstruct movement and line of sight. Three pursuing UAVs must search for and capture an evading UAV. The authors divide the task into three phases: search, pursuit, and reacquisition, depending on whether the target is currently visible or has been detected previously. The method extends MAPPO with an attention-based LSTM for predicting target motion, geometric features for interception, and an individual team dual-critic architecture. The two critics learn team-average and agent-specific residual returns, respectively, and their advantages are combined for policy optimization. Phase-dependent rewards encourage spatial coverage, coordinated pursuit, and target reacquisition. The authors evaluate PA-MAPPO against MAPPO, HAPPO, HASAC, and FACMAC in simulation, using three training seeds and 300 evaluation scenarios per seed. PA-MAPPO reports the highest mean Success Capture Rate (SCR) (74.67%) and improvements in mission efficiency and visual-contact recovery. Ablations examine the contributions of trajectory prediction, task representation, and dual-critic learning.

Critical strengths
  1. the paper addresses cooperative multi-UAV pursuit under limited visibility and occlusion, which is a practically relevant problem.
  2. clear formulation, the division into search, pursuit, and reacquisition phases provides a structured approach to the problem.
  3. PA-MAPPO reports improved capture success and mission efficiency compared with the evaluated MARL baselines.
  4. the ablation results suggest that trajectory prediction and geometric task features improve performance in the tested environment.
Critical weaknesses
  1. The experiments use only three pursuers, one scripted evader, and a single environment configuration. It is unclear whether the method generalizes to other scenarios.
  2. Only three training seeds are used, which makes the statistical evidence weak. The improvement over HASAC is also small (74.67% vs. 72.22% capture success).
  3. The baseline results raise questions. HAPPO achieves only 9.56% capture success compared with 65.67% for MAPPO. The paper does not clearly explain whether the baselines received the same rewards, observations, training budgets, and hyperparameter tuning.
  4. No learning curves or quantitative safety results are provided.
  5. The method mainly combines existing techniques, including LSTM-based prediction, geometric features, dual critics, and reward shaping. The algorithmic novelty is not clearly established.
  6. The dual-critic ablation is difficult to explain. Replacing it with conventional twin critics reduces capture success to 25.83%, much lower than plain MAPPO (65.67%). More controlled comparisons are needed to understand this result.
  7. The residual-reward critic may effectively introduce individual reward information rather than provide a fundamentally different credit-assignment mechanism. This distinction is not investigated.
  8. Phase-dependent reward shaping is presented as a contribution but is never independently ablated. Prediction accuracy is also not evaluated against a simple constant-velocity baseline.

Overall: The problem is relevant, but the proposed algorithmic contribution is not sufficiently demonstrated. The limited experiments, unclear baseline comparisons, and incomplete ablations make it difficult to determine where the reported improvements come from. These concerns require additional experiments and clarification before the paper can be properly assessed.

Questions for authors
  1. Were the same rewards, observations, and training budgets used for all the baseline methods? How were the baselines tuned? The large difference between HAPPO (9.56% SCR) and MAPPO (65.67% SCR) is surprising. Could the authors explain this gap?
  2. Why does the conventional twin-critic variant perform much worse (25.83% SCR) than even plain MAPPO (65.67%)? It would help to see a comparison using only the team critic while keeping the other components unchanged. An ablation of the phase-dependent reward is also needed to understand its contribution.
  3. The experiments use only three training seeds and one evader strategy. Can the authors test with more seeds and different evader strategies, team sizes, and obstacle configurations to show whether the results generalize?
  4. The residual reward is defined as the difference between an agent's reward and the team-average reward. How does this improve credit assignment compared with simply using individual rewards? A direct comparison would help clarify the benefit of the proposed dual critic.
Reviewer_9PezOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose a phase-aware MAPPO method for multi-UAV pursuit under partial observability. The framework combines multi-step evader trajectory prediction, an interception-oriented task representation, individual-team dual critics, and phase-dependent rewards to support search, pursuit, and reacquisition. Extensive experiments on cooperative multi-UAV pursuit in an urban workspace containing static obstacles demonstrate the method’s effectiveness in this environment.

Critical strengths
  1. The paper is well organized.
  2. The technical presentation is clear and easy to follow.
  3. The evaluation considers several performance metrics, including episode reward, capture success rate, and mission time.
Critical weaknesses

My main concerns are the insufficient discussion of related work, the unclear justification for the problem settings and methodological choices, and the limited experimental evidence supporting these choices and the method’s generalizability.

Major concerns

  1. Insufficient literature review. Reinforcement learning for UAVs and pursuit–evasion (PE) [1–3] is an active research area. The paper cites only 12 references and does not sufficiently discuss relevant recent developments. The literature review is insufficient to provide a comprehensive overview of the relevant research, leaving the novelty and significance of the claimed contributions unclear.

  2. Insufficient justification of the problem formulation. The authors do not clearly justify the adopted settings or explain how they inform the proposed method. For example, the kinematic model closely resembles the Dubins car model [4], but the authors provide neither a citation for this modeling choice nor a justification of its suitability for the UAVs considered. Given that “multi-UAV” appears in the title, the authors should explain how these modeling assumptions reflect the intended multi-UAV application.

  3. Insufficient motivation for the method. The authors clearly describe their method, but the reasons for several design choices remain unclear. For example, Section 3.2 describes an LSTM-based decoder without explaining why this architecture was chosen over other trajectory-prediction approaches.

  4. Insufficient experimental details. Although the paper reports a training and evaluation protocol and a hyperparameter table, the authors should clarify the training configurations of the individual baseline methods, particularly whether their hyperparameters were tuned for this task. Deep reinforcement learning can be sensitive to hyperparameters [5,6]. FACMAC appears to perform poorly, and the authors should explain these results. I think its poor performance may be related to hyperparameter settings or implementation issues.

  5. Insufficient ablation studies. Section 4.3 states that “w/o LSTM removes multi-step trajectory prediction,” with the corresponding results shown in Figure 4. This comparison demonstrates the benefit of including trajectory prediction, but it does not isolate the contribution of the LSTM architecture. The justification for adopting LSTM specifically therefore remains unclear.

Minor concerns

  1. The authors state that “The predictor is trained separately from PPO using simulator-generated history–future pairs.” The pursuers’ policies change during training, and the evader’s behavior depends on the pursuers’ actions. The trajectory distribution may therefore shift as training progresses. How does the predictor handle these distribution shifts? The authors should clarify how the prediction data and predictor are updated and report quantitative metrics evaluating prediction accuracy and reliability.

  2. The experiments use static obstacles and a single type of evader policy. Simple heuristics may also perform well under these settings, and the authors should include such methods as baselines. The authors should also consider more dynamic scenarios and a broader range of evader policies to evaluate the generalizability of the proposed method.

  3. Missing AI use statement. According to the ICLR 2027 Author Guidelines, an AI use statement is required rather than optional. I could not find such a statement in the manuscript.

References

[1] C. de Souza, R. Newbury, A. Cosgun, P. Castillo, B. Vidolov, and D. Kulić. “Decentralized Multi-Agent Pursuit Using Deep Reinforcement Learning.” IEEE Robotics and Automation Letters, 6(3):4552–4559, 2021. DOI: 10.1109/LRA.2021.3068952.

[2] Y. Xiang, S. Li, R. Li, Z. Zhao, and H. Zhang. “Decentralized Consensus Inference-Based Hierarchical Reinforcement Learning for Multiconstrained UAV Pursuit-Evasion Game.” IEEE Transactions on Neural Networks and Learning Systems, 36(10):18229–18243, 2025. DOI: 10.1109/TNNLS.2025.3582909.

[3] Y. Liu, C. Liu, Y. Meng, B. Jiang, and X. Wang. “Observer-Based Multi-Agent Reinforcement Learning for Pursuit-Evasion Game With Multiple Unknown Uncertainties.” IEEE Transactions on Automation Science and Engineering, 22:14332–14345, 2025. DOI: 10.1109/TASE.2025.3560757.

[4] L. E. Dubins. “On Curves of Minimal Length with a Constraint on Average Curvature, and with Prescribed Initial and Terminal Positions and Tangents.” American Journal of Mathematics, 79(3):497–516, 1957. DOI: 10.2307/2372560.

[5] P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. “Deep Reinforcement Learning That Matters.” Proceedings of the AAAI Conference on Artificial Intelligence, 32(1):3207–3214, 2018. DOI: 10.1609/aaai.v32i1.11694.

[6] T. Eimer, M. Lindauer, and R. Raileanu. “Hyperparameters in Reinforcement Learning and How To Tune Them.” Proceedings of the 40th International Conference on Machine Learning, PMLR 202:9104–9149, 2023.

Questions for authors

Please see the weaknesses above for detailed concerns.

  1. Could you clarify the main differences between the proposed method and the most closely related work?
  2. Could you explain the reason behind the chosen environment settings and the use of an LSTM-based trajectory predictor?
Reviewer_Lj2KOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PA-MAPPO for multi-UAV cooperative pursuit-evasion under limited field of view and building occlusion. The method combines attention-LSTM target trajectory prediction, interception geometry features, individual residual and team dual critics, and a reward design with three phases: search, tracking, and reacquisition. In a 2D simulation with three pursuing UAVs against one rule-based evading UAV, the method achieves a 74.67% capture success rate.

Critical strengths
  1. Search, tracking, and reacquisition after occlusion involve different information conditions and behavioral requirements. The paper addresses these issues separately through trajectory prediction and phase-based rewards, with a clear application motivation.
  2. In addition to capture success rate, the paper reports task duration, search time, and lost-contact time, and includes ablations of trajectory prediction, task representation, and the dual critics. The results provide preliminary support for the effectiveness of the complete system.
Critical weaknesses
  1. The method simultaneously adds prediction features, handcrafted geometric representations, dual critics, and complex rewards, but there is no ablation of the phase rewards, nor is it clear whether all baselines use the same rewards and available information. Therefore, the current results make it difficult to distinguish the role of algorithmic improvements from that of task engineering design. In addition, there is a lack of comparison with and sufficient distinction from NAGC, which directly studies the same type of limited-field-of-view urban pursuit-evasion problem.
  2. The experiments cover only a fixed number of UAVs, a fixed building scale, and one repulsive-force evasion strategy, and have not yet demonstrated generalization to other evasion behaviors or environment configurations. Relative to the strongest baseline HASAC, the success rate improves by only 2.45 percentage points.
Questions for authors
  1. Do all baselines use the same phase rewards, target history, and communication information? After unifying these conditions, does PA-MAPPO's advantage still hold? Why was NAGC, which addresses the same task, not compared?
  2. What is the relationship between weighting the individually normalized residual advantage and team advantage and the defined team return objective? Compared with a single critic that directly learns individual returns, does the gain come from value decomposition or from changes in reward weighting?
  3. The predicted trajectory starts from the last observation time and has a length of only five steps. When occlusion exceeds five steps, how is temporal alignment performed or stale predictions handled? Is there a constant-velocity extrapolation control, as well as prediction error and reacquisition performance statistics broken down by occlusion duration?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.