CoSeg-MARL: Cooperative Multi-Agent Reinforcement Learning for Brain Tumor MRI segmentation

Accurate segmentation of brain tumors from multimodal magnetic resonance imaging (MRI) remains challenging due to the heterogeneous appearance of tumor sub-regions, including necrosis (NEC), edema (ED), and enhancing tumor (ET), and the need to maintain consistency among anatomically related segmentation predictions. In this paper, we propose CoSeg-MARL, a communicative multi-agent reinforcement learning framework based on Multi-Agent Deep Deterministic Policy Gradient (MADDPG) for iterative brain tumor segmentation from multimodal MRI. The framework employs three specialized agents that collaboratively refine segmentation predictions through cooperative sequential decision-making. Each agent specializes in one of three hierarchically related tumor regions whole tumor (WT), tumor core (TC), and enhancing tumor (ET) and coordinates with the other agents through shared observations, shared convolutional representations, and explicit message passing. Each agent generates continuous refinement actions comprising four global segmentation controls and a coarse spatial coefficient grid, which are transformed into updated segmentation masks through a deterministic applicator. A hybrid training objective combines supervised Dice and binary cross-entropy losses with a reinforcement learning policy-gradient term, while the reward function encourages segmentation improvement and penalizes hierarchical inconsistencies. Experiments on 2D MRI slices from a patient-level subset of BraTS 2020 evaluate the proposed framework under two initialization strategies: refinement from zero-initialized logits (cold start) and refinement from pretrained U-Net predictions (warm start). CoSeg-MARL achieves mean Dice scores of 0.821 and 0.916, respectively. Compared with the pretrained backbone, warm-start refinement improves the Dice scores for whole tumor, tumor core, and enhancing tumor segmentation from 0.811, 0.830, and 0.838 to 0.927, 0.916, and 0.907, respectively.