acceptodds
Under review as a conference paper at ICLR 2027

CoSeg-MARL: Cooperative Multi-Agent Reinforcement Learning for Brain Tumor MRI segmentation

Abstract

Accurate segmentation of brain tumors from multimodal magnetic resonance imaging (MRI) remains challenging due to the heterogeneous appearance of tumor sub-regions, including necrosis (NEC), edema (ED), and enhancing tumor (ET), and the need to maintain consistency among anatomically related segmentation predictions. In this paper, we propose CoSeg-MARL, a communicative multi-agent reinforcement learning framework based on Multi-Agent Deep Deterministic Policy Gradient (MADDPG) for iterative brain tumor segmentation from multimodal MRI. The framework employs three specialized agents that collaboratively refine segmentation predictions through cooperative sequential decision-making. Each agent specializes in one of three hierarchically related tumor regions whole tumor (WT), tumor core (TC), and enhancing tumor (ET) and coordinates with the other agents through shared observations, shared convolutional representations, and explicit message passing. Each agent generates continuous refinement actions comprising four global segmentation controls and a coarse spatial coefficient grid, which are transformed into updated segmentation masks through a deterministic applicator. A hybrid training objective combines supervised Dice and binary cross-entropy losses with a reinforcement learning policy-gradient term, while the reward function encourages segmentation improvement and penalizes hierarchical inconsistencies. Experiments on 2D MRI slices from a patient-level subset of BraTS 2020 evaluate the proposed framework under two initialization strategies: refinement from zero-initialized logits (cold start) and refinement from pretrained U-Net predictions (warm start). CoSeg-MARL achieves mean Dice scores of 0.821 and 0.916, respectively. Compared with the pretrained backbone, warm-start refinement improves the Dice scores for whole tumor, tumor core, and enhancing tumor segmentation from 0.811, 0.830, and 0.838 to 0.927, 0.916, and 0.907, respectively.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Tumor Segmentation, Multi-Modal, Multi-Agent, DDPG

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 2 reviews, above 0% of the others.
Reviewer_12PqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CoSeg-MARL, a communicative multi-agent reinforcement learning framework for iterative brain tumor segmentation from multimodal MRI. Three agents specialize in whole tumor (WT), tumor core (TC), and enhancing tumor (ET), exchanging information through shared representations and explicit message passing. Each agent predicts global segmentation controls and a coarse spatial refinement grid, which a deterministic applicator converts into updated predictions. Training combines supervised Dice and binary cross-entropy losses with an actor–critic objective and hierarchy-aware rewards. Evaluation uses 2D slices from a 200-patient subset of BraTS 2020 with patient-level partitioning. The paper reports mean Dice scores of 0.821 for cold-start refinement and 0.916 for refinement initialized by a pretrained U-Net, compared with 0.826 for the backbone alone.

Critical strengths
  1. The agent formulation reflects the structure of the segmentation task. Assigning agents to WT, TC, and ET provides a coherent way to model region-specific refinement while accounting for their nested anatomical relationships. Combining global controls with spatial refinement coefficients also offers a flexible action representation.

  2. The reported warm-start improvements are substantial across multiple segmentation measures. Table 1 reports improvements over the pretrained backbone in Dice and ASSD for all three regions. Patient-level partitioning is also an appropriate precaution against leakage between slices from the same patient. If verified, these results would support the potential value of the complete refinement pipeline.

Critical weaknesses
  1. Inconsistent Dice and IoU values. Table 1 reports warm-start WT Dice of 0.927 and IoU of 0.725. For the same binary masks and consistent averaging, these values are incompatible: mean Dice of 0.927 implies mean IoU of at least approximately 0.864. Similar inconsistencies appear in other entries. This raises a substantial concern about metric computation or reporting.

  2. The contribution of reinforcement learning is not established. The actor also receives direct Dice and BCE supervision, but no supervised-only refinement baseline is provided. Consequently, the reported gains cannot be attributed specifically to reinforcement learning.

  3. The benefit of multi-agent cooperation is not isolated. There is no comparison against a single policy jointly refining all three regions or against agents without explicit communication. Since all agents already access shared masks, logits, and representations, the additional value of message passing remains unclear.

  4. Hierarchy preservation is motivated but not directly evaluated. The paper does not report hierarchy-violation rates or isolate the effects of the hierarchy reward and hierarchy-control action. Improved Dice alone does not establish more anatomically consistent predictions.

  5. The deterministic applicator is insufficiently specified. The paper describes threshold adjustment, temperature scaling, morphology, and hierarchy enforcement without giving the exact update equations, parameter mappings, or operation order. These choices determine the transition dynamics and are essential for reproduction.

  6. The optimization procedure is incomplete and potentially inconsistent with the stated algorithm. The hybrid loss weights and gradient paths are unspecified. Moreover, the critics are described as receiving each agent’s own action rather than the joint action, leaving the relationship to standard MADDPG unclear.

  7. The fairness of the baseline comparison is uncertain. Table 2 does not clearly explain whether all methods were retrained on the same subset and evaluated using identical preprocessing, patient splits, and slice-level metrics. References associated with some entries concern different segmentation settings, making the provenance of the reported numbers particularly important.

  8. The evaluation has limited coverage. Experiments use a selected 200-patient subset of one dataset and report slice-level averages. The criteria for selecting patients and informative slices are insufficiently defined, and volumetric performance is not evaluated. This limits conclusions about full-volume brain tumor segmentation.

  9. The test-time refinement procedure is unclear. Training termination can depend on ground-truth Dice, but the inference stopping rule and maximum number of refinement steps are not specified. These details affect both reproducibility and the validity of the reported evaluation.

10 Computational cost and training stability are not assessed. The framework adds iterative actor evaluations and multiple critics during training, but reports no inference latency, memory usage, parameter counts, or variability across independent runs. The practical cost of the reported improvements therefore remains unknown.

  1. Several reporting discrepancies require correction. Table 2 gives a Dice standard deviation of 4.80, whereas the accompanying text states 2.64%. Distance units are unclear for the 2D evaluation, and ASSD is compared with ABD without establishing equivalence. These discrepancies compound concerns about the reliability of the results.
Questions for authors
  1. Were Dice and IoU calculated from identical binary masks, samples, and averaging weights? How do you explain the incompatible values in Table 1, particularly warm-start WT Dice of 0.927 versus IoU of 0.725?

  2. How are slices with absent WT, TC, or ET handled when computing Dice, IoU, HD95, ASSD, and RVD? Are the same slices included in every metric?

  3. What precisely defines an “informative” slice, and how were the 200 patients selected? Were these criteria fixed before training, and do they use tumor annotations?

  4. Are the final table results calculated exclusively on the 30 test patients? Which split is used for the Dice curves in Figure 4, checkpoint selection, and hyperparameter tuning?

  5. What stopping criterion is used when ground-truth masks are unavailable? Please specify the maximum refinement horizon and confirm that test labels are not used to select the best intermediate prediction.

  6. What are the exact equations and operation order for converting the four global controls and spatial grid into updated logits and masks? How are conflicting hierarchy controls from different agents resolved?

  7. What is the complete actor objective, including all loss weights? Through which outputs and operations are the supervised gradients propagated, particularly when thresholding and morphology are involved?

  8. Do the critics receive the joint actions of all agents or only their respective agent’s action? If only the latter, how are the effects of other agents’ simultaneous updates accounted for?

  9. Which Table 2 results were obtained through your own experiments, and which were taken from previous publications? Were all entries evaluated on the same BraTS version, patient split, tumor targets, and metric definitions?

10 Does the full method outperform the same refinement actor trained using only supervised losses under an identical training and inference budget?

  1. What happens when explicit communication is removed, when one policy predicts all three regions, and when hierarchy penalties are disabled? Please report both segmentation accuracy and hierarchy-violation rates.

  2. How consistent are the gains across independent training seeds and individual test patients? What inference time and number of refinement steps are required relative to the pretrained U-Net?

Reviewer_NDgcOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoSeg-MARL, an iterative brain tumor segmentation framework with three agents specializing in whole tumor (WT), tumor core (TC), and enhancing tumor (ET). The framework combines shared representations, explicit communication, continuous refinement actions, and a hybrid supervised and reinforcement learning objective. On a 200-patient subset of BraTS 2020, the authors report an improvement in mean slice-wise Dice from 0.826 for the pretrained U-Net backbone to 0.916 after warm-start refinement.

Critical strengths

Coordinating predictions using the nested relationships among WT, TC, and ET is a reasonable task-specific motivation. The paper explicitly describes patient-level splitting and distinguishes slice-wise evaluation from volumetric evaluation. However, two fundamental concerns prevent the current submission from supporting acceptance.

Critical weaknesses

1.Missing ablations for the main contributions. Comparing the backbone, cold-start, and warm-start settings does not isolate reinforcement learning, communication, or hierarchical constraints. Refinement also introduces additional capacity, iterative computation, and Dice supervision, preventing attribution of the improvement to MARL. 2.An unexplained departure from MADDPG. Section 3.4.2 describes critics conditioned on the shared state and only their own agent’s action, whereas standard MADDPG conditions centralized critics on joint actions. The modification and corresponding update rules are insufficiently specified, making the claimed algorithm difficult to assess.

Questions for authors

1.With architecture, initialization, supervised losses, and training budget held constant, how much improvement does reinforcement learning provide? Please report separate ablations removing reinforcement learning, message passing, and hierarchical constraints to establish each component’s contribution. 2.Does the implemented critic receive the joint actions of all agents? If it receives only its own agent’s action, please explain the departure from standard MADDPG and provide the complete critic objective and actor update equations.

1 comment

Since I have not found the authors' responses to my concerns, I will keep my rating.

Comments on the paper

1 comment

The authors did not respond and address the concerns. I will keep my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.