acceptodds
Under review as a conference paper at ICLR 2027

MedRefineAgent: Risk-Aware Multi-Turn Mask Refinement for Medical Image Segmentation

Abstract

Medical image segmentation has advanced along two largely separate lines: prompt-free specialist models that automatically produce complete masks, and promptable foundation models that refine regions through points, boxes, or other user prompts. In practical clinical pipelines, however, the input is often neither an empty image nor a manually prompted target, but an imperfect coarse mask produced by an automatic segmenter. This setting calls for a unified mechanism that can inherit the efficiency of prompt-free segmentation while exploiting the local correction ability of promptable models. In this paper, we propose MedRefineAgent, a risk-aware multi-turn agent for automatic coarse-to-fine medical mask refinement across both 2D images and 3D volumes. Specifically, 1) a prompt-free backbone provides the initial coarse mask and an auxiliary reference for automatic risk localization; 2) a promptable local refiner is invoked only on high-risk regions through a structured action space of region, repair primitive, and hyper-parameter; and 3) conservative repair control coordinates gating, action pruning/reranking, and best-so-far rollback to suppress harmful edits. A teacher-distilled training pipeline further transfers search-based repair trajectories into lightweight test-time policies. Extensive experiments on abdominal, cardiac, endoscopic, skin-lesion, and fundus benchmarks show that MedRefineAgent consistently improves coarse predictions and achieves competitive or leading Dice performance across diverse 2D/3D medical segmentation settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
CV: Medical and Biological Imaging

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_DghcOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper treats medical segmentation as a repair problem starting from a coarse mask. An nnU-Net provides the initial prediction and a reference mask from complementary folds, a risk map picks candidate regions, and MedSAM2 is called through a structured action space. A learned controller plus proxy rollback decides whether to keep an edit, and teacher trajectories are distilled into the test-time policy. The evaluation covers 2D and 3D benchmarks.

Critical strengths

The clinical setup is sensible: in practice one often starts from a coarse automatic mask, not from a blank prompt. I also like that the action space is explicit and that the method tries to prevent harmful edits instead of just running SAM repeatedly. Covering CT, MRI, endoscopy, dermoscopy, and fundus images is a useful breadth check.

Critical weaknesses

The Kvasir results are what stopped me. Table 9 is described as the development split with 880 cases and reports 88.51 Dice, which is exactly the main-table result and the five-turn ablation result. Also, repairable recall changes with the number of teacher trajectories, although that should be a property of the teacher and available actions, not of student training size. This looks like possible overlap between the teacher/tuning split and the reported evaluation. Until the authors give an exact case-level split and explain those numbers, I cannot trust the headline gain.

Even if the split is clean, the paper does not separate the controller from simpler use of the same references. There is no direct coarse+reference fusion baseline and no fixed one-step MedSAM2 prompt under the same budget. Most of the gain is already there at turn 1; later turns add a tiny amount of Dice at a much higher runtime. So the current evidence supports a reasonable one-step repair pipeline, but not the full multi-turn agent with gating, ranking, and rollback.

Questions for authors
  1. What are the exact Kvasir training, development, and test sizes, and why does the 880-case development result equal the main test result to two decimal places?
  2. Why does repairable recall change from 45.83% to 79.17% with more teacher trajectories if it is defined from the teacher’s available actions?
  3. Can you add a direct coarse/reference fusion baseline and a fixed one-step MedSAM2 baseline with the same reference information?
  4. Can you report paired per-case bootstrap or McNemar tests for the Dice gains and a table that isolates the learned gate, ranker, and rollback?
Reviewer_brEgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MedRefineAgent automatically repairs an existing medical segmentation mask. Disagreement with an auxiliary reference identifies candidate regions, and a structured policy selects prompt-based or morphological repairs. A search teacher supervises lightweight gating and ranking modules. At inference, reference-based proxy scores guide lookahead, stopping, and rollback. Experiments cover abdominal and cardiac CT/MRI and three native 2D datasets, with ablations on turn budget, action space, localization, and distillation.

Critical strengths

Automatically correcting a coarse mask without test-time clicks is a useful problem. The explicit region, primitive, and parameter action space makes the repair procedure inspectable, while stopping and rollback address the practical risk of damaging an already reasonable prediction.

The reported improvements span multiple modalities and anatomical targets. Tables 1 and 2 show gains of 1.12–3.07 Dice points over the designated coarse predictor. The paper also clearly distinguishes the ground-truth-guided teacher from the deployed policy and acknowledges that disagreement and proxy scores do not provide a safety guarantee.

Critical weaknesses
  1. The comparisons do not yet isolate the controller's contribution under the same information and supervision budget. The agent uses a fold-0 coarse prediction, a reference built from four complementary folds, and a promptable refiner. Tables 1 and 2 include reference-only scores, which is useful, but omit the direct-fusion control discussed in the text. The paper also states that comparison rows without a same-protocol marker may use different splits or prompts. A comparison with direct fusion and a simple automatic local-refinement policy using the same inputs is needed to establish what the learned sequential controller adds.

  2. The deployed controller's individual mechanisms and reliability are insufficiently tested. Table 4 uses oracle ground-truth selection and therefore measures action-space potential rather than the learned policy's benefit. The turn-budget study does not separately test the gate, ranker, progressive schedule, lookahead, or rollback. On Kvasir, the first turn already raises Dice from 0.8544 to 0.8806; five turns reach 0.8851 while runtime increases from 15.41 to 33.57 seconds. Moreover, the five-turn degradation rate is 1.00% in Table 3 but 2.05% for the distilled student in Table 10. Without consistent protocols, uncertainty estimates, and deployed-policy controls, the evidence for the added multi-turn machinery remains limited.

Questions for authors
  1. How do direct fusion and a simple one-step or greedy refinement policy perform with the same coarse/reference predictions, refiner, and development data? Which SAMRefiner and SegRefiner results were reproduced under exactly this protocol?
  2. What does each deployed control component add over that common baseline? In particular, does the progressive schedule outperform an unrestricted action library at matched compute, and how much does proxy rollback reduce actual degradation?
  3. Please give the case counts and separation of segmentation training, teacher/controller training, hyperparameter selection, and final testing for each dataset. What explains the different degradation rates in Tables 3 and 10?

Comments on the paper

1 comment

The authors did not respond and address the concerns. I will keep original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.