MedRefineAgent: Risk-Aware Multi-Turn Mask Refinement for Medical Image Segmentation

Medical image segmentation has advanced along two largely separate lines: prompt-free specialist models that automatically produce complete masks, and promptable foundation models that refine regions through points, boxes, or other user prompts. In practical clinical pipelines, however, the input is often neither an empty image nor a manually prompted target, but an imperfect coarse mask produced by an automatic segmenter. This setting calls for a unified mechanism that can inherit the efficiency of prompt-free segmentation while exploiting the local correction ability of promptable models. In this paper, we propose MedRefineAgent, a risk-aware multi-turn agent for automatic coarse-to-fine medical mask refinement across both 2D images and 3D volumes. Specifically, 1) a prompt-free backbone provides the initial coarse mask and an auxiliary reference for automatic risk localization; 2) a promptable local refiner is invoked only on high-risk regions through a structured action space of region, repair primitive, and hyper-parameter; and 3) conservative repair control coordinates gating, action pruning/reranking, and best-so-far rollback to suppress harmful edits. A teacher-distilled training pipeline further transfers search-based repair trajectories into lightweight test-time policies. Extensive experiments on abdominal, cardiac, endoscopic, skin-lesion, and fundus benchmarks show that MedRefineAgent consistently improves coarse predictions and achieves competitive or leading Dice performance across diverse 2D/3D medical segmentation settings.