MISeg-Agent: A Human-in-the-Loop Agentic Framework for Error Diagnosis and Executable Prompt Review in 3D Medical Image Segmentation

Foundation models can provide useful initial predictions for dense structured tasks, but their outputs often require correction before use in high-stakes domains. In 3D medical image segmentation, such correction is difficult because errors may be sparse, distributed across slices, and costly for users to locate manually. We propose MISeg-Agent, a human-in-the-loop agentic framework for reliable correction of 3D medical segmentation masks. Given a 3D volume and a free-form target request, MISeg-Agent selects a compatible segmentation foundation model to produce an initial rough mask, then assists human-guided refinement through an auditable loop of error diagnosis, prompt proposal, executable review, and user-controlled commitment. We first formulate and benchmark model-agnostic voxel-wise segmentation error diagnosis: given an image and an imperfect mask from a black-box segmentation model, the task is to predict which voxels are correct, false negative, or false positive. We construct the benchmark using rough masks generated by multiple foundation models and evaluate cross-source error diagnosis performance. An Error Diagnosis Agent trained on this benchmark estimates voxel-wise error beliefs from the current image-mask pair. A Prompt Policy Agent then converts these beliefs into positive and negative point or scribble recommendations on informative slices. For reviewed point candidates, an Expert Review Agent uses nnInteractive as a 3D executable probe to test whether the candidate produces a locally responsive, directionally supported, and foreground-preserving correction. The user may accept, reject, relabel, or redraw prompts, and only user-approved refinements are committed. MISeg-Agent reframes interactive segmentation from user-driven manual correction to agent-assisted, evidence-validated, and human-controlled correction. Experiments on 3D CT segmentation datasets show that MISeg-Agent improves user-guided correction, supports efficient interaction, and makes agent-generated prompts more reliable through executable expert review.