acceptodds
Withdrawn from ICLR 2027

MISeg-Agent: A Human-in-the-Loop Agentic Framework for Error Diagnosis and Executable Prompt Review in 3D Medical Image Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Foundation models can provide useful initial predictions for dense structured tasks, but their outputs often require correction before use in high-stakes domains. In 3D medical image segmentation, such correction is difficult because errors may be sparse, distributed across slices, and costly for users to locate manually. We propose MISeg-Agent, a human-in-the-loop agentic framework for reliable correction of 3D medical segmentation masks. Given a 3D volume and a free-form target request, MISeg-Agent selects a compatible segmentation foundation model to produce an initial rough mask, then assists human-guided refinement through an auditable loop of error diagnosis, prompt proposal, executable review, and user-controlled commitment. We first formulate and benchmark model-agnostic voxel-wise segmentation error diagnosis: given an image and an imperfect mask from a black-box segmentation model, the task is to predict which voxels are correct, false negative, or false positive. We construct the benchmark using rough masks generated by multiple foundation models and evaluate cross-source error diagnosis performance. An Error Diagnosis Agent trained on this benchmark estimates voxel-wise error beliefs from the current image-mask pair. A Prompt Policy Agent then converts these beliefs into positive and negative point or scribble recommendations on informative slices. For reviewed point candidates, an Expert Review Agent uses nnInteractive as a 3D executable probe to test whether the candidate produces a locally responsive, directionally supported, and foreground-preserving correction. The user may accept, reject, relabel, or redraw prompts, and only user-approved refinements are committed. MISeg-Agent reframes interactive segmentation from user-driven manual correction to agent-assisted, evidence-validated, and human-controlled correction. Experiments on 3D CT segmentation datasets show that MISeg-Agent improves user-guided correction, supports efficient interaction, and makes agent-generated prompts more reliable through executable expert review.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Interactive Medical Image Segmentation, Multi-Agent Collaboration, Error-Guided Refinement, Human-in-the-Loop

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YimGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes MISeg-Agent, a human-in-the-loop framework for 3D medical image segmentation that selects pretrained models based on benchmark performance and computational requirements. It uses a trained error-diagnosis network to identify segmentation errors, proposes corrective point prompts, and tests them using nnInteractive before human review and approval. The framework is evaluated on seven CT datasets involving 120 cases and 360 human-review sessions, reporting segmentation accuracy, error-detection performance, and human interaction efficiency.

Critical strengths

The framework is evaluated on seven public CT datasets covering different anatomical structures, with 120 validation cases and 360 human-review sessions involving three participants.

The authors also investigate error-diagnosis performance across masks generated by different pretrained segmentation models.

Critical weaknesses

1. Unavailable Source Code: The paper claims that its source code is publicly available. However, as of October 16, 2026, the provided GitHub repository contains only a readme.txt file, with no implementation available. This prevents reviewers from inspecting the framework, verifying implementation details, or reproducing the reported results.

2. Limited Methodological Novelty: The proposed framework primarily integrates existing pretrained segmentation models, a conventional 3D U-Net for error detection, heuristic point selection, and nnInteractive for segmentation refinement. While the integration may offer practical utility, the paper does not establish a sufficiently distinctive methodological contribution beyond combining existing components into an agentic workflow.

3. Unfair Segmentation Performance Comparisons: Table 1 compares MISeg-Agent, which incorporates human-guided corrections, against pretrained segmentation models operating without human intervention. The reported improvement from 0.915 to 0.943 Dice therefore does not establish the superiority of the proposed framework. Furthermore, the interactive segmentation baselines receive unfiltered prompts without initial segmentation masks, making these comparisons difficult to interpret fairly.

4. Absence of Controlled Human Evaluation: Although the authors conduct 360 participant–case sessions involving three participants, they do not provide a controlled comparison against conventional human-assisted segmentation workflows. A meaningful evaluation would compare users correcting the same initial masks with and without the proposed agentic assistance. Without such a comparison, it remains unclear whether the framework actually improves annotation accuracy or reduces human effort.

5. Insufficient Evidence for Annotation Efficiency: The authors report an average human review time of approximately 35 seconds per case and compare this with annotation times from previous studies. However, these comparisons involve different datasets, annotation procedures, and measurement protocols. Furthermore, the reported time measures recorded human review activity rather than the complete workflow. Consequently, the claimed efficiency benefits are not convincingly established.

6. Lack of Component-Level Ablation Studies: The framework contains several components, including model routing, error diagnosis, prompt proposal, executable review, and human-guided refinement. However, the paper does not provide systematic ablations to establish how much each component contributes to the final segmentation performance or annotation efficiency. In particular, it remains unclear whether simpler approaches, such as direct interactive correction of pretrained masks without the proposed agentic coordination, could achieve comparable results.

Questions for authors

Please refer to the Critical Weaknesses section.

Reviewer_wqxROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MISeg-Agent, a human-in-the-loop framework for reviewing and correcting 3d medical image segmentation. it selects a pretrained model, predicts likely segmentation errors, and propose where needs to be corrected.

The framework is evaluated on 120 cases from seven CT datasets with three participants, reporting final segmentation accuracy and recorded human review activity. I feel the entire idea is interesting and can be a useful tool, but considering the lack of novelty, lack of experiments proving the effectiveness/generalization, this is below the ICLR acceptance bar.

Critical strengths
  1. This paper addresses a practical problem: locating segmentation errors and deciding how to correct them can save a lot of human effort.
  2. testing candidate clicks before providing to human is useful.
  3. error diagnosis uses only the image and mask, allowing the integration with different pretrained models without requiring internal features, can generalize to more frameworks.
Critical weaknesses
  1. The main comparisons do not isolate the framework's benefit. MISeg-agent receives initial masks and human review, while the other point based baselines receive nothing. It seems not be a fair comparison.
  2. all reported results use 120 case validation cohort. I feel it seems the model selecting, designing, and thresholding datasets are the same as final reporting datasets. It is highly possible current decision logit already tuned on these datasets. I think this workflow needs more external dataset test. Like how this workflow works on tasks it didn't see before.
  3. The comparison with other studies use different protocols, i feel the current efficiency analysis doesn't under a fair comparison now.
  4. The diagnosis model is trained on initial segmentation errors, it is unclear about how its accuracy on the human corrected/edited masks.
Questions for authors
  1. how the routine rules, diagnosis model selection/tuning and screening threshold are selected/tuned based on the same dataset for reporting final results?
Reviewer_A4CbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents MISeg-Agent, a human-in-the-loop framework for 3D medical image segmentation. The system combines pretrained model selection, voxel-level error diagnosis, corrective point proposal, executable prompt review using nnInteractive, and human approval of mask updates. The motivation is practical: pretrained models can provide good initial segmentations, but residual errors still need to be located and corrected, and interactive segmentation models do not by themselves tell users where or how to intervene. The paper evaluates the framework on seven CT datasets. The final human-guided workflow achieves an overall Dice of 0.943, and the authors additionally evaluate error diagnosis and report interaction statistics from three participants. Overall, I think the problem is useful and the system is reasonably well designed. However, I have significant concerns about the experimental setup, particularly the lack of matched comparisons and component-level ablations. These make it difficult to determine how much of the final improvement actually comes from the proposed agentic framework.

Critical strengths

The paper addresses an important issue in medical image segmentation: correcting residual errors from pretrained models without requiring users to manually inspect an entire 3D volume.

The combination of model routing, error diagnosis, point proposal, executable prompt testing, and human approval forms a coherent workflow. I particularly like the idea of testing a candidate interaction before recommending it to the user rather than directly modifying the segmentation.

Critical weaknesses

The main comparison is not sufficiently controlled. This is my primary concern. MISeg-Agent uses a strong pretrained initial mask, error diagnosis, screened prompts, human editing, and human approval. In contrast, the ScribblePrompt and nnInteractive baselines in Table 1 receive unfiltered generated points without rough-mask initialization or human editing. Therefore, the reported 0.943 Dice cannot be directly attributed to the proposed agentic framework. The paper itself notes that these workflows differ in initialization and human input.

The interactive segmentation baselines appear much weaker under the main evaluation protocol than under a more realistic interaction protocol. In Table 1, nnInteractive obtains only 0.369 Dice with unfiltered generated points. However, under the separate participant-supplied point protocol, it reaches 0.912 with five points and 0.928 with ten points. This large difference suggests that the prompt protocol, rather than the proposed framework alone, explains a substantial portion of the apparent improvement.

There is no convincing component ablation for the complete system. I would like to see matched experiments starting from the same initial masks, for example: initial mask only; + human correction; + error diagnosis; + proposed points; + executable review; and the complete MISeg-Agent. Without such experiments, it is difficult to understand which components are actually necessary.

The automatic correction results are not particularly convincing. Figure 1(b) shows an initial Dice of 0.927, while applying all prompts reduces it to 0.701 and screened prompts obtain 0.892. Screening is clearly better than using unscreened prompts, but it still performs worse than simply retaining the initial segmentation. This suggests that human decisions play a crucial role in obtaining the final reported performance.

The human study is too small to strongly support the efficiency claims. Only three participants are included. More importantly, there is no matched human-only or standard interactive-segmentation control using the same initial masks, cases, interface, and completion criteria. The reported average review/editing time of 35.18 seconds is interesting, but comparisons with annotation times from previous studies are not controlled because the tasks and timing protocols differ.

The methodological novelty is somewhat limited without stronger empirical evidence for the orchestration itself. Many individual components—pretrained model selection, 3D U-Net-based error prediction, point proposal, nnInteractive refinement, and rule-based screening—are relatively standard. The main novelty appears to be their integration into an agentic workflow. This can still be valuable, but it makes controlled experiments demonstrating the benefit of this particular orchestration especially important.

Generalization is limited. The current experiments are restricted to CT. Moreover, “model-agnostic” means that the diagnosis model does not receive the source model identity or internal features, rather than demonstrating generalization to arbitrary unseen segmentation models.Evaluation on unseen segmentation models, external cohorts, and additional modalities such as MRI would strengthen this claim.

Questions for authors

Can the authors provide a controlled ablation where all methods start from exactly the same pretrained initial masks and use the same human interaction budget? In particular, I would like to see initial mask + human/nnInteractive correction compared directly against the complete MISeg-Agent.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.