- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies MLLM-based agents that refine a segmentation mask over several rounds by sending a positive click, a negative click or a stop signal to a frozen MedSAM, with a focus on ultrasound images. It argues that existing agents such as MedSAM-Agent and IBISAgent choose actions without an explicit estimate of the current error, and that trajectory-level RL never compares alternative actions at the same state. REVIVE adds two components. A small Residual-Error Verifier predicts false-negative and false-positive maps from MedSAM outputs, and these maps are turned into action-specific tokens for the MLLM. During training, COAR evaluates all three action types at each state visited by the policy, scores them against the ground truth, and applies a pairwise ranking loss to the action-type logits, on top of the GRPO objective of MedSAM-Agent. Against ten MLLM-based methods, REVIVE has the highest Dice on six of seven ultrasound datasets and on one of four polyp datasets. An ablation on BUS-BRA raises Dice from 0.766 (GRPO only) to 0.834 with both components. A further table reports that REVIVE recovers from 88% of its erroneous actions, against 25% and 41% for the two agent baselines.
Critical strengths
- The paper starts from a reasonable observation. The frozen segmentation tool can be queried cheaply during training, so the outcome of every action type can be computed at the states the policy actually visits and used as supervision at each step. The resulting ranking loss is simple and fits into the existing training recipe of MedSAM-Agent.
- The diagnosis module is designed so that it can be used at test time. The error maps are predicted from signals of the frozen tool, and each action gets its own evidence: false negatives outside the mask for a positive click, false positives inside the mask for a negative click, and both for stop.
- The evaluation covers seven ultrasound and four polyp datasets with ten MLLM-based baselines and four metrics. It also looks at what the agent does after a wrong action (Table 4), not only at the final Dice.
Critical weaknesses
- The comparison with other methods does not show that the gains come from the proposed components. The paper does not say whether any baseline was trained or fine-tuned on the data used for REVIVE, and for the seven ultrasound datasets no training set or test split is given. The tables suggest that the baselines were run as released: LISA++ and SAM4MLLM have recall close to 1 and precision about equal to IoU on most datasets, so they return almost the whole image. The authors' own GRPO-only model M1 reaches 0.77 Dice on BUS-BRA (Table 3), while MedSAM-Agent, whose GRPO training and reward design M1 retains, is listed at 0.54 (Table 1). About three quarters of the gap between MedSAM-Agent and REVIVE is therefore present before either component is added. The text also overstates the results: REVIVE is behind Citrus-V on Kvasir-SEG and CVC-ClinicDB and behind MedSAM-Agent on FH-PS-AoP and CVC-300, yet Citrus-V is said to be "still outperformed" (L413-415).
- The evidence for the two components and for the recovery claim is thin. The ablation uses one dataset (BUS-BRA) and reports no variance over runs, and the diagnosis module alone adds 0.014 Dice. The accuracy of the Residual-Error Verifier is never reported, although the diagnosis step depends on it and the paper itself stresses that errors in ultrasound are hard to identify. Simpler uses of the same signals are not tested, such as applying the rule of Eq. 7 to the predicted maps without the MLLM, or relabeling on-policy states with Eq. 7 in the style of DAgger (Ross et al., 2011, cited in the paper). The columns of Table 4 are not defined (what counts as an error or a recovery, how many cases, which datasets), and the table includes only external agents, not M1 to M3, so the higher recovery rate cannot be attributed to COAR. Its Error column reads 22.5% for REVIVE and 22.3% for IBISAgent, although the abstract claims reduced error-action misalignment.
- The description of the method is incomplete and partly inconsistent with the experiments. The MLLM backbone, the stopping threshold, the quality measure S, the verifier architecture and the maximum number of rounds are not given. The action space is defined as positive click, negative click and stop (L206-208), but Table 4 has a BBox column that reads 100% for REVIVE and a Point column that reads 29.1%; how the initial mask is obtained is never explained. In Eq. 13 the STOP score is a margin to a threshold while the click scores are one-step gains, so for a mask below the threshold a click that lowers Dice can still be ranked above STOP. Fig. 1 shows the agent comparing the outcomes of the alternative actions and selecting the best one, which needs the ground truth and can therefore happen only during training. It is unclear what the "verify" step consists of at test time.
Questions for authors
- Were the baselines fine-tuned on the same training data or run with the released checkpoints? IBISAgent, whose protocol is followed (L376), fine-tuned the general-purpose baselines on its own training data. Could MedSAM-Agent and IBISAgent be retrained on the same split, and MedSAM with the initial prompt alone be added?
- What text prompt and target did each baseline receive? On CAMUS, MedSAM-Agent and IBISAgent have precision above 0.9 but recall of about one third, and they score below LISA++ and SAM4MLLM, which segment almost the whole image.
- What are the training sets and test splits for the seven ultrasound datasets, which MLLM is used, and what are the stopping threshold, the measure S and the maximum number of rounds? Is the tool the box-prompted MedSAM of Ma et al. (2024) or MedSAM2, as in MedSAM-Agent and IBISAgent, and how are clicks passed to it?
- How accurate are the predicted FN/FP maps on test images, and what is the result when they are used directly (clicking at the largest predicted component) without the MLLM policy? In COAR, how are the click positions of the non-selected action types obtained (mentioned only in the conclusion, L474-475), and how often does the outcome-based ranking differ from the rule of Eq. 7?