- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper looks at how to place corrective clicks automatically for promptable segmentation models (SAM and its variants) on medical images. The user gives one click, and the proposed method, AlphaClick, then adds ten more clicks by itself, with no access to ground truth at test time. The authors treat click placement as a sequential decision problem in which the frozen mask decoder serves as an exact simulator. A small policy-value network is trained on trajectories generated with ground truth on the training images. At test time, an AlphaZero-style MCTS searches over about 55 candidate clicks, using the policy as a prior and the value network to score the leaves. With SAM ViT-B on 80 datasets from IMed-361M, the Dice after ten corrective clicks goes from 0.40 to 0.69, while the best baseline stays at 0.43. Dice also goes up with the number of simulations per click, from 0.650 with one simulation to 0.693 with sixteen. The paper also reports results with SAM2.1 and MedSAM2, and on 12 datasets held out from training.
Critical strengths
- The paper builds on an appealing observation. The decoder already gives the exact result of any click at very low cost, so what planning lacks is only a way to score a mask without ground truth. Tree search does not seem to have been used for click selection before.
- The evaluation is large (80 datasets, seven modalities, three backbones). The authors also report things that are often left out, such as runtime, oracle and ceiling rows that use ground truth, bootstrap intervals, and qualitative examples chosen by a fixed rule, including cases where their own method fails.
Critical weaknesses
-
The paper credits its gains to look-ahead search and test-time scaling, but its own numbers point elsewhere. In Table 4, the policy with no search at all already gets 0.637. That is roughly 80% of the total improvement over the initial click, and it is already more than 0.2 above every baseline. Search adds another 0.056, and the curve levels off after 8 to 16 simulations (Table 6 is flat from 16 to 64). The argument that scaling can continue rests on Fig. 3(c), where the leaf values are mixed with the true Dice, so it says little about what the actual method can do. There is also no experiment showing that planning several clicks ahead is better than looking one click ahead.
-
The comparison with baselines is not very informative. None of the baselines really improves over the initial click (0.40 versus 0.43 at best in Table 1), the best one is a simple entropy heuristic run on the authors' own candidate set, and PPO and PPD were designed for a reference-image setting and had to be adapted (Appendix C). AlphaClick, on the other hand, is trained with ground truth on the training splits of the same 80 datasets. The fair comparison would be against other ways of using that supervision: a small network that refines the mask directly, a fine-tuned decoder, a model that predicts where the error is and clicks there, or simply ranking all candidates one step ahead with the same value network. A medical interactive model such as IMIS-Net, which the paper cites, would also be a useful reference.
-
The evidence for generalization and for practical usefulness is not convincing. On the 12 held-out datasets, the margin over the best baseline is less than half of what it is in the main experiment (0.12 versus 0.27), and only the average is given. Several of these datasets also seem to share images with datasets that remain in training, for example MSD_Spleen and AbdomenCT1K, BTCV and the Learn2Reg abdominal sets, or ISIC 2017 and ISIC 2016/2018, and the paper does not say whether duplicates were removed. On the practical side, ten automatic clicks (0.694) still end up slightly below what two simulated user clicks achieve (0.700 in Table 1).
Questions for authors
- Could the authors add a depth-1 search, or a one-step look-ahead over all candidates with a matched number of decoder calls?
- How were the baselines tuned for this setting? Is it possible to add at least one baseline that uses the same training data, such as a mask refinement network?
- Could the authors report per-dataset results for the held-out experiment, ideally on datasets that do not share sources with the training set?
- When the decoder is called with the updated click set, is the previous mask also passed as a mask prompt? And how is SAM's multi-mask output handled for the first click? In Figs. 4 to 6 the initial mask often covers most of the image, which may affect the 0.40 starting point.
4 comments
Response to Reviewer (2/2)
Q3: Held-out results
“Could the authors report per-dataset results for the held-out experiment, ideally on datasets that do not share sources with the training set?”
A3: Some separately named datasets in IMed-361M share source images or patients. After excluding nine related training datasets and retraining on the remaining 59, macro Dice@10 rises from 0.6648 to 0.6979 on the same 1,600 targets. We checked the actual training samples against the evaluation targets, confirming the overlaps you raised for MSD Spleen, BTCV, and ISIC, as well as overlap for Kvasir and CHAOS. Drishti’s official cup/disc partitions coincide, with no training–test overlap among the 51 evaluated cup images. The 59-dataset retraining removes all overlaps identified in this check.
The full per-dataset results at 64 simulations per click are:
| Dataset | Targets | Initial click | Entropy | AlphaClick: 68 training sets | AlphaClick: 59 training sets | Change |
|---|---|---|---|---|---|---|
| ACDC | 176 | 0.5221 | 0.5210 | 0.6068 | 0.6599 | +0.0531 |
| BTCV | 200 | 0.4536 | 0.4853 | 0.6854 | 0.7017 | +0.0163 |
| CHAOS_Task_4_t1 | 87 | 0.5989 | 0.6387 | 0.7291 | 0.7768 | +0.0477 |
| MSD_Heart | 133 | 0.6247 | 0.6502 | 0.7646 | 0.7699 | +0.0053 |
| MSD_Spleen | 146 | 0.6507 | 0.6842 | 0.9066 | 0.9269 | +0.0203 |
| TDSC-ABUS2023 | 200 | 0.4275 | 0.4749 | 0.5658 | 0.6232 | +0.0574 |
| cvc_clinicdb | 123 | 0.4064 | 0.4722 | 0.7681 | 0.8102 | +0.0421 |
| drishti_gs_cup | 51 | 0.3886 | 0.4677 | 0.6287 | 0.6842 | +0.0555 |
| gamma | 20 | 0.4584 | 0.5020 | 0.5506 | 0.5557 | +0.0051 |
| isic2017_task1 | 150 | 0.5515 | 0.5449 | 0.6821 | 0.5687 | -0.1134 |
| kvasir_seg_aliyun | 200 | 0.4910 | 0.5147 | 0.7246 | 0.7750 | +0.0504 |
| sz_cxr | 114 | 0.4739 | 0.5536 | 0.3658 | 0.5224 | +0.1566 |
| Macro mean | 1,600 | 0.5039 | 0.5425 | 0.6648 | 0.6979 | ≈ +0.033 |
Both AlphaClick models use 64 simulations per click. Macro means give equal weight to datasets; the reported aggregate values use the evaluation outputs before rounding the per-dataset entries.
Eleven of the twelve datasets improve relative to the 68-dataset model; eleven also exceed Entropy, giving a macro advantage of 0.1554. Excluding ISIC 2016 and ISIC 2018 removes all dermoscopy data from training, so the ISIC 2017 row is now an unseen-modality evaluation. The full comparison supports strong held-out performance after excluding related training data, addressing the source-overlap concern directly.
The held-out restriction applies to AlphaClick’s training. AlignSAM and PPD retain their checkpoints trained on the training splits of all 80 datasets, including these 12 evaluation datasets; PPO uses an annotated training image from each evaluation dataset as its reference. Random, Entropy, and SimSAM are training-free, and their results are unchanged between the 68- and 59-dataset settings. These comparison conditions are explicit in Appendix C and the held-out table notes.
Section 4.4, Table 7, and Appendix K (Table 10) now give the revised setting, overlap findings, and all twelve results.
Q4: Mask prompts and multimask selection
“When the decoder is called with the updated click set, is the previous mask also passed as a mask prompt? And how is SAM's multi-mask output handled for the first click? In Figs. 4 to 6 the initial mask often covers most of the image, which may affect the 0.40 starting point.”
A4: Yes. After the initial click, AlphaClick passes the previous state’s low-resolution mask logits as mask_input, together with the accumulated click set. The previous probability map is also an input to the policy-value network. At the first click, the main protocol uses multimask_output=False. Section 3.1 now states these decoder settings explicitly. PPO and PPD construct a prompt set and decode it once, as described in Appendix C.
To test whether first-click multimask selection changes the comparison, we also selected the highest predicted-IoU mask from multimask_output=True, without using ground truth:
| First-click setting | Initial Dice | Entropy | AlphaClick, 16 simulations |
|---|---|---|---|
| Single output | 0.4008 | 0.4260 | 0.6929 |
| Multimask, highest predicted IoU | 0.4192 | 0.4385 | 0.6964 |
AlphaClick retains a 0.2579 Dice advantage over Entropy with multimask selection. This control directly checks the effect of the initial mask-selection rule on the comparison. Appendix L, Table 12 reports the results.
Thank you for the thorough and extensive rebuttal. You have addressed all of my concerns to my satisfaction. Accordingly, I will raise my recommendation from 2 to 3, as I now have considerably greater confidence in the paper's contribution and merits.
Thank you for your updated recommendation
Thank you for taking the time to go through our rebuttal and for raising your recommendation to 3! Your thoughtful questions pushed us to strengthen both our experimental controls and the presentation of the revised manuscript.
Response to Reviewer (1/2)
Thank you for the thoughtful questions on the search controls, baseline supervision, and evaluation protocol. We provide detailed responses below.
Q1: Depth-one control
A1: The learned policy provides most of the gain over the initial mask: policy-only clicking raises Dice from 0.4008 to 0.6366, accounting for 80.7% of the improvement to 16-simulation MCTS. Your depth-one control separates the additional effects of evaluating candidate masks and planning several clicks ahead. We added both matched-budget and exhaustive comparisons.
At 161 decoder calls per image, value-ranked one-step lookahead reaches 0.6685 and multi-step MCTS reaches 0.6929. The successive gains of 0.0319 and 0.0244 account for 10.9% and 8.4% of the total improvement. The one-step control evaluates the top 16 policy candidates at each corrective step; ranking by the current-Dice head gives 0.6612. Exhaustively evaluating all approximately 55 candidates reaches 0.6738 at approximately 551 calls, recovering 66.1% of the gain from policy-only to MCTS.
Multi-step planning adds 0.0244 at matched cost (95% paired bootstrap CI [0.0191, 0.0298]) and 0.0191 over exhaustive one-step lookahead (CI [0.0141, 0.0242]), with approximately 1/3.4 as many decoder calls as the exhaustive control. For the deployed model, the main search gains occur from 1 to 16 simulations (0.6504 → 0.6929), followed by a plateau (0.6937 at 64). Figure 3(c) uses oracle-assisted leaf values to investigate how evaluation accuracy affects this curve. Sections 4.3 and 4.5 and Appendix I, Table 8 now distinguish the observed scaling, the gain decomposition, and this diagnostic.
Q2: Baseline tuning and supervision
A2: Our evaluation spans 80 medical datasets across seven imaging modalities, with substantial variation in image appearance and segmentation targets. In this heterogeneous setting, the evaluated RL-based prompting methods achieve limited aggregate improvement over the initial click. AlignSAM and PPD use the released implementations and the benchmark’s official training splits. AlignSAM was evaluated over learning rates {1e-5, 5e-5, 1e-4} with validation-based checkpoint selection; PPD’s retrained agents were selected using 400 validation images from 40 datasets (Appendices C and J).
The prompting baselines test how to choose corrections for a frozen decoder. Your proposed mask-refinement baseline tests a different use of the same supervision, so we added two direct segmentation alternatives using the same training-image pool: a 3.38M-parameter mask-refinement U-Net, trained with BCE + Dice to refine the initial mask, and a fine-tuned SAM mask decoder, trained with the same losses and iterative simulated clicks. We also evaluated official IMIS-Net weights on the same targets:
The advantage persists against direct segmentation supervision and combines with decoder adaptation. Section 4.2 and Appendix J (Table 9) report the protocols and results.
Regarding the practical-usefulness concern in Weakness 3, AlphaClick reaches 0.6929 after the shared initial click without additional human-selected corrections, close to the 0.7002 obtained with two ground-truth-guided oracle corrections. Dice@2 counts those two corrections after the initial click; the oracle is a test-time ground-truth reference.