acceptodds
Under review as a conference paper at ICLR 2027

AlphaClick: Scaling Interactive Medical Image Segmentation at Test-time via Monte Carlo Tree Search

Abstract

Powered by large-scale training data, promptable segmentation models such as the Segment Anything Model (SAM) have shown impressive generalization in medical image segmentation, where a few corrective clicks can turn a coarse mask into an accurate one. However, these clicks still have to be provided by experts, which is costly and limits the scalability of interactive segmentation in clinical practice. Existing automatic prompting methods, built on heuristics or reinforcement learning (RL) policies, decide clicks without looking ahead and cannot turn additional test-time computation into better segmentation. In this paper, we propose AlphaClick, a novel RL framework that scales interactive medical image segmentation at test time through Monte Carlo tree search (MCTS). Specifically, AlphaClick formulates corrective clicking as a sequential decision process and trains a policy-value network with the Dice improvement of each click as the reward. The policy proposes promising clicks, and the value predicts the final segmentation quality reachable from the current state. At inference, MCTS uses the policy to expand candidate clicks and the value to evaluate simulated click sequences, so that more simulations yield better click decisions without any ground truth. Extensive experiments on 80 public datasets covering seven imaging modalities show that AlphaClick outperforms existing automatic prompting methods by a large margin, reaching a Dice of 0.69 after ten corrective clicks compared with 0.43 for the strongest baseline, and that its accuracy improves steadily as the search grows from one to sixteen simulations per click. Results on three segmentation backbones and twelve unseen datasets further confirm its generality. In conclusion, AlphaClick makes test-time computation a new dimension for improving interactive segmentation and provides a promising solution for reducing the annotation burden in medical imaging. The code will be made publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
interactive medical image segmentation, Monte Carlo tree search, reinforcement learning, test-time scaling, Segment Anything Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hSDVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at how to place corrective clicks automatically for promptable segmentation models (SAM and its variants) on medical images. The user gives one click, and the proposed method, AlphaClick, then adds ten more clicks by itself, with no access to ground truth at test time. The authors treat click placement as a sequential decision problem in which the frozen mask decoder serves as an exact simulator. A small policy-value network is trained on trajectories generated with ground truth on the training images. At test time, an AlphaZero-style MCTS searches over about 55 candidate clicks, using the policy as a prior and the value network to score the leaves. With SAM ViT-B on 80 datasets from IMed-361M, the Dice after ten corrective clicks goes from 0.40 to 0.69, while the best baseline stays at 0.43. Dice also goes up with the number of simulations per click, from 0.650 with one simulation to 0.693 with sixteen. The paper also reports results with SAM2.1 and MedSAM2, and on 12 datasets held out from training.

Critical strengths
  1. The paper builds on an appealing observation. The decoder already gives the exact result of any click at very low cost, so what planning lacks is only a way to score a mask without ground truth. Tree search does not seem to have been used for click selection before.
  2. The evaluation is large (80 datasets, seven modalities, three backbones). The authors also report things that are often left out, such as runtime, oracle and ceiling rows that use ground truth, bootstrap intervals, and qualitative examples chosen by a fixed rule, including cases where their own method fails.
Critical weaknesses
  1. The paper credits its gains to look-ahead search and test-time scaling, but its own numbers point elsewhere. In Table 4, the policy with no search at all already gets 0.637. That is roughly 80% of the total improvement over the initial click, and it is already more than 0.2 above every baseline. Search adds another 0.056, and the curve levels off after 8 to 16 simulations (Table 6 is flat from 16 to 64). The argument that scaling can continue rests on Fig. 3(c), where the leaf values are mixed with the true Dice, so it says little about what the actual method can do. There is also no experiment showing that planning several clicks ahead is better than looking one click ahead.

  2. The comparison with baselines is not very informative. None of the baselines really improves over the initial click (0.40 versus 0.43 at best in Table 1), the best one is a simple entropy heuristic run on the authors' own candidate set, and PPO and PPD were designed for a reference-image setting and had to be adapted (Appendix C). AlphaClick, on the other hand, is trained with ground truth on the training splits of the same 80 datasets. The fair comparison would be against other ways of using that supervision: a small network that refines the mask directly, a fine-tuned decoder, a model that predicts where the error is and clicks there, or simply ranking all candidates one step ahead with the same value network. A medical interactive model such as IMIS-Net, which the paper cites, would also be a useful reference.

  3. The evidence for generalization and for practical usefulness is not convincing. On the 12 held-out datasets, the margin over the best baseline is less than half of what it is in the main experiment (0.12 versus 0.27), and only the average is given. Several of these datasets also seem to share images with datasets that remain in training, for example MSD_Spleen and AbdomenCT1K, BTCV and the Learn2Reg abdominal sets, or ISIC 2017 and ISIC 2016/2018, and the paper does not say whether duplicates were removed. On the practical side, ten automatic clicks (0.694) still end up slightly below what two simulated user clicks achieve (0.700 in Table 1).

Questions for authors
  1. Could the authors add a depth-1 search, or a one-step look-ahead over all candidates with a matched number of decoder calls?
  2. How were the baselines tuned for this setting? Is it possible to add at least one baseline that uses the same training data, such as a mask refinement network?
  3. Could the authors report per-dataset results for the held-out experiment, ideally on datasets that do not share sources with the training set?
  4. When the decoder is called with the updated click set, is the previous mask also passed as a mask prompt? And how is SAM's multi-mask output handled for the first click? In Figs. 4 to 6 the initial mask often covers most of the image, which may affect the 0.40 starting point.
4 comments
AuthorsOpenReview ↗

Response to Reviewer (1/2)

Thank you for the thoughtful questions on the search controls, baseline supervision, and evaluation protocol. We provide detailed responses below.

Q1: Depth-one control

“Could the authors add a depth-1 search, or a one-step look-ahead over all candidates with a matched number of decoder calls?”

A1: The learned policy provides most of the gain over the initial mask: policy-only clicking raises Dice from 0.4008 to 0.6366, accounting for 80.7% of the improvement to 16-simulation MCTS. Your depth-one control separates the additional effects of evaluating candidate masks and planning several clicks ahead. We added both matched-budget and exhaustive comparisons.

At 161 decoder calls per image, value-ranked one-step lookahead reaches 0.6685 and multi-step MCTS reaches 0.6929. The successive gains of 0.0319 and 0.0244 account for 10.9% and 8.4% of the total improvement. The one-step control evaluates the top 16 policy candidates at each corrective step; ranking by the current-Dice head gives 0.6612. Exhaustively evaluating all approximately 55 candidates reaches 0.6738 at approximately 551 calls, recovering 66.1% of the gain from policy-only to MCTS.

Multi-step planning adds 0.0244 at matched cost (95% paired bootstrap CI [0.0191, 0.0298]) and 0.0191 over exhaustive one-step lookahead (CI [0.0141, 0.0242]), with approximately 1/3.4 as many decoder calls as the exhaustive control. For the deployed model, the main search gains occur from 1 to 16 simulations (0.6504 → 0.6929), followed by a plateau (0.6937 at 64). Figure 3(c) uses oracle-assisted leaf values to investigate how evaluation accuracy affects this curve. Sections 4.3 and 4.5 and Appendix I, Table 8 now distinguish the observed scaling, the gain decomposition, and this diagnostic.

Q2: Baseline tuning and supervision

“How were the baselines tuned for this setting? Is it possible to add at least one baseline that uses the same training data, such as a mask refinement network?”

A2: Our evaluation spans 80 medical datasets across seven imaging modalities, with substantial variation in image appearance and segmentation targets. In this heterogeneous setting, the evaluated RL-based prompting methods achieve limited aggregate improvement over the initial click. AlignSAM and PPD use the released implementations and the benchmark’s official training splits. AlignSAM was evaluated over learning rates {1e-5, 5e-5, 1e-4} with validation-based checkpoint selection; PPD’s retrained agents were selected using 400 validation images from 40 datasets (Appendices C and J).

The prompting baselines test how to choose corrections for a frozen decoder. Your proposed mask-refinement baseline tests a different use of the same supervision, so we added two direct segmentation alternatives using the same training-image pool: a 3.38M-parameter mask-refinement U-Net, trained with BCE + Dice to refine the initial mask, and a fine-tuned SAM mask decoder, trained with the same losses and iterative simulated clicks. We also evaluated official IMIS-Net weights on the same targets:

ComparisonDice
Direct mask refinement0.5694
Fine-tuned SAM, shared initial click0.5842
Fine-tuned SAM + 10 Entropy corrections0.6015
Official IMIS-Net, shared initial click0.6183
AlphaClick, 16 simulations, frozen SAM0.6929
AlphaClick on fine-tuned SAM0.7214

The advantage persists against direct segmentation supervision and combines with decoder adaptation. Section 4.2 and Appendix J (Table 9) report the protocols and results.

Regarding the practical-usefulness concern in Weakness 3, AlphaClick reaches 0.6929 after the shared initial click without additional human-selected corrections, close to the 0.7002 obtained with two ground-truth-guided oracle corrections. Dice@2 counts those two corrections after the initial click; the oracle is a test-time ground-truth reference.

AuthorsOpenReview ↗

Response to Reviewer (2/2)

Q3: Held-out results

“Could the authors report per-dataset results for the held-out experiment, ideally on datasets that do not share sources with the training set?”

A3: Some separately named datasets in IMed-361M share source images or patients. After excluding nine related training datasets and retraining on the remaining 59, macro Dice@10 rises from 0.6648 to 0.6979 on the same 1,600 targets. We checked the actual training samples against the evaluation targets, confirming the overlaps you raised for MSD Spleen, BTCV, and ISIC, as well as overlap for Kvasir and CHAOS. Drishti’s official cup/disc partitions coincide, with no training–test overlap among the 51 evaluated cup images. The 59-dataset retraining removes all overlaps identified in this check.

The full per-dataset results at 64 simulations per click are:

DatasetTargetsInitial clickEntropyAlphaClick: 68 training setsAlphaClick: 59 training setsChange
ACDC1760.52210.52100.60680.6599+0.0531
BTCV2000.45360.48530.68540.7017+0.0163
CHAOS_Task_4_t1870.59890.63870.72910.7768+0.0477
MSD_Heart1330.62470.65020.76460.7699+0.0053
MSD_Spleen1460.65070.68420.90660.9269+0.0203
TDSC-ABUS20232000.42750.47490.56580.6232+0.0574
cvc_clinicdb1230.40640.47220.76810.8102+0.0421
drishti_gs_cup510.38860.46770.62870.6842+0.0555
gamma200.45840.50200.55060.5557+0.0051
isic2017_task11500.55150.54490.68210.5687-0.1134
kvasir_seg_aliyun2000.49100.51470.72460.7750+0.0504
sz_cxr1140.47390.55360.36580.5224+0.1566
Macro mean1,6000.50390.54250.66480.6979≈ +0.033

Both AlphaClick models use 64 simulations per click. Macro means give equal weight to datasets; the reported aggregate values use the evaluation outputs before rounding the per-dataset entries.

Eleven of the twelve datasets improve relative to the 68-dataset model; eleven also exceed Entropy, giving a macro advantage of 0.1554. Excluding ISIC 2016 and ISIC 2018 removes all dermoscopy data from training, so the ISIC 2017 row is now an unseen-modality evaluation. The full comparison supports strong held-out performance after excluding related training data, addressing the source-overlap concern directly.

The held-out restriction applies to AlphaClick’s training. AlignSAM and PPD retain their checkpoints trained on the training splits of all 80 datasets, including these 12 evaluation datasets; PPO uses an annotated training image from each evaluation dataset as its reference. Random, Entropy, and SimSAM are training-free, and their results are unchanged between the 68- and 59-dataset settings. These comparison conditions are explicit in Appendix C and the held-out table notes.

Section 4.4, Table 7, and Appendix K (Table 10) now give the revised setting, overlap findings, and all twelve results.

Q4: Mask prompts and multimask selection

“When the decoder is called with the updated click set, is the previous mask also passed as a mask prompt? And how is SAM's multi-mask output handled for the first click? In Figs. 4 to 6 the initial mask often covers most of the image, which may affect the 0.40 starting point.”

A4: Yes. After the initial click, AlphaClick passes the previous state’s low-resolution mask logits as mask_input, together with the accumulated click set. The previous probability map is also an input to the policy-value network. At the first click, the main protocol uses multimask_output=False. Section 3.1 now states these decoder settings explicitly. PPO and PPD construct a prompt set and decode it once, as described in Appendix C.

To test whether first-click multimask selection changes the comparison, we also selected the highest predicted-IoU mask from multimask_output=True, without using ground truth:

First-click settingInitial DiceEntropyAlphaClick, 16 simulations
Single output0.40080.42600.6929
Multimask, highest predicted IoU0.41920.43850.6964

AlphaClick retains a 0.2579 Dice advantage over Entropy with multimask selection. This control directly checks the effect of the initial mask-selection rule on the comparison. Appendix L, Table 12 reports the results.

Thank you for the thorough and extensive rebuttal. You have addressed all of my concerns to my satisfaction. Accordingly, I will raise my recommendation from 2 to 3, as I now have considerably greater confidence in the paper's contribution and merits.

AuthorsOpenReview ↗

Thank you for your updated recommendation

Thank you for taking the time to go through our rebuttal and for raising your recommendation to 3! Your thoughtful questions pushed us to strengthen both our experimental controls and the presentation of the revised manuscript.

Reviewer_iccUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents AlphaClick, a framework for automatically selecting corrective point prompts for interactive medical image segmentation. The method formulates corrective clicking as a sequential decision problem and trains a policy-value network using ground-truth-guided trajectories. At inference time, Monte Carlo tree search (MCTS), with Gumbel-based root action selection, uses the policy to propose candidate clicks and the value function to evaluate multi-step click sequences without accessing the test-time ground-truth mask.

The method is evaluated on 80 datasets from IMed-361M across seven imaging modalities, using SAM ViT-B, SAM2.1, and MedSAM2 as frozen segmentation backbones. AlphaClick substantially outperforms several heuristic and reinforcement-learning prompting baselines. The experiments also show that increasing the number of MCTS simulations improves performance, particularly from one to sixteen simulations per click, and that the method generalizes to twelve datasets excluded from training.

Critical strengths
  1. Broad empirical evaluation with consistently strong quantitative results. The paper evaluates the method on 80 datasets spanning seven medical imaging modalities and three segmentation backbones. AlphaClick substantially outperforms the included automatic prompting baselines, e.g., achieving a Dice score of approximately 0.69 after ten corrective clicks compared with approximately 0.43 for the strongest baseline on SAM ViT-B. The evaluation on twelve held-out datasets also provides useful evidence of cross-dataset generalization. This breadth is a meaningful strength because it reduces the likelihood that the reported gains are specific to a small number of datasets or one segmentation backbone.

  2. A clear and technically coherent formulation of corrective clicking as planning. The paper correctly observes that the utility of a click can depend on subsequent clicks and therefore treats click selection as a multi-step planning problem rather than an independent one-step decision. The integration of a compact candidate set, a policy-value network, value-based tree search, and tree reuse is technically coherent. The ablation comparing the full method with the policy-only variant provides evidence that explicit search contributes beyond the learned prompting policy.

Critical weaknesses
  1. Limited methodological novelty. The core method directly combines a U-Net-based policy-value network with Gumbel AlphaZero search, including PUCT selection, sequential halving, value backup, and tree reuse. The network is trained using ground-truth-derived policy, value, and ranking targets. These are established components, and the main novelty appears to be their application to corrective point prompting rather than a new learning or planning technique.
  2. Limited evidence for sustained test-time scaling. Most gains are already obtained with one simulation, and performance nearly saturates after sixteen simulations despite substantially increased inference cost. This weakens the central claim that additional test-time computation consistently translates into better segmentation.
  3. Practical relevance and evaluation limitations. The initial click is derived from the ground-truth mask, which does not reflect a fully automatic deployment setting. Moreover, several baselines perform close to or below the initial-click result, raising concerns about whether the comparisons sufficiently demonstrate advantages over well-optimized alternatives.
Questions for authors
  1. How do the authors distinguish the proposed training procedure from supervised imitation and value learning? Which component specifically requires the method to be characterized as reinforcement learning rather than supervised learning followed by MCTS?

  2. Could the authors report results for the policy-only model and MCTS under matched test-time decoder-call budgets? This would help separate the contribution of the learned policy, candidate simulation, and multi-step tree search.

  3. How sensitive is AlphaClick to the initial click? In particular, what happens when the initial click is sampled from a realistic user-click distribution, perturbed away from the ground-truth component center, or generated by an automatic detector?

  4. Why does performance nearly saturate after sixteen simulations and after approximately four corrective clicks? Does this result from value-function errors, limitations of the candidate set, or the segmentation backbone becoming insensitive to additional prompts?

  5. Several baselines remain close to or below the initial-click result, whereas AlphaClick improves substantially. Could the authors provide stronger evidence that the baseline implementations and retrained checkpoints are appropriately optimized for this multi-dataset medical setting, particularly for AlignSAM and PPD?

4 comments
AuthorsOpenReview ↗

Response to Reviewer (1/2)

We thank the reviewer for the constructive review. We provide detailed responses below.

Q1: Learning and novelty

“How do the authors distinguish the proposed training procedure from supervised imitation and value learning? Which component specifically requires the method to be characterized as reinforcement learning rather than supervised learning followed by MCTS?”

A1: The learning stage fits policy and value estimates from ground-truth-guided offline trajectories using supervised objectives; test-time MCTS uses those estimates to plan click sequences. The MDP formulation specifies the action, exact decoder transition, Dice-improvement reward, and return. In Equation 4, the policy target is a softmax of candidate Dice gains. The value target is terminal Dice along a trajectory, equal to the trajectory return plus current Dice; the sibling-ranking loss trains the ordering of alternative continuations. This separates the learning procedure from the planning algorithm explicitly.

The training data contain a greedy trajectory and exploratory and adversarial continuations, providing both favorable and unfavorable states for value learning. Training on greedy trajectories alone reduces the same 16-simulation MCTS from 0.6929 to 0.6691. This ablation supports the role of trajectory diversity in learning estimates useful to search (Appendix I).

Our contribution is making value-guided planning effective for corrective clicking: a ground-truth-free candidate space with positive and negative clicks, an exact and inexpensive frozen-decoder simulator, a history-conditioned policy-value network, and sibling-ranking supervision for closely related states. Gumbel selection and tree reuse allocate small search budgets efficiently. The matched-budget comparison isolates the resulting planning benefit: 0.6929 versus 0.6685 at 161 calls per image. Section 3 and the revised introduction explain this contribution through its actual training objectives and test-time operation.

Q2: Matched decoder-call budgets

“Could the authors report results for the policy-only model and MCTS under matched test-time decoder-call budgets? This would help separate the contribution of the learned policy, candidate simulation, and multi-step tree search.”

A2: The policy supplies most of the gain over the initial mask: directly committing its highest-prior clicks raises Dice from 0.4008 to 0.6366 using 11 decoder calls per image. To separate candidate evaluation from multi-step planning, we allocate the same 161-call budget to one-step simulation of the top 16 policy candidates at each corrective step:

Allocation of test-time computationCalls/imageDice@10
Policy only110.6366
Top-16 one-step, current-Dice scoring1610.6612
Top-16 one-step, value scoring1610.6685
16-simulation multi-step MCTS1610.6929
Exhaustive one-step, value scoring≈5510.6738

The successive improvements from the initial mask are 0.2358 from policy-only clicking, 0.0319 from matched-budget one-step evaluation, and 0.0244 from multi-step planning, or 80.7%, 10.9%, and 8.4% of the total gain. Exhaustive one-step evaluation recovers 66.1% of the gain from policy-only to MCTS; MCTS adds 0.0191 with approximately 1/3.4 as many calls. Thus, the learned policy establishes a strong starting point, and planning improves the accuracy obtained from the test-time computation. Section 4.5 and Appendix I, Table 8 report this decomposition and the controlled comparisons.

AuthorsOpenReview ↗

Response to Reviewer (2/2)

Q3: Initial-click sensitivity

“How sensitive is AlphaClick to the initial click? In particular, what happens when the initial click is sampled from a realistic user-click distribution, perturbed away from the ground-truth component center, or generated by an automatic detector?”

A3: The original center click provides a shared target specification for every method. We evaluated the trained model without retraining under three alternative initializations: a center-biased Gaussian truncated inside the target (σ=0.25R\sigma = 0.25R), near-boundary locations at 0.65–0.85R0.65\text{--}0.85R, and a simulated noisy detector-box prompt obtained by perturbing the target-box center by up to 20% of box width and height. Here RR is the inscribed radius used by the sampling protocol.

Initial-click settingInitial DiceEntropyPPDAlphaClick, 16 simulations
Original component center0.40080.42600.42050.6929
Gaussian user-click simulation0.36420.39800.39150.6845
Near-boundary click0.29150.33410.32800.6672
Simulated noisy detector-box prompt0.31800.35120.34900.6588

The gain persists across all three initial-click conditions. For example, near-boundary initialization lowers the initial Dice to 0.2915, while AlphaClick reaches 0.6672 after automatic corrections. Appendix L, Table 12 specifies the perturbations and reports the comparisons.

Q4: Saturation

“Why does performance nearly saturate after sixteen simulations and after approximately four corrective clicks? Does this result from value-function errors, limitations of the candidate set, or the segmentation backbone becoming insensitive to additional prompts?”

A4: The deployed model gains mainly from 1 to 16 simulations and then plateaus: Dice rises from 0.6504 to 0.6929, reaching 0.6937 at 64 simulations. The click curve similarly flattens after approximately four corrections. The diagnostics identify value-ranking precision as an important factor in this behavior. On sibling states, the learned value orders pairs with Dice differences of at least 0.05 correctly 84.6% of the time, versus 57.2% for differences between 0.005 and 0.02. Thus, it distinguishes large improvements more reliably than small refinements. Figure 3(c) tests the effect of more accurate leaf evaluation directly: mixing in true Dice enables further gains with larger search budgets. This is an oracle-assisted diagnostic of search capacity.

The click-axis references show that further improvement remains possible with the same frozen decoder. From four to ten corrective clicks, the candidate ceiling rises from 0.8154 to 0.8306, and unrestricted oracle clicks rise from 0.8032 to 0.8833. The decoder remains responsive, and the current candidate set still contains useful corrections. Together with the ranking measurements, these results explain why accurate evaluation of small differences matters as search progresses.

At 16 simulations, MCTS reaches 0.6929 with 161 decoder calls, compared with 0.6685 for one-step lookahead at the same cost. This comparison establishes the benefit of multi-step planning within the observed scaling range. Section 4.3 and Appendix I now state the plateau explicitly and connect it to the value-ranking diagnostic.

Q5: Baseline optimization

“Several baselines remain close to or below the initial-click result, whereas AlphaClick improves substantially. Could the authors provide stronger evidence that the baseline implementations and retrained checkpoints are appropriately optimized for this multi-dataset medical setting, particularly for AlignSAM and PPD?”

A5: Our evaluation spans 80 medical datasets across seven imaging modalities, with substantial variation in image appearance and segmentation targets. In this heterogeneous setting, the evaluated RL-based prompting methods achieve limited aggregate improvement over the initial click. We therefore examined their training and model-selection settings in detail.

We used the released implementations with SAM ViT-B and trained AlignSAM and PPD on the official training splits used by our benchmark. AlignSAM’s learning-rate sweep covers {1e-5, 5e-5, 1e-4}, with validation-based checkpoint selection. PPD’s attack and defense agents were retrained, and checkpoint selection used 400 validation images from 40 datasets. Appendix C and Appendix J report the implementation and model-selection settings.

To test whether AlphaClick’s advantage also holds against direct uses of the same segmentation supervision, we added fine-tuned SAM (0.6015 with Entropy corrections) and direct mask refinement (0.5694), plus the official IMIS-Net reference (0.6183 with the shared initial click). AlphaClick reaches 0.6929 on frozen SAM and 0.7214 when combined with the fine-tuned decoder. Table 9 reports these additional comparisons.

Thank you for the responses. The new ablations and additional baselines address my main concerns, and the residual concerns are acceptable given the strong experimental evidence. Considering the other reviewers' opinions as well, I have raised my score to 3.

AuthorsOpenReview ↗

Thank you for your feedback and updated score

Many thanks for following up on our responses and updating your score! We are pleased that the new ablations and baselines addressed your main concerns, and your questions helped make the revised paper stronger.

Reviewer_Zmr5OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes AlphaClick, which automatically places corrective clicks for promptable medical image segmentation after a single initial click, without ground truth or a human at test time. It frames corrective clicking as a sequential decision process and trains a policy-value network on ground-truth-guided trajectories. At test time, it plans each click with Monte Carlo tree search, using the frozen mask decoder as the transition model. On 80 IMed-361M datasets with SAM ViT-B, Dice after ten corrective clicks rises from 0.40 to 0.69, against 0.43 for the strongest baseline. The authors report further gains on two additional backbones and on held-out datasets, and conclude that lookahead search makes test-time computation a new axis for improving interactive segmentation.

[POST-REBUTTAL] As outlined in their responses and my comment below, the authors did a good job addressing my major concerns, especially my questions about the one-step control experiments and potential held-out data/split overlap issues (and how their performance might be affected when such overlaps are no longer present). As such, I raise my score from 2 to 3.

Critical strengths
  1. A well-designed and carefully specified search framework: The decoder provides exact, inexpensive transitions (1.46 ms per batched call on SAM ViT-B), which makes value-guided search well suited to this problem. The design choices are well matched to it:
    • a value target defined as the final reachable Dice;
    • a ranking loss that trains the ordering of sibling states that search depends on;
    • Gumbel root selection for small simulation budgets;
    • tree reuse across clicks.

The leaf-value diagnostic on 24 datasets (Fig. 3(c)) is particularly informative. Mixing leaf values with the true Dice separates the capacity of the search from the accuracy of the value estimate, and shows that with exact leaf values the search exceeds the candidate ceiling at 1024 simulations. This is consequential because it identifies value estimation, rather than search, as the limiting factor.

  1. Scale and rigor of the evaluation: The evaluation covers 80 datasets across seven modalities, three backbones, a fixed and documented target-selection rule, a paired bootstrap over datasets, and ground-truth reference points (oracle clicks and candidate ceiling).
Critical weaknesses

1. Most of the improvement comes from the learned policy rather than from test-time search, and the controls needed to isolate multi-step lookahead are missing.

  • Committing the top-prior candidate without search reaches 0.6366 (Table 4). This accounts for 0.236 of the 0.292 improvement over the initial click obtained with 16 simulations. Search contributes 0.056.
  • Scaling largely saturates by 8 simulations per click (0.6899; Table 6). From 16 to 64 simulations, Dice changes by 0.0008 while time per image increases from 1.73 s to 5.65 s (Table 2).
  • The title and abstract present test-time scaling as the central contribution. The evidence instead supports a narrower conclusion: a policy trained on ground-truth-derived click targets accounts for most of the gain, and search adds a modest, quickly saturating improvement.
  • Two controls are needed to support the lookahead claim specifically:
    • (a) Exhaustive one-step lookahead: decode all ~55 candidates and select the one with the highest VθV_\theta or d^θ\hat{d}_\theta. This costs a number of decoder calls comparable to 64 simulations and tests whether multi-step search adds anything beyond learned one-step reranking.
    • (b) The same search with SAM's built-in IoU-prediction output as a training-free value function. This tests whether the learned value network is necessary.

2. The comparison does not include the most natural uses of the same supervision, and the held-out evaluation is not cleanly out-of-distribution.

  • AlphaClick is trained with ground truth from the training splits of the same 80 datasets used for evaluation. Of the baselines, Entropy and SimSAM are training-free. AlignSAM and PPD are trained on the same splits, but were designed for different settings: PPD optimizes prompt sets from reference-image feature matching, and AlignSAM targets open-context prompting with a CLIP vocabulary. PPO likewise refines prompts from reference-image matching, per test image. All of these baselines remain near or below the initial click, and PPO falls below Random (0.1664 vs. 0.1751). Their failure is therefore weak evidence for the proposed approach.
  • The natural competitors are more direct uses of the same annotations: fine-tuning the SAM mask decoder, or a lightweight refinement head, on the same training images with the same single click; or a medical promptable model developed on IMed-361M, such as IMIS-Net, evaluated with a single click.
  • The MedSAM2 result suggests that this is important. On a medically fine-tuned backbone, the gain over the initial click falls to about 0.10, compared with 0.29 on SAM ViT-B (Table 3). Part of the improvement may therefore compensate for domain mismatch that in-domain fine-tuning would also address.
  • The unseen-dataset experiment does not fully resolve this. The gain over the strongest baseline falls to 0.123, under half the 0.268 on the full benchmark, although the held-out datasets also start from a higher initial Dice (0.5088).
  • In addition, several of the 12 held-out datasets share images or patients with datasets that remain in training (Table 5):
    • drishti_gs_cup and drishti_gs_od annotate the same fundus images;
    • CHAOS T1 and T2 are different MR sequences of the same patients;
    • kvasir_seg_aliyun appears to derive from kvasir_seg;
    • the ISIC 2016, 2017, and 2018 segmentation datasets have substantial image overlap, so isic2017_task1 is unlikely to be disjoint from isic2016_task1 and isic2018_task1.
Questions for authors
  1. One-step lookahead (clarifying the main claim): What Dice@10 does exhaustive one-step lookahead over the full candidate set achieve when candidates are ranked by VθV_\theta or by d^θ\hat{d}_\theta? If it approaches the 16- or 64-simulation result, the contribution of multi-step search would be substantially smaller than claimed.
  2. SAM IoU output as value: How does the same search perform with SAM's IoU-prediction output in place of VθV_\theta?
  3. Same-supervision baselines: How does AlphaClick compare with fine-tuning the SAM ViT-B mask decoder on the same 35,551 training images with the same initial click? How does it compare with IMIS-Net evaluated with the same single click on these test targets?
  4. Data splits: Are the IMed-361M official splits defined at the patient or volume level? Could 2D slices from the same patient appear in both the training and test splits of a dataset?
  5. Held-out datasets: Can the authors confirm whether any images or patients in the 12 held-out datasets appear in the 68 training datasets, given the dataset pairs listed above? If overlap exists, can results be reported with all related datasets also excluded from training?
  6. Model selection: On what data were mm, τ\tau, κ\kappa, ϵ\epsilon, and the candidate-slot allocation chosen? Appendix B states that complete test splits were evaluated before the main experiments; were any design decisions informed by those results?
  7. Training variance: Each configuration is run once. How much does Dice@10 vary across independently trained policy-value networks, relative to the 0.003 difference between 8 and 16 simulations?
6 comments
AuthorsOpenReview ↗

Response to Reviewer (1/4)

We thank you for your insightful comments. Detailed responses are provided below.

Q1: One-step lookahead

“What Dice@10 does exhaustive one-step lookahead over the full candidate set achieve when candidates are ranked by Vθ or by d^θ? If it approaches the 16- or 64-simulation result, the contribution of multi-step search would be substantially smaller than claimed.”

A1: The learned policy accounts for most of the improvement over the initial mask, and one-step evaluation recovers a substantial part of the search gain. Dice rises from 0.4008 initially to 0.6366 with the policy alone. Your proposed control helps determine how much further improvement comes from evaluating candidates one click ahead and from planning subsequent clicks. We added both matched-budget and exhaustive one-step controls using the same learned policy and scoring heads on all 80 datasets.

MethodDecoder calls/imageTime (s/image)Dice@10
Policy only110.210.6366
Matched one-step, current-Dice head1611.560.6612
Matched one-step, value head1611.580.6685
Exhaustive one-step, current-Dice head≈5514.820.6654
Exhaustive one-step, value head≈5514.850.6738
MCTS, 8 simulations810.940.6899
MCTS, 16 simulations1611.730.6929

Along the sequence initial mask → policy-only → matched-budget one-step → MCTS, the Dice increments are 0.2358, 0.0319, and 0.0244, accounting for 80.7%, 10.9%, and 8.4% of the total 0.2921 improvement. Exhaustive one-step lookahead recovers 66.1% of the 0.0563 gain from policy-only to MCTS. These results quantify the substantial role of candidate evaluation raised in your question.

Multi-step planning adds 0.0244 at matched cost (paired 95% bootstrap CI [0.0191, 0.0298]) and 0.0191 over exhaustive one-step lookahead (CI [0.0141, 0.0242]), using approximately 1/3.4 as many decoder calls as the exhaustive control. Its contribution is the additional accuracy and computational efficiency of planning click sequences. For the deployed model, Dice rises from 0.6504 to 0.6929 between 1 and 16 simulations and then plateaus, reaching 0.6937 at 64. Sections 4.3 and 4.5 and Appendix I (Table 8) now explain the gain decomposition and the observed scaling range.

Q2: SAM IoU output as value

“How does the same search perform with SAM's IoU-prediction output in place of Vθ?”

A2: SAM’s native IoU head scores the current mask, whereas our value head predicts the final target Dice reached by a continuation. Search uses the latter to compare the potential of different click sequences.

We tested the proposed replacement while keeping the learned policy prior, candidate expansion, and 16-simulation MCTS unchanged. Replacing the learned value with SAM’s native IoU score reduces Dice@10 from 0.6929 to 0.4482. On sibling states, the within-image Spearman correlation with true Dice is 0.14 for SAM’s native score versus 0.71 for our value head. The controlled replacement therefore supports learning a task-specific value for search. The setup and results are reported in Appendix I, Table 8.

AuthorsOpenReview ↗

Response to Reviewer (2/4)

Q3: Same-supervision baselines

“How does AlphaClick compare with fine-tuning the SAM ViT-B mask decoder on the same 35,551 training images with the same initial click? How does it compare with IMIS-Net evaluated with the same single click on these test targets?”

A3: The original experiments freeze each segmentation backbone to evaluate AlphaClick as a test-time planner. To test alternative uses of the same supervision, we added a fine-tuned SAM mask decoder and a direct mask-refinement network, both using the same training-image pool and held-out validation procedure. The decoder uses BCE + Dice supervision with iterative simulated clicks; the refinement network uses the same 3.38M-parameter U-Net architecture as AlphaClick to predict the mask directly. We also evaluated official IMIS-Net weights on the same 12,448 targets and shared initial click.

MethodTraining sourceDice after initial clickDice after 10 automatic corrections
Direct mask refinementSame training-image pool0.5694—
Fine-tuned SAM decoderSame training-image pool0.58420.6015 (Entropy)
IMIS-NetOfficial IMed-361M weights0.61830.5940 (Entropy)
AlphaClick on frozen SAMSame training-image pool0.40080.6929
AlphaClick on fine-tuned SAMSame training-image pool0.58420.7214

AlphaClick exceeds these direct segmentation alternatives, and combining it with decoder fine-tuning improves performance further. The latter result directly addresses the domain-mismatch concern: planning remains useful after the decoder has been adapted to medical images. Section 4.2 and Appendix J (Table 9) report these comparisons, distinguishing the two matched-supervision baselines from the official pretrained IMIS-Net reference.

Q4: Within-dataset splits

“Are the IMed-361M official splits defined at the patient or volume level? Could 2D slices from the same patient appear in both the training and test splits of a dataset?”

A4: We use the train/test assignments released with IMed-361M. These official splits are predominantly volume-disjoint, but they are not uniformly patient-disjoint. Comparing our index with the official image lists confirmed that these assignments are preserved; sampling within each split does not move images between training and test.

Among the 65 datasets derived from 3D volumes, 63 have no volume shared across the official train/test splits. The exceptions are Brain-MRI and braimMRI, where slices from the same volume occur on both sides. Five further datasets—ACDC, CETUS2014, CMRxMotions, mnms2, and CHAOS_Task_4_t1—have disjoint volumes but share patients across phases, views, or acquisitions. We also identified shared source videos in cvc_clinicdb, endovis15, and kvasircapsule_seg, and fellow-eye images from the same patients in PAPILA.

To assess the contribution of these 11 datasets to the reported comparison, we recomputed macro Dice@10 on the remaining 69 datasets using the existing predictions. The same datasets were excluded from every method’s evaluation aggregate:

MethodOriginal 80 datasetsRemaining 69 datasets
Initial click0.40080.3978
Entropy0.42600.4222
SimSAM0.41140.4073
AlignSAM0.38350.3816
PPD0.42050.4156
AlphaClick, 16 simulations0.69290.6855
AlphaClick, 64 simulations0.69370.6855

AlphaClick retains a 0.2633 Dice advantage over Entropy on the remaining 69 datasets. This supports the main performance conclusion after excluding the identified datasets from evaluation. Appendix K reports the split findings and this analysis (Table 11). The held-out experiment separately excludes each evaluation dataset’s entire training split; cross-dataset overlap and the additional training exclusions are addressed below.

AuthorsOpenReview ↗

Response to Reviewer (3/4)

Q5: Held-out datasets

“Can the authors confirm whether any images or patients in the 12 held-out datasets appear in the 68 training datasets, given the dataset pairs listed above? If overlap exists, can results be reported with all related datasets also excluded from training?”

A5: Some separately named datasets in IMed-361M share source images or patients. We compared the images actually used to generate the 68-dataset model’s training trajectories against the 1,600 held-out evaluation targets. The check confirms image- or patient-level overlap in five of the six dataset relationships raised across the reviews. Drishti has no training–test overlap: its cup and disc annotations use the same official partition, so the 51 evaluated cup images are also test images in the disc dataset.

Held-out datasetRelated training dataset(s)Actual overlap affecting evaluation targets
drishti_gs_cupdrishti_gs_od0/51; identical train/test partitions
kvasir_seg_aliyunkvasir_seg29/200 images, re-encoded copies
isic2017_task1ISIC 2016 / 201824/150 identical images: 1 through ISIC 2016 and 23 through ISIC 2018
CHAOS_Task_4_t1CHAOS_Task_4_t218/87 targets from a patient also represented in training
MSD_SpleenAbdomenCT1K68/146 targets at patient level, including 1 identical image
BTCVLearn2Reg abdominal CTCT / MRCT-CT200/200 targets at patient level; all three test patients represented in training

We excluded nine related training datasets and retrained on the remaining 59. On the same targets with 64 simulations per click, macro Dice@10 increases from 0.6648 to 0.6979. The full per-dataset comparison is:

DatasetTargetsInitial clickEntropyAlphaClick: 68 training setsAlphaClick: 59 training setsChange
ACDC1760.52210.52100.60680.6599+0.0531
BTCV2000.45360.48530.68540.7017+0.0163
CHAOS_Task_4_t1870.59890.63870.72910.7768+0.0477
MSD_Heart1330.62470.65020.76460.7699+0.0053
MSD_Spleen1460.65070.68420.90660.9269+0.0203
TDSC-ABUS20232000.42750.47490.56580.6232+0.0574
cvc_clinicdb1230.40640.47220.76810.8102+0.0421
drishti_gs_cup510.38860.46770.62870.6842+0.0555
gamma200.45840.50200.55060.5557+0.0051
isic2017_task11500.55150.54490.68210.5687-0.1134
kvasir_seg_aliyun2000.49100.51470.72460.7750+0.0504
sz_cxr1140.47390.55360.36580.5224+0.1566
Macro mean1,6000.50390.54250.66480.6979≈ +0.033

Both AlphaClick models use 64 simulations per click. Macro means give equal weight to datasets; the reported aggregate values use the evaluation outputs before rounding the per-dataset entries.

Eleven of twelve datasets improve relative to the original model. ISIC 2017 is the sole decrease; removing ISIC 2016 and ISIC 2018 leaves no dermoscopy dataset in training, making this an unseen-modality evaluation. The 59-dataset model also exceeds Entropy on eleven of twelve datasets, with a macro advantage of 0.1554. These results show that strong held-out performance persists after removing the identified related training data.

The held-out restriction applies to AlphaClick’s training. AlignSAM and PPD retain their checkpoints trained on the training splits of all 80 datasets, including these 12 evaluation datasets; PPO uses an annotated training image from each evaluation dataset as its reference. Random, Entropy, and SimSAM are training-free, and their results are unchanged between the 68- and 59-dataset settings. These comparison conditions are explicit in Appendix C and the held-out table notes.

Section 4.4, Table 7, and Appendix K (Table 10) now report the updated training setting and results.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful assessment of AlphaClick. The revised manuscript, with changes marked in blue, addresses the main questions about multi-step planning, supervision, and evaluation splits:

  • The learned policy provides most of the initial improvement, and planning adds further gains. Dice rises from 0.4008 initially to 0.6366 with the policy alone, 0.6685 with one-step lookahead, and 0.6929 with multi-step MCTS. The latter two use the same 161 decoder calls per image. Exhaustive one-step lookahead reaches 0.6738 at approximately 551 calls. Search gains are concentrated between 1 and 16 simulations and plateau thereafter (Sections 4.3 and 4.5; Appendix I, Table 8).
  • The gain persists against direct uses of the same supervision. Fine-tuned SAM and direct mask refinement reach 0.6015 and 0.5694, respectively. Official IMIS-Net weights provide an additional medical-model reference. Combining the fine-tuned SAM decoder with AlphaClick reaches 0.7214 (Appendix J, Table 9).
  • The additional overlap checks account for shared data sources and released splits in IMed-361M. Some separately named IMed-361M datasets share source images or patients. After excluding nine related training datasets and retraining on the remaining 59, macro Dice rises from 0.6648 to 0.6979 on the same 1,600 targets with 64 simulations per click, with improvements on 11 of 12 datasets (Appendix K, Table 10). Separately, excluding 11 datasets with overlap in IMed-361M’s released train/test splits leaves a 0.2633 Dice advantage over Entropy on the remaining 69 datasets (Appendix K, Table 11).
  • The requested robustness checks support the comparisons. Three independently trained networks consistently improve from 8 to 16 simulations; initial-click perturbations and alternative first-click multimask selection preserve strong performance (Appendix L, Table 12).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.