acceptodds
Under review as a conference paper at ICLR 2027

SAMLens: Canonical Quality Can Hide Negative-Point Reversal in Specialized Segment Anything Models

Abstract

Specialized versions of the Segment Anything Model (SAM) keep their point-prompt interface, yet release checks usually validate only the single positive prompt used in training. Among the prompts that remain available, a negative point has the least ambiguous meaning: the mask should not grow into the object it marks. We introduce SAMLens, a paired audit that gives reference and specialized endpoints identical images and prompts and reports each prompt relation both as an absolute response and as a change. On dense cell images, full mask-decoder tuning reverses this instruction: a negative point inside a neighboring object raises the predicted overlap with that object from 0.03 to 0.53, whereas endpoints of other SAM-family adaptations whose canonical Dice lies within 0.03 of this endpoint keep the object excluded or add only a small part of it. The reversal recurs in two further training seeds on cell images and in three on histopathology nuclei, and no other audited adaptation shows a reversal of comparable size. Changes in same-target stability and target switching do not separate the reversed endpoint from the others. A candidate decomposition places the reversal in the returned masks rather than in score-based selection, and shows that the relative decline seen after partial tuning is mostly response-range compression. Canonical Dice and change-only summaries of prompt behavior can therefore hide whether a specialized segmenter still follows negative points.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Segment Anything Model, promptable segmentation, negative prompts, behavioral auditing, model specialization, interactive segmentation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Dk2BOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SAMLens, a paired audit for specialized SAM models. It compares a reference and a specialized endpoint on the same images and prompt transformations, and reports both absolute responses and changes. The main probe adds a negative point inside a neighboring object. On LIVECell, full mask-decoder tuning with positive-point-only supervision increases the neighboring-object overlap from about 0.03 to 0.53, despite a canonical Dice of 0.693. The effect recurs on PanNuke and across additional seeds, while other audited adaptations do not show a reversal of comparable size. The paper also argues that change-only metrics can confuse response compression with an actual increase in distractor coverage, and that the main effect appears in the returned masks rather than only in mask selection.

Critical strengths
  • The paper identifies a potentially important blind spot in evaluating specialized promptable segmenters. Canonical Dice, same-target stability, and target switching do not reveal the negative-point behavior, while the proposed absolute response exposes a qualitatively different failure.

  • The paired design and the absolute-versus-change comparison are useful. They make the distinction between “the negative point has little response left to produce” and “the negative point increases distractor coverage” clear. The effect is also reported across two dense domains and multiple seeds, rather than being based on a single qualitative example.

Critical weaknesses
  • The contribution is currently closer to an empirical audit than to a sufficiently general ICLR contribution. SAMLens combines several reasonable evaluation ideas, but the main finding is tied to SAM ViT-B, two dense datasets, and one canonical-only specialization setup. The paper does not show that the audit or the failure generalizes to other specialization methods, architectures, or realistic interaction settings. Thus the practical importance is plausible but not yet broad enough for a clear ICLR contribution.

  • The paper does not isolate the mechanism behind the reported reversal. A negative point is interpreted as an instruction to suppress the whole neighboring object, although the prompt itself is point-based. The measured increase in distractor overlap can also involve mask expansion, loss of the original target, or a prompt-induced mask switch. Moreover, the main audit evaluates SAM's selected multimask output, while training uses the single-mask output and positive prompts. Full and partial tuning also update different modules and very different numbers of parameters. Therefore, the results establish an interesting failure pattern, but do not establish whether it is caused by the full decoder, the two-way transformer, model capacity, or the training procedure.

Questions for authors
  • Is the intended contract of a negative point point-level or object-level? In particular, does SAMLens define a failure when the clicked pixel is included, when the whole neighboring object becomes included, or both? This distinction affects the interpretation of the main claim.

  • The main experiment trains the single-mask output but evaluates the selected multimask output. How much of the reported effect is present when the same output path used for training is used for evaluation? This would clarify whether the finding is a specialization failure or partly an inference-path mismatch.

  • Do the authors have evidence that separates full-decoder tuning from the particular modules, parameter count, or training budget used here? A transformer-only comparison or a parameter- and schedule-matched control would help establish the mechanism behind the effect.

  • Answers to these questions would materially affect my assessment, especially whether the paper demonstrates a general auditing insight or a failure specific to one specialization recipe. Depending on the authors’ clarifications and additional evidence, I may reconsider my rating.

Reviewer_js12OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper introduces SAMLens: a test that checks whether a SAM model finetuned for a special domain like cell images or medical usecases still reacts to clicks in the same way as the original SAM.(such as "negative click" which select a region that user don't want to include)

SAMLens gives the original and finetuned model the same images and clicks, and checks what happens when a negative click is put on a nearby object. The authors find that finetuning the whole mask decoder keeps a good positive click score but breaks the negative click. After positive click only finetuning, a negative click on the nearby object even adds it to the mask instead of removing it.

In conclusion, this paper mainly shows:

  • Two models with the same normal test score can react to the same negative click in opposite ways.
  • Finetuning the whole mask decoder makes the negative click work backwards in every seed and on both cell datasets.
  • The usual prompt checks do not catch this problem and choosing another of SAM's output masks does not fix it.
Critical strengths
  • The finding is clear and practical bcuz two models with the same normal score can react to the same negative click in opposite ways, and since this repeats across seeds and datasets it gives a cheap check before release.
  • The related work and the stat are carefully done as each prior line is compared with what this paper measures and the analysis follows a dated plan with image level statistics.
  • This kind of "catastrophic forgetting" phenomenon on "behavior" was under explored for such detection or classification heavy models. This paper provide an important direction to works on.
Critical weaknesses
  1. The paper never justifies that negative clicks are actually needed for specialized SAM models, because its only support is one sentence citing general interactive segmentation work, and it leaves open whether the reversal costs annotators anything in practice (Section 8). Since these models are trained and released for single positive-click use, people may test only the positive click simply because that is all they use, in which case the reversal is real but may not matter.
  2. The finding looks like a special case of known finetuning damage such as forgetting [6][7], which the paper itself cites, but the paper does not say what is new beyond this or why the click works backwards instead of just getting weaker, and ProMISe [3], which it calls the closest work, is never tested. MedicoSAM [1] already reports a forgetting-like effect in which finetuning without iterative multi-prompt training breaks point and multi-prompt interaction. The paper cites it only for correction gains.
  3. The evidence is much narrower and less clean than the title suggests. It uses one SAM size, two cell datasets and one training recipe, and the advice to "audit the extent of adaptation" rests on three configurations rather than a sweep of learning rate or training length. Some controls also blur the picture, because the far-background click is also harmed by LoRA and the normal score gain flips under one click policy, while the main analyses are post-hoc and the training recipe is only partly reported.
  4. The only fix the main text talks about is retraining the IoU head, but the paper itself says this cannot remove the reversal. The other repair methods are only in the appendix and are only tried on partial tuning which does not have the reversal at all, and they also lose Dice. So no fix is tried on the fully tuned decoder which is the real problem, and this makes the work much less valuable. For this task the easiest fix is to just train with both positive and negative clicks made from the instance masks, which is already what micro-sam [2] and MedicoSAM [1] do in every training step. The micro-sam model that the paper itself tests also does not show the reversal.

References

  • [1] A. Archit, L. Freckmann and C. Pape. MedicoSAM: Robust Improvement of SAM for Medical Imaging. IEEE TMI 2026
  • [2] A. Archit et al. Segment Anything for Microscopy. Nature Methods 2025
  • [3] J. Wang et al. ProMISe: Promptable Medical Image Segmentation using SAM. arXiv:2403.04164, 2024
  • [4] M. M. Rahman et al. PP-SAM: Perturbed Prompts for Robust Adaption of Segment Anything Model for Polyp Segmentation. CVPRW 2024
  • [5] J. Ma et al. Segment Anything in Medical Images. Nature Communications 2024
  • [6] J. Kirkpatrick et al. Overcoming Catastrophic Forgetting in Neural Networks. PNAS 2017
  • [7] A. Kumar et al. Fine-Tuning Can Distort Pretrained Features and Underperform Out-of-Distribution. ICLR 2022
Questions for authors
  1. Is there any evidence, such as tool logs, a user study or the documented workflow of micro-sam [2] or MedSAM [5], that users of specialized SAM models rely on negative clicks, and how many extra corrections does the reversal cause?
  2. Why does full decoder tuning make the negative click work backwards instead of just weaker?
  3. Can you compare against ProMISe [3] or a prompt-robust method like PP-SAM [4] under the same test?
  4. Does the problem still appear with a lower learning rate or shorter training, and do your repair methods work on the fully tuned decoder? What prompts are used to train PCPA and do they include negative points?

Besides the questions above, please also refer to the weakness part.

References

  • [1] A. Archit, L. Freckmann and C. Pape. MedicoSAM: Robust Improvement of SAM for Medical Imaging. IEEE TMI 2026
  • [2] A. Archit et al. Segment Anything for Microscopy. Nature Methods 2025
  • [3] J. Wang et al. ProMISe: Promptable Medical Image Segmentation using SAM. arXiv:2403.04164, 2024
  • [4] M. M. Rahman et al. PP-SAM: Perturbed Prompts for Robust Adaption of Segment Anything Model for Polyp Segmentation. CVPRW 2024
  • [5] J. Ma et al. Segment Anything in Medical Images. Nature Communications 2024
  • [6] J. Kirkpatrick et al. Overcoming Catastrophic Forgetting in Neural Networks. PNAS 2017
  • [7] A. Kumar et al. Fine-Tuning Can Distort Pretrained Features and Underperform Out-of-Distribution. ICLR 2022 [1] A. Archit, L. Freckmann and C. Pape. MedicoSAM: Robust Improvement of SAM for Medical Imaging. IEEE TMI 2026 [2] A. Archit et al. Segment Anything for Microscopy. Nature Methods 2025 [3] J. Wang et al. ProMISe: Promptable Medical Image Segmentation using SAM. arXiv:2403.04164, 2024 [4] M. M. Rahman et al. PP-SAM: Perturbed Prompts for Robust Adaption of Segment Anything Model for Polyp Segmentation. CVPRW 2024 [5] J. Ma et al. Segment Anything in Medical Images. Nature Communications 2024 [6] J. Kirkpatrick et al. Overcoming Catastrophic Forgetting in Neural Networks. PNAS 2017 [7] A. Kumar et al. Fine-Tuning Can Distort Pretrained Features and Underperform Out-of-Distribution. ICLR 2022
Reviewer_yr2nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies whether domain-specific fine-tuning preserves the prompt behavior of segmentation models. SAMLens compares reference and adapted endpoints under the same prompts and reports both absolute responses and changes in target preservation, switching and negative-point suppression. The experiments show that full mask-decoder tuning can improve single-positive-point Dice while reversing the response to a negative point.

Critical strengths
  1. The motivation is interesting. Checking whether domain adaptation preserves interactive behavior addresses a meaningful problem beyond segmentation accuracy.

  2. Comparing the same prompts before and after fine-tuning is an interesting analysis approach. Checking all returned masks also helps determine whether the model generates poor masks or selects the wrong one.

  3. The paper is easy to follow, and the tables and figures are informative.

Critical weaknesses
  1. The evaluation is limited to independent single-positive and positive-plus-negative queries, without accumulated clicks or previous-mask input. The scope therefore does not cover the repeated correction process used in interactive annotation.

  2. The analysis locates the reversal in returned masks but does not explain why full-decoder tuning causes the behavior. The adaptation settings change multiple factors, so the evidence does not isolate the responsible training objective or decoder component.

  3. The paper does not provide an effective solution for the reversal. Reranking cannot remove an error already present in the returned candidates, while the supplementary repair experiments show trade-offs without resolving the full-decoder case.

Questions for authors

The research direction is interesting, but the practical effect on iterative annotation and a supported direction for fixing the reversal remain unclear. For the specific points, please refer to Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.