- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SAMLens, a paired audit for specialized SAM models. It compares a reference and a specialized endpoint on the same images and prompt transformations, and reports both absolute responses and changes. The main probe adds a negative point inside a neighboring object. On LIVECell, full mask-decoder tuning with positive-point-only supervision increases the neighboring-object overlap from about 0.03 to 0.53, despite a canonical Dice of 0.693. The effect recurs on PanNuke and across additional seeds, while other audited adaptations do not show a reversal of comparable size. The paper also argues that change-only metrics can confuse response compression with an actual increase in distractor coverage, and that the main effect appears in the returned masks rather than only in mask selection.
Critical strengths
-
The paper identifies a potentially important blind spot in evaluating specialized promptable segmenters. Canonical Dice, same-target stability, and target switching do not reveal the negative-point behavior, while the proposed absolute response exposes a qualitatively different failure.
-
The paired design and the absolute-versus-change comparison are useful. They make the distinction between “the negative point has little response left to produce” and “the negative point increases distractor coverage” clear. The effect is also reported across two dense domains and multiple seeds, rather than being based on a single qualitative example.
Critical weaknesses
-
The contribution is currently closer to an empirical audit than to a sufficiently general ICLR contribution. SAMLens combines several reasonable evaluation ideas, but the main finding is tied to SAM ViT-B, two dense datasets, and one canonical-only specialization setup. The paper does not show that the audit or the failure generalizes to other specialization methods, architectures, or realistic interaction settings. Thus the practical importance is plausible but not yet broad enough for a clear ICLR contribution.
-
The paper does not isolate the mechanism behind the reported reversal. A negative point is interpreted as an instruction to suppress the whole neighboring object, although the prompt itself is point-based. The measured increase in distractor overlap can also involve mask expansion, loss of the original target, or a prompt-induced mask switch. Moreover, the main audit evaluates SAM's selected multimask output, while training uses the single-mask output and positive prompts. Full and partial tuning also update different modules and very different numbers of parameters. Therefore, the results establish an interesting failure pattern, but do not establish whether it is caused by the full decoder, the two-way transformer, model capacity, or the training procedure.
Questions for authors
-
Is the intended contract of a negative point point-level or object-level? In particular, does SAMLens define a failure when the clicked pixel is included, when the whole neighboring object becomes included, or both? This distinction affects the interpretation of the main claim.
-
The main experiment trains the single-mask output but evaluates the selected multimask output. How much of the reported effect is present when the same output path used for training is used for evaluation? This would clarify whether the finding is a specialization failure or partly an inference-path mismatch.
-
Do the authors have evidence that separates full-decoder tuning from the particular modules, parameter count, or training budget used here? A transformer-only comparison or a parameter- and schedule-matched control would help establish the mechanism behind the effect.
-
Answers to these questions would materially affect my assessment, especially whether the paper demonstrates a general auditing insight or a failure specific to one specialization recipe. Depending on the authors’ clarifications and additional evidence, I may reconsider my rating.