acceptodds
Under review as a conference paper at ICLR 2027

SAM-RISE: Signed Black-Box Saliency for Vision-Language Models via Semantic Region Ablation

Abstract

The most capable vision-language models (VLMs) are closed and reachable only through APIs, which rules out interpretability tools that need gradients, attention maps, or activations. Black-box occlusion saliency, as in RISE and D-RISE, needs only the model’s output, but it does not transfer to VLMs unchanged: a VLM reads a tight occlusion mask as an object in the scene (masking a cat’s eyes yields “the cat has black eyes”), and probability-weighted masks saturate on a confident model. We introduce SAM-RISE, a black-box saliency method built around both failures. It removes a few dozen semantic regions from Segment Anything, each as a grey bounding box, scores the change in the unsaturated logit margin, and keeps both signs: positive evidence, regions whose removal lowers confidence in the answer, and negative evidence, regions whose removal raises it. When an API returns no token probabilities, sampled answers take their place. Semantic regions pay off in three ways. At the small query budgets a paid API imposes, they localise objects as well as random masks with 2 to 4× fewer queries. Negative evidence is actionable: subtracting it corrects over-confident “yes” answers on NaturalBench for two model families. And because the same part can be matched across images, per-image maps aggregate into class-level maps that show whether one image’s explanation is typical of its class, and that separate classes the model recognises from pixels from those it answers from language priors. We evaluate on MS-COCO, ImageNet, and NaturalBench, compare against white-box Grad-CAM and attention rollout and against human judgments, and run SAM-RISE on the closed Gemini model.

Primary area:
interpretability and explainable AI
Keywords:
Black-box Saliency Maps, Vision Language Models, Interpretability, Negative Evidence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xWyxOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper shows that standard black-box occlusion saliency (RISE/D-RISE) fails on VLMs for two reasons. Tight masks get read as objects ("the cat has black eyes"), and probability weights saturate on a confident model, so the D-RISE map is almost exactly a count of how often each pixel was kept. The proposed SAM-RISE ablates around 30 SAM regions as grey bounding boxes, scores changes in the logit margin, and keeps both positive and negative evidence. When log-probabilities aren't available, it estimates them from 20 sampled answers. Experiments cover pointing and insertion on COCO and ImageNet, a comparison with Grad-CAM and rollout, a negative-evidence correction on NaturalBench (Qwen3-VL, Gemma 4), class-level population maps on ImageNet, and a small run on Gemini.

Critical strengths

The diagnosis in Section 3 is the most useful part for me. Showing that VLMs re-interpret tight masks as content, and that D-RISE's published estimator collapses to the mask-count field (correlation 0.9994), is a clean and practical warning. Anyone running occlusion probes on VLMs should know this, whatever happens to the method itself.

I also appreciate that the paper is honest about where it loses. It openly says random masks win given hundreds of queries, and that the Grad-CAM comparison and the user study use selected examples.

Critical weaknesses
  • The gains are modest relative to how they're framed. The "negative evidence is actionable" claim rests on a +0.2 joint-accuracy change on Qwen (not significant) and +1.5 on Gemma, with a true-positive cost, about 32 extra calls per question, and a damping exponent (k = 5) that is needed for it to help at all. I wonder whether a much cheaper baseline, like a global bias shift toward "no" or simple recalibration of P(yes), would get similar true-negative gains. Without that comparison I can't tell whether the region-level signal matters, or whether it's just lowering yes-confidence overall.
  • The evaluation doesn't quite match the method's own argument. Section 3 argues the bounding box is the right occluder, but the pointing benchmark uses the mask-shaped occluder. The advantage over D-RISE holds only below about 75–100 queries. The white-box comparison and the human study (14 showcase pairs, 11 raters) are on selected examples, and the Gemini evidence is 5 pairs. Each piece is reasonable alone, but together the evidence for "SAM-RISE is the better tool" feels thinner than the paper suggests. It reads more like a strong diagnosis paper with a reasonable method attached.
Questions for authors
  1. Does the NaturalBench gain survive against a simple calibration baseline (a global shift or threshold tuned on held-out data)? This would most directly change my assessment.
  2. Why does the pointing benchmark use the mask-shaped occluder instead of the bounding box? What are the numbers with the bbox?
  3. The bbox covers about 1.55× the mask area. You mention a dilated-outline control and a matched-area random-box null as future work. Could you run at least a small version? Right now the "signal tracks the part, not the context" claim is only correlational.
  4. For the population maps, how sensitive are the results to the choice of query image that seeds the anchors? Only 10 classes are covered.

A minor thing is, relational questions seem to fail (Fig. 10). How often does this happen on NaturalBench overall?

2 comments

We thank the reviewer for the careful reading, and for crediting the diagnosis and the paper's candour about where SAM-RISE loses.

W1 / Q1. Does the NaturalBench gain survive a calibration baseline? We ran the controls the reviewer suggests on the stored scores, with no new queries. NaturalBench's 1,900 groups include 450 multiple-choice groups. Following our protocol, these are posed as yes/no questions and count as correct when the model answers "No", so any reduction of "yes" confidence is rewarded on them. We therefore report the full benchmark (the paper's protocol) and the 1,450 binary groups, where a correct answer has its usual meaning. Values are joint accuracy (%).

Qwen3-VL, all 1,900Qwen3-VL, 1,450 binaryGemma 4, all 1,900Gemma 4, 1,450 binary
Baseline23.530.325.225.0
Global threshold, 5-fold CV25.332.628.225.1
Ours, pooled masks23.730.326.725.0
Global "no" shift, same TN gain23.4 (p = 0.30)29.5 (p = 0.07)26.5 (p = 0.74)24.3 (p = 0.08)
Ours, δ shuffled across items22.8 (p = 0.01)28.9 (p = 0.002)27.1 (p = 0.86)23.5 (p = 0.004)
Ours, SAM boxes only23.630.326.325.2
Global "no" shift, same TN gain23.3 (p = 0.21)29.8 (p = 0.13)26.5 (p = 0.61)24.2 (p = 0.003)
Ours, δ shuffled across items23.4 (p = 0.14)30.0 (p = 0.09)26.6 (p = 0.88)24.4 (p = 0.02)

The p-values compare our joint accuracy with each control: exact McNemar for the shift, and a permutation p over 500 draws for the shuffle. The shuffle keeps the damping and each item's own margin, and moves only δ between items.

We draw three conclusions, and the revised Sec. 6.4 reports both protocols and all controls.

  1. A tuned threshold is the better accuracy fix. Any monotone recalibration of P(yes) (Platt, temperature, bias) makes the same decisions as a global threshold, and that threshold beats our correction on both models and protocols. We therefore no longer present the correction as an accuracy method.
  2. On binary groups, the region signal chooses which answers to lower. Joint accuracy is unchanged. True negatives rise by 2.2 (Qwen) and 1.6 (Gemma) points, and true positives fall by 2.8 and 2.7. At the same true-negative gain, our correction keeps 0.5 to 1.0 points more joint accuracy than a uniform shift (p = 0.003 to 0.13). Moving the same δ values to random items costs 0.4 to 1.5 points (p = 0.002 to 0.09). On the full set, the multiple-choice groups reward any lowering of "yes", so there the shift and shuffle match our correction.
  3. The effect is small. We present it as evidence that negative-evidence regions identify answers worth distrusting, a diagnostic property of the maps. The cost (about 32 calls per question) and the role of the damping exponent are stated up front.

W2 / Q2. The pointing benchmark used the mask occluder; what are the bounding-box numbers? We reran the benchmark with the bounding-box occluder on the same 500 COCO and 500 ImageNet images, with the same SAM regions, model, prompt and call budget. Only the shape of the hidden region changes.

Pointing gameMask occluder (paper)Box occluderp (McNemar)
ImageNet, GT box0.7960.6549e-10
COCO, GT box0.8020.5942e-16
COCO, GT mask0.7260.4384e-26

Boxes lose 14 to 29 points, and on ImageNet they fall below the image-centre control (0.704). The two operators serve different purposes, and the revision states which one is used where. The box is the right operator for measuring a region's effect: it removes the re-interpretation failure of Sec. 3 and doubles the deep-drop tail. The mask is the right operator for localising. A box of a large or elongated region also hides neighbouring content, often the object itself, so a background region can inherit the object's score drop and win the argmax. We see the same effect in App. I, where a background box can span the whole frame. A box also hides about 1.6× the mask's pixels, so this run changes shape and area together, and the dilated-outline control (Q3) separates the two. With area-normalised scoring, boxes do better than masks (e.g. 0.566 vs 0.480 on COCO, GT box), but both stay far below the headline scoring. The revised Table 6 reports both occluders. On the budget crossover we agree, and the paper says so. SAM-RISE wins per query below about 75 to 100 calls, the regime a paid API imposes, and random masks win given hundreds. On SAM-RISE's own regions and budget, LIME ties SAM-RISE (0.740 vs 0.726 PG, p = 0.52), while KernelSHAP and PixelSHAP are worse. With 2 to 4 times the queries, LIME and SHAP pass SAM-RISE by 3 to 5 points.

Closed models and the 5 Gemini pairs. SAM-RISE needs only the model's answers, so it applies to any model unchanged. We are running it on two more closed models from two more providers, Claude Fable 5.1 (Anthropic) and GPT-6 Astra (OpenAI), and extending Gemini 3.8 Flash from 5 to all 16 image-question pairs. All use the unchanged pipeline (same SAM regions, prompt, grey boxes and 20 sampled answers). Each is scored quantitatively with the metric of Sec. 6.3: is the region that flips Qwen's log-probability answer among the model's top regions? We will include these results in the revised paper.

Selected examples. The white-box and user-study comparisons are now labelled illustrative in the main text. The quantitative claims rest on the 500 + 500-image benchmark, the matched-budget LIME/SHAP/PixelSHAP comparison, and NaturalBench on two models.

Q3. Dilated outline and matched-area random box. We agree that the current evidence is correlational (within box-area deciles, a box's background fraction does not predict its effect; Spearman 0.03 over 66,018 boxes). We are running the causal version: for each SAM region, the bounding box, the outline dilated to the box's area, and a random box of the same size placed off the region, compared paired on the same images. We will include the results in the revised paper.

Q4. Sensitivity of population maps to the query image. We are rerunning the population study with several query images per class and on more classes. We will report the spread of the class-averaged drop across query images and whether the separation between recognised and language-prior classes holds, and include the results in the revised paper.

Q5. How often do relational questions fail? We measured this on NaturalBench (Qwen3-VL, the 534 groups with SAM masks, 1,578 yes/no items). For each item, we checked whether removing any single SAM region as a grey box flips the answer. A flipping region exists for 42.7% of items, and for the other 57% no single region is decisive. Questions with relational or counting words are not worse than the rest (44.7% vs 41.7% have a flipping region). The limit is the single-region probe in general, not relations in particular, and the revised limitations section reports this.

Reviewer_Mhb9OpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce SAM-RISE, a black-box saliency method designed to address two failure points of existing methods when applied to modern, closed-sourced VLMs, such as Gemini 3.8. They identify these two failure points as (i) tightly masked regions are misinterpreted by the VLMs and (ii) probability-weighted saliency masks are saturated and uninformative in confident models. To address the first issue, SAM-RISE uses SAM-suggested regions but occludes their bounding boxes instead of tight masks. The second issue is addressed by using logit margins rather than probabilities to avoid saturation, capturing both positive and negative evidence.

The authors benchmark SAM-RISE against D-RISE, showing that their method achieves higher accuracy per API call. They also compare it against white-box methods (GRAD-CAM and Attention Rollout), showing that their method works better on a small set of 20 examples.

Critical strengths

First, the proposed method is well motivated. The authors identify two limitations of existing methods on modern VLMs, and introduce a new algorithm that explicitly addresses them.

Second, the authors provide a thorough experimental evaluation (section 6), including comparisons with both black-box and white-box methods. Their results demonstrate that SAM-RISE outperforms existing methods in certain settings, particularly at small query budgets.

Finally, the paper is well written and structured nicely, making it easy to understand and follow along.

Critical weaknesses

First, if I am not mistaken, the only closed-source VLM evaluated is Gemini 3.8 Flash. Given the paper's focus on black-box interpretability, I would have liked to see evaluations on additional closed-source models, ideally from other companies, such as GPT or Claude. Moreover, the evaluation on Gemini itself appears limited to only five image-question pairs (Section 6.3), with primarily qualitative results (Figure 6), and no quantitative results against existing methods.

Second, several of the paper's experiments are conducted on relatively small evaluation sets. For example, Section 6.2 uses only 20 image-question pairs and 11 human raters, while Section 6.5 covers only 10 classes. Although the main quantitative experiments are more extensive, I would have liked to see these additional evaluations conducted on larger and more representative sets to better support the generalizability of the findings.

Questions for authors

Following my weaknesses, I have the following questions:

(1) Have you tested your method on other closed-source models, such as GPT or Claude? If so, does SAM-RISE work without modification, and do your conclusions still hold, or are model-specific adaptations required?

(2) Could you expand the evaluations in Sections 6.2 and 6.5 to include more image-question pairs and ImageNet classes, respectively? In particular, do the findings remain consistent when evaluated on larger, more representative datasets?

1 comment

We thank the reviewer for the positive assessment of the motivation and writing.

W1 / Q1. Other closed-source models, and only 5 Gemini pairs. SAM-RISE is black-box and architecture-agnostic. It reads only the model's answers and never touches weights, gradients, attention or tokenizer internals, so a new model needs only a new API endpoint. We are running it on two more closed models from two more providers, Claude Fable 5.1 (Anthropic) and GPT-6 Astra (OpenAI). Neither returns usable token probabilities, so both use the same 20 sampled answers as Gemini. We are also extending all three closed models from 5 to all 16 image-question pairs, with a quantitative comparison. The metric is the one Sec. 6.3 validated against log-probabilities: is the region that flips Qwen3-VL's answer among the closed model's top regions? We will include these results, with request counts and cost, in the revised paper.

Does SAM-RISE work without modification? Yes. The pipeline (SAM regions, prompt, grey-box occluder, sampled estimator) is identical across models, and only the API client differs, to respect each provider's interface. Gemini accepts one answer per request, and Claude Fable 5.1 does not accept a temperature setting, so it is sampled at its default. No model-specific change to the method is needed.

W2 / Q2. Small evaluation sets. The headline results use full sets: 500 + 500 images for pointing, insertion and deletion; all 1,900 NaturalBench groups on two models; and new matched-budget comparisons against LIME, KernelSHAP and PixelSHAP on the same 1,000 images. The small sets are the white-box and user-study showcases, which the revision labels as illustrative, and the population study. We are expanding the population study to more ImageNet classes and several query images per class, and will report whether the separation between recognised and language-prior classes (13.4 vs at most 1.2 logits) holds. We are also expanding the white-box comparison and the user study to more image-question pairs. We will include these results in the revised paper.

Reviewer_pjbUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a black-box saliency method, called SAM-RISE, to explain vision-language models using only their outputs, without accessing gradients, attention, or internal activations. It identifies two failures of conventional occlusion saliency: models can interpret tightly shaped masks as objects rather than missing content, and highly confident answer probabilities obscure meaningful changes. SAM-RISE uses Segment Anything to propose semantic regions, occludes their bounding boxes with grey, and measures changes in the yes-versus-no logit margin to distinguish supporting evidence from evidence that suppresses the answer. When token probabilities are unavailable, it estimates scores from 20 sampled answers per input, enabling explanations of closed models such as Gemini but multiplying the query cost. Extensive experiments have been conducted in the paper to compare the proposed saliency method and D-RISE baseline on a series of benchmarks. Its main limitations include restricted model and class coverage, selected examples in the white-box and human-preference comparisons, sampling-related uncertainty, and the possibility that bounding-box occlusion itself introduces distribution shifts.

Critical strengths
  1. This paper has conducted a comprehensive evaluation and ablation study that are included in the appendix. These experiments , including section 3.3 and 6.1, has compared more baselines than a saturated probability estimator. Figure 4 and Table 3, 4, 5 show both low-budget benefits and advantage of dense random masks.
  2. This paper has a strong motivation to make powerful, closed-source vision-language models interpretable.
Critical weaknesses
  1. Eq. (2) only decreases the yes margin as the δ\delta is always positive. Therefore no correct no answer can flip to yes by construction, and true-negative gains alone do not establish a region-specific correction mechanism. The negative evidence correction might simply making the model less likely to answer 'yes'.

  2. The method selects anchors from one query image and matches them to other images using DINOv2 embedding similarity, without establishing that those matches consistently represent the same semantic part.

  3. It is good to compare the method with adapted random-mask benchmark. However, closely related signed semantic approaches such as LIME, SHAP, and PixelSHAP are not evaluated under matched region pools and query budgets. These comparison would be informative to find out how much improvement comes from semantic proposals rather than the specific SAM-RISE scoring.

  4. In the experiments of the paper, the proposed method measures each region’s effect while leaving the rest of the image intact. If two distinct cues independently support the same answer, removing either may have little effect, causing both to appear unimportant despite their collective relevance. It would be great if the authors could come up with some more experiments to do coalition-based ablations.

Questions for authors

A few detailed questions:

  1. How often are yes/no aliases missing from the returned top-20 list, and do full-logit results change the ranking or correction gains?
  2. How do boxes compare with dilated outlines and matched random boxes when removed area and location are controlled?
2 comments

We thank the reviewer for the precise technical questions; several led to new experiments.

W1. Eq. 2 can only lower the yes margin. Correct, and the revision says so plainly. The correction is one-sided by design, so "never flips a correct no" is a property of the estimator, not evidence that it works. The real test is whether the region signal picks which "yes" answers to lower better than lowering every "yes". We ran that test against a global "no" shift with the same true-negative gain, a shuffle of δ across items, and a cross-validated global threshold. On the 1,450 binary NaturalBench groups, our correction keeps 0.5 to 1.0 points more joint accuracy than the matched shift (p = 0.003 for Gemma with SAM boxes, 0.07 to 0.13 elsewhere), and shuffling δ costs 0.4 to 1.5 points (p = 0.002 to 0.09). So the region signal is informative. However, it does not raise joint accuracy on these groups, and a tuned global threshold improves accuracy more on both models. The paper's joint-accuracy gain on all 1,900 groups comes from the 450 multiple-choice groups, which our protocol poses as yes/no and credits when the model answers "No". We therefore reframe the result as diagnostic rather than an accuracy method, and the revised Sec. 6.4 reports both protocols and all controls (full table in our reply to Reviewer xWyx).

W2. DINOv2 matches may not represent the same part. We agree that the paper does not establish this directly. The indirect evidence is that grid cells matched the same way carry no consistent evidence across a gallery (0.019 for 3×3 and 0 for finer grids, vs 0.080 for SAM regions, Table 9). If DINOv2 matches were arbitrary, SAM regions would collapse the same way. For a direct test, we are using PartImageNet's part annotations to measure how often the matched region covers the same annotated part as the anchor, against a random-region baseline. We will include the results in the revised paper.

W3. LIME, SHAP and PixelSHAP under matched regions and budgets. We ran all three on SAM-RISE's exact SAM regions, flat-fill occluder, logit-margin score and map painting, at 1, 2, 4 and 8 times SAM-RISE's per-image budget. LIME and SHAP use the released lime / shap samplers and fits, and PixelSHAP uses its released estimator. We evaluate on 500 COCO and 500 ImageNet images, with paired McNemar tests against SAM-RISE.

At SAM-RISE's budget (1×)COCO PG maskpImageNet PG boxp
SAM-RISE0.726-0.796-
LIME0.7400.520.8000.89
KernelSHAP (default L1)0.6840.040.7800.40
PixelSHAP, as released0.674<0.0010.7680.049
LIME fit to P(yes)0.6143e-60.7560.042

At matched budget, LIME ties SAM-RISE on pointing and is better on COCO deletion (paired ΔDel −0.090 [−0.137, −0.035]). KernelSHAP and PixelSHAP are worse. PixelSHAP as released never occludes pixels of kept objects. With SAM-RISE's occlusion it ties SAM-RISE (0.720 vs 0.726, p = 0.55), since its first queries are the same leave-one-out ablations. With 2 to 4 times the queries, LIME and SHAP pass SAM-RISE by 3 to 5 points, and D-RISE (our adaptation) is still stronger at about 150 queries. Our saturation diagnosis carries over: fitting LIME to P(yes), its usual target, drops it to 0.614, and scoring PixelSHAP by answer agreement drops it to 0.466. We revise the claim accordingly. The low-budget advantage comes from the semantic region pool combined with an unsaturated margin score. Among methods on that pool, SAM-RISE is the cheapest and gives signed per-region effects directly.

W4. Redundant cues and coalitions. The reviewer is right, and the new results quantify it. LIME's random coalitions drop about half the regions per query. That exposes redundant cues that one-at-a-time removal misses, which is why LIME is better on COCO deletion at matched budget. On NaturalBench (Qwen3-VL, 1,578 yes/no items with SAM masks), no single region flips the answer for 57% of items. The revision adds coalition sampling over SAM regions as a variant and discusses the cost trade-off.

Q1. Yes/no aliases missing from the top-20 list. We did not log alias misses directly. A missing alias is set to log-probability −20, which can only matter for answers whose margin is already around 19 logits or more. Only 2.2% of Qwen's 7,600 NaturalBench baseline answers are in that range (median |margin| 7.3). For these the decision is unaffected, and only the size of an already-saturated margin is capped.

Q2. Boxes vs. dilated outlines and matched random boxes. One new data point bears on this. On the pointing benchmark (500 COCO + 500 ImageNet images), swapping each SAM mask for its bounding box lowers pointing accuracy by 14 to 29 points (p < 1e-9). The extra area a box hides often covers the object itself, which helps when measuring a region's effect but hurts localisation. The paper's current evidence on context is correlational. Within box-area deciles, the background fraction of a box does not predict its effect (Spearman 0.03 over 66,018 boxes). We are running the causal controls the reviewer suggests, on the same images: the outline dilated to the box's area, and a random box of the same size placed off the region. We will include the results in the revised paper.

Closed models. SAM-RISE needs only the model's answers. Beyond Gemini 3.8 Flash, we are running it unchanged on Claude Fable 5.1 and GPT-6 Astra and will include the results in the revised paper (reply to Reviewer Mhb9).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.