acceptodds
Under review as a conference paper at ICLR 2027

Interaction Steering for Compositional Robustness in Vision Models

Abstract

Modern vision models are trained through outcome supervision: the predicted class should match the label. However, this supervision does not specify how visual factors should contribute jointly, allowing models to rely on spurious context or overlook useful combinations of object parts. We introduce Interaction Steering, a framework for mechanism supervision in vision models. Interaction Steering applies finite-difference operators to counterfactual edits of a model's score, isolating interactions such as object-background, object-co-occurring-object, and part-part effects. These operators audit how named visual factors jointly affect the score and define auxiliary steering and guard losses that move their interactions toward specified targets. Counterfactual edits are used only for these auxiliary losses, while the primary supervised objective remains on natural images. On Waterbirds, Interaction Steering increases worst-group accuracy from to on ResNet-50 and from to on DINOv2 ViT-B/14. On UrbanCars, it improves all three official shortcut gaps, including the combined background-plus-co-object gap. On PACO, it shifts the mean audited part-part interaction from to , while natural Top-1 changes by percentage points on a held-out PACO-LVIS test manifest covering five object categories. Further experiments retain Waterbirds gains with masks derived from the classifier itself and show larger interaction increases for the selected PACO part pair than for other audited part pairs. Interaction Steering enables direct supervision of how visual factors jointly affect a model's score, supporting both shortcut suppression and useful synergy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
computer vision, shortcut learning, spurious correlations, counterfactual learning, feature interactions, mechanism supervision, robust generalization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_c13FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Interaction Steering to control selected visual interactions. Finite differences across edited images measure how named factors affect model scores together. Auxiliary losses suppress unwanted interactions or encourage selected part interactions. The main classification loss stays on natural images. Inference uses only the classifier. Experiments show robustness gains on Waterbirds and UrbanCars, and targeted interaction changes on PACO.

Critical strengths
  1. The same interaction framework supports both suppression and enhancement. It gives users a direct way to specify which visual factors should interact. This is a useful methodological contribution, even though the mathematical tools are established.

  2. Waterbirds worst-group accuracy rises from 81.26% to 89.04% on ResNet-50. DINOv2 improves from 89.55% to 94.81%. Joint steering reduces all UrbanCars shortcut gaps relative to ERM. Alternative mask tests further strengthen the evidence.

Critical weaknesses
  1. finite difference and explanation-based training are both pretty mature methods, and CDEP can also constrain feature interactions, but why not use CDEP? not clear what extra value the proposed method has, novelty also needs clearer sorting out.

  2. the matched PACO control is useful, but Waterbirds and UrbanCars lack similar control, still unclear how separating the target loss contributes to robustness and to sensitivity of the guard weights, also some components’ effects still unclear.

  3. Full CF-Aug can improve Waterbirds accuracy, while IS lowers the measured interaction score, these trade-offs are useful, but when to choose IS still not very clear.

  4. PACO shows controllable interaction changes, but the target loss doesn’t bring clear extra accuracy gain. it only covers five annotated categories and controlled removal, natural occlusion and unseen categories not tested yet. these limitations narrow the applicability.

Questions for authors
  1. Which capabilities distinguish IS from interaction penalties such as CDEP?

  2. Can you add matched controls on Waterbirds and UrbanCars that remove only the target loss? How sensitive are the results to guard weights?

  3. Which practical failures should stronger part interactions help resolve? The authors should show a test with natural occlusion or unseen classes.

1 comment

Thank you for your feedback. Our general comment defines the robustness measures used below and provides the code link.

W1 / Q1: CDEP. CDEP can constrain feature interactions, and we will acknowledge it as the closest prior method. It regularizes decomposition-based explanations on unedited images; IS instead constrains margin differences across individual and joint edits of named factors, with a target for each pair. We evaluated CDEP using its official loss in our Waterbirds setup, with λ = 10 as in CDEP’s code:

WaterbirdsWGA (%) ↑Corrupted BG-Gap (pp) ↓Only-BG acc. (%) ↓
ERM81.373.691.3
CDEP85.266.291.0
IS89.036.757.5

CDEP improves WGA and its penalty reaches zero during training, but Only-BG agreement stays near ERM’s level: the penalty is met while the model still relies on the background. IS has a smaller corrupted BG-Gap than CDEP on every seed.

W2 / Q2: Target removal and guard weights. Our control removes only the interaction target and keeps all guards, the training schedule and the checkpoint selection; on UrbanCars it reruns the paper’s full schedule with the target weight set to zero.

WaterbirdsWGA (%) ↑Avg. acc. (%) ↑Corrupted BG-Gap (pp) ↓Only-BG acc. (%) ↓
IS without the target88.895.547.363.5
IS89.095.536.757.5
UrbanCarsID acc. (%) ↑BG Gap (pp) ↑CoObj Gap (pp) ↑BG+CoObj Gap (pp) ↑
IS without the target95.9−11.4−3.9−26.5
IS95.1−3.8−4.8−16.6

On Waterbirds, most of the WGA gain remains without the target, and adding it lowers both robustness measures at the same WGA. On UrbanCars, the target narrows the background and combined gaps, at a small cost in ID accuracy and in the co-object gap.

Halving or doubling the guard weights gives spoof FPR95 of 21.2% / 21.8% and Waterbirds corrupted BG-Gap of 35.8 / 35.0 pp (WGA −0.6 / +0.8 points), close to the reported 22.2% and 36.7 pp and below the values without the target (27.3%, 47.3 pp).

We also found that the lam_obj and lam_obj_floor terms listed in Appendix D.2 contribute no gradient, so the reported Waterbirds models were trained with cross-entropy, the interaction penalty and the guard_weight guards; we will correct the objective description.

W3: Full CF-Aug, LLE and when to use IS. Under matched fine-tuning (same ERM checkpoint, same edits, same final 20 epochs), IS and CF-Aug achieve similar WGA (89.0% vs. 88.5%), and IS has a smaller corrupted BG-Gap (36.7 vs. 46.9 pp) and lower Only-BG agreement (57.5% vs. 72.4%). Full CF-Aug (edits for all 50 epochs) achieves higher WGA (92.5%) but also higher Only-BG agreement (70.5%); CF-Aug + IS has a smaller corrupted BG-Gap than Full CF-Aug (32.2 vs. 34.3 pp).

LLE (Li et al., CVPR 2023) achieves a smaller combined gap (−5.9 vs. −16.6 pp) using three last-layer heads and a learned shift classifier; IS steers both interactions within a single classifier and ranks second on this gap in Figure 3.

IS is most useful when the shortcut or desired relation can be named and edited. It can also complement an existing training objective. Across five tested pairings—Waterbirds CF-Aug and GroupDRO, UrbanCars CF-Aug, and PACO PartDrop and outlier exposure—adding IS lowers the mean corrupted BG-Gap or spoof FPR95, with any accuracy decrease of at most 1.3 points. The GroupDRO comparison includes 20 more training epochs, so it does not isolate IS from further training.

W4 / Q3: Practical failure. Encouraging joint evidence from named parts (e.g., a car’s wheels and window/body) addresses false recognition from an isolated familiar part, which we measure by spoof FPR95 with parts pasted onto objects outside the training categories. This tests rejection of unfamiliar objects carrying a familiar part, rather than recognition of unseen categories.

All PACO arms are new runs from the paper’s ERM checkpoints, so values differ from the paper’s tables. Adding the target reduces mean FPR95 from 27.3% to 22.2%, with a reduction on every seed, while Top-1 accuracy remains similar (98.1% without the target and 97.7% with it).

Adding IS to other objectives using the same part removals reduces FPR95 from 33.6% to 26.9% for PartDrop and from 31.1% to 23.8% for outlier exposure (Hendrycks et al., ICLR 2019).

We also evaluate 1,543 held-out PACO photos in which a named part is not visible because of occlusion, truncation, or viewpoint. At each model’s threshold accepting 95% of genuine test objects, acceptance on this subset is nearly unchanged by the target (67.0% with it vs. 66.8% without it).

W5 / W6. Our anonymous repository contains the editing scripts, split and anchor lists, configurations, and instructions for reproducing the evaluations. We will enlarge the labels in Figures 2 and 4 and simplify Table 1.

Reviewer_rFBAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Vision classifiers can exploit any signal from images to make prediction labels. Some of these signals might be relevant, while others may lead to shortcut or spurious learning. These shortcuts can involve foreground–background or object–object interactions, while part–part interactions can provide useful evidence for recognition. The paper proposes Interaction Steering, a learning methodology that mitigates undesirable interactions while encouraging useful ones. The authors propose finite-difference methods based on each of these interactions and add them as auxiliary learning signals alongside the natural-image classification loss. The edits are performed using segmentation masks, object removal, inpainting, and background replacement or masking. Overall, experiments show improvements in robustness using ResNet-50 and DINOv2 ViT-B/14 on Waterbirds and UrbanCars. Experiments on PACO demonstrate increased positive part–part interactions with largely unchanged natural classification accuracy.

Critical strengths
  1. The paper proposes a principled approach as to how to use finite differences to distinguish the contributions of individual visual factors from their interactions. Previous approaches usually suppress background information. The proposed method targets how contextual information effect object evidence while preserving the objectness aspect of the object. This is an interesting and intuitive way of addressing shortcut learning.

  2. The experiments on different datasets actually show the effect of proposed objective on the specific interactions rather than blanket classification metric improvements.

Critical weaknesses
  1. While Waterbirds and UrbanCars evaluate robustness under controlled distribution shifts, the paper does not investigate whether the learned interactions generalize to independent datasets or unseen domains. The experiments still show that interaction steering helps in in-domain learning. To stress test if the effect of short cut learning has been effectively mitigated or part-part interaction is effectively encouraged, evaluations on unseen categories, novel part configurations, or natural occlusions would provide stronger evidence.

  2. The use of image masking and contrasting model outputs has been explored in prior work for VLMs such as Contrastive Region Guidance [1]. Interaction-based regularization has also been studied by Zhang et al. [2], Although Interaction Steering extends these ideas to targeted semantic interactions in vision classifiers, the methodological novelty appears somewhat incremental. A clearer distinction from these approaches would strengthen the contribution.

References

  1. Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training, ECCV, 2024.

  2. Interpreting and Boosting Dropout from a Game-Theoretic View, ICLR, 2021.

Questions for authors
  1. The PACO experiments demonstrate improvements in the measured second-order interactions, but natural classification accuracy remains largely unchanged. Can the authors demonstrate whether encouraging positive interactions leads to better recognition under unseen part configurations or natural occlusions, rather than primarily modifying output margins on the targeted masked configurations?

  2. Methods such as Contrastive Region Guidance (ECCV 2024) demonstrate that contrasting predictions from original and masked images can influence recognition behavior. Can the authors evaluate a simpler first-order masking-based regularization or guidance baseline using the same counterfactual edits? This would help isolate the benefit of explicitly supervising second-order interactions.

  3. The finite-difference measurements assume that the edits isolate the intended visual factors. However, masking, inpainting, and object replacement may introduce additional visual cues that influence the classifier’s outputs. Although the authors evaluate alternative masking strategies, can they further establish whether the measured interactions reflect genuine semantic dependencies rather than artifacts of the counterfactual generation procedure?

  4. Could the authors include a randomized masking baseline, such as an MAE-based approach to investigate whether explicitly steering part–part interactions offers advantages over learning visual relationships through random patch masking?

1 comment

Thank you for your feedback. Our general comment defines the robustness measures used below and provides the code link.

W1 / Q1: Beyond the targeted configurations. On PACO, IS encourages joint evidence from two named parts, such as a car’s wheels and window/body, using training edits that only remove parts. Our spoof evaluation instead pastes one familiar part onto an object outside the classifier’s training categories—a configuration not shown during training—and includes 7,320 pairs. All PACO arms are new runs from the paper’s ERM checkpoints, so values differ from the paper’s tables. PartDrop labels part-removed images with the class; outlier exposure (Hendrycks et al., ICLR 2019) pushes predictions on them toward the uniform distribution.

PACOTop-1 (%) ↑Spoof FPR95 (%) ↓AUROC ↑
Plain fine-tuning97.933.20.920
PartDrop98.033.60.902
PartDrop + IS98.026.90.949
Outlier exposure92.331.10.939
Outlier exposure + IS96.023.80.959
IS without the synergy target98.127.30.949
IS with the sign reversed98.128.80.941
IS with synergy on random regions97.927.70.950
IS97.722.20.960

IS achieves the lowest FPR95 on every seed and the highest mean AUROC.

For unseen backgrounds, we paste Waterbirds test birds onto 2,800 Places365 scenes of 28 types absent from Waterbirds: IS reduces corrupted BG-Gap from ERM’s 25.3 to 17.0 pp, below CF-Aug trained from the same ERM checkpoint (19.5 pp) on every seed.

For naturally missing parts, we evaluate 1,543 held-out PACO photos in which a named part is not visible because of occlusion, truncation, or viewpoint. Encouraging joint evidence could reduce acceptance of these objects, but we observe no decrease from adding the synergy target: at each model’s spoof-test threshold, which accepts 95% of genuine test objects, acceptance on this subset is 67.0% with the target and 66.8% without it. Adding IS to PartDrop likewise preserves acceptance (69.5% vs. 69.4%).

W2: Positioning. CRG (ECCV 2024) is a training-free guidance method for VLMs that contrasts outputs for an image and for a copy with the prompted region blacked out, a first-order contrast. Zhang et al. (ICLR 2021) analyze how dropout suppresses interactions, relate interaction strength to overfitting, and propose an interaction loss based on second-order differences over sampled groups.

IS applies this established finite-difference principle to named image factors, using edits that retain the rest of the image. It specifies a target for each pair: object–context interactions are penalized outside a tolerance around zero, while part–part interactions are encouraged to exceed a synergy threshold. The controls above show that both the named regions and the target’s sign matter: random-region and reversed-sign targets do not reproduce the spoof-detection benefit of the intended target. We will cite both works and clarify how IS builds on them.

Q2: First-order baselines. The W1 table includes two alternatives that supervise each edited image’s output, PartDrop and outlier exposure, and a matched control that removes only the true-class synergy target; this control is not purely first-order because it keeps the wrong-class interaction guard. These alternatives yield FPR95 values of 27–34%. Adding IS to PartDrop or outlier exposure, or restoring the true-class target in the matched control, lowers FPR95 on every seed.

Q3: Artifacts. Random regions edited by the same procedure do not reproduce the named target’s benefit on PACO (27.7% FPR95). On Waterbirds, placebo masks taken from other images leave WGA near ERM’s level (80.4% vs. 80.5%, one seed). We also vary the appearance and location of the pasted part in the spoof evaluation. IS retains its advantage on every seed:

Spoof FPR95 (%) ↓Original pasteHarmonized, featheredMoved to a new position
Plain fine-tuning33.235.029.8
IS without the synergy target27.331.824.9
IS22.227.920.4

Q4: Random masking. First, we replace the named parts with randomly placed regions within the object, retaining the same mask area, fill, and synergy target. This control stays near IS without the target (27.7% FPR95, AUROC 0.950, vs. 22.2% and 0.960 for IS).

Second, we repeat the PACO study on a ViT-B/16 pretrained with MAE random patch masking (He et al., CVPR 2022), using the same IS settings:

PACO, MAE ViT-B/16Top-1 (%) ↑Spoof FPR95 (%) ↓AUROC ↑
Plain fine-tuning98.923.80.927
IS without the synergy target98.813.60.973
IS98.55.70.982

IS further reduces spoof acceptance on every seed, showing that its benefit extends to a backbone pretrained with random patch masking.

Reviewer_4t2TOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Interaction Steering (IS), a framework for explicitly controlling how visual factors jointly influence a model’s predictions. Given predefined interaction factors and suitable editing operations, IS constructs counterfactual images by changing factors individually and jointly, then uses differences of the model’s class margins to measure their interactions. This becomes an auxiliary training objective that regularizes undesirable object–background or object–co-occurring-object interactions, or encourage positive synergy between informative object parts. Experiments on Waterbirds and UrbanCars demonstrate improved worst-group accuracy (WGA), while experiments on PACO show stronger interactions between selected object parts with largely preserved classification accuracy.

Critical strengths

The authors turn counterfactual differences into training objectives that support both (1) suppressing undesirable object–context interactions and (2) encouraging useful part–part synergy. This provides a clear way to specify how selected visual factors jointly influence predictions. The ability to express both behaviors within the same framework is meaningful.

Critical weaknesses
  • The benefit of IS remains unclear. On Waterbirds, Full CF-Aug uses the same masks and generator but outperforms IS in both worst-group accuracy (92.52% vs. 89.04%) and average accuracy (96.45% vs. 95.54%). Although IS achieves a lower interaction magnitude, this demonstrates control of the directly optimized quantity without establishing an additional downstream benefit. A controlled comparison of CF-Aug with and without IS, together with a direct DaC comparison, would help isolate the value of the proposed objective.
  • Task-specific assumptions limit the applicability of the proposed method. IS requires human supervision to specify the relevant factors, their desired relationships, and guard losses. Although these requirements are consistent with its objective, the experiments largely concern curated object-context and object-part interventions. The authors provide limited ablation studies on how misspecified factors or targets affect the proposed method or whether it transfers to less separable attributes such as gender or age related vision cues.
Questions for authors
  • How were the PACO target (K=0.5), the suppression tolerances, and the guard-loss weights selected?
  • What additional value does explicit interaction supervision provide over using the same edits for counterfactual augmentation?

Minor Comments

  • Figure organization and readability could be improved. Figure 2 combines the interaction distribution plots, counterfactual examples, and a quantitative performance table in a dense layout. The plot legends and axis labels are difficult to read at normal viewing size. Additionally, the UrbanCars comparison is presented as a table but captioned “Figure 3”; treating it as a numbered table and updating the corresponding references would improve consistency.
2 comments

Thank you for your feedback. Our general comment defines the robustness measures used below and provides the code link.

Q1: Parameter selection. The values were chosen through manual search during development. We have now added a sensitivity analysis:

SettingPACO spoof FPR95 (%) ↓Waterbirds WGA vs reported (points) ↑Waterbirds corrupted BG-Gap (pp) ↓
Reported22.20.036.7
PACO target K = 0.25 / 1 / 2 (reported 0.5)22.2 / 20.8 / 20.0——
Waterbirds tolerance τ ×0.5 / ×2—0.0 / 0.036.6 / 37.5
Guard weights ×0.5 / ×221.2 / 21.8−0.6 / +0.835.8 / 35.0
No target27.3—47.3

Across these settings, spoof FPR95 and corrupted BG-Gap remain close to the reported values and below those obtained without the target. Increasing the synergy target K slightly lowers spoof acceptance, changing the tolerance τ has little effect, and halving or doubling the guard weights changes WGA by less than one percentage point.

W1 / Q2: CF-Aug with and without IS, and DaC. We first compare IS with CF-Aug under matched fine-tuning and with DaC. We then evaluate whether adding IS to existing methods improves accuracy and the additional robustness measures.

IS fine-tunes a 30-epoch ERM model for 20 epochs, using masks and edits only during fine-tuning. The paper’s Full CF-Aug uses labelled edits throughout all 50 training epochs. For a matched comparison, we trained CF-Aug from the same ERM checkpoint, using the same edits for the same final 20 epochs, with and without IS. We also continued training the paper’s GroupDRO model with IS for 20 additional epochs. All checkpoints are selected by validation WGA, as in the paper:

WaterbirdsWGA (%) ↑Avg. acc. (%) ↑Corrupted BG-Gap (pp) ↓Only-BG acc. (%) ↓
ERM81.393.073.691.3
Full CF-Aug (edits for 50 epochs)92.596.534.370.5
DaC (official code)92.394.25.059.2
IS (edits for the last 20 epochs)89.095.536.757.5
CF-Aug (edits for the last 20 epochs)88.595.946.972.4
CF-Aug + IS (edits for the last 20 epochs)90.196.032.255.6
GroupDRO88.494.850.481.5
GroupDRO + IS (20 more epochs)90.396.029.759.7

Under matched fine-tuning, IS and CF-Aug achieve similar WGA (89.0 vs. 88.5), while IS has a smaller corrupted BG-Gap (36.7 vs. 46.9) and lower Only-BG agreement (57.5 vs. 72.4). Full CF-Aug achieves higher WGA than IS, but also higher Only-BG agreement (70.5 vs. 57.5).

DaC freezes the ERM backbone and retrains the last layer on images combining objects with other images’ backgrounds. In this comparison, it achieves higher WGA and a substantially smaller corrupted BG-Gap than IS. Last-layer retraining is also effective on our checkpoints: DFR (Kirichenko et al., ICLR 2023), fitted on group-balanced validation data, achieves 91.9% WGA on the ERM backbone and 92.5% on the IS backbone, higher on every seed (Appendix I.3), so IS and last-layer retraining can be combined. IS also sets an explicit target for each named pair, so the same objective encourages part–part synergy on PACO. We will include DaC in the revised comparison.

Adding IS to CF-Aug raises WGA from 88.5 to 90.1 while preserving average accuracy, and lowers both corrupted BG-Gap (46.9 → 32.2) and Only-BG agreement (72.4 → 55.6). Continuing GroupDRO training with IS also improves these measures, although this comparison includes 20 more training epochs and so does not isolate IS from further training.

On UrbanCars, adding IS to CF-Aug for 20 epochs lowers corrupted BG-Gap (92.0 → 86.0) and narrows the BG+CoObj gap (−43.8 → −30.7). On PACO, it lowers spoof FPR95 for both PartDrop (33.6 → 26.9) and outlier exposure (Hendrycks et al., ICLR 2019; 31.1 → 23.8). Across these five pairings, the background or part-spoof diagnostic improves, with any accuracy decrease limited to 1.3 percentage points.

W2: Specification. The tested incorrect specifications do not reproduce the benefit of the intended target. On Waterbirds (one seed), placebo masks of similar size taken from other images leave WGA near ERM’s level (80.4% vs. 80.5%, compared with 88.3% for IS). On PACO, spoof FPR95 is 22.2% with synergy on the named parts, 28.8% with the sign reversed, 27.7% with random regions, and 27.3% without the target.

Ground-truth bird masks are not required: predicted masks yield 89.5% WGA with a COCO segmenter and 88.1% with CAM + GrabCut (Appendix E.1). The practitioner must specify the factors and their desired relation. Extending the approach to age or gender would require justified targets and an editor that isolates the intended attribute; we will state these requirements explicitly in the revision.

Minor comments. We will split Figure 2, enlarge its labels and legends, and present the UrbanCars comparison as a numbered table.

I appreciate the authors for additional experiments with clarifications. The matched CF-Aug comparison provides evidence of an incremental benefit from adding IS and the sensitivity/specification controls address several of my concerns.

However, my main concern about the significance of the proposed method remains. DaC achieves higher WGA than IS (92.3% vs. 89.0%) and a much smaller corrupted background gap (5.0 vs. 36.7). CF-Aug + IS also remains below DaC in worst-group accuracy and has a larger corrupted background gap. I also note that the major additional experiments, diagnostic definitions, and discussion have not been incorporated into the revised manuscript. My main concern, however, would remain even with this integration.

For these reasons, I maintain my recommendation

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: robustness measures and code

We thank all reviewers for their time and thoughtful feedback. We begin with two points common to our responses below.

Code. Our anonymous repository contains the code, instructions for reproducing the experiments, and result tables for the paper and these responses: https://anonymous.4open.science/r/Interaction-Steering-E960/

Robustness measures. Worst-group accuracy (WGA) and average accuracy measure predictive performance, but do not fully reveal whether a model relies on spurious context or isolated parts. We therefore report three additional diagnostic measures adapted from the robustness literature:

  • Corrupted BG-Gap: the accuracy difference between matching backgrounds (a waterbird on water) and conflicting backgrounds (a waterbird on land), with only the object blurred, downsampled, or reduced in contrast. These corruptions are adapted from ImageNet-C (Hendrycks & Dietterich, ICLR 2019). The measure combines the background comparisons of Xiao et al. (ICLR 2021), replacing random backgrounds with conflicting ones, with the object-only corruptions of Salient ImageNet (Singla & Feizi, ICLR 2022). On UrbanCars, context includes both the background and the co-occurring object. We interpret the gap alongside predictive accuracy, since a small gap alone does not establish robustness.
  • Only-BG accuracy: the percentage of test images assigned to the background-associated class after the bird is removed by inpainting, following the background-only tests of Xiao et al. (2021).
  • Spoof FPR95: the false-positive rate at 95% true-positive rate (Liang et al., ICLR 2018) on PACO spoofs, constructed by pasting a real object part, such as a car wheel, onto an object outside the classifier’s training categories, such as a box. For each donor class, we set a margin threshold that accepts 95% of genuine test objects and measure the percentage of spoofs that also pass it. We then average over classes. AUROC measures the separation between genuine objects and spoofs across thresholds.

All results are means over three seeds unless otherwise noted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.