- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Interaction Steering to control selected visual interactions. Finite differences across edited images measure how named factors affect model scores together. Auxiliary losses suppress unwanted interactions or encourage selected part interactions. The main classification loss stays on natural images. Inference uses only the classifier. Experiments show robustness gains on Waterbirds and UrbanCars, and targeted interaction changes on PACO.
Critical strengths
-
The same interaction framework supports both suppression and enhancement. It gives users a direct way to specify which visual factors should interact. This is a useful methodological contribution, even though the mathematical tools are established.
-
Waterbirds worst-group accuracy rises from 81.26% to 89.04% on ResNet-50. DINOv2 improves from 89.55% to 94.81%. Joint steering reduces all UrbanCars shortcut gaps relative to ERM. Alternative mask tests further strengthen the evidence.
Critical weaknesses
-
finite difference and explanation-based training are both pretty mature methods, and CDEP can also constrain feature interactions, but why not use CDEP? not clear what extra value the proposed method has, novelty also needs clearer sorting out.
-
the matched PACO control is useful, but Waterbirds and UrbanCars lack similar control, still unclear how separating the target loss contributes to robustness and to sensitivity of the guard weights, also some components’ effects still unclear.
-
Full CF-Aug can improve Waterbirds accuracy, while IS lowers the measured interaction score, these trade-offs are useful, but when to choose IS still not very clear.
-
PACO shows controllable interaction changes, but the target loss doesn’t bring clear extra accuracy gain. it only covers five annotated categories and controlled removal, natural occlusion and unseen categories not tested yet. these limitations narrow the applicability.
Questions for authors
-
Which capabilities distinguish IS from interaction penalties such as CDEP?
-
Can you add matched controls on Waterbirds and UrbanCars that remove only the target loss? How sensitive are the results to guard weights?
-
Which practical failures should stronger part interactions help resolve? The authors should show a test with natural occlusion or unseen classes.
Thank you for your feedback. Our general comment defines the robustness measures used below and provides the code link.
W1 / Q1: CDEP. CDEP can constrain feature interactions, and we will acknowledge it as the closest prior method. It regularizes decomposition-based explanations on unedited images; IS instead constrains margin differences across individual and joint edits of named factors, with a target for each pair. We evaluated CDEP using its official loss in our Waterbirds setup, with λ = 10 as in CDEP’s code:
CDEP improves WGA and its penalty reaches zero during training, but Only-BG agreement stays near ERM’s level: the penalty is met while the model still relies on the background. IS has a smaller corrupted BG-Gap than CDEP on every seed.
W2 / Q2: Target removal and guard weights. Our control removes only the interaction target and keeps all guards, the training schedule and the checkpoint selection; on UrbanCars it reruns the paper’s full schedule with the target weight set to zero.
On Waterbirds, most of the WGA gain remains without the target, and adding it lowers both robustness measures at the same WGA. On UrbanCars, the target narrows the background and combined gaps, at a small cost in ID accuracy and in the co-object gap.
Halving or doubling the guard weights gives spoof FPR95 of 21.2% / 21.8% and Waterbirds corrupted BG-Gap of 35.8 / 35.0 pp (WGA −0.6 / +0.8 points), close to the reported 22.2% and 36.7 pp and below the values without the target (27.3%, 47.3 pp).
We also found that the lam_obj and lam_obj_floor terms listed in Appendix D.2 contribute no gradient, so the reported Waterbirds models were trained with cross-entropy, the interaction penalty and the guard_weight guards; we will correct the objective description.
W3: Full CF-Aug, LLE and when to use IS. Under matched fine-tuning (same ERM checkpoint, same edits, same final 20 epochs), IS and CF-Aug achieve similar WGA (89.0% vs. 88.5%), and IS has a smaller corrupted BG-Gap (36.7 vs. 46.9 pp) and lower Only-BG agreement (57.5% vs. 72.4%). Full CF-Aug (edits for all 50 epochs) achieves higher WGA (92.5%) but also higher Only-BG agreement (70.5%); CF-Aug + IS has a smaller corrupted BG-Gap than Full CF-Aug (32.2 vs. 34.3 pp).
LLE (Li et al., CVPR 2023) achieves a smaller combined gap (−5.9 vs. −16.6 pp) using three last-layer heads and a learned shift classifier; IS steers both interactions within a single classifier and ranks second on this gap in Figure 3.
IS is most useful when the shortcut or desired relation can be named and edited. It can also complement an existing training objective. Across five tested pairings—Waterbirds CF-Aug and GroupDRO, UrbanCars CF-Aug, and PACO PartDrop and outlier exposure—adding IS lowers the mean corrupted BG-Gap or spoof FPR95, with any accuracy decrease of at most 1.3 points. The GroupDRO comparison includes 20 more training epochs, so it does not isolate IS from further training.
W4 / Q3: Practical failure. Encouraging joint evidence from named parts (e.g., a car’s wheels and window/body) addresses false recognition from an isolated familiar part, which we measure by spoof FPR95 with parts pasted onto objects outside the training categories. This tests rejection of unfamiliar objects carrying a familiar part, rather than recognition of unseen categories.
All PACO arms are new runs from the paper’s ERM checkpoints, so values differ from the paper’s tables. Adding the target reduces mean FPR95 from 27.3% to 22.2%, with a reduction on every seed, while Top-1 accuracy remains similar (98.1% without the target and 97.7% with it).
Adding IS to other objectives using the same part removals reduces FPR95 from 33.6% to 26.9% for PartDrop and from 31.1% to 23.8% for outlier exposure (Hendrycks et al., ICLR 2019).
We also evaluate 1,543 held-out PACO photos in which a named part is not visible because of occlusion, truncation, or viewpoint. At each model’s threshold accepting 95% of genuine test objects, acceptance on this subset is nearly unchanged by the target (67.0% with it vs. 66.8% without it).
W5 / W6. Our anonymous repository contains the editing scripts, split and anchor lists, configurations, and instructions for reproducing the evaluations. We will enlarge the labels in Figures 2 and 4 and simplify Table 1.