WHEN CONTEXT HELPS: RECOGNIZER-RELATIVE BENEFIT INTERFACES

Visual context is not intrinsically helpful: the same surrounding evidence can reduce one recognizer’s loss and increase another’s. We study what can transfer when this per-example benefit target changes with the recognizer and no target benefit labels are available. The answer is not a universal gate. A matched source router trained directly on routed NLL transfers a substantially worse benefit ordering than explicit benefit supervision, including on unseen supervised CNNs. More importantly, under one frozen CLIP-trained decoder and a fixed target benefit event, the target recognizer’s own label-free response relations are the highest-AUC evidence origin in all 20 non-source target–dataset cases across four datasets. This is not built into the target definition: the interface contains neither the ground-truth label nor the target correct-class probability, and the decoder and normalization are learned only on the source. Finally, target responses can remain predictively useful where the frozen source interpretation fails: structural relations improve targetsupervised prediction on ImageNet-9 while the same source-learned augmentation degrades it. These separations identify a recognizer-relative benefit interface: portable response coordinates need not imply a portable decoder or decision rule.