Receiver State Shapes Functional Compatibility: Internal Message Adaptation for Model Stitching

Model stitching connects a frozen source model and a frozen target receiver through a learned interface, with functional compatibility commonly assessed by downstream task accuracy. However, downstream accuracy alone does not fully characterize functional compatibility, as the effect of a stitching mismatch can vary sharply with receiver state. In particular, receiver states can have nearly identical unperturbed accuracy yet respond very differently to the same controlled perturbation. Controlled token and attention interventions identify messages from patch tokens to CLS as a key pathway through which stitching mismatch affects downstream function. Guided by this pathway, we introduce RIMA, Receiver Internal Message Adaptation, which learns state conditioned corrections to CLS attention outputs while keeping the pretrained models and stitch frozen. This receiver state dependence is further validated across model families, stitching directions, interface locations, and objectives. We then evaluate RIMA across heterogeneous model pairs and classification benchmarks against conventional stitch optimization and parameter matched receiver adaptation. On DeiT B to DeiT S, RIMA adds 0.02M trainable parameters and improves ImageNet 1K top 1 accuracy from 68.538% to 73.752%. These results indicate that receiver state shapes the functional impact of stitching mismatch, while internal message adaptation can recover degraded performance.