Adapting Foundation Models with Mechanistic Evidence

Foundation models present a promising paradigm for producing estimators with broad domain understanding, serving as strong priors that can be adapted to specialized settings. Current practices often rely on scaling generic architectures, domain-specific fine-tuning, and retrieval-augmented workflows. These paradigms are practically feasible given their deployment simplicity and ability to reuse existing infrastructure, but they often struggle to faithfully embody mechanistic domain knowledge and respect known symmetries or conservation laws in structured environments. This work introduces Mechanistic Residual Adaptation (MechRA), a framework that orchestrates the utilization of domain-specific knowledge characterized by expert models during downstream adaptation. A foundation model is used as global signal guidance, while observations from expert models (which demonstrate key invariances, symmetries, and other properties) facilitate mechanistic adjustment towards more plausible, task-specific outputs. We evaluate this approach under various foundation model types and on a diverse set of downstream tasks, and observe the method's ability to learn and orchestrate task-relevant knowledge provided by expert models translates to performance advantages over other fine-tuning methods and task-specific models.