acceptodds
Under review as a conference paper at ICLR 2027

Adapting Foundation Models with Mechanistic Evidence

Abstract

Foundation models present a promising paradigm for producing estimators with broad domain understanding, serving as strong priors that can be adapted to specialized settings. Current practices often rely on scaling generic architectures, domain-specific fine-tuning, and retrieval-augmented workflows. These paradigms are practically feasible given their deployment simplicity and ability to reuse existing infrastructure, but they often struggle to faithfully embody mechanistic domain knowledge and respect known symmetries or conservation laws in structured environments. This work introduces Mechanistic Residual Adaptation (MechRA), a framework that orchestrates the utilization of domain-specific knowledge characterized by expert models during downstream adaptation. A foundation model is used as global signal guidance, while observations from expert models (which demonstrate key invariances, symmetries, and other properties) facilitate mechanistic adjustment towards more plausible, task-specific outputs. We evaluate this approach under various foundation model types and on a diverse set of downstream tasks, and observe the method's ability to learn and orchestrate task-relevant knowledge provided by expert models translates to performance advantages over other fine-tuning methods and task-specific models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
foundation model, physics-informed ML, simulation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_LhCiOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a new foundation models adaptation framework based on "mechanistic evidence" or "mechanistic guidance". In a nutshell, this consists on integrating, at inference time, domain knowledge about a problem (e.g.,physical information), complementing the knowledge that foundation models acquired through pre-training. The authors experiment mainly with time-series foundation model and tabular foundation models, demosntrating that their technique reduce errors in a variety of scenarios, especially when compared to LoRA.

Critical strengths

First, the problem treated in this paper is interesting and necessary, connecting the fields of parameter efficient finetuning or adaptation and the integration of domain knowledge into a pretrained model. Second, the choice of foundation models (Chronos-2 and TabICL) and applications is relevant and interesting as well, connecting the paper to emerging fields such as time series and tabular foundation models.

Critical weaknesses

While I think the paper goes in an interesting direction, there are several problems with the writing, and especially the technical language used in this paper. Here is a non-exhaustive list of problems,

  • In the caption of figure 2, the authors mention "Context features X=χ(u,y^0)X = \chi(u, \hat{y}_0) initialize the workspace BB". None of these symbols is defined beforehand, and, to make things worse, the notion of workspace is not well defined in this context. The rest of the caption goes into multiple directions with more undefined symbols and terms.

  • Section 3.1: "paired-data-trained bridge", "deterministic experts produce singleton evidence"

  • Section 3.2: LN\text{LN}, which I assume is short for Layer Norm, is never defined. A citation to (Ba et al., 2016) is needed. The notion of "message" is not well defined (around eq. 4). The op [a,b][a, b] is used here without a proper defintiion or comment (for instance, is it concatenation? I assume so...).

  • Section 3.3: qjq_j is never defined. The notion of "slot" is also not well defined (e.g., is it a token?). A lot is described in plain text (e.g., "context reaches the head through initialized slots, and base-output features also enter through the query" should reference an equation).

  • Section 3.4 introduces alternatives to different normalization strategies based on the denominator DikD_{ik} without a proper motivation or explanation as to why these choices matter.

And honestly, many others. Overarching across the whole paper, it seems that a lot of the text has been AI generated, without a careful "human polishing pass", which makes the overall text much more difficult to read than it has to be.

References

Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. "Layer normalization." arXiv preprint arXiv:1607.06450 (2016).

Questions for authors

N/A

1 comment

Thank you for your precise comments regarding the presentation of our paper and pointing to several confusing uses of notation/phrases. We've incorporated each of your suggestions, as well as made a broader pass to help improve clarity and polish the general flow of the paper's presentation. Some passages indeed reflect structure from a writing phase with an AI tool in the loop (as disclosed), and this explains some (although not all) of the artifacts you noted. For instance, Figure 1 was regrettably positioned too early in the paper based on layout constraints, despite the caption having been written assuming a reader would have first visited the methodology section. We nevertheless agree these hurt the presentation of the work, and have incorporated your suggestions, among others, in our updated manuscript.

For larger content additions and experimental extensions, please also see our global comment (also reproduced here for convenience):


We’ve attempted to holistically extend our core evaluations and address the primary concerns raised, including the following key elements:

  • Additional trials for existing environments: for the three primary reported environments, the results tables originally provided mean scores averaged across 5 trials (but standard deviations were omitted due to space constraints). We’ve re-run all evaluations over 10 trials and supply the full tables with standard deviations in Appendix E, with the mean-value tables remaining in the primary paper body.
  • New real-world environments/tasks: while the originally reported environments (sphere, thermal, and bistable) leverage realistic utilities as expert models in their respective domains, each fundamentally use data generated from the provided mechanistic rules/equations. We've added two new environments that form prediction tasks from real-world time series datasets and use capable domain-specific tools: solar for predicting photovoltaic power (using the public PVDAQ dataset), and streamflow for predicting rainfull-runoff patterns (using the public CAMELS-US dataset). Full environment details, expert samples, and method performance are provided in Appendix B.4 and B.5, with results and samples in Appendix E.2 and E.3.
  • Additional method variants: in the new set of reported trials in the primary paper body, we additionally include plain pooled expert methods and detailed bypass methods that were not included in the original tables. The thermal and bistable (as well as the new solar and streamflow) environments additionally report direct expert evaluations fitted to historical data, providing more direct comparisons to the baseline expert signal across the variable tested regimes.
  • Extra details for tasks and expert models: in Appendix B, we've provided several new types of figures that help contextualize the environment data and sample behavior from contributing expert models. These show, e.g., the prediction context and measurement horizons used across regimes for trajectory forecasting, as well key properties/sample rollouts of fitted expert models used during adaptation.
  • Ablations for the primary adapter method: in Appendix F, we provide a large set of controlled ablations for the proposed method measured across the primary sphere, thermal, and bistable tasks. These tables show performance changes when disabling specific module components in a trained model, helping reveal the relative impact of granular pieces of the adapter and its robustness to their removal. This can be contrasted with the primary performance tables, which themselves provide several method variants, each of which are trained from scratch in independent sessions.
  • Writing improvements: the updated manuscript polishes several local writing choices to help improve clarity and the natural flow of the document (although the core structure and layout remains virtually the same). We additionally include new references and expand acronyms, as suggested by reviewers.
Reviewer_CQuFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

MechRA adapts a frozen predictor using context-conditioned outputs from mechanistic experts. Expert encoders produce tokens that write gated messages into a shared latent workspace, and an output-query decoder predicts an additive correction to the base forecast. Optional bypasses expose expert information directly to the decoder. Experiments on synthetic geometric, thermal, and stochastic systems compare several adaptation and routing variants under controlled expert mismatch.

Critical strengths
  1. The evidence interface respects how mechanistic tools are actually available. A simulator may provide informative outputs without exposing differentiable internals. Encoding its case-specific outputs and learning their relevance is therefore a practical alternative to requiring gradients through the simulator. Fixed observation maps and the exclusion of future targets from evidence preparation make the intended information boundary clear.
  2. Residual adaptation provides a reasonable way to combine a prior predictor with imperfect specialists. The base supplies an initial forecast while expert evidence can inform corrections in regimes where it is useful. Controlled expert mismatch makes this more meaningful than assuming every simulator is correct. The full-model improvements on SPHERE and THERMAL show that the complete integration pipeline can exploit complementary information.
Critical weaknesses
  1. The results do not establish that the shared workspace provides the claimed integration benefit. The full decoder can access evidence through a bypass, so its improvement does not by itself demonstrate that localized writes and workspace fusion are needed. In THERMAL, the canonical expert route has overall RMSE 1.297, worse than 1.235 for the no-expert adapter, while the full model reaches 0.956. Tables 1-3 do not report the described pooled and bypass-only comparisons. A learned gate also does not inherently represent expert reliability; ordinary task loss may encourage reliance on a misleading expert within the training distribution. Matched route comparisons and controlled expert corruption would directly test the proposed selective-integration mechanism.
  2. The role of the foundation-model prior is not separated from learning a task predictor from experts. An expressive additive residual can replace much of the base forecast, so freezing the base does not guarantee that its prior remains important in the final prediction. Context and expert outputs may already provide enough information to solve these synthetic tasks. A predictor receiving the same information but no foundation-model output is needed to establish the value of foundation-model adaptation rather than supervised expert fusion alone. Likewise, mechanistic inputs do not enforce physical consistency in an unconstrained residual head; predictive error and physical validity should remain distinct claims, as the paper's own limitations acknowledge.
Questions for authors
  1. Evidence affecting the recommendation: With the same expert tokens, context inputs, and comparable capacity, how do full, workspace-only, bypass-only, and pooled models compare? In particular, what explains the THERMAL improvement from the workspace configurations to the full model?
  2. Evidence affecting the recommendation: How does a context-plus-experts predictor without foundation-model outputs compare with MechRA and with the direct expert forecasts? Is the pretrained prediction necessary for the reported gains, or does it mainly serve as an initial offset that the residual learns to replace?
  3. Evidence affecting the recommendation: When an expert becomes misleading or unavailable, do the gates suppress its influence and preserve useful predictions? Please distinguish changes to expert quality from removing a route after joint training. Existing mismatch interventions, together with radial or thermal consistency measures, would help establish what the learned integration actually achieves beyond lower prediction error.
1 comment

Thank you for your time and effort providing feedback on our work. Please find our responses to your questions and comments below. We'd like to additionally refer you to our global reply, which provides a high-level summary of all core changes made to the updated manuscript.


W1, Q1: Regarding the original 1.297 (FM+MechRA (experts)) versus 1.235 (FM+MechRA (no experts)) scores mentioned on the thermal task: it's worth noting neither of these method variants constitute the full proposed method. The reliability of experts in a given environment is subject to a high degree of variation, and without the learned workspace components, for instance, to integrate (and sometimes ignore) expert signal more carefully, having no expert signal can prove better when they otherwise lead the simple adapter astray. In this setting we saw, and continue to see in the newly reported values, that the full MechRA pipeline outperforms both of these variants, indicating simple expert presence may not, on its own, yield performance benefits even when the full adaptation can make use of their signal.

Regarding the bypass variants: in our updated manuscript, the primary tables have been regenerated to include a larger set of trials and methods, including the described bypass methods to allow for direct comparisons (which use the same expert tokens and context inputs, and match architectural capacity). Further, the full MechRA method does not have a direct evidence bypass: it can only access signal from experts through writes into the workspace. Only the bypass-specific tested variants have simple, direct pass-through of expert samples.

W2, Q2, Q3: Regarding the sufficiency of the expert models: in the updated manuscript, the thermal and bistable (as well as the new solar and streamflow; please see the global comment) environments now report direct expert evaluations fitted to historical data, providing more direct comparisons to the baseline expert signal across the variable tested regimes. In addition, Appendix B now supplies detailed figures and analysis of expert formulations, assumptions, and several sample roll-outs on environment context. For instance, the results from the thermal environment now provide a baseline ridge regression learner optimized over expert input. This can be contrasted with the MechRA (pooled bypass), which also allows a direct means of passing expert information to the downstream residual head. These results point toward there being available utility from both the foundation model prior and the expert models, but only when they can be properly coordinated. Alone, e.g., with Ridge regression or FM raw, the raw signal from either source may indeed not be demonstrably useful for final prediction, but here the full MechRA pipeline demonstrates an ability to leverage and contextualize both for significantly improved performance across regimes. As you mention, we additionally do not make the claim that this leads to a formal, physically consistent adjustment, but more a coordinated alignment of expert guidance when it proves useful for structured tasks.

In addition to providing cross-regime performance to test incorporation of experts (even when unreliable), specific architectural interventions have been added to Appendix F. These are controlled ablations that test the role of introduced mechanisms (like gates, bypasses, etc) on a given model after training, indicating to what extent their presence or omission impacts performance.

Reviewer_gb57OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MechRA adapts a frozen predictor by integrating mechanistic expert outputs through a gated latent workspace and an optional evidence bypass. Evaluation covers three synthetic forecasting environments.

Critical strengths

The interface accommodates heterogeneous, non-differentiable experts without modifying the backbone. The full model gives useful improvements. The paper also explicitly distinguishes expert evidence from guaranteed physical consistency.

Critical weaknesses
  1. The workspace’s added value is unclear. Pooled, bypass-only, reconstruction, and expert-removal results are absent from the supplied PDF. Without matched controls, improvements cannot be separated from direct evidence access or additional capacity, leaving the advantage over simpler fusion unresolved.

  2. Expert benefits are inconsistent across tasks and integration routes. Canonical MechRA worsens THERMAL RMSE relative to the no-expert adapter (1.297 versus 1.235). Full MechRA improves BISTABLE RMSE only from 0.232 to 0.229 (Tables 2–3). A matched model without FM predictions is also needed to establish the backbone’s contribution.

  3. Statistical and implementation details are missing. Tables 1–3 omit seed counts and uncertainty estimates. Appendix D describes bootstrapping without reporting intervals; checkpoint identifiers and native LoRA settings are unspecified. Small gains and baseline competitiveness are difficult to assess.

  4. Practical evidence remains limited. The synthetic tasks test controlled expert mismatch, but use fixed observation maps and, for BISTABLE, a prescribed grid posterior. Generalization beyond these settings and total inference costs, including expert evidence generation, remain untested or unreported.

Questions for authors
  1. Can you report matched pooled/bypass-only controls and an expert-based model without FM predictions?
  2. Can you provide seed-level results, paired intervals, and complete checkpoint/LoRA settings?
  3. How do performance and total inference cost change beyond the constructed synthetic settings?
1 comment

Thank you for your time and effort providing detailed feedback on our work. Please find our responses to your questions and comments below. We'd like to additionally refer you to our global reply, which provides a high-level summary of all core changes made to the updated manuscript.


W1: We have added detailed ablations for the primary sphere, thermal, and bistable tasks in Appendix F for the proposed method. These tables show performance (RMSE) changes when disabling specific module components in trained models, helping reveal the relative impact of the adapter's fundamental design and its robustness to component removal after optimization has taken place.

We'd like to additionally note that these tables are meant to complement the primary performance tables reported in the main paper body, which themselves compare several method variants covering your named interactions (pooled experts, bypass routes, presence or omission of experts), where each method variant is trained from scratch in independent sessions. The controlled ablations in Appendix F actively disable components after learning, which helps reveal how coordinated modules remain after removal. The primary tables instead provide insight into the fundamental ability for method variants to recover the behavior of the full architecture without the presence of certain components.

W2, Q2: In our updated manuscript, the primary tables have been regenerated to include a larger set of trials and methods, including your requested expert-only reference points and stronger evidence for MechRA's performance. With the now-reported 10 trial averages (with full trial standard deviations additionally provided in Appendix F), the thermal task shows MechRA outperforming all competing methods across metrics and regimes. For the bistable environment, sensitivities are generally smaller, leading generally to lower overhead for gains. Here, the full MechRA pipeline underperforms marginally relative to its no-expert variant on the high noise regime due the unreliable expert signal, but otherwise maintains consistent outperformance on other tested regimes.

Regarding the original 1.297 (FM+MechRA (experts)) versus 1.235 (FM+MechRA (no experts)) scores mentioned on the thermal task: it's worth noting neither of these method variants constitute the full proposed method. The reliability of experts in a given environment is subject to a high degree of variation, and without the learned workspace components, for instance, to integrate (and sometimes ignore) expert signal more carefully, having no expert signal can prove better when they otherwise lead the simple adapter astray. In this setting we saw, and continue to see in the newly reported values, that the full MechRA pipeline outperforms both of these variants, indicating simple expert presence may not, on its own, yield performance benefits even when the full adaptation can make use of their signal.

W3: As mentioned in W2, we have re-run all primary results (originally reported only as means of 5 trials) over 10 trials, and report the full standard deviations in Appendix E.

W4, Q1, Q3: To address limitations of the synthetic benchmarks, we've added two new real-world environments that constitute prediction tasks from real-world time series datasets, and include multiple real domain-specific tools. The new environments are 1) solar for predicting photovoltaic power (using the public PVDAQ dataset), and 2) streamflow for predicting rainfull-runoff patterns (using the public CAMELS-US dataset). Full environment details along with samples from experts and method results are provided in Appendix B.4 and B.5.

Note that we also provide direct expert comparisons in core task tables, evaluating the performance of the context-fitted analytical models and simulations. We additionally provide a large set of new environment and expert details and figures in Appendix B, highlighting their core mechanical assumptions and showing sample roll-outs. Please additionally see our global comment for a full summary of all manuscript changes and new figures.

Reviewer_RbxiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed using a smaller adapter to fit the residual of the foundation model given the expert input.

Critical strengths
  1. The design is intuitive.
Critical weaknesses
  1. Missing component ablation, not sure how those things contribute to the final performance.

  2. Not sure fitting residual is better than do the foundation model and expert model ensemble.

  3. Limited evaluation. only synthetic dataset.

Questions for authors

NAN.

1 comment

Thank you for your time and effort providing feedback on our work. Please find our responses to your questions and comments below. We'd like to additionally refer you to our global comment, which provides a high-level summary of all core changes made to the updated manuscript.


W1: We have added detailed ablations for the primary sphere, thermal, and bistable tasks in Appendix F. These tables show performance (RMSE) changes when disabling specific module components in trained models, helping reveal the relative impact of the adapter's fundamental design and its robustness to component removal. We'd like to note that these tables are meant to complement the primary performance tables reported in the main paper body, which themselves provide several method variants (with different components omitted or swapped out), each of which are trained from scratch in independent sessions.

W2: In the original performance tables, the "FM (experts)" entries effectively constitute FM-native integration of expert samples (formally treated as covariates alongside the canonical history context). Ordinary fine-tuning then allows predictions/roll-outs from both the foundation model and expert models to be trivially combined in an ensemble-style manner, although more sophisticated schemes are supported. In our updated manuscript, we additionally supply an even more direct ensemble-style analog with the "FM+pooled experts" row in the newly generated tables, which constitutes a simple learned combination of FM and expert outputs for prediction. This entry remains fairly competitive across tasks, especially when experts are highly predictive of the task output, but it is consistently outperformed by more sophisticated fine-tuning methods, including the proposed MechRA method, which allows for more nuanced interaction of FM prior knowledge with experts.

W3: Thank you for this suggestion, we agreed our paper would benefit from more grounded, real-world environments. While the originally reported environments (sphere, thermal, and bistable) leverage realistic utilities as expert models in their respective domains, each fundamentally use data generated from the described mechanistic rules/equations. We've added two new real-world environments that constitute prediction tasks from real-world time series datasets, and include multiple real domain-specific tools. The new environments are 1) solar for predicting photovoltaic power (using the public PVDAQ dataset), and 2) streamflow for predicting rainfull-runoff patterns (using the public CAMELS-US dataset). Full environment details along with samples from experts and method results are provided in Appendix B.4 and B.5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.