- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes a new foundation models adaptation framework based on "mechanistic evidence" or "mechanistic guidance". In a nutshell, this consists on integrating, at inference time, domain knowledge about a problem (e.g.,physical information), complementing the knowledge that foundation models acquired through pre-training. The authors experiment mainly with time-series foundation model and tabular foundation models, demosntrating that their technique reduce errors in a variety of scenarios, especially when compared to LoRA.
Critical strengths
First, the problem treated in this paper is interesting and necessary, connecting the fields of parameter efficient finetuning or adaptation and the integration of domain knowledge into a pretrained model. Second, the choice of foundation models (Chronos-2 and TabICL) and applications is relevant and interesting as well, connecting the paper to emerging fields such as time series and tabular foundation models.
Critical weaknesses
While I think the paper goes in an interesting direction, there are several problems with the writing, and especially the technical language used in this paper. Here is a non-exhaustive list of problems,
-
In the caption of figure 2, the authors mention "Context features initialize the workspace ". None of these symbols is defined beforehand, and, to make things worse, the notion of workspace is not well defined in this context. The rest of the caption goes into multiple directions with more undefined symbols and terms.
-
Section 3.1: "paired-data-trained bridge", "deterministic experts produce singleton evidence"
-
Section 3.2: , which I assume is short for Layer Norm, is never defined. A citation to (Ba et al., 2016) is needed. The notion of "message" is not well defined (around eq. 4). The op is used here without a proper defintiion or comment (for instance, is it concatenation? I assume so...).
-
Section 3.3: is never defined. The notion of "slot" is also not well defined (e.g., is it a token?). A lot is described in plain text (e.g., "context reaches the head through initialized slots, and base-output features also enter through the query" should reference an equation).
-
Section 3.4 introduces alternatives to different normalization strategies based on the denominator without a proper motivation or explanation as to why these choices matter.
And honestly, many others. Overarching across the whole paper, it seems that a lot of the text has been AI generated, without a careful "human polishing pass", which makes the overall text much more difficult to read than it has to be.
References
Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. "Layer normalization." arXiv preprint arXiv:1607.06450 (2016).
Questions for authors
N/A
Thank you for your precise comments regarding the presentation of our paper and pointing to several confusing uses of notation/phrases. We've incorporated each of your suggestions, as well as made a broader pass to help improve clarity and polish the general flow of the paper's presentation. Some passages indeed reflect structure from a writing phase with an AI tool in the loop (as disclosed), and this explains some (although not all) of the artifacts you noted. For instance, Figure 1 was regrettably positioned too early in the paper based on layout constraints, despite the caption having been written assuming a reader would have first visited the methodology section. We nevertheless agree these hurt the presentation of the work, and have incorporated your suggestions, among others, in our updated manuscript.
For larger content additions and experimental extensions, please also see our global comment (also reproduced here for convenience):
We’ve attempted to holistically extend our core evaluations and address the primary concerns raised, including the following key elements:
sphere,thermal, andbistable) leverage realistic utilities as expert models in their respective domains, each fundamentally use data generated from the provided mechanistic rules/equations. We've added two new environments that form prediction tasks from real-world time series datasets and use capable domain-specific tools:solarfor predicting photovoltaic power (using the public PVDAQ dataset), andstreamflowfor predicting rainfull-runoff patterns (using the public CAMELS-US dataset). Full environment details, expert samples, and method performance are provided in Appendix B.4 and B.5, with results and samples in Appendix E.2 and E.3.thermalandbistable(as well as the newsolarandstreamflow) environments additionally report direct expert evaluations fitted to historical data, providing more direct comparisons to the baseline expert signal across the variable tested regimes.sphere,thermal, andbistabletasks. These tables show performance changes when disabling specific module components in a trained model, helping reveal the relative impact of granular pieces of the adapter and its robustness to their removal. This can be contrasted with the primary performance tables, which themselves provide several method variants, each of which are trained from scratch in independent sessions.