- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes adaptation compilation: learning from previous adaptations to predict the behavioral outcomes of candidate adaptation configurations before executing them. A predictor estimates acquisition, transfer, boundedness, and preservation, and a downstream utility selects a configuration. The experiments consider four approximately budget-matched LoRA configurations across five learning families, with held-out episode evaluation, alternative utility functions, leave-one-family-out evaluation, and replication on a second model backbone.
Critical strengths
-
Predicting multiple behavioral outcomes rather than only a scalar score allows the same predictions to support different downstream preferences. This is a useful formulation of adaptation selection.
-
The study explicitly separates selection headroom, prediction accuracy, and realized selection utility. Its leave-one-family-out and second-backbone results also make the practical boundaries visible rather than reporting only successful settings.
Critical weaknesses
-
The compiler uses standard regression to predict outcomes for a fixed set of configurations, then selects one using a utility function. Its methodological difference from feature-based algorithm selection is still unclear.
-
The experiments select among only four LoRA configurations, which is narrower than the idea of compiling adaptation programs suggests. The small gain over task-specific defaults on Llama, together with the Gemma and unseen-family results, leaves the practical significance limited.
Questions for authors
-
The paper connects adaptation compilation to per-instance algorithm selection and meta-learning. What is the precise methodological advance beyond learning a multi-output surrogate over task features and candidate configurations, followed by utility-based selection?
-
In the current experiments, the output is one of four fixed LoRA configurations. What does the “program compilation” abstraction contribute beyond configuration selection, and which properties of the proposed framework are actually demonstrated by this fixed-library realization?
-
Given the modest improvement over objective-fixed selection on Llama, the lack of improvement on Gemma, and the unseen-family results, what generalizable new knowledge does the study establish beyond the fact that configuration preferences depend on the task distribution? Please distinguish the contribution that is empirically established from the broader future vision of adaptation compilation.
We thank the reviewer for the thoughtful assessment and agree that the distinction between our demonstrated contributions and the broader vision of adaptation compilation warrants clarification. Changes and additions are highlighted in the revised PDF for ease of review.
. . .
Methodological positioning (Q1). We have clarified in Sec. 3 the relationship to per-instance algorithm selection and surrogate-based optimization. We do not claim a novel regression or selection algorithm. Rather, our contribution is to formulate and empirically investigate whether the multidimensional behavioral consequences of fine-tuning can be predicted before adaptation, and whether these predictions remain useful for selecting adaptation strategies under changing behavioral objectives.
Compilation abstraction (Q2). We have clarified in Sec. 3 that the present implementation compiles learning objectives into executable choices from a fixed library of adaptation programs. This realization demonstrates the separation of behavioral outcome prediction from utility-dependent program selection, but neither synthesizes new programs nor establishes generalization to unseen configurations. We explicitly distinguish these demonstrated properties from the broader compilation abstraction.
Empirical scope (Q3). We have improved Sec. 11 to articulate three findings: (i) adaptation outcomes exhibit predictable structure beyond learning-objective identity within represented regimes; (ii) multidimensional predictions can support program selection across changing utility preferences without retraining; and (iii) selection headroom alone does not guarantee reliable decisions, particularly under backbone or learning-family shifts. We acknowledge the modest gains over strong defaults and the negative Gemma and leave-one-family-out results. Additional direct-selection comparisons in Sec. 7 and Appendix D further clarify when multidimensional prediction provides an empirical advantage.