acceptodds
Under review as a conference paper at ICLR 2027

Compiling Learning Problems into Adaptation Programs for Language Models

Abstract

Model adaptation is typically governed by a fixed recipe, even though different update programs can produce substantially different behavioral outcomes. We introduce adaptation compilation, which reframes where, how, and to what extent a model should adapt as a joint prediction and decision problem. Rather than searching over candidate programs anew for each learning episode, a compiler learns from prior adaptations to predict a vector-valued counterfactual response surface over candidate programs—their expected effects on acquisition, transfer, boundedness, and preservation—and selects a program before adaptation begins. Because this predicted geometry captures multiple behavioral consequences rather than a single winner or scalar score, it can be reused under different downstream priorities without retraining. Across five learning types, preferred programs vary meaningfully across episodes, and this variation is predictable from pre-adaptation information. On Llama-3.1-8B, compiler-selected programs approach exhaustive search while outperforming global and objective-specific defaults. Replication on Gemma-2-9B preserves program heterogeneity and selection headroom, but shows that exploiting this headroom requires accounting for uncertainty when departing from strong defaults. Together, these results show that adaptation search can be amortized across related learning problems, turning prior adaptation experience into a basis for deciding how future learning should occur.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language models, adaptation, parameter-efficient fine-tuning, LoRA, meta-learning, program selection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mRmsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes adaptation compilation: learning from previous adaptations to predict the behavioral outcomes of candidate adaptation configurations before executing them. A predictor estimates acquisition, transfer, boundedness, and preservation, and a downstream utility selects a configuration. The experiments consider four approximately budget-matched LoRA configurations across five learning families, with held-out episode evaluation, alternative utility functions, leave-one-family-out evaluation, and replication on a second model backbone.

Critical strengths
  1. Predicting multiple behavioral outcomes rather than only a scalar score allows the same predictions to support different downstream preferences. This is a useful formulation of adaptation selection.

  2. The study explicitly separates selection headroom, prediction accuracy, and realized selection utility. Its leave-one-family-out and second-backbone results also make the practical boundaries visible rather than reporting only successful settings.

Critical weaknesses
  1. The compiler uses standard regression to predict outcomes for a fixed set of configurations, then selects one using a utility function. Its methodological difference from feature-based algorithm selection is still unclear.

  2. The experiments select among only four LoRA configurations, which is narrower than the idea of compiling adaptation programs suggests. The small gain over task-specific defaults on Llama, together with the Gemma and unseen-family results, leaves the practical significance limited.

Questions for authors
  1. The paper connects adaptation compilation to per-instance algorithm selection and meta-learning. What is the precise methodological advance beyond learning a multi-output surrogate over task features and candidate configurations, followed by utility-based selection?

  2. In the current experiments, the output is one of four fixed LoRA configurations. What does the “program compilation” abstraction contribute beyond configuration selection, and which properties of the proposed framework are actually demonstrated by this fixed-library realization?

  3. Given the modest improvement over objective-fixed selection on Llama, the lack of improvement on Gemma, and the unseen-family results, what generalizable new knowledge does the study establish beyond the fact that configuration preferences depend on the task distribution? Please distinguish the contribution that is empirically established from the broader future vision of adaptation compilation.

1 comment

We thank the reviewer for the thoughtful assessment and agree that the distinction between our demonstrated contributions and the broader vision of adaptation compilation warrants clarification. Changes and additions are highlighted in the revised PDF for ease of review.

. . .

Methodological positioning (Q1). We have clarified in Sec. 3 the relationship to per-instance algorithm selection and surrogate-based optimization. We do not claim a novel regression or selection algorithm. Rather, our contribution is to formulate and empirically investigate whether the multidimensional behavioral consequences of fine-tuning can be predicted before adaptation, and whether these predictions remain useful for selecting adaptation strategies under changing behavioral objectives.

Compilation abstraction (Q2). We have clarified in Sec. 3 that the present implementation compiles learning objectives into executable choices from a fixed library of adaptation programs. This realization demonstrates the separation of behavioral outcome prediction from utility-dependent program selection, but neither synthesizes new programs nor establishes generalization to unseen configurations. We explicitly distinguish these demonstrated properties from the broader compilation abstraction.

Empirical scope (Q3). We have improved Sec. 11 to articulate three findings: (i) adaptation outcomes exhibit predictable structure beyond learning-objective identity within represented regimes; (ii) multidimensional predictions can support program selection across changing utility preferences without retraining; and (iii) selection headroom alone does not guarantee reliable decisions, particularly under backbone or learning-family shifts. We acknowledge the modest gains over strong defaults and the negative Gemma and leave-one-family-out results. Additional direct-selection comparisons in Sec. 7 and Appendix D further clarify when multidimensional prediction provides an empirical advantage.

Reviewer_7HDVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces adaptation compilation, a framework for predicting which fine-tuning configuration will work best for a given learning task. Rather than using a fixed LoRA configuration or searching over configurations for every new task, the authors train a predictor on previous adaptation experiments to estimate the outcomes of different configurations. These outcomes include acquisition, transfer, boundedness, and preservation, allowing configurations to be selected according to different priorities without retraining the predictor.

The authors evaluate four LoRA configurations across five synthetic learning tasks, using Llama-3.1-8B and Gemma-2-9B. On Llama, the predictor accurately estimates adaptation outcomes and selects configurations that outperform fixed defaults, approaching the performance of exhaustive search. The authors also find that representations from the frozen model are nearly sufficient for predicting adaptation outcomes. However, the method generalizes poorly to entirely unseen task types and fails to improve over fixed baselines on Gemma.

Critical strengths

(1) The paper presents an interesting approach to learning how a model should be adapted. Rather than predicting a single optimal configuration, the authors predict how each configuration will affect several aspects of model behavior, since the best configuration may depend on whether we care more about acquisition, transfer, boundedness, or preservation.

(2) The experiments demonstrate that adaptation outcomes can be predicted accurately using relatively inexpensive information, suggesting that the consequences of different adaptation configurations are predictable from information available before training.

Critical weaknesses

(1) The experiments are too limited to establish how broadly the method applies. The authors evaluate only four LoRA configurations and five synthetic learning tasks. Although the proposed framework is general, the experiments only demonstrate that a predictor can choose among a small, fixed set of configurations. It is unclear whether the approach would work with a larger space of possible layer placements, different LoRA ranks, or more realistic fine-tuning tasks. Take the results in Section 9 for example. When an entire task family is excluded from training, prediction error increases from 0.040 to 0.339, and the compiler performs worse than the global fixed baseline. This result suggests that the method is learning patterns specific to the task families in its training data, rather than more general relationships between learning problems and adaptation strategies.

(2) The gains over strong baselines are modest, and the method does not consistently improve performance across models. For example, on Llama, choosing a fixed configuration for each task type already achieves a mean utility of 0.600, compared with 0.618 for the exhaustive oracle. The compiler improves this to 0.607, an absolute gain of only 0.007. Of the 100 test episodes, the compiler improves performance on 22, makes it worse on 7, and produces the same result on the remaining 71. And the results on Gemma are more concerning. Here, the compiler achieves a mean utility of 0.435, compared with 0.438 for the fixed baseline. The authors attribute this to unreliable decisions based on small predicted differences between configurations. This result calls into question whether the method can reliably exploit the variation in optimal configurations that motivates the paper.

Questions for authors

(1) How does the method compare with directly predicting the best adaptation configuration? For example, could the authors train a classifier or ranking model using the same episode representations and adaptation records? This would help establish whether predicting the full behavioral outcome vector improves selection, rather than just providing the flexibility to change utility functions afterward.

(2) Can the predictor generalize to configurations it has not seen during training? The experiments train and evaluate on the same four LoRA configurations. It would be informative to exclude one configuration during training and test whether the predictor can estimate its performance, or to evaluate on a larger set of configurations.

(3) Can the authors test whether a simple confidence threshold improves performance on Gemma? The paper identifies small, unreliable predicted advantages as the main reason the compiler sometimes performs worse than fixed defaults. It seems possible that requiring a minimum predicted improvement before switching configurations could address much of this problem.

(4) How does the computational cost compare with ordinary configuration search? Constructing the training and validation datasets requires approximately 2,000 adaptation runs per backbone. How many subsequent adaptation tasks would be needed to recover this cost? A comparison of the total computation required by compilation and conventional search would help clarify when the method is practically worthwhile.

2 comments

We thank the reviewer for the constructive feedback and agree that the reliability and computational practicality of adaptation compilation warrant further examination. Changes and additions are highlighted in the revised PDF for ease of review.

. . .

Direct selection baselines (Q1). We have expanded Sec. 7 and Appendix D.4-D.6 to compare geometry prediction with direct program classification and scalar-utility regression using matched pre-adaptation features, historical adaptation records, and episode partitions. These analyses include utility-conditioned alternatives, held-out preference specifications, paired statistical comparisons, and robustness experiments varying preference-training coverage. The cross-backbone comparisons in Sec. 10 and Appendix G further clarify when multidimensional prediction offers an advantage over direct selection.

Generalization to unseen configurations (Q2). We conducted a leave-one-configuration-out experiment on Llama, withholding each adaptation program entirely from predictor training and validation (Sec. 9; Appendix F). Predictions for excluded programs improve upon a training-only mean baseline, indicating partial generalization through structural program representations. However, prediction errors increase substantially, and admitting unseen programs reduces selection utility on average. Additional outcome-level diagnostics identify configuration-dependent prediction biases that contribute to harmful selections. These findings distinguish generalization to new episodes within a represented program library from the more challenging problem of selecting previously unseen configurations.

Conservative switching on Gemma (Q3). We evaluated a minimum predicted-utility improvement threshold, selected using validation episodes (Sec. 10; Appendix G.8). This reduces harmful program switches and improves realized utility relative to the original compiler, bringing performance approximately in line with objective-fixed selection. We additionally report paired confidence intervals, per-objective analyses, and threshold-sensitivity diagnostics. We clarify that the threshold represents a predicted-utility margin rather than calibrated confidence and does not establish superiority over fixed defaults (Sec. 11.3).

Computational efficiency (Q4). We have added Appendix H, reporting measured feature-extraction, adaptation, evaluation, and predictor-selection costs from a 20-run GPU pilot comparing compilation with exhaustive configuration search. We additionally provide pilot-calibrated amortization estimates, explicitly distinguishing observed execution times from extrapolated offline costs and documenting their assumptions.

Finally, we have revised Sec. 11 to distinguish the empirically established capabilities of the current compiler from the broader research vision, acknowledging the restricted configuration space, controlled learning objectives, modest improvements over strong defaults, and limitations in generalization across learning families and model backbones.

Thank you for the thorough rebuttal and additional experiments. I appreciate that the authors have directly addressed all four of my questions. The new results clarify the strengths and limitations of the proposed approach, and I raise my score.

Reviewer_JVWKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

Authors propose adaptation compilation, which learns from previous adaptation episodes to predict the acquisition, transfer, boundedness, and preservation outcomes of candidate adaptation programs, then selects a program under a specified utility. Predicting multiple outcomes allows priorities to change without retraining the predictor. Evaluation covers four approximately budget-matched LoRA configurations and five controlled learning families. On Llama-3.1-8B, compiler selection achieves utility 0.607, compared with 0.600 for objective-specific defaults and 0.618 for exhaustive oracle selection. Feature ablations, alternative utilities, leave-one-family-out evaluation, and an independently trained Gemma replication further characterize the approach, revealing substantial limitations beyond the favorable Llama setting.

Critical strengths
  • Predicting a behavioral outcome vector enables program selection under different priorities without retraining. The alternative-utility experiments demonstrate this flexibility, making the formulation useful for settings where adaptation objectives may change (Table 6).

  • Objective-specific defaults provide a meaningful comparison beyond simply identifying the learning family. The Llama improvement is supported by a paired gain of 0.0072 with a 95% confidence interval of [0.0011, 0.0136]. The ablations and openly reported negative results strengthen the empirical analysis (§7; Appendices D–G).

Critical weaknesses
  • Experiments cover only four predefined LoRA configurations and five controlled learning families. The modest improvement on Llama disappears on Gemma, where compiler utility is 0.435 versus 0.438 for objective-specific defaults. Leave-one-family-out selection also underperforms the global default, 0.533 versus 0.556. Although acknowledged, these results substantially limit the generality and practical effectiveness established by the paper (Section 9–10).
  • The main baselines use fixed policies or mean outcome predictions. There is no direct program classifier or scalar-utility predictor trained on the same episode features and adaptation records. Consequently, the experiments do not isolate the empirical advantage of predicting the full behavioral vector over conventional episode-conditioned selection (§§6–7).
Questions for authors
  1. How does the proposed predictor compare with direct program classification or scalar-utility prediction using the same features, adaptation records, and validation budget, including under changing utility weights?

  2. What are the offline record-collection and per-episode feature-extraction costs, and how many future adaptation episodes are required to amortize them relative to searching the four candidate programs?

2 comments

We thank the reviewer for these constructive suggestions. Changes and additions are highlighted in the revised PDF for ease of review.

. . .

Q1. We have expanded Experiment 3 (Sec. 7) and added analyses in Appendix D.4-D.6, comparing direct program classification and scalar-utility regression against geometry prediction using the same pre-adaptation features, historical adaptation records, and episode partitions. These analyses include utility-conditioned alternatives, held-out utility preferences, paired statistical comparisons, and robustness experiments varying preference-training coverage. We also extend the cross-backbone comparisons in Sec. 10 and Appendix G to clarify when multidimensional prediction offers an advantage and where that advantage does not generalize.

Q2. To address computational efficiency, we added Appendix H, reporting measured feature-extraction, adaptation, evaluation, and predictor-selection costs from a 20-run GPU pilot. We additionally analyze offline record-collection costs and provide pilot-calibrated estimates of the number of subsequent adaptation episodes required for amortization relative to exhaustive configuration search, explicitly distinguishing measured execution times from extrapolated costs and documenting their assumptions and limitations.

Finally, we have clarified the empirical scope of our findings in Sec. 11, particularly the restricted program space, limited generalization to unseen learning families, and inconsistent selection improvements across backbones.

Thank you for the additional experiments and clarifications. I have no further questions at this point, I have raised my soundness to 3 and my overall assessment remains unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.