acceptodds
Under review as a conference paper at ICLR 2027

When Contextual Skills Can Be Weighted: The Compilability and Reachability of In-Context to In-Weight Learning for LLM Agents

Abstract

Large language models can rapidly acquire new task capabilities from instructions, rules, and examples provided in context, but repeatedly invoked skills incur persistent computation and context overhead. We study how such context-induced capabilities can be compiled into reusable parameter updates so that the model preserves the desired behavior after the skill is removed. We show that successful compilation depends not only on optimization, but also on whether finite calibration data identify the target behavior, whether the target behavior is realizable within the model's parameterization, and how well limited update capacity can approximate it. We derive a characterization for each of these conditions: identifiability, structural realizability, and low-rank approximation. Based on these results, we develop a screening procedure that evaluates candidate calibration sets, update layers, and ranks before expensive behavioral optimization. In a fully enumerated, theoretically supported configuration space, the procedure reaches 95% of the best configuration in that space while using less candidate-training GPU time than random search. These results show that our theory can both explain the structural limits of skill-to-weight compilation and directly reduce its practical search cost.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
in-context learning, model adaptation, parameter-efficient fine-tuning, low-rank adaptation, large language models, skill compilation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hVA4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to compile a contextual skill into model weights, where the skill is present during calibration but removed at deployment, so its behavior must be absorbed into a low-rank weight update. The authors cast this as a shared-control problem over attention values and derive three structural limits—identifiability from finite calibration data, realizability under grouped-query parameter sharing, and low-rank representability—all measurable on the frozen model before training. Based on these, they propose Preflight, a training-free ordering of candidate compilations that ranks calibration sets, layers, and ranks before any adapter is trained. On a fully enumerated 90-configuration menu across 18 skill seeds, Preflight reaches 95% of the best configuration with 4.76× less candidate-training GPU time than random search, or 1.86× once its own cost is charged, and the ordering generalizes across skill families and model architectures.

Critical strengths
  1. It reframes skill compilation as a pre-analyzable structural problem with three orthogonal, measurable limits. The paper derives identifiability (the causal-prefix bound in Proposition 1), realizability (the GQA condition in Eq. 4), and low-rank representability (Proposition 2), corresponding to the calibration set, the architecture, and the rank budget, all measurable on the frozen model before training. The validation in Fig. 2 is quantitative (Eq. 3 holds on 60/60 cells; Eq. 5 has a median observed/predicted ratio of 1.013). This turns design choices from hyperparameter search into decisions readable from structural signals.

  2. The Preflight ordering is validated on a fully enumerated menu and reproduced prospectively. With 18 skill seeds, 90 configurations, and 1,656 training runs, the cost replay is exact rather than estimated. At the 95% target, Preflight reduces candidate-training time by 4.76× over random search, or 1.86× once its own cost is charged; the priority order was frozen to disk before any adapter trained and reproduced on a second skill family, Qwen3-8B, and Llama-3.1-8B (1.74×–5.82×), ruling out post-hoc selection.

  3. Appendix S's "control geometry vs. task optimization" is a counterintuitive and independently valuable finding. A standard SFT recipe monotonically increases the control residual (from 1.0 to 3.56/5.78), with its minimum occurring at steps 5 and 8, before the task is learned. This shows that task optimization and the control object are anti-aligned, which directly informs why compilation sometimes works and sometimes does not.

Critical weaknesses
  1. The paper is hard to read. The definitions of the core objects Z\mathbf{Z}, DV\mathbf{D}_V, V∗V^*, B0B^0 are scattered across Section 2.2, Appendix E, and Appendix I, forcing the reader to jump back and forth to assemble the full picture. The notation is dense, and there are inconsistencies between the main text and the appendix (e.g., HSϵH_S^\epsilon vs. HhcH_h^c). How Stage II derives the predicted error in Eq. 5 from a single activation capture is not explained in the main text and requires Appendix Q. This directly affects both the reviewing experience and the assessment of the method's reproducibility.

  2. The effectiveness of the ordering depends on the discriminative power of the menu, but no predictive criterion is provided. Table 6 shows that on Llama-3.1-8B the composite ordering falls behind both baselines (1.74×1.74\times vs. 2.88×2.88\times/2.61×2.61\times), with one seed reversing. The authors attribute this to the low discriminative power of that menu (5–8 of 30 configurations reach the target), which implies that Preflight may be harmful on low-discriminative menus. Yet the paper gives no actionable criterion for when Preflight should be used.

  3. The optimism of the Stage II predicted error is not quantified. Appendix Q explicitly states that the spectrum is computed per KV group and then summed, whereas the rank constraint applies to the combined update, making E^ℓ,rprobe\widehat{E}_{\ell,r}^{\mathrm{probe}} an optimistic lower bound. However, the actual gap is not reported. If the gap is large, Stage II may systematically favor higher ranks, undermining the reliability of the ordering.

Questions for authors
  1. Can the authors provide an actionable criterion for when Preflight should be used? Table 6 shows that on Llama-3.1-8B the composite ordering falls behind both baselines (1.74×1.74\times vs. 2.88×2.88\times/2.61×2.61\times), which the authors attribute to the low discriminative power of that menu. Is there a pre-training statistic that predicts menu discriminative power, so that a user can decide in advance whether Preflight will help or hurt?

  2. Can the authors quantify the optimism of the Stage II predicted error? Appendix Q states that E^ℓ,rprobe\widehat{E}_{\ell,r}^{\mathrm{probe}} is an optimistic lower bound because the spectrum is computed per KV group and then summed, while the rank constraint applies to the combined update. What is the actual gap between this per-group estimate and the true combined rank-rr error on the experimental menu, and does this gap systematically bias Stage II toward higher ranks?

3 comments
AuthorsOpenReview ↗

Official Reply to Reviewer hVA4 (1/3)

W1: The paper is hard to read. The definitions of the core objects Z\mathbf{Z}, DV\mathbf{D}_V, V∗V^*, B0B^0 are scattered across Section 2.2, Appendix E, and Appendix I, forcing the reader to jump back and forth to assemble the full picture. The notation is dense, and there are inconsistencies between the main text and the appendix (e.g., HSϵH_S^\epsilon vs. HhcH_h^c). How Stage II derives the predicted error in Eq. 5 from a single activation capture is not explained in the main text and requires Appendix Q. This directly affects both the reviewing experience and the assessment of the method's reproducibility.

Reply to W1: Thank you for your careful reading. We apologize for the difficulty caused by the presentation of the theory in the main text. We make three revisions to improve clarity and reproducibility, with the modified sections marked in blue.

1. Clarifying the three execution settings. In §2.2, we now distinguish three executions when introducing matched-state routing (B0B^0). Skill-conditioned execution defines the target behavior. The matched-state counterfactual fixes the retained states (ZSZ_S) and recomputes routing after removing the Skill, serving only to derive the local control relation. Skill-free execution represents the actual deployment setting, and is used for both Stage II scoring and all task evaluations of the compiled model.

2. Correcting inconsistencies between the main text and appendices. We correct Appendix E.5, where B0B^0 is incorrectly called "compiled routing," to consistently refer to matched-state routing. We also replace the remaining "Stage III" references in Appendix U with "Stage II" and correct the reference to Eq. 3 in §3, which should instead point to the ranking key KK.

3. Explaining the complete Stage II computation in the main text. We add a paragraph to §3 detailing how Eq. 5 is evaluated from activation captures. We select 64 probes of the most common sequence length from the calibration pool, shared across all candidates, and append the interpreter's answer to each. We run one forward pass with the Skill and one without it. For each KV group, we stack the value-projection inputs and head-output differences into XgX_g and DgD_g, then apply Eq. 5 to compute the error at each rank. Summing and normalizing across groups yields E^ℓ,rprobe\widehat{E}^{\mathrm{probe}}_{\ell,r}, which is combined with the 1.05 rank-dominance rule and Stage I labels to construct the final ranking key.

We believe these revisions make the theoretical framework easier to follow and the Preflight procedure easier to reproduce.

AuthorsOpenReview ↗

Official Reply to Reviewer hVA4 (2/3)

W2: The effectiveness of the ordering depends on the discriminative power of the menu, but no predictive criterion is provided. Table 6 shows that on Llama-3.1-8B the composite ordering falls behind both baselines (1.74× vs. 2.88×/2.61×), with one seed reversing. The authors attribute this to the low discriminative power of that menu (5–8 of 30 configurations reach the target), which implies that Preflight may be harmful on low-discriminative menus. Yet the paper gives no actionable criterion for when Preflight should be used.

Q1: Can the authors provide an actionable criterion for when Preflight should be used? Table 6 shows that on Llama-3.1-8B the composite ordering falls behind both baselines (1.74× vs. 2.88×/2.61×), which the authors attribute to the low discriminative power of that menu. Is there a pre-training statistic that predicts menu discriminative power, so that a user can decide in advance whether Preflight will help or hurt?

Reply to W2 and Q1: Thank you for highlighting this important practical concern. We agree that an effective candidate ordering does not necessarily justify the cost of Preflight, particularly when the candidate menu has limited discriminative power. We add a practical adoption criterion and stopping protocol in §4.4, allowing users to decide whether to use Preflight before committing to a full search.

1. A Practical Criterion for Adopting Preflight.

Rather than relying on an indirect measure of menu discriminative power, we compare the expected search savings directly with the screening cost. Let FF denote screening overhead, cc the cost of training and validating one candidate, and ndirectn_{\mathrm{direct}} and nPFn_{\mathrm{PF}} the numbers of candidates required to reach a validation target. Preflight is worthwhile when

ndirect−nPF>F/c.n_{\mathrm{direct}}-n_{\mathrm{PF}}>F/c.

Users can measure FF and cc on the target model and hardware, and estimate the number of candidates saved from similar tasks or a small pilot, whose cost is also included. This criterion accounts for menu discriminative power, model architecture, and hardware costs without requiring an exact prediction of the best configuration. We also specify a stopping protocol with a validation target and total budget fixed before searching.

2. Validation Across Models and Candidate Spaces.

The criterion agrees with the observed outcomes in all four tested settings:

ModelCandidatesScreening costCandidates savedDecision
Gemma-2-2B (main)903.66Use (1.86× net)
Gemma-2-2B (second)302.16Use (2.05× net)
Qwen3-8B (second)3017.76Skip
Llama-3.1-8B (second)3023.52Skip

Screening cost is reported in candidate-training equivalents.

Our results provide a concrete decision rule: use Preflight only when the estimated number of candidates saved exceeds the screening cost measured in candidate-training equivalents; otherwise, skip it. This estimate can be obtained from comparable tasks or a small pilot before the full search. For Llama-3.1-8B, screening costs 23.5 candidate-training equivalents but saves only two candidates, so the rule recommends skipping Preflight even when the ordering improves. This makes the distinction clear: better ranking does not necessarily translate into net savings, and our criterion accounts for both.

AuthorsOpenReview ↗

Official Reply to Reviewer hVA4 (3/3)

W3: The optimism of the Stage II predicted error is not quantified. Appendix Q explicitly states that the spectrum is computed per KV group and then summed, whereas the rank constraint applies to the combined update, making E^l,rprobe\hat{E}^{probe}_{l,r} an optimistic lower bound. However, the actual gap is not reported. If the gap is large, Stage II may systematically favor higher ranks, undermining the reliability of the ordering.

Q2: Can the authors quantify the optimism of the Stage II predicted error? Appendix Q states that E^l,rprobe\hat{E}^{probe}_{l,r} is an optimistic lower bound because the spectrum is computed per KV group and then summed, while the rank constraint applies to the combined update. What is the actual gap between this per-group estimate and the true combined rank-rr error on the experimental menu, and does this gap systematically bias Stage II toward higher ranks?

Reply to W3 and Q2: Thank you for highlighting this important issue. We agree that the optimism of the group-wise relaxation needs to be quantified, particularly its effect on rank selection. We add a quantitative comparison in Appendix Q and clarify in §3 that Stage II uses the relaxed score for ranking rather than as an exact solution to the global rank constraint.

1. Quantifying the Relaxation Gap and Its Effect on Selection.

We compute two bounds on the optimal error under the shared global rank-rr constraint: the group-wise SVD solution gives a lower bound LL, while alternating least squares with multiple initializations provides a feasible upper bound UU.

The gap is small, and configuration selection remains stable. Across 20 seeds, the median relative gaps U/L−1U/L-1 are 3.7%, 6.4%, and 4.0% at ranks 4, 16, and 64, respectively. The two rankings have a median Kendall's τb\tau_b of 0.94, with 93% of configuration pairs having a provably fixed order based on non-overlapping bounds. Both methods retain identical candidate sets and select the same top configuration (L25, r64) in 20/20 seeds.

Importantly, the relaxation does not systematically favor higher ranks. The median error ratio between ranks 4 and 64 is 1.66 under the relaxation and 1.72 under the global constraint, indicating that the latter penalizes lower ranks more strongly.

2. Impact on Search Efficiency.

We further apply both rankings to the same trained candidates and compare their speedups over random search in reaching 95% of the best candidate performance.

Rank constraint for scoringOriginal 18 seedsProspective 6 seeds
Group-wise rank-rr (Preflight)4.76×4.03×
Global rank-rr (feasible UU)4.71×4.35×

The results are also consistent across additional task and model settings. The speedups remain unchanged on Gemma-2-2B (5.82×) and Qwen3-8B (4.87×), while global-rank scoring improves Llama-3.1-8B from 1.74× to 2.10×.

These results show that the group-wise relaxation slightly underestimates the global error, but preserves configuration selection and search efficiency without systematically favoring higher ranks. Its lower computational cost therefore makes it a suitable ranking proxy for Preflight.


We sincerely appreciate your constructive feedback, which has helped us improve the paper. Thank you again for your time and careful consideration. We welcome any further questions or discussion.

Reviewer_fj79OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper explores how skills described as text in an AI agent's context can be compiled into weight updates that are reusable, with the skill description being removed from the context. The authors frame this as a shared-control problem over attention values, where they analyze whether a (shared) linear transformation can make the post-skill-deletion outputs to match the pre-skill-deletion attention outputs through a calibration set. They also present an algorithm called preflight, whose goal is to find the best possible training configuration for this skill compilation procedure, through some well specified structural measurements.

Critical strengths
  • In my opinion, the paper is very well motivated, and relevant. Agentic workflows can incorporate large skill descriptions in their context, that are persistent across user queries. Therefore, exploring whether those skills can be internalized within the model's weights, without sacrificing match of the performance is a valuable direction, since it can substantially reduce deployment costs by a serious margin.
  • I found the framining of the compilation problem interesting and insightful. The way the authors set the problem up allowed for theoretically supported limitations (finite calibration, parameter sharing, adapter capacity) that the authors were able to explicitely measure in their experiments.
Critical weaknesses
  • The main experiments focus quite heavily on preflight. I understand that this is one of the paper’s contributions, but the question I find most interesting is when, and how well, a skill can be internalized within the model’s weights without sacrificing much of the performance. I would have liked to see more emphasis on this question in the main text, and a clearer explanation of how the structural measurements relate to actual task performance.
Questions for authors
  • How much does the length of the skill description play a role in whether it is able for the model to internalize it? Does the method work equally well for longer descriptions, and how does length affect the compilation cost, task performance, and deployment cost?
  • Could the authors discuss more directly in the main text when skill compilation works well and when it does not?
2 comments
AuthorsOpenReview ↗

Official Reply to Reviewer fj79 (1/2)

W1: The main experiments focus quite heavily on preflight. I understand that this is one of the paper’s contributions, but the question I find most interesting is when, and how well, a skill can be internalized within the model’s weights without sacrificing much of the performance. I would have liked to see more emphasis on this question in the main text, and a clearer explanation of how the structural measurements relate to actual task performance.

Q2: Could the authors discuss more directly in the main text when skill compilation works well and when it does not?

Reply to W1 and Q2: Thank you for recognizing the motivation and framing of our work. We appreciate the opportunity to enhance our work based on your insightful suggestions. We agree that when and how well a Skill can be internalized without losing task performance deserves more emphasis in the main text. We add a discussion of compilation success and failure conditions in §4.1, supported by new experiments in Appendix X.

1. Conditions for Successful Compilation and Failure Boundaries.

Compilation works well when the calibration data cover the required primitives, the target behavior is single-step, and the base model can execute the Skill in context. On our main task family, the compiled model not only retains the Skill's capabilities without the Skill in context, but also outperforms in-context execution. The following results are evaluated on a held-out test set never used for selection (20 seeds):

SettingTest Accuracy
Base model without Skill0.000
Skill provided in context0.432
Compiled, single-layer Preflight search0.712
Compiled, default multi-layer configuration0.830

Our experiments identify three distinct failure modes.

  • Insufficient calibration coverage: when primitives required at test time are absent from calibration, training accuracy reaches 1.000 while test accuracy remains 0.000.
  • Composition depth: for unseen compositions, final-answer supervision alone yields an accuracy of 0.026, which increases to 0.427 when intermediate results are included in the training targets.
  • Limited base-model capability: when the base model cannot execute the Skill even in context, its accuracy is only 0.001. We also find that meaning-preserving increases in Skill description length do not change the selected compilation configurations.

2. Relationship Between Structural Metrics and Task Performance.

Our structural metrics characterize necessary conditions for compilation and are used to filter and rank candidates before training, rather than directly predict final accuracy. From this perspective, their observed relationships with task performance are consistent with their intended purpose:

Identifiability. Calibration sets with higher prefix complexity achieve better task performance, with an average gain of 0.186 at r64 and positive gains across all 20 seeds.

Capacity score. Its ranking of layer-rank configurations agrees with their actual task accuracy, with a median within-seed Kendall τb\tau_b of 0.67. Ranking candidates by this score reduces the training cost required to reach 95% of the best accuracy in the configuration menu by a factor of 4.76.

Necessary but not sufficient. Structural feasibility does not guarantee behavioral success. In Appendix S, task-only training causes the control residuals to increase steadily (reaching 3.56 and 5.78), while task accuracy still exceeds the 0.95 threshold. In the composition experiments, structural residuals remain nearly unchanged across composition depths, yet accuracy drops from 0.686 to 0.010.

We believe this distinction is central to the problem: reproducing a control signal does not guarantee correct behavior. Structural metrics provide a low-cost way to eliminate infeasible candidates and rank feasible ones, while final behavioral performance must still be verified on a validation set.

AuthorsOpenReview ↗

Official Reply to Reviewer fj79 (2/2)

Q1: How much does the length of the skill description play a role in whether it is able for the model to internalize it? Does the method work equally well for longer descriptions, and how does length affect the compilation cost, task performance, and deployment cost?

Reply to Q1: Thank you for this thoughtful and constructive question. We find it particularly interesting because Skill-description length and the complexity of the behavior it specifies are two distinct factors. Separating them helps clarify both the limits of Skill compilation and its practical benefits. Motivated by your suggestion, we conduct a meaning-preserving length ablation, with full results in Appendix X.2.

Experimental design. We keep the underlying rules fixed and vary description length across 84 Skills from seven task families. For each Skill, we generate three independent rewriting series at 1×, 2×, 4×, and 8× the original token count. An independent parser reconstructs the rules and checks their behavior on boundary cases. All 1,092 descriptions pass these semantic checks.

Configuration selection and task performance. On eight representative Skills, we recompute Preflight scores for all 96 rewrites and replay selection under a fixed budget. All 96/96 preserve the preferred layer-rank configuration, the candidates visited within budget, and the final selection. The compiled models retain identical test accuracy (difference 0.000). In contrast, in-context accuracy varies with description length and task. For example, select_copy_transform drops from 0.271 to 0.109 at 8× length, while other tasks remain stable.

Compilation and deployment costs. Longer descriptions increase only the one-time activation capture cost, by at most roughly one candidate training run at 8× length. Spectral decomposition and adapter training costs remain unchanged. In contrast, deployment benefits increase substantially with description length. The table reports throughput gains from compilation at batch 64 with 16 generated tokens per request.

Task familyOriginal4×8×8× peak memory (context/compiled)
policy_tool1.8×6.6×14.7×43.4/5.8 GB
select_copy_v21.7×4.9×10.6×37.7/6.1 GB
nonce_dsl1.6×4.0×9.1×26.7/5.7 GB
secret_arithmetic1.1×1.8×3.3×12.8/5.7 GB
select_copy_transform1.9×7.1×N/AN/A
kv_memory5.1×N/AN/AN/A

N/A indicates that in-context execution at batch 64 exceeds 80 GB of GPU memory. At 8× length, throughput improves by up to 14.7×, and some in-context settings become infeasible within 80 GB of memory.

These results show that longer Skill descriptions can be compiled without changing the selected configurations or their task performance, while offering greater deployment savings.


Finally, we sincerely appreciate your constructive comments about our work. Your valuable feedback has been helpful to our paper. If you have any further questions, we would be very pleased to engage in further discussion. Thank you once again for your time and attention.

Reviewer_c8SYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to compile textual skills into model parameters while reducing configuration search costs.The authors model skill effects as local control over attention values and analyze constraints from finite calibration data, parameter sharing, and low rank updates.They propose Preflight to rank candidate combinations of calibration sets, update layers, and adapter ranks before training.

Critical strengths

The main experiment involves an exhaustive enumeration of 90 configurations encompassing calibration sets, update layers, and rank selection evaluated across 18 skill seeds. This supports a significant finding of the paper: structural information within a frozen model can be leveraged to prioritize the order in which candidate configurations are tested during adapter training. When the primary setup reaches 95% of the optimal score, it offers a 4.76× advantage in terms of candidate training costs; even after accounting for the overhead of the selection process, a net advantage of 1.86× remains.

Critical weaknesses
  1. The tasks used for behavioral evaluation have limited representativeness, leaving a gap between them and the actual capabilities required of an agent in real world scenarios. The core behavioral experiments focus on conditional tool selection and field copying transformations, with explicit rules and correct answers generated by deterministic interpreters. While such setups facilitate the control of variables, they fail to adequately capture the multi-step interactions, feedback loops from tool execution, error recovery processes, and complex workflows inherent to real world agent operations.
  2. The theoretical contributions rely heavily on standard linear algebra tools, with a relatively limited degree of independent theoretical innovation. In terms of the overall contribution, the added value is primarily concentrated on the construction of specific control objects and the use of structural quantities for configuration ordering. These derivations, in themselves, do not represent a breakthrough in terms of novel mathematical tools or learning theory.
Questions for authors

1.Appendix L quantifies the differences between states but does not verify whether these differences affect the ranking of candidates. Provide a comparison of scores in a scenario where no "Skill" is present, and clarify whether the ranking and task performance remain stable. 2.The group wise rank-r scoring in Appendix Q relaxes the actual global rank-r constraint, yet it does not verify whether the original ordering is preserved. Please provide a comparison between the scores obtained under the global rank-r constraint and the resulting configuration selections. 3.Section4.2 primarily examines control errors, yet the link between these errors and final task performance has not been fully established. Provide an analysis that directly predicts task accuracy based on structural scores, and specify the conditions under which failure occurs. 4.The causal prefix rank bound does not directly justify the 0.90 threshold or the strict priority assigned to calibration-set labels. Please explain the threshold choice and provide sensitivity tests and an analysis of interactions between calibration sets, layers, and ranks. 5.Some settings in Appendix U do not break even after screening overhead, yet the conditions for adopting Preflight remain unclear. Please specify which tasks, models, and candidate space sizes are expected to yield net savings, and how users can assess this beforehand. 6.Appendix V evaluates alternative adapters but does not provide a comparison of cost and quality under a consistent budget. Please clarify whether updating a single layer is required for deployment, and compare total compilation time, parameter count, and task performance with reasonable default configurations. 7.Reaching 95% of the menu’s best score is a retrospective evaluation criterion, while an executable stopping protocol remains unspecified. Explain how the target is determined before search and how validation, configuration selection, and final testing are separated.

7 comments
AuthorsOpenReview ↗

Official Reply to Reviewer c8SY (1/5)

W1: The tasks used for behavioral evaluation have limited representativeness, leaving a gap between them and the actual capabilities required of an agent in real world scenarios. The core behavioral experiments focus on conditional tool selection and field copying transformations, with explicit rules and correct answers generated by deterministic interpreters. While such setups facilitate the control of variables, they fail to adequately capture the multi-step interactions, feedback loops from tool execution, error recovery processes, and complex workflows inherent to real world agent operations.

Reply to W1: We thank the reviewer for highlighting execution feedback and error recovery as important aspects of Agent Skill evaluation. To address these concerns, we conduct three additional experiments on tool failure recovery, unseen step compositions, and public instruction tasks. We summarize the main results below, with experimental details in Appendix X.

Tool Execution and Error Recovery. We extend our tool-selection tasks to include execution feedback, where the model must respond to tool failures by invoking a fallback or escalating. We evaluate six Skill seeds on Gemma-2-2B at two difficulty levels. The in-context baseline receives the Skill manual and three demonstrations, while the compiled model operates without Skill text.

DifficultyFailure trajectoriesFull success: In-contextFull success: CompiledRecovery: In-contextRecovery: Compiled
Easy37740.8%77.9%35.7%76.8%
Hard37939.2%75.7%31.8%77.0%

Results are averaged equally across six seeds. The compiled model achieves higher full-trajectory success on 6/6 seeds at both difficulty levels. When tool observations are hidden, its recovery rate drops from 76.8% to 16.6% on easy tasks and from 77.0% to 7.2% on hard tasks. These results show that the compiled model can use runtime feedback to recover from tool failures without the Skill document.

Cross-Step Dependencies and Unseen Compositions. We further test whether compiled Skills can generalize to unseen combinations of learned atomic operations. Across six Skill specifications, we hold out eight compositions per Skill and evaluate on 128 test examples per specification.

MethodAdaptersTraining stepsCalls per requestUnseen accuracy
Final-answer supervision112012.6%
Intermediate + final supervision1120142.7%
Stepwise, shared adapter1480258.3%
Stepwise, adapter per operation4480265.4%

With identical training data, adapter configuration, and training steps, intermediate supervision improves all six Skills, increasing unseen-composition accuracy by 40.1 percentage points on average. Stepwise execution achieves up to 65.4% accuracy when using the model's own intermediate outputs, although these settings use more training steps.

Public Instruction Tasks. We also evaluate eight Super-NaturalInstructions tasks covering extraction, classification, and structured outputs. We train Gemma-2-2B on 32 examples per task, with three independent calibration samples, and evaluate normalized exact match on 128 test examples. Both the base and compiled models are tested without Skill text.

Task IDBase modelBest constant answerCompiled, 3-run mean
9261.6%2.3%87.0%
14110.0%2.3%67.4%
12050.0%56.2%95.8%
14180.0%45.3%48.4%
6720.0%27.3%28.6%
1731 (QuaRTz)1.6%7.8%67.2%
3300.8%1.6%68.0%
13828.6%3.1%31.2%

The best constant answer is the fixed response matching the most test references. The compiled model outperforms this baseline on 8/8 tasks in mean accuracy and in 21/24 individual runs.

We include the full experimental settings and results in the revised appendix: X.1 for unseen compositions and X.4 for tool feedback and public instruction tasks.

AuthorsOpenReview ↗

Official Reply to Reviewer c8SY (2/5)

W2: The theoretical contributions rely heavily on standard linear algebra tools, with a relatively limited degree of independent theoretical innovation. In terms of the overall contribution, the added value is primarily concentrated on the construction of specific control objects and the use of structural quantities for configuration ordering. These derivations, in themselves, do not represent a breakthrough in terms of novel mathematical tools or learning theory.

Reply to W2: We thank the reviewer for encouraging the development of new theoretical tools to explain LoRA. Our main contribution is a framework for analyzing the identifiability, feasibility, and capacity of Skill compilation. Beyond standard linear algebra, our analysis builds on the computational structure of Transformers. Softmax renormalization determines which computations must be compensated for after removing the Skill. Causal prefixes limit the number of independent state directions that calibration data can constrain. The shared projections in GQA determine whether the corrections required by different attention heads can be realized simultaneously.

Specifically, our paper makes three main contributions.

We show that even a very short Skill can require a nearly full-rank update. The direct value contribution of a single token has rank at most one. Yet, under the conditions in Appendix F.2, exactly recovering the local output by updating only the value projection requires a rank of at least dv−1d_v-1, where dvd_v is the value dimension (Eq. 57).

We identify architectural constraints that cannot be resolved by increasing the rank. Even when each attention head can be fitted independently, sharing the same value projection can make their targets incompatible. In a comparison with matched equation counts on Llama-3.1-70B, the median joint-control residuals are 0.125, 0.260, and 0.361 when 2, 4, and 8 heads share a projection, respectively (Appendix N).

We turn structural analysis into a method for selecting configurations before training. Across a complete grid of 90 candidates for each of 18 Skill seeds, random ordering requires 4.76 times as much candidate training time as Preflight to reach 95% of the best performance available in the candidate set (§4.3).

We believe these results address your concerns and strengthen our contribution to the theoretical understanding and practical application of Skill compilation.

AuthorsOpenReview ↗

Official Reply to Reviewer c8SY (3/5)

Q1: Appendix L quantifies the differences between states but does not verify whether these differences affect the ranking of candidates. Provide a comparison of scores in a scenario where no "Skill" is present, and clarify whether the ranking and task performance remain stable.

Reply to Q1: Thank you for raising the question of how differences in student states affect candidate selection. We first clarify that the Skill-free setting is already the student's deployment condition in the original paper and is also the state used by Preflight for scoring. The teacher retains the Skill to define the target behavior, while the student is scored and evaluated without it. Removing the Skill from the teacher would change the target itself.

Appendix L already compares the residuals of a fixed update under matched-state and Skill-free execution. To address your concern about candidate selection, we further compare the resulting rankings and their search efficiency.

Across 18 seeds, the two rankings have a median Kendall's τb\tau_b of 0.57, and both scoring methods select 64 as the preferred LoRA rank. Applying these rankings to the same trained candidates, we obtain the following speedups over random search in reaching 95% of the best candidate performance:

Student state for scoringTraining-time speedup
Skill-free (current Preflight)4.76×
Matched-state4.88×

Although the rankings differ, both achieve similar search efficiency. We include this comparison at the end of Appendix L in the revised manuscript.

Q2: The group wise rank-r scoring in Appendix Q relaxes the actual global rank-r constraint, yet it does not verify whether the original ordering is preserved. Please provide a comparison between the scores obtained under the global rank-r constraint and the resulting configuration selections.

Reply to Q2: As noted in Section 3, group-wise rank-rr scoring is an optimistic relaxation of the global rank-rr constraint. We now compare it with feasible solutions under the global constraint, using the group-wise score and the feasible global score as lower and upper bounds on the optimal residual.

The relaxation has little effect on configuration selection. Across 20 seeds, the median Kendall's τb\tau_b between the two rankings is 0.94. Both methods select the same top configuration and retain identical candidate sets in 20/20 seeds. Moreover, the bounds certify the relative ordering of 93% of all configuration pairs.

The search efficiency also remains stable. On the original 18 seeds, the training-time speedup over random search is 4.76× with group-wise scoring and 4.71× with global scoring. On six prospective seeds, the corresponding speedups are 4.03× and 4.35×. Results on additional models show the same trend.

These results show that the cheaper group-wise relaxation preserves both configuration selection and search efficiency despite its optimistic residual estimates. We add the detailed score comparisons, bounds, and selection results to Appendix Q and clarify this finding in Section 3.

Q3: Section4.2 primarily examines control errors, yet the link between these errors and final task performance has not been fully established. Provide an analysis that directly predicts task accuracy based on structural scores, and specify the conditions under which failure occurs.

Reply to Q3: Thank you for highlighting the link between structural scores and task performance. Our paper already validates structural scores for configuration selection using final task performance. Predicting accuracy directly from these scores, however, is a more challenging goal. Section 4.2 examines structural properties before training, while §4.3 evaluates their value for configuration selection through 90 configurations and 18 seeds, an experiment you also recognize in the Strengths.

Appendix S shows that task recovery and control residual reduction are not equivalent. Task-only training increases residuals from 1 to 3.56 and 5.78 at two layers. At the better-conditioned layer, both task-only and control-regularized training achieve 1.000 accuracy, although the latter reduces the residual to 0.3956. Thus, structural scores can guide selection without directly predicting final accuracy.

Appendix P and Table 4 identify two failure conditions. Without required primitives in calibration data, training accuracy reaches 1.000 but test accuracy remains 0.000. Even with identifiable primitives and sufficient capacity, compositional generalization can fail: across nine configurations spanning a 6000-fold parameter range, training accuracy reaches 1.000, yet unseen compositions remain at constant-predictor performance.

We clarify these success and failure conditions in §4.1 and provide further analysis in Appendix X.3.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Comments to All Reviewers

We sincerely thank the reviewers (Reviewer c8SY, Reviewer fj79, Reviewer hVA4) for their careful reading and constructive comments.

We are encouraged that the reviewers recognized the importance and practical relevance of compiling contextual Skills into reusable model updates (Reviewer fj79); found our shared-control formulation insightful and the three structural limits measurable before training (Reviewer fj79, Reviewer hVA4); and highlighted the significance of our exhaustive configuration study and the empirical effectiveness of Preflight in reducing search costs (Reviewer c8SY, Reviewer hVA4). We are particularly grateful that Reviewer hVA4 recognized the distinction between control geometry and task optimization in Appendix S as an independently valuable finding.

We addressed all the reviewer comments below and will incorporate them into the revision. The revised version primarily includes the following significant updates, with the modified sections marked in blue:

  • We add closed-loop tool execution with error recovery, unseen two-step compositions, and eight public instruction tasks (Reviewers c8SY and fj79).
  • In Section 4.1, we state directly when compilation works and how it fails, and summarize the evidence in one paragraph with a table (Reviewer fj79).
  • We compare matched-state and Skill-free scores on the same candidate grid, including the resulting search savings (Reviewer c8SY).
  • We quantify the global rank-rr relaxation with two-sided bounds, and report the selected configuration and the search cost under both scores (Reviewers hVA4 and c8SY).
  • We add threshold and interaction sensitivity for the calibration-set rule (Reviewer c8SY).
  • We give an executable cost-based rule that decides before any training whether Preflight should be used, and we separate screening cost from search benefit (Reviewers c8SY and hVA4).
  • We compare single-layer and multi-layer deployment under one time budget, in total cost, parameter count, and held-out accuracy (Reviewer c8SY).
  • We separate the validation target from the stopping protocol, so that neither depends on the best score in the menu (Reviewer c8SY).
  • We consolidate the notation and the Stage II procedure in the main text, unify the matched-state routing symbol across the main text and the appendix, and add a reproducibility check for the Stage II scores (Reviewer hVA4).
  • We measure the screening cost on the GPU under one convention and update the end-to-end accounting throughout (Reviewer c8SY).

If our rebuttal addresses the concerns, we earnestly and kindly ask the reviewers to consider raising the rating and supporting us for acceptance.

AuthorsOpenReview ↗

Final Author Summary to the Area Chair and Reviewers

Dear Area Chair and Reviewers,

As the discussion period concludes, we sincerely thank the Area Chair and all three reviewers for their time and constructive feedback.

We are encouraged by the reviewers' positive assessments of our work. Reviewer fj79 recognized the importance and practical relevance of Skill compilation and found our shared-control formulation insightful. Reviewer hVA4 highlighted the three measurable structural limits, the rigorous evaluation of Preflight, and the distinction between control geometry and task optimization as an independently valuable finding. Reviewer c8SY recognized the significance of our exhaustive evaluation of 90 configurations and the resulting reduction in search costs.

In response to the reviews, we have strengthened the paper in three main aspects.

  • First, we extend the behavioral evaluation with closed-loop tool execution and error recovery, unseen compositions, and eight public instruction tasks. We also clarify the conditions under which Skill compilation succeeds or fails and examine the effect of Skill-description length on compilation and deployment.

  • Second, we strengthen the theoretical and empirical analysis. We quantify the gap between group-wise and global rank constraints, verify the stability of configuration rankings, and examine the sensitivity of calibration-set selection. We also clarify the theoretical notation and provide a more complete description of the Stage II procedure.

  • Third, we establish a clearer basis for practical deployment. We introduce a fixed-budget evaluation with independent validation and testing, an executable stopping protocol, and a cost-based criterion for deciding when Preflight is worthwhile. We further compare single-layer and multi-layer adapters under a common budget.

These results reinforce our central contribution: Skill compilation is subject to three measurable structural limits, namely identifiability, realizability, and capacity. Our framework characterizes these limits before training and uses structural measurements to guide configuration selection. On the exhaustive 90-configuration menu, Preflight achieves a 4.76$\times$ reduction in candidate-training cost and a 1.86$\times$ net speedup over random search.

We appreciate the positive assessments from Reviewers fj79 and hVA4, who recognized the importance of Skill compilation, the value of our structural framework, and the practical benefits of Preflight. We are also grateful to Reviewer c8SY for acknowledging that our responses addressed their concerns and raising their rating from 2 to 3.

Once again, we sincerely thank you for your feedback and comments throughout this insightful review process, which has helped us improve the clarity, rigor, and practical relevance of our work. We hope that the revised manuscript and additional evidence will support a favorable final assessment.

Best regards,

Authors of submission-42154

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.