When Contextual Skills Can Be Weighted: The Compilability and Reachability of In-Context to In-Weight Learning for LLM Agents

Large language models can rapidly acquire new task capabilities from instructions, rules, and examples provided in context, but repeatedly invoked skills incur persistent computation and context overhead. We study how such context-induced capabilities can be compiled into reusable parameter updates so that the model preserves the desired behavior after the skill is removed. We show that successful compilation depends not only on optimization, but also on whether finite calibration data identify the target behavior, whether the target behavior is realizable within the model's parameterization, and how well limited update capacity can approximate it. We derive a characterization for each of these conditions: identifiability, structural realizability, and low-rank approximation. Based on these results, we develop a screening procedure that evaluates candidate calibration sets, update layers, and ranks before expensive behavioral optimization. In a fully enumerated, theoretically supported configuration space, the procedure reaches 95% of the best configuration in that space while using $4.76\times$ less candidate-training GPU time than random search. These results show that our theory can both explain the structural limits of skill-to-weight compilation and directly reduce its practical search cost.