Towards Reliable Skill Internalization via Transferability and Non-Interference

LLM agents can extract reusable skills from successful task experiences and internalize them into model parameters, enabling skill-guided behavior without explicit instructions at inference time. However, a single successful trajectory does not uniquely identify a reusable skill: multiple candidates may explain the same success but behave differently on new tasks. Internalizing an insufficiently validated skill may therefore reinforce behavior that fails to transfer to new tasks or interferes when the skill is unnecessary. We introduce TANI, a framework for validating and refining extracted skills before internalization. It evaluates candidate skills for transferability across tasks requiring the same knowledge and non-interference on tasks where that knowledge is unnecessary. The framework generates diagnostic tasks to test both properties and uses execution outcomes to select and refine candidate skills. We evaluate the effectiveness of our framework by internalizing retained skills using on-policy self-distillation (OPSD). On ${\tau }^2$-bench Telecom and Retail, TANI achieves 50.0% and 70.0% Pass@3, respectively, outperforming existing baselines. Our analyses show that source-task success alone is insufficient for identifying skills suitable for internalization. Evaluating both transferability and non-interference enables more effective skill selection and refinement.