Scalable Skill Retrieval for Machine Learning Agents via Evolving State Modeling

Machine learning agents solve tasks through sequences of costly experiments, during which their knowledge needs evolve with observed results. Static retrieval, which retrieves skills only once at the beginning of a task, cannot adapt to these changing needs. We study skill retrieval as a sequential decision problem conditioned on experimental progress. Preliminary study on MLE-bench reveal that repeatedly retrieving from the latest experimental state may introduce irrelevant guidance, disrupt promising experiments, and consume resources without improving outcomes. This observation motivates three design goals: retrieving the right skill to address the actual problem encountered in the current experiment; retrieving at the right time, only when the model’s own knowledge is insufficient to solve the current problem; and using the retrieved skill in the right way, correctly executing it in the current experimental context. We propose an experiment-state-aware retrieval framework that maintains a structured record of attempted approaches, validation evidence, unresolved failures, and remaining resources. The framework triggers retrieval when this evidence identifies a concrete need, selects skills according to their applicability and feasibility within the remaining budget, and integrates them into targeted experiments with explicit evaluation criteria. Subsequent observations update the experimental state and inform future retrieval decisions. We outline a controlled evaluation on MLE-bench that compares static, naive dynamic, and selective state-aware retrieval under matched resource budgets, measuring final task performance, experimental efficiency, and skill adoption. This work investigates when adaptive access to procedural knowledge can improve long-horizon experimentation and when retrieval instead becomes a source of interference.