A Deconfounded Memory Mechanism for Machine Learning Engineer Agent Systems

The memory mechanisms of large language model (LLM)-based agents enable cross-episode self-improvement, expanding their ability to automate the end-to-end process of machine learning (ML) research and development (R&D). However, existing ML engineer agents typically promote or demote hypotheses in memory based on the final outcome of the episode in which they are tested. Such outcome-based memory updates are highly confounded: a negative outcome may result from factors unrelated to the hypothesis itself, causing valid hypotheses to be incorrectly demoted and permanently excluded from subsequent research. We refer to genuinely valid hypotheses that receive a confounded negative outcome in an R&D episode as false-negative hypotheses. In this paper, we propose DEC-MEM, a principled memory mechanism for deconfounding episode outcomes. When an R&D episode yields a negative outcome, DEC-MEM first counterfactually probes the episode to identify the causes of the outcome, collecting thorough evidence within a minimal computational budget. We then employ a write policy that stores each hypothesis together with the attributed causes of the outcome and supporting evidence to guide future R&D episodes. During memory retrieval in future episodes, we rank stored memory items using a novel metric that uncovers items associated with confounded negative outcomes, giving their hypotheses an opportunity to be re-examined rather than permanently discarded. In addition, we propose novel evaluation metrics to quantify how well a memory mechanism identifies false-negative hypotheses and re-examines them in later episodes. Extensive experiments across R&D streams show that DEC-MEM identifies false-negative hypotheses more effectively than state-of-the-art baselines and improves cumulative R&D performance by re-examining these hypotheses. Our code is available at https://anonymous.4open.science/r/DEC-MEM-2D12.