acceptodds
Under review as a conference paper at ICLR 2027

A Deconfounded Memory Mechanism for Machine Learning Engineer Agent Systems

Abstract

The memory mechanisms of large language model (LLM)-based agents enable cross-episode self-improvement, expanding their ability to automate the end-to-end process of machine learning (ML) research and development (R&D). However, existing ML engineer agents typically promote or demote hypotheses in memory based on the final outcome of the episode in which they are tested. Such outcome-based memory updates are highly confounded: a negative outcome may result from factors unrelated to the hypothesis itself, causing valid hypotheses to be incorrectly demoted and permanently excluded from subsequent research. We refer to genuinely valid hypotheses that receive a confounded negative outcome in an R&D episode as false-negative hypotheses. In this paper, we propose DEC-MEM, a principled memory mechanism for deconfounding episode outcomes. When an R&D episode yields a negative outcome, DEC-MEM first counterfactually probes the episode to identify the causes of the outcome, collecting thorough evidence within a minimal computational budget. We then employ a write policy that stores each hypothesis together with the attributed causes of the outcome and supporting evidence to guide future R&D episodes. During memory retrieval in future episodes, we rank stored memory items using a novel metric that uncovers items associated with confounded negative outcomes, giving their hypotheses an opportunity to be re-examined rather than permanently discarded. In addition, we propose novel evaluation metrics to quantify how well a memory mechanism identifies false-negative hypotheses and re-examines them in later episodes. Extensive experiments across R&D streams show that DEC-MEM identifies false-negative hypotheses more effectively than state-of-the-art baselines and improves cumulative R&D performance by re-examining these hypotheses. Our code is available at https://anonymous.4open.science/r/DEC-MEM-2D12.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multi-agent system, ML engineer agent system

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PrLHOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies false-negative hypothesis lock-in in LLM-based ML engineering agents, where valid hypotheses are prematurely discarded due to implementation errors, resource limitations, or inappropriate evaluation metrics. To address this, it proposes DEC-MEM, which combines counterfactual failure attribution, cause-specific memory writing, and uncertainty-aware retrieval. The paper also introduces lock-in evaluation metrics and demonstrates improvements over existing memory mechanisms across multiple LLMs and ML engineering tasks.

Critical strengths
  1. The paper identifies false-negative hypothesis lock-in as an important limitation of existing memory mechanisms in autonomous ML engineering agents. DEC-MEM addresses this through cost-aware counterfactual probing, cause-specific memory writing, and frontier-aware retrieval, while introducing FNF1 and FNLR metrics for evaluation. Treating unsuccessful experiments as potentially confounded rather than direct evidence of hypothesis invalidity is a meaningful conceptual contribution beyond conventional outcome-based memory management.

  2. DEC-MEM is evaluated against multiple memory baselines across three LLM backbones using fixed-episode, fixed-compute, ablation, and open-world settings. It consistently achieves substantial FNLR reductions, while ablation studies highlight the importance of frontier-aware retrieval. The breadth of comparisons, consistency of results, and component-level analyses provide strong evidence that DEC-MEM effectively addresses false-negative hypothesis lock-in in the evaluated settings.

Critical weaknesses
  1. The main evaluation relies on a controlled benchmark with predefined hypotheses, cached outcomes, and synthetic failures, whose diagnostic structure closely matches DEC-MEM's attribution mechanism and may favor the proposed approach. While open-world experiments are included, they are limited in scope. Consequently, it remains unclear whether the reported gains generalize to realistic, open-ended ML engineering with naturally occurring and interacting failure conditions.

  2. DEC-MEM relies on manually specified diagnostic likelihoods, restrictive failure-cause assumptions, and counterfactual probes with unverified causal validity. Moreover, the experiments do not clearly separate gains from accurate failure attribution versus increased hypothesis retention and exploration, nor do they compare against simpler alternatives under matched budgets. Without stronger evidence isolating the contribution of causal attribution, the methodological novelty and effectiveness of DEC-MEM remain insufficiently substantiated.

Questions for authors
  1. How sensitive is DEC-MEM to misspecified detector likelihoods, prior distributions, and correlated failure causes? Can the authors provide attribution-calibration analyses and evaluate performance on unseen or simultaneous failure types not explicitly modeled in the current diagnostic framework?

  2. To what extent are the reported improvements due to accurate failure-cause attribution rather than simply retaining and revisiting failed hypotheses? How does DEC-MEM compare with simpler retain-all, random-retry, or uncertainty-based retrieval strategies under identical computational and memory budgets?

  3. Do the reported improvements persist in open-ended ML engineering environments with dynamically generated hypotheses and naturally occurring failures? Broader end-to-end evaluations, paired confidence intervals, and compute-normalized comparisons of final model performance would help substantiate the practical benefits.

7 comments
AuthorsOpenReview ↗

Authors Response 1/5

We sincerely thank the reviewer for recognizing the conceptual contribution and the breadth and consistency of the evaluation. We address each concern and question below with new results. We also revised our paper based on your suggestions, with changes highlighted in blue.

W1. Independence of the benchmark and DEC-MEM

Reviewer: The main evaluation relies on a controlled benchmark with predefined hypotheses, cached outcomes, and synthetic failures, whose diagnostic structure closely matches DEC-MEM's attribution mechanism and may favor the proposed approach.

Authors:

We would like to clarify that our controlled benchmark is independent from DEC-DEM and does not favor DEC-MEM. This is because, DEC-MEM does not have access to the benchmark's hidden fault information. The benchmark determines which harness fault is injected and how likely it is to leave a log trace. DEC-MEM is given neither the fault identity nor these probabilities. It diagnoses failures from observed logs and executed probes, using its own likelihoods defined for failure-cause categories rather than individual injected fault types. We have added to the manuscript to clarify this independence.

AuthorsOpenReview ↗

Authors Response 2/5

W2 / Q2. Contribution of accurate failure attribution

Reviewer: To what extent are the reported improvements due to accurate failure-cause attribution rather than simply retaining and revisiting failed hypotheses?

Authors:

Failure attribution determines which hypotheses memory refutes or retains. To test whether attribution accuracy matters, we keep the cause-specific write policy and frontier retrieval unchanged and vary only how failures are diagnosed (DeepSeek-v4-pro, 10 seeds):

Failure diagnosisFNF1 ↑FNLR ↓CPG ↑
LLM diagnosis0.2050.5580.359
DEC-MEM probes0.3750.3920.396
Oracle diagnosis0.7120.2320.520

Replacing LLM diagnosis with DEC-MEM's probes reduces lock-in (FNLR: 0.558 to 0.392) and increases cumulative gain (CPG: 0.359 to 0.396). Oracle diagnosis improves both further. Thus, the same memory rules perform better when supplied with more accurate failure diagnoses.

W2 / Q2. Simpler alternatives under matched budgets

Reviewer: How does DEC-MEM compare with simpler retain-all, random-retry, or uncertainty-based retrieval strategies under identical computational and memory budgets?

Authors:

We compare DEC-MEM with simpler memory strategies under the same measured compute budget. Each stream receives the CPU-time budget required by a no-memory agent to complete 30 episodes, with every experiment and probe charged to that budget. We use 10 seeds per backbone; paired values below refer to DeepSeek-v4-pro / GPT-OSS-120B.

  • Forced random-retry tests whether retrying failed hypotheses at approximately DEC-MEM's rate is sufficient. In 35% of episodes, it overrides the agent's choice with a randomly selected failed hypothesis. Its FNLR is significantly higher than DEC-MEM's: 0.469 / 0.507 versus 0.336 / 0.241. The paired differences, computed as DEC-MEM minus random-retry, are −0.134 / −0.266, with both confidence intervals excluding zero.

  • Retain-all with uncertainty-based retrieval removes failure cause attribution and probing of DEC-MEM and keep all hypotheses from episodes with negative outcome. This strategy keeps lock-in low but has much lower failure-attribution accuracy (FNF1): approximately 0.08 versus 0.39–0.40 for DEC-MEM.

We also compare final best-task performance (macro), defined as the best macro-F1 gain over each task's baseline, averaged across tasks:

Memory strategyDeepSeek-v4-proGPT-OSS-120B
DEC-MEM0.0700.069
Retain-all0.0490.061
Forced random-retry0.0620.061

DEC-MEM achieves the highest mean final best-task performance among these methods on both backbones, although the paired comparisons do not show statistically significant differences.

AuthorsOpenReview ↗

Authors Response 3/5

W2 / Q1. Sensitivity to detector likelihoods and priors

Reviewers: How sensitive is DEC-MEM to misspecified detector likelihoods, prior distributions?

Authors:

We test sensitivity to the probabilities used for failure attribution. The likelihoods describe how probable an observation is under each failure cause; the priors specify each cause's probability before observing evidence. Using DeepSeek-v4-pro, we modify these probabilities in five settings and compare the results with DEC-MEM's default setting:

Attribution settingFNLR ↓FNF1 ↑CPG ↑
Default DEC-MEM (Table 1)0.3920.3750.396
Multiply each likelihood by log-normal noise, σ = 0.50.3510.3090.452
Multiply each likelihood by log-normal noise, σ = 1.00.3330.3400.342
Swap the likelihoods assigned to CPT and MSN0.3110.3280.317
Assign equal prior probabilities to all four causes0.4410.3230.345
Bias the prior toward hypothesis invalidity: P(HYP) = 0.700.3370.3340.397
No memory (reference)0.4360.1000.201
  • Lock-in: None of the five changes produces a statistically significant difference in FNLR relative to default DEC-MEM; all paired 95% confidence intervals include zero.
  • Attribution accuracy: FNF1 decreases from 0.375 to 0.309–0.340, but remains 0.21–0.24 above no memory, with all paired confidence intervals excluding zero.

Thus, these probability changes reduce diagnostic accuracy, but we detect no significant change in lock-in.

Reviewer_2tcYOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This work studies a practical failure mode in ML engineer agents: a useful hypothesis may be abandoned because an unsuccessful experiment reflects implementation errors, insufficient compute, or unsuitable evaluation rather than hypothesis invalidity. DEC-MEM combines cost-aware diagnostic probes, cause-specific memory updates, and retrieval that revisits uncertain, underexplored hypotheses. The evaluation includes controlled classification tasks and supplementary open-ended AIDE experiments.

Critical strengths

The problem is concrete and relevant to agents that learn from experimental histories. Separating unsuccessful executions from invalid hypotheses is a useful design principle, and the paper connects diagnosis, evidence preservation, and subsequent hypothesis selection into a coherent mechanism. The controlled benchmark makes otherwise unavailable validity and failure-cause labels observable, enabling targeted evaluation. Comparisons across memory mechanisms, component ablations, and matched-budget experiments provide useful evidence. In particular, removing frontier-aware retrieval substantially increases lock-in, supporting the importance of actually revisiting unresolved hypotheses rather than merely retaining them. The supplementary experiments on four open-ended tasks provide encouraging evidence beyond the fixed candidate setting.

Critical weaknesses

The main evaluation uses predefined hypotheses and cached outcomes, with injected implementation faults. This improves experimental control but limits conclusions about diagnosing naturally occurring failures in open-ended research, particularly when obtaining a verified implementation is itself difficult. Several baselines are mechanism-level adaptations with omitted components or altered retrieval, so comparisons should not be interpreted as comprehensive evaluations of the original systems.

The reverse failure mode receives insufficient attention: an invalid hypothesis may be repeatedly preserved because failures are incorrectly attributed to implementation, compute, or measurement. The appendix acknowledges delayed rejection, but provides no explicit failure-loop case study or hypothesis-level wasted-budget analysis. This concern is supported by the attribution results: only 68 of 160 HYP-labelled failures are correctly identified. That is episode-level evidence, however, and does not establish persistent looping. The reported examples predominantly illustrate successful rescue.

Cost accounting is informative but incomplete. Probe costs are assigned normalized values rather than measured execution costs. Moreover, the log-only ablation achieves slightly higher mean cumulative gain than full DEC-MEM while avoiding probe expenditure, although variability prevents a decisive conclusion. Measurement-related failures are also too sparse to substantiate broad four-cause attribution claims.

Questions for authors

How often are genuinely invalid hypotheses revisited, and what fraction of total experimental and probing budgets is spent before their rejection?

Can the authors provide an unsuccessful rescue trajectory and clarify whether any per-hypothesis retry or cumulative-budget limit prevents repeated external attribution?

Under measured end-to-end costs, how does DEC-MEM compare with log-only memory and a simpler budget-matched retry policy, particularly on final best-task performance?

6 comments
AuthorsOpenReview ↗

Authors Response 1/5

We deeply appreciate the reviewer for the detailed and constructive review, and for recognizing the coherence of the designed memory mechanism, the value of the controlled benchmark, and the importance of the frontier-aware memory retrieval module. We address each concern and question below. We also revised the manuscript based on your suggestions, with changes highlighted in blue.

W1. Controlled evaluation and baseline scope

Reviewer: The main evaluation uses predefined hypotheses and cached outcomes, with injected implementation faults. This improves experimental control but limits conclusions about diagnosing naturally occurring failures in open-ended research, particularly when obtaining a verified implementation is itself difficult.

Authors:

We thank the reviewer for acknowledging that our controlled benchmark makes false-negative lock-in "observable, enabling targeted evaluation." This design makes false-negative lock-in measurable. Specifically, FNLR is defined over hypotheses that are actually valid, so computing it requires ground-truth validity labels.

We do evaluate scenarios beyond predefined hypotheses and cached outcomes. Specifically, we perform open-world R&D streams on MLE-bench Lite experiments (Fig. 3), where the agent generates its own hypotheses and failures occur naturally. Because the ground-truth validity of these hypotheses is unavailable, we cannot compute FNLR in this setting. We therefore use the final performance of the R&D stream (the medal score used by real Kaggle leaderboards) as a proxy for DEC-MEM's effectiveness. Figure 3 shows that DEC-MEM significantly improves the medal-score performance of AIDE ML agent systems.

Reviewer: Several baselines are mechanism-level adaptations with omitted components or altered retrieval, so comparisons should not be interpreted as comprehensive evaluations of the original systems.

Authors:

We adapt baselines to share the same agent system workflow backbone (i.e., experimental environment setup, episode loop, backbone LLM) so we can isolate the effect of their memory mechanisms. Running each original baseline agent system end to end would account in effects unrelated to their memory mechanisms. To assess robustness of our DEC-MEM's effectiveness across agent workflows, we additionally run the memory mechanisms within other agent syste workflow backbones (i.e., AIDE's, as shown in Fig. 3). We have also revised Section 5.1 to state this scope explicitly.

AuthorsOpenReview ↗

Authors Response 2/5

W2 / Q1. Revisiting invalid hypotheses and budget use

Reviewer: How often are genuinely invalid hypotheses revisited, and what fraction of total experimental and probing budgets is spent before their rejection?

Authors:

We measured this on the streams of Table 1 (10 seeds per backbone). For each memory mechanism, the table below reports how many times, on average, each invalid hypothesis is tested, and what share of the stream's total budget is spent on testing invalid hypotheses. Each cell lists the values for agent systems with DeepSeek-v4-pro / GPT-OSS-120B / Qwen3.5-397B backbones, respectively.

memorytests per invalid hypothesisshare of total computation budget spent on invalid hypotheses
DEC-MEM1.91 / 1.43 / 1.7331.1% / 12.3% / 30.1%
Reflexion1.42 / 1.60 / 1.4925.3% / 25.2% / 25.8%
Memp1.26 / 1.53 / 1.4037.6% / 35.7% / 39.3%
ExpeL1.00 / 1.00 / 1.0034.2% / 31.5% / 34.9%
Mem01.00 / 1.00 / 1.0033.6% / 30.2% / 26.4%
ReasoningBank1.00 / 1.02 / 1.0030.8% / 31.0% / 31.1%
A-MEM1.00 / 1.00 / 1.0034.3% / 34.2% / 38.0%

We observe that DEC-MEM's share of computation is within the baselines' range on DeepSeek-v4-pro and Qwen3.5-397B, and below it on GPT-OSS-120B. ExpeL, Mem0, ReasoningBank and A-MEM essentially never re-test an invalid hypothesis, yet they spend a similar or larger share of their budget trying a larger number of different invalid ones. This is because they do not, as DEC-MEM does, tell which of hypotheses in past failed episodes were attributed to the hypotheses themselves.

AuthorsOpenReview ↗

Authors Response 3/5

W2 / Q2. Unsuccessful rescue and limits on repeated external attribution

Reviewer: Can the authors provide an unsuccessful rescue trajectory

Authors:

The table below traces an unsuccessful rescue of the invalid hypothesis word_bigrams on AG News (extending the TF-IDF classifier's word n-gram range to (1,2)) in a Qwen3.5-397B stream.

episodeattributed cause (P)true cause
1CPT (0.87)HYP
17HYP (0.83)HYP

In episode 1, DEC-MEM misattributed the failure to insufficient compute (CPT): performance trends suggested that word_bigrams might benefit from more compute, so it retained the hypothesis for re-examination. Only in episode 17 did further experiments and probes reveal that performance plateaued below the success threshold despite larger compute budgets, leading DEC-MEM to attribute the failure to the hypothesis itself (HYP).

The reverse failure mode receives insufficient attention: an invalid hypothesis may be repeatedly preserved because failures are incorrectly attributed to implementation, compute, or measurement. The appendix acknowledges delayed rejection, but provides no explicit failure-loop case study or hypothesis-level wasted-budget analysis. This concern is supported by the attribution results: only 68 of 160 HYP-labelled failures are correctly identified. That is episode-level evidence, however, and does not establish persistent looping. The reported examples predominantly illustrate successful rescue.

Authors:

We respectfully clarify that the episode-level attribution errors that the reviewer points out (a negative episode whose true cause is HYP but which DEC-MEM attributes to HNS, CPT or MSN) do not compound into persistent loops, because our DEC-MEM is self-correcting:

  1. Every negative outcome of an episode attributed to HYP, CPT, or MSN adds evidence to the hypothesis, and the frontier score (Sfrontier(m)S_{frontier}(m) defined in Section 4.4) decays with accumulated evidence. Each such re-test therefore lowers the hypothesis's own priority for re-visitation.
  2. Our counterfactual probing stage performs probing experiments in addition to the original R&D episode to maximally ensure the correct failure attribution. If the probing experiments show that the episode failure cause is the hypothesis itself (i.e., HYP), the hypothesis is refuted, and its confidence ss will be reduced. The discard rule (Sec. 4.3) requires both confidence below the threshold and negative evidence from at least ν\nu distinct experimental settings.

Reviewer: clarify whether any per-hypothesis retry or cumulative-budget limit prevents repeated external attribution?

We now implement a limit on repeated external attributions of the same hypothesis (i.e., attributing episode failure cause outside of its hypothesis): After two consecutive external attributions of the same hypothesis with no positive outcome in between, the next negative outcome is written as HYP, so the hypothesis's validity confidence ss starts to decrease. This experiment uses a separate set of 10 seeds from Table 1, with DEC-MEM rerun on those seeds as the baseline:

DeepSeek-v4-pro: DEC-MEM → + limitGPT-OSS-120B: DEC-MEM → + limit
FNLR ↓0.384 → 0.3700.168 → 0.216
FNF1 ↑0.398 → 0.4950.461 → 0.519
tests per invalid hypothesis ↓1.72 → 1.581.85 → 1.76

We observe that the limit can make the failure cause attribution more accurate in some settings (e.g., DeepSeek-v4-pro FNF1 +0.097, paired 95% CI [+0.034, +0.162]) and re-test invalid hypotheses less often while maintain DEC-MEM's effectiveness w.r.t. FNLR. In other words, setting appropriate limit to repeated external attributions does not cause DEC-MEM to abandon much more valid hypotheses. We have added this analysis to the paper, and we sincerely thank the reviewer for the valuable suggestion on improving DEC-MEM.

Reviewer_S6zVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work proposes DEC-MEM, a memory mechanism for deconfounding episode outcomes, attempting to diagnose the problem when a trajectory ends in failure. It then employs a write policy to store the hypothesis of the ML engineering task and the failure reason to improve future machine learning (ML) research and development.

Critical strengths

This work attempts to address the issue that the reason for a failed trajectory may be confounded and may not be attributable to the hypothesis itself. This research question and motivation are interesting and make sense.

Critical weaknesses
  1. The motivation for emphasizing memory is not clear, and the definition of memory is unclear in this paper, especially in the preliminaries. Diagnosing why an experiment fails is well-motivated and important, but what role memory plays is unclear to me.

  2. In Section 3, the work introduces two novel metrics, FNF1 and FNLR, but this part is hard to follow. In particular, how does “FNF1 evaluate memory writing”? Its definition in Definition 1 does not explicitly relate to memory writing.

Overall, the paper writing is hard to understand.

  1. For failure diagnosis, this work gives four predefined classes, but the diagnostic rules seem task-specific, and their generalizability is unclear. How is the precision of the failure-mode checks ensured?

  2. In addition, this work highlights the memory mechanism, but does its actual implementation mainly consist of storing hypotheses and failure-analysis examples in a memory table with confidence levels? The technical novelty and difficulty appear limited. How are these confidence levels determined, and how is their reliability ensured?

  3. How are the memory items retrieved? Are they relevant only to the same task, or can they generalize across different tasks? If the memory is mainly used within the same task, what is the difference from a traditional ReAct agent that includes previous failure analysis in its context?

Questions for authors

What do {HYP, HNS, CPT, MSN} stand for? It would be better to give an explicit explanation before their first appearance in line 124. and see Weaknesses

5 comments
AuthorsOpenReview ↗

Authors Response 1/4

We sincerely thank the reviewer for recognizing the importance and motivation of the research. We address each concern and question below and have revised the paper accordingly, with changes highlighted in blue in the revised manuscript.

W1. The definition and role of memory

Reviewer: The motivation for emphasizing memory is not clear, and the definition of memory is unclear in this paper, especially in the preliminaries.

Authors:

Definition. Thank you for this suggestion. The original submission introduced the memory mechanism at the end of the first paragraph. We now provide a more rigorous definition in Section 2: memory is a store MtM_t of an agent's past experiences in text form, carried across R&D episodes [1]. The agent can write to it (Mt=Write(Mt−1,et)M_t = \mathrm{Write}(M_{t-1}, e_t)) and retrieve from it (Rt+1=Retrieve(Mt,qt+1)R_{t+1} = \mathrm{Retrieve}(M_t, q_{t+1})) to inform its future behavior.

Motivation. Memory plays a key role because it is an important and essential channel through which agents can self-improve across episodes. For example, past episode outcomes influence which hypotheses are tested in later episodes.

Reviewer: Diagnosing why an experiment fails is well-motivated and important, but what role memory plays is unclear to me.

Authors:

Memory is not used to diagnose why an experiment fails. Instead, memory carries diagnoses to guide future episodes. Our paper identifies a failure mode in existing memory mechanisms: they treat episodes with negative outcomes as evidence that the hypotheses examined in those episodes are invalid. Careful failure diagnosis is therefore needed to disentangle confounding factors and prevent memory from harming performance in future episodes. Additionally, our method is not only about identifying the true cause of an episode’s negative outcome; it is also about designing the memory architecture and how memories are written, updated, and retrieved across episodes to facilitate optimal performance in ML agent systems.

W2. How FNF1 evaluates memory writing

Reviewer: In Section 3, the work introduces two novel metrics, FNF1 and FNLR, but this part is hard to follow. In particular, how does “FNF1 evaluate memory writing”? Its definition in Definition 1 does not explicitly relate to memory writing. Overall, the paper writing is hard to understand.

Authors:

FNF1 measures the F1 score for how accurately a memory mechanism identifies the cause of failure in episodes with negative outcomes. Because these identified failure causes are written directly into memory, the accuracy of this identification directly determines the quality of the recorded memory. Therefore, we state that FNF1 evaluates memory writing. We have included this rationale in Section 3 for further clarification.

[1] Park, Joon Sung, et al. "Generative agents: Interactive simulacra of human behavior." Proceedings of the 36th annual acm symposium on user interface software and technology. 2023.

AuthorsOpenReview ↗

Authors Response 2/4

W3. Generalizability and precision of failure diagnosis

Reviewer: For failure diagnosis, this work gives four predefined classes, but the diagnostic rules seem task-specific, and their generalizability is unclear.

Authors:

In this work, we focus only on machine learning engineer agent systems, which have a highly homogeneous task: solving a machine learning problem by proposing a hypothesis, implementing the code, running experiments, and measuring performance. The four predefined failure causes, i.e., HYP (hypothesis), HNS (harnessing/code implementation and execution), CPT (compute budget), and MSN (performance measurement), each correspond to a stage of the episode. This structure helps the memory mechanism accurately locate failure causes at each stage and simplifies retrieval for agents, since these predefined classes standardize the written memory format. Although the tasks are similar, each task produces different evidence for each cause class, allowing precise identification of the true causes.

Reviewer: How is the precision of the failure-mode checks ensured?

Authors:

To precisely identify failure causes in episodes with negative outcomes, we set clear identification criteria at both stages of causal-aware attribution. During execution log examination, evidence detectors only label a possible failure cause if they detect keywords or patterns in the log that reveal a specific failure. In the counterfactual probing stage, we define a success standard for each failure cause type. For HNS (harness), we substitute the original code with a verified correct implementation and check whether this change eliminates the negative outcome. For CPT (compute), we vary the compute budget during code execution and examine performance trends to determine if compute starvation caused the failure. For MSN (measurement), we re-score the episode’s outputs using an alternative metric aligned with the task objective. If the episode receives a positive outcome under this new metric, it indicates that inappropriate performance measurement likely caused the observed negative outcome.

Experimentally, we achieve high precision in failure diagnosis, as shown in Appendix I.2, Tables 14–15. For example, we identify Harness (HNS) failures with a precision of 0.745 and a recall of 0.935, and CPT failures with a precision of 0.728. DEC-MEM's FNF1 score exceeds all baselines on every LLM backbone (paired Wilcoxon p<=0.002p <= 0.002 for each comparison).

Q1. Definitions of HYP, HNS, CPT, and MSN

Reviewer: What do {HYP, HNS, CPT, MSN} stand for? It would be better to give an explicit explanation before their first appearance in line 124.

Authors:

Thank you for pointing this out, and we apologize for the oversight. The acronyms were defined in Definition 1 around line 128 and discussed in Remark 1. We agree that they should be defined before Definition 1 and have made this change in the updated manuscript:

AcronymMeaningThe episode outcome is negative because…
HYPHypothesisthe idea (e.g., model design) does not help under the tested conditions
HNSHarnessthe implementation or execution is faulty (e.g. a bug)
CPTComputethe compute allocation is too small for the hypothesis effect to appear
MSNMeasurementan inappropriate evaluation metric hides the hypothesis effect

Only HYP is evidence against the hypothesis itself.

AuthorsOpenReview ↗

Authors Response 3/4

W4. Technical novelty and confidence estimates

Reviewer: In addition, this work highlights the memory mechanism, but does its actual implementation mainly consist of storing hypotheses and failure-analysis examples in a memory table with confidence levels? The technical novelty and difficulty appear limited.

Authors:

We would like to clarify that our contribution goes well beyond storing hypotheses and failure-analysis examples in a memory table with confidence levels. First, we identify the phenomenon of false-negative lock-in in ML agent systems and formulate the problem of memory under confounded R&D outcomes. Second, we introduce the DEC-MEM method, which consists of three main innovations: (1) a causal-aware evidence attribution module to accurately deconfound failure causes for episodes with negative outcomes; (2) a memory item architecture and cause-specific write policy for handling confounded outcomes; and (3) a retrieval policy for deciding when and how to use cached memory to support more informed and effective ML R&D in future episodes.

Reviewer: How are these confidence levels determined, and how is their reliability ensured?

Authors:

How confidence ss is determined. ss is DEC-MEM's estimated probability that hypothesis hh is valid: s=1s = 1 means certainly valid and s=0s = 0 certainly invalid. It starts at 0.5 (valid and invalid equally likely) and is updated by Bayes style rule each time hh is tested with outcome oo: s←sP(o∣h valid)sP(o∣h valid)+(1−s)P(o∣h invalid).s \leftarrow \frac{s P(o \mid h \text{ valid})}{s P(o \mid h \text{ valid}) + (1-s) P(o \mid h \text{ invalid})}. A positive outcome raises ss. A negative outcome lowers ss only when DEC-MEM attributes it to HYP, and the more confident that attribution, the larger the drop. A negative outcome attributed to HNS, CPT or MSN leaves ss unchanged, since it carries no evidence about whether hh is valid.

How the reliability of ss is ensured. Reliable ss should satisfy the following: for all episodes where a failure cause attribution is made with confidence kk (e.g., 0.9), approximately kk proportion (i.e., 90%) of those episodes should have correct failure attribution. We measure this with the expected calibration error (ECE): we group episodes by DEC-MEM's confidence in its top cause and, within each group, measure the gap between the confidence and how often the top cause is actually the true cause among those episodes (0 means ss is perfectly reliable). On the negative episodes of the R&D streams in Table 1:

DeepSeek-v4-pro / GPT-OSS-120B / Qwen3.5-397B
Top cause is correct0.64 / 0.63 / 0.68 (always pick the most common cause: 0.52 / 0.45 / 0.46)
ECE0.24 / 0.23 / 0.22
ECE w/ temperature scaling0.13 / 0.08 / 0.07

We found that DEC-MEM's attribution is far more accurate than picking the most common cause across episodes, but can be slightly overconfident (ECE 0.22–0.24). To further improve this, we can perform temperature scaling, which softens the cause probabilities as P(z)1/TP(z)^{1/T} (renormalised), with T>1T>1. The ECE can be further reduced to as low as ~0.1. We thank the reviewer for motivating this analysis and we have added it to the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.