- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies false-negative hypothesis lock-in in LLM-based ML engineering agents, where valid hypotheses are prematurely discarded due to implementation errors, resource limitations, or inappropriate evaluation metrics. To address this, it proposes DEC-MEM, which combines counterfactual failure attribution, cause-specific memory writing, and uncertainty-aware retrieval. The paper also introduces lock-in evaluation metrics and demonstrates improvements over existing memory mechanisms across multiple LLMs and ML engineering tasks.
Critical strengths
-
The paper identifies false-negative hypothesis lock-in as an important limitation of existing memory mechanisms in autonomous ML engineering agents. DEC-MEM addresses this through cost-aware counterfactual probing, cause-specific memory writing, and frontier-aware retrieval, while introducing FNF1 and FNLR metrics for evaluation. Treating unsuccessful experiments as potentially confounded rather than direct evidence of hypothesis invalidity is a meaningful conceptual contribution beyond conventional outcome-based memory management.
-
DEC-MEM is evaluated against multiple memory baselines across three LLM backbones using fixed-episode, fixed-compute, ablation, and open-world settings. It consistently achieves substantial FNLR reductions, while ablation studies highlight the importance of frontier-aware retrieval. The breadth of comparisons, consistency of results, and component-level analyses provide strong evidence that DEC-MEM effectively addresses false-negative hypothesis lock-in in the evaluated settings.
Critical weaknesses
-
The main evaluation relies on a controlled benchmark with predefined hypotheses, cached outcomes, and synthetic failures, whose diagnostic structure closely matches DEC-MEM's attribution mechanism and may favor the proposed approach. While open-world experiments are included, they are limited in scope. Consequently, it remains unclear whether the reported gains generalize to realistic, open-ended ML engineering with naturally occurring and interacting failure conditions.
-
DEC-MEM relies on manually specified diagnostic likelihoods, restrictive failure-cause assumptions, and counterfactual probes with unverified causal validity. Moreover, the experiments do not clearly separate gains from accurate failure attribution versus increased hypothesis retention and exploration, nor do they compare against simpler alternatives under matched budgets. Without stronger evidence isolating the contribution of causal attribution, the methodological novelty and effectiveness of DEC-MEM remain insufficiently substantiated.
Questions for authors
-
How sensitive is DEC-MEM to misspecified detector likelihoods, prior distributions, and correlated failure causes? Can the authors provide attribution-calibration analyses and evaluate performance on unseen or simultaneous failure types not explicitly modeled in the current diagnostic framework?
-
To what extent are the reported improvements due to accurate failure-cause attribution rather than simply retaining and revisiting failed hypotheses? How does DEC-MEM compare with simpler retain-all, random-retry, or uncertainty-based retrieval strategies under identical computational and memory budgets?
-
Do the reported improvements persist in open-ended ML engineering environments with dynamically generated hypotheses and naturally occurring failures? Broader end-to-end evaluations, paired confidence intervals, and compute-normalized comparisons of final model performance would help substantiate the practical benefits.
7 comments
Authors Response 2/5
W2 / Q2. Contribution of accurate failure attribution
Reviewer: To what extent are the reported improvements due to accurate failure-cause attribution rather than simply retaining and revisiting failed hypotheses?
Authors:
Failure attribution determines which hypotheses memory refutes or retains. To test whether attribution accuracy matters, we keep the cause-specific write policy and frontier retrieval unchanged and vary only how failures are diagnosed (DeepSeek-v4-pro, 10 seeds):
| Failure diagnosis | FNF1 ↑ | FNLR ↓ | CPG ↑ |
|---|---|---|---|
| LLM diagnosis | 0.205 | 0.558 | 0.359 |
| DEC-MEM probes | 0.375 | 0.392 | 0.396 |
| Oracle diagnosis | 0.712 | 0.232 | 0.520 |
Replacing LLM diagnosis with DEC-MEM's probes reduces lock-in (FNLR: 0.558 to 0.392) and increases cumulative gain (CPG: 0.359 to 0.396). Oracle diagnosis improves both further. Thus, the same memory rules perform better when supplied with more accurate failure diagnoses.
W2 / Q2. Simpler alternatives under matched budgets
Reviewer: How does DEC-MEM compare with simpler retain-all, random-retry, or uncertainty-based retrieval strategies under identical computational and memory budgets?
Authors:
We compare DEC-MEM with simpler memory strategies under the same measured compute budget. Each stream receives the CPU-time budget required by a no-memory agent to complete 30 episodes, with every experiment and probe charged to that budget. We use 10 seeds per backbone; paired values below refer to DeepSeek-v4-pro / GPT-OSS-120B.
-
Forced random-retry tests whether retrying failed hypotheses at approximately DEC-MEM's rate is sufficient. In 35% of episodes, it overrides the agent's choice with a randomly selected failed hypothesis. Its FNLR is significantly higher than DEC-MEM's: 0.469 / 0.507 versus 0.336 / 0.241. The paired differences, computed as DEC-MEM minus random-retry, are −0.134 / −0.266, with both confidence intervals excluding zero.
-
Retain-all with uncertainty-based retrieval removes failure cause attribution and probing of DEC-MEM and keep all hypotheses from episodes with negative outcome. This strategy keeps lock-in low but has much lower failure-attribution accuracy (FNF1): approximately 0.08 versus 0.39–0.40 for DEC-MEM.
We also compare final best-task performance (macro), defined as the best macro-F1 gain over each task's baseline, averaged across tasks:
| Memory strategy | DeepSeek-v4-pro | GPT-OSS-120B |
|---|---|---|
| DEC-MEM | 0.070 | 0.069 |
| Retain-all | 0.049 | 0.061 |
| Forced random-retry | 0.062 | 0.061 |
DEC-MEM achieves the highest mean final best-task performance among these methods on both backbones, although the paired comparisons do not show statistically significant differences.
Authors Response 3/5
W2 / Q1. Sensitivity to detector likelihoods and priors
Reviewers: How sensitive is DEC-MEM to misspecified detector likelihoods, prior distributions?
Authors:
We test sensitivity to the probabilities used for failure attribution. The likelihoods describe how probable an observation is under each failure cause; the priors specify each cause's probability before observing evidence. Using DeepSeek-v4-pro, we modify these probabilities in five settings and compare the results with DEC-MEM's default setting:
| Attribution setting | FNLR ↓ | FNF1 ↑ | CPG ↑ |
|---|---|---|---|
| Default DEC-MEM (Table 1) | 0.392 | 0.375 | 0.396 |
| Multiply each likelihood by log-normal noise, σ = 0.5 | 0.351 | 0.309 | 0.452 |
| Multiply each likelihood by log-normal noise, σ = 1.0 | 0.333 | 0.340 | 0.342 |
| Swap the likelihoods assigned to CPT and MSN | 0.311 | 0.328 | 0.317 |
| Assign equal prior probabilities to all four causes | 0.441 | 0.323 | 0.345 |
| Bias the prior toward hypothesis invalidity: P(HYP) = 0.70 | 0.337 | 0.334 | 0.397 |
| No memory (reference) | 0.436 | 0.100 | 0.201 |
- Lock-in: None of the five changes produces a statistically significant difference in FNLR relative to default DEC-MEM; all paired 95% confidence intervals include zero.
- Attribution accuracy: FNF1 decreases from 0.375 to 0.309–0.340, but remains 0.21–0.24 above no memory, with all paired confidence intervals excluding zero.
Thus, these probability changes reduce diagnostic accuracy, but we detect no significant change in lock-in.
Authors Response 1/5
We sincerely thank the reviewer for recognizing the conceptual contribution and the breadth and consistency of the evaluation. We address each concern and question below with new results. We also revised our paper based on your suggestions, with changes highlighted in blue.
W1. Independence of the benchmark and DEC-MEM
Authors:
We would like to clarify that our controlled benchmark is independent from DEC-DEM and does not favor DEC-MEM. This is because, DEC-MEM does not have access to the benchmark's hidden fault information. The benchmark determines which harness fault is injected and how likely it is to leave a log trace. DEC-MEM is given neither the fault identity nor these probabilities. It diagnoses failures from observed logs and executed probes, using its own likelihoods defined for failure-cause categories rather than individual injected fault types. We have added to the manuscript to clarify this independence.