- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how shared failure banks in autonomous research systems may prematurely eliminate promising research directions. The authors propose evidence-graded negative memory that allows uncertain failures to be revisited.
Critical strengths
The problem is relevant, and the paper includes extensive theoretical and empirical analyses.
Critical weaknesses
-
I find both the problem formulation and the proposed method to be incremental. Noisy failures can lead to premature rejection, and retaining evidence and allowing reconsideration are natural remedies. Although the paper formalizes this risk, I am not convinced that it establishes a sufficiently substantial advance over existing uncertainty-aware memory and sequential decision principles.
-
The connection between the theory and the evaluated method is incomplete. The theory emphasizes the value of future research opportunities, but the experiments mainly use a fixed posterior threshold. They do not directly establish whether adapting the threshold to this future value provides additional benefit. It is unclear what new algorithmic insight is provided beyond standard uncertainty-aware decision-making.
-
The strongest gains come from controlled experiments with an algorithmic proposer and enforced locks. Results with LLM proposers are mixed: graded memory improves the enforced DeepSeek setting, but not the soft-lock setting, and performs worse on average than binary memory in the enforced GPT-4o setting. These results support a conditional effect, but leave unclear how to identify systems where the intervention will help.
-
The presentation makes the contribution difficult to assess. The organization is confusing, important findings are buried in excessive details, and the presentation of experiments and figures could be substantially improved.
Questions for authors
-
Which theoretical or empirical finding provides the main advance over existing uncertainty-aware memory and sequential decision principles? Please clarify what new knowledge this finding establishes beyond the intuitive risk of premature rejection.
-
Is future research value explicitly used in the evaluated method? If so, which analysis or experiment demonstrates its contribution? If not, how does this theoretical principle inform the fixed-threshold method used in the experiments?
-
Do the current experiments identify conditions under which graded memory should improve discovery outcomes in a new research system? How do these conditions explain the different results for DeepSeek and GPT-4o?