- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies the use of memory to retain historical experience in sequential experimentation and causal querying. The central idea is to preserve queryable experimental records, thereby mitigating catastrophic forgetting that may arise when shared predictive models are updated sequentially, and to abstain when the available evidence does not adequately support a query. Through a series of small-scale experiments, the paper illustrates several roles of memory, such as directly estimating interventional probabilities or supplying evidence for inference and experiment selection.
Critical strengths
- The paper addresses a meaningful problem: when data from different experimental contexts arrive sequentially, a continually updated predictive model may fail to retain earlier experience. Maintaining historical records separately from the predictive model is a reasonable and intuitive design choice. The sequential learning experiments illustrate its potential to mitigate forgetting.
Critical weaknesses
- The main contribution is unclear. The analysis of the memory mechanism is relatively limited, while much of the subsequent discussion focuses on how to respond when memory is insufficient.
- The role of memory is not clearly established across experiments. Memory is used to directly estimate probabilities, provide evidence for inference or experiment selection, and supplement predictive models in different experiments. These settings involve fundamentally different roles of memory, making it difficult to understand what general conclusions about the proposed memory mechanism can be drawn from the experimental results.
Questions for authors
- The paper is not well-written. It introduces numerous technical details without sufficiently explaining their meanings or motivations. Both the methodology and experimental results are difficult to follow, and the main ideas are obscured by excessive technical details.
- The experimental section is poorly organized and difficult to follow. Experimental settings, methods, results, and analysis are often mixed together, making it difficult to identify the specific research questions addressed by each experiment and how the results support the paper's main claims.
- The final part of the appendix appears to contain an internal workflow or working-directory description unrelated to the paper’s technical content. The authors should check and ensure a clean submission.