- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper examines the viability of using a small discrete memory of tokens which is rewritten after steps to store relevant information. The paper diagnoses that the failure to succeed here is primarily a problem of knowing which facts to write rather than having enough space to write. Training a writer model to function better against the reader's loss yields a modest +7% improvement in keeping the critical facts over short horizons (but not long horizons). These modest gains are traced to credit assignment issues.
Critical strengths
The paper does careful experimentation to address and diagnose how a system (re)writing a discrete memory approach for a reader model fails and how to train it to perform better.
Critical weaknesses
There's a different approach to gaining learning signal which is likely significantly advantageous here. In particular, AggreVaTe https://arxiv.org/abs/1406.5979 diagnoses the problem and provides an approach to solve via rollout with a reference policy which seems valid here. There are followups like LOLS which can bring performance even higher.
Separately, the few% performance gain here seems near to the noise level in evaluations since observed noise is of a similar scale (table 2 says 3.3 / 15.6 / 12.2).
Another less serious issue is that portions of the language have formalized definitions of the literature but are used in an informal way here. The authors may want to examine:
L. Li, T. J. Walsh, and M. L. Littman. Towards a unified theory of state abstraction for MDPs. ISAIM 2006.
J. Subramanian, A. Sinha, R. Seraj, and A. Mahajan. Approximate information state for approximate planning and reinforcement learning in partially observed systems. JMLR 23(12):1–83, 2022
Write-time regret related: L. Peshkin, N. Meuleau, and L. P. Kaelbling. Learning policies with external memory. In Proceedings of the 16th International Conference on Machine Learning (ICML), pages 307–314, 1999 N. Meuleau, L. Peshkin, K.-E. Kim, and L. P. Kaelbling. Learning finite-state controllers for partially observable environments. In Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence (UAI), pages 427–436, 1999.
Questions for authors
No questions.