- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper presents a study comparing transformer model predictions with those of an informed Bayes model, that is, a model that knows both the source family and the prior over the data-generating processes from which the data are sampled. The data-generating processes considered in this paper include coin flips and several variants of first-order Markov processes.
Critical strengths
None
Critical weaknesses
The research question could be quite compelling with sharper framing. At present, however, the contribution feels limited in novelty. In particular, the central question of the paper appears to have been studied previously in [1] for example with other papers already cited. Although the setup there differs somewhat, it would be important to cite this work and clarify how the present paper extends or differs from it.
I also feel the writing would benefit from substantial revision before the paper is ready for submission. In its current form, it is difficult to follow, which makes it hard to properly evaluate the main claims.
A few specific points:
- The abstract is difficult to read and would benefit from a substantial rewrite. The paper main point only became clear to me once I reached the introduction, so it may help to bring some of that clarity forward into the abstract.
- The caption of Figure 2 includes the phrase “these estimates await numerical verification.”
Given these issues, I seriously doubt the authors thoroughly checked their paper before submission, which raises concerns about the overall quality and rigor of the work.
[1] Mikulik, Vladimir, et al. "Meta-trained agents implement bayes-optimal agents." Advances in neural information processing systems 33 (2020): 18691-18703.
Questions for authors
The caption of Figure 2 includes the phrase “these estimates await numerical verification.” This is confusing for what is presented as a completed paper. If the verification has since been done, the caption should be updated; if not, it would be worth completing before submission (Clarification).
Response to reviewer q9hP
We disagree that the issues identified justify the general conclusion that the work lacks rigor. Appendices D and E document an extensive validation protocol: checks against closed-form cases, independent integration refinements and sampling runs, explicit numerical-error thresholds, checks of probability normalization and causal predictions, and independent recomputation of reported statistics. Numerical error is assessed separately from variation across test sequences. Code, saved results and reproduction instructions are also available.
The Figure 2 caveat concerns two Bayesian estimates at t=128 for L=2 and L=3. Their verification has now been completed, and we will update the caption. The other three behavioral tests and the t=64 copying comparison in Table 1 do not depend on those two estimates.
Mikulik et al. is relevant prior work, and we will cite it. It studies LSTM meta-learners on conditionally independent prediction tasks and bandits. Our contribution concerns transformers on structured Markov sources with dependencies across contexts, together with four quantitative behavioral tests beyond matching prediction curves.
The review identifies no specific error in a derivation or experimental result. We would appreciate clarification of the technical grounds for the soundness assessment.