- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper suggests bayesian cheatsheet, which self-reflects on the problem solving behavior, then cluster using HDP-GMM. Afterwards, it searches by posterior predictive, renewing sufficient statistics in online manner.
Critical strengths
- Interesting approach to adapt the external memory under probabilistic prior.
Critical weaknesses
-
The core motivating claims (that probabilistic growth avoids singleton or overloaded clusters, and that the hierarchy yields domain-aware yet cross-domain-shared components) are never directly tested: no cluster-size statistics, no HDP vs. flat DP-GMM ablation, and only LLM-generated labels showing every component tied to a single domain.
-
The results could be very sensitive to hyper-parameter settings: , ε, γ, , PCA dimension, etc.
Questions for authors
Can the authors provide sensitivity analysis to hyper-parameters?
We thank the reviewer for their review and feedback; we appreciate that the reviewer found our approach to be “interesting.” We address the concerns and questions raised in the review below:
Thank you for pointing this out! Regarding singleton clusters, we find that Bayesian Cheatsheet never forms singleton clusters (either in the offline fitting or with online adaptation), while the k-means experiments (both hard and soft versions) have a small number of singletons (< 5). More crucially, we find that the components with BC-TTT have no more than ~2k behaviors (in the largest component) and no fewer than 110, whereas the k-means components have as many as ~19k behaviors and as few as 1 (the aforementioned singletons). Our compute cluster is unexpectedly down at the moment, but we would be happy to include a more thorough analysis on this point as soon as possible (once the cluster becomes available again), and in our camera-ready revision!
We also include an ablation comparing the HDP-GMM (hierarchical) structure against a flat DP-GMM (domain-pooled) in Appendix F.3. Our findings show that the hierarchical structure achieves clear performance gains on all three benchmarks, suggesting that domain-aware structure and retrieval are effective.
We include a series of ablations in Appendix F, entailing the number of behaviors retrieved (k’; F.1), the refit interval (g; F.2), the component creation mechanism (threshold-based inspired by DP-means, Bernoulli-based sampling; F.4), the inclusion of the KL-based component merging mechanism with ϵ threshold (with and without KL-merge; F.4), the novelty threshold (τnew; F.5), and the PCA dimension (F.6).
We hope that these responses have addressed your concerns; if so, we respectfully ask that you reconsider and increase your score. We would also be happy to address any further concerns that you may have. Thank you for your time and consideration!