acceptodds
Under review as a conference paper at ICLR 2027

Test-Time Adaptation of Reasoning Strategies with Bayesian Nonparametric Memory

Abstract

While modern large language models (LLMs) have been trained to reason through verbalized chains-of-thought, the generation cost grows substantially due to suboptimal paths to reach the final answer. Furthermore, as new insights are discovered while observing various input queries (e.g. through self-reflection), limited mechanisms exist for carrying forward these findings to be applied to subsequent problems. One can view the list of such strategies or behaviors as a growing cheatsheet, with elements retrieved from this memory module at inference-time. In this work, we consider cheatsheets, with learned clusters of behaviors. We introduce a Hierarchical Dirichlet Process Gaussian Mixture Model (HDP-GMM) over behavior embeddings, which shares components across domains while allowing domain-specific mixing weights, and uses the posterior predictive to retrieve relevant behaviors for a query; we call this a . This mechanism allows for cheap adaptation in an online test-time training (TTT) setting, softly updating the mixture's sufficient statistics following each sample and enabling the creation of new components when the synthesized behaviors are sufficiently novel. We demonstrate that Bayesian Cheatsheet achieves clear performance gains relative to existing memory modules across reasoning benchmarks such as AIME'25, Omni-MATH, and PhysReason, even in the cold-start setting. We show that the Bayesian Cheatsheet is an adaptively reorganizing memory module, as behaviors can be re-assigned to components through a single step of collapsed Gibbs sampling. Our findings highlight the value of Bayesian-inspired memory modules for effective test-time adaptation and the role of structure in metacognitive reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
test-time training, llm reasoning, structured memory, bayesian nonparametrics

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_aLzhOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper suggests bayesian cheatsheet, which self-reflects on the problem solving behavior, then cluster using HDP-GMM. Afterwards, it searches by posterior predictive, renewing sufficient statistics in online manner.

Critical strengths
  • Interesting approach to adapt the external memory under probabilistic prior.
Critical weaknesses
  • The core motivating claims (that probabilistic growth avoids singleton or overloaded clusters, and that the hierarchy yields domain-aware yet cross-domain-shared components) are never directly tested: no cluster-size statistics, no HDP vs. flat DP-GMM ablation, and only LLM-generated labels showing every component tied to a single domain.

  • The results could be very sensitive to hyper-parameter settings: τnew\tau_{new}, ε, γ, αd\alpha_d, PCA dimension, etc.

Questions for authors

Can the authors provide sensitivity analysis to hyper-parameters?

1 comment

We thank the reviewer for their review and feedback; we appreciate that the reviewer found our approach to be “interesting.” We address the concerns and questions raised in the review below:

The core motivating claims (that probabilistic growth avoids singleton or overloaded clusters, and that the hierarchy yields domain-aware yet cross-domain-shared components) are never directly tested: no cluster-size statistics, no HDP vs. flat DP-GMM ablation

Thank you for pointing this out! Regarding singleton clusters, we find that Bayesian Cheatsheet never forms singleton clusters (either in the offline fitting or with online adaptation), while the k-means experiments (both hard and soft versions) have a small number of singletons (< 5). More crucially, we find that the components with BC-TTT have no more than ~2k behaviors (in the largest component) and no fewer than 110, whereas the k-means components have as many as ~19k behaviors and as few as 1 (the aforementioned singletons). Our compute cluster is unexpectedly down at the moment, but we would be happy to include a more thorough analysis on this point as soon as possible (once the cluster becomes available again), and in our camera-ready revision!

We also include an ablation comparing the HDP-GMM (hierarchical) structure against a flat DP-GMM (domain-pooled) in Appendix F.3. Our findings show that the hierarchical structure achieves clear performance gains on all three benchmarks, suggesting that domain-aware structure and retrieval are effective.

The results could be very sensitive to hyper-parameter settings: , ε, γ, , PCA dimension, etc. Can the authors provide sensitivity analysis to hyper-parameters?

We include a series of ablations in Appendix F, entailing the number of behaviors retrieved (k’; F.1), the refit interval (g; F.2), the component creation mechanism (threshold-based inspired by DP-means, Bernoulli-based sampling; F.4), the inclusion of the KL-based component merging mechanism with ϵ\epsilon threshold (with and without KL-merge; F.4), the novelty threshold (τnew\tau_{new}; F.5), and the PCA dimension (F.6).


We hope that these responses have addressed your concerns; if so, we respectfully ask that you reconsider and increase your score. We would also be happy to address any further concerns that you may have. Thank you for your time and consideration!

Reviewer_mTxBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper organizes a cheatsheet of natural-language reasoning behaviors with a hierarchical Dirichlet process Gaussian mixture over behavior embeddings. Components are shared across domains with domain-specific mixing weights, retrieval scores components by posterior predictive density and returns each component's medoid, and at test time newly mined behaviors are absorbed through conjugate sufficient-statistic updates, novelty-triggered component creation, and periodic collapsed-Gibbs refits. With Qwen3-32B and Granite-4.2-30B, the method is compared against flat retrieval, online hard/soft k-means, Dynamic Cheatsheet, and ACE on AIME'25, Omni-MATH (hard subset), and PhysReason, including a cold-start setting.

Critical strengths
  1. The formulation is novel and principled. Replacing LLM-driven cheatsheet curation with a nonparametric mixture gives a clean retrieve / update / refit abstraction in which k-means, soft k-means, and the HDP variant are instances of the same online procedure, and conjugacy makes test-time updates cheap and deterministic.

  2. The comparison is designed fairly. All methods share the same behavior corpus, retrieval budget, and backbones, the k-means baselines are built as equally capable online methods, and the cold-start experiment removes the dependence on the offline corpus.

Critical weaknesses
  1. The experiments are not rigorous enough to support the reported gains. No variance or confidence intervals are given despite three seeds, and on AIME'25 a single trial moves the score by more than one point, so most headline gaps in Table 1 and Figure 1 lie within a handful of trials. The refit interval and retrieval budget are selected directly on the test benchmarks (Tables 5 and 6), and varying the refit interval alone shifts Omni-MATH by an amount comparable to the gain over ACE. The grading protocol for Omni-MATH and PhysReason and the composition of AIME'25 are not stated.

  2. The benefit of the Bayesian machinery is never isolated. The three advertised mechanisms (refit, novelty-driven growth, cross-domain sharing) show no clear positive effect: refit is worth almost nothing in Table 6 and hurts on Granite in Table 1, novelty triggers only a few times per benchmark, and the hierarchy reduces to two domains with each benchmark in one of them, with no flat DP-GMM ablation. The online soft k-means baseline already captures most of the improvement. Since BC retrieves from a few dozen medoids while the flat baselines retrieve from the full noisy corpus, a random-medoid control is needed to separate structure from medoid quality.

Questions for authors
  1. Please report per-seed standard deviations, state the grading protocol and the AIME'25 composition, and re-select the refit interval and k' on a held-out set. Do the conclusions of Tables 1 and 6 still hold?

  2. Please report the entropy of the responsibilities and the distribution of the novelty score during online updates, and how the Student-t log densities are computed numerically. In 128 dimensions these quantities are typically near one-hot, which would make the soft update equivalent to hard k-means. A flat DP-GMM ablation and a random-medoid control would resolve whether the gain comes from the hierarchy or from medoid quality.

2 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1/2)

We thank the reviewer for taking the time to review our paper and for their thorough feedback. We appreciate that the reviewer finds our method’s formulation to be “novel and principled” with the “clean retrieve / update / refit abstraction” and is appreciative of our experimental design with strong baselines. We address the concerns and questions raised in the review through our responses below:

No variance or confidence intervals are given despite three seeds, and on AIME'25 a single trial moves the score by more than one point, so most headline gaps in Table 1 and Figure 1 lie within a handful of trials.

In our revised version, we have included the standard deviations across 3 seeds for Tables 1, 2, and 4 (the main results). It is true that AIME’25 is a small dataset (30 samples), yet it has been used as a prominent benchmark in language model reasoning due to being fairly challenging for small to medium-size language models. As such, consistently improving by even a single question is quite non-trivial. We also note that the PhysReason and Omni-MATH datasets, which are larger, have noticeably smaller standard deviations. Accordingly, we suggest that our results are indeed statistically significant.

The refit interval and retrieval budget are selected directly on the test benchmarks (Tables 5 and 6), and varying the refit interval alone shifts Omni-MATH by an amount comparable to the gain over ACE.

… and re-select the refit interval and k' on a held-out set. Do the conclusions of Tables 1 and 6 still hold?

As per the reviewer’s request, we ran ablations of the number of behaviors to be retrieved (k’) and the refit interval (g) selected based on a held-out set of problems from Omni-MATH. Specifically, as our test set consists of problems with a difficulty score ≥ 7.0, our held-out set is an equal-size subset of the remaining problems (difficulty < 7.0), of 916 samples. This ended up yielding the same parameters chosen previously, k’ = 5 and g = 50. Unfortunately, our compute cluster is facing an unexpected outage which prevents us from adding these tables at the moment, but we intend to add them as soon as possible, once resolved!

The grading protocol for Omni-MATH and PhysReason and the composition of AIME'25 are not stated.

We use the Omni-Judge grader introduced with the Omni-MATH dataset, and PhysReason uses the answer-level grader introduced in their work (PSAS-A). Regarding AIME’25, we adopt it as it is one of the most popular mathematical reasoning benchmarks in language model evaluation over the last year; it consists of 30 (fairly) challenging problems across a range of topics, including number theory, combinatorics, geometry, and algebra. If the reviewer has further questions on the nature of these problems, we would be happy to discuss them!

The three advertised mechanisms (refit, novelty-driven growth, cross-domain sharing) show no clear positive effect: refit is worth almost nothing in Table 6 and hurts on Granite in Table 1, novelty triggers only a few times per benchmark, and the hierarchy reduces to two domains with each benchmark in one of them, with no flat DP-GMM ablation. The online soft k-means baseline already captures most of the improvement.

Novelty-driven growth is the key mechanism driving the gains of “Online, no refit” against “Static” in Table 1, across both models and all benchmarks. It is natural for novelty to only trigger a few times when there is an offline-mined memory -- in qualitative terms, new clusters should only be created when the proposed new behaviors are different from existing ones. In the cold-start setting, there is no pre-initialized memory, so novelty-driven component creation occurs at nearly every sample early on, and then eventually decreases for longer benchmarks like Omni-MATH. The refit mechanism is optional, and as noted above, g = 50 achieves the strongest performance; it is not a required element of Bayesian Cheatsheet, although it allows for reorganization of the behavior mixture.

There is also a noticeable difference in the behavior of component creation (specifically, in the number of components spawned) when considering our threshold-based creation scheme (inspired by DP-means) ablated against a Bernoulli-based sampling procedure, which we include in Appendix F.4. Accordingly, we find that the threshold-based method with component merging based on KL achieves the best performance on Omni-MATH, although the other configurations are also competitive.

We have added an ablation comparing the hierarchical structure (HDP-GMM) against a flat DP-GMM (domain-pooled) in Appendix F.3, showing that the hierarchy (domain-awareness and cross-domain sharing) does yield clearly better performance on all three benchmarks.

AuthorsOpenReview ↗

Official Comment by Authors (Part 2/2)

Since BC retrieves from a few dozen medoids while the flat baselines retrieve from the full noisy corpus, a random-medoid control is needed to separate structure from medoid quality.

We have included a random-medoid selection experiment in the table below; as this method benefits from the HDP-GMM-structured memory that we introduce through the Bayesian Cheatsheet method, we label this method “BC-random.” Specifically, we see this as an ablation of the behavior selection mechanism, between 1. k’’ random medoids (which we term BC-random), 2. top-k’ nearest medoids (BC-centroids), and 3. posterior predictive (BC-PP). We indeed find that the random medoid selection performs worse than more careful retrieval methods in our BC framework, which further highlights the value of a native, probabilistic scheme like the posterior predictive as an effective instantiation of the retrieve module.

MethodQwen3-32B AIME'25Qwen3-32B OMQwen3-32B PRGranite-4.2-30B AIME'25Granite-4.2-30B OMGranite-4.2-30B PR
Base model71.1 ± 4.318.1 ± 0.716.4 ± 1.173.3 ± 3.424.8 ± 0.616.4 ± 1.1
Random behavior selection72.2 ± 3.818.9 ± 0.718.6 ± 1.373.3 ± 3.427.5 ± 0.816.4 ± 1.2
Direct behavior retrieval72.2 ± 3.418.6 ± 0.623.5 ± 1.374.4 ± 4.128.2 ± 0.619.5 ± 1.1
Metacognitive Reuse, k′=575.6 ± 3.017.8 ± 0.721.7 ± 1.177.8 ± 3.328.4 ± 0.922.6 ± 1.4
BC-random (ours)72.2 ± 3.618.3 ± 0.923.9 ± 1.277.8 ± 3.928.1 ± 1.121.7 ± 1.5
BC-centroids (ours)74.4 ± 4.220.5 ± 0.824.8 ± 1.381.1 ± 3.129.1 ± 0.723.9 ± 1.4
BC-PP (ours)77.8 ± 3.522.3 ± 0.827.0 ± 1.380.0 ± 2.931.6 ± 0.723.9 ± 1.2

Please report the entropy of the responsibilities and the distribution of the novelty score during online updates, and how the Student-t log densities are computed numerically. In 128 dimensions these quantities are typically near one-hot, which would make the soft update equivalent to hard k-means.

This is a great question, and we would like to separate the update operation from the retrieve operation. Indeed, folding a new behavior discovered at test time into the mixture is often one-hot, closer to hard assignment. However, the retrieve operation is not one-hot, and when, for example, 2-3 components share the mass, a given behavior can be selected by multiple components. We chose to allow soft assignment and use a threshold-based component creation mechanism based on the success of soft k-means and the DP-means work, respectively. The Student-t log densities depend on Mahalanobis distance and log-determinant, so we compute this in practice with a cached Cholesky solve with log-sum-exp normalization, which we find matches scipy.stats.multivariate_t within 5×10−135\times10^{-13}.

We would be glad to include a further analysis in the camera-ready version, given the aforementioned cluster outage that we are currently facing.


We hope that these responses have addressed your questions and concerns; if so, we would like to respectfully ask you to further your support of our work. We would also be happy to address any further concerns that you may have. Thank you very much for your time and consideration!

Reviewer_YMbFOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents the extension of LLMs with a memory extension through structured Bayesian Cheatsheets, where textually summarized strategies (behaviors) are clustered through a Hierarchical Dirichlet Process Gaussian Mixture Model (HDP-GMM) of behavior embeddings. The Bayesian cheatsheet uses this clustering for efficient selection in inference and is extended at test time for formation of new behaviors. This memory extension and strategy leads to strong performance gains on hard reasoning benchmarks like AIME25, Omni-MATH, PhysReason.

Critical strengths

The technique strongly improves benchmark performance. It provides a clear angle on continual learning, with memory extension at test time, while allowing this memory to be interpretable as all is in natural language. The clustering in embedding space allows of a scalable architecture going beyond standard architectures and graphs. They also tweak their baselines (k-means) to a non-trivial degree, really proving the strength of their method with adaptive GMMs.

Critical weaknesses

It would be great to measure the statistical significance of their results more thoroughly. The method introduces quite some hyperparameters, with unclear origin of exact values (e.g. refitting intervals).

Questions for authors

Since this method relies on an embedding model and subsequent PCA, it would be interesting to see the impact of the dimension on results here. Could you re-run with a different set of dimensions? It would be interesting to see if "only a handful directions" define the behavior space meaningfully. Similarly a t-SNE plot of behaviors would be interesting. While domain specific pools might be desirable for explainability reasons, could you ablate "domain-free" modeling - letting the behavior space alone define the partitioning?

1 comment

We thank the reviewer for taking the time to review our paper, and for their insightful feedback. We appreciate that the reviewer finds that our method “provides a clear angle on continual learning,” notes our performance gains, and values the strengthened k-means baselines adapted to the test-time training setting. We address the concerns and questions raised in the review through our responses below:

It would be great to measure the statistical significance of their results more thoroughly. The method introduces quite some hyperparameters, with unclear origin of exact values (e.g. refitting intervals).

Regarding hyperparameters, in our revision, we have included ablations in Appendix F, studying different values of k’ (number of behaviors retrieved), the refit interval g, the novelty threshold, the PCA dimension, the component creation mechanism, the inclusion of merging components based on KL, as well as the role of the hierarchy against a flat (domain-pooled) structure. Different hyperparameters may be suitable for different datasets (for example, g = 50 cannot be used on AIME’25, which has 30 problems), and given our work centers around test-time training, it is appropriate for the method to be adapted accordingly. To the former point, we include the standard deviations across 3 seeds for Tables 1, 2, and 4 (the main results).

Since this method relies on an embedding model and subsequent PCA, it would be interesting to see the impact of the dimension on results here. Could you re-run with a different set of dimensions? It would be interesting to see if "only a handful directions" define the behavior space meaningfully. Similarly a t-SNE plot of behaviors would be interesting.

In our revision, we have included an ablation of the PCA dimension in Appendix F.6, spanning dim = 64, 96, 128 (default for the main results), 192, and 256. We find that the dimension of 96 is actually slightly better than 128 on AIME’25 and PhysReason and slightly worse on Omni-MATH. Our compute cluster is currently facing an unexpected outage, but we have generated the t-SNE plots as requested (for each PCA dimension analyzed above) and intend to include them in a revision as soon as possible. Specifically, we find that for all dimensions tested, we see a separation between the math and physics behaviors (while there is some overlap with components/behaviors being shared cross-domain), with new behaviors discovered at test-time often lying close, but not directly aligned to the offline-mined behaviors.

While domain specific pools might be desirable for explainability reasons, could you ablate "domain-free" modeling - letting the behavior space alone define the partitioning?

As noted above, we have added an ablation of the hierarchical version against a flat (domain-pooled) structure in Appendix F.3. Our findings show that even empirically (beyond human explainability and control reasons), the HDP-GMM is preferable over the flat DP-GMM version on all three benchmarks.

Comments on the paper

1 comment

We thank the reviewers again for taking the time to review our work, and for their valuable feedback and consideration. We have updated our manuscript, incorporating their feedback; we express the revised text in navy-colored font for the purpose of outlining the changes made in this discussion period. We address and provide further clarifications on common points raised in the reviews, and revisions made:

Hyperparameter tuning and selection: The reviewers noted the presence of several hyperparameters, which require ablation and careful selection. We add an extended Appendix F, focusing on further ablations: this includes a comparison of hierarchical structure (HDP-GMM) against a flat, domain-pooled setup (DP-GMM), requested by all three reviewers (YMbF, mTxB, and aLzh); threshold-based vs. sampling-based methods of component creation; ablating the novelty threshold; and analysis of the PCA dimension on downstream performance, as requested by reviewers YMbF and aLzh. These ablations provide supporting evidence of the design choices made regarding the hierarchical structure and component creation scheme.

The revision also substantiates our choice of threshold-based selection inspired by the DP-means method, suggesting that deterministic component creation can be valuable rather than probabilistic methods, as shown through our ablations in Appendix F.4.

Statistical Significance: Reviewers YMbF and mTxB noted measuring the statistical significance of our findings as part of their concerns. Accordingly, as our key results are an average over three seeds, in our revision, we include standard deviations for all major results (see Tables 1, 2, and 4), which confirms that our findings are indeed significant.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.