acceptodds
Under review as a conference paper at ICLR 2027

Epistemic Uncertainty for Test-Time Discovery

Abstract

Verifier-guided test-time search with large language models seeks one exceptional program within a small budget. TTT-DISCOVER trains the generating policy at test time with an entropic objective that emphasises the best sampled candidates, yet it can only credit programs the policy actually samples. We ask whether disagreement among policy adapters is a useful additional exploration signal. Uncertainty-Guided Test-Time Training (UG-TTT) maintains a small ensemble of low-rank adapters over a frozen base model, measures per-token disagreement as the mutual information between the next token and the ensemble member (an epistemic-uncertainty proxy, not a calibrated posterior estimate), and adds a rollout-level aggregate of it to the advantage as an exploration bonus. At a matched budget of 384 rollouts on six TTT-DISCOVER tasks, UG-TTT attains the higher maximum reward in five of the six seed-42 comparisons and a 0.03% lower one on the Erdos problem. Circle-packing differences are positive at all ˝ three seeds (with differing comparator configurations), outcomes on an autocorrelation inequality and a heuristic-contest task are mixed across seeds, and the single-cell denoising verifier’s error falls by 8.7% on Qwen3-8B and by 2.9% on a Qwen3-14B pair. In a one-run-per-arm component study on circle packing, the mutual-information bonus attains the highest maximum reward, and removing the adapter regulariser leaves disagreement nearly unchanged, indicating that diversified initialisation supplies the signal over six optimiser steps. Diversity depends on the window and the measure: at the end of training every seed-42 mathematical baseline has concentrated on one solution family while UG-TTT keeps at least three, and a rule-free structural measure favours UG-TTT in 9 of 12 run pairs over the late epochs and in 7 of 12 over the whole run. Ensemble disagreement is thus a practical exploration signal for budgeted test-time search; whether it beats simpler uncertainty bonuses, and whether diversity mediates its gains, remain open.

Primary area:
foundation or frontier models, including LLMs
Keywords:
test-time training, LLM-driven scientific discovery, reinforcement learning with verifiable rewards, uncertainty-guided exploration, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_fwtNOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes UG-TTT, which adds a LoRA policy ensemble, a token-disagreement bonus, and nuclear-norm regularization to TTT-DISCOVER (ICML 2026). The paper evaluates maximum verifier reward and program diversity under a 384-rollout budget on six tasks, with additional seed, model, and component studies.

Critical strengths
  1. The paper addresses a practically relevant question in verifier-guided program search.
  2. The appendices provide substantial implementation detail and openly report mixed results and limitations.
Critical weaknesses
  1. Limited insight and insufficient isolation of the proposed mechanism

W1 — Limited methodological insight. The method largely combines established ensemble disagreement and reward shaping with TTT-DISCOVER. Propositions 1–3 establish elementary component properties, not conditions linking the proposed proxy to useful exploration; the empirical analysis does not establish a compelling mechanism either (Section 2; Appendix E).

W2 — The disagreement bonus is not adequately isolated. No frozen-ensemble control separates adaptation from diversified initial sampling. In Table 3, MI and the entropy comparator also differ in vocabulary coverage and aggregation, preventing a clean assessment of the particular disagreement signal (Section 3.3; Limitations).

  1. Limited empirical support for a reliable practical advantage

W3 — Nonuniform and thin comparisons. CP26 seed 42 uses a degenerate five-member comparator, while AC1 seed 42 changes stopping only for UG-TTT. With at most three run pairs per task and mixed seed outcomes, the headline results do not establish a robust advantage under a consistent comparison protocol (Tables 1–2).

W4 — Practical benefit remains insufficiently established. Matching rollout counts does not match total compute, given the ensemble’s additional scoring and backward passes. The strongest denoising gains are measured on the repeatedly queried search verifier, without untouched evaluation (Section 3; Appendix B).

Questions for authors
  1. Regarding W1: Beyond assembling the components, what specific new insight about effective exploration is established, and which result supports it?
  2. Regarding W2: What evidence separates the MI bonus from diversified ensemble sampling and from a full-vocabulary entropy bonus with matched aggregation?
  3. Regarding W3: Which gains are reproducible against a standard single-adapter baseline when both arms use matched stopping and a consistent protocol across seeds?
  4. Regarding W4: Does the method retain an advantage under matched total compute? Do the selected denoisers improve performance on data never queried during search?
Reviewer_ZqUUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper builds on TTT-Discover (Learning to Discover at Test Time, Yuksekgonul et al., 2026), which searches (with branching through rollout noise and selection by verifier score) for programs at test-time, fine-tuning LoRA adapters across search paths (for a single task) using RL on the output token sequences (reasoning and program). This paper claims this method has issues of collapsing into a single solution mode in the program pool, and proposes UG-TTT, which uses an ensemble of five LoRA adapters for different rollouts, with exploration bonuses on disagreement among the adapters (mutual information between their next-token distributions) to incentivise diversity, and nuclear-norm regularization. Their experiments compare the scores across tasks with TTT-Discover, at a much smaller budget, with very limited evidence of gains, low significance, low effect sizes and ablations showing conflicting evidence that effects are due to the added methods as argued.

Critical strengths

Experimenting with diversity mechanisms in test-time discovery searches is a timely problem.

The results and arguments, including the negative results, are reported transparently.

Critical weaknesses

The main claim is not supported and is confounded. The experimental results don’t show clear evidence that there is a performance gain. There is no clear evidence that the added mechanisms are doing what they are supposed to do, e.g. no evidence links the bonus to either reward or diversity; six AdamW steps at 4e-5 will barely move the adapters; most of the disagreement signal comes from initialisation (Sec. 3.3). The paper is titled “Epistemic Uncertainty”, indicating the mechanism would be due to disagreement among adapted hypotheses, while an init-noise signal is a different method with a different name.

The paper doesn’t seem ready for publication as is, written as a sequence of numbers, caveats and self-corrections, with very limited, or even contradictory, experimental evidence. There are not enough necessary baseline comparisons, and the present results in fact indicate that simpler methods would have a similar effect, e.g. the component pilot shows the bonus within 0.001 of a plain token-entropy bonus. The model also includes LoRA dropout, a cheaper source of a similar kind of perturbation (the paper notes each member distribution is already an MC-dropout draw). It is not clarified, e.g. if simply more dropout would generate similar diversity. ThetaEvolve (Wang et al. 2025) is not cited nor used as a baseline, despite being already referenced and compared against in TTT-Discover. The ethics statement is unrelated to ethical concerns.

Questions for authors
  • Does a frozen five-adapter ensemble without training reach the same R_max as UG-TTT? This would probe whether it is doing something beyond ensemble sampling.
  • How does a K=1 baseline with matched compute compare with UG-TTT?
  • Does increased LoRA dropout get similar effects to the proposed method?
Reviewer_yxLqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper extends TTT-DISCOVER, a test-time RL method that trains an LLM policy to find the best verified program within a fixed budget. UG-TTT replaces its single LoRA adapter with an ensemble of five adapters and uses the mutual information (MI) between their next-token distributions to measure uncertainty. It then adds an aggregate of this MI over each rollout as an exploration bonus to the advantage. With 384 rollouts on six tasks, it achieves a higher maximum reward on 5/6 tasks for seed 42, although the improvements are mostly very small.

Critical strengths
  1. The idea is simple and cheap to add on top of TTT-DISCOVER, since all adapters share one frozen base model.
  2. The paper is honest about its negative and mixed results, potential confounds, and limitations.
Critical weaknesses

1. Limited novelty and unclear gains. The method mainly combines existing components on top of TTT-DISCOVER, while simpler uncertainty signals, such as average NLL, NLL variance, or top-K probability entropy, are not properly compared. The gain over token entropy is only 0.0010, and all results use only 384 rollouts, which is also different to the published setting (512 rollouts), making it unclear whether the improvement persists at larger budgets.

2. Experimental setup and robustness. The evaluation uses inconsistent seeds with seed 42 for all tasks, other seeds, e.g. 2, 3 only for CP26, AC1. Why not report mean/std over same three seeds for every task? Moreover, the reported gains are often smaller than seed variation, and most of the gain comes from denoising, which is evaluated only on the search verifier. Many design choices lack motivation or ablation, e.g. why top-7% (rather than, e.g., the midpoint of the cited 5-12% range), why 8 rollouts x 8 parents per step, and why K=5 when K=2 scored highest in the only sweep. The role of gamma in Eq. 4 is unclear where Remark 1 says beta-coupling is not load-bearing, so why keep it? Why not use U from Eq. 3 directly with a constant coefficient in Eq. 5?

3. Theory is not well connected to the method. The propositions are mostly textbook-level or restatements of prior work, and they do not explain why the MI bonus should improve exploration or maximum reward.

4. Presentation and clarity. The structure is hard to follow: Limitations come before Related Work, and the captions of Fig. 1 and Tab. 1 are long, number-heavy and repeat the main text. Key evidence is pushed to the appendices. Some methodological details are also insufficiently explained. For example: abstract use "TTT-DISCOVER" without defining it, and the name refers both to a method and a task suite ("six TTT-DISCOVER tasks"). L159 ("measured per token") is imprecise, since MI is computed per decoding step over the full next-token distribution. The split between thinking tokens and code (Sec. 2.1) is introduced, but its role in the method is not explained. The last sentence of the abstract (L33-34) leaves the central question open (whether MI beats simpler bonuses), which reads like an unfinished study rather than a result.

5. Reproducibility. The paper claims that code, logs, and scripts are publicly available, but no anonymized repository or supplementary material is provided. Given the many run-specific settings, these claims cannot be independently verified.

Questions for authors

See Weakness.

Reviewer_mTydOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether disagreement among language-model policy adapters improves verifier-guided test-time program search. UG-TTT maintains an ensemble of LoRA adapters over a frozen base model, computes token-level mutual information (MI) across member predictions, and incorporates an aggregated disagreement bonus into entropic policy advantages. The method additionally employs diversified adapter initialization and nuclear-norm regularization. Experiments cover six TTT-Discover tasks, with additional random seeds, a Qwen3-14B comparison, component ablations, and two diversity measures. Under a budget of 384 rollouts, UG-TTT achieves higher maximum verifier reward in five of six seed-42 comparisons, while results across additional seeds are mixed.

Critical strengths
  1. Meaningful problem formulation and exploration mechanism The paper addresses the challenge of discovering high-quality solutions under limited test-time search budgets, where reward optimization may lead to premature concentration on existing solution patterns. Incorporating ensemble disagreement into policy advantages provides a technically motivated approach to encouraging exploration beyond reward-driven exploitation.
  2. Broad methodological analysis and empirical investigation The paper evaluates six discovery tasks, includes additional seeds and a second base model, and provides component ablations. The analysis of adapter initialization and nuclear-norm regularization offers useful insights into the source and behavior of ensemble disagreement. The authors also report mixed and negative results, demonstrating reasonable experimental transparency.
Critical weaknesses
  1. Unclear attribution of improvements to uncertainty-guided test-time learning Diversified LoRA initialization introduces predictive disagreement before any test-time updates, making it difficult to distinguish the benefits of MI-guided learning from ensemble sampling alone. The absence of a frozen-ensemble control leaves the contribution of test-time adaptation unclear, particularly given the substantial performance gap already observed before the first update in AC1.
  2. Limited robustness and confounding factors in experimental comparisons Results are inconsistent across random seeds, particularly on AC1 and AHC039. Some comparisons also involve different baseline configurations or asymmetric streaming settings. Furthermore, the component study uses only one run per configuration, while comparisons against alternative uncertainty bonuses change both the uncertainty measure and aggregation strategy. The additional computational overhead of UG-TTT is also not evaluated under matched compute budgets.
Questions for authors
  1. Regarding the contribution of test-time adaptation Can the authors provide a frozen-ensemble search control using the same diversified adapter initialization and rollout budget? Such a comparison would help determine whether the observed improvements arise from MI-guided policy updates or from the increased exploration coverage provided by multiple initially distinct adapters.
  2. Regarding the effectiveness of MI-based exploration Can the authors clarify whether the advantage of MI over simpler uncertainty bonuses persists when using the same aggregation strategy and evaluation settings? Given the relatively small differences observed in the component study and the mixed results across seeds, additional controlled evidence would materially strengthen the claim that MI provides a distinct exploration benefit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.